Benchmarks

This section describes how to start benchmark with the dAIEdge-VLab Python API.

Type of benchmark

Currently, there is two type of benchmarks possible :

Benchmark using random data (TYPE1).
Benchmark using your own dataset (TYPE2).
Benchmark for on device training experiment (TYPE3).

For TYPE1 and TYPE2, if random data is used, more metrics are gatherd and provided in the report.json. If your own dataset is used, you will get the raw output computed by your model in addition to a likely less detailed report.json. The metrics provided depends on the target implementation.

For TYPE3, the target will run an on device training experiment and extract some key metrics. The target get as input a model, a test dataset, a train dataset and some parameters for the on device training. The experiment provides the report.json with the metrics, a user.log and an error.log. The trained model is also provided as an artifact.

Summary of the artifacts

The following table help you understand what you can expect form the different benchmark types :

Artifact	TYPE1	TYPE2	TYPE3
`report.json`	YES	YES (less detailed)	YES (odt version)
`user.log`	YES	YES	YES
`error.log`	YES	YES	YES
`raw_output.bin`	NO	YES	NO
`trained_model.extension`	NO	NO	YES

The report.json contains the keys described here.
The report.json for on device training contains the keys described here.
The user.log and error.log files are simple text file.
The raw_output.bin contains the concatenated outputs for each input contained in the dataset provided.

Launch benchmark with the API

The type of benchmark run is implicit. If you do not provide a dataset, the TYPE1 is selected, otherwise the TYPE2 is launched.

Launch a TYPE1 benchmark :

from daiedge_vlab import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "rpi5", 
    runtime = "tflite", 
    model_path = "./path/to/my_model.tflite"
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print(result["report"])
print(result["user_log"])
print(result["error_log"])

Launch a TYPE2 benchmark :

To enable the TYPE2 benchmark you need to provide a dataset. There are several ways to do so, for more details follow the Provide a Dataset section.

from daiedge_vlab import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "rpi5", 
    runtime = "tflite", 
    model_path = "./path/to/my_model.tflite",
    dataset = "dataset_name" # or dataset file object
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print(result["report"])
print(result["user_log"])
print(result["error_log"])
print(result["raw_output"])

Launch a TYPE3 benchmark :

To enable the TYPE3 benchmark you need to provide a model, a test dataset, a train dataset and some parameters for the on device training. The parameters are provided with a OnDeviceTrainingConfig object.

from daiedge_vlab import dAIEdgeVLabAPI, OnDeviceTrainingConfig

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")
# Create the on device training config
config = OnDeviceTrainingConfig("./config.json")

# Start the benchmark with the specified target, runtime, dataset and model.
benchmark_id = api.startOdtBenchmark(
    "rpi5",
    "tflite",
    "./path/to/odt/capable/model.tflite",
    config
)

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id, save_path="./results")

# Use the result
print(result["report"])
print(result["user_log"])
print(result["error_log"])
print(result["model_output"])  # The trained model is provided as an artifact

The config.json file should contain the parameters for the on device training. Here is an example of such a file :

config.json

{
    "learning_parameters" : {
      "batch_size": 32,
      "epochs": 10,
      "loss_function": "sparse_categorical_crossentropy"
    },
    "input": {
      "name": "input",
      "shape": [28, 28, 1],
      "dtype": "float32",
      "train_file": "mnist_train_input.bin",
      "test_file": "mnist_test_input.bin"
    },
    "output": {
      "name": "output",
      "shape": [],
      "dtype": "int32",
      "train_file": "mnist_train_target.bin",
      "test_file": "mnist_test_target.bin"
    }
}

It is also possible to provide the OnDeviceTrainingConfig object directly instead of a file.

Get available configurations

As you may have realized, you need to know what value to provide for the target and runtime parameters. To get the list of available configuration, you can :

Go to dAIEdge-VLab Available Target page
Use the method getAvailableConfigurations

Method `getAvailableConfigurations`

To use this method you simply need to :

from daiedge_vlab import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")
# Get the possible configs
configs = api.getAvailableConfigurations()
# Use the congis
print(configs)

The config dictionary you get as a response has the following structure :

config.json

{
    "target_id1":{
        "name": "target_name",
        "engines": {
            "engine1": ["format_supported_1", "e.g. tflite"], 
            "engine2": ["format_supported_1", "e.g. ONNX"], 
        },
        "dev": true,
        "versions_tag": {
            "v1": "v1.0.0"
        },
        "target_type": "target_type, e.g. MCU",
        "memory_size_for_models": 640000,
        "description" : "Target short description"
    },
    "target_id2":{
        "name": "target_name",
        "engines": {
            "engine1": ["format_supported_1", "e.g. tflite"], 
        },
        "dev": true,
        "versions_tag": {
            "v1": "v1.0.0"
        },
        "target_type": "target_type, e.g. NPU",
        "memory_size_for_models": -1,
        "description" : "Target short description"
    },
}

In the example above, there are three possible configuration :

Config number	`target`	`runtime`
1	target_id1	engine1
2	target_id1	engine2
3	target_id2	engine1

Therefor, to start the config number 2 you should :

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "target_id1", 
    runtime = "engine2", 
    model_path = "./path/to/my_model.tflite"
    )

Advanced options

This section descibes other methods that can be usefull in less common situations.

Get your benchmark history

You may want to get the list of your previous benchmark and retrieve the old results.

from daiedge_vlab import dAIEdgeVLabAPI
# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")
# Get your history
history = api.getBenchmarks()
# Use the history - here get the results for all successfull benchmark
for benchmark in history:
    print(benchmark["platform"])
    print(benchmark["engine"])
    if benchmark["status"] == "success" : 
        result = api.getBenchmarkResult(benchmark["id"])
        # Use result

Here is all the available keys in the benchmark dictionary used in the for loop :

benchmark.json

{
    "date": "2025-03-04T08:40:55.470248+00:00", 
    "engine": "engine_name", 
    "id": 1, 
    "mean_inf_time": 500, 
    "model": "model_file_name.tflite", 
    "platform": "target_id", 
    "status": "success"
}

The status key may take the following values : success, failed, canceled, waiting, running, pending, created.

Non-bloking results

You may not want to actively wait for your benchmark result. The dAIEdge-VLab Python API implement a simple callback system to get notify when the result is available.

from daiedge_vlab import dAIEdgeVLabAPI

# This method will be invoked once the result is available
def callback(result): 
    # Use the result
    print(result["report"])
    print(result["user_log"])
    print(result["error_log"])
    print(result["raw_output"])
    print(result["model_output"])

if __name__ == "__main__" : 
    # Authentify to the dAIEdge-VLab
    api = dAIEdgeVLabAPI("./path/to/setup.yaml")

    # Start a benchmark for a given target, runtime and model
    benchmark_id = api.startBenchmark(
        target = "target_id1", 
        runtime = "engine2", 
        model_path = "./path/to/my_model.tflite",
        callback = callback
        )
    # Do something else

Setup Provide a Dataset