Benchmarks

This section describes how to start benchmark with the dAIEdge-VLab Python API.

Type of benchmark

Currently, there is two type of benchmarks possible :

Benchmark using random data (TYPE1).
Benchmark using your own dataset (TYPE2).

If random data is used, more metrics are gatherd and provided in the report.json. If your own dataset is used, you will get the raw output computed by your model in addition to a likely less detailed report.json. The metrics provided depends on the target implementation.

Summary of the artifacts

The following table help you understand what you can expect form the different benchmark types :

Artifact	TYPE1	TYPE2
`report.json`	YES	YES (less detailed)
`user.log`	YES	YES
`error.log`	YES	YES
`raw_output.bin`	NO	YES

The report.json contains the keys described here.
The user.log and error.log files are simple text file.
The raw_output.bin contains the concatenated outputs for each input contained in the dataset provided.

Launch benchmark with the API

The type of benchmark run is implicit. If you do not provide a dataset, the TYPE1 is selected, otherwise the TYPE2 is launched.

Launch a TYPE1 benchmark :

from dAIEdgeVLabAPI import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "rpi5", 
    runtime = "tflite", 
    model_path = "./path/to/my_model.tflite"
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print(result["report"])
print(result["user_log"])
print(result["error_log"])

Launch a TYPE2 benchmark :

To enable the TYPE2 benchmark you need to provide a dataset. There are several ways to do so, for more details follow the Provide a Dataset section.

from dAIEdgeVLabAPI import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "rpi5", 
    runtime = "tflite", 
    model_path = "./path/to/my_model.tflite",
    dataset = "dataset_name" # or dataset file object
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print(result["report"])
print(result["user_log"])
print(result["error_log"])
print(result["raw_output"])

Get available configurations

As you may have realized, you need to know what value to provide for the target and runtime parameters. To get the list of available configuration, you can :

Go to dAIEdge-VLab Available Target page
Use the method getAvailableConfigurations

Method `getAvailableConfigurations`

To use this method you simply need to :

from dAIEdgeVLabAPI import dAIEdgeVLabAPI

# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")
# Get the possible configs
configs = api.getAvailableConfigurations()
# Use the congis
print(configs)

The config dictionary you get as a response has the following structure :

config.json

{
    "target_id1":{
        "name": "target_name",
        "engines": {
            "engine1": ["format_supported_1", "e.g. tflite"], 
            "engine2": ["format_supported_1", "e.g. ONNX"], 
        },
        "dev": true,
        "versions_tag": {
            "v1": "v1.0.0"
        },
        "target_type": "target_type, e.g. MCU",
        "memory_size_for_models": 640000,
        "description" : "Target short description"
    },
    "target_id2":{
        "name": "target_name",
        "engines": {
            "engine1": ["format_supported_1", "e.g. tflite"], 
        },
        "dev": true,
        "versions_tag": {
            "v1": "v1.0.0"
        },
        "target_type": "target_type, e.g. NPU",
        "memory_size_for_models": -1,
        "description" : "Target short description"
    },
}

In the example above, there are three possible configuration :

Config number	`target`	`runtime`
1	target_id1	engine1
2	target_id1	engine2
3	target_id2	engine1

Therefor, to start the config number 2 you should :

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = "target_id1", 
    runtime = "engine2", 
    model_path = "./path/to/my_model.tflite"
    )

Advanced options

This section descibes other methods that can be usefull in less common situations.

Get your benchmark history

You may want to get the list of your previous benchmark and retrieve the old results.

from dAIEdgeVLabAPI import dAIEdgeVLabAPI
# Authentify to the dAIEdge-VLab
api = dAIEdgeVLabAPI("./path/to/setup.yaml")
# Get your history
history = api.getBenchmarks()
# Use the history - here get the results for all successfull benchmark
for benchmark in history:
    print(benchmark["platform"])
    print(benchmark["engine"])
    if benchmark["status"] == "success" : 
        result = api.getBenchmarkResult(benchmark["id"])
        # Use result

Here is all the available keys in the benchmark dictionary used in the for loop :

benchmark.json

{
    "date": "2025-03-04T08:40:55.470248+00:00", 
    "engine": "engine_name", 
    "id": 1, 
    "mean_inf_time": 500, 
    "model": "model_file_name.tflite", 
    "platform": "target_id", 
    "status": "success"
}

The status key may take the following values : success, failed, canceled, waiting, running, pending, created.

Non-bloking results

You may not want to actively wait for your benchmark result. The dAIEdge-VLab Python API implement a simple callback system to get notify when the result is available.

from dAIEdgeVLabAPI import dAIEdgeVLabAPI

# This method will be invoked once the result is available
def callback(result): 
    # Use the result
    print(result["report"])
    print(result["user_log"])
    print(result["error_log"])

if __name__ == "__main__" : 
    # Authentify to the dAIEdge-VLab
    api = dAIEdgeVLabAPI("./path/to/setup.yaml")

    # Start a benchmark for a given target, runtime and model
    benchmark_id = api.startBenchmark(
        target = "target_id1", 
        runtime = "engine2", 
        model_path = "./path/to/my_model.tflite",
        callback = callback
        )
    # Do something else

Setup Provide a Dataset