User Documentation
dAIEdge-VLab Python API

dAIEdge-VLab Python API

This section describes the dAIEdge-VLab API.

⚠️
Warning : This dAIEdge-VLab API is in developpement and is only available using the API sources directly.

What dose the API provide ?

The dAIEdge-VLab Python API provide a simple way to interact with the targets available in the dAIEdge-VLab using Python. This allows you to build developpement pipeline with hardware in the loop.

Here are the key points that the dAIEdge-VLab API allows you to do :

  • Run benchmark of your model on the availbles targets
  • Provide your pre-processed dataset for the benchmark
  • Retrive the report.json generated by the benchmark
  • Retrive the different logs files generated by the benchmark
  • Retrive the raw output generated by your model

Quick start

ℹ️
Install : This dAIEdge-VLab API is in developpement and is currenlty not available using pip. Use the API sources instead.
  • Download the API sources here.

Install the Python API depedencies :

pip install PyYaml
pip install requests

Step-by-step user guide

⏱️
Time needed : 15 min to 30 min

This step-by-step guide is here to help you start with the dAIEdgeVLabAIP and create a first simple script that will launch benchmark on remote hardware. It is assumed that you are familiarized with the concepts of AI models, dataset and Python. However, you do not need to be an expert in any of these.

Overview

This diagram explains the overall pipeline that enables the dAIEdgeVLab Python API :

sequenceDiagram
    participant User Computer
    participant dAIEdge-VLab
    participant Remote Device
    User Computer->>dAIEdge-VLab: Model, Target, Runtime, (Dataset)
    dAIEdge-VLab->>Remote Device: Model, (Dataset)
    Remote Device->>Remote Device: Perfom the inferences
    Remote Device->>dAIEdge-VLab: report, logs, (raw output)
    dAIEdge-VLab->>User Computer: report, logs, (raw output)
    User Computer->>User Computer: Analyse data

Some key elements :

  • User computer : Your computer with the script we are about to create
  • Remote Device / Target : The remote hardware you want your model to be evaluated on
  • Runtime : The software that will execute your model
  • Model : The AI model you want to evaluate
  • Dataset : The data that will be fed to your model to perform the inferences (optional)
  • Raw output : The raw output of the model. Only available if a dataset was provided
  • dAIEdge-VLab : The main link between you and the remote devices

Follow the next few steps to learn how to use the dAIEdge-VLab Python API.

Follow this section if your opertating system is Linux based.

Requirements

First, make sure you have Python install on your computer and an IDE. We recommend the following :

Note that you can use another version of Python 3.

Prepare your environment

  1. Open your IDE and select a new empty work directory
  2. Open a terminal in this directory
  3. Create a new Python virtual environment with : 
    python -m venv .venv
  4. Activate the environment with : 
    source .venv/bin/activate
  5. Install the package PyYaml with : 
    pip install PyYaml
  6. Install the package requests : 
    pip install requests
  7. Create a Python file : 
    touch main.py
  8. Copy in this directory your trained model and the according pre-processed dataset. If you don’t have these files you can download and use the following ones:
    mnist.tflite dataset_mnist.bin

Learn how to prepare your dataset here.

Install the dAIEdge-VLab API

Download and place in your work directory the following Python script. This is the script that contains the dAIEdgeVLabAPI object that interact with the dAIEdge-VLab.

Setup the API

Start by creating a setup.yaml file in the work directory :

touch setup.yaml

In this file place the API configuration and your credentials :

setup.yaml
api:
    url: "vlab.daiedge.eu"
    port: "443"
user : 
    email: "your.email@example.com"
    password: "your-password"

If you are using git to track your project, it is heavily recommended adding the setup.yaml file in your .gitignore:

.gitignore
setup.yaml

Start your first benchmark

Open the main.py file you created earlier. Then add the following content :

  • Modify the variables TARGET, RUNTIME and MODEL to match your needs.
  • Have a look at this page to find all the available configuration TARGET and RUNTIME.
main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Mean us : ", result["report"]["inference_latency"]["mean"])

To run this program simply run the following command in your terminal :

python main.py
ℹ️
Execution time : The execution time depends on the availability of the target and its implementation. You may wait several minutes. For the rpi5 - tflite you can expect to wait around 20 secondes.

In the previous example, we started a benchmark on the selected target using the trained model in your local computer. Finally, we printed the results and you should see something like :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE1', 'date': '2025-03-06T18:00:58.990195+00:00', 'energy_efficiency': None, 'flash_size': 62416101376, 'flash_usage': 0.0007047220033020731, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [{'layer_name': ' Delegate/Copy (NC X32):0', 'max': 0.0017816, 'mean': 0.0017816, 'min': 0.0017816, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:1', 'max': 6.69511, 'mean': 6.69511, 'min': 6.69511, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:2', 'max': 1.91131, 'mean': 1.91131, 'min': 1.91131, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:3', 'max': 0.00228613, 'mean': 0.00228613, 'min': 0.00228613, 'std': 0}, {'layer_name': ' Delegate/Softmax (NC F32):4', 'max': 0.000961751, 'mean': 0.000961751, 'min': 0.000961751, 'std': 0}], 'max': 7875, 'mean': 8.61145, 'min': 5, 'std': 85, 'troughput': None}, 'load_accelerator': None, 'load_cpu': 79.94444444444444, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 63426, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': 0.19776433598306642, 'ram_size': 4241719296, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Mean us : 8.38573
  • Have a look at this page to learn more about the report format.

Make a simple AI App

Now we will try to provide a pre-processed dataset to the dAIEdge-VLab to assess the accuracy of the model. Modify the main.py file to add the dataset in the startBenchmark method :

main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Open the dataset
dataset = open("dataset_mnist.bin", "rb")

# Start a benchmark for a given target, runtime, model and dataset
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL,
    dataset = dataset
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Binary:\n", result["raw_output"])

Run the program with :

python main.py

Now the result should look like this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'

This is not very helpful as is, however, the binary we got back is the actual raw output computed by the model on the selected target for the given dataset. We simply need to interpret it correctly. To do so, we need to know some key information :

  • The number of samples that composed the dataset. The dataset dataset_mnist.bin is composed 256 images, so 256 samples.
  • The output size of the model. The model mnist.tflite output an array of 10 values (10 classes).
  • The type of the output of the model. The model mnist.tflite outputs values of type float32.
  • The labels of each sample that compose the dataset. For the dataset_mnist.bin they are the following:
labels = [ 
    5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7,
    3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1,
    4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4,
    6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0,
    2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3,
    4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3,
    2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0,
    0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5,
    8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3 
]

Once we know all the needed information we will modify the main.py script to interpret the data correctly. But first we will install numpy as it is very useful to handle situations like this. Run the next command in your terminal :

pip install numpy

Then add this line at the top of your main.py file :

top of main.py
import numpy as np

Finally, place the following lines at the bottom of the file :

bottom of main.py
# Store the array of expected labes
labels = [ 5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7, 3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1, 4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4, 6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0, 2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3, 4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3, 2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0, 0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5, 8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3]

# Set the number of sampels that composes the dataset
nb_samples = 256

# Convert the binary file to an array of float32, then reshape it as (nb_samples, 10)
output_array = np.frombuffer(result["raw_output"], dtype=np.float32)
output_array = output_array.reshape(nb_samples, 10)

# Find the max argument in each nb_samples array of 10 float values
predictions = np.argmax(output_array, axis=1)
# Compute the accuracy of the prediction 
accuracy = np.mean(predictions == labels)

print("Accuracy: ", accuracy)

Run the script and you should see this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'
Accuracy:  0.984375

If you encounter any issue, you can download the full project here. Do not hesitate contacting the dAIEdge-VLab Team if you keep struggling.

Follow this section if your opertating system is Windows.

Requirements

First, make sure you have Python install on your computer and an IDE. We recommend the following :

Note that you can use another version of Python 3.

Prepare your environment

  1. Open your IDE and select a new empty work directory
  2. Open a terminal in this directory
  3. Create a new Python virtual environment with : 
    python -m venv .venv
  4. Activate the environment with : 
    .\.venv\Scripts\activate
  5. Install the package PyYaml with : 
    pip install PyYaml
  6. Install the package requests : 
    pip install requests
  7. Create a Python file using the IDE. Name this file mian.py
  8. Copy in this directory your trained model and the according pre-processed dataset. If you don’t have these files you can download and use the following ones:
    mnist.tflite dataset_mnist.bin

Learn how to prepare your dataset here.

Install the dAIEdge-VLab API

Download and place in your work directory the following Python script. This is the script that contains the dAIEdgeVLabAPI object that interact with the dAIEdge-VLab.

Setup the API

Start by creating a setup.yaml file in the work directory with the IDE. In this file place the API configuration and your credentials :

setup.yaml
api:
    url: "vlab.daiedge.eu"
    port: "443"
user : 
    email: "your.email@example.com"
    password: "your-password"

If you are using git to track your project, it is heavily recommended adding the setup.yaml file in your .gitignore:

.gitignore
setup.yaml

Start your first benchmark

Open the main.py file you created earlier. Then add the following content :

  • Modify the variables TARGET, RUNTIME and MODEL to match your needs.
  • Have a look at this page to find all the available configuration TARGET and RUNTIME.
main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Mean us : ", result["report"]["inference_latency"]["mean"])

To run this program simply run the following command in your terminal :

python main.py
ℹ️
Execution time : The execution time depends on the availability of the target and its implementation. You may wait several minutes. For the rpi5 - tflite you can expect to wait around 20 secondes.

In the previous example, we started a benchmark on the selected target using the trained model in your local computer. Finally, we printed the results and you should see something like :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE1', 'date': '2025-03-06T18:00:58.990195+00:00', 'energy_efficiency': None, 'flash_size': 62416101376, 'flash_usage': 0.0007047220033020731, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [{'layer_name': ' Delegate/Copy (NC X32):0', 'max': 0.0017816, 'mean': 0.0017816, 'min': 0.0017816, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:1', 'max': 6.69511, 'mean': 6.69511, 'min': 6.69511, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:2', 'max': 1.91131, 'mean': 1.91131, 'min': 1.91131, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:3', 'max': 0.00228613, 'mean': 0.00228613, 'min': 0.00228613, 'std': 0}, {'layer_name': ' Delegate/Softmax (NC F32):4', 'max': 0.000961751, 'mean': 0.000961751, 'min': 0.000961751, 'std': 0}], 'max': 7875, 'mean': 8.61145, 'min': 5, 'std': 85, 'troughput': None}, 'load_accelerator': None, 'load_cpu': 79.94444444444444, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 63426, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': 0.19776433598306642, 'ram_size': 4241719296, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Mean us : 8.38573
  • Have a look at this page to learn more about the report format.

Make a simple AI App

Now we will try to provide a pre-processed dataset to the dAIEdge-VLab to assess the accuracy of the model. Modify the main.py file to add the dataset in the startBenchmark method :

main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Open the dataset
dataset = open("dataset_mnist.bin", "rb")

# Start a benchmark for a given target, runtime, model and dataset
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL,
    dataset = dataset
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Binary:\n", result["raw_output"])

Run the program with :

python main.py

Now the result should look like this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'

This is not very helpful as is, however, the binary we got back is the actual raw output computed by the model on the selected target for the given dataset. We simply need to interpret it correctly. To do so, we need to know some key information :

  • The number of samples that composed the dataset. The dataset dataset_mnist.bin is composed 256 images, so 256 samples.
  • The output size of the model. The model mnist.tflite output an array of 10 values (10 classes).
  • The type of the output of the model. The model mnist.tflite outputs values of type float32.
  • The labels of each sample that compose the dataset. For the dataset_mnist.bin they are the following:
labels = [ 
    5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7,
    3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1,
    4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4,
    6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0,
    2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3,
    4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3,
    2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0,
    0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5,
    8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3 
]

Once we know all the needed information we will modify the main.py script to interpret the data correctly. But first we will install numpy as it is very useful to handle situations like this. Run the next command in your terminal :

pip install numpy

Then add this line at the top of your main.py file :

top of main.py
import numpy as np

Finally, place the following lines at the bottom of the file :

bottom of main.py
# Store the array of expected labes
labels = [ 5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7, 3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1, 4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4, 6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0, 2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3, 4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3, 2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0, 0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5, 8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3]

# Set the number of sampels that composes the dataset
nb_samples = 256

# Convert the binary file to an array of float32, then reshape it as (nb_samples, 10)
output_array = np.frombuffer(result["raw_output"], dtype=np.float32)
output_array = output_array.reshape(nb_samples, 10)

# Find the max argument in each nb_samples array of 10 float values
predictions = np.argmax(output_array, axis=1)
# Compute the accuracy of the prediction 
accuracy = np.mean(predictions == labels)

print("Accuracy: ", accuracy)

Run the script and you should see this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'
Accuracy:  0.984375

If you encounter any issue, you can download the full project here. Do not hesitate contacting the dAIEdge-VLab Team if you keep struggling.

Follow this section if your opertating system is MacOS.

Requirements

First, make sure you have Python install on your computer and an IDE. We recommend the following :

Note that you can use another version of Python 3.

Prepare your environment

  1. Open your IDE and select a new empty work directory
  2. Open a terminal in this directory
  3. Create a new Python virtual environment with : 
    python -m venv .venv
  4. Activate the environment with : 
    source .venv/bin/activate
  5. Install the package PyYaml with : 
    pip install PyYaml
  6. Install the package requests : 
    pip install requests
  7. Create a Python file : 
    touch main.py
  8. Copy in this directory your trained model and the according pre-processed dataset. If you don’t have these files you can download and use the following ones:
    mnist.tflite dataset_mnist.bin

Learn how to prepare your dataset here.

Install the dAIEdge-VLab API

Download and place in your work directory the following Python script. This is the script that contains the dAIEdgeVLabAPI object that interact with the dAIEdge-VLab.

Setup the API

Start by creating a setup.yaml file in the work directory :

touch setup.yaml

In this file place the API configuration and your credentials :

setup.yaml
api:
    url: "vlab.daiedge.eu"
    port: "443"
user : 
    email: "your.email@example.com"
    password: "your-password"

If you are using git to track your project, it is heavily recommended adding the setup.yaml file in your .gitignore:

.gitignore
setup.yaml

Start your first benchmark

Open the main.py file you created earlier. Then add the following content :

  • Modify the variables TARGET, RUNTIME and MODEL to match your needs.
  • Have a look at this page to find all the available configuration TARGET and RUNTIME.
main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Start a benchmark for a given target, runtime and model
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Mean us : ", result["report"]["inference_latency"]["mean"])

To run this program simply run the following command in your terminal :

python main.py
ℹ️
Execution time : The execution time depends on the availability of the target and its implementation. You may wait several minutes. For the rpi5 - tflite you can expect to wait around 20 secondes.

In the previous example, we started a benchmark on the selected target using the trained model in your local computer. Finally, we printed the results and you should see something like :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE1', 'date': '2025-03-06T18:00:58.990195+00:00', 'energy_efficiency': None, 'flash_size': 62416101376, 'flash_usage': 0.0007047220033020731, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [{'layer_name': ' Delegate/Copy (NC X32):0', 'max': 0.0017816, 'mean': 0.0017816, 'min': 0.0017816, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:1', 'max': 6.69511, 'mean': 6.69511, 'min': 6.69511, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:2', 'max': 1.91131, 'mean': 1.91131, 'min': 1.91131, 'std': 0}, {'layer_name': ' Delegate/Fully Connected (NC F32) GEMM:3', 'max': 0.00228613, 'mean': 0.00228613, 'min': 0.00228613, 'std': 0}, {'layer_name': ' Delegate/Softmax (NC F32):4', 'max': 0.000961751, 'mean': 0.000961751, 'min': 0.000961751, 'std': 0}], 'max': 7875, 'mean': 8.61145, 'min': 5, 'std': 85, 'troughput': None}, 'load_accelerator': None, 'load_cpu': 79.94444444444444, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 63426, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': 0.19776433598306642, 'ram_size': 4241719296, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Mean us : 8.38573
  • Have a look at this page to learn more about the report format.

Make a simple AI App

Now we will try to provide a pre-processed dataset to the dAIEdge-VLab to assess the accuracy of the model. Modify the main.py file to add the dataset in the startBenchmark method :

main.py
from dAIEdgeVLabAPI import dAIEdgeVLabAPI

TARGET = "rpi5"
RUNTIME = "tflite" 
MODEL = "mnist.tflite"

api = dAIEdgeVLabAPI("setup.yaml")

# Open the dataset
dataset = open("dataset_mnist.bin", "rb")

# Start a benchmark for a given target, runtime, model and dataset
benchmark_id = api.startBenchmark(
    target = TARGET, 
    runtime = RUNTIME, 
    model_path = MODEL,
    dataset = dataset
    )

# Blocking method - wait for the results
result = api.waitBenchmarkResult(benchmark_id)

# Use the result
print("Report:\n", result["report"])
print("User log:\n", result["user_log"])
print("Error log:\n", result["error_log"])
print("Binary:\n", result["raw_output"])

Run the program with :

python main.py

Now the result should look like this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'

This is not very helpful as is, however, the binary we got back is the actual raw output computed by the model on the selected target for the given dataset. We simply need to interpret it correctly. To do so, we need to know some key information :

  • The number of samples that composed the dataset. The dataset dataset_mnist.bin is composed 256 images, so 256 samples.
  • The output size of the model. The model mnist.tflite output an array of 10 values (10 classes).
  • The type of the output of the model. The model mnist.tflite outputs values of type float32.
  • The labels of each sample that compose the dataset. For the dataset_mnist.bin they are the following:
labels = [ 
    5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7,
    3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1,
    4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4,
    6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0,
    2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3,
    4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3,
    2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0,
    0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5,
    8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3 
]

Once we know all the needed information we will modify the main.py script to interpret the data correctly. But first we will install numpy as it is very useful to handle situations like this. Run the next command in your terminal :

pip install numpy

Then add this line at the top of your main.py file :

top of main.py
import numpy as np

Finally, place the following lines at the bottom of the file :

bottom of main.py
# Store the array of expected labes
labels = [ 5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7, 3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1, 4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7, 1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4, 6, 7, 4, 6, 8, 0, 7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0, 2, 0, 2, 7, 1, 8, 6, 4, 1, 6, 3, 4, 5, 9, 1, 3, 3, 8, 5, 4, 7, 7, 4, 2, 8, 5, 8, 6, 7, 3, 4, 6, 1, 9, 9, 6, 0, 3, 7, 2, 8, 2, 9, 4, 4, 6, 4, 9, 7, 0, 9, 2, 9, 5, 1, 5, 9, 1, 2, 3, 2, 3, 5, 9, 1, 7, 6, 2, 8, 2, 2, 5, 0, 7, 4, 9, 7, 8, 3, 2, 1, 1, 8, 3, 6, 1, 0, 3, 1, 0, 0, 1, 7, 2, 7, 3, 0, 4, 6, 5, 2, 6, 4, 7, 1, 8, 9, 9, 3, 0, 7, 1, 0, 2, 0, 3, 5, 4, 6, 5, 8, 6, 3, 7, 5, 8, 0, 9, 1, 0, 3, 1, 2, 2, 3, 3]

# Set the number of sampels that composes the dataset
nb_samples = 256

# Convert the binary file to an array of float32, then reshape it as (nb_samples, 10)
output_array = np.frombuffer(result["raw_output"], dtype=np.float32)
output_array = output_array.reshape(nb_samples, 10)

# Find the max argument in each nb_samples array of 10 float values
predictions = np.argmax(output_array, axis=1)
# Compute the accuracy of the prediction 
accuracy = np.mean(predictions == labels)

print("Accuracy: ", accuracy)

Run the script and you should see this :

main.py output
Report: 
{'GFLOPs': None, 'accuracy': None, 'ambiant_temperature': None, 'benchmark_type': 'TYPE2', 'date': '2025-03-06T18:07:41.229616+00:00', 'energy_efficiency': None, 'flash_size': None, 'flash_usage': None, 'inference_engine': 'tflite', 'inference_latency': {'latency_per_layers': [], 'max': 3.9425e-05, 'mean': 1.54326015625e-05, 'min': 1.5074e-05, 'std': 1.5830454102567805e-06, 'troughput': None}, 'load_accelerator': None, 'load_cpu': None, 'model_file_name': 'mnist_model_small.tflite', 'model_size': 439860, 'nb_inference': 0, 'nb_parameters_model': None, 'postprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'power_consumption': None, 'preprocess_time': {'max': None, 'mean': None, 'min': None, 'std': None}, 'ram_peak_usage': None, 'ram_size': None, 'target': 'rpi5', 'target_id': None, 'temperature': None, 'version': 0, 'version_tag': 'v1.0.0'}
User log: 
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Error log:

Binary: 
b'\x86\xe2\x8e*Gy\xb41\x0b?\x0c0\xf0\xdd\xe8;\x12\xbe\x7f"C.~?z=2):\xdeX+\xff\nW/\x9a\x99\x932\xce\xff\x7f?\xdb\xf6\x0f-\xf8\xef=69d\x8d.C\x9cX/6P...'
Accuracy:  0.984375

If you encounter any issue, you can download the full project here. Do not hesitate contacting the dAIEdge-VLab Team if you keep struggling.

You now know how to use the dAIEdgeVLabAPI. Look at the next sections for a deeper overview of the API and its capabilities. You will also find a complete application example with the generation of a pre-processed dataset, the training of the model and the evaluation of the performances.

Setup