Benchmark Report Keys

This page describes all the keys that are used in the benchmark report. The report output is a json file that contains all the keys even if they are not used.

Report available keys

General information keys

• version (type: str): This field is used to keep track of the report version. Currently, this field is not used.
• date (type: str) : Date and time of the generation of the report. Format : YY-MM-DD hh:mm:ss
• target (type: str): Target name
• target_id (type: str): Unique ID of the target
• version_tag (type: str): Version of the target used for the benchmark.
• inference_engine (type: str): Inference engine used to run the model.
• model_file_name (type: str): Name of the model used for the benchmark.
• benchmark_type (type: str): This field contains the type of benchmark from which the measures from. (TYPE1 : random data, TYPE2 : dataset preprocessed, TYPE3: raw dataset).
• nb_infernece (type: int): Number of inference performed for the benchmark

Hardware specific keys

• ram_size (type: int): Size in bytes of the available RAM of the system.
• ram_peak (type: float): Percentage of the peak RAM usage.
• flash_size (type int): Size in bytes of the available FLASH.
• flash_usage (type: float): Percentage of the FLASH usage. Might not be relevant for Linux targets.
• load_cpu (type: float): Percentage of the CPU usage.
• load_accelerator (type: float): Percentage of the accelerator (GPU, NPU, …) usage.
• temperature (type: float): Temperature of the system (?) in °C.
• ambiant_temperature (type: float): Temperature in °C of the room where the system is.
• power_consumption (type: float): Power consumption of the system in Watt.
• enery_efficiency (type: float): Energy efficiency in Operation Per Watt (OPW).
• FLOPs (type: float): Operations per seconds
• inference_latency (type: json): This key contains a json composed of all the latency measures.
  • mean (type: float): Mean in seconds of all the inferences performed.
  • std (type: flaot): Standard deviation of all the inferences performed.
  • min (type: float): Minimum inference time in seconds.
  • max (type: float): Maximum inference time in seconds.
  • troughput (type: float): Throughput of the system in inference per seconds.
  • latency_per_layers (type: list): This key contains a list of json composed of all the layer’s latency measures.
    • <element list> (type: json): Each layer is represented as a json.
      • layer_name (type: str): Name of the layer
      • mean (type: float): Mean in seconds of the layer <layer_name>.
      • std (type: flaot): Standard deviation of the layer <layer_name>.
      • min (type: float): Minimum inference time in seconds for the layer <layer_name>.
      • max (type: float): Maximum inference time in seconds for the layer <layer_name>.
• preprocess_time (type: json): Timing of the preprocessing.
  • mean (type: float): Mean in seconds of all the preprocessing time.
  • std (type: float): std time of all the preprocessing time.
  • min (type: float): min time of all the preprocessing time.
  • max (type: float): max time of all the preprocessing time.
• postprocess_time (type: json): Timing of the post-processing.
  • mean (type: float): Mean in seconds of all the post-processing time.
  • std (type: float): std time of all the post-processing time.
  • min (type: float): min time of all the post-processing time.
  • max (type: float): max time of all the post-processing time.

Hardware agnostic keys

• model_size (type: int): Size in bytes of the model on the target (after any compression / optimization performed by the engine).
• nb_parameters_model (type: int): Number of parameters of the model.
• accuracy (type: float): Accuracy of the model (value between 0 and 1).

Report generation

The generated report should contain all the previously described keys. If you do not have a value for a given key, replace the value of this key by the keyword null. Example :

...
"ram_size": 4096,
"ram_peak": 0.7,
"flash_size": null,
...

Keys automatically overrided

The dAIEdge-VLab CI/CD automatically override some keys of the report generated by the Target. These keys are overrided to ensure their proper value.