Benchmark Report Generator

This page describes how to generate a benchmark report using a simple python object. The report is a JSON file that contains the benchmark results.

Usage

A first python generator is available here. This generator helps you create a report that follows the next guidelines. You will need to download the report_template.json to make it work.

The following example shows how to use the ReportGenerator class. Only a subset of the available functions is shown:

report = ReportGenerator("report_template.json")

# Add information
report.addInfo(
    target=os.environ['TARGET'], 
    engine=os.environ['RUNTIME'], 
    benchmark_type="TYPE1", 
    nb_inference=10
    )
    
report.addInfoModel(
    model_file_name=os.environ['MODEL_FILENAME'], 
    model_size=None, 
    nb_parameters_model=None
    )
    
report.addLatency(mean=100, std=100, min=100, max=100)
report.computeAndAddLayerLatency(layer_name="Layer 1", latency=array_latenecy_layer1)

report.save("AI_Manager/out/report.json")

ReportGenerator Class Documentation

The ReportGenerator class is designed to generate a valid report.json file for the dAIEdge-VLab. It loads a JSON template, updates it with various metrics and information, and saves the updated report to a specified location.

Class: ReportGenerator

Constructor

__init__(self, path: str)

Initializes the ReportGenerator class with the path to a JSON template.

• Parameters:
  • path (str): The file path to the JSON template of the report.

Private Methods

_loadJson(self)

Loads the JSON template from the specified path, initializes a timestamp, and clears all layer latency details.

Public Methods

addLayerLatency(self, layer_name: str = None, mean: float = None, std: float = None, min: float = None, max: float = None)

Adds latency metrics for a specific layer.

• Parameters:
  • layer_name (str): Description or name of the layer.
  • mean (float): Mean latency in seconds across all inferences.
  • std (float): Standard deviation of latency across all inferences.
  • min (float): Minimum latency recorded across all inferences.
  • max (float): Maximum latency recorded across all inferences.

addLatency(self, mean: float, std: float, min: float, max: float)

Adds overall latency metrics for the model.

• Parameters:
  • mean (float): Mean latency in seconds across all inferences.
  • std (float): Standard deviation of latency across all inferences.
  • min (float): Minimum latency recorded across all inferences.
  • max (float): Maximum latency recorded across all inferences.

addTroughput(self, troughput: float)

Adds throughput metrics for the model.

• Parameters:
  • troughput (float): Throughput in inferences per second.

addInfo(self, target: str, engine: str, benchmark_type: str, nb_inference: int)

Adds general information about the benchmark.

• Parameters:
  • target (str): Target name.
  • engine (str): Inference engine used.
  • benchmark_type (str): Type of benchmark (e.g., random data, dataset preprocessed, raw dataset).
  • nb_inference (int): Number of inferences performed for the benchmark.

addInfoModel(self, model_file_name: str, model_size: int = None, nb_parameters_model: int = None)

Adds information about the model.

• Parameters:
  • model_file_name (str): Name of the model used for the benchmark.
  • model_size (int, optional): Size in bytes of the model after compression or optimization.
  • nb_parameters_model (int, optional): Number of parameters in the model.

computeAndAddLayerLatency(self, layer_name: str, latency: list[float])

Computes and adds latency metrics for a specific layer.

• Parameters:
  • layer_name (str): Description or name of the layer.
  • latency (list[float]): List of latencies for the layer.

computeAndAddLatency(self, latency: list[float])

Computes and adds overall latency metrics for the model.

• Parameters:
  • latency (list[float]): List of latencies for the model.

addRAMInfo(self, ram_size: int = None, ram_peak: float = None)

Adds information about RAM usage.

• Parameters:
  • ram_size (int, optional): Total RAM size in bytes.
  • ram_peak (float, optional): Peak RAM usage as a percentage (0 to 1).

addFLASHInfo(self, flash_size: int = None, flash_usage: float = None)

Adds information about FLASH memory usage.

• Parameters:
  • flash_size (int, optional): Total FLASH size in bytes.
  • flash_usage (float, optional): Percentage of FLASH usage (0 to 1).

addTemperature(self, ambiant_temp: float = None, system_temp: float = None)

Adds temperature metrics.

• Parameters:
  • ambiant_temp (float, optional): Ambient temperature in °C.
  • system_temp (float, optional): System temperature in °C.

addPowerInfo(self, power_consumption: float = None, energy_efficiency: float = None, GFLOPs: float = None)

Adds power consumption metrics.

• Parameters:
  • power_consumption (float, optional): Power consumption of the system in watts.
  • energy_efficiency (float, optional): Energy efficiency in operations per watt (OPW).
  • GFLOPs (float, optional): GFLOPs of the system.

addCPULoad(self, cpu_load: float)

Adds CPU load metric.

• Parameters:
  • cpu_load (float): Percentage of CPU usage (0 to 1).

addAcceleratorLoad(self, accelerator_load: float)

Adds load metrics for a hardware accelerator (e.g., GPU, NPU).

• Parameters:
  • accelerator_load (float): Percentage of accelerator usage (0 to 1).

addPreprocessingTiming(self, mean: float, std: float, min: float, max: float)

Adds preprocessing timing metrics.

• Parameters:
  • mean (float): Mean preprocessing time in seconds.
  • std (float): Standard deviation of preprocessing time.
  • min (float): Minimum preprocessing time.
  • max (float): Maximum preprocessing time.

addPostprocessingTiming(self, mean: float, std: float, min: float, max: float)

Adds postprocessing timing metrics.

• Parameters:
  • mean (float): Mean postprocessing time in seconds.
  • std (float): Standard deviation of postprocessing time.
  • min (float): Minimum postprocessing time.
  • max (float): Maximum postprocessing time.

addApplicationTiming(self, preprocess_time: list[float] = None, postprocess_time: list[float] = None)

Computes and adds application timing metrics.

• Parameters:
  • preprocess_time (list[float], optional): List of preprocessing times.
  • postprocess_time (list[float], optional): List of postprocessing times.

addModelPerformances(self, accuracy: float = None)

Adds performance metrics of the model.

• Parameters:
  • accuracy (float, optional): Accuracy of the model (0 to 1).

save(self, path: str)

Saves the report to the specified file path.

• Parameters:
  • path (str): File path where the report will be saved.