S100 OpenExplorer

# #The hb_verifier Tool

The hb_verifier is a consistency verification tool that supports performing cosine similarity comparisons between ONNX models, between ONNX model and HBIR model, between HBIR model and HBIR model, and output consistency comparisons between HBIR model and HBM model.

The cosine similarity indicates the consistency between quantized models at different stages. As the cosine similarity gets closer to 1, it indicates that the outputs of the two quantized models being compared are closer to each other.

Consistency comparison prints output consistency information for the comparison models, including output name, consistency, number of inconsistent elements, maximum absolute error, and maximum relative error.

## #Support Range

- Fbc represents the floating-point HBIR model, which refers to the HBIR model before convert exported after the model conversion by PTQ API method or QAT pipeline.

- Qbc represents the fixed-point HBIR model.

Comparison Models | Comparison Scenarios | Whether the Model is from hb_compile | Input Data Requirement
Onnx vs Onnx | Cosine similarity comparison | Not interested | One original data
Onnx vs Fbc | Cosine similarity comparison | No | Two sets of data corresponding to the Onnx and Fbc models
Onnx vs Qbc | Cosine similarity comparison | Yes | One original data
Onnx vs Qbc | Cosine similarity comparison | No | One original data One runtime data
Fbc vs Qbc | Cosine similarity comparison | No | One original data One runtime data
Qbc vs Hbm | Output consistency comparison | Yes | One runtime data
Qbc vs Hbm | Output consistency comparison | No | One runtime data
Hbm vs Hbm | Output consistency comparison | Not interested | One runtime data

Attention

1. For Fbc (floating-point HBIR model), if some preprocessing nodes have been inserted, the raw data needs to be modified to match the model inputs.

2. Currently, the `hb_verifier` tool does not support the cosine similarity comparison for the `ptq_model.onnx model` generated during the model quantization and compilation process when `separate_batch` or `separate_name` has been specified.

3. All models involved in the comparison must be the outputs of conversion and compilation from the same pipeline(PTQ/QAT).

## #Usage

`Usage: hb_verifier [OPTIONS] Options: -h, --help Show this message and exit. --version Show the version and exit. -m, --model TEXT The types of parameters supported include onnx/bc/hbm models, comma-separated. [required] -i, --input TEXT Original model input files. [required] -c, --compare_digits INTEGER The number of decimal places to compare. --ip TEXT Board ip (comma-separated, number should match HBM model count). -u, --username TEXT Board username (comma-separated, number should match HBM model count). -p, --password TEXT Board password (comma-separated, number should match HBM model count). --port TEXT Board ssh port (comma-separated, number should match HBM model count).`

## #Parameters Introduction

Parameter Name | Parameter Description
`-h, --help` | Display help information and exit.
`--version` | Display version information and exit.
`-m, --model` | Specify the model input, supports ONNX model (*.onnx), HBIR model (*.bc) file, HBM model (*.hbm) file. The two model addresses are separated by a “,”: `-m model1,model2`.
`-i, --input` | Specify the data to be used for inference testing; only *.npy files are supported. For multi-input models, there are two ways to pass input data, separated by commas: input_name1:input_data_1.npy,input_name2:input_data_2.npy, … input_data_1.npy,input_data_2.npy… If you need to input data for two different models, please use two `-i` parameters. Attention: The input data must include preprocessing operations, such as color conversion, mean, scale, etc., which should be done externally when preparing the input. If there are no special requirements, the raw data can use the calibration dataset as input; runtime data needs to be consistent with the input of the quantized model. The order of input data names must match the actual input order of the model.
`-c, --compare_digits` | Set the numerical precision of the comparison inference result (i.e. the number of decimal places to compare the value), if not specified, the tool will compare to five decimal places by default.
`--ip` | Set the IP address of the development board, default is empty. Multiple entries need to be separated by commas.
`-u, --usename` | Set the board SSH user name, defaults to `root`. Multiple entries need to be separated by commas.
`-p, --password` | Set the board SSH password, default is empty. Multiple entries need to be separated by commas.
`--port` | Set the board SSH port, default is `22`. Multiple inputs need to be separated by commas.

## #Usage Examples

### #Cosine Similarity Comparisons

#### #ONNX Model and ONNX Model

Cosine similarity comparisons were performed between the ONNX model and the ONNX model.

Take the model optimization stage model `optimized_float_model.onnx` and the model calibration stage model `calibrated_model.onnx` as an example:

`hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_calibrated_model.onnx \ -i input.npy`

#### #ONNX Model and HBIR Model

1. Cosine similarity comparisons were performed between the ONNX model and the floating-point HBIR model.

   Take the model optimization stage model `optimized_float_model.onnx` and the floating-point HBIR model `float_model.bc` as an example:

   `hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_float_model.bc \ -i input1.npy,input2.npy`

2. Cosine similarity comparisons were performed between the ONNX model and the fixed-point model HBIR model.

   - If the model is compiled by hb_compile, take the model optimization stage model `optimized_float_model.onnx` and the model quantization stage fixed-point model `quantized_model.bc` as an example:

     `hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_quantized_model.bc \ -i input.npy`

   - If the model is compiled by hb_compile and there are multiple input data, take the model optimization stage model `optimized_float_model.onnx` and the model quantization stage fixed-point model `quantized_model.bc` as an example:

     `hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_quantized_model.bc \ -i input1.npy,input2.npy`

   - If the model is not compiled by hb_compile, take the original floating-model `original_float_model.onnx` and the fixed-point model `quantized_model.bc` as an example:

     `hb_verifier -m googlenet_original_float_model.onnx,googlenet_quantized_model.bc \ -i input.npy \ -i runtime_input.npy`

   - If the model is not compiled by hb_compile and there are multiple input data, take the original floating-model `original_float_model.onnx` and the fixed-point model `quantized_model.bc` as an example:

     `hb_verifier -m googlenet_original_float_model.onnx,googlenet_quantized_model.bc \ -i input1.npy,input2.npy \ -i runtime_input1.npy,runtime_input2.npy`

#### #HBIR Model and HBIR Model

Cosine similarity comparisons were performed between the HBIR model and the HBIR model.

- If the model is not compiled by hb_compile, take the floating-point model `float_model.bc` and the fixed-point model `quantized_model.bc` as an example:

  `hb_verifier -m googlenet_float_model.bc,googlenet_quantized_model.bc \ -i input.npy \ -i runtime_input.npy`

- If the model is not compiled by hb_compile and there are multiple input data, take the floating-point model `float_model.bc` and the fixed-point model `quantized_model.bc` as an example:

  `hb_verifier -m googlenet_float_model.bc,googlenet_quantized_model.bc \ -i input1.npy,input2.npy -i runtime_input1.npy,runtime_input2.npy`

### #Output Consistency Comparisons

#### #HBIR Model and HBM Model

Output consistency comparisons were performed between the HBIR model and the HBM model.

- Take the model quantization stage fixed-point model `quantized_model.bc` and the model compilation stage model `googlenet.hbm` as an example:

  `hb_verifier -m googlenet_quantized_model.bc,googlenet.hbm \ -i runtime_input.npy`

- Take the model quantization stage fixed-point model `quantized_model.bc` and the model compilation stage model `googlenet.hbm` as an example, and setting the ip to `None` indicates using the simulator:

  `hb_verifier -m googlenet_quantized_model.bc,googlenet.hbm \ -i runtime_input.npy \ --ip None,10.10.10.10`

#### #HBM Model and HBM Model

Output consistency comparisons of HBM model across simulator and development board.

Take the model compilation stage model `googlenet.hbm` as an example. Setting the ip to `None` indicates using the simulator:

`hb_verifier -m googlenet.hbm,googlenet.hbm \ -i runtime_input.npy \ --ip None,10.10.10.10`

## #Output Contents

### #Cosine Similarity Comparison

The cosine similarity information for the compared models will be printed within the terminal, as shown in the example below:

`+-----------------------------------+----------------------------+-------------------+ | NodeName | TensorName | ConsineSimilarity | +-----------------------------------+----------------------------+-------------------+ | Conv_0 | 365 | 0.999978 | | Relu_2 | 367 | 0.999976 | | MaxPool_3 | 368 | 0.999901 | | Conv_4 | 369 | 0.999112 | | Relu_6 | 371 | 0.999212 | | ... | ... | ... | | Conv_189 | 554 | 0.984707 | | Relu_191 | 556 | 0.989554 | | Concat_192 | 557 | 0.989221 | | GlobalAveragePool_193 | 558 | 0.995896 | | Gemm_195 | transposed_replaced_output | 0.995836 | | Gemm_195_transpose_output_reshape | output | 0.995836 | +-----------------------------------+----------------------------+-------------------+`

Among them:

- NodeName represents the name of the operator.
- TensorName represents the Tensor name of the first output of the operator.
- ConsineSimilarity represents the calculated cosine similarity.

### #Output Consistency Comparison

`+------------+-------------+---------------------+--------------+--------------+ | OutputName | Consistency | Mismatched Elements | Max Abs Diff | Max Rel Diff | +------------+-------------+---------------------+--------------+--------------+ | output | True | 0/1000 (0.00%) | 0.0 | 0.0 | +------------+-------------+---------------------+--------------+--------------+`

Among them:

- OutputName represents the output name.

- Consistency represents whether the output is consistent or not.

- Mismatched Elements represents the number and percentage of inconsistent elements.

- Max Abs Diff represents the maximum absolute error.

- Max Rel Diff represents the maximum relative error.

On This Page