#The hb_verifier Tool

The hb_verifier is a consistency verification tool that supports performing cosine similarity comparisons between ONNX models, between ONNX model and HBIR model, between HBIR model and HBIR model, and output consistency comparisons between HBIR model and HBM model.

The cosine similarity indicates the consistency between quantized models at different stages. As the cosine similarity gets closer to 1, it indicates that the outputs of the two quantized models being compared are closer to each other.

Consistency comparison prints output consistency information for the comparison models, including output name, consistency, number of inconsistent elements, maximum absolute error, and maximum relative error.

#Support Range

• Fbc represents the floating-point HBIR model, which refers to the HBIR model before convert exported after the model conversion by PTQ API method or QAT pipeline.

• Qbc represents the fixed-point HBIR model.

Comparison Models	Comparison Scenarios	Whether the Model is from hb_compile	Input Data Requirement
Onnx vs Onnx	Cosine similarity comparison	Not interested	One original data
Onnx vs Fbc	Cosine similarity comparison	No	Two sets of data corresponding to the Onnx and Fbc models
Onnx vs Qbc	Cosine similarity comparison	Yes	One original data
Onnx vs Qbc	Cosine similarity comparison	No	One original data One runtime data
Fbc vs Qbc	Cosine similarity comparison	No	One original data One runtime data
Qbc vs Hbm	Output consistency comparison	Yes	One runtime data
Qbc vs Hbm	Output consistency comparison	No	One runtime data
Hbm vs Hbm	Output consistency comparison	Not interested	One runtime data

#Usage

#Parameters Introduction

Parameter Name	Parameter Description
-h, --help	Display help information and exit.
--version	Display version information and exit.
-m, --model	Specify the model input, supports ONNX model (*.onnx), HBIR model (*.bc) file, HBM model (*.hbm) file. The two model addresses are separated by a “,”: -m model1,model2.
-i, --input	Specify the data to be used for inference testing; only *.npy files are supported. For multi-input models, there are two ways to pass input data, separated by commas: input_name1:input_data_1.npy,input_name2:input_data_2.npy, … input_data_1.npy,input_data_2.npy… If you need to input data for two different models, please use two -i parameters. Attention: The input data must include preprocessing operations, such as color conversion, mean, scale, etc., which should be done externally when preparing the input. If there are no special requirements, the raw data can use the calibration dataset as input; runtime data needs to be consistent with the input of the quantized model. The order of input data names must match the actual input order of the model.
-c, --compare_digits	Set the numerical precision of the comparison inference result (i.e. the number of decimal places to compare the value), if not specified, the tool will compare to five decimal places by default.
--ip	Set the IP address of the development board, default is empty. Multiple entries need to be separated by commas.
-u, --usename	Set the board SSH user name, defaults to root. Multiple entries need to be separated by commas.
-p, --password	Set the board SSH password, default is empty. Multiple entries need to be separated by commas.
--port	Set the board SSH port, default is 22. Multiple inputs need to be separated by commas.

#Usage Examples

#Cosine Similarity Comparisons

#ONNX Model and ONNX Model

Cosine similarity comparisons were performed between the ONNX model and the ONNX model.

Take the model optimization stage model optimized_float_model.onnx and the model calibration stage model calibrated_model.onnx as an example:

#ONNX Model and HBIR Model

1. Cosine similarity comparisons were performed between the ONNX model and the floating-point HBIR model.

   Take the model optimization stage model optimized_float_model.onnx and the floating-point HBIR model float_model.bc as an example:

   hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_float_model.bc \ -i input1.npy,input2.npy

2. Cosine similarity comparisons were performed between the ONNX model and the fixed-point model HBIR model.

   • If the model is compiled by hb_compile, take the model optimization stage model optimized_float_model.onnx and the model quantization stage fixed-point model quantized_model.bc as an example:

     hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_quantized_model.bc \ -i input.npy

   • If the model is compiled by hb_compile and there are multiple input data, take the model optimization stage model optimized_float_model.onnx and the model quantization stage fixed-point model quantized_model.bc as an example:

     hb_verifier -m googlenet_optimized_float_model.onnx,googlenet_quantized_model.bc \ -i input1.npy,input2.npy

   • If the model is not compiled by hb_compile, take the original floating-model original_float_model.onnx and the fixed-point model quantized_model.bc as an example:

     hb_verifier -m googlenet_original_float_model.onnx,googlenet_quantized_model.bc \ -i input.npy \ -i runtime_input.npy

   • If the model is not compiled by hb_compile and there are multiple input data, take the original floating-model original_float_model.onnx and the fixed-point model quantized_model.bc as an example:

     hb_verifier -m googlenet_original_float_model.onnx,googlenet_quantized_model.bc \ -i input1.npy,input2.npy \ -i runtime_input1.npy,runtime_input2.npy

#HBIR Model and HBIR Model

Cosine similarity comparisons were performed between the HBIR model and the HBIR model.

• If the model is not compiled by hb_compile, take the floating-point model float_model.bc and the fixed-point model quantized_model.bc as an example:

  hb_verifier -m googlenet_float_model.bc,googlenet_quantized_model.bc \ -i input.npy \ -i runtime_input.npy

• If the model is not compiled by hb_compile and there are multiple input data, take the floating-point model float_model.bc and the fixed-point model quantized_model.bc as an example:

  hb_verifier -m googlenet_float_model.bc,googlenet_quantized_model.bc \ -i input1.npy,input2.npy -i runtime_input1.npy,runtime_input2.npy

#Output Consistency Comparisons

#HBIR Model and HBM Model

Output consistency comparisons were performed between the HBIR model and the HBM model.

• Take the model quantization stage fixed-point model quantized_model.bc and the model compilation stage model googlenet.hbm as an example:

  hb_verifier -m googlenet_quantized_model.bc,googlenet.hbm \ -i runtime_input.npy

• Take the model quantization stage fixed-point model quantized_model.bc and the model compilation stage model googlenet.hbm as an example, and setting the ip to None indicates using the simulator:

  hb_verifier -m googlenet_quantized_model.bc,googlenet.hbm \ -i runtime_input.npy \ --ip None,10.10.10.10

#HBM Model and HBM Model

Output consistency comparisons of HBM model across simulator and development board.

Take the model compilation stage model googlenet.hbm as an example. Setting the ip to None indicates using the simulator:

#Output Contents

#Cosine Similarity Comparison

The cosine similarity information for the compared models will be printed within the terminal, as shown in the example below:

Among them:

• NodeName represents the name of the operator.
• TensorName represents the Tensor name of the first output of the operator.
• ConsineSimilarity represents the calculated cosine similarity.

#Output Consistency Comparison

Among them:

• OutputName represents the output name.

• Consistency represents whether the output is consistent or not.

• Mismatched Elements represents the number and percentage of inconsistent elements.

• Max Abs Diff represents the maximum absolute error.

• Max Rel Diff represents the maximum relative error.