#Accuracy Tuning Practice

This chapter introduces the post-training quantization(PTQ) accuracy tuning pipeline using the precision problems encountered in actual use as an example. Please read the Model Accuracy Tuning chapter firstly to Understand relevant theoretical knowledge and tool usage.

Typical accuracy issues include:

1. The all node type int16 quantization accuracy meets the requirements and Accuracy Debug Tool can provide relatively accurate sorting of sensitive nodes;
2. The all node type int16 quantization accuracy meets the requirements, but setting a large number of sensitive nodes to higher precision cannot effectively improve quantization accuracy;
3. The all node type int16 quantization accuracy does not meet the standard, under the premise of full BPU quantization of the model, we hope to further improve the quantization accuracy.

#Sensitive Node Analysis

Accuracy Debug Tool provides an interface for calculating node quantization sensitivity. It can calculate the impact of each operator's quantization on the output results, set nodes with high quantization loss to higher precision, and complete accuracy tuning. The tuning pipeline is described using the HybridNets model as an example.

Using HMCT default INT8 quantization, percentile is selected as the calibration algorithm, the calibration accuracy does not meet the requirements (the accuracy of det and ll_seg decreases by more than 1%):

#All Node Type INT16

First, set all_node_type to INT16 and select percentile for the calibration algorithm. At this point, the calibration accuracy meets the requirements, and we can use INT8+INT16 mixed precision to complete the tuning:

#Mixed Precision Debugging

Compile the INT8 calibration model based on the percentile calibration algorithm selected for INT16 model, configure debug_mode: "dump_calibration_data" in the yaml file to save the calibration data, and output the node quantization sensitivity through get_sensitivity_of_nodes:

Sorting by cosine similarity from front to back, gradually set the operator INT16 quantization, and the calibration model accuracy will increase until it meets the requirements:

From the above test table, we can see that if the sensitive nodes with a sensitivity value less than or equal to 0.99994 are set as INT16 nodes, the calibration accuracy meets the requirements:

A complete accuracy tuning deployment example is available at: HybriNets Accuracy Tuning Deployment Example.

#Sensitive Node Analysis Failure

If using the Accuracy Debug Tool to set sensitive nodes to higher precision fails to effectively improve model accuracy, we can first try specifying output nodes to filter out irrelevant nodes. Additionally, observe the model output error and select other metric to improve the correlation between sensitivity ranking and precision. Furthermore, by analyzing the model structure, typical substructures with a higher risk of quantization loss (such as model outputs, inputs, and structures with specific physical meaning) can be set to higher precision to complete accuracy tuning. This tuning pipeline is described using the YoloP model as an example.

Using HMCT default INT8 quantization, percentile is selected as the calibration algorithm, the calibration accuracy does not meet the requirements (the accuracy of det decreases by more than 1%):

#All Node Type INT16

First, set all_node_type to INT16 and select percentile for the calibration algorithm. At this point, the calibration accuracy meets the requirements, and we can use INT8+INT16 mixed precision to complete the tuning:

#Mixed Precision Debugging

Compile the INT8 calibration model based on the percentile calibration algorithm selected for INT16 model, configure debug_mode: "dump_calibration_data" in the yaml file to save the calibration data, and output the node quantization sensitivity through get_sensitivity_of_nodes:

Sorting by cosine similarity from front to back, gradually set the operator INT16 quantization. However, even with a large number of sensitive nodes set to INT16, the accuracy still failed to meet the requirements:

Observing the accuracy results of the INT8 calibrated model, only det branch does not reach 99%. When calculating nodes sensitivity through get_sensitivity_of_nodes interface, we can use the -o option to specify det output, so that a better sensitivity ranking can be obtained:

Sorting by cosine similarity from front to back, setting the operator to INT16 quantization, and focusing only on the det output can filter out useless nodes, but the final accuracy still does not meet the requirements:

Observe the output similarity of the INT8 calibrated model. The L1 and L2 distances of the det branch deviate greatly from the floating point. Try replacing cosine similarity with other metrics:

When calculating nodes sensitivity through the get_sensitivity_of_nodes interface, we can specify mse as the metric to improve the discrimination between different nodes:

Sort by mse similarity from front to back, set the operator to higher precision, and finally achieve the required accuracy after adding a large number of INT16 nodes:

#Subgraph Structure Analysis

Even if we only focus on improving the det output based on mse metric, setting a large number of sensitive nodes for higher precision still cannot effectively improve the accuracy. Furthermore, considering that only the det task currently does not meet the requirements, it is inferred that quantization loss is less likely to occur in nodes in the da_seg and ll_seg branches, as well as in the shared backbone. Focus on the accuracy tuning of det branch, combined with model structure analysis, try to specify the det output position subgraph to use higher precision and test the calibration accuracy:

A complete accuracy tuning deployment example is available at: YoloP Accuracy Tuning Deployment Example.

#Quantization Loss Compensation

Due to hardware constraints and latency, when set all_node_type to INT16, nodes with int8 precision still exist in model, including weights of Conv and ConvTranspose, Resize, GridSample and the second input of MatMul. PTQ introduces an identical operator to compensate for the quantization loss caused by int8 precision, further improving the model's accuracy with all nodes deployed on BPU, and complete accuracy tuning. The Lane model is used as an example to illustrate the tuning pipeline.

Using HMCT default INT8 quantization, max calibration algorithm with asymmetric and per-channel is selected, the accuracy does not meet the requirements(average cosine similarity of all outputs is less than 0.99):

#All Node Type INT16

First, set all_node_type to INT16 and select max for the calibration algorithm. At this point, the calibration accuracy meets the requirements(average cosine similarity of occlusion and box_arrow outputs does not reach 0.99), further accuracy tuning is required:

#Upper Limit Accuracy of INT16

Since INT8 precision nodes still exist in model after setting all_node_type to INT16, we can modify the qtype of all calibration nodes to INT16 through IR interface provided by HMCT to obtain a true INT16 model:

Verify that the average similarity of the true INT16 calibrated model on all outputs, which can meet requirements, so that accuracy can be improved by compensating quantization error.

#Compensate Quantization Loss

The analysis process of compensation needs to be based on the calibrated model configured with all_node_type int16, nodes sensitivity is output through get_sensitivity_of_nodes:

Then modify the calibrated model according to the nodes sensitivity sorting, and increase the quantization precision from INT8 to INT16 until occlusion and box_arrow meet requirements:

As shown in the table above, increasing the weight quantization precision of Conv_360, Conv_3, and Conv_338 from INT8 to INT16 allows all output similarities to meet requirements. To deploy all nodes on the BPU, a similar operator is introduced to compensate for the quantization loss caused by int8 precision, and the final precision will be improved to close to INT16:

#Mixed Precision Debugging

After compensating for the weight quantization losses of Conv_360, Conv_3, and Conv_338, the accuracy of calibrated model with all_node_type int16 configured can meet requirements. Then attempted to optimize the INT8 calibrated model with error compensated. The accuracy of the INT8 calibrated model is as follows:

Output nodes sensitivity through get_sensitivity_of_nodes:

Sorting by cosine similarity from front to back, gradually setting the operator INT16 quantization, the calibrated model similarity will also increase:

Based on the above table, we optimized model by setting sensitive nodes with a sensitivity value less than or equal to 0.9995 to INT16. Except for cls_arrow and box_arrow, average similarity of all other outputs was no less than 0.99. Observing that cls_arrow and box_arrow share the same branch, we additionally tried configuring arrow output head subgraph to INT16, quantization configuration and output similarity are as follows:

Currently, only the average similarity of the box_arrow does not meet requirements. Specify box_arrow output to regain sensitivity sorting:

Sort by cosine similarity from front to back, and gradually set the operator INT16 quantization until box_arrow meets the requirements:

Finally, by setting some sensitive nodes to INT16, the average similarities of all model outputs meet the requirements. The quantization configuration and output similarity are as follows:

A complete accuracy tuning deployment example is available at: Lane Accuracy Tuning Deployment Example.

#Accuracy Tuning Techniques

The PTQ accuracy tuning pipeline requires constant modification of node configurations, model compilation and accuracy verification. The entire process is time-consuming and expensive to debug. Based on this, we provide the IR interface to support you to directly modify quantization parameters in calibrated_model.onnx for rapid verification.