English

Model Inference

The hrt_model_exec infer tool can be used for model inference, where the input data are defined by user and one frame is inferred.

The program runs a single frame of data in a single thread and outputs the time of the model execution.

Support Range

The support range of the model: QAT(Quantized Awareness Training) model, PTQ (Post-Training Quantization) model.

Usage

Usage:
hrt_model_exec [Option...] [Parameter]

[Option]                      [instruction]
---------------------------------------------------------------------------------------------------------------
-h --help                     Display this information
-v --version                  Display this version

[Option]                      [Parameter]
---------------------------------------------------------------------------------------------------------------
--model_file                  [string]: Model file paths, separate by comma, each represents one model file path.
--model_name                  [string]: Model name.
                                        When model_file has one more model and Subcommand is infer or perf,
                                        "model_name" must be specified! 
--input_file                  [string]: Input file paths, separate by comma, each represents one input.
                                        The extension of files should be one of [jpg, JPG, jpeg, JPEG, png, PNG, bin, txt]
                                        bin for binary such as image data, nv12 or yuv444 etc.
                                        txt for plain data such as image info.
--core_id                     [string]: core id, 0 for any core, 1 for core 0, 2 for core 1 and etc, default is 0.
                                        Please confirm the number of bpu cores on the board before setting up.
                                        When you need to specify multiple cores, separate them with commas, such as "1,2".
--frame_count                 [int]   : frame count for run loop, default 200, valid when perf_time is 0 in perf mode; default 1 for infer mode.
--dump_intermediate           [string]: dump intermediate layer input and output. The default is 0. Subcommand must be infer.
--enable_dump                 [bool]  : flag for dump infer input and output. The default is false. Subcommand must be infer.
--dump_precision              [int]   : Output dump precision for float32/float64 in txt file. Default is 9 decimal places.
--dequantize_process          [bool]  : dequantize the model infer output. The default is false.
                                        Subcommand must be infer, enable_dump set as true
--remove_padding_process      [bool]  : remove padding of the model infer output. The default is false.
                                        Subcommand must be infer, enable_dump set as true
--dump_format                 [string]: output dump format, only support [bin, txt]. The default is bin.
                                        Subcommand must be infer.
--dump_txt_axis               [int]   : The txt file of dump is expanded according to the specified axis;
                                        the default is -1, which means there is only one data per line;
                                        Subcommand must be infer, dump_format must be txt.
                                        range:[0, tensor_dimension].
--enable_cls_post_process     [bool]  : flag for classfication post process, only for ptq model now.
                                        Subcommand must be infer.
--dump_path                   [string]: dump file path, --enable_dump or --dump_intermediate will dump model nodes inputs and outputs files.
--input_img_properties        [string]: Specify the color space of the image type input. Each image needs to specify the color space, separated by commas.
                                        The supported color spaces are [Y, UV].
--input_valid_shape           [string]: Complete the validshape of the model input, allowing only the dynamic part to change. Provide two ways to set:
                                        1. This only needs to be set when the validShape of the model input is dynamic. 2. Set for all inputs.
                                        Different inputs are separated by semicolons, and different dimensions are separated by commas.
                                        For example: --input_valid_shape="1,376,376,1;1,188,188,2".
--input_stride                [string]: Complete the stride of the model input, allowing only the dynamic part to change. Provide two ways to set:
                                        1. This only needs to be set when the stride of model input is dynamic. 2. Set for all inputs.
                                        Different inputs are separated by semicolons, and different dimensions are separated by commas.
                                        For example: --input_stride="144384,384,1,1;72192,384,2,1".

[Examples]
---------------------------------------------------------------------------------------------------------------
hrt_model_exec infer              
   --model_file                 
   --model_name                  
   --core_id                    
   --input_file                  
   --input_img_properties        
   --input_valid_shape            
   --input_stride               
   --frame_count                 
   --dump_intermediate            
   --enable_dump                  
   --dump_precision               
   --dequantize_process         
   --dump_path                     
   --remove_padding_process        
   --dump_format                   
   --dump_txt_axis                 
   --enable_cls_post_process

Parameters Introduction

Parameter	Data type	Parameter Description	Correlated Parameters
`-h, --help`	None.	Display the help information.	None.
`-v, --version`	None.	View the version number of the DNN prediction library of the tool.	None.
`infer`	None.	Perform model inference and get model inference results.	This parameter should be used together with `input_file` to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
`model_file`	string	Model file path, multiple paths can be separated by commas.	None.
`model_name`	string	Specify the name of a model.	None.
`input_file`	string	Model input information. The input of the image type, it must have one of the following file name suffixes: `PNG`/`JPG`/`JPEG`/`png`/`jpg`/`jpeg`/`bin`/`txt`. The inputs should be separated by commas `,`, such as `xxx.jpg,input.txt`.	This parameter needs to be used in conjunction with the subcommand `infer`to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
`core_id`	string	Specify the running core, `0` means arbitrary core, `1` means core0, `2` means core1 and etc. Default value is `0`. When you need to specify multiple cores, separate them with commas, such as `"1,2"`.	None.
`input_img_properties`	string	The color space information of the model image input, range [`Y`, `UV`].	This parameter should be used together with `input_file`, each image type input in `input_file` needs to specify a `Y`/`UV` type, and each input color space needs to be separated by an English character comma `,`, such as: `Y,UV`.
`input_valid_shape`	string	Model dynamic `validShape` input information. If the model input attribute `validShape` contains `-1`, the `-1` part needs to be completed, and multiple `validShape` are separated by English semicolons. For example: `--input_valid_shape="1,376,376,1;1,188,188,2"`.	None.
`input_stride`	string	Model dynamic `stride` input information. If the model input attribute `stride` contains `-1`, the `-1` part needs to be completed, and multiple `stride`s are separated by English semicolons. For example:`--input_stride="50176,224,1,1;25088,224,2,1"`.	None.
`frame_count`	int	The number of running frames of the execution model. When the subcommand is `infer`, defaults to `1`. When the subcommand is `perf`, defaults to `200`.	When the subcommand is `perf`, it takes effect when `perf_time` is not set.
`dump_intermediate`	string	dump model each layer of input and output, range [0, 3]. Default value is `0`. When `dump_intermediate=0`, the dump function is turned off by default. When `dump_intermediate=1`, the input and output data of each node layer in the model are saved as `bin`, where inputs and outputs of node are `stride` data. When `dump_intermediate=2`, the input and output data of each node layer in the model are saved as `bin` and `txt`, where the inputs and outputs of node are `stride` data. When `dump_intermediate=3`, the input and output data of each node layer in the model are saved as `bin` and `txt`, where the inputs and outputs of node are `valid` data.	None.
`enable_dump`	bool	Enables dump model input and output. Default value is `false`.	None.
`dump_precision`	int	Controls the number of decimal places of the float type data output in txt format. Default value is `9`.	None.
`dequantize_process`	bool	Inverse quantization of model output. Default value is `false`.	Effective when the `enable_dump` parameter is `true`.
`remove_padding_process`	bool	Remove padding of model output. Default value is `false`.	Effective when the `enable_dump` parameter is `true`.
`dump_format`	string	The format of the dump model input and output, range [`bin`, `txt`]. Default value is `bin`.	None.
`dump_txt_axis`	int	Control line feed rules for txt format input and output. If output dimension = n, then parameter range: [0, n]. Defaults to -1, which means one data per row.	None.
`enable_cls_post_process`	bool	Enables classification post-processing. Default value is `false`.	This parameter needs to be used in conjunction with the subcommand `infer`. Currently, it only supports post-processing of the ptq classification model and printing of classification results.
`dump_path`	string	The path of dump model input and output.	Effective when the `enable_dump` or `dump_intermediate` parameter is set.

Usage Example

Model Inference

Single Model Inference

hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.bin

../aarch64/bin/hrt_model_exec infer --model_file=resnet50_224x224_nv12.hbm --input_file=zebra_cls.jpeg,zebra_cls.jpeg --input_img_properties=Y,UV --input_stride=50176,224,1,1;25088,224,2,1
Load model to DDR cost 1965.03ms.
[I][35143][06-28][10:39:51:373][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,224,224,1), and the image [zebra_cls.jpeg] will be scaled to 224x224
[I][35143][06-28][10:39:51:377][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,112,112,2), and the image [zebra_cls.jpeg] will be scaled to 224x224

---------------------Frame 0 begin---------------------
Infer time: 1.464 ms
---------------------Frame 0 end---------------------

Multiple Models Inference

hrt_model_exec infer --model_file=xxx.hbm,xxx.hbm --model_name=xx --input_file=xxx.jpg

Model Input Instructions

Multi-input Model Description

The tool infer supports inference for multiple input models, supporting image input, binary file input, and text file input, with input data separated by commas. The model input information can be viewed via model_info.

hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.jpg,input.txt

Dynamic input instructions

If the model input is dynamic, you need to use the input_valid_shape and input_stride parameters to complete the dynamic information according to the actual input situation. You can choose to specify the parameters in the following two ways:

Only give the validShape or stride information of the dynamic input.
Give the validShape or stride information of all inputs. The information of non-dynamic inputs must be consistent with the model information.

Note

The tool will automatically complete the dynamic input information to the maximum extent, which is convenient for you to perform performance evaluation more easily. You can choose whether to let the tool automatically complete it according to the actual situation.

If the input stride is dynamic and validShape is fixed, you can not specify the input_stride parameter. The tool will automatically complete it according to the minimum alignment rule and print the alignment information.
If both validShape and stride are dynamic:
- If the input is a picture type, you need to specify the color space information of the picture input_img_properties, and the other parameters can be left unspecified. The tool will fill in the input_valid_shape and input_stride information according to the size of the picture.
- If the input is a type other than a picture or is not specified, you need to set the input_valid_shape information, and the input_stride information will be automatically filled in and printed.

Taking the model in the Dynamic Input Introduction section as an example, you can run the model with the following command:

# Only given dynamically input information 
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2" --input_stride="49280,224,1,1;24640,224,2,1"

# Given all the input information
hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2;1,4" --input_stride="49280,224,1,1;24640,224,2,1;16,4"

Image type input instructions

When input_file is given an image input, you need to use the input_img_properties parameter to specify which color space of the image you want to use as input to the model. Currently, only Y and UV color spaces are supported.

hrt_model_exec infer --model_file=xxx.hbm --input_file="img.jpg,img.jpg,input_roi.bin" --input_img_properties="Y,UV"

On This Page

The hb_compile Tool

QAT

qconfig_setter

Model Export

Horizon Operator

Model Inference API Introduction

Data Type and Data Structure

Function Interface

Model Inference Tool Introduction

The hrt_model_exec Tool Introduction

The hbm_infer Tool Introduction

Data Structure

Functional Interface

Operator Support List

Operator BPU Constraint List

#Model Inference

#Support Range

#Usage

#Parameters Introduction

#Usage Example

#Model Inference

#Single Model Inference

#Multiple Models Inference

#Model Input Instructions

#Multi-input Model Description

#Dynamic input instructions

#Image type input instructions

Model Inference

Support Range

Usage

Parameters Introduction

Usage Example

Model Inference

Single Model Inference

Multiple Models Inference

Model Input Instructions

Multi-input Model Description

Dynamic input instructions

Image type input instructions