#Basic Sample User Guide

The Basic Sample Package is designed to help you familiarize yourself with and learn the interfaces and various advanced functions related to model inference. The Basic Sample Package can also be used as a starting point for new projects to help you quickly build a basic framework and conduct further development and customization on this basis.

The Basic Sample Package provides samples in three aspects:

• Provides a quick start sample based on the resnet50 model. You can experience and develop applications based on this sample to lower the development threshold.

• Provides a sample of using the model inference interface. You can use this sample to better familiarize yourself with the use of various basic functions.

• Provides advanced sample of various special functions. You can choose the appropriate function for model inference according to the actual usage scenario.

#Release Description

The Basic Sample package is located in the samples/ucp_tutorial/dnn/basic_samples/ path of the horizon_s100_open_explorer release package and consists the following main contents:

RELEASE PACKAGE	DESCRIPTIONS
basic_samples	including sample codes and scripts.

Directory of the Basic Sample Package is shown as below:

• The code directory contains source code.

• code/00_quick_start: A quick start sample, based on the model inference interface, using resnet50 to perform single image model inference and result analysis.

• code/01_api_tutorial: Samples of using the model inference interface, including adding/removing padding sample padding, quantization and dequantization samples quanti, memory usage samples mem, model loading and information acquisition samples model.

• code/02_advanced_samples: Advanced samples of model inference, including cropping an image as model input sample crop, the inference sample roi_infer of the model with roi input and the multi-model batch inference sample multi_model_batch.

• The code/build.sh is a application compilation script, and you need to specify the compilation parameters, called by build_aarch64.sh and build_x86.sh.

• The code/build_aarch64.sh is a quick application compilation script, and the compiled application runs in the aarch64 linux environment.

• The code/build_qnx.sh is a quick application compilation script, and the compiled application runs in the aarch64 qnx environment.

• The code/build_x86.sh is a quick application compilation script, and the compiled application runs in the x86 environment.

• The code/resolve.sh is the program running dependency acquisition script, which needs to be executed before the user compiles the model.

• The runtime folder contains scripts to run samples, preset data and related models.

#Environment Configuration

#Prepare the Dev Board

1. After getting the development board, upgrade the system image file to the version recommended by the sample package.

2. Make sure the local development machine and development board can be connected remotely.

#Compilation

Take the following steps to perform the compilation:

1. Follow the instructions in the [Environment Deployment] (../../env_install) section to install the cross-compilation tool.

2. In the ucp_tutorial/dnn/basic_samples/code directory, there is a pre-configured build script build.sh . The options -a x86 , -a aarch64 , and -a aarch64-qnx represent three different architectures. Additionally, the directory also contains three separate build scripts: build_x86.sh , build_aarch64.sh , and build_qnx.sh.

3. After executing the compilation script, the executable programs and dependency files required to run the samples will be generated and copied to the runtime/script or runtime/script_x86 directory.

#How to Use (Basic Samples)

Sample scripts are in the runtime/script folder, and the directory structure after compilation is shown as follows:

Before running, deploy the runtime directory to the board end and execute the corresponding script.

#quick_start

Under 00_quick_start directory, we provide a quick-start sample to introduce the use process of model inference. Taking the resnet50 as an example, it shows the process of model inference for nv12 input and rgb input respectively, including the complete process code from data preparation to model inference, to execution of post-processing, and finally to generating classification results.

The code is mainly divided into six parts:

• Load the model and get the model handle: the main interfaces involved are hbDNNInitializeFromFiles, hbDNNGetModelNameList and hbDNNGetModelHandle.

• Prepare input and output tensors, and complete the dynamic information of the input: the main interfaces involved are hbDNNGetInputCount, hbDNNGetOutputCount, hbDNNGetInputTensorProperties, hbDNNGetOutputTensorProperties, hbDNNGetInputName, hbDNNGetOutputName and hbUCPMallocCached.

• Prepare input data: For nv12 input, the image needs to be converted into color space, and for rgb input, pre-processing from U8 to S8 is required. The interface involved is hbUCPMemFlush.

• Model inference: The main interfaces involved are hbDNNInferV2, hbUCPSubmitTask and hbUCPWaitTaskDone.

• Output data post-processing: The resnet50 model is a classification model, and you can obtain classification results with high confidence through post-processing. Before processing the output, you need to call the hbUCPMemFlush interface for data synchronization.

• Release tasks, memory and models: The main interfaces involved are hbUCPReleaseTask, hbUCPFree and hbDNNRelease.

The directory structure of the script is as follows:

When using it, enter the 00_quick_start directory, and then directly execute sh run_resnet_rgb.sh and sh run_resnet_nv12.sh. After the script is successfully executed, it will print the names of the model input and output and the 5 classification results with the highest confidence. The classification result with the highest confidence number 340 indicates that it is a zebra.

#api_tutorial

The API tutorial refers to the samples in the 01_api_tutorial folder. These samples help developers understand how to use the model inference APIs. This folder contains the following scripts:

#padding

This sample helps you become familiar with how to add padding to model input and how to remove padding from model output. The main data types involved are validShape and stride in hbDNNTensorProperties.

In order to speed up calculations, the model has alignment requirements for input data, see alignment rules for details. This sample is divided into two parts:

• For model input, when additional alignment operations are required for data, use the add_padding method provided in the sample to add padding.

• For model output, when the output contains padding, use the remove_padding method provided in the sample to remove padding to obtain valid data.

When using it, directly enter the 01_api_tutorial directory, and then directly execute sh padding.sh. After the script is successfully executed, it will print the original input and the input data after adding padding, the original output and the output data after removing padding.

#quanti

This sample mainly helps you get familiar with the quantization and dequantization of the model input and output. The data types involved are quantiType, scale, and quantizeAxis in hbDNNTensorProperties.

When the model input contains quantization information, the input floating point number needs to be quantized according to the specified rules, when the model output contains quantization information, the output needs to be dequantized before post-processing the data. For the processing method, please refer to hbDNNQuantiScale.

This sample is mainly divided into four parts:

• Quantization of input per tensor

• Quantization of input per axis

• Dequantization of output per tensor

• Dequantization of output per axis

When using it, just go to the 01_api_tutorial directory and execute sh quanti.sh directly. After the script is successfully executed, each part will print the original input/output and the quantized/dequantized results.

#model

This sample mainly helps you get familiar with the use of interfaces related to model loading and model information acquisition.

The interfaces involved mainly include hbDNNInitializeFromFiles, hbDNNGetModelNameList, hbDNNGetModelHandle, hbDNNGetInputCount, hbDNNGetOutputCount, hbDNNGetInputTensorProperties, hbDNNGetOutputTensorProperties and hbDNNRelease.

When using it, directly enter the 01_api_tutorial directory, and then execute sh model.sh directly. After the script is executed successfully, the basic information of the model will be printed out.

#sys_mem

This sample mainly helps you get familiar with the use of memory interfaces. The main interfaces involved are hbUCPMalloc, hbUCPMallocCached, hbUCPMemFlush and hbUCPFree.

For cacheable memory, the usage steps are as follows:

• Apply for cacheable memory.

• Write data to memory.

• Call the hbUCPMemFlush interface and use the HB_SYS_MEM_CACHE_CLEAN parameter to synchronize data to DDR.

• After model inference or operator calculation, call the hbUCPMemFlush interface and use the HB_SYS_MEM_CACHE_INVALIDATE parameter to synchronize data to the cache.

• Read data for processing.

• Release memory.

When using it, directly enter the 01_api_tutorial directory and execute sh sys_mem.sh.

#advanced_samples

This sample refers to the samples in the 02_advanced_samples directory, which is intended to introduce the use of advanced samples. Its directory contains the following scripts:

#crop

This sample mainly helps you get familiar with how to crop images and use them as model input for inference. The overall process of the sample code is the same as 00_quick_start, and the difference lies mainly in the preparation of input data.

The principle of cropping in this sample is to offset the memory address of the existing image to the upper left corner of the ROI, and mask the redundant part of the image by controlling the size of the stride to prepare the input of the model.

Limitations of sample use:

• Image: The image resolution is required to be large, at least larger than the input of the model, and the image is read into the BPU memory in advance.

• Model: The input validShape of the model is required to be fixed and the stride is dynamic, so that the image can be cropped by controlling the size of the stride.

• Cropping position: Since cropping is to offset the image memory, and the first address of the input memory requires 32 alignment, there is a limit on the size of the offset.

Preparation of input data:

• Image: The size of the zebra image is 376x376. We read the image and convert it into Y and UV input and store them in two BPU memories after 32 alignment. The width of the zebra image after 32 alignment is image_width_stride = 384.

• Model: The model has two inputs

  • y: validShape = (1,224,224,1), stride = (-1,-1, 1,1)

  • uv: validShape = (1,112,112,2), stride = (-1,-1,2,1)

• Cropping: We need to crop a 224x224 area in the image as the input of the model. Selecting (64,50) as the upper left corner point can just get most of the zebra image, and the offset memory address is exactly 32 aligned, which meets the requirements. As shown in the figure below.

• Input tensor preparation:

  • The stride of the Y input is [224 * image_width_stride, image_width_stride, 1, 1], and the coordinates of the upper left corner of the cropped image are [0, 50, 64, 0], then the address offset is 50 * image_width_stride + 64 * 1.

  • The stride of the UV input is [112 * image_width_stride, image_width_stride, 2, 1], and the coordinates of the upper left corner of the cropped image are [0, 25, 32, 0], then the address offset is 25 * image_width_stride + 32 * 2.

When using, enter directly 02_advanced_samples directory, and then directly execute sh run_crop.sh. After the script is successfully executed, the five classification results with the highest confidence will be printed. The classification result with the highest confidence number 340 indicates that it is a zebra.

#multi_model_batch

This sample mainly helps you get familiar with the function of batch processing of small models.

When there are multiple small models running, if each model task runs separately, the scheduling time of the entire framework will be relatively large. In order to avoid performance loss, you can combine multiple small model tasks into one task for inference to reduce the proportion of framework scheduling time.

This sample takes two small models googlenet and resnet50 as an example, and creates/adds tasks by calling the hbDNNInferV2 interface multiple times.

When using it, directly enter the 02_advanced_samples directory and execute sh run_multi_model_batch.sh. After the script is successfully executed, the classification results with the highest confidence of the two models will be printed. The classification number 340 indicates that it is a zebra.

#roi_infer

This sample mainly helps you understand how to prepare data for model inference when there is a roi input.

The overall process of the sample code is the same as 00_quick_start, and the difference lies mainly in the preparation of input data. The sample model is mobilenetv1, and the input is 3 in total, as shown below:

• name: data_y; validShape: (1,-1,-1,1); stride: (-1,-1,1,1);

• name: data_uv; validShape: (1,-1,-1,2); stride: (-1,-1,2,1);

• name: data_roi; validShape: (1,4); stride: (16,4);

data_y and data_uv are dynamic inputs. They are read from the image and converted into Y and UV data for alignment as input, and validShape and stride are completed according to the size of the image. For dynamic input, please refer to Introduction to Dynamic Input.

data_roi is the coordinates of the upper left and lower right corners of the ROI area in the image, in the order of left, upper, right, and lower.

When using it, directly enter the 01_api_tutorial directory, and then execute sh roi_infer.sh directly. After the script is successfully executed, it will print the five classification results with the highest confidence for the two sets of inputs of the model. The classification result with the highest confidence number 340 indicates that it is a zebra.

#Helper Tools (Log)

There are 2 types of logs: sample log and dnn log. Wherein, sample log refers to the log in the Basic Sample Package release package, while dnn log refers to the log in the embedded dnn library. Developers can specify logs according to their own needs.

#Sample Log

The sample log mainly uses hlog, which is mainly divided into 7 levels:

The log level can be set to 0, 1, 2, 3, 4, 5 and 6, corresponding to trace, debug, info, warn, error, critical and never, with the default being info.

#dnn Log

For the configuration of dnn logs, please read the Configuration Info section in the Model Inference API Introduction.