English

Eager Mode

The horizon_plugin_pytorch currently supports quantization using the eager mode, however, we no longer recommend using this mode. For quantization in eager mode, currently you are still supported by our horizon_plugin_pytorch. The overall flow of the Eager mode is also based on PyTorch's quantization interface and ideas, so we recommend that you read the Eager mode section of the PyTorch official document.

Difference from fx mode

The main differences between using eager mode and fx mode in horizon_plugin_pytorch are:

The eager mode only supports module operators. Before carryout model quantization, you need to manually replace the functional operators in the floating-point model with Module-type operators in PyTorch or proprietary operators defined in horizon_plugin_pytorch, including but not limited to:

Original floating-point operator	Operator to be replaced
torch.nn.functional.relu	torch.nn.ReLU()
a + b / torch.add	horizon.nn.quantized.FloatFunctional().add
Tensor.exp	horizon.nn.Exp()
torch.nn.functional.interpolate	horizon.nn.Interpolate()

You must manually define the operators to be fused and explicitly call the fusion function before carryout model quantization, and also specify the use of the fuser_func provided in horizon_plugin_pytorch when calling it. As shown below:

import torch
from torch import nn
import horizon_plugin_pytorch as horizon


class ConvBNReLU(nn.Sequential):
    def __init__(self, in_channels, out_channels, kernel_size):
        super(ConvBNReLU, self).__init__(
            nn.Conv2d(
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=kernel_size
            ),
            nn.BatchNorm2d(num_features=out_channels),
            nn.ReLU()
        )

    # specify the operators that can be fuse
    def fuse_model(self):
        torch.quantization.fuse_modules(
            self,
            ['0', '1', '2'],
            inplace=True,
            # specify the fuse function provided in horizon_plugin_pytorch
            fuser_func=horizon.quantization.fuse_known_modules,
        )

float_model = ConvBNReLU(1, 1, 1)
# call the fuse function explicitly
float_model.fuse_model()

print(float_model)
# ConvBNReLU(
#   (0): ConvReLU2d(
#     (0): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))
#     (1): ReLU()
#   )
#   (1): Identity()
#   (2): Identity()
# )

The eager mode cannot use QconfigSetter because there is no computation graph, and the qconfig can only be configured by setting attributes.

model.qconfig = QConfig(...)

On This Page

PTQ Conversion Tools

The hb_compile Tool

PTQ Conversion Steps

PTQ Conversion Samples

FAQ and Common Failure Resolutions

Appendix

Tutorial

Advanced Tutorial

API Reference

QAT

qconfig_setter

Model Export

Horizon Operator

FAQ and Common Failure Resolutions

Model Inference Development

Model Inference API Introduction

Data Type and Data Structure

Function Interface

Model Inference Tool Introduction

The hrt_model_exec Tool Introduction

The hbm_infer Tool Introduction

UCP Common API Introduction

Data Structure

Functional Interface

UCP Performance Analysis Tools

FAQ and Error code

Model Deployment Principle and Process

Model Deployment Practice Guidance Examples

HMCT API Reference

Toolchain Operator Support Constraint List

Operator Support List

Operator BPU Constraint List

Community Quality Articles

Eager Mode

Difference from fx mode

The hb_compile Tool

QAT

qconfig_setter

Model Export

Horizon Operator

Model Inference API Introduction

Data Type and Data Structure

Function Interface

Model Inference Tool Introduction

The hrt_model_exec Tool Introduction

The hbm_infer Tool Introduction

Data Structure

Functional Interface

Operator Support List

Operator BPU Constraint List

#Eager Mode

#Difference from fx mode

Eager Mode

Difference from fx mode