User Guide

Disclaimer

This software is provided “as is,” without any express or implied warranties. The authors and contributors shall not be held liable for any damages arising from its use. The code may be incomplete or insufficiently tested. Users are solely responsible for evaluating its suitability and assume all associated risks.
Note: Contributions are welcome. Please ensure thorough testing before deploying in critical systems.

Note

Native Mode:

To achieve optimal performance for model data preprocessing, we should try to use the "native" mode:
- When initializing the model, set the input and output data types to "native" (the default is "float");
- When passing data to the QAI AppBuilder inference function(Inference), provide the format required by the model itself;
- You can obtain the required input and output data format types of the model via the getInputDataType() and getOutputDataType() functions;
- For specific usage, please refer to the Wisper example code.

Supported model's file format:

.bin & .dlc & .so & .onnx.

Environment Setup

** For Python developers, from v2.0.0, we don't need to prepare the below libraries since we've included these libraries into QAI AppBuilder extension(*.whl).

Qualcomm® AI Runtime SDK

Qualcomm® AI Runtime SDK is designed to provide unified, low-level APIs for AI development. It can be downloaded from Qualcomm software center:
https://softwarecenter.qualcomm.com/#/catalog/item/Qualcomm_AI_Runtime_SDK
Or from QPM [this option expected to be deprecated soon]
https://qpm.qualcomm.com/#/main/tools/details/Qualcomm_AI_Runtime_SDK

1. Libraries from Qualcomm® AI Runtime SDK:

We need below libraries from Qualcomm® AI Runtime SDK for using AppBuilder on Snapdragon X Elite(Windows on Snapdragon device).

Note: For X64 and ARM64EC program, use the libraries under the folder 'arm64x-windows-msvc'; for ARM64 program, use the libraries under the folder 'aarch64-windows-msvc'.

We have 2 options to get these runtime libraries:

Download QAIRT Runtime and extract the dependency libraries to app folder.
Install Qualcomm® AI Runtime SDK and copy the dependency libraries to app folder.

If use x64 Python, use the libraries below from Qualcomm® AI Runtime SDK:

C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\arm64x-windows-msvc\QnnHtp.dll  (backend for running model on HTP)
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\arm64x-windows-msvc\QnnCpu.dll  (backend for running model on CPU)
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\arm64x-windows-msvc\QnnHtpPrepare.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\arm64x-windows-msvc\QnnSystem.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\arm64x-windows-msvc\QnnHtpV73Stub.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\hexagon-v73\unsigned\libQnnHtpV73Skel.so
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\hexagon-v73\unsigned\libqnnhtpv73.cat

If use ARM64 Python, use the libraries below from Qualcomm® AI Runtime SDK(ARM64 Python has better performance in Snapdragon X Elite platform):

C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\aarch64-windows-msvc\QnnHtp.dll  (backend for running model on HTP)
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\aarch64-windows-msvc\QnnCpu.dll  (backend for running model on CPU)
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\aarch64-windows-msvc\QnnHtpPrepare.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\aarch64-windows-msvc\QnnSystem.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\aarch64-windows-msvc\QnnHtpV73Stub.dll
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\hexagon-v73\unsigned\libQnnHtpV73Skel.so
C:\Qualcomm\AIStack\QAIRT\{SDK Version}\lib\hexagon-v73\unsigned\libqnnhtpv73.cat

We can copy these libraries to one folder. E.g.: C:\<Project Name>\qnn\

python.md can help setup the x64 Python environment automatically.
In WoS platform, ARM64 Python has better performance, but some Python extensions don't work for ARM64 Python today. For detailed help information on how to setup environment for using ARM64 Python, you can refer to python_arm64.md

2. API from AppBuilder Python binding extension for Python projects.

There're several Python classes from this extension:

QNNContext - The context of QNN model, used to initialize the QNN model, run the inference and destroy the model resource.
QNNContextProc - It's similar with QNNContext but support load the model into a separate processes.
QNNShareMemory - It's used to create processes share memory while using QNNContextProc.
QNNConfig - It's for configuring Qualcomm® AI Runtime SDK libraries path, runtime(CPU/HTP), log leverl, profiling level.
PerfProfile - Set the HTP perf profile.
OnnxRuntimeContext - A backend for running .onnx model files via onnxruntime_qnn (QNN HTP Execution Provider).
More Apis: getInputShapes()/getInputDataType()/getInputName()/getOutputShapes()/getOutputDataType()/getOutputName()/getGraphName(), refer to usage sample codes in real_esrgan_x4plus.py.

Notes:

a. Plese use the API LogLevel.SetLogLevel() for Python project to initialize the log function before you call any other APIs. b. Refer to Python sample code:
https://github.com/qualcomm/qai-appbuilder/tree/main/samples/python

Sample Code(Python)

from qai_appbuilder import (QNNContext, Runtime, LogLevel, ProfilingLevel, PerfProfile, QNNConfig)
import os

execution_ws = os.getcwd()
des_dir = execution_ws + "\\qnn"

def SetQNNConfig():
    QNNConfig.Config(Runtime.HTP, LogLevel.DEBUG, ProfilingLevel.BASIC)

class TextEncoder(QNNContext):
    #@timer
    def Inference(self, input_data):
        input_datas=[input_data]
        output_data = super().Inference(input_datas)[0]

        # Output of Text encoder should be of shape (1, 77, 768)
        output_data = output_data.reshape((1, 77, 768))
        return output_data

SetQNNConfig()  # We need to call this function to configure the basic environment before using any other AppBuilder functions.

model_text_encoder  = "text_encoder"
text_encoder_model = "models\\text_encoder_quantized.serialized.v73.bin"

# Instance for TextEncoder 
text_encoder = TextEncoder(model_text_encoder, text_encoder_model)

# Burst the HTP before inference.
PerfProfile.SetPerfProfileGlobal(PerfProfile.BURST)

# Run the inference of text encoder on tokens.
user_text_embedding = text_encoder.Inference(tokens)

# Release the HTP.
PerfProfile.RelPerfProfileGlobal()

# Destroy the model and free the memory resource.
del(text_encoder)

3. API from 'libappbuilder.dll' for C++ projects.

bool LibAppBuilder::ModelInitialize(...)

std::string model_name: Model name such as "unet", "text_encoder", "controlnet_canny". Model name must be unique for different model files.
std::string proc_name: This is an optional parameter, needed just when you want the model to be executed in a separate process. If use process name, this model will be loaded in 'QAIAppSvc' service process. One service process can load many models. There can be many service processes.
std::string model_path: The path of model.
std::string backend_lib_path: The path of 'QnnHtp.dll'
std::string system_lib_path: The path of 'QnnSystem.dll'

bool LibAppBuilder::ModelInference(...)

std::string model_name: Model name used in 'ModelInference'.
std::string proc_name: Process name used in 'ModelInference'. This is an optional parameter, needed just when you want the model to be executed in a separate process.
std::string share_memory_name: Share memory name used in 'CreateShareMemory'. This is an optional parameter, use it with 'proc_name' together.
std::vector<uint8_t>& inputBuffers*: All input data required for the model.
std::vector<size_t>& inputSize: The size of input data in 'inputBuffers'. This is an optional parameter, use it with 'proc_name' together.
std::vector<uint8_t>& outputBuffers*: Used to save all the output data of the model.
std::vector<size_t>& outputSize: The size of output data in 'outputBuffers'.

bool LibAppBuilder::ModelDestroy(...)

bool LibAppBuilder::CreateShareMemory(...)

std::string share_memory_name: Share memory name. This share memory will be used to store model input & output data.
size_t share_memory_size: The one with the larger memory size of the model input and output data. For example: total size of model input data size is 10M, out put data size is 16M, we can set 'share_memory_size' to 16M.

bool LibAppBuilder::DeleteShareMemory(...)

std::string share_memory_name: Share memory name.

Helper function for printing log:

bool SetLogLevel(int32_t log_level)
void QNN_ERR(const char* fmt, ...)
void QNN_WAR(const char* fmt, ...)
void QNN_INF(const char* fmt, ...)
void QNN_VEB(const char* fmt, ...)
void QNN_DBG(const char* fmt, ...)

log_level:

    QNN_LOG_LEVEL_ERROR   = 1 
    QNN_LOG_LEVEL_WARN    = 2 
    QNN_LOG_LEVEL_INFO    = 3 
    QNN_LOG_LEVEL_VERBOSE = 4 
    QNN_LOG_LEVEL_DEBUG   = 5

Notes:

a. For C++(Visual Studio) projects, you need to set 'Runtime Library' to 'Multi-threaded DLL (/MD)'. Please refer to below link for detailed information: https://learn.microsoft.com/en-us/cpp/build/reference/md-mt-ld-use-run-time-library?view=msvc-170

b. Plese use the API SetLogLevel() for C++ project to initialize the log function before you call any other APIs.

c. Refer to C++ sample code:
https://github.com/qualcomm/qai-appbuilder/tree/main/samples/c%2B%2B

Sample Code(C++: load model to local process.)

#include "LibAppBuilder.hpp"

LibAppBuilder libAppBuilder;
SetLogLevel(2);
std::vector<uint8_t*> inputBuffers;
std::vector<uint8_t*> outputBuffers;
std::vector<size_t> outputSize;

libAppBuilder.ModelInitialize(model_name, model_path, backend_lib_path, system_lib_path);

// Fill the inputBuffers before inference.
... 

libAppBuilder.ModelInference(model_name, inputBuffers, outputBuffers, outputSize);

// Use the data in outputBuffers.
... 

// Free the memory in outputBuffers.
for (int j = 0; j < outputBuffers.size(); j++) {
    free(outputBuffers[j]);
}
outputBuffers.clear();
outputSize.clear();

libAppBuilder.ModelDestroy(model_name);

DLC Support

load .bin model file after covert .dlc file, refer to dlc2bin.

load .dlc model file directly from QAIRT version 2.41.0.251128 onwards.

You only need to put the .dlc model file at the same folder of your .bin file, and modify corresponding model path in your app. A new .dlc.bin file will be generated after load and run .dlc model at the first time, then will load that new .dlc.bin file when run your app later to save time. Usage sample codes can refer to real_esrgan_x4plus.py.

ONNX Support

From PR #126, QAI AppBuilder supports running .onnx model files directly via the new OnnxRuntimeContext class, which uses onnxruntime_qnn (the QNN HTP Execution Provider for ONNX Runtime) to accelerate inference on the Snapdragon NPU (HTP).

Overview

Feature	Details
Model format	Standard `.onnx` (FP32 or FP16)
Backend	`onnxruntime_qnn` — QNN HTP Execution Provider
Fallback	Automatically falls back to `CPUExecutionProvider` if QNN EP is unavailable
Platform	Windows on Snapdragon (WoS, ARM64); also works on x86 with CPU fallback
API class	`OnnxRuntimeContext` (from `qai_appbuilder`)

Prerequisites

Install onnxruntime_qnn (the QNN-enabled ONNX Runtime package for Snapdragon):

pip install onnxruntime_qnn

Note: onnxruntime_qnn is a separate package from the standard onnxruntime. It bundles the QNN Execution Provider and the required QNN HTP backend libraries. It is available for ARM64 (WoS) and x64 Windows.

API: OnnxRuntimeContext

OnnxRuntimeContext provides the same interface as QNNContext for metadata queries and inference, so existing code patterns work with minimal changes.

Constructor

OnnxRuntimeContext(model_name: str, model_path: str, use_cpu: bool = False)

Parameter	Description
`model_name`	A unique name string for this model instance
`model_path`	Path to the `.onnx` model file
`use_cpu`	If `True`, forces `CPUExecutionProvider` (skips QNN HTP EP); default `False`

Sample scripts using OnnxRuntimeContext

yolov8_det-onnx.py — YOLOv8 object detection via ONNX Runtime QNN EP
real_esrgan_x4plus.py — Real-ESRGAN super-resolution with --onnx flag (auto-generates FP16 ONNX from PyTorch weights)

Navigation Menu

Repository navigation

user_guide.md

user_guide.md

User Guide

Disclaimer

Note

Native Mode:

Supported model's file format:

Environment Setup

Qualcomm® AI Runtime SDK

1. Libraries from Qualcomm® AI Runtime SDK:

2. API from AppBuilder Python binding extension for Python projects.

Notes:

Sample Code(Python)

3. API from 'libappbuilder.dll' for C++ projects.

bool LibAppBuilder::ModelInitialize(...)

bool LibAppBuilder::ModelInference(...)

bool LibAppBuilder::ModelDestroy(...)

bool LibAppBuilder::CreateShareMemory(...)

bool LibAppBuilder::DeleteShareMemory(...)

Helper function for printing log:

Notes:

Sample Code(C++: load model to local process.)

DLC Support

load .bin model file after covert .dlc file, refer to dlc2bin.

load .dlc model file directly from QAIRT version 2.41.0.251128 onwards.

ONNX Support

Overview

Prerequisites

API: OnnxRuntimeContext

Constructor

Sample scripts using OnnxRuntimeContext

FilesExpand file tree

user_guide.md

Latest commit

History

user_guide.md

File metadata and controls

User Guide

Disclaimer

Note

Native Mode:

Supported model's file format:

Environment Setup

Qualcomm® AI Runtime SDK

1. Libraries from Qualcomm® AI Runtime SDK:

2. API from AppBuilder Python binding extension for Python projects.

Notes:

Sample Code(Python)

3. API from 'libappbuilder.dll' for C++ projects.

bool LibAppBuilder::ModelInitialize(...)

bool LibAppBuilder::ModelInference(...)

bool LibAppBuilder::ModelDestroy(...)

bool LibAppBuilder::CreateShareMemory(...)

bool LibAppBuilder::DeleteShareMemory(...)

Helper function for printing log:

Notes:

Sample Code(C++: load model to local process.)

DLC Support

load .bin model file after covert .dlc file, refer to dlc2bin.

load .dlc model file directly from QAIRT version 2.41.0.251128 onwards.

ONNX Support

Overview

Prerequisites

API: OnnxRuntimeContext

Constructor

Sample scripts using OnnxRuntimeContext