QNN_TARGET_ARCH when running qnn-model-lib-generator and qnn-context-binary-generator

[xiaoxiaosuaxuan]

I want to run a neural network on the htp backend on Android device. My host device is linux x64.

Following the documentation https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/qnn_tutorial_linux_host_linux_target.html?product=1601111740009302, I first use qnn-model-lib-generator to get a .so model file, then use qnn-context-binary-generator to get the serialized binary file.

According to the documenation, when run qnn-model-lib-generator, the "-t" option should be set to "aarch64-android". But doing so leads the following qnn-context-binary-generator command fails, specifically:

if the backend is set to "${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtp.so", the error is:

0.2ms [ ERROR ] Unable to load backend. dlerror(): /opt/qcom/aistack/qairt/2.31.0.250130/lib/aarch64-android/libQnnHtp.so: cannot open shared object file: No such file or directory
0.2ms [ ERROR ] Failed to load backendLib! 
0.2ms [ ERROR ] Could not load backend : /opt/qcom/aistack/qairt/2.31.0.250130/lib/aarch64-android/libQnnHtp.so!

if the target of the model library is set to "aarch64-android", ther error is:

 27.6ms [ ERROR ] Unable to load model. dlerror(): ./linear_model_lib/aarch64-android/liblinear_model.so: cannot open shared object file: No such file or directory
 27.6ms [ ERROR ] Failed to load ModelLib!
 27.6ms [ ERROR ] Could not load model : ./linear_model_lib/aarch64-android/liblinear_model.so!

It seems that on linux host, qnn-context-binary-generator can't open .so file whose target is "aarch64-android". And if I replace all these files to the "x86_64-linux-clang" version(including set the target of qnn-model-lib-generator to "x86_64-linux-clang", and set the backend when running qnn-context-binary-generator to "x86_64-linux-clang/libQnnHtp.so"), I can successfully generate the context binary file.

However, I still failed to run the context binary file with qnn-net-run on my Android device, with the "initialization failure" or "device creation failure" errors.

[bhushan23]

Hi @xiaoxiaosuaxuan

Your observation of using x86_64-linux-clang is correct during compilation. You should be using libs from host that you are using to compile. Target specified in htp_config handles ahead of time compilation for target device.

However, I still failed to run the context binary file with qnn-net-run on my Android device, with the "initialization failure" or "device creation failure" errors.

Could you please share logs? what error you are getting?
It could be due to not specifying htp_config during compilation. Here's an example of htp_config https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-100/htp_htp.html

[xiaoxiaosuaxuan]

@bhushan23 Thanks for your timely reply!

Now I am able to run the context binary file with htp backend on my device. However, the generated result is incorrect: it produces only zeros. The following is the detailed precedure:

The model is a simple fp16 linear layer (input dim 128, output dim 32), I use the following command to generate model .cpp files from linear_model.onnx:

qnn-onnx-converter \
    --input_network ./linear_model.onnx \
    --float_bitwidth 16 \
    --act_bitwidth 16 \
    --weights_bitwidth 16\
    --output_path ./linear_model_qnn/linear_model.cpp

Then I use the following command to generate the model library on my host:

QNN_TARGET_ARCH=x86_64-linux-clang
model_pth=./linear_model_qnn/linear_model

${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-model-lib-generator \
        -c "${model_pth}.cpp" \
        -b "${model_pth}.bin" \
        -o linear_model_lib \
        -t ${QNN_TARGET_ARCH}

Then I use the following command to generate the context binary on my host:

QNN_TARGET_ARCH=x86_64-linux-clang

cmd="${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-context-binary-generator" \
    --backend  "libQnnHtp.so"\
    --model "./linear_model_lib/${QNN_TARGET_ARCH}/liblinear_model.so" \
    --output_dir "./linear_model_ctx_bin" \
    --binary_file "linear_model_ctx_fp16" \
    --config_file "./configs_ctx_bin/backend_ext.json" \

• The content of backend_ext.json is :

  {
    "backend_extensions": {
        "shared_library_path": "libQnnHtpNetRunExtensions.so", 
        "config_file_path": "./configs_ctx_bin/htp_backend_config.json"         
    }

• The content of htp_backend_config.sjon is:

   {
   "devices": [
     {
       "cores":[{
         "perf_profile": "burst",
         "rpc_control_latency": 100
       }]
     }
   ]
 }

Finally, I create a bundle directory with all related files and push it to my Android device, and run the network using the following command:

  export LD_LIBRARY_PATH=./lib
  export ADSP_LIBRARY_PATH=./lib
  
  chmod +x ./qnn-net-run 
  
  ./qnn-net-run \
     --backend "libQnnHtp.so" \
     --input_list "./input_list.txt" \
     --retrieve_context "./linear_model_ctx_fp16.bin" \
     --output_dir "./output"

The network finish execution successfully, but the preduced result contains all zeros.

[quic-bhushans]

Hi @xiaoxiaosuaxuan could you please use qairt-converter instead of qnn-onnx-converter?

qairt-converter is new path from QNN team to convert onnx models.
one reason for all zero output could be passing act_bitwidth and other flags which triggers qairt-quantizer

since you already have quantized aimet model, you don't need to invoke qairt-quantizer.