QAT_HW does not work in OpenSSL-3.0.16.

[Taeil-Nam]

Hi team,

• Version info

  • OS: Ubuntu 18.04 (Kernel version : 4.15.0-243-generic)
  • OpenSSL: 3.0.16
  • QAT Driver: QAT.L.4.27.0-00006
  • QAT Engine: v1.8.1

• symptoms

  • QAT_HW is not operate in OpenSSL-3.0.16 with error below:

[DEBUG][329005.373283] PID [5509] Thread [7f63a7964780][qat_hw_ecx.c:194:qat_pkey_ecx_keygen()] QAT HW ECX Started
[DEBUG][329005.373293] PID [5509] Thread [7f63a7964780][qat_hw_ecx.c:259:qat_pkey_ecx_keygen()] EVP_PKEY_X25519
[WARN][329005.373297] PID [5509] Thread [7f63a7964780][qat_hw_init.c:314:get_instance()] No working instance is available
[DEBUG][329005.373301] PID [5509] Thread [7f63a7964780][qat_hw_init.c:318:get_instance()] inst type: ASYM, inst_num = -1
[WARN][329005.373306] PID [5509] Thread [7f63a7964780][qat_hw_ecx.c:301:qat_pkey_ecx_keygen()] Failed to get an instance

When I built QAT Engine, I configured as below:

./configure \
--with-qat_hw_dir=/mydirectory/ssl_qat/qat/ \
--with-openssl_dir=/mydirectory/openssl-3.0.16/ \
--with-openssl_install_dir=/mydirectory/ssl_qat \
--enable-qat_small_pkt_offload \
--disable-qat_hw_ciphers \
--disable-qat_hw_dh \
--disable-qat_hw_ecdh \
--disable-qat_hw_prf \
--disable-qat_hw_lenstra_protection \
--disable-qat_hw_dsa \
--disable-qat_hw_ecdsa \
--enable-qat_hw_rsa \
--enable-qat_provider \
--enable-qat_warnings \
--enable-qat_debug \
--with-qat_debug_file=/tmp/qat.log

Following is openssl info.

root# openssl version -a
QAT_HW RSA for Provider Enabled
QAT_HW ECX25519 for Provider Enabled
QAT_HW ECX448 for Provider Enabled
OpenSSL 3.0.16 11 Feb 2025 (Library: OpenSSL 3.0.16 11 Feb 2025)
built on: Tue Feb  3 00:24:26 2026 UTC
platform: linux-x86_64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
OPENSSLDIR: "/mydirectory/ssl_qat/ssl"
ENGINESDIR: "/mydirectory/ssl_qat/lib/engines-3"
MODULESDIR: "/mydirectory/ssl_qat/lib/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x21cbfbb

root# openssl list -providers
QAT_HW RSA for Provider Enabled
QAT_HW ECX25519 for Provider Enabled
QAT_HW ECX448 for Provider Enabled
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.16
    status: active
  qatprovider
    name: QAT Provider for QAT_HW
    version: v1.8.1
    status: active

I attached the qat_debug file and dh895xcc_dev0.conf file.
qat_debug.txt
dh895xcc_dev0.conf.txt

I don't know the cause of this symptoms..
Could you please let me know the solution?

Thank you.

[venkatesh6911]

@Taeil-Nam Please paste the output of adf_ctl status
From the debug output, it looks the QAT Engine application got the asymc instance with first process but could not find for the second. Are you running two userspace processes ?

[Taeil-Nam]

Hi @venkatesh6911 ,

Here is the output of adf_ctl status below:

root# ./adf_ctl status
Checking status of all devices.
There is 1 QAT acceleration device(s) in the system:
 qat_dev0 - type: dh895xcc,  inst_id: 0,  node_id: 1,  bsf: 0000:83:00.0,  #accel: 6 #engines: 12 state: up

I'm running 12 userspace processes and each process load the QAT provider.

[venkatesh6911]

Can you increase the NumProcesses from 1 to 12 in the device config file (dh895xcc_dev0.conf) ?

[SHIM]
NumberCyInstances = 1
NumberDcInstances = 0
NumProcesses = 12
LimitDevAccess = 1

Please restart the adf_ctl service after the config file update with adf_ctl restart

[Taeil-Nam]

I modified device config file (dh895xcc_dev0.conf) as you requested.

root# cat /etc/dh895xcc_dev0.conf | grep -A 5 -B 3 SHIM
##############################################
# User Process Instance Section
##############################################
[SHIM]
NumberCyInstances = 1
NumberDcInstances = 0
NumProcesses = 12
LimitDevAccess = 1

root# ./adf_ctl restart
Restarting all devices.
Processing /etc/dh895xcc_dev0.conf

root# ./adf_ctl status
Checking status of all devices.
There is 1 QAT acceleration device(s) in the system:
 qat_dev0 - type: dh895xcc,  inst_id: 0,  node_id: 1,  bsf: 0000:83:00.0,  #accel: 6 #engines: 12 state: up

I restarted my program.

root# ps -ef | grep proxy
root     17639     1  0 14:59 pts/5    00:00:00 ./ssl_proxy
root     17650 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17651 17639  5 14:59 pts/5    00:00:01 ./ssl_proxy
root     17652 17639  5 14:59 pts/5    00:00:01 ./ssl_proxy
root     17653 17639  5 14:59 pts/5    00:00:01 ./ssl_proxy
root     17654 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17655 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17656 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17657 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17658 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17659 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17660 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy
root     17661 17639  6 14:59 pts/5    00:00:01 ./ssl_proxy

But same symptoms occur as below:

root# cat /tmp/qat.log
[WARN][14923.266420] PID [17659] Thread [7f7caa37c780][e_qat.c:1126:bind_qat()] QAT Warnings enabled.
[DEBUG][14923.266432] PID [17659] Thread [7f7caa37c780][e_qat.c:1127:bind_qat()] QAT Debug enabled.
[WARN][14923.266435] PID [17659] Thread [7f7caa37c780][e_qat.c:1128:bind_qat()] (null) - Reference implementation of QAT crypto engine(qat_hw) v1.8.1
[DEBUG][14923.266925] PID [17659] Thread [7f7caa37c780][e_qat.c:1145:bind_qat()] QAT HW device available
[DEBUG][14923.266968] PID [17659] Thread [7f7caa37c780][qat_evp.c:964:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_128 registration succeeded
[DEBUG][14923.266973] PID [17659] Thread [7f7caa37c780][qat_evp.c:964:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_192 registration succeeded
[DEBUG][14923.266977] PID [17659] Thread [7f7caa37c780][qat_evp.c:976:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_256 registration succeeded
[DEBUG][14923.266986] PID [17659] Thread [7f7caa37c780][e_qat.c:615:qat_engine_init()] QAT Engine initialization:
[DEBUG][14923.266990] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:507:qat_hw_init()] QAT_HW initialization:
[DEBUG][14923.266993] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:508:qat_hw_init()] - External polling: OFF
[DEBUG][14923.266996] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:509:qat_hw_init()] - Heuristic polling: OFF
[DEBUG][14923.266999] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:510:qat_hw_init()] - SW Fallback: OFF
[DEBUG][14923.267002] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:511:qat_hw_init()] - Inline polling: OFF
[DEBUG][14923.267005] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:512:qat_hw_init()] - Internal poll interval: 100000ns
[DEBUG][14923.267008] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:513:qat_hw_init()] - Epoll timeout: 1000ms
[DEBUG][14923.267011] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:514:qat_hw_init()] - Event driven polling mode: OFF
[DEBUG][14923.267014] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:515:qat_hw_init()] - Instance for thread: OFF
[DEBUG][14923.267017] PID [17659] Thread [7f7caa37c780][qat_hw_init.c:516:qat_hw_init()] - Max retry count: 5
[DEBUG][14923.324409] PID [17660] Thread [7f7caa37c780][qat_hw_init.c:553:qat_hw_init()] Found 1 Cy instances
[DEBUG][14923.324459] PID [17660] Thread [7f7caa37c780][qat_hw_init.c:672:qat_hw_init()] Started Instance No: 0 Located on Device: 0
Asymmetric instances sequence: 0
Symmetric instances sequence: 0
[DEBUG][14923.324468] PID [17660] Thread [7f7caa37c780][qat_hw_init.c:696:qat_hw_init()] Instances 1, SVM instances 0, contig instances 1
Asym sequence for thread mapping: 0
Sym sequence for thread mapping: 0
[DEBUG][14923.324545] PID [17660] Thread [7f7ca55ff700][qat_hw_polling.c:137:qat_timer_poll_func()] timer_poll_func started
[DEBUG][14923.340015] PID [17658] Thread [7f7ca55ff700][qat_hw_polling.c:149:qat_timer_poll_func()] qat_timer_poll_func_thread = 0x7f7ca55ff700
[DEBUG][14924.583018] PID [17656] Thread [7f7caa37c780][e_qat.c:663:qat_engine_finish_int()] ---- QAT Engine Finishing...

[DEBUG][14924.583029] PID [17656] Thread [7f7caa37c780][e_qat.c:665:qat_engine_finish_int()] RSA Priv retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583039] PID [17656] Thread [7f7caa37c780][e_qat.c:667:qat_engine_finish_int()] RSA Pub retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583042] PID [17656] Thread [7f7caa37c780][e_qat.c:670:qat_engine_finish_int()] ECDH keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583046] PID [17656] Thread [7f7caa37c780][e_qat.c:673:qat_engine_finish_int()] ECDH derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583049] PID [17656] Thread [7f7caa37c780][e_qat.c:675:qat_engine_finish_int()] ECX keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583053] PID [17656] Thread [7f7caa37c780][e_qat.c:677:qat_engine_finish_int()] ECX derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583056] PID [17656] Thread [7f7caa37c780][e_qat.c:679:qat_engine_finish_int()] ECDSA sign retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583059] PID [17656] Thread [7f7caa37c780][e_qat.c:682:qat_engine_finish_int()] SM4-CBC retries: 0, HW requests: 0, SW requests: 0
[DEBUG][14924.583064] PID [17656] Thread [7f7caa37c780][qat_hw_init.c:801:qat_hw_finish_int()] ---- QAT Finishing...

[DEBUG][14924.583235] PID [17656] Thread [7f7ca55ff700][qat_hw_polling.c:215:qat_timer_poll_func()] timer_poll_func finishing - pid = 17656

[DEBUG][14924.586527] PID [17656] Thread [7f7caa37c780][qat_hw_init.c:901:qat_hw_finish_int()] Calling pthread_key_delete()
[DEBUG][14924.587473] PID [17707] Thread [7f7caa37c780][e_qat.c:615:qat_engine_init()] QAT Engine initialization:
[DEBUG][14924.587510] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:507:qat_hw_init()] QAT_HW initialization:
[DEBUG][14924.587517] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:508:qat_hw_init()] - External polling: OFF
[DEBUG][14924.587520] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:509:qat_hw_init()] - Heuristic polling: OFF
[DEBUG][14924.587523] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:510:qat_hw_init()] - SW Fallback: OFF
[DEBUG][14924.587527] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:511:qat_hw_init()] - Inline polling: OFF
[DEBUG][14924.587530] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:512:qat_hw_init()] - Internal poll interval: 100000ns
[DEBUG][14924.587533] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:513:qat_hw_init()] - Epoll timeout: 1000ms
[DEBUG][14924.587536] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:514:qat_hw_init()] - Event driven polling mode: OFF
[DEBUG][14924.587539] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:515:qat_hw_init()] - Instance for thread: OFF
[DEBUG][14924.587542] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:516:qat_hw_init()] - Max retry count: 5
[DEBUG][14924.641070] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:553:qat_hw_init()] Found 1 Cy instances
[DEBUG][14924.641179] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:672:qat_hw_init()] Started Instance No: 0 Located on Device: 0
Asymmetric instances sequence: 0
Symmetric instances sequence: 0
[DEBUG][14924.641195] PID [17707] Thread [7f7caa37c780][qat_hw_init.c:696:qat_hw_init()] Instances 1, SVM instances 0, contig instances 1
Asym sequence for thread mapping: 0
Sym sequence for thread mapping: 0
[DEBUG][14924.641511] PID [17707] Thread [7f7ca55ff700][qat_hw_polling.c:137:qat_timer_poll_func()] timer_poll_func started
[DEBUG][15038.640740] PID [17661] Thread [7f7caa37c780][qat_hw_ecx.c:194:qat_pkey_ecx_keygen()] QAT HW ECX Started
[DEBUG][15038.640749] PID [17661] Thread [7f7caa37c780][qat_hw_ecx.c:259:qat_pkey_ecx_keygen()] EVP_PKEY_X25519
[WARN][15038.640754] PID [17661] Thread [7f7caa37c780][qat_hw_init.c:314:get_instance()] No working instance is available
[DEBUG][15038.640757] PID [17661] Thread [7f7caa37c780][qat_hw_init.c:318:get_instance()] inst type: ASYM, inst_num = -1
[WARN][15038.640761] PID [17661] Thread [7f7caa37c780][qat_hw_ecx.c:301:qat_pkey_ecx_keygen()] Failed to get an instance
...

[Taeil-Nam]

I found some QAT logs on dmesg -T as below:

[Fri Feb  6 14:58:59 2026] QAT: Stopping all acceleration devices.
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: qat_dev0 stopped 12 acceleration engines
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: Disable arbiter.
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: Resetting device qat_dev0
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: Secondary bus reset
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: Parameter 'StorageEnabled' has been deprecated
[Fri Feb  6 14:58:59 2026] dh895xcc 0000:83:00.0: Starting acceleration device qat_dev0.
[Fri Feb  6 14:59:00 2026] dh895xcc 0000:83:00.0: qat_dev0 started 12 acceleration engines
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17684 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17687 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17689 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17688 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17690 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17691 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17692 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17693 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17694 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17695 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17696 ssl_proxy exit with orphan rings
[Fri Feb  6 14:59:09 2026] dh895xcc 0000:83:00.0: Process 17707 ssl_proxy exit with orphan rings

As we can see from the QAT debug output in my previous comment and the logs above, it seems that process [17707] is related to these symptoms.

[venkatesh6911]

Can you run an OpenSSL speed test with qatprovider and see if the instance allocation issue still persists.
taskset -c 1-2 openssl speed -async_jobs 2 -multi 2 rsa2048

[Taeil-Nam]

I ran an OpeenSSL speed test as you requested and QAT_HW operated successfully.

root# taskset -c 1-2 openssl speed -provider qatprovider -async_jobs 2 -multi 2 rsa2048
QAT_HW RSA for Provider Enabled
QAT_HW ECX25519 for Provider Enabled
QAT_HW ECX448 for Provider Enabled
Forked child 0
Forked child 1
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+R1:20432:2048:10.00
+DTP:2048:public:rsa:10
+R1:20441:2048:10.01
+DTP:2048:public:rsa:10
+R2:100827:2048:10.00
Got: +F2:2:2048:2043.200000:10082.700000 from 0
+R2:100698:2048:10.00
Got: +F2:2:2048:2042.057942:10069.800000 from 1
version: 3.0.16
built on: Mon Feb  9 01:20:39 2026 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x21cbfbb
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000245s 0.000050s   4085.3  20152.5

root# cat /sys/kernel/debug/qat_dh895xcc_0000:83:00.0/fw_counters
+------------------------------------------------+
| FW Statistics for Qat Device                   |
+------------------------------------------------+
| Firmware Requests [AE  0]:              131129 |
| Firmware Responses[AE  0]:              131129 |
| RAS Events        [AE  0]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  1]:                   0 |
| Firmware Responses[AE  1]:                   0 |
| RAS Events        [AE  1]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  2]:                   0 |
| Firmware Responses[AE  2]:                   0 |
| RAS Events        [AE  2]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  3]:              131129 |
| Firmware Responses[AE  3]:              131129 |
| RAS Events        [AE  3]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  4]:                   0 |
| Firmware Responses[AE  4]:                   0 |
| RAS Events        [AE  4]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  5]:                   0 |
| Firmware Responses[AE  5]:                   0 |
| RAS Events        [AE  5]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  6]:              131129 |
| Firmware Responses[AE  6]:              131129 |
| RAS Events        [AE  6]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  7]:                   0 |
| Firmware Responses[AE  7]:                   0 |
| RAS Events        [AE  7]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  8]:                   0 |
| Firmware Responses[AE  8]:                   0 |
| RAS Events        [AE  8]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  9]:              131129 |
| Firmware Responses[AE  9]:              131129 |
| RAS Events        [AE  9]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE 10]:                   0 |
| Firmware Responses[AE 10]:                   0 |
| RAS Events        [AE 10]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE 11]:                   0 |
| Firmware Responses[AE 11]:                   0 |
| RAS Events        [AE 11]:                   0 |
+------------------------------------------------+

root#head -n 200 qat.log
[DEBUG][258136.071494] PID [1935] Thread [7f687e6181c0][qat_prov_init.c:557:qat_get_params_from_core()] get_params is NULL. Using the default params
[WARN][258136.071521] PID [1935] Thread [7f687e6181c0][e_qat.c:1126:bind_qat()] QAT Warnings enabled.
[DEBUG][258136.071526] PID [1935] Thread [7f687e6181c0][e_qat.c:1127:bind_qat()] QAT Debug enabled.
[WARN][258136.071531] PID [1935] Thread [7f687e6181c0][e_qat.c:1128:bind_qat()] (null) - Reference implementation of QAT crypto engine(qat_hw) v1.8.1
[DEBUG][258136.071772] PID [1935] Thread [7f687e6181c0][e_qat.c:1145:bind_qat()] QAT HW device available
[DEBUG][258136.071799] PID [1935] Thread [7f687e6181c0][qat_evp.c:964:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_128 registration succeeded
[DEBUG][258136.071807] PID [1935] Thread [7f687e6181c0][qat_evp.c:964:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_192 registration succeeded
[DEBUG][258136.071811] PID [1935] Thread [7f687e6181c0][qat_evp.c:976:qat_create_ccm_cipher_meth()] OpenSSL SW AES_CCM_256 registration succeeded
[DEBUG][258136.071819] PID [1935] Thread [7f687e6181c0][e_qat.c:615:qat_engine_init()] QAT Engine initialization:
[DEBUG][258136.071825] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:507:qat_hw_init()] QAT_HW initialization:
[DEBUG][258136.071830] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:508:qat_hw_init()] - External polling: OFF
[DEBUG][258136.071835] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:509:qat_hw_init()] - Heuristic polling: OFF
[DEBUG][258136.071839] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:510:qat_hw_init()] - SW Fallback: OFF
[DEBUG][258136.071844] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:511:qat_hw_init()] - Inline polling: OFF
[DEBUG][258136.071848] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:512:qat_hw_init()] - Internal poll interval: 10000ns
[DEBUG][258136.071852] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:513:qat_hw_init()] - Epoll timeout: 1000ms
[DEBUG][258136.071857] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:514:qat_hw_init()] - Event driven polling mode: OFF
[DEBUG][258136.071861] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:515:qat_hw_init()] - Instance for thread: OFF
[DEBUG][258136.071866] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:516:qat_hw_init()] - Max retry count: 5
[DEBUG][258136.124842] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:553:qat_hw_init()] Found 1 Cy instances
[DEBUG][258136.124892] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:672:qat_hw_init()] Started Instance No: 0 Located on Device: 0
Asymmetric instances sequence: 0
Symmetric instances sequence: 0
[DEBUG][258136.124908] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:696:qat_hw_init()] Instances 1, SVM instances 0, contig instances 1
Asym sequence for thread mapping: 0
Sym sequence for thread mapping: 0
[DEBUG][258136.124974] PID [1935] Thread [7f687bfff700][qat_hw_polling.c:137:qat_timer_poll_func()] timer_poll_func started
[DEBUG][258136.125004] PID [1935] Thread [7f687bfff700][qat_hw_polling.c:149:qat_timer_poll_func()] qat_timer_poll_func_thread = 0x7f687bfff700
[DEBUG][258136.125096] PID [1935] Thread [7f687e6181c0][e_qat.c:663:qat_engine_finish_int()] ---- QAT Engine Finishing...

[DEBUG][258136.125105] PID [1935] Thread [7f687e6181c0][e_qat.c:665:qat_engine_finish_int()] RSA Priv retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125111] PID [1935] Thread [7f687e6181c0][e_qat.c:667:qat_engine_finish_int()] RSA Pub retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125117] PID [1935] Thread [7f687e6181c0][e_qat.c:670:qat_engine_finish_int()] ECDH keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125122] PID [1935] Thread [7f687e6181c0][e_qat.c:673:qat_engine_finish_int()] ECDH derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125127] PID [1935] Thread [7f687e6181c0][e_qat.c:675:qat_engine_finish_int()] ECX keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125131] PID [1935] Thread [7f687e6181c0][e_qat.c:677:qat_engine_finish_int()] ECX derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125136] PID [1935] Thread [7f687e6181c0][e_qat.c:679:qat_engine_finish_int()] ECDSA sign retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125150] PID [1935] Thread [7f687e6181c0][e_qat.c:682:qat_engine_finish_int()] SM4-CBC retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.125156] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:801:qat_hw_finish_int()] ---- QAT Finishing...

[DEBUG][258136.125232] PID [1935] Thread [7f687bfff700][qat_hw_polling.c:215:qat_timer_poll_func()] timer_poll_func finishing - pid = 1935
[DEBUG][258136.128380] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:901:qat_hw_finish_int()] Calling pthread_key_delete()
[DEBUG][258136.128584] PID [1935] Thread [7f687e6181c0][e_qat.c:663:qat_engine_finish_int()] ---- QAT Engine Finishing...

[DEBUG][258136.128595] PID [1935] Thread [7f687e6181c0][e_qat.c:665:qat_engine_finish_int()] RSA Priv retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128600] PID [1935] Thread [7f687e6181c0][e_qat.c:667:qat_engine_finish_int()] RSA Pub retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128605] PID [1935] Thread [7f687e6181c0][e_qat.c:670:qat_engine_finish_int()] ECDH keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128610] PID [1935] Thread [7f687e6181c0][e_qat.c:673:qat_engine_finish_int()] ECDH derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128615] PID [1935] Thread [7f687e6181c0][e_qat.c:675:qat_engine_finish_int()] ECX keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128620] PID [1935] Thread [7f687e6181c0][e_qat.c:677:qat_engine_finish_int()] ECX derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128624] PID [1935] Thread [7f687e6181c0][e_qat.c:679:qat_engine_finish_int()] ECDSA sign retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128629] PID [1935] Thread [7f687e6181c0][e_qat.c:682:qat_engine_finish_int()] SM4-CBC retries: 0, HW requests: 0, SW requests: 0
[DEBUG][258136.128635] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:801:qat_hw_finish_int()] ---- QAT Finishing...

[DEBUG][258136.128621] PID [1937] Thread [7f687e6181c0][e_qat.c:615:qat_engine_init()] QAT Engine initialization:
[DEBUG][258136.128652] PID [1935] Thread [7f687e6181c0][qat_hw_init.c:901:qat_hw_finish_int()] Calling pthread_key_delete()
[DEBUG][258136.128658] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:507:qat_hw_init()] QAT_HW initialization:
[DEBUG][258136.128665] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:508:qat_hw_init()] - External polling: OFF
[DEBUG][258136.128671] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:509:qat_hw_init()] - Heuristic polling: OFF
[DEBUG][258136.128676] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:510:qat_hw_init()] - SW Fallback: OFF
[DEBUG][258136.128681] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:511:qat_hw_init()] - Inline polling: OFF
[DEBUG][258136.128685] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:512:qat_hw_init()] - Internal poll interval: 10000ns
[DEBUG][258136.128698] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:513:qat_hw_init()] - Epoll timeout: 1000ms
[DEBUG][258136.128704] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:514:qat_hw_init()] - Event driven polling mode: OFF
[DEBUG][258136.128709] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:515:qat_hw_init()] - Instance for thread: OFF
[DEBUG][258136.128714] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:516:qat_hw_init()] - Max retry count: 5
[DEBUG][258136.128980] PID [1938] Thread [7f687e6181c0][e_qat.c:615:qat_engine_init()] QAT Engine initialization:
[DEBUG][258136.129016] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:507:qat_hw_init()] QAT_HW initialization:
[DEBUG][258136.129021] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:508:qat_hw_init()] - External polling: OFF
[DEBUG][258136.129026] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:509:qat_hw_init()] - Heuristic polling: OFF
[DEBUG][258136.129031] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:510:qat_hw_init()] - SW Fallback: OFF
[DEBUG][258136.129035] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:511:qat_hw_init()] - Inline polling: OFF
[DEBUG][258136.129048] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:512:qat_hw_init()] - Internal poll interval: 10000ns
[DEBUG][258136.129053] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:513:qat_hw_init()] - Epoll timeout: 1000ms
[DEBUG][258136.129058] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:514:qat_hw_init()] - Event driven polling mode: OFF
[DEBUG][258136.129063] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:515:qat_hw_init()] - Instance for thread: OFF
[DEBUG][258136.129068] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:516:qat_hw_init()] - Max retry count: 5
[DEBUG][258136.181786] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:553:qat_hw_init()] Found 1 Cy instances
[DEBUG][258136.181838] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:672:qat_hw_init()] Started Instance No: 0 Located on Device: 0
Asymmetric instances sequence: 0
Symmetric instances sequence: 0
[DEBUG][258136.181862] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:696:qat_hw_init()] Instances 1, SVM instances 0, contig instances 1
Asym sequence for thread mapping: 0
Sym sequence for thread mapping: 0
[DEBUG][258136.181959] PID [1937] Thread [7f687bfff700][qat_hw_polling.c:137:qat_timer_poll_func()] timer_poll_func started
[DEBUG][258136.181971] PID [1937] Thread [7f687bfff700][qat_hw_polling.c:149:qat_timer_poll_func()] qat_timer_poll_func_thread = 0x7f687bfff700
[DEBUG][258136.182037] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:553:qat_hw_init()] Found 1 Cy instances
[DEBUG][258136.182083] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:672:qat_hw_init()] Started Instance No: 0 Located on Device: 0
Asymmetric instances sequence: 0
Symmetric instances sequence: 0
[DEBUG][258136.182105] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:696:qat_hw_init()] Instances 1, SVM instances 0, contig instances 1
Asym sequence for thread mapping: 0
Sym sequence for thread mapping: 0
[DEBUG][258136.182194] PID [1938] Thread [7f687bfff700][qat_hw_polling.c:137:qat_timer_poll_func()] timer_poll_func started
[DEBUG][258136.185198] PID [1937] Thread [7f687e6181c0][qat_prov_sign_rsa.c:1370:qat_signature_rsa_sign_init()] qat_rsa_sign_init.
[DEBUG][258136.185216] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:1010:qat_rsa_priv_enc()] QAT HW RSA Started.
[DEBUG][258136.185229] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:318:get_instance()] inst type: ASYM, inst_num = 0
[DEBUG][258136.185235] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:454:build_decrypt_op_buf()] - Started
[DEBUG][258136.185240] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:466:build_decrypt_op_buf()] flen = 36, padding = 1
[DEBUG][258136.185309] PID [1938] Thread [7f687e6181c0][qat_prov_sign_rsa.c:1370:qat_signature_rsa_sign_init()] qat_rsa_sign_init.
[DEBUG][258136.185334] PID [1938] Thread [7f687e6181c0][qat_hw_rsa.c:1010:qat_rsa_priv_enc()] QAT HW RSA Started.
[DEBUG][258136.185345] PID [1938] Thread [7f687e6181c0][qat_hw_init.c:318:get_instance()] inst type: ASYM, inst_num = 0
[DEBUG][258136.185351] PID [1938] Thread [7f687e6181c0][qat_hw_rsa.c:454:build_decrypt_op_buf()] - Started
[DEBUG][258136.185356] PID [1938] Thread [7f687e6181c0][qat_hw_rsa.c:466:build_decrypt_op_buf()] flen = 36, padding = 1
[DEBUG][258136.185728] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:587:build_decrypt_op_buf()] - Finished
[DEBUG][258136.185741] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:203:qat_rsa_decrypt()] - Started
RSA Decrypt Request: 0x5573f7ed99d0
instance_handle = 0x5573f7e6aa80
op_done = 0x7ffd60315860
opData: pRecipientPrivateKey->version = 1
opData: pRecipientPrivateKey->privateKeyRepType = 2
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.modulusN.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.privateExponentD.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime1P.pData: Length 128, Address 0x7f687b201400
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime2Q.pData: Length 128, Address 0x7f687b201800
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.exponent1Dp.pData: Length 128, Address 0x7f687b201c00
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.exponent2Dq.pData: Length 128, Address 0x7f687b202000
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.coefficientQInv.pData: Length 128, Address 0x7f687b202400
qat_rsa_decrypt: opData: inputData.pData: Length 256, Address 0x7f687b202800
output_buf = 0x5573f7ed99f0
[DEBUG][258136.185848] PID [1938] Thread [7f687e6181c0][qat_hw_rsa.c:587:build_decrypt_op_buf()] - Finished
[DEBUG][258136.185857] PID [1938] Thread [7f687e6181c0][qat_hw_rsa.c:203:qat_rsa_decrypt()] - Started
RSA Decrypt Request: 0x5573f7ed9d70
instance_handle = 0x5573f7e6aa80
op_done = 0x7ffd60315860
opData: pRecipientPrivateKey->version = 1
opData: pRecipientPrivateKey->privateKeyRepType = 2
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.modulusN.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.privateExponentD.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime1P.pData: Length 128, Address 0x7f687b201400
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime2Q.pData: Length 128, Address 0x7f687b201800
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.exponent1Dp.pData: Length 128, Address 0x7f687b201c00
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.exponent2Dq.pData: Length 128, Address 0x7f687b202000
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.coefficientQInv.pData: Length 128, Address 0x7f687b202400
qat_rsa_decrypt: opData: inputData.pData: Length 256, Address 0x7f687b202800
output_buf = 0x5573f7ed9d90
[DEBUG][258136.186662] PID [1937] Thread [7f687bfff700][qat_hw_callback.c:214:qat_crypto_callbackFn()] status 0 verifyResult 1
RSA Decrypt Output: 0x5573f7ed99f0
qat_rsa_decrypt: output_buf: Length 256, Address 0x7f687b202c00
[DEBUG][258136.186680] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:434:qat_rsa_decrypt()] - Finished
[DEBUG][258136.186690] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:162:rsa_decrypt_op_buf_free()] - Started
[DEBUG][258136.186696] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:184:rsa_decrypt_op_buf_free()] - Finished
[DEBUG][258136.186701] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:1136:qat_rsa_priv_enc()] - Finished
[DEBUG][258136.186720] PID [1937] Thread [7f687e6181c0][qat_prov_sign_rsa.c:1370:qat_signature_rsa_sign_init()] qat_rsa_sign_init.
[DEBUG][258136.186728] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:1010:qat_rsa_priv_enc()] QAT HW RSA Started.
[DEBUG][258136.186733] PID [1937] Thread [7f687e6181c0][qat_hw_init.c:318:get_instance()] inst type: ASYM, inst_num = 0
[DEBUG][258136.186739] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:454:build_decrypt_op_buf()] - Started
[DEBUG][258136.186743] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:466:build_decrypt_op_buf()] flen = 36, padding = 1
[DEBUG][258136.186753] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:587:build_decrypt_op_buf()] - Finished
[DEBUG][258136.186758] PID [1937] Thread [7f687e6181c0][qat_hw_rsa.c:203:qat_rsa_decrypt()] - Started
RSA Decrypt Request: 0x5573f7ed99f0
instance_handle = 0x5573f7e6aa80
op_done = 0x7ffd60315860
opData: pRecipientPrivateKey->version = 1
opData: pRecipientPrivateKey->privateKeyRepType = 2
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.modulusN.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep1.privateExponentD.pData: Length 0, Address (nil)
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime1P.pData: Length 128, Address 0x7f687b201400
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.prime2Q.pData: Length 128, Address 0x7f687b201800
qat_rsa_decrypt: opData: pRecipientPrivateKey->privateKeyRep2.exponent1Dp.pData: Length 128, Address 0x7f687b201c00
...

[venkatesh6911]

Thanks for the testrun. Can you share details about your test application ssl_proxy ?
The fact that there are no instance allocation problems when running OpenSSL Speed indicate that there are no underlying QAT HW resource allocation issues.

[Taeil-Nam]

The ssl_proxy acts as a SSL/TLS proxy between a client and a server.
It uses OpenSSL for SSL/TLS processing and QAT_HW to accelerate the TLS handshake.

I am migrating the system from OpenSSL 1.1.1 (QAT Engine) to OpenSSL 3.0.16 (QAT Provider).

The ssl_proxy spawns 12 child processes, and each process loads the provider using the following OpenSSL APIs:

OSSL_LIB_CTX_load_config(NULL, PROVD_OSSL_CNF);
OSSL_PROVIDER_set_default_search_path(NULL, PROVD_PATH);
OSSL_PROVIDER_load(NULL, "default");
OSSL_PROVIDER_load(NULL, "qatprovider");

Each process establishes SSL/TLS connections with the client and the server using SSL_accept() and SSL_connect().

When a process terminates, it unloads the QAT Provider as follows:

OSSL_PROVIDER_unload(g_PvQat);

It operates normally when QAT is not used.

[venkatesh6911]

From your replies,
"I am migrating the system from OpenSSL 1.1.1 (QAT Engine) to OpenSSL 3.0.16 (QAT Provider)."

Q: I assume that the ssl proxy application works fine with 1.1.1 based QAT Engine and this problem is observed only with provider ?

"Each process establishes SSL/TLS connections with the client and the server using SSL_accept() and `SSL_connect()."

Q: Are you forking again for every connection ? If so, it is very likely that there is an instance contention problem.

[Taeil-Nam]

Q1: I assume that the ssl proxy application works fine with 1.1.1 based QAT Engine and this problem is observed only with provider ?
A1: Yes.

Q2: Are you forking again for every connection ? If so, it is very likely that there is an instance contention problem.
A2: No.

1. fork()
 ├─ child1 → Load QAT Provider
 ├─ child2 → Load QAT Provider
 ├─ child3 → Load QAT Provider
  ...
 ├─ child12 → QAT load

2. Each child process handles SSL/TLS connections without performing any additional fork().

I tested QAT_HW with the following combination, and it worked successfully:

QAT Engine	QAT Driver	OpenSSL	Result
v1.6.0	QAT.L.4.23.0-00001	v3.0.16	Success

I then upgraded the QAT Engine to v1.7.0 but same problem was observed:

QAT Engine	QAT Driver	OpenSSL	Result
v1.7.0	QAT.L.4.23.0-00001	v3.0.16	Fail

During the test, I didn't modify the code of the ssl_proxy and any other configurations.

[venkatesh6911]

Q1: I assume that the ssl proxy application works fine with 1.1.1 based QAT Engine and this problem is observed only with provider ? A1: Yes.

Q2: Are you forking again for every connection ? If so, it is very likely that there is an instance contention problem. A2: No.

1. fork()
 ├─ child1 → Load QAT Provider
 ├─ child2 → Load QAT Provider
 ├─ child3 → Load QAT Provider
  ...
 ├─ child12 → QAT load

2. Each child process handles SSL/TLS connections without performing any additional fork().

I tested QAT_HW with the following combination, and it worked successfully:

QAT Engine QAT Driver OpenSSL Result
v1.6.0 QAT.L.4.23.0-00001 v3.0.16 Success
I then upgraded the QAT Engine to v1.7.0 but same problem was observed:

QAT Engine QAT Driver OpenSSL Result
v1.7.0 QAT.L.4.23.0-00001 v3.0.16 Fail
During the test, I didn't modify the code of the ssl_proxy and any other configurations.

Both v1.6.0 and v1.7.0 were built and tested as providers ?

[Taeil-Nam]

Yes.