panic: dpaa2_ni_rx: failed to recycle buffer: error=12

[bzfbd]

More likely so I don't forget; I can reproduce this running an iperf3 in server mode and sending some UDP packets in from wifi (note wifi!) on a 2.5G (10G) SFP+... I'll leave this here to not forget; I wonder how we extremly quickly exhaust the 9k jumbo zone. Makes me wonder if we are leaking mbufs or what's going on here...

db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
     mbuf_jumbo_9k   9216  144627    1785      183914       0     254 1349332992        0
              mbuf    256  144630    1674      421305       0     254   37453824        0

[zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
dpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_see[  5]   3.06-4.0d_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/n0   sec  90.0 KBdpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/nytes   781 Kbitsdpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/n/sec  4.660 ms  dpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/n0/64 (0%)  
dpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
dpaa2_ni8: dpaa2_buf_seed_pool: dpaa2_buf_seed_rxb() failed: error=12/ndpaa2_ni8: dpaa2_buf_seed_rxb: m_getjcl() failed
panic: dpaa2_ni_rx: failed to recycle buffer: error=12
cpuid = 7
time = 266
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x38
vpanic() at vpanic+0x1a0
panic() at panic+0x48
dpaa2_ni_rx() at dpaa2_ni_rx+0x370
dpaa2_ni_cleanup_task() at dpaa2_ni_cleanup_task+0x100
taskqueue_run_locked() at taskqueue_run_locked+0x17c
taskqueue_thread_loop() at taskqueue_thread_loop+0xc0
fork_exit() at fork_exit+0x78
fork_trampoline() at fork_trampoline+0x18
KDB: enter: panic
[ thread pid 0 tid 100191 ]
Stopped at      kdb_enter+0x48: str     xzr, [x19, #128]

[dsalychev]

Do you have INVARIANTS enabled for that kernel? There's a place at the beginning of the dpaa2_buf_seed_rxb() which is supposed to verify whether buffer is ready to be used for reception:

#if defined(INVARIANTS)
	DPAA2_BUF_ASSERT_RXPREP(buf);
	if (dma_mtx != NULL) {
		mtx_assert(dma_mtx, MA_OWNED);
	}
#endif /* INVARIANTS */	

[dsalychev]

and sending some UDP packets in from wifi (note wifi!) on a 2.5G (10G) SFP+

I'm not sure I understand it correctly. Do you have SFP+ connected to WiFi access point?

[bzfbd]

and sending some UDP packets in from wifi (note wifi!) on a 2.5G (10G) SFP+

I'm not sure I understand it correctly. Do you have SFP+ connected to WiFi access point?

I just mentioned it as the RJ45 SPF+ in the Ten64 is a 10G Ethernet but runing at 2.5 as the AP has a 2.5 Ethernet. But also this happens in the first 2s of traffic so this is less likely an issue there but probably also going to happen on 1G ethernet as it simply happens too easily. Need to go and investigate as soon as I find time.

[dsalychev]

Could you try INVARIANTS as well, please?

[bzfbd]

Could you try INVARIANTS as well, please?

I am running an extended GENRIC kernel on main, so INVARIANTS are on

[dsalychev]

@bzfbd Could you take a look at the dev.dpaa2_io sysctl? I've added two counters in https://github.com/mcusim/freebsd-src/tree/bugs/27 to understand whether DQRR entries are fine and channel cleanup tasks scheduled.