Skip to content

IPv6: SocketAddrV6 through UdpSocket#62

Merged
gspivey merged 2 commits into
developmentfrom
agent/ipv6-socket-addr
May 29, 2026
Merged

IPv6: SocketAddrV6 through UdpSocket#62
gspivey merged 2 commits into
developmentfrom
agent/ipv6-socket-addr

Conversation

@gspivey
Copy link
Copy Markdown
Owner

@gspivey gspivey commented May 29, 2026

Roadmap Item

IPv6 sub-task 3: SocketAddrV6 through UdpSocketbind / send_to / recv_from / connect / local_addr / peer_addr accept and return IPv6 addresses. set_only_v6 / only_v6 socket option. AddressFamily state on the socket so the send/recv paths pick the right wire format.

Changes

  • bind() now accepts IPv6 addresses ([::]:port, [2001:db8::1]:port, etc.)
  • send_to() builds IPv6/UDP frames when destination is SocketAddrV6, using build_udp6_frame_into and NDP for neighbor resolution
  • recv_from() already returned SocketAddrV6 (RX path was done in prior PRs); now the socket can be bound to IPv6 so the full path works end-to-end
  • connect() / peer_addr() / local_addr() work with IPv6 addresses
  • AddressFamily enum (IPv4 / IPv6) and address_family() accessor
  • set_only_v6() / only_v6() socket option (IPV6_V6ONLY equivalent) — when true, IPv4 frames are rejected
  • NdpHandler field on socket for IPv6 neighbor resolution on TX path
  • Gratuitous NA sent on bind for IPv6 sockets (parallel to GARP for IPv4)
  • bind_with_backend() also accepts IPv6 addresses

Tests Added

18 new unit tests (625 total, all passing):

  • bind with various IPv6 addresses (unspecified, specific, loopback, link-local)
  • connect with IPv6 peer
  • send_to with IPv6 destination (verifies path is reachable; stub returns WouldBlock)
  • recv_from returning V6 source address
  • set_only_v6/only_v6 accessors
  • only_v6 filtering (rejects IPv4, accepts IPv6)
  • set_only_v6 on IPv4 socket returns error
  • address_family accessor
  • connected socket filtering with IPv6
  • roundtrip (bind V6 + receive V6 frame)

Tradeoffs

  • Dual-stack default: IPv6 sockets accept both IPv4 and IPv6 by default (matching Linux behavior). Use set_only_v6(true) to restrict.
  • NDP resolution fallback: When NDP cache miss occurs, a solicitation is sent and broadcast MAC is used as fallback (stub mode always hits this path). Real hardware will resolve via NDP reply.
  • No IPv6 routing table: IPv6 sockets use a default routing table (no auto-detection from /proc/net/ipv6/route yet). MTU is inherited from the DPDK port config.

Implement IPv6 sub-task 3 from the roadmap: SocketAddrV6 through UdpSocket.

Changes:
- bind() now accepts IPv6 addresses ([::]:port, [2001:db8::1]:port, etc.)
- send_to() builds IPv6/UDP frames when destination is SocketAddrV6
- recv_from() already returned SocketAddrV6 (RX path was done); now the
  socket can be bound to IPv6 so the full path works end-to-end
- connect()/peer_addr()/local_addr() work with IPv6 addresses
- Added AddressFamily enum (IPv4/IPv6) and address_family() accessor
- Added set_only_v6()/only_v6() socket option (IPV6_V6ONLY equivalent)
- When only_v6 is true, IPv4 frames are rejected in process_frame_zerocopy
- Added NdpHandler field for IPv6 neighbor resolution on TX path
- Gratuitous NA sent on bind for IPv6 sockets (parallel to GARP for IPv4)
- bind_with_backend() also accepts IPv6 addresses

18 new tests covering:
- bind with various IPv6 addresses (unspecified, specific, loopback, link-local)
- connect with IPv6 peer
- send_to with IPv6 destination
- recv_from returning V6 source address
- set_only_v6/only_v6 accessors
- only_v6 filtering (rejects IPv4, accepts IPv6)
- address_family accessor
- connected socket filtering with IPv6
- roundtrip (bind V6 + receive V6 frame)
@github-actions
Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 8f630df1

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.7M 11.2M 1.0x 89
RX recv_from 64B 3.6M 4.8M 0.8x 210
TX send_to 1400B 1.8M 1.8M 1.0x 545
RX recv_from 1400B 1.1M 1.3M 0.9x 799

Avg sync/async ratio: 0.9x, worst: 1.0x

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 8f630df1

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.8M 11.3M 1.0x 88
RX recv_from 64B 3.5M 4.7M 0.8x 214
TX send_to 1400B 1.8M 1.8M 1.0x 568
RX recv_from 1400B 1.1M 1.2M 0.9x 843

Avg sync/async ratio: 0.9x, worst: 1.0x

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-08e2fdc71a26ef575 (DPDK ENI: 10.0.1.18)
  • Receiver: i-0eebe3d9d67ff71eb (DPDK ENI: 10.0.1.160)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0b3edbbb4e9f74671 (DPDK ENI: 10.0.1.128)
  • Receiver: i-073e238024c706662 (DPDK ENI: 10.0.1.57)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

1 similar comment
@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Deploy

Deploying PerfTestStack...
Configs: plain-rust,rust-dpdk,tokio-dpdk,native-dpdk
Packet sizes: 64,512,1400,8500

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed (Run 26632384723)

Branch: 62/merge | Commit: 8f630df1

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.160:9000 (MAC: 02:eb:01:ad:a5:05)
echo listening on 10.0.1.160:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.18:9000 (MAC: 02:1d:b4:ca:b9:bf)
echo listening on 10.0.1.18:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.160:9000: 'arp-probe #1'
Test complete
[2026-05-29T10:46:57Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T10:46:57Z] INFO: Test: udp_send_receive
[2026-05-29T10:46:58Z] INFO: UDP send/receive succeeded
[2026-05-29T10:46:58Z] INFO: Test: echo_roundtrip
[2026-05-29T10:46:59Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T10:46:59Z] INFO: Test: payload_integrity
[2026-05-29T10:46:59Z] INFO: Response received, checking payload match...
[2026-05-29T10:46:59Z] INFO: Payload integrity verified (found in response)
[2026-05-29T10:46:59Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T10:46:59Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-29T10:54:01Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-29T10:54:01Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-29T10:54:01Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-29T10:54:01Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-29T10:53:00Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-29T10:53:00Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-29T10:53:00Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-29T10:53:00Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.160:9000 (MAC: 02:eb:01:ad:a5:05)
echo listening on 10.0.1.160:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.18:9000 (MAC: 02:1d:b4:ca:b9:bf)
echo listening on 10.0.1.18:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-05-29T10:43:19Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.160:9000
Bind address: 10.0.1.18:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.18:32768 (MAC: 02:1d:b4:ca:b9:bf)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.160:9000: 'arp-probe #1'
Test complete
[2026-05-29T10:43:21Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T10:43:21Z] INFO: Test: udp_send_receive
[2026-05-29T10:43:22Z] INFO: UDP send/receive succeeded
[2026-05-29T10:43:22Z] INFO: Test: echo_roundtrip
[2026-05-29T10:43:24Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T10:43:24Z] INFO: Test: payload_integrity
[2026-05-29T10:43:25Z] INFO: Response received, checking payload match...
[2026-05-29T10:43:25Z] INFO: Payload integrity verified (found in response)
[2026-05-29T10:43:25Z] INFO: Test: jumbo_diagnostics
[2026-05-29T10:43:25Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-05-29T10:43:25Z] INFO: Interface MTU:
  9001
  65536
[2026-05-29T10:43:25Z] INFO:   ens5: MTU=9001
[2026-05-29T10:43:25Z] INFO:   lo: MTU=65536
[2026-05-29T10:43:25Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-05-29T10:43:25Z] INFO: DPDK port config (from echo server log):
[2026-05-29T10:43:25Z] INFO:   (no MTU info in echo log)
[2026-05-29T10:43:25Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-05-29T10:43:25Z] INFO: Test: jumbo_echo_8000
[2026-05-29T10:43:26Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.160:9000
Bind address: 10.0.1.18:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.18:32768 (MAC: 02:1d:b4:ca:b9:bf)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.160:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.160:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.160:9000 (expected 8000, OK)
Test complete
[2026-05-29T10:43:26Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-05-29T10:43:26Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-29T10:43:26Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-29T10:46:57Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.160:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.160:9000: 'arp-probe #1'
Test complete
[2026-05-29T10:46:57Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T10:46:57Z] INFO: Test: udp_send_receive
[2026-05-29T10:46:58Z] INFO: UDP send/receive succeeded
[2026-05-29T10:46:58Z] INFO: Test: echo_roundtrip
[2026-05-29T10:46:59Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T10:46:59Z] INFO: Test: payload_integrity
[2026-05-29T10:46:59Z] INFO: Response received, checking payload match...
[2026-05-29T10:46:59Z] INFO: Payload integrity verified (found in response)
[2026-05-29T10:46:59Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T10:46:59Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-29T10:54:01Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-29T10:54:01Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-29T10:54:01Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-29T10:54:01Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-05-29T10:52:58Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-05-29T10:52:58Z] INFO: Local IP: 10.0.1.18, Peer IP: 10.0.1.160, Port: 9000
[2026-05-29T10:52:58Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.160       0x1         0x2         02:eb:01:ad:a5:05     *        ens5
10.0.1.68        0x1         0x2         02:e5:ae:54:a3:d3     *        ens5
10.0.1.1         0x1         0x2         02:9e:7b:3b:fe:81     *        ens5
10.0.1.67        0x1         0x2         02:63:56:d7:92:a3     *        ens5
10.0.1.18        0x1         0x2         02:1d:b4:ca:b9:bf     *        ens5
[2026-05-29T10:52:58Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-05-29T10:52:58Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-05-29T10:52:58Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1916248 May 29 10:41 /opt/dpdk-stdlib/target/release/test-client
[2026-05-29T10:52:58Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.160 --port 9000 --bind-ip 10.0.1.18 --count 10 --delay 200
[2026-05-29T10:53:00Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.160:9000
Bind address: 10.0.1.18:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.18:32768 (MAC: 02:1d:b4:ca:b9:bf)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.160:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-29T10:53:00Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-29T10:53:00Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-29T10:53:00Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-29T10:53:00Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eebe3d9d67ff71eb (receiver)
Command ID: bb17c36e-b277-48c8-a8c5-3c42abf8f8b5

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eebe3d9d67ff71eb (receiver)
Command ID: 75ea13ab-b793-406c-aeac-b0091e4ac9fd

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eebe3d9d67ff71eb (receiver)
Command ID: b63cf7c4-7b6f-4b64-b2c9-6d800b44102c

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0eebe3d9d67ff71eb (receiver)
Command ID: 6e78448e-8fdd-43af-aaf4-14dc6cc8b82c

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-08e2fdc71a26ef575 (sender)
Command ID: 6ee887aa-52bd-4649-88c6-a58e9162e059

=== STDOUT ===


=== STDERR ===


Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:74:74:76:b7 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-04f33ba4769178fbd
    altname device-number-0.0
    inet 10.0.1.16/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2362sec preferred_lft 2362sec
    inet6 fe80::23:74ff:fe74:76b7/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:18:14:7f:5f:f3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-099cf8f0f2769097e
    altname device-number-0.0
    inet 10.0.1.124/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2386sec preferred_lft 2386sec
    inet6 fe80::18:14ff:fe7f:5ff3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-29T10:42:39Z
hostname: ip-10-0-1-16.ec2.internal
kernel: 6.18.30-61.116.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:74:74:76:b7 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-04f33ba4769178fbd
    altname device-number-0.0
    inet 10.0.1.16/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3351sec preferred_lft 3351sec
    inet6 fe80::23:74ff:fe74:76b7/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.1 dev ens5 lladdr 02:9e:7b:3b:fe:81 REACHABLE 
10.0.1.67 dev ens5 lladdr 02:63:56:d7:92:a3 STALE 
10.0.1.68 dev ens5 lladdr 02:e5:ae:54:a3:d3 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.16 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.16 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.16 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.16 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:23:74:74:76:b7/ 02:eb:01:ad:a5:05/ 

--- ENI: 02:23:74:74:76:b7/ ---
  device-number: 0
  local-ipv4s: 10.0.1.16
  subnet-id: subnet-0e68f15d89361ede5
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:eb:01:ad:a5:05/ ---
  device-number: 1
  local-ipv4s: 10.0.1.160
  subnet-id: subnet-0e68f15d89361ede5
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:9e:7b:3b:fe:81 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.16 ens5
Unicast reply from 10.0.1.1 [02:9E:7B:3B:FE:81]  0.536ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 29 10:42 .
drwxr-xr-x. 14 root root     3100 May 29 10:41 ..
crw-------.  1 root root 243,   0 May 29 10:42 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 29 10:38 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.053438] x2apic enabled
[    0.057628] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.057736] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.067496] audit: type=2000 audit(1780051095.712:1): state=initialized audit_enabled=0 res=1
[    0.087086] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.148252] ACPI: Interpreter enabled
[    0.148252] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.167055] pci 0000:00:05.0: enabling Extended Tags
[    0.233171] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.247202] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.250592] IPI shorthand broadcast: enabled
[   10.565850] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[   10.600540] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[   10.748751] VFIO - User Level meta-driver version: 0.3
[   11.690842] audit: type=1305 audit(1780051106.930:128): op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   11.856709] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[   11.887468] ena 0000:00:05.0: ENA device version: 0.10
[   11.888192] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[   11.994712] ena 0000:00:05.0: ENA Large LLQ is disabled
[   12.011907] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:23:74:74:76:b7
[   12.044321] ena 0000:00:05.0 ens5: renamed from eth0
[  189.813402] pci 0000:00:06.0: enabling Extended Tags
[  189.817398] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  189.830219] ena 0000:00:06.0: ENA device version: 0.10
[  189.830981] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  189.930203] ena 0000:00:06.0: ENA Large LLQ is disabled
[  189.942204] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:eb:01:ad:a5:05
[  189.949344] ena 0000:00:06.0 ens6: renamed from eth0
[  249.724352] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  249.725759] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-29T10:42:31Z
hostname: ip-10-0-1-124.ec2.internal
kernel: 6.18.30-61.116.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:18:14:7f:5f:f3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-099cf8f0f2769097e
    altname device-number-0.0
    inet 10.0.1.124/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3360sec preferred_lft 3360sec
    inet6 fe80::18:14ff:fe7f:5ff3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.21 dev ens5 lladdr 02:d4:10:6a:80:dd STALE 
10.0.1.68 dev ens5 lladdr 02:e5:ae:54:a3:d3 REACHABLE 
10.0.1.1 dev ens5 lladdr 02:9e:7b:3b:fe:81 REACHABLE 
10.0.1.67 dev ens5 lladdr 02:63:56:d7:92:a3 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.124 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.124 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.124 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.124 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:18:14:7f:5f:f3/ 02:1d:b4:ca:b9:bf/ 

--- ENI: 02:18:14:7f:5f:f3/ ---
  device-number: 0
  local-ipv4s: 10.0.1.124
  subnet-id: subnet-0e68f15d89361ede5
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:1d:b4:ca:b9:bf/ ---
  device-number: 1
  local-ipv4s: 10.0.1.18
  subnet-id: subnet-0e68f15d89361ede5
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:9e:7b:3b:fe:81 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.124 ens5
Unicast reply from 10.0.1.1 [02:9E:7B:3B:FE:81]  0.537ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 29 10:42 .
drwxr-xr-x. 14 root root     3100 May 29 10:41 ..
crw-------.  1 root root 243,   0 May 29 10:42 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 29 10:38 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.056417] x2apic enabled
[    0.060653] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.060766] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.070539] audit: type=2000 audit(1780051095.941:1): state=initialized audit_enabled=0 res=1
[    0.090111] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.159250] ACPI: Interpreter enabled
[    0.159250] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.172254] pci 0000:00:05.0: enabling Extended Tags
[    0.235236] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.248799] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.252515] IPI shorthand broadcast: enabled
[   10.397859] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[   10.433766] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[   10.538527] VFIO - User Level meta-driver version: 0.3
[   11.467458] audit: type=1305 audit(1780051105.710:118): op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   11.665230] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[   11.686321] ena 0000:00:05.0: ENA device version: 0.10
[   11.687077] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[   11.796424] ena 0000:00:05.0: ENA Large LLQ is disabled
[   11.808391] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:18:14:7f:5f:f3
[   11.852621] ena 0000:00:05.0 ens5: renamed from eth0
[  209.271799] pci 0000:00:06.0: enabling Extended Tags
[  209.275823] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  209.285382] ena 0000:00:06.0: ENA device version: 0.10
[  209.286139] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  209.385362] ena 0000:00:06.0: ENA Large LLQ is disabled
[  209.397360] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:1d:b4:ca:b9:bf
[  209.404583] ena 0000:00:06.0 ens6: renamed from eth0
[  242.499035] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  242.500466] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.066897] pid_max: default: 32768 minimum: 301
[    0.170672] iommu: Default domain type: Translated
[    0.170672] NetLabel:  unlabeled traffic allowed by default
[    0.194111] PCI: CLS 0 bytes, default 64
[    0.275022] nvme nvme0: 2/0/0 default/read/poll queues
[    0.448864] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.448985] systemd[1]: No hostname configured, using default hostname.
[    0.506820] systemd[1]: Queued start job for default target initrd.target.
[    7.271722] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  291.652933] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4372)
[  505.871945] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4853)
[ 1110.639175] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6023)

sender-dmesg-crashes.log

[    0.069913] pid_max: default: 32768 minimum: 301
[    0.175919] iommu: Default domain type: Translated
[    0.175919] NetLabel:  unlabeled traffic allowed by default
[    0.204085] PCI: CLS 0 bytes, default 64
[    0.284777] nvme nvme0: 2/0/0 default/read/poll queues
[    0.431475] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.431597] systemd[1]: No hostname configured, using default hostname.
[    0.518236] systemd[1]: Queued start job for default target initrd.target.
[    7.048989] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  927.684489] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:5626)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  291.651399] vfio-pci 0000:00:06.0: reset done
[  291.652933] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4372)
[  291.654435] vfio-pci 0000:00:06.0: resetting
[  291.871250] vfio-pci 0000:00:06.0: reset done
[  369.923290] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  370.936169] ena 0000:00:06.0: ENA device version: 0.10
[  370.936930] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  371.038101] ena 0000:00:06.0: ENA Large LLQ is disabled
[  371.050724] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:eb:01:ad:a5:05
[  371.061111] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[  390.933712] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  390.935108] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  505.654225] vfio-pci 0000:00:06.0: resetting
[  505.870465] vfio-pci 0000:00:06.0: reset done
[  505.871945] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4853)
[  505.873369] vfio-pci 0000:00:06.0: resetting
[  506.090178] vfio-pci 0000:00:06.0: reset done
[  581.965797] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  582.978411] ena 0000:00:06.0: ENA device version: 0.10
[  582.979182] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  583.078401] ena 0000:00:06.0: ENA Large LLQ is disabled
[  583.090347] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:eb:01:ad:a5:05
[  583.098952] ena 0000:00:06.0 ens6: renamed from eth0
[ 1029.101865] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1029.103287] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1110.416884] vfio-pci 0000:00:06.0: resetting
[ 1110.637616] vfio-pci 0000:00:06.0: reset done
[ 1110.639175] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6023)
[ 1110.640617] vfio-pci 0000:00:06.0: resetting
[ 1110.857432] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[  927.685925] vfio-pci 0000:00:06.0: resetting
[  927.902822] vfio-pci 0000:00:06.0: reset done
[  990.791693] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  991.801002] ena 0000:00:06.0: ENA device version: 0.10
[  991.801768] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  991.900984] ena 0000:00:06.0: ENA Large LLQ is disabled
[  991.912925] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:1d:b4:ca:b9:bf
[  991.925413] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1021.874707] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1021.876052] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1126.206933] vfio-pci 0000:00:06.0: resetting
[ 1126.421918] vfio-pci 0000:00:06.0: reset done
[ 1126.423469] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6068)
[ 1126.425081] vfio-pci 0000:00:06.0: resetting
[ 1126.641717] vfio-pci 0000:00:06.0: reset done
[ 1126.920042] vfio-pci 0000:00:06.0: resetting
[ 1127.131681] vfio-pci 0000:00:06.0: reset done
[ 1127.133216] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6088)
[ 1127.134857] vfio-pci 0000:00:06.0: resetting
[ 1127.351680] vfio-pci 0000:00:06.0: reset done
[ 1128.641941] vfio-pci 0000:00:06.0: resetting
[ 1128.861749] vfio-pci 0000:00:06.0: reset done
[ 1128.863286] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6111)
[ 1128.864944] vfio-pci 0000:00:06.0: resetting
[ 1129.081685] vfio-pci 0000:00:06.0: reset done
[ 1130.371847] vfio-pci 0000:00:06.0: resetting
[ 1130.601762] vfio-pci 0000:00:06.0: reset done
[ 1130.603319] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6134)
[ 1130.604924] vfio-pci 0000:00:06.0: resetting
[ 1130.821683] vfio-pci 0000:00:06.0: reset done

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 62/merge | Commit: 8f630df1

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.57:9000 (MAC: 02:8e:4f:a1:61:a7)
echo listening on 10.0.1.57:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.128:9000 (MAC: 02:16:47:dc:6c:c9)
echo listening on 10.0.1.128:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.57:9000: 'arp-probe #1'
Test complete
[2026-05-29T10:47:00Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T10:47:00Z] INFO: Test: udp_send_receive
[2026-05-29T10:47:01Z] INFO: UDP send/receive succeeded
[2026-05-29T10:47:01Z] INFO: Test: echo_roundtrip
[2026-05-29T10:47:02Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T10:47:02Z] INFO: Test: payload_integrity
[2026-05-29T10:47:02Z] INFO: Response received, checking payload match...
[2026-05-29T10:47:02Z] INFO: Payload integrity verified (found in response)
[2026-05-29T10:47:02Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T10:47:02Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Instances Ready

  • TRex: i-0abb53353ff1e6de9 (10.0.1.84)
  • DUT: i-08eb8b5b75ff1530d (10.0.1.35)
  • Instance type: unknown

@github-actions
Copy link
Copy Markdown

[Perf] Stage: TRex Config

Starting TRex configuration (MAC discovery + NIC binding)...

@github-actions
Copy link
Copy Markdown

[Perf] Stage: TRex Config OK

  • TX: 0000:00:06.0 MAC: 02:23:aa:d5:e4:93
  • RX: 0000:00:07.0 MAC: 02:db:0a:19:d9:11
  • Gateway MAC: 02:ef:ac:06:8b:0f
    Starting TRex server...

@github-actions
Copy link
Copy Markdown

[Perf] Stage: TRex Started

TRex server running. Beginning benchmarks...

@github-actions
Copy link
Copy Markdown

[Perf] DUT Ready

DUT instance i-08eb8b5b75ff1530d SSM working, build complete.

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Benchmark (1/4)

Running plain-rust benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust preflight

TRex API OK: 2 ports
  Port 0: 02:23:aa:d5:e4:93
  Port 1: 02:db:0a:19:d9:11
PREFLIGHT_OK

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: plain-rust result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:23:aa:d5:e4:93

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,995 pps, Drop: 1.4351%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,893 pps, Drop: 0.3164%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 550,494 pps, Drop: 21.3579%, Lat avg: 0.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,893 pps, Drop: 0.3164%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 401,830 pps, Drop: 42.5958%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,973 pps, Drop: 0.7333%, Lat avg: 86.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,947 pps, Drop: 0.3008%, Lat avg: 183.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 389,116 pps, Drop: 44.412%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 42,718 pps, Drop: 38.9737%, Lat avg: 102.0 us
  Target: 140,000 pps ... TX: 125,292 pps, RX: 124,343 pps, Drop: 0.7574%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 125,236 pps, RX: 121,835 pps, Drop: 2.7157%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/plain-rust.json
EXIT_CODE=0

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Benchmark (2/4)

Running rust-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:23:aa:d5:e4:93
  Port 1: 02:db:0a:19:d9:11
PREFLIGHT_OK

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: rust-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:23:aa:d5:e4:93

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,982 pps, Drop: 0.727%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,969 pps, Drop: 0.2947%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 695,587 pps, Drop: 0.6304%, Lat avg: 282.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,992 pps, Drop: 0.2879%, Lat avg: 161.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 693,903 pps, Drop: 0.871%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 174.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 348,997 pps, Drop: 0.2864%, Lat avg: 284.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 558,263 pps, Drop: 20.2481%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 519.0 us
  Target: 140,000 pps ... TX: 125,313 pps, RX: 124,362 pps, Drop: 0.7589%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 125,313 pps, RX: 123,676 pps, Drop: 1.3059%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/rust-dpdk.json
EXIT_CODE=0

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Benchmark (3/4)

Running tokio-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: tokio-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:23:aa:d5:e4:93
  Port 1: 02:db:0a:19:d9:11
PREFLIGHT_OK

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: tokio-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:23:aa:d5:e4:93

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 68,999 pps, Drop: 1.4307%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 311,412 pps, Drop: 11.025%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 310,799 pps, Drop: 55.6001%, Lat avg: 0.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,000 pps, Drop: 0.7143%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 224,052 pps, Drop: 35.985%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 223,081 pps, Drop: 68.1312%, Lat avg: 0.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 69,000 pps, Drop: 1.4286%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 138,997 pps, Drop: 0.7162%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 145,402 pps, Drop: 58.4565%, Lat avg: 0.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 145,560 pps, Drop: 79.2057%, Lat avg: 0.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 53,460 pps, Drop: 23.628%, Lat avg: 0.0 us
  Target: 140,000 pps ... TX: 125,218 pps, RX: 56,798 pps, Drop: 54.6405%, Lat avg: 0.0 us
  Target: 350,000 pps ... TX: 125,305 pps, RX: 56,808 pps, Drop: 54.664%, Lat avg: 0.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/tokio-dpdk.json
EXIT_CODE=0

@github-actions
Copy link
Copy Markdown

[Perf] Stage: Benchmark (4/4)

Running native-dpdk benchmark...
Packet sizes: 64,512,1400,8500 | Duration: 30s/step | Target PPS: 70000,140000,350000,700000

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: native-dpdk preflight

TRex API OK: 2 ports
  Port 0: 02:23:aa:d5:e4:93
  Port 1: 02:db:0a:19:d9:11
PREFLIGHT_OK

@github-actions
Copy link
Copy Markdown

[Perf] Benchmark Diag: native-dpdk result

SSM exit: 0

Output (last 30 lines)
Duration per step: 30s
Source MAC: 02:23:aa:d5:e4:93

--- Packet size: 64B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 46.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 140,000 pps, Drop: 0.0%, Lat avg: 69.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,985 pps, Drop: 0.0044%, Lat avg: 72.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 698,590 pps, Drop: 0.2014%, Lat avg: 176.0 us

--- Packet size: 512B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 45.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 140,000 pps, Drop: 0.0%, Lat avg: 58.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 349,980 pps, Drop: 0.0057%, Lat avg: 78.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 698,803 pps, Drop: 0.171%, Lat avg: 365.0 us

--- Packet size: 1400B ---
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 58.0 us
  Target: 140,000 pps ... TX: 140,000 pps, RX: 139,989 pps, Drop: 0.008%, Lat avg: 61.0 us
  Target: 350,000 pps ... TX: 350,000 pps, RX: 350,000 pps, Drop: 0.0%, Lat avg: 67.0 us
  Target: 700,000 pps ... TX: 700,000 pps, RX: 647,469 pps, Drop: 7.5045%, Lat avg: 911.0 us

--- Packet size: 8500B ---
  (jumbo: capping at 418,133 pps for 30.0 Gbps limit)
  Target: 70,000 pps ... TX: 70,000 pps, RX: 70,000 pps, Drop: 0.0%, Lat avg: 66.0 us
  Target: 140,000 pps ... TX: 125,302 pps, RX: 125,302 pps, Drop: 0.0001%, Lat avg: 8495.0 us
  Target: 350,000 pps ... TX: 125,295 pps, RX: 124,337 pps, Drop: 0.7643%, Lat avg: 8951.0 us
  Target: 700,000 pps ... SKIPPED (exceeds 30.0 Gbps cap)

Results written to /tmp/perf-results/native-dpdk.json
EXIT_CODE=0

@github-actions
Copy link
Copy Markdown

[Perf] Diag: testpmd log

testpmd output (last 30 lines)
=== testpmd stats (last 30 lines) ===
  ############################################################################
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 121469099      RX-dropped: 0             RX-total: 121469099
  RX-error: 423977
  RX-nombufs: 0             
  TX-packets: 121467654      TX-dropped: 1445          TX-total: 121469099
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 121469099      RX-dropped: 0             RX-total: 121469099
  TX-packets: 121467654      TX-dropped: 1445          TX-total: 121469099
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Shutting down port 0...
ena_rx_queue_release(): Rx queue 0:0 released
ena_tx_queue_release(): Tx queue 0:0 released
Closing ports...
Port 0 is closed
Done

Bye...
=== ENA port stats ===
N/A (vfio-pci)

Performance test results from GH Actions run 26633424088:
- rust-dpdk 700K/64B: 695,587 RX (0.6% drop) — no regression vs Run #25
- rust-dpdk 700K/512B: 693,903 RX (0.9% drop) — no regression
- tokio-dpdk caps at ~311K PPS — unchanged
- native-dpdk baseline: 698,590 at 700K/64B

Conclusion: IPv6 socket address support is performance-neutral.
@gspivey gspivey merged commit 84ea3f0 into development May 29, 2026
5 checks passed
@github-actions
Copy link
Copy Markdown

[Perf] Stage: Results

[11:54:11] INFO Generating markdown summary...

Performance Test Results — unknown

Commit: 7cd3da0c
Timestamp: 2026-05-29T11:54:11.956970+00:00

64B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% 46.0 210.0 35.8 35.8
native-dpdk 140,000 140,000 140,000 0.00% 69.0 258.0 71.7 71.7
native-dpdk 350,000 350,000 349,985 0.00% 72.0 898.0 179.2 179.2
native-dpdk 700,000 700,000 698,590 0.20% 176.0 1340.0 358.4 357.7
rust-dpdk 70,000 70,000 69,000 1.43% 0 32,961 0 0 0.0 0.0 35.8 35.3
rust-dpdk 140,000 140,000 138,982 0.73% 0 41,882 0 0 0.0 0.0 71.7 71.2
rust-dpdk 350,000 350,000 348,969 0.29% 0 34,841 0 0 0.0 0.0 179.2 178.7
rust-dpdk 700,000 700,000 695,587 0.63% 0 31,775 0 0 282.0 0.0 358.4 356.1
tokio-dpdk 70,000 70,000 68,999 1.43% 0 31,489 0 0 0.0 0.0 35.8 38.6
tokio-dpdk 140,000 140,000 139,000 0.71% 0 39,196 0 0 0.0 0.0 71.7 77.8
tokio-dpdk 350,000 350,000 311,412 11.03% 0 19,532 0 0 0.0 0.0 179.2 174.4
tokio-dpdk 700,000 700,000 310,799 55.60% 0 23,589 0 0 0.0 0.0 358.4 174.1
plain-rust 70,000 70,000 68,995 1.44% 0.0 0.0 35.8 35.3
plain-rust 140,000 140,000 139,000 0.71% 0.0 0.0 71.7 71.2
plain-rust 350,000 350,000 348,893 0.32% 0.0 0.0 179.2 178.6
plain-rust 700,000 700,000 550,494 21.36% 0.0 0.0 358.4 281.9

512B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% 45.0 347.0 286.7 286.7
native-dpdk 140,000 140,000 140,000 0.00% 58.0 215.0 573.4 573.4
native-dpdk 350,000 350,000 349,980 0.01% 78.0 434.0 1433.6 1433.5
native-dpdk 700,000 700,000 698,803 0.17% 365.0 2564.0 2867.2 2862.3
rust-dpdk 70,000 70,000 69,000 1.43% 0 41,864 0 0 0.0 0.0 286.7 282.6
rust-dpdk 140,000 140,000 139,000 0.71% 0 32,666 0 0 0.0 0.0 573.4 569.3
rust-dpdk 350,000 350,000 348,992 0.29% 0 41,843 0 0 161.0 0.0 1433.6 1429.5
rust-dpdk 700,000 700,000 693,903 0.87% 0 34,399 0 0 0.0 0.0 2867.2 2842.2
tokio-dpdk 70,000 70,000 69,000 1.43% 0 35,315 0 0 0.0 0.0 286.7 285.9
tokio-dpdk 140,000 140,000 139,000 0.71% 0 31,094 0 0 0.0 0.0 573.4 576.0
tokio-dpdk 350,000 350,000 224,052 35.98% 0 20,190 0 0 0.0 0.0 1433.6 928.5
tokio-dpdk 700,000 700,000 223,081 68.13% 0 12,153 0 0 0.0 0.0 2867.2 924.5
plain-rust 70,000 70,000 69,000 1.43% 0.0 0.0 286.7 282.6
plain-rust 140,000 140,000 139,000 0.71% 0.0 0.0 573.4 569.3
plain-rust 350,000 350,000 348,893 0.32% 0.0 0.0 1433.6 1429.1
plain-rust 700,000 700,000 401,830 42.60% 0.0 0.0 2867.2 1645.9

1400B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% 58.0 215.0 784.0 784.0
native-dpdk 140,000 140,000 139,989 0.01% 61.0 4974.0 1568.0 1567.9
native-dpdk 350,000 350,000 350,000 0.00% 67.0 430.0 3920.0 3920.0
native-dpdk 700,000 700,000 647,469 7.50% 911.0 2225.0 7840.0 7251.6
rust-dpdk 70,000 70,000 69,000 1.43% 0 31,963 0 0 0.0 0.0 784.0 772.8
rust-dpdk 140,000 140,000 139,000 0.71% 0 41,901 0 0 174.0 0.0 1568.0 1556.8
rust-dpdk 350,000 350,000 348,997 0.29% 0 31,575 0 0 284.0 0.0 3920.0 3908.8
rust-dpdk 700,000 700,000 558,263 20.25% 0 32,943 0 0 0.0 0.0 7840.0 6252.6
tokio-dpdk 70,000 70,000 69,000 1.43% 0 39,937 0 0 0.0 0.0 784.0 776.1
tokio-dpdk 140,000 140,000 138,997 0.72% 0 31,113 0 0 0.0 0.0 1568.0 1563.4
tokio-dpdk 350,000 350,000 145,402 58.46% 0 8,150 0 0 0.0 0.0 3920.0 1635.5
tokio-dpdk 700,000 700,000 145,560 79.21% 0 7,785 0 0 0.0 0.0 7840.0 1637.3
plain-rust 70,000 70,000 69,000 1.43% 0.0 0.0 784.0 772.8
plain-rust 140,000 140,000 138,973 0.73% 86.0 0.0 1568.0 1556.5
plain-rust 350,000 350,000 348,947 0.30% 183.0 0.0 3920.0 3908.2
plain-rust 700,000 700,000 389,116 44.41% 0.0 0.0 7840.0 4358.1

8500B packets

Config Target PPS TX pps RX pps Drop % NIC imissed NIC ierrors NIC nombuf App Drops Lat Avg (us) Lat Max (us) TX Mbps RX Mbps
native-dpdk 70,000 70,000 70,000 0.00% 66.0 138.0 4760.0 4760.0
native-dpdk 140,000 125,302 125,302 0.00% 8495.0 9213.0 8520.6 8520.5
native-dpdk 350,000 125,295 124,337 0.76% 8951.0 9759.0 8520.0 8454.9
rust-dpdk 70,000 70,000 69,000 1.43% 0 33,610 0 0 519.0 0.0 4760.0 4692.0
rust-dpdk 140,000 125,313 124,362 0.76% 0 28,459 0 0 0.0 0.0 8521.3 8456.6
rust-dpdk 350,000 125,313 123,676 1.31% 0 17,066 0 0 0.0 0.0 8521.3 8410.0
tokio-dpdk 70,000 70,000 53,460 23.63% 0 4,222 0 0 0.0 0.0 4760.0 3637.9
tokio-dpdk 140,000 125,218 56,798 54.64% 0 13,810 0 0 0.0 0.0 8514.8 3865.0
tokio-dpdk 350,000 125,305 56,808 54.66% 0 4,849 0 0 0.0 0.0 8520.8 3865.7
plain-rust 70,000 70,000 42,718 38.97% 102.0 0.0 4760.0 2904.8
plain-rust 140,000 125,292 124,343 0.76% 0.0 0.0 8519.9 8455.3
plain-rust 350,000 125,236 121,835 2.72% 0.0 0.0 8516.1 8284.8

NIC Drops Instrumentation Self-Check

Compares (NIC-FINAL − NIC-BASELINE) one-shot snapshots against the sum of per-tick [PERF] deltas over the reporter's lifetime. These MUST match exactly — a mismatch means per-tick delta bookkeeping is losing data.

Config Status imissed (expected / actual / Δ) ierrors (expected / actual / Δ) rx_nombuf (expected / actual / Δ)
native-dpdk no instrumentation
rust-dpdk OK 0 / 0 / 0 418,223 / 418,223 / 0 0 / 0 / 0
tokio-dpdk OK 0 / 0 / 0 266,133 / 266,133 / 0 0 / 0 / 0
plain-rust no instrumentation

expected = FINAL − BASELINE (raw NIC counter delta across reporter lifetime). actual = sum of per-tick [PERF] delta fields. Any Δ ≠ 0 is a bug in the tick loop's bookkeeping.

plain-rust Kernel NIC Drops (ethtool -S delta)

ethtool snapshots not available — baseline or final file missing in $LOGS_DIR/dut-plain-rust-ethtool-*.txt.

Full results artifact

@github-actions
Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 103d6c19

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 9.1M 10.9M 0.8x 92
RX recv_from 64B 3.4M 4.2M 0.8x 236
TX send_to 1400B 1.5M 1.6M 1.0x 615
RX recv_from 1400B 941.0K 1.0M 0.9x 957

Avg sync/async ratio: 0.9x, worst: 1.0x

Good: Async wrapper is within 1.0x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 103d6c19

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.1M 10.2M 1.1x 98
RX recv_from 64B 3.3M 3.7M 0.9x 269
TX send_to 1400B 1.7M 1.7M 1.0x 605
RX recv_from 1400B 991.8K 1.0M 1.0x 976

Avg sync/async ratio: 1.0x, worst: 1.1x

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-06ce7b186762431d0 (DPDK ENI: 10.0.1.34)
  • Receiver: i-050ac081763788bda (DPDK ENI: 10.0.1.221)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-08fa131210ebebc9c (DPDK ENI: 10.0.1.236)
  • Receiver: i-06cf53222d0d1f6d0 (DPDK ENI: 10.0.1.233)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

1 similar comment
@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed (Run 26635758677)

Branch: 62/merge | Commit: 103d6c19

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.221:9000 (MAC: 02:e3:f7:7a:d7:ad)
echo listening on 10.0.1.221:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.34:9000 (MAC: 02:39:f4:44:fd:5b)
echo listening on 10.0.1.34:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.221:9000: 'arp-probe #1'
Test complete
[2026-05-29T12:07:02Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T12:07:02Z] INFO: Test: udp_send_receive
[2026-05-29T12:07:03Z] INFO: UDP send/receive succeeded
[2026-05-29T12:07:03Z] INFO: Test: echo_roundtrip
[2026-05-29T12:07:04Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T12:07:04Z] INFO: Test: payload_integrity
[2026-05-29T12:07:04Z] INFO: Response received, checking payload match...
[2026-05-29T12:07:04Z] INFO: Payload integrity verified (found in response)
[2026-05-29T12:07:04Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T12:07:04Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-29T12:14:02Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-29T12:14:02Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-29T12:14:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-29T12:14:02Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-29T12:13:03Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-29T12:13:03Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-29T12:13:03Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-29T12:13:03Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.221:9000 (MAC: 02:e3:f7:7a:d7:ad)
echo listening on 10.0.1.221:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.34:9000 (MAC: 02:39:f4:44:fd:5b)
echo listening on 10.0.1.34:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-05-29T12:03:18Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.221:9000
Bind address: 10.0.1.34:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.34:32768 (MAC: 02:39:f4:44:fd:5b)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.221:9000: 'arp-probe #1'
Test complete
[2026-05-29T12:03:19Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T12:03:19Z] INFO: Test: udp_send_receive
[2026-05-29T12:03:21Z] INFO: UDP send/receive succeeded
[2026-05-29T12:03:21Z] INFO: Test: echo_roundtrip
[2026-05-29T12:03:22Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T12:03:22Z] INFO: Test: payload_integrity
[2026-05-29T12:03:23Z] INFO: Response received, checking payload match...
[2026-05-29T12:03:23Z] INFO: Payload integrity verified (found in response)
[2026-05-29T12:03:23Z] INFO: Test: jumbo_diagnostics
[2026-05-29T12:03:23Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-05-29T12:03:23Z] INFO: Interface MTU:
  9001
  65536
[2026-05-29T12:03:23Z] INFO:   ens5: MTU=9001
[2026-05-29T12:03:23Z] INFO:   lo: MTU=65536
[2026-05-29T12:03:23Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-05-29T12:03:23Z] INFO: DPDK port config (from echo server log):
[2026-05-29T12:03:23Z] INFO:   (no MTU info in echo log)
[2026-05-29T12:03:23Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-05-29T12:03:23Z] INFO: Test: jumbo_echo_8000
[2026-05-29T12:03:25Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.221:9000
Bind address: 10.0.1.34:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.34:32768 (MAC: 02:39:f4:44:fd:5b)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.221:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.221:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.221:9000 (expected 8000, OK)
Test complete
[2026-05-29T12:03:25Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-05-29T12:03:25Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-29T12:03:25Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-29T12:07:02Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.221:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.221:9000: 'arp-probe #1'
Test complete
[2026-05-29T12:07:02Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T12:07:02Z] INFO: Test: udp_send_receive
[2026-05-29T12:07:03Z] INFO: UDP send/receive succeeded
[2026-05-29T12:07:03Z] INFO: Test: echo_roundtrip
[2026-05-29T12:07:04Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T12:07:04Z] INFO: Test: payload_integrity
[2026-05-29T12:07:04Z] INFO: Response received, checking payload match...
[2026-05-29T12:07:04Z] INFO: Payload integrity verified (found in response)
[2026-05-29T12:07:04Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T12:07:04Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-29T12:14:02Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-29T12:14:02Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-29T12:14:02Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-29T12:14:02Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-05-29T12:13:00Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-05-29T12:13:00Z] INFO: Local IP: 10.0.1.34, Peer IP: 10.0.1.221, Port: 9000
[2026-05-29T12:13:00Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.34        0x1         0x2         02:39:f4:44:fd:5b     *        ens5
10.0.1.22        0x1         0x2         02:d3:e0:64:67:8f     *        ens5
10.0.1.1         0x1         0x2         02:c9:7f:c6:af:ef     *        ens5
10.0.1.234       0x1         0x2         02:2f:be:a7:aa:8d     *        ens5
10.0.1.221       0x1         0x2         02:e3:f7:7a:d7:ad     *        ens5
[2026-05-29T12:13:00Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-05-29T12:13:00Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-05-29T12:13:00Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1916248 May 29 12:01 /opt/dpdk-stdlib/target/release/test-client
[2026-05-29T12:13:00Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.221 --port 9000 --bind-ip 10.0.1.34 --count 10 --delay 200
[2026-05-29T12:13:03Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.221:9000
Bind address: 10.0.1.34:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.34:32768 (MAC: 02:39:f4:44:fd:5b)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.221:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-29T12:13:03Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-29T12:13:03Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-29T12:13:03Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-29T12:13:03Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-050ac081763788bda (receiver)
Command ID: 4d3aff78-35b8-4e17-866c-5f7378f87bd5

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-050ac081763788bda (receiver)
Command ID: e8f29c79-0476-494e-92c5-ea5065ba3d88

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-050ac081763788bda (receiver)
Command ID: 20e469b5-4323-4ea4-9903-f8e34ff5798d

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-050ac081763788bda (receiver)
Command ID: 443af933-7393-414e-91b6-084c085831ad

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-06ce7b186762431d0 (sender)
Command ID: 759c7993-6ab1-4f55-ac6c-787e94d2750d

=== STDOUT ===


=== STDERR ===


Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:57:03:60:57:d1 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-06a45495afba1add2
    altname device-number-0.0
    inet 10.0.1.71/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2341sec preferred_lft 2341sec
    inet6 fe80::57:3ff:fe60:57d1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:fb:d0:eb:e3:41 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-09120eb2afda8e6f1
    altname device-number-0.0
    inet 10.0.1.31/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2364sec preferred_lft 2364sec
    inet6 fe80::fb:d0ff:feeb:e341/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-29T12:02:40Z
hostname: ip-10-0-1-71.ec2.internal
kernel: 6.18.30-61.116.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:57:03:60:57:d1 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-06a45495afba1add2
    altname device-number-0.0
    inet 10.0.1.71/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3326sec preferred_lft 3326sec
    inet6 fe80::57:3ff:fe60:57d1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.234 dev ens5 lladdr 02:2f:be:a7:aa:8d REACHABLE 
10.0.1.1 dev ens5 lladdr 02:c9:7f:c6:af:ef REACHABLE 
10.0.1.22 dev ens5 lladdr 02:d3:e0:64:67:8f REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.71 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.71 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.71 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.71 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:57:03:60:57:d1/ 02:e3:f7:7a:d7:ad/ 

--- ENI: 02:57:03:60:57:d1/ ---
  device-number: 0
  local-ipv4s: 10.0.1.71
  subnet-id: subnet-000b24c232d6be084
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:e3:f7:7a:d7:ad/ ---
  device-number: 1
  local-ipv4s: 10.0.1.221
  subnet-id: subnet-000b24c232d6be084
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:c9:7f:c6:af:ef REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.71 ens5
Unicast reply from 10.0.1.1 [02:C9:7F:C6:AF:EF]  0.542ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 29 12:02 .
drwxr-xr-x. 14 root root     3100 May 29 12:01 ..
crw-------.  1 root root 243,   0 May 29 12:02 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 29 11:58 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.054137] printk: legacy console [ttyS0] enabled
[    0.055176] x2apic enabled
[    0.059420] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.059527] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.069287] audit: type=2000 audit(1780055879.584:1): state=initialized audit_enabled=0 res=1
[    0.088861] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.157496] ACPI: Interpreter enabled
[    0.157496] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.169987] pci 0000:00:05.0: enabling Extended Tags
[    0.233227] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.246985] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.250455] IPI shorthand broadcast: enabled
[    4.225848] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    4.261962] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    4.331787] VFIO - User Level meta-driver version: 0.3
[    4.736427] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    4.744784] ena 0000:00:05.0: ENA device version: 0.10
[    4.745491] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    4.844709] ena 0000:00:05.0: ENA Large LLQ is disabled
[    4.857232] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:57:03:60:57:d1
[    4.903155] ena 0000:00:05.0 ens5: renamed from eth0
[  230.477217] pci 0000:00:06.0: enabling Extended Tags
[  230.481218] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  230.491630] ena 0000:00:06.0: ENA device version: 0.10
[  230.492413] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  230.591609] ena 0000:00:06.0: ENA Large LLQ is disabled
[  230.603485] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e3:f7:7a:d7:ad
[  230.610947] ena 0000:00:06.0 ens6: renamed from eth0
[  267.484626] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  267.486069] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-29T12:02:32Z
hostname: ip-10-0-1-31.ec2.internal
kernel: 6.18.30-61.116.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:fb:d0:eb:e3:41 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-09120eb2afda8e6f1
    altname device-number-0.0
    inet 10.0.1.31/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3334sec preferred_lft 3334sec
    inet6 fe80::fb:d0ff:feeb:e341/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.81 dev ens5 lladdr 02:64:79:bf:24:89 STALE 
10.0.1.22 dev ens5 lladdr 02:d3:e0:64:67:8f REACHABLE 
10.0.1.1 dev ens5 lladdr 02:c9:7f:c6:af:ef REACHABLE 
10.0.1.234 dev ens5 lladdr 02:2f:be:a7:aa:8d REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.31 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.31 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.31 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.31 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:39:f4:44:fd:5b/ 02:fb:d0:eb:e3:41/ 

--- ENI: 02:39:f4:44:fd:5b/ ---
  device-number: 1
  local-ipv4s: 10.0.1.34
  subnet-id: subnet-000b24c232d6be084
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:fb:d0:eb:e3:41/ ---
  device-number: 0
  local-ipv4s: 10.0.1.31
  subnet-id: subnet-000b24c232d6be084
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:c9:7f:c6:af:ef REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.31 ens5
Unicast reply from 10.0.1.1 [02:C9:7F:C6:AF:EF]  0.529ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 29 12:02 .
drwxr-xr-x. 14 root root     3100 May 29 12:01 ..
crw-------.  1 root root 243,   0 May 29 12:02 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 29 11:58 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.053291] printk: legacy console [ttyS0] enabled
[    0.054323] x2apic enabled
[    0.058543] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.058650] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.068421] audit: type=2000 audit(1780055879.804:1): state=initialized audit_enabled=0 res=1
[    0.087992] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.148709] ACPI: Interpreter enabled
[    0.148709] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.167966] pci 0000:00:05.0: enabling Extended Tags
[    0.212837] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.219382] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.222040] IPI shorthand broadcast: enabled
[    3.975696] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    4.007496] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    4.074492] VFIO - User Level meta-driver version: 0.3
[    4.638375] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    4.658354] ena 0000:00:05.0: ENA device version: 0.10
[    4.659083] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    4.774680] ena 0000:00:05.0: ENA Large LLQ is disabled
[    4.786716] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:fb:d0:eb:e3:41
[    4.821303] ena 0000:00:05.0 ens5: renamed from eth0
[  230.779830] pci 0000:00:06.0: enabling Extended Tags
[  230.783842] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  230.792232] ena 0000:00:06.0: ENA device version: 0.10
[  230.792984] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  230.892206] ena 0000:00:06.0: ENA Large LLQ is disabled
[  230.904309] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:39:f4:44:fd:5b
[  230.911266] ena 0000:00:06.0 ens6: renamed from eth0
[  260.606652] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  260.608102] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.068667] pid_max: default: 32768 minimum: 301
[    0.173649] iommu: Default domain type: Translated
[    0.173649] NetLabel:  unlabeled traffic allowed by default
[    0.192776] PCI: CLS 0 bytes, default 64
[    0.271222] nvme nvme0: 2/0/0 default/read/poll queues
[    0.428730] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.428853] systemd[1]: No hostname configured, using default hostname.
[    0.495271] systemd[1]: Queued start job for default target initrd.target.
[    3.252136] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  303.663789] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4428)
[  527.410580] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4867)
[ 1128.742214] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6026)

sender-dmesg-crashes.log

[    0.067805] pid_max: default: 32768 minimum: 301
[    0.171633] iommu: Default domain type: Translated
[    0.171633] NetLabel:  unlabeled traffic allowed by default
[    0.191965] PCI: CLS 0 bytes, default 64
[    0.245484] nvme nvme0: 2/0/0 default/read/poll queues
[    0.419735] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.419860] systemd[1]: No hostname configured, using default hostname.
[    0.488593] systemd[1]: Queued start job for default target initrd.target.
[    2.952435] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  945.167065] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:5630)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  303.662222] vfio-pci 0000:00:06.0: reset done
[  303.663789] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4428)
[  303.665254] vfio-pci 0000:00:06.0: resetting
[  303.882090] vfio-pci 0000:00:06.0: reset done
[  383.182917] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  384.196165] ena 0000:00:06.0: ENA device version: 0.10
[  384.196932] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  384.298123] ena 0000:00:06.0: ENA Large LLQ is disabled
[  384.310798] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e3:f7:7a:d7:ad
[  384.319145] ena 0000:00:06.0 ens6: renamed from eth0
[  403.853822] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  403.855036] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  527.185678] vfio-pci 0000:00:06.0: resetting
[  527.409018] vfio-pci 0000:00:06.0: reset done
[  527.410580] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4867)
[  527.412057] vfio-pci 0000:00:06.0: resetting
[  527.628897] vfio-pci 0000:00:06.0: reset done
[  601.239422] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  602.252291] ena 0000:00:06.0: ENA device version: 0.10
[  602.253046] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  602.354214] ena 0000:00:06.0: ENA Large LLQ is disabled
[  602.366912] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:e3:f7:7a:d7:ad
[  602.378419] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1043.893582] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1043.894990] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1128.530981] vfio-pci 0000:00:06.0: resetting
[ 1128.740649] vfio-pci 0000:00:06.0: reset done
[ 1128.742214] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6026)
[ 1128.743660] vfio-pci 0000:00:06.0: resetting
[ 1128.960515] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[  945.168527] vfio-pci 0000:00:06.0: resetting
[  945.385364] vfio-pci 0000:00:06.0: reset done
[ 1006.665425] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1007.678418] ena 0000:00:06.0: ENA device version: 0.10
[ 1007.679163] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1007.780330] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1007.793104] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:39:f4:44:fd:5b
[ 1007.801613] ena 0000:00:06.0 ens6: renamed from eth0
[ 1037.018254] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1037.019502] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1144.190985] vfio-pci 0000:00:06.0: resetting
[ 1144.413197] vfio-pci 0000:00:06.0: reset done
[ 1144.414747] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6073)
[ 1144.416381] vfio-pci 0000:00:06.0: resetting
[ 1144.633083] vfio-pci 0000:00:06.0: reset done
[ 1144.911245] vfio-pci 0000:00:06.0: resetting
[ 1145.133090] vfio-pci 0000:00:06.0: reset done
[ 1145.134627] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6093)
[ 1145.136222] vfio-pci 0000:00:06.0: resetting
[ 1145.353050] vfio-pci 0000:00:06.0: reset done
[ 1146.642858] vfio-pci 0000:00:06.0: resetting
[ 1146.863079] vfio-pci 0000:00:06.0: reset done
[ 1146.864633] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6116)
[ 1146.866229] vfio-pci 0000:00:06.0: resetting
[ 1147.083004] vfio-pci 0000:00:06.0: reset done
[ 1148.372802] vfio-pci 0000:00:06.0: resetting
[ 1148.603056] vfio-pci 0000:00:06.0: reset done
[ 1148.604601] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6139)
[ 1148.606196] vfio-pci 0000:00:06.0: resetting
[ 1148.822960] vfio-pci 0000:00:06.0: reset done

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 62/merge | Commit: 103d6c19

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.233:9000 (MAC: 02:4d:b2:06:b4:0f)
echo listening on 10.0.1.233:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.236:9000 (MAC: 02:9e:b4:e4:d7:0f)
echo listening on 10.0.1.236:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.233:9000: 'arp-probe #1'
Test complete
[2026-05-29T12:07:12Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-29T12:07:12Z] INFO: Test: udp_send_receive
[2026-05-29T12:07:13Z] INFO: UDP send/receive succeeded
[2026-05-29T12:07:13Z] INFO: Test: echo_roundtrip
[2026-05-29T12:07:14Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-29T12:07:14Z] INFO: Test: payload_integrity
[2026-05-29T12:07:14Z] INFO: Response received, checking payload match...
[2026-05-29T12:07:14Z] INFO: Payload integrity verified (found in response)
[2026-05-29T12:07:14Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-29T12:07:14Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant