Skip to content

ICMPv6 error handling#58

Merged
gspivey merged 3 commits into
developmentfrom
agent/icmpv6-error-handling
May 20, 2026
Merged

ICMPv6 error handling#58
gspivey merged 3 commits into
developmentfrom
agent/icmpv6-error-handling

Conversation

@gspivey
Copy link
Copy Markdown
Owner

@gspivey gspivey commented May 20, 2026

Roadmap item

IPv6 task 8: ICMPv6 error handling — Destination Unreachable, Packet Too Big (with Next-Hop MTU), Time Exceeded, and Parameter Problem parsed and matched back to the originating socket.

Changes

Adds ICMPv6 error message parsing and handling to dpdk-udp/src/icmpv6.rs, mirroring the existing IPv4 ICMP error handling pattern in icmp.rs:

  • Icmpv6ErrorInfo struct: captures error type/code, error source, original src/dst IP and ports, MTU
  • parse_icmpv6_error(): parses raw ICMPv6 error frames, walks extension headers in the embedded original packet to locate the UDP header
  • Icmpv6Action enum: Reply(Vec<u8>) | Error(Icmpv6ErrorInfo) — parallels IcmpAction
  • process_icmpv6_full() on Icmpv6Handler: combined echo + error processing
  • RX path integration in lib.rs: errors matched to socket via original source port, queued via existing error_queue/take_error() infrastructure

Error type → io::ErrorKind mapping

ICMPv6 Error io::ErrorKind
Port Unreachable ConnectionRefused
Admin Prohibited PermissionDenied
Hop Limit / Frag Reassembly Exceeded TimedOut
Parameter Problem InvalidData
Packet Too Big, No Route, others Other

Tests added

24 new unit tests covering:

  • All error type parsing (Dest Unreachable variants, Packet Too Big, Time Exceeded, Parameter Problem)
  • Edge cases (truncated frames, IPv4 frames, echo packets rejected as errors)
  • to_io_error() mapping for each error kind
  • process_icmpv6_full() handler routing (echo reply, error for our IP, ignore other IPs)
  • Synthetic performance benchmark (~10K iterations)

Tradeoffs

  • Error queue is shared with IPv4 ICMP errors (same queue_icmp_error path, same 16-entry bound). This matches the existing design where take_error() returns errors from both protocols.
  • Port matching uses only the source port (same as IPv4 path). Full 5-tuple matching would require connected-socket state that doesn't exist yet for IPv6.
  • The build_icmpv6_error_frame helper is #[cfg(test)] only — not exposed in the public API.

Agent Router added 3 commits May 20, 2026 08:15
Implement ICMPv6 error message parsing and handling (IPv6 roadmap task 8):

- Destination Unreachable (type 1): no route, admin prohibited, beyond
  scope, address unreachable, port unreachable
- Packet Too Big (type 2): carries Next-Hop MTU
- Time Exceeded (type 3): hop limit exceeded, fragment reassembly
- Parameter Problem (type 4): erroneous header, unrecognized next header

Adds:
- Icmpv6ErrorInfo struct with to_io_error() mapping (mirrors IcmpErrorInfo)
- parse_icmpv6_error() extracts original IPv6+UDP headers from error payload
- Icmpv6Action enum (Reply | Error) paralleling IcmpAction
- process_icmpv6_full() on Icmpv6Handler for combined echo+error handling
- RX path integration: errors matched to socket via original src port,
  queued via existing error_queue/take_error() infrastructure

24 new tests covering all error types, edge cases, io::Error mappings,
handler routing, and a synthetic parse performance benchmark.
gspivey pushed a commit that referenced this pull request May 20, 2026
Implement ICMPv6 error message parsing and handling (IPv6 roadmap task 8):

- Destination Unreachable (type 1): no route, admin prohibited, beyond
  scope, address unreachable, port unreachable
- Packet Too Big (type 2): carries Next-Hop MTU
- Time Exceeded (type 3): hop limit exceeded, fragment reassembly
- Parameter Problem (type 4): erroneous header, unrecognized next header

Adds Icmpv6ErrorInfo, parse_icmpv6_error(), Icmpv6Action enum,
process_icmpv6_full() on Icmpv6Handler, and RX path integration via
the existing error_queue/take_error() infrastructure.

24 new tests, no performance regression (Run #17 in perf-test-log.md).
@gspivey gspivey merged commit 628ab39 into development May 20, 2026
1 check passed
@gspivey gspivey deleted the agent/icmpv6-error-handling branch May 20, 2026 12:33
@github-actions
Copy link
Copy Markdown

Synthetic Performance Results (run)

Commit: 28e10240

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 12.0M 11.3M 1.1x 88
RX recv_from 64B 3.8M 4.8M 0.8x 207
TX send_to 1400B 1.8M 1.8M 1.0x 549
RX recv_from 1400B 1.1M 1.2M 0.9x 810

Avg sync/async ratio: 0.9x, worst: 1.1x

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

Synthetic Performance Results — Graviton (run)

Commit: 28e10240

✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)
✅ synthetic UDP socket bound to 10.0.0.1:9000 (MAC: 02:00:00:00:00:01)

Synthetic UDP Performance Results

Measures framework overhead: sync dpdk_udp::UdpSocket vs async (std::sync::Mutex + try_recv_from).

Test Payload Sync PPS Async PPS Ratio (sync/async) Async ns/op
TX send_to 64B 11.5M 10.9M 1.1x 91
RX recv_from 64B 3.5M 4.7M 0.8x 214
TX send_to 1400B 1.6M 1.6M 1.0x 613
RX recv_from 1400B 1.0M 1.1M 1.0x 930

Avg sync/async ratio: 0.9x, worst: 1.1x

Good: Async wrapper is within 1.1x of sync — minimal framework overhead.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0d826e4b5a8d7ae79 (DPDK ENI: 10.0.1.130)
  • Receiver: i-04761bd8476a12885 (DPDK ENI: 10.0.1.136)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Deploy

Infrastructure ready.

  • Sender: i-0b90c592bd3c8b6db (DPDK ENI: 10.0.1.240)
  • Receiver: i-04fe31b97fa3c22e7 (DPDK ENI: 10.0.1.89)
  • Both instances SSM-ready.

@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

1 similar comment
@github-actions
Copy link
Copy Markdown

[CI] Stage: Summary

All tests PASSED.

ARP seeding: kernel /proc/net/arp (automatic)

  • tier1-dpdk-echo: 6 tests, 0 failures
  • tier2-kernel-interop: 4 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures
  • tier3-iperf-interop: 1 tests, 0 failures

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed — Graviton (run)

Branch: 58/merge | Commit: 28e10240

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures
  • tier2-kernel-interop.xml: 4 tests, 0 failures
  • tier3-iperf-sends.xml: 1 tests, 0 failures
  • tier3-our-app-sends.xml: 1 tests, 0 failures

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.89:9000 (MAC: 02:e3:d8:cb:2f:ab)
echo listening on 10.0.1.89:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.240:9000 (MAC: 02:8b:d5:5e:89:81)
echo listening on 10.0.1.240:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.89:9000: 'arp-probe #1'
Test complete
[2026-05-20T12:48:40Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-20T12:48:40Z] INFO: Test: udp_send_receive
[2026-05-20T12:48:41Z] INFO: UDP send/receive succeeded
[2026-05-20T12:48:41Z] INFO: Test: echo_roundtrip
[2026-05-20T12:48:42Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-20T12:48:42Z] INFO: Test: payload_integrity
[2026-05-20T12:48:42Z] INFO: Response received, checking payload match...
[2026-05-20T12:48:42Z] INFO: Payload integrity verified (found in response)
[2026-05-20T12:48:42Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-20T12:48:42Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

@github-actions
Copy link
Copy Markdown

✅ Integration Tests Passed (Run 26162064326)

Branch: 58/merge | Commit: 28e10240

Test Results

  • tier1-dpdk-echo.xml: 6 tests, 0 failures, skipped
  • tier2-kernel-interop.xml: 4 tests, 0 failures, skipped
  • tier3-iperf-sends.xml: 1 tests, 0 failures, skipped
  • tier3-our-app-sends.xml: 1 tests, 0 failures, skipped

Application Logs (last 20 lines)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.136:9000 (MAC: 02:8b:0b:9c:38:bf)
echo listening on 10.0.1.136:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.130:9000 (MAC: 02:48:7d:a4:0f:99)
echo listening on 10.0.1.130:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.136:9000: 'arp-probe #1'
Test complete
[2026-05-20T12:48:01Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-20T12:48:01Z] INFO: Test: udp_send_receive
[2026-05-20T12:48:02Z] INFO: UDP send/receive succeeded
[2026-05-20T12:48:02Z] INFO: Test: echo_roundtrip
[2026-05-20T12:48:03Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-20T12:48:03Z] INFO: Test: payload_integrity
[2026-05-20T12:48:03Z] INFO: Response received, checking payload match...
[2026-05-20T12:48:03Z] INFO: Payload integrity verified (found in response)
[2026-05-20T12:48:03Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-20T12:48:03Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-20T12:55:28Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-20T12:55:28Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-20T12:55:28Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-20T12:55:28Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-20T12:54:29Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-20T12:54:29Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-20T12:54:29Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-20T12:54:29Z] INFO: our-app-sends test complete
Full Application Logs (last 200 lines each)

receiver-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.136:9000 (MAC: 02:8b:0b:9c:38:bf)
echo listening on 10.0.1.136:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-echo-server.log

EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.130:9000 (MAC: 02:48:7d:a4:0f:99)
echo listening on 10.0.1.130:9000 (MTU=9001, max_udp_payload=8973)
Shutting down gracefully...

sender-test-client.log

[2026-05-20T12:43:43Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.136:9000
Bind address: 10.0.1.130:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.130:32768 (MAC: 02:48:7d:a4:0f:99)
Backend: dpdk
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.136:9000: 'arp-probe #1'
Test complete
[2026-05-20T12:43:44Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-20T12:43:44Z] INFO: Test: udp_send_receive
[2026-05-20T12:43:46Z] INFO: UDP send/receive succeeded
[2026-05-20T12:43:46Z] INFO: Test: echo_roundtrip
[2026-05-20T12:43:48Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-20T12:43:48Z] INFO: Test: payload_integrity
[2026-05-20T12:43:48Z] INFO: Response received, checking payload match...
[2026-05-20T12:43:48Z] INFO: Payload integrity verified (found in response)
[2026-05-20T12:43:48Z] INFO: Test: jumbo_diagnostics
[2026-05-20T12:43:48Z] INFO: === JUMBO FRAME DIAGNOSTICS ===
[2026-05-20T12:43:48Z] INFO: Interface MTU:
  9001
  65536
[2026-05-20T12:43:48Z] INFO:   ens5: MTU=9001
[2026-05-20T12:43:48Z] INFO:   lo: MTU=65536
[2026-05-20T12:43:48Z] INFO: Routing table MTU column:
Iface	Destination	Gateway 	Flags	RefCnt	Use	Metric	Mask		MTU	Window	IRTT                                                       
ens5	00000000	0101000A	0003	0	0	512	00000000	0	0	0                                                                             
ens5	0200000A	0101000A	0007	0	0	512	FFFFFFFF	0	0	0                                                                             
ens5	0001000A	00000000	0001	0	0	512	00FFFFFF	0	0	0                                                                             
ens5	0101000A	00000000	0005	0	0	512	FFFFFFFF	0	0	0                                                                             
[2026-05-20T12:43:48Z] INFO: DPDK port config (from echo server log):
[2026-05-20T12:43:48Z] INFO:   (no MTU info in echo log)
[2026-05-20T12:43:48Z] INFO: === END JUMBO DIAGNOSTICS ===
[2026-05-20T12:43:48Z] INFO: Test: jumbo_echo_8000
[2026-05-20T12:43:50Z] INFO: Jumbo output: UDP Test Client
Target: 10.0.1.136:9000
Bind address: 10.0.1.130:0
Payload size: 8000 bytes
Count: 3
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.130:32768 (MAC: 02:48:7d:a4:0f:99)
Backend: dpdk
Sending packets...
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.136:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.136:9000 (expected 8000, OK)
Sent 8000 bytes (binary payload)
Received 8000 bytes from 10.0.1.136:9000 (expected 8000, OK)
Test complete
[2026-05-20T12:43:50Z] INFO: Jumbo frame echo succeeded: 3/3 responses with correct size
[2026-05-20T12:43:50Z] INFO: JUnit XML written to: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-20T12:43:50Z] INFO: Tier 1 sender tests complete. Results: /tmp/test-results/tier1-dpdk-echo.xml
[2026-05-20T12:48:01Z] INFO: Test: arp_resolution
UDP Test Client
Target: 10.0.1.136:9000
Bind address: 0.0.0.0:0
Message: 'arp-probe'
Count: 1
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
DPDK bind failed (Port init failed: Invalid port ID: 0), falling back to tokio
Backend: tokio
Sending packets...
Sent 12 bytes: 'arp-probe #1'
Received 12 bytes from 10.0.1.136:9000: 'arp-probe #1'
Test complete
[2026-05-20T12:48:01Z] INFO: ARP resolution succeeded (got response from peer)
[2026-05-20T12:48:01Z] INFO: Test: udp_send_receive
[2026-05-20T12:48:02Z] INFO: UDP send/receive succeeded
[2026-05-20T12:48:02Z] INFO: Test: echo_roundtrip
[2026-05-20T12:48:03Z] INFO: Echo roundtrip succeeded: 5/5 responses received
[2026-05-20T12:48:03Z] INFO: Test: payload_integrity
[2026-05-20T12:48:03Z] INFO: Response received, checking payload match...
[2026-05-20T12:48:03Z] INFO: Payload integrity verified (found in response)
[2026-05-20T12:48:03Z] INFO: JUnit XML written to: /tmp/test-results/tier2-kernel-interop.xml
[2026-05-20T12:48:03Z] INFO: Tier 2 sender tests complete. Results: /tmp/test-results/tier2-kernel-interop.xml

receiver-test-client-iperf.log

[2026-05-20T12:55:28Z] INFO: iperf-sends: sent 10 packets, received 10 responses
[2026-05-20T12:55:28Z] INFO: iperf-sends: PASS (sent >= 5 packets)
[2026-05-20T12:55:28Z] INFO: JUnit XML written to: /tmp/test-results/tier3-iperf-sends.xml
[2026-05-20T12:55:28Z] INFO: iperf-sends test complete

sender-test-client-iperf.log

[2026-05-20T12:54:26Z] INFO: Pre-flight: checking DPDK state and ARP cache...
[2026-05-20T12:54:26Z] INFO: Local IP: 10.0.1.130, Peer IP: 10.0.1.136, Port: 9000
[2026-05-20T12:54:26Z] INFO: /proc/net/arp contents:
IP address       HW type     Flags       HW address            Mask     Device
10.0.1.136       0x1         0x2         02:8b:0b:9c:38:bf     *        ens5
10.0.1.143       0x1         0x2         02:72:b5:f6:2d:07     *        ens5
10.0.1.130       0x1         0x2         02:48:7d:a4:0f:99     *        ens5
10.0.1.1         0x1         0x2         02:4e:bf:43:5a:c9     *        ens5
[2026-05-20T12:54:26Z] INFO: DPDK runtime state:
No /var/run/dpdk/ directory
[2026-05-20T12:54:26Z] INFO: vfio-pci bindings:
0000:00:06.0
bind
module
new_id
remove_id
uevent
unbind
[2026-05-20T12:54:26Z] INFO: Test binary: /opt/dpdk-stdlib/target/release/test-client
-rwxr-xr-x. 2 root root 1902792 May 20 12:41 /opt/dpdk-stdlib/target/release/test-client
[2026-05-20T12:54:26Z] INFO: Launching test-client: /opt/dpdk-stdlib/target/release/test-client --target 10.0.1.136 --port 9000 --bind-ip 10.0.1.130 --count 10 --delay 200
[2026-05-20T12:54:29Z] INFO: Test client output: UDP Test Client
Target: 10.0.1.136:9000
Bind address: 10.0.1.130:0
Message: 'dpdk-to-kernel-test-payload'
Count: 10
EAL: Detected CPU lcores: 2
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:00:06.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Warning: Some RX offloads not supported by device (flags: 0x1)
Warning: Some TX offloads not supported by device (flags: 0x1)
✅ DPDK UDP socket bound to 10.0.1.130:32768 (MAC: 02:48:7d:a4:0f:99)
Backend: dpdk
Sending packets...
Sent 30 bytes: 'dpdk-to-kernel-test-payload #1'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #1'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #2'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #2'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #3'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #3'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #4'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #4'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #5'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #5'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #6'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #6'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #7'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #7'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #8'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #8'
Sent 30 bytes: 'dpdk-to-kernel-test-payload #9'
Received 30 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #9'
Sent 31 bytes: 'dpdk-to-kernel-test-payload #10'
Received 31 bytes from 10.0.1.136:9000: 'dpdk-to-kernel-test-payload #10'
Test complete
[2026-05-20T12:54:29Z] INFO: our-app-sends: sent 10 packets, received 10 responses
[2026-05-20T12:54:29Z] INFO: our-app-sends: PASS (sent >= 5 packets)
[2026-05-20T12:54:29Z] INFO: JUnit XML written to: /tmp/test-results/tier3-our-app-sends.xml
[2026-05-20T12:54:29Z] INFO: our-app-sends test complete
⚠️ SSM Command Failures (receiver-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-04761bd8476a12885 (receiver)
Command ID: 52539efc-e4cf-4ee1-aa2a-d5f374f9ec0f

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-04761bd8476a12885 (receiver)
Command ID: 0f19617e-7f0d-4d9f-8af1-8481d0fa6e9a

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-04761bd8476a12885 (receiver)
Command ID: d1a5b6b7-7cd5-42ad-b619-28eb1789f006

=== STDOUT ===


=== STDERR ===


=== Polling timeout after 30s ===
Status: InProgress
Instance: i-04761bd8476a12885 (receiver)
Command ID: 27373aae-caa3-415a-afa3-dbc66c04b358

=== STDOUT ===


=== STDERR ===


⚠️ SSM Command Failures (sender-ssm-failure.log)
=== Polling timeout after 30s ===
Status: InProgress
Instance: i-0d826e4b5a8d7ae79 (sender)
Command ID: 9d0b6e10-904c-48a0-af1d-75907408fff2

=== STDOUT ===


=== STDERR ===


Network & PCI State

receiver-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:2d:d0:7c:cc:c3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0c5071120a5e56cbd
    altname device-number-0.0
    inet 10.0.1.77/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2305sec preferred_lft 2305sec
    inet6 fe80::2d:d0ff:fe7c:ccc3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

sender-network-interfaces.log

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:20:29:1f:7a:89 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ebcbcb9a1807bb33
    altname device-number-0.0
    inet 10.0.1.211/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2326sec preferred_lft 2326sec
    inet6 fe80::20:29ff:fe1f:7a89/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

receiver-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-20T12:43:04Z
hostname: ip-10-0-1-77.ec2.internal
kernel: 6.18.25-57.109.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:2d:d0:7c:cc:c3 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0c5071120a5e56cbd
    altname device-number-0.0
    inet 10.0.1.77/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3349sec preferred_lft 3349sec
    inet6 fe80::2d:d0ff:fe7c:ccc3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.143 dev ens5 lladdr 02:72:b5:f6:2d:07 REACHABLE 
10.0.1.1 dev ens5 lladdr 02:4e:bf:43:5a:c9 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.77 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.77 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.77 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.77 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:2d:d0:7c:cc:c3/ 02:8b:0b:9c:38:bf/ 

--- ENI: 02:2d:d0:7c:cc:c3/ ---
  device-number: 0
  local-ipv4s: 10.0.1.77
  subnet-id: subnet-0064262d5913eafeb
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:8b:0b:9c:38:bf/ ---
  device-number: 1
  local-ipv4s: 10.0.1.136
  subnet-id: subnet-0064262d5913eafeb
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:4e:bf:43:5a:c9 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.77 ens5
Unicast reply from 10.0.1.1 [02:4E:BF:43:5A:C9]  0.537ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 20 12:42 .
drwxr-xr-x. 14 root root     3100 May 20 12:42 ..
crw-------.  1 root root 243,   0 May 20 12:42 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 20 12:38 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.054222] printk: legacy console [ttyS0] enabled
[    0.055318] x2apic enabled
[    0.060022] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.060135] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.069715] audit: type=2000 audit(1779280725.462:1): state=initialized audit_enabled=0 res=1
[    0.079417] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.129352] ACPI: Interpreter enabled
[    0.129505] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.148786] pci 0000:00:05.0: enabling Extended Tags
[    0.234178] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.249994] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.253514] IPI shorthand broadcast: enabled
[    5.515127] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    5.548819] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    5.667634] VFIO - User Level meta-driver version: 0.3
[    6.110089] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    6.123466] ena 0000:00:05.0: ENA device version: 0.10
[    6.124221] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    6.225098] ena 0000:00:05.0: ENA Large LLQ is disabled
[    6.237725] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:2d:d0:7c:cc:c3
[    6.275483] ena 0000:00:05.0 ens5: renamed from eth0
[  210.389798] pci 0000:00:06.0: enabling Extended Tags
[  210.393920] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  210.404167] ena 0000:00:06.0: ENA device version: 0.10
[  210.404954] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  210.504135] ena 0000:00:06.0: ENA Large LLQ is disabled
[  210.515970] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:8b:0b:9c:38:bf
[  210.523524] ena 0000:00:06.0 ens6: renamed from eth0
[  246.088080] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  246.089558] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===

sender-networking-diag-baseline.txt

=== NETWORKING DIAGNOSTICS ===
timestamp: 2026-05-20T12:42:57Z
hostname: ip-10-0-1-211.ec2.internal
kernel: 6.18.25-57.109.amzn2023.x86_64

=== DPDK PORT STATUS ===

Network devices using DPDK-compatible driver
============================================
0000:00:06.0 'Elastic Network Adapter (ENA) ec20' drv=vfio-pci unused=ena

Network devices using kernel driver
===================================
0000:00:05.0 'Elastic Network Adapter (ENA) ec20' if=ens5 drv=ena unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

=== IP ADDRESSES ===
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:20:29:1f:7a:89 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0ebcbcb9a1807bb33
    altname device-number-0.0
    inet 10.0.1.211/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 3356sec preferred_lft 3356sec
    inet6 fe80::20:29ff:fe1f:7a89/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

=== ARP TABLE ===
10.0.1.147 dev ens5 lladdr 02:6a:7f:7a:17:15 REACHABLE 
10.0.1.1 dev ens5 lladdr 02:4e:bf:43:5a:c9 REACHABLE 

=== ROUTE TABLE ===
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.211 metric 512 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.211 metric 512 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.211 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.211 metric 512 

=== IMDS: ENI INFORMATION ===
ENI MACs found: 02:20:29:1f:7a:89/ 02:48:7d:a4:0f:99/ 

--- ENI: 02:20:29:1f:7a:89/ ---
  device-number: 0
  local-ipv4s: 10.0.1.211
  subnet-id: subnet-0064262d5913eafeb
  subnet-cidr: 10.0.1.0/24

--- ENI: 02:48:7d:a4:0f:99/ ---
  device-number: 1
  local-ipv4s: 10.0.1.130
  subnet-id: subnet-0064262d5913eafeb
  subnet-cidr: 10.0.1.0/24


=== GATEWAY ARP TEST ===
Gateway IP: 10.0.1.1
Gateway ARP entry:
10.0.1.1 dev ens5 lladdr 02:4e:bf:43:5a:c9 REACHABLE 

arping result:
ARPING 10.0.1.1 from 10.0.1.211 ens5
Unicast reply from 10.0.1.1 [02:4E:BF:43:5A:C9]  0.534ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

=== HUGEPAGE STATUS ===
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:     18432 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

=== VFIO STATUS ===
total 0
drwxr-xr-x.  2 root root       80 May 20 12:42 .
drwxr-xr-x. 14 root root     3100 May 20 12:42 ..
crw-------.  1 root root 243,   0 May 20 12:42 noiommu-0
crw-rw-rw-.  1 root root  10, 196 May 20 12:38 vfio

noiommu mode:
Y

=== DPDK SHARED MEMORY ===
no /var/run/dpdk/ directory (clean state)

=== DPDK-RELATED DMESG (last 30 lines) ===
[    0.054719] printk: legacy console [ttyS0] enabled
[    0.055863] x2apic enabled
[    0.060574] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    0.060688] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.070256] audit: type=2000 audit(1779280725.674:1): state=initialized audit_enabled=0 res=1
[    0.080105] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.129884] ACPI: Interpreter enabled
[    0.130032] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.149765] pci 0000:00:05.0: enabling Extended Tags
[    0.224412] SGI XFS with ACLs, security attributes, quota, no debug enabled
[    0.229956] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    0.233389] IPI shorthand broadcast: enabled
[    5.403496] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    5.438971] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    5.522650] VFIO - User Level meta-driver version: 0.3
[    6.043545] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.16.1g
[    6.054867] ena 0000:00:05.0: ENA device version: 0.10
[    6.055669] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    6.155134] ena 0000:00:05.0: ENA Large LLQ is disabled
[    6.167456] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0500000, mac addr 02:20:29:1f:7a:89
[    6.209524] ena 0000:00:05.0 ens5: renamed from eth0
[  210.355211] pci 0000:00:06.0: enabling Extended Tags
[  210.359300] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  210.373479] ena 0000:00:06.0: ENA device version: 0.10
[  210.374223] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  210.475366] ena 0000:00:06.0: ENA Large LLQ is disabled
[  210.488310] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:48:7d:a4:0f:99
[  210.496073] ena 0000:00:06.0 ens6: renamed from eth0
[  239.252318] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  239.253788] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device

=== DPDK-RELATED PROCESSES ===
no DPDK processes running

=== END DIAGNOSTICS ===
⚠️ Crash Diagnostics

receiver-dmesg-crashes.log

[    0.069081] pid_max: default: 32768 minimum: 301
[    0.152724] iommu: Default domain type: Translated
[    0.152750] NetLabel:  unlabeled traffic allowed by default
[    0.173303] PCI: CLS 0 bytes, default 64
[    0.257378] nvme nvme0: 2/0/0 default/read/poll queues
[    0.391593] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.391718] systemd[1]: No hostname configured, using default hostname.
[    0.460717] systemd[1]: Queued start job for default target initrd.target.
[    3.721182] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  282.637080] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4353)
[  540.605968] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4842)
[ 1168.213232] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6056)

sender-dmesg-crashes.log

[    0.069615] pid_max: default: 32768 minimum: 301
[    0.153718] iommu: Default domain type: Translated
[    0.153745] NetLabel:  unlabeled traffic allowed by default
[    0.173850] PCI: CLS 0 bytes, default 64
[    0.258717] nvme nvme0: 2/0/0 default/read/poll queues
[    0.435794] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    0.435919] systemd[1]: No hostname configured, using default hostname.
[    0.491243] systemd[1]: Queued start job for default target initrd.target.
[    3.587875] systemd[1]: systemd 252.23-11.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  985.117406] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:5692)
Kernel Console (dmesg)

receiver-console-output.log (PCI/driver events only)

[  282.635468] vfio-pci 0000:00:06.0: reset done
[  282.637080] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4353)
[  282.638592] vfio-pci 0000:00:06.0: resetting
[  282.855332] vfio-pci 0000:00:06.0: reset done
[  361.753733] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  362.763507] ena 0000:00:06.0: ENA device version: 0.10
[  362.764287] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  362.863479] ena 0000:00:06.0: ENA Large LLQ is disabled
[  362.875396] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:8b:0b:9c:38:bf
[  362.888386] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[  382.536968] vfio-pci 0000:00:06.0: Adding to iommu group 0
[  382.538401] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[  540.384051] vfio-pci 0000:00:06.0: resetting
[  540.604345] vfio-pci 0000:00:06.0: reset done
[  540.605968] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:4842)
[  540.607477] vfio-pci 0000:00:06.0: resetting
[  540.824209] vfio-pci 0000:00:06.0: reset done
[  614.197176] vfio-pci 0000:00:06.0: Removing from iommu group 0
[  615.210073] ena 0000:00:06.0: ENA device version: 0.10
[  615.210875] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[  615.312110] ena 0000:00:06.0: ENA Large LLQ is disabled
[  615.324290] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:8b:0b:9c:38:bf
[  615.336034] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1083.657186] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1083.658695] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1167.988975] vfio-pci 0000:00:06.0: resetting
[ 1168.211608] vfio-pci 0000:00:06.0: reset done
[ 1168.213232] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (echo:6056)
[ 1168.214720] vfio-pci 0000:00:06.0: resetting
[ 1168.431513] vfio-pci 0000:00:06.0: reset done

sender-console-output.log (PCI/driver events only)

[  985.118906] vfio-pci 0000:00:06.0: resetting
[  985.335683] vfio-pci 0000:00:06.0: reset done
[ 1046.512098] vfio-pci 0000:00:06.0: Removing from iommu group 0
[ 1047.523865] ena 0000:00:06.0: ENA device version: 0.10
[ 1047.524614] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[ 1047.623833] ena 0000:00:06.0: ENA Large LLQ is disabled
[ 1047.635979] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0508000, mac addr 02:48:7d:a4:0f:99
[ 1047.647853] ena 0000:00:06.0 ens6: renamed from eth0 (while UP)
[ 1076.817550] vfio-pci 0000:00:06.0: Adding to iommu group 0
[ 1076.819017] vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
[ 1183.675925] vfio-pci 0000:00:06.0: resetting
[ 1183.905034] vfio-pci 0000:00:06.0: reset done
[ 1183.906633] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6134)
[ 1183.908239] vfio-pci 0000:00:06.0: resetting
[ 1184.124882] vfio-pci 0000:00:06.0: reset done
[ 1184.412917] vfio-pci 0000:00:06.0: resetting
[ 1184.644906] vfio-pci 0000:00:06.0: reset done
[ 1184.646492] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6154)
[ 1184.648124] vfio-pci 0000:00:06.0: resetting
[ 1184.864874] vfio-pci 0000:00:06.0: reset done
[ 1186.154982] vfio-pci 0000:00:06.0: resetting
[ 1186.374914] vfio-pci 0000:00:06.0: reset done
[ 1186.376512] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6177)
[ 1186.378146] vfio-pci 0000:00:06.0: resetting
[ 1186.594875] vfio-pci 0000:00:06.0: reset done
[ 1187.874842] vfio-pci 0000:00:06.0: resetting
[ 1188.094910] vfio-pci 0000:00:06.0: reset done
[ 1188.096487] vfio-pci 0000:00:06.0: vfio-noiommu device opened by user (tokio-rt-worker:6200)
[ 1188.098153] vfio-pci 0000:00:06.0: resetting
[ 1188.314841] vfio-pci 0000:00:06.0: reset done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant