keep-alive packets trigger fevent for raw flows

[th1j5]

A (minimal) working example to trigger the bug is found on this branch

1. start sudo irmd --stdout
2. make local layer

irm ipcp create type local name local_ipcp
irm ipcp bootstrap type local name local_ipcp layer local_layer
irm name register oping_server layer local_layer
irm bind prog oping name oping_server

3. start oping --listen
4. start oping -n oping_server -c 5

observe after the oping client has stopped sending pings (but is still connected and sending keep-alives due to sleep(50))
that on the server side (printf statements added in branch) we get:

Type of event: 1.
message len: -11
Type of event: 1.
message len: -11
Type of event: 1.
message len: -11
Type of event: 32.
message len: -1006

The -1006 is #define EFLOWPEER 1006 /* Flow is down (peer timed out) */
but those three flow_read events before (-11) are EAGAIN 11 Resource temporarily unavailable

using the poor mans debugger (printf), I can say it is due to https://ouroboros.rocks/cgit/ouroboros/tree/src/lib/dev.c#n1345

[th1j5]

This (the message len of -EAGAIN) was the expected behaviour on a non-blocking flow.
The question remains if an 'fqueue' event needs to be emitted for each keep-alive packet

[th1j5]

Thus, keep-alive packets trigger an fevent (with type FLOW_PKT) when they should not.

I'm not sure if they should be filtered out in fqueue_filter (they should not take this if-statement) or elsewhere
I'm not familiar enough with the codebase to say more

[dstaesse]

This is indeed a bug, for non-FRCT flows, the keepalives aren't filtered out due to that if-statement.

@th1j5 can you check that this is fixed in

https://github.com/dstaesse/ouroboros/tree/be-fix-fevent-keepalive

[th1j5]

No, the keepalives are indeed filtered out in fevent,
but now the client cannot dealloc (no idea why, a bit late for me to figure it out tonight)
see https://github.com/th1j5/ouroboros/tree/bug-fevent-keepalive for a MWE (instructions are the same as first comment)

[th1j5]

This is what I get for the client when it's blocked, so it will have to do with a lock, I guess

(gdb) bt
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f6b3a311098) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6b3a311018, cond=0x7f6b3a311070) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f6b3a311070, mutex=0x7f6b3a311018) at pthread_cond_wait.c:647
#3  0x00007f6b39ede691 in shm_rbuff_fini (rb=0x55e405143340) at /mnt/data/UGent/2021-2022/Thesis/tools/ouroboros/src/lib/shm_rbuff_pthr.c:288
#4  0x00007f6b3a10c869 in flow_dealloc (fd=64) at /mnt/data/UGent/2021-2022/Thesis/tools/ouroboros/src/lib/dev.c:997
#5  0x000055e404fa1286 in client_main () at /mnt/data/UGent/2021-2022/Thesis/tools/ouroboros/src/tools/oping/oping_client.c:275
#6  0x000055e404fa2235 in main (argc=0, argv=0x7fffca896390) at /mnt/data/UGent/2021-2022/Thesis/tools/ouroboros/src/tools/oping/oping.c:273

[dstaesse]

No, there's probably unexpected packets left in the buffer (it only exits when all remaining queued packets are cleaned up). I staged packets after the keepalives from event using the mechanism we have for partial reads. That staging area will probably still contain an packet reference on exit. I will look into it tonight.

[dstaesse]

@th1j5, this doesn't seem to reproduce for me...

I compiled your branch, and had no issues with dealloc client side... Also added a 50 second sleep, which I didn't find anymore in the MWE, and client dealloc also still works:

[dstaesse@heteropoda build]$ oping -n oping -c 5
Pinging oping with 64 bytes of data (5 packets):

64 bytes from oping: seq=0 time=0.611 ms
64 bytes from oping: seq=1 time=0.441 ms
64 bytes from oping: seq=2 time=0.448 ms
64 bytes from oping: seq=3 time=0.450 ms
64 bytes from oping: seq=4 time=0.424 ms

--- oping ping statistics ---
5 packets transmitted, 5 received, 0 out-of-order, 0% packet loss, time: 5001.413 ms
rtt min/avg/max/mdev = 0.424/0.475/0.611/0.077 ms
Sleeping 50 seconds.
deallocing client
deallocing client done

Server also deallocated just fine.

Does it always reproduce for you?

[th1j5]

No, it seems it's not reliably reproduced, I just tested it and the different results were:

1. I reproduced the problem
2. It deallocated just fine
3. oping client stopped midway after 3 pings
   (this with commit https://github.com/th1j5/ouroboros/commit/726e6b5a34d5dffeb5e5a728bc5862a16ac84055)

[dstaesse]

Which glibc version are you using?

[dstaesse@heteropoda build]$ ldd --version
ldd (GNU libc) 2.35

Reason I'm asking is that I'm aware of this issue in glibc:
https://sourceware.org/bugzilla/show_bug.cgi?id=25847

The issue only happens with the fevent fix, and not on be?

[th1j5]

$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31

and yes, it only happened on the fevent fix, never encountered it on be