send_bundle reorders

[erenon]

If I understand the man correctly, if I populate a single bufring, and submit multiple send bundle operations, buffers should go out in-order (the order they were pushed to the buf ring). However, it doesn't seem to be the case?

Repro: https://github.com/erenon/sendbundle_reorder

Is my code broken, or my understanding of how send bundle works incorrect?

[axboe]

I'll take a look, but it looks like you're frantically dumping a ton of sends in there. That's not really the best way to do it. Or are you just doing that to show the issue?

Can you explain the logic of the reproducer?

[erenon]

The use-case: there's an application, that wants to send on a TCP connection. The application doesn't want to wait for the send completion after a send (if sqpoll is on, it is not enough to just submit), if there's something new to send, it wants to send right away. Simple sends do not guarantee ordering, so I'm using send bundle here.

During stress testing, I found this strange behaviour, and I do not understand what is at fault (my code, my understanding, io_uring, something else). The reproducer is intentionally bursty, to reliably trigger the issue. The reproducer logic is the following:

• start a TCP server
• connect to it
• register the client fd with the ring
• register a buf ring big enough for NUM_SENDS
• pre-allocate NUM_SENDS buffers
• in a loop, submit a send bundle op, and reap completions, if any
• in the server, check if messages are received in order. panic if not.

let me know if something else is needed.

[axboe]

I figured this was a crazy version of what you want to do, just so to show the issue more easily. All good, I'll take a closer look. Got some travel going on this week, so may take a bit longer than usual..

[axboe]

Two fixes here:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=io_uring-6.19

but even with that, I don't believe it can work reliably like that - if you get a short send on one of the sends, then the receiving side will still see reordering as the gap between bufferN - some_bytes and bufferN+1 will not have been sent in order.

[axboe]

OK took a closer look - the way you do it can't ever work reliably, as you have multiple sends active on the socket. It's quite possible that one send will be partial because the socket buffer fills up, and then you have another send pick the next buffer in the range. You cannot rely on that kind of ordering. The test case should work with bundles as long as you only keep a single send active at the time.

I did make a tweak on the kernel side, as I noticed that the bundle sends will keep retrying even past a full socket buffer. If we run into a full socket buffer, we should continue the current bundle, but not start any new ones.

[erenon]

So is my interpretation of the man incorrect?

By using provided buffers with send requests, the application can prevent any kind of reordering of the outgoing data which can otherwise occur if the application has more than one send request inflight for a single socket. This provides better pipelining of data, where previously the app needed to manually serialize sends.

https://man7.org/linux/man-pages/man3/io_uring_prep_send.3.html

What can an application do, to prevent reordering, with send bundles, other than serializing sends (my understanding: that means not having multiple sends active on a socket at the same time)?