io_uring_register_ifq() returns EEIXST when testing zero-copy RX

[youngmin-kaist]

Problem: io_uring_register_ifq() returns -17, EEIXST

I'm testing io_uring zero-copy RX and sometimes encountering an EEXIST error from io_uring_register_ifq() when I run the program multiple times.

Test environment
Kernel version: 6.19.0-061900rc5-generic
liburing version: 2.11
HW: NVIDIA connectx-7 NIC, fw_version: 28.42.1000

Reason
I looked at the source code and found that it returns EEXIST when rxq has mp_ops

	rxq = __netif_get_rx_queue(dev, rxq_idx);
	if (rxq->mp_params.mp_ops) {
		NL_SET_ERR_MSG(extack, "designated queue already memory provider bound");
		return -EEXIST;
	}

I ran bpftrace to check this is the problem.

'kprobe:__net_mp_open_rxq
/((struct net_device *)arg0)->ifindex == 3/
{
  $dev = (struct net_device *)arg0;
  $qid = arg1;
  $rxq = (struct netdev_rx_queue *)($dev->_rx + $qid);

  @watch[tid] = 1;
  printf("enter ifindex=%d q=%d mp_ops=%p mp_priv=%p\n",
         $dev->ifindex, $qid,
         $rxq->mp_params.mp_ops,
         $rxq->mp_params.mp_priv);
}

kretprobe:__net_mp_open_rxq
/@watch[tid]/
{
  printf("ret=%d\n", retval);
  delete(@watch[tid]);
}'

And found that rxq has mp_ops.

enter ifindex=3 q=6 mp_ops=0xffffffffa56a32e0 mp_priv=0xff294f20e2856cc0
ret=-17

I tried to reload the interface using ifconfig down/up, it sometimes work but not always. How can I solve the problem deterministically?

[axboe]

I this timing dependent? IIRC, the zero copy queues are town down as part of ring async exit. CC @isilence

[isilence]

@youngmin-kaist, what's the use pattern? Is it recreating a ring + zcrx in the same process for the same queue? Is there any delay between io_uring_queue_exit() and io_uring_register_ifq()? If no, does adding a ~1s helps? Do you have a reproducer?

[axboe]

@isilence as background, though I'm sure you've seen that too, Jakub recently added wait delays to work around this problem in CI / selftests too. Hence I assumed this is probably the same issue, the fact that it'll be X time after the ring is closed until the resources are freed.

[isilence]

That's my first thought as well. And the same problem bit users before even w/o zcrx, so maybe it's time to add some variation on "signal an eventfd on io_uring death".

[axboe]

Yep, I do think it'd be nice to have something suitable to quiesce on ring exit.

[youngmin-kaist]

@isilence,
I recreate the ring and ZCRX for the same RX queue every time I launch the executable (i.e., this is not a re-creation within the same process; each run starts a new process).

io_uring_queue_exit() is called at the end of the process. In most cases, re-launching the executable immediately works fine. However, sometimes io_uring_register_ifq() returns -EEXIST.

In some cases, waiting for a while before launching the executable again resolves the issue, but not always. Sometimes the problem disappears after reloading the interface using ifconfig down/up. In other cases, reloading the interface does not help either, and the only workaround is to wait for some time before running the executable again.

[isilence]

Hmm, the delay should've normally fixed it, some maybe something gets blocked, e.g. io_uring exit loop. I have patches returning queue removal back before the exit loop, so maybe it'll help, but let's try to trace it better. I'll cook up a script tomorrow.

[isilence]

@youngmin-kaist, can you paste the full output of the bpftrace below? I extended your script a bit. You might need to comment some probes out if it can't find symbols.

config = { missing_probes=warn; }

kprobe:io_register_zcrx_ifq
{
	$ctx = arg0;
	@cur_ctx[curtask] = $ctx;
	printf("io_register_zcrx_ifq: ctx=%p\n", $ctx);
}

kretprobe:io_register_zcrx_ifq
{
	delete(@cur_ctx[curtask]);
}

kprobe:io_ring_exit_work
{
	$off = offsetof(struct io_ring_ctx, exit_work);
	$ctx = (struct io_ring_ctx *)(arg0 - $off);
	printf("io_ring_exit_work: ctx=%p\n", $ctx);
}

kprobe:io_put_zcrx_ifq
{
	$z = (struct io_zcrx_ifq *)arg0;
	$c = $z->refs.refs.counter;
	printf("io_put_zcrx_ifq: zcrx=%p, qid=%d cnt=%d\n", $z, $z->if_rxq, $c);
}

kprobe:io_zcrx_ifq_free
{
	$z = (struct io_zcrx_ifq *)arg0;
	printf("io_zcrx_ifq_free: zcrx=%p, qid=%d\n", $z, $z->if_rxq);
}

kprobe:io_close_queue
{
	$z = (struct io_zcrx_ifq *)arg0;
	printf("io_close_queue: zcrx=%p, qid=%d\n", $z, $z->if_rxq);
}

kprobe:__net_mp_close_rxq
{
	$dev = (struct net_device *)arg0;
	$qid = arg1;
	$rxq = (struct netdev_rx_queue *)($dev->_rx + $qid);
	$p = (struct pp_memory_provider_params *)arg2;

	printf("__net_mp_close_rxq: ifindex=%d q=%d mp_ops=%p mp_priv=%p tgt_ops=%p tgt_priv=%p\n",
		 $dev->ifindex, $qid,
		 $rxq->mp_params.mp_ops,
		 $rxq->mp_params.mp_priv,
		 $p->mp_ops,
		 $p->mp_priv);
}

kprobe:__net_mp_open_rxq
{
	$dev = (struct net_device *)arg0;
	$qid = arg1;
	$rxq = (struct netdev_rx_queue *)($dev->_rx + $qid);
	$p = (struct pp_memory_provider_params *)arg2;

	@watch[tid] = 1;
	printf("__net_mp_open_rxq: ifindex=%d q=%d mp_ops=%p mp_priv=%p new_ops=%p new_priv=%p ioring_ctx=%p\n",
		 $dev->ifindex, $qid,
		 $rxq->mp_params.mp_ops,
		 $rxq->mp_params.mp_priv,
		 $p->mp_ops,
		 $p->mp_priv,
		 @cur_ctx[curtask]);
}

kretprobe:__net_mp_open_rxq
/@watch[tid]/
{
	printf("ret=%d\n", retval);
	delete(@watch[tid]);
}

[isilence]

By the way, the branch I mention is below. I needed it for other purposes, but if you're able to test custom kernels, it'd be interesting to know if it improves your problem.

https://github.com/isilence/linux/tree/zcrx/test-notif

[youngmin-kaist]

@isilence Here is the bpftrace result. I commented out only the io_ring_exit_work probe.
I ran the program twice, using 16 threads with one RXQ per thread.
The EEXIST errors occurred during the second run.

I'll also try that linux branch. Thank you.

zcrx_trace.txt

[isilence]

@youngmin-kaist, thanks. I don't see anything even trying to close that rx queue and was interested to see when the corresponding io_ring_exit_work() happens. Was there a problem with the io_ring_exit_work() probe?

[youngmin-kaist]

The error is ERROR: 'struct io_ring_ctx' has no field named 'exit_work'
I`m not sure it is a proper way, I can get the bits offset of exit_work

$ bpftool btf dump file /sys/kernel/btf/vmlinux | grep exit_work
        'exit_work' type_id=489 bits_offset=8768

So, I modify the io_uring_exit_work() probe
$ctx = (struct io_ring_ctx *)((uint64)arg0 - 1096);

And this is the result of bpftrace with io_ring_exit_work() probe when the problem happens.

zcrx_trace2.txt

[isilence]

Thanks, so we're destroying the io_uring instance, but it's not executed in time as per above. Likely, you're closing an io_uring with a bunch of inflight requests, and it needs to process/cancel first before killing the rx queue.

Moving the close a bit earlier as in the branch should help. I'll mark the patches for stable. If you give your full name + email, I can add you to the commit message in a form of a Reported-by tag to give some credit.

[youngmin-kaist]

Actually, there are no inflight requests when the program exits. When the process starts, each thread issues 100 connect() requests to the server and waits for all of them to complete. After all threads receive completions for all 100 connect() calls, the program exits. So, there must be no requests to process.

Anyway, I hope the problem is fixed! My name is Youngmin Choi (youngminchoi94@gmail.com)

Also, I have another question about io_uring zcrx with multi-threading/rxq. I implemented L7 proxy using zcrx and zctx, which forwards zcrx'ed data using zctx. When using fewer than 8 cores (1 rxq per each core), all CPU cores are 100% utilized given enough traffic. However, when using more than 8 cores, some cores are not fully utilized, showing 50%~90%.
I am not sure why the CPU stops being the bottleneck when using more than 8 cores. I checked other possible bottlenecks such as client/server CPU usage, link bandwidth, PCIe bandwidth etc and all of them seem fine. I also verified the flow steering rule and make sure that traffic are evenly distributed to across cores (RXQs). (FYI, I tested with 3 machines, each equipped with 1 connectx-7 400G NIC. client generates HTTP traffic using 'h2load' and server runs Nginx.)

Do you have any idea what could cause this behavior, or what I should check next?

[isilence]

Actually, there are no inflight requests when the program exits. When the process starts, each thread issues 100 connect() requests to the server and waits for all of them to complete. After all threads receive completions for all 100 connect() calls, the program exits. So, there must be no requests to process.

Ok, that's interesting. Should get fixed either way, but might be useful to know what blocks it. It might give some clue if you add timestamps to the script, so we know what order of delays before a close we're talking about.

Also, I have another question about io_uring zcrx with multi-threading/rxq. I implemented L7 proxy using zcrx and zctx, which forwards zcrx'ed data using zctx. When using fewer than 8 cores (1 rxq per each core), all CPU cores are 100% utilized given enough traffic. However, when using more than 8 cores, some cores are not fully utilized, showing 50%~90%. I am not sure why the CPU stops being the bottleneck when using more than 8 cores. I checked other possible bottlenecks such as client/server CPU usage, link bandwidth, PCIe bandwidth etc and all of them seem fine. I also verified the flow steering rule and make sure that traffic are evenly distributed to across cores (RXQs). (FYI, I tested with 3 machines, each equipped with 1 connectx-7 400G NIC. client generates HTTP traffic using 'h2load' and server runs Nginx.)

Do you have any idea what could cause this behavior, or what I should check next?

It's hard to say. Try to increase packet sizes and see if that helps. You can try to use some iperf / etc. and see if you can push more traffic than your test program. Check if the tx side is the bottleneck here, maybe increase the socket tx queue size. If you're keeping a limited number of inflight reqs, double check you're not getting stuck waiting on send zc notifications. You can try to increase the number of connections and see if that changes anything.