failure to reconnect with -persist

[mxork]

I still need to do a proper job getting logs together, but maybe someone can tell me I'm being dumb before I put too much time into this.

Currently, filesystems I have mounted from nbd devices panic on I/O failure if I restart the corresponding server. I would like nbd to renegotiate the connection if the server drops, and -persist seems to do the right thing . However, if I set up a test environment on the local machine and restart the server after connecting:

$ nbd-client -N default localhost /dev/nbd0 -persist -nonetlink -nofork             
Negotiation: ..size = 1024MB
bs=1024, sz=1073741824 bytes
timeout=5
<restart nbd-server here>
Kernel call returned.
sock, done

The call simply returns, and does not attempt to reconnect. From the log message, it does not take the branch at

nbd/nbd-client.c

Line 1292 in 128fd55

if (ioctl(nbd, NBD_DO_IT) < 0) {

, instead seeming to take the branch at

nbd/nbd-client.c

Line 1329 in 128fd55

} else {

. I have no idea why the ioctl call would return >= 0, but it seems to.

I realize that the filesystem may also need some love to get the desired behavior, but that's moot if nbd does not renegotiate.

[nand11]

Hi,

TL;DR: Here is my experience with the -persist option of nbd-client, can you help me?

I want to use NBD over an unreliable network connection. So I was trying to use nbd-client with the -persist option. A while ago somone on irc (#nbd on oftc.net) told me that -persist is broken on recent kernels and I should try to use -nonetlink (Thanks!). So I did that (using debian buster). I also started nbd-client with -nofork to get debug outputs.

Test 1:

I connected nbd, interrupted the network connection and nbd-client exited with

Kernel call returned.
sock, done

Okay, that's not what I wanted so I had a look at the sourcecode of nbd-client (debian source package nbd-3.19). Near the end of main(), there is this section that gets executed:

  if (ioctl(nbd, NBD_DO_IT) < 0) {
           [...]
   } else {
           /* We're on 2.4. It's not clearly defined what exactly
            * happened at this point. Probably best to quit, now
            */
           fprintf(stderr, "Kernel call returned.\n");
           cont=0;
   }

Okay, we are not on Kernel 2.4, my kernel version is 4.19.

Test 2:

I found out that ioctl returned 0 in case of a network disconnect. So I changed

  if (ioctl(nbd, NBD_DO_IT) < 0) {

to

  if (ioctl(nbd, NBD_DO_IT) <= 0) {

and things got better. After a network disconnect, nbd-client tried to reconnect:

nbd,16756: Kernel call returned: Success
Reconnecting
Error: Socket failed: Connection refused
Reconnecting
Error: Socket failed: Connection refused
Reconnecting
Error: Socket failed: Connection refused
Reconnecting

A connection retry was done about once per second. Yay! I restored my network connection and it worked:

Negotiation: ..size = 2844688MB
bs=512, sz=2982871564288 bytes
timeout=120
nbd,31830: Kernel call returned: Success
Reconnecting
Error: Socket failed: Connection refused
Reconnecting
Error: Socket failed: Connection refused
Reconnecting
Negotiation: ..size = 2844688MB
bs=512, sz=2982871564288 bytes
timeout=120

Test 3:

The test above was done without a mounted filesystem. Now I mounted a filesystem (readonly, via LUKS crypto) and tried it again. Unfortunately, the result was a little bit different. The kernel said it was busy and retries were attempted quickly after another. nbd-client printed:

Negotiation: ..size = 2844688MB
bs=512, sz=2982871564288 bytes
timeout=120
nbd,16756: Kernel call returned: Device or resource busy
Reconnecting
Negotiation: ..size = 2844688MB
bs=512, sz=2982871564288 bytes
timeout=120
nbd,16756: Kernel call returned: Device or resource busy
Reconnecting

The reconnect in nbd-client does not print an error, but the next iteration of

ioctl(nbd, NBD_DO_IT)

returns "Device or resource busy".

Now I am at a point where I would need to debug or understand the kernel nbd driver, what I have not attempted yet. (__nbd_ioctl calls nbd_start_device_ioctl for NBD_DO_IT, this calls nbd_start_device, and this returns -EBUSY if nbd->task_recv is set.
https://github.com/torvalds/linux/blob/e595dd94515ed6bc5ba38fce0f9598db8c0ee9a9/drivers/block/nbd.c#L1232
Is this where it fails?)

best regards,
nand11

[axos88]

This is still an issue with version 3.18

[0rtz]

I see the same behaviour as nand11 with version 3.21

[fff7d1bc]

I can confirm that there's no attempt to reconnect on 3.21.

[yoe]

So.

-persist uses an old quirk in the ioctl configuration interface that used to work at some point, but seems to have been lost after a few maintainer changes for the kernel module of nbd.

The netlink interface does not have anything to support -persist; when using the netlink interface, nbd-client has exited long before the connection is dropped, and so there is no way for it to discover that this has happened.

In order for -persist to work again, I think we need to go back to the drawing board. Meanwhile a possible workaround could be to use the multiple connection feature; if the connection drops, and you still have another connection open, then that allows you to continue working (but obviously that doesn't work if you only have one server and that one is being restarted).

For now, I think I'll just disable -persist (better to not have a feature rather than one that does nothing), and talk to the kernel maintainer to see how we can fix this.

[fff7d1bc]

In that case, would you also consider enabling timeout by default? From what I see, even if I don't use -persist, like

nbd-client  -nofork  -N test1 HOSTNAME /dev/nbd0

And the server reboots, I get a block device and nbd-client forever in D state, until I reboot. Seems like timeout is essential here.

[yoe]

It's actually in the D state until the TCP timeout, which is a per-system setting that defaults to 2 hours and 12 minutes (IIRC).

The timeout thing is another of those bad ideas that I should probably get rid of; it triggers if the device is perfectly happily connected but idle. This may be a good idea in some cases, but not in most.

[chabad360]

Well that would explain why my kiosks randomly fail (well, clearly not as random as I thought) if they sit idle for too long. How would I go about disabling the timeout?

[yoe]

If you don't explicitly pass the -t or -timeout parameter to nbd-client, it shouldn't be set. If you still see things going wrong there, please file a (separate) bug.

[chabad360]

Wait, now I'm confused. The timeout is set by default, or by default it never times out?

Cause at this point, this has become quite a problem for me, but I've never been able (or had enough time and patience) to really track it down. I'm not sure if my kiosks are failing from this timeout (seems to happen only during idle) or if it's a lucky network failure.

I'm thinking of switching to ISCSI, but the CoW feature of NBD is very useful to me.

(Does the timeout in nbd-server also close on idle?)

[yoe]

Neither the client nor the server timeout should be set by default (which means neither should time out by default).

The TCP keepalive probes are set, and it's not possible to switch them off. As long as the remote end is still functioning properly, these shouldn't interrupt your connection, however.

[bauen1]

Hi, is there any chance this is getting fixed in the near future, or a way to work around this issue ?
Otherwise I'll also have to investigate replacing nbd with iSCSI or something else.

I have setup backups on a remote server for my laptop, for this a wireguard VPN is setup between the hosts, and the server runs an nbd-server.
When connecting with -persist, eventually after a day or so, attempting to access the mounted filesystem will result in IO errors due to nbd having dropped the connection, forcing me to unmount everything uncleanly, reconnect and remount everything again.

[yoe]

A simple workaround is to make sure the connection never remains idle for too long. Just touching a file in the mounted NBD file system every once in a while should do that.

[wtarreau]

That's a very sad situation, I was testing NBD as a really appealing candidate for remote backups, but ended up on this non-working persist situation, and the D state as well when timeout is not set. Yes I think we should rework all of this a few ways:

• always set TCP_USER_TIMEOUT on linux >= 2.6.37. This one is very clean, as it only counts failures to ACK sent packets ;
• also enable SO_KEEPALIVE with a short TCP_KEEPIDLE value (e.g. configured timeout divided by 3 or so retries).

But this would only be used to make sure the timeout doesn't kill idle connections and actually only kills dead ones (killing idle connections didn't happen in my tests). The fact that the daemon cannot automatically reconnect by default with -p is a problem that clearly indicates a logic error in the code, but if it fails on EBUSY once the block device is in use, we have a much bigger problem which is that NBD is basically unusable for any real-world purpose since TCP connections eventually fail. I can easily reproduce this here by trying to restart nbd-client after a network error:

19:02:11.161069 ioctl(4, NBD_SET_SOCK, 3) = 0
19:02:11.176188 rt_sigprocmask(SIG_SETMASK, ~[KILL PIPE TERM RTMIN RT_1], ~[KILL PIPE TERM STOP RTMIN RT_1], 8) = 0
19:02:11.176416 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 7755 attached
, child_tidptr=0xffff8e2290f0) = 7755
[pid  7447] 19:02:11.177218 ioctl(4, NBD_DO_IT <unfinished ...>
[pid  7755] 19:02:11.177284 set_robust_list(0xffff8e229100, 24 <unfinished ...>
[pid  7447] 19:02:11.177338 <... ioctl resumed>) = -1 EBUSY (Device or resource busy)
[pid  7755] 19:02:11.177398 <... set_robust_list resumed>) = 0
[pid  7447] 19:02:11.177481 write(2, "nbd,7447: Kernel call returned: Device or resource busy\n", 56nbd,7447: Kernel call returned: Device or resource busy
) = 56
[pid  7447] 19:02:11.177651 close(3 <unfinished ...>
[pid  7755] 19:02:11.177698 openat(AT_FDCWD, "/sys/block/nbd0/pid", O_RDONLY <unfinished ...>
[pid  7447] 19:02:11.177783 <... close resumed>) = 0
[pid  7447] 19:02:11.177869 close(4 <unfinished ...>

Worse, it loops like crazy eating the CPU trying to do that in loops.

I agree that it may become important to get back to the blackboard. It seems to me we're dealing with a bunch of chicken-and-egg problems here. Maybe we're just missing a "reconnect" operation to communicate with the kernel instead of the "connect" one, I don't know.

[nand11]

Oh, this thread is still active? Then it may make sense to share my experience.

I tried iscsi over an unreliable network connection and it worked. It is slow over this kind of network, as expected, but it can handle reconnects. I am using the standard debian packages targetcli-fb (server) and open-iscsi (client).

For iscsi, the client demon iscsid does connection-level error processing. But I have not looked at the source code to see how things are handled differently between nbd and iscsi.