Support commands retrying when global lock is held

[shonfeder]

We fixed the bad hanging behavior reported in #12685 by ensuring that the commands would not end up hanging forever waiting for an RPC server what wasn't going to appear. This indeed prevents hanging, but it still omits some helpful low-hanging fruit: rather than just failing with error, we can make commands wait to obtain the lock.

This waiting behavior should be

• conditionally enabled (probably opt-out, I'd think, but we can solicit feedback to decide this)
• with configurable retries/timeout
• designed so it wouldn't interfere with the likely near term addition of a daemon build server modality

The need for this is currently reported by users making heavy use of AI agents.

[rgrinberg]

I don't remember why or when this waiting was introduced, but i think it is almost always unnecessary and useless? It dune is running in batch mode (not watch mode), it is enough to just kill the currently running dune and re-run the command with the combined build queries.

[shonfeder]

FAIU, waiting was removed in #12685 . For a single user, re-running the command is easy. But apparently the functionality requested here would be a help for working with agent swarms.

[rgrinberg]

Waiting has not been removed. It is still present when trying to $ dune build concurrently. This is something that was recently introduced.

The irony is that the waiting is what ends up messing up the agents. It's the only time I've ran into this issue at all. So erroring out immediately and never waiting is an improvement is mostly there to improve agent iteration loops.

[Alizter]

We shouldn't even be waiting in this way at the moment. Currently all dune build's wait for an RPC server to start as soon as they see a build lock which is wrong.

At some point, we had a retry loop that did work, but it got refactored and broken. I would prefer if we just didn't wait at all and instead implement the daemon feature for the users who would like it.

[shonfeder]

Are you two saying that the claimed fix that closed #12685 did not actually fix or close the issue? We should reopen it in that case.

But AFAIK, it has been claimed the waiting was removed.