Wait for all timeseries deletions to complete

[bnaecker]

This is an attempt to address #10052, a flaky test deleting a timeseries by name. It's hard to generate a repro for that failure, since it relies on racing the deletion across all the ClickHouse nodes. But this should probably address it by ensuring we completely wait for the deletion to occur on all replicas before returning.

This does make the query slower but (1) we don't run this today and (2) we definitely don't run multinode ClickHouse, which is the only place this would matter anyway.

[bnaecker]

Ping on this quick one @jmcarp :)

[jmcarp]

This makes sense to me. I have some questions, but I don't see any harm in merging.

• Is this a regression, or has the test been flaky since we added it?
• Can we add a comment explaining the query settings? I'm guessing the goal is to ensure that the delete is synchronous so that subsequent test assertions pass, but might as well write it down.
• Nit, and I don't even know that I like this idea, but if we really didn't want to make the delete feel slow, we could leave the query as-is, and have the test poll clickhouse's system.mutations until there aren't any pending mutations. But it's not clear that we care if this function is slow, and I'm guessing it doesn't run often.

All of which is to say, LGTM.

[bnaecker]

* Is this a regression, or has the test been flaky since we added it?

I think this is a long-standing issue, but it seems to be worse after #10012.

* Can we add a comment explaining the query settings? I'm guessing the goal is to ensure that the delete is synchronous so that subsequent test assertions pass, but might as well write it down.

Yes, that's right. I'll add a note.

* Nit, and I don't even know that I like this idea, but if we really didn't want to make the delete feel slow, we could leave the query as-is, and have the test poll clickhouse's system.mutations until there aren't any pending mutations. But it's not clear that we care if this function is slow, and I'm guessing it doesn't run often.

That's true, but we don't care for a few reasons. We don't run this ever outside of tests, because no customers are running multinode ClickHouse yet. That worked stalled out. Second, it's during schema migrations, which we don't currently do in either single- or multi-node deployments.