[RFC] support scale down without search only replica

[guojialiang92]

Is your feature request related to a problem? Please describe

We have already supported the scale down of primary and replica under the read-write separation architecture. When performing the scale down operation, it is necessary to ensure the existence of search-only replicas.

However, in production practice (e.g., in the multi-tenant scenario of AI agents), there are indexes with very low write throughput. To save resources, only primaries and replicas are included, and no search-only replicas are configured. In clusters that support remote store, we should support scaling down without search-only shard.

Currently, when determining the status of an index that is in the scale down state, the following rules will be applied.

• When activeShards is 0, the status is red.
• When activeShards > 0
  • activeShards < totalShards, status is yellow
  • activeShards == totalShards, status is green

Under the current judgment rules, scaling down when the search-only replica is 0 will cause the cluster to turn red, leading to unnecessary concerns.

Describe the solution you'd like

Remove the restriction that a search-only replica must exist when scaling down.

I think the rules for index status can be simplified. In the scaled down state, only two states, yellow and green, should be retained. The new judgment rule is:

• activeShards < totalShards, status is yellow
• activeShards == totalShards, status is green

Doing so brings the following benefits:

• The red state indicates that there are unassigned primary shards, making the state more explicit.
• When search only shard is 0, perform scale down, and the index status remains green to avoid affecting the monitoring alerts of the cluster.

Related component

Storage

Describe alternatives you've considered

No response

Additional context

No response

[guojialiang92]

Hi @vinaykpud @prudhvigodithi @mch2
You are the main developers of read-write separation. Could you please take a look and provide suggestions?

[vinaykpud]

Thanks for the RFC and PR. I've gone through the code changes and wanted to confirm my understanding of the safety guarantees here.

question:

1. When scaled to zero without search replicas, is it guaranteed that totalShards (i.e., shardRoutingTable.size()) will always be 0? If for any reason a failure in scale-to-zero can cause a shard entry to remain in the routing table, it would report health as YELLOW instead of RED.

[guojialiang92]

@vinaykpud Thank you for your reply. Regarding the status judgment, I will explain some details.

We will set INDEX_BLOCKS_SEARCH_ONLY_SETTING to true during the finalize phase of scale down. Only when INDEX_BLOCKS_SEARCH_ONLY_SETTING is true will we make a judgment based on activeShards and totalShards and return a yellow or green status.

If any exception occurs during the scale down process, INDEX_BLOCKS_SEARCH_ONLY_SETTING will remain false, and if the primary shard cannot be found or the primary shard is not allocated, a red status will still be returned.

[guojialiang92]

Hi, @vinaykpud
I've added tests in the PR and ensured test coverage. Could you help review the code?