Description
In opensearch k-NN, we want merge operations to be executed with a dynamic number of threads. To do this, we are giving a TaskExecutor a dynamic number of threads here: https://github.com/opensearch-project/k-NN/blob/e327675aa7c404ab0cac533a826bca0febb53fdd/src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/KNN9120PerFieldKnnVectorsFormat.java#L94-L106
The problem is that every time a merge operation occurs, we spawn a new executor that is never shut down. We would like a close function to be added as an interface with default noop behavior. This function would be called whenever a merge operation is completed. That way we can shut down our merge thread pool every time it is done executing. Is this feasible?
cc @navneet1v @shatejas
Description
In opensearch k-NN, we want merge operations to be executed with a dynamic number of threads. To do this, we are giving a TaskExecutor a dynamic number of threads here: https://github.com/opensearch-project/k-NN/blob/e327675aa7c404ab0cac533a826bca0febb53fdd/src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/KNN9120PerFieldKnnVectorsFormat.java#L94-L106
The problem is that every time a merge operation occurs, we spawn a new executor that is never shut down. We would like a
closefunction to be added as an interface with default noop behavior. This function would be called whenever a merge operation is completed. That way we can shut down our merge thread pool every time it is done executing. Is this feasible?cc @navneet1v @shatejas