Golden Query Generation
Given an embedding dataset on HuggingFace, we need to be able to evaluate the quality of the ANN results against true KNN. This requires generating representative queries (not random). We can do this in a few ways:
Techniques
- Manually craft representative queries (not scalable)
- Use an LLM to generate queries based on the dataset description (requires scripting)
- Leave out a set of embeddings from the given dataset as a holdout "validation" set (probably shittiest but easiest)
- Could just delete points randomly from collection, but this has limitations
Recall Calculation
Given set of golden queries, execute the brute force queries. This will require GPUs.
Given a framework for calculating brute force queries using GPUs.
Golden Query Generation
Given an embedding dataset on HuggingFace, we need to be able to evaluate the quality of the ANN results against true KNN. This requires generating representative queries (not random). We can do this in a few ways:
Techniques
Recall Calculation
Given set of golden queries, execute the brute force queries. This will require GPUs.
Given a framework for calculating brute force queries using GPUs.