We may need to build a Redis command compatibility test for Kvrocks

[Tangruilin]

Search before asking

• I had searched in the issues and found no similar issues.

Motivation

From a design perspective, Kvrocks should maintain full backward compatibility with Redis. For commands that exist in Redis, Kvrocks needs to have consistent external behavior
However, during testing, I discovered that the TDIGEST.MERGE command has a compatibility issue, refer to: #3410

Even the smartest and most technically capable people make mistakes.
Therefore, we should integrate compatibility testing into CI to verify the behavior of Redis and Kvrocks for the same Redis commands, and continuously enrich the test suite.

Solution

Establish compatibility testing in CI and continuously enrich the test suite.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Tangruilin]

@git-hulk Please take a look at this issue. Could we consider establishing a compatibility test in Kvrocks to avoid incompatibility issues with Redis?

At the same time, for requirements to be compatible with Redis commands, this could significantly reduce the review burden.

[git-hulk]

@Tangruilin Thanks for your report.

Could we consider establishing a compatibility test in Kvrocks to avoid incompatibility issues with Redis?

Could you please elaborate a bit about the compatibility test plan? we now have Go tests will cover this scenario, but not totally for testing compatibility.

[Tangruilin]

@Tangruilin Thanks for your report.

Could we consider establishing a compatibility test in Kvrocks to avoid incompatibility issues with Redis?

Could you please elaborate a bit about the compatibility test plan? we now have Go tests will cover this scenario, but not

@Tangruilin Thanks for your report.

Could we consider establishing a compatibility test in Kvrocks to avoid incompatibility issues with Redis?

Could you please elaborate a bit about the compatibility test plan? we now have Go tests will cover this scenario, but not totally for testing compatibility.

I can start by explaining the difference between the two approaches.

The current Go E2E tests are post-hoc verification, relying on developers to manually predict the expected output of commands and set up corresponding test assertions. Of course, some
developers carefully check the Redis documentation and compare each command one by one, but even so, omissions are inevitable.

The compatibility testing approach, on the other hand, only requires us to provide Redis commands as the test set. The test automatically compares the outputs from Redis and kvrocks, and
if any inconsistency is found, the test fails.

The advantages of the compatibility testing approach are:

1. We don't need to think about command outputs—we only need kvrocks and Redis to produce consistent results, which reduces the cognitive burden of writing tests. (Of course, we should
   still keep the Go E2E tests as they serve as kvrocks's functional baseline tests.)
2. We can easily use AI to generate a large number of boundary test cases to test kvrocks and Redis compatibility in edge scenarios. (This is difficult to achieve with the current Go E2E
   tests, because AI might design outputs that make the Go E2E tests pass, but AI cannot determine what Redis actually outputs.)

If you think this is feasible and valuable, I can write a proposal for us to discuss the implementation. Perhaps we can create a dedicated directory for compatibility testing within the
current Go test framework, sending the same commands to both Redis and kvrocks and comparing their results. We would only need special handling for format differences between kvrocks and
Redis responses, as well as scenarios involving unordered structures like HASH.

[Tangruilin]

The current Go tests primarily help us ensure that existing logic is not broken, while compatibility testing aims to guarantee that a feature's external behavior is consistent with Redis from the moment the code is merged into kvrocks.