Performance: Pipelined SET+EXPIRE is ~2.3x slower than MSET

[REASY]

There is a significant performance difference when comparing Pipelined SET + EXPIRE operations against Native MSET commands.

Benchmark details

• The total number of elements in the benchmark is 100 000.
• Keys are randomly (Alphanumeric) generated strings of length 80.
• Values are randomly generated bytes of length 20.

According to the End-to-End Benchmarks: Write Throughput results (single-threaded AsyncRedisClientV1 with batch size 10000):

1. MSET achieves an average throughput of ~1.41 M elem/s or ~134.96 MiB/s with average latency 70.664 ms.
2. Pipelined SET + EXPIRE (Manual) achieves an average throughput of ~610.88 K elem/s or ~58.25 MiB/s with significantly higher latency, 163.70 ms.

To Reproduce

Steps to reproduce the behavior:

1. Clone https://github.com/REASY/dragonfly-playground-rs
2. Start the Dragonfly instance using the provided Docker Compose configuration (v1.34.2).
3. Build the benchmark image: dragonfly-playground-rs-dev:0.1.0.
4. Run the write_throughput benchmark:

   docker run -it --network=host \
     dragonfly-playground-rs-dev:0.1.0 \
     -c "REDIS_BENCH_SERVER=127.0.0.1:6379 REDIS_BENCH_MULTI_SET=1 cargo bench --bench write_throughput"

Environment

• OS: Ubuntu 22.04.5 LTS
• Kernel: 5.15.0-153-generic, x86_64
• Containerized?: Docker
• Dragonfly Version: v1.34.2 (image ghcr.io/dragonflydb/dragonfly:v1.34.2)
• Hardware: Bare-metal, 2 x AMD EPYC 7643 48-Core Processor (192 vCPU), 1536 GiB RAM

docker-compose.yml

name: dragonfly-playground-rs
services:
  dragonfly:
    image: ghcr.io/dragonflydb/dragonfly:v1.34.2
    environment:
      DFLY_max_multi_bulk_len: 2000100
      DFLY_pipeline_buffer_limit: 872415232
      DFLY_pipeline_queue_limit: 2000100
      DFLY_maxmemory: "40g"
    ulimits:
      memlock: -1
    ports:
      - "6379:6379"
    deploy:
      resources:
        limits:
          cpus: "32"
          memory: "48g"
    cpuset: "0-32"

Reproducible Code Snippet

The logic producing the lower performance (Pipelined Manual SET + EXAT) is defined in redis_client.rs, the benchmark write_throughput.rs

[romange]

why it is a bug?

[REASY]

why it is a bug?

Ah, misclicked, this isn't a bug, more a question, sorry.