dragonfly_commands_duration_seconds is declared as histogram but exposes no buckets

[kcj-3054]

There are no accompanying:

• _bucket
• _sum
• _count

series exposed for this metric.

Questions

1. Is dragonfly_commands_duration_seconds intended to be:

   • a true Prometheus histogram (with buckets planned), or
   • a cumulative counter representing total time spent per command?

2. If it is intentionally cumulative:

   • should this metric be declared as counter instead of histogram?
   • is the recommended usage to apply rate() / increase() for "time spent per interval"?

3. Are there any plans to expose proper histogram buckets (or latency percentiles)
   similar to what redis-exporter provides?

Context

Without buckets or _sum/_count, it’s not possible to compute:

• average latency
• p95 / p99 latency using histogram_quantile()

At the moment, the only meaningful interpretation seems to be
“cumulative command execution time in seconds”, which behaves like a counter.

Clarification would help users build correct Grafana dashboards and avoid misinterpreting
this metric.

Thanks!

[romange]

@abhijat can you answer please

[abhijat]

@kcj-3054 we are changing this metric to a counter. That change should be merged soon.

Do you need this metric to be a histogram though, or does consuming it as the correct type (counter) work for your use case?

[kcj-3054]

Title: Inquiry regarding dragonfly_commands_duration_seconds metric definition and documentation

Hi @abhijat

I'm currently working on monitoring Dragonfly and had a few questions regarding the metrics, as I couldn't find detailed official documentation for them.

Although the code comments refer to it as a histogram, when I checked the actual values of dragonfly_commands_duration_seconds, they appear to be cumulative counts converted into seconds (Total Duration). Could you confirm if my understanding is correct?

I've consulted LLMs like ChatGPT for an explanation, but the # HELP and # TYPE lines (e.g., dragonfly_commands_duration_seconds) don't provide enough detail to fully understand the internal logic. Are there any internal guides or comprehensive documentation available for these metrics?

Ultimately, my goal is to visualize the average latency per command in Grafana. Currently, I am using the following PromQL:

sum by (cmd) (increase(dragonfly_commands_duration_seconds{pod="$pod"}[$__rate_interval]))
/
sum by (cmd) (increase(dragonfly_commands_total{pod="$pod"}[$__rate_interval]))
Is this the recommended way to calculate the average execution time per command, or is there a more standard metric I should be using?

Thanks in advance for your help!

[abhijat]

Although the code comments refer to it as a histogram, when I checked the actual values of dragonfly_commands_duration_seconds, they appear to be cumulative counts converted into seconds (Total Duration). Could you confirm if my understanding is correct?

This understanding is correct, it is a counter.

your proposed formula should work once the PR to fix the type lands.

If you want percentile style latency (p50,p99 etc) you can also use info latencystats command which exports per command latency statistics via the cli client (not on the prometheus endpoint).

[kcj-3054]

Thanks for the clarification regarding INFO LATENCYSTATS.

Since I am relying entirely on the Prometheus endpoint (/metrics) for scraping and visualizing data in Grafana, I cannot utilize the CLI-based statistics for continuous monitoring.

Given this limitation, I've decided to stick with calculating the mean latency using the available counters, as shown below:

sum by (cmd) (increase(dragonfly_commands_duration_seconds{pod="$pod"}[1m]))
/
sum by (cmd) (increase(dragonfly_commands_total{pod="$pod"}[1m]))
It would be great if Dragonfly could expose histogram buckets (e.g., _bucket) via the Prometheus endpoint in the future to allow for P99/P90 calculations directly in Grafana.

Thanks again for your help!

[romange]

this is the issue tracking the wrong type:
#6378