Speed up local parametric testing capabilities by 2-5x

[bm1549]

Motivation

Iterating on parametric tests locally is slow because run.sh rebuilds the library Docker image on every invocation, even when nothing has changed. This is wasted time when only the test code is changing.

Changes

Adds --skip-parametric-build (and SKIP_PARAMETRIC_BUILD=1) to run.sh: when the flag is set and the library image already exists, the build step is skipped (~3–4s saved per run).

What was dropped from the earlier version of this PR:

The original PR also added a +1 flag that forced a single xdist worker and enabled library container reuse between tests. Per reviewer feedback, this was a blocker:

• run.sh aims to stay close to bare pytest; the right way to set worker count is -n <int> (standard pytest CLI), not a custom +1 shorthand.
• Parametric tests are designed to run with different library_env parameters — sequential container reuse causes test pollution failures, which is the primary use case for this scenario.

The --skip-parametric-build flag is the main benefit for local dev iteration and carries no correctness risk.

Example:

# Edit test code, then re-run quickly without rebuilding the image
TEST_LIBRARY=nodejs ./run.sh PARAMETRIC --skip-parametric-build tests/parametric/test_tracer.py

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

.cursor/rules/parametric-testing.mdc                                    @DataDog/system-tests-core
AGENTS.md                                                               @DataDog/system-tests-core
conftest.py                                                             @DataDog/system-tests-core
docs/understand/scenarios/parametric.md                                 @DataDog/system-tests-core
utils/_context/_scenarios/parametric.py                                 @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core

[datadog-datadog-prod-us1]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 36 Tests failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[chi] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected at least one feature_flag.evaluations metric for flag 'eval-metric-basic-flag', but found none. All eval metrics: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7f40202c5670>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
    
...

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[echo] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected at least one feature_flag.evaluations metric for flag 'eval-metric-basic-flag', but found none. All eval metrics: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7f94242d0ce0>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
    
...

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[gin] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected at least one feature_flag.evaluations metric for flag 'eval-metric-basic-flag', but found none. All eval metrics: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7f328deacf80>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
    
...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: da87747 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[nccatoni]

These changes could cause reliability issues. While it might be fine to speedup local development you should make sure that all parametric tests do indeed pass. You should also get a review from @cbeauchesne (OOO this week)

[nccatoni]

Are you sure that docker doesn't already cache the images in this case ?

[bm1549]

Going based on empirical evidence, it saved an additional ~1-2 seconds per run since we avoid the layer cache validation

[cbeauchesne]

I'm not sure to get how this works, what if agent_env/otlp ports change from one test to another, it won't be applied on session_agent, right ?

[cbeauchesne]

If I understand correctly, this option is only meant for local dev.

I usually strongly recommand to use nodeid among run.sh when running on local on parametric tests, it allows to run only the subset of test. I'm wondering what is the benefit of this change using this setup ?

[bm1549]

@cbeauchesne Great question — I ran benchmarks to compare the two approaches on TEST_LIBRARY=nodejs with tests/parametric/test_tracer.py.

Single test re-run (the targeted use case)

Approach	Run 1	Run 2	Run 3	Avg wall time
Nodeid only (./run.sh PARAMETRIC <nodeid>)	12.2s	9.8s	10.5s	~10.5s
PR approach (./run.sh PARAMETRIC +1 --skip-parametric-build <nodeid>)	5.5s	6.0s	5.7s	~5.7s

~1.8x faster for a single test re-run. The savings come from:

1. --skip-parametric-build skips the Docker image build step (~3–4s saved). Without this flag, run.sh rebuilds/revalidates the parametric library image on every invocation, even when nothing changed.
2. +1 avoids spawning 10 xdist workers (gw0–gw9) for a single test.

Running a whole file (e.g. test_tracer.py, ~14 tests)

Approach	Wall time	Test failures
Nodeid only (./run.sh PARAMETRIC tests/parametric/test_tracer.py)	~20s	0
PR approach (./run.sh PARAMETRIC +1 --skip-parametric-build tests/parametric/test_tracer.py)	~70s	3 failures (container pollution)

For a whole file, nodeid-only is ~3.5x faster because xdist runs tests in parallel. The PR approach runs tests sequentially and the container reuse causes test failures via state pollution.

Summary

You're right that nodeid already narrows the test subset — and for running multiple tests, nodeid-only is both faster and more reliable. The benefit of +1 --skip-parametric-build is specifically for the common "edit → re-run one test → edit → re-run" loop, where the Docker image is never changing. The combination makes that cycle ~1.8x faster by skipping the image build each time.

The docs in this PR try to call this out (the "single-worker runs may cause failures" warning), but I agree the use case is narrower than I initially presented. Happy to adjust the recommendation to be more explicit that +1 is only for single-test re-runs.

[cbeauchesne]

+1 avoids spawning 10 xdist workers (gw0–gw9) for a single test.

We try to keep run.sh as close as possible from a bare pytest call. It means we must stick to the pytest CLI, in that case, it'll be -n <int>.

The PR approach runs tests sequentially and the container reuse causes test failures via state pollution.

To me, that's a blocker (I think it's related to my point here). The full point of parametric test is to have different parameters for a single tests, so that's by design the main use case of this scenario.

WDYT of just keeping the "no rebuild" feature, since it'll be the main benefit for the local dev using one nodeid ?

[bm1549]

WDYT of just keeping the "no rebuild" feature, since it'll be the main benefit for the local dev using one nodeid ?

Works for me! Just made the edits so it only does no rebuild