[Flaky Test] x-pack/metricbeat/mbreceiver TestReceiverStatus

[mauri870]

Flaky Test

• Test Name: TestReceiverStatus
• Link: https://github.com/elastic/beats/blob/665ea4a/x-pack/metricbeat/mbreceiver/receiver_test.go#L363
• Branch: main
• Artifact Link: https://buildkite.com/elastic/beats-xpack-metricbeat/builds/29547#019d1f5e-767e-46d0-bec0-aafcd057dcf0/L684-L720
• Notes:

Stack Trace

=== Failed
=== FAIL: x-pack/metricbeat/mbreceiver TestReceiverStatus/degraded_input (0.03s)
    oteltest.go:191:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1774347828603654737/elastic/beats-xpack-metricbeat/x-pack/otel/oteltest/oteltest.go:191
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.25.8/go/src/runtime/asm_amd64.s:1693
        	Error:      	Not equal:
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestReceiverStatus/degraded_input
    oteltest.go:192:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1774347828603654737/elastic/beats-xpack-metricbeat/x-pack/otel/oteltest/oteltest.go:192
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.25.8/go/src/runtime/asm_amd64.s:1693
        	Error:      	Not equal:
        	            	expected: *errors.errorString(&errors.errorString{s:"Error fetching data for metricset benchmark.info: benchmark input degraded"})
        	            	actual  : <nil>(<nil>)
        	Test:       	TestReceiverStatus/degraded_input
    oteltest.go:193:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1774347828603654737/elastic/beats-xpack-metricbeat/x-pack/otel/oteltest/oteltest.go:193
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.25.8/go/src/runtime/asm_amd64.s:1693
        	Error:      	Not equal:
        	            	expected: map[string]interface {}{"inputs":map[string]interface {}{"benchmark-id":map[string]interface {}{"error":"Error fetching data for metricset benchmark.info: benchmark input degraded", "status":"StatusRecoverableError"}}}
        	            	actual  : map[string]interface {}{"inputs":map[string]interface {}{"benchmark-id":map[string]interface {}{"error":"benchmark/info is starting", "status":"StatusStarting"}}}
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,4 +3,4 @@
        	            	   (string) (len=12) "benchmark-id": (map[string]interface {}) (len=2) {
        	            	-   (string) (len=5) "error": (string) (len=74) "Error fetching data for metricset benchmark.info: benchmark input degraded",
        	            	-   (string) (len=6) "status": (string) (len=22) "StatusRecoverableError"
        	            	+   (string) (len=5) "error": (string) (len=26) "benchmark/info is starting",
        	            	+   (string) (len=6) "status": (string) (len=14) "StatusStarting"
        	            	   }
        	Test:       	TestReceiverStatus/degraded_input
    oteltest.go:94: Ingested Logs: map[r1:[{"@timestamp":"2026-03-24T10:30:34.821Z","agent":{"ephemeral_id":"030fe3c1-0f68-457b-8fc1-53955f3d4c5d","id":"cc0bb5f2-e7ab-42a3-90e0-0ebea912b4e5","name":"bk-agent-prod-gcp-1774347828603654737","type":"metricbeat","version":"9.4.0"},"benchmark":{"info":{"counter":1}},"ecs":{"version":"8.0.0"},"event":{"dataset":"benchmark.info","duration":1770,"module":"benchmark"},"host":"xxxxx","metricset":{"name":"info","period":1000},"service":{"type":"benchmark"}} {"@timestamp":"2026-03-24T10:30:34.821Z","agent":{"ephemeral_id":"030fe3c1-0f68-457b-8fc1-53955f3d4c5d","id":"cc0bb5f2-e7ab-42a3-90e0-0ebea912b4e5","name":"bk-agent-prod-gcp-1774347828603654737","type":"metricbeat","version":"9.4.0"},"ecs":{"version":"8.0.0"},"error":{"message":"benchmark input degraded"},"event":{"dataset":"benchmark.info","duration":14770,"module":"benchmark"},"host":"xxxxx","metricset":{"name":"info","period":1000},"service":{"type":"benchmark"}}]]
=== FAIL: x-pack/metricbeat/mbreceiver TestReceiverStatus (1.08s)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

tl;dr: this flake is most likely a status-timing race in the test harness (CheckReceivers), where transient StatusStarting is asserted too strictly before degraded status is emitted.

Recommendation

Update x-pack/otel/oteltest/oteltest.go so status assertions inside require.EventuallyWithT use the collector test handle (ct) and retry until the expected terminal status appears, instead of failing on transient startup state.

Findings

• x-pack/metricbeat/mbreceiver/receiver.go:23-33 starts BeatReceiver in a goroutine and returns immediately, so startup/status ordering is asynchronous.

• metricbeat/mb/module/wrapper.go:177 emits startup message ("... is starting"), which matches the observed failure value ("benchmark/info is starting", StatusStarting).

• Degraded status is only emitted later after fetch error (metricbeat/mb/module/wrapper.go:327-330), matching expected degraded message path.

• In x-pack/otel/oteltest/oteltest.go:170-194, assertions are inside EventuallyWithT, but status checks use assert.Equal(t, ...) (outer t) instead of ct, making intermediate mismatches non-retryable and flaky.

• MockHost stores only the latest event (x-pack/otel/oteltest/oteltest.go:41-45), so transient startup events can be observed at assertion time.

Verification

Commands run:

go test ./x-pack/metricbeat/mbreceiver -run TestReceiverStatus/degraded_input -count=30

Result in this runner:

github.com/aws/aws-sdk-go-v2/service/ec2: .../compile: signal: killed
FAIL github.com/elastic/beats/v7/x-pack/metricbeat/mbreceiver [build failed]

This failure is environmental (OOM/compile kill), so it did not provide a reliable repro signal for the flake itself.

I also ran a narrower repeated subtest command (shell 10), which exited 0 in this environment.

Detailed Action Plan

1. In x-pack/otel/oteltest/oteltest.go:190-194, change status assertions to use ct inside EventuallyWithT.

2. Gate strict equality assertions on status only after the expected target status is reached (ignore transient StatusStarting during polling).

3. Keep current equality checks for error + attributes, but evaluate them only when status equals expected status.

4. Stress test locally/CI with:

go test ./x-pack/metricbeat/mbreceiver -run TestReceiverStatus -count=100

5. Optionally audit other EventuallyWithT uses in x-pack/otel/oteltest for accidental t use in retry blocks.

Related Items

Type	Link	Relevance
Issue	#49618	Current flaky failure report
File	x-pack/metricbeat/mbreceiver/receiver_test.go:363-429	Flaky test definition
File	x-pack/metricbeat/mbreceiver/receiver.go:23-33	Async receiver start
File	x-pack/otel/oteltest/oteltest.go:170-194	Retry/assert logic likely causing flake
File	metricbeat/mb/module/wrapper.go:177,327-330	Starting vs degraded status emission path
PR	#49571	Recent receiver refactor context (not direct proof of root cause)

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mauri870]

/ai Thanks, even though you failed to confirm the solution with tests because of the OOM, your reasoning is spot on. I tested locally and confirmed this patch fixes the issue:

diff --git a/x-pack/otel/oteltest/oteltest.go b/x-pack/otel/oteltest/oteltest.go
index 512514e4d0..e58e864874 100644
--- a/x-pack/otel/oteltest/oteltest.go
+++ b/x-pack/otel/oteltest/oteltest.go
@@ -188,9 +188,9 @@ func CheckReceivers(params CheckReceiversParams) {
                        require.NotNil(ct, host.getEvent(), "expected not nil, got nil")

                        if params.Status != nil {
-                               assert.Equal(t, params.Status.Status(), host.Evt.Status())
-                               assert.Equal(t, params.Status.Err(), host.Evt.Err())
-                               assert.Equal(t, params.Status.Attributes().AsRaw(), host.Evt.Attributes().AsRaw())
+                               assert.Equal(ct, params.Status.Status(), host.Evt.Status())
+                               assert.Equal(ct, params.Status.Err(), host.Evt.Err())
+                               assert.Equal(ct, params.Status.Attributes().AsRaw(), host.Evt.Attributes().AsRaw())
                        }

                        if params.AssertFunc != nil {

Stresstest output:

./script/stresstest.sh ./x-pack/metricbeat/mbreceiver '^TestReceiverStatus$' -p 32 
...
4m0s: 3379 runs so far, 0 failures, 32 active
4m5s: 3443 runs so far, 0 failures, 32 active
4m10s: 3525 runs so far, 0 failures, 32 active
4m15s: 3603 runs so far, 0 failures, 32 active
4m20s: 3667 runs so far, 0 failures, 32 active
4m25s: 3746 runs so far, 0 failures, 32 active
4m30s: 3814 runs so far, 0 failures, 32 active
4m35s: 3878 runs so far, 0 failures, 32 active

Please create a PR with the fix.