fix(auth): fix OpenTelemetry test flake in grpctransport - #20191
Open
macastelaz wants to merge 1 commit into
Open
fix(auth): fix OpenTelemetry test flake in grpctransport#20191macastelaz wants to merge 1 commit into
macastelaz wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the OpenTelemetry test setup in auth/grpctransport/grpctransport_otel_test.go by moving the creation and cleanup of the in-memory exporter and tracer provider from the outer test functions into the individual subtests. This ensures test isolation and prevents state leakage between subtests. I have no feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes a subtle race condition in grpctransport_otel_test.go that caused
flaky failures in TestDial_OpenTelemetry_Disabled and other OpenTelemetry
tests (e.g. Issue #20175).
Root Cause
The flake was caused by cross-test leakage of OpenTelemetry spans. Tests
looped over several sub-test conditions (t.Run), but the test set up the
tracetest.NewInMemoryExporter() and the global TracerProvider only once
outside of this loop, attempting to clear out previous test data by calling
exporter.Reset() at the start of each iteration.
However, the OpenTelemetry gRPC stats handler runs asynchronously. For the
test cases simulating an error (e.g., client timeout or client cancelled),
the client.Echo method returned immediately, but the background goroutine
tearing down the gRPC stream hadn't generated its stats.End event yet. If
these delayed spans were flushed after the next test case ("telemetry
disabled") called exporter.Reset(), they were written to the shared
exporter. The "telemetry disabled" test case would then assert that
len(spans) == 0 but would instead find 1 from the previous test causing the
flake.
The Fix
Scope the OpenTelemetry exporter and TracerProvider initialization strictly
inside the t.Run block so a completely fresh exporter instance is created
per sub-test. This ensures delayed background spans end up in the old
provider's memory where they are harmlessly discarded.
Evidence of Fix
Ran the transport tests with -race and a high -count locally to ensure the
race condition is resolved and no cross-test pollution occurs.
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_timeout
=== RUN
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_cancelled
=== RUN TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_error
--- PASS: TestDial_OpenTelemetry_Disabled (0.01s)
--- PASS: TestDial_OpenTelemetry_Disabled/telemetry_disabled (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_timeout (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_cancelled (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_error (0.00s)