Skip to content

fix(auth): fix OpenTelemetry test flake in grpctransport - #20191

Open
macastelaz wants to merge 1 commit into
mainfrom
fix-grpc-otel-test-flake
Open

fix(auth): fix OpenTelemetry test flake in grpctransport#20191
macastelaz wants to merge 1 commit into
mainfrom
fix-grpc-otel-test-flake

Conversation

@macastelaz

Copy link
Copy Markdown

Description

Fixes a subtle race condition in grpctransport_otel_test.go that caused
flaky failures in TestDial_OpenTelemetry_Disabled and other OpenTelemetry
tests (e.g. Issue #20175).

Root Cause

The flake was caused by cross-test leakage of OpenTelemetry spans. Tests
looped over several sub-test conditions (t.Run), but the test set up the
tracetest.NewInMemoryExporter() and the global TracerProvider only once
outside of this loop, attempting to clear out previous test data by calling
exporter.Reset() at the start of each iteration.

However, the OpenTelemetry gRPC stats handler runs asynchronously. For the
test cases simulating an error (e.g., client timeout or client cancelled),
the client.Echo method returned immediately, but the background goroutine
tearing down the gRPC stream hadn't generated its stats.End event yet. If
these delayed spans were flushed after the next test case ("telemetry
disabled") called exporter.Reset(), they were written to the shared
exporter. The "telemetry disabled" test case would then assert that
len(spans) == 0 but would instead find 1 from the previous test causing the
flake.

The Fix

Scope the OpenTelemetry exporter and TracerProvider initialization strictly
inside the t.Run block so a completely fresh exporter instance is created
per sub-test. This ensures delayed background spans end up in the old
provider's memory where they are harmlessly discarded.

Evidence of Fix

Ran the transport tests with -race and a high -count locally to ensure the
race condition is resolved and no cross-test pollution occurs.

$ cd auth/grpctransport && go test -v -race -count 20 .                  
=== RUN   TestDial_OpenTelemetry_Disabled                                
=== RUN   TestDial_OpenTelemetry_Disabled/telemetry_disabled             
=== RUN                                                                  

TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_timeout
=== RUN
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_cancelled
=== RUN TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_error
--- PASS: TestDial_OpenTelemetry_Disabled (0.01s)
--- PASS: TestDial_OpenTelemetry_Disabled/telemetry_disabled (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_timeout (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_cancelled (0.00s)
--- PASS:
TestDial_OpenTelemetry_Disabled/telemetry_disabled_client_error (0.00s)

# ... (repeated across other tests for 20 iterations without failure)    
                                                                         
PASS                                                                     
ok      cloud.google.com/go/auth/grpctransport    5.689s                        
```***    

@macastelaz
macastelaz requested review from a team as code owners July 22, 2026 18:45

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the OpenTelemetry test setup in auth/grpctransport/grpctransport_otel_test.go by moving the creation and cleanup of the in-memory exporter and tracer provider from the outer test functions into the individual subtests. This ensures test isolation and prevents state leakage between subtests. I have no feedback to provide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants