feat: add OpenTelemetry bridge for AI observability - #306
Open
posthog[bot] wants to merge 12 commits into
Open
Conversation
Adds a posthogotel Go module that forwards OpenTelemetry AI spans to PostHog AI observability. It keeps only spans that follow a known AI semantic convention (gen_ai., llm., ai., traceloop.) and sends them over OTLP/HTTP to the /i/v0/ai/otel endpoint with the project API key as a bearer token. The bridge is a separate Go module so the core posthog-go SDK stays free of OpenTelemetry dependencies. It ports the posthog-python OTel bridge: a SpanProcessor for TracerProvider setups and an Exporter for frameworks that accept only a span exporter. An example wires it to a Google ADK Go agent through the shared tracer provider. Generated-By: PostHog Desktop Task-Id: 5d65718b-f73a-48c4-9c4d-0fdd2d00fa17
Author
🦔 PostHog Review reviewed this pull requestFound 3 must fix, 8 should fix, 0 consider. Published 11 findings (view the review). Resolved comments: 8 fixed, 3 left for you |
Contributor
posthog-go Compliance ReportDate: 2026-08-31 09:49:25 UTC ✅ All Tests Passed!111/111 tests passed Capture_V1 Tests✅ 94/94 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
Author
|
PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏 |
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
Comment on lines
+6
to
+8
| go.opentelemetry.io/otel v1.28.0 | ||
| go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0 | ||
| go.opentelemetry.io/otel/sdk v1.28.0 |
Author
There was a problem hiding this comment.
Upgrade dependencies with reachable security vulnerabilities
Why we think it's a valid issue
- Checked: Ran
govulncheck v1.7.0over./...in theotelmodule with the module cache populated, then parsed the JSON output to separate findings by reachability level (CALLED / IMPORTED / REQUIRED). Also read the four cited advisories fromvuln.go.dev, the.modfiles of newer OTel SDK releases, the rootgo.mod, and the CI workflow. - Found: Two of the four cited IDs are confirmed CALLED and are fixable from this
go.mod.GO-2026-4394resolves togo.opentelemetry.io/otel/sdk— it is CVE-2026-24051, arbitrary code execution through PATH hijacking, introduced in 1.21.0 and fixed in 1.40.0, so thev1.28.0pin atotel/go.mod:8is in range.GO-2026-4985resolves togo.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp— the exporter does not limit the size of the HTTP response body it reads, so a large response causes memory exhaustion and process termination. The advisory listsotlptracehttpas fixed in 1.43.0, andotel/go.mod:7pinsv1.28.0. - Found: The
x/nethalf of the finding is misattributed.govulnchecktracesGO-2026-4918andGO-2026-5026to stdlib, not togolang.org/x/net. Both advisories carry astdlibentry alongside thegolang.org/x/netentry, and the reached path runs through the toolchain's ownnet/http. Every one of the 13golang.org/x/netfindings in this module sits at REQUIRED level, which means the module is in the graph but no vulnerable symbol is reached. Raisinggolang.org/x/netto v0.55.0 atotel/go.mod:22would therefore clear neither cited ID. Those two are fixed by the consumer's Go toolchain:net/httpis fixed in 1.25.10 / 1.26.3 forGO-2026-4918and 1.25.13 / 1.26.6 forGO-2026-5026. - Found: The fix has a real cost the author must weigh.
go.opentelemetry.io/otel/sdk@v1.40.0declaresgo 1.24.0and@v1.43.0declaresgo 1.25.0. ClearingGO-2026-4985needs 1.43.0 or newer, sootel/go.mod:3must move fromgo 1.21togo 1.25. The PR body gives Go 1.21 parity as the reason for the current pin. That parity is preserved for the core SDK either way, because the rootgo.modstays atgo 1.21and this is a separate module — only users who opt into the bridge would need Go 1.25. Theotel-bridgematrix at.github/workflows/unit-tests.yml:38tests Go 1.21 and would need updating. - Impact: This PR introduces a new public module. Any customer who runs
go get github.com/posthog/posthog-go/oteland thengovulncheckgets a CALLED arbitrary-code-execution finding and a CALLED memory-exhaustion finding on day one. Many teams gate dependency adoption on exactly that scan, so the pin works against the PR's stated goal of unblocking Go teams. Because the module is new, changing the pin costs no migration — only a decision about the minimum Go version. - Impact: The memory-exhaustion path is not theoretical for this module. The exporter built at
otel/config.goreads the response from the PostHog endpoint on every export, which is the code path the advisory describes. - Impact: One caveat for the suggested CI step. The same scan reports 17 CALLED stdlib findings that come from the scanning toolchain, not from this
go.mod. Addinggovulncheckto theotel-bridgejob without scoping it would make the job permanently red for reasons the module cannot fix.
Issue description
The module pins dependencies with published vulnerabilities in reachable code. OTel SDK v1.28.0 has the PATH-hijacking flaw GO-2026-4394. The OTLP HTTP exporter has the memory-exhaustion flaw GO-2026-4985. x/net v0.26.0 also has reachable GO-2026-4918 and GO-2026-5026 paths.
Suggested fix
Upgrade all OTel packages to v1.43.0 or newer and x/net to v0.55.0 or newer. Raise the nested module's Go version or backport the upstream fixes if necessary. Add govulncheck to the otel CI job.
Prompt to fix with AI (copy-paste)
## Context
@otel/go.mod#L6-8
@otel/go.mod#L21
<issue_description>
The module pins dependencies with published vulnerabilities in reachable code. OTel SDK v1.28.0 has the PATH-hijacking flaw GO-2026-4394. The OTLP HTTP exporter has the memory-exhaustion flaw GO-2026-4985. x/net v0.26.0 also has reachable GO-2026-4918 and GO-2026-5026 paths.
</issue_description>
<issue_validation>
- **Checked:** Ran `govulncheck v1.7.0` over `./...` in the `otel` module with the module cache populated, then parsed the JSON output to separate findings by reachability level (CALLED / IMPORTED / REQUIRED). Also read the four cited advisories from `vuln.go.dev`, the `.mod` files of newer OTel SDK releases, the root `go.mod`, and the CI workflow.
- **Found:** Two of the four cited IDs are confirmed CALLED and are fixable from this `go.mod`. `GO-2026-4394` resolves to `go.opentelemetry.io/otel/sdk` — it is CVE-2026-24051, arbitrary code execution through PATH hijacking, introduced in 1.21.0 and fixed in 1.40.0, so the `v1.28.0` pin at `otel/go.mod:8` is in range. `GO-2026-4985` resolves to `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` — the exporter does not limit the size of the HTTP response body it reads, so a large response causes memory exhaustion and process termination. The advisory lists `otlptracehttp` as fixed in 1.43.0, and `otel/go.mod:7` pins `v1.28.0`.
- **Found:** The `x/net` half of the finding is misattributed. `govulncheck` traces `GO-2026-4918` and `GO-2026-5026` to **stdlib**, not to `golang.org/x/net`. Both advisories carry a `stdlib` entry alongside the `golang.org/x/net` entry, and the reached path runs through the toolchain's own `net/http`. Every one of the 13 `golang.org/x/net` findings in this module sits at REQUIRED level, which means the module is in the graph but no vulnerable symbol is reached. Raising `golang.org/x/net` to v0.55.0 at `otel/go.mod:22` would therefore clear neither cited ID. Those two are fixed by the consumer's Go toolchain: `net/http` is fixed in 1.25.10 / 1.26.3 for `GO-2026-4918` and 1.25.13 / 1.26.6 for `GO-2026-5026`.
- **Found:** The fix has a real cost the author must weigh. `go.opentelemetry.io/otel/sdk@v1.40.0` declares `go 1.24.0` and `@v1.43.0` declares `go 1.25.0`. Clearing `GO-2026-4985` needs 1.43.0 or newer, so `otel/go.mod:3` must move from `go 1.21` to `go 1.25`. The PR body gives Go 1.21 parity as the reason for the current pin. That parity is preserved for the core SDK either way, because the root `go.mod` stays at `go 1.21` and this is a separate module — only users who opt into the bridge would need Go 1.25. The `otel-bridge` matrix at `.github/workflows/unit-tests.yml:38` tests Go 1.21 and would need updating.
- **Impact:** This PR introduces a new public module. Any customer who runs `go get github.com/posthog/posthog-go/otel` and then `govulncheck` gets a CALLED arbitrary-code-execution finding and a CALLED memory-exhaustion finding on day one. Many teams gate dependency adoption on exactly that scan, so the pin works against the PR's stated goal of unblocking Go teams. Because the module is new, changing the pin costs no migration — only a decision about the minimum Go version.
- **Impact:** The memory-exhaustion path is not theoretical for this module. The exporter built at `otel/config.go` reads the response from the PostHog endpoint on every export, which is the code path the advisory describes.
- **Impact:** One caveat for the suggested CI step. The same scan reports 17 CALLED stdlib findings that come from the scanning toolchain, not from this `go.mod`. Adding `govulncheck` to the `otel-bridge` job without scoping it would make the job permanently red for reasons the module cannot fix.
</issue_validation>
## Task
Investigate the issue and solve it
<potential_solution>
Upgrade all OTel packages to v1.43.0 or newer and x/net to v0.55.0 or newer. Raise the nested module's Go version or backport the upstream fixes if necessary. Add govulncheck to the otel CI job.
</potential_solution>
Author
There was a problem hiding this comment.
Escalating this one for a human decision rather than applying it automatically — it sits on three things an unattended fixer shouldn't decide on its own: a third-party dependency upgrade, a minimum-Go-version change, and a CI matrix edit.
The substance looks right and worth doing. Two of the four advisories are genuinely reachable (CALLED) and fixable from this module: GO-2026-4394 (arbitrary code execution via PATH hijacking) in go.opentelemetry.io/otel/sdk, fixed in 1.40.0; and GO-2026-4985 (unbounded HTTP response read → memory exhaustion) in otlptracehttp, fixed in 1.43.0 — and the exporter this module builds does read the endpoint's response on every export, so that path is real. The catch is the cost the PR already weighed the other way: OTel sdk v1.40.0 requires go 1.24 and v1.43.0 requires go 1.25, so clearing GO-2026-4985 means moving otel/go.mod from go 1.21 to go 1.25 and updating the otel-bridge CI matrix (currently Go 1.21). That only affects users who opt into the bridge — the root module stays at go 1.21 — but dropping the stated Go 1.21 parity for the bridge is a maintainer call, not a mechanical one.
The other two IDs (GO-2026-4918, GO-2026-5026) are misattributed to golang.org/x/net: govulncheck traces them to stdlib net/http, they sit at REQUIRED (not reachable) level here, and they're fixed by the consumer's Go toolchain — so bumping x/net would clear neither and isn't needed.
What a human needs to decide: (1) accept raising the bridge module's minimum to Go 1.25 (and update the CI matrix accordingly) in exchange for upgrading OTel to >= 1.43.0, or keep Go 1.21 parity and accept the two CALLED findings for now. If you want, also consider whether to scope any govulncheck CI step to this module's own findings, since the reviewer noted the raw scan reports toolchain stdlib findings the module can't fix. I've left the code untouched.
Comment on lines
+52
to
+57
| func newOTLPExporter(ctx context.Context, apiKey string, cfg config) (sdktrace.SpanExporter, error) { | ||
| return otlptracehttp.New(ctx, | ||
| otlptracehttp.WithEndpointURL(cfg.host+ingestPath), | ||
| otlptracehttp.WithHeaders(map[string]string{ | ||
| "Authorization": "Bearer " + apiKey, | ||
| }), |
Author
There was a problem hiding this comment.
Normalize the API key before using it
Why we think it's a valid issue
- Checked: Ran a throwaway test inside the module that builds a
SpanProcessoragainst anhttptestserver for five key shapes and records what the server actually receives, then traced the server-side bearer parsing inrust/capture/src/otel/mod.rsandrust/capture/src/token.rs. The temporary test was deleted; the tree is clean. - Found: A newline in the key breaks every export, permanently. With
"phc_nl\n"and with"\nphc_lead",NewSpanProcessorreturned no error, and thenForceFlushfailed withtraces export: Post "...": net/http: invalid header field value for "Authorization". The server received no request at all — Go'snet/httprejects the value before sending. The constructor's guard atotel/processor.go:22accepts these keys precisely becausestrings.TrimSpacetrims\n, so the check passes and the untrimmed value then goes tootel/config.go:56. - Found: A leading space produces a polluted token rather than a clean rejection. With
" phc_pad "the server receivedAuthorization: "Bearer phc_pad". On the server,rust/capture/src/otel/mod.rs:124-135doesauth_header.starts_with("Bearer ")and then the fixed slice&auth_header[7..], so the token becomes" phc_pad".validate_tokeninrust/capture/src/token.rs:45-68checks only empty, length, ASCII,phx_prefix, and null byte — a space passes all five — and never trims. The untrimmed token flows on intoCapturedEvent.tokenatfiltering.rs:125. There is no 401 from this handler; the token simply carries whitespace into an exact-match project lookup downstream. - Found: The finding's wording needs one correction that matters for the tests it asks for. Trailing space and trailing tab are harmless:
" phc_pad "arrived with the trailing spaces already gone, and"phc_tab\t"arrived as a clean"Bearer phc_tab"and exported successfully, because an HTTP recipient strips trailing optional whitespace. Only leading whitespace pollutes the token, and only a newline breaks the request outright. A test that pads the key with a trailing space would pass today and prove nothing. - Found: The module already treats whitespace as insignificant everywhere else.
WithHostatotel/config.go:35-36computesstrings.TrimSpace(host)and stores the trimmed value. The key path computes the same trim atotel/processor.go:22andotel/exporter.go:22but discards it, passing the original tonewOTLPExporter. This is an inconsistency inside one file, not a missing hypothetical guard. - Impact: The realistic trigger is a secret read from a file. Kubernetes secrets mounted as files and Docker secrets under
/run/secrets/routinely end with a trailing newline, andos.ReadFilepreserves it. A Go service wired that way gets a constructor that reports success and then loses every span, with an error naming neither the API key nor whitespace. That is a total failure with a misleading diagnostic. - Impact: The current state is the one state that should not ship: the code validates a trimmed key and then uses an untrimmed one. Either fix is defensible — normalise with
apiKey = strings.TrimSpace(apiKey)before use, or reject a key that is not already trimmed with a clear error. Both are a line per constructor and both remove the accept-then-fail-opaquely behaviour.
Issue description
The constructors use TrimSpace only for the empty check. They pass the original key into the Authorization header. A key with surrounding whitespace can produce an invalid token or header and make every export fail.
Suggested fix
Assign apiKey = strings.TrimSpace(apiKey) before validation and pass the normalized value. Add constructor tests for leading and trailing whitespace.
Prompt to fix with AI (copy-paste)
## Context
@otel/config.go#L52-57
<issue_description>
The constructors use TrimSpace only for the empty check. They pass the original key into the Authorization header. A key with surrounding whitespace can produce an invalid token or header and make every export fail.
</issue_description>
<issue_validation>
- **Checked:** Ran a throwaway test inside the module that builds a `SpanProcessor` against an `httptest` server for five key shapes and records what the server actually receives, then traced the server-side bearer parsing in `rust/capture/src/otel/mod.rs` and `rust/capture/src/token.rs`. The temporary test was deleted; the tree is clean.
- **Found:** A newline in the key breaks every export, permanently. With `"phc_nl\n"` and with `"\nphc_lead"`, `NewSpanProcessor` returned no error, and then `ForceFlush` failed with `traces export: Post "...": net/http: invalid header field value for "Authorization"`. The server received **no request at all** — Go's `net/http` rejects the value before sending. The constructor's guard at `otel/processor.go:22` accepts these keys precisely because `strings.TrimSpace` trims `\n`, so the check passes and the untrimmed value then goes to `otel/config.go:56`.
- **Found:** A leading space produces a polluted token rather than a clean rejection. With `" phc_pad "` the server received `Authorization: "Bearer phc_pad"`. On the server, `rust/capture/src/otel/mod.rs:124-135` does `auth_header.starts_with("Bearer ")` and then the fixed slice `&auth_header[7..]`, so the token becomes `" phc_pad"`. `validate_token` in `rust/capture/src/token.rs:45-68` checks only empty, length, ASCII, `phx_` prefix, and null byte — a space passes all five — and never trims. The untrimmed token flows on into `CapturedEvent.token` at `filtering.rs:125`. There is no 401 from this handler; the token simply carries whitespace into an exact-match project lookup downstream.
- **Found:** The finding's wording needs one correction that matters for the tests it asks for. Trailing space and trailing tab are harmless: `" phc_pad "` arrived with the trailing spaces already gone, and `"phc_tab\t"` arrived as a clean `"Bearer phc_tab"` and exported successfully, because an HTTP recipient strips trailing optional whitespace. Only **leading** whitespace pollutes the token, and only a **newline** breaks the request outright. A test that pads the key with a trailing space would pass today and prove nothing.
- **Found:** The module already treats whitespace as insignificant everywhere else. `WithHost` at `otel/config.go:35-36` computes `strings.TrimSpace(host)` and stores the trimmed value. The key path computes the same trim at `otel/processor.go:22` and `otel/exporter.go:22` but discards it, passing the original to `newOTLPExporter`. This is an inconsistency inside one file, not a missing hypothetical guard.
- **Impact:** The realistic trigger is a secret read from a file. Kubernetes secrets mounted as files and Docker secrets under `/run/secrets/` routinely end with a trailing newline, and `os.ReadFile` preserves it. A Go service wired that way gets a constructor that reports success and then loses every span, with an error naming neither the API key nor whitespace. That is a total failure with a misleading diagnostic.
- **Impact:** The current state is the one state that should not ship: the code validates a trimmed key and then uses an untrimmed one. Either fix is defensible — normalise with `apiKey = strings.TrimSpace(apiKey)` before use, or reject a key that is not already trimmed with a clear error. Both are a line per constructor and both remove the accept-then-fail-opaquely behaviour.
</issue_validation>
## Task
Investigate the issue and solve it
<potential_solution>
Assign apiKey = strings.TrimSpace(apiKey) before validation and pass the normalized value. Add constructor tests for leading and trailing whitespace.
</potential_solution>
Author
There was a problem hiding this comment.
This is a real bug, but it touches how the project API key credential is handled and how the Authorization: Bearer header is built, so per our policy on auth/secrets-handling changes I'm escalating it for a human rather than applying it automatically.
What's happening: both constructors run strings.TrimSpace(apiKey) only for the empty-key check and then pass the original, untrimmed key into the header ("Bearer " + apiKey). A trailing newline — which os.ReadFile preserves on Kubernetes/Docker secret files mounted at /run/secrets/... — makes Go's net/http reject the header value, so the constructor reports success and then every export fails with invalid header field value for "Authorization", an error that names neither the key nor whitespace. Leading whitespace instead yields a polluted "Bearer phc_..." token. (As the thread's own correction notes, trailing spaces/tabs are harmless — an HTTP recipient strips them — so only leading whitespace and newlines actually bite.)
What a human needs to decide: whether to normalize the key with apiKey = strings.TrimSpace(apiKey) before use (consistent with how WithHost already trims), or reject a non-trimmed key with a clear error. Either is one line per constructor and both remove the current accept-then-fail-opaquely behaviour; the choice and the sign-off on a credential-handling change should be a maintainer's. I've left the code untouched.
Comment on lines
+1
to
+10
| module github.com/posthog/posthog-go/otel | ||
|
|
||
| go 1.21 | ||
|
|
||
| require ( | ||
| go.opentelemetry.io/otel v1.28.0 | ||
| go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0 | ||
| go.opentelemetry.io/otel/sdk v1.28.0 | ||
| go.opentelemetry.io/proto/otlp v1.3.1 | ||
| google.golang.org/protobuf v1.34.2 |
Author
There was a problem hiding this comment.
Add dependency updates for the nested module
Why we think it's a valid issue
- Checked: The repository's
.github/dependabot.yml, the CI workflow this PR modifies, and the actual freshness of the pins in the new module. - Found: The premise is exactly right.
.github/dependabot.ymldeclares three ecosystems —gomod,npm, andgithub-actions— and every one of them setsdirectory: "/". There is no/otelentry. Dependabot'sdirectorypoints at the location of a single manifest, and eachgo.modis its own manifest, so thegomodblock covers only the root module and will never open a PR for the nested one. - Found: The suggested policy matches what already exists, so the fix is a copy of the current block with a new directory. The
gomodentry usesschedule.interval: weekly,cooldown.default-days: 7, and ago-dependenciesgroup withpatterns: ["*"]. Adirectories: ["/", "/otel"]list on the existing block is an equally small alternative to a second entry. - Found: The author already solved the identical problem for the other automation. This PR adds an
otel-bridgejob at.github/workflows/unit-tests.yml:38-59withworking-directory: otel, and the PR body explains it exists because "the root./...does not reach a nested module." The same reasoning applies to Dependabot, and only the CI half was done. - Found: The consequence is already visible in the file being flagged, not hypothetical. Running
govulncheckagainst this module reportsGO-2026-4394— arbitrary code execution through PATH hijacking ingo.opentelemetry.io/otel/sdk, fixed in 1.40.0 — andGO-2026-4985— unbounded OTLP HTTP response reads inotlptracehttp, fixed in 1.43.0 — both at CALLED reachability against thev1.28.0pins atotel/go.mod:6-8. The module ships stale on day one. - Impact: Without a Dependabot entry, nothing in the repository will ever surface the next advisory for this module's OpenTelemetry, gRPC, protobuf, or
golang.org/x/nettree. The root module keeps getting weekly grouped PRs while the nested one silently drifts, and every future fix depends on somebody thinking to run a manual scan. - Impact: This pairs with the separate finding about upgrading the vulnerable pins, and the two should be treated as one piece of work. That one clears the current advisories; this one is the mechanism that keeps them clear. Fixing only the pins leaves the same gap open for the next release.
Issue description
The nested go.mod creates an independent dependency graph. The repository's Dependabot configuration scans only directory /. It will not update this module's OpenTelemetry and network dependencies. Security and compatibility fixes will require manual discovery.
Suggested fix
Add a gomod Dependabot entry for /otel. Use the same weekly grouping policy as the root module.
Prompt to fix with AI (copy-paste)
## Context
@otel/go.mod#L1-10
<issue_description>
The nested go.mod creates an independent dependency graph. The repository's Dependabot configuration scans only directory /. It will not update this module's OpenTelemetry and network dependencies. Security and compatibility fixes will require manual discovery.
</issue_description>
<issue_validation>
- **Checked:** The repository's `.github/dependabot.yml`, the CI workflow this PR modifies, and the actual freshness of the pins in the new module.
- **Found:** The premise is exactly right. `.github/dependabot.yml` declares three ecosystems — `gomod`, `npm`, and `github-actions` — and every one of them sets `directory: "/"`. There is no `/otel` entry. Dependabot's `directory` points at the location of a single manifest, and each `go.mod` is its own manifest, so the `gomod` block covers only the root module and will never open a PR for the nested one.
- **Found:** The suggested policy matches what already exists, so the fix is a copy of the current block with a new directory. The `gomod` entry uses `schedule.interval: weekly`, `cooldown.default-days: 7`, and a `go-dependencies` group with `patterns: ["*"]`. A `directories: ["/", "/otel"]` list on the existing block is an equally small alternative to a second entry.
- **Found:** The author already solved the identical problem for the other automation. This PR adds an `otel-bridge` job at `.github/workflows/unit-tests.yml:38-59` with `working-directory: otel`, and the PR body explains it exists because "the root `./...` does not reach a nested module." The same reasoning applies to Dependabot, and only the CI half was done.
- **Found:** The consequence is already visible in the file being flagged, not hypothetical. Running `govulncheck` against this module reports `GO-2026-4394` — arbitrary code execution through PATH hijacking in `go.opentelemetry.io/otel/sdk`, fixed in 1.40.0 — and `GO-2026-4985` — unbounded OTLP HTTP response reads in `otlptracehttp`, fixed in 1.43.0 — both at CALLED reachability against the `v1.28.0` pins at `otel/go.mod:6-8`. The module ships stale on day one.
- **Impact:** Without a Dependabot entry, nothing in the repository will ever surface the next advisory for this module's OpenTelemetry, gRPC, protobuf, or `golang.org/x/net` tree. The root module keeps getting weekly grouped PRs while the nested one silently drifts, and every future fix depends on somebody thinking to run a manual scan.
- **Impact:** This pairs with the separate finding about upgrading the vulnerable pins, and the two should be treated as one piece of work. That one clears the current advisories; this one is the mechanism that keeps them clear. Fixing only the pins leaves the same gap open for the next release.
</issue_validation>
## Task
Investigate the issue and solve it
<potential_solution>
Add a gomod Dependabot entry for /otel. Use the same weekly grouping policy as the root module.
</potential_solution>
Author
There was a problem hiding this comment.
Escalating this to sit alongside the related dependency-upgrade thread, rather than applying it automatically. The premise is correct: .github/dependabot.yml has a single gomod block at directory: "/" and no /otel entry, so Dependabot only ever updates the root module — the nested otel module's OpenTelemetry, gRPC, protobuf, and x/net tree will silently drift and every future advisory there depends on someone running a manual scan.
Why a human should take this one: it's repository dependency-automation policy, and it's tightly coupled to the still-open decision on upgrading this module's vulnerable pins. A /otel Dependabot entry would immediately open PRs bumping go.opentelemetry.io/otel/sdk and otlptracehttp to their latest (1.40+/1.43+), which require Go 1.24/1.25 and clash with the module's intentional go 1.21 pin — so whether those grouped bumps are even mergeable, and whether you want ignore rules for the Go-version-raising majors, depends on how you resolve the minimum-Go question. There are also two equally clean ways to write it that are a maintainer's call: a second gomod entry with directory: "/otel", or switching the existing block to directories: ["/", "/otel"]. Either mirrors the current weekly / 7-day-cooldown / go-dependencies group policy. I've left the config untouched so you can decide the shape together with the pin upgrade.
The PostHog AI observability OTLP endpoint rejects requests carrying more than 100 spans with a non-retryable HTTP 400, which drops the whole batch. NewSpanProcessor used the OTel SDK default batch size (512), so any window producing more than 100 AI spans silently lost data. Cap the batch span processor at a shared maxSpansPerRequest (100) constant and add a regression test that emits more than 100 AI spans and asserts no export request exceeds the limit. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The example queued the span and immediately logged "sent AI span to PostHog", then relied on the deferred Shutdown to flush. The batch span processor's Shutdown returns only context errors, so a rejected export (bad API key or host) never surfaced and the program printed success and exited 0 regardless. Call provider.ForceFlush before the success message so the export error is surfaced and reported, and print the success line only once the export is confirmed. On failure the program returns rather than exiting hard so the deferred Shutdown still runs. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's ADK Go section implied a complete integration ("no further
code"). The bridge forwards spans only, but ADK Go emits prompt and
response message content as OpenTelemetry log records, not span attributes,
so those fields arrive empty in PostHog. Document the gap so users do not
read the empty prompt/response as a broken integration.
Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI Gateway captures its own $ai_generation on every routed call. A service that both routes through the gateway and exports spans through this bridge double-counts and double-bills every generation, with no signal that it is happening. The sibling posthog-python and posthog-js SDKs guard against this with a warning; the Go port dropped it. Port the guard: SpanProcessor.OnEnd now inspects the AI span's server.address / url.full attributes against the known gateway hosts and logs a warning (matching the siblings' plain-warn behaviour, never dropping the span). Host list and detection logic mirror posthog-python's posthog/ai/gateway.py. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
otlptracehttp.WithEndpointURL swallows a URL parse error: it logs and keeps its localhost:4318 default, so a malformed WithHost value produces a working-looking processor that never reaches PostHog while the request still carries the Authorization header. A scheme-less host such as "us.i.posthog.com" parses but leaves the endpoint empty. Both are silent data loss reachable straight from the documented POSTHOG_ENDPOINT path. Validate the resolved host in newConfig and return an error from NewSpanProcessor / NewExporter, completing the input validation the module already performs on the API key. The host must be an absolute http or https URL with a hostname; query and fragment are left alone as the review scoped. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI observability endpoint rejects requests carrying more than 100 spans with a non-retryable HTTP 400 that discards the whole batch. Capping the SpanProcessor's batch size covered only that entry point; the caller-supplied Exporter path (wired with its own batch processor, default 512) could still hand an oversized slice to a single request, because nothing below this module splits a batch. Wrap the OTLP exporter in a chunkingExporter that splits each ExportSpans call into slices of at most maxSpansPerRequest. This lives in newOTLPExporter, so it protects both the SpanProcessor and the Exporter regardless of the feeding processor's batch size. Adds a 101+-span regression test for the Exporter path. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's usage snippet reused the construction context for the deferred provider.Shutdown and discarded its error. When that context is already canceled at shutdown — the standard signal.NotifyContext graceful shutdown idiom — the OTel SDK returns ctx.Err() before running the batch processor's final export, so every buffered AI span is dropped with no error surfaced. This is the module's primary copy-paste path and it contradicted the fresh-timeout-context pattern the shipped example already uses. Mirror the example: a fresh context.WithTimeout and a reported error. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The usage snippet built a new sdktrace.TracerProvider and made it global. For the flagship case — an ADK Go agent inside a service that already has OpenTelemetry configured — this loses AI spans: the OTel global delegation fires only once, so an already-installed provider's tracers (including ADK's, cached at package init) stay bound to it and never reach the PostHog processor, while the replacement provider also drops the user's resource, sampler, and existing exporters. The README prose already said "register it on your TracerProvider", which the code contradicted. Show provider.RegisterSpanProcessor(processor) on the application's own *sdktrace.TracerProvider, and shut down the processor (not the provider) so PostHog's setup does not tear down the user's tracing pipeline. Keeps the fresh-context, error-reported shutdown from the previous fix. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
Member
|
Live US Cloud validation succeeded for the expanded example: the span was ingested as cc @PostHog/team-ai-observability |
Member
|
ingested event example {
"createdAt": "2026-08-31T09:44:37.958000+00:00",
"event": "$ai_generation",
"id": "01a05734-e115-7103-8a19-54b4da519853",
"properties": {
"$ai_ingestion_source": "otel",
"$ai_span_id": "3d0d99fac071acbe",
"$ai_trace_id": "b46ea40bb3555a8eb0be877729a2013b",
"$geoip_disable": true,
"example.run_id": "20260831T094437.958644000Z",
"gen_ai.response.finish_reasons": [
"stop"
],
"gen_ai.response.id": "chatcmpl-posthog-go-example",
"service.name": "posthog-go-otel-example",
"$ip": "84.115.233.26",
"$ai_input_tokens": 18,
"$ai_output_tokens": 11,
"$ai_model": "gpt-4o-mini-2024-07-18",
"$ai_provider": "openai",
"$ai_base_url": "api.openai.com",
"$ai_span_name": "chat posthog-go OTel example",
"$ai_latency": 0.051059583,
"$ai_cache_reporting_exclusive": false,
"$ai_input_cost_usd": 0.0000027,
"$ai_output_cost_usd": 0.0000066,
"$ai_request_cost_usd": 0,
"$ai_web_search_cost_usd": 0,
"$ai_total_cost_usd": 0.0000093,
"$ai_model_cost_used": "openai/gpt-4o-mini-2024-07-18",
"$ai_cost_model_source": "openrouter",
"$ai_cost_model_provider": "openai",
"$ai_input": [
{
"role": "system",
"content": "Answer concisely."
},
{
"role": "user",
"content": "What is PostHog?"
}
],
"$ai_output_choices": [
{
"role": "assistant",
"content": "PostHog is an open-source product analytics platform."
}
]
},
"sentiment": null
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
$ai_*event arrived with$lib = posthog-go, which matched the code: grepping the SDK for$ai_generation,$ai_span, or anythingllm-shaped returned nothing. Those users got a flat "no" on a product they were otherwise ready to pay for.posthog-pythonships a generic OpenTelemetry bridge that keepsgen_ai./llm./ai./traceloop.spans and forwards them to/i/v0/ai/otel. Google's Agent Development Kit (ADK) for Go emits exactly thosegen_ai.*spans — there was just nothing on the Go side pointing at them.This PR ports that bridge to Go.
github.com/posthog/posthog-go/otelmodule (posthogotel):SpanProcessor— recommended integration; register it on aTracerProvider. Filters, batches, and exports AI spans.Exporter— for frameworks that accept only a span exporter.IsAISpan— the shared prefix filter (gen_ai.,llm.,ai.,traceloop.), matched against the span name and every attribute key.{host}/i/v0/ai/otelover OTLP/HTTP withAuthorization: Bearer <project api key>;WithHostswitches cloud region.example/wires the processor to the tracer provider an ADK Go agent emits through.Why a separate Go module
The core SDK today has a lean dependency graph. Folding the OTel SDK,
otlptracehttp, and their protobuf/grpc transitive tree into the rootgo.modwould burden everyposthog-goconsumer for an optional feature. A nested module keeps those dependencies opt-in, so only users who import.../otelpull them.💚 How did you test it?
IsAISpan: name-prefix match, attribute-key match, and a non-AI negative, using realReadOnlySpans from aSpanRecorder.httptestOTLP server that decodes the protobuf payload and asserts:gen_ai.chat) reaches the endpoint when an AI and a non-AI span are emitted together, for bothSpanProcessorandExporter;/i/v0/ai/oteland the header isAuthorization: Bearer <key>;WithHostnormalisation (blank falls back to default, trailing slash trimmed).go build,go vet, andgo test -racepass in the module. Added anotel-bridgeCI job (Go 1.25 and 1.x), since the root./...does not reach a nested module.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset fileThe changeset release workflow versions the root module and this PR does not touch it. The
otelmodule versions independently throughotel/vX.Y.Zgit tags, so no changeset is added.🤖 Agent context
Autonomy: Fully autonomous
posthog-pythonposthog/ai/otel/bridge (spans.py,exporter.py,processor.py) as the reference and ported its behaviour..gofiles in the root package: the report suggested following theerror_tracking*.gopattern, but that product added no third-party dependencies, whereas the OTel bridge pulls in grpc/protobuf. Isolating it keeps the core SDK lean.v1.43.0and the bridge module to Go 1.25 to avoid reachable vulnerabilities in older OTel releases; the separate core module remains on Go 1.21.gen_ai.*span, with the ADK hookup documented.Created with PostHog Desktop from this inbox report.