Skip to content

fix(client): do not panic when the HTTP client cannot be built - #247

Draft
posthog[bot] wants to merge 4 commits into
mainfrom
posthog-self-driving/fixcli-stop-sourcemap-inject-from-1076b9
Draft

fix(client): do not panic when the HTTP client cannot be built#247
posthog[bot] wants to merge 4 commits into
mainfrom
posthog-self-driving/fixcli-stop-sourcemap-inject-from-1076b9

Conversation

@posthog

@posthog posthog Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

  • A posthog-cli sourcemap inject run stops before it processes any source map when the container has no CA certificates. The telemetry client kills a build command that must not depend on telemetry.
  • Both client() constructors call HttpClient::builder().build().unwrap(). The comment says the unwrap is as safe as HttpClient::new, but the TLS trust-store setup makes build() fallible.
  • reqwest asks rustls_platform_verifier::Verifier::new() for the platform roots. With an empty root store it returns No CA certificates were loaded from the system, so build() returns Err and the unwrap panics.
  • The SDK already has a graceful-degradation path (a disabled client). The panic happens before that path can run.
Before After
Trust store present client sends events client sends events
No CA certificates panic, caller stops warning, disabled client, caller continues

Changes

  • A shared helper takes the build result. On error it logs a warning and sets the options to disabled, so the local-evaluation poller and the background transport (both gated on is_disabled()) never start.
  • Client.client is now Option<HttpClient>. A private http() accessor returns Error::Connection instead of a panic if a request path is ever reached with no HTTP client.
  • Same change in the blocking client and the async client.

💚 How did you test it?

  • New unit test: a failed build gives no HTTP client and disables the options.
  • Reproduced the real failure with an empty trust store (SSL_CERT_FILE and SSL_CERT_DIR pointed at an empty certificate set) and a small example program:
Before and after, both clients

Before (async client, on main):

thread 'main' panicked at src/client/async_client.rs:134:10:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Builder,
source: General("No CA certificates were loaded from the system") }

After, with the same environment:

posthog client constructed, capture: ()          # async client
blocking client constructed, capture: ()         # blocking client
  • cargo test, cargo test --no-default-features, cargo test --no-default-features --features capture-v1,error-tracking, cargo fmt -- --check, and cargo clippy -- -D warnings pass. (cargo clippy --all-targets has failures on main that this branch does not change.)

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

Agent context

  • Considered a fallback HTTP client on failure. There is none to build: any second build() fails for the same reason.
  • The unwrap_or_default() calls in the capture transport have the same latent problem, but they are only reached when the client is enabled, which now implies a successful build. Left as is.

Created with PostHog Desktop from this inbox report.

reqwest's build() is fallible because it sets up the TLS trust store. In a container without CA certificates it returns an error, and the unwrap stopped the calling program.

Both clients now log a warning, disable themselves, and return a no-op client instead.

Generated-By: PostHog Desktop
Task-Id: e3125f9b-fea1-42ef-83b2-4e8bcbddc7bb
@posthog

posthog Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

🦔 PostHog Review reviewed this pull request

Found 0 must fix, 1 should fix, 1 consider.

Published 2 findings (view the review).

Resolved comments: 2 fixed

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

posthog-rs-v0 Compliance Report

Date: 2026-09-05 13:21:10 UTC
Duration: 15568ms

✅ All Tests Passed!

46/46 tests passed


Capture Tests

29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields 148ms
Format Validation.Event Has Uuid 160ms
Format Validation.Event Has Lib Properties 159ms
Format Validation.Distinct Id Is String 157ms
Format Validation.Token Is Present 158ms
Format Validation.Custom Properties Preserved 158ms
Format Validation.Event Has Timestamp 157ms
Retry Behavior.Retries On 503 5160ms
Retry Behavior.Does Not Retry On 400 2159ms
Retry Behavior.Does Not Retry On 401 2159ms
Retry Behavior.Respects Retry After Header 5119ms
Retry Behavior.Implements Backoff 15109ms
Retry Behavior.Retries On 500 5109ms
Retry Behavior.Retries On 502 5104ms
Retry Behavior.Retries On 504 5105ms
Retry Behavior.Max Retries Respected 15109ms
Deduplication.Generates Unique Uuids 112ms
Deduplication.Preserves Uuid On Retry 5024ms
Deduplication.Preserves Uuid And Timestamp On Retry 10040ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5036ms
Deduplication.No Duplicate Events In Batch 26ms
Deduplication.Different Events Have Different Uuids 23ms
Compression.Sends Gzip When Enabled 23ms
Batch Format.Uses Proper Batch Structure 22ms
Batch Format.Flush With No Events Sends Nothing 23ms
Batch Format.Multiple Events Batched Together 101ms
Error Handling.Does Not Retry On 403 2069ms
Error Handling.Does Not Retry On 413 2082ms
Error Handling.Retries On 408 5082ms

Feature_Flags Tests

17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 67ms
Request Payload.Flags Request Uses V2 Query Param 53ms
Request Payload.Flags Request Hits Flags Path Not Decide 28ms
Request Payload.Flags Request Omits Authorization Header 29ms
Request Payload.Token In Flags Body Matches Init 29ms
Request Payload.Groups Round Trip 38ms
Request Payload.Groups Default To Empty Object 40ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 49ms
Request Payload.Disable Geoip Omitted Defaults To False 31ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 61ms
Request Lifecycle.No Flags Request On Init Alone 23ms
Request Lifecycle.No Flags Request On Normal Capture 48ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 41ms
Request Lifecycle.Mock Response Value Is Returned To Caller 31ms
Retry Behavior.Retries Flags On 502 231ms
Retry Behavior.Retries Flags On 504 241ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 38ms

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

posthog-rs-v1 Compliance Report

Date: 2026-09-05 13:21:35 UTC
Duration: 22998ms

✅ All Tests Passed!

111/111 tests passed


Capture_V1 Tests

94/94 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint 151ms
Endpoint And Method.Does Not Use Legacy Endpoints 149ms
Required Headers.Has Authorization Bearer Header 148ms
Required Headers.Has Content Type Json 150ms
Required Headers.Has Posthog Sdk Info Format 150ms
Required Headers.Has Posthog Attempt Header 149ms
Required Headers.Has Posthog Request Id 149ms
Required Headers.Has Posthog Request Timestamp 149ms
Required Headers.Has User Agent 149ms
Body Format.Body Has Created At And Batch 147ms
Body Format.No Api Key In Body 139ms
Body Format.No Sent At In Body 139ms
Event Format.Event Has Required Root Fields 138ms
Event Format.Event Uuid Is Valid 138ms
Event Format.Event Timestamp Is Rfc3339 138ms
Event Format.Distinct Id Is String 137ms
Event Format.Distinct Id At Root Not Properties 141ms
Event Format.Custom Properties Preserved 137ms
Event Format.Set Properties Preserved 137ms
Event Format.Set Once Properties Preserved 137ms
Event Format.Groups Properties Preserved 120ms
Event Format.Sdk Generates Uuid If Not Provided 120ms
Event Format.Event Has Required Root Fields Batch 152ms
Event Format.Event Uuid Is Valid Batch 155ms
Event Format.Event Timestamp Is Rfc3339 Batch 165ms
Event Format.Distinct Id Is String Batch 155ms
Event Format.Distinct Id At Root Not Properties Batch 154ms
Event Format.Custom Properties Preserved Batch 155ms
Event Format.Set Properties Preserved Batch 167ms
Event Format.Set Once Properties Preserved Batch 152ms
Event Format.Groups Properties Preserved Batch 140ms
Event Format.Sdk Generates Uuid If Not Provided Batch 139ms
Batch Behavior.Multiple Events In Single Batch 189ms
Batch Behavior.Batch Envelope Smoke 175ms
Batch Behavior.Flush With No Events Sends Nothing 85ms
Batch Behavior.Flush At Triggers Batch 1123ms
Batch Behavior.Created At Reflects Batch Creation Time 103ms
Deduplication.Generates Unique Uuids 187ms
Deduplication.Different Events Same Content Different Uuids 118ms
Deduplication.Preserves Uuid On Retry 5123ms
Deduplication.Preserves Timestamp On Retry 5082ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5101ms
Deduplication.No Duplicate Events In Batch 145ms
Header Behavior On Retry.Attempt Header Starts At One 76ms
Header Behavior On Retry.Attempt Header Increments On Retry 10064ms
Header Behavior On Retry.Request Id Preserved On Retry 5066ms
Header Behavior On Retry.Different Requests Have Different Request Ids 2069ms
Header Behavior On Retry.Request Timestamp Changes On Retry 5061ms
Response Format Validation.Success Response Has Uuid Keyed Results 63ms
Response Format Validation.Success Response Has Ok For Each Event 41ms
Response Format Validation.Success No Retry After When All Ok 34ms
Response Format Validation.Success Retry After Present When Retry Events 31ms
Response Format Validation.Success No Retry After When Drop Only 32ms
Response Format Validation.Response Echoes Request Id 30ms
Retry Behavior.Retries On 408 5033ms
Retry Behavior.Retries On 500 5024ms
Retry Behavior.Retries On 503 5025ms
Retry Behavior.Retries On 504 5024ms
Retry Behavior.Retryable Errors Have Retry After 2025ms
Retry Behavior.Respects Retry After On Retryable Error 8024ms
Retry Behavior.Does Not Retry On 400 2026ms
Retry Behavior.Does Not Retry On 401 2029ms
Retry Behavior.Does Not Retry On 402 2030ms
Retry Behavior.Does Not Retry On 413 2026ms
Retry Behavior.Does Not Retry On 415 2025ms
Retry Behavior.Non Retryable Errors Have No Retry After 2026ms
Retry Behavior.Implements Backoff 15029ms
Retry Behavior.Max Retries Respected 15023ms
Partial Batch Handling.Handles 200 Full Success 2044ms
Partial Batch Handling.Handles 200 With All Ok 3043ms
Partial Batch Handling.Does Not Retry Dropped Events 3045ms
Partial Batch Handling.Does Not Retry Limited Events 3035ms
Partial Batch Handling.Prunes Ok Events On Partial Retry 5031ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry 5025ms
Partial Batch Handling.Retries Only Retry Events From Partial 5026ms
Partial Batch Handling.Partial Retry Preserves Uuids 5024ms
Partial Batch Handling.Partial Retry Attempt Header Increments 5034ms
Partial Batch Handling.Partial Retry Request Id Preserved 5035ms
Partial Batch Handling.Respects Retry After On Partial 5025ms
Partial Batch Handling.Unknown Result Treated As Terminal 3024ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry 3034ms
Compression.Sends Gzip Content Encoding 32ms
Compression.No Content Encoding When Disabled 21ms
Compression.Compressed Body Is Decompressible 23ms
Error Handling.Does Not Retry On Unknown 4Xx 2024ms
Event Options.Cookieless Mode Override 24ms
Event Options.Disable Skew Correction Override 22ms
Event Options.Process Person Profile Override 21ms
Event Options.Product Tour Id Override 23ms
Event Options.Unset Options Omitted 22ms
Event Options.Options Override In Batch 21ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties 22ms
Geoip And Historical Migration.Historical Migration Set In Body 23ms
Geoip And Historical Migration.Historical Migration Absent By Default 23ms

Feature_Flags Tests

17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 14ms
Request Payload.Flags Request Uses V2 Query Param 14ms
Request Payload.Flags Request Hits Flags Path Not Decide 14ms
Request Payload.Flags Request Omits Authorization Header 14ms
Request Payload.Token In Flags Body Matches Init 14ms
Request Payload.Groups Round Trip 14ms
Request Payload.Groups Default To Empty Object 15ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 14ms
Request Payload.Disable Geoip Omitted Defaults To False 16ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 15ms
Request Lifecycle.No Flags Request On Init Alone 11ms
Request Lifecycle.No Flags Request On Normal Capture 22ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 18ms
Request Lifecycle.Mock Response Value Is Returned To Caller 15ms
Retry Behavior.Retries Flags On 502 218ms
Retry Behavior.Retries Flags On 504 219ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 23ms

@posthog

posthog Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review

Found 1 should fix, 1 consider.

Comment thread src/client/async_client.rs
Comment thread src/client/common.rs
posthog Bot added 3 commits September 5, 2026 13:04
The capture worker built its blocking reqwest client with unwrap_or_default(), but Default for reqwest::blocking::Client calls Client::new(), which panics on the same failures build() reports. The blocking builder can also fail where the client's async build succeeded, because it spawns its own thread and runtime.

Pipeline::new now returns None on a build failure and logs a warning, and run_worker signals any queued completions and returns instead of panicking on a thread whose panic is swallowed by join().

Generated-By: PostHog Desktop
Task-Id: ef58b299-44ac-4dc4-be2e-66379d7c883a
FlagPoller::new and AsyncFlagPoller::new unwrapped the reqwest builder, so a direct user of either public poller still panicked in the container this PR is about — one without CA certificates, where the TLS trust-store setup makes build() fail.

Both pollers now hold an Option client: a failed build logs a warning, load_flags returns Error::Connection, and start keeps the poller stopped instead of running a thread or task that could only log failures. The blocking poller's polling thread reuses the client built in new instead of building a second one, which removes the third unwrap; the async poller already cloned its client that way.

Generated-By: PostHog Desktop
Task-Id: 6b381563-d746-41ae-94d3-5fb69dd64ec9
reqwest's Display for a builder failure writes the fixed text "builder error" and nothing else, so the warnings this PR adds in place of the panic said only that the build failed. Verified against the resolved reqwest 0.13.4 (Display writes the kind, Debug adds the source) and reproduced with an empty trust store: Display gives "builder error", Debug gives reqwest::Error { kind: Builder, source: General("No CA certificates were loaded from the system") }.

The three build-failure warnings — the shared client helper, the transport worker, and the flag pollers — now format the error with Debug, which prints the source the operator needs. The helpers take a Debug bound instead of Display; every caller already passes a reqwest::Error or a &str.

Generated-By: PostHog Desktop
Task-Id: 6b381563-d746-41ae-94d3-5fb69dd64ec9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants