Skip to content

Improve reverse proxy active health check logging - #2

Closed
ryans-posthog wants to merge 2 commits into
masterfrom
posthog-code/improve-health-check-logging
Closed

ryans-posthog wants to merge 2 commits into
masterfrom
posthog-code/improve-health-check-logging

Conversation

@ryans-posthog

Copy link
Copy Markdown
Owner

Summary

  • Log host-down transitions: Added an info-level "host is down" log when a host becomes unhealthy. Previously only recovery ("host is up") was logged, making it difficult to correlate downtime events in logs.
  • Include upstream dial address: Health transition logs ("host is up" / "host is down") now include the upstream dial address, making it easier to identify which backend is affected in multi-upstream configurations.
  • Include expected values in failure logs: The "unexpected status code" log now shows both status_code and expected_status. The "response body failed expectations" log now includes the expected_body_pattern regex.

Test plan

  • Verify the project compiles with go build ./...
  • Set up a reverse proxy with active health checks and an intentionally failing backend; confirm "host is down" appears in logs with the upstream address
  • Configure expect_status and verify the log includes both actual and expected status codes when they don't match
  • Configure expect_body and verify the log includes the expected pattern when the body doesn't match

Created with PostHog Code

Add more context to active health check log messages to help operators
diagnose upstream issues:

- Log "host is down" when a host transitions to unhealthy (previously
  only "host is up" was logged on recovery, making it hard to correlate)
- Include upstream dial address in health transition logs ("host is up"
  and "host is down") for easier identification in multi-upstream setups
- Include expected_status in "unexpected status code" log so operators
  can see both actual and expected values at a glance
- Include expected_body_pattern in "response body failed expectations"
  log to clarify what regex the body was checked against

Generated-By: PostHog Code
Task-Id: e12d0e86-41aa-489e-a671-495c07b60b5d
@ryans-posthog

Copy link
Copy Markdown
Owner Author

Expand the logging a bit mre

Address feedback to expand logging further:

- Log active health checker configuration at provision time (method,
  interval, timeout, thresholds, expected status/body, upstream count)
- Add upstream dial address to all health check failure logs (HTTP
  request failed, unexpected status, status out of tolerances, body
  read failure, body mismatch) for consistent upstream identification
- Add method and URL to "HTTP request failed" log for full request
  visibility
- Add expected_range "2xx" to "status code out of tolerances" log
- Clarify passive health check error messages ("could not count
  passive health failure" / "could not forget passive health failure")
  and include fail_duration for debugging expiry issues

Generated-By: PostHog Code
Task-Id: e12d0e86-41aa-489e-a671-495c07b60b5d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant