Skip to content
This repository was archived by the owner on Sep 26, 2026. It is now read-only.
This repository was archived by the owner on Sep 26, 2026. It is now read-only.

Track intermittent Postgres SSL EOF on /package (Sentry DRAGONFLY-MAINFRAME-1WT) #384

Description

@import-pandas-as-numpy

Summary

Sentry issue DRAGONFLY-MAINFRAME-1WT is a recurring but low-frequency psycopg2/SQLAlchemy database error in dragonfly-mainframe:

  • OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected
  • Project: dragonfly-mainframe
  • Environment: production
  • Endpoint: /package
  • Latest occurrences: 2026-04-10 19:41:34Z and 2026-04-10 19:43:34Z
  • Matching events seen over the last 90 days: 4

This appears to affect an internal service caller, not a direct external user path.

What We Know

  • Current Sentry issue: https://vipyrsec.sentry.io/issues/DRAGONFLY-MAINFRAME-1WT
  • Active release on recent events: dragonfly-mainframe@be6a87f9ae471bc738eadacfa8017544e5c6cb70
  • Stack trace fails during SQL execution in the /package query path
  • The SQLAlchemy engine is configured with a pool but does not enable connection liveness checks such as pool_pre_ping or a recycle policy

Relevant code:

  • src/mainframe/database.py
  • src/mainframe/endpoints/package.py

Likely Cause

A stale pooled PostgreSQL connection is being reused after the database or network side has already closed it. The connection only fails when the next query executes, surfacing as SSL SYSCALL error: EOF detected.

Priority

Low. The error is intermittent, low-volume, and the caller is an internal service. This does not look like an urgent outage, but it is a reasonable reliability hardening task.

Suggested Fix

  • Enable pool_pre_ping=True on the SQLAlchemy engine
  • Consider a conservative pool_recycle value if the upstream database or proxy aggressively closes idle connections
  • Verify whether the internal caller already retries /package requests before changing priority

Definition of Done

  • Engine configuration updated to avoid reusing dead pooled connections
  • Local checks pass
  • Sentry issue volume remains flat or drops after deploy

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions