feat(observability): spool Sentry events to disk until confirmed

CLAUDE.md

@@ -50,6 +50,7 @@ Domain terms are defined in `GLOSSARY.md`. Nothing is created at import time. Th
 - `src/croupier/printing.py` — **Message** (raw ESC/POS `content: bytes` plus `network_host`, `network_timeout`) and **Receipt printing**: `print_receipt(message, connect=connect_network)`. Blocking; always attempts `close()`; errors propagate. The `connect` parameter is the seam: production uses `escpos.Network`, tests use `escpos.printer.Dummy`. `open()` and `_raw()` run inside a `try/finally`; a close failure is narrow-excepted (PEP 758 unparenthesized form, `except OSError, AttributeError:`), logged via `logger.exception` (promoted to a Sentry event) and never replaces the original error — broader exceptions intentionally surface. This is the only place that calls `printer._raw()`.
 - `src/croupier/main.py` — `build_broker(settings, connect=...)` builds the `RabbitBroker` whose single `handle_message` subscriber runs `print_receipt` via `asyncio.to_thread`; `create_app(settings=None, connect=...)` wraps it in an `AsgiFastStream` through lite-bootstrap's `FastStreamBootstrapper` (Sentry + structlog); `main()` calls `create_app()`, disables Windows console QuickEdit and runs `uvicorn.run`. Compatible with `uvicorn --factory`. Bootstrap reconfigures global state (structlog, Sentry init); call once per process.
 - **SentryMiddleware** — `BaseMiddleware[Any, bytes]` registered on the broker only when `sentry_dsn` is set. Opens a per-message `sentry_sdk.isolation_scope()`, tags it with `correlation_id` (skipped when empty), `logger.exception(...)` on unhandled exceptions (auto-promoted to a Sentry event by `LoggingIntegration`) and re-raises so the reject→DLX/DLQ path still runs. `IgnoredException` is re-raised without capture. Plain passthrough when `sentry_sdk.is_initialized()` is `False`. Exception class is filterable via Sentry's built-in `error.type` index.
+- `src/croupier/spool.py` — `SpoolTransport` / `make_spool_transport(spool_dir)`, passed to Sentry through `sentry_additional_params` (default dir `~/.croupier-sentry-spool`, see `DEFAULT_SPOOL_DIR`). Each envelope (and, on POSIX, its directory entry) is fsynced to disk before the send and deleted only on 2xx or a permanent rejection (400, 401, 403, 413); dead link, 408, 429, 5xx and an envelope with rate-limited items keep it, and leftovers replay at startup and every 60 s. Capped at 500 files, oldest evicted. It relies on private `sentry-sdk` methods (`_worker`, `_send_envelope`, `_flush_client_reports`, `_check_disabled`, `_request`); `tests/test_spool.py` fails on a `sentry-sdk` bump that breaks them. A duplicate after a crash is accepted (Sentry dedups by event ID). See `docs/adr/0002-sentry-events-spool-to-disk.md`.
 - **Per-deployment branching** — `queue_name` is the fleet-unique identifier surfaced as the Sentry `queue_name` tag (`receipt.dispatch.<branch_id>`), so signals stay separable when multiple instances share infrastructure.
 - **Health** — `GET /health/` payload comes from lite-bootstrap (no override). There is no HTTP receipt route; receipts arrive only via the queue.
 - **Logging** — structlog → JSON on stdout via lite-bootstrap's `LoggingInstrument` (`service_debug=False`). No file handler.

docs/adr/0002-sentry-events-spool-to-disk.md

@@ -0,0 +1,56 @@
+# 2. Sentry events spool to disk until Sentry confirms them
+
+Date: 2026-10-21 — Status: accepted
+
+## Context
+
+Croupier runs on branch computers with poor ethernet that also lose power. The
+stock `sentry-sdk` transport keeps events in an in-memory queue and, when a
+send fails, counts the loss and drops the event. Anything captured during an
+outage, or still queued when the machine loses power, never reaches Sentry.
+Logs do not reach disk either (stdout only), so for an outage the spool is the
+only on-disk record.
+
+Sentry ships offline buffering for its JavaScript, Electron and .NET SDKs but
+not for Python (`getsentry/sentry-python#3514` was closed without
+implementation; `getsentry/develop#476` collects the cross-SDK design notes).
+The only third-party option, `sentry-offline-transport` 1.0.0, saves an
+envelope only *after* a failed send, ignores non-2xx responses, has no size
+cap, replays only at startup, deletes before re-sending and writes
+non-atomically.
+
+## Decision
+
+`croupier.spool.SpoolTransport`, installed through lite-bootstrap's
+`sentry_additional_params={"transport": ...}`:
+
+- Every envelope is written to `~/.croupier-sentry-spool/<dsn-hash>/` (temp
+  file, `fsync`, rename) before the send is attempted.
+- The file is deleted only on a definitive answer: 2xx, or a permanent 4xx
+  (400, 401, 403, 413). A dead link, 429, 5xx, or a send the SDK skipped while
+  rate-limited keeps it. `sentry-sdk` does not raise on non-2xx, so the status
+  is read by overriding `_request`.
+- Leftovers replay at startup and every 60 seconds. A path is claimed under a
+  lock before it is written visible or read for replay, so an envelope is never
+  sent twice concurrently.
+- It never takes the app down: an unusable spool directory at startup degrades
+  to the stock in-memory transport, a write failure falls back to a plain send,
+  and a file that cannot be read (for example locked by antivirus) is kept
+  rather than deleted. Filenames stay in capture order even if the clock jumps
+  back after a power loss.
+- The spool holds at most 500 files; the oldest are evicted first. Corrupt files
+  and client-report envelopes are discarded or never spooled.
+
+## Consequences
+
+- A crash after Sentry accepted an event but before the file is deleted replays
+  it as a duplicate. Sentry dedups by event ID; do not add machinery to prevent
+  it. Eviction can likewise delete a file that is mid-send, which is harmless.
+- The transport depends on private `sentry-sdk` internals: `_worker`,
+  `_send_envelope`, `_flush_client_reports`, `_check_disabled` and `_request`. A `sentry-sdk` bump
+  that changes them fails `tests/test_spool.py` (including the kill-and-recover
+  and stress tests), so a Renovate PR will not go green silently.
+- Events replayed after a long outage arrive late but keep their original
+  timestamps.
+- Every event costs one small synchronous disk write on the capturing thread.
+  Croupier's error volume is low.

src/croupier/main.py

@@ -25,13 +25,16 @@
 from croupier.printing import PrinterFactory
 from croupier.printing import connect_network
 from croupier.printing import print_receipt
+from croupier.settings import DEFAULT_SPOOL_DIR
 from croupier.settings import Settings
 from croupier.settings import load_settings
+from croupier.spool import make_spool_transport
 
 if TYPE_CHECKING:
     from collections.abc import Awaitable
     from collections.abc import Callable
     from collections.abc import Iterator
+    from pathlib import Path
 
     from faststream.message import StreamMessage
 
@@ -237,6 +240,7 @@ async def handle_message(body: Message) -> None:
 def create_app(
     settings: Settings | None = None,
     connect: PrinterFactory = connect_network,
+    spool_dir: Path = DEFAULT_SPOOL_DIR,
 ) -> AsgiFastStream:
     """ASGI app factory that wires Sentry and structlog through lite-bootstrap.
 
@@ -263,6 +267,9 @@ def create_app(
         # Process-wide Sentry tags. lite-bootstrap calls sentry_sdk.set_tags()
         # after init.
         sentry_tags={"queue_name": settings.queue_name},
+        # Branch computers lose ethernet and power; the spool keeps events on
+        # disk until Sentry confirms them and replays them after a restart.
+        sentry_additional_params={"transport": make_spool_transport(spool_dir)},
     )
     return FastStreamBootstrapper(config).bootstrap()
 

src/croupier/settings.py

@@ -8,6 +8,7 @@
 from pydantic import HttpUrl
 
 DEFAULT_SETTINGS_PATH = Path.home() / ".croupier.json"
+DEFAULT_SPOOL_DIR = Path.home() / ".croupier-sentry-spool"
 
 
 class Settings(BaseModel):