feat(cdc): native SQLite CDC source (db.cdc.sqlite)#351
Open
skhaz wants to merge 2 commits into
Open
Conversation
Why: The runtime could stream row changes from Postgres (db.cdc.postgres) but had no equivalent for SQLite. SQLite has no logical-replication slot, so the native mechanism is the preupdate hook, which observes row changes on the connection the runtime writes through. What: Adds a db.cdc.sqlite registry kind backed by a supervised Source that installs SQLite preupdate + commit/rollback hooks on the target db.sql.sqlite pool's writer connection and emits insert/update/delete (plus a gap-free snapshot bootstrap) through the existing, engine-agnostic cdc Lua module (cdc.stream/list_sources/source). How: - service/sql: a build-tagged hook seam (sqlite_preupdate_hook) registers a ConnectHook-enabled driver and installs/clears hooks on a raw *sqlite3.SQLiteConn; the factory selects the driver transparently. - service/cdc/sqlite: the Source buffers preupdate rows per transaction and flushes atomically on commit via a bounded handoff to a drain goroutine. Column names/affinity resolve over a dedicated read-only connection, and checkpoints write through a separate plain-driver connection, so the writer-blocking commit hook can never deadlock against schema resolution or checkpoint writes. A laggard subscriber is closed rather than allowed to stall the writer. Durable snapshot/offset state lives in wippy_cdc_offsets in the source DB. - api/service/cdc: db.cdc.sqlite kind, SQLiteConfig, and a composite inspector/streamer so the Lua module observes both engines. - boot: kind-specific listeners (db.cdc.postgres + db.cdc.sqlite) feed the composite. Capture is in-process and live-only: changes made while the runtime is down, or by an external writer, are not captured. The checkpoint exists for snapshot-gating and idempotent dedupe, not replay. Building requires the sqlite_preupdate_hook tag (added to the Makefile); without it the source fails loudly instead of silently capturing nothing.
Why: PR #351 bolted SQLite CDC specifics onto the generalized service/sql driver (a build-tagged preupdate-hook file, a mutable driver-name global, and CDC-only errors), and the package dispatched engines through kind switches. Adding a database therefore meant editing core dispatch code, and the generalized driver carried engine- and CDC-specific knowledge it should not have. What: service/sql core now exposes only two public seams, RegisterEngine and RegisterDriver, and dispatches purely via engineFor(kind); the manager, factory, and ConnPool.UpdateConfig no longer switch on engine kind. An EngineConfig contract lets the generic create/update lifecycle validate and read lifecycle settings without knowing the concrete type. Built-in engines move into self-registering sub-packages that use only the public API: engine/standard (Postgres and MySQL, each with its own DSN builder, removing the buildDSN/getDriver kind switch), engine/sqlite (DSN, WAL, single-writer tuning), and engine/all (blank-imports the built-ins). Boot blank-imports engine/all as the single wiring point. The database/sql driver override is applied centrally in createPool, so engines stay override-agnostic. All SQLite CDC hook code (custom sqlite3_wippy driver, sink registry, preupdate scan, install/clear-on-raw, CDC errors) moves to service/cdc/sqlite/hook.go and registers its driver through RegisterDriver, leaving service/sql with zero CDC knowledge. Adding a database is now a new self-registering package that touches no existing core file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The runtime streams Postgres row changes via
db.cdc.postgres(#323) but had no SQLite equivalent. SQLite has no logical-replication slot, so the native mechanism is the preupdate hook, which observes row changes on the connection the runtime writes through.What
Adds a
db.cdc.sqliteregistry kind backed by a supervisedSourcethat installs SQLite preupdate + commit/rollback hooks on the targetdb.sql.sqlitepool's writer connection and emitsinsert/update/delete(plus a gap-freesnapshotbootstrap) through the existing, engine-agnosticcdcLua module (cdc.stream/list_sources/source).service/sql: build-tagged hook seam (sqlite_preupdate_hook) registering aConnectHook-enabled driver; the factory selects it transparently (1-line change).service/cdc/sqlite: preupdate rows buffered per-transaction, flushed atomically on commit via a bounded handoff to a drain goroutine. Column names/affinity resolve over a dedicated read-only connection and checkpoints write through a separate plain-driver connection, so the writer-blocking commit hook can never deadlock against schema resolution or checkpoints. A laggard subscriber is closed loudly rather than allowed to stall the writer. Durable snapshot/offset state lives inwippy_cdc_offsetsin the source DB.api/service/cdc:db.cdc.sqlitekind,SQLiteConfig, and a composite inspector/streamer so the Lua module observes both engines.boot: kind-specific listeners (db.cdc.postgres+db.cdc.sqlite) feed the composite.Limitation (by design)
Capture is in-process and live-only: changes made while the runtime is down, or by an external process writing the file, are not captured (unlike a Postgres slot that replays). The checkpoint exists for snapshot-gating and idempotent dedupe, not replay. This is the trade for zero schema intrusion and lowest overhead.
Setup
Build with the new tag (already wired into the Makefile):
make build-wippy-local. Withoutsqlite_preupdate_hookthe source fails loudly instead of silently capturing nothing.Registry usage:
Lua:
local s = cdc.stream("app:cdc"); local c = s:channel():receive()->{ op, table, before, after, source, lsn }.Testing
golangci-lint(tagsrace,sqlite_preupdate_hook): 0 issues.-race): decode/affinity, subscribers, manager, composite, config; and against a real WAL SQLite file: insert/update/delete, value fidelity (blob/text/NULL), rollback-discard, snapshot bootstrap, restart-keeps-checkpoint, table allowlist, and laggard-subscriber-does-not-stall-writes.mode=roon WAL).cdc.stream-> Lua channel):