Add a full database docdb SQLite database - #1502
Open
msuozzo wants to merge 2 commits into
Open
Conversation
msuozzo
force-pushed
the
push-pnkpvutxolvu
branch
from
August 28, 2026 19:52
d5cf0bc to
feed925
Compare
In order to provide better support and interoperability with local-first workflows, this SQLite data layer provides flexible downstream query and storage without requiring changes to the existing firestore schema and usage. Updates can be streamed in periodically to ensure freshness and full snapshots are made ~daily which ensures accuracy.
Scan Firestore into docdb doc tables (attempts, runs, agent sessions and iterations, scratch VMs and execs, repo metrics) and write the result as a single rebuild-v<N>.db, gzip-encoded where the destination supports it. The registry in tables.go is the entire snapshot schema: entity columns are extractions over the source documents (the strategy oneof labels itself via json_each) so no row structs or Go projection code exist anywhere. The database meta watermark records the scan start so incremental replay can resume from it. It will run as a scheduled job periodically (daily?). The object name carries the schema version, so a version bump ensures new consumers will require deployment instead of getting unexpected data. And the schema safety is enforced with a golden test which pins the materialized column shapes (generated columns included) against a committed golden per SchemaVersion. Additive changes may regenerate it in-place with -update, while removals, renames, retypes, and key changes fail until the version is bumped, and -update itself refuses to rewrite an incompatible golden. In terms of mapping safety, a saturation test stores a fully populated instance of every source type and asserts every column reads non-NULL with its declared storage type, so an extraction path whose identity departs from its struct (a renamed field, a newly added json tag) fails rather than silently reading as empty data. forge-parent: krxuzqzswono
msuozzo
force-pushed
the
push-pnkpvutxolvu
branch
from
August 31, 2026 16:05
feed925 to
224417c
Compare
wbxyz
approved these changes
Sep 1, 2026
| agentLogsBucket = flag.String("agent-logs-bucket", "", "GCS bucket for agent build logs") | ||
| agentTimeoutSeconds = flag.Int("agent-timeout-seconds", 3600, "Seconds to allow agent to run") | ||
| rebuildJobName = flag.String("rebuild-job-name", "", "Name of the pre-created Cloud Run Job for rebuilds") | ||
| analyticsURI = flag.String("analytics-uri", "", "URI for snapshots (supported schemea: gs, file). Empty disables /snapshot/rollup") |
Member
There was a problem hiding this comment.
Does this need to be set in terraform? And does the orchestrator need to be granted access?
| // Rollup scans the source and writes the snapshot database to the configured | ||
| // analytics destination, replacing the published object wholesale. | ||
| func Rollup(ctx context.Context, _ RollupRequest, deps *RollupDeps) (*RollupResponse, error) { | ||
| if deps.Dest == nil { |
Member
There was a problem hiding this comment.
Also check for deps.Source being nil?
wbxyz
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scan Firestore into docdb doc tables (attempts, runs, agent sessions
and iterations, scratch VMs and execs, repo metrics) and write the
result as a single rebuild-v.db, gzip-encoded where the destination
supports it. The registry in tables.go is the entire snapshot schema:
entity columns are extractions over the source documents (the strategy
oneof labels itself via json_each) so no row structs or Go
projection code exist anywhere. The database meta watermark records the
scan start so incremental replay can resume from it. It will run as a
scheduled job periodically (daily?).
The object name carries the schema version, so a version bump ensures
new consumers will require deployment instead of getting unexpected
data. And the schema safety is enforced with a golden test which pins
the materialized column shapes (generated columns included) against a
committed golden per SchemaVersion. Additive changes may regenerate it
in-place with -update, while removals, renames, retypes, and key changes
fail until the version is bumped, and -update itself refuses to rewrite
an incompatible golden.
In terms of mapping safety, a saturation test stores a fully populated
instance of every source type and asserts every column reads non-NULL
with its declared storage type, so an extraction path whose identity
departs from its struct (a renamed field, a newly added json tag) fails
rather than silently reading as empty data.