Support local snapshot database in benchmark runs - #1509
Open
msuozzo wants to merge 7 commits into
Open
Conversation
In order to provide better support and interoperability with local-first workflows, this SQLite data layer provides flexible downstream query and storage without requiring changes to the existing firestore schema and usage. Updates can be streamed in periodically to ensure freshness and full snapshots are made ~daily which ensures accuracy.
RepoMetrics.MeasuredAt previously stamped the record write time, which overstates freshness. Inference clones through the git cache with no freshness bound, so the measured contents can long predate the measurement. CloneFunc is made to return gitx.Repo, a git.Repository plus its clone-time provenance, so every cloner reports the fetch time uniformly.
Attempts and scratch execs are mutable (status transitions, timings, costs), so incremental snapshot replay needs a clock recording the last write, not the first. Every write site stamps Updated alongside its other mutations. Repo metrics gain Updated next to MeasuredAt, which keeps its fetch-time meaning while Updated carries the write clock. The attempts range scans are collection-group queries and need single-field index exemptions. The other collections are top-level, where Firestore's automatic single-field indexes already serve updated scans. forge-parent: pknmrsotsovq
Scan Firestore into docdb doc tables (attempts, runs, agent sessions and iterations, scratch VMs and execs, repo metrics) and write the result as a single rebuild-v<N>.db, gzip-encoded where the destination supports it. The registry in tables.go is the entire snapshot schema: entity columns are extractions over the source documents (the strategy oneof labels itself via json_each) so no row structs or Go projection code exist anywhere. The database meta watermark records the scan start so incremental replay can resume from it. It will run as a scheduled job periodically (daily?). The object name carries the schema version, so a version bump ensures new consumers will require deployment instead of getting unexpected data. And the schema safety is enforced with a golden test which pins the materialized column shapes (generated columns included) against a committed golden per SchemaVersion. Additive changes may regenerate it in-place with -update, while removals, renames, retypes, and key changes fail until the version is bumped, and -update itself refuses to rewrite an incompatible golden. In terms of mapping safety, a saturation test stores a fully populated instance of every source type and asserts every column reads non-NULL with its declared storage type, so an extraction path whose identity departs from its struct (a renamed field, a newly added json tag) fails rather than silently reading as empty data. forge-parent: yzlrnvktsvvw forge-parent: krxuzqzswono
Delta writes one docdb segment of every document written since the newest segment already exported, recovering its resume point from the segment name so the delta writer itself is stateless. This allows a crashed or skipped run to be repaired by the following run. All collection groups gain the requisite single-field indices for their updated range scans. Segments live under a schema-versioned prefix (deltas-v<N>/), so a reader only ever replays segments matching its base's era. forge-parent: pnkpvutxolvu
Implement Reader and SessionReader over the snapshot. Notably, we reuse the shared client-side pipeline (pending filter, verdict cleaning, latest-per-package, limits) so semantics match the Firestore reader. Access is serialized with Querier so the cache can swap the database between calls. forge-parent: motxuwqtlvtt
run-bench -local -db <PATH> writes its run and verdicts into a snapshot database on disk instead of the per-attempt file tree, creating the file on first use, and rederives the trend tables when the run finishes. The result is the same artifact the rollup publishes, so every snapshot consumer (the dashboard sqlite rundex, ctl snapshot) reads it unchanged, and because writes go through the guarded upserts, re-runs and replayed upstream delta segments compose with local results safely. A database from another schema era is refused: rebuilding from raw documents is the only migration. Cross-process writers coordinate through SQLite file locking. forge-parent: ykunzvmpqqpk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run-bench -local -db writes its run and verdicts into a snapshot
database on disk instead of the per-attempt file tree, creating the file
on first use, and rederives the trend tables when the run finishes. The
result is the same artifact the rollup publishes, so every snapshot
consumer (the dashboard sqlite rundex, ctl snapshot) reads it unchanged,
and because writes go through the guarded upserts, re-runs and replayed
upstream delta segments compose with local results safely. A database
from another schema era is refused: rebuilding from raw documents is the
only migration. Cross-process writers coordinate through SQLite file
locking.