improve: speed up checking channels for changed messages

[hannesrudolph]

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

Agents and other integrations pulling incremental data from an archive need to ask, "Which messages in this channel changed since my last check?" That includes older messages edited or otherwise updated later. On channels with substantial history, finding the latest update or retrieving a small batch of changed messages can become unnecessarily expensive, slowing repeated polling as the archive grows.

Why This Change Was Made

Add a covering index on messages(channel_id, updated_at, id) so per-channel updated-time queries can use the required ordering without a temporary sort. Ensure it exists through the existing initialization and index-maintenance paths, including when reopening an already-versioned archive, while retaining the existing indexes and schema version.

User Impact

Agents and read-only archive consumers can find the latest update cursor and retrieve changed-message batches with less database work. The existing channel/creation-time index continues to support reads ordered by when messages were originally posted; the additional index supports reads ordered by when archived records were last updated.

Existing records and command behavior are preserved. On an existing archive, the first writable open that runs index maintenance will build the index once. That work can delay the command, hold up other writers, and use additional disk space; the index also has ongoing storage and message-write costs, including for installations that do not use these queries.

Evidence

The query-plan test verifies that an incremental channel query uses the new index and does not require USE TEMP B-TREE. An existing-archive regression removes the index from a populated current-schema database, reopens it, and verifies index creation while preserving the message, queued embedding job, and schema version.

A synthetic in-memory SQLite 3.51.0 fixture used 100,000 rows, including 80,000 in the queried channel. Median read timings across 15 repetitions were:

Query	Before	After
Next 500 records after an update cursor	4.2263 ms	2.1856 ms
Latest update cursor in a channel	8.7049 ms	0.0010 ms

Both query plans stopped using a temporary sort. The additional index occupied 2,797,568 bytes in that fixture. These are synthetic read measurements, not production latency or storage estimates; large-archive index-build time and write-throughput impact have not yet been measured.

Validation:

• go test -count=1 ./internal/store passed.
• go vet ./internal/store passed.
• Formatting and git diff --check passed.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge. Reviewed September 10, 2026, 4:15 PM ET / 20:15 UTC.

ClawSweeper review

What this changes

Adds a SQLite index that speeds per-channel changed-message queries, with tests for fresh databases and existing archive upgrades.

Merge readiness

⛔ Blocked before merge - 3 items remain

The optimization remains useful and is absent from main and v0.14.1. No concrete correctness defect was found; the outstanding choice is whether to accept the documented, unmeasured large-archive upgrade cost.

Likely related people: steipete and vkehfdl1 are unverified routing candidates, with low confidence.

Priority: P2
Reviewed head: f1b5c8cf575f9aee59c5c2f7623309135b130651
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused optimization with fresh-schema and upgrade-preservation coverage and no identified correctness defect.
Proof confidence	🌊 off-meta tidepool	Not applicable: The MEMBER-authored PR is exempt from the ordinary contributor proof gate. Store initialization and cursor planning have supplemental tests and synthetic timing evidence; large persisted-archive upgrade costs remain a separate maintainer risk decision.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The MEMBER-authored PR is exempt from the ordinary contributor proof gate. Store initialization and cursor planning have supplemental tests and synthetic timing evidence; large persisted-archive upgrade costs remain a separate maintainer risk decision.
Evidence reviewed	8 items	Verified introduced change: The verified test merge has the pinned main and PR head as its two parents. Its production delta adds only the two index declarations; existing indexes and schema version remain intact. Current main still lacks the index: Main's query-index maintenance creates creation-time indexes but no messages(channel_id, updated_at, id) index. The existing SQL interface already supports arbitrary read-only archive queries, so this improves an existing access path without adding a competing API. Latest release check: The v0.14.1 schema and index-maintenance declarations likewise lack the proposed update-cursor index.
Findings	None	None.
Security	None	None.

How this fits together

Discrawl stores archived Discord messages in SQLite for CLI queries and external consumers. The added index orders each channel’s records by update time and message ID, helping consumers retrieve incremental changes.

flowchart LR
  A[Archived messages] --> B[SQLite archive]
  C[Writable archive open] --> D{Index exists?}
  D -->|No| E[Build update-time index]
  D -->|Yes| B
  E --> B
  F[Channel and update cursor] --> B
  B --> G[Ordered changed messages]

Loading

Decision needed

Question	Recommendation
Should this index land with its documented but unmeasured large-archive build and write costs?	Measure archive upgrade costs: Record build duration, disk growth, and write impact on a representative persisted archive before accepting the synchronous initialization cost.

Why: The data-preservation tests support structural compatibility, but synthetic read timings do not establish an acceptable operational cost for existing large archives.

Before merge

• Resolve merge risk (P1) - The first writable open builds the index synchronously across an existing archive, potentially delaying commands and competing writers; representative build duration, disk headroom, and ongoing write overhead remain unmeasured.
• Complete next step (P2) - Resolve the synchronous index-build tradeoff through representative archive measurements or explicit maintainer acceptance of the documented unmeasured cost.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	production +2, tests +59; 0 removed	The production growth is limited to one index declared in both existing initialization paths and is justified by the cursor-query optimization.

Merge-risk options

Maintainer options:

1. Measure the first writable open (recommended)
   Use a representative persisted archive to quantify the synchronous build and subsequent message-write cost.
2. Accept the initialization tradeoff
   A maintainer can explicitly accept the documented one-time writer delay and continuing storage overhead.

Technical review

Best possible solution:

Retain the additive index and existing query contracts, with representative archive-upgrade costs measured or explicitly accepted before landing.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a correctness-bug reproduction: this is a performance optimization supported by source inspection and submitted synthetic timings; no runtime benchmark was executed during review.

Is this the best way to solve the issue?

Yes, the additive covering index follows the existing maintenance pattern and preserves creation-time queries; the remaining question is its operational cost on large archives.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against 5ef4faad55d0.

Labels

Label changes:

• add P2: This is a bounded archive-query performance improvement without evidence of an urgent user-facing regression.
• add merge-risk: 🚨 availability: Building the new index during the first writable open can delay commands and other writers on large archives.
• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The MEMBER-authored PR is exempt from the ordinary contributor proof gate. Store initialization and cursor planning have supplemental tests and synthetic timing evidence; large persisted-archive upgrade costs remain a separate maintainer risk decision.

Label justifications:

• P2: This is a bounded archive-query performance improvement without evidence of an urgent user-facing regression.
• merge-risk: 🚨 availability: Building the new index during the first writable open can delay commands and other writers on large archives.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The MEMBER-authored PR is exempt from the ordinary contributor proof gate. Store initialization and cursor planning have supplemental tests and synthetic timing evidence; large persisted-archive upgrade costs remain a separate maintainer risk decision.

Evidence

What I checked:

• Verified introduced change: The verified test merge has the pinned main and PR head as its two parents. Its production delta adds only the two index declarations; existing indexes and schema version remain intact. (internal/store/store.go:540, e2e5e79e56d5)
• Current main still lacks the index: Main's query-index maintenance creates creation-time indexes but no messages(channel_id, updated_at, id) index. The existing SQL interface already supports arbitrary read-only archive queries, so this improves an existing access path without adding a competing API. (internal/store/store.go:703, 5ef4faad55d0)
• Latest release check: The v0.14.1 schema and index-maintenance declarations likewise lack the proposed update-cursor index. (internal/store/store.go:703, 2aef26b2df8a)
• Upgrade coverage: Writable opening runs query-index maintenance even for the current schema version. The new regression reopens a populated archive without the index and checks index creation, unchanged schema version, retained message content, and the pending embedding job. Fresh-schema and legacy-schema tests also cover index creation. Tests were inspected, not executed in this read-only review. (internal/store/updated_cursor_index_test.go:10, f1b5c8cf575f)
• Submitted performance evidence and limits: The captured PR body reports a synthetic SQLite 3.51.0 fixture with 100,000 rows, faster cursor reads, removal of temporary sorting, and 2,797,568 bytes of additional index storage. It explicitly acknowledges that large-archive build time and write-throughput impact remain unmeasured. This is synthetic evidence, not a production upgrade trace. (f1b5c8cf575f)
• Existing index-maintenance precedent: Current-main history and GitHub commit metadata identify steipete as the author of an earlier addition of the embedding-job ordering index to the same baseline and maintenance paths. This supports routing relevance, not attribution for this PR. (internal/store/store.go, 146e72773ab8)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• vkehfdl1: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Quantify build duration, disk growth, and write overhead on a representative persisted archive to resolve the operational uncertainty.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.