Skip to content

Comment sync reads the attachment index and stops listing R2 (#934 phases 3-4) #955

Description

@zachdunn

Summary

Follow-up to #934. The R2 Class A spike is resolved (#935 scoped the fan-out to the target's prefixes, and Class A operations have been flat since 2026-09-02). Phases 1 and 2 of the attachment-index plan are live: github_attachments is dual-written from every write path (#938, #945) and the shadow compare runs behind the attachment-index-shadow flag (#940, report script in #948).

The comment sync still lists R2, one or two ListObjects per sync. This issue tracks finishing the plan so it reads only D1. Low priority: cost is no longer the driver, this is for zero-R2 syncs and for retiring the shadow flag rather than leaving it running with no owner.

Remaining work

  • Phase 3: read from the index behind a flag. gatherAttachments (apps/api/src/github-comment.ts) renders from listAttachmentsForTarget (apps/api/src/github-attachment-index.ts) when the flag is on, with list-on-miss fallback. Lane-aware URLs from lane_id via objectPublicUrls; poster lane from the row instead of createLaneResolver. Tests asserting zero listObjects calls under the flag.
  • Backfill. D1-only pass from file_metadata gh.ref rows (source='backfill'), then an offline R2 walk (apps/api/scripts/backfill-attachment-index.mjs, modelled on backfill-gh-metadata.mjs) for anything the metadata pass missed.
  • Reconcile job. reconcileAttachmentIndex scoped to one target: diff R2 against the index, repair, emit a drift metric. Hook into the explicit resync path and the post-rotation resync.
  • Phase 4: delete the fan-out once shadow diffs have been clean for a full release cycle. Keep listActivePrefixIds for rotation and the CLI's local-gh fallback.
  • Retire the shadow flag and the shadow module after phase 4.
  • Optional: a GET /:workspace/github/attachments endpoint served from the index so the CLI's local-gh fallback stops fanning out client-side.

Before starting phase 3

Run node scripts/attachment-index-shadow-report.mjs --hours 72 (runbook in docs/ops.md) and confirm matches across more than one write path. As of 2026-09-06 the shadow is 100% match with no missing or extra rows, but every attachment-bearing sync came from a single private repo, so plain-prefix (public repo) targets, private-prefix rotation, and pre-#938 targets are still unverified. Workers Logs only reach back about three days, so windows longer than 72h return less, not more.

Full design, DDL, and write-path inventory are in the scoping comment on #934.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions