feat: preserve full Note Tweets across sync and rendering

[eferm]

Problem

Birdclaw currently stores Xurl's top-level text for every tweet. For a Note Tweet, that field is only a compatibility preview; the complete body and its entities live under note_tweet.

This truncates Note Tweets in the database, full-text search, exports, profile analysis, research, replies, quotes, and retweets.

New feature	Result
Full Note Tweet ingestion	Stores note_tweet.text and its matching entities
Full-text indexing	Makes text beyond the compatibility preview searchable
Higher-fidelity retention	Prevents later preview-only payloads from replacing a stored full body
Portable storage	Preserves Note Tweets through backup export, import, and merge
Read-model propagation	Carries complete Note Tweets through timelines, conversations, replies, quotes, and retweets
X-style presentation	Shows a 280-code-point preview with a "Show more" control

Change

Xurl tweet requests now include note_tweet. A shared content helper chooses the Note Tweet body and entities when present.

The canonical representation is stored as:

• tweets.text: complete Note Tweet body
• tweets.entities_json: entities belonging to the complete body
• tweets.note_tweet_json: durable marker containing the Note Tweet body and entities
• tweets_fts: indexed from the canonical stored text

The marker lets ingestion distinguish a complete Note Tweet from a later payload that carries only the top-level preview. Archive and backup merges use the same rule: the richer marked representation wins.

Read models propagate the marker through primary timeline items, replies, quotes, retweets, and conversations. Backup schema version 9 carries it through export, replace, and merge.

New readers continue importing schemas 1–8. Birdclaw 0.12.1 and older reject v9 backups, so every installation sharing a backup repository must be upgraded before its first v9 export or sync.

The shared rich-text renderer uses that marker to reproduce X's browsing behavior:

• preview the first 280 Unicode code points
• finish at a word boundary
• show "Show more" only when text was collapsed
• reveal the complete body when clicked
• render only entities that fall inside the visible preview
• preserve entity offsets around emoji and Twitter Article links

Scope

Xurl's note_tweet field is the source of the Note Tweet marker.

Archive imports continue to use archive tweet text. When an archive row meets an Xurl-marked Note Tweet, the marked full body remains canonical.

Tweet edit histories continue through the existing revision path.

Ordinary tweets continue rendering their complete stored text. Note Tweets use the shared collapsed renderer everywhere TimelineCard, EmbeddedTweetCard, or ConversationThread is used.

Real behavior proof

This was exercised against a personal Likes database populated through Xurl.

The following read-only query compares the retained raw Xurl payload with the canonical tweet row and FTS entry:

$ sqlite3 -json ~/.birdclaw/birdclaw.sqlite "
with note_rows as (
  select
    t.id,
    t.text as stored_text,
    json_extract(tc.raw_json, '$.text') as preview_text,
    json_extract(tc.raw_json, '$.note_tweet.text') as full_text
  from tweet_collections tc
  join tweets t on t.id = tc.tweet_id
  where tc.kind = 'likes'
    and json_type(tc.raw_json, '$.note_tweet.text') = 'text'
)
select
  count(*) as xurl_note_tweets,
  sum(stored_text = full_text) as stored_full_text,
  sum(length(full_text) > length(preview_text))
    as full_text_longer_than_preview,
  sum(exists(
    select 1
    from tweets_fts f
    where f.tweet_id = note_rows.id
      and f.text = note_rows.full_text
  )) as fts_contains_full_text
from note_rows;
"

{
  "xurl_note_tweets": 1511,
  "stored_full_text": 1511,
  "full_text_longer_than_preview": 1425,
  "fts_contains_full_text": 1511
}

All 1,511 raw Xurl Note Tweets use the complete note_tweet.text as their canonical stored and indexed body. In 1,425 cases, that body is longer than Xurl's top-level preview.

Tests

$ bun --no-env-file run --bun check
# format, lint, and TypeScript passed

$ node ./scripts/run-vitest.mjs run \
    src/lib/tweet-repository.test.ts \
    src/lib/query-models.test.ts \
    src/lib/backup.test.ts
# 146 tests passed

$ node ./scripts/run-vitest.mjs run src/components/TweetRichText.test.tsx
# 10 tests passed

$ git diff --check
# passed

Backend coverage verifies:

• an ordinary preview is upgraded when note_tweet arrives
• a later preview-only payload retains the complete body, entities, marker, and FTS entry
• archive merges preserve an existing marked Note Tweet
• backup replace and merge preserve the richer representation and rebuild FTS correctly
• timeline and conversation models propagate Note Tweets through replies, quotes, and retweets
• Xurl requests include the note_tweet field

Frontend coverage verifies:

• long Note Tweets collapse at a word boundary
• text beyond the preview remains hidden while collapsed
• "Show more" reveals the complete body
• the control disappears after expansion

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: found issues before merge. Reviewed September 1, 2026, 7:39 AM ET / 11:39 UTC.

ClawSweeper review

What this changes

The PR stores full X Note Tweet text and entities through local ingestion, search, backups, read models, exports, and an expandable 280-code-point presentation.

Merge readiness

⚠️ Needs maintainer review before merge - 4 items remain

Keep open: the Note Tweet retention work has credible real-data proof, but its introduced schema-v9 writer breaks existing shared-backup sync with v0.12.1 readers unless maintainers choose and implement a compatibility or coordinated-migration contract.

Priority: P1
Reviewed head: 5beae12b3cdbaec02450c1ae347b96cda530738a
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The core retention result has strong real-data evidence, but the introduced shared-backup format break is a merge blocker.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The changed ingestion and FTS owners in the PR are exercised by an after-fix query against a personal Xurl-populated database, with all 1,511 observed Note Tweets retaining and indexing their full body; that proof does not resolve the separate upgrade-compatibility failure.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The changed ingestion and FTS owners in the PR are exercised by an after-fix query against a personal Xurl-populated database, with all 1,511 observed Note Tweets retaining and indexing their full body; that proof does not resolve the separate upgrade-compatibility failure.
Evidence reviewed	6 items	Introduced schema writer: The PR changes the backup manifest writer from schema v8 to v9. Existing reader rejects v9: The v0.12.1-era backup reader accepts manifests only through its declared schema version, which was v8; a v9 manifest fails before import rows are read. Current-main provenance: Fetched current main descends from the PR base and has no changes to the backup reader, backup codecs, or backup documentation, so the v8 reader behavior remains current-main behavior outside this PR.
Findings	1 actionable finding	[P1] Preserve cross-version backup sync before writing schema v9
Security	None	None.

How this fits together

Birdclaw fetches tweet data from Xurl, stores and indexes it locally, then exposes it through timeline views, exports, and Git-backed backups. This PR changes the canonical tweet representation and the backup interchange format consumed by other Birdclaw installations.

flowchart LR
  A[Xurl tweet payload] --> B[Canonical tweet storage]
  B --> C[Full-text search]
  B --> D[Timeline and exports]
  B --> E[Git backup]
  E --> F[Other Birdclaw installations]
  D --> G[Expandable tweet rendering]

Loading

Decision needed

Question	Recommendation
Should shared Birdclaw backups remain readable by v0.12.1 installations, or should the first schema-v9 write require an explicit coordinated migration gate?	Preserve shared-backup compatibility: Keep compatible backup output or defer schema-v9 writing until all supported readers can safely consume it, with mixed-version sync coverage.

Why: The PR documents a simultaneous-upgrade requirement but does not prevent an upgraded device from making an existing shared repository unreadable to another supported installation.

Before merge

• Preserve cross-version backup sync before writing schema v9 (P1) - The new writer emits schema v9, but the current v0.12.1 reader rejects manifests above v8 before any import or merge. One upgraded device therefore makes a shared backup repository unusable by an older installation; retain a compatible contract or add an intentional, tested migration gate before changing the writer.
• Resolve merge risk (P1) - After one upgraded installation exports schema v9, a v0.12.1 installation sharing that backup repository rejects the manifest before it can import or sync, interrupting an existing multi-machine workflow.
• Complete next step (P2) - A maintainer must select the permanent backup compatibility and migration policy before a safe repair path can be defined.

Findings

• [P1] Preserve cross-version backup sync before writing schema v9 — src/lib/backup.ts:49

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production versus test delta	production +362/-138, tests +362/-30, docs +6	The feature spans persistent storage, backup interchange, read models, and UI, so the shared-backup compatibility boundary needs deliberate review.

Merge-risk options

Maintainer options:

1. Keep existing installations interoperable (recommended)
   Revise the backup contract so an installation on v0.12.1 does not fail after another installation processes Note Tweets, and add mixed-version import or sync coverage.
2. Approve a coordinated schema migration
   Explicitly accept the multi-device upgrade requirement only after adding a preflight gate and recovery behavior that prevents accidental shared-repository lockout.
3. Pause the schema-changing portion
   Land no schema-v9 writer until maintainers select a supported backup upgrade policy.

Technical review

Best possible solution:

Adopt a maintainer-approved backup upgrade contract that either preserves supported shared-repository compatibility or visibly gates a coordinated migration with recovery, then add mixed-version sync coverage.

Do we have a high-confidence way to reproduce the issue?

Yes—source proves the deterministic path: this PR writes schema v9 while the unchanged v0.12.1 reader rejects every manifest above v8 before importing rows.

Is this the best way to solve the issue?

No—the retention implementation is broadly coherent, but a documentation-only simultaneous-upgrade instruction is not a safe replacement for an explicit shared-backup compatibility or migration contract.

Full review comments:

• [P1] Preserve cross-version backup sync before writing schema v9 — src/lib/backup.ts:49
  The new writer emits schema v9, but the current v0.12.1 reader rejects manifests above v8 before any import or merge. One upgraded device therefore makes a shared backup repository unusable by an older installation; retain a compatible contract or add an intentional, tested migration gate before changing the writer.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 658b979c7e93.

Labels

Label justifications:

• P1: An existing multi-machine backup workflow stops when an older installation encounters the PR's v9 manifest.
• merge-risk: 🚨 compatibility: The introduced backup schema writer exceeds the version accepted by currently supported v0.12.1 readers.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The changed ingestion and FTS owners in the PR are exercised by an after-fix query against a personal Xurl-populated database, with all 1,511 observed Note Tweets retaining and indexing their full body; that proof does not resolve the separate upgrade-compatibility failure.
• proof: sufficient: Contributor real behavior proof is sufficient. The changed ingestion and FTS owners in the PR are exercised by an after-fix query against a personal Xurl-populated database, with all 1,511 observed Note Tweets retaining and indexing their full body; that proof does not resolve the separate upgrade-compatibility failure.

Evidence

What I checked:

• Introduced schema writer: The PR changes the backup manifest writer from schema v8 to v9. (src/lib/backup.ts:49, 5beae12b3cdb)
• Existing reader rejects v9: The v0.12.1-era backup reader accepts manifests only through its declared schema version, which was v8; a v9 manifest fails before import rows are read. (src/lib/backup.ts:2998, d64bcdf6b054)
• Current-main provenance: Fetched current main descends from the PR base and has no changes to the backup reader, backup codecs, or backup documentation, so the v8 reader behavior remains current-main behavior outside this PR. (src/lib/backup.ts:2998, 658b979c7e93)
• Prior blocker remains unresolved: The prior completed ClawSweeper review identified the same schema-v9 cross-version sync break at this exact head; the current head is unchanged. (src/lib/backup.ts:49, 5beae12b3cdb)
• Real retention-path proof: The PR body records an after-fix query against a personal Xurl-populated database: all 1,511 observed Note Tweets matched their full stored and FTS-indexed body, including 1,425 longer than the compatibility preview. (src/lib/tweet-repository.ts:152, 5beae12b3cdb)
• Likely area owner history: Recent merged backup and ingestion refactors were authored by Peter Steinberger, including backup-merge reconciliation and consolidated tweet ingestion. (src/lib/backup-table-codecs.ts:495, 1353329ec9c2)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Choose and implement the shared v8/v9 backup contract.
• Add mixed-version import or sync coverage for that contract.
• Rebase onto current main and refresh review after the compatibility decision.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (19 earlier review cycles; latest 8 shown)

• reviewed 2026-08-29T21:34:55.920Z sha 5beae12 :: needs maintainer review before merge. :: none
• reviewed 2026-08-30T01:20:07.223Z sha 5beae12 :: needs maintainer review before merge. :: none
• reviewed 2026-08-30T04:44:46.294Z sha 5beae12 :: needs maintainer review before merge. :: none
• reviewed 2026-08-30T08:06:50.732Z sha 5beae12 :: found issues before merge. :: [P1] Preserve cross-version backup sync before writing schema v9
• reviewed 2026-08-30T15:55:38.565Z sha 5beae12 :: found issues before merge. :: [P1] Preserve cross-version backup sync before writing schema v9
• reviewed 2026-08-31T10:07:37.348Z sha 5beae12 :: found issues before merge. :: [P1] Preserve cross-version backup sync before writing schema v9
• reviewed 2026-08-31T19:04:17.203Z sha 5beae12 :: found issues before merge. :: [P1] Preserve cross-version backup sync before writing schema v9
• reviewed 2026-09-01T10:04:29.604Z sha 5beae12 :: found issues before merge. :: [P1] Preserve cross-version backup sync before writing schema v9