Skip to content

Add a measured raw-XDR extraction boundary - #3

Merged
tmosleyIII merged 1 commit into
mainfrom
feature/raw-view-extraction
Aug 5, 2026
Merged

Add a measured raw-XDR extraction boundary#3
tmosleyIII merged 1 commit into
mainfrom
feature/raw-view-extraction

Conversation

@tmosleyIII

Copy link
Copy Markdown
Contributor

Context

The backfill path is moving toward raw ledger XDR as its handoff between the archive source and extraction. go-stellar-sdk v0.7.1 gives us zero-copy XDR views, but exposing those experimental SDK types throughout every consumer would couple the entire pipeline to their lifetime and API stability.

This change creates a narrow boundary in stellar-extract: callers can hand us raw XDR, stable callers can keep using LedgerInput, and individual table families can migrate to borrowed views only when measurement shows a real win.

Design

LedgerViewInput owns the extraction contract, but not the input bytes. Its views borrow the caller's XDR buffer, so extraction must finish before the upstream ledger stream advances or reuses that buffer. Typed output rows do not retain the borrowed bytes.

The complete ExtractAllFromXDR path deliberately performs one parsed decode today. Contract events are the first view-native family through ExtractContractEventsView, using one cached LedgerTransactionViewRange walk and decoding only event payloads retained in output rows.

This is an explicit split rather than a transparent replacement:

  • LedgerInput remains the stable parsed API.
  • LedgerViewInput makes borrowed lifetimes visible at the call site.
  • ExtractAllFromXDR removes JSON and protobuf from the file-backfill boundary without claiming the complete surface is zero-copy.
  • table families move only after exact differential parity and a full-surface performance win.

A rejected version

The first implementation replaced contract events inside the complete ExtractAll result while every other table still used the parsed ledger. That paid for both representations. On a real mainnet ledger it made complete extraction about 13% slower and increased allocations.

This PR does not ship that path. The complete surface stays on one parsed decode, and the view-native contract-event API is available explicitly. Once enough related families migrate, ExtractAllView can stop materializing the parsed graph.

ExtractLedgerTxParts was also considered for contract events. It omits diagnostic events, and V3 event gating requires the paired envelope. LedgerTransactionViewRange preserves diagnostic, operation, and transaction-level semantics in one API. The parts API remains a candidate when transactions and fee-derived tables migrate together.

Evidence

The differential gate uses mainnet ledger 58,752,000 from the SDK fixture corpus:

  • 1,278,080 bytes
  • 249 transactions
  • 2,229 contract-event rows
  • exact extractor-error parity
  • exact row counts for every LedgerData family
  • exact contract-event field parity after normalizing the intentionally nondeterministic CreatedAt

On an AMD Ryzen 9 7940HS, contract events from raw XDR changed from roughly 11.6-11.8 ms and 14.90 MB allocated to 8.4-8.6 ms and 8.98 MB allocated. That is approximately 28% less time, 40% fewer allocated bytes, and 34% fewer allocations. The complete direct-XDR entry point remains within noise of the existing manual decode plus ExtractAll sequence.

Verification

  • go test ./...
  • go test -race ./...
  • go vet ./...
  • go build ./...
  • real-ledger differential parity under the race detector
  • real-ledger benchmarks with three repeated samples
  • go mod tidy with no module-file drift
  • git diff --check

Follow-up

The next integration PR consumes this boundary from the bounded backfill pipeline. Transactions and operations are the next likely view migrations, followed by change-reader-backed state tables. Each migration must improve the complete path rather than only its isolated microbenchmark.

@tmosleyIII
tmosleyIII merged commit 5e8c77e into main Aug 5, 2026
2 checks passed
@tmosleyIII
tmosleyIII deleted the feature/raw-view-extraction branch August 5, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant