Add a measured raw-XDR extraction boundary - #3
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The backfill path is moving toward raw ledger XDR as its handoff between the archive source and extraction.
go-stellar-sdkv0.7.1 gives us zero-copy XDR views, but exposing those experimental SDK types throughout every consumer would couple the entire pipeline to their lifetime and API stability.This change creates a narrow boundary in
stellar-extract: callers can hand us raw XDR, stable callers can keep usingLedgerInput, and individual table families can migrate to borrowed views only when measurement shows a real win.Design
LedgerViewInputowns the extraction contract, but not the input bytes. Its views borrow the caller's XDR buffer, so extraction must finish before the upstream ledger stream advances or reuses that buffer. Typed output rows do not retain the borrowed bytes.The complete
ExtractAllFromXDRpath deliberately performs one parsed decode today. Contract events are the first view-native family throughExtractContractEventsView, using one cachedLedgerTransactionViewRangewalk and decoding only event payloads retained in output rows.This is an explicit split rather than a transparent replacement:
LedgerInputremains the stable parsed API.LedgerViewInputmakes borrowed lifetimes visible at the call site.ExtractAllFromXDRremoves JSON and protobuf from the file-backfill boundary without claiming the complete surface is zero-copy.A rejected version
The first implementation replaced contract events inside the complete
ExtractAllresult while every other table still used the parsed ledger. That paid for both representations. On a real mainnet ledger it made complete extraction about 13% slower and increased allocations.This PR does not ship that path. The complete surface stays on one parsed decode, and the view-native contract-event API is available explicitly. Once enough related families migrate,
ExtractAllViewcan stop materializing the parsed graph.ExtractLedgerTxPartswas also considered for contract events. It omits diagnostic events, and V3 event gating requires the paired envelope.LedgerTransactionViewRangepreserves diagnostic, operation, and transaction-level semantics in one API. The parts API remains a candidate when transactions and fee-derived tables migrate together.Evidence
The differential gate uses mainnet ledger 58,752,000 from the SDK fixture corpus:
LedgerDatafamilyCreatedAtOn an AMD Ryzen 9 7940HS, contract events from raw XDR changed from roughly 11.6-11.8 ms and 14.90 MB allocated to 8.4-8.6 ms and 8.98 MB allocated. That is approximately 28% less time, 40% fewer allocated bytes, and 34% fewer allocations. The complete direct-XDR entry point remains within noise of the existing manual decode plus
ExtractAllsequence.Verification
go test ./...go test -race ./...go vet ./...go build ./...go mod tidywith no module-file driftgit diff --checkFollow-up
The next integration PR consumes this boundary from the bounded backfill pipeline. Transactions and operations are the next likely view migrations, followed by change-reader-backed state tables. Each migration must improve the complete path rather than only its isolated microbenchmark.