Skip to content

Fix large extract pmtiles from large source file - #289

Open
smounir wants to merge 1 commit into
protomaps:mainfrom
Mappy:bugfix/fix-extract-from-large-source-file
Open

smounir wants to merge 1 commit into
protomaps:mainfrom
Mappy:bugfix/fix-extract-from-large-source-file

Conversation

@smounir

@smounir smounir commented Oct 6, 2026

Copy link
Copy Markdown

Fix OOM in extract when the source is a local file

Problem

FileBucket.NewRangeReaderEtag allocated a []byte of the full requested range length and filled it with ReadAt. This makes extract allocate the entire merged range in memory.

With --overfetch=0 on a contiguous zoom level, mergeRanges coalesces adjacent tiles into very large ranges. On a 2.9 TB source (--minzoom=18 --maxzoom=18), one of the 254 requests was ~218 GB, so makeslice failed:

fatal error: runtime: out of memory
...
pmtiles.FileBucket.NewRangeReaderEtag  bucket.go:136
pmtiles.Extract.func2                  extract.go:538

GOMEMLIMIT doesn't help, because this is a single allocation. The HTTP and cloud bucket implementations already stream the response body, so only FileBucket was affected.

Fix

FileBucket.NewRangeReaderEtag now returns an io.SectionReader over the open file, wrapped in a ReadCloser whose Close() closes the file. Memory use no longer depends on the range size.

  • I also removed the defer file.Close(). It was registered before the error check and would have closed the file before any streaming read.
  • Behavior is otherwise unchanged:
    • 404 if the file is missing.
    • 412 if the etag doesn't match.
    • 206 otherwise.
    • A range running past EOF is truncated, as before.
  • The old 416 branch ("Expected to read N bytes...") was unreachable, since ReadAt always returns an error when it reads fewer bytes than requested. It's removed.

Tests

Added TestFileBucketRangeReader, which covers:

  • a range in the middle of the file
  • a range truncated at EOF
  • an offset past EOF
  • a missing key
  • a stale etag (412)

@smounir
smounir force-pushed the bugfix/fix-extract-from-large-source-file branch from de22fa9 to 39bcb45 Compare October 6, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant