[philipko100]

Summary

DeserializeEntries reads the entry count from the directory stream, then builds the result with make([]EntryV3, 0) + append. Since the slice starts at zero capacity, append reallocates and copies the backing array as it grows. For a full-planet leaf directory (~62k entries) that's 27 reallocations, each discarding the previous backing array as transient garbage.

The exact entry count is already known before the append loops, so this PR sizes the slice capacity to it up front: make([]EntryV3, 0, numEntries). The decode logic is otherwise unchanged.

Why it's safe

• Contents/length identical. Only the initial capacity hint changes; the varint reads and delta/run-length/offset math are untouched, so the returned []EntryV3 has the same elements, order, and len(). The slice still starts at length 0, so a truncated stream behaves exactly as before.
• No caller relies on cap() or appends into the returned slice. All call sites (server.go, show.go, extract.go, IterateEntries) consume the result read-only; the only append (extract.go) appends the result into another slice.
• No new trust in numEntries. The value already bounds the existing append loop, so this introduces no new assumption about untrusted input.

Benchmark

Decoding a realistic 62,746-entry leaf directory (gzip), allocations measured with -benchmem:

before:  8,849,368 B/op   216 allocs/op
after:   1,579,005 B/op   190 allocs/op

~82% fewer bytes allocated per decode, 26 fewer allocations, with no change in decode time. The win scales with directory size and matters most for servers that decode leaf directories on the hot path for scattered tile requests.

[bdon]

Have you measured if this makes a material difference?

[philipko100]

Hey Brandon, yes I did! By preallocating the slice to the known entry count instead of growing it, this eliminates the 27 reallocations it needs deserialize a 62,746-entry leaf directory page. This made our code reduce wasted allocation of memory from 8+ MB to 1.5 MB for deserializing a leaf. Since we know the slice size ahead of time (i.e. how large a leaf directory is), we can pre-allocate this memory ahead of time for optimization.

Since we run this DeserializeEntires function quite frequently when handling tiles at surges, we have been using this code in our team to help slow memory spikes and garbage collection pressure. I'd thought I would make a PR to add this to the go-pmtiles code to have a single source of truth for decoding PMTile entries.

Here's a code snippet you can run to validate this memory material difference:

package pmtiles
  
  import (
        "bytes"
        "testing"
  )

  // makeLeafEntries builds a directory resembling a real full-planet leaf: ~62k
  // entries with a mix of run-length-1 tiles and short runs, monotonically
  // increasing tile IDs, and contiguous offsets. Large enough that the un-presized
  // make([]EntryV3, 0) + append loop reallocates ~27 times.
  func makeLeafEntries(n int) []EntryV3 {
        entries := make([]EntryV3, 0, n)
        var offset uint64
        for i := 0; i < n; i++ {
                length := uint32(100 + i%900)
                runLength := uint32(1)
                if i%7 == 0 {
                        runLength = uint32(1 + i%4)
                }
                entries = append(entries, EntryV3{
                        TileID:    uint64(i) * 3,
                        Offset:    offset,
                        Length:    length,
                        RunLength: runLength,
                })
                offset += uint64(length)
        }
        return entries
  }

  func BenchmarkDeserializeEntries(b *testing.B) {
        entries := makeLeafEntries(62746)
        serialized := SerializeEntries(entries, Gzip)
        b.ResetTimer()
        b.ReportAllocs()
        for i := 0; i < b.N; i++ {
                DeserializeEntries(bytes.NewBuffer(serialized), Gzip)
        }
}

go test ./pmtiles/ -bench=DeserializeEntries -benchmem -run=^$ -count=5

[bdon]

Released here: https://github.com/protomaps/go-pmtiles/releases/tag/v1.31.1

Thanks!

[philipko100]

That's awesome, thanks Brandon!