[4.1] fix(fileservice): backport memory cache allocator

[VioletQwQ-0]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Related issues: #26992, #27567

What this PR does / why we need it:

This Draft PR backports the memory-cache backing-capacity accounting and the dedicated jemalloc allocator to 4.1-dev.

The change is based on upstream PR #26997 (allocator replacement) and #26783 (backing-capacity accounting), adapted to the exact 4.1-dev base 0d5195e5f93292939e2c5bca66a9746993315d25 (ad93dd48c66bbabc82db3078125b36d4b3e90aaa candidate). It adds:

• allocator-backed capacity accounting before cache admission;
• one isolated jemalloc arena per configured memory cache, with bounded pending-buffer reuse and explicit reclaim/close ownership;
• foreign-cache-data rehoming and reservation transfer on insert/reject/evict/close;
• service-scoped logical, physical/backing-overhead, and allocator statistics;
• the pinned jemalloc 5.3.1 source archive and static build verification.

The backport preserves the 4.1 FileService/objectio surface and does not import unrelated main-only DISK-V2, ONNX, CopyObject, or validation-pipeline changes. No SQL syntax, wire protocol, storage format, or default public configuration is changed.

This update also closes the two startup/remote-cache safety gaps identified during review:

• remote-cache responses are accepted only for indexes sent to that target, with nil, range, duplicate, and hit-payload-size checks; malformed responses fall back to local storage;
• malformed remote-cache requests are rejected before indexing or reading, and partial cache data is released on read failure;
• embedded FileService construction tracks created counters and closes all already-created services on later failure;
• embedded operator/cluster startup transfers ownership before service start and rolls back partially-created services, including retryable cleanup for dynamically added CN services.

Validation

• bash -n: not applicable (no shell source changed).
• GOWORK=off go build -mod=readonly ./...: PASS.
• GOWORK=off go vet -mod=readonly on all changed production packages: PASS.
• Targeted normal tests for pkg/common/malloc, pkg/fileservice/fifocache, pkg/fileservice, and pkg/objectio: PASS.
• CGo-linked targeted tests with the repository CGo wrapper: PASS.
• Remote-cache malformed-response/request and embedded FileService/cluster rollback tests: normal, CGo, and race paths PASS.
• Focused race stress: 36 individually named allocator/cache tests, -race -count=2, PASS; one full owning-package race run, PASS.
• Jemalloc source SHA-256: 3826bc80232f22ed5c4662f3034f799ca316e819103bdc7bb99018a421706f92, local static build PASS.
• BVT: N/A. Physical allocator arena ownership, backing capacity, native reclaim, and race transitions are not deterministically observable through SQL/BVT without conflating them with host memory state; the deterministic package/CGo/race tests above exercise the production paths instead.

The combined pkg/fileservice + pkg/embed package run remains affected by the existing TestDefaultTmpFileServiceUsesServiceDataDir baseline failure (the default TMP config has an empty data directory); the new tests and all affected production packages compile, vet, and pass their targeted normal/CGo/race runs.

QA is required before this Draft is made Ready. This PR does not deploy to mo-50 and does not claim the #26992 workload has been rerun on 4.1-dev.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →