Support for deployments larger than 2GB

[kriszyp]

Problem

Payload-based deploy_component is capped at ~2 GB. The CLI (bin/cliOperations.js) calls packageDirectory(), which returns the tar+gzip as a Buffer, CBOR-encodes the whole request, and POSTs it. The server (server/operationsServer.ts, body limit REQ_MAX_BODY_SIZE ~1 GB) reads the body whole before dispatch. Node.js's hard 2 GB Buffer ceiling is the absolute upper bound, and the configured body limit is lower in practice.

Replication compounds the problem: server.replication.replicateOperation() relays operations as single payloads over the inter-node WebSocket. Multi-GB payloads exceed both buffer and frame-size limits there as well.

Plan

1. CLI — stream the package as multipart/form-data

bin/cliOperations.js switches deploy_component from CBOR-encoded body to multipart/form-data:

• One field part per operation property (operation, project, package, flags, etc.) serialized as JSON.
• One file part carrying the tar+gzip payload, piped from a streaming variant of packageDirectory(). No intermediate Buffer.
• Sent via Node https with Transfer-Encoding: chunked — no Content-Length precomputation.

Touches bin/cliOperations.js, components/packageComponent.ts (expose streaming variant), and utility/common_utils.js (multipart-aware httpRequest).

2. Operations API server — receive multipart and stream into extraction

server/operationsServer.ts accepts a multipart body when Content-Type: multipart/form-data:

• Field parts are decoded into the same req object shape the JSON/CBOR path produces.
• The file part is exposed as a Readable and passed straight into deployComponent, replacing req.payload: Buffer with req.payload: Readable in this path.
• The existing extraction pipeline in components/Application.ts:217 (gunzip-maybe → tar-fs.extract) is already stream-native — it accepts the file part directly, eliminating the in-memory hop.
• REQ_MAX_BODY_SIZE is bypassed on the multipart deploy path (the limit only meaningfully constrains the small field parts; the file part is streamed and bounded by disk only).

Touches server/operationsServer.ts, server/serverHelpers/serverHandlers.js, server/serverHelpers/serverUtilities.ts, components/operations.js, components/Application.ts.

3. Replication — direct HTTPS relay instead of WebSocket-framed operation

WebSocket-framed replication cannot carry multi-GB payloads. Replicated deploy_component (and restart_service, for symmetry) switch to a direct-relay scheme:

1. Origin completes local receive: streams the file part to a temporary tarball on disk while extraction runs (or pipes through a PassThrough to a temp file in parallel). This temp file is the source for all peer relays so the CLI uploads once.
2. Origin calls create_authentication_tokens over the existing replication channel to mint a short-lived (~5 min TTL) operation JWT, scoped to the specific operations needed on each peer (deploy_component, restart_service).
3. Origin opens a direct HTTPS connection to each peer's operations API (default port 9925) using that JWT and streams the same multipart/form-data request the CLI sent. Peers process it as a normal local deploy with replication suppressed (no re-fanout).
4. Per-peer semantics: peer failures do not abort other peers. Origin retries each peer a small number of times on transient transport errors (connection reset, TLS handshake, 502/503) but does not retry on application errors (npm install failure, validation, auth). Final result is a per-peer status grid surfaced to the CLI via SSE (Improved deployment visibility #526); the CLI exits non-zero if any peer ultimately failed.
5. Temp tarball is deleted on completion (success or failure).

Open questions / things to nail down during implementation:

• JWT scope mechanics — does create_authentication_tokens already support per-operation scoping at sufficient granularity, or do we need a new scope dimension? Audit this before relying on it.
• Peer addressing — reuse the replication peer configuration (host/port/TLS settings) rather than rediscovering nodes. Confirm the operations API port and TLS posture are consistently configured across pro deployments.
• Coexistence with mTLS-proxied operations APIs: the JWT path must work where peers expect client certs too.

Touches the replication module under replication/ (in harper-pro) and server/Server.ts's replication interface — sendOperationToNode() currently throws ''Replication not implemented'' (see server/Server.ts:96); this work fills it in for the deploy/restart case.

4. Backward compatibility

The existing CBOR/JSON request path stays unchanged for small payloads and external operations API clients. Multipart is opt-in on the CLI side and selected automatically for deploy_component (we may keep CBOR for tiny payloads, or just unify on multipart — to be decided when measuring overhead).

Files touched (approximate)

• bin/cliOperations.js, components/packageComponent.ts, utility/common_utils.js
• server/operationsServer.ts, server/serverHelpers/serverHandlers.js, server/serverHelpers/serverUtilities.ts
• components/operations.js, components/Application.ts
• server/Server.ts and the replication module (harper-pro replication/)
• New helper: a small multipart parser / formatter shared by CLI and server (likely under utility/)

Related

• Improved deployment visibility #526 — progress reporting / visibility. The SSE channel defined there is what surfaces per-peer relay status to the CLI for this design.

[kriszyp]

Appears to be a duplicate of / consolidate under #641 — 2GB payload cap — folded into the deployment-tracking redesign.

— Claude (Opus 4.8), issue-backlog triage