SCALING.md

Stress, scaling and system limits

Generated by python -m tools.scaling against docker compose up: a gateway, three workers and the CV service, each capped at 512 MB, the limit of the four lab containers. CPUs are this machine's (12 logical, x86_64); the lab containers see 120, so the memory envelope here is the lab's and the CPU envelope is tighter. Raw rows are in docs/scaling_raw.jsonl.

Workload: closed loop, POST /analyzeArea through the gateway. 70% small selections (0.9-1.5 km side, one village) and 30% large (4-5 km side, ~20 km2), at random places inside the committed tile region. Each request has a distinct curve number, so none is answered from the result cache, and states its rainfall, so Open-Meteo's latency is not what is measured. Every analysis includes the resolution ensemble.

The occasional 422 in the errors column is not a failure. A randomly placed box sometimes holds no channel a pond can sit on (no_site_in_selection), or only ground beside a watercourse (no_ground_clear_of_watercourse), and saying so is the right answer. No request in the matrix failed for any other reason.

Scaling with workers

60 requests from 6 concurrent clients, least-busy dispatch.

workers	throughput	p50	p95	p99	mean small	mean large	errors	peak worker memory
1 worker	1.34 req/s	4.11 s	7.65 s	8.92 s	3.89 s	5.7 s	none	204 MiB
2 workers	2.10 req/s	2.60 s	4.50 s	4.96 s	2.41 s	3.48 s	none	220 MiB
3 workers	3.54 req/s	1.17 s	2.99 s	3.08 s	1.25 s	2.59 s	422 no_site_in_selection: 1	224 MiB

Three workers give 2.6x the throughput of one, and p95 falls from 7.7 s to 3.0 s. Each worker runs one analysis at a time (worker_slots=1), so the queue, not a worker, absorbs the burst.

Round-robin against least-busy

Same load, three workers, 40% large selections so the jobs are uneven. Round-robin is what the existing Go load balancer on sys1 does; least-busy is the gateway's dispatcher, which knows each worker's queue.

strategy	throughput	p50	p95	p99	mean small	mean large	errors	peak worker memory
round-robin	2.63 req/s	2.13 s	3.68 s	4.66 s	1.73 s	2.76 s	422 no_ground_clear_of_watercourse: 1	222 MiB
least-busy	2.62 req/s	2.08 s	3.34 s	4.11 s	1.59 s	2.89 s	none	239 MiB

least-busy wins on the tail, modestly: p95 3.34 s against 3.68 s (9% lower), p99 4.11 s against 4.66 s, with the same throughput. Round-robin hands a job to the next worker in turn even when it is busy with a large selection and another worker is idle; the job waits behind the large one. With one slot per worker, least-busy never queues a job while a worker is free. The gap is small here because three workers drain a six-client queue quickly; it grows with the spread of job sizes. Both ship: the Go balancer for the synchronous endpoint, the gateway for jobs, whose polling has to reach the worker that holds the job.

Concurrency

clients	throughput	p50	p95	p99	mean small	mean large	errors	peak worker memory
1 client	1.10 req/s	0.56 s	1.62 s	1.63 s	0.56 s	1.6 s	none	239 MiB
3 clients	3.29 req/s	0.56 s	2.11 s	2.12 s	0.56 s	1.84 s	422 no_ground_clear_of_watercourse: 1	232 MiB
6 clients	3.26 req/s	1.52 s	3.01 s	3.15 s	1.35 s	2.6 s	422 no_ground_clear_of_watercourse: 1	254 MiB
12 clients	3.21 req/s	3.53 s	5.19 s	5.73 s	3.01 s	4.41 s	422 no_site_in_selection: 1	270 MiB

Throughput stops rising once every worker is busy; beyond that, more clients only lengthen the queue, which shows up as latency and not as errors.

The result cache

	latency
First request for the sample area (computed)	2.07 s
The same area again, 19 times from 4 clients	p50 10 ms, p95 40 ms

A repeated selection with the same parameters is answered by the gateway without touching a worker. Elevation tiles are cached separately on each worker's disk and never expire; the demo region is committed with the repository (data/tiles).

What the limits look like

Situation	What the user gets	Time
A selection over 100 km2	422 aoi_too_large, naming the limit	25 ms
24 large selections at once, queue capped at 4	7 answered; 17 x 503 busy	p95 5.5 s
The largest selection tested, 93 km2	200, grid 17.82 m x [838, 787]	2.4 s

A refusal is immediate and says why: the queue limit turns overload into a 503 busy the client can retry, instead of requests that all time out together. A selection too large for the memory is a 422 aoi_too_large before any work starts, the same pattern Phase 2 used for ensemble_unavailable.

Memory

Peak worker memory over the whole matrix: 276 MiB of 512. Under sustained large selections: 276 MiB. The map path's grids are small (the sample sheet is 60,606 cells with its buffer against 638,472 on the contour path), which is why the resolution ensemble, switched off on the lab since Phase 11 because it did not fit, runs on every map-path request here. The adaptive resolution caps any selection at a million cells, the envelope the Phase 2 sheet proved fits.

On the lab's four containers

deploy/deploy.sh puts the gateway on sys1 (the Phase 2 URL, 5229) and a worker on each of sys2-4; deploy/start-lb.sh starts the existing Go balancer on sys1:6000 for the round-robin comparison. python -m tools.loadtest --url http://10.1.75.53:5229 --docker '' runs the same workload there, and deploy/status.sh reads each container's cgroup peak (memory.peak, and anon from memory.stat, never memory.current, which counts reclaimable page cache).