[tadamcz]

In the bloom68 production runs, live sandbox pods kept being evicted and recreated mid-sample (k8s_sandbox PodReplacedError, "Pod ... has been replaced"), in bursts of 5 to 9 unrelated samples within minutes. One burst coincided to the second with a runner pod evicted for "ephemeral local storage usage exceeds the total limit of containers 20Gi". The sandbox pods had a memory limit and no ephemeral-storage request at all.

Why they were evicted

Under node disk pressure the kubelet ranks pods first by whether their ephemeral-storage usage exceeds their request (pkg/kubelet/eviction/helpers.go). With no request, any usage at all exceeds zero, so every sandbox pod sat at the front of the eviction queue regardless of how little it had written. Usage is the container's writable layer plus logs and emptyDir volumes; pulled image layers do not count against the pod, only against the node.

The change

get_values_file_content now gives both services a request-only ephemeral-storage reservation, in the chart-native values file that Hawk feeds to the agent-env chart:

Service	Request
default (agent)	20Gi
comparator	8Gi

Request only, no limit, as the inspect_k8s_sandbox docs describe for a scheduling floor ("no eviction cap"): a limit would evict a pod for writing more, which is the failure this fixes. The memory limit still defaults its own request, so memory scheduling is unchanged. The compose file is untouched; Compose has no disk-request concept. Verified by rendering the values through the METR fork of the chart at the revision Hawk's runner pins (helm template): both containers come out with the intended limits and requests.

Sizing

Measured on the local images at the 67338a15 pin (writable layer via docker ps -s):

Container	Action	Writable layer
agent	lake build Submission.Spec importing Mathlib	5 MB
comparator	reset script plus a full accepting check	90 MB

Mathlib (7 GB) and the corpus (13 GB) are image layers and /tmp stays empty, so steady-state usage is megabytes. The requests are sized for the tails, doubled for margin: the agent is root and can reach the Mathlib tree through the .lake/packages symlink, so a copy-up rebuild is roughly 7 GB on top of the 100 MB submission cap and any Sage or Julia caches; the comparator's build is Landlock-confined to .lake, so its tail is a large solution's oleans plus the two lean4export closures of a Mathlib-heavy proof, hundreds of MB.

Node capacity

Hawk's standard CPU pools have a 200Gi data volume by default, about 180Gi allocatable. Memory already caps packing at seven 32Gi pods on the largest node type, so seven agent pods reserve 140Gi, which fits. If Kueue admission is enabled on the cluster, the sandboxes queue needs an ephemeral-storage quota or these workloads will sit queued (Hawk's Kueue docs call this out).

Not in this PR

Node disk pressure itself. Images share the node filesystem with writable layers, and a node serving several FC pins can carry well over 100 GB of images; the request protects the pods in the eviction ranking but does not stop pressure from image churn. Raising karpenterNodeDataVolumeSize in the Hawk stack would.