Repository navigation
fix(k8s): cut prod Elasticsearch heap to 4g - #110
Merged
Merged
Conversation
On the prod node (4 vCPU / 16 GiB, ES cgroup limited to 13Gi), a 6g heap left only ~6.8 GB of page cache for a 15.2 GB index. The node ran at ~74% iowait and the ES search thread pool saturated; API p95 was several seconds. Measured old-gen usage was 224 MB with zero full GCs, so 6g was heavily oversized. Cutting to 4g frees ~2 GB back into page cache. Measured after applying in prod: iowait 74% -> 5%, idle 1.6% -> 75%, search pool queue drained to 0, /search median 1 ms and p95 289 ms over 1016 real requests. The change was already applied live to the prod StatefulSet via kubectl patch; this commit brings git in line with the running cluster. deploy/k8s/tests/prod-manifest.sh pins the rendered value so the heap cannot silently drift back.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On the prod node (4 vCPU / 16 GiB, ES cgroup limited to 13Gi), a 6g heap left only ~6.8 GB of page cache for a 15.2 GB index. The node ran at ~74% iowait, the ES search thread pool saturated, and API p95 was several seconds.
Measured old-gen usage was 224 MB with zero full GCs, so 6g was heavily oversized. Cutting to 4g frees ~2 GB back into page cache.
Measured effect in prod
/searchmedian/searchp95Measured over 1016 real requests.
Note
The change was already applied live to the prod StatefulSet via
kubectl patch. This commit brings git in line with the running cluster.Changes
deploy/k8s/overlays/prod/elasticsearch.prod.yaml:ES_JAVA_OPTS-Xms6g -Xmx6g->-Xms4g -Xmx4gdeploy/k8s/tests/prod-manifest.sh: pin the rendered value so the heap cannot silently drift backTests
deploy/k8s/tests/prod-manifest.sh— passdeploy/k8s/tests/prod-secrets-preflight.sh— passdeploy/k8s/tests/prod-publication-guardrails.sh— fails identically on pristinemain(dataprep-monthly must scan every 4 hours); pre-existing and unrelated to this change