Repository navigation
fix(k8s): prune superseded deces indices after a prod restore - #113
Merged
Merged
Conversation
Every restore lands in a new deces-esdata_<dataprep>_<data> index and moves the deces alias, but nothing removed the index it replaced. On 2026-09-14 three full copies (45.5 GB) sat on the 59 GB volume, cut the page cache left for the live index and pushed iowait to 57-74 %; they were pruned by hand. The 2026-10-10 restore left one more (deces-esdata_44607e77_6c73c977). prune_stale_indices() deletes every deces-* index other than the target and whatever the alias still resolves to: - it runs right after the alias switch is verified and the marker written, and when the restore is skipped because the snapshot is already live, so a later release clears copies left by earlier restores - it prunes nothing when the alias cannot be read - it is best-effort: a failed delete is logged and does not fail the Job - every pruned index stays recoverable from its S3 snapshot prod-restore-prune.sh extracts the function from the rendered prod overlay with awk (no YAML parser) and runs it against a stubbed Elasticsearch in 7 scenarios; it fails if either call site goes missing.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Each prod restore creates
deces-esdata_<dataprep>_<data>and moves thedecesalias, but the superseded index was never deleted.deces-esdata_44607e77_6c73c977(15.2 GB) next to the live index.What
prune_stale_indices()inoverlays/prod/elasticsearch-restore.job.yamldeletes everydeces-*index except the restore target and any index thedecesalias still resolves to.switch_alias, the live count check andwrite_marker; and on the "snapshot already live, skipping restore" path, so the next release clears copies left by restores that predate this changeTests
deploy/k8s/tests/prod-restore-prune.sh(new, bash + awk, no Python/YAML parser): extracts the function from the rendered prod overlay and runs 7 scenarios against a stubbed Elasticsearch, plus checks both call sitesdeploy/k8s/tests/prod-manifest.shasserts the function ships; passes