Skip to content

fix(k8s): free stale Elasticsearch rollback before restore - #107

Merged
rhanka merged 1 commit into
mainfrom
fix/prod-elasticsearch-restore-space
Sep 13, 2026
Merged

rhanka merged 1 commit into
mainfrom
fix/prod-elasticsearch-restore-space

Conversation

@rhanka

@rhanka rhanka commented Sep 13, 2026

Copy link
Copy Markdown
Member

The new monthly snapshot was created successfully, but production Elasticsearch refused to allocate its 15.3 GB restore shard because two prior 15 GB generations filled the 45 GiB PVC beyond the 90% disk watermark. The restore job now recovers from an incomplete red target, preserves the live alias target, removes only older unreferenced restore generations before a real restore, and retains all S3 snapshots.

The monthly wrapper timeout is raised from 10 to 125 minutes so it can wait for the child release-prod job, whose own timeout is 120 minutes.

Validation:

  • production evidence: live alias resolved to the August index; only the incomplete target and July rollback are selected for deletion
  • embedded restore script passes POSIX sh -n
  • manifest parses and passes kubectl apply --dry-run=client
  • prod overlay passes kubectl kustomize
  • git diff --check
  • AGY Gemini 3.8 high second-pass review: PASS, no blocker/high/medium findings

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: f9085856-7683-452e-9589-bf90fa9b5f6e


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@rhanka
rhanka merged commit c5fe36a into main Sep 13, 2026
11 checks passed
@rhanka
rhanka deleted the fix/prod-elasticsearch-restore-space branch September 13, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants