Conversation
hysds 3.3.3 adds scripts/reap_orphaned_job_failed.py, a mozart daemon that deletes job_failed documents a later attempt has superseded. This config is written from scratch rather than rendered from hysds's template, so without this block the release-regression rig would never exercise the new component. It runs with --dry-run, as the upstream block does: the daemon deletes failure records, and a venue reconciles a dry-run sweep against its own audit before letting it delete. --lookback-days 1 matches the redis job-status TTL; the daemon refuses a longer window without --dry-run.
pymonger
marked this pull request as draft
September 2, 2026 20:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hysds 3.3.3 (hysds#223) adds
scripts/reap_orphaned_job_failed.py, a mozart daemon that deletesjob_faileddocuments a later attempt has superseded. This repo writes its mozart supervisord config from scratch rather than rendering hysds's template, so without a block here the release-regression rig never runs the new component.Held as a draft, deliberately. As it stands this PR would add a program that cannot work, and merging it would give a green run that proves nothing:
HYSDS_VERSION=v6.1.2in.env.example(hysds 3.1.1).install.shuses that to pull the images and locate the bundle, and the daemon ships only from 3.3.3, so supervisord would exhauststartretriesand park the program FATAL on every install.indexer.confhas no filter block, so every job doc goes to a concretejob_status-currentindex: there are no datedjob_status-2*indices, nojob_failed, and no job-failed move. The reaper's candidate pattern would match nothing and every sweep would reportscanned: 0, meeting the success criterion vacuously.What this needs before it lands
A retrofit of the rig, not just this block: the indexer's job-failed move and its paired delete, a
job_status-currentalias spanningjob_failedand the dailies, and a 3.3.3 bundle pin. Until then the block stays out, and hysds#223 says plainly that the rig does not get it in this release.One thing the un-retrofitted rig is good for
That empty shape is exactly the state hysds#223's guard reports as ABSENT: with neither home present, every probe is a clean miss and every mozart supervisory writer declines to write. That was confirmed on a dev venue before any job indices existed:
Worth running deliberately once, as the cheapest demonstration that the ABSENT routing behaves as designed.
Change
One supervisord program block beside
watchdog_job_timeouts, matching the upstream one:--lookback-days 1, matching the redis job-status TTL, and no--delete-orphans, so the daemon reports what it would delete and deletes nothing.Test plan
supervisorctl status reap_orphaned_job_failedis RUNNING withsweep summarylines in its log, and that a manufactured orphan is reported.