Skip to content

Run hysds 3.3.3's orphan reaper on mozart, dry-run - #1

Draft
pymonger wants to merge 1 commit into
mainfrom
HC-648-reaper-block
Draft

pymonger wants to merge 1 commit into
mainfrom
HC-648-reaper-block

Conversation

@pymonger

@pymonger pymonger commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

hysds 3.3.3 (hysds#223) adds scripts/reap_orphaned_job_failed.py, a mozart daemon that deletes job_failed documents a later attempt has superseded. This repo writes its mozart supervisord config from scratch rather than rendering hysds's template, so without a block here the release-regression rig never runs the new component.

Held as a draft, deliberately. As it stands this PR would add a program that cannot work, and merging it would give a green run that proves nothing:

  • The rig pins HYSDS_VERSION=v6.1.2 in .env.example (hysds 3.1.1). install.sh uses that to pull the images and locate the bundle, and the daemon ships only from 3.3.3, so supervisord would exhaust startretries and park the program FATAL on every install.
  • On a 3.3.3 bundle it gets worse rather than better. The rig's indexer.conf has no filter block, so every job doc goes to a concrete job_status-current index: there are no dated job_status-2* indices, no job_failed, and no job-failed move. The reaper's candidate pattern would match nothing and every sweep would report scanned: 0, meeting the success criterion vacuously.

What this needs before it lands

A retrofit of the rig, not just this block: the indexer's job-failed move and its paired delete, a job_status-current alias spanning job_failed and the dailies, and a 3.3.3 bundle pin. Until then the block stays out, and hysds#223 says plainly that the rig does not get it in this release.

One thing the un-retrofitted rig is good for

That empty shape is exactly the state hysds#223's guard reports as ABSENT: with neither home present, every probe is a clean miss and every mozart supervisory writer declines to write. That was confirmed on a dev venue before any job indices existed:

a payload nobody has written        -> absent
an index name that is not a daily   -> absent
no index given at all               -> absent

Worth running deliberately once, as the cheapest demonstration that the ABSENT routing behaves as designed.

Change

One supervisord program block beside watchdog_job_timeouts, matching the upstream one: --lookback-days 1, matching the redis job-status TTL, and no --delete-orphans, so the daemon reports what it would delete and deletes nothing.

Test plan

  • Retrofit the rig as above, then confirm supervisorctl status reap_orphaned_job_failed is RUNNING with sweep summary lines in its log, and that a manufactured orphan is reported.

hysds 3.3.3 adds scripts/reap_orphaned_job_failed.py, a mozart daemon that
deletes job_failed documents a later attempt has superseded. This config is
written from scratch rather than rendered from hysds's template, so without
this block the release-regression rig would never exercise the new
component.

It runs with --dry-run, as the upstream block does: the daemon deletes
failure records, and a venue reconciles a dry-run sweep against its own audit
before letting it delete. --lookback-days 1 matches the redis job-status TTL;
the daemon refuses a longer window without --dry-run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant