Golden re-take during a roll records a terminal error and is never retried #292

Problem

A control-plane upgrade that re-takes golden snapshots (a golden recorded in an older snapshot layout, #282) can leave every ActorTemplate with a terminal golden error when the re-take runs while the worker pool and the atelet DaemonSet are still rolling.

Observed on a 1.7.0 → 1.8.1 step with 42 ActorTemplates: the new ate-api and ate-controller started within ~12 s and the reconciler immediately resumed all 42 golden actors (atespace ate-golden) for the re-take. All 42 went CRASHED within 8 s, and GoldenSnapshotStatus.error_message now reads:

• 41× GoldenActorCrashed: actor ate-golden/<uid> crashed: resume failed: assigned worker is draining
• 1× GoldenActorCrashed: … ateom.RunWorkload … ateom.sock: connect: no such file or directory

During the roll ate-api logged about 500 Failed to reconcile actor template … CallAteletRestore … errors (Unavailable, selected atelet … has no assigned IPs, found 2 atelet pods on node) and 42 Crashing actor without freeing its worker. Since then it has been quiet: resync skips a template whose error_message is set (goldenSnapshotDone), reconcileOne returns at once on it, and the Control service has no way to clear it (no UpdateActorTemplate, no re-take request). Deleting and re-creating the template would be a reset.

Impact today is latent: the recorded 1.7-layout golden tags are kept, and new actors still resume from them through the restore-time conversion (#277). But the goldens are never re-taken in the current layout, and nothing retries them. Every Session and actor snapshot is intact. The same step on two other clusters re-took every golden; one of them retried 9 transient no assigned IPs errors within 2 s.

Proposed solution

1. A golden actor crash caused by the roll is transient: a draining or gone worker, a worker pod or ateom that went away, an atelet without assigned IPs or with two pods on one node, an ateom socket not there yet, an Unavailable atelet. The reconciler reverts the crashed golden actor, which tears down and frees its worker, and retries the take with the work queue's backoff. Such a crash is never recorded as a terminal golden error.
2. A golden error recorded by an earlier release for such a transient crash is not terminal. The reconciler clears it and takes the golden again on its next pass, so rolling out the fix heals the affected templates without a reset. The recorded older-layout tag stays the golden until the new one is complete.
3. The pull request says whether the control plane should wait for worker-pool readiness before a re-take after an upgrade.

Acceptance criteria

• Unit tests cover the transient classification, the revert and retry, and the healing of a recorded transient error.
• A non-transient crash (an image the registry refuses, a container with no runnable process) still fails the template.
• A template with a recorded transient GoldenActorCrashed error is re-taken once the fixed control plane runs, with no reset and no snapshot deleted except the replaced golden tag.