[BUG] Retry re-marks host for build and invalidates in-flight PXE token #758

Description

A retried or overlapping move/rebuild can invalidate the provisioning token of a host that is already PXE booting. The host then loops on 401 Unauthorized against /unattended/provision until it is powered off and boots PXE again.

Root cause

Foreman.prepare_host_provisioning() calls mark_for_build() before writing the OS/medium/ptable:

• src/quads/tools/external/foreman.py:615
• mark_for_build() unconditionally PUTs {"host": {"build": 1}} (foreman.py:444-446).

Foreman generates a new provisioning token whenever the host is marked for build. The running installer does not re-read the Foreman host token; it retries the kickstart URL embedded in the PXE template from its boot. A second mark after that boot rotates the token and makes the installer's URL stale.

There is also an ordering window on Dell/HPE:

• src/quads/plugins/builtin/release/standard.py:192 powers the host on before prepare_host_provisioning() at standard.py:207.
• If a previous attempt left a one-time PXE boot target set, the host can boot and fetch its PXE template before the new attempt marks it for build.
• mark_for_build() then rotates the token the host just received.

There is no guard that skips build=1 when the host is already marked for build, and the failure path clears QUADS' build flag but not Foreman's build state/token.

Observed sequence

From Foreman production logs (hostname, IPs, and tokens omitted):

12:20:13  PUT /api/v2/hosts/<id> {"host":{"build":1}}
12:20:24  GET /unattended/provision?token=<prior-generation> -> 401
12:21:42  PUT /api/v2/hosts/<id> {"host":{"build":1}}

The host was already using a token from an earlier generation when a later move attempt marked it for build again. Retrying Anaconda cannot recover because it keeps requesting the same URL.

Expected behavior

A retry must not invalidate the token of a host that is already provisioning. Marking an already-building host should be a no-op, or QUADS must guarantee the host is not booting before rotating the token.

Suggested fix

• Make mark_for_build() idempotent: read the host's current build state and return without a PUT when build is already true.
• Avoid powering on a host with a pending one-time PXE target before Foreman build state and parameters are finalized.
• Keep token generation as close as possible to the reboot that will consume it, and serialize per-host rebuilds.
• Add a regression test that asserts a second prepare_host_provisioning()/mark_for_build() call does not PUT build=1 again when the host is already building.

Impact

A failed or delayed boot followed by a retry leaves the host stuck in the installer with an invalid token. The current recovery is manual: mark the host for build again and power cycle it so it fetches a fresh PXE template.