zedagent: app-instance config changes are dropped when the same config activates a baseOS, and are never retried

[eriknordmark]

Summary

When one EdgeDevConfig carries both a baseOS activation and app-instance changes, parseConfig applies the volume half and discards the app half. The discarded half is never retried on its own — it is only re-applied if the controller subsequently changes the config again. If the baseOS activation never completes, the app-instance changes in that config are lost indefinitely, and the device reports no error about them.

Where this bites hardest: the dropped app change can be the very thing the baseOS activation is waiting for, in which case the device deadlocks and the update can never proceed.

Mechanism

All references are origin/master at c71ee6314.

1. The app half is skipped, and the parse is not recorded as done.
pkg/pillar/cmd/zedagent/parseconfig.go:174-211 — volumes are parsed at line 182, then app instances are skipped and the function returns at line 197, before lastProcessedConfig is updated at line 211:

parseVolumeConfig(getconfigCtx, config)                       // :182 — applied
...
if ((source == fromController || source == fromLOC) && activateNewBaseOS) ||
    (getconfigCtx.configProcessingRV == skipConfigUpdate) {
    log.Noticef("parseConfig: Ignoring config as a new baseOS image is being activated")
    return skipConfigUpdate                                   // :197 — app half dropped
}
...
parseAppInstanceConfig(getconfigCtx, config)                  // :204 — never reached
...
getconfigCtx.lastProcessedConfig = getconfigCtx.lastReceivedConfig  // :211 — never reached

2. Nothing retries it. The next config fetch short-circuits on an unchanged hash, so parseConfig is not called again:

• pkg/pillar/cmd/zedagent/handleconfig.go:1246 — if hash == prevConfigHash { ... return false, nil, nil }
• pkg/pillar/cmd/zedagent/handleconfig.go:987-992 — if !changed { ... goto cfgReceived }, so inhaleDeviceConfig (and therefore parseConfig) is skipped entirely.

lastProcessedConfig is left behind, but nothing reads it to decide whether to re-parse. There is no timer, no retry, and no error surfaced to the controller for the dropped app-instance changes.

The deferral is only released by an unrelated controller-side edit: on the next changed config, parseBaseOS sees the same baseOS element hash and returns early with activateNewBaseOSFlag = false (parseconfig.go:309-312), so app instances are parsed then.

Why the asymmetry matters — the deadlock case

Because volumes are parsed but app instances are not, a config that removes both an app and its volume leaves the app behind, and the app's VolumeRefConfig keeps the volume alive:

volumemgr computes a volume's refcount as vcRefCount + vrcRefCount (pkg/pillar/cmd/volumemgr/handlevolume.go:389-415) and only tears the volume down at RefCount == 0 (maybeDeleteVolume, :306-309). Deleting the VolumeConfig drops vcRefCount to 0, but the surviving AppInstanceConfig keeps zedmanager publishing a VolumeRefConfig, so vrcRefCount stays 1 and the VolumeStatus is never deleted. gcVolumes will not reclaim it either — it only reaps on-disk volumes that have no published VolumeStatus (initialvolumestatus.go:174).

So any baseOS activation whose precondition is "no volumes" can never be satisfied: the app delete that would release the volume is in the half that was dropped, and the config hash never changes again.

On master today this is reachable through the cross-flavor refusal at pkg/pillar/cmd/baseosmgr/handlebaseos.go:222-233: a controller that pushes an EVE-k image to a kvm device (or the reverse) together with an app change gets the app change dropped, and since the activation is refused outright it is never re-applied. It becomes a hard deadlock in the in-flight kvm→k boot-disk-conversion work, where that refusal is being narrowed to "refuse only while volumes exist" — the release condition then is the dropped app delete.

Observed

Seen once in a 14-iteration kvm→k conversion soak (1 of 14), on a build carrying the volume-conditional form of the cross-flavor gate. Sequence, from the device's newlog:

time (UTC)	event
00:47:15	device reboots: Watchdog report for IMGB … : 3 /run/newlogd.pid
00:48:40	controller deletes the app instance (device is down, so it never sees this config)
00:49:36	on boot: Use backup config due to boot reason BootReasonNone → app re-created from the checkpoint, VolumeConfig re-created
00:49:46	controller pushes the kvm→k BaseOs update
00:49:56	device fetches one config containing the app delete and the baseOS activation → Volume config delete ×3 applied, then parseConfig: Ignoring config as a new baseOS image is being activated
00:49:57–00:50:42	baseosmgr refuses 151×: Upgrade to EVE-k (…-k-amd64) from non EVE-k (…-kvm-amd64) is not supported while volumes exist
00:50:42 → +44 min	nothing further. No Volume status delete in the entire capture; getVolumeStatusByLocation: found volume /persist/vault/volumes/<uuid>#0.container re-logged every ~5 min with no destroy; UsageFromStatus: Volume <uuid>#0 not found in VolumeConfigs confirms the VolumeConfig is gone while the VolumeStatus survives.

The device stayed healthy the whole time — controller reachable (/ping 200), vault unlocked, logging normally — it simply never acted on the update again. The app also stayed RUNNING despite being absent from the controller's config.

Suggested fixes

Any one of these breaks the trap; the first two are the smallest:

1. Do not skip app-instance deletions. Removals do not depend on the new baseOS, and they are what a "no apps/volumes" precondition is waiting for.
2. Make the deferral retryable. On skipConfigUpdate, reset prevConfigHash (or gate re-parsing on lastProcessedConfig != lastReceivedConfig rather than on the response hash) so the deferred half is retried on the next fetch instead of waiting for a controller edit.
3. Bound the deferral. If a baseOS activation is in an error state (BaseOsStatus carries an error), stop deferring app-instance processing — the activation is not going to complete.

Related, and worth deciding together: the parse currently applies parseVolumeConfig but not parseAppInstanceConfig, which is what splits an app+volume removal in half. Either both should be deferred or neither.

Environment

• Defect located in origin/master @ c71ee6314 (line numbers above).
• Reproduced incidentally on an EVE build carrying the in-flight kvm→k conversion branch; the zedagent and volumemgr code paths involved are unmodified from master.