No unique soft serial when provisioning from a cloned live image

[eriknordmark]

Describe the unexpected behaviour

A device provisioned by cloning a live image never gets a unique soft serial number. Depending on how the live image was produced it ends up either with no soft serial at all, or with a soft serial that is identical on every device cloned from that image.

Only the installer creates one. pkg/installer/install:566-570 takes eve_soft_serial= off the kernel command line (this is what the iPXE flow uses, see pkg/eve/installer/ipxe.efi.cfg:17), falls back to an existing /config/soft_serial, and otherwise generates a fresh UUID. When the installer is skipped, nothing on the device ever creates the file:

• The in-repo live build target produces config.img straight from conf/ (Makefile → tools/makeconfig.sh → pkg/mkconf/make-config). Nothing in that path generates a serial, and conf/ ships only the onboarding certificate/key, server and root-certificate.pem. So /config/soft_serial is simply absent. (.gitignore:24 reserves conf/soft_serial as a manual drop-in, which is the only supported workaround today.)
• docker run lfedge/eve live (pkg/eve/runme.sh:154-165) does generate one with uuidgen — but it writes it into the image with mcopy … ::/soft_serial. Every device cloned from that image therefore reports the same soft serial. This is arguably the worse of the two outcomes, because it looks provisioned but is not unique.
• Nothing in pillar or any other on-device component ever writes the file. hardware.GetSoftSerial() (pkg/pillar/hardware/model.go:171) only reads it, and getOverride() returns "" when it is missing.

Impact

/config/soft_serial is one of the two serial numbers presented in the register API call, and the whole point of it is to avoid depending on guessable hardware serial numbers — see docs/SECURITY-ARCHITECTURE.md and docs/DEPLOYMENT.md ("Unique identification: serial vs. soft serial numbers").

With a cloned live image, client selfRegister sends SoftSerial: "" (pkg/pillar/cmd/client/client.go:603-616), and every subsequent controller request carries an empty DevSoftSerial header (cmd/zedagent/handleconfig.go, cmd/loguploader, cmd/diag, conntester/controller.go, cmd/scepclient). So:

• Onboarding falls back entirely on the hardware serial, which is often absent, non-unique or guessable — precisely the case the soft serial exists to cover. Cloned live images also share one onboarding certificate, so the serial is the only discriminator left.
• In the runme.sh case a fleet of devices registers with one shared soft serial, which is worse than an empty one: it is non-unique but looks like a valid device identifier, and the controller cannot tell the devices apart by it.

Expected behavior

A device booting EVE for the first time from a cloned live image should end up with a unique /config/soft_serial, generated before the device certificate is created and before client selfRegister runs, and the value should be recoverable by the operator (the installer prints it and deposits it on the INVENTORY partition; a live image has neither).

Additional context

Two constraints make the fix less obvious than it looks:

1. /config is a tmpfs copy. pkg/storage-init/storage-init.sh copies the CONFIG partition into a RAM tmpfs at /config and remounts it read-only. Persisting a new file means writing both the tmpfs copy and the real partition, the way pkg/pillar/scripts/device-steps.sh does when it creates the device certificate.

2. PCR 14. measure-config is deliberately the last onboot container (images/rootfs.yml.in:40-43) and extends PCR 14 with a measurement of /config. /config/soft_serial is on the exclude list (pkg/measure-config/src/measurefs.go:148) so its content is not hashed — but its existence is: absent exclude-list entries are backfilled as exist: false and measured as file:/config/soft_serial exist:false. PCR 14 is part of DefaultDiskKeySealingPCRs (pkg/pillar/evetpm/tpm.go:125-138). Creating the file after measure-config has already run therefore flips PCR 14 on the next boot and the sealed vault key stops unsealing; UnsealDiskKeyWithRecovery only recovers the PCR index selection, not changed values, so recovery depends on the controller-held key backup.

That rules out the otherwise obvious spot (device-steps.sh, right before tpmmgr createDeviceCert) and points at pillar's onboot phase, which runs before measure-config in the same boot.

Suggested fix, gated so that it can only ever fire on a genuine first boot:

• In pkg/pillar/scripts/onboot.sh, if /config/soft_serial is missing and /config/device.cert.pem does not exist yet, generate a UUID and write it to both the tmpfs /config and the CONFIG partition. The device-certificate gate keeps already-deployed devices from gaining the file on upgrade, which would flip PCR 14 under an already-sealed vault.
• Honour eve_soft_serial= from /proc/cmdline in the same place, matching the installer and iPXE convention, so an operator cloning live images can inject a known per-device serial via grub.
• Log the resulting value prominently, since a live image has no INVENTORY partition to deposit it on.

Separately, pkg/eve/runme.sh do_live() should probably stop baking a shared serial into the image (or at least warn), so that the "one serial for the whole fleet" case does not silently persist.