evetest: TestNetworkConfigFallback never re-activates the newer DPC after the network is restored

[eriknordmark]

TestDeviceConnectivitySuite/TestNetworkConfigFallback fails on every image, at the same assertion and with near-identical timing:

master	resize-allprs	appvol-allprs
FAIL 612.4 s	FAIL 622.0 s	FAIL 610.9 s

failover_test.go:642: TEST FAILURE:
    Timed out after 474.9s / 474.6s / 470.8s.
    Expected to satisfy: The newer DPC becomes active again once the network matches it

The device never promotes the newer DPC back to active after the network is restored to match it; the suite's captured systemAdapter shows the newer entry stuck with lastFailed set while the older one holds lastSucceeded. The recorded error on the failing port is a DNS timeout against the device's own resolver:

All attempts to connect to https://adam.evetest:443/api/v2/edgedevice/ping failed:
send via eth0 with src IP 10.99.99.5: Get "https://adam.evetest:443/api/v2/edgedevice/ping":
dial tcp: lookup adam.evetest on 127.0.0.1:53: read udp 127.0.0.1:57493->127.0.0.1:53: i/o timeout

The three timeout values span 4.1 s across three separate images, which is the signature of a deterministic failure rather than a race.

Worth noting for triage: the rest of this suite is genuinely flaky on my host — six other subtests failed on one or two images and passed on the others, in both directions. TestNetworkConfigFallback is the only one that failed on all three, which is why it is the only one reported here.

Environment

Three EVE images, all -kvm-amd64 unless noted, all built from master f94089001 or an integration branch based on it; evetest 1.5; device TPM-less; EVETEST_SUITE_MAX_FAILURES=-1 so the whole suite runs. One run per image, same test tree and same harness image for all three, only EVETEST_EVE_VERSION differing.

[milan-zededa]

Addressed in #6482

[eriknordmark]

Still fails with #6480 (tree 9934b28b7, same master EVE image): FAIL 409.4 s, now 7 failures in 7 runs.

The failure moved earlier -- 409 s here against a tight 611-622 s across the previous six runs -- which fits #6480's changes to failover_test.go shortening the wait. It gives up sooner but the newer DPC still never becomes active again.

Not fixed; leaving open.

[milan-zededa]

same master EVE image

And what image version is that? Use the very latest that includes all my recent DPC fixes (#6460, #6464, #6465, #6466).
It passes for me: https://github.com/milan-zededa/eve/actions/runs/34381743343/job/102568307997

[eriknordmark]

Still fails with #6480 -- 7 of 7

Re-ran with the #6480 tree (9934b28b7, harness 1.6, adam 0.0.82) against the same master image: FAIL 409.4 s. Seven failures in seven runs, spanning three EVE images, two evetest trees and both TPM and TPM-less devices.

The failure now arrives ~200 s sooner (409 s vs a tight 611-622 s over the previous six runs), consistent with #6480 shortening the wait. It gives up earlier; the newer DPC still never becomes active.

Where I think it goes wrong

Reading the test rather than the symptom, phase 3 looks incomplete. It repairs the network by editing only the subnet and gateway of network0:

for _, n := range fixedModel.Networks {
    if n.LogicalLabel == "network0" {
        n.Ipv4.Subnet = "10.99.99.0/24"
        n.Ipv4.GwIp   = "10.99.99.1"
    }
}

but the broken DPC applied in phase 2 also pins a DNS server:

const dnsServer0IP = "10.16.16.25"
...
DNSServers: []net.IP{evetest.IPAddress(dnsServer0IP)},

10.16.16.25 lives on the SDN's own 10.16.16.0/24 endpoint network, reached by routing from the device network -- which in TwoMgmtPorts is 172.20.20.0/24. Phase 3 moves network0 to 10.99.99.0/24 but changes nothing about how 10.16.16.0/24 is reached from it.

That matches the device-side evidence captured when it failed: the newer DPC's port carries dns:{DNSservers:"10.16.16.25"} and the recorded error is a DNS timeout, not a routing or TCP failure:

All attempts to connect to https://adam.evetest:443/api/v2/edgedevice/ping failed:
send via eth0 with src IP 10.99.99.5: dial tcp: lookup adam.evetest on 127.0.0.1:53:
read udp 127.0.0.1:57493->127.0.0.1:53: i/o timeout

So EVE gets its static address and gateway as intended, and then cannot resolve the controller, so the DPC's connectivity test keeps failing and NIM never promotes it back to index 0 -- exactly what the assertion waits for.

What I have not verified is that SDN routing to 10.16.16.0/24 is actually broken from the relocated network0; I have the test source and the device's own error, not the SDN's routing state at that moment. That is the check I would do first: after UpdateNetworkModel(fixedModel), confirm from the device whether 10.16.16.25 is reachable at all. If it is not, the fix is for phase 3 to repair DNS reachability alongside the subnet -- or for phase 2 to use a DNS server that survives the subnet change.

[milan-zededa]

against the same master image

Did you see my comment: #6483 (comment) ?
It was fixed in different PRs (#6460, #6464, #6465, #6466) and it was a set of EVE bugs, not test flakiness (not related to #6480)