[DX-3791] devenv tests Docker logs panics scan

[Tofel]

Summary

• Add single-stream Docker log fanout support in framework, so test cleanup consumers read one shared stream instead of opening parallel streams.
• Introduce stream-injected cleanup consumers in framework (SaveContainerLogsFromStreams, PrintFailedContainerLogsFromStreams), add PrintFailedContainerLogs in framework, and fix SaveContainerLogs shared-slice race.
• Add CTF convenience APIs in framework (CTFContainersListOpts, CTFContainersLogsOpts, StreamCTFContainerLogsFanout) to reduce callsite boilerplate.
• Move system-tests failed-log printing to framework delegation (remove duplicated implementation in system-tests/lib/infra/docker.go).
• Add products.ScanLogsFromStreams and shared products.CleanupContainerLogs(settings) in devenv, then migrate non-functional devenv tests to use this helper (supports per-test scanner exclusions via settings).

Requires:

• passing legacy functional tests ✅
• passing legacy non-functional tests ✅
• merged CTF PR ✅
• startup panic in Docker container correctly found ✅

[github-actions]

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[github-actions]

✅ No conflicts with other open PRs targeting develop

[trunk-io]

     

Failed Test	Failure Summary	Logs
TestConfigureForwarder/set_config_without_mcms	The test failed without a clear error message, but it appears to be related to a configuration or registration process in a blockchain or smart con...	Logs ↗︎

View Full Report ↗︎ ⋅ Docs

[Copilot]

Pull request overview

Risk Rating: MEDIUM — touches shared test harness cleanup behavior, introduces/uses concurrent log-stream processing, and updates a shared testing-framework dependency.

This PR centralizes Docker container log scanning/saving/failed-log printing into the testing framework and devenv/products, then migrates test callsites to use the new helpers to avoid parallel log streams and reduce duplicated cleanup logic.

Changes:

• Bump chainlink-testing-framework/framework to v0.15.17 across modules (system-tests, devenv, core/scripts).
• Remove system-tests/lib/infra/docker.go (duplicated log/panic scanning + failed-log printing) and delegate to framework helpers.
• Add products.ScanLogsFromStreams + products.CleanupContainerLogs(settings) and migrate many devenv tests to use the shared cleanup helper.

Reviewed changes

Copilot reviewed 35 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
system-tests/tests/test-helpers/before_suite.go	Switches panic/failed-log handling to framework helpers in test cleanup.
system-tests/tests/go.mod	Bumps CTF framework dependency to v0.15.17.
system-tests/tests/go.sum	Updates sums for CTF framework v0.15.17.
system-tests/lib/infra/docker.go	Removes duplicated Docker log printing + panic scanning implementation.
system-tests/lib/go.mod	Bumps CTF framework dependency to v0.15.17.
system-tests/lib/go.sum	Updates sums for CTF framework v0.15.17.
system-tests/lib/cre/environment/jobs.go	Uses framework failed-container log printing on JD start failures.
system-tests/lib/cre/environment/dons.go	Uses framework failed-container log printing on DON startup failures.
system-tests/lib/cre/environment/blockchains/blockchains.go	Uses framework failed-container log printing on deployer selection errors.
devenv/products/logs.go	Adds stream-based log scanning and a consolidated cleanup helper using fanout.
devenv/tests/vrfv2plus/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/replay_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/pending_block_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/multiple_keys_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/migration_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/bhs_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2plus/bhf_test.go	Migrates cleanup and adds per-test allowed-message settings.
devenv/tests/vrfv2plus/batch_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2/multiple_keys_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2/bhs_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrfv2/batch_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/vrf/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/ocr2/soak_test.go	Switches to log-stream fanout for scanning/panic detection; saves logs on failure.
devenv/tests/ocr2/smoke_test.go	Migrates cleanup and adds per-test allowed-message settings.
devenv/tests/ocr2/chaos_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/logpoller/logpoller_test.go	Replaces direct scanning with consolidated cleanup helper.
devenv/tests/keepers/smoke_test.go	Replaces per-subtest scan+save with consolidated cleanup helper.
devenv/tests/flux/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/features/reorg_finality_violation_test.go	Migrates cleanup and adds per-test allowed-message settings.
devenv/tests/features/jd_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/directrequest/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/cron/smoke_test.go	Migrates cleanup to products.CleanupContainerLogs.
devenv/tests/automation/smoke_test.go	Restores full test list and migrates cleanup to consolidated helper.
devenv/tests/automation/load_test.go	Uses log-stream fanout for scanning/panic detection; saves logs on failure.
devenv/go.mod	Bumps CTF framework dependency to v0.15.17.
devenv/go.sum	Updates sums for CTF framework v0.15.17.
core/scripts/go.mod	Bumps CTF framework dependency to v0.15.17.
core/scripts/go.sum	Updates sums for CTF framework v0.15.17.

Comments suppressed due to low confidence (1)

devenv/products/logs.go:107

• The goroutine in ScanLogsFromStreams closes over the loop variable stream. In Go, the iteration variable is reused, so multiple goroutines can end up scanning the same (last) stream, causing missed logs and flaky/incorrect scan results. Capture the current stream per-iteration (e.g., assign to a new local or pass it as a parameter to the closure).

	verifyLogsGroup := &errgroup.Group{}
	for _, stream := range logStream {
		verifyLogsGroup.Go(func() error {
			verifyErr := verifyLogStream(
				stream,
				settings.FailingLogLevel,
				settings.Threshold,
				settings.AllowedMessages...,
			)
			// ignore processing errors
			if verifyErr != nil && !strings.Contains(verifyErr.Error(), MultipleLogsAtLogLevelErr) &&
				!strings.Contains(verifyErr.Error(), OneLogAtLogLevelErr) {
				l.Error().Err(verifyErr).Msg("Error processing CL node logs")

				return nil

				// if it's not a processing error, we want to fail the test; we also can stop processing logs all together at this point
			} else if verifyErr != nil &&
				(strings.Contains(verifyErr.Error(), MultipleLogsAtLogLevelErr) ||
					strings.Contains(verifyErr.Error(), OneLogAtLogLevelErr)) {
				return verifyErr
			}
			return nil
		})

[cl-sonarqube-production]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube