bug(plugins/container): unbounded blocking in engine bootstrap hangs Falco startup

[leogr]

Describe the bug

StartWorker bootstraps all enabled container engines synchronously, on the caller's thread, before returning. The context it uses is derived from context.Background() with cancel only, so it has no deadline. As a consequence, a runtime socket that accepts a connection but never answers blocks the call forever.

Since StartWorker is invoked from start_async_events, which libs calls from sinsp::open_common() on Falco's main thread, this hangs Falco's startup permanently, before the event loop is even reached.

Reported downstream at falcosecurity/falco#3953

The chain:

• StartWorker runs generator(ctx) and then engine.List(ctx) for each engine, serially and inline, with no deadline 👇
  

  plugins/plugins/container/go-worker/worker_api.go

  Lines 66 to 83 in 21d3ee6

  	for _, generator := range generators {
  	engine, err := generator(ctx)
  	if err != nil {
  	continue
  	}
  	containerEngines = append(containerEngines, engine)
  	if _, ok := enabledEngines[engine.Name()]; !ok {
  	enabledEngines[engine.Name()] = make([]string, 0)
  	}
  	enabledEngines[engine.Name()] = append(enabledEngines[engine.Name()], engine.Sock())
  	// List all pre-existing containers and run `goCb` on all of them
  	containers, err := engine.List(ctx)
  	if err == nil {
  	for _, ctr := range containers {
  	goCb(ctr.String(), true, true)
  	}
  	}
  	}

• newPodmanEngine calls bindings.NewConnection, which pings the service with a GET /_ping. For unix:// URIs, the podman bindings build an http.Client with no Timeout and a net.Dialer with no timeout 👇
  

  plugins/plugins/container/go-worker/pkg/container/podman.go

  Lines 33 to 39 in 21d3ee6

  	func newPodmanEngine(ctx context.Context, _ *slog.Logger, socket string) (Engine, error) {
  	conn, err := bindings.NewConnection(ctx, enforceUnixProtocolIfEmpty(socket))
  	if err != nil {
  	return nil, err
  	}
  	return &podmanEngine{pCtx: conn, socket: socket}, nil
  	}

• podmanEngine.List discards its ctx parameter and uses the connection context instead 👇
  

  plugins/plugins/container/go-worker/pkg/container/podman.go

  Line 188 in 21d3ee6

  func (pc *podmanEngine) List(_ context.Context) ([]event.Event, error) {

• Sockets are not reachability-probed. Generators() creates an engine for any path whose os.Stat does not return NotExist 👇
  

  plugins/plugins/container/go-worker/pkg/container/engine.go

  Lines 61 to 85 in 21d3ee6

  	func Generators() ([]EngineGenerator, error) {
  	generators := make([]EngineGenerator, 0)
  	c := config.Get()
  	for engineName, engineGen := range engineGenerators {
  	eCfg, ok := c.SocketsEngines[string(engineName)]
  	if !ok || !eCfg.Enabled {
  	continue
  	}
  	// For each specified socket, return a closure to generate its engine
  	for _, socket := range eCfg.Sockets {
  	// Properly account for HOST_ROOT env variable
  	socket = filepath.Join(config.GetHostRoot(), socket)
  	// Even if `stat` returns an err that is not NotExist,
  	// try to generate an engine for the socket.
  	if _, statErr := os.Stat(socket); !os.IsNotExist(statErr) {
  	generators = append(generators, func(ctx context.Context) (Engine, error) {
  	return engineGen(ctx, slog.With("engine", engineName), socket)
  	})
  	}
  	}
  	}
  	return generators, nil
  	}

Affected engines:

Engine	Connect timeout	List bounded
podman	none	no (ignores ctx)
docker	none	no
containerd	none	no
cri	5s	yes

So cri is the only one already protected 👇

plugins/plugins/container/go-worker/pkg/container/cri.go

Line 43 in 21d3ee6

func newCriEngine(ctx context.Context, logger *slog.Logger, socket string) (Engine, error) {

Aggravating factor: rootless podman auto-discovery

When engines.podman.sockets is empty, the plugin adds /run/podman/podman.sock and scans /run/user/*/podman/podman.sock, enabling everything it finds 👇

plugins/plugins/container/src/plugin_config.cpp

Lines 66 to 93 in 21d3ee6

	if(cfg.engines.podman.sockets.empty())
	{
	cfg.engines.podman.sockets.emplace_back("/run/podman/podman.sock");
	try
	{
	for(const auto& entry : std::filesystem::directory_iterator(
	cfg.host_root + "/run/user"))
	{
	if(entry.is_directory())
	{
	if(std::filesystem::exists(entry.path().string() +
	"/podman/podman.sock"))
	{
	// Remove host root since it will be later added by
	// go-worker itself
	auto root = entry.path().string().substr(
	cfg.host_root.length());
	cfg.engines.podman.sockets.emplace_back(
	root + "/podman/podman.sock");
	}
	}
	}
	}
	catch(...)
	{
	// No error; perhaps /run/user does not exist.
	}
	}

That scan is active on host installs, since Falco ships engines unset ("We use default config values for engines key" in falco.yaml).

Those sockets are socket-activated by systemd, per user. Falco runs as root, so connect() succeeds because systemd owns the listener, but the response only arrives once that user's podman.service starts. If the user manager is not running, no response ever arrives.

N.B. The Helm chart pins podman.sockets: ["/run/podman/podman.sock"], which suppresses the scan. So Kubernetes deployments are largely shielded, and host RPM/DEB installs are the exposed configuration.

How to reproduce it

1. Install Falco from the RPM/DEB packages, so that engines is left at its defaults
2. Make a unix socket available at /run/user/<uid>/podman/podman.sock that accepts connections but never writes a response. A socket-activated rootless podman.service that cannot start reproduces this naturally (this is what the reporter hit). For a synthetic repro, something like socat UNIX-LISTEN:<path>,fork SYSTEM:'sleep infinity' should do, though I have not verified that exact invocation yet
3. Start Falco

Falco logs up to Trying to open the right engine! and then hangs indefinitely. SIGINT and SIGTERM have no effect, since they only set a flag that the event loop polls, so only SIGKILL works.

Expected behaviour

An unresponsive or misbehaving runtime socket must never be able to block Falco's startup. The engine bootstrap should be bounded, and an engine that does not answer in time should be reported as unavailable instead of hanging.

Environment

• Container plugin version: 0.7.1, and also reproducible on main (no timeout changes have landed on go-worker since 0.7.1)
• Falco version: 0.44.1
• libs version: 0.25.4
• OS: Rocky Linux 10.2, kernel 6.12.0-211.42.1.el10_2.x86_64
• Installation method: RPM, also reported with the Docker/Podman images
• Drivers: reproduced with both kmod and modern_ebpf, since the plugin bootstrap runs before the driver matters

Proposed fix

In order of risk, from lowest:

1. Bound the bootstrap. Wrap each generator(ctx) and engine.List(ctx) in a context.WithTimeout inside StartWorker. This just normalizes the behaviour to what cri already does, and the timeout should be configurable.
2. Honour the passed ctx. podmanEngine.List currently discards it, so (1) alone would have no effect for podman. The other engines should be audited for the same problem.
3. Set explicit client timeouts where the SDK allows it, i.e. client.WithTimeout for docker and containerd.WithTimeout for containerd. The podman bindings own their http.Client, so the ctx deadline from (1) is the only lever there.
4. Follow-up, larger change: do not block startup at all. Move the pre-existing container enumeration into the worker goroutine and let StartWorker return immediately. N.B. this needs care, since capture_open enriches the initial thread table right afterwards, so it would race against the merge of s_preexisting_containers. I'd do 1-3 first and treat this one separately.

Additional context

There is also a design question worth discussing. Auto-enabling every /run/user/*/podman/podman.sock found on disk gives a root-running Falco a hard dependency on arbitrary users' socket-activated services. At the very least, those sockets should be reachability-probed with a short timeout before being enabled. Arguably, the scan should be opt-in. wdyt? 🤔

cc @deepskyblue86 @FedeDP

[leogr]

UPDATE: the reporter's socket probe came back, and it corrects two points in the description above 👇

falcosecurity/falco#3953 (comment)

/run/podman/podman.sock -> 000 TIMEOUT
/var/run/docker.sock -> 000 TIMEOUT
/run/user/1000/podman/podman.sock -> 200
/run/user/1001/podman/podman.sock -> 200

1. The rootless sockets are not the trigger. They answer correctly. The sockets that do not answer are /run/podman/podman.sock and /var/run/docker.sock, i.e. the defaults we ship for the podman and docker engines. So the "aggravating factor" section above is misleading: the /run/user/* scan is still a design concern worth discussing, but it is not what hangs Falco here. The shipped defaults alone are enough to reproduce.

N.B. With the podman-docker package, /var/run/docker.sock is a symlink to /run/podman/podman.sock, so both engines end up blocking on the very same socket.

2. My note about the Helm chart was right, but for the wrong reason. I wrote that the chart shields users because it pins podman.sockets. That is not the mechanism. The chart pins exactly /var/run/docker.sock and /run/podman/podman.sock, which are the two failing paths here. What actually shields Kubernetes users is that the chart mounts no runtime socket by default (mounts.volumes and mounts.volumeMounts are both empty, and no template adds a hostPath for them), so os.Stat reports NotExist and Generators() skips them.

So anyone who does mount /var/run/docker.sock or /run/podman/podman.sock into the Falco pod, which is a fairly common pattern, is exposed exactly like a host install.

This raises the severity in my view. It is not a rootless-podman edge case, it is any unresponsive socket among the defaults.

Also worth noting for whoever picks this up: dockerEngine.List does pass its ctx down to ContainerList, unlike podmanEngine.List which drops it. So fix (2) in the description is podman-specific, while fix (1), the bounded context in StartWorker, is what actually covers all of them.

[leogr]

UPDATE 2: the reporter confirmed the mechanism, and the environment cause turns out to be a clean, reproducible one 👇

falcosecurity/falco#3953 (comment)

Confirmed:

sudo curl ... --max-time 10 --unix-socket /run/podman/podman.sock http://d/_ping
000 in 10.002253s (exit 28)

curl exit 28 is a timeout, not a refusal. The socket accepts the connection and never answers. And /var/run/docker.sock is a symlink to /run/podman/podman.sock, so the docker and podman engines both block on the same socket.

Root environment cause, and it is not an exotic broken state:

podman.socket    Listen: /run/podman/podman.sock (Stream)      <- active
podman.service   Drop-In: /etc/systemd/system/podman.service.d/override.conf
                 └─ /usr/bin/podman ... system service --time=0 tcp:0.0.0.0:12979

podman.socket is enabled and listening on the unix socket, but podman.service has been overridden to serve the API over TCP. Nothing services the unix socket anymore, so systemd accepts the connection and the client waits forever.

That is a perfectly valid podman configuration. Any admin who moves the podman API to TCP and leaves podman.socket enabled gets an orphaned unix socket that behaves as a black hole. So the reproducer is much more mundane than "a broken user service", which is what I assumed in UPDATE 1. This is a supported setup that hangs Falco unkillably.

A second, related gap. The plugin cannot use a TCP endpoint at all, even though the podman bindings support it. Generators() gates each engine on os.Stat of the socket string 👇

plugins/plugins/container/go-worker/pkg/container/engine.go

Lines 71 to 81 in 21d3ee6

	// For each specified socket, return a closure to generate its engine
	for _, socket := range eCfg.Sockets {
	// Properly account for HOST_ROOT env variable
	socket = filepath.Join(config.GetHostRoot(), socket)
	// Even if `stat` returns an err that is not NotExist,
	// try to generate an engine for the socket.
	if _, statErr := os.Stat(socket); !os.IsNotExist(statErr) {
	generators = append(generators, func(ctx context.Context) (Engine, error) {
	return engineGen(ctx, slog.With("engine", engineName), socket)
	})
	}

filepath.Join collapses the double slash first, so the stat never matches:

tcp://0.0.0.0:12979    -> Join = tcp:/0.0.0.0:12979    -> IsNotExist = true  -> engine skipped

Meanwhile enforceUnixProtocolIfEmpty is explicitly written to preserve an existing scheme, and bindings.NewConnection accepts tcp://. So the intent to support non-unix endpoints seems to be there, but the os.Stat gate silently defeats it.

Worth deciding whether to handle this in the same PR or track it separately. At minimum, the stat gate should only apply to unix socket paths.

Release note. No Falco 0.44.2 is planned, since Falco 0.45 lands in about a month. So the fix should go out as a container plugin release in the meantime, and get picked up by the CONTAINER_VERSION pin in Falco's CMakeLists.txt for 0.45.