Tracks the fix for GHSA-x9f2-h2rj-vp63:
https://github.com/cisagov/Malcolm/security/advisories/GHSA-x9f2-h2rj-vp63
Several Malcolm API endpoints make outbound requests to other Malcolm
services without a timeout. If one of those services accepts a connection
but stops responding, the API request stays open until gunicorn kills the
worker. Because the API runs a single sync worker by default, a slow
dependency can make the whole API unresponsive, including /mapi/ping,
which the container healthcheck uses.
The readiness endpoint (/mapi/ready) is the most affected, since it runs
more than a dozen checks one after another and their timeouts add up.
Changes:
- Add explicit timeouts to every outbound
requests call in
api/project/__init__.py, configurable through
MALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC (default 5) and
MALCOLM_API_HTTP_READ_TIMEOUT_SEC (default 30)
- Bound each
/mapi/ready check with MALCOLM_API_READY_TIMEOUT_SEC
(default 5), and run the checks concurrently so the endpoint's total time
is roughly that of its slowest check
- Add
socket_timeout to the Redis/Valkey connections used by /mapi/ready
and the keyspace endpoint
- Add an optional
timeout argument to malcolm_utils.check_socket
(default unchanged at 10 seconds)
- Run the API with
--workers 2 --threads 4 in both compose files so one
slow request can't block the API or its healthcheck
Tracks the fix for GHSA-x9f2-h2rj-vp63:
https://github.com/cisagov/Malcolm/security/advisories/GHSA-x9f2-h2rj-vp63
Several Malcolm API endpoints make outbound requests to other Malcolm
services without a timeout. If one of those services accepts a connection
but stops responding, the API request stays open until gunicorn kills the
worker. Because the API runs a single sync worker by default, a slow
dependency can make the whole API unresponsive, including
/mapi/ping,which the container healthcheck uses.
The readiness endpoint (
/mapi/ready) is the most affected, since it runsmore than a dozen checks one after another and their timeouts add up.
Changes:
requestscall inapi/project/__init__.py, configurable throughMALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC(default 5) andMALCOLM_API_HTTP_READ_TIMEOUT_SEC(default 30)/mapi/readycheck withMALCOLM_API_READY_TIMEOUT_SEC(default 5), and run the checks concurrently so the endpoint's total time
is roughly that of its slowest check
socket_timeoutto the Redis/Valkey connections used by/mapi/readyand the keyspace endpoint
timeoutargument tomalcolm_utils.check_socket(default unchanged at 10 seconds)
--workers 2 --threads 4in both compose files so oneslow request can't block the API or its healthcheck