feat(rq): configurable HTTP source fetch retries

[dan-m62]

Summary

This exposes two new settings for the RQ engine so operators can control how HTTP source fetches are retried, and wires them into RQOrchestratorConfig. It is the docling-serve half of the retry work; the behaviour itself lives in docling-project/docling-jobkit#200, and both follow up on #648.

The new settings mirror the existing eng_ray_* retry knobs, so the two engines are configured the same way.

What this changes

Two new settings (defaults chosen to match the Ray engine):

Environment variable	Default	Description
DOCLING_SERVE_ENG_RQ_MAX_TASK_RETRIES	3	Additional attempts when fetching an HTTP source fails with a transient error (429, 502, 503, 504, or a connection/timeout failure). Permanent failures (401/403/404/413/415/422, oversize) are never retried. Set to 0 to disable retrying.
DOCLING_SERVE_ENG_RQ_RETRY_DELAY	5.0	Fixed delay, in seconds, between HTTP source fetch retry attempts.

They are wired into RQOrchestratorConfig at both construction sites — the API's orchestrator_factory.py and the rq-worker command in __main__.py. Both are needed: the fetch (and therefore the retry) happens in the worker process, so the worker's config is the one that actually governs retry behaviour.

Files

• docling_serve/settings.py — eng_rq_max_task_retries, eng_rq_retry_delay.
• docling_serve/orchestrator_factory.py — pass them into RQOrchestratorConfig (API).
• docling_serve/__main__.py — pass them into RQOrchestratorConfig (rq-worker).
• docs/configuration.md — document the two variables.

Consistency with the Ray engine

The names (max_task_retries / retry_delay), defaults (3 / 5.0) and settings style match the existing eng_ray_max_task_retries / eng_ray_retry_delay, so operators configure both engines the same way.

Compatibility

RQOrchestratorConfig ignores unknown keyword arguments, so this change is safe to merge against the current released docling-jobkit — the new arguments are simply dropped until the jobkit change (docling-project/docling-jobkit#) is released, at which point the retries take effect. Merging jobkit first is recommended so the settings are functional on release.

Testing

Validated end-to-end by running the API and an rq-worker from this branch (with the docling-jobkit branch installed) against a live Redis. Setting the variables and submitting a task whose HTTP source returns 503 produced the configured behaviour in the worker log:

DOCLING_SERVE_ENG_RQ_MAX_TASK_RETRIES=2
DOCLING_SERVE_ENG_RQ_RETRY_DELAY=1

retry 1/2 after 1.0s
retry 2/2 after 1.0s

(With the defaults left in place the worker retries 3 times with a 5.0s delay, as expected.)

Related

• Original issue: RQ engine silently reports success for unfetchable HTTP sources (no error surfaced, no retry) #648
• docling-jobkit PR (the behaviour): feat(rq): retry transient HTTP source fetch failures docling-jobkit#200

[github-actions]

✅ DCO Check Passed

Thanks @dan-m62, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?: