Temp [ improve worker observability ]

[bKP451]

don’t give crayons to more kids than the box has.

Summary

Improve Delayed::Worker#work_off observability and failure handling around worker thread setup.

What Changed

• Add debug-level logs around job reservation, batch dispatch, thread waiting, and batch completion inside work_off
• Include job name and id when a queued job is handed off for thread execution
• Include job context in the thread crash log by using job_say
• Distinguish thread crashes that happen before perform from those that happen after it starts
• Route pre-perform thread crashes through normal job error handling instead of only logging them
• Cap the worker thread pool size based on the Active Record connection pool so a batch does not fan out beyond available connections

Why

The worker previously had two observability gaps:

• work_off did not log enough detail to show where a batch was in the reserve and dispatch flow
• thread setup failures could be logged without enough job context, and pre-perform crashes were not handled through the normal job failure path

This change makes worker execution easier to trace and improves behavior when a thread fails before the job body starts running.

Behavior

• Adds new debug-level logs for the work_off loop
• Preserves existing handling for crashes that happen after perform has started
• Changes pre-perform thread crashes so they are treated like job failures instead of being logged and dropped
• Limits thread fan-out per batch to reduce the chance of exhausting the Active Record connection pool

[smudge]

Thanks for this PR!

I know you've marked it WIP/Temp, but I have also noticed that it's possible to trigger ActiveRecord::ConnectionTimeoutError if the pool size is lower than max_claims, so this fix definitely makes sense to me and I would be very open to addressing it. 👍

For the sake of reviewing changes independently, I think it would make sense to split the observability/logging changes out into a separate PR, and keep this PR focused on the thread pool / connection pool / max claims reconciliation. (LMK if that doesn't make sense!)

[smudge]

There may be a way to more proactively (e.g. during worker boot/initialization) establish if Delayed::Worker.max_claims > Delayed::Job.connection_pool.size rather than performing this thread_pool_size logic on every pickup loop. (My understanding is that Delayed::Job.connection_pool.size is informed by the pool size config in database.yml, and should not change once the app has loaded.)

The current pickup strategy is also intended to avoid picking up more work than the worker can immediately begin working off (to avoid holding unworked jobs in memory), so it may make sense to raise or warn up front (again, during boot / worker initialization) if a misconfiguration is detected.