[danieroux]

Nested inputs wait for every batch of the current round

Repro

(def env
  (pci/register
    [(pco/resolver 'entity->nested
       {::pco/input  [:entity/id]
        ::pco/output [{:entity/nested [:nested/id]} :entity/sibling-key]}
       (fn [_ {:entity/keys [id]}]
         {:entity/nested      {:nested/id id}
          :entity/sibling-key id}))
     (pco/resolver 'nested-batch
       {::pco/input  [:nested/id]
        ::pco/output [:nested/value]
        ::pco/batch? true}
       (fn [_ items] (mapv (fn [{:nested/keys [id]}] {:nested/value (str "nested-value-of-" id)}) items)))
     (pco/resolver 'sibling-batch
       {::pco/input  [:entity/sibling-key]
        ::pco/output [:entity/sibling-value]
        ::pco/batch? true}
       (fn [_ items] (mapv (fn [{:entity/keys [sibling-key]}] {:entity/sibling-value (str "sibling-value-of-" sibling-key)}) items)))
     (pco/resolver 'needs-nested-and-sibling
       {::pco/input  [{:entity/nested [:nested/value]} :entity/sibling-value]
        ::pco/output [:entity/result]}
       (fn [_ {:entity/keys [nested sibling-value]}]
         {:entity/result (str (:nested/value nested) " and " sibling-value)}))]))

(p.eql/process env {:entity {:entity/id "entity"}} [{:entity [:entity/result]}])

Expected:

{:entity {:entity/result "nested-value-of-entity and sibling-value-of-entity"}}

On main:

Required attributes missing at path [:entity]:
- Attribute :entity/result was expected to be returned from resolver needs-nested-and-sibling but inputs were missing:
  - Attribute :entity/nested was expected to be returned from resolver entity->nested but it failed to provide it.

The error message is misleading: entity->nested did provide :entity/nested. What was missing was :nested/value inside it.

What happens

:entity
├─ :entity/nested ────── nested-batch ──▶ :nested/value ─────────┐
└─ :entity/sibling-key ─ sibling-batch ─▶ :entity/sibling-value ─┤
                                                                 ▼
                                            needs-nested-and-sibling ─▶ :entity/result

nested-batch (inside :entity/nested) and sibling-batch (on :entity) both hold in the same round. run-batches-pending! empties ::batch-pending*, then runs the round's batches one after the other. Each batch resumes its entities as soon as it finishes:

1. nested-batch runs and merges :nested/value into the root data.
2. sibling-batch runs and resumes :entity. But :entity still has its own copy of :entity/nested, without :nested/value.
3. missing-maybe-in-pending-batch? finds nothing pending, so needs-nested-and-sibling fails instead of waiting.

If the batches run in the other order, :entity is resumed before nested-batch runs. That batch is already out of ::batch-pending*, so the result is the same.

It only happens for a non-root entity (here :entity sits under a join). At the root there's no separate copy to go stale.

The fix

Keep the round's batches in ::batch-pending* until all of them have run, then remove them (remove-ran-batches-keeping-new-batch-items!). Batch items added during the round are kept for the next round. A nested input on any batch of the round now waits, and run-batches-waiting! resets the entity from the root data before running it again.

Done for the serial and async runners. The parallel runner passes the new tests unchanged.

Tests

The first commit adds a test that fails on main with two cases, one per batch order, on the serial and async runners. The second commit is the fix. The full suite passes.