Skip to content

feat(checker): arrow tasks with captures, tasks without a destination, and a seeded Rng - #644

Merged
amritk merged 26 commits into
mainfrom
feature/threads-complete/task-forms
Oct 16, 2026
Merged

amritk merged 26 commits into
mainfrom
feature/threads-complete/task-forms

Conversation

@amritk

@amritk amritk commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Part of #575: stage task-forms of "Threads to completion" (WP29 P2). s.spawn(() => work(a)) is now the whole spelling of a task, and parallel code has random numbers that match under Node.

{
  using s = scope()
  s.spawn(() => binInto(xs0, hist))   // captures const locals; answers void; no destination
  s.spawn(addAll, new Job(xs1, hist)) // a named entry, no destination
  s.spawn((n: i32): i32 => sumOf(xs) * k + n, 1, out, 0) // an arrow with a destination
}
const r = new Rng(seed)               // nish/random: the same bits natively and under Node

Changes

  • Arrow tasks (liftArrowArgument / captureOuter in src/generics.ts). An arrow given to spawn captures as feat(checker): parallelMap, parallelFrom and arrows that read the enclosing function's consts #609's data-parallel arrows do. Each name it reads is appended to its parameters, and every instance carries it as %cap.N. The method-call path now passes captures as well (Emitter.captureOperands, emitCall in src/emit-classes.ts). Each capture is copied into the task's payload after the written operands, and $run hands it back (emitSpawnTask in src/emit-parallel.ts). NL2391 ("must be a top-level function named at the call, not an arrow") is retired: the lifted arrow is named after its function (main$arrow0), which is what a debugger shows for the thread.
  • A capture is the task's argument to every scope rule (src/parallel.ts):
    • Region: an arrow task's captures are handed on (handOnCaptures). Writing them before the join, by any name, helper or new, is therefore refused, and a destination cannot be captured.
    • R7: RegionState.taskArgs now holds the capture types too, with their names (taskArgRoots).
    • R8: taskMutexMessageFor walks each capture (NL2510).
    • Arguments: taskArgumentMessage, recordSenders and handsNullableChannel read the task's argument through taskArgTypeOf, which is -1 for a thunk.
    • Capturable types: a task may capture a Mutex. It may not capture a guard, a channel (its senders are counted from the argument), a scope, a Result, a value that may be absent or a Secret (NL2509, captureObstacle(…, task)). A let (NL2506), this (NL2507) and an assignment (NL2508) each get the task's own wording.
  • NL2290 at the join (RefWalk.holdTaskInputs / readHeld in src/arrays.ts). A task reads its argument and its captures when the scope joins. So a name it was handed is read again at the end of the block that declares the scope, and a push between the spawn and the join that may move an element it refers to is refused. This also closes the same gap for a task's argument (reject_thread_task_arg_stale_join), which compiled before.
  • No destination. The language has no overloads, so ThreadScope gets spawnWith(entry, arg) and spawnThunk(entry), and the checker picks one by argument count (spawnFormFor, src/members.ts). Their tasks (runWith, runThunk) are filed like runTask's, and their $finish returns without storing anything. An entry that answers a value is NL2511. Under Node every form reaches spawn, whose new first branch (slotWritten(at), never true natively) runs a task with no slot at once. That branch moves the IR of every program that spawns; nish-cmp reports 0 undeclared differences.
  • RegionState.live() now means "a spawn on this scope has been passed" (started), so a scope whose tasks have no destinations still has a region.
  • A write into an object a function made itself is private while it stays in the object's own slots (composeArguments in src/attributes.ts). An argument edge rooted at the function's own object (ROOT_OWN) marks no shared write when nothing the callee writes can be anything but that object: const r = new Rng(seed); r.nextI32(). When the path has steps (h.b, hs[0]) or the callee writes below its parameter's slots, and a DirtMark says the caller's memory or memory of unknown origin may be there, the write is a shared one (reachesForeign, then markPastWrite). Round 1 (H1): the first head skipped every ROOT_OWN edge, so hs.push(box); hs[0].bump() and h.r = shared; h.r.nextI32() set only writesReach, which the task and body rule never reads, and raced. Each of the reviewer's shapes and its siblings is now a negative, and each one compiled on a compiler built without the fix (reject_thread_task_private_object_element, _field, _ctor_alias, _declarator, _capture, _array_rng, _array_own, reject_par_private_object_field_rng, _array_rng, _for_of); the siblings (_map, _set, _nested, _forward, _forward_store) are refused by both. Round 2: _array_own now gets the argument into the array by a push of new HB(box) rather than by a direct store, which noteSlotWrite refused without this rule; _declarator reads the holder's field back in the statement's second declarator (const h = new HB(box), b = h.b). reject_par_private_object_held and _kept pin the refusal once the caller's array is inside the object. The --emit-checked dumps move where a function's only shared write was into its own object (dump_shared_write_field, dump_checked, checked.txt).
  • std/random.ts. Rng holds one i32 of state. Each step adds the Weyl increment 0x9E3779B9 and answers MurmurHash3's fmix32 of the new state. nextF64 builds 53 bits from two steps. The two wraps are wrappingAdd and wrappingMul from nish:unsafe, which Node answers with | 0 and Math.imul, so tests/link/std_random prints the same lines both ways. I used plain xorshift32 first, but seeds 1, 2, 3 gave visibly correlated first draws, because xorshift is linear in its seed. MurmurHash3 is public domain, and its constants are marked where they appear; THIRD_PARTY_NOTICES.md lists it. Registered in stdModuleNames and in release.yml's three presence gates.
  • Docs: docs/LANGUAGE.md ("Scoped tasks": the three forms and the capture rules with their cases; "Nor is a write into an object's own slots, when the object is one this call made"; "Random numbers: Rng in nish/random"), docs/IR_COOKBOOK.md (thread-task-arrow, std-random), docs/guide/using-nish/threads.md and standard-library.md, docs/RUN_UNDER_NODE.md, docs/wp29-thread-surface.md and std/README.md.
  • Removed: reject_thread_task_arrow, reject_par_capture_spawn and nl2391_thread_task_arrow, whose programs now compile. NL2391 is listed as retired in tests/wordings/unreachable.txt.

Negatives, one per route

All are tests/cases/reject_thread_task_capture_* unless named otherwise:

  • Parent writes before the join:
    • a store: _parent_write
    • a second name: _alias
    • a helper: _helper
    • new writing it: _new
    • this handed out of a constructor: _this_escape
    • the second declarator of a using: _declarator
  • Destination: _dst
  • Task writes:
    • a store: _task_write
    • new run inside the task: _task_new
    • this handed out of a constructor inside the task: _task_this_escape
  • Mutex: R7 _mutex_lock, R8 _mutex_deep (NL2510)
  • Refused types (NL2509): _guard, _channel, _secret
  • Rules feat(checker): parallelMap, parallelFrom and arrows that read the enclosing function's consts #609 closed, through a task capture:
    • NL2394: _or_return
    • NL2290: _stale_record, _stale_join, and reject_thread_task_arg_stale_join for the argument
    • NL2445: _secret_moved
  • Wording:
    • _let (NL2506), _this (NL2507), _assign (NL2508)
    • _nested (NL2501)
    • reject_thread_task_void_result and _arrow (NL2511)
  • Private-object rule (round 1, H1, and its siblings): reject_thread_task_private_object_element, _field, _ctor_alias, _declarator, _capture, _array_rng, _map, _set, _nested, _array_own, _forward, _forward_store; reject_par_private_object_field_rng, _array_rng, _for_of, _held, _kept

tests/wordings/nl2506–nl2511 pin each new message whole.

The IR

The full modules are the committed goldens. Below is what each new program adds.

tests/link/thread_task_arrow (main.ll): the captures passed at each spawn

call void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow0(%struct.ThreadScope* %23, %struct.Mutex$$Tally* %24, %struct.nish_array* %25)
call void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow1(%struct.ThreadScope* %26, %struct.Mutex$$Tally* %27, %struct.nish_array* %28, i32 %29)
call void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow2(%struct.ThreadScope* %30, i8* %31, %struct.Mutex$$Tally* %32, i32 %33)
call void @nish.ThreadScope.spawn$i32$i32$fn.18.scaledParts$arrow0(%struct.ThreadScope* %12, i32 0, %struct.nish_array* %13, i32 0, %struct.nish_array* %xs, i32 %scale)
call void @nish.ThreadScope.spawn$i32$i32$fn.24.Weights.scaledSum$arrow0(%struct.ThreadScope* %11, i32 1, %struct.nish_array* %12, i32 0, %struct.nish_array* %xs, %struct.Weights* %13)

define hidden void @nish_main$arrow0(%struct.Mutex$$Tally* noundef nonnull readonly align 8 dereferenceable(8) nocapture %tally, %struct.nish_array* noundef nonnull align 8 dereferenceable(24) readonly nocapture %xs) #0
define hidden noundef i32 @scaledParts$arrow0(i32 noundef %n, %struct.nish_array* noundef nonnull align 8 dereferenceable(24) readonly nocapture %xs, i32 noundef %scale) #0
define hidden noundef i32 @Weights.scaledSum$arrow0(i32 noundef %k, %struct.nish_array* noundef nonnull align 8 dereferenceable(24) readonly nocapture %xs, %struct.Weights* noundef nonnull readonly align 8 dereferenceable(4) nocapture %self) #0

tests/link/thread_task_arrow (threads.ll): a thunk task and its payload

define void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow0(%struct.ThreadScope* noundef nonnull align 8 dereferenceable(4) %this, %struct.Mutex$$Tally* noundef %cap.0, %struct.nish_array* noundef %cap.1) #1 {
entry:
  %task.payload = alloca { %struct.Mutex$$Tally*, %struct.nish_array* }, align 8
  ; %cap.0 and %cap.1 stored into its two fields
  call void @nish_scope_spawn(i8* %3, void (i8*)* @...$run, void (i8*)* @...$finish, i8* %2, i64 ptrtoint (...))
  ret void
}

define internal void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow0$run(i8* noundef %p) #1 {
  ; both fields loaded back
  call void @nish_main$arrow0(%struct.Mutex$$Tally* %2, %struct.nish_array* %4)
  ret void
}

define internal void @nish.ThreadScope.spawnThunk$void$fn.16.nish_main$arrow0$finish(i8* noundef %p) #1 {
entry:
  ret void
}

tests/link/thread_task_void (threads.ll): s.spawn(entry, arg), and a sender counted from its argument

define void @nish.ThreadScope.spawnWith$$Pipe$void$fn.7.produce(%struct.ThreadScope* noundef nonnull align 8 dereferenceable(4) %this, %struct.Pipe* noundef nonnull align 8 dereferenceable(16) %arg) #1 {
entry:
  %task.payload = alloca { %struct.Pipe* }, align 8
  store %struct.Pipe* %arg, %struct.Pipe** %0
  %3 = getelementptr inbounds %struct.Channel$i32, %struct.Channel$i32* %2, i32 0, i32 2
  call void @nish_channel_count(i64* %3, i64 1)
  call void @nish_scope_spawn(i8* %5, void (i8*)* @...produce$run, void (i8*)* @...produce$finish, i8* %4, i64 ptrtoint (...))
  ret void
}
; $run: call void @produce$(%struct.Pipe* %2), then nish_channel_count(..., i64 -1); $finish: ret void

Every program that spawns (threads.ll): the branch only Node takes

define internal noundef zeroext i1 @nish.slotWritten(i32 noundef %at) #0 {
  ; at < 0 || at >= 0: true for every i32
}
  %0 = call i1 @nish.slotWritten(i32 %at)
  %1 = xor i1 %0, true
  br i1 %1, label %if.then, label %if.end
if.then:                ; Node only: `s.spawn(entry, arg)` or `s.spawn(() => ...)`
  %2 = call double @sumOf$(%struct.nish_array* %arg)
  ret void

tests/link/std_random (random.ll)

define noundef i32 @nish.Rng.nextI32(%struct.Rng* noundef nonnull align 8 dereferenceable(4) nocapture %this) #1 {
  %2 = add i32 %1, -1640531527          ; wrappingAdd(state, 0x9E3779B9)
  ; stored back into `state`
  %6 = tail call i32 @nish.mix(i32 %5)
  ret i32 %6
}
; @nish.mix: x ^= x >>> 16; x = mul i32 x, -2048144789; x ^= x >>> 13; x = mul i32 x, -1028477387; x ^= x >>> 16

The std_random parallel body's new Rng(i + 1) stays inside the element (no NL9012), and its tasks lock only to add a count.

Testing

On this head, with main at ce97fa2 (#621, #639, #645, #647, #640) merged in, each command was run bare and its exit status read:

  • npm run check, npm run lint, npm run lint:dead: exit 0
  • WASI_SYSROOT=/usr npm test (wasi-libc and libclang-rt-18-dev-wasm32 installed), with LLVM 18 on PATH and no DEGRADED: line: 18,361 passed, 0 failed, 1 skipped, exit 0. The skip is the aarch64 static link (no crt1.o here), as on feat(checker): parallelMap, parallelFrom and arrows that read the enclosing function's consts #609
  • node tests/run.js performance: the ratchet matches tests/perf-baseline.json exactly, with no new NL9007 or NL9011
  • node docs/check-links.mjs: ok
  • No runtime/*.c change
  • node scripts/gen-diagnostic-codes.mjs --check and docs/cookbook/regen.sh --check: exit 0
  • nish-cmp against the 0.18.0 seed (run by npm test with NISH_BOOTSTRAP set): 0 undeclared differences
  • Goldens regenerated with node tests/self/goldens.js --update, node tests/run.js <filter> --update-link-goldens, docs/cookbook/regen.sh and node tests/differential/goldens.js --update, never by hand; the diffs were read. After fix(checker): refuse an exported function that takes a C symbol, and spell private ones apart #640, checked-self.txt moves by CheckContext's new field, and fix(checker): refuse an exported function that takes a C symbol, and spell private ones apart #640's two new thread link programs (thread_no_strict_hidden_task, thread_private_prefixed_names) take spawn's slotWritten branch in their threads.ll. No new test program or std/random.ts exports a C-reserved name (NL3037)
  • Each new link program prints the same natively and under Node (the threads-under-node list now includes thread_task_arrow, thread_task_void and std_random)
  • Not used in src/'s own source (the rolling freeze)
  • H1 pins, checked against a neutralised fix: a stage1 built from this head with the ROOT_OWN branch in composeArguments restored to a bare continue compiles reject_thread_task_private_object_element, _field, _ctor_alias, _declarator, _capture, _array_rng, _array_own, and reject_par_private_object_field_rng, _array_rng, _for_of. This head refuses each one with its pinned .err ("through Box.bump" or "through Rng.nextI32"), so reverting the fix fails those cases. The siblings (_map, _set, _nested, _forward, _forward_store) are refused by both builds.

Related issues

Part of #575.


The squash commit, which is the changelog entry:

feat(checker): arrow tasks with captures, tasks without a destination, and a seeded Rng

A scope's task can now be an arrow written at the `spawn`, and it may read
the `const` locals and the parameters of the function it is written in:
`s.spawn(() => binInto(xs, hist))` is the whole spelling of a task. Each
name the arrow reads is copied into the task's payload beside its argument
and read on the task's thread when the scope joins, so every scope rule
holds of it as of the argument: the parent may not write it before the
join by any name, helper or constructor, a destination is never captured,
and a captured `Mutex` is held to R7 and R8. A guard, a channel, a scope, a
`Result`, a value that may be absent and a `Secret` are not captured, and
neither is a `let` or `this`. The arrow is lifted into a function named
after the one it is written in, which is what a debugger shows for the
task's thread.

`s.spawn(entry, arg)` and `s.spawn(() => ...)` run a task that answers
`void` and has nothing to store; the four-argument form still stores an
answer. A task's argument and captures are read again at the join, so a
`push` that may move an element one of them refers to is refused between
the spawn and the end of the block.

A function may write the own slots of an object it made itself without
that counting as a shared write, as it already could an array. A write
below those slots counts as a write into whatever they may hold, so once
the caller's memory is inside the object it is refused as any shared
write is. That is what lets a parallel body or a task step a generator of
its own.

`nish/random` adds `Rng`: `new Rng(seed)`, `nextI32()` and `nextF64()`, a
Weyl sequence through all 2^32 states with MurmurHash3's finaliser on each.
Its wraps are `nish:unsafe`'s, so every answer is the same bits natively
and under Node.

Refs: docs/LANGUAGE.md#scoped-tasks-using-s--scope, docs/LANGUAGE.md#random-numbers-rng-in-nishrandom
Tests: tests/link/thread_task_arrow, tests/link/thread_task_void, tests/link/std_random, tests/cases/reject_thread_task_capture_*, tests/cases/reject_thread_task_void_result, tests/cases/reject_thread_task_arg_stale_join, tests/cases/reject_thread_task_private_object_*, tests/cases/reject_par_private_object_*

claude added 21 commits October 12, 2026 11:40
…losing function's consts

The Rayon one-liners are now spellable with `nish/threads`: a new array
from a map, a new array from an index, and a body that reads the
function's local constants.

```ts
const ys = parallelMap(xs, (x) => x * k + b)          // k, b: const locals
const sq = parallelFrom(n, (i) => i * i)              // Array.from({ length: n }, ...)
parallelMapInto(xs, dst, (x) => table[x % 8] * k)     // table: a const array, read-only
```

`parallelMap(src, f)` and `parallelFrom(n, f)` allocate their result on
the calling thread before the region and then run the existing map:
`parallelMap` is `parallelMapInto` into that array, and `parallelFrom`
the same region over the indices through a chunk loop of its own. Their
`f` is held to P1's rules unchanged, and a result that is not a number,
a `boolean` or an enum is refused at the call, before `std/threads.ts`
is instantiated for it.

An arrow passed to any of the four forms may now read the `const`
locals and the parameters of the function it is written in. Each name
it reads is appended to the arrow's parameters; every instance between
the call and the body carries the value as a parameter of its own, and
the region packs it into the context the runtime already passes. A
scalar is copied and a reference is read-only: a store through it is a
shared write and is refused as one, through an alias, a field, an
element, a helper or a method included. A `let` is refused with a
message telling the user to bind a `const`, as are `this`, a read from
outside a nested arrow, a lock, a guard, a channel, a scope and a
`Result`. In `parallelMapInto`, a capture that could reach `dst` is
refused, except a captured `const` array of its own beside a `dst` that
is another such `const`. No closure is made, and no other arrow
captures: the arrow is still a function of its own with a direct call.

Refs: docs/LANGUAGE.md#a-data-parallel-arrow-reads-its-functions-const-locals
Tests: tests/link/par_capture, tests/link/par_capture_large, tests/link/par_map_new, tests/link/par_from, tests/cases/reject_par_capture_*

Part of #575
…text loop keeps its length facts

The loop that splits a region's arguments into its context fields read
each element after a push, which ends the length facts, and kept two
bounds checks the performance ratchet counts against src/.
…ete/data-forms

# Conflicts:
#	tests/self/goldens/checked-self.txt
#	tests/self/goldens/checked.txt
…ete/data-forms

# Conflicts:
#	docs/AI.md
#	tests/self/goldens/checked-self.txt
…function's locals

Every rule about a local of the enclosing function now sees a capture as a
read of that local where the call runs the body: a captured parameter keeps
a link to the local it was made from (`Local.captureOf`), the walks that
compare locals resolve through it (`sourceLocal`), and the walks that pass
over an arrow read its captures at the arrow (`capturedReads`). That closes
a scope destination read through `parallelFrom` (NL2394), a stale inline
record reference read after a push or a pop (NL2290), and an array read
after `secret(raw)` took it (NL2445).

`dst` is resolved where the call is checked, in a generic caller's own
tables, so two fresh `const`s are no longer refused there, and a parameter
array beside a fresh `dst` is admitted: a fresh array is memory no older
binding held.
…ete/data-forms

# Conflicts:
#	tests/self/goldens/checked.txt
…ete/data-forms

# Conflicts:
#	tests/run.js
#	tests/self/goldens/checked-self.txt
…tability warnings

The portability rows that follow a local through its reads skipped an arrow
argument's body, so a record copied into an array (NL8003) or a byte offset
compared with a count (NL8001) went unreported when the read was a capture
inside an arrow given to `parallelMap`, `parallelFrom`, `parallelMapInto` or
`parallelReduce`. A captured read now counts as a read of the local where the
call runs the arrow, as it already did for the scope, element-reference and
`Secret` rules. A refused region call inside a scope also reports its arrow's
captured destination read, and NL2494 says that `this` is refused for its
spelling: `const self = this` is captured.

Refs: docs/LANGUAGE.md#a-data-parallel-arrow-reads-its-functions-const-locals
Tests: tests/link/port_rec_capture_copy, tests/link/port_str_capture_offset, tests/link/port_rec_capture_quiet, tests/cases/reject_par_capture_*
…ete/data-forms

# Conflicts:
#	tests/self/goldens/checked.txt
…low an offset out of its answer

NL8009 lined a function's written parameters up with its parameter types from
the end, and an arrow handed to a data-parallel form has the names it captures
appended after the written ones, so each was checked against another's type:
a false warning beside a captured `i64`, a missed one beside a captured `i32`.
The alignment now ends at the written parameters.

NL8001 followed a captured byte offset into the arrow's `return` and read it
as `main`'s exit code, and lost it at a concise body. A value an arrow answers
with is now stored in the array its form writes, and `parallelReduce`'s answer
is followed on from the call.

Refs: docs/LANGUAGE.md#a-data-parallel-arrow-reads-its-functions-const-locals
Tests: tests/link/port_num_capture_wide, tests/link/port_str_capture_result
…ete/data-forms

# Conflicts:
#	tests/self/goldens/checked.txt
…ete/data-forms

# Conflicts:
#	tests/self/goldens/checked-self.txt
#	tests/self/goldens/checked.txt
…ete/data-forms

main took NL2492-NL2499 for typed JSON's options (#630), so the
data-parallel capture refusals move to NL2500-NL2505, in the same order.
…ete/data-forms

main took NL2492-NL2499 for typed JSON's options (#630), so the
data-parallel capture refusals move to NL2500-NL2505, in the same order.
#616's hidden-symbol spelling moves the par_map_new and par_from goldens,
regenerated with --update-link-goldens.
…ete/task-forms

# Conflicts:
#	docs/LANGUAGE.md
#	docs/guide/using-nish/threads.md
#	src/arrays.ts
#	src/codes.ts
#	src/context.ts
#	src/emit-parallel.ts
#	src/expressions.ts
#	src/generics.ts
#	src/parallel.ts
#	tests/self/goldens/checked-self.txt
#	tests/self/goldens/checked.txt
…ete/task-forms

# Conflicts:
#	docs/guide/using-nish/threads.md
#	tests/self/goldens/checked-self.txt
#	tests/self/goldens/checked.txt
…sk made itself is a shared write when the caller's memory may be there
…ete/task-forms

# Conflicts:
#	tests/link/caps_parallel/expected.caps.json
#	tests/link/panics_parallel_length/expected.panics
#	tests/self/goldens/checked.txt
…d a second declarator

reject_thread_task_private_object_array_own now gets the task's argument into
its own array by pushing a holder made around it, rather than by a direct
store that the slot-write rule refused on its own; it compiles without
reachesForeign and is refused with it. reject_thread_task_private_object_declarator
reads the holder's field back in the statement's second declarator. Unused
imports leave two cases.
…ete/task-forms

# Conflicts:
#	src/codes.ts
#	tests/run.js
#	tests/self/goldens/checked-self.txt
@amritk
amritk merged commit cbb2213 into main Oct 16, 2026
28 checks passed
@amritk
amritk deleted the feature/threads-complete/task-forms branch October 16, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants