Skip to content

Split inference includeLogits bool into orthogonal tokens + logits axes - #297

Merged
stikves merged 11 commits into
apple:mainfrom
stikves:sukru/logits-tokens-seam
Oct 28, 2026
Merged

stikves merged 11 commits into
apple:mainfrom
stikves:sukru/logits-tokens-seam

Conversation

@stikves

@stikves stikves commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaces the InferenceOptions.includeLogits: Bool flag with two request axes shared by all four engines (sequential, pipelined, static/ANE, VLM):

  • tokens: TokenRequest (.none / .sample): whether the engine samples and returns a token.
  • logits: LogitsRequest (.none / .lastPosition / .allPositions): which positions' logits it returns.

Presets cover the common pairs:

Preset tokens logits Use
.prefill .none .none prefill and warmup chunks
.extend (alias .default) .sample .none decode
.eval .none .allPositions perplexity and scoring
.guided .none .lastPosition constrained decoding
.sampleWithLogits .sample .lastPosition decode and keep the last-row logits

Motivation

One boolean had to answer two separate questions: whether to sample a token, and which logits to return.

  • The pipelined engine samples on the GPU and never copies logits to the CPU, so it can only return a token.
  • Perplexity and evaluation need logits at every position; constrained decoding needs only the last row.
  • Logit inspection and scoring want a token and its logits together.

With an action axis and an output axis, a caller asks for exactly what it needs, and an engine can reject a request it can't serve before running a forward pass.

Scope

This PR changes the request API and migrates every call site. Engine behavior is unchanged:

  • Engines read only returnsLogits (logits != .none). They still sample when tokens is .none, and return one [vocabSize] row per step for both .lastPosition and .allPositions. The doc comments on TokenRequest and LogitsRequest say so.
  • The pipelined engine still rejects any request that returns logits.

Compatibility

Source-compatible. The default (tokens: .sample, logits: .none) matches the old includeLogits: false, so decode is unchanged. init(includeLogits:) and the includeLogits property stay as deprecated shims: true maps to logits: .lastPosition, false to logits: .none, with tokens: .sample.

Verification

  • On the current head (merged with main at cc3c234), the package and tests build, and 94 tests pass across the InferenceOptions, generate(), BatchScheduler, cancel, stop-reason, idempotent-engine and pipelined-constrained suites. These include tests for every preset, returnsLogits, and both deprecated shims.
  • Before that merge, greedy decode was byte-identical to the base commit on a 135M fp16 model and a 0.8B hybrid model, on both the sequential and pipelined engines, and the pipelined engine's rejection of logit-returning requests matched the base binary. The merge changed one call site (CompletionHandler.scoreOnEngine, now .eval); the golden check was not re-run after it.

Follow-ups

Left for separate changes:

  • Skip the CPU logits readback for .none and narrow it to the last row for .lastPosition on the sequential and VLM engines.
  • Skip sampling when tokens is .none. InferenceOutput.tokenId is non-optional today, so this also needs an output-side change.
  • Support .allPositions on the static and pipelined engines.
  • Engine consolidation and warmup work built on these options.

@stikves
stikves marked this pull request as ready for review September 27, 2026 20:58
@stikves stikves self-assigned this Sep 27, 2026
Replace InferenceOptions.includeLogits: Bool with two orthogonal axes —
tokens: TokenRequest {.none, .sample} and logits: LogitsRequest
{.none, .lastPosition, .allPositions} — plus presets (.prefill / .extend /
.eval / .guided / .sampleWithLogits). Default (.sample, .none) equals the old
includeLogits: false, so decode is byte-identical. All call sites migrated to
presets; engines derive returnsLogits = (logits != .none); the pipelined engine
rejects logits != .none (was includeLogits).
…binations

Add an InferenceOptions.returnsLogits computed property (equivalent to
logits != .none) and route all engines (sequential, static-shape, VLM,
pipelined guard) and the MockEngine through it instead of open-coding the
comparison. Document on the raw initializer that the (tokens, logits) space
is intentionally permissive with no invalid combinations, that the presets
cover the meaningful pairings, and that (tokens: .sample, logits: .allPositions)
is the one representable pair without a dedicated preset. Pure readability
refactor; behavior unchanged.
Comment thread swift/Sources/CoreAILanguageModels/InferenceEngines/InferenceEngine.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/InferenceEngines/InferenceEngine.swift Outdated
Reintroduce the old InferenceOptions.includeLogits boolean as a deprecated
initializer parameter and computed property that bridge to the (tokens, logits)
axes: true -> logits: .lastPosition, false -> .none, with tokens still sampled.
This avoids a hard public-API break from splitting the bool into the orthogonal
axes — external callers keep compiling, with a deprecation warning pointing to
the new API. Addresses review feedback.
Reword the .prefill preset from "no token or logits generation" to describe
what the request needs. The (tokens, logits) axes express intent, and engines
do not yet gate sampling on tokens: .none, so the old wording over-claimed
runtime behavior. Addresses review feedback.
@tjia1818 tjia1818 closed this Oct 12, 2026
@stikves stikves reopened this Oct 27, 2026
@stikves
stikves requested a review from a team as a code owner October 27, 2026 03:59
stikves and others added 3 commits October 26, 2026 21:05
…recate

# Conflicts:
#	swift/Sources/Tools/llm-server/CompletionHandler.swift
Doc comments state what each option and preset does, and note that engines
currently sample under both TokenRequest cases and return one row per step for
both LogitsRequest positions. Tests cover returnsLogits and the deprecated
includeLogits initializer and accessor.

@carinapeng carinapeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address rest of comments, otherwise thanks for the changes from prev review :)

@stikves
stikves merged commit 7fd2063 into apple:main Oct 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants