Repository navigation
Split inference includeLogits bool into orthogonal tokens + logits axes - #297
Merged
Merged
Conversation
stikves
marked this pull request as ready for review
September 27, 2026 20:58
Replace InferenceOptions.includeLogits: Bool with two orthogonal axes —
tokens: TokenRequest {.none, .sample} and logits: LogitsRequest
{.none, .lastPosition, .allPositions} — plus presets (.prefill / .extend /
.eval / .guided / .sampleWithLogits). Default (.sample, .none) equals the old
includeLogits: false, so decode is byte-identical. All call sites migrated to
presets; engines derive returnsLogits = (logits != .none); the pipelined engine
rejects logits != .none (was includeLogits).
stikves
force-pushed
the
sukru/logits-tokens-seam
branch
from
September 27, 2026 21:25
132916f to
9431478
Compare
…binations Add an InferenceOptions.returnsLogits computed property (equivalent to logits != .none) and route all engines (sequential, static-shape, VLM, pipelined guard) and the MockEngine through it instead of open-coding the comparison. Document on the raw initializer that the (tokens, logits) space is intentionally permissive with no invalid combinations, that the presets cover the meaningful pairings, and that (tokens: .sample, logits: .allPositions) is the one representable pair without a dedicated preset. Pure readability refactor; behavior unchanged.
carinapeng
reviewed
Oct 8, 2026
carinapeng
reviewed
Oct 8, 2026
carinapeng
reviewed
Oct 8, 2026
Reintroduce the old InferenceOptions.includeLogits boolean as a deprecated initializer parameter and computed property that bridge to the (tokens, logits) axes: true -> logits: .lastPosition, false -> .none, with tokens still sampled. This avoids a hard public-API break from splitting the bool into the orthogonal axes — external callers keep compiling, with a deprecation warning pointing to the new API. Addresses review feedback.
Reword the .prefill preset from "no token or logits generation" to describe what the request needs. The (tokens, logits) axes express intent, and engines do not yet gate sampling on tokens: .none, so the old wording over-claimed runtime behavior. Addresses review feedback.
…recate # Conflicts: # swift/Sources/Tools/llm-server/CompletionHandler.swift
Doc comments state what each option and preset does, and note that engines currently sample under both TokenRequest cases and return one row per step for both LogitsRequest positions. Tests cover returnsLogits and the deprecated includeLogits initializer and accessor.
kevchengcodes
approved these changes
Oct 27, 2026
carinapeng
reviewed
Oct 27, 2026
carinapeng
reviewed
Oct 27, 2026
carinapeng
approved these changes
Oct 27, 2026
carinapeng
left a comment
Contributor
There was a problem hiding this comment.
Please address rest of comments, otherwise thanks for the changes from prev review :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
InferenceOptions.includeLogits: Boolflag with two request axes shared by all four engines (sequential, pipelined, static/ANE, VLM):tokens: TokenRequest(.none/.sample): whether the engine samples and returns a token.logits: LogitsRequest(.none/.lastPosition/.allPositions): which positions' logits it returns.Presets cover the common pairs:
tokenslogits.prefill.none.none.extend(alias.default).sample.none.eval.none.allPositions.guided.none.lastPosition.sampleWithLogits.sample.lastPositionMotivation
One boolean had to answer two separate questions: whether to sample a token, and which logits to return.
With an action axis and an output axis, a caller asks for exactly what it needs, and an engine can reject a request it can't serve before running a forward pass.
Scope
This PR changes the request API and migrates every call site. Engine behavior is unchanged:
returnsLogits(logits != .none). They still sample whentokensis.none, and return one[vocabSize]row per step for both.lastPositionand.allPositions. The doc comments onTokenRequestandLogitsRequestsay so.Compatibility
Source-compatible. The default
(tokens: .sample, logits: .none)matches the oldincludeLogits: false, so decode is unchanged.init(includeLogits:)and theincludeLogitsproperty stay as deprecated shims:truemaps tologits: .lastPosition,falsetologits: .none, withtokens: .sample.Verification
mainat cc3c234), the package and tests build, and 94 tests pass across the InferenceOptions,generate(), BatchScheduler, cancel, stop-reason, idempotent-engine and pipelined-constrained suites. These include tests for every preset,returnsLogits, and both deprecated shims.CompletionHandler.scoreOnEngine, now.eval); the golden check was not re-run after it.Follow-ups
Left for separate changes:
.noneand narrow it to the last row for.lastPositionon the sequential and VLM engines.tokensis.none.InferenceOutput.tokenIdis non-optional today, so this also needs an output-side change..allPositionson the static and pipelined engines.