Skip to content

[browsing] Consume the with-tasks routes as compact NDJSON streams#2122

Merged
frankrousseau merged 1 commit into
cgwire:mainfrom
frankrousseau:stream-compact-with-tasks
Jul 8, 2026
Merged

[browsing] Consume the with-tasks routes as compact NDJSON streams#2122
frankrousseau merged 1 commit into
cgwire:mainfrom
frankrousseau:stream-compact-with-tasks

Conversation

@frankrousseau

Copy link
Copy Markdown
Contributor

Problem

  • The all-assets and all-shots views download the full with-tasks payload as one JSON document: heavy on the wire and built entirely in server memory, on a route called constantly (up to ~10k shots on series).

Solution

  • Request the new Kitsu-only options of [perf] Rework the with-tasks views: flat queries, stream and compact options zou#1143 (stream=true&compact=true): the server streams NDJSON, one entity per line, as positional arrays described by the header field lists (payload halved, server peak 132→58MB on assets and 100→33MB on shots).
  • Decode rows in the API client by mapping field names from the header (never by position); stores keep receiving the same list of dicts.
  • Fall back to the legacy plain JSON request on any non-NDJSON answer, so an up-to-date Kitsu keeps working against an older Zou.

Assets and shots full-project views now request stream=true and
compact=true: the server streams one entity per line as positional
arrays described by the header field lists, which halves the payload
and keeps the response out of the server memory. Rows are mapped back
by field name (never by position) in the API client, so the stores
keep receiving the same list of dicts. Any non-NDJSON answer (older
Zou, proxy) falls back to the legacy plain JSON request.
@frankrousseau frankrousseau merged commit 9a427ea into cgwire:main Jul 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant