[perf] Rework the with-tasks views: flat queries, stream and compact options#1143
Merged
Merged
Conversation
get_assets_and_tasks joined Entity x Task x TaskPersonLink in one query: one row per (asset, task, assignee) carrying the full asset row each time, sorted and materialized before a Python dedup. On a large production that meant 100k+ wide tuples for a few thousand assets, and the episode filter joined EntityLink on top, duplicating rows again. Assets, tasks and assignee links are now three narrow flat queries; episode casting and assigned-to become EXISTS filters so nothing multiplies rows, the sort only covers assets and no ORM instance is built per row. Output is byte-identical (episode filter now covered by a dedicated test). Tests: tests/assets, tests/services/test_assets_service.py, tests/export.
Profiling on a real 5k assets / 75k tasks project showed the remaining time was dominated by Python assembly, not SQL: 455k serialize_value calls (type-dispatch for values that are always DateTime) and 385k uuid.__str__ (plus the UUID result processor) per response. Task and link uuids are now cast to text by PostgreSQL and the six task dates go through a specialized serializer. Bench on that project (median of 4 runs, subscriptions mocked): 2.03s before the branch, 1.13s after (-45%), byte-identical payload. Tests: tests/assets, tests/services/test_assets_service.py, tests/export.
Full-project views (up to 10k entities for series) are fetched constantly by Kitsu and must stay unpaginated because the client operates on the whole list, so the response cost has to drop instead: - compact=true encodes assets and tasks as positional value arrays with the field names in the response header (self-describing: clients map by name, adding a field cannot break them). Halves the raw payload. - stream=true sends NDJSON (header line then one asset per line) through a generator, so the response is never held in memory. The service exposes prepare_assets_and_tasks(), which does all DB and request-bound work eagerly and returns a pure-formatting generator that a streaming response can consume outside the request context. get_assets_and_tasks() keeps its exact output for other callers. Measured on the 5k assets / 75k tasks project, per request: - peak memory: 132MB -> 58MB with stream (any encoding) - raw payload: 35MB -> 16MB with compact (gzip input halved too) - time: unchanged (~1s, after the -45% of the previous commits) Tests: tests/assets (3 new roundtrip tests), tests/export, tests/services/test_assets_service.py.
Same treatment as assets for the all-shots view (up to 10k shots for series, fetched constantly by Kitsu): three narrow flat queries with EXISTS filters instead of the Entity x Task x TaskPersonLink join, no per-row ORM or generic serialization, uuid::text cast in SQL, and the Kitsu-oriented stream (NDJSON, memory-flat) and compact (self-describing positional arrays) response options. The shared date serializer moves to fields.serialize_datetime and the assets service now uses it too. Default output stays byte-identical. Measured on the 2k shots / 40k tasks project, per request: - time: 1.68s -> 0.59s (-65%) - stream+compact: 0.43s, peak memory 100MB -> 33MB, payload 21MB -> 9MB Tests: tests/shots (3 new roundtrip tests), tests/services/test_shots_service.py, tests/assets.
The episodes/sequences/scenes, edits and concepts with-tasks builders were near-identical copies of the same row-multiplying join. They now delegate the task and assignee fetching to entities_service.fetch_entity_task_map: two narrow flat queries with uuid::text casts, grouped by entity, plus a task-dict builder driven by an explicit per-view field list so every response keeps its exact current shape (no field added or removed, per-view assigned_to semantics preserved, concept links fetched flat). Assets and shots keep their specialized builders (compact/stream field contracts); this core is the template to fold them in later (ARCH-3). Tests: tests/entities, tests/edits, tests/concepts, tests/shots, tests/export, tests/breakdown.
frankrousseau
force-pushed
the
perf-assets-and-tasks
branch
from
0254c13 to
47e7c05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Solution
Six roundtrip tests assert stream/compact reassemble exactly to the default output; all existing suites pass unchanged.