perf: SW cache for indexed reads + denormalised image dimensions#3495
Open
karlitschek wants to merge 1 commit into
Open
perf: SW cache for indexed reads + denormalised image dimensions#3495karlitschek wants to merge 1 commit into
karlitschek wants to merge 1 commit into
Conversation
Two related performance changes that show up most when navigating
between Photos / Videos / Favorites tabs and on big libraries.
Service-worker runtime cache (vite.config.ts)
- The SW was already registered with one CacheFirst rule for the
preview endpoint. Added strategies for the indexed-read paths
the perf-index branch introduced:
/api/v1/index/timeline → StaleWhileRevalidate (200 entries,
6h TTL). Returns the previous response instantly for tab
transitions while the SW kicks off a fresh fetch in the
background — feels "instant" on repeat visits.
/api/v1/index/search → NetworkFirst with 4s timeout fall-back
(50 entries, 30s TTL). Search wants freshness; the cache is
only there to keep the previous results visible while the
new query lands.
/api/v1/index/status → NetworkFirst with 3s timeout fall-back
(4 entries, 60s TTL). Migration banner needs near-real-time
updates while the backfill runs.
/api/v1/transcode/{id}/seg-N.ts → CacheFirst (2000 entries,
30d TTL). Segments are content-addressed so safe to cache
aggressively.
- Existing preview cache rule generalised to also match
`publicPreview` (was missing on shared-album views).
Denormalised image dimensions on `oc_photos_index`
- New migration adds `width` / `height` (nullable INT UNSIGNED).
- `PhotoIndexService` now reads `photos-size` via
`IFilesMetadataManager` at index time and writes both columns
alongside the existing `taken_at`. NULL until metadata is ready
— the metadata pipeline is async, so a freshly-written file may
hit the listener before its size is computed; a subsequent
NodeWritten / backfill pass fills the columns.
- `extractTakenAt` now delegates to a shared `extractIndexFields`
so the metadata read path is one method, not two slightly-
different ones drifting over time.
- Mapper SELECTs (`getEnrichedTimelineForUser`, `searchUserTimeline`)
return width/height; the GROUP BY adds them to the keyed columns.
- `IndexController.composeItem` reads `photos-size` from the row
first, falls back to the metadata blob for old rows that haven't
been re-indexed since the migration. Frontend keeps reading
`attributes['metadata-photos-size'].width/height` — same shape
in the response, just sourced from the index table.
Bumped app version to `7.0.0-dev.3` so `occ upgrade` re-runs the
migration loader.
Signed-off-by: Frank Karlitschek <frank@nextcloud.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related performance changes that show up most when navigating between Photos / Videos / Favorites tabs and on big libraries.
Service-worker runtime cache (vite.config.ts)
publicPreview(was missing on shared-album views).Denormalised image dimensions on
oc_photos_indexwidth/height(nullable INT UNSIGNED).PhotoIndexServicenow readsphotos-sizeviaIFilesMetadataManagerat index time and writes both columns alongside the existingtaken_at. NULL until metadata is ready — the metadata pipeline is async, so a freshly-written file may hit the listener before its size is computed; a subsequent NodeWritten / backfill pass fills the columns.extractTakenAtnow delegates to a sharedextractIndexFieldsso the metadata read path is one method, not two slightly- different ones drifting over time.getEnrichedTimelineForUser,searchUserTimeline) return width/height; the GROUP BY adds them to the keyed columns.IndexController.composeItemreadsphotos-sizefrom the row first, falls back to the metadata blob for old rows that haven't been re-indexed since the migration. Frontend keeps readingattributes['metadata-photos-size'].width/height— same shape in the response, just sourced from the index table.Bumped app version to
7.0.0-dev.3soocc upgradere-runs the migration loader.