Skip to content

perf: SW cache for indexed reads + denormalised image dimensions#3495

Open
karlitschek wants to merge 1 commit into
frank/feat/photos-preview-warmupfrom
frank/feat/photos-perf-sw-and-denorm
Open

perf: SW cache for indexed reads + denormalised image dimensions#3495
karlitschek wants to merge 1 commit into
frank/feat/photos-preview-warmupfrom
frank/feat/photos-perf-sw-and-denorm

Conversation

@karlitschek

Copy link
Copy Markdown
Member

Two related performance changes that show up most when navigating between Photos / Videos / Favorites tabs and on big libraries.

Service-worker runtime cache (vite.config.ts)

  • The SW was already registered with one CacheFirst rule for the preview endpoint. Added strategies for the indexed-read paths the perf-index branch introduced: /api/v1/index/timeline → StaleWhileRevalidate (200 entries, 6h TTL). Returns the previous response instantly for tab transitions while the SW kicks off a fresh fetch in the background — feels "instant" on repeat visits. /api/v1/index/search → NetworkFirst with 4s timeout fall-back (50 entries, 30s TTL). Search wants freshness; the cache is only there to keep the previous results visible while the new query lands. /api/v1/index/status → NetworkFirst with 3s timeout fall-back (4 entries, 60s TTL). Migration banner needs near-real-time updates while the backfill runs. /api/v1/transcode/{id}/seg-N.ts → CacheFirst (2000 entries, 30d TTL). Segments are content-addressed so safe to cache aggressively.
  • Existing preview cache rule generalised to also match publicPreview (was missing on shared-album views).

Denormalised image dimensions on oc_photos_index

  • New migration adds width / height (nullable INT UNSIGNED).
  • PhotoIndexService now reads photos-size via IFilesMetadataManager at index time and writes both columns alongside the existing taken_at. NULL until metadata is ready — the metadata pipeline is async, so a freshly-written file may hit the listener before its size is computed; a subsequent NodeWritten / backfill pass fills the columns.
  • extractTakenAt now delegates to a shared extractIndexFields so the metadata read path is one method, not two slightly- different ones drifting over time.
  • Mapper SELECTs (getEnrichedTimelineForUser, searchUserTimeline) return width/height; the GROUP BY adds them to the keyed columns.
  • IndexController.composeItem reads photos-size from the row first, falls back to the metadata blob for old rows that haven't been re-indexed since the migration. Frontend keeps reading attributes['metadata-photos-size'].width/height — same shape in the response, just sourced from the index table.

Bumped app version to 7.0.0-dev.3 so occ upgrade re-runs the migration loader.

Two related performance changes that show up most when navigating
between Photos / Videos / Favorites tabs and on big libraries.

Service-worker runtime cache (vite.config.ts)
- The SW was already registered with one CacheFirst rule for the
  preview endpoint. Added strategies for the indexed-read paths
  the perf-index branch introduced:
    /api/v1/index/timeline → StaleWhileRevalidate (200 entries,
       6h TTL). Returns the previous response instantly for tab
       transitions while the SW kicks off a fresh fetch in the
       background — feels "instant" on repeat visits.
    /api/v1/index/search → NetworkFirst with 4s timeout fall-back
       (50 entries, 30s TTL). Search wants freshness; the cache is
       only there to keep the previous results visible while the
       new query lands.
    /api/v1/index/status → NetworkFirst with 3s timeout fall-back
       (4 entries, 60s TTL). Migration banner needs near-real-time
       updates while the backfill runs.
    /api/v1/transcode/{id}/seg-N.ts → CacheFirst (2000 entries,
       30d TTL). Segments are content-addressed so safe to cache
       aggressively.
- Existing preview cache rule generalised to also match
  `publicPreview` (was missing on shared-album views).

Denormalised image dimensions on `oc_photos_index`
- New migration adds `width` / `height` (nullable INT UNSIGNED).
- `PhotoIndexService` now reads `photos-size` via
  `IFilesMetadataManager` at index time and writes both columns
  alongside the existing `taken_at`. NULL until metadata is ready
  — the metadata pipeline is async, so a freshly-written file may
  hit the listener before its size is computed; a subsequent
  NodeWritten / backfill pass fills the columns.
- `extractTakenAt` now delegates to a shared `extractIndexFields`
  so the metadata read path is one method, not two slightly-
  different ones drifting over time.
- Mapper SELECTs (`getEnrichedTimelineForUser`, `searchUserTimeline`)
  return width/height; the GROUP BY adds them to the keyed columns.
- `IndexController.composeItem` reads `photos-size` from the row
  first, falls back to the metadata blob for old rows that haven't
  been re-indexed since the migration. Frontend keeps reading
  `attributes['metadata-photos-size'].width/height` — same shape
  in the response, just sourced from the index table.

Bumped app version to `7.0.0-dev.3` so `occ upgrade` re-runs the
migration loader.

Signed-off-by: Frank Karlitschek <frank@nextcloud.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@karlitschek

karlitschek commented May 3, 2026

Copy link
Copy Markdown
Member Author

@codecov

codecov Bot commented May 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant