Feature/cluster representative images

README.md

@@ -30,6 +30,39 @@ Visual exploration and clustering tool for image embeddings. Users can either br
 
 **Precalculated Embeddings** - Load parquet files (or directories of parquets) with precomputed embeddings, apply dynamic cascading filters, and explore clusters with taxonomy tree navigation. See [Data Format](docs/DATA_FORMAT.md) for the expected schema and [Backend Pipeline](docs/BACKEND_PIPELINE.md) for how embeddings flow through clustering and visualization.
 
+**Representative Images** - After KMeans, both apps show the three records closest to each cluster's centroid. Each tile carries the record name, a short id that copies the full id on click, and a source link that opens the image in a new tab; images that fail to load are skipped in favor of the next-closest record, and images over the 25 MiB download cap are shown as a placeholder with their source link intact. The example below is KMeans on [BioCLIP 2](https://huggingface.co/imageomics/bioclip-2) embeddings of a sample of canid images. See [Representative Images](docs/REPRESENTATIVE_IMAGES.md) for how selection and fallback work.
+
+<table>
+  <tr>
+    <td colspan="3"><b>Cluster 1</b></td>
+  </tr>
+  <tr>
+    <td>Canis simensis<br><code>dbb9ca8f</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/334339733/original.jpg" target="_blank">source</a></td>
+    <td>Canis simensis<br><code>70f9ab9f</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/134581592/original.jpeg" target="_blank">source</a></td>
+    <td>Canis simensis<br><code>4de7a1cf</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/345255557/original.jpg" target="_blank">source</a></td>
+  </tr>
+  <tr>
+    <td><img src="docs/images/readme_rep_c1_1.png" alt="Ethiopian wolf" width="180"></td>
+    <td><img src="docs/images/readme_rep_c1_2.png" alt="Ethiopian wolf" width="180"></td>
+    <td><img src="docs/images/readme_rep_c1_3.png" alt="Ethiopian wolf" width="180"></td>
+  </tr>
+  <tr>
+    <td colspan="3"><b>Cluster 2</b></td>
+  </tr>
+  <tr>
+    <td>Canis lycaon<br><code>ea070e30</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/176686074/original.jpeg" target="_blank">source</a></td>
+    <td>Canis lycaon<br><code>55af17f4</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/176686067/original.jpeg" target="_blank">source</a></td>
+    <td>Canis lycaon<br><code>a7edd26f</code> · <a href="https://inaturalist-open-data.s3.amazonaws.com/photos/104417928/original.jpg" target="_blank">source</a></td>
+  </tr>
+  <tr>
+    <td><img src="docs/images/readme_rep_c2_1.png" alt="paw print in snow" width="180"></td>
+    <td><img src="docs/images/readme_rep_c2_2.png" alt="paw print in snow" width="180"></td>
+    <td><img src="docs/images/representatives_tile_too_large.png" alt="placeholder shown for an image over the download cap" width="180"></td>
+  </tr>
+</table>
+
+<sub>Photos [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/deed.en): Björn Reese, Miguel A. Casado, Bill Fintel, corium.</sub>
+
 ## Installation
 
 Requires Python 3.10–3.13. We recommend using [uv](https://github.com/astral-sh/uv) for package management.

apps/precalculated/app.py

@@ -26,7 +26,10 @@ def app():
         render_projection_section,
         render_kmeans_section,
     )
-    from apps.precalculated.components.data_preview import render_data_preview
+    from apps.precalculated.components.data_preview import (
+        render_data_preview,
+        render_cluster_representatives,
+    )
     from shared.components.visualization import render_scatter_plot
     from shared.components.summary import render_clustering_summary
 
@@ -71,9 +74,10 @@ def app():
     with col_preview:
         render_data_preview()
 
-    # Bottom: Taxonomy summary
+    # Bottom: Taxonomy summary + representative images
     st.markdown("---")
     render_clustering_summary(show_taxonomy=True)
+    render_cluster_representatives()
 
 
 if __name__ == "__main__":

apps/precalculated/components/data_preview.py

@@ -6,80 +6,20 @@
 import streamlit as st
 import pandas as pd
 import numpy as np
-import requests
-import time
-from typing import Optional
-from PIL import Image
-from io import BytesIO
 
 from shared.utils.logging_config import get_logger
+from shared.utils.representatives import find_cluster_representatives
+from shared.utils.images import (
+    ImageTooLarge,
+    fetch_images_concurrent,
+    get_record_image,
+    resolve_record_image_url,
+)
+from shared.components.representatives import render_representative_images
 
 logger = get_logger(__name__)
 
 
-@st.cache_data(ttl=300, show_spinner=False)
-def _fetch_image_from_url_cached(url: str, timeout: int = 5) -> Optional[bytes]:
-    """Internal cached function to fetch image bytes."""
-    if not url or not isinstance(url, str):
-        return None
-
-    try:
-        if not url.startswith(('http://', 'https://')):
-            return None
-
-        response = requests.get(url, timeout=timeout, stream=True)
-        response.raise_for_status()
-
-        content_type = response.headers.get('content-type', '').lower()
-        if not content_type.startswith('image/'):
-            return None
-
-        return response.content
-
-    except Exception:
-        return None
-
-
-def fetch_image_from_url(url: str, timeout: int = 5) -> Optional[bytes]:
-    """
-    Fetch an image from a URL with logging.
-    Uses caching internally but logs the request.
-    """
-    if not url or not isinstance(url, str):
-        return None
-
-    if not url.startswith(('http://', 'https://')):
-        logger.warning(f"[Image] Invalid URL scheme: {url[:50]}...")
-        return None
-
-    logger.info(f"[Image] Fetching: {url[:80]}...")
-    start_time = time.time()
-
-    result = _fetch_image_from_url_cached(url, timeout)
-
-    elapsed = time.time() - start_time
-    if result:
-        logger.info(f"[Image] Loaded: {len(result)/1024:.1f}KB in {elapsed:.3f}s")
-    else:
-        logger.warning(f"[Image] Failed to load: {url[:50]}...")
-
-    return result
-
-
-def get_image_from_url(url: str) -> Optional[Image.Image]:
-    """Get image from URL with caching and logging."""
-    image_bytes = fetch_image_from_url(url)
-    if image_bytes:
-        try:
-            image = Image.open(BytesIO(image_bytes))
-            logger.info(f"[Image] Opened: {image.size[0]}x{image.size[1]} {image.mode}")
-            return image
-        except Exception as e:
-            logger.error(f"[Image] Failed to open: {e}")
-            return None
-    return None
-
-
 def render_data_preview():
     """Render the data preview panel (record details on point click)."""
     df_plot = st.session_state.get("data", None)
@@ -110,15 +50,17 @@ def render_data_preview():
 
         st.markdown("### Record Details")
 
-        # Try to display image if identifier/url column exists (cached to prevent re-fetch)
-        image_cols = ['identifier', 'image_url', 'url', 'img_url', 'image']
-        for img_col in image_cols:
-            if img_col in record.index and pd.notna(record[img_col]):
-                url = record[img_col]
-                image = get_image_from_url(url)
-                if image is not None:
-                    st.image(image, width=280)
-                    break
+        # Try to display an image, falling back across the record's URL
+        # columns until one loads (process-cached). An over-cap image keeps
+        # its URL visible so the user can still open it in a browser.
+        image = get_record_image(record)
+        if isinstance(image, ImageTooLarge):
+            st.warning(
+                f"Image not downloaded — {image.describe()} exceeds the fetch limit."
+            )
+            st.markdown(f"[Open original image]({image.url})")
+        elif image is not None:
+            st.image(image, width=280)
 
         st.markdown(f"**UUID:** `{selected_uuid}`")
 
@@ -276,3 +218,86 @@ def render_cluster_analysis():
         st.code(tree_output, language="text")
     else:
         st.info(f"No valid '{color_by}' values to compare with KMeans clusters.")
+
+
+def render_cluster_representatives():
+    """Render representative images per KMeans cluster for the precalculated app.
+
+    Representatives are the members closest to each cluster centroid (computed
+    on the full-dimensional embeddings). Images are fetched from each record's
+    URL column; URLs that fail to load are skipped and the next-closest
+    candidate is tried (fallback), so transient/broken URLs don't leave gaps.
+    """
+    df_plot = st.session_state.get("data", None)
+    embeddings = st.session_state.get("embeddings", None)
+    if df_plot is None or embeddings is None:
+        return
+
+    kmeans_cols = sorted(
+        [c for c in df_plot.columns if c.startswith("KMeans (k=")],
+        key=lambda c: int(c.split("=")[1].rstrip(")")),
+    )
+    if not kmeans_cols:
+        return  # nothing to show until a KMeans run exists
+
+    st.markdown("### Representative Images")
+    st.caption(
+        "Members closest to each cluster centroid. Images load from each "
+        "record's URL; unreachable images are skipped automatically."
+    )
+
+    selected_col = st.selectbox(
+        "KMeans result",
+        options=kmeans_cols,
+        index=len(kmeans_cols) - 1,
+        key="representatives_kmeans_selector",
+        help="Which KMeans run to show representatives for.",
+    )
+
+    # Guard: embeddings must align row-for-row with df_plot.
+    if len(embeddings) != len(df_plot):
+        st.info("Re-run projection and KMeans to view representatives.")
+        return
+
+    n_per_cluster = 3
+    representatives = find_cluster_representatives(
+        embeddings, df_plot[selected_col].values, n_per_cluster=n_per_cluster
+    )
+
+    # Warm the cache concurrently. Representatives are oversampled for fallback,
+    # but we only need a few successes per cluster — prefetch a prefix (2x the
+    # display count, first URL per record) in parallel. Deeper fallbacks (later
+    # candidates, or a record's alternate URL columns) resolve on-demand below.
+    prefetch_per_cluster = n_per_cluster * 2
+    prefetch_urls = [
+        resolve_record_image_url(df_plot.iloc[idx])
+        for idxs in representatives.values()
+        for idx in idxs[:prefetch_per_cluster]
+    ]
+    with st.spinner("Loading representative images..."):
+        fetch_images_concurrent([u for u in prefetch_urls if u])
+
+    def _resolve(idx):
+        # Prefetched URLs hit the process cache inside get_record_image; broken
+        # URLs fall back to the record's next URL column, and anything not
+        # prefetched falls back to a single synchronous fetch (also cached).
+        return get_record_image(df_plot.iloc[idx])
+
+    def _caption(idx):
+        row = df_plot.iloc[idx]
+        for col in ("scientific_name", "species", "common_name", "uuid"):
+            if col in row.index and pd.notna(row[col]):
+                return str(row[col])
+        return None
+
+    def _details(idx):
+        row = df_plot.iloc[idx]
+        return {"id": str(row["uuid"]), "url": resolve_record_image_url(row)}
+
+    render_representative_images(
+        representatives,
+        resolve_image=_resolve,
+        n_per_cluster=n_per_cluster,
+        caption_fn=_caption,
+        details_fn=_details,
+    )

docs/REPRESENTATIVE_IMAGES.md

@@ -0,0 +1,107 @@
+# Representative images per cluster
+
+After a KMeans run, both apps show a **Representative Images** panel: for every cluster, up to three images of the records that sit closest to the cluster's center. This page explains how those records are chosen, how their images are loaded, what happens when an image cannot be shown.
+
+## 1. Overview: What the panel shows
+
+Each cluster is one card. Inside it, every tile has a short header above the image:
+
+- the record's name (`scientific_name`, else `species`, `common_name`, or the uuid; in Embed & Explore the tile shows a single `File path` label instead),
+- a **short id** that copies the full uuid (or file path) to the clipboard when clicked; hovering previews the full value,
+- a **source** link that opens the fetched image URL in a new tab.
+
+The example card below mixes two rendered images with one that was too large to download. The third tile is the app's own placeholder, drawn by the same function the app uses, and its "source" link still opens the original.
+
+<table>
+  <tr>
+    <td width="33%" align="center"><b>Cluster 2</b></td>
+    <td width="33%"></td>
+    <td width="33%"></td>
+  </tr>
+  <tr>
+    <td>Sombrero Galaxy<br><code>3f1c9a0b</code> · <a href="https://cdn.esahubble.org/archives/images/large/heic0407a.jpg">source</a></td>
+    <td>Horsehead Nebula<br><code>9a77e2d1</code> · <a href="https://cdn.esahubble.org/archives/images/large/heic1307a.jpg">source</a></td>
+    <td>Cosmic Reef<br><code>51be0c3e</code> · <a href="https://cdn.esahubble.org/archives/images/large/heic2007a.jpg">source</a></td>
+  </tr>
+  <tr>
+    <td><img src="images/representatives_tile_1.png" alt="rendered image" width="100%"></td>
+    <td><img src="images/representatives_tile_2.png" alt="rendered image" width="100%"></td>
+    <td><img src="images/representatives_tile_too_large.png" alt="placeholder for an image over the download cap" width="100%"></td>
+  </tr>
+</table>
+
+<sub>Example images: ESA/Hubble (CC BY 4.0). The third source is a 97 MiB file, above the app's 25 MiB download cap, so the app shows the placeholder instead of fetching it.</sub>
+
+A cluster whose records all fail to load shows the caption *"No images available for this cluster."* instead of tiles.
+
+## 2. How representatives are chosen
+
+For each cluster, the app takes the mean of its members' embeddings (the **centroid**, computed in the full embedding space, not in the 2D projection) and ranks the members by squared Euclidean distance to it. The closest three are the representatives.
+
+In symbols: let $x_i \in \mathbb{R}^D$ be the embedding of record $i$ (for BioCLIP 2, $D = 768$) and $C_k = \{\, i : \text{label}_i = k \,\}$ the members of cluster $k$. The centroid and each member's distance to it are
+
+$$
+\mu_k = \frac{1}{|C_k|} \sum_{i \in C_k} x_i,
+\qquad
+d_i = \lVert x_i - \mu_k \rVert^2 = \sum_{j=1}^{D} (x_{ij} - \mu_{kj})^2 .
+$$
+
+Sorting the members of $C_k$ by $d_i$ ascending gives the ranked list $r_1, r_2, \dots$; the app keeps the first
+
+$$
+K = \min\big(n \cdot o,\; |C_k|\big), \qquad n = 3,\; o = 4,
+$$
+
+as **candidates**, and the **representatives shown** are the first $n$ of those whose image can be displayed. With every image loadable they are simply $r_1, r_2, r_3$.
+
+The ranking is **oversampled** by the factor $o$: the app keeps 4 times as many candidates as it intends to show (12 for 3 slots, or the whole cluster if it is smaller). The extra candidates are backups, used only when an image cannot be shown. Representatives therefore never change because of a broken URL: the same three records are tried first every time; a failure moves on to the fourth-closest, and so on.
+
+Two details of the implementation: ties in $d_i$ are broken by record order, and $\mu_k$ is computed on the embeddings as stored in the dataset, whereas KMeans itself clusters L2-normalized copies, so for unnormalized embeddings a member's rank can differ slightly from its rank by distance to the normalized centroid.
+
+<img src="images/representatives_ranking.png" alt="ranking of cluster members by distance to the centroid" width="720">
+
+Two consequences worth knowing:
+
+- Representatives depend only on the embeddings and the KMeans labels, not on the projection method; PCA, t-SNE and UMAP views of the same KMeans run share the same representatives.
+- A cluster with fewer than three loadable images shows fewer tiles rather than borrowing from another cluster.
+
+## 3. How images are loaded, and what happens when they cannot be
+
+```mermaid
+flowchart TD
+    A[KMeans labels + embeddings] --> B[rank members per cluster<br/>closest to centroid first, keep 12]
+    B --> C[prefetch the first URL of the first 6 candidates per cluster<br/>8 threads, results cached per process]
+    C --> D{for each candidate, in rank order}
+    D --> E[walk the record's URL columns in order<br/>identifier, image_url, url, img_url, image]
+    E --> F{fetch one URL}
+    F -->|HTTP error, not an image,<br/>undecodable| G[try next column]
+    F -->|Content-Length or stream<br/>exceeds 25 MiB| H[remember as too large,<br/>try next column]
+    F -->|image decodes| I[image for this record]
+    G --> J{columns left?}
+    H --> J
+    J -->|yes| E
+    J -->|no, and a too-large URL was seen| K[too large for this record]
+    J -->|no| L[no image for this record:<br/>skip, slot not consumed]
+    I --> M{3 images collected?}
+    K --> M
+    L --> M
+    M -->|not yet, candidates left| D
+    M -->|done| N[render: images first,<br/>too-large placeholders fill empty slots,<br/>nothing at all: No images available]
+```
+
+In words, there are two fallback layers:
+
+1. **Per record, across URL columns.** A record may hold image URLs in several columns (`identifier`, `image_url`, `url`, `img_url`, `image`; see [`shared/utils/images.py`](../shared/utils/images.py)). The app tries them in that order and uses the first one that actually loads. A broken column moves on to the next. A column whose image is over the size cap also moves on; it is remembered only if no column loads.
+2. **Per cluster, across candidates.** The renderer walks the ranked candidates until it has three real images. A record with nothing loadable is skipped and consumes no slot. A record whose best outcome is "too large" is set aside; real images fill the slots first, and the set-aside records fill only the slots that would otherwise stay empty, as placeholder tiles. If there is nothing at all, the cluster shows the empty caption.
+
+<img src="images/representatives_candidate_walk.png" alt="how the renderer walks the ranked candidates" width="900">
+
+### Why there is a size cap
+
+The cap originates from a [GitHub Copilot review comment on the feature PR](https://github.com/Imageomics/emb-explorer/pull/42#discussion_r3598642716), which pointed out that the first version read every response into memory with no limit, so a malicious or misconfigured URL returning a very large payload could drive memory use up and stall the app. The app now refuses any image larger than 25 MiB: it checks the declared Content-Length before reading a single byte, and it also counts bytes while streaming and stops as soon as the limit is passed, since a server can omit or misstate Content-Length.
+
+Every fetch result is cached for the lifetime of the app process, including failures and too-large verdicts, so switching views or re-running KMeans does not refetch, and a dead URL is tried at most once per session. Fetches time out after 5 seconds.
+
+## 4. Known limits
+
+HTTP 429 (rate limiting) and 503 are treated like any other failure, with no retry. Hosts that throttle the app's User-Agent, Wikimedia Commons for example, can make a cluster look empty.

pyproject.toml

@@ -37,6 +37,7 @@ dependencies = [
     "pandas>=2.0.0",
     "pillow>=9.0.0",
     "pyarrow>=10.0.0",
+    "requests>=2.28.0",
     # Visualization
     "altair>=5.0.0",
     # Machine learning
@@ -97,12 +98,6 @@ emb-embed-explore = "apps.embed_explore.app:main"
 emb-precalculated = "apps.precalculated.app:main"
 list-models = "shared.utils.models:print_available_models"
 
-[tool.hatch.version]
-path = "shared/__init__.py"
-
-[tool.hatch.metadata]
-allow-direct-references = true
-
 [tool.hatch.build.targets.wheel]
 packages = ["shared", "apps"]