perf: add pg_trgm GIN indexes for backoffice organization search by pieterbeulque · Pull Request #14325 · polarsource/polar

pieterbeulque · 2026-09-09T19:50:30Z

Problem

The backoffice organization search (server/polar/backoffice/organizations_v2/endpoints.py) filters name, slug and email with a leading-wildcard ILIKE '%q%':

search_term = f"%{q}%"
stmt = stmt.where(
    or_(
        Organization.name.ilike(search_term),
        Organization.slug.ilike(search_term),
        Organization.email.ilike(search_term),
    )
)

The leading % means no btree index can serve the predicate, so Postgres does a full sequential scan of the now-large organizations table. One recent search hit the 30s statement timeout and returned a 500.

Fix

Add pg_trgm GIN trigram indexes on organizations.name, organizations.slug and organizations.email. Trigram GIN indexes make ILIKE '%term%' index-backed, turning the seq-scan into an index scan.

New migration ensures the extension exists (CREATE EXTENSION IF NOT EXISTS pg_trgm) and creates the three indexes with gin_trgm_ops.
Indexes are built with CREATE INDEX CONCURRENTLY (via op.get_context().autocommit_block(), the same pattern the repo already uses), so the deploy is non-blocking — no table lock in production. A previously-interrupted concurrent build is cleaned up first (DROP INDEX ... IF EXISTS CONCURRENTLY) so re-runs are safe.
downgrade drops the three indexes (also CONCURRENTLY / IF EXISTS) but intentionally leaves the pg_trgm extension in place, since other objects may rely on it.
The indexes are also declared on the Organization model's __table_args__ and pg_trgm is registered as a tracked alembic_utils PGExtension (alongside citext / uuid-ossp), matching how the codebase tracks GIN indexes and extensions. uv run alembic check reports no drift.

The endpoint query is unchanged — the ILIKE stays as-is; the indexes just make it fast.

Ship safety

CREATE INDEX CONCURRENTLY cannot run inside a transaction, and Alembic wraps migrations in one by default — handled here with autocommit_block. Per the repo's ship-safety conventions this migration should ship on its own so the concurrent index build runs outside a transaction.

Verification

uv run alembic upgrade head applies cleanly against the local DB; the three GIN indexes and the pg_trgm extension are created.
downgrade removes all three indexes; re-upgrade restores them (round-trip tested).
uv run alembic check → no new operations. uv run task lint and mypy on the touched files pass.

🤖 Generated with Claude Code

vercel · 2026-09-09T19:50:36Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
orbit	Ready	Preview	Sep 10, 2026 1:05pm UTC
polar-test	Ready	Preview	Sep 10, 2026 1:05pm UTC

github-actions · 2026-09-09T19:51:21Z

⚠️ Migration Isolation Check Failed

This PR contains database migrations along with other code changes. To ensure safe deployments, please split this into separate PRs,
or verify that your changes will not break. Keep in mind that the API is deployed before the workers.

Migration PR: Only model changes and the migration file
Code PR: All other changes (can be merged after migration PR)

Files that should be in a separate PR:

server/polar/backoffice/customers/endpoints.py
server/polar/backoffice/orders/endpoints.py
server/polar/backoffice/organizations_v2/endpoints.py
server/polar/backoffice/organizations_v2/views/list_view.py
server/polar/backoffice/products/endpoints.py
server/polar/backoffice/search.py
server/polar/backoffice/subscriptions/endpoints.py
server/polar/backoffice/webhooks/endpoints.py
server/polar/kit/db/models/base.py
server/tests/backoffice/organizations_v2/test_endpoints.py

Why?

Migrations are deployed separately and run before code changes. Mixing them can cause deployment issues if the new code depends on the migration.

github-actions · 2026-09-09T19:51:45Z

OpenAPI Changes

No changes detected in the OpenAPI schema.

cubic-dev-ai

2 issues found across 3 files

Confidence score: 3/5

server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py does not make one- or two-character organization searches index-backed, so short queries may remain slow; enforce a minimum search length before running the query.
server/polar/models/organization.py leaves Organization.slug searches unable to use the trigram index because CITEXT selects a different operator; create the index with the compatible text/operator expression and verify the generated query plan.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py">

<violation number="1" location="server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py:55">
P2: When `q` is one or two characters, these indexes do not make the organization search index-backed because `pg_trgm` cannot extract a trigram from the pattern. Enforce a minimum search length before running this query or add a separate strategy for short searches.</violation>
</file>

<file name="server/polar/models/organization.py">

<violation number="1" location="server/polar/models/organization.py:542">
P2: Slug searches still cannot use this trigram index because `Organization.slug` is `CITEXT`, so `ilike()` resolves to citext's operator rather than the `text` operator in `gin_trgm_ops`. Create the trigram index on `slug::text` and cast the slug expression to `text` in the backoffice predicate (updating both the migration and ORM declaration).</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-09-09T19:55:29Z

+                [column],
+                postgresql_concurrently=True,
+                postgresql_using="gin",
+                postgresql_ops={column: "gin_trgm_ops"},


P2: When q is one or two characters, these indexes do not make the organization search index-backed because pg_trgm cannot extract a trigram from the pattern. Enforce a minimum search length before running this query or add a separate strategy for short searches.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py, line 55: <comment>When `q` is one or two characters, these indexes do not make the organization search index-backed because `pg_trgm` cannot extract a trigram from the pattern. Enforce a minimum search length before running this query or add a separate strategy for short searches.</comment> <file context> @@ -0,0 +1,68 @@ + [column], + postgresql_concurrently=True, + postgresql_using="gin", + postgresql_ops={column: "gin_trgm_ops"}, + ) + </file context>

cubic-dev-ai · 2026-09-09T19:55:29Z

+            postgresql_ops={"name": "gin_trgm_ops"},
+        ),
+        Index(
+            "ix_organizations_slug_trgm",


P2: Slug searches still cannot use this trigram index because Organization.slug is CITEXT, so ilike() resolves to citext's operator rather than the text operator in gin_trgm_ops. Create the trigram index on slug::text and cast the slug expression to text in the backoffice predicate (updating both the migration and ORM declaration).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/polar/models/organization.py, line 542: <comment>Slug searches still cannot use this trigram index because `Organization.slug` is `CITEXT`, so `ilike()` resolves to citext's operator rather than the `text` operator in `gin_trgm_ops`. Create the trigram index on `slug::text` and cast the slug expression to `text` in the backoffice predicate (updating both the migration and ORM declaration).</comment> <file context> @@ -530,6 +530,26 @@ class Organization(RateLimitGroupMixin, RecordModel): + postgresql_ops={"name": "gin_trgm_ops"}, + ), + Index( + "ix_organizations_slug_trgm", + "slug", + postgresql_using="gin", </file context>

cubic-dev-ai

1 existing issue remains and no new issues found across 5 files (changes from recent commits).

Confidence score: 4/5

server/polar/models/organization.py adds an index on (slug::text), but existing backoffice Organization.slug.ilike(...) predicates may not use it, reducing search performance; update those predicates to cast Organization.slug to Text consistently.

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic}

cubic-dev-ai

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic}

cubic-dev-ai

0 issues found across 9 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic}

The backoffice organization search filters name/slug/email with a leading-wildcard `ILIKE '%q%'`, which no btree index can serve, so it seq-scans the now-large organizations table. One recent search hit the 30s statement timeout and returned a 500. Add pg_trgm GIN trigram indexes on name, slug and email so those `ILIKE` predicates become index-backed. The indexes are built CONCURRENTLY so the deploy is non-blocking, and pg_trgm is registered as a tracked alembic_utils entity alongside citext/uuid-ossp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two follow-ups on the pg_trgm indexes: `slug` is CITEXT, so `slug ILIKE '%q%'` resolves to citext's `~~*` operator instead of the `text` one in `gin_trgm_ops` — the index was never eligible and the slug branch of the search kept seq-scanning. Index the `slug::text` expression and cast the column the same way in the search predicate. Verified with EXPLAIN: the three-way OR now bitmap-ORs all three trigram indexes, where the uncast predicate seq-scans even with `enable_seqscan = off`. pg_trgm also can't extract a trigram from a one- or two-character pattern, so short searches fell back to the same full scan — and the search box fires on every keystroke. Require three characters and show a "Keep Typing" hint instead of running the query. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGuVPEaf3zemquZPN39rFx

`lazy_counts_url` forwards the raw query params, so a whitespace-only `q` reached `status_counts` unnormalized: the list treated it as no search and excluded deleted organizations, while the tab counts saw a truthy `q` and included them. Share the normalization between both endpoints so they always derive the same deleted filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGuVPEaf3zemquZPN39rFx

The CITEXT slug trap wasn't limited to the organization list: every backoffice search that matches an organization by name or slug had the same uncast `slug ILIKE`, and because the OR's slug branch has no index path, the planner couldn't use the name index either. Move the predicate into `backoffice/search.py` so the cast lives in one place, and use it from the organization, order, customer, product, subscription and webhook searches. The order search gains an index-backed plan (its organization filter is its own AND-ed clause): EXPLAIN ANALYZE over 60k organizations goes from a seq scan discarding 59,999 rows to a BitmapOr over both trigram indexes. The other four OR the organization predicate together with columns of the driving table, so Postgres has to evaluate them as a join filter and no index applies until that shape changes — the cast is correctness-neutral there, and keeps them from re-introducing the trap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGuVPEaf3zemquZPN39rFx

cubic-dev-ai

0 issues found across 8 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic}

…arch

Co-authored-by: linear-code[bot] <222613912+linear-code[bot]@users.noreply.github.com>

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

Confidence score: 3/5

server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py changes the migration parent so environments that already applied 1b299ae956f3 under its old lineage may skip 382c4661fdb2, leaving the intended migration unapplied despite being stamped at the current head; preserve a compatible Alembic migration path for those environments.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py">

<violation number="1" location="server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py:16">
P2: When an environment has already applied `1b299ae956f3` with its old parent, this reparenting makes `382c4661fdb2` an ancestor that Alembic will skip because the database is already stamped at the current head. Preserve the old parent and add a merge/bridge revision, or explicitly reconcile those databases, so the member index migration still runs.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

cubic-dev-ai · 2026-09-10T13:04:54Z

+
+# revision identifiers, used by Alembic.
+revision = "1b299ae956f3"
+down_revision = "382c4661fdb2"


P2: When an environment has already applied 1b299ae956f3 with its old parent, this reparenting makes 382c4661fdb2 an ancestor that Alembic will skip because the database is already stamped at the current head. Preserve the old parent and add a merge/bridge revision, or explicitly reconcile those databases, so the member index migration still runs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/migrations/versions/2026-09-09-2145_add_pg_trgm_gin_indexes_for_backoffice_.py, line 16: <comment>When an environment has already applied `1b299ae956f3` with its old parent, this reparenting makes `382c4661fdb2` an ancestor that Alembic will skip because the database is already stamped at the current head. Preserve the old parent and add a merge/bridge revision, or explicitly reconcile those databases, so the member index migration still runs.</comment> <file context> @@ -13,7 +13,7 @@ # revision identifiers, used by Alembic. revision = "1b299ae956f3" -down_revision = "a0bc64d272f1" +down_revision = "382c4661fdb2" branch_labels: tuple[str] | None = None depends_on: tuple[str] | None = None </file context>

vercel Bot deployed to Preview – orbit September 9, 2026 19:52 View deployment

vercel Bot deployed to Preview – polar-test September 9, 2026 19:53 View deployment

cubic-dev-ai Bot reviewed Sep 9, 2026

View reviewed changes

pieterbeulque requested a review from frankie567 as a code owner September 10, 2026 05:57

vercel Bot deployed to Preview – orbit September 10, 2026 05:59 View deployment

vercel Bot deployed to Preview – polar-test September 10, 2026 06:00 View deployment

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread server/polar/backoffice/organizations_v2/endpoints.py Outdated

vercel Bot deployed to Preview – orbit September 10, 2026 06:33 View deployment

vercel Bot deployed to Preview – polar-test September 10, 2026 06:34 View deployment

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

vercel Bot deployed to Preview – orbit September 10, 2026 07:42 View deployment

vercel Bot deployed to Preview – polar-test September 10, 2026 07:43 View deployment

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

frankie567 approved these changes Sep 10, 2026

View reviewed changes

pieterbeulque and others added 3 commits September 10, 2026 11:01

pieterbeulque force-pushed the pieter/pg-trgm-org-search branch from 332ffd9 to f36677b Compare September 10, 2026 11:03

vercel Bot deployed to Preview – orbit September 10, 2026 11:05 View deployment

pieterbeulque force-pushed the pieter/pg-trgm-org-search branch from f36677b to 098a875 Compare September 10, 2026 11:05

vercel Bot deployed to Preview – orbit September 10, 2026 11:08 View deployment

vercel Bot deployed to Preview – polar-test September 10, 2026 11:09 View deployment

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

pieterbeulque added this pull request to the merge queue Sep 10, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 10, 2026

Merge remote-tracking branch 'origin/main' into pieter/pg-trgm-org-se…

11dc083

…arch

fix(migrations): update pg_trgm migration head

dcdcf57

Co-authored-by: linear-code[bot] <222613912+linear-code[bot]@users.noreply.github.com>

vercel Bot deployed to Preview – orbit September 10, 2026 13:04 View deployment

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

vercel Bot deployed to Preview – polar-test September 10, 2026 13:05 View deployment

pieterbeulque added this pull request to the merge queue Sep 10, 2026

Merged via the queue into main with commit 021bbbd Sep 10, 2026
27 of 29 checks passed

pieterbeulque deleted the pieter/pg-trgm-org-search branch September 10, 2026 13:13

Conversation

pieterbeulque commented Sep 9, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Ship safety

Verification

Uh oh!

vercel Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Migration Isolation Check Failed

Files that should be in a separate PR:

Why?

Uh oh!

github-actions Bot commented Sep 9, 2026

OpenAPI Changes

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pieterbeulque commented Sep 9, 2026 •

edited by cubic-dev-ai Bot

Loading

vercel Bot commented Sep 9, 2026 •

edited

Loading

github-actions Bot commented Sep 9, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading