diff --git a/pydatalab/advancedSearch.md b/pydatalab/advancedSearch.md new file mode 100644 index 000000000..afe2ba99d --- /dev/null +++ b/pydatalab/advancedSearch.md @@ -0,0 +1,835 @@ +# Advanced Search + +Advanced Search lets you filter any item list in datalab using structured, field-level conditions — no raw query strings required. It dynamically pulls the available fields and operators directly from the data model, so you never need to update the UI when a new field is added to the backend. + +--- + +## Table of Contents + +1. [For Users](#for-users) + - [Opening the Panel](#opening-the-panel) + - [Selecting an Item Type](#selecting-an-item-type) + - [Adding Filter Rules](#adding-filter-rules) + - [Combining Rules — AND / OR](#combining-rules--and--or) + - [Nested Groups](#nested-groups) + - [Sorting Results](#sorting-results) + - [Live Preview Count](#live-preview-count) + - [Running the Search](#running-the-search) + - [Clearing Filters](#clearing-filters) + - [Screenshots](#screenshots) +2. [For Developers](#for-developers) + - [Architecture Overview](#architecture-overview) + - [Component Tree](#component-tree) + - [Mounting AdvancedQueryBuilder in a New View](#mounting-advancedquerybuilder-in-a-new-view) + - [Backend: List Views and Item Types](#backend-list-views-and-item-types) + - [How Dynamic Fields Are Discovered](#how-dynamic-fields-are-discovered) + - [API Endpoints](#api-endpoints) + - [Query Tree Format](#query-tree-format) + - [Available Operators](#available-operators) + - [Editor Components](#editor-components) + - [Cursor-Based Pagination](#cursor-based-pagination) + - [How to Add a New List View](#how-to-add-a-new-list-view) + - [How to Add a New Item Type / Model](#how-to-add-a-new-item-type--model) + - [Pinning Fields Manually per Model](#pinning-fields-manually-per-model) + - [Customising Field Labels and Groups](#customising-field-labels-and-groups) + - [Adding a New Operator](#adding-a-new-operator) + - [Adding a Custom Editor Component](#adding-a-custom-editor-component) + +--- + +## For Users + +### Opening the Panel + +Every item list page (Samples, Cells, Starting Materials, Equipment) has an **Advanced Search** button in the toolbar — it looks like a funnel icon. Click it to open the search panel as a modal dialog. + +When filters are active, the button turns purple and shows a short summary of your current conditions. You can click the **×** on the button to clear all filters instantly without opening the panel. + +> _[Screenshot: toolbar with the Advanced Search button highlighted]_ + +--- + +### Selecting an Item Type + +At the top of the panel you will see radio buttons for each item type available in that view — for example **Sample** and **Cell** on the Samples page. + +Choose the type you want to search across. The field list below updates automatically to show only the fields that belong to that type. + +> **Note:** If you already have rules built and you switch types, datalab will warn you how many of those rules would become invalid and ask you to confirm before clearing them. + +> _[Screenshot: type selector with Sample and Cell options]_ + +--- + +### Adding Filter Rules + +Click **New Rule** to add a condition. Each rule has three parts: + +| Part | What it does | +|---|---| +| **Field** | The data attribute you want to filter on (e.g. Name, Formula, Date) | +| **Operator** | How to match the value (e.g. *contains*, *equals*, *is set*, *before*) | +| **Value** | The value to compare against (not shown for operators like *is set*) | + +The operator list changes depending on the field type: + +- **Text fields** — `contains`, `equals`, `is set`, `is not set` +- **Number fields** — `greater than`, `less than`, `equals`, `is set` +- **Date fields** — `in range`, `before`, `after`, `is set` +- **Enum fields** (e.g. Status, Cell Format) — `is one of`, `equals`, `is set` +- **Constituent fields** (electrodes, electrolyte) — `contains`, `does not contain` with an item reference picker + +To remove a rule, click the trash icon on the right of that row. + +> _[Screenshot: a rule row with field "Name", operator "contains", value "LFP"]_ + +--- + +### Combining Rules — AND / OR + +When you have two or more rules, an **AND / OR** toggle appears above them. + +- **AND** — all conditions must be true (narrower results) +- **OR** — at least one condition must be true (broader results) + +Click the toggle to switch between them. The combinator applies to every rule in that group. + +> _[Screenshot: AND/OR toggle with two rules]_ + +--- + +### Nested Groups + +Click **Add group** to create a sub-group of rules with its own AND / OR combinator. This lets you build more complex logic, for example: + +``` +(Name contains "LFP") AND ( + (Cell Format equals "coin") OR (Cell Format equals "pouch") +) +``` + +Groups can be nested up to a depth defined per view (up to 5 levels for item views). Remove a group with the **Remove group** link that appears inside it. + +> _[Screenshot: two rules plus a nested group with its own OR combinator]_ + +--- + +### Sorting Results + +The **Sort by** row at the top of the rule area lets you pick which field to sort results by. Only fields that support sorting appear in the dropdown. Click the **↓ Desc / ↑ Asc** button next to it to flip the direction. + +If no sort is chosen, results default to newest first (by date). + +> _[Screenshot: sort row with "Date" selected and descending arrow]_ + +--- + +### Live Preview Count + +As you build or change your conditions, the footer of the panel shows a live count — for example **Found: 42 items** — without you having to click Search. This count updates automatically after a short pause (400 ms) every time you change a rule or switch types. + +The count shows `200+ items` when the result set exceeds 200 (the preview limit). + +If any rule is incomplete (missing a value), the preview pauses until the rule is valid. + +> _[Screenshot: footer showing "Found: 17 items"]_ + +--- + +### Running the Search + +Click **Search** in the bottom-right corner. datalab fetches all matching items (up to 2 000) and replaces the table content with the results. The Advanced Search button in the toolbar now shows a summary of the active filters. + +The table works normally on the filtered results — you can still sort columns, select rows, export, etc. + +> _[Screenshot: table showing search results with the active filter summary in the toolbar]_ + +--- + +### Clearing Filters + +- Click the **×** on the Advanced Search button in the toolbar — clears all filters instantly and restores the full list. +- Or open the panel and click **Cancel** to close without applying changes. + +--- + +### Screenshots + +> _Add screenshots here before turning this into a presentation._ + +--- + +--- + +## For Developers + +### Architecture Overview + +Advanced Search is split cleanly between a Vue 3 frontend and a Flask/MongoDB backend. Neither side hard-codes field names or operators — everything is derived at runtime from Pydantic model schemas. + +``` +Browser Backend (Flask + MongoDB) +────────────────────── ───────────────────────────────────────── +AdvancedQueryBuilder.vue GET /query-capabilities → is search enabled? + │ GET /query-types → list of item types + ├─ QueryGroup.vue GET /query-schema → fields + operators per type + │ └─ QueryRule.vue POST /query → run query, returns items + cursor + │ └─ .vue GET /query-options/ → autocomplete for constituent picker + │ + └─ server_fetch_utils.js → wraps all five endpoints above +``` + +The full backend logic lives in: + +``` +pydatalab/src/pydatalab/routes/v0_1/query.py +``` + +The frontend components live in: + +``` +webapp/src/components/AdvancedQueryBuilder.vue ← entry point +webapp/src/components/QueryGroup.vue +webapp/src/components/QueryRule.vue +webapp/src/components/queryEditors/ ← one file per editor type +``` + +--- + +### Component Tree + +``` +AdvancedQueryBuilder Manages state: selected type, schema, rules, sort, +│ preview, pagination. Emits query-results upward. +│ +├─ QueryGroup Recursive. Renders a list of rules/sub-groups with +│ │ an AND/OR combinator toggle. Handles add/remove/update. +│ └─ QueryRule One filter row: field select → operator select → editor. +│ │ Looks up available operators from the schema returned +│ │ by the backend — never hard-codes them. +│ └─ Swapped in based on the operator's "editor" key: +│ TextEditor +│ NumberEditor +│ DatetimeEditor +│ DatetimeRangeEditor +│ StringListEditor +│ EnumEditor +│ ChemicalFormulaEditor +│ ConstituentSelectorEditor (fetches item references from the server) +│ FallbackEditor (catch-all for unknown editor types) +``` + +--- + +### Mounting AdvancedQueryBuilder in a New View + +`AdvancedQueryBuilder` is a self-contained component. The parent page only needs to: + +1. Pass a `listView` name (must match a key in `QUERY_VIEWS` on the backend). +2. Listen for the `query-results` event. + +**Minimal example** + +```vue + + + +``` + +**With pre-loaded query config** (avoids one extra network round-trip) + +If your page already fetches capabilities (e.g. `DynamicDataTable` does this automatically), pass the config object directly: + +```vue + +``` + +`queryOptions.options` comes from `GET /query-capabilities` and has this shape: + +```json +{ + "queryRoute": "/query", + "listViewName": "samples", + "resource": "items", + "item_types": [ + { "id": "samples", "label": "Sample", "queryable": true }, + { "id": "cells", "label": "Cell", "queryable": true } + ] +} +``` + +**DynamicDataTable integration** + +`DynamicDataTable` does all of this automatically via `DynamicDataTableButtons`. It calls `fetchAdvancedQueryConfig(dataType)` on mount, passes the result to `DynamicDataTableButtons`, which renders `AdvancedQueryBuilder` when `advancedQueryConfig.isEnabled` is true. No extra code is needed for views already backed by a registered list view. + +--- + +### Backend: List Views and Item Types + +The backend organises searchable data into **list views**, defined in `QUERY_VIEWS` at the top of `query.py`: + +```python +QUERY_VIEWS: dict[str, dict] = { + "samples": { + "resource": "items", + "collection": "items", + "types": ["samples", "cells"], # which item types live here + "model_by_type": ITEM_MODELS, + "view_contexts": ["samples"], # frontend dataType strings that map here + "default_sort": [("date", -1)], + }, + "starting_materials": { + "resource": "items", + "collection": "items", + "types": ["starting_materials"], + "model_by_type": ITEM_MODELS, + "view_contexts": ["startingMaterials", "starting_materials"], + "default_sort": [("date", -1)], + }, + # equipment, collections, users, groups ... +} +``` + +`view_contexts` is the bridge between the frontend `dataType` prop and the backend view name. When `DynamicDataTable` calls `GET /query-capabilities?data_type=startingMaterials`, the backend finds the view whose `view_contexts` list contains that string. + +--- + +### How Dynamic Fields Are Discovered + +This is the core of what makes advanced search dynamic. When the frontend requests `GET /query-schema?list_view=samples&item_type=cells`, the backend: + +1. Calls `model.schema(by_alias=False)` on the Pydantic model for that type (e.g. `Cell`). +2. Walks every property recursively via `_iter_schema_fields`, skipping internal fields listed in `_SKIP_FIELDS`. +3. Inspects each field's JSON Schema type/format to assign operators: + - `"format": "date-time"` or field name contains `"date"` → date operators + - `"type": "number"` or `"integer"` → numeric operators + - `"type": "string"` → text operators + - `$ref` pointing to an enum → enum operators + - Known constituent array fields → constituent operators +4. Merges UI hints from `_FIELD_UI` (custom label, group, sortable flag, editor overrides). +5. Returns the complete field + operator list as JSON. + +**Result:** add a field to a Pydantic model → it automatically appears in the advanced search UI the next time the schema endpoint is called. No frontend changes needed. + +--- + +### API Endpoints + +All endpoints are registered on the `QUERY` blueprint and mounted at `/api/v0.1/`. + +--- + +#### `GET /query-capabilities` + +Checks whether advanced search is enabled for a given frontend data type context. + +| Parameter | Type | Description | +|---|---|---| +| `data_type` | string | The frontend `dataType` value (e.g. `"samples"`, `"startingMaterials"`) | + +**Response** + +```json +{ + "data_type": "samples", + "advanced_query": { + "isEnabled": true, + "listViewName": "samples", + "resource": "items", + "queryRoute": "/query", + "options": { "...": "..." }, + "capabilities": { + "combinators": ["and", "or"], + "allow_negation": false, + "max_rules": 50, + "max_depth": 5 + } + }, + "views": ["...all views..."] +} +``` + +Returns `"advanced_query": null` when no view matches `data_type`. + +--- + +#### `GET /query-types` + +Returns the list of queryable item types for a list view. + +| Parameter | Type | Description | +|---|---|---| +| `list_view` | string | e.g. `"samples"` | + +**Response** + +```json +{ + "list_view": "samples", + "item_types": [ + { "id": "samples", "label": "Sample", "description": "...", "queryable": true }, + { "id": "cells", "label": "Cell", "description": "...", "queryable": true } + ] +} +``` + +--- + +#### `GET /query-schema` + +Returns all searchable fields and their operators for a specific item type. + +| Parameter | Type | Description | +|---|---|---| +| `list_view` | string | e.g. `"samples"` | +| `item_type` | string (repeatable) | e.g. `"cells"` | + +**Response (excerpt)** + +```json +{ + "version": "1.0", + "list_view": "samples", + "fields": [ + { + "id": "name", + "label": "Name", + "group": "Basic", + "sortable": true, + "operators": [ + { "id": "contains", "label": "contains", "value_required": true, "editor": "text" }, + { "id": "eq", "label": "equals", "value_required": true, "editor": "text" }, + { "id": "is_set", "label": "is set", "value_required": false }, + { "id": "is_not_set", "label": "is not set", "value_required": false } + ] + }, + { + "id": "cell_format", + "label": "Cell Format", + "group": "Cell", + "sortable": true, + "operators": [ + { + "id": "in", "label": "is one of", "value_required": true, + "editor": "enum", + "value_schema": { "type": "array", "items": { "enum": ["coin","pouch","..."] } } + } + ] + } + ], + "capabilities": { "combinators": ["and", "or"], "max_depth": 5 } +} +``` + +--- + +#### `POST /query` + +Executes a query and returns one page of matching items with a cursor for the next page. + +**Request body** + +```json +{ + "list_view": "samples", + "item_types": ["cells"], + "where": { + "kind": "group", + "combinator": "and", + "children": [ + { "kind": "rule", "field": "name", "operator": "contains", "value": "LFP" }, + { "kind": "rule", "field": "cell_format", "operator": "eq", "value": "coin" } + ] + }, + "sort": [{ "field": "date", "direction": "desc" }], + "page": { "limit": 50, "cursor": "" } +} +``` + +`where` can be omitted (returns all items of the given types). +`page.cursor` is omitted on the first request; pass the value from `next_cursor` to fetch the next page. + +**Response** + +```json +{ + "query": { + "common_type": { "id": "cells", "label": "Cell" }, + "selected_item_types": ["cells"] + }, + "items": [ { "item_id": "...", "name": "...", "_id": "...", "...": "..." } ], + "page": { "limit": 50, "next_cursor": "", "has_more": true } +} +``` + +Each item is a summary object (not the full document). Fields returned: `_id` (string), `item_id`, `name`, `chemform`, `type`, `date`, `refcode`, `status`, `characteristic_chemical_formula`, `nblocks`, `nfiles`, `blocks`, `creators`, `groups`, `collections`. + +--- + +#### `GET /query-options/` + +Autocomplete endpoint for the constituent-selector editor. Currently the only valid `source_id` is `datalab:item-reference`. + +| Parameter | Type | Description | +|---|---|---| +| `q` | string | Search string matched against name, item_id, refcode | +| `limit` | int | Max results (default 20, max 100) | +| `item_type` | string (repeatable) | Filter by item type | +| `cursor` | string | Pagination cursor | + +--- + +### Query Tree Format + +Every query sent to `POST /query` is a tree of `group` and `rule` nodes. + +``` +Node = Group | Rule + +Group: + kind: "group" + combinator: "and" | "or" + children: Node[] ← can contain Rules and nested Groups + +Rule: + kind: "rule" + field: string ← field id from /query-schema + operator: string ← operator id from that field's operators list + value: any | undefined ← omit when value_required is false +``` + +**Example — (name contains "LFP") AND ((format = coin) OR (format = pouch))** + +```json +{ + "kind": "group", + "combinator": "and", + "children": [ + { "kind": "rule", "field": "name", "operator": "contains", "value": "LFP" }, + { + "kind": "group", + "combinator": "or", + "children": [ + { "kind": "rule", "field": "cell_format", "operator": "eq", "value": "coin" }, + { "kind": "rule", "field": "cell_format", "operator": "eq", "value": "pouch" } + ] + } + ] +} +``` + +--- + +### Available Operators + +Operators are defined in the `OPERATORS` dict in `query.py`. Each compiles to a MongoDB filter fragment. + +The table below shows both the **id** (the token sent in the API) and the **label** (the text shown to the user in the dropdown). + +| `id` (sent to API) | `label` (shown in UI) | Value type | Editor | +|---|---|---|---| +| `contains` | contains | string | `text` | +| `eq` | equals | any | `text` | +| `is_set` | is set | — | — | +| `is_not_set` | is not set | — | — | +| `in` | is one of | array | `string-list` or `enum` | +| `gt` | greater than | number | `number` | +| `lt` | less than | number | `number` | +| `before` | before | datetime string | `datetime` | +| `after` | after | datetime string | `datetime` | +| `date_range` | in range | `[start, end]` | `datetime-range` | +| `has_constituent` | contains | item refcode/id | `constituent-selector` | +| `not_has_constituent` | does not contain | item refcode/id | `constituent-selector` | + +When building a rule in the frontend, use the **id** as the `operator` field. The **label** is only for display. When defining a new operator in `query.py`, you set both independently. + +--- + +### Editor Components + +`QueryRule` maps the `editor` key from the operator definition to a Vue component via `editorMap`: + +| `editor` key | Component | Used for | +|---|---|---| +| `text` | `TextEditor` | Plain string input | +| `number` | `NumberEditor` | Numeric input | +| `datetime` | `DatetimeEditor` | Single date/time picker | +| `datetime-range` | `DatetimeRangeEditor` | Start + end date pickers | +| `string-list` | `StringListEditor` | Comma-separated or tag-style list | +| `enum` | `EnumEditor` | Dropdown built from `value_schema.items.enum` | +| `chemical-formula` | `ChemicalFormulaEditor` | Plain text input for now (formula-aware editor is a TODO) | +| `constituent-selector` | `ConstituentSelectorEditor` | Async item search, fetches from `/query-options/` | +| _(anything else)_ | `FallbackEditor` | Plain text fallback | + +--- + +### Cursor-Based Pagination + +`POST /query` uses cursor pagination instead of offset pagination. This avoids the "skipping rows" problem that can occur when documents are inserted between page fetches. + +**How the cursor is built** + +The backend appends `("_id", 1)` to the sort spec as a tie-breaker, then fetches `limit + 1` documents. If the result has more than `limit` rows, the `_id` of the last returned document is base64-encoded into `next_cursor`: + +```python +# backend: _encode_cursor / _decode_cursor +next_cursor = base64.urlsafe_b64encode(str(last_id).encode()).decode() +``` + +**How the cursor is applied on the next request** + +When the client sends `page.cursor`, the backend decodes the `_id` and adds: + +```json +{ "_id": { "$gt": "" } } +``` + +This is combined with the original `$match` filter via `$and`, so only documents with a greater `_id` are returned. + +> **Known limitation:** The current cursor is based only on `_id`. Because `_id` is always appended to the sort as the last key, pagination stays consistent for most cases. However, for large datasets with a custom sort field that has many ties, some edge cases can produce unexpected ordering between pages. A composite cursor (encoding the sort field value alongside `_id`) would be more robust. + +**Frontend behaviour** + +`AdvancedQueryBuilder.submitQuery` loops automatically, collecting pages of 200 items each until `has_more` is false or 2 000 items are accumulated (safety cap). The full collected array is then emitted once via `query-results` and displayed in the table. + +--- + +### How to Add a New List View + +1. Add an entry to `QUERY_VIEWS` in `query.py`: + +```python +QUERY_VIEWS["experiments"] = { + "resource": "items", + "collection": "items", + "types": ["experiments"], + "model_by_type": ITEM_MODELS, + "view_contexts": ["experiments"], # must match the frontend dataType prop + "default_sort": [("date", -1)], +} +``` + +2. Register `Experiment` in `ITEM_MODELS` (see `pydatalab/models/__init__.py`). + +3. The frontend picks this up automatically — no changes needed in Vue components. + +--- + +### How to Add a New Item Type / Model + +1. Create the Pydantic model class (e.g. `pydatalab/models/experiments.py`): + +```python +from pydatalab.models.items import Item +from pydantic import Field + +class Experiment(Item): + """A model for representing an experiment.""" + + type: str = Field("experiments", const="experiments", pattern="^experiments$") + + protocol: str | None + temperature_c: float | None +``` + +2. Register it in `ITEM_MODELS` in `pydatalab/models/__init__.py`: + +```python +from pydatalab.models.experiments import Experiment + +ITEM_MODELS = { + ..., + "experiments": Experiment, +} +``` + +3. That is all. The schema endpoint will automatically discover `protocol` (text operators) and `temperature_c` (numeric operators) next time it is called. + +--- + +### Pinning Fields Manually per Model + +By default, every field that appears in the Pydantic schema and has a recognised type is shown in the advanced search UI. If you want to **restrict** the search to a specific subset of fields — or expose them in a specific order — add `query_options_list` to your model's schema extra config. + +When `query_options_list` is present it **completely replaces** the auto-discovered field list for that model. See `_query_options_list` in `query.py` for the full specification. + +The codebase uses **Pydantic v1**. Add a `Config` inner class with `schema_extra`: + +```python +class Experiment(Item): + """A model for representing an experiment.""" + + type: str = Field("experiments", const="experiments", pattern="^experiments$") + protocol: str | None + temperature_c: float | None + + class Config: + schema_extra = { + "query_options_list": ["name", "item_id", "date", "temperature_c"] + } +``` + +Only those four fields will appear in the advanced search panel for `Experiment` items. + +**Advanced example — overriding label, operators, and editor** + +Each entry in `query_options_list` can be a plain field-ID string or a dict that overrides specific properties: + +```python +"query_options_list": [ + "name", + "date", + { + "id": "temperature_c", + "label": "Temperature (°C)", + "operators": ["gt", "lt", "eq", "is_set"], + }, + { + "id": "chemform", + "label": "Formula", + "operators": ["eq", "contains"], + "editor_override": {"eq": "chemical-formula"}, + }, +] +``` + +Each entry is either: +- A **plain string** — field ID looked up in the auto-discovered registry; uses all its defaults. +- A **dict** — overrides any of `label`, `operators`, `editor_override`, `mongo_path`, `group`, `sortable`. + +> **Important:** If any field ID in the list is invalid (typo or not present in the schema), the backend raises a `ValueError` at request time with a message listing the unknown entries and the available fields. This prevents the silent fallback to the full registry. + +--- + +### Customising Field Labels and Groups + +`_FIELD_UI` in `query.py` is a plain dict that maps field names to display metadata. It applies globally across all models that have that field. + +```python +_FIELD_UI: dict[str, dict] = { + "name": {"label": "Name", "group": "Basic", "sortable": True}, + "chemform":{"label": "Formula", "group": "Chemistry", "sortable": True, + "editor_override": {"eq": "chemical-formula", "contains": "text"}}, + # ... +} +``` + +| Key | Type | Effect | +|---|---|---| +| `label` | string | Human-readable column header in the UI | +| `group` | string | Groups fields in the dropdown (`Basic`, `Chemistry`, `Cell`, `Synthesis`, `Provenance`, `Other`) | +| `sortable` | bool | Whether the field appears in the Sort By dropdown | +| `editor_override` | dict | Maps operator `id` → editor key, overriding the default for that operator | + +To add a new field to `_FIELD_UI`, simply add an entry. Fields not listed get a title-cased label and are placed in `Other`. + +--- + +### Adding a New Operator + +Operators are defined in the `OPERATORS` dict in `query.py`. Add an entry: + +```python +OPERATORS["regex"] = { + "label": "matches regex", + "value_required": True, + "editor": "text", + "compile": lambda path, value: { + path: {"$regex": str(value), "$options": "m"} + }, +} +``` + +> **Security note:** For operators like `contains` that treat user input as a **literal string**, always use `re.escape()` — the built-in `contains` operator already does this. For a dedicated `regex` operator the user is intentionally writing a pattern, so do **not** escape it — escaping would turn `^LFP.*` into `\^LFP\.\*` and break it. Instead, consider limiting regex complexity or restricting who can use this operator. + +The keys each operator entry must have: + +| Key | Required | Description | +|---|---|---| +| `label` | yes | Displayed in the operator dropdown | +| `value_required` | yes | If `False`, no editor is shown and `value` is ignored | +| `editor` | when `value_required` is `True` | Key into the frontend `editorMap` in `QueryRule.vue` | +| `compile` | yes | `(mongo_path, value) → dict` — produces the MongoDB filter fragment | +| `options_source` | no | For constituent-selector only: the source ID passed to `/query-options/` | + +After adding to `OPERATORS`, also add the operator `id` to the relevant field's `operator_ids` list — either via `_field_to_operators` logic, via `_FIELD_UI` `editor_override`, or via `query_options_list` on the model. + +--- + +### Adding a Custom Editor Component + +1. Create `webapp/src/components/queryEditors/MyEditor.vue`. The component must: + - Accept a `modelValue` prop (the current value). + - Accept a `valueSchema` prop (optional — the JSON Schema for allowed values, passed from the backend). + - Emit `update:modelValue` when the value changes. + +```vue + + + +``` + +2. Register it in `QueryRule.vue`: + +```js +import MyEditor from "@/components/queryEditors/MyEditor.vue"; + +const editorMap = { + // existing entries ... + "my-editor": "MyEditor", +}; + +// and add MyEditor to the components: { ... } option +``` + +3. Reference it in an operator definition on the backend: + +```python +OPERATORS["my_op"] = { + "label": "my operator", + "value_required": True, + "editor": "my-editor", # must match the key in editorMap + "compile": lambda p, v: {p: v}, +} +``` diff --git a/pydatalab/src/pydatalab/routes/v0_1/__init__.py b/pydatalab/src/pydatalab/routes/v0_1/__init__.py index 93b3a22db..71e0a0366 100644 --- a/pydatalab/src/pydatalab/routes/v0_1/__init__.py +++ b/pydatalab/src/pydatalab/routes/v0_1/__init__.py @@ -12,6 +12,7 @@ from .healthcheck import HEALTHCHECK from .info import INFO from .items import ITEMS +from .query import QUERY from .remotes import REMOTES from .users import USERS @@ -29,6 +30,7 @@ INFO, GRAPHS, EXPORT, + QUERY, ) __all__ = ("BLUEPRINTS", "OAUTH", "__api_version__", "OAUTH_PROXIES") diff --git a/pydatalab/src/pydatalab/routes/v0_1/query.py b/pydatalab/src/pydatalab/routes/v0_1/query.py new file mode 100644 index 000000000..463e1df93 --- /dev/null +++ b/pydatalab/src/pydatalab/routes/v0_1/query.py @@ -0,0 +1,1042 @@ +import base64 +import re +import uuid +from datetime import datetime, timezone +from typing import Any + +from bson import ObjectId +from flask import Blueprint, jsonify, request + +from pydatalab.models import ITEM_MODELS +from pydatalab.models.collections import Collection +from pydatalab.models.people import Group, Person +from pydatalab.mongo import flask_mongo +from pydatalab.permissions import active_users_or_get_only, get_default_permissions +from pydatalab.routes.v0_1.items import collections_lookup, creators_lookup, groups_lookup + +QUERY = Blueprint("query", __name__) + + +@QUERY.before_request +@active_users_or_get_only +def _(): ... + + +QUERY_VIEWS: dict[str, dict] = { + "samples": { + "resource": "items", + "collection": "items", + "types": ["samples", "cells"], + "model_by_type": ITEM_MODELS, + "view_contexts": ["samples"], + "default_sort": [("date", -1)], + }, + "starting_materials": { + "resource": "items", + "collection": "items", + "types": ["starting_materials"], + "model_by_type": ITEM_MODELS, + "view_contexts": ["startingMaterials", "starting_materials"], + "default_sort": [("date", -1)], + }, + "equipment": { + "resource": "items", + "collection": "items", + "types": ["equipment"], + "model_by_type": ITEM_MODELS, + "view_contexts": ["equipment"], + "default_sort": [("date", -1)], + }, + "collections": { + "resource": "collections", + "collection": "collections", + "types": ["collections"], + "model_by_type": {"collections": Collection}, + "view_contexts": ["collections"], + "default_sort": [("_id", -1)], + }, + "users": { + "resource": "users", + "collection": "users", + "model": Person, + "view_contexts": ["users"], + "default_sort": [("display_name", 1)], + }, + "groups": { + "resource": "groups", + "collection": "groups", + "model": Group, + "view_contexts": ["groups"], + "default_sort": [("display_name", 1)], + }, +} +LIST_VIEWS = QUERY_VIEWS + +_SKIP_FIELDS: set[str] = { + "type", + "immutable_id", + "creator_ids", + "group_ids", + "blocks_obj", + "display_order", + "file_ObjectIds", + "revision", + "revisions", + "version", + "relationships", + "files", + "collections", + "creators", + "groups", +} + +_FIELD_UI: dict[str, dict] = { + "name": {"label": "Name", "group": "Basic", "sortable": True}, + "item_id": {"label": "Item ID", "group": "Basic", "sortable": True}, + "refcode": {"label": "Refcode", "group": "Basic", "sortable": True}, + "description": {"label": "Description", "group": "Basic", "sortable": False}, + "date": {"label": "Date", "group": "Basic", "sortable": True}, + "status": {"label": "Status", "group": "Basic", "sortable": True}, + "chemform": { + "label": "Formula", + "group": "Chemistry", + "sortable": True, + "editor_override": {"eq": "chemical-formula", "contains": "text"}, + }, + "characteristic_chemical_formula": { + "label": "Active material formula", + "group": "Chemistry", + "sortable": True, + "editor_override": {"eq": "chemical-formula"}, + }, + "smiles": {"label": "SMILES", "group": "Chemistry", "sortable": False}, + "inchi_key": {"label": "InChI key", "group": "Chemistry", "sortable": False}, + "inchi": {"label": "InChI", "group": "Chemistry", "sortable": False}, + "GHS_codes": {"label": "GHS hazard codes", "group": "Chemistry", "sortable": False}, + "CAS": {"label": "CAS number", "group": "Chemistry", "sortable": False}, + "molar_mass": {"label": "Molar mass (g/mol)", "group": "Chemistry", "sortable": True}, + "characteristic_mass": { + "label": "Characteristic mass (mg)", + "group": "Chemistry", + "sortable": True, + }, + "characteristic_molar_mass": { + "label": "Characteristic molar mass", + "group": "Chemistry", + "sortable": True, + }, + "cell_format": {"label": "Cell format", "group": "Cell", "sortable": True}, + "cell_format_description": { + "label": "Cell format description", + "group": "Cell", + "sortable": False, + }, + "cell_preparation_description": { + "label": "Cell preparation", + "group": "Cell", + "sortable": False, + }, + "supplier": {"label": "Supplier", "group": "Provenance", "sortable": True}, + "location": {"label": "Location", "group": "Provenance", "sortable": True}, + "manufacturer": {"label": "Manufacturer", "group": "Provenance", "sortable": True}, + "serial_numbers": {"label": "Serial numbers", "group": "Provenance", "sortable": False}, + "contact": {"label": "Contact", "group": "Provenance", "sortable": False}, + "barcode": {"label": "Barcode", "group": "Provenance", "sortable": False}, + "chemical_purity": {"label": "Chemical purity", "group": "Chemistry", "sortable": False}, + "synthesis_description": { + "label": "Synthesis description", + "group": "Synthesis", + "sortable": False, + }, + "synthesis_constituents": { + "label": "Synthesis constituent", + "group": "Synthesis", + "sortable": False, + }, + "date_opened": {"label": "Date opened", "group": "Provenance", "sortable": True}, + "last_modified": {"label": "Last modified", "group": "Basic", "sortable": True}, + "active_ion_charge": {"label": "Active ion charge", "group": "Cell", "sortable": True}, + "positive_electrode": { + "label": "Positive electrode constituent", + "group": "Cell", + "sortable": False, + }, + "negative_electrode": { + "label": "Negative electrode constituent", + "group": "Cell", + "sortable": False, + }, + "electrolyte": {"label": "Electrolyte constituent", "group": "Cell", "sortable": False}, +} + +OPERATORS: dict[str, dict] = { + "contains": { + "label": "contains", + "value_required": True, + "editor": "text", + "compile": lambda p, v: {p: {"$regex": re.escape(str(v)), "$options": "i"}}, + }, + "eq": { + "label": "equals", + "value_required": True, + "editor": "text", + "compile": lambda p, v: {p: v}, + }, + "is_set": { + "label": "is set", + "value_required": False, + "compile": lambda p, v: {p: {"$exists": True, "$nin": [None, ""]}}, + }, + "is_not_set": { + "label": "is not set", + "value_required": False, + "compile": lambda p, v: {"$or": [{p: {"$exists": False}}, {p: None}, {p: ""}]}, + }, + "in": { + "label": "is one of", + "value_required": True, + "editor": "string-list", + "compile": lambda p, v: {p: {"$in": list(v) if not isinstance(v, list) else v}}, + }, + "gt": { + "label": "greater than", + "value_required": True, + "editor": "number", + "compile": lambda p, v: {p: {"$gt": v}}, + }, + "lt": { + "label": "less than", + "value_required": True, + "editor": "number", + "compile": lambda p, v: {p: {"$lt": v}}, + }, + "before": { + "label": "before", + "value_required": True, + "editor": "datetime", + "compile": lambda p, v: {p: {"$lt": _parse_dt(v)}}, + }, + "after": { + "label": "after", + "value_required": True, + "editor": "datetime", + "compile": lambda p, v: {p: {"$gt": _parse_dt(v)}}, + }, + "date_range": { + "label": "in range", + "value_required": True, + "editor": "datetime-range", + "compile": lambda p, v: _compile_date_range(p, v), + }, + "has_constituent": { + "label": "contains", + "value_required": True, + "editor": "constituent-selector", + "options_source": "datalab:item-reference", + "compile": lambda p, v: { + p: { + "$elemMatch": { + "$or": [ + {"item.refcode": str(v)}, + {"item.item_id": str(v)}, + ] + } + } + }, + }, + "not_has_constituent": { + "label": "does not contain", + "value_required": True, + "editor": "constituent-selector", + "options_source": "datalab:item-reference", + "compile": lambda p, v: { + "$nor": [ + { + p: { + "$elemMatch": { + "$or": [ + {"item.refcode": str(v)}, + {"item.item_id": str(v)}, + ] + } + } + } + ] + }, + }, +} + + +def _parse_dt(s: str) -> datetime: + s = str(s).rstrip("Z") + for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%dT%H:%M", "%Y-%m-%d"): + try: + return datetime.strptime(s, fmt).replace(tzinfo=timezone.utc) + except ValueError: + continue + raise ValueError(f"Cannot parse datetime: {s!r}") + + +def _compile_date_range(path: str, value: Any) -> dict: + if not isinstance(value, (list, tuple)) or len(value) != 2: + raise ValueError("date_range requires [start, end]") + return {path: {"$gte": _parse_dt(value[0]), "$lte": _parse_dt(value[1])}} + + +def _resolve_ref(ref: str, definitions: dict) -> dict: + name = ref.split("/")[-1] + return definitions.get(name, {}) + + +def _resolve_schema_node(field_def: dict, definitions: dict) -> dict: + for candidate in (field_def, *field_def.get("allOf", []), *field_def.get("anyOf", [])): + ref = candidate.get("$ref", "") + if ref: + return _resolve_ref(ref, definitions) + return field_def + + +def _json_schema_extra(model: type) -> dict: + config = getattr(model, "Config", None) + schema_extra = getattr(config, "schema_extra", {}) if config else {} + if isinstance(schema_extra, dict): + return schema_extra + + model_config = getattr(model, "model_config", {}) + if isinstance(model_config, dict): + return model_config.get("json_schema_extra", {}) or {} + + return {} + + +def _query_options_list(model: type) -> list | None: + """Return the explicit query field list for *model*, or ``None`` to use auto-discovery. + + To pin specific fields for a model in the advanced-search UI, add a ``Config`` + class with ``query_options_list`` to that model. When present it **replaces** + the automatically derived field registry entirely:: + + class Config: + schema_extra = { + "query_options_list": [ + "name", + "item_id", + "date", + {"id": "chemform", "operators": ["eq", "contains"], "label": "Formula"}, + ] + } + + Each entry is either a plain field-ID string (looks up defaults from the registry) + or a dict accepted by :func:`_normalise_query_option` (overrides label/operators/path). + """ + extra = _json_schema_extra(model) + options = extra.get("query_options_list") + return options if isinstance(options, list) else None + + +def _field_to_operators( + field_def: dict, definitions: dict, field_name: str +) -> tuple[list[str], dict, dict]: + """Returns (operator_ids, editor_override_per_op, value_schema_override_per_op).""" + fmt = field_def.get("format", "") + ftype = field_def.get("type", "") + + ref = "" + for candidate in (field_def, *field_def.get("allOf", []), *field_def.get("anyOf", [])): + ref = candidate.get("$ref", "") + if ref: + break + + if ref: + resolved = _resolve_ref(ref, definitions) + enum_values = resolved.get("enum") + if enum_values: + vs = {"enum": enum_values} + vs_array = {"type": "array", "items": vs} + ui = _FIELD_UI.get(field_name, {}) + eo = ui.get("editor_override", {"in": "enum", "eq": "enum"}) + if "in" not in eo: + eo["in"] = "enum" + if "eq" not in eo: + eo["eq"] = "enum" + return ["in", "eq", "is_set"], eo, {"in": vs_array, "eq": vs} + + if fmt == "date-time" or (ftype == "string" and "date" in field_name): + eo = {"date_range": "datetime-range", "before": "datetime", "after": "datetime"} + return ["date_range", "before", "after", "is_set"], eo, {} + + if ftype in ("number", "integer"): + return ["gt", "lt", "eq", "is_set"], {}, {} + + if ftype == "string": + ui = _FIELD_UI.get(field_name, {}) + eo = dict(ui.get("editor_override", {})) + return ["contains", "eq", "is_set", "is_not_set"], eo, {} + + return [], {}, {} + + +def _iter_schema_fields( + properties: dict, definitions: dict, prefix: str = "", depth: int = 0 +) -> list[tuple[str, dict]]: + fields: list[tuple[str, dict]] = [] + for field_name, field_def in properties.items(): + path = f"{prefix}.{field_name}" if prefix else field_name + if not prefix and field_name in _SKIP_FIELDS: + continue + + resolved = _resolve_schema_node(field_def, definitions) + node = resolved or field_def + nested_properties = node.get("properties") + array_items = node.get("items", {}) + array_node = _resolve_schema_node(array_items, definitions) if array_items else {} + + if node.get("type") == "array" and array_node.get("properties") and depth < 3: + fields.extend( + _iter_schema_fields(array_node["properties"], definitions, path, depth + 1) + ) + continue + + if nested_properties and depth < 3: + fields.extend(_iter_schema_fields(nested_properties, definitions, path, depth + 1)) + continue + + fields.append((path, field_def)) + + return fields + + +def _normalise_query_option( + option: str | dict, registry: dict[str, dict] +) -> tuple[str, dict] | None: + if isinstance(option, str): + return (option, registry[option]) if option in registry else None + + if not isinstance(option, dict): + return None + + field_id = option.get("id") or option.get("field") or option.get("path") + if not field_id: + return None + + base = dict(registry.get(field_id, {})) + operator_ids = ( + option.get("operators") + or base.get("operator_ids") + or [ + "contains", + "eq", + "is_set", + ] + ) + operator_ids = [op_id for op_id in operator_ids if op_id in OPERATORS] + if not operator_ids: + return None + + base.update( + { + "mongo_path": option.get("mongo_path") + or option.get("path") + or base.get("mongo_path", field_id), + "label": option.get("label") or base.get("label", field_id.replace("_", " ").title()), + "group": option.get("group") or base.get("group", "Other"), + "sortable": option.get("sortable", base.get("sortable", False)), + "operator_ids": operator_ids, + "editor_override": option.get("editor_override") or base.get("editor_override", {}), + "value_schema_override": option.get("value_schema_override") + or base.get("value_schema_override", {}), + } + ) + if "subfields" in option: + base["subfields"] = option["subfields"] + + return field_id, base + + +def _build_model_field_registry(model: Any, type_id: str) -> dict[str, dict]: + try: + schema = model.schema(by_alias=False) + definitions = schema.get("definitions", {}) + properties = schema.get("properties", {}) + except Exception: + definitions = {} + properties = {} + + registry: dict[str, dict] = {} + for field_name, field_def in _iter_schema_fields(properties, definitions): + operator_ids, editor_override, value_schema_override = _field_to_operators( + field_def, definitions, field_name + ) + if not operator_ids: + continue + + ui = _FIELD_UI.get(field_name, {}) + label = ui.get("label") or field_name.replace("_", " ").title() + group = ui.get("group", "Other") + sortable = ui.get("sortable", field_def.get("type") in ("string", "number", "integer")) + + registry[field_name] = { + "mongo_path": field_name, + "label": label, + "group": group, + "sortable": sortable, + "operator_ids": operator_ids, + "editor_override": editor_override, + "value_schema_override": value_schema_override, + } + + _constituent_fields = { + "synthesis_constituents": ("Synthesis constituent", "Synthesis"), + "positive_electrode": ("Positive electrode constituent", "Cell"), + "negative_electrode": ("Negative electrode constituent", "Cell"), + "electrolyte": ("Electrolyte constituent", "Cell"), + } + for cf_name, (cf_label, cf_group) in _constituent_fields.items(): + if cf_name in properties: + registry[cf_name] = { + "mongo_path": cf_name, + "label": cf_label, + "group": cf_group, + "sortable": False, + "operator_ids": ["has_constituent", "not_has_constituent"], + "editor_override": {}, + "value_schema_override": {}, + } + + explicit_options = _query_options_list(model) + if explicit_options: + explicit_registry: dict[str, dict] = {} + invalid: list[str] = [] + for option in explicit_options: + normalised = _normalise_query_option(option, registry) + if normalised: + field_id, field_config = normalised + explicit_registry[field_id] = field_config + else: + label = option if isinstance(option, str) else repr(option) + invalid.append(label) + if invalid: + raise ValueError( + f"{model.__name__}.Config.schema_extra['query_options_list'] contains invalid " + f"entries: {invalid}. Available fields: {sorted(registry)}" + ) + if not explicit_registry: + raise ValueError( + f"{model.__name__}.Config.schema_extra['query_options_list'] is set but " + f"produced no valid fields" + ) + registry = explicit_registry + + return registry + + +def _build_field_registry(type_id: str) -> dict[str, dict]: + return _build_model_field_registry(ITEM_MODELS[type_id], type_id) + + +def _get_field_registry( + selected_types: list[str], model_by_type: dict | None = None +) -> dict[str, dict]: + models = model_by_type or ITEM_MODELS + registries = [_build_model_field_registry(models[t], t) for t in selected_types] + common_ids = set(registries[0].keys()) + for r in registries[1:]: + common_ids &= set(r.keys()) + return {fid: registries[0][fid] for fid in common_ids} + + +def _build_field_registry_for_view( + list_view: str, view: dict, selected_types: list[str] | None = None +) -> dict[str, dict]: + if view.get("model"): + # single-model view + return _build_model_field_registry(view["model"], list_view) + + model_by_type = view.get("model_by_type") or ITEM_MODELS + types = selected_types or view.get("types") or list(model_by_type.keys()) + return _get_field_registry(types, model_by_type) + + +def _query_type_entry(type_id: str, model_by_type: dict) -> dict: + model = model_by_type.get(type_id) + schema = model.schema(by_alias=False) if model else {} + label = schema.get("title", type_id.replace("_", " ").title()) + description = schema.get("description") or "" + mro_names = [c.__name__ for c in model.__mro__] if model else [] + parent_type = None + for t, m in model_by_type.items(): + if t != type_id and m.__name__ in mro_names[1:]: + parent_type = t + break + + return { + "id": type_id, + "label": label, + "description": description, + "parent_type": parent_type, + "queryable": bool(model and _build_model_field_registry(model, type_id)), + } + + +def _model_label(model: Any | None, fallback: str) -> str: + if not model: + return fallback + try: + return model.schema(by_alias=False).get("title", fallback) + except Exception: + return fallback + + +def _item_type_entry(type_id: str) -> dict: + return _query_type_entry(type_id, ITEM_MODELS) + + +def _view_capability(list_view: str, view: dict) -> dict: + if view.get("model_by_type"): + query_types = [ + _query_type_entry(type_id, view["model_by_type"]) for type_id in view.get("types", []) + ] + elif view.get("model"): + # single model view + query_types = [_query_type_entry(list_view, {list_view: view["model"]})] + else: + query_types = [] + + capabilities = { + "combinators": ["and", "or"], + "allow_negation": False, + "max_rules": 50, + "max_in_values": 100, + "max_depth": 5 if view.get("resource") == "items" else 0, + } + + return { + "isEnabled": any(t["queryable"] for t in query_types), + "listViewName": list_view, + "resource": view["resource"], + "queryRoute": "/query", + "options": { + "queryRoute": "/query", + "listViewName": list_view, + "resource": view["resource"], + "query_types": query_types, + "item_types": query_types, + }, + "capabilities": capabilities, + } + + +def _view_matches_context(view: dict, data_type: str | None) -> bool: + return bool(data_type and data_type in view.get("view_contexts", [])) + + +def _get_query_view(list_view: str | None) -> dict | None: + return QUERY_VIEWS.get(list_view or "") + + +def _selected_query_types(body_or_args, view: dict) -> list[str]: + if hasattr(body_or_args, "getlist"): + return ( + body_or_args.getlist("query_type") + or body_or_args.getlist("item_type") + or [view["types"][0]] + ) + return body_or_args.get("query_types") or body_or_args.get("item_types") or [view["types"][0]] + + +def _compile_rule(rule: dict, field_registry: dict) -> dict: + field_id = rule.get("field") + op_id = rule.get("operator") + value = rule.get("value") + + field_def = field_registry.get(field_id) + if not field_def: + raise ValueError(f"Unknown field: {field_id!r}") + if op_id not in field_def["operator_ids"]: + raise ValueError(f"Operator {op_id!r} not valid for field {field_id!r}") + + op_def = OPERATORS.get(op_id or "") + if not op_def: + raise ValueError(f"Unknown operator: {op_id!r}") + if op_def["value_required"] and value is None: + raise ValueError(f"Operator {op_id!r} on field {field_id!r} requires a value") + + try: + return op_def["compile"](field_def["mongo_path"], value) + except Exception as exc: + raise ValueError(f"Invalid value for {field_id!r}/{op_id!r}: {exc}") from exc + + +def _compile_node(node: dict, field_registry: dict, depth: int = 0) -> dict: + if depth > 5: + raise ValueError("Query too deeply nested (max depth 5)") + kind = node.get("kind") + if kind == "rule": + return _compile_rule(node, field_registry) + if kind == "group": + children = [_compile_node(c, field_registry, depth + 1) for c in node.get("children", [])] + children = [c for c in children if c] + if not children: + return {} + if len(children) == 1: + return children[0] + return {"$and" if node.get("combinator", "and") == "and" else "$or": children} + raise ValueError(f"Unknown node kind: {kind!r}") + + +def _encode_cursor(oid: ObjectId) -> str: + return base64.urlsafe_b64encode(str(oid).encode()).decode() + + +def _decode_cursor(s: str) -> ObjectId: + return ObjectId(base64.urlsafe_b64decode(s.encode()).decode()) + + +def _error(status: int, code: str, message: str, details: list | None = None): + return jsonify( + { + "error": { + "code": code, + "message": message, + "details": details or [], + "request_id": str(uuid.uuid4())[:8], + } + } + ), status + + +_SUMMARY_PROJECT = { + "_id": {"$toString": "$_id"}, + "item_id": 1, + "name": 1, + "chemform": 1, + "type": 1, + "date": 1, + "refcode": 1, + "status": 1, + "characteristic_chemical_formula": 1, + "nblocks": {"$size": "$display_order"}, + "nfiles": {"$size": "$file_ObjectIds"}, + "blocks": { + "$map": { + "input": {"$objectToArray": {"$ifNull": ["$blocks_obj", {}]}}, + "as": "b", + "in": {"blocktype": "$$b.v.blocktype", "title": "$$b.v.title"}, + } + }, + "creators": {"display_name": 1, "gravatar_hash": 1}, + "groups": {"display_name": 1, "group_id": 1}, + "collections": {"collection_id": 1, "title": 1}, +} + + +@QUERY.route("/query-capabilities", methods=["GET"]) +def get_query_capabilities(): + data_type = request.args.get("data_type") + views = [ + { + "view_contexts": view.get("view_contexts", []), + **_view_capability(list_view, view), + } + for list_view, view in LIST_VIEWS.items() + ] + selected = next( + ( + _view_capability(list_view, view) + for list_view, view in LIST_VIEWS.items() + if _view_matches_context(view, data_type) + ), + None, + ) + + return jsonify({"data_type": data_type, "advanced_query": selected, "views": views}) + + +@QUERY.route("/query-types", methods=["GET"]) +def get_item_types(): + list_view = request.args.get("list_view") + if not list_view: + return _error(400, "MISSING_PARAM", "list_view is required") + view = LIST_VIEWS.get(list_view) + if not view: + return _error(404, "NOT_FOUND", f"Unknown list_view: {list_view!r}") + if view.get("model_by_type"): + types = view.get("types", []) + item_types = [_query_type_entry(t, view["model_by_type"]) for t in types] + elif view.get("model"): + item_types = [_query_type_entry(list_view, {list_view: view["model"]})] + else: + item_types = [] + + return jsonify({"list_view": list_view, "item_types": item_types}) + + +@QUERY.route("/query-schema", methods=["GET"]) +def get_query_schema(): + list_view = request.args.get("list_view") + if not list_view: + return _error(400, "MISSING_PARAM", "list_view is required") + view = LIST_VIEWS.get(list_view) + if not view: + return _error(404, "NOT_FOUND", f"Unknown list_view: {list_view!r}") + # determine selected types and model map for this view + if view.get("model_by_type"): + model_map = view["model_by_type"] + selected_types = request.args.getlist("item_type") or [view.get("types", [])[0]] + if not selected_types or selected_types[0] is None: + return _error(400, "MISSING_PARAM", "item_type is required for this list_view") + for t in selected_types: + if t not in model_map: + return _error(404, "NOT_FOUND", f"Unknown item type: {t!r}") + if t not in view.get("types", []): + return _error(422, "INVALID_TYPE", f"Type {t!r} is not in list_view {list_view!r}") + field_registry = _build_field_registry_for_view(list_view, view, selected_types) + elif view.get("model"): + # single-model view; ignore item_type parameter + model_map = {list_view: view["model"]} + selected_types = [list_view] + field_registry = _build_field_registry_for_view(list_view, view, selected_types) + else: + return _error(400, "INVALID_VIEW", "This list_view has no model information") + + fields = [] + _group_order = { + "Basic": 0, + "Chemistry": 1, + "Cell": 2, + "Synthesis": 3, + "Provenance": 4, + "Other": 9, + } + for field_id, fdef in sorted( + field_registry.items(), key=lambda x: (_group_order.get(x[1]["group"], 9), x[0]) + ): + eo = fdef.get("editor_override", {}) + vs_override = fdef.get("value_schema_override", {}) + operators = [] + for op_id in fdef["operator_ids"]: + op_def = OPERATORS[op_id] + entry: dict = { + "id": op_id, + "label": op_def["label"], + "value_required": op_def["value_required"], + } + if op_def.get("editor") or op_id in eo: + entry["editor"] = eo.get(op_id, op_def.get("editor", "text")) + if op_id in vs_override: + entry["value_schema"] = vs_override[op_id] + if op_def.get("options_source"): + entry["options_source"] = op_def["options_source"] + operators.append(entry) + fields.append( + { + "id": field_id, + "label": fdef["label"], + "group": fdef["group"], + "sortable": fdef["sortable"], + "operators": operators, + } + ) + + common_type_id = selected_types[0] + model = model_map[common_type_id] + common_label = model.schema(by_alias=False).get("title", common_type_id) + + return jsonify( + { + "version": "1.0", + "list_view": list_view, + "selected_item_types": [ + {"id": t, "label": _model_label(model_map.get(t), t)} for t in selected_types + ], + "common_type": {"id": common_type_id, "label": common_label}, + "capabilities": { + "combinators": ["and", "or"], + "allow_negation": False, + "max_rules": 50, + "max_in_values": 100, + "max_depth": 5 if view.get("resource") == "items" else 0, + }, + "fields": fields, + } + ) + + +@QUERY.route("/query", methods=["POST"]) +def run_query(): + body = request.get_json(silent=True) + if not body: + return _error(400, "MALFORMED_REQUEST", "Request body must be JSON") + + list_view = body.get("list_view") + if not list_view or list_view not in LIST_VIEWS: + return _error(400, "MISSING_PARAM", "list_view is required") + + view = LIST_VIEWS[list_view] + # determine model map and selected types + if view.get("model_by_type"): + model_map = view["model_by_type"] + item_types = body.get("item_types") + if not item_types or not isinstance(item_types, list): + return _error(400, "MISSING_PARAM", "item_types must be a non-empty array") + for t in item_types: + if t not in model_map: + return _error(404, "NOT_FOUND", f"Unknown item type: {t!r}") + if t not in view.get("types", []): + return _error(422, "INVALID_TYPE", f"Type {t!r} is not in list_view {list_view!r}") + field_registry = _build_field_registry_for_view(list_view, view, item_types) + elif view.get("model"): + model_map = {list_view: view["model"]} + item_types = [list_view] + field_registry = _build_field_registry_for_view(list_view, view, item_types) + else: + return _error(400, "INVALID_VIEW", "This list_view has no model information") + where = body.get("where", {"kind": "group", "combinator": "and", "children": []}) + + try: + compiled_filter = _compile_node(where, field_registry) + except ValueError as exc: + return _error(422, "INVALID_QUERY", str(exc)) + + page_opts = body.get("page") or {} + limit = min(int(page_opts.get("limit", 50)), 200) + cursor = page_opts.get("cursor") + + if view.get("resource") == "items": + match: dict = {"type": {"$in": item_types}} + else: + match = {} + match.update(get_default_permissions(user_only=False, inherit_from_collections=False)) + if compiled_filter: + match = {"$and": [match, compiled_filter]} + + if cursor: + try: + cursor_clause = {"_id": {"$gt": _decode_cursor(cursor)}} + match = ( + {"$and": [match, cursor_clause]} + if "$and" not in match + else {**match, "$and": match["$and"] + [cursor_clause]} + ) + except Exception: + return _error(400, "INVALID_CURSOR", "Cursor is invalid") + + sort_spec = body.get("sort") or [{"field": "date", "direction": "desc"}] + mongo_sort: list[tuple] = [] + for s in sort_spec: + f = s.get("field") + direction = -1 if s.get("direction", "desc") == "desc" else 1 + if ( + f + and f in field_registry + and field_registry[f].get("mongo_path") + and field_registry[f]["sortable"] + ): + mongo_sort.append((field_registry[f]["mongo_path"], direction)) + mongo_sort = mongo_sort or [("date", -1)] + mongo_sort.append(("_id", 1)) + + pipeline = [{"$match": match}, {"$sort": dict(mongo_sort)}, {"$limit": limit + 1}] + if view.get("resource") == "items": + pipeline.extend( + [ + {"$lookup": creators_lookup()}, + {"$lookup": groups_lookup()}, + {"$lookup": collections_lookup()}, + {"$project": _SUMMARY_PROJECT}, + ] + ) + + raw = list(flask_mongo.db[view.get("collection")].aggregate(pipeline)) + has_more = len(raw) > limit + items = raw[:limit] + + next_cursor = None + if has_more and items: + raw_id = items[-1].get("_id") + if raw_id: + last_id = ObjectId(raw_id) if isinstance(raw_id, str) else raw_id + next_cursor = _encode_cursor(last_id) + + common_type_id = item_types[0] + return jsonify( + { + "query": { + "common_type": { + "id": common_type_id, + "label": model_map[common_type_id] + .schema(by_alias=False) + .get("title", common_type_id), + }, + "selected_item_types": item_types, + }, + "items": items, + "page": {"limit": limit, "next_cursor": next_cursor, "has_more": has_more}, + } + ) + + +@QUERY.route("/query-options/", methods=["GET"]) +def get_query_options(source_id: str): + if source_id != "datalab:item-reference": + return _error(404, "NOT_FOUND", f"Unknown options source: {source_id!r}") + + q = request.args.get("q", "").strip() + limit = min(int(request.args.get("limit", 20)), 100) + item_types = request.args.getlist("item_type") or list(ITEM_MODELS.keys()) + + match: dict = {"type": {"$in": item_types}} + match.update(get_default_permissions(user_only=False, inherit_from_collections=False)) + if q: + escaped = re.escape(q) + match["$or"] = [ + {"name": {"$regex": escaped, "$options": "i"}}, + {"item_id": {"$regex": escaped, "$options": "i"}}, + {"refcode": {"$regex": escaped, "$options": "i"}}, + ] + + cursor_str = request.args.get("cursor") + if cursor_str: + try: + match["_id"] = {"$gt": _decode_cursor(cursor_str)} + except Exception: + return _error(400, "INVALID_CURSOR", "Cursor is invalid") + + docs = list( + flask_mongo.db.items.find( + match, {"_id": 1, "item_id": 1, "name": 1, "refcode": 1, "type": 1, "chemform": 1} + ) + .sort("_id", 1) + .limit(limit + 1) + ) + + has_more = len(docs) > limit + docs = docs[:limit] + + options = [ + { + "value": d.get("refcode") or d["item_id"], + "label": f"{d.get('name') or d['item_id']} — {d.get('refcode') or d['item_id']}", + "metadata": { + "item_id": d["item_id"], + "refcode": d.get("refcode"), + "name": d.get("name"), + "chemform": d.get("chemform"), + "type": { + "id": d["type"], + "label": ITEM_MODELS.get(d["type"], type("", (), {"schema": lambda **_: {}})()) + .schema(by_alias=False) + .get("title", d["type"]), + }, + }, + } + for d in docs + ] + + return jsonify( + { + "options": options, + "next_cursor": _encode_cursor(docs[-1]["_id"]) if has_more and docs else None, + "has_more": has_more, + } + ) diff --git a/pydatalab/tests/server/test_query.py b/pydatalab/tests/server/test_query.py new file mode 100644 index 000000000..58a32a981 --- /dev/null +++ b/pydatalab/tests/server/test_query.py @@ -0,0 +1,114 @@ +def test_query_capabilities_are_view_driven(client): + expected = { + "samples": "samples", + "equipment": "equipment", + "startingMaterials": "starting_materials", + } + + for data_type, list_view in expected.items(): + response = client.get(f"/query-capabilities?data_type={data_type}") + assert response.status_code == 200 + assert response.json["advanced_query"]["isEnabled"] is True + assert response.json["advanced_query"]["listViewName"] == list_view + assert response.json["advanced_query"]["options"]["item_types"] + + +def test_query_schema_supports_multiple_list_views(client): + cases = { + "samples": ("samples", "chemform"), + "equipment": ("equipment", "manufacturer"), + "starting_materials": ("starting_materials", "CAS"), + } + + for list_view, (item_type, expected_field) in cases.items(): + response = client.get(f"/query-schema?list_view={list_view}&item_type={item_type}") + assert response.status_code == 200 + field_ids = {field["id"] for field in response.json["fields"]} + assert expected_field in field_ids + + +def test_query_execution_across_list_views( + client, + insert_default_sample, + insert_default_equipment, + insert_default_starting_material, +): + cases = [ + ("samples", "samples", "name", "other_sample", insert_default_sample.item_id), + ( + "equipment", + "equipment", + "manufacturer", + "science inc.", + insert_default_equipment.item_id, + ), + ( + "starting_materials", + "starting_materials", + "chemform", + "Na2CO3", + insert_default_starting_material.item_id, + ), + ] + + for list_view, item_type, field, value, item_id in cases: + response = client.post( + "/query", + json={ + "list_view": list_view, + "item_types": [item_type], + "where": { + "kind": "group", + "combinator": "and", + "children": [ + {"kind": "rule", "field": field, "operator": "eq", "value": value} + ], + }, + }, + ) + assert response.status_code == 200 + assert item_id in {item["item_id"] for item in response.json["items"]} + + +def test_query_execution_for_collections(client, database, default_collection): + database.collections.insert_one(default_collection.dict(exclude_unset=False)) + try: + response = client.post( + "/query", + json={ + "list_view": "collections", + "where": { + "kind": "group", + "combinator": "and", + "children": [ + {"kind": "rule", "field": "title", "operator": "contains", "value": "My"} + ], + }, + }, + ) + assert response.status_code == 200 + assert default_collection.collection_id in { + item["collection_id"] for item in response.json["items"] + } + finally: + database.collections.delete_one({"collection_id": default_collection.collection_id}) + + +def test_invalid_list_view_returns_error(client): + response = client.get("/query-schema?list_view=missing") + assert response.status_code == 404 + + response = client.post("/query", json={"list_view": "missing", "where": {}}) + assert response.status_code in (400, 404) + + +def test_malformed_query_does_not_500(client): + response = client.post( + "/query", + json={ + "list_view": "samples", + "item_types": ["samples"], + "where": {"kind": "rule", "field": "name", "operator": "contains"}, + }, + ) + assert response.status_code == 422 diff --git a/webapp/src/components/AdvancedQueryBuilder.vue b/webapp/src/components/AdvancedQueryBuilder.vue new file mode 100644 index 000000000..56fee7329 --- /dev/null +++ b/webapp/src/components/AdvancedQueryBuilder.vue @@ -0,0 +1,693 @@ + + + + + diff --git a/webapp/src/components/DynamicDataTable.vue b/webapp/src/components/DynamicDataTable.vue index 08c3a4e75..e703ee86c 100644 --- a/webapp/src/components/DynamicDataTable.vue +++ b/webapp/src/components/DynamicDataTable.vue @@ -9,7 +9,7 @@ v-model:filters="filters" v-model:selection="itemsSelected" v-model:select-all="allSelected" - :value="data" + :value="advancedQueryResults !== null ? advancedQueryResults : data" :data-testid="computedDataTestId" selection-mode="checkbox" paginator @@ -47,6 +47,7 @@ :selected-columns="selectedColumns" :collection-id="collectionId" :all-users="allUsersForBulk" + :advanced-query-config="advancedQueryConfig" @update:filters="updateFilters" @update:selected-columns="onToggleColumns" @open-create-item-modal="createItemModalIsOpen = true" @@ -62,6 +63,7 @@ @users-data-changed="$emit('users-data-changed')" @bulk-invalidate-tokens="handleItemsUpdated" @bulk-delete-groups="$emit('groups-data-changed')" + @advanced-query-results="handleAdvancedQueryResults" />