OMOP samples

[ofermend]

What this adds

• A new OMOP sample showing how to analyze observational health data in the OMOP Common Data Model using Malloy. OMOP is the dominant open standard (maintained by OHDSI) for standardizing electronic health records, used across hundreds of academic medical centers and research networks. The sample is built around a generic semantic model (omop.malloy) that conforms to OMOP CDM v5.3 and v5.4.
• We use here the synthea-covid19-10k dataset (~10,700 synthetic patients, OMOP CDM v5.3) which uses Synthetic data (no PHI).

The model captures some OMOP-specific querying challenges Malloy handles well:

• Symmetric aggregates: counting conditions, drugs, visits, and procedures per patient in one query without fan-out errors.
• Reusable vocabulary joins: the concept dictionary reused under plain-English aliases (gender, drug_concept, condition_concept, race, ethnicity) so queries ask for readable names instead of numeric codes.
• Named views: cohort logic and demographic breakdowns defined once and reused across analyses.

The specific dataset has no laboratory results (measurement/observation tables are empty) and unpopulated drug dosing fields, so lab/vital and prescription-length analyses are out of scope. Noted in the README.

[jswir]

On the "Table 1" layout discussion — I dug into why the vertical (journal-style) version needs so much more Malloy than the horizontal nested one, and tested an alternative. Sharing in case it's useful (all of this is run against the Synthea COVID data in this PR and reproduces the vertical table cell-for-cell).

Why the vertical version is hard: it needs two things Malloy has no native primitive for — (1) wide→long / UNPIVOT to stack Sex/Race/Ethnicity/Age-band into one column, and (2) mixed row formats in one column (mean (SD) for continuous rows, n (%) for categorical). I confirmed via the docs that Malloy has no native UNION and no native UNPIVOT today, so the array-literal line trick in this PR is genuinely the idiomatic pure-Malloy way to emulate UNPIVOT. The complexity is inherent, not accidental.

A simpler alternative — write each section as a plain block query that emits the same columns, and let SQL do only the stacking via %{ } query interpolation. Each block formats its own column, so the pick … when ord ladder disappears:

query: blk_age is covid_cohort -> {
  group_by: ord is 2, Characteristic is 'Age at index — yr, mean (SD)', Subgroup is ''
  aggregate:
    Survived is concat(round(age_at_index.avg() {where: vital_status='Survived'},1)::string, ' (',
                       round(age_at_index.stddev() {where: vital_status='Survived'},1)::string, ')')
    Died     is concat(round(age_at_index.avg() {where: vital_status='Died'},1)::string, ' (',
                       round(age_at_index.stddev() {where: vital_status='Died'},1)::string, ')')
}
query: blk_sex is covid_cohort -> {
  group_by: ord is 5, Characteristic is 'Sex — no. (%)', Subgroup is sex
  aggregate:
    Survived is concat(count() {where: vital_status='Survived'}::string, ' (',
      round(count() {where: vital_status='Survived'} / all(count() {where: vital_status='Survived'}) * 100, 1)::string, '%)')
    Died     is concat(count() {where: vital_status='Died'}::string, ' (',
      round(count() {where: vital_status='Died'} / all(count() {where: vital_status='Died'}) * 100, 1)::string, '%)')
}
// ... one block per section (patients, visits, follow-up, race, ethnicity, age band) ...

source: table1 is duckdb.sql("""
  SELECT * FROM (%{ blk_patients })
  UNION ALL BY NAME SELECT * FROM (%{ blk_age })
  UNION ALL BY NAME SELECT * FROM (%{ blk_sex })
  -- ...the rest...
""")

run: table1 -> { order_by: ord, Subgroup; select: Characteristic, Subgroup, Survived, Died }

Honest tradeoff: this drops to the duckdb.sql() escape hatch, so it's DuckDB-bound and less "pure Malloy" — for a teaching sample, the array-unnest version is arguably the better example precisely because it stays native. (UNION ALL BY NAME is also DuckDB-specific; plain positional UNION ALL is portable.) So I'd treat this as an optional third cell that illustrates the pragmatic path, not a replacement.

The real takeaway / possible feature: the fact that the clean version requires either an obscure array-unnest or a SQL escape hatch is good evidence for a language feature — a native unpivot / wide-to-long (or first-class union) — and/or a renderer that stacks sibling nest:s vertically under shared columns (the nested query already computes 100% of the data; only the layout differs). # transpose exists but only flips a single-row, many-measure result; it doesn't merge sibling nests into one column.

Separate heads-up (unrelated to layout): on the repo's pinned @malloydata/malloy@0.0.244, covid_cohort failed to compile for me — n_visits is patient.total_visits errors visits not found (a named measure defined as a join-many visits.count(), referenced through join_one: patient). Writing it inline as patient.visits.count() compiles and runs fine. Worth checking it doesn't trip npm test / CI on the pinned version (it may already be fixed in a newer runtime).

[ofer-mendelevitch-thenvoi]

Thanks @jswir for the investigation about table-1 and detailed explanation. I reflected your thoughts in the PR itself to capture that nuance until such time as Malloy has better solution for this.

[mtoy-googly-moogly]

Really like this sample — the OMOP model and the symmetric-aggregate / vocabulary-join story are great, and the package bump is clean this time (exact pins, no stray deps). One blocker around CI, though.

The OMOP data is gitignored and fetched on-demand, but the test harness isn't told to skip these notebooks — so this will turn main red on merge.

Every omop/*.malloynb does import 'omop.malloy' → omop_synthea_covid.malloy → duckdb.table('omop/data/*.parquet'). Those parquet files aren't committed (they come from setup.sh's ~840 MB download). The DuckDB test suite (tests/duckdb.spec.ts) globs every top-level dir not in EXCEPT_LIST and compiles each .malloynb; omop isn't excluded, so all ~12 notebooks fail at compile with failed-to-fetch-table-schema (IO Error: No files found that match the pattern "omop/data/concept.parquet"). I reproduced this locally.

The fork-PR run masks it (it dies earlier at the GCloud-auth step because fork PRs don't get secrets), but on push-to-main secrets are present, the full suite runs, and it'll fail on these notebooks. It also breaks the repo's stated invariant that the DuckDB suite runs with no setup.

Fix — pick one (there's precedent: apijson.malloynb is in EXCEPT_LIST because it needs the network):

1. Add 'omop' to EXCEPT_LIST in tests/duckdb.spec.ts (and the bigquery spec) so the suite skips it. Simplest; downside is nothing validates these notebooks in CI.
2. Add a separate opt-in CI job that runs omop/scripts/setup.sh and then compiles only omop/. Real coverage, at the cost of the 840 MB download per run.
3. Commit the parquet (presumably why it was gitignored — likely too big).

My preference is 1, optionally plus 2 as a dedicated job. Happy to wire up whichever you pick.