An end-to-end germline variant-calling platform built to production-grade clinical bioinformatics patterns

Built by Quentin Clayssen · solo-designed, solo-built, AI-orchestrated

Highlights · Architecture · Validation · Quickstart · Walkthrough · Docs · For Recruiters →

_{The Streamlit demo (demo/) — no database, cloud account, or LLM required. See demo/README.md.}

✨ Highlights

🧬 Validated like a lab Benchmarked with hap.py against the GIAB HG002 v4.2.1 truth set: SNV F1 0.9927 over all of chr20 at 33.7×, measured against a ≥ 0.99 acceptance criterion that must be re-met after any caller, reference or filter change.	🔏 Traceable by design Every result carries a provenance stamp (git commit, pipeline version, reference build, SHA-256 checksums). Results are insert-only: corrections are new records, never edits.	🤖 Responsible AI A QLoRA fine-tuned LLM drafts summaries and a ReAct agent does ACMG variant interpretation, both behind enforced guardrails and clinician sign-off.
☁️ Cloud as code AWS CDK (6 stacks: S3, DynamoDB, Step Functions, Lambda, IAM, CloudWatch) with Jest guardrail tests for its integrity invariants, plus Azure Bicep for the API and web app.	📊 Data engineering Star-schema warehouse built two ways (SQL and dbt), scheduled by Airflow, and two Metabase dashboards provisioned from version-controlled definitions.	🛠️ Engineering discipline 31 ADRs, nf-core-style Nextflow DSL2 with nf-test, pytest + Jest suites, and 11 CI workflows covering lint, security, DB integrity, Docker and release.

Tech Stack

Layer	Tools
Bioinformatics	Nextflow DSL2 · nf-test · fastp · FastQC · MultiQC · BWA-MEM2 · GATK HaplotypeCaller · DeepVariant · hap.py · bcftools · Biocontainers (digest-pinned)
Cloud & IaC	AWS CDK (TypeScript) · S3 · DynamoDB · Step Functions · Lambda · EventBridge · CloudWatch/SNS · Azure Bicep (Container Apps, Static Web Apps)
Data	PostgreSQL (insert-only + immutability triggers) · star schema · dbt · Airflow · Metabase as code · Parquet
AI / ML	PyTorch · Hugging Face Transformers + PEFT (QLoRA) · ReAct agent over ClinVar / gnomAD / ACMG · guardrails + human-in-the-loop
Apps	FastAPI (OpenAPI 3.1) · React + TypeScript + Vite · Streamlit
Quality	pytest · Jest CDK guardrail tests · GitHub Actions (11 workflows) · Dependabot · 31 ADRs

---

Overview

This platform implements the full path from sequencing data to clinical insight:

QC | Alignment | Variant Calling | Truth-Set Benchmarking | Postgres Provenance | Metabase Dashboards | LLM Report Drafting

Nextflow style, not the nf-core template. The pipeline follows nf-core conventions (DSL2, one process per module, conf/modules.config, per-process versions.yml, module meta.yml) but is hand-written — it does not include the template's subworkflows/nf-core/utils_* backbone or nf-schema validation, and would not pass nf-core lint cleanly. It does have an nf-test suite (pipeline/tests/, stub mode) covering the QC, calling and validation modules plus a full-workflow run, executed in CI.

Scope honesty. This is a portfolio project. It demonstrates the validation methodology and traceability patterns ISO 15189 / NATA accreditation requires (benchmarking against a reference truth set, insert-only provenance, audit trails, change control). It is not an accredited clinical test and must not be used for clinical decision-making.

---

Architecture

Lambda functions handle orchestration and metadata; heavy bioinformatics compute (alignment, variant calling) runs via Nextflow + Docker locally or on compute instances.

flowchart TD
    subgraph Local["Local / Compute"]
        NF["Nextflow DSL2 pipeline<br/>(QC → align → call → validate → export)"]
    end

    subgraph AWS["AWS (CDK-managed)"]
        S3["S3 Data Lake<br/>raw/ • aligned/ • called/ • validated/"]
        EB["EventBridge"]
        SF["Step Functions<br/>state machine"]

        subgraph Lambdas["Lambda Functions (orchestration)"]
            L1["ingestion_trigger"]
            L2["qc_orchestrator"]
            L3["variant_calling"]
            L4["validation_checker"]
            L5["export_handler"]
            L6["metadata_ingestor"]
            L7["report_generator"]
        end

        HEAL["healer<br/>(self-healing / retry)"]
        DDB["DynamoDB<br/>(primary metadata store)"]
        PG["Postgres<br/>(Metabase read-replica)"]
        MB["Metabase Dashboard<br/>(QC trends, turnaround, pass/fail)"]
        CW["CloudWatch + SNS<br/>(alarms & notifications)"]
    end

    subgraph AI["AI Report Layer"]
        QLORA["QLoRA-tuned LLM<br/>structured JSON → draft summary"]
        AGENT["ReAct Agent<br/>VCF → ClinVar/gnomAD/ACMG<br/>variant interpretation"]
    end

    NF -- "FASTQ / VCF / metrics.json<br/>(provenance-stamped)" --> S3
    S3 -- "object created" --> EB
    EB --> SF
    SF --> L1 --> L2 --> L3 --> L4 --> L5 --> L6 --> L7
    SF -.-> HEAL
    L6 --> DDB
    DDB -- "sync" --> PG
    PG --> MB
    L7 --> QLORA --> AGENT
    SF --> CW

Loading

---

Validation Summary

The pipeline is benchmarked on GIAB HG002 / NA24385, all of chromosome 20, at a measured 33.7× mean depth (downsampled from GIAB's 300× data — see ADR-0037), against the v4.2.1 high-confidence truth set using hap.py. Full methodology, the depth comparison and the limitations in docs/VALIDATION.md; the raw outputs are in docs/validation-evidence/HG002_chr20_35x/.

Metric	GATK HaplotypeCaller	DeepVariant	Source
SNV Precision	0.9903	—	hap.py summary.csv
SNV Recall	0.9951	—	hap.py summary.csv
SNV F1	0.9927	—	hap.py summary.csv
INDEL F1	0.9862	—	hap.py summary.csv
Truth SNVs benchmarked	71,333	—	hap.py summary.csv

SNV F1 meets the >= 0.99 acceptance criterion, with a narrow margin on precision (0.9903); the pipeline's stricter QC layer grades this run warn. DeepVariant comparison is planned.

---

Quickstart

# 1. Stage the tiny test data (subset, ships in-repo) and run the stub DAG
cd pipeline
nextflow run main.nf -profile test,docker -stub

# 2. Full local run on GIAB HG002 chr20.
#    Nothing is downloaded automatically: `-profile test` points at the tiny
#    committed placeholders, so stage the real data first (~10 GB).
./scripts/fetch_testdata.sh && ./scripts/preflight.sh
nextflow run main.nf -profile test,docker \
  --reference <staged.fa> --truth_vcf <staged.vcf.gz> --truth_bed <staged.bed>

# 3. Deploy the AWS storage/metadata layer. Compute stays local by design —
#    `-profile aws` provisions no cloud executor, it publishes results to the
#    S3 data lake and turns on metadata ingest (ADR-0017, ADR-0018).
cd ../infra && npm ci && npx cdk synth && npx cdk deploy --all
cd ../pipeline
export CGP_S3_BUCKET=... CGP_DB_URL=...
nextflow run main.nf -profile docker,aws

See docs/SOP-run-pipeline.md for the full operating procedure and acceptance criteria.

---

Repository Layout

clinical-genomics-platform/
│
├── pipeline/               Nextflow DSL2 modules: QC → align → call → validate → export
├── infra/                  AWS CDK app: S3 data lake, DynamoDB metadata, Step Functions +
│                           Lambda orchestration, scoped IAM, CloudWatch (6 stacks)
├── lambdas/                Handlers for the serverless path: ingestion, metadata, QC
│                           orchestration, validation, reporting, export, LLM healer
├── db/                     Postgres schema + migrations (samples, runs, QC, provenance, audit)
├── dashboards/metabase/    Version-controlled dashboard + question definitions
├── dbt/                    dbt analytics layer rebuilding the star schema (ADR-0025)
├── orchestration/          Airflow DAG for the warehouse ETL (ADR-0023, ADR-0026)
├── api/                    FastAPI REST service + OpenAPI docs over runs/QC/provenance
├── web/                    React + Vite UI for agentic variant review and sign-off
├── demo/                   Streamlit walkthrough over the committed fixtures
├── emr-pipeline/           Separate ICU/VVED demo: mock EMR → insert-only warehouse →
│                           LLM-structured consult notes → stub ANZICS-APD export (ADR-0029)
├── ai-report/              PyTorch QLoRA fine-tune + inference for AI-drafted summaries
├── ai-report/agent/        ReAct variant interpretation agent (ACMG classification)
├── docker/                 Dockerfile.tools (helper scripts) + Dockerfile.demo; per-stage
│                           tool containers are pinned in each module's `container` directive
├── scripts/                fetch_testdata.sh, preflight.sh, and other helpers
├── docs/                   Validation report, SOP, beginner's guide, glossary, ADRs
├── tests/                  Unit tests + small committed fixtures
└── .github/workflows/      CI/CD: lint, security, DB validation, coverage, Docker, release, maintenance

---

Documentation

Audience	Resource
New to the domain	Beginner's Guide and Glossary
Recruiter / Hiring manager	For Recruiters
Technical decisions	Architecture Decision Records
Accuracy and benchmarking	Validation Report
ML component	Model Card and ADR-0007
Agentic AI	Agent Design and Agent Model Card
Operations	SOP and Runbook
Standards interoperability	GA4GH Alignment

---

Milestones

The build is demoable at every stage — see docs/MILESTONES.md for the full timeline.

---

Dashboard Preview

_{QC trends, turnaround time, and validation pass/fail — powered by Metabase over Postgres, provisioned entirely as code (dashboards/metabase/dashboard_manifest.yaml + provision_metabase.py).}

_{Both dashboards, scrolled through live — not a mockup.}

Ops Validation pass rate, per-run turnaround, runs/week throughput.

Analytics SNV F1 by pipeline version, turnaround SLA p50/p95, cohort explorer.

Reproduce it yourself: docker compose up -d, complete Metabase's one-time setup at localhost:3000, then MB_USERNAME=... MB_PASSWORD=... python dashboards/metabase/provision_metabase.py — see dashboards/metabase/README.md.

---

Capability Walkthrough

_{Every screenshot is a real capture of the running service, not a mockup. Click to enlarge.}

Agentic variant review (React + FastAPI) The agent calls ClinVar / gnomAD / ACMG tools step by step; a clinician signs off through the insert-only review-decisions endpoint. web/ · ADR-0027	REST API (FastAPI, OpenAPI 3.1) uvicorn api.main:app --reload. Fixture-backed by default, no database required.
Variant interpretation (Streamlit) Agent reasoning trace and ACMG classification over committed fixtures.	Pipeline data explorer (Streamlit) KPIs, filters and the SNV F1 trend. PYTHONPATH=. streamlit run demo/app.py
Submit a variant (React) Entry point to the agentic review flow.	dbt lineage Sources → staged views → star-schema marts. dbt/ · ADR-0025

---

Contributing

This is a solo portfolio project. Issues, questions, and feedback are welcome — open an issue or start a discussion. Unsolicited pull requests are unlikely to be merged, but feel free to fork.

License

MIT — see LICENSE. Reference data (GIAB) is public domain / CC0 from NIST.