[adRn-s]

Why

dissectBCL --forcelanesplit (-F) can put one project into <flowcell>_lanes_1 and <flowcell>_lanes_2, each with only that lane's samples (e.g. a Multiome project: lane 1 GEX, lane 2 ATAC). BRB processed the project in each lane folder with all its Parkour samples, picked the pipeline from the first sample's protocol, and had no Multiome support.

What changes

• Per-lane samples: in a *_lanes_<n> folder a project keeps only samples that have FASTQs there. A project left with none is skipped and logged. Normal and _lanes_1_2 folders are unchanged.
• One pipeline per protocol: an analysis type with several protocols is split, and each part gets its own output dir (_<protocol> suffix). Single-protocol paths are unchanged.
• Multiome (scMultiome): Chromium_NextGEM_Multiome_GEX/_ATAC go to a new pipeline that runs the 10x_ARC wrapper (feat: add 10x_ARC wrapper for joint cellranger-arc Multiome runs 10X_snakepipe#21) once per <base>_RNA + <base>_ATAC pair.
  • The first lane pass that finds both halves ready runs the pair. A half counts as ready if its FASTQs are in this folder, or in a sibling lane folder that has fastq.made. Lanes can be processed in any order.
  • Pairs with <base>.done in a sibling lane's output are not rerun.
  • A group still waiting for the other lane is not marked analysis.done.
  • FASTQ names are read from disk, because dissectBCL rewrites umlauts and spaces.
• Pairs that can't run (waiting for another lane, missing partner in Parkour, duplicate base name, one half in several lane folders) are:
  • logged as warnings;
  • written to multiome.attention in the output dir;
  • shown as ATTENTION in the finished e-mail, which holds back the deepSeq notice;
  • printed in red by run_brb.
• Crash teardown race: runOneGroup now re-checks registry.aborted after writing running.pid. Before, a teardown between the check and the write left a live marker behind. This is what made TestCrashAndRestart fail now and then under load. A new regression test forces the timing.
• BigRedButton.ini: [10x] ARC= added.

Known limitations

• GEX and ATAC on different flowcells: Parkour returns one half only, so the pair is reported as unpaired and has to be run by hand.
• Pairing depends on the _RNA/_ATAC name endings.
• If the partner lane is never demultiplexed, the ATTENTION row is the only signal.

A possible follow-up is a Parkour endpoint with per-library lanes, flowcells and a Multiome partner field. Not part of this PR.

Testing

• pytest: 388 passed. ruff check BRB tests: clean.
• New tests/test_PushButton_lanesplit.py (fake flowcell in tmp_path, modelled on a real -F flowcell): per-lane dispatch, protocol split, Multiome lane order, hand-over, done-elsewhere, unpaired, duplicate, both-lanes, umlaut names, note cleanup. ATTENTION e-mail test in tests/test_email.py.
• End-to-end smoke with a fake cellranger-arc: runFlowcell → 10x_ARC → snakemake, lanes processed in the order 2, 1, 2. 2 arc calls (one per pair), no reruns.

Before deploying

• Run the real cellranger-arc on rapidus from an editable env of 10X_snakepipe#21.
• Set ARC= in BigRedButton_dev.ini. _prod only after release.

[adRn-s]

Reviewed incl. the teardown race fix. Test suite on e5032bb: 389 passed locally.

Race fix (runOneGroup): correct. Leftover marker case (worker passes check → teardown clears absent marker → worker writes marker) is closed by re-checking after writeMarker. Comments inline: first check now redundant; one residual window remains (driver started after cancelGroup snapshot) — pre-existing, OK as follow-up.

Multiome: design reads fine. Blockers before merge:

1. ATAC FASTQ read set — see 10X_snakepipe#21; wrapper hardcodes R1/R2/I1/I2, BRB passes names only from _R1.
2. dm6/GRCz11 unreachable via arcGenomeKey (inline).
3. Retry after a failed cellranger-arc run likely hits the pipestance _lock (see #21).
4. Real cellranger-arc run on rapidus (already noted in PR).

Also note: flowcell-level analysis.done is still touched by run.py → markFinished even when pairs are waiting, so the lane folder is never revisited. Fine because the other lane's pass runs the pair, but the "group not marked analysis.done" part has no effect on its own — worth saying so in the docstring.

[adRn-s]

Update after inspecting the real run (261028_A00931_0938_BHVLWKDRX7, project 4077, mm10, processed with #21 + BRB#174):

• Retracting blocker 1 (ATAC read names). dissectBCL output is <lib>_ATAC_{R1,R2,I1,I2}.fastq.gz (no R3, no _UMI_), so the hardcoded set matches. fq_ln symlinks + libraries.csv are correct; GEX from _lanes_1, ATAC from _lanes_2.
• Both pairs Pipestance completed successfully!, 3 of 3 steps (100%) done, full outs/ (web_summary, fragments, h5, cloupe). genome resolved to the mm10 ARC ref.
• Pair ran from the _lanes_1 pass; _lanes_2/Analysis_4077_Xing_Akhtar/Multiome_mm10/ holds only analysis.done → cross-lane pairing works as designed.

Still open: _lock on retry after a failed run (not exercised: no failure), dm6/GRCz11 genome-key normalisation, stale-symlink rerun, log overwrite. None block this mm10 use case.

[adRn-s]

Handoff for the next agent session

Related PRs (one feature: 10x Multiome with GEX and ATAC on separate lanes): dissectBCL#364 (docs) -> BigRedButton#174 (pipeline) -> 10X_snakepipe#21 (10x_ARC wrapper). Reviews are already posted on all three; this lists what is still open.

Verified on a real run (/data/akhtar/sequencing_data12/261028_A00931_0938_BHVLWKDRX7_lanes_1/Analysis_4077_Xing_Akhtar/Multiome_mm10, run with the code of #21 + #174):

• dissectBCL ATAC output is <lib>_ATAC_{R1,R2,I1,I2}.fastq.gz, which matches the wrapper's hardcoded set. No change needed there.
• GEX from _lanes_1, ATAC from _lanes_2 pair correctly. libraries.csv is correct. Both pairs finished (Pipestance completed successfully!, 3/3 steps), with full outs/. _lanes_2 holds only analysis.done.
• BRB test suite on the fix: handle lane-split flowcells and run 10x Multiome with cellranger-arc #174 head: 389 passed (SETUPTOOLS_SCM_PRETEND_VERSION=0.0.1 uv run --with pytest pytest -q).

Warnings

• Contract between fix: handle lane-split flowcells and run 10x Multiome with cellranger-arc #174 and ET: telegraphHome needs to account for different location #21: wrapper CLI (--pair ID:GEX:ATAC, --gex-indir, --atac-indir, -o), [10x] ARC= config key, and the <outdir>/<pair>.done file. Change one side -> change the other.
• Do NOT edit BigRedButton_prod.ini / *_prod configs until the PR is merged and released, and only with explicit human confirmation. _dev files may be updated alongside the PR.
• Ask before every git push. No state-changing commands on rapidus/parkour hosts without confirmation.
• The failure paths below were NOT exercised on a real run (the real run had no failures).

Open items for THIS PR (BigRedButton#174), in priority order

1. Genome key - BRB/PushButton.py arcGenomeKey (~L38-45): fixed list lacks dm6/GRCz11. Resolve together with 10X_snakepipe#21 item 2 (normalise in the wrapper, remove this helper and its callers/tests).
2. Run race fix (runOneGroup) - fix is correct (re-check registry.aborted after writeMarker, ~L1310, with local marker cleanup). Cleanup: the earlier check at ~L1306-1308 is now redundant; drop it so there is one check point.
3. Residual race, pre-existing, follow-up OK - abort latched after that check: cancelAllGroups snapshots the handle before runManagedSubprocess does Popen + add_process, so the driver can start unkilled. Idea: GroupHandle.cancelled flag set in cancelGroup, checked under handle._lock in add_process (kill immediately if set). Needs a regression test if done.
4. Multiple wrapper calls - scMultiome ~L1209 builds one call per (gexDir, atacDir) group into the same -o; only the last call's 10X_BRB.log survives and 10X_ARC.config.yaml is overwritten. Coordinate with 10X_snakepipe#21 item 4.
5. Stale note - multiome.attention (~L1194, MULTIOME_NOTE) written in the lane's output dir says "waiting" and is not updated when the sibling lane's pass runs the pair. Say in the note text that it reflects this pass only (not seen in the real run).
6. Docstring - run.py still calls markFinished (touches flowcell-level analysis.done) even when pairs are waiting; document that the hold-back only works because the sibling lane's pass runs the pair.
7. ARC= must exist in BigRedButton_dev.ini (dev only, see warnings).