Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
328 lines (226 loc) · 21.7 KB

CONTRIBUTING.md

File metadata and controls

328 lines (226 loc) · 21.7 KB
Raw
title Contributing
markdownPlugin checklist

nf-core/scdownstream: Contributing guidelines

Hi there! Thanks for taking an interest in improving nf-core/scdownstream.

This page describes the recommended nf-core way to contribute to both nf-core/scdownstream and nf-core pipelines in general, including:

Note

If you need help using or modifying nf-core/scdownstream, ask on the nf-core Slack #scdownstream channel (join our Slack here).

General contribution guidelines

Contribution quick start

To contribute code to any nf-core pipeline:

  • Ensure you have Nextflow, nf-core tools, and nf-test installed. See the nf-core/tools repository for instructions.
  • Check whether a GitHub issue about your idea already exists. If an issue does not exist, create one so that others are aware you are working on it.
  • Fork the nf-core/scdownstream repository to your GitHub account.
  • Create a branch on your forked repository and make your changes following pipeline conventions (if applicable).
  • To fix major bugs, name your branch patch and follow the patch release process.
  • Update relevant documentation within the docs/ folder, use nf-core/tools to update nextflow_schema.json, and update CITATIONS.md.
  • Run and/or update tests. See Testing for more information.
  • Lint your code with nf-core/tools.
  • Submit a pull request (PR) against the dev branch and request a review.

If you are not used to this workflow with Git, see the GitHub documentation or Git resources for more information.

Use of AI and LLMs

The nf-core stance on the use of AI and LLMs is that humans are still ultimately responsible for their submitted code, regardless of the tools they use.

If you're using AI tools, try to stick by these guidelines:

  • Keep PRs as small and focused as possible
  • Avoid any unnecessary changes, such as moving or refactoring code (unless that is the explicit intention of the PR)
  • Review all generated code yourself before opening a PR, and ensure that you understand it
  • Engage with the community review process and expect to make revisions

For more detail, see the blog post for a statement from the nf-core/core team.

Getting help

For further information and help, see the nf-core/scdownstream documentation or ask on the nf-core #scdownstream Slack channel (join our Slack here).

GitHub Codespaces

You can contribute to nf-core/scdownstream without installing a local development environment on your machine by using GitHub Codespaces.

GitHub Codespaces is an online developer environment that runs in your browser, complete with VS Code and a terminal. Most nf-core repositories include a devcontainer configuration, which creates a GitHub Codespaces environment specifically for Nextflow development. The environment includes pre-installed nf-core tools, Nextflow, and a few other helpful utilities via a Docker container.

To get started, open the repository in Codespaces.

Testing

Once you have made your changes, run the pipeline with nf-test to test them locally. For additional information, use the --verbose flag to view the Nextflow console log output.

nf-test test --tag test --profile +docker --verbose

If you have added new functionality, ensure you update the test assertions in the .nf.test files in the tests/ directory. Update the snapshots with the following command:

nf-test test --tag test --profile +docker --verbose --update-snapshots

When you create a pull request with changes, GitHub Actions will run automatic tests. Pull requests are typically reviewed when these tests are passing.

Two types of tests are typically run:

Lint tests

nf-core has a set of guidelines which all pipelines must follow. To enforce these, run linting with nf-core/tools:

nf-core pipelines lint <pipeline_directory>

If you encounter failures or warnings, follow the linked documentation printed to screen. Generated pipeline logos can differ across CPU architectures. A local files_unchanged lint failure for a logo can therefore occur even when CI passes. Leave the logo images unchanged and use the CI result for this check. For more information about linting tests, see nf-core/tools API documentation.

Pipeline tests

Each nf-core pipeline should be set up with a minimal set of test data. GitHub Actions runs the pipeline on this data to ensure it runs through and exits successfully. If there are any failures then the automated tests fail. These tests are run with the latest available version of Nextflow and the minimum required version specified in the pipeline code.

Patch release

Warning

Only in the unlikely event of a release that contains a critical bug.

  • Create a new branch patch on your fork based on upstream/main or upstream/master.
  • Fix the bug and use nf-core/tools to bump the version to the next semantic version, for example, 1.2.3 to 1.2.4.
  • Open a Pull Request from patch directly to main/master with the changes.

Pipeline contribution conventions

nf-core semi-standardises how you write code and other contributions to make the nf-core/scdownstream code and processing logic more understandable for new contributors and to ensure quality.

Add a new pipeline step

To contribute a new step to the pipeline, follow the general nf-core coding procedure. Please also refer to the pipeline-specific contribution guidelines:

  • Define the corresponding input channel into your new process from the expected previous process channel.
  • Install a module with nf-core/tools, or write a local module (see default processes resource requirements), and add it to the target <workflow>.nf.
  • Define the output channel if needed. Mix the version output channel into ch_versions and relevant files into ch_multiqc.
  • Add new or updated parameters to nextflow.config with a default value.
  • Add new or updated parameters and relevant help text to nextflow_schema.json with nf-core/tools.
  • Add validation for relevant parameters to the pipeline utilisation section of utils_nfcore_\_pipeline/main.nf subworkflow.
  • Perform local tests to validate that the new code works as expected.
    • If applicable, add a new test in the tests directory.
  • Update usage.md, output.md, and citation.md as appropriate.
  • Lint the code with nf-core/tools.
  • Update any diagrams or pipeline images as necessary.
  • Update MultiQC config assets/multiqc_config.yml so relevant suffixes, file name cleanup, and module plots are in the appropriate order.
  • If applicable, create a MultiQC module.
  • Add a description of the output files and, if relevant, images from the MultiQC report to docs/output.md.

To update the minimum required Nextflow version, see the Nextflow version bumping section below. For more information about pipeline contributions, see pipeline-specific contribution guidelines.

Channel naming schemes

Use the following naming schemes for channels to make the channel flow easier to understand:

  • Initial process channel: ch_output_from_<process>
  • Intermediate and terminal channels: ch_<previousprocess>_for_<nextprocess>

Default parameter values

Parameters should be initialised and defined with default values within the params scope in nextflow.config. They should also be documented in the pipeline JSON schema. Declare boolean and numeric parameter types in the params block in main.nf, without duplicating their defaults. Use nullable numeric types, such as Integer? and Float?, when the default is null. Keep string parameters and the boolean-or-string help parameter untyped.

To update nextflow_schema.json, run:

nf-core pipelines schema build

The schema builder interface that loads in your browser should automatically update the defaults in the parameter documentation.

Default processes resource requirements

If you write a local module, specify a default set of resource requirements for the process.

Sensible defaults for process resource requirements (CPUs, memory, time) should be defined in conf/base.config. Specify these with generic withLabel: selectors, so they can be shared across multiple processes and steps of the pipeline.

nf-core provides a set of standard labels that you should follow where possible, as seen in the nf-core pipeline template. These labels define resource defaults for single-core processes, modules that require a GPU, and different levels of multi-core configurations with increasing memory requirements.

Values assigned within these labels can be dynamically passed to a tool using the ${task.cpus} and ${task.memory} Nextflow variables in the script: block of a module (see an example in the modules repository).

Nextflow version bumping

If you use a new feature from core Nextflow, bump the minimum required Nextflow version in the pipeline with:

nf-core pipelines bump-version --nextflow . <min_nf_version>

Images and figures guidelines

If you update images or graphics, follow the nf-core style guidelines.

Pipeline specific contribution guidelines

Harshil alignment

Use Harshil alignment where applicable. Do not add more spaces than the widest element requires.

Linting and formatting

Stage your changes and run prek before each commit. The hooks in .pre-commit-config.yaml run Prettier, nextflow lint, and ruff for Python code. The ruff rules are configured in ruff.toml.

Local modules

Local modules follow the nf-core module specifications.

Meta map fields

In addition to id, local modules and subworkflows use the following meta fields:

Field Set by Meaning
batch_col Samplesheet obs column holding batch labels, used for batch-aware QC and feature selection.
symbol_col Samplesheet var column holding gene symbols.
counts_layer Samplesheet, ambient correction, unification Selected counts source; defaults to X and becomes X after unification.
condition_col, donor_col Samplesheet obs columns used for differential expression and pseudobulking.
integration Integration step Name of the integration method that produced the embedding.
subset CLUSTER_TARGETS, SUB_INTEGRATE global for the full dataset, otherwise the name of the subset.
resolution Clustering step Leiden resolution of the clustering.
obs_key workflows/scdownstream.nf obs column that defines the groups for per-group analyses.
analyses, de_methods_resolved Analysis plan (utils_nfcore_scdownstream_pipeline) Per-group analyses and DE methods to run. null means all.

Channel operations must preserve these fields. Add new fields to this table when you introduce them.

Ambient correction resolves the expression-source fields before unification:

Field Before unification After unification
counts_layer Physical source of selected analysis counts. X.
raw_counts_layer Selected original input source before this correction stage. raw_counts.
ambient_correction_layer Generated correction source, or null when bypassed. ambient_corrected_<method>, or null.
ambient_correction_method Applied method, or none. Unchanged.
analysis_counts_source raw_counts or canonical ambient_corrected_<method>. Unchanged.

An explicit sample selection flag overrides the global flag. Disabled correction and method none select raw counts. Unification preserves required sources before replacing reserved names, then applies a common gene grouping and order to X and every retained layer. X holds the selected analysis counts; raw_counts retains the original raw counts, and ambient_corrected_<method> retains corrected counts. Normalisation writes derived layers such as log1p and scran without replacing X.

The two provenance fields also become categorical obs columns. Merging fills unavailable method layers from each sample's raw_counts and retains provenance independently of the ordinary observation-column intersection. Historical bases require explicit raw_counts. Missing historical provenance remains unknown, without inference from layer names.

Process templates

Many processes in this pipeline use templates containing Python or R scripts. Nextflow interprets strings prefixed with $ as Nextflow variables - this is especially relevant for R scripts. The process fails if such a variable is not defined in the associated Nextflow process. To use a literal $ character in a template, escape it with a backslash (\$). In R templates, this applies to every use of the $ operator, for example adata\$as_SingleCellExperiment() or rowData(sce)\$highly_deviant.

Environment variables that apply to all Python processes, such as NUMBA_CACHE_DIR and MPLCONFIGDIR, are set in the env scope of nextflow.config. Do not set them in individual templates. Pass task.cpus to the tool explicitly, for example with threadpool_limits(int("${task.cpus}")) in Python. Set a fixed random seed for stochastic methods, for example with set.seed() in R.

Write H5AD outputs with compression="gzip" in Python or compression = "gzip" in R. Keep the write call and its compression argument on the same line for the h5ad-gzip pre-commit check.

Templates should follow this structure:

  1. Read inputs.
  2. Perform processing.
  3. Write outputs:
    • Primary outputs, such as updated AnnData files.
    • Fragments, such as added obs columns or layers, in relevant formats.
    • MultiQC inputs.
    • versions.yml containing the versions of the language and all relevant packages.

Fragments

A fragment contains only the data that a module adds to an AnnData object. workflows/scdownstream.nf collects fragments on the ch_obs, ch_var, ch_obsm, ch_obsp, and ch_uns channels, and on their _per_sample counterparts (including ch_layers_per_sample) before combination. ADATA_EXTEND merges them into the H5AD files, once per sample after quality control (FINALIZE_QC_ANNDATAS) and once at the end of the pipeline (FINALIZE). Emit each fragment on a channel named after its AnnData slot and use a format that modules/local/adata/extend can read:

Slot Format Index
obs Pickled or CSV pandas.DataFrame obs_names
var Pickled or CSV pandas.DataFrame var_names
obsm Pickled pandas.DataFrame, file name X_<name>.pkl obs_names
obsp NumPy .npy file containing a sparse matrix None
uns Pickled Python object None
layers .npz, .mtx, or .npy matrix Same cell and gene order as the input

The file stem becomes the key in obsm, obsp, uns, and layers.

MultiQC custom content

Modules that produce plots for the MultiQC report write a <prefix>_mqc.json file with an embedded base64 PNG image. Set parent_id, parent_name, and parent_description from meta.integration so that MultiQC groups the section with the other results of the same integration. modules/local/scanpy/leiden/templates/leiden.py contains a reference implementation. Add new sections to assets/multiqc_config.yml in the order of the pipeline steps.

Output directories

conf/modules.config publishes results into numbered top-level directories that follow the order of the pipeline steps, for example 02_quality_control or 05_cluster_dimred. Place outputs of new steps into the matching directory. Guard intermediate outputs with enabled: params.save_intermediates. Update docs/output.md with the new files.

Python-only profile

The python_only profile in conf/python_only.config replaces R-based default tools with Python alternatives. If you add an R-based tool as a new default, add a Python alternative to this profile. If no alternative exists, document the limitation in the comment block of conf/python_only.config.

Testing conventions

Write pipeline tests in nf-test and extend the existing module or subworkflow .nf.test suites where applicable. Do not add standalone Python tests or test-only processes for assertions.

Ambient count-retention tests use the existing nft-anndata metadata interface: layer keys, dimensions, cell/gene indices, categorical provenance, and existing scalar QC fields. Assertions must not read, decode, compare, sum, or slice X or layer arrays. Do not add matrix-reading helpers, backend readers, or production diagnostic passes solely for tests. Use sequential assertions and place the snapshot assertion last.

These metadata checks do not prove equality of retained matrix values or exact raw-fallback values. Review source copying, alias handling, sparse-zero aggregation, and common axes directly. RDS round trips use the production conversion modules and inspect resulting AnnData metadata, not expression arrays.

CI runs three nf-test categories, selected by tag:

  • modules_local for local module tests.
  • subworkflows_local for local subworkflow tests.
  • pipeline for end-to-end tests in tests/.

Tag new tests accordingly.

docs/reproducibility.md classifies every local module and subworkflow by the reproducibility of its outputs and assigns a snapshot strategy to each. When you add a module or subworkflow, add a row to this file and write the test according to the assigned strategy. Snapshot files must never be edited by hand; regenerate them with nf-test test --update-snapshot.

Test data must not be committed to this repository. Generate test inputs in nf-test setup blocks with existing production modules or subworkflows on data from test-datasets (for example DECOUPLER_PSEUDOBULK followed by EDGEPYTHON_DIFFERENTIAL to obtain DE tables). Do not create modules or processes solely to generate fixtures. Only add a fixture if no module can produce it, or if the setup modules would make the test too slow. Store such fixtures, and the scripts that generate them, in the scdownstream branch of nf-core/test-datasets and reference them through params.pipelines_testdata_base_path.

For AnnData API usage, consult the nf-anndata README and nft-anndata README.