Skip to content

TSPS-1127 make all disk sizes dynamic and set bootDiskSizeGb to 0 for all tasks #1925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jsotobroad merged 7 commits into from
Aug 17, 2026
+58 −28
Merged

TSPS-1127 make all disk sizes dynamic and set bootDiskSizeGb to 0 for all tasks #1925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
afbe62c
make all disk sizes dynamic and set bootDiskSizeGb to 0 for all tasks
Aug 14, 2026
1d71fa6
forgot a "gb" param in size call
Aug 14, 2026
e0bf173
update changelog and pipeline versions
Aug 15, 2026
4110662
Updated pipeline_versions.txt with all pipeline version information
actions-user Aug 15, 2026
124b3cb
set default sample batch size to 1000
Aug 17, 2026
5c6d3fc
update changelog
Aug 17, 2026
8603c13
kick tests
Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions pipeline_versions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@ Glimpse2LowPassImputation 1.0.6 2026-08-12
Glimpse2LowPassImputationBatch 1.0.4 2026-08-12
Glimpse2LowPassImputationQC 1.0.5 2026-07-20
Glimpse2LowPassImputationQuotaConsumed 1.0.0 2026-06-30
Glimpse2SVImputation 0.0.18 2026-08-13
Glimpse2SVImputationBatch 0.0.13 2026-08-13
Glimpse2SVImputation 0.0.19 2026-08-17
Glimpse2SVImputationBatch 0.0.14 2026-08-17
IlluminaGenotypingArray 1.12.27 2026-01-21
Imputation 1.1.23 2025-10-03
ImputationBeagle 3.0.1 2026-02-23
JointGenotyping 1.7.3 2025-08-11
MultiSampleSmartSeq2SingleNucleus 2.2.8 2026-07-10
MultilevelHierarchicallyPasteVcfsStreaming 0.0.5 2026-08-13
MultilevelHierarchicallyPasteVcfsStreaming 0.0.6 2026-08-17
Multiome 7.0.2 2026-07-10
Optimus 9.2.0 2026-07-10
PairedTag 3.0.2 2026-07-10
PeakCalling 1.0.1 2025-08-11
Pipeline Name Version Date of Last Commit
PreprocessPLsGVCF 0.0.10 2026-08-13
PreprocessPLsGVCF 0.0.11 2026-08-17
RNAWithUMIsPipeline 1.0.20 2026-01-21
ReblockGVCF 2.4.4 2026-01-29
SlideSeq 3.6.8 2026-07-10
Expand Down
6 changes: 6 additions & 0 deletions pipelines/wdl/glimpse/sv_imputation/Glimpse2SVImputation.changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
# 0.0.19
2026-08-17 (Date of Last Commit)

* make all task disk sizes dynamic and set bootDiskSizeGb to 0
* update default value of sample_batch_size to 1000

# 0.0.18
2026-08-13 (Date of Last Commit)

Expand Down
8 changes: 4 additions & 4 deletions pipelines/wdl/glimpse/sv_imputation/Glimpse2SVImputation.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ import "./Glimpse2SVImputationBatch.wdl" as Glimpse2SVImputationBatch
import "../../../../tasks/wdl/Glimpse2SVImputationTasks.wdl" as Glimpse2SVImputationTasks

workflow Glimpse2SVImputation {
String pipeline_version = "0.0.18"
String preprocess_pls_gvcf_pipeline_version = "0.0.10"
String batch_pipeline_version = "0.0.13"
String pipeline_version = "0.0.19"
String preprocess_pls_gvcf_pipeline_version = "0.0.11"
String batch_pipeline_version = "0.0.14"

input {
# if both array inputs and gvcf_manifest are provided, array inputs take precedence
Array[File]? input_gvcfs
Array[File]? input_gvcf_idxs
Array[String]? sample_ids
File? gvcf_manifest
Int sample_batch_size = 500
Int sample_batch_size = 1000

String output_prefix

Expand Down
5 changes: 5 additions & 0 deletions pipelines/wdl/glimpse/sv_imputation/Glimpse2SVImputationBatch.changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 0.0.14
2026-08-17 (Date of Last Commit)

* make all task disk sizes dynamic and set bootDiskSizeGb to 0

# 0.0.13
2026-08-13 (Date of Last Commit)

Expand Down
19 changes: 10 additions & 9 deletions pipelines/wdl/glimpse/sv_imputation/Glimpse2SVImputationBatch.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import "../../../../tasks/wdl/Glimpse2SVImputationTasks.wdl" as Glimpse2SVImputa

workflow Glimpse2SVImputationBatch {
# if this changes, update the batch_pipeline_version value in Glimpse2SVImputation.wdl
String pipeline_version = "0.0.13"
String pipeline_version = "0.0.14"

input {
File input_preprocessed_joint_vcf
Expand Down Expand Up @@ -167,7 +167,7 @@ task GLIMPSE2Phase {
}
}

Int disk_size_gb = ceil(size(input_vcf, "GiB") + size(panel_split_chunk_bin, "GiB") + size(genetic_map, "GiB") + 30)
Int disk_size_gb = ceil(size(input_vcf, "GiB") + size(panel_split_chunk_bin, "GiB") + size(genetic_map, "GiB") + 40)

@jsotobroad jsotobroad Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change was to help with the batch size testing, this will need to change when we scale test as well so just tryign to help with that now


command <<<
set -euxo pipefail
Expand Down Expand Up @@ -211,7 +211,7 @@ task GLIMPSE2Phase {
cpu_cores: threads,
mem_gb: 16,
disk_gb: disk_size_gb,
boot_disk_gb: 10,
boot_disk_gb: 0,
use_ssd: true,
preemptible_tries: 30,
max_retries: 3,
Expand Down Expand Up @@ -242,7 +242,7 @@ task GLIMPSE2Ligate {
RuntimeAttr? runtime_attr_override
}

Int disk_size_gb = 2 * ceil(size(phased_vcfs, "GB")) + 10
Int disk_size_gb = ceil(2.1*size(phased_vcfs, "GB")) + 10

command <<<
set -euox pipefail
Expand All @@ -263,7 +263,7 @@ task GLIMPSE2Ligate {
cpu_cores: 2,
mem_gb: 18,
disk_gb: disk_size_gb,
boot_disk_gb: 10,
boot_disk_gb: 0,
use_ssd: true,
preemptible_tries: 0,
max_retries: 1,
Expand Down Expand Up @@ -298,7 +298,7 @@ task PopAndMarginalizeCollisions {
RuntimeAttr? runtime_attr_override
}

Int disk_gb = 10 + 3 * ceil(size([posteriors_vcf, panel_bubble_split_sites_only_vcf, panel_id_split_vcf_gz], "GB"))
Int disk_gb = ceil(3*size(posteriors_vcf, "GB")) + ceil(size([panel_bubble_split_sites_only_vcf, panel_id_split_vcf_gz], "GB")) + 10

command <<<
set -euox pipefail
Expand All @@ -319,9 +319,9 @@ task PopAndMarginalizeCollisions {
#########################
RuntimeAttr default_attr = object {
cpu_cores: 2,
mem_gb: 6,
mem_gb: 8,

@jsotobroad jsotobroad Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also a fix from the glimpse batch testing

disk_gb: disk_gb,
boot_disk_gb: 10,
boot_disk_gb: 0,
use_ssd: true,
preemptible_tries: 2,
max_retries: 1,
Expand Down Expand Up @@ -349,7 +349,7 @@ task UpdateHeader {

Int mem_gb = 2
Int cpu = 1
Int disk_size_gb = ceil(2.1 * size(bcf_to_reheader, "GiB") + 10)
Int disk_size_gb = ceil(2.1 * size(bcf_to_reheader, "GiB")) + 10
Int max_retries = 1
String docker
}
Expand Down Expand Up @@ -382,6 +382,7 @@ task UpdateHeader {
runtime {
docker: docker
disks: "local-disk " + disk_size_gb + " SSD"
bootDiskSizeGb: 0
memory: mem_gb + " GiB"
cpu: cpu
maxRetries: max_retries
Expand Down
5 changes: 5 additions & 0 deletions ...l/glimpse/sv_imputation/MultilevelHierarchicallyPasteVcfsStreaming.changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 0.0.6
2026-08-17 (Date of Last Commit)

* make all task disk sizes dynamic and set bootDiskSizeGb to 0

# 0.0.5
2026-08-13 (Date of Last Commit)

Expand Down
10 changes: 5 additions & 5 deletions pipelines/wdl/glimpse/sv_imputation/MultilevelHierarchicallyPasteVcfsStreaming.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import "../../../../tasks/wdl/Glimpse2SVImputationTasks.wdl" as Glimpse2SVImputa

workflow MultilevelHierarchicallyMergeVcfs {
# if this changes, update the multi_level_paste_pipeline_version value in PreprocessPLsGVCF.wdl
String pipeline_version = "0.0.5"
String pipeline_version = "0.0.6"

input {
Array[String]? vcfs_array
Expand Down Expand Up @@ -189,9 +189,9 @@ task CreateBatches {
#########################
RuntimeAttr default_attr = object {
cpu_cores: 1,
mem_gb: 4,
mem_gb: 2,
disk_gb: 10,
boot_disk_gb: 10,
boot_disk_gb: 0,
disk_type: "HDD",
preemptible_tries: 2,
max_retries: 1,
Expand Down Expand Up @@ -226,7 +226,7 @@ task MergeVcfs {
}

# Dynamically sizes disk if localizing, defaults to 50GB if streaming
Int disk_gb = if length(vcfs_localize) > 0 then 10 + 2 * ceil(size(vcfs_localize, "GiB")) else 50
Int disk_gb = if length(vcfs_localize) > 0 then ceil(2.1*size(vcfs_localize, "GiB")) + 10 else ceil(1.1*size(vcfs_localize, "GiB")) + 10

command <<<
set -euox pipefail
Expand Down Expand Up @@ -339,7 +339,7 @@ task MergeVcfs {
cpu_cores: 2,
mem_gb: 4,
disk_gb: disk_gb,
boot_disk_gb: 10,
boot_disk_gb: 0,
disk_type: "SSD",
preemptible_tries: 3,
max_retries: 0,
Expand Down
5 changes: 5 additions & 0 deletions pipelines/wdl/glimpse/sv_imputation/PreprocessPLsGVCF.changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 0.0.11
2026-08-17 (Date of Last Commit)

* make all task disk sizes dynamic and set bootDiskSizeGb to 0

# 0.0.10
2026-08-13 (Date of Last Commit)

Expand Down
8 changes: 4 additions & 4 deletions pipelines/wdl/glimpse/sv_imputation/PreprocessPLsGVCF.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import "../../../../tasks/wdl/Glimpse2SVImputationTasks.wdl" as Glimpse2SVImputa

workflow PreprocessPLsGVCF {
# if this changes, update the preprocessing_pls_gvcf_pipeline_version value in Glimpse2SVImputation.wdl
String pipeline_version = "0.0.10"
String multi_level_paste_pipeline_version = "0.0.5"
String pipeline_version = "0.0.11"
String multi_level_paste_pipeline_version = "0.0.6"
input {
File input_gvcf_manifest

Expand Down Expand Up @@ -85,7 +85,7 @@ task PreprocessPLs {
RuntimeAttr? runtime_attr_override
}

Int disk_size_gb = 10 + 2 * ceil(size([input_vcf, panel_bubble_split_sites_only_vcf], "GB"))
Int disk_size_gb = ceil(2*size([input_vcf, panel_bubble_split_sites_only_vcf], "GB")) + 10

File sample_names_list = write_lines(sample_names)

Expand Down Expand Up @@ -116,7 +116,7 @@ task PreprocessPLs {
cpu_cores: 1,
mem_gb: 2,
disk_gb: disk_size_gb,
boot_disk_gb: 10,
boot_disk_gb: 0,
use_ssd: true,
preemptible_tries: 4,
max_retries: 1,
Expand Down
12 changes: 10 additions & 2 deletions tasks/wdl/Glimpse2SVImputationTasks.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ task MergeSampleChunksVcfsWithPaste {
runtime {
docker: "us.gcr.io/broad-dsde-methods/bcftools_bgzip:beagle_imputation_v1.0.0"
disks: "local-disk " + disk_size_gb + " HDD"
bootDiskSizeGb: 0
memory: mem_gb + " GiB"
cpu: cpu
preemptible: preemptible
Expand Down Expand Up @@ -117,6 +118,7 @@ task ExtractAnnotations {
runtime {
docker: docker_extract_annotations
disks: "local-disk " + disk_size_gb + " HDD"
bootDiskSizeGb: 0
memory: mem_gb + " GiB"
cpu: cpu
preemptible: preemptible
Expand Down Expand Up @@ -210,6 +212,7 @@ EOF
runtime {
docker: docker_merge
disks: "local-disk " + disk_size_gb + " HDD"
bootDiskSizeGb: 0
memory: mem_gb + " GiB"
cpu: cpu
preemptible: preemptible
Expand Down Expand Up @@ -251,6 +254,7 @@ task CreateVcfIndexAndMd5 {
runtime {
docker: gatk_docker
disks: "local-disk ${disk_size_gb} SSD"
bootDiskSizeGb: 0
memory: "${memory_mb} MiB"
cpu: cpu
preemptible: preemptible
Expand Down Expand Up @@ -286,6 +290,7 @@ task FilterVcfByInfo {
runtime {
docker: docker
disks: "local-disk " + disk_size_gb + " HDD"
bootDiskSizeGb: 0
memory: mem_gb + " GiB"
cpu: cpu
preemptible: preemptible
Expand Down Expand Up @@ -340,6 +345,7 @@ task SplitVcfManifestIntoBatches {
docker: "us.gcr.io/broad-dsde-methods/python-data-slim:1.0"
cpu: 1
disks: "local-disk 10 HDD"
bootDiskSizeGb: 0
memory: "1 GiB"
preemptible: 3
noAddress: true
Expand Down Expand Up @@ -387,6 +393,7 @@ task ConvertInputArraysToManifest {
cpu: 1
memory: "1 GiB"
disks: "local-disk 10 HDD"
bootDiskSizeGb: 0
preemptible: 3
noAddress: true
}
Expand Down Expand Up @@ -430,8 +437,9 @@ task ParseVcfManifestIntoArrays {
runtime {
docker: "us.gcr.io/broad-dsde-methods/python-data-slim:1.0"
cpu: 1
memory: "4 GiB"
memory: "1 GiB"
disks: "local-disk 10 HDD"
bootDiskSizeGb: 0
preemptible: 3
noAddress: true
}
Expand Down Expand Up @@ -472,7 +480,7 @@ task ConcatBcfs {
cpu: 1
memory: "4 GiB"
disks: "local-disk " + disk_gb + " SSD"
bootDiskSizeGb: 10
bootDiskSizeGb: 0
preemptible: 3
maxRetries: 0
docker: "us.gcr.io/broad-gotc-prod/bcftools-vcftools:2.0.0-1.24-0.1.17-1784569943"
Expand Down
Loading