HTSJDK 5.0 forthcoming soon - looking for early adopters

[tfenne]

Hi all! A 5.0.0 release of HTSJDK will be forthcoming shortly. I've been actively working on a number of areas including CRAM 3.1 write support, and performance improvements to BAM and CRAM reading and writing.

There's a draft release with good release notes: HTSJDK 5.0.0 release draft

The majority of changes have not hit master yet, but are on dev, which I will merge to master (preserving all commits) at release time.

There are a number of breaking changes in this release(!) To try and ensure they are low-impact, I have also tested both picard and fgbio, both of which rely heavily on HTSJDK, with a pre-release version of 5.0.0 matching the current dev branch.

@jrobinso, @lindenb, @alecw: If you have a few minutes over the next few days to test out the version on dev within your projects (I can publish a snapshot to sonatype central if that's helpful), that would certainly de-risk things. OTOH, I'm very open to rapidly shipping 5.0.1 etc. if issues arise.

[alecw]

Hi @tfenne , please excuse my prehistoric build tool skills. If you could push a snapshot to central I think I should be able to test.

[jrobinso]

Yes a snapshot would be helpful. Thanks for your work on this, I feared it might be abandoned.

[jrobinso]

BTW the link to the release draft gives a 404 error.

[tfenne]

Thanks @alecw and @jrobinso. I'm not sure why the release link isn't working (maybe because it's a draft?). But the releases page has it up top at: https://github.com/samtools/htsjdk/releases

I just published a SNAPSHOT release from the head of dev. It's in the sonatype central snapshot repo as 5.0.0-4f25db8-SNAPSHOT (in my hands sbt resolves it at https://central.sonatype.com/repository/maven-snapshots/com/github/samtools/htsjdk/5.0.0-4f25db8-SNAPSHOT/htsjdk-5.0.0-4f25db8-20260427.171619-1.jar).

You'll need the following for a gradle build, or the equivalent for your build tool of choice:

maven {
      name = "sonatypeSnapshots"
      url = "https://central.sonatype.com/repository/maven-snapshots/"
      mavenContent { snapshotsOnly() }
  }

If you run into trouble, just reply here and I'll see what I can do to help!

[jmarshall]

FYI draft releases are only visible to members of the samtools org or people with write privileges to the htsjdk repo or something like that.

[tfenne]

TIL 🤦 . Thanks @jmarshall.

[alecw]

In case someone else is a clueless as me...gradle changes:

repositories {
    mavenCentral()
+    maven {
+        name = "sonatypeSnapshots"
+        url = uri("https://central.sonatype.com/repository/maven-snapshots/")
+        mavenContent { snapshotsOnly() }
+    }
 }
 
dependencies {
+    implementation("com.github.samtools:htsjdk:5.0.0-4f25db8-SNAPSHOT")

[alecw]

Hi @tfenne ,

Our workflows are a mixture of Picard CLP invocation, our own Java CLPs built on top of Picard and htsjdk, and of course lots of other stuff. I haven't tested our Picard invocations, because I didn't figure out how to create a Picard executable jar with the dev htsjdk. However, I have tested a soup-to-nuts workflow cascade in which our own Java CLPs use the new htsjdk, and it checks out perfectly. If you want me to test the Picard CLPs also I could probably figure out how to build the jar, but I figure you've probably already done such testing.

[jrobinso]

Hi @tfenne

I did a build and quick smoke test, loading a 1KG bam, platinum genomes cram, 1KG vcf with genotypes, and an annotation (tribble) track. No issues. If there are particular files you want me to try send me the url(s). I'm o.k. with you releasing this when you are ready, we'll deal with any issues as they arise.

We're also working on a big IGV release, 3.0, sometime before the end of May. That would be a target date for updating our htsjdk dependency, if its ready by then.

[tfenne]

Thank you both @alecw and @jrobinso! I see no reason we couldn't push out a release of HTSJDK within the week. Probably better to get it out there, and iterate quickly if issues come up, as I feel beyond the three of us it's unlikely many other users will test a pre-release.

[jrobinso]

Sounds good. Ping us here when it's released, for me reminders never hurt.

[lindenb]

@tfenne FYI my tests are PASSing with htsjdk5 https://gitlab.univ-nantes.fr/pierre.lindenbaum/jvarkit/-/jobs/291011 , so there is no problem on my side.

[tfenne]

Thanks @lindenb !

[jrobinso]

@tfenne We've run into an issue with this dependency "com.fulcrumgenomics:jlibdeflate:0.1.0". We've been unable to notarize our app (Mac OS), it complains about that jar. I suspect because it has a JNI component. I'm not sure we'll be able to work around this. I see from the release notes there is an automatic fallback if the library can't be loaded, is there a way to specify that the jar itself not be included?

[jrobinso]

@tfenne On never mind, we can deal with this on our side. We have a precedent, "snappy.jar" has to be deleted from the bundle before Mac notarization for the same reason (embedded unsigned dylib files). We can try the same thing for jlibdeflate.

Alternatively, we might trying code signing those internal dylib files. In any event this isn't really and htsjdk problem, but be aware that anyone using the htsjdk in a notarized Mac app will have this issue. I suspect we are the only client doing that.