Baselines are sometimes deleted after a thresholds check?

[MahdiBM]

I encountered a very weird bug yesterday which consumed half of my day; I'm using that custom branch of mine that has static range-thresholds support, not the official package-benchmark release, and so the issue might be related to my changes, but I doubt that.

I wonder if you have any idea or have you noticed such a bug too, or not.

The bug is; for these package benchmark failures, there were no baselines written to .benchmarkBaselines (or that they written, but also deleted at some point in the CI):

This is happening in CI, and the CI works consistently well with cpuUser or malloc metrics. However something looks to go wrong when using throughput, most of the times, not always.

The CI (swift-dns/.github/workflows/benchmarks.yml) basically does these:

• Run the benchmarks:

          swift package -c release \
            --package-path Benchmarks \
            --allow-writing-to-package-directory \
            benchmark baseline update \
            '${{ steps.commit-info.outputs.title }}'

• Read the benchmark results:

          swift package -c release \
            --package-path Benchmarks \
            --allow-writing-to-package-directory \
            benchmark baseline read \
            '${{ steps.commit-info.outputs.title }}' \
            --no-progress \
            --format markdown \
            >> benchmark.md

• Compare benchmark results against thresholds:

          set +e
          swift package -c release \
            --package-path Benchmarks \
            --allow-writing-to-package-directory \
            benchmark thresholds check \
            '${{ steps.commit-info.outputs.title }}' \
            --path "$PWD/${{ env.THRESHOLDS_PATH }}" \
            --skip-loading-benchmark-targets \
            --no-progress \
            --format markdown \
            >> comparison.md
          echo "exit-status=$?" >> "${GITHUB_OUTPUT}"
          set -e

• If benchmark failed, update static-range-threshold-files(new in my PRs) based on the failed benchmark, to then commit them to a new branch so I can easily merge the new thresholds to the primary branch I'm working on, if I want to:

          swift package -c release \
            --package-path Benchmarks \
            --allow-writing-to-package-directory \
            benchmark thresholds update \
            '${{ steps.commit-info.outputs.title }}' \
            --path "$PWD/${{ env.THRESHOLDS_PATH }}" \
            --range

And ... the issue is that by the time the last script above is about to get triggered, the baselines are gone. I've 100% confirmed this by injecting commands into the CI steps. The folders and all are there, just that the ending folders contain no result.json anymore.

Now that I'm thinking with a fresh mind, I think it must be that the compare step somehow deletes the baseline?

In this specific case, this would happen very consistently. So always, there were no baseline result.json files anymore.

[MahdiBM]

Also I notice, the reported 'Current-PR' columns are not correct. There is 0 chance they all have a throughput of 14 million / s.
Some of them should exceed 50 million (e.g. IPv4 Zero String Description), some of them should be ~20 million (e.g. IPv4 Mixed String Description).

So ... something is going really wrong here?

[MahdiBM]

Checking the old benchmark-comments, I can see that they were not necessarily stuck on 14 million. They would change, sometimes some would change and the other would not, which makes sense. But still, the numbers look very off to me.