KIT DSN data archive format

[jharveyb]

I see that the existing GHA cronjob to archive the KIT data downloads a set of specific plaintext files:

https://github.com/bitcoin-data/bitcoin-stats-archive/blob/master/download-dsn-bitcoin-monitoring-data.sh

But on the Monitoring project website, I don't see these files mentioned:

https://www.dsn.kastel.kit.edu/bitcoin/data.html

I do, however, see a link to 'single file' downloads, which seems to have all 10 years of data:

https://www.dsn.kastel.kit.edu/bitcoin/snapshots/

So, is there a difference between these two sources? I figure that there isn't but it's not clear to me from the Montioring website that that is true.

[0xB10C]

This repository only downloads the aggregated data for the plots displayed on https://www.dsn.kastel.kit.edu/bitcoin/index.html. You find the links to the files below the charts. E.g.:

While I do keep a copy of the https://www.dsn.kastel.kit.edu/bitcoin/data.html snapshots, they are too big to upload them on GitHub I presume.

[jharveyb]

This repository only downloads the aggregated data for the plots displayed on https://www.dsn.kastel.kit.edu/bitcoin/index.html. You find the links to the files below the charts.

Ok, makes sense. AFAICT, all of those stats should be computable from the raw data on the snapshots page, so backing up only the snapshots would be enough to recreate these graphs.

While I do keep a copy of the https://www.dsn.kastel.kit.edu/bitcoin/data.html snapshots, they are too big to upload them on GitHub I presume.

I agree; are you saying that you have a copy of all 10 years of snapshots, or a partial mirror?

I'm beginning to explore how we can compress the full dataset here.

Please let me know if that would be interesting to you.

[0xB10C]

Ok, makes sense. AFAICT, all of those stats should be computable from the raw data on the snapshots page, so backing up only the snapshots would be enough to recreate these graphs.

also, cc @matthias-g

Hm I don't think the snapshots contain e.g. data on INVs as in https://www.dsn.kastel.kit.edu/bitcoin/data/invstat.gpd?

I agree; are you saying that you have a copy of all 10 years of snapshots, or a partial mirror?

I do have a copy of the 10 years of snapshots (i.e. curl -s 'https://www.dsn.kastel.kit.edu/bitcoin/snapshots/?curl' | xargs -P 4 wget). In compressed ZFS, it only takes up 3.6GB or so.

I'm beginning to explore how we can compress the full dataset here. Please let me know if that would be interesting to you.

Cool! I'll take a look at your repo.

In the past, I've encoded that data in away where I record when e.g. a node version changed:

  "nodeid-as-hash": {
        "versions": [
            {
                "start": "2025-07-10"
                "end": "2025-07-21",
                "version": "/Satoshi:29.0.0/"
            },
            {
                "start": "2025-07-21",
                "end": "2025-08-02",
                "version": "/Satoshi:28.1.0/Knots:20250305/"
            },
            {
                "start": "2025-08-02",
                "end": "2025-09-16",
                "version": "/Satoshi:29.0.0/"
            },
            {
                "start": "2025-09-16",
                "end": "2025-10-16",
                "version": "/Satoshi:29.1.0/"
            }
        ]
    },

I guess that could be applied to other fields than version too. Not sure if it beats ZSTD by a lot though.

[jharveyb]

Hm I don't think the snapshots contain e.g. data on INVs as in https://www.dsn.kastel.kit.edu/bitcoin/data/invstat.gpd?

I think most of those values are just computed over the meandelay and meandelayblock data of all peers:

At least, that's my interpretation of the fields given the description on the schema at the bottom of this page:

https://www.dsn.kastel.kit.edu/bitcoin/data.html

It should be straightforward to cross-check that for a specific day though, by computing it again from the raw data.

I do have a copy of the 10 years of snapshots (i.e. curl -s 'https://www.dsn.kastel.kit.edu/bitcoin/snapshots/?curl' | xargs -P 4 wget). In compressed ZFS, it only takes up 3.6GB or so.

Ok, nice - ZFS can use ZSTD for block compression, so that may explain your savings there.

[jharveyb]

As a small update, I was looking at the data again and trying out various options for compression.

Tl;dr just ZSTD 22 is pretty great at 15x space savings. The next big improvement is packing data monthly vs. daily (35x), and then reordering measurements and sorting by node_id.

The fields used in the raw data have changed a few times, so that makes mapping directly to Parquet a little more complicated, but still doable. More details on the observed schema changes here.

[0xB10C]

have you looked into zstd's --patch-from across daily files?