feat(data): recover stale headers and remove known invalid blocks - #139
feat(data): recover stale headers and remove known invalid blocks#139deadmanoz wants to merge 3 commits into
Conversation
| 914373,00000000000000000000a4a5644ce7e315feeeb1c586903c077993bb5f7825a2,00a04d220640c91602474b848f12f5be4863d350159d9b460de400000000000000000000eb0d363171fc2d3d48a279b35dd33e1f5cb4a64a23a320dc4ab5872599631f8dec26c468ac1102175a300e23 | ||
| 914371,000000000000000000013cad1d0d9a4d168c0b2dbae847fa8a2c3ba0704252c0,00807125c47328320e42272d035cad0073643fa35146f55f79ee00000000000000000000bd7c87b7c163b9061084615f9abbc6e0c9c0e7bb366f90cffa4e63664c99b59dbb21c468ac110217d5ba1524 | ||
| 914362,0000000000000000000170e712b2da3ba73fb618753fc44d0e4b6b1db3e3fc4b,006008203f7a2b385508a05cc2b32721756145dfa9317aba094d0100000000000000000066c1de0382413a8f224243d73d37aceb1f717dfee876aede87f3c51232e25d3a8e0cc468ac11021736448df5 | ||
| 914362,0000000000000000000098d9f82cd465d4a1e55fabb659d1df2464648f796777,0000783b3f7a2b385508a05cc2b32721756145dfa9317aba094d0100000000000000000048db5b54c9ce3436b157658cae512a7304da82898c59e6b7032158e1f90fe1f3860ec468ac110217672d7fe9 | ||
| 914360,00000000000000000001c7664aeb6f4ce7f29e3f799eb3aafb924255d6643573,0040ad291cc22dade83c2ce7b7f3893473422c283cd632b3daf3010000000000000000001471b3e94f959aff7e362f7812cd16f941cdc5cb40d6d04f61a32c8c80684b698f09c468ac110217dcd29561 | ||
| 914357,0000000000000000000135a4e20e4402f5f3b5798b2d49ec9f51614feb224c93,00408e25b66a8748eaa244fb8cd41fc32bec0b8d499f393c5ca80100000000000000000023d8a8125d4fd339b6f39bfebbccffe32a429fdb306554e77b777b4ee9968d7944f7c368ac1102175a3d528d | ||
| 914344,000000000000000000020806ba231682bab800c757366c169b2e17259a5e3aaf,00e0e7232b929e89995b68b87553a7afea0383bd467d7b6461a80100000000000000000029e9f2b78f60359141574b1e775b2a24ce6f15649c45e26fa6929e1d4ad13926b0e3c368ac1102170d0169c2 | ||
| 914344,0000000000000000000075bdcb14ab9b07e87c6c5a43addb81ff6dd74c14d16e,004003202b929e89995b68b87553a7afea0383bd467d7b6461a80100000000000000000004d3821459fe3c6043b90896ea17224f239437c2f57018871d6bc9cf6928b33b53e1c368ac110217a3166e1c | ||
| 914342,00000000000000000000ba485fa4bbba0416890572ff6b5853cdc24e92d7434f,0040d6233b995334cbefe9c09ad38bef90527fc484097ea234540000000000000000000092ca6058a40183784413f8d87dce7b73265297f065d5d94705d00d7254c09006c6d9c368ac1102173b3741be | ||
| 914334,0000000000000000000092003874d2b869e8b986339f5bcf8c3a03f5dddd972c,000000208cd96f2fa9d9cad5cdf3c980606f42a17f06b2a3b9b6000000000000000000009d7c2fbbda61aad6c58a0c0f6871b806fea480c13326bf60bbd14dfb936f20347dcac368ac110217231a45e9 | ||
| 914332,000000000000000000016c286d182701403f8eb2e322a844a019c933ba6082cb,0000ff3f0da0aa5fdbe2efc140729414992410bbcc7e7f1bf75301000000000000000000909e269c6ea2caaf2e24c4aeff22f5e9e7eecd3a8ca4e1bc9deb9975f78bb6ed49c5c368ac110217fc063011 | ||
| 914332,0000000000000000000122a9c97762f507cc57df067f07b826ab0a81cb4e787f,00c00b200da0aa5fdbe2efc140729414992410bbcc7e7f1bf7530100000000000000000088fb0b73c4d1102ae215c0bb5ae799023d1e9209eb80d379f7d5bcb6c12e4e5f40c1c368ac1102171873973f | ||
| 914331,00000000000000000000439817c63926873f46ca396974268b1f9f6eae84b1c1,00c00620a47f3806fb0107f54898997acfa3653ec56e00f68f0301000000000000000000c3ab851af94d4c486579d4643d037470aa9b87f0c3900adb5412c6235d3801753bbcc368ac110217a0067493 | ||
| 914329,00000000000000000002097c6c16d3e4065640c744d544d35dd797d06189a4ab,00603b2406947da999591fc6164db64a8445971cc55cbb3a69d60000000000000000000022881561a7507ad88f3f278a4c19731e8913045a69f5f48e46b7517b17cd296808b5c368ac110217cd975dad | ||
| 914319,000000000000000000008223e92e8a65a3400761c5307b01f0b87270bd24bafa,00e00f20471bb0c301ca76c341e8b7c6bedfc1dcd13ad194ae990100000000000000000097234e6f8960bf95bfe73a5a5d410768d33a434b5ab5f8987c423cce1e2e470dbca5c368ac1102172924c946 | ||
| 914318,00000000000000000001de4957ebb1c9e08d28fd604f017cb288bed2481455ed,0020b02b5860bdd0995c065f1955fdb0d6ee2a685ee7ba689002000000000000000000007f8d5b586b268c6fafd1c4834fc783b57d3d339df85a161b28463aeda881153a81a3c368ac11021782664964 | ||
| 914301,00000000000000000000a4b0e87728228afec5a5333d32c1de4631708be29e5f,00809533d16a6144a65746f1d8f02dd56b97dd177f015eeece7900000000000000000000eeb8feaaa4c85347c6c108d17918484a0224394d9d6159cd0b83a267530d99973261c368ac110217367d72d7 | ||
| 914282,00000000000000000001f6d6612ed8d40c023858249952fdee7d7d99c081f0cf,0040f5266d72296a250a099e32dc257c1a4a55561e5f16950df400000000000000000000054908d135392a44d6b16be8ea7c431cf04b52535673bf0ee53f271991cbdb00923cc368ac110217029af326 | ||
| 914282,00000000000000000001208a381476ad1cd93142df2fdc9f978a991a496e0aa0,00805a256d72296a250a099e32dc257c1a4a55561e5f16950df4000000000000000000003b2a2cc1873981673bf45cfe9b243500867034e01a5f80afaa182ca4829ad953303ec368ac110217a909062b | ||
| 914261,00000000000000000000ce42c5b1a6fb01d652fa82f25458f6c3d53364fb9f69,004003206c6d9589037eaf069a7cc98db8db221f152706ed6b680000000000000000000090a7aa08434e10e255df2dac9953660d7e084a653ecb0c28fca6bf12644897867a0bc368ac110217ede60a89 |
There was a problem hiding this comment.
oh, cool to see these! I wonder why we didn't have them before.
There was a problem hiding this comment.
I only added the namecoin stales previously, this is from (majority of) the rest of the chains
|
I went looking for the full blocks behind these headers and came up empty, so noting it here in case it saves someone the trip. I tried the explorers that still keep blocks which lost the race — blockchain.info and BlockCypher both do, for recent ones — plus four public Bitcoin nodes and the three fork-observer instances. None of them had any of the 589. The 325 new ones turn up nowhere at all, and they're also absent from the old node snapshots this dataset was built from. That fits with them never having spread across the network, which would explain why we didn't have them before. The 264 filled-in headers do leave a trace. 87 of them sat on a node that had the whole block stored at the time — Syke's from the bitcointalk thread, apoelstra's, and jgarzik's. So those blocks did exist somewhere. If any of those machines is still running untouched, they could still be pulled off them. The other 177 were header-only there too. |
| Dataset of stale headers and blocks observed on the Bitcoin network. | ||
|
|
||
| Known consensus-invalid headers and blocks belong in the | ||
| [invalid-blocks dataset](https://github.com/bitcoin-data/invalid-blocks/tree/321395ecffa69b97dd9bb6a06325b6fdf4b6440d). |
There was a problem hiding this comment.
maybe just this:
| [invalid-blocks dataset](https://github.com/bitcoin-data/invalid-blocks/tree/321395ecffa69b97dd9bb6a06325b6fdf4b6440d). | |
| [invalid-blocks dataset](https://github.com/bitcoin-data/invalid-blocks). |
|
Can you include a note on why the removed blocks are invalid in the commit message of the second commit? i.e. reason per hash |
|
I wonder how we can make sure that these headers/hashes aren't re-added at some point. Maybe we should mark them as known-invalid in the vanity check. |
They are "recovered from merge-mining commitments" (see PR description). |
Yeah, no worries |
I guess we can cross check against the invalid-blocks repo? |
yeah, good idea |
|
fyi you pushed a Merge branch 'master' into update-stale-blocks commit. you probably meant to rebase |
Follow-up to bitcoin-data#94 (Namecoin). 308 direct stales plus 17 stale-fork descendants recovered from the merge-mining commitments of 20 child chains, heights 160,948 to 947,985. Every header satisfies its own PoW target and Bitcoin's canonical nBits at its claimed height; 314 rows land at previously-unrepresented heights. The same recovery evidence also fills the missing headers of 264 existing hash-only rows (heights 179,641 to 472,549), each verified to hash to the recorded value, satisfy its own PoW target, and match Bitcoin's canonical nBits at its recorded height. Methodology and per-chain recovery evidence: https://github.com/deadmanoz/merge-mining-research
Remove 31 existing entries with named consensus failures and their four block files at heights 477115, 783426, 784121 and 809478. Preserve all other rows and binaries, including the 325 additions and 264 header fills. Point known-invalid contributions to the companion dataset in the README. The replacement records and binaries are published at: https://github.com/bitcoin-data/invalid-blocks/tree/321395ecffa69b97dd9bb6a06325b6fdf4b6440d All four replacement bodies match byte-for-byte and by SHA-256. Header identity, PoW, canonical-parent context and version enforcement were checked for each removal; the transaction merkle roots and ordering failures were independently reproduced. The two F2Pool sigops failures use documented historical rejection evidence, not a fresh UTXO replay. The sanity and Bitcoin Core 31.1 checks pass on the combined dataset.
9cdae6c to
e0f1c6e
Compare
|
Found another block for removal, added as a separate (3rd) commit because it's a bit more unusual. Discovered from: https://bitcointalk.org/index.php?topic=2041607.0 |
1Hash block 00000000000000000182acdf5657c93a0769dc6f9004047496b2e15efc6a4232 spends an output whose parent transaction confirmed only in the same-height competitor. That is bad-txns-inputs-missingorspent at connect time, not an in-block ordering error. The replacement record and binary are in bitcoin-data/invalid-blocks#5. Point the README at the companion repository rather than the pre-474294 tree snapshot.
f6ae24e to
73af152
Compare
|
nice finding! |
Recovered from AuxPoW and merge-mining commitments in namecoin, i0coin, ixcoin, devcoin, RSK, Elastos and Syscoin, surfaced by deadmanoz/merge-mining-research as strict and weak orphan candidates that #139 does not carry. Each header double-SHA256s to the hash recorded here, the BIP34 height in the recovered coinbase scriptSig matches the height column for all 11, and none of them is the main-chain block at its height. 363732, 363733 and 363735 are from the July 2015 BIP66 SPV-mining fork listed at https://en.bitcoin.it/wiki/July_2015_Forks, which the dataset otherwise covers with two header-less rows. 225436 belongs to the March 2013 BDB fork chain of #64 and is the eighth of those 32 headers to be recovered; its parent 225435 is still missing, so check-with-bitcoind.py will report it as building on an unknown parent rather than confirm its height. Ten further candidates from the same source are left out: five are recovered independently from the Decker-Wattenhofer capture, and four December 2012 headers build on parents that are on no main chain and in no stale dataset, so their ancestry cannot be authenticated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DN4RVKx7AhCy6sR7xu2WtE
…ture The orphan capture behind Decker and Wattenhofer, "Information Propagation in the Bitcoin Network" (P2P 2013), survives in NStifter/mergedmonitor at fork-analysis/decker-wattenhofer/orphans.tar.bz2, the artifact repository for Stifter et al., "Echoes of the Past: Recovering Blockchain Metrics From Merged Mining". Their measurement node recorded 1,804 blocks at fork heights between 142257 and 200206, stored as per-block JSON rather than as headers. The capture holds both branches of each fork. 1,192 of the blocks are the main-chain block at their height, checked against a public Esplora API, and are left out. 1,396 of the records are getblock dumps carrying version, merkle root, time, bits, nonce and height, but no parent hash, so the header cannot be read off directly. The parent is recoverable by trial: build the header against each known block at height - 1 and keep the candidate that hashes to the block's own hash. 1,364 take the main-chain block at that height; the remaining 32 build on another stale block and need a second pass over the stale set itself. The other 408 records are blockchain.info-style dumps carrying prev_block and mrkl_root directly, so their headers assemble without the search. Their main_chain flag agreed with the main-chain check on every block sampled. All 1,804 headers double-SHA256 to the hash the archive filed them under. For the blocks already in this file the recovered height always matches, and where a header was already present the bytes are identical. 87 rows that #139 already adds or fills are left to that pull request. Both changesets were compared first and agree on the height and the header bytes for every one of them, by way of two independent recovery routes. Five of the 32 deep-fork blocks also appear in Namecoin AuxPoW commitments via deadmanoz/merge-mining-research, and both routes produce byte-identical headers.
3 participants
Summary
This PR adds 325 stale-block headers recovered from merge-mining commitments and fills missing headers for 264 existing entries. The additions include 17 stale-fork descendants; the recovery methodology and evidence are in merge-mining-research.
A separate commit removes 31 entries with confirmed consensus failures and their four block files. A follow-up commit also removes height 474294 and its body, the 1Hash omitted-unconfirmed-parent case. Those 32 records belong in the invalid-blocks dataset. The README also clarifies the limits of the repository’s validation.
Testing
The sanity, Bitcoin Core 31.1 and live main-chain exclusion checks pass locally. Eight headers have unavailable ancestry and are reported as unchecked, including two additions at heights 234108 and 305610 whose parents remain hash-only. These checks do not reconstruct historical UTXO state or establish full consensus validity.