feat(state): add ChainstateManager::validate_block()

CHANGELOG.md

@@ -15,6 +15,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `Transaction::check` for context-free consensus validation of a transaction. Returns `TxCheckResult::Valid` on success or `TxCheckResult::Invalid(TxValidationResult)` on failure.
 - Added `TxValidationResult` enum with all transaction validation result variants.
 - Added `ChainParams::new_signet` and `ContextBuilder::signet` to configure a custom signet from a user-provided challenge.
+- Added `TransactionExt::is_coinbase` to check whether a transaction is a coinbase transaction.
+- Added `Coin::new` to construct a coin from a transaction output, confirmation height, and coinbase flag, for use with `ChainstateManager::validate_block` when validating a block without the full UTXO set present.
+- Added `TxOutPoint::new` to construct a new outpoint from a transaction ID and output index.
+- Added `ChainstateManager::validate_block` to validate a block against caller-supplied spent coins, without requiring the full UTXO set to be present. Returns a `ValidateBlockResult` enum carrying the validation state on failure.
 
 ### Changed
 - The `verify` function's `flags` parameter now uses `ScriptVerificationFlags` instead of `u32`, making the type explicit in the public API.

libbitcoinkernel-sys/CHANGELOG.md

@@ -13,6 +13,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - New `btck_tx_validation_state_create`, `btck_tx_validation_state_get_validation_mode`, `btck_tx_validation_state_get_tx_validation_result` and `btck_tx_validation_state_destroy` for managing and inspecting transaction validation state
 - New `btck_transaction_check` for context-free consensus validation of transactions (reachable results: `UNSET` on success, `CONSENSUS` on failure)
 - New `btck_chain_parameters_create_signet` for creating chain parameters for a custom signet from a user-provided challenge.
+- New `btck_transaction_is_coinbase` for checking whether a transaction is a coinbase transaction
+- New `btck_coin_create` for constructing a `btck_Coin` from a transaction output, confirmation height, and coinbase flag
+- New `btck_transaction_out_point_create` for constructing a `btck_TransactionOutPoint` from a txid and output index
+- New `btck_chainstate_manager_validate_block` for validating a block against caller-supplied spent coins (as parallel `btck_TransactionOutPoint`/`btck_Coin` arrays) without requiring the full UTXO set to be present
 
 ### Changed
 - `btck_block_header_create` now asserts that the input buffer is non-null and exactly 80 bytes; previously non-null buffer of any length was accepted

libbitcoinkernel-sys/bitcoin/.github/workflows/ci.yml

@@ -463,6 +463,12 @@ jobs:
             timeout-minutes: 120
             file-env: './ci/test/00_setup_env_mac_cross_intel.sh'
 
+          - name: 'NetBSD Cross'
+            warp-runner: 'warp-ubuntu-latest-x64-8x'
+            fallback-runner: 'ubuntu-latest'
+            timeout-minutes: 120
+            file-env: './ci/test/00_setup_env_netbsd_cross.sh'
+
           - name: 'FreeBSD Cross'
             warp-runner: 'warp-ubuntu-latest-x64-8x'
             fallback-runner: 'ubuntu-latest'

libbitcoinkernel-sys/bitcoin/SECURITY.md

@@ -13,7 +13,6 @@ The following keys may be used to communicate sensitive information to developer
 
 | Name | Fingerprint |
 |------|-------------|
-| Pieter Wuille | 133E AC17 9436 F14A 5CF1  B794 860F EB80 4E66 9320 |
 | Michael Ford | E777 299F C265 DD04 7930  70EB 944D 35F9 AC3D B76A |
 | Ava Chow | 1528 1230 0785 C964 44D3  334D 1756 5732 E08E 5E41 |
 | Niklas Gögge | 2CBB F208 E594 BF43 9B5F 276C 7465 CFFF 6793 242E |

libbitcoinkernel-sys/bitcoin/ci/test/00_setup_env_netbsd_cross.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or https://opensource.org/license/mit.
+
+export LC_ALL=C.UTF-8
+
+export CONTAINER_NAME=ci_netbsd_cross
+export CI_IMAGE_NAME_TAG="mirror.gcr.io/ubuntu:26.04"
+export APT_LLVM_V="22"
+export HOST=x86_64-unknown-netbsd
+export NETBSD_VERSION=11.0_RC6
+export NETBSD_SDK_BASENAME="netbsd-${HOST}-${NETBSD_VERSION}"
+export PACKAGES="clang-${APT_LLVM_V} llvm-${APT_LLVM_V} lld-${APT_LLVM_V}"
+export SYSROOT="--sysroot=${DEPENDS_DIR}/SDKs/${NETBSD_SDK_BASENAME}"
+export DEP_OPTS="build_CC=clang build_CXX=clang++ \
+ CC='clang --target=${HOST} ${SYSROOT}' \
+ CXX='clang++ --target=${HOST} ${SYSROOT} -stdlib=libstdc++' \
+ LDFLAGS='-fuse-ld=lld -lgcc_s' \
+ AR=llvm-ar-${APT_LLVM_V} \
+ NM=llvm-nm-${APT_LLVM_V} \
+ OBJCOPY=llvm-objcopy-${APT_LLVM_V} \
+ OBJDUMP=llvm-objdump-${APT_LLVM_V} \
+ RANLIB=llvm-ranlib-${APT_LLVM_V} \
+ STRIP=llvm-strip-${APT_LLVM_V}"
+export GOAL="install"
+export BITCOIN_CONFIG="\
+ --preset=dev-mode \
+ -DBUILD_GUI=OFF \
+ -DREDUCE_EXPORTS=ON \
+ -DWITH_USDT=OFF \
+"
+export RUN_UNIT_TESTS=false
+export RUN_FUNCTIONAL_TESTS=false

libbitcoinkernel-sys/bitcoin/ci/test/01_base_install.sh

@@ -118,6 +118,17 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
   tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
 fi
 
+if [ -n "$NETBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${NETBSD_SDK_BASENAME}" ]; then
+  mkdir -p "${DEPENDS_DIR}/SDKs/${NETBSD_SDK_BASENAME}"
+  for NETBSD_SDK_FILENAME in base.tar.xz comp.tar.xz; do
+    NETBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${NETBSD_SDK_FILENAME}"
+    if [ ! -f "$NETBSD_SDK_PATH" ]; then
+      ${CI_RETRY_EXE} curl --location --fail "https://cdn.netbsd.org/pub/NetBSD/NetBSD-${NETBSD_VERSION}/amd64/binary/sets/${NETBSD_SDK_FILENAME}" -o "$NETBSD_SDK_PATH"
+    fi
+    tar -C "${DEPENDS_DIR}/SDKs/${NETBSD_SDK_BASENAME}" -xf "$NETBSD_SDK_PATH"
+  done
+fi
+
 if [ -n "$FREEBSD_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${FREEBSD_SDK_BASENAME}" ]; then
   FREEBSD_SDK_FILENAME="base-${FREEBSD_VERSION}.txz"
   FREEBSD_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${FREEBSD_SDK_FILENAME}"

libbitcoinkernel-sys/bitcoin/contrib/seeds/README.md

@@ -15,8 +15,7 @@ DNS seed, virtu's crawler, and asmap community AS map data. Run the following co
 from the `/contrib/seeds` directory:
 
 ```
-curl https://bitcoin.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
-curl https://21.ninja/seeds.txt.gz | gzip -dc >> seeds_main.txt
+curl https://21.ninja/seeds.txt.gz | gzip -dc > seeds_main.txt
 curl https://mainnet.achownodes.xyz/seeds.txt.gz | gzip -dc >> seeds_main.txt
 curl https://signet.achownodes.xyz/seeds.txt.gz | gzip -dc > seeds_signet.txt
 curl https://testnet.achownodes.xyz/seeds.txt.gz | gzip -dc > seeds_test.txt

libbitcoinkernel-sys/bitcoin/depends/hosts/netbsd.mk

@@ -1,5 +1,6 @@
 netbsd_CFLAGS=
 netbsd_CXXFLAGS=
+netbsd_LDFLAGS=
 
 ifneq ($(LTO),)
 netbsd_AR = $(host_toolchain)gcc-ar

libbitcoinkernel-sys/bitcoin/doc/release-notes-35634.md

@@ -1,9 +1,8 @@
 Indexes
 -------
 
-- The transaction output spender index (`-txospenderindex`) now stores one byte
-  less per spender entry for newly indexed blocks. Existing indexes remain
-  compatible and no action is required. To apply the same saving to entries
-  indexed before upgrading, stop the node, delete the
+- The transaction output spender index (`-txospenderindex`) now uses less disk
+  space. Existing indexes remain compatible and no action is required. To reclaim
+  the space for data indexed before upgrading, stop the node, delete the
   `<datadir>/indexes/txospenderindex/` directory, and restart with
-  `-txospenderindex` enabled to rebuild it.
+  `-txospenderindex` enabled to rebuild it. (#35634, #35568)

libbitcoinkernel-sys/bitcoin/doc/release-notes/release-notes-29.4.md

@@ -0,0 +1,107 @@
+Bitcoin Core version 29.4 is now available from:
+
+  <https://bitcoincore.org/bin/bitcoin-core-29.4/>
+
+This release includes various bug fixes and performance
+improvements, as well as updated translations.
+
+Please report bugs using the issue tracker at GitHub:
+
+  <https://github.com/bitcoin/bitcoin/issues>
+
+To receive security and update notifications, please subscribe to:
+
+  <https://bitcoincore.org/en/list/announcements/join/>
+
+How to Upgrade
+==============
+
+If you are running an older version, shut it down. Wait until it has completely
+shut down (which might take a few minutes in some cases), then run the
+installer (on Windows) or just copy over `/Applications/Bitcoin-Qt` (on macOS)
+or `bitcoind`/`bitcoin-qt` (on Linux).
+
+Upgrading directly from a version of Bitcoin Core that has reached its EOL is
+possible, but it might take some time if the data directory needs to be migrated. Old
+wallet versions of Bitcoin Core are generally supported.
+
+Compatibility
+==============
+
+Bitcoin Core is supported and tested on operating systems using the
+Linux Kernel 3.17+, macOS 13+, and Windows 10+. Bitcoin
+Core should also work on most other Unix-like systems but is not as
+frequently tested on them. It is not recommended to use Bitcoin Core on
+unsupported systems.
+
+Notable changes
+===============
+
+This release fixes an issue where the chainstate database would repeatedly
+rewrite large portions of itself, causing excessive disk reads and writes
+during normal operation.
+
+### Validation
+
+- #35209 validation: correct lifetime of precomputed tx data
+- #35465 coins: compact chainstate regularly
+
+### Leveldb
+
+- #61(bitcoin-core/leveldb): Disable seek compaction
+
+### Net
+
+- #34093 netif: fix compilation warning in QueryDefaultGatewayImpl()
+
+### Wallet
+
+- #35228 wallet: use outpoint when estimating input size
+
+### Build
+
+- #34228 depends: Unset SOURCE_DATE_EPOCH in gen_id script
+- #34848 cmake: Migrate away from deprecated SQLite3 target
+
+### Test
+
+- #34918 fuzz: [refactor] Remove unused g_setup pointers
+
+### Doc
+
+- #34510 doc: fix broken bpftrace installation link
+- #34561 wallet: rpc: manpage: fix example missing `fee_rate` argument
+- #34671 doc: Update Guix install for Debian/Ubuntu
+- #35283 doc: mention -DWITH_ZMQ=ON in BSD build guides
+
+### CI
+
+- #35202 ci: restore sockets in i686, no IPC job
+- #35378 ci: switch runners from cirrus to warpbuild
+- #35408 ci: 35378 followups
+
+### Misc
+
+- #35175 multi_index: fix compilation failure with boost >= 1.91
+
+Credits
+=======
+
+Thanks to everyone who directly contributed to this release:
+
+- andrewtoth
+- Cory Fields
+- Daniel Pfeifer
+- darosior
+- fanquake
+- Hennadii Stepanov
+- jayvaliya
+- junbyjun1238
+- Lőrinc
+- MarcoFalke
+- SomberNight
+- ToRyVand
+- willcl-ark
+
+As well as to everyone that helped with translations on
+[Transifex](https://explore.transifex.com/bitcoin/bitcoin/).

libbitcoinkernel-sys/bitcoin/src/dbwrapper.cpp

@@ -136,12 +136,12 @@ static void SetMaxOpenFiles(leveldb::Options *options) {
              options->max_open_files, default_open_files);
 }
 
-static leveldb::Options GetOptions(size_t nCacheSize)
+static leveldb::Options GetOptions(size_t nCacheSize, bool bloom_filter)
 {
     leveldb::Options options;
     options.block_cache = leveldb::NewLRUCache(nCacheSize / 2);
     options.write_buffer_size = nCacheSize / 4; // up to two write buffers may be held in memory simultaneously
-    options.filter_policy = leveldb::NewBloomFilterPolicy(10);
+    options.filter_policy = bloom_filter ? leveldb::NewBloomFilterPolicy(10) : nullptr;
     options.compression = leveldb::kNoCompression;
     options.info_log = new CBitcoinLevelDBLogger();
     if (leveldb::kMajorVersion > 1 || (leveldb::kMajorVersion == 1 && leveldb::kMinorVersion >= 16)) {
@@ -225,7 +225,7 @@ CDBWrapper::CDBWrapper(const DBParams& params)
     DBContext().iteroptions.verify_checksums = true;
     DBContext().iteroptions.fill_cache = false;
     DBContext().syncoptions.sync = true;
-    DBContext().options = GetOptions(params.cache_bytes);
+    DBContext().options = GetOptions(params.cache_bytes, params.bloom_filter);
     DBContext().options.create_if_missing = true;
     DBContext().options.max_file_size = params.max_file_size;
     assert(!(params.testing_env && params.memory_only));

libbitcoinkernel-sys/bitcoin/src/dbwrapper.h

@@ -50,6 +50,8 @@ struct DBParams {
     //! If true, store data obfuscated via simple XOR. If false, XOR with a
     //! zero'd byte array.
     bool obfuscate = false;
+    //! If true, build a LevelDB bloom filter to accelerate point lookups.
+    bool bloom_filter = true;
     //! Passed-through options.
     DBOptions options{};
     //! If non-null, use this as the leveldb::Env instead of the default.

libbitcoinkernel-sys/bitcoin/src/index/base.cpp

@@ -65,13 +65,14 @@ CBlockLocator GetLocator(interfaces::Chain& chain, const uint256& block_hash)
     return locator;
 }
 
-BaseIndex::DB::DB(const fs::path& path, size_t n_cache_size, bool f_memory, bool f_wipe, bool f_obfuscate) :
+BaseIndex::DB::DB(const fs::path& path, size_t n_cache_size, bool f_memory, bool f_wipe, bool f_obfuscate, bool f_bloom) :
     CDBWrapper{DBParams{
         .path = path,
         .cache_bytes = n_cache_size,
         .memory_only = f_memory,
         .wipe_data = f_wipe,
         .obfuscate = f_obfuscate,
+        .bloom_filter = f_bloom,
         .options = [] { DBOptions options; node::ReadDatabaseArgs(gArgs, options); return options; }()}}
 {}
 
@@ -273,12 +274,23 @@ void BaseIndex::Sync()
     }
 }
 
-bool BaseIndex::Commit()
+void BaseIndex::Commit()
 {
     // Don't commit anything if we haven't indexed any block yet
     // (this could happen if init is interrupted).
     bool ok = m_best_block_index != nullptr;
     if (ok) {
+        // Don't commit if the index best block is not an ancestor of the chainstate's last flushed
+        // block. Otherwise, after an unclean shutdown, the index could be
+        // persisted ahead of a chainstate it can no longer roll back to, which
+        // would corrupt indexes with state (e.g. coinstatsindex).
+        const CBlockIndex* index_tip = m_best_block_index.load();
+        const CBlockIndex* last_flushed = WITH_LOCK(::cs_main, return m_chainstate->GetLastFlushedBlock());
+        if (!last_flushed || last_flushed->GetAncestor(index_tip->nHeight) != index_tip) {
+            LogDebug(BCLog::COINDB, "Skipping commit, index is ahead of flushed chainstate (index height %d, last flush at height %d)",
+                    index_tip->nHeight, last_flushed ? last_flushed->nHeight : -1);
+            return;
+        }
         CDBBatch batch(GetDB());
         ok = CustomCommit(batch);
         if (ok) {
@@ -288,9 +300,7 @@ bool BaseIndex::Commit()
     }
     if (!ok) {
         LogError("Failed to commit latest %s state", GetName());
-        return false;
     }
-    return true;
 }
 
 bool BaseIndex::Rewind(const CBlockIndex* current_tip, const CBlockIndex* new_tip)

libbitcoinkernel-sys/bitcoin/src/index/base.h

@@ -65,7 +65,7 @@ class BaseIndex : public CValidationInterface
     {
     public:
         DB(const fs::path& path, size_t n_cache_size,
-           bool f_memory = false, bool f_wipe = false, bool f_obfuscate = false);
+           bool f_memory = false, bool f_wipe = false, bool f_obfuscate = false, bool f_bloom = true);
 
         /// Read block locator of the chain that the index is in sync with.
         /// Note, the returned locator will be empty if no record exists.
@@ -94,14 +94,10 @@ class BaseIndex : public CValidationInterface
     CThreadInterrupt m_interrupt;
 
     /// Write the current index state (eg. chain block locator and subclass-specific items) to disk.
-    ///
-    /// Recommendations for error handling:
-    /// If called on a successor of the previous committed best block in the index, the index can
-    /// continue processing without risk of corruption, though the index state will need to catch up
-    /// from further behind on reboot. If the new state is not a successor of the previous state (due
-    /// to a chain reorganization), the index must halt until Commit succeeds or else it could end up
-    /// getting corrupted.
-    bool Commit();
+    /// Will skip the commit if no block has been indexed yet or if the index's best block is
+    /// ahead of the chainstate's last flushed block. This avoids persisting state an unclean shutdown
+    /// could not roll back from. A later call commits when the chainstate has flushed far enough.
+    void Commit();
 
     /// Loop over disconnected blocks and call CustomRemove.
     bool Rewind(const CBlockIndex* current_tip, const CBlockIndex* new_tip);

libbitcoinkernel-sys/bitcoin/src/index/txospenderindex.cpp

@@ -62,7 +62,7 @@ struct DBKey {
 };
 
 TxoSpenderIndex::TxoSpenderIndex(std::unique_ptr<interfaces::Chain> chain, size_t n_cache_size, bool f_memory, bool f_wipe)
-    : BaseIndex(std::move(chain), "txospenderindex", "txospenderidx"), m_db{std::make_unique<DB>(gArgs.GetDataDirNet() / "indexes" / "txospenderindex" / "db", n_cache_size, f_memory, f_wipe)}
+    : BaseIndex(std::move(chain), "txospenderindex", "txospenderidx"), m_db{std::make_unique<DB>(gArgs.GetDataDirNet() / "indexes" / "txospenderindex" / "db", n_cache_size, f_memory, f_wipe, /*f_obfuscate=*/false, /*f_bloom=*/false)}
 {
     if (!m_db->Read("siphash_key", m_siphash_key)) {
         FastRandomContext rng(false);

libbitcoinkernel-sys/bitcoin/src/ipc/libmultiprocess/doc/usage.md

@@ -10,7 +10,7 @@ _libmultiprocess_ is a library and code generator that allows calling C++ class
 
 The `*.capnp` data definition files are consumed by the _libmultiprocess_ code generator and each `X.capnp` file generates `X.capnp.c++`, `X.capnp.h`, `X.capnp.proxy-client.c++`, `X.capnp.proxy-server.c++`, `X.capnp.proxy-types.c++`, `X.capnp.proxy-types.h`, and `X.capnp.proxy.h` output files. The generated files include `mp::ProxyClient<Interface>` and `mp::ProxyServer<Interface>` class specializations for all the interfaces in the `.capnp` files. These allow methods on C++ objects in one process to be called from other processes over IPC sockets.
 
-The `ProxyServer` objects help translate IPC requests from a socket to method calls on a local object. The `ProxyServer` objects are just used internally by the `mp::ServeStream(loop, socket, wrapped_object)` and `mp::ListenConnections(loop, socket, wrapped_object)` functions, and aren't exposed externally. The `ProxyClient` classes are exposed, and returned from the `mp::ConnectStream(loop, socket)` function and meant to be used directly. The classes implement methods described in `.capnp` definitions, and whenever any method is called, a request with the method arguments is sent over the associated IPC connection, and the corresponding `wrapped_object` method on the other end of the connection is called, with the `ProxyClient` method blocking until it returns and forwarding back any return value to the `ProxyClient` method caller.
+The `ProxyServer` objects help translate IPC requests from a socket to method calls on a local object. The `ProxyServer` objects are just used internally by the `mp::ServeStream(loop, socket, wrapped_object)` and `mp::ListenConnections(loop, socket, wrapped_object[, max_connections])` functions, and aren't exposed externally. The `ProxyClient` classes are exposed, and returned from the `mp::ConnectStream(loop, socket)` function and meant to be used directly. The classes implement methods described in `.capnp` definitions, and whenever any method is called, a request with the method arguments is sent over the associated IPC connection, and the corresponding `wrapped_object` method on the other end of the connection is called, with the `ProxyClient` method blocking until it returns and forwarding back any return value to the `ProxyClient` method caller.
 
 ## Example