Spendblock plus coinsfetcher#184
Open
sedited wants to merge 63 commits into
Open
Conversation
Adds build configuration, benchmarking CI workflows, Python dependencies, plotting tools, and documentation for benchcoin. Co-authored-by: David Gumberg <davidzgumberg@gmail.com> Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
- Fix empty chart: use get_chart_data() instead of to_dict() so JS
filters can match config strings ("450", "32000") instead of objects
- Capture machine specs on self-hosted runner during build job and pass
via --machine-specs flag to nightly append, instead of detecting on
the ubuntu-latest publish runner
Run LogParser + PlotGenerator from bench/analyze.py during artifact
copying to produce static PNG charts from debug.log files. This
pre-generates the same 11 chart types that were previously rendered
client-side via JavaScript.
Changes to report.py:
- Import HAS_MATPLOTLIB, LogParser, PlotGenerator from bench.analyze
- _copy_network_artifacts: generate plots after each debug.log with
"{network}-{name}" prefix (e.g. "450-uninstrumented-pr")
- _copy_artifacts: generate plots for single-directory mode, including
when input_dir == output_dir
- _prepare_graphs_data: add "plots" key with relative paths to PNGs
- generate(): reorder to copy artifacts before HTML rendering so
_prepare_graphs_data can find the generated plot files
Plot generation is guarded by HAS_MATPLOTLIB for graceful fallback
when matplotlib is unavailable.
The pr-report.html template previously included debug-log-charts.html which fetched multi-hundred-MB debug.log.gz files in the browser, decompressed them with pako.js, parsed every line, and rendered 11 Plotly charts client-side. This made report pages unresponsive. Now that report.py pre-generates the charts as static PNGs: - pr-report.html: replace the debug-log-charts.html include with an img loop over graph.plots, using loading="lazy" - debug-log-charts.html: delete (344 lines of client-side JS) - base.html: remove pako.js and Plotly CDN scripts (both are independently included by pr-chart.html and nightly-chart.html via their own script tags) The debug.log download link is preserved.
Rewrite to document the TOML config + matrix entry workflow, removing stale references to the old two-commit comparison CLI, --datadir requirement, profiles, and BENCH_DATADIR env var.
Debug logs were consuming 388MB on gh-pages. They are already uploaded as CI artifacts with 90-day retention during benchmark runs. - Remove gzip compression and copying of debug logs in report generation - Remove debug log extraction in publish-results workflow - Replace per-graph "Download debug.log" links with a single link to the CI run page where artifacts can be downloaded - Keep matplotlib plot generation from debug logs (plots are still generated during report phase, just the raw logs aren't published)
The PR comment with result links was posted before GitHub Pages finished deploying, leading to broken links. Add a wait-for-pages job that polls for the pages-build-deployment run matching our exact gh-pages commit, then blocks until it completes.
Previously, prune=10000 was causing flushes of the UTXO set when block pruning was taking please, resulting in logs like: ❯ zcat 32000-instrumented-pr-debug.log.gz | rg UTXO 2026-02-12T07:22:57Z * Using 31990.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space) 2026-02-12T07:28:51Z [warning] Flushing large (2 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:33:10Z [warning] Flushing large (3 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:37:23Z [warning] Flushing large (4 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:42:03Z [warning] Flushing large (4 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:46:34Z [warning] Flushing large (5 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:51:10Z [warning] Flushing large (6 GiB) UTXO set to disk, it may take several minutes 2026-02-12T07:55:57Z [warning] Flushing large (7 GiB) UTXO set to disk, it may take several minutes 2026-02-12T08:00:35Z [warning] Flushing large (8 GiB) UTXO set to disk, it may take several minutes 2026-02-12T08:05:16Z [warning] Flushing large (8 GiB) UTXO set to disk, it may take several minutes 2026-02-12T08:10:00Z [warning] Flushing large (8 GiB) UTXO set to disk, it may take several minutes 2026-02-12T08:14:36Z [warning] Flushing large (8 GiB) UTXO set to disk, it may take several minutes 2026-02-12T08:16:47Z [warning] Flushing large (8 GiB) UTXO set to disk, it may take several minutes and generally interrupting benchmarking. Remove this effect by setting prune to such a high value it will never trigger. Prune is **required** to permit us to continue syncing from a pruned datadir.
Adds a separate benchmark job (benchmark-noav) that runs IBD with -assumevalid=0 to measure full script verification performance. Uses a dedicated TOML config with uninstrumented-only matrix, and prefixes artifacts with noav- so the publish workflow can handle them alongside existing runs.
Introduce CoinsViewOverlay::StartFetching, which maps all input prevouts of a block to a new m_inputs vector of InputToFetch elements. Returns a ResetGuard which is lifetime bound to the block, while the InputToFetch elements are lifetime bound to the block as well. Introduce StopFetching to clear the m_inputs vector. CCoinsViewCache::Reset is made virtual and is overridden in CoinsViewOverlay. StopFetching is called on Reset, so the InputToFetch objects will not exceed the lifetime of the block. Introduce ProcessInput to fetch the utxo of an individual input in m_inputs. Each caller fetches the input at m_input_head and increments it, so each call will fetch the next input in the queue. Fetch coins from the m_inputs vector in FetchCoinFromBase by scanning all inputs until we discover the input with the correct outpoint. This is designed deliberately so multiple threads can call ProcessInput independently. Co-authored-by: l0rinc <pap.lorinc@gmail.com> Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Inputs spending outputs of an earlier transaction in the same block won't be in the cache or the db. They also won't be requested by FetchCoinFromBase, so we can filter them out to not waste time trying to fetch them. Build an unordered set of seen txids while flattening m_inputs and skip any prevout whose hash is already in the set. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Provides a worst-case upper bound on the number of inputs that can fit in a block, so callers (e.g. parallel input prefetching) can pre-allocate stable storage and rule out reallocation of per-input state. Cherry-picked from PR bitcoin#9938 (Lock-Free CheckQueue), with MAX_TXINS_PER_BLOCK renamed to MAX_INPUTS_PER_BLOCK to match the call site. Co-authored-by: Jeremy Rubin <jeremy.l.rubin@gmail.com>
Prepares for ProcessInput to be called from multiple threads. This flag acts as a memory fence around InputToFetch::coin. There is no lock guarding reads and writes of the coin field. Instead we use the flag's release/acquire semantics to ensure that when the main thread reads the coin it will have happened after a worker thread has finished writing it. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Prepares for ProcessInput to be called from multiple threads. ProcessInput reads from base. For ProcessInput to be safe to call in parallel on separate threads, it must not be mutated. Flush, Sync, and SetBackend can modify base, so we override these and StopFetching before calling the base class. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Add a configuration option for the number of worker threads used for parallel UTXO input fetching during block connection. Default is 4 threads, max is 16, 0 disables parallel fetching.
Prepares for ProcessInput to be called from multiple threads. Introduce a ThreadPool shared pointer to CoinsViewOverlay. A pool managed externally can be passed in the constructor. A global thread pool is used in fuzz harnesses since iterations can happen faster than the OS can create and tear down thread pools. This can cause a memory leak when fuzzing. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Leverages the thread pool to fetch inputs on multiple threads, while the overlay serves inputs on the main thread. This is a performance improvement over blocking the main thread to fetch inputs. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: l0rinc <pap.lorinc@gmail.com> Co-authored-by: sedited <seb.kung@gmail.com>
Move the responsibility of retrieving coins from GetP2SHSigOpCount to its caller. This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins. Define two explicit template specializations for both a span of references to coins and a span of coins. This allows using it for both Coin entries referenced from the CCoinsViewCache, and from contiguous memory, like the vector in CBlockUndo.
Move the responsibility of retrieving coins from GetTransactionSigOpCost to its caller. This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins. Define two explicit template specializations for both a span of references to coins and a span of coins. This allows using it for both Coin entries referenced from the CCoinsViewCache, and from contiguous memory, like the vector in CBlockUndo.
Move the responsibility of retrieving coins from CheckTxInputs to its caller. The missing inputs check will be moved in an upcoming commit. This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins. Define two explicit template specializations for both a span of references to coins and a span of coins. This allows using it for both Coin entries referenced from the CCoinsViewCache, and from contiguous memory, like the vector in CBlockUndo.
This should introduce no behaviour change, but avoids a few extra vector allocations and coins lookups
…InputScripts Move the responsibility of retrieving coins from the CCoinsViewCache in CheckInputScripts to its caller. Add a helper method in CCoinsViewCache to collect all Coin's outputs spent by a transaction's inputs. Callers of CCoinsViewCache are updated to either pre-fetch the spent outputs, or pass in an empty vector if the precomputed transaction data has already been initialized with the required outputs. This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins.
Move the BIP30 checks from ConnectBlock to a new SpendBlock method. This is a move-only change, more content to SpendBlock is added in the next commits. The goal is to move logic requiring access to the CCoinsViewCache out of ConnectBlock and to the new SpendBlock method. SpendBlock will in future handle all UTXO set interactions that previously took place in ConnectBlock. Callers of ConnectBlock now also need to call SpendBlock before. This is enforced in a future commit by adding a CBlockUndo argument that needs to be passed to both. This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins. Co-authored-by: w0xlt <94266259+w0xlt@users.noreply.github.com>
This is a part of a series of commits for removing access to the CCoinsViewCache in consensus verification functions. The goal is to allow calling verification functions with pre-fetched, or a user-defined set of coins.
Move the remaining UTXO-related operations from ConnectBlock to SpendBlock. This includes moving the existence check, the UpdateCoins call, and CBlockUndo generation. ConnectBlock now takes a pre-populated CBlockUndo as an argument and no longer accesses or manipulates the UTXO set. Co-authored-by: w0xlt <94266259+w0xlt@users.noreply.github.com>
sedited
force-pushed
the
spendblock_plus_coinsfetcher
branch
from
63d2759 to
141dcd4
Compare
Benchmark ResultsComparison to nightly master (median of last 7 runs):
|
willcl-ark
force-pushed
the
master
branch
5 times, most recently
from
b207ac1 to
c3647b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.