feat: add gas visibility metrics for estimation, pool and bundles - #1341
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files
... and 4 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The changes are instrumentation-only with comprehensive unit tests added, and the remaining review notes are minor metric HELP-text wording fixes.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Adds new Prometheus metrics across types, sim, rpc, pool, builder, and the CLI metrics exporter to make AA-stage failures and gas accounting inefficiencies (notably around >2²⁴ gas / EIP-8037 “state gas”) observable without changing functional behavior.
Changes:
- Introduce a bounded
AaErrorCodeparser and a sharedrecord_aa_errorhelper to count AA errors by stage/entrypoint without leaking revert text. - Add pool and builder “gas efficiency / compensation” histograms + counters (mined op efficiency, admission verification efficiency, bundle fee vs compensation, charged-gas vs tx gas-used ratio).
- Add estimator internal instrumentation (eth_call counts, non-convergence, clamped estimates, error kinds) and wire per-metric histogram buckets in the CLI exporter.
| File | Description |
|---|---|
| crates/types/src/validation_results.rs | Adds AaErrorCode parsing and exposes AA code extraction from ValidationRevert + tests. |
| crates/types/src/pool/error.rs | Exposes AA code extraction for SimulationViolation so callers can record stage metrics. |
| crates/types/src/lib.rs | Re-exports AaErrorCode and introduces shared entry_point_metrics module. |
| crates/types/src/entry_point_metrics.rs | New shared AA-error counter metric + stage enum and tests. |
| crates/types/Cargo.toml | Adds metrics-util dev-dependency for recorder-based metric tests. |
| crates/sim/src/simulation/mod.rs | Adds SimulationError::aa_error_code() helper for metric labeling. |
| crates/sim/src/lib.rs | Re-exports estimator metric constants and record_estimation_error. |
| crates/sim/src/estimation/v0_7.rs | Records “clamped estimate” metrics and labels paymaster verification field correctly. |
| crates/sim/src/estimation/v0_6.rs | Records “clamped estimate” metrics and adds/extends tests for estimator metrics. |
| crates/sim/src/estimation/mod.rs | Adds estimator search/nonconvergence/clamp/error metrics and their recording points + tests. |
| crates/sim/src/estimation/estimate_verification_gas.rs | Threads a field label through binary search for verification estimation metrics. |
| crates/sim/src/estimation/estimate_call_gas.rs | Threads field="call" into the binary search metric recording. |
| crates/sim/Cargo.toml | Adds metrics-util dev-dependency for metric tests. |
| crates/rpc/src/eth/router.rs | Records estimator error-kind metrics and AA validation errors during eth_estimateUserOperationGas. |
| crates/pool/src/mempool/uo_pool.rs | Records AA errors at pool admission + mined/admission gas-efficiency metrics; updates tests. |
| crates/pool/src/mempool/pool.rs | Updates pool tests for expanded MinedOp fields. |
| crates/pool/src/mempool/mod.rs | Introduces and exports pool gas metric constants. |
| crates/pool/src/mempool/gas_metrics.rs | New pool gas-efficiency histograms (mined op and admission verification) with bounded labels + tests. |
| crates/pool/src/lib.rs | Re-exports pool gas metric constants. |
| crates/pool/src/chain.rs | Extends MinedOp with success + actualGasUsed and preserves them when parsing events; tests. |
| crates/pool/Cargo.toml | Adds metrics-util dev-dependency for metric tests. |
| crates/builder/src/transaction_tracker.rs | Captures UserOperationEvent logs from receipts into TrackerUpdate::Mined (no extra RPC). |
| crates/builder/src/lib.rs | Exports bundle metric constants for CLI bucket wiring. |
| crates/builder/src/bundle_sender.rs | Records mined-bundle metrics on receipt processing. |
| crates/builder/src/bundle_proposer.rs | Records AA errors at bundle revalidation / handleOps simulation / onchain revert parsing. |
| crates/builder/src/bundle_metrics.rs | New builder bundle compensation/fee counters and gas-compensation ratio + op-count histograms with tests. |
| crates/builder/Cargo.toml | Adds dependencies needed for log decoding + metrics-util dev-dependency. |
| Cargo.lock | Locks new dependencies (metrics-util, rundler-contracts, alloy-sol-types) pulled into crates/tests. |
| bin/rundler/src/cli/metrics.rs | Wires per-metric histogram buckets via set_buckets_for_metric for the new ratio/count histograms. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
jakehobbs
reviewed
Sep 28, 2026
jakehobbs
left a comment
jakehobbs
left a comment
Contributor
There was a problem hiding this comment.
Codex review: I found two cases where the new metrics can misrepresent production behavior.
| .collect::<Vec<_>>(); | ||
| let sponsored_ops = events | ||
| .iter() | ||
| .filter(|event| event.actual_gas_cost.is_zero()) |
Contributor
There was a problem hiding this comment.
When an op has bundler_sponsorship but retains nonzero fee caps, the proposer still accepts it without clearing those caps, so its UserOperationEvent.actualGasCost can be positive. This classifies a sponsored bundle as has_bundler_sponsored_op=no and records zero sponsored ops. Classify sponsorship from the op permission or another tracked identifier rather than from zero gas cost. — Codex
Contributor
There was a problem hiding this comment.
This should not actually be possible. The proxy (wallet server) in front of wallet should reject in this case.
Contributor
Author
There was a problem hiding this comment.
Checked the whole path. The zero-fee check only exists in wallet-server's wallet_sendPreparedCalls, which rejects a sponsored op unless maxFeePerGas, maxPriorityFeePerGas, preVerificationGas and paymasterVerificationGasLimit are all 0. eth_sendUserOperation with an x-alchemy-policy-id header forwards sponsored ops without checking fees, and neither the paymaster nor rundler checks them. So an op sent that way with nonzero fees would pay actualGasCost on chain and be counted as has_bundler_sponsored_op="no".
I'm keeping zero-cost detection here: detecting from permissions would mean changing the bundle sender's state, and this PR is metrics-only. The gap is noted in Known limits and I'll raise the eth_sendUserOperation check with the wallet-server owners separately.
Add the rundler_entry_point_aa_errors counter, labelled by stage, aa_code and entry_point, so validation failures such as AA26 are visible in Prometheus instead of being folded into generic JSON-RPC error codes. Stages: - estimation: RevertInValidation from eth_estimateUserOperationGas - pool_admission: simulateValidation failure in UoPool::add_operation - bundle_revalidation: op re-simulation during bundle building - bundle_handle_ops: FailedOp from the handleOps simulation - onchain: FailedOp decoded from a reverted bundle transaction The aa_code label is parsed with AaErrorCode::from_message, which only accepts "AA" followed by two digits (otherwise "none"), so the label stays bounded and never carries revert text. The hooks only read the error and do not change control flow, return values or events.
Add rundler_op_pool_mined_op_gas_efficiency, a histogram of actualGasUsed / gas limit for each mined op that was in the pool. The limit is what the entry point budgets for the prefund, including preVerificationGas (v0.6 counts the verification gas limit 3x with a paymaster). Ops whose pool copy has a different hash are skipped. Labels are entry_point, success, has_paymaster, has_factory, has_7702_auth and fresh_nonce_slot, all booleans except the entry point. success=false exposes call-phase failures, which never carry an AA code on v0.7+. MinedOp now keeps success and actualGasUsed from UserOperationEvent. The histogram gets ratio buckets instead of the global millisecond buckets.
Add rundler_op_pool_admission_verification_gas_efficiency, a histogram of (pre_op_gas - preVerificationGas) / total verification gas limit from the pool admission simulation, with the same op labels as the mined op histogram minus success. The value is recorded on the scale of the reject threshold: for v0.6 ops with a paymaster, where the check halves the threshold, it is measured against one verification gas limit and can exceed 1. An op is rejected exactly when its recorded value is below the configured threshold. The existing efficiency check only computed this ratio when its reject threshold is above 0 (default 0). The ratio is now computed once by gas_metrics::verification_gas_efficiency, recorded for every op that passes simulation, including ops the threshold then rejects, and reused by the check, whose outcome is unchanged. An op with a zero verification gas limit is still accepted and is not recorded.
The bundler is the handleOps beneficiary, so a mined bundle is paid back the sum of actualGasCost of its UserOperationEvents. Nothing compared that with what the bundle transaction cost. The transaction tracker now keeps the UserOperationEvents from the receipt it already fetches, with their emitter address. The sender keeps only events from the pinned entry point and records: - builder_bundle_fee_paid_gwei and builder_bundle_compensation_gwei counters (entry_point, sender, success), for every mined bundle - builder_bundle_gas_compensation_ratio (sum of actualGasUsed / gas used) and builder_bundle_fee_compensation_ratio (compensation / fee) histograms, labelled by entry_point, bundle_size (1, 2-4, 5-9, 10+) and has_bundler_sponsored_op - builder_bundle_ops and builder_bundle_bundler_sponsored_ops histograms per entry point Histograms are recorded for successful bundles only, since a reverted receipt has no events. The fee is gas used times effective gas price and leaves out the separate L1 data fee on OP-stack chains.
Gas estimation only had timing histograms. Add:
- gas_estimator_eth_calls{entry_point, field, outcome}: eth_calls a
binary search used, recorded on every exit and counting a call that
failed. outcome is success, revert, error, or not_converged when the
search used all max_gas_estimation_rounds, which today only surfaces
as GasEstimationError::Other
- gas_estimator_clamped_estimates{entry_point, field}: estimates whose
buffered value was cut down to max_verification_gas or
max_bundle_execution_gas
- gas_estimator_errors{entry_point, kind}: eth_estimateUserOperationGas
errors by GasEstimationError variant, recorded in the RPC router
field is verification, paymaster_verification or call. The search
loop moves into binary_search, wrapped by run_binary_search, which
records the metric and returns the same estimates and errors as
before.
Bundler sponsored ops pay nothing on chain and are billed separately by the paymaster service, so a compensation / fee ratio is meaningless for bundles that contain them. Remove builder_bundle_fee_compensation_ratio; the aggregate for paying bundles can still be computed from the counters. Add has_bundler_sponsored_op (yes, no, or unknown for bundles without user operation events such as reverted ones) to the fee paid and compensation counters, and use the same values on the gas compensation ratio. Drop the sender label from the counters to keep their cardinality independent of the number of sender EOAs. Sponsored ops are detected by an actualGasCost of 0. When the bundle gas price is unknown or 0 every op costs 0, so has_bundler_sponsored_op is unknown and builder_bundle_bundler_sponsored_ops is not recorded. Skip builder_bundle_gas_compensation_ratio on chains that price DA in preVerificationGas but leave it out of the receipt gas used, such as OP stack chains, where sum(actualGasUsed) includes DA gas and the ratio sits far above 1. Chains that include DA gas in the gas limit, such as Arbitrum, keep it.
cargo-deny rejects metrics-util being declared in five crates without a shared workspace dependency. Declare it once in the workspace and inherit it, enabling the debugging feature only in the test-only dependencies.
inakov
force-pushed
the
ivan/gas-visibility-metrics
branch
from
September 30, 2026 09:17
2428825 to
603ad4e
Compare
rado-alchemy
approved these changes
Sep 30, 2026
jakehobbs
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instrumentation only: new metrics, no change to estimation, pool, builder or RPC behaviour. Every returned value, error and state transition is unchanged. One commit per metric group.
Motivation
On a Glamsterdam devnet, sponsored EIP-7702 ops failed with
AA26 over verificationGasLimitand no metric showed it.eth_callruns withmax_gas_estimation_gas(550M), far above 2^24, so state gas is paid from a reservoir thatgasleft(), and therefore the EntryPoint's metering, cannot see.actualGasUsedon chain.These metrics make that visible before we change estimation, so the fix's effect can be seen in Grafana.
Proposed Changes
types,pool,builder,rpc):rundler_entry_point_aa_errors{stage, aa_code, entry_point}.stageisestimation,pool_admission,bundle_revalidation,bundle_handle_opsoronchain.aa_codeisAAplus two digits, ornone. It's parsed with a check that never panics, and never contains revert text.onchainis recorded before the submission proxy handles the revert, so it's counted even when the proxy attributes it.pool):rundler_op_pool_mined_op_gas_efficiency, a histogram ofactualGasUsed / gas limit. The limit is the EntryPoint's own prefund formula, includingpreVerificationGas.entry_point,success,has_paymaster,has_factory,has_7702_auth,fresh_nonce_slot.success=falseexposes call-phase failures, which never produce an AA code on v0.7+.MinedOpnow keepssuccessandactualGasUsedfromUserOperationEvent.pool):rundler_op_pool_admission_verification_gas_efficiency, with the same labels minussuccess.builder): the bundler is thehandleOpsbeneficiary, so a mined bundle is paid back ΣactualGasCostof itsUserOperationEvents.rundler_builder_bundle_fee_paid_gweiandrundler_builder_bundle_compensation_gwei(counters). Labels:entry_point,success,has_bundler_sponsored_op(yes/no/unknown).unknownis for bundles without events, such as reverted ones, and for a bundle gas price that is unknown or 0, since then every op has anactualGasCostof 0.rundler_builder_bundle_gas_compensation_ratio: ΣactualGasUsed÷ receiptgasUsed. Labels:entry_point,bundle_size(1/2-4/5-9/10+),has_bundler_sponsored_op. Below 1.0 means ops were charged for less gas than the bundle used, which is the direct EIP-8037 signal. Not recorded on chains that price DA inpreVerificationGasbut leave it out of the receiptgasUsed, such as OP-stack chains; Arbitrum includes DA gas in the gas limit and keeps it.rundler_builder_bundle_opsandrundler_builder_bundle_bundler_sponsored_ops: op counts per successful bundle, so the share of bundler-sponsored ops can be computed. The sponsored count isn't recorded when the bundle gas price is unknown or 0.has_bundler_sponsored_op="no".sim,rpc), withfield=verification/paymaster_verification/call:rundler_gas_estimator_eth_calls{entry_point, field, outcome}:eth_calls per binary search, recorded on every exit and counting a call that failed. Loweringmax_gas_estimation_gasmeans more continuation calls.outcomeissuccess,revert,error, ornot_convergedwhen the search used allmax_gas_estimation_rounds. Today that failure is only aGasEstimationError::Other; its count isrundler_gas_estimator_eth_calls_count{outcome="not_converged"}.rundler_gas_estimator_clamped_estimates{entry_point, field}: estimates whose buffered value was cut down tomax_verification_gasormax_bundle_execution_gas, which removes the buffer.rundler_gas_estimator_errors{entry_point, kind}:eth_estimateUserOperationGaserrors byGasEstimationErrorvariant, recorded in the RPC router. Several variants share one JSON-RPC code.set_buckets_for_metric, because the global buckets are sized for milliseconds. The bucket lists are defined next to the metrics, and tests assert that the real metric names match them.Cardinality
senderlabel on the new metrics.Testing
make fmt,make lintandmake test-unitpass: 575 tests, 29 of them new.metrics_util::debugging::DebuggingRecorderand check metric names, labels and values. They're added as a dev-dependency torundler-types,rundler-pool,rundler-builderandrundler-sim.stage="pool_admission",aa_code="AA26"increments.Known limits
actualGasUsedincludes the v0.7+ unused-gas penalty, so mined-op efficiency reads slightly high for ops with a large unused call gas limit.time_to_mine.actualGasCostof 0. Only wallet-server'swallet_sendPreparedCallsrequires sponsored ops to have zero fees;eth_sendUserOperationwith a policy header, the paymaster and rundler don't check. A sponsored op with nonzero fees would pay on chain and be counted ashas_bundler_sponsored_op="no".