define non-custodial pool payouts extension

[GitGab19]

This PR is heavily inspired by the draft PR opened by @warioishere (#195) and all the discussions in #192.

I changed names, message fields and overall structure of the extension.

Let me know what you think about it @warioishere.

[warioishere]

looks good to me

two questions on how this is meant to handle payout sets that change over time (sliding-window PPLNS; or a mod ewhere the window resets on a found block, so the set before vs after a block is a different, already-settled epoch):

1. Freshness, and recovery on rejection. A RequestPayoutOutputs. Success can be stale by the time the job is declared. Since §5.2/§5.3 validate the declared coinbase before acknowledging, we'd make that check epoch-aware on the validating side and reject a superseded set from our own state (no extra wire data). Is that the intended home for freshness, and would you (a) state normatively that a Success is single-use / the JDC SHOULD request a fresh set per declared job, and (b) define how the JDC learns a rejection was caused by staleness, so it re-requests RequestPayoutOutputs and retries rather than treating it as a generic declaration failure? The re-fetch message itself already exists, only the "stale → re-request" signal is missing. Without it, a JDC that caches and reuses a set (e.g. "only re-request when the value changes") silently builds stale coinbases against a sliding-window or reset pool, which neither consensus nor the value field catches?

2. Exact sum. A per-recipient split won't hit Σ = available_payout_value exactly (rounding + sub-dust). We place 100% on-chain by folding the remainder into kept outputs. Is that the intended way to satisfy the rule, or did you have specific residual handling in mind?

[GitGab19]

About your point n.1, I added a clarification about the single-use and freshness requirements of every RequestPayoutOutputs.Success:

- A `RequestPayoutOutputs.Success` response is single-use. The JDC MUST request a fresh payout output set for each custom job it declares: before each `DeclareMiningJob` in Full-Template mode, and before each `SetCustomMiningJob` in Coinbase-only mode. The JDC MUST NOT reuse a payout output set across multiple custom jobs.
- The validating party MUST treat each `RequestPayoutOutputs.Success` as a single-use pending payout output set. It MUST reject a `DeclareMiningJob` or `SetCustomMiningJob` if the payout output set is unknown or already used.
- The validating party MAY reject a payout output set that is stale or superseded according to its payout-accounting policy.
- When a job is rejected because its payout output set is stale or superseded, the validating party SHOULD use the error code `stale-payout-outputs`. Upon receiving this error, the JDC SHOULD request a fresh payout output set before retrying the job declaration.

About your point n.2, I added the following clarification:

Any rounding, dust-threshold, or payout-policy residual MUST be represented in `coinbase_tx_outputs`, so that the returned output set sums exactly to `available_payout_value`. This extension does not define how residual value is assigned; the JDS assigns it according to its payout policy.

[warioishere]

Happy with this as the model, I'll align my pool-side implementation to this spec and can report back if a first version is available for testing if you start to adapt the jd-client

[warioishere]

blitzpool runs with the curent spec of this extention and could be tested against a jd-client that is built against 0x0003

[GitGab19]

Great job @warioishere, I'll try to prepare a draft solution in JDC as soon as I can!

[GitGab19]

The thing I don't like of this proposal is that we introduce a latency between a NewTemplate and the new jobs containing the final coinbase_tx_outputs returned by the RequestPayoutOutputs.Success..

[warioishere]

I thought about this latency in my own PR (#195), adressing it is what I tried to do in the draft I put together. Since this reworks #195, let me put the question back rather than just restate my take: there may be a reason the stricter shape is better that I'm not seeing.

The way I read this is,, and maybe I didnt think about single-use properly, the round-trip ends up on the critical path because the response is single-use, bound to the exact available_payout_value, and has no fallback, so after each NewTemplate (worst on a prev_hash change) the JDC has to wait for RequestPayoutOutputs.Success before it can declare anything.

In #195 I left two ways out, and I'm wondering whether either fits here:

• Fall back to AllocateMiningJobToken.Success.coinbase_tx_outputs when there's no fresh response yet, declare right away, refine on the next job. The first job after a tip uses a slightly stale split, but for a PPLNS-style window that barely moves in one round-trip. Is there a validation reason that's off the table once the extension is negotiated?

• Or loosen "single-use" to "valid within the current prev_hash window," so the JDC can prefetch the next set and have it ready when the template lands.

Single-use is clearly nice for clean per-job attribution on the validating side, I'm just not sure it's worth a round-trip on every job. Could be missing context, so how do you see the tradeoff?

[GitGab19]

Fall back to AllocateMiningJobToken.Success.coinbase_tx_outputs when there's no fresh response yet, declare right away, refine on the next job.

What do you mean with fallback to AllocateMiningJobToken.Success.coinbase_tx_outputs? Temporary allocating the reward to the pool address? Or do you mean using the last RequestPayoutOutputs.Success.coinbase_tx_outputs values while waiting the response to the fresher RequestPayoutOutputs message?

[warioishere]

sorry, reading my comment back from a couple of weeks ago, i muddled two separate things, which is probably exactly why the fallback line came across as ambiguous. let me try to untangle it. If it doesn't sound logic, be easy with me because I am just into this since 2 years or so :)

On the question: by "fall back to AllocateMiningJobToken.Success.coinbase_tx_outputs" I meant the plain §6.4.3 base rule, declare against whatever the pool committed at token time, purely as an availability net when there's no usable response yet (initial connection, JDS unreachable). Not "temporarily pay the pool address" as a policy, what goes in that field is the pool's choice. But honestly that fallback is a side thing, it isnt what removes the latency you flagged, and I shouldn't have leaned on it.

The part that actually matters is your second reading, reusing a prior RequestPayoutOutputs.Success instead of blocking on a fresh one. It made me question whether "single-use per declared job" is the granularity we really need. A payout set only gets consumed when a block is actually found with it, re-declaring jobs on the same tip without finding a block settles nothing, so blocking on a fresh set before each one seems to buy freshness we never use. The one thing that must never happen is the same set settling two found blocks, but a found block advances the tip, so its set is already stale for the next one, which stale-payout-outputs catches anyway.

The obvious objection to "let the JDC reuse" is cherry-picking, a JDC shopping the most favorable old set. So maybe the tight version is: the JDC may reuse only the single most-recently-issued set, until a newer one is issued or the tip advances. There's only ever one reusable set (always the latest), so there's nothing to shop, latency's gone on the hot path; and it's actually less validator state than tracking many. Freshness still holds, the pool can re-issue as often as it likes in the background, the JDC just isn't forced to wait per job.

The bit I'm genuinely unsure about is the value: the last set sums to its issued available_payout_value, so under fee drift it won't match the new one exactly. Is that a real problem for you, or is the residual something the JDC carries (or would Σ ≤ available_payout_value be enough)? And, do you see a settlement or attribution case where "reuse the latest" breaks, that per-job single-use was actually guarding against? I'd rather have your read on that than assume.

[GitGab19]

I think I just found another simpler option to solve the latency issue, and allow optimistic mining when running this extension.

The idea is to add a bool field to RequestPayoutOutputs.Success, something like allow_proportional_reuse:

• if false, current semantics stay as-is: these exact outputs are single-use.
• if true, the JDC can optimistically use the same outputs set on a newer template by keeping the same scripts, but re-assigning the new available_payout_value proportionally across them. In parallel JDC always sends the usual RequestPayoutOutputs for the fresh value, and switches to that once the expected RequestPayoutOutputs.Success arrives.

By using a BOOL, this would let JDS/Pool opt into this only when their payout policy is fine with it.

[warioishere]

Maybe im missing something, but isnt that part already covered by how jd-client works today? Shares found before SetCustomMiningJob.Success are cached per template and flushed once the pool job_id is known, and downstream jobs go out before the declaration anyway. So the declaration of the optimistic job isnt really an extra message, its just the normal declaration of that template, only with the rescaled outputs — or do you see a case where thats not enough?

The only real extra I can see is the re-declare mid-template when the fresh Success arrives. Couldnt that switch be optional? JDC just rides the scaled job until the next template, and the fresh set primes the next job. Then the message count stays the same as base protocol and RequestPayoutOutputs becomes a pure background refresh.

One thing about validating the rescaled set: for the original set we settled that residual handling is pool policy, becaue the JDS builds the set itself. But now the JDC computes the amounts and the pool has to check them — floor vs round, or where the residual goes, easily ends up a few sats apart between implementations, and the exact multiset validation would reject the job over that. Instead of pinning a normative rounding algorithm, how would you stand to a small per-output tolerance? Scripts have to match the issued set exactly, sum has to equal the new value exactly, each amount within a few sats of proportional — any skew a JDC could play with is capped by the tolerance then. Or did you already have something in mind for this?

One thing we think the bool actually needs to work reliably: the latest issued set must be able to back more then one job on the same tip. Fees can drift again before the fresh Success arrives, and if the set already counts as used after the first scaled job, the second declaration gets rejected — so the JDC either has to do the blocking round-trip again (which defeats the point of the reuse) or worse treats it as a fatal error. So with the bool set, single-use should bind to a found block / tip change instead of per declaration — which is basicly the consumption model we discussed above anyway.

[warioishere]

Thinking about this a bit more while walking through the timing, something occured to me: within a tip the waiting is actually almost free, isnt it? When a fee-update template arrives and the fresh set is still in flight, the JDC just keeps mining the previous job — same tip, still valid, just slightly stale fees for one round-trip. No wasted work at all.

The round-trip only really hurts on the tip change, where the old job becomes unusable and the miners stall (or grind the old tip) until the Success arrives. So im wondering: should the proportional reuse also cover the first job after SetNewPrevHash, bridging with the last set of the previous tip until the fresh one arrives? If the reuse is within-tip only, the bool mostly helps the case that didnt hurt in the first place. From the payout side a sliding window barely moves in those few hundred ms, so the bridge seems harmless — but it would need a small carve-out from the staleness rule for exactly that one bridging job.

[warioishere]

For what its worth, this tip-change case is also where I ended up in #195: §5.2 there named the prev_hash transition as the dominant latency cost, and I tried to cover it with a predictive prefetch (§3.2) and the §6.4.3 fallback (§3.4), so the critical path never blocks. Im not saying #195 had the better overall shape, the strict single-use accounting here fixed real freshness holes my draft was too loose about, but maybe those two escape hatches are worth a second look for exactly this case, with your bool as the opt-in.

[plebhash]

One detailed I missed is that while optimistically working on the newer job with the proportionally scaled outputs, JDC likely needs to send the usual set of DeclareMiningJob and SetCustomMiningJob messages, in order to let JDS&Pool verify eventual shares found in that short period of time.

I don't know if there's a way to avoid this TBH.

say there were 3 miners getting paid:

• A gets 50% of available_payout_value
• B gets 30%
• C gets 20%

but B started sending less and C started sending more work, so at some point the distribution changes:

• A gets: 50%
• B gets 25%
• C gets 25%

so eventually an optimistic DeclareMiningJob might get rejected

and when that happens, the optimistic JDC work will be lost

assuming that JDC always sends a RequestPayoutOutputs at the same time it sends the optimistic DeclareMiningJob, eventually the new RequestPayoutOutputs.Success will arrive, so the window of lost work will be small... but it won't be null

---

on the other hand, if this optimistic approach isn't part of the extension, the tradeoffs are:

• a permanent delay of RequestPayoutOutput round-trip time (RTT) on JDC's ability to declare freshly updated/increased template revenues
• on every chain tip update, JDC's hashrate becomes stale/dead for 1 RTT

---

so there's negative sides to both, but hard to quantify which one is worse

[plebhash]

another important detail to keep in mind is that with this optimistic approach, we come back to the fundamental problem of how to deal with weights

• we need to establish an algorithm that will lead to deterministic rounding of floating point numbers
• the payout distribution might end up with dust

[plebhash]

the Pool in Coinbase-only mode

this sentence doesn't make much sense

[plebhash]

• AllocateMiningJobToken.Success.coinbase_tx_outputs continues to define the coinbase output reservation associated with the mining job token, as specified by the base Job Declaration Protocol.

this sentence feels a bit too vague

in the base spec, AllocateMiningJobToken.Success.coinbase_tx_outputs carries heavy semantic meaning with regards to the locking scripts: the first output is designated as the pool payout output, and any additional outputs imposed by JDS are expected to be present in the declared job.

under this extension, that semantic meaning is much looser: the field's remaining purpose is to provide the constraints that JDC communicates to its TP via CoinbaseOutputConstraints (block space and sigops) before it gets a NewTemplate.

so the locking scripts are essentially only there to "save up space/sigops" for the coinbase tx on the template

I wouldn't try to change any of this logic (either on the base spec or on this extension), but this distinction feels important and we're missing a more explicitly clear explanation here

[plebhash]

serialized size AND sigops

[plebhash]

Suggested change

	| 3 | JDC -> TP | `CoinbaseOutputConstraints` | Reserves enough template space for outputs associated with the token. |
	| 3 | JDC -> TP | `CoinbaseOutputConstraints` | Reserves enough template blockspace and sigops for outputs associated with the token. |

[GitGab19]

Closing in favor #203