feat(clmm): orphaned-position lifecycle and recovery, bin_count passthrough, Raydium via Gateway - #217
Closed
fengtality wants to merge 18 commits into
Closed
feat(clmm): orphaned-position lifecycle and recovery, bin_count passthrough, Raydium via Gateway#217fengtality wants to merge 18 commits into
fengtality wants to merge 18 commits into
Conversation
… listing, DB-aware stop, resolve
API-side of the gateway#678 retry-ownership work (canonical design:
docs/retry-architecture.md in the companion gateway PR). Executors created
via the API run in-process with no controller, so the API owns the
"react to a stranded position" role:
- Stop on a terminal executor returns already_terminated with close_type,
position_address, orphaned_position, and hold_reason instead of the 404
dead end #678 hit (terminal executors are popped from memory within one
tick, so "not in memory" almost always means "already terminated").
404 is reserved for ids the DB has never seen.
- Completion flags stranded exposure in the persisted final state:
an involuntary hold (POSITION_HOLD with hold_reason set — an LP close
that exhausted its retries) or a legacy FAILED-with-position gets
orphaned_position: true and an error-level log. Voluntary holds never
match (a successful close clears position_address first).
- GET /executors/positions/orphaned lists recovery candidates
(SQL-filtered to lp_executor; involuntary holds, FAILED-with-position,
and SYSTEM_CLEANUP restarts flagged needs_onchain_reconciliation).
- POST /executors/{id}/resolve-orphan marks a candidate recovered after
the position is closed externally, silencing listings and warnings.
- bots/controllers lp_rebalancer mirror: halt + skip accounting for
executors that ended with a live position (re-creating one would mint a
second position on top of the stranded one).
Validated live on mainnet: forced close-failure cascade terminated as the
involuntary hold, surfaced in the orphan listing with hold_reason,
re-stop returned already_terminated, and resolve-orphan cleared it after
a direct gateway close recovered all funds + rent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HahKfEY9rvKnZijrzUAFSq
This was referenced Aug 13, 2026
Greptile SummaryThe PR adds durable orphaned-position detection, listing, resolution, and recovery metadata for CLMM executors while preserving the controller’s intentionally process-local safety latch. It also adds CLMM
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains within the eligible follow-up scope. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| bots/controllers/generic/lp_rebalancer/lp_rebalancer.py | Adds a process-local halt when an LP executor terminates with a live position; the reviewed restart and resolution behavior matches the explicitly accepted design. |
| services/executor_service.py | Persists orphan metadata, discovers recovery candidates, resolves recovered records, and handles terminal executor stop requests. |
| routers/executors.py | Exposes orphan listing and resolution endpoints backed by ExecutorService. |
| routers/gateway_clmm.py | Adds pool-address fallback and bin-count forwarding while unifying Raydium with the Gateway CLMM path. |
| database/repositories/executor_repository.py | Adds persistence queries supporting terminal executor lookup and orphan recovery. |
| models/executors.py | Extends executor API contracts with orphan lifecycle and recovery fields. |
Reviews (12): Last reviewed commit: "test(gateway): pin the amm quote-liquidi..." | Re-trigger Greptile
GET /gateway/clmm/pool-info accepts bin_count and forwards it to Gateway's unified trading/clmm/pool-info, so the per-tick liquidity distribution is reachable for orca, raydium, uniswap and pancakeswap (Meteora always returns its own bins). The response model already carried bins. Requests with bin_count > 0 skip the direct-Raydium-API shortcut: that API returns no bin distribution, and only Gateway computes it from on-chain ticks. Also wraps pre-existing long lines and drops an unused import in routers/gateway_clmm.py — the flake8 pre-commit hook lints the whole file and would not otherwise accept a commit touching it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…her connector /gateway/clmm/pool-info special-cased Raydium: it skipped Gateway entirely, called api-v3.raydium.io directly, and reshaped that response to look like Gateway's. That divergence cost real data — the transform hardcoded active_bin_id to None, bin_step to 1, and bins to [] — and it meant Raydium could not answer bin_count at all, since only Gateway computes the tick distribution. Raydium now takes the same path as meteora/orca/uniswap/pancakeswap. The Raydium API helpers and their aiohttp import go with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An lp_executor that exhausts its close retries terminates as an involuntary
POSITION_HOLD with the position still on-chain. Recovering it means closing that
position by address — which this API could not do.
Two things blocked it:
- /gateway/clmm/close and /collect-fees read the position's pool only from the
gateway_clmm_positions table, and 404'd when it was absent. An lp_executor
opens its position straight from the bot to Gateway, so it is never in that
table: on a live deployment the table was empty and every orphan 404'd.
Both endpoints now accept pool_address on the request, resolving
database-first and erroring with 400 (a bad request, not a missing position)
naming pool_address as the fix.
Gateway's close needs only position_address; pool_address is used to snapshot
pending fees before the close so they can be reported.
- /executors/positions/orphaned reported connector_name and trading_pair but
not the DEX or the pool, so a caller had nothing to build a close from. Note
connector_name holds the *network* for an lp_executor ("solana-mainnet-beta").
The DEX ("orca/clmm") and pool live in the executor's stored config; both are
now surfaced as lp_provider and pool_address.
Also wraps two pre-existing over-length lines in gateway_trading.py, which the
whole-file flake8 hook fails on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mt84XBEMVxbbyMG8fDxDKj
The CLMM work on this branch depends on Gateway changes that ship in hummingbot/gateway#679 and are not in the `latest` tag, so a container started from the default image cannot serve the endpoints this branch calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mt84XBEMVxbbyMG8fDxDKj
Gateway's /trading/clmm/positions-owned takes no pool filter — its handler reads only connector, chainNetwork and walletAddress — so the pool_address this API required, forwarded, and documented as a filter never filtered anything: every caller always got the wallet's full position list labeled as one pool's. Remove the parameter end to end (request model, router, gateway_client) so the contract says what actually happens; each returned row carries its own pool_address for callers that want one pool. Includes flake8 fixes in gateway_transaction_poller.py that the pre-commit hook now enforces on the touched file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…routes Audit fixes, most severe first: - close: read the confirmed transaction data Gateway returns — the removed base/quote amounts and positionRentRefunded — record them on the CLOSE event and surface them in a new CLMMClosePositionResponse. The rent tracked as locked at open was never reconciled as refunded. - open/add/remove: prefer Gateway's confirmed on-chain amounts (baseTokenAmountAdded/Removed) over the requested amounts when persisting and responding; requested amounts remain the submitted-not-confirmed fallback. Also fixes REMOVE_LIQUIDITY events never persisting: the event payload carried a "percentage" key GatewayCLMMEvent has no column for, so create_event raised into the log-and-continue handler on every call. - new endpoints mirroring Gateway routes hapi never exposed: POST /gateway/clmm/quote-position (pre-trade deposit split), POST /gateway/clmm/create-pool (CLMM pools; AMM had this, CLMM did not), GET /gateway/clmm/position-info (single position by address). - positions_owned/position-info: pass through rewardTokenAddress / rewardAmount (farm rewards; populated by pancakeswap-sol today) instead of dropping them. - amm create-pool: expose openTime (Raydium CPMM) and slippagePct (Uniswap seeding) which Gateway accepts. - open: reject extra_params keys other than strategyType with a 400 — Gateway's unified open silently ignores everything else. - drop dead surface: dynamicFeePct/minBinId/maxBinId on pool-info (Gateway's declared response schema strips them before serialization; nothing consumes them) and the camelCase pageSize field (renamed page_size; no consumers). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…st swap quote - CLMM and AMM create-pool drop their per-connector named fields; connector params ride extra_params under Gateway's own names (the clmm open contract), spread into the payload, with unknown keys rejected loudly — Gateway destructures a fixed set and silently ignores the rest. Meteora's required configAddress is enforced at the router. - CLMMPositionInfo drops reward_token_address/reward_amount: no Gateway connector populates them (the only assignments are commented out) and the schema fields are being removed from Gateway's trading responses. - CLMMPoolInfoRequest documents bin_count, mirroring Gateway's binCount. - SwapQuoteResponse mirrors what /trading/swap/quote actually returns: gains min_amount_out/max_amount_in/price_impact_pct/pool_address/ route_path, slippage_pct reflects Gateway's applied value, and the phantom gas_estimate (never returned by Gateway) and deprecated expected_amount are gone — nothing consumed either. - Fix the stale module docstring claiming AMM support was removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…ding routes
- Stop forcing slippage 1.0 everywhere: all request models default
slippage_pct=None and the routers omit the key when unset, so Gateway
applies the connector's configured slippagePct (the schema-default
shadowing this used to cause was removed Gateway-side). The swap DB
record and quote response echo the applied/requested value or None —
SwapQuoteResponse.slippage_pct is now Optional and no longer backfills
a fabricated 1.0 (also fixes the falsy-`or` that swallowed slippagePct=0).
- extra_params convention extended to every surface with connector-specific
params, matching the executor stack's LPExecutorConfig.extra_params:
swap quote/execute gain approximateIfNoExactOut (Solana routers; query
values stringified for aiohttp), clmm add gains strategyType (same
contract as open), each guarded by loud unknown-key rejection since
Gateway silently drops unrecognized keys.
- CLMM remove exposes the standard slippage_pct field (Orca-only today).
- create-pool guards pinned to what Gateway actually destructures:
clmm {binStep, feeBps, ammConfigIndex}; amm {configAddress,
ammConfigIndex} + first-class seeding slippage_pct. The phantom keys
(fee/tickSpacing/ammConfig/gasPrice/maxGas/feeConfigIndex/openTime)
passed the guard and were silently ignored by Gateway.
- Fix /clmm/open crash: clmm_pool_info was called without its required
pool_address since the pool-info signature change (TypeError → 500 on
every open).
- Contract tests updated and extended for all of the above.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
- Pending CLMM open no longer 500s with an orphaned on-chain position: Gateway's OpenPositionResponse carries positionAddress only inside the confirmed-only data object (the schema strips everything else, so the old top-level fallback keys could never arrive). A submitted-not- confirmed open now returns 200 with position_address=None and the signature to poll; the poller's discovery sweep records the position once it lands. A confirmed response without an address is still a loud 500, and the confirmed path now reports status "confirmed". - extra_params validation hardened via a shared routers/gateway_extras helper: unknown keys, keys sent to a connector that ignores them, and wrong-typed values (incl. null, and bool-vs-int subclass traps) all 400 locally instead of being silently dropped or reaching Gateway as the string "None". - ROUTER_CONNECTORS gains dflow/okx/titan so bare names route to /router instead of misrouting to /clmm and 404ing. - Swap DB record: price is quote-per-base for BOTH sides (BUY was inverted), the pending fallback keeps tokenIn/tokenOut denominations (BUY no longer stores a base amount in the quote-denominated input column), side is normalized to uppercase, and the dead poolAddress read (never in the execute response schema) is an explicit None. - clmm_fetch_pools speaks each connector's real schema: meteora page/includeUnverified + "field:direction" sortBy; orca sortBy/sortDirection/verifiedOnly, with page>0 rejected loudly for orca instead of a silent no-op that echoed the requested page. - AMM quote/execute swap uppercase side like the unified path. - Honesty fixes: swap amount documented as base-denominated for BUY (ExactOut), poll docstring documents txStatus -2 NOT_FOUND as terminal, activeBinId is not meteora-only, and swap listings report a recorded slippage of 0 as 0 instead of null. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…ng failed txs as submitted Follow-ups from the re-audit of cd3cab7: - The position discovery sweep now covers every Solana CLMM connector whose open can return submitted-not-confirmed (meteora, raydium, pancakeswap-sol, plus orca for externally-created positions) — previously meteora-only, so the pending-open path's "the poller records it once it lands" contract was false for raydium/pancakeswap-sol and those positions were permanently orphaned from the DB. - get_transaction_status_from_response (both routers) maps Gateway's negative statuses to FAILED instead of folding them into SUBMITTED: a failed EVM swap (status -1, zeroed amounts) is now recorded and returned as failed, and the swap execute response reports confirmed/submitted/failed honestly instead of a hardcoded "submitted". - The CLMM open pending branch rejects the EVM late-revert shape (data present without a position address, or negative status) with a loud 500 instead of returning 200 "submitted" for a tx that definitively failed on-chain. - gas_fee falsy-zero in swap listings fixed the same way as slippage_pct one line above it (a recorded 0 must not report as null). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…d residuals
Poller (the load-bearing cluster):
- _check_transaction_status classifies on txStatus ALONE: Gateway returns
txStatus 0 WITH an error message for transient poll failures ("poll
again, don't give up") — the error field no longer promotes a pending
tx to FAILED (one RPC hiccup used to permanently fail in-flight swaps
and close events). -2 NOT_FOUND is DROPPED (terminal after a 180s
blockhash grace) instead of polled for an hour and mislabeled timeout;
0 is an explicit PENDING distinct from None (no information).
- The age timeout fires only after a successful poll says pending —
never while Gateway is unreachable (an outage used to mass-FAIL
everything over an hour old, including confirmed txs). One
availability gate per cycle replaces per-call pings.
- Position close detection: a single position-info 404/500 no longer
closes a position — 3 consecutive misses required (mirrors the
lp_executor gate); the router refresh no longer closes on absence from
one positions-owned read; discovery skips reopening positions closed
within a 300s grace so a lagging listing can't flap CLOSED->OPEN.
- Failed txs record their gas fee; fee of exactly 0 survives.
Fee bookkeeping (double-count / phantom-fee cluster):
- close/collect endpoints mutate the position ONLY when Gateway
confirmed inline; submitted txs are booked once by the poller's
confirm path (which now books close fees before closing); failed txs
mutate nothing. Previously every pending collect double-counted and
every failed close permanently inflated *_fee_collected.
- ADD_LIQUIDITY raises the PnL baseline (initial amounts) on confirm —
pnl_summary no longer counts added capital as profit.
Honesty and contract:
- add/remove/close/collect responses report confirmed/submitted/failed
instead of hardcoded "submitted"; writes on unrecorded positions log
loudly instead of silently dropping the event.
- Swap summary: quote-denominated volume per quote token (was summing
the base leg across mixed pairs while claiming quote); status filter
case-insensitive; tz-aware time filters; 10k-row cap logged.
- CLMM remove renames percentage -> percentage_to_remove (matches AMM
and Gateway; position.percentage still means range width).
- close/collect honor an explicit request wallet (same precedence as
open/add) and drop the required-but-unused pool_address 400.
- get_native_gas_token single-sourced in gateway_client (three drifted
copies produced MATIC/None/UNKNOWN for one chain); status mapping
single-sourced in gateway_extras.
- Wallet placeholder check matches Gateway's real "<chain-wallet-address>"
template; unreachable Gateway raises 503 instead of "No wallet
configured" 400 or "'error' in None" crashes; hardware wallet
addresses included in discovery/balance sweeps; deprecated /pools
maps Solana routers to solana; position-info 503s on connection error.
- Dead code removed per convention: unused request models, legacy
_poll_open_positions wrapper, poll_transaction_once, unused repo
helpers. DISCOVERED event type documented everywhere event types are
enumerated.
Accepted residuals documented in place, NOT fixed (by decision):
pending-tx amounts/price never backfilled from txData; discovery-time
entry price/synthetic history for pending opens; int-only extra_params
strictness; legacy lowercase side rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…and gas fields A fee or gas value of exactly 0 was stored as None (unknown) on close and collect events, contradicting the is-not-None convention adopted everywhere else and leaving pending-fee columns stale on the poller's confirm path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…_api_keys A None result from the client (connection error mid-batch) was filtered out as if it succeeded, so the endpoint reported keys updated that never reached Gateway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…ools; uppercase side on read Gateway folded /trading/amm/quote-swap and execute-swap into the unified /trading/swap route (connector as name/type, pool resolved internally), so the hapi proxies, models, and client methods go with them — swaps always go through /gateway/swap regardless of connector type. /gateway/clmm/pools takes a network parameter instead of hardcoding mainnet-beta (the last endpoint on the surface without one), and the swap repository serves side uppercase so legacy lowercase rows cannot leak into strict consumers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
…or codes The hardcoded ROUTER_CONNECTORS roster silently misrouted every connector Gateway added after it was written (a bare name fell through to /clmm and 404'd). Connector trading types now come from Gateway's own config/connectors listing, cached per client, preferring router then clmm then amm; an unknown name raises instead of guessing. Gateway's machine-readable error code (TRANSACTION_TIMEOUT, SLIPPAGE_EXCEEDED, ...) was flattened into prose before GatewayError was raised, leaving callers unable to tell retryable from terminal failures. It now rides GatewayError.code. Also: the swap execute response no longer lower-cases its status (every read surface reports uppercase), and swaps file under the base venue name so 'jupiter' and 'jupiter/router' land in one history bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
The one client verb with no path/payload test; also pins that an omitted slippage stays omitted and an explicit 0 is sent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166iQoxKce23GkUwuQJxdkr
Contributor
Author
|
Superseded — reopening against the current branch. The work continues in a new PR, linked from the same cross-repo set; condor #204 stays open and is referenced there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
API-side of the gateway#678 retry-ownership work, plus the CLMM pool-info plumbing that goes with it. Canonical design:
docs/retry-architecture.md(in the gateway PR), esp. the two-topology section: executors created via the API run in-process with no controller, so the API owns the "react to a stranded position" role.Orphaned-position lifecycle
already_terminatedwithclose_type,position_address,orphaned_position, andhold_reasoninstead of the 404 dead end #678 hit (terminal executors are popped from memory within one tick, so "not in memory" almost always means "already terminated"). 404 is reserved for ids the DB has never seen.POSITION_HOLDwithhold_reason— an LP close that exhausted its retries) or a legacyFAILED-with-position getsorphaned_position: truepersisted plus an error-level log. Voluntary holds never match (a successful close clearsposition_addressfirst).GET /executors/positions/orphaned: recovery candidates, SQL-filtered tolp_executor— involuntary holds,FAILED-with-position, andSYSTEM_CLEANUPrestarts (flaggedneeds_onchain_reconciliation).POST /executors/{id}/resolve-orphan: mark recovered after the position is closed externally; silences listings and agent warnings.bots/controllerslp_rebalancer mirror: halt + skip accounting for executors that ended with a live position.Closing an orphaned position
The lifecycle above could flag an orphan but not clear it — the close itself failed.
/gateway/clmm/closeand/collect-feesacceptpool_address. Both read a position's pool only fromgateway_clmm_positionsand404'd when absent. Anlp_executoropens its position straight from the bot to Gateway, so it is never in that table — on a live deployment the table was empty and every orphan404'd. Resolution is now database-first, then the request field, then a400(a bad request, not a missing position) namingpool_addressas the fix. Gateway's close needs onlyposition_address;pool_addressis used to snapshot pending fees before the close so they can be reported./executors/positions/orphanedreportslp_providerandpool_address. It previously gave a caller nothing to build a close from:connector_nameholds the network for anlp_executor(solana-mainnet-beta), not the DEX. The DEX (orca/clmm) and pool live in the executor's stored config and are now surfaced.CLMM pool-info:
bin_count, and Raydium consistencyGET /gateway/clmm/pool-infoacceptsbin_countand forwards it to Gateway's unifiedtrading/clmm/pool-info, making the per-tick liquidity distribution reachable for orca, raydium, uniswap and pancakeswap (Meteora always returns its own bins). The response model already carriedbins.api-v3.raydium.iodirectly, and reshaping that response to imitate Gateway's. That divergence cost real data — the transform hardcodedactive_bin_idtoNone,bin_stepto1, andbinsto[]— and it meant Raydium could not answerbin_countat all, since only Gateway computes the tick distribution. Raydium now takes the same path as every other CLMM connector; the Raydium API helpers and theiraiohttpimport go with it.Companion PRs
docs/retry-architecture.md(in the gateway PR)binCounton unified CLMM pool-infoPOSITION_HOLD, fresh position reads, bounded pending-tx pollingbin_countpassthrough, Raydium routed through Gatewaybin_countonget_pool_info(1.5.8)bin_countonget_pool_infoNote
routers/gateway_clmm.pyandmodels/gateway_trading.pyalso gain some unrelated line-wrapping and a dropped unused import: the flake8 pre-commit hook lints the whole file and would not otherwise accept a commit touching it.Validation
36 gateway-client contract tests pass. Validated live on mainnet with the companion branches deployed:
hold_reason, re-stop returnedalready_terminated, and resolve-orphan cleared it after a direct gateway close recovered all funds + rent.lp_providerandpool_addressextract from reallp_executorconfig rows, and thatgateway_clmm_positionsis empty for executor-opened positions (the condition that made every orphan close404). The endpoints themselves are not yet re-verified live — that needs a rebuilt API image.bin_count=5returns populated bins for orca and raydium through this endpoint;bin_count=0returns none. Raydium via Gateway now reports a realactive_bin_idwhere the bypass reportedNone.🤖 Generated with Claude Code
https://claude.ai/code/session_01Mt84XBEMVxbbyMG8fDxDKj