[Bug] codex-native event consumers remain blocked after app-server disconnect

[PattaraS]

Description

Follow-up from reviewing #7508: the codex-native app-server event iterator still waits after its WebSocket reader exits. Native forwarding or thread discovery can remain blocked waiting for events that can no longer arrive, until an outer timeout or session teardown intervenes.

Current main includes #7355, which fixes pending response-bearing RPCs (closed issue #7352). Those RPCs now fail promptly. The remaining gap is CodexAppServerClient.iter_events(): it waits on _events.get(), while _reader_loop() exits without notifying that queue. The iterator's documented “until the connection closes” behavior therefore does not hold.

The production consumers include the native forwarder's event loop and wait_for_thread_started(). In particular, the latter's timeout=None path cannot reach its “event stream ended before thread startup” error after disconnect. This report establishes the client disconnect boundary; it does not establish a universal user-visible timeout or field frequency.

Steps to reproduce

Using the production client and a real loopback WebSocket server:

1. Complete the initialize handshake.
2. Start an event consumer and a turn/start RPC.
3. Close the server connection without replying, with close code 1000 (normal) or 1011 (error).
4. Wait for the reader to finish, then check both callers.

Observed on current main for both close codes:

{"close_code": 1000, "reader_finished": true, "rpc_error": "ConnectionError", "event_consumer_still_waiting": true}
{"close_code": 1011, "reader_finished": true, "rpc_error": "ConnectionError", "event_consumer_still_waiting": true}

Expected: event consumers also wake and terminate or receive a disconnect error promptly, preserving notifications already queued before the disconnect. A caller should not need to discover the dead reader and cancel the consumer itself.

Self-contained reproduction (no credentials or Codex binary required)

After installing the checkout's frozen development dependencies, save this as /tmp/repro_native_event_disconnect.py:

import asyncio
import contextlib
import json

from websockets.asyncio.server import serve

from omnigent.harnesses.codex_native.app_server import CodexAppServerClient


async def check(code):
    async def handler(ws):
        async for raw in ws:
            message = json.loads(raw)
            if message.get('method') == 'initialize':
                await ws.send(json.dumps({'id': message['id'], 'result': {}}))
            elif message.get('method') == 'turn/start':
                await ws.close(code=code, reason='local disconnect probe')
                return

    async with serve(handler, '127.0.0.1', 0) as server:
        port = server.sockets[0].getsockname()[1]
        client = CodexAppServerClient(ws_url=f'ws://127.0.0.1:{port}')
        await client.connect()
        events = client.iter_events()
        waiting_event = asyncio.create_task(anext(events))
        rpc = asyncio.create_task(client.request('turn/start', {'threadId': 'probe'}))
        reader = client._reader_task
        try:
            with contextlib.suppress(Exception):
                await asyncio.wait_for(asyncio.shield(reader), timeout=3)
            assert reader.done()
            done, _ = await asyncio.wait([rpc, waiting_event], timeout=0.3)
            result = {
                'close_code': code,
                'reader_finished': reader.done(),
                'rpc_error': type(rpc.exception()).__name__ if rpc.done() else None,
                'event_consumer_still_waiting': not waiting_event.done(),
            }
            print(json.dumps(result), flush=True)
            assert rpc in done and isinstance(rpc.exception(), ConnectionError)
            assert waiting_event not in done, 'event hang no longer reproduces'
        finally:
            rpc.cancel()
            waiting_event.cancel()
            await asyncio.gather(rpc, waiting_event, return_exceptions=True)
            await client.close()
            await events.aclose()


async def main():
    for code in (1000, 1011):
        await check(code)


asyncio.run(main())

Run from the repository root:

OMNIGENT_DISABLE_KEYRING=1 PYTHONPATH="$PWD/sdks/python-client:$PWD/sdks/ui:$PWD" \
  uv run --no-sync python /tmp/repro_native_event_disconnect.py

The assertions confirm the current defect; a successful run is reproduction evidence, not proof of a fix. The script cleans up its own tasks and connections.

Version

Main e047da5f734e0db77a6d4c32abb3dce433a2e216, using frozen dependencies. The six existing tests/test_codex_native_app_server_rpc.py tests also pass on this revision, confirming the RPC fix is retained while the event-consumer gap remains.

OS / Platform or device

macOS; local Python process and real loopback WebSockets. No live model provider, native Codex CLI, browser session, or Windows/Linux run was used for this reproduction.

Harness / Harness mode

Codex, Native (codex-native). #7508 addresses the separate request-driven codex harness.

Observed impact

Sessions whose app-server disconnects while a native event consumer is waiting. Frequency in production is unknown. Forwarding or thread discovery may remain pending rather than promptly recognizing the disconnected app-server.

Authentication type

Not authentication-related; the reproduction uses no credentials.

Fix scope and validation

• Wake event consumers on normal disconnect, reader failure, and explicit close; preserve already-buffered notifications and use a fresh event queue after reconnect.
• Exercise the production thread-discovery and forwarding consumers, including cancellation and cleanup, so stream termination results in the intended failure/teardown behavior.
• Retain fix(codex): release pending RPCs on reader disconnect #7355's RPC response-before-close, disconnect, cancellation, and failed-send behavior.
• Evaluate native reconnect/recovery separately; do not automatically replay messages or tool work merely because the transport closed.

Related work

• fix(codex): recover promptly after app-server stdout closes #7508 and [Bug] codex harness waits for the watchdog after app-server stdout closes #7510 cover stdout disconnect handling for the request-driven codex harness.
• Merged fix(codex): release pending RPCs on reader disconnect #7355 / closed Native Codex RPCs remain pending after app-server disconnection #7352 cover native RPCs and are already present in the tested main revision.
• Closed, unmerged fix(codex-native): recover late thread startup and stop on disconnect #6833 / closed Slow native Codex thread startup permanently disables later chat turns #7341 discussed event-stream termination alongside late thread startup. That implementation may be reusable, but the event-iterator defect still reproduces on the revision above. This ticket tracks the remaining disconnect behavior specifically.

[github-actions]

🤖 Automated triage

• Bot assessment: Medium impact
• Priority: P2-medium
• Evidence: a controlled test
• Why: A confirmed Codex-native disconnect bug can leave event forwarding or thread discovery pending and delay session progress, but the affected scenario is scoped and callers can mitigate it through cancellation or teardown.

This automated assessment uses the issue content and repository signals. Maintainers can override the priority label.

[github-actions]

Thanks for reporting this. #7341, #7352, #7510 may be related — could you take a look in case they already cover this?

If it turns out to be the same problem, please close this one and add your details there. Otherwise leave a note and we'll pick it up here.