intersect-proxy

A generic, configurable resilience proxy for INTERSECT microservices. The proxy sits in front of a target service — transparently adopting that service's INTERSECT identity so upstream callers need no changes — and adds retry, communication timeouts, and active/standby failover on top of the SDK's messaging.

It contains no domain-specific logic: what to route where, how aggressively to retry, and how long to wait are all configuration; the only place workflow knowledge enters is a small set of overridable hooks.

The proxy is currently written for an older version of intersect as it was originally written for a specific service demo. I am in the process of testing it against the latest intersect version and should be updating the repo soon.

AI usage

I've been recently playing with claude some. I've used it in this repo for things like helping with the readme, code comments, CI, and tests.

Why

The INTERSECT SDK gives you connection-level reconnect and a per-call timeout that silently drops a request when it expires — it never tells the caller. It has no retry of failed calls, no fallback between services, and no health-based routing. This proxy fills that gap as a reusable component, following the resilience patterns in the INTERSECT architecture (Monitoring, Retry, Active/Standby).

Model

Requests are handled in two phases, matching an asynchronous compute workflow where the reply to a request is an acknowledgement, not the result:

• Ack phase (synchronous to the caller): forward the request, wait up to ack_timeout for an acknowledgement, retry the same target per the retry policy, then fail over to the next target.
• Result phase (asynchronous): after a good ack, arm a result_timeout watchdog; if the result streams back in time it is relayed upstream, otherwise the job is failed over to a standby and re-driven. Streaming results (many messages per job) renew the watchdog so a mid-stream stall also fails over.

Detection runs on two independent paths:

• Synchronous — a call that times out or errors during a request flips its target unhealthy at once.
• Always-on active health-check (the paper's watchdog) — a background loop pings every target's status operation every heartbeat_interval, marks it unhealthy after missed_intervals missed pings, and revives it when pings resume — so a service going down is detected even while idle.

caller ──▶ [ Resilience Proxy ] ──primary──▶ target A   (retry, then…)
                     │            ╲─standby──▶ target B   (…fail over)
   results ◀─────────┘  (relayed upstream when they stream back)

Usage

Subclass ResilienceProxyCapability, set the fronted service's capability name, and declare @intersect_message methods that delegate to the engine:

from intersect_sdk import intersect_message, intersect_status
from intersect_proxy import ResilienceProxyCapability, ProxyConfig

class MyProxy(ResilienceProxyCapability):
    intersect_sdk_capability_name = 'TargetCapability'   # identity being fronted

    @intersect_status()
    def status(self) -> str:
        return 'Up'

    @intersect_message()
    def run_compute_job(self, payload: str) -> str:
        return self._proxy_request('run_compute_job', payload)   # ack phase + result watchdog

Configure it with YAML (see examples/):

capability_name: TargetCapability
strategy: active_standby
strategy_group_order: [primary, standby]
retry:    { count: 1, delay: 0.5, backoff: 2.0 }
timeouts: { ack_timeout: 5.0, result_timeout: 60.0, heartbeat_interval: 30.0, missed_intervals: 3 }
targets:
  - { name: primary-a, group: primary, destination: org.fac.sys.sub.primary,
      operation_map: { run_compute_job: TargetCapability.run_compute_job } }
  - { name: standby-b, group: standby, destination: org.fac.sys.sub.standby,
      operation_map: { run_compute_job: TargetCapability.run_compute_job } }

Examples

examples/ is a generic, broker-backed demo (run against python-sdk/docker-compose.yml): two toy Worker instances behind the proxy, driven through run.sh, which walks a client through the happy path, an injected error (retry then failover), recovery/failback once the primary answers again, an injected hang (ack-timeout failover), and a hard outage (killing the primary process). See examples/README.md for details.

Fault injection (demos)

maybe_inject(target) in a demo service reads a live fault mode from a control file, flipped with:

export PROXY_DEMO_FAULT_DIR=/tmp/proxy-faults
intersect-proxy inject primary-a hang     # next call hangs -> ack timeout -> failover
intersect-proxy inject primary-a error    # next call errors -> retry -> failover
intersect-proxy inject primary-a none     # recover

It is inert unless PROXY_DEMO_FAULT_DIR is set.

Extension points (designed-in, not yet built)

• Load balancing / scaling — add a RoundRobin/weighted RoutingStrategy (the seam and a reference RoundRobin already exist) to spread load across many instances in a group.
• Registration-driven discovery — the TargetRegistry has an add/remove API a future register endpoint can call so downstream services register themselves at runtime.
• Invisible proxy — with registration in place, enabling the proxy makes services register with it directly and clients route through it unknowingly (also needs python-sdk support).

Development

pip install -e ".[dev]"
pytest tests/unit          # broker-free unit tests (fake transport + fake timers)
ruff check . && mypy src/intersect_proxy