Skip to content

fix(linear): back off on rate limits - #2

Open
shaunr0b wants to merge 6 commits into
mainfrom
fix/linear-rate-limit-backoff
Open

fix(linear): back off on rate limits#2
shaunr0b wants to merge 6 commits into
mainfrom
fix/linear-rate-limit-backoff

Conversation

@shaunr0b

@shaunr0b shaunr0b commented May 29, 2026

Copy link
Copy Markdown

Summary

  • classify Linear RATELIMITED GraphQL responses separately from generic API failures
  • back off project polling using Linear retry/duration details
  • show tracker backoff separately from per-issue retry queue in the dashboard

Tests

  • TMPDIR=/tmp/symphony-test mix test test/symphony_elixir/workspace_and_config_test.exs test/symphony_elixir/orchestrator_status_test.exs

Summary by CodeRabbit

  • New Features

    • Linear API rate-limit handling with tracker backoff and adaptive poll scheduling.
    • Configurable required labels for issue filtering.
    • Retry mechanism now distinguishes capacity-wait (slot) retries from standard retries; UI and payloads reflect new kind/status fields.
    • Dashboard shows tracker backoff and improved retry/queue labeling.
    • Agent run preflight refusals and completions are surfaced to orchestration/status views.
  • Tests

    • Expanded tests for rate limiting, required labels, retry/capacity-wait, and preflight/refusal behavior.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 29, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3f288d38-f00c-43ce-ae2c-ec9f6bd5bca2

📥 Commits

Reviewing files that changed from the base of the PR and between e115121 and cb9fb03.

📒 Files selected for processing (3)
  • elixir/lib/symphony_elixir/agent_runner.ex
  • elixir/lib/symphony_elixir/orchestrator.ex
  • elixir/test/symphony_elixir/core_test.exs

📝 Walkthrough

Walkthrough

Adds tracker-required-labels and label-filtering in Linear client; classifies Linear GraphQL rate limits and computes orchestrator backoff; introduces slot-wait retry semantics with delay_type and snapshot visibility; updates dashboard/Presenter/LiveView and tests; small Codex handler for MCP elicitation requests.

Changes

Issue Filtering and Linear Rate Limit Resilience

Layer / File(s) Summary
Tracker required labels configuration
elixir/lib/symphony_elixir/config/schema.ex, elixir/test/support/test_support.exs, elixir/test/symphony_elixir/core_test.exs
Tracker schema adds required_labels (string list, default []); changeset accepts it; test helpers emit/read the workflow key and tests assert default/override behavior.
Linear Client label-based filtering
elixir/lib/symphony_elixir/linear/client.ex, elixir/test/symphony_elixir/workspace_and_config_test.exs
Fetch entry points compute required_label_filter() and thread label_filter through pagination and batch-by-id fetching; adds fetch_issue_states_by_ids_for_test/3 to exercise filtering; filter_by_required_labels/2 normalizes and filters issues by labels.
Linear rate limit detection
elixir/lib/symphony_elixir/linear/client.ex, elixir/test/symphony_elixir/workspace_and_config_test.exs, elixir/test/symphony_elixir/orchestrator_status_test.exs
GraphQL body classification detects RATELIMITED errors, extracts rate-limit details, and returns {:error, {:linear_rate_limited, details}}; decode_linear_response/3 surfaces rate-limit tuples; tests verify classification and propagation.

Orchestrator Retry Scheduling and Dashboard Rendering

Layer / File(s) Summary
Orchestrator poll scheduling and rate-limit backoff
elixir/lib/symphony_elixir/orchestrator.ex, elixir/test/symphony_elixir/orchestrator_status_test.exs
State adds next_poll_delay_ms and tracker_backoff; :run_poll_cycle uses pop_next_poll_delay/1; Linear rate-limit returns set next poll delay and store tracker_backoff; linear_rate_limit_backoff_ms_for_test/2 computes bounded delay.
Slot-wait retry handling with delay_type
elixir/lib/symphony_elixir/orchestrator.ex, elixir/test/symphony_elixir/core_test.exs
schedule_issue_retry/4 persists delay_type; slot-wait retries use dedicated delay constant and info logging; orchestrator reschedules retries with :slot_wait when capacity exhausted; retry_delay/2 selects delay by delay_type; snapshot exposes delay_type.
Dashboard and web presentation
elixir/lib/symphony_elixir/status_dashboard.ex, elixir/lib/symphony_elixir_web/presenter.ex, elixir/lib/symphony_elixir_web/live/dashboard_live.ex, elixir/test/symphony_elixir/extensions_test.exs, elixir/test/symphony_elixir/orchestrator_status_test.exs
StatusDashboard inserts a tracker_backoff line when present and formats capacity-wait retry rows specially; Presenter adds kind, maps "waiting_for_slot" status, and omits last_error for capacity waits; LiveView updates queue table columns and row helpers to show Type/Attempt/Due/Status.
MCP server elicitation request handler
elixir/lib/symphony_elixir/codex/app_server.ex
New private clause maybe_handle_approval_request/8 handles "mcpServer/elicitation/request" by delegating id/params into existing auto-answer/approval flow.
AgentRunner and blocked completions
elixir/lib/symphony_elixir/agent_runner.ex, elixir/test/symphony_elixir/core_test.exs
AgentRunner treats worker completion payloads as {:ok, completion} and forwards {:agent_run_completed, issue_id, completion}; workspace preflight exit code 64 is converted into a :workspace_preflight_refused completion; Codex turn terminal outcomes return structured completions for max-turns and issue-done.

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐰 I hopped through labels, rate limits, and queues,

Filtering issues by names we choose;
When Tracker sighs and Linear says "slow,"
Orchestrator waits — we soften the flow.
MCP asks kindly, the rabbit replies with a shoehorned prose.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The pull request provides a summary of changes and test commands but does not follow the required template structure with Context, TL;DR, Alternatives, and checkbox Test Plan sections. Restructure the description to match the template: add Context section explaining why the change is needed, expand TL;DR to a concise summary, add Alternatives section, and use the checkbox format for Test Plan.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: implementing backoff behavior when Linear API rate limits are encountered.
Docstring Coverage ✅ Passed Docstring coverage is 84.21% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/linear-rate-limit-backoff

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
elixir/lib/symphony_elixir/status_dashboard.ex (1)

315-321: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

tracker_backoff is dropped before render, so the new line never appears.

The snapshot shaping logic omits :tracker_backoff in both snapshot_with_samples/2 and snapshot_payload/0, so format_tracker_backoff_line/1 always sees nil.

Suggested fix
          {:ok,
           %{
             running: running,
             retrying: retrying,
             codex_totals: codex_totals,
             rate_limits: Map.get(snapshot, :rate_limits),
-            polling: Map.get(snapshot, :polling)
+            polling: Map.get(snapshot, :polling),
+            tracker_backoff: Map.get(snapshot, :tracker_backoff)
           }},
@@
          {:ok,
           %{
             running: running,
             retrying: retrying,
             codex_totals: codex_totals,
             rate_limits: Map.get(snapshot, :rate_limits),
-            polling: Map.get(snapshot, :polling)
+            polling: Map.get(snapshot, :polling),
+            tracker_backoff: Map.get(snapshot, :tracker_backoff)
           }}

Also applies to: 562-568

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@elixir/lib/symphony_elixir/status_dashboard.ex` around lines 315 - 321, The
snapshot shaping code drops :tracker_backoff so format_tracker_backoff_line/1
always gets nil; update the snapshot builders (snapshot_with_samples/2 and
snapshot_payload/0) to include the tracker_backoff value (e.g. add
tracker_backoff: Map.get(snapshot, :tracker_backoff) or include the local
tracker_backoff variable in the returned maps) so format_tracker_backoff_line/1
receives the real value; apply the same addition in the other snapshot
construction site referenced around the second occurrence so both render paths
include :tracker_backoff.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@elixir/lib/symphony_elixir/linear/client.ex`:
- Line 143: fetch_issues_by_states/1 currently calls do_fetch_by_states(...,
nil, nil) which bypasses tracker.required_labels; change the call in
fetch_issues_by_states/1 so the label_filter argument is set to the tracker's
required_labels (or the result of a helper that builds the proper label filter)
instead of nil, i.e. pass tracker.required_labels (or
build_label_filter(tracker)) into do_fetch_by_states/4 so required labels are
enforced for this public path.

In `@elixir/lib/symphony_elixir/orchestrator.ex`:
- Around line 257-261: When Tracker.fetch_candidate_issues() succeeds we should
clear stale tracker_backoff immediately instead of only inside choose_issues/2;
update the with-chain so that after the {:ok, issues} <-
Tracker.fetch_candidate_issues() match you bind a new_state = %{state |
tracker_backoff: nil} (or otherwise set tracker_backoff: nil) and use new_state
for the subsequent available_slots(new_state) check and when calling
choose_issues(issues, new_state); apply the same change to the other occurrence
around lines 315-317 so successful fetches always reset tracker_backoff even if
available_slots/1 is false.

In `@elixir/test/symphony_elixir/core_test.exs`:
- Around line 705-710: The test spawns occupying_worker but may leak it if an
assertion fails; ensure the spawned process is always cleaned up by moving its
creation/cleanup into an on_exit callback (or wrap in try ... after) so you
always send :stop and/or monitor/kill the process; update the block that creates
occupying_worker (and the similar block around lines 751-753) to register
cleanup in on_exit/after referencing the occupying_worker variable so the
process is terminated regardless of test failures.

---

Outside diff comments:
In `@elixir/lib/symphony_elixir/status_dashboard.ex`:
- Around line 315-321: The snapshot shaping code drops :tracker_backoff so
format_tracker_backoff_line/1 always gets nil; update the snapshot builders
(snapshot_with_samples/2 and snapshot_payload/0) to include the tracker_backoff
value (e.g. add tracker_backoff: Map.get(snapshot, :tracker_backoff) or include
the local tracker_backoff variable in the returned maps) so
format_tracker_backoff_line/1 receives the real value; apply the same addition
in the other snapshot construction site referenced around the second occurrence
so both render paths include :tracker_backoff.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d2ef0645-264c-47d3-80dc-0319bded63df

📥 Commits

Reviewing files that changed from the base of the PR and between 2c18518 and e115121.

📒 Files selected for processing (12)
  • elixir/lib/symphony_elixir/codex/app_server.ex
  • elixir/lib/symphony_elixir/config/schema.ex
  • elixir/lib/symphony_elixir/linear/client.ex
  • elixir/lib/symphony_elixir/orchestrator.ex
  • elixir/lib/symphony_elixir/status_dashboard.ex
  • elixir/lib/symphony_elixir_web/live/dashboard_live.ex
  • elixir/lib/symphony_elixir_web/presenter.ex
  • elixir/test/support/test_support.exs
  • elixir/test/symphony_elixir/core_test.exs
  • elixir/test/symphony_elixir/extensions_test.exs
  • elixir/test/symphony_elixir/orchestrator_status_test.exs
  • elixir/test/symphony_elixir/workspace_and_config_test.exs


true ->
do_fetch_by_states(project_slug, normalized_states, nil)
do_fetch_by_states(project_slug, normalized_states, nil, nil)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

fetch_issues_by_states/1 is bypassing tracker.required_labels.

Line 143 passes nil for label_filter, so this public fetch path can return issues that other paths would correctly filter out.

Proposed fix
-          do_fetch_by_states(project_slug, normalized_states, nil, nil)
+          do_fetch_by_states(project_slug, normalized_states, nil, required_label_filter())
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
do_fetch_by_states(project_slug, normalized_states, nil, nil)
do_fetch_by_states(project_slug, normalized_states, nil, required_label_filter())
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@elixir/lib/symphony_elixir/linear/client.ex` at line 143,
fetch_issues_by_states/1 currently calls do_fetch_by_states(..., nil, nil) which
bypasses tracker.required_labels; change the call in fetch_issues_by_states/1 so
the label_filter argument is set to the tracker's required_labels (or the result
of a helper that builds the proper label filter) instead of nil, i.e. pass
tracker.required_labels (or build_label_filter(tracker)) into
do_fetch_by_states/4 so required labels are enforced for this public path.

Comment on lines 257 to 261
with :ok <- Config.validate!(),
{:ok, issues} <- Tracker.fetch_candidate_issues(),
true <- available_slots(state) > 0 do
choose_issues(issues, state)
choose_issues(issues, %{state | tracker_backoff: nil})
else

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Clear stale tracker_backoff after successful tracker fetches.

tracker_backoff is only reset when choose_issues/2 runs. If fetch succeeds but available_slots(state) > 0 is false, stale backoff state persists.

Suggested fix
-    with :ok <- Config.validate!(),
-         {:ok, issues} <- Tracker.fetch_candidate_issues(),
-         true <- available_slots(state) > 0 do
-      choose_issues(issues, %{state | tracker_backoff: nil})
+    with :ok <- Config.validate!(),
+         {:ok, issues} <- Tracker.fetch_candidate_issues() do
+      state = %{state | tracker_backoff: nil}
+
+      if available_slots(state) > 0 do
+        choose_issues(issues, state)
+      else
+        state
+      end
     else
@@
-      false ->
-        state
     end

Also applies to: 315-317

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@elixir/lib/symphony_elixir/orchestrator.ex` around lines 257 - 261, When
Tracker.fetch_candidate_issues() succeeds we should clear stale tracker_backoff
immediately instead of only inside choose_issues/2; update the with-chain so
that after the {:ok, issues} <- Tracker.fetch_candidate_issues() match you bind
a new_state = %{state | tracker_backoff: nil} (or otherwise set tracker_backoff:
nil) and use new_state for the subsequent available_slots(new_state) check and
when calling choose_issues(issues, new_state); apply the same change to the
other occurrence around lines 315-317 so successful fetches always reset
tracker_backoff even if available_slots/1 is false.

Comment thread elixir/test/symphony_elixir/core_test.exs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant