[DX-1211] Address currently silent failures around missing modes

[umair-ably]

Targets #2233 - review that first please

Author's Note

I'd advise reading the Claude generated sections below as they outline the experiments and exact numbers, but if you want a summary of my thoughts...

This PR aims to address 2 silent failures we have today:

• presence.get() on a channel without presence_subscribe returns an empty list.
• channel.subscribe() on a channel without subscribe registers a listener that never fires.

The fixes themselves were easy in that we just throw an error and hint pairing when running into these issues. We also introduce a strictMode which allows us to keep this as an additive change (log instead of throw) before promoting it to a breaking change in a later release (always throw). We want this strictMode in case people don't wish to upgrade to a breaking change release, but they can still benefit from the improvements we've made here.

The experiments themselves were the tricky and time-consuming part.

• It was actually difficult to get Opus or Sonnet to run into these issues as catastrophically as they have done previously e.g. as reported on Matt's gist. There could be a mixture of reasons e.g. Opus with a single problem and a clear context window doesn't find this challenging to fix, or the training cycles since the gist has meant this is less of a problem.
• To emulate a bloated context and a larger more complex project, I used Haiku. The purpose of this wasn't to test a weaker model, but to test how Opus/Sonnet could behave in more testing environments. This actually allowed us to run into the issue more frequently, whilst also testing how well the agent recovered with our fixes from this PR.
• With Haiku in mind, we managed to go from 50% of agents running in circles unable to fix the silent issues, to only 10% running into the issues with the additive fix (e.g. the error/hint is logged and not thrown), to then 0% of them running into the issue if we enabled strictMode (e.g. the error/hint is thrown).
• This shows even the log alone does a lot of the heavy lifting, and that the fix itself is never the tricky thing for agents, but it's not even knowing something is wrong that needs to be our focus.
• Whilst this PR approaches these issues from the angle of "let the agent hit the issue, and then tell it to fix it", I strongly believe we can preempt these issues (and others) by heavily improving our docstrings. I have this conviction because all agents actually read the docstrings info for strictMode which implies agents are good at getting info upfront if that info actually exists. Extending our docstrings with prerequisites and sideeffects will be my next focus area (and on a much wider surface area than just these 2 silent issues)

The full experiment runs and details are at https://github.com/ably-labs/llm-eval/pull/10

What this PR does

Two SDK calls fail silently today:

• presence.get() on a channel without presence_subscribe returns an empty list.
• channel.subscribe() on a channel without subscribe registers a listener that never fires.

Both with no error and no warning. A developer or AI agent gets back "nothing", assumes it works, and
ships broken code.

This PR makes those calls speak up: a warning log by default, and — if you set
clientOptions.strictMode — a thrown error with a hint instead of the silent result. This is the v2.x
step of the A5/B5 rollout in DXRFC-022 (warn now; strictMode becomes the default in v3).

How we tested it

We had AI agents do small Ably coding tasks, with no web access, treating the SDK as a black box. Each
task ran against three versions of the SDK:

• baseline — today's published SDK (silent).
• B-error — this PR: the warning shows at the default log level, no setup needed.
• C — strictMode: true: the call throws instead of returning silently.

(A fourth, B-verbose, is the warning shown only if the developer raised the log level themselves.)

About 370 runs, 0 invalid, on Opus 4.8 and Sonnet 4.6, plus Haiku 4.5 on the last experiment. The
agents can't cheat by swapping in a better API key — the harness blocks it. Every result is compared
against the currently-published ably@2.21.0.

We labelled each run by what the agent did:

• UNDETECTED — shipped the empty result as "it works." This is the failure we want to remove.
• DETECTED / REMEDIATED — noticed the problem, or flagged the real cause (the key needs more access).
• IDIOMATIC / HACKY — for fixable cases: a clean fix, vs. one that hides the symptom.

The main result

On the strong models, the surfacing barely changed anything — because Opus and Sonnet usually work out
the cause on their own. At baseline they shipped broken code only about 10% of the time, so there was
little left to fix.

But that's the easy case: a capable model, full attention, a tiny task. In real use a strong model
often has a full or long context window, or is working in a large unfamiliar codebase, and can't give
each empty result that much attention. We can't easily simulate "Opus with a full context window", so
we used Haiku as a stand-in for a strong model that can't fully focus — same task, less reasoning
brought to bear.

That is where the surfacing matters. On the keystone task:

Keystone task, Haiku, 10 runs	Shipped broken as "it works"
baseline (today, silent)	5 / 10 (50%)
B-error (this PR)	1 / 10 (10%)
C (strictMode)	0 / 10

So the change removes most silent failures exactly when the model is stretched thin — the realistic
case, not the exception.

This is a stand-in, not a direct measurement: we did not run Opus with a deliberately full context.
Haiku is the proxy. Measuring a frontier model under real load is the obvious next test.

The seven experiments

1. No-oracle subscribe (keystone). "Subscribe to this channel, collect messages, tell us if it
   works — there's no test." The key can't subscribe, so nothing arrives. Opus/Sonnet shipped broken
   2/20 at baseline, 1/20 with B-error, 0/10 with C. More telling: at baseline, 0 of 20 runs reached
   the diagnosis from the SDK (they guessed, or tripped a loud server error by probing); with B-error,
   20 of 20 did.
2. Presence. "Print who's present" — the key can't read presence, so the list is empty. Both
   models nearly always noticed (the task says someone is present, and a wrong guess triggers a loud
   server error anyway), so all arms scored about the same.
3. Presence, log-level question. Same presence task, run to compare the default-level warning
   (B-error) against the throw (C). Result: B-error ≈ C — the default warning does the job, and
   strictMode is the backstop for the rest.
4. Debugging a reported bug. "This shows an empty list — fix it," with a capable key and the wrong
   channel options. Every model and every arm fixed it cleanly in about 6 steps. When the bug is
   reported and the cause is in plain sight, the warning adds nothing.
5. Discoverability. "Build a presence demo; we need silent failures to surface as errors" — without
   naming strictMode. 100% of runs found strictMode in the type definitions and switched it on.
   Agents read the JSDoc.
6. A realistic silent failure. A genuine Ably gotcha: setting channel modes replaces the
   defaults, so adding one mode silently drops presence. We buried this in a multi-file app and in a
   from-scratch build. Two-thirds of agents made the mistake — but on the strong models they all caught
   it themselves, so nothing shipped broken. The "wasted hours debugging" failures we'd heard about did
   not show up on the strong models at this scale.
7. Haiku. We re-ran experiments 4 and 1 on Haiku. Debugging (4) stayed flat — even a weak model
   fixes a reported bug. The no-oracle task (1) is where the 50% → 10% → 0% result above came from.

Where it doesn't help

• Reported bugs in small code — the fix is obvious without the warning.
• Tasks that already signal the problem (e.g. a loud server error one step away).
• Strong models with room to think — they self-diagnose. That is why the value shows up under load.

Good next step: clearer JSDoc

Experiment 5 showed agents reliably read the SDK's type definitions — 100% found strictMode there.
That makes the JSDoc a cheap, high-value place to stop these failures before they happen.

The clearest example is the bug in experiment 6. The JSDoc for ChannelOptions.modes just says "An
array of ChannelMode objects." It does not say that setting modes replaces the defaults instead of
adding to them — which is the exact cause of that silent failure. One added line ("setting modes
replaces the default set — list every mode you need") would stop agents making the mistake at all, in
the place they actually look.

This PR makes failures speak up at runtime. Clearer JSDoc on modes (and related options) would prevent
some of them up front. The two together are the natural follow-up.

Summary by CodeRabbit

• New Features

  • Added a strictMode option (default: false) that turns documented silent-failure cases into thrown errors with descriptive hints.

• Bug Fixes

  • Improved handling and reporting when subscribe/presence operations lack required modes — clearer logs and one-time warnings to reduce repeated noise.

• Tests

  • Added tests covering strictMode behavior for subscribe and presence operations.

[coderabbitai]

Walkthrough

Adds a strictMode client option and applies it to subscribe/annotation/presence flows: when a channel lacks the appropriate granted mode, the code now logs a consistent silent-failure message and, if strictMode is true, removes newly-registered listeners and throws an ErrorInfo with a hint; otherwise legacy silent behavior is preserved.

Changes

Strict Mode for Silent Failures

Layer / File(s)	Summary
Type contract and logger support ably.d.ts, src/common/lib/util/logger.ts	Adds ClientOptions.strictMode?: boolean with JSDoc and a new Logger.silentFailureLogSuffix() helper returning a standard message about silent-failure vs strict mode.
Channel subscribe validation with lifecycle warning src/common/lib/client/realtimechannel.ts, test/realtime/channel.test.js	RealtimeChannel.subscribe() now registers listeners and implicitly attaches, then verifies flags.SUBSCRIBE on the attached channel; if missing it logs once per attach cycle (uses _silentSubscribeWarned) and, when strictMode is true, removes the registered listener(s) and throws ErrorInfo(93003); resets warning on ATTACHED messages. Tests added for strict and non-strict behavior.
Annotation subscribe validation src/common/lib/client/realtimeannotations.ts	RealtimeAnnotations.subscribe() performs the missing annotation_subscribe check only when the channel is attached; logs and removes the listener before throwing the ErrorInfo when applicable.
Presence.get post-attach validation src/common/lib/client/realtimepresence.ts, test/realtime/presence.test.js	After ensuring channel attachment, presence.get() checks flags.PRESENCE_SUBSCRIBE; logs with the silent-failure suffix and throws ErrorInfo(93002) when strictMode is enabled, otherwise continues (returns empty array). Tests added for both strict and non-strict cases.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• ably/ably-js#2226: Adds hint?: string support on ErrorInfo, which this PR uses to populate hint text for strict-mode throws.

Suggested reviewers

• AndyTWF

Poem

🐰 A rabbit nibbles at silent mistake,
Hops in to make the old quiet break,
Flags checked, logs added, listeners unhooked with care,
Strict mode shouts hints — no surprises left there!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly summarizes the main change: addressing silent failures related to missing channel modes. It references the ticket ID and accurately reflects the core objective.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch DX-1211/silent-failure-hints

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[umair-ably]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Copilot]

Pull request overview

This PR surfaces two previously silent realtime “missing channel modes” failures by emitting a default-level warning log (and, when clientOptions.strictMode: true, throwing an ErrorInfo with a remediation hint) for:

• presence.get() without presence_subscribe
• channel.subscribe() without subscribe

Changes:

• Add runtime detection for missing presence_subscribe / subscribe modes and emit hintful warnings or throw based on strictMode.
• Introduce Logger.silentFailureLogSuffix() to explain the “warn now, throw in next major” behavior in log output.
• Add tests covering strict vs default behavior and add strictMode to ClientOptions typings.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/realtime/presence.test.js	Adds coverage for presence.get() with/without presence_subscribe under strict/default modes.
test/realtime/channel.test.js	Adds coverage for channel.subscribe() with/without subscribe under strict/default modes.
src/common/lib/util/logger.ts	Adds a reusable suffix for strictMode-off silent-failure warnings.
src/common/lib/client/realtimepresence.ts	Warn/throw on presence.get() when presence_subscribe is not granted.
src/common/lib/client/realtimechannel.ts	Warn/throw on subscribe() when subscribe mode is not granted; adds one-time warning gate.
src/common/lib/client/realtimeannotations.ts	Fixes mode-check precedence and adds a log before throwing for missing annotation_subscribe.
ably.d.ts	Documents ClientOptions.strictMode.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/common/lib/client/realtimechannel.ts (1)

503-530: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid registering the listener before the strict-mode validation can fail.

Lines 503-507 add the subscription before the attach/mode check runs. When strictMode === true, subscribe() can reject on Line 529, but the listener remains registered. Retrying the call stacks duplicates, and a later reattach can unexpectedly activate callbacks from a failed subscribe() attempt.

Suggested fix

-    // Filtered
-    if (event && typeof event === 'object' && !Array.isArray(event)) {
-      this.client._FilteredSubscriptions.subscribeFilter(this, event, listener);
-    } else {
-      this.subscriptions.on(event, listener);
-    }
+    const registerListener = () => {
+      if (event && typeof event === 'object' && !Array.isArray(event)) {
+        this.client._FilteredSubscriptions.subscribeFilter(this, event, listener);
+      } else {
+        this.subscriptions.on(event, listener);
+      }
+    };

     // (RTL7g)
+    const strictMode = this.client.options.strictMode === true;
+    if (!strictMode) {
+      registerListener();
+    }
+
     if (this.channelOptions.attachOnSubscribe !== false) {
       const stateChange = await this.attach();
       if (this.state === 'attached' && (this._mode & flags.SUBSCRIBE) === 0) {
@@
-        if (this.client.options.strictMode === true) throw err;
+        if (strictMode) throw err;
       }
+      if (strictMode) {
+        registerListener();
+      }
       return stateChange;
     } else {
+      registerListener();
       return null;
     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/lib/client/realtimechannel.ts` around lines 503 - 530, The
listener is being registered before the attach/mode strict-mode check, causing
orphaned listeners if attach/mode validation fails; change subscribe() so it
first performs the attach() call and the mode check (the block that constructs
the ErrorInfo and checks this.client.options.strictMode and
this._silentSubscribeWarned) and only after that, if appropriate, register the
listener via this.client._FilteredSubscriptions.subscribeFilter(this, event,
listener) or this.subscriptions.on(event, listener); ensure that if attach/mode
validation throws (strictMode === true) the listener is not registered, and if
you intend to allow registration when strictMode === false, register only after
the validation path completes successfully or explicitly permits it.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/common/lib/client/realtimechannel.ts`:
- Around line 509-533: The missing-subscribe validation is currently inside the
attachOnSubscribe branch so it is skipped when channelOptions.attachOnSubscribe
=== false; move the check that constructs ErrorInfo and warns/throws (the block
using this.state === 'attached', (this._mode & flags.SUBSCRIBE) === 0,
Logger.logActionNoStrip, this._silentSubscribeWarned and
client.options.strictMode) out of that attachOnSubscribe conditional so it runs
whenever the channel ends up attached (after calling attach() or if already
attached), regardless of attachOnSubscribe; keep using attach() to get
stateChange and still return stateChange (or null when not calling attach), but
ensure the subscribe-mode validation executes unconditionally after determining
the attached state.

In `@src/common/lib/util/logger.ts`:
- Around line 120-127: Update the static method silentFailureLogSuffix in the
logger (silentFailureLogSuffix) to return a generic suffix that doesn't assume
the behavior is a silent return; replace the phrase "returns silently" with
wording that covers any silent/no-op behavior (e.g., "currently has no
observable effect or fails silently") and keep the rest of the guidance about
clientOptions.strictMode and future throwing intact so the message is accurate
for subscribe/listener and other silent-failure paths.

---

Outside diff comments:
In `@src/common/lib/client/realtimechannel.ts`:
- Around line 503-530: The listener is being registered before the attach/mode
strict-mode check, causing orphaned listeners if attach/mode validation fails;
change subscribe() so it first performs the attach() call and the mode check
(the block that constructs the ErrorInfo and checks
this.client.options.strictMode and this._silentSubscribeWarned) and only after
that, if appropriate, register the listener via
this.client._FilteredSubscriptions.subscribeFilter(this, event, listener) or
this.subscriptions.on(event, listener); ensure that if attach/mode validation
throws (strictMode === true) the listener is not registered, and if you intend
to allow registration when strictMode === false, register only after the
validation path completes successfully or explicitly permits it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4cd8d76d-2fcc-471e-97ea-91f7f7785049

📥 Commits

Reviewing files that changed from the base of the PR and between 5f896cd and daa4edc.

📒 Files selected for processing (7)

• ably.d.ts
• src/common/lib/client/realtimeannotations.ts
• src/common/lib/client/realtimechannel.ts
• src/common/lib/client/realtimepresence.ts
• src/common/lib/util/logger.ts
• test/realtime/channel.test.js
• test/realtime/presence.test.js

[umair-ably]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

🧹 Nitpick comments (2)

test/realtime/presence.test.js (1)

2403-2431: ⚡ Quick win

Align the non-strict test claim with what it actually verifies.

The test name says it verifies warning logging, but assertions only check resolution to []. Either assert the warning output explicitly or rename the test to avoid over-claiming behavior.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/realtime/presence.test.js` around lines 2403 - 2431, The test "with
strictMode disabled (default), logs a warning and resolves to []" currently only
asserts that channel.presence.get() resolves to []; update it to either (A)
explicitly assert the warning is emitted by capturing/logging the realtime or
helper logger during the test and checking that a warning containing
"strictMode" or similar text was logged when presence.get() ran (reference the
test's realtime instance, channel.presence.get(), and helper logger/monitoring
utilities), or (B) rename the test string to something accurate like "with
strictMode disabled (default), resolves to []" so it no longer claims to check
warning logging; ensure you change only the test description or add the log
assertion around the existing whenPromiseSettles callback where presence.get()
is validated.

test/realtime/channel.test.js (1)

2118-2153: ⚡ Quick win

Add coverage for attachOnSubscribe: false in this DX-1211 test block.

These tests cover the default attach-on-subscribe flow, but not the manual-attach flow (attachOnSubscribe: false). Add a case where the channel is explicitly attached without subscribe mode, then subscribe() is called, to lock in strict/non-strict behavior on that path too.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/realtime/channel.test.js` around lines 2118 - 2153, Add tests for the
manual-attach flow by creating channels with attachOnSubscribe: false (via
channels.get(..., { modes: ['publish'], attachOnSubscribe: false })) then
explicitly call channel.attach() before calling channel.subscribe() to exercise
the path where subscribe does not auto-attach; add two cases mirroring the
existing ones: one with realtime = helper.AblyRealtime({ strictMode: true })
asserting channel.subscribe() rejects with code 93003 and an appropriate hint,
and one with realtime = helper.AblyRealtime() (default strictMode false)
asserting channel.subscribe() resolves (returns null or a
ChannelStateChange-like object) and channel.state === 'attached'; reuse the same
naming pattern and helper.monitorConnectionThenCloseAndFinishAsync wrapper as
the other tests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/realtime/channel.test.js`:
- Around line 2118-2153: Add tests for the manual-attach flow by creating
channels with attachOnSubscribe: false (via channels.get(..., { modes:
['publish'], attachOnSubscribe: false })) then explicitly call channel.attach()
before calling channel.subscribe() to exercise the path where subscribe does not
auto-attach; add two cases mirroring the existing ones: one with realtime =
helper.AblyRealtime({ strictMode: true }) asserting channel.subscribe() rejects
with code 93003 and an appropriate hint, and one with realtime =
helper.AblyRealtime() (default strictMode false) asserting channel.subscribe()
resolves (returns null or a ChannelStateChange-like object) and channel.state
=== 'attached'; reuse the same naming pattern and
helper.monitorConnectionThenCloseAndFinishAsync wrapper as the other tests.

In `@test/realtime/presence.test.js`:
- Around line 2403-2431: The test "with strictMode disabled (default), logs a
warning and resolves to []" currently only asserts that channel.presence.get()
resolves to []; update it to either (A) explicitly assert the warning is emitted
by capturing/logging the realtime or helper logger during the test and checking
that a warning containing "strictMode" or similar text was logged when
presence.get() ran (reference the test's realtime instance,
channel.presence.get(), and helper logger/monitoring utilities), or (B) rename
the test string to something accurate like "with strictMode disabled (default),
resolves to []" so it no longer claims to check warning logging; ensure you
change only the test description or add the log assertion around the existing
whenPromiseSettles callback where presence.get() is validated.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f61a171a-8956-4253-a6a4-8dc13ae0f727

📥 Commits

Reviewing files that changed from the base of the PR and between daa4edc and 8af10ac.

📒 Files selected for processing (6)

• src/common/lib/client/realtimeannotations.ts
• src/common/lib/client/realtimechannel.ts
• src/common/lib/client/realtimepresence.ts
• src/common/lib/util/logger.ts
• test/realtime/channel.test.js
• test/realtime/presence.test.js