[core] fix None yield from restarted streaming generator with application errors by rueian · Pull Request #65121 · ray-project/ray

rueian · 2026-07-30T00:26:15Z

Description

Previously, there was a special path in the streaming generator reconstruction:

If the retried streaming generator stops with an application exception, it will try to fill return_objects(0) for all refs that have not yet been restored. But return_objects(0) is None if the first streaming generator succeeds.

The result is that, in the case of lineage reconstruction, we can see ray.get(ref) unexpectedly return None if the restarted streaming generator stops with an application exception.

This PR removes the special path. We will still treat the exception as the recovered object for the corresponding ref, but if the restarted streaming generator produces a different number of yields than its first execution, the user will get a StreamingGeneratorReplayInconsistentError exception on the following ray.get instead.

…tion errors Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

chatgpt-codex-connector · 2026-07-30T23:01:00Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

gemini-code-assist

Code Review

This pull request ensures that streaming generator replays failing with an application error are correctly validated for object count consistency, and prevents copying the static None return onto stream ObjectRefs during such replays. Specifically, FailStreamingGeneratorReplayIfInconsistent is now executed for both successful and application-error completions. Additionally, regression and unit tests have been added to verify these behaviors. There are no review comments, so I have no feedback to provide.

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

Kunchd

Thanks for the fix! I'm a little confused on the exact desired behavior that we want here, so I left a few questions.

Kunchd · 2026-08-05T01:10:31Z

+  // the inconsistent objects before the failure propagates. Applies to both
+  // successful and application-error completions: retries must reproduce the
+  // same object count, and app-error replays often report fewer IDs.
+  if (FailStreamingGeneratorReplayIfInconsistent(task_id, reply)) {


If FailStreamingGeneratorReplayIfInconsistent returns true, it means one of the following:

An error (application or system level) has caused the generator to terminate before it could reach the same point as the first execution. In this case, should we return the actual error that caused the issue instead of STREAMING_GENERATOR_REPLAY_INCONSISTENT?

The generator was non-deterministic and returned a different number of results. In this case, we should return STREAMING_GENERATOR_REPLAY_INCONSISTENT.

Here we only deal with application errors. System-level failures are handled via FailPendingTask, which doesn't have the FailStreamingGeneratorReplayIfInconsistent check.

The actual application error should still be observed by users since an application error will be treated as a normal yield. For example, if the user has ref1 and ref2 originated from a streaming generator on hand but the actual objects are lost, and the reconstructed streaming generator fails with an Exception1 before yielding anything. Then ray.get(ref1) will raise Exception1 and ray.get(reg2) will raise STREAMING_GENERATOR_REPLAY_INCONSISTENT

Is the application error is non-retriable, I don't think we actually report the application error as a generator yield (see https://github.com/ray-project/ray/blob/master/python/ray/_raylet.pyx#L1911-L1925). Instead the error is stored as a return value: https://github.com/ray-project/ray/blob/master/python/ray/_raylet.pyx#L2411-L2420. In this case, wouldn't we be missing this error if we don't do what we did in the past and explicitly populate the streaming generator with the error?

Also, maybe I'm missing something, but I don't know if FailPendingTask is guaranteed to trigger on system failures. It seems like we only report a non-ok status for system errors if the output was not written (see https://github.com/ray-project/ray/blob/master/src/ray/core_worker/task_execution/task_receiver.cc#L123-L133). I'm not sure if it's possible for the output to be correctly written, but we still receive a system error afterwards.

If the application error is non-retriable, we do report it as a yield. ray.get(gen.completed()) will be None though. The user can only get the error via ray.get(next(gen)).

If the application error is retriable, the expectation is that we never report it as a yield. The user can only get the error via ray.get(gen.completed()) after the retries are exhausted.

In this case, wouldn't we be missing this error if we don't do what we did in the past and explicitly populate the streaming generator with the error?

We will only miss an application error if it meets both of the following:

It is retriable so we don't report it as a yield.

The streaming generator has succeeded once so it can't be injected into gen.completed().

At this point, I think we still shouldn't make the retriable application error a yield. That is not aligned with the expectation of marking an error retriable. Users should still expect to only get the retriable error via ray.get(gen.completed()) after the retries are exhausted. This has nothing to do with the FailStreamingGeneratorReplayIfInconsistent.

Also, maybe I'm missing something, but I don't know if FailPendingTask is guaranteed to trigger on system failures. It seems like we only report a non-ok status for system errors if the output was not written (see https://github.com/ray-project/ray/blob/master/src/ray/core_worker/task_execution/task_receiver.cc#L123-L133). I'm not sure if it's possible for the output to be correctly written, but we still receive a system error afterwards.

I think we really don't care about the system errors in the path of task_receiver.cc#L123-L133 because when objects_valid is true for a streaming generator, it means the streaming generator has ended already by itself, not by the system error. So if it has inconsistent yields, then we should raise STREAMING_GENERATOR_REPLAY_INCONSISTENT.

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

…ator-retry-none Keep master's first-execution EOF error materialization, and drop the re-execution app-error path that copied return_objects(0) onto stream refs. Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 768956c. Configure here.}

Kunchd

Thanks for clarifying! I left one last comment from my interpretation of what's expected of the test.

Kunchd · 2026-08-10T21:48:31Z

+        app_error_reply.add_streaming_generator_return_ids();
+    return_id_proto->set_object_id(spec.StreamingGeneratorReturnId(i).Binary());
+    return_id_proto->set_is_plasma_object(true);
+  }


One final comment. From my walk through the code, the current invocations in the test doesn't fully replicate the states for a resubmission. When an error is reported, we should actually invoke HandleReportGeneratorItemReturns on them as well. And this should in turn cause the error to be added to plasma via put_in_local_plasma_callback and the assertion for plasma_put_error_types should not be false below.

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

rueian added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Jul 30, 2026

rueian changed the title ~~[core] fix None yield from restarted streaming generator with applica…~~ [core] fix None yield from restarted streaming generator with application errors Jul 30, 2026

[core] fix None yield from restarted streaming generator with applica…

81b3443

…tion errors Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

rueian force-pushed the fix-streaming-generator-retry-none branch from b705c88 to 81b3443 Compare July 30, 2026 02:48

more tests

0080a7b

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

rueian marked this pull request as ready for review July 30, 2026 23:00

rueian requested a review from a team as a code owner July 30, 2026 23:00

gemini-code-assist Bot reviewed Jul 30, 2026

View reviewed changes

nit

e8266c8

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

Kunchd reviewed Aug 5, 2026

View reviewed changes

rueian added 2 commits August 4, 2026 20:23

review

b6c36a8

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

cursor Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread src/ray/core_worker/task_manager.cc

rueian and others added 5 commits August 5, 2026 15:54

Merge branch 'master' into fix-streaming-generator-retry-none

ea524e5

Merge branch 'master' into fix-streaming-generator-retry-none

e226f10

Merge branch 'master' into fix-streaming-generator-retry-none

874dcd4

Merge branch 'master' into fix-streaming-generator-retry-none

0c9b171

Merge branch 'master' into fix-streaming-generator-retry-none

0477c86

Kunchd reviewed Aug 10, 2026

View reviewed changes

rueian added 2 commits August 11, 2026 16:40

Merge branch 'master' into fix-streaming-generator-retry-none

2b47abe

replicate resubmission

ddbe55c

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

Navigation Menu

Uh oh!

[core] fix None yield from restarted streaming generator with application errors - #65121

[core] fix None yield from restarted streaming generator with application errors#65121
rueian wants to merge 12 commits into
ray-project:masterfrom
rueian:fix-streaming-generator-retry-none

rueian commented Jul 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Jul 30, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Kunchd left a comment

Uh oh!

Uh oh!

Kunchd Aug 5, 2026

Uh oh!

rueian Aug 5, 2026

Uh oh!

Kunchd Aug 5, 2026

Uh oh!

rueian Aug 5, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Kunchd left a comment

Uh oh!

Kunchd Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rueian commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

chatgpt-codex-connector Bot commented Jul 30, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Kunchd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kunchd Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

rueian Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Kunchd Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

rueian Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kunchd left a comment

Choose a reason for hiding this comment

Uh oh!

Kunchd Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rueian commented Jul 30, 2026 •

edited

Loading