Skip to content

[docs] add initial documentation for Ray sandboxing - #65503

Open
andrewsykim wants to merge 20 commits into
ray-project:masterfrom
andrewsykim:ray-sandbox-userguide
Open

[docs] add initial documentation for Ray sandboxing#65503
andrewsykim wants to merge 20 commits into
ray-project:masterfrom
andrewsykim:ray-sandbox-userguide

Conversation

@andrewsykim

Copy link
Copy Markdown
Member

Description

Add initial documentation for Ray's experimental sandboxing library.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

@andrewsykim
andrewsykim requested review from a team as code owners August 16, 2026 01:04

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for Ray Sandboxes, an experimental feature that utilizes gVisor to provide lightweight, kernel-isolated execution environments on Ray clusters. The changes include a new deployment guide for Kubernetes (GKE) and a detailed core concept guide covering architecture, APIs, and security models. The review feedback highlights several formatting and syntax issues that need to be addressed, such as bullet point inconsistencies, incorrect step numbering, a shell command prefix, and syntax errors in both Markdown and RST files (including incorrect cross-references and Markdown syntax used inside an RST file).

- {ref}`kuberay-rayservice-deepseek-example`
- {ref}`kuberay-verl`
- {ref}`kuberay-agent-sandbox`
* {ref}`kuberay-sandboxing`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For consistency with the other items in this list, please use a hyphen (-) instead of an asterisk (*) for the bullet point.

kubectl get pods -l job-name=rayjob-sandbox

# Stream the demo logs
$ kubectl logs -f -l job-name=rayjob-sandbox

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For consistency with the other commands in this guide, please remove the leading $ prefix from this command.

Suggested change
$ kubectl logs -f -l job-name=rayjob-sandbox
kubectl logs -f -l job-name=rayjob-sandbox

```
---

## (Optional) Step 6: Verify Isolation and Security Guarantees

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The step numbering skips from Step 3 to Step 6. Please renumber this to Step 4.

Suggested change
## (Optional) Step 6: Verify Isolation and Security Guarantees
## (Optional) Step 4: Verify Isolation and Security Guarantees


---

## (Optional) Step 7: Build a Custom Ray Image with Pre-installed `runsc`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The step numbering skips from Step 3 to Step 7. Please renumber this to Step 5.

Comment thread doc/source/cluster/kubernetes/examples/ray-sandboxing.md Outdated
Comment thread doc/source/ray-core/sandboxes.rst Outdated
Comment thread doc/source/ray-core/sandboxes.rst Outdated
@ray-gardener ray-gardener Bot added docs An issue or change related to documentation core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Aug 16, 2026
@andrewsykim
andrewsykim force-pushed the ray-sandbox-userguide branch from 7ae9d6b to 130b8b6 Compare August 16, 2026 02:44

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 130b8b6. Configure here.

Comment thread python/ray/experimental/sandbox/sandbox.py Outdated
print(result.stdout)
ray.get(pool.close.remote())
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add another section here Example (advanced): Passing configurations directly to gvisor where we show some more of the most common ways to use _oci_spec_transform_fn, e.g. for host mounts, setting up networking and passing capabilities. And maybe a note that we are iterating on this and how to best expose these without needing _oci_spec_transform_fn and we would love to hear about people's feedback in an issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add here that _by default_, the working directory is the only writable path in the sandbox and that by setting readonly = False the whole file system can be made writeable (and make it clear to the reader that writes will go into an overlayfs filesystem so multiple file systems with the same image that do writes won't interfere with each other). [I believe this is true, let me know if it is not]

@pcmoritz pcmoritz Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as I made here https://github.com/ray-project/ray/pull/65503/changes#r3793075723 -- this one is arguably more important since the docs to create is linked from the main docs very prominently.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should say something like Create a _remote_ sandbox environment. or similar to make the difference to the SandboxRuntime clearer.

Comment thread doc/source/ray-core/sandboxes.md Outdated
### Core Components

1. **High-Level Helper ({func}`ray.experimental.sandbox.create <ray.experimental.sandbox.create>`):** Spawns a Ray actor that encapsulates the sandbox lifecycle and returns an `ActorHandle`.
2. **Sandbox Actor ({class}`ray.experimental.sandbox.Sandbox <ray.experimental.sandbox.Sandbox>`):** A Ray actor managing scheduling, lifecycle, command execution, and file I/O for an isolated sandbox instance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can modify this a little bit and say something like A Ray actor that serves as a proxy to forward command execution and file I/O to the isolated sandbox instance and manages scheduling and lifecycle of the sandbox or similar, to make the mental model clear to the user that the actor is not actually run in the sandbox.

Comment thread doc/source/ray-core/sandboxes.md Outdated

To install `runsc` on a Linux worker node, see the [gVisor installation guide](https://gvisor.dev/docs/user_guide/install/).

## Usage Patterns and Examples

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine this section is actually the most useful for users so we should move it up. Some of the above sections are more interesting from the internal implementation perspective which users will generally be less interested in, it would make more sense to move those down a little more.

@pcmoritz

pcmoritz commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Thanks a lot for putting this together, I finished reviewing it now. It looks great, I left a bunch of suggestions.

Also cc @dstrodtman who might have additional comments or suggestions :)

@pcmoritz
pcmoritz requested a review from dstrodtman August 17, 2026 04:06
@andrewsykim

Copy link
Copy Markdown
Member Author

thanks for the review, all comments addressed

@dstrodtman dstrodtman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs style review (Anyscale docs team)

Douglas Strodtman, technical writer on the Anyscale docs team. @pcmoritz cc'd me above, so here's a full pass. Claude Code assisted with the mechanical sweeps and the verification queries below; I read every comment before posting and I stand behind each one.

Scope: style, grammar, and Ray docs conventions only, against the Ray documentation style guide. I don't own whether any of the described behavior is true. Everything technical below is a question for you and @pcmoritz, not a request. Happy to give this a docs-side approval once the content questions settle.

Worth saying plainly: this is the first Claude-aided review I've run against the Kubernetes and KubeRay conventions that landed in #65239, so the guide is getting its first real workout here. I read every recommendation and they all match what I'd have written by hand. But if a specific term or rule reads wrong to you, that's useful signal about the rule rather than a verdict on this PR, and I'd rather hear it and refine the guide than have you apply something that doesn't fit. You reviewed #65239, so you're the right person to push back.

I read the existing threads first. @pcmoritz's six comments and the bot findings all look applied at b1029a1, including the section reorder, the "remote sandbox" wording, the proxy-actor description, and the readonly docstrings. Where I comment on that text I'm styling your applied fix, not reopening the request.

One bot suggestion to skip: the gemini comment asking to point the resource-isolation cross-reference at resource-isolation-with-cgroupv2. I checked the tree; .. _resource-isolation: is the label defined in doc/source/ray-core/resource-isolation-with-cgroupv2.rst and resource-isolation-with-cgroupv2 isn't a label anywhere. What you have is correct.

What the inline comments cover

The bulk is sentence-case headings, about 20 across the three pages. Nothing in CI catches this: Google.Headings is disabled and .vale.ini doesn't scope Vale to Ray Core or KubeRay pages at all, so it's a human check every time. I batched them into two sweep comments rather than 20 separate ones.

The one structural change I'd push for is the "Benefits of gVisor" H4. It's a bolded benefits list, which the guide names twice as something to cut, and it's the page's only H4. The suggestion folds it into two prose paragraphs and keeps every claim.

The rest: numbered lists that aren't sequences, first-person plural, e.g. and via and "utilizing," a few passive constructions with a nameable actor, and bare {ref} links so text tracks titles.

Questions for you and @pcmoritz

  1. _oci_spec_transform_fn framing. runtime.py documents it as "PRIVATE -- development/testing only ... No stability guarantees," while the new advanced section presents it as the supported escape hatch. Since @pcmoritz asked for that section, softening the docstring is probably the right direction, but it should say one thing.
  2. RAY_SANDBOX_IGNORE_CGROUPS=1 in Troubleshooting. It maps to _ignore_cgroups, also marked "PRIVATE -- testing only" in runtime.py. Is it something users in cgroup-restricted environments should reach for?
  3. The inline RayJob script. It's a hand-copy of the sample YAML and has already drifted by one line (see the L93 comment). A literalinclude or a link to the sample would keep them from separating further.
  4. Two adjacent sandbox examples. The examples index now lists "Sandboxed Code Execution with Ray and Agent Sandbox" next to "Deploying Ray Sandboxes using KubeRay." They're genuinely different things, but the titles don't say so. One clause on each would help a reader pick.
  5. Untested snippets. The Python blocks are bare fences, so the doc tests don't run them, and a couple aren't runnable standalone (sb is undefined in the Step 4 blocks). Fine if that's intentional for illustrative fragments.

Nice work on this. It's a lot of new surface and the structure is right.

Comment thread doc/source/ray-core/sandboxes.md Outdated
Comment thread doc/source/ray-core/sandboxes.md Outdated
Comment thread doc/source/ray-core/sandboxes.md Outdated
Comment thread doc/source/ray-core/sandboxes.md Outdated
Comment thread doc/source/ray-core/sandboxes.md Outdated
Comment thread doc/source/cluster/kubernetes/examples/ray-sandboxing.md
Comment thread doc/source/cluster/kubernetes/examples/ray-sandboxing.md
Comment thread doc/source/cluster/kubernetes/examples/ray-sandboxing.md Outdated
Comment thread doc/source/cluster/kubernetes/examples/ray-sandboxing.md Outdated
Comment thread doc/source/ray-core/api/sandboxes.md Outdated
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
andrewsykim and others added 5 commits August 17, 2026 17:35
Co-authored-by: Douglas Strodtman <douglas@anyscale.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
Co-authored-by: Douglas Strodtman <douglas@anyscale.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
Co-authored-by: Douglas Strodtman <douglas@anyscale.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
Co-authored-by: Douglas Strodtman <douglas@anyscale.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
Co-authored-by: Douglas Strodtman <douglas@anyscale.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
@pcmoritz pcmoritz added the go add ONLY when ready to merge, run all tests label Aug 17, 2026

@pcmoritz pcmoritz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks a lot for writing this!

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
@andrewsykim
andrewsykim force-pushed the ray-sandbox-userguide branch from bbff47a to 5a0ab45 Compare August 17, 2026 18:37
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
@richardliaw richardliaw changed the title add initial documentation for Ray sandboxing [docs] add initial documentation for Ray sandboxing Aug 17, 2026
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community core Issues that should be addressed in Ray Core docs An issue or change related to documentation go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ray fails to serialize self-reference objects

3 participants