fix: support non-interactive SSH on SSH Instances (VS Code Remote-SSH, scp/rsync, SFTP) by thxCode · Pull Request #29 · gpustack/gpustack-operator

thxCode · 2026-07-10T04:22:26Z

Summary

SSH-enabled Instances render a two-container Pod (main = user workload, sshd = Alpine sidecar) whose sshd routes every session through ForceCommand /chroot.sh, which nsenters + chroots into main, drops all capabilities, and launched an interactive login shell. The script never inspected $SSH_ORIGINAL_COMMAND: it unconditionally printed a banner and ran $SHELL -l, so any non-interactive request — VS Code Remote-SSH's server bootstrap, ssh host <cmd>, scp, rsync, the SFTP subsystem — had its command silently discarded and its stream corrupted by the banner. The remote server never started; VS Code then failed forwarding to the absent server with "TCP port forwarding may be disabled, or the remote server may have crashed" — the symptom in gpustack/gpustack#5801.

This fixes the sidecar image (pack/ssh-server) so chroot.sh mirrors a stock SSH server, and enables forwarding in sshd_config. The operator only bumps the default instance-ssh-server-image tag.

Root cause (two independent causes)

F1 — chroot.sh ignored $SSH_ORIGINAL_COMMAND, so the non-interactive command never ran and the banner corrupted the stream.
F3 — the Alpine base ships an active AllowTcpForwarding no, so even a running server could not forward.

What changed

chroot.sh dispatches on $SSH_ORIGINAL_COMMAND through one enter() chokepoint that preserves the capability-drop confinement (--bounding-set=-all --inh-caps=-all): empty → interactive banner + login shell (byte-unchanged); the pinned sftp-server path / internal-sftp → SFTP subsystem; else → $SHELL -c <cmd> (no banner).
SFTP is served by a bundled fully static (CGO_ENABLED=0) github.com/pkg/sftp binary staged into main (a musl sidecar server cannot exec in main's glibc rootfs; internal-sftp would serve the sidecar's FS). Staged atomically (temp inode + rename) so concurrent sessions do not race on the running executable; matched by the exact pinned Subsystem path.
sshd_config enables AllowTcpForwarding / AllowStreamLocalForwarding and pins Subsystem sftp.
Default instance-ssh-server-image → gpustack/ssh-server:v1.3.0.
New CPU-only e2e guard (gpustack-operator-e2e CASE 21) + a manual VS Code / sshfs runbook.

Verification (real cluster)

RED on ssh-server:v1.2.0: CASE 21 fails — command discarded, banner leaked, forwarding refused.
GREEN on ssh-server:v1.3.0: CASE 21 passes 9/9 (exec no-banner, interactive banner preserved, sftp put/get into main, loopback TCP forward, capability-strip + mknod denied). GPU-slice-over-SSH regressions (CASE 13/15) stay green.
Concurrency smoke: 10 simultaneous sftp sessions round-trip intact (atomic staging); a command ending in /sftp-server runs instead of being mis-routed.

Release note

The release must publish gpustack/ssh-server:v1.3.0 (built from this branch's pack/ssh-server) alongside the operator so the default resolves. Already-running Instances pick up the new sidecar image on Pod recreation (no forced restart).

Fixes gpustack/gpustack#5801.

Copilot

Pull request overview

This PR fixes SSH-enabled Instance behavior so non-interactive SSH sessions (VS Code Remote-SSH bootstrap, ssh host <cmd>, scp/rsync, SFTP) execute correctly through the sshd sidecar’s ForceCommand chroot/nsenter confinement, and enables TCP/stream-local forwarding needed for VS Code port forwarding. It also bumps the operator’s default SSH sidecar image tag and adds an e2e guard + manual verification runbook.

Changes:

Update chroot.sh to dispatch based on $SSH_ORIGINAL_COMMAND (interactive banner + login shell vs exec-channel command vs SFTP subsystem), while preserving capability-drop confinement.
Update the ssh-server image to enable forwarding and provide an SFTP subsystem backed by a statically built github.com/pkg/sftp server staged into the main container filesystem.
Bump the default instance-ssh-server-image to gpustack/ssh-server:v1.3.0 and add CASE 21 + a manual VS Code/sshfs verification guide.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`pkg/worker/settings/value.go`	Bumps default SSH sidecar image tag to `v1.3.0`.
`pack/ssh-server/rootfs/chroot.sh`	Adds exec-channel/SFTP routing using `$SSH_ORIGINAL_COMMAND` through a single confined `enter()` path; keeps interactive banner behavior.
`pack/ssh-server/Dockerfile`	Enables forwarding, pins SFTP subsystem path, and builds/copies a static SFTP server binary into the sidecar.
`docs/settings.md`	Updates documented default `instance-ssh-server-image` tag.
`.claude/skills/gpustack-operator-e2e/SKILL.md`	Documents new CASE 21 coverage and when it should run.
`.claude/skills/gpustack-operator-e2e/references/manual-ssh-verification.md`	Adds manual verification steps for VS Code Remote-SSH and sshfs.
`.claude/skills/gpustack-operator-e2e/cases/case-21.sh`	Adds an e2e case validating exec-channel behavior, SFTP round-trip into `main`, and TCP forwarding.

- add case-21 asserting `ssh host '<cmd>'` runs in main with no banner - assert the exec path is capability-stripped and host mknod denied - keep interactive login (banner + shell) as a regression guard - round-trip a loopback tcp port-forward through the instance - register case 21 in the e2e skill case table and prose Task 0.1 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- dispatch chroot.sh on $SSH_ORIGINAL_COMMAND instead of always launching a login shell - empty request keeps the interactive banner + login shell, byte-for-byte - a command request runs `$SHELL -c "<cmd>"` in the target with no banner - factor the nsenter+chroot+cap-drop chain into a shared enter() so every path stays confined - move the tty clear into the interactive branch so it never corrupts a command stream Task 1.1 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- replace Alpine's active `AllowTcpForwarding no` with `yes` so port-forwarding works - add `AllowStreamLocalForwarding yes` for unix-socket forwarding - pin `Subsystem sftp /usr/lib/ssh/sftp-server`, replacing the shipped internal-sftp - drop the shipped active lines first so sshd reads exactly ours (no duplicate Subsystem) Task 1.2 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

- add a cross-compiled build stage for a CGO-free github.com/pkg/sftp server - serve SFTP over stdin/stdout so sshd can invoke it as the sftp subsystem - pin pkg/sftp v1.13.7 and strip the binary - COPY it into the image at /usr/local/bin/gpustack-sftp-server for staging into main Task 2.1 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- add an sftp branch to chroot.sh matching the pinned sftp-server path and internal-sftp - stage the bundled static server into the target's /tmp before the cap-drop - run it inside the target's rootfs so sftp/sshfs/scp operate on main's files - keep the same confinement and suppress the banner on this path Task 2.2 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- point instance-ssh-server-image at gpustack/ssh-server:v1.3.0 - sync the default in docs/settings.md Task 3.2 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- add an sftp put/get round-trip check to case-21 - assert the uploaded file lands in main's filesystem, not the sidecar - renumber the port-forward step and refresh the case header + SKILL.md prose Task 4.1 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

- document a VS Code Remote-SSH connect + integrated terminal in main - document an sshfs mount round-trip through the bundled sftp-server - link the runbook from the e2e skill references Task 4.2 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

- match the exact pinned sftp-server path so an exec command ending in /sftp-server is not silently rerouted to the sftp branch - stage the sftp-server atomically (temp inode then rename) so a concurrent session cannot hit ETXTBSY or a partial-write window - require EPERM (rc==1) in the case-21 mknod check, not any non-zero rc Follows the end-of-build review of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

+REQUEST="${SSH_ORIGINAL_COMMAND:-}"
+
 prepare_environment

 nsenter_bin="$(command -v nsenter)"
 chroot_bin="$(command -v chroot)"
 setpriv_bin="$(command -v setpriv)"


Copilot AI review requested due to automatic review settings July 10, 2026 04:22

Copilot started reviewing on behalf of thxCode July 10, 2026 04:22 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread pack/ssh-server/Dockerfile Outdated

Comment thread .claude/skills/gpustack-operator-e2e/cases/case-21.sh Outdated

Comment thread .claude/skills/gpustack-operator-e2e/SKILL.md Outdated

thxCode added 3 commits July 10, 2026 12:35

Copilot AI review requested due to automatic review settings July 10, 2026 04:37

thxCode force-pushed the fix/ssh-instance-noninteractive-ssh branch from f574d9c to b580e21 Compare July 10, 2026 04:37

Copilot started reviewing on behalf of thxCode July 10, 2026 04:37 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread .claude/skills/gpustack-operator-e2e/cases/case-21.sh

Comment thread .claude/skills/gpustack-operator-e2e/references/manual-ssh-verification.md Outdated

Comment thread .claude/skills/gpustack-operator-e2e/references/manual-ssh-verification.md Outdated

thxCode force-pushed the fix/ssh-instance-noninteractive-ssh branch from b580e21 to 2a5d1ac Compare July 10, 2026 06:04

Copilot AI review requested due to automatic review settings July 10, 2026 06:04

Copilot started reviewing on behalf of thxCode July 10, 2026 06:04 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread pack/ssh-server/Dockerfile

thxCode added 5 commits July 10, 2026 14:20

build(settings): bump the default ssh-server image to v1.3.0

e91977a

- point instance-ssh-server-image at gpustack/ssh-server:v1.3.0 - sync the default in docs/settings.md Task 3.2 of ssh-instance-noninteractive-ssh. Signed-off-by: thxCode <thxcode0824@gmail.com>

Copilot AI review requested due to automatic review settings July 10, 2026 06:37

thxCode force-pushed the fix/ssh-instance-noninteractive-ssh branch from 2a5d1ac to effdd69 Compare July 10, 2026 06:37

Copilot started reviewing on behalf of thxCode July 10, 2026 06:38 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread pack/ssh-server/rootfs/chroot.sh

Copilot AI review requested due to automatic review settings July 10, 2026 06:53

thxCode force-pushed the fix/ssh-instance-noninteractive-ssh branch from effdd69 to 9636a5e Compare July 10, 2026 06:53

Copilot started reviewing on behalf of thxCode July 10, 2026 06:53 View session

thxCode merged commit 512f5dc into main Jul 10, 2026
4 checks passed

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread pack/ssh-server/rootfs/chroot.sh

Comment on lines +359 to 365

REQUEST="${SSH_ORIGINAL_COMMAND:-}"

prepare_environment

nsenter_bin="$(command -v nsenter)"

chroot_bin="$(command -v chroot)"

setpriv_bin="$(command -v setpriv)"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: support non-interactive SSH on SSH Instances (VS Code Remote-SSH, scp/rsync, SFTP)#29

fix: support non-interactive SSH on SSH Instances (VS Code Remote-SSH, scp/rsync, SFTP)#29
thxCode merged 9 commits into
mainfrom
fix/ssh-instance-noninteractive-ssh

thxCode commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

thxCode commented Jul 10, 2026

Summary

Root cause (two independent causes)

What changed

Verification (real cluster)

Release note

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants