Add noninteractive, isolated Python environment tooling for agents (#1900)

# Add noninteractive, isolated Python environment tooling for agents

## Summary

Add a private, versioned `__pythonTools` capability to the existing flat
extension export so Python agent tools can configure environments,
inspect interpreters/packages, and install packages without requiring
extension-owned pickers or dialogs.

The companion Python-extension change consumes this capability. The
public API contract is unchanged, and host tool approval remains in
force.

## Motivation

Agent requests currently reuse interactive environment workflows, which
can pause for interpreter, manager, package, or environment-creation
choices. A selected global interpreter can also become the
package-installation target.

This change gives agents an explicit noninteractive path with isolation
by default, actionable errors, and scoped results, without recording why
an interpreter was selected.

## Behavior

- Automatic configuration reuses a usable isolated environment or
creates one for the project. A selected System/Pyenv base creates a venv
using that interpreter; Conda base creates a project Conda environment
with the same Python major/minor version.
- An explicit `pythonPath` still selects that exact existing
interpreter. Package tools reject global/base environments even when
selected explicitly.
- Queries do not create environments or change selection. Package
installation does not implicitly configure an environment.
- Configuration and installation require an open, trusted, local
workspace. Results identify the effective resource, environment, and
execution information.
- Root selections survive reload, including when
`python.defaultInterpreterPath` is configured. Nested targets persist
exact project entries without replacing root defaults.
- Existing Poetry/Pipenv project caches and enabled PEP 723 scripts have
noninteractive discovery/setup paths.
- Errors and partial creation results are returned explicitly.
Same-scope operations are serialized; cancellation and timeouts await
owned-process cleanup and report uncertainty instead of falsely claiming
that a process stopped.
- Failed human venv initialization invalidates partial discovery and
readiness state, so subsequent public lookups retry discovery instead of
remaining on a stale global fallback. Private discovery remains
independent of human onboarding.

## Human-flow compatibility

Built-in managers opt in through internal symbol capabilities and
explicit operation flags. Normal public selection, creation, package,
and execution entry points retain their interactive behavior, including
public quick-create's `.venv-N` suffixing.

This is not a claim that all shared code is untouched:

- Venv collection events can arrive earlier relative to base-manager
onboarding.
- Public Conda command failures no longer generate the extra unhandled
rejection caused by the old unused `finally()` promise.
- Agent selections, settings, and package changes intentionally remain
visible to subsequent human operations.

## Validation

- Full ESLint and TypeScript checks pass.
- Windows unit suite after rebasing onto current upstream `main`:
**2,752 passing, 7 pending**. Six new failure-path assertions failed
before the retry fix and now pass; successful initialization remains
cached. The new upstream venv deletion, unresolved-selection recovery,
refresh selection-race, uv bootstrap, and scoped Conda discovery tests
also pass with the combined implementation.
- Reviewer regression coverage verifies that private Conda execution
preserves shell metacharacters as one argument with shell execution
disabled, resolves Windows batch launchers to a real executable or fails
actionably, and scopes both package queries and installation
independently when two Poetry projects share one environment.
- Linux private/public boundary and owned-process suites with the retry
fix: **116 passing**.
- Real VS Code comparisons against the current base verified manual
selection, package install/remove, Run Python File, debugging, unittest
discovery/execution, switching back to global Python, and public
quick-create.
- Real-host agent verification covered fresh/global setup, selected-base
preservation, reload, multi-root, nested targets, Conda, Poetry/Pipenv,
PEP 723, cancellation, no-workspace behavior, and invalid input.
- Critical flows were repeated against freshly built production VSIX
contents. Installed files were compared with the packages rather than
inferred from unchanged version numbers.
- An additional real VS Code before/after retry check used the loaded
extension's actual venv manager, native discovery, a real newly created
venv, public selection and Run Python File. A one-shot post-discovery
base failure was injected only in the isolated test host. Both direct
initialization and public-get retries recovered without manual manager
refresh after the fix; the pre-fix build required refresh.

## Known findings before merge

- Ordinary project `.python-version` / `pyproject.toml` interpreter
constraints are not automatically resolved. Selected base interpreters
and `defaultInterpreterPath` are honored; PEP 723 constraints are
handled separately.
- macOS/remote E2E, arbitrary third-party integrations, and real-chat
model choices/approval combinations have not been exhaustively verified.

## Companion change

Python consumer PR:
[microsoft/vscode-python#26208](microsoft/vscode-python#26208).

The consumer checks the private version/method shape and retains a
compatibility route when the capability is unavailable.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Author: StellaHuang95

.github/instructions/testing-workflow.instructions.md

@@ -20,6 +20,7 @@ This guide covers the full testing lifecycle:
 
 - Pip commands that return JSON must pass `--disable-pip-version-check`; the process helper combines stderr with stdout, so update notices can otherwise make valid JSON unparseable (1).
 - When a view subscribes to a newly added provider event, TypeMoq-based view tests must return a real `EventEmitter.event`; an unstubbed event yields an undefined disposable and fails during teardown (1).
+- Test agent selections across a full reload with `python.defaultInterpreterPath` set. An effective manager value equal to the extension default does not prove a workspace value was saved; tool-owned persistence must inspect `workspaceValue` or startup can restore the global interpreter (1).
 
 ### When to Use This Guide
 

CONTRIBUTING.md

@@ -98,6 +98,152 @@ The npm package under [`api/`](./api) is the public API facade other extensions
 - `src/api.ts`, `src/types.ts`, and `src/publicErrors.ts` are validated on every PR by the extension's own lint and TypeScript compile.
 - **Versioning and compatibility:** the published package version in [`api/package.json`](./api/package.json) is maintained independently of the extension version in [`package.json`](./package.json) — the two do not need to match. Compatibility is based on the API shape exported by the installed Python Environments extension at runtime. Package updates must preserve backwards-compatible contracts unless the API package version intentionally communicates a breaking change; consumers should treat newly added members as optional when they may run against older installed extension versions. Any PR that edits `src/api.ts`, `src/types.ts`, or `src/publicErrors.ts` must bump `api/package.json` (use the `skip api version` label to bypass) and add an entry to [`api/CHANGELOG.md`](./api/CHANGELOG.md) (use the `skip api changelog` label to bypass).
 
+## Internal Python agent-tool bridge
+
+The extension's existing **flat** export also provides `__pythonTools.version === 1`.
+This is an internal convention, not a security boundary or a supported public API.
+Its types live only in `src/internal/pythonToolsApi.ts`; do not copy them into
+`src/types.ts`, `src/api.ts`, or the published API package. Consumers must check the
+version and all three methods at runtime before using the private API. The Python
+consumer uses its previous public Environments integration when that capability
+is absent or incompatible; that compatibility route can still prompt. It never
+falls back after a private operation starts, including errors, partial results,
+timeouts, or cancellation. Environments-disabled users retain the legacy Python
+route.
+Explicit compatibility selection persists through the public Environments setter, not
+just the Python extension's cached interpreter path, so later package operations
+use the same environment. Reusing an already-selected environment is read-only;
+it does not reselect or wait for an interpreter-change event.
+
+| Method | Request | Result |
+| --- | --- | --- |
+| `configureEnvironment(request, token)` | `{ resourcePath?: string, pythonPath?: string }` | Select an explicit interpreter exactly; otherwise reuse an isolated environment or create one for the project. Success includes `created`. |
+| `getEnvironment(request, token)` | `{ resourcePath?: string, includePackages?: boolean }` | Resolve the real selected environment, not the public API's short-timeout cache. `includePackages: true` returns `{ name, version?: string }[]`, including `[]`, or an error. |
+| `installPackages(request, token)` | `{ resourcePath?: string, packages: string[] }` | Install into an isolated environment. Does not create or change selection implicitly, or modify any global/base interpreter. |
+
+Automatic configuration uses isolation by default, regardless of whether a global
+interpreter was selected automatically or by a person. Virtual environments are
+recognized by `pyvenv.cfg`; non-base Conda and inline-script environments are also
+isolated. Existing isolated environments outside the workspace remain reusable.
+When no isolated environment is selected, configuration can resolve Poetry/Pipenv
+caches through the native finder's project association. It does not infer ownership
+from the cache location or record selection history; ambiguous project matches
+require an explicit `pythonPath`. Read-only queries do not select these candidates.
+There is no selection-origin bookkeeping or persistence. Old origin records are
+ignored; ordinary current-interpreter selection and its persistence are unchanged.
+An explicit `pythonPath` selects that interpreter without creating an environment.
+This can select global Python for execution or queries, but does not authorize
+package installation into it: `ENVIRONMENT_NOT_ISOLATED` directs the caller to
+configure without `pythonPath` or select an existing isolated environment.
+
+All methods require a VS Code `CancellationToken` and a trusted workspace.
+`resourcePath` accepts an absolute path or a `file:` URI within an open workspace.
+The Python tools default an omitted or empty path to the first workspace folder and
+pass it explicitly. They disclose the target in preparation/results and instruct agents
+to reuse the returned `resourcePath`. Editor focus does not change this default.
+At the direct private API boundary, omission selects the only workspace root;
+zero/multiple roots still produce actionable errors. The consumer default does not
+relax that backend validation.
+Other explicit input is never replaced by that default. Because the route is
+workspace scoped, read-only tools keep their previous Environments behaviour when
+no folder is open. Configuration and installation with a compatible private API
+report `NO_WORKSPACE` instead of opening a picker or modifying global Python.
+Nested projects and per-script selections retain their scope; a file is
+never used as an environment creation directory. There is no User/global settings
+fallback. An explicit `pythonPath` can switch away from an unavailable manager.
+New nested targets persist as exact project entries without changing root defaults;
+their automatic dependency discovery is restricted to that target directory.
+Tool selections at the workspace root persist an explicit workspace manager, even
+when it matches the extension default. This prevents `python.defaultInterpreterPath`
+from restoring global Python on reload without rewriting that interpreter setting.
+Ordinary human selections retain their existing settings-write behavior.
+
+Results are `{ status: 'success', environment, resourcePath?, created?, packages? }`
+or `{ status: 'error', code, message, environment?, resourcePath? }`. Errors retain
+an already-created environment when subsequent package installation or selection
+fails; retries reuse a resolvable `.venv`/`.conda` rather than creating a suffixed
+directory. Selecting an existing environment does not reinstall project
+dependencies. A resolvable `.venv`/`.conda` is only reused when it actually contains
+the resolved isolated environment. A new subdirectory can reuse an isolated parent-project
+selection; `resourcePath` chooses the target, not a requirement to create a fresh
+environment.
+Register an independent project when it needs a separate selection.
+Reads and writes for the same effective target share a cancellable
+queue; waiting behind an operation does not use the discovery timeout. Different
+targets remain independent. Discovery timeouts report `NOT_READY`; a configured
+environment manager that never registers reports `UNSUPPORTED_MANAGER` naming the id;
+subprocess
+timeouts report `TIMEOUT`. Cancellation throws `CancellationError` after owned
+process cleanup. Cancellation is cooperative, not a rollback: completed writes or
+package changes are not undone. A completed action remains successful if a
+cancellation request arrives on its return. If cleanup fails,
+`PROCESS_TERMINATION_FAILED` takes precedence:
+the process may still be modifying files, so consumers must not blindly retry.
+Cleanup waits for process closure and owned POSIX group completion; an exited
+Windows parent cannot turn a failed tree termination into successful cancellation.
+If a Windows parent has already exited when cancellation starts, descendant
+cleanup cannot be confirmed; the tool reports that uncertainty instead of
+signalling a potentially reused PID.
+Only internal `toolExecution` calls enable these subprocess policies. Public
+progress tokens do not enable agent timeouts, input suppression, or process-tree
+cleanup semantics. Public package calls retain their existing execution defaults.
+
+Internal symbol capabilities opt built-in managers into prompt-free operations;
+they are unrelated to the human UI's `quickCreateConfig`. Creation supports venv,
+Conda, and enabled PEP 723 inline scripts (the latter retain their existing cache
+ownership/metadata rules). System and Pyenv base interpreters route creation to
+venv; Conda base routes to an isolated Conda prefix. Poetry and Pipenv environments
+can be reused, but unsupported creation and contributed managers without this
+capability fail explicitly. Private package operations use the existing headless pip/uv, Conda, and
+Poetry helpers with the real token and strict, uncached package listing. Venv
+dependency validation fails rather than asking to continue. No tool installs a
+base Python automatically. Conda response descriptors clone `execInfo.activatedRun`
+to a standalone `conda run --prefix ... --no-capture-output ...` command; shared
+public descriptors are not changed.
+Venv creation uses the selected Python 3 as its base, falling back to the latest
+discovered Python 3 only when none is selected. Conda creation requests the selected
+Python major/minor version. This does not add a project version-constraint resolver
+for `pyproject.toml` or `.python-version`. To choose a particular base, select it
+first and then configure without `pythonPath`; the path parameter itself is an
+exact selection request, not a venv base-interpreter argument.
+Bare Conda commands configured through `python.condaPath` are resolved through
+PATH for private execution and returned command prefixes.
+Conda base is isolated even when inherited through `CONDA_PREFIX`, and failed
+Conda discovery does not authorize replacement creation. Public Conda refresh remains best-effort while
+the private readiness check retains the discovery error. Poetry verifies the
+requested project's actual environment prefix before package operations. Agent
+inventory uses `poetry run pip list --format=json`, including when Poetry supplies
+pip itself; the public lockfile-based `poetry show` path is unchanged.
+Private Poetry installs also apply the lockfile with `install --no-root`, since
+`add` alone skips already-declared dependencies missing from the environment.
+Direct installs into shared PEP 723
+caches return `IMMUTABLE_ENVIRONMENT`; edit script metadata and configure again.
+With the inline-script feature enabled, configuring a Python file with PEP 723
+metadata can create its cache on the first tool invocation, without first using a
+CodeLens or registering a script project manually. Invalid metadata fails rather
+than falling back to ordinary project creation. This opt-in configuration route
+does not change human or read-only routing, which still requires a validated
+association. Target the project directory or supply `pythonPath` for ordinary
+project/exact-interpreter configuration instead.
+An agent joining a human-owned inline cache build gets `ENVIRONMENT_BUSY` without
+waiting for or cancelling the human operation.
+
+Private discovery is separate from public initialization and its onboarding.
+Agent reads never start or await installation or missing-manager questions, even
+when a concurrent public initialization is waiting for input. Ordinary startup,
+public API, and human workflows retain their prompts; independently triggered
+startup UI can still appear while a tool runs. There is no global "suppress UI"
+switch. Exact nested-project persistence is also opt-in to the private setter;
+ordinary manager-setting updates retain their existing defaults.
+Human venv initialization retries discovery and base preparation after a failure,
+including failures after discovery has completed. A partial initialization is not
+reused as a successful cache on the next public lookup; private discovery still
+does not wait for human onboarding.
+
+Keep tests for both internal and human routes when changing these helpers. The
+focused suites are in `src/test/internal`, with shared creation/package and
+inline-script regressions under `src/test/managers`.
+
 ## Questions or Issues?
 
 - **Questions**: Start a [discussion](https://github.com/microsoft/vscode-python/discussions/categories/q-a)

src/common/inlineScript/metadata.ts

@@ -3,7 +3,7 @@
 
 import * as tomljs from '@iarna/toml';
 import * as fs from 'fs/promises';
-import { Uri } from 'vscode';
+import { l10n, Uri } from 'vscode';
 import { traceVerbose, traceWarn } from '../logging';
 import { PythonVersion } from '../pythonVersion';
 import { PythonVersionSpecifier } from '../pythonVersionSpecifier';
@@ -654,8 +654,13 @@ export function sliceHeaderBytes(text: string): string {
  *  - any I/O error (logged at `traceVerbose`);
  *  - any of the malformed-metadata cases handled by
  *    `readInlineScriptMetadata`.
+ * @param uri The local script to read.
+ * @param strict Throw on I/O errors or invalid metadata instead of treating them as absent.
  */
-export async function readInlineScriptMetadataFromFile(uri: Uri): Promise<InlineScriptMetadata | undefined> {
+export async function readInlineScriptMetadataFromFile(
+    uri: Uri,
+    strict = false,
+): Promise<InlineScriptMetadata | undefined> {
     if (uri.scheme !== 'file') {
         traceVerbose(`inline script metadata: skipping non-file URI scheme '${uri.scheme}'`);
         return undefined;
@@ -671,11 +676,22 @@ export async function readInlineScriptMetadataFromFile(uri: Uri): Promise<Inline
             await handle.close();
         }
     } catch (err) {
+        if (strict) {
+            throw err;
+        }
         traceVerbose(`inline script metadata: failed to read ${uri.fsPath}:`, err);
         return undefined;
     }
 
-    return readInlineScriptMetadata(text, uri.fsPath);
+    const result = parseInlineScriptMetadata(text, uri.fsPath);
+    if (
+        strict &&
+        (result.kind === 'invalid' ||
+            (result.kind === 'parsed' && result.problems.some((problem) => problem.severity === 'error')))
+    ) {
+        throw new Error(l10n.t('Fix the PEP 723 metadata in {0} before configuring its environment.', uri.fsPath));
+    }
+    return result.kind === 'parsed' ? result.metadata : undefined;
 }
 
 /**

src/common/lockfile.apis.ts

@@ -8,6 +8,7 @@ import * as path from 'path';
 export interface AcquireFileLockOptions {
     readonly timeoutMs: number;
     readonly retryIntervalMs: number;
+    readonly checkCancellation?: () => void;
 }
 
 export interface AcquiredFileLock {
@@ -52,6 +53,7 @@ export async function acquireFileLock(filePath: string, options: AcquireFileLock
     const deadline = Date.now() + options.timeoutMs;
 
     while (true) {
+        options.checkCancellation?.();
         try {
             await fsapi.mkdir(lockPath);
             try {