diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cac6af86b..9de207fbd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -81,6 +81,15 @@ Thank you for your interest in contributing to the Python Environments extension ### Testing Run unit tests with the different configurations in the "Run and Debug" panel +### Experimentation and telemetry + +See [Experimentation infrastructure](./docs/experimentation.md) for the internal TAS +service, publisher configuration, lifecycle and consent behavior, deterministic tests, +and the baseline measurement inventory. The new assignments endpoint is fixed by the +platform, but live experimentation remains unconfigured until VS Code exposes an approved +DevDeviceId provider. This infrastructure does not enable a feature or publish an +experimental setting. + ## Contributor License Agreement (CLA) This project requires contributors to sign a Contributor License Agreement (CLA). When you submit a pull request, a CLA bot will automatically check if you need to provide a CLA and guide you through the process. You only need to do this once across all Microsoft repositories. diff --git a/docs/experimentation.md b/docs/experimentation.md new file mode 100644 index 000000000..c5684d3ab --- /dev/null +++ b/docs/experimentation.md @@ -0,0 +1,222 @@ +# Experimentation infrastructure + +The extension owns one internal TAS service per activation. This infrastructure +supports general experimentation and measurement without enabling product features. +The new assignments endpoint and parameter name are platform-owned. Live TAS +configuration remains absent until VS Code provides an approved DevDeviceId API to +the extension. + +## Settings rollout versus extension attribution + +An `onExP` settings rollout does not require this service merely to apply its +treatment. Declare the setting with a safe package default and the `onExP` tag, +configure `config.` in ExP, and read the effective value through +`workspace.getConfiguration()`. VS Code core retrieves the assignment and applies +the treatment as a default; explicit user and workspace values still take +precedence. Do not add a second TAS treatment gate for the same setting. + +Use this service when an extension-owned experiment needs treatment variables, or +when Python Environments telemetry must carry the matching assignment context for +an experiment scorecard. + +## Publisher configuration + +`initializeExperimentation()` reads an optional top-level `experimentation` object +from the installed extension's `package.json`. This is publisher-owned metadata, +**not** a VS Code setting. The endpoint and identity parameter cannot be overridden +by publisher or workspace configuration. + +The following is a schema illustration, not deployable configuration. The manifest +entry must remain absent until the DevDeviceId provider is approved and wired. +Do not copy names from legacy `X-*` headers. + +```json +{ + "experimentation": { + "targetPopulation": "public", + "assignmentParameters": { + "": "extensionVersion", + "": "language" + } + } +} +``` + +Supported populations are `public`, `insider`, `internal`, and `team`. Select one +explicitly with the owners; an `internal` value is not authentication or proof of +employee status. Version and language bindings are optional. The identity binding +`"devdeviceid": "devDeviceId"` is added by the platform configuration and cannot be +overridden. Generic SDK parameters cannot be overridden. + +The approved full endpoint is: + +```text +https://exp.individual.githubcopilot.com/api/v1/assignments +``` + +The host is currently fixed rather than taken from a Copilot token. Business and +Enterprise networks may block the Individual endpoint; those failures must remain +nonfatal and their telemetry unattributed. Validate attribution coverage before +using extension events in a scorecard. + +The public extension API does not currently expose DevDeviceId. The service accepts +an injected provider so the approved API can be connected without falling back to +MachineId. Until that provider is available, configured initialization fails closed +and makes no TAS request. The legacy MachineId read elsewhere in the service only +namespaces cache data for the SDK's inherited legacy request; it is not the new +assignments identity. + +Absent configuration reports `notConfigured` and makes no TAS requests. Invalid +configuration reports an error and also makes no requests; it does not silently +fall back to a legacy-only integration. + +## Service lifecycle and queries + +The service is initialized after logging is available and is registered in the +extension context's subscriptions. It does not block extension activation on +networking and is not part of the public Python Environments API. + +Future extension-owned experiments can use the internal facade: + +```typescript +const experiments = getExperimentationService(); +await experiments?.initializePromise; +const enabled = experiments?.getTreatmentVariable('approvedFeatureName', false) ?? false; +``` + +Use constant **bare** names in the `vscode` namespace, not `/vscode/` prefixes or +user-provided strings. Boolean, string, and finite-number defaults are supported. +Missing or wrong-typed values use the caller's default. These are synchronous +snapshot queries, not refresh requests. + +- `initializePromise` covers module/cache initialization, with a five-second + deadline. It is not evidence of a successful server request. +- `initialFetch` covers the first attempt, with a fifteen-second upper bound. + Completion is not proof of success either; use the endpoint outcomes below. +- A valid warm cache is usable before networking completes. A cold lookup returns + its default without consuming the SDK's empty snapshot until a fetched snapshot + is committed. A later request failure does not erase an already usable snapshot. +- The SDK normally polls every thirty minutes. Once a treatment is consumed, its + ordinary polls persist newer assignments for later sessions rather than replacing + the consumed in-memory snapshot. The facade does not expose a force-refresh API. +- Consumers still own their feature's startup-versus-dynamic policy. In particular, + a settings-based experiment should read VS Code's effective setting, not add a + second TAS Boolean gate. + +Cache data lives in `context.globalState`, namespaced by the approved configuration, +extension version, resolved assignment parameters, and the SDK's built-in targeting +values: VS Code version, application name, language, and legacy MachineId. The new +assignments identity is DevDeviceId. The namespace is hashed; identifiers and +endpoints are not emitted in diagnostics. This prevents a snapshot from being reused +after its population, endpoint, version, identity, or audience context changes. +Malformed cache data is ignored with a warning. + +Revoking telemetry consent disposes the SDK, aborts outstanding requests, clears +shared attribution, and makes queries use defaults. Re-enabling consent creates a +replacement instance. Old callbacks and new writes from stopped instances are ignored. +Initialization failures are nonfatal and explicitly logged; reload or a consent +transition starts a new attempt. Automated extension-test hosts do not start live TAS. + +The current SDK performs a legacy GET **and** the configured assignments POST. +New-endpoint variables take precedence when both return the same name. The common +HTTPS transport supplies cancellation, a ten-second request deadline, and a two-MiB +response cap to both endpoints. It uses Node HTTPS like the SDK, retaining the +extension host's HTTP hooks; proxy behavior must still be verified in the deployment +environments. There is no insecure TLS or redirect fallback. New experiments must +use the shared GitHub/DevDiv workspace and the assignments POST; the inherited +legacy request is SDK behavior and is not a supported fallback for this extension. + +`tas-client` requires Node 22. TypeScript 5.8 or newer is needed to type-check the +current wrapper's CommonJS-to-ESM declarations without disabling library checking. + +## Telemetry and diagnostics + +The SDK adapter forwards an allowlist of classified events/properties through the +existing sender. `abexp.assignmentcontext` is shared by normal events, error events, +and SDK events, including automatically captured `unhandlederror` exceptions. +It is copied into each event; per-event data cannot override it. +Raw request headers, audience filters, and identifiers are not forwarded. + +Automatic exceptions use an extension-owned VS Code telemetry logger for consent +and data cleaning. The reporter's duplicate automatic collection is disabled using +the SDK's public `ignoreUnhandledErrors` option. The cleaned +exception is labelled at capture time, then forwarded through the reporter's raw +send method with an additional error-consent and lifetime check. This preserves +the existing event name and common properties without bypassing consent or +changing the assignment on events that are already queued. + +The reporter exists only within an active registration. Late completions after +disposal cannot recreate it, and an obsolete disposer cannot end a newer registration. + +| Signal | Meaning | +| --- | --- | +| `EXPERIMENTATION.INITIALIZATION` | `notConfigured`, `cacheReady`, `error`, or `timeout`, with initialization duration and initial cache presence. Consent-disabled instances are observable locally, not by sending telemetry against that choice. | +| `tas-call` | Independent `legacy` or `assignments` request outcome: `Success`, `ServerError`, `NoResponse`, or `GenericError`. | +| `assignments-validation` | Counts and response data version from the assignments provider. | +| `query-expfeature` | A treatment was queried; this is not proof that a feature was enabled, exposed, or used. | +| `call-tas-error`, `call-assignments-error` | Classified fetch failure categories. | + +`service.diagnostics` separates cache readiness/presence, availability of a usable +snapshot, first-attempt completion, and the latest outcomes of each endpoint. +For example, a cached value can remain usable while the latest request reports +`NoResponse`. Output-channel diagnostics do not log treatment values or identities. + +Events sent before attribution is available remain unattributed; the sender does +not retroactively label them using a later assignment. Scorecards must account for +this, especially for early startup events. Validate attribution coverage in A/A; +do not assume that every event is assigned just because initialization completed. + +## Baseline measurement inventory + +Reuse these existing signals before adding new events. Names below are the source +event names; confirm the telemetry ingestion system's normalized names when defining +the scorecard. Only consented, attributed traffic belongs in assignment comparisons. + +| Area | Existing events | Candidate baseline measurements and limitations | +| --- | --- | --- | +| Activation | `EXTENSION.ACTIVATION_DURATION` | Activation-duration distribution and activated-device denominator. Emitted on successful return, so it does not count every early activation failure. | +| Async setup | `EXTENSION.MANAGER_REGISTRATION_DURATION`, `SETUP.HANG_DETECTED`, `MANAGER_READY.TIMEOUT` | Setup duration, failure-stage distribution, hangs and readiness timeouts. Async setup is distinct from activation. | +| Discovery | `ENVIRONMENT_DISCOVERY`, `MANAGER.LAZY_INIT`, `PET.INIT_DURATION`, `PET.CONFIGURE`, `PET.REFRESH`, `PET.RESOLVE` | Duration and error/timeout rates by manager/operation. A refresh is not a unique user or environment; cancellation is grouped with timeout in the manager refresh wrapper. | +| Discovery recovery | `PET.PROCESS_RESTART`, `PET.JSON_CLI_FALLBACK`, `GLOBAL_ENV.CACHE` | Restart/fallback rates and cache hits/stale entries, with attempt versus user denominators kept separate. | +| Selection | `ENV_SELECTION.STARTED`, `ENV_SELECTION.RESULT`, `ENV_SELECTION.COMPLETED` | Selection latency, resolution path and coverage, including deferred global work. | +| Package operations | `PACKAGE_MANAGEMENT` | Success/error/cancellation rates and duration by manager and trigger source. A fulfilled manager operation is not independently verified package state. | +| Environment creation | `CREATE_ENVIRONMENT`, `VENV.CREATION` | Entry-point and quick/custom intent counts. These do not prove successful creation, usability, or an end-to-end completion. | +| Base Python installation | `UV.PYTHON_INSTALL_PROMPTED`, `UV.PYTHON_INSTALL_STARTED`, `UV.PYTHON_INSTALL_COMPLETED`, `UV.PYTHON_INSTALL_FAILED` | Prompt/start/completion/failure counts. Define denominators and cancellation semantics before calling this a conversion funnel. | +| Project/environment usage | `ADD_PROJECT`, `PROJECT_STRUCTURE`, `ENVIRONMENT_TOOL_USAGE` | Adoption and workspace-shape context. Startup observations are not retention by themselves. | + +Gaps to settle with metric owners: + +- An arm-independent, feature-specific exposure/eligibility event for each future + experiment. General treatment queries are not substitutes for actual exposure. +- A consistent end-to-end environment-creation outcome and duration. Existing intent + events cannot be treated as successful creations. +- An agreed activated/eligible-device or session denominator, retention window, + cancellation policy, and attribution-coverage threshold. +- Coverage for failures before successful activation and for user flows spanning + multiple components. Any new correlation data needs privacy review. + +This inventory preserves existing event semantics. + +## Validation and external onboarding + +Unit tests use a fake SDK for lifecycle cases. A separate contract test loads the +installed SDK with a fake transport to verify dual requests, assignment merging, +bare variable names, and shared attribution without contacting TAS. + +Before live use, obtain DevDeviceId access and confirm the audience/population, shared +GitHub/DevDiv workspace group, access, and scorecard with the experimentation owners. +The old workspace and old TAS endpoint are not supported for new experiments. Obtain +the integration and metrics reviews described in the onboarding guidance. An A/A can +validate allocation, attribution, data quality and baseline stability without exposing +a new setting or changing product behavior. A real `tas-call` with +`callType = assignments` and `outcome = Success`, a known new-endpoint assignment, +and tagged subsequent telemetry must all agree; a cached value alone is not proof +that onboarding works. + +Some older checklists still require `vscode.abexp.features`; the current SDK no +longer maintains it. Use the current assignment-context guidance instead. The +documented CodeExpOwners mail alias has been reported unavailable; confirm the +current onboarding and review contacts with the platform owners. + +- [Extension onboarding](https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/23415/Onboard-a-VS-Code-extension) +- [Current TAS integration guidance](https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/23416/Step-1a-Integrate-TAS) diff --git a/package-lock.json b/package-lock.json index 2806ef7c3..1fdcb4813 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,12 +10,13 @@ "dependencies": { "@iarna/toml": "^2.2.5", "@renovatebot/pep440": "^4.2.4", - "@vscode/extension-telemetry": "^0.9.7", + "@vscode/extension-telemetry": "^1.5.2", "@vscode/test-cli": "^0.0.10", "dotenv": "^16.4.5", "fs-extra": "^11.2.0", "stack-trace": "0.0.10", "vscode-jsonrpc": "^9.0.0-next.5", + "vscode-tas-client": "^0.3.3", "which": "^4.0.0" }, "devDependencies": { @@ -38,7 +39,7 @@ "ts-loader": "^9.4.3", "ts-mockito": "^2.6.1", "typemoq": "^2.1.0", - "typescript": "^5.1.3", + "typescript": "^5.8.3", "webpack": "^5.105.0", "webpack-cli": "^5.1.1" }, @@ -606,73 +607,72 @@ } }, "node_modules/@microsoft/1ds-core-js": { - "version": "4.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/1ds-core-js/-/1ds-core-js-4.3.4.tgz", - "integrity": "sha512-3gbDUQgAO8EoyQTNcAEkxpuPnioC0May13P1l1l0NKZ128L9Ts/sj8QsfwCRTjHz0HThlA+4FptcAJXNYUy3rg==", + "version": "4.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/1ds-core-js/-/1ds-core-js-4.4.4.tgz", + "integrity": "sha1-xUJD3qPmdbAoQZbeQE54ZxTMT/g=", "license": "MIT", "dependencies": { - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "node_modules/@microsoft/1ds-post-js": { - "version": "4.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/1ds-post-js/-/1ds-post-js-4.3.4.tgz", - "integrity": "sha512-nlKjWricDj0Tn68Dt0P8lX9a+X7LYrqJ6/iSfQwMfDhRIGLqW+wxx8gxS+iGWC/oc8zMQAeiZaemUpCwQcwpRQ==", + "version": "4.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/1ds-post-js/-/1ds-post-js-4.4.4.tgz", + "integrity": "sha1-WaFHAkhS9nNdZ7+1/OM4Cd9a7x0=", "license": "MIT", "dependencies": { - "@microsoft/1ds-core-js": "4.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "node_modules/@microsoft/applicationinsights-channel-js": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-channel-js/-/applicationinsights-channel-js-3.3.4.tgz", - "integrity": "sha512-Z4nrxYwGKP9iyrYtm7iPQXVOFy4FsEsX0nDKkAi96Qpgw+vEh6NH4ORxMMuES0EollBQ3faJyvYCwckuCVIj0g==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-channel-js/-/applicationinsights-channel-js-3.4.4.tgz", + "integrity": "sha1-Ls+QmUHA4yVD3S3OAE3+LfYqNeU=", "license": "MIT", "dependencies": { - "@microsoft/applicationinsights-common": "3.3.4", - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" }, "peerDependencies": { "tslib": ">= 1.0.0" } }, "node_modules/@microsoft/applicationinsights-common": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-common/-/applicationinsights-common-3.3.4.tgz", - "integrity": "sha512-4ms16MlIvcP4WiUPqopifNxcWCcrXQJ2ADAK/75uok2mNQe6ZNRsqb/P+pvhUxc8A5HRlvoXPP1ptDSN5Girgw==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-common/-/applicationinsights-common-3.4.4.tgz", + "integrity": "sha1-zy7uCxCK/bu8Ak3K2O7Y2wREXhA=", "license": "MIT", "dependencies": { - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" }, "peerDependencies": { "tslib": ">= 1.0.0" } }, "node_modules/@microsoft/applicationinsights-core-js": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-core-js/-/applicationinsights-core-js-3.3.4.tgz", - "integrity": "sha512-MummANF0mgKIkdvVvfmHQTBliK114IZLRhTL0X0Ep+zjDwWMHqYZgew0nlFKAl6ggu42abPZFK5afpE7qjtYJA==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-core-js/-/applicationinsights-core-js-3.4.4.tgz", + "integrity": "sha1-8yehNw34Vcl88sWOJj4djZxJ5Mg=", "license": "MIT", "dependencies": { "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" }, "peerDependencies": { "tslib": ">= 1.0.0" @@ -680,53 +680,52 @@ }, "node_modules/@microsoft/applicationinsights-shims": { "version": "3.0.1", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-shims/-/applicationinsights-shims-3.0.1.tgz", - "integrity": "sha512-DKwboF47H1nb33rSUfjqI6ryX29v+2QWcTrRvcQDA32AZr5Ilkr7whOOSsD1aBzwqX0RJEIP1Z81jfE3NBm/Lg==", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-shims/-/applicationinsights-shims-3.0.1.tgz", + "integrity": "sha1-OGW3Os6EBbnEYYzFxXHy/jh28G8=", "license": "MIT", "dependencies": { "@nevware21/ts-utils": ">= 0.9.4 < 2.x" } }, "node_modules/@microsoft/applicationinsights-web-basic": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-web-basic/-/applicationinsights-web-basic-3.3.4.tgz", - "integrity": "sha512-OpEPXr8vU/t/M8T9jvWJzJx/pCyygIiR1nGM/2PTde0wn7anl71Gxl5fWol7K/WwFEORNjkL3CEyWOyDc+28AA==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-web-basic/-/applicationinsights-web-basic-3.4.4.tgz", + "integrity": "sha1-qwgTikWVAaf/F+0ybS0MdPKAzHc=", "license": "MIT", "dependencies": { - "@microsoft/applicationinsights-channel-js": "3.3.4", - "@microsoft/applicationinsights-common": "3.3.4", - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-channel-js": "3.4.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" }, "peerDependencies": { "tslib": ">= 1.0.0" } }, "node_modules/@microsoft/dynamicproto-js": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/@microsoft/dynamicproto-js/-/dynamicproto-js-2.0.3.tgz", - "integrity": "sha512-JTWTU80rMy3mdxOjjpaiDQsTLZ6YSGGqsjURsY6AUQtIj0udlF/jYmhdLZu8693ZIC0T1IwYnFa0+QeiMnziBA==", + "version": "2.0.5", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/dynamicproto-js/-/dynamicproto-js-2.0.5.tgz", + "integrity": "sha1-9NCLCwbFcNaerXzSq69BQCUITGc=", "license": "MIT", "dependencies": { - "@nevware21/ts-utils": ">= 0.10.4 < 2.x" + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "node_modules/@nevware21/ts-async": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/@nevware21/ts-async/-/ts-async-0.5.3.tgz", - "integrity": "sha512-UsF7eerLsVfid7iV1oXF80qXBwHNBeqSqfh/nPZgirRU1MACmSsj83EZKS2ViFHVfSGG6WIuXMGBP6KciXfYhA==", + "version": "0.5.5", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@nevware21/ts-async/-/ts-async-0.5.5.tgz", + "integrity": "sha1-UJZI48PDqpq2E8AMce7k/pvYo8M=", "license": "MIT", "dependencies": { - "@nevware21/ts-utils": ">= 0.11.5 < 2.x" + "@nevware21/ts-utils": ">= 0.12.2 < 2.x" } }, "node_modules/@nevware21/ts-utils": { - "version": "0.14.0", - "resolved": "https://registry.npmjs.org/@nevware21/ts-utils/-/ts-utils-0.14.0.tgz", - "integrity": "sha512-WoeqTIXQ8WPhl+lD2NbMHoAQ4sJl0n7EoRoDmVJui//Usg512enl9q1fdbVobuZt3omnxnmVsDrNIvPBvFgddQ==", + "version": "0.16.0", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@nevware21/ts-utils/-/ts-utils-0.16.0.tgz", + "integrity": "sha1-5KOcjrEi54ESMVx5oAkWI4t/PEw=", "funding": [ { "type": "github", @@ -736,7 +735,8 @@ "type": "other", "url": "https://buymeacoffee.com/nevware21" } - ] + ], + "license": "MIT" }, "node_modules/@nodelib/fs.scandir": { "version": "2.1.5", @@ -1519,14 +1519,16 @@ } }, "node_modules/@vscode/extension-telemetry": { - "version": "0.9.7", - "resolved": "https://registry.npmjs.org/@vscode/extension-telemetry/-/extension-telemetry-0.9.7.tgz", - "integrity": "sha512-2GQbcfDUTg0QC1v0HefkHNwYrE5LYKzS3Zb0+uA6Qn1MBDzgiSh23ddOZF/JRqhqBFOG0mE70XslKSGQ5v9KwQ==", + "version": "1.5.2", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@vscode/extension-telemetry/-/extension-telemetry-1.5.2.tgz", + "integrity": "sha1-KsL23kVWWOJgGA/l6r9zvNbaUUY=", "license": "MIT", "dependencies": { - "@microsoft/1ds-core-js": "^4.3.0", - "@microsoft/1ds-post-js": "^4.3.0", - "@microsoft/applicationinsights-web-basic": "^3.3.0" + "@microsoft/1ds-core-js": "^4.4.1", + "@microsoft/1ds-post-js": "^4.4.1", + "@microsoft/applicationinsights-common": "^3.4.1", + "@microsoft/applicationinsights-core-js": "^3.4.1", + "@microsoft/applicationinsights-web-basic": "^3.4.1" }, "engines": { "vscode": "^1.75.0" @@ -6738,6 +6740,15 @@ "node": ">= 6" } }, + "node_modules/tas-client": { + "version": "0.4.6", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/tas-client/-/tas-client-0.4.6.tgz", + "integrity": "sha1-HkdkngHPE02WM4flsbf3OjgXJ8I=", + "license": "MIT", + "engines": { + "node": ">=22" + } + }, "node_modules/terminal-link": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/terminal-link/-/terminal-link-4.0.0.tgz", @@ -7012,10 +7023,11 @@ } }, "node_modules/typescript": { - "version": "5.3.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.3.3.tgz", - "integrity": "sha512-pXWcraxM0uxAS+tN0AG/BF2TyqmHO014Z070UsJ+pFvYuRSq8KH8DmWpnbXe0pEPDHXZV3FcAbJkijJ5oNEnWw==", + "version": "5.9.3", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha1-W09Z4VMQqxeiFvXWz1PuR27eZw8=", "dev": true, + "license": "Apache-2.0", "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -7158,6 +7170,18 @@ "node": ">=14.0.0" } }, + "node_modules/vscode-tas-client": { + "version": "0.3.3", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/vscode-tas-client/-/vscode-tas-client-0.3.3.tgz", + "integrity": "sha1-wNaaNp8vARLk2hz7VbmYkSba6Jg=", + "license": "MIT", + "dependencies": { + "tas-client": "^0.4.6" + }, + "engines": { + "vscode": "^1.85.0" + } + }, "node_modules/watchpack": { "version": "2.5.1", "resolved": "https://registry.npmjs.org/watchpack/-/watchpack-2.5.1.tgz", @@ -7887,106 +7911,104 @@ } }, "@microsoft/1ds-core-js": { - "version": "4.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/1ds-core-js/-/1ds-core-js-4.3.4.tgz", - "integrity": "sha512-3gbDUQgAO8EoyQTNcAEkxpuPnioC0May13P1l1l0NKZ128L9Ts/sj8QsfwCRTjHz0HThlA+4FptcAJXNYUy3rg==", + "version": "4.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/1ds-core-js/-/1ds-core-js-4.4.4.tgz", + "integrity": "sha1-xUJD3qPmdbAoQZbeQE54ZxTMT/g=", "requires": { - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/1ds-post-js": { - "version": "4.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/1ds-post-js/-/1ds-post-js-4.3.4.tgz", - "integrity": "sha512-nlKjWricDj0Tn68Dt0P8lX9a+X7LYrqJ6/iSfQwMfDhRIGLqW+wxx8gxS+iGWC/oc8zMQAeiZaemUpCwQcwpRQ==", + "version": "4.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/1ds-post-js/-/1ds-post-js-4.4.4.tgz", + "integrity": "sha1-WaFHAkhS9nNdZ7+1/OM4Cd9a7x0=", "requires": { - "@microsoft/1ds-core-js": "4.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/applicationinsights-channel-js": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-channel-js/-/applicationinsights-channel-js-3.3.4.tgz", - "integrity": "sha512-Z4nrxYwGKP9iyrYtm7iPQXVOFy4FsEsX0nDKkAi96Qpgw+vEh6NH4ORxMMuES0EollBQ3faJyvYCwckuCVIj0g==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-channel-js/-/applicationinsights-channel-js-3.4.4.tgz", + "integrity": "sha1-Ls+QmUHA4yVD3S3OAE3+LfYqNeU=", "requires": { - "@microsoft/applicationinsights-common": "3.3.4", - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/applicationinsights-common": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-common/-/applicationinsights-common-3.3.4.tgz", - "integrity": "sha512-4ms16MlIvcP4WiUPqopifNxcWCcrXQJ2ADAK/75uok2mNQe6ZNRsqb/P+pvhUxc8A5HRlvoXPP1ptDSN5Girgw==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-common/-/applicationinsights-common-3.4.4.tgz", + "integrity": "sha1-zy7uCxCK/bu8Ak3K2O7Y2wREXhA=", "requires": { - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/applicationinsights-core-js": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-core-js/-/applicationinsights-core-js-3.3.4.tgz", - "integrity": "sha512-MummANF0mgKIkdvVvfmHQTBliK114IZLRhTL0X0Ep+zjDwWMHqYZgew0nlFKAl6ggu42abPZFK5afpE7qjtYJA==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-core-js/-/applicationinsights-core-js-3.4.4.tgz", + "integrity": "sha1-8yehNw34Vcl88sWOJj4djZxJ5Mg=", "requires": { "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/applicationinsights-shims": { "version": "3.0.1", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-shims/-/applicationinsights-shims-3.0.1.tgz", - "integrity": "sha512-DKwboF47H1nb33rSUfjqI6ryX29v+2QWcTrRvcQDA32AZr5Ilkr7whOOSsD1aBzwqX0RJEIP1Z81jfE3NBm/Lg==", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-shims/-/applicationinsights-shims-3.0.1.tgz", + "integrity": "sha1-OGW3Os6EBbnEYYzFxXHy/jh28G8=", "requires": { "@nevware21/ts-utils": ">= 0.9.4 < 2.x" } }, "@microsoft/applicationinsights-web-basic": { - "version": "3.3.4", - "resolved": "https://registry.npmjs.org/@microsoft/applicationinsights-web-basic/-/applicationinsights-web-basic-3.3.4.tgz", - "integrity": "sha512-OpEPXr8vU/t/M8T9jvWJzJx/pCyygIiR1nGM/2PTde0wn7anl71Gxl5fWol7K/WwFEORNjkL3CEyWOyDc+28AA==", + "version": "3.4.4", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/applicationinsights-web-basic/-/applicationinsights-web-basic-3.4.4.tgz", + "integrity": "sha1-qwgTikWVAaf/F+0ybS0MdPKAzHc=", "requires": { - "@microsoft/applicationinsights-channel-js": "3.3.4", - "@microsoft/applicationinsights-common": "3.3.4", - "@microsoft/applicationinsights-core-js": "3.3.4", + "@microsoft/applicationinsights-channel-js": "3.4.4", + "@microsoft/applicationinsights-core-js": "3.4.4", "@microsoft/applicationinsights-shims": "3.0.1", "@microsoft/dynamicproto-js": "^2.0.3", - "@nevware21/ts-async": ">= 0.5.2 < 2.x", - "@nevware21/ts-utils": ">= 0.11.3 < 2.x" + "@nevware21/ts-async": ">= 0.5.5 < 0.6.0", + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@microsoft/dynamicproto-js": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/@microsoft/dynamicproto-js/-/dynamicproto-js-2.0.3.tgz", - "integrity": "sha512-JTWTU80rMy3mdxOjjpaiDQsTLZ6YSGGqsjURsY6AUQtIj0udlF/jYmhdLZu8693ZIC0T1IwYnFa0+QeiMnziBA==", + "version": "2.0.5", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@microsoft/dynamicproto-js/-/dynamicproto-js-2.0.5.tgz", + "integrity": "sha1-9NCLCwbFcNaerXzSq69BQCUITGc=", "requires": { - "@nevware21/ts-utils": ">= 0.10.4 < 2.x" + "@nevware21/ts-utils": ">= 0.14.0 < 2.x" } }, "@nevware21/ts-async": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/@nevware21/ts-async/-/ts-async-0.5.3.tgz", - "integrity": "sha512-UsF7eerLsVfid7iV1oXF80qXBwHNBeqSqfh/nPZgirRU1MACmSsj83EZKS2ViFHVfSGG6WIuXMGBP6KciXfYhA==", + "version": "0.5.5", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@nevware21/ts-async/-/ts-async-0.5.5.tgz", + "integrity": "sha1-UJZI48PDqpq2E8AMce7k/pvYo8M=", "requires": { - "@nevware21/ts-utils": ">= 0.11.5 < 2.x" + "@nevware21/ts-utils": ">= 0.12.2 < 2.x" } }, "@nevware21/ts-utils": { - "version": "0.14.0", - "resolved": "https://registry.npmjs.org/@nevware21/ts-utils/-/ts-utils-0.14.0.tgz", - "integrity": "sha512-WoeqTIXQ8WPhl+lD2NbMHoAQ4sJl0n7EoRoDmVJui//Usg512enl9q1fdbVobuZt3omnxnmVsDrNIvPBvFgddQ==" + "version": "0.16.0", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@nevware21/ts-utils/-/ts-utils-0.16.0.tgz", + "integrity": "sha1-5KOcjrEi54ESMVx5oAkWI4t/PEw=" }, "@nodelib/fs.scandir": { "version": "2.1.5", @@ -8569,13 +8591,15 @@ } }, "@vscode/extension-telemetry": { - "version": "0.9.7", - "resolved": "https://registry.npmjs.org/@vscode/extension-telemetry/-/extension-telemetry-0.9.7.tgz", - "integrity": "sha512-2GQbcfDUTg0QC1v0HefkHNwYrE5LYKzS3Zb0+uA6Qn1MBDzgiSh23ddOZF/JRqhqBFOG0mE70XslKSGQ5v9KwQ==", + "version": "1.5.2", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@vscode/extension-telemetry/-/extension-telemetry-1.5.2.tgz", + "integrity": "sha1-KsL23kVWWOJgGA/l6r9zvNbaUUY=", "requires": { - "@microsoft/1ds-core-js": "^4.3.0", - "@microsoft/1ds-post-js": "^4.3.0", - "@microsoft/applicationinsights-web-basic": "^3.3.0" + "@microsoft/1ds-core-js": "^4.4.1", + "@microsoft/1ds-post-js": "^4.4.1", + "@microsoft/applicationinsights-common": "^3.4.1", + "@microsoft/applicationinsights-core-js": "^3.4.1", + "@microsoft/applicationinsights-web-basic": "^3.4.1" } }, "@vscode/test-cli": { @@ -12255,6 +12279,11 @@ } } }, + "tas-client": { + "version": "0.4.6", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/tas-client/-/tas-client-0.4.6.tgz", + "integrity": "sha1-HkdkngHPE02WM4flsbf3OjgXJ8I=" + }, "terminal-link": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/terminal-link/-/terminal-link-4.0.0.tgz", @@ -12437,9 +12466,9 @@ } }, "typescript": { - "version": "5.3.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.3.3.tgz", - "integrity": "sha512-pXWcraxM0uxAS+tN0AG/BF2TyqmHO014Z070UsJ+pFvYuRSq8KH8DmWpnbXe0pEPDHXZV3FcAbJkijJ5oNEnWw==", + "version": "5.9.3", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha1-W09Z4VMQqxeiFvXWz1PuR27eZw8=", "dev": true }, "uc.micro": { @@ -12533,6 +12562,14 @@ "resolved": "https://registry.npmjs.org/vscode-jsonrpc/-/vscode-jsonrpc-9.0.0-next.6.tgz", "integrity": "sha512-KCSvUNsFiVciG9iqjJKBZOd66CN3ZKohDlYRmoOi+pd8l15MFLZ8wRG4c+wuzePGba/8WcCG2TM+C/GVlvuaeA==" }, + "vscode-tas-client": { + "version": "0.3.3", + "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/vscode-tas-client/-/vscode-tas-client-0.3.3.tgz", + "integrity": "sha1-wNaaNp8vARLk2hz7VbmYkSba6Jg=", + "requires": { + "tas-client": "^0.4.6" + } + }, "watchpack": { "version": "2.5.1", "resolved": "https://registry.npmjs.org/watchpack/-/watchpack-2.5.1.tgz", diff --git a/package.json b/package.json index 561157277..6a367b835 100644 --- a/package.json +++ b/package.json @@ -752,19 +752,20 @@ "ts-loader": "^9.4.3", "ts-mockito": "^2.6.1", "typemoq": "^2.1.0", - "typescript": "^5.1.3", + "typescript": "^5.8.3", "webpack": "^5.105.0", "webpack-cli": "^5.1.1" }, "dependencies": { "@iarna/toml": "^2.2.5", "@renovatebot/pep440": "^4.2.4", - "@vscode/extension-telemetry": "^0.9.7", + "@vscode/extension-telemetry": "^1.5.2", "@vscode/test-cli": "^0.0.10", "dotenv": "^16.4.5", "fs-extra": "^11.2.0", "stack-trace": "0.0.10", "vscode-jsonrpc": "^9.0.0-next.5", + "vscode-tas-client": "^0.3.3", "which": "^4.0.0" }, "overrides": { diff --git a/src/common/env.apis.ts b/src/common/env.apis.ts index 21369eded..be202130b 100644 --- a/src/common/env.apis.ts +++ b/src/common/env.apis.ts @@ -1,4 +1,5 @@ -import { env, Uri } from 'vscode'; +import { Disposable, env, Uri, version } from 'vscode'; +import type { TelemetryLogger, TelemetryLoggerOptions, TelemetrySender } from 'vscode'; export function launchBrowser(uri: string | Uri): Thenable { return env.openExternal(uri instanceof Uri ? uri : Uri.parse(uri)); @@ -7,3 +8,38 @@ export function launchBrowser(uri: string | Uri): Thenable { export function clipboardWriteText(text: string): Thenable { return env.clipboard.writeText(text); } + +/** Read VS Code's usage-telemetry consent. */ +export function isTelemetryEnabled(): boolean { + return env.isTelemetryEnabled === true; +} + +/** Subscribe to usage-telemetry consent changes. */ +export function onDidChangeTelemetryEnabled(listener: (enabled: boolean) => void): Disposable { + return env.onDidChangeTelemetryEnabled(listener); +} + +/** Read the legacy MachineId targeting value that vscode-tas-client adds automatically. */ +export function getMachineId(): string { + return env.machineId; +} + +/** Read the display language for an approved audience binding. */ +export function getLanguage(): string { + return env.language; +} + +/** Read the VS Code application version used by built-in experimentation targeting. */ +export function getVSCodeVersion(): string { + return version; +} + +/** Read the VS Code application name used by built-in experimentation targeting. */ +export function getAppName(): string { + return env.appName; +} + +/** Create a VS Code logger that applies telemetry consent and data cleaning. */ +export function createTelemetryLogger(sender: TelemetrySender, options?: TelemetryLoggerOptions): TelemetryLogger { + return env.createTelemetryLogger(sender, options); +} diff --git a/src/common/experimentation/configuration.ts b/src/common/experimentation/configuration.ts new file mode 100644 index 000000000..d49eb1356 --- /dev/null +++ b/src/common/experimentation/configuration.ts @@ -0,0 +1,87 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +export type ExperimentationPopulation = 'public' | 'insider' | 'internal' | 'team'; +export type AssignmentParameterSource = 'devDeviceId' | 'extensionVersion' | 'language'; + +export const EXPERIMENTATION_ASSIGNMENTS_ENDPOINT = + 'https://exp.individual.githubcopilot.com/api/v1/assignments'; +export const EXPERIMENTATION_IDENTITY_PARAMETER = 'devdeviceid'; + +export interface ExperimentationConfiguration { + readonly assignmentsEndpoint: string; + readonly targetPopulation: ExperimentationPopulation; + readonly identityParameter: string; + readonly assignmentParameters: Readonly>; +} + +const GENERIC_PARAMETERS = new Set([ + EXPERIMENTATION_IDENTITY_PARAMETER, + 'vscode_core_appversion', + 'vscode_core_build', + 'vscode_core_extensionname', + 'extensionname', + 'vscode_core_targetpopulation', +]); + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +/** Read publisher configuration; return undefined if absent and throw if invalid. */ +export function readExperimentationConfiguration(manifest: unknown): ExperimentationConfiguration | undefined { + if (!isRecord(manifest) || manifest.experimentation === undefined) { + return undefined; + } + const config = manifest.experimentation; + if (!isRecord(config)) { + throw new Error('The experimentation manifest entry must be an object.'); + } + if (config.assignmentsEndpoint !== undefined || config.identityParameter !== undefined) { + throw new Error( + 'The experimentation endpoint and identity parameter are platform-owned and cannot be overridden.', + ); + } + const population = config.targetPopulation; + if (population !== 'public' && population !== 'insider' && population !== 'internal' && population !== 'team') { + throw new Error('Experimentation requires an explicitly approved targetPopulation.'); + } + if (config.assignmentParameters !== undefined && !isRecord(config.assignmentParameters)) { + throw new Error('Experimentation requires approved assignment parameter bindings.'); + } + + const parameters: Record = { + [EXPERIMENTATION_IDENTITY_PARAMETER]: 'devDeviceId', + }; + for (const [name, source] of Object.entries(config.assignmentParameters ?? {})) { + if (!/^[A-Za-z][A-Za-z0-9_.-]*$/.test(name) || /^x-/i.test(name) || GENERIC_PARAMETERS.has(name)) { + throw new Error( + 'Assignment parameters must use new API names and must not replace generic SDK parameters.', + ); + } + if (source !== 'extensionVersion' && source !== 'language') { + throw new Error( + 'Unsupported assignment parameter source. The DevDeviceId binding is platform-owned.', + ); + } + parameters[name] = source; + } + if (Object.keys(parameters).length > 45) { + throw new Error('Experimentation supports at most 45 assignment parameters.'); + } + + return { + assignmentsEndpoint: EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + targetPopulation: population, + identityParameter: EXPERIMENTATION_IDENTITY_PARAMETER, + assignmentParameters: Object.freeze(parameters), + }; +} + +/** Read and validate the extension version from its manifest. */ +export function getExperimentationExtensionVersion(manifest: unknown): string { + if (!isRecord(manifest) || typeof manifest.version !== 'string' || !manifest.version.trim()) { + throw new Error('Experimentation requires the extension version.'); + } + return manifest.version; +} diff --git a/src/common/experimentation/service.ts b/src/common/experimentation/service.ts new file mode 100644 index 000000000..f761d04cd --- /dev/null +++ b/src/common/experimentation/service.ts @@ -0,0 +1,451 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import type { Disposable, Memento } from 'vscode'; +import type { ExperimentationConfig, IExperimentationService } from 'vscode-tas-client'; +import { ENVS_EXTENSION_ID } from '../constants'; +import { + getAppName, + getLanguage, + getMachineId, + getVSCodeVersion, + isTelemetryEnabled, + onDidChangeTelemetryEnabled, +} from '../env.apis'; +import { traceError, traceInfo, traceVerbose, traceWarn } from '../logging'; +import { EventNames } from '../telemetry/constants'; +import { setSharedTelemetryProperty } from '../telemetry/reporter'; +import { sendTelemetryEvent } from '../telemetry/sender'; +import { createDeferred, Deferred } from '../utils/deferred'; +import { + ExperimentationConfiguration, + ExperimentationPopulation, + getExperimentationExtensionVersion, + readExperimentationConfiguration, +} from './configuration'; +import { ExperimentationStorage } from './storage'; +import { ASSIGNMENT_CONTEXT_PROPERTY, ExperimentationTelemetry, TasCall } from './telemetry'; +import { createExperimentationFetch } from './transport'; + +export const EXPERIMENTATION_INITIALIZATION_TIMEOUT_MS = 5_000; +const INITIAL_FETCH_TIMEOUT_MS = 15_000; +type TreatmentValue = boolean | number | string; + +function trimVersionSuffix(version: string): string { + return version.split(/-[a-zA-Z0-9]+$/)[0]; +} + +export type ExperimentationClient = Pick< + IExperimentationService, + 'initializePromise' | 'initialFetch' | 'getTreatmentVariable' | 'dispose' +>; +export type ExperimentationClientOptions = Omit & { + targetPopulation: ExperimentationPopulation; +}; +export type ExperimentationClientFactory = ( + options: ExperimentationClientOptions, +) => ExperimentationClient | Promise; +export type DevDeviceIdProvider = () => string | undefined; + +export interface ExperimentationContext { + readonly extension: { readonly packageJSON: unknown }; + readonly globalState: Memento; +} + +export interface ExperimentationDiagnostics { + readonly state: 'notConfigured' | 'disabled' | 'initializing' | 'ready' | 'failed' | 'disposed'; + readonly cacheState: 'present' | 'absent' | 'unknown'; + readonly hasUsableSnapshot: boolean; + readonly initialFetch: 'notStarted' | 'pending' | 'completed' | 'failed' | 'timeout'; + readonly legacyFetch: 'notObserved' | TasCall['outcome']; + readonly assignmentsFetch: 'notObserved' | TasCall['outcome']; +} + +interface SdkRun { + active: boolean; + readonly controller: AbortController; + readonly initialized: Deferred; + readonly fetched: Deferred; + readonly startedAt: number; + client?: ExperimentationClient; + initializationTimer?: ReturnType; + fetchTimer?: ReturnType; +} + +async function createSdk(options: ExperimentationClientOptions): Promise { + const sdk = await import('vscode-tas-client'); + const populations = { + public: sdk.TargetPopulation.Public, + insider: sdk.TargetPopulation.Insiders, + internal: sdk.TargetPopulation.Internal, + team: sdk.TargetPopulation.Team, + }; + return sdk.getExperimentationServiceFromConfig({ + ...options, + targetPopulation: populations[options.targetPopulation], + }); +} + +/** Own a consent-scoped SDK and expose defaulted snapshot queries. */ +export class ExperimentationService implements Disposable { + private readonly configuration?: ExperimentationConfiguration; + private readonly version: string = ''; + private readonly consentSubscription?: Disposable; + private run?: SdkRun; + private disposed = false; + private consent = false; + private readonly warnedQueries = new Set(); + private snapshot: ExperimentationDiagnostics = { + state: 'notConfigured', + cacheState: 'unknown', + hasUsableSnapshot: false, + initialFetch: 'notStarted', + legacyFetch: 'notObserved', + assignmentsFetch: 'notObserved', + }; + + constructor( + private readonly context: ExperimentationContext, + private readonly createClient: ExperimentationClientFactory = createSdk, + private readonly getDevDeviceId: DevDeviceIdProvider = () => undefined, + disabledForTests = false, + ) { + try { + this.configuration = readExperimentationConfiguration(context.extension.packageJSON); + if (!this.configuration) { + traceInfo( + '[experimentation] Not configured; no TAS requests will be made. See docs/experimentation.md.', + ); + this.reportInitialization('notConfigured', 0); + return; + } + this.version = getExperimentationExtensionVersion(context.extension.packageJSON); + if (disabledForTests) { + this.snapshot = { ...this.snapshot, state: 'disabled' }; + traceVerbose('[experimentation] Live TAS disabled in automated extension tests.'); + return; + } + this.consent = isTelemetryEnabled(); + this.consentSubscription = onDidChangeTelemetryEnabled((enabled) => { + if (enabled === this.consent) { + return; + } + this.consent = enabled; + this.restart(); + }); + this.restart(); + } catch (error) { + this.snapshot = { ...this.snapshot, state: 'failed' }; + traceError( + '[experimentation] Invalid configuration or unavailable consent API; no TAS requests started:', + error, + ); + this.reportInitialization('error', 0); + } + } + + /** Settle after cache initialization or bounded failure, not network success. */ + public get initializePromise(): Promise { + return this.run?.initialized.promise ?? Promise.resolve(); + } + + /** Settle after the first fetch attempt; diagnostics report its outcome. */ + public get initialFetch(): Promise { + return this.run?.fetched.promise ?? Promise.resolve(); + } + + /** Copy cache readiness and endpoint outcomes. */ + public get diagnostics(): ExperimentationDiagnostics { + return { ...this.snapshot }; + } + + /** Read a bare vscode treatment name, using the default if unavailable or mistyped. */ + public getTreatmentVariable(name: string, defaultValue: boolean): boolean; + public getTreatmentVariable(name: string, defaultValue: number): number; + public getTreatmentVariable(name: string, defaultValue: string): string; + public getTreatmentVariable(name: string, defaultValue: TreatmentValue): TreatmentValue { + if (!/^[A-Za-z][A-Za-z0-9_.-]{0,255}$/.test(name)) { + this.warnQuery(name, 'Invalid treatment name; use a constant bare variable name.'); + return defaultValue; + } + const run = this.run; + if (!run?.client || !this.isActive(run) || this.snapshot.state !== 'ready') { + return defaultValue; + } + if (!this.snapshot.hasUsableSnapshot) { + // Do not consume the SDK's empty snapshot before a cold fetch commits. + return defaultValue; + } + try { + const value = run.client.getTreatmentVariable('vscode', name); + if (value === undefined) { + return defaultValue; + } + if (typeof value !== typeof defaultValue || (typeof value === 'number' && !Number.isFinite(value))) { + this.warnQuery(name, 'Treatment type does not match its default.'); + return defaultValue; + } + return value; + } catch (error) { + this.warnQuery(name, 'Treatment lookup failed; using its default.'); + traceVerbose('[experimentation] Treatment lookup error:', error); + return defaultValue; + } + } + + /** Stop requests and polling, invalidate callbacks, and clear attribution. */ + public dispose(): void { + if (this.disposed) { + return; + } + this.disposed = true; + this.consentSubscription?.dispose(); + this.stopRun(); + this.snapshot = { ...this.snapshot, state: 'disposed' }; + } + + private restart(): void { + if (this.disposed || !this.configuration) { + return; + } + this.stopRun(); + this.snapshot = { + state: this.consent ? 'initializing' : 'disabled', + cacheState: 'unknown', + hasUsableSnapshot: false, + initialFetch: 'notStarted', + legacyFetch: 'notObserved', + assignmentsFetch: 'notObserved', + }; + if (!this.consent) { + traceVerbose('[experimentation] Telemetry consent disabled; TAS stopped.'); + return; + } + const run: SdkRun = { + active: true, + controller: new AbortController(), + initialized: createDeferred(), + fetched: createDeferred(), + startedAt: Date.now(), + }; + this.run = run; + run.initializationTimer = setTimeout( + () => this.failInitialization(run, 'timeout'), + EXPERIMENTATION_INITIALIZATION_TIMEOUT_MS, + ); + void this.initialize(run).catch((error) => { + if (this.isActive(run)) { + traceError('[experimentation] SDK initialization failed:', error); + this.failInitialization(run, 'error'); + } + }); + } + + private async initialize(run: SdkRun): Promise { + const configuration = this.configuration!; + const identity = this.getDevDeviceId(); + if (typeof identity !== 'string' || !identity.trim()) { + throw new Error('The approved DevDeviceId experimentation identity is unavailable.'); + } + const language = getLanguage(); + const values = { devDeviceId: identity, extensionVersion: this.version, language }; + const parameters = new Map(); + for (const [name, source] of Object.entries(configuration.assignmentParameters)) { + const value = values[source]; + if (typeof value !== 'string' || !value.trim()) { + throw new Error('A configured experimentation audience value is unavailable.'); + } + parameters.set(name, value); + } + // The SDK adds these filters independently and merges both endpoint responses into one cached snapshot. + const sdkTargetingValues = { + applicationVersion: trimVersionSuffix(getVSCodeVersion()), + build: getAppName(), + clientId: getMachineId(), + language, + }; + const storage = new ExperimentationStorage( + this.context.globalState, + configuration, + parameters, + sdkTargetingValues, + this.version, + () => this.isActive(run), + ); + const cached = storage.hasCachedAssignments(); + this.snapshot = { ...this.snapshot, cacheState: cached ? 'present' : 'absent', hasUsableSnapshot: cached }; + const telemetry = new ExperimentationTelemetry( + () => this.isActive(run), + (call) => { + this.snapshot = { + ...this.snapshot, + ...(call.callType === 'assignments' + ? { assignmentsFetch: call.outcome } + : { legacyFetch: call.outcome }), + }; + }, + () => { + // A provider success is not yet a committed SDK snapshot. + if (this.snapshot.assignmentsFetch === 'Success' || this.snapshot.legacyFetch === 'Success') { + this.snapshot = { ...this.snapshot, hasUsableSnapshot: true }; + } + }, + ); + const fetch = createExperimentationFetch(run.controller.signal); + this.snapshot = { ...this.snapshot, initialFetch: 'pending' }; + const client = await this.createClient({ + extensionName: ENVS_EXTENSION_ID, + extensionVersion: this.version, + targetPopulation: configuration.targetPopulation, + memento: storage, + telemetry, + assignmentsEndpoint: configuration.assignmentsEndpoint, + assignmentsFilterProviders: [{ getFilters: () => new Map(parameters) }], + fetch: (url, init) => { + if (!this.isActive(run)) { + return Promise.reject(new Error('Experimentation request cancelled.')); + } + return fetch(url, init); + }, + }); + if (!this.isActive(run)) { + client.dispose(); + return; + } + run.client = client; + this.observeInitialFetch(run, client); + await client.initializePromise; + if (!this.isActive(run)) { + return; + } + clearTimeout(run.initializationTimer); + this.snapshot = { ...this.snapshot, state: 'ready' }; + traceVerbose( + `[experimentation] Cache initialization complete (${this.snapshot.cacheState}); ` + + 'this does not prove a successful fetch.', + ); + this.reportInitialization('cacheReady', Date.now() - run.startedAt); + run.initialized.resolve(); + } + + private observeInitialFetch(run: SdkRun, client: ExperimentationClient): void { + const complete = (result: 'completed' | 'failed' | 'timeout') => { + if (!this.isActive(run) || run.fetched.completed) { + return; + } + clearTimeout(run.fetchTimer); + this.snapshot = { ...this.snapshot, initialFetch: result }; + if (result !== 'completed' || this.snapshot.assignmentsFetch !== 'Success') { + traceWarn( + '[experimentation] Initial fetch did not confirm assignments API success; inspect endpoint outcomes.', + ); + } else { + traceVerbose('[experimentation] Initial assignments API fetch confirmed successful.'); + } + run.fetched.resolve(); + if (result === 'timeout') { + this.stopRun(); + this.snapshot = { ...this.snapshot, state: 'failed' }; + } + }; + run.fetchTimer = setTimeout(() => complete('timeout'), INITIAL_FETCH_TIMEOUT_MS); + void client.initialFetch.then( + () => complete('completed'), + (error) => { + if (this.isActive(run)) { + traceWarn('[experimentation] Initial fetch rejected:', error); + } + complete('failed'); + }, + ); + } + + private isActive(run: SdkRun): boolean { + return !this.disposed && run.active && this.run === run && this.consent && isTelemetryEnabled(); + } + + private failInitialization(run: SdkRun, result: 'error' | 'timeout'): void { + if (!this.isActive(run)) { + return; + } + traceWarn(`[experimentation] Initialization ${result}; treatment queries will use their defaults.`); + this.stopRun(); + this.snapshot = { + ...this.snapshot, + state: 'failed', + initialFetch: + this.snapshot.initialFetch === 'pending' + ? result === 'timeout' ? 'timeout' : 'failed' + : this.snapshot.initialFetch, + }; + this.reportInitialization(result, Date.now() - run.startedAt); + } + + private stopRun(): void { + const run = this.run; + if (run) { + run.active = false; + clearTimeout(run.initializationTimer); + clearTimeout(run.fetchTimer); + run.controller.abort(); + try { + run.client?.dispose(); + } catch (error) { + traceWarn('[experimentation] Failed to dispose the SDK:', error); + } + run.initialized.resolve(); + run.fetched.resolve(); + this.run = undefined; + } + setSharedTelemetryProperty(ASSIGNMENT_CONTEXT_PROPERTY, undefined); + this.snapshot = { ...this.snapshot, hasUsableSnapshot: false }; + } + + private warnQuery(name: string, message: string): void { + if (!this.warnedQueries.has(name)) { + this.warnedQueries.add(name); + traceWarn(`[experimentation] ${message}`); + } + } + + private reportInitialization( + result: 'notConfigured' | 'disabled' | 'cacheReady' | 'error' | 'timeout', + duration: number, + ): void { + sendTelemetryEvent(EventNames.EXPERIMENTATION_INITIALIZATION, duration, { + result, + cacheState: this.snapshot.cacheState, + }); + } +} + +let activeService: ExperimentationService | undefined; + +/** Register the activation's internal service without awaiting networking. */ +export function initializeExperimentation( + context: ExperimentationContext & { subscriptions: Disposable[] }, + getDevDeviceId?: DevDeviceIdProvider, +): ExperimentationService { + if (activeService) { + return activeService; + } + const testExecution = [ + 'VSC_PYTHON_CI_TEST', 'VSC_PYTHON_INTEGRATION_TEST', 'VSC_PYTHON_SMOKE_TEST', 'VSC_PYTHON_E2E_TEST', + ].some((name) => !!process.env[name]); + const service = new ExperimentationService(context, createSdk, getDevDeviceId, testExecution); + activeService = service; + context.subscriptions.push({ + dispose: () => { + service.dispose(); + if (activeService === service) { + activeService = undefined; + } + }, + }); + return service; +} + +/** Get the activation's internal experimentation service. */ +export function getExperimentationService(): ExperimentationService | undefined { + return activeService; +} diff --git a/src/common/experimentation/storage.ts b/src/common/experimentation/storage.ts new file mode 100644 index 000000000..e2751c57f --- /dev/null +++ b/src/common/experimentation/storage.ts @@ -0,0 +1,99 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import { createHash } from 'crypto'; +import type { Memento } from 'vscode'; +import { traceWarn } from '../logging'; +import type { ExperimentationConfiguration } from './configuration'; + +export const TAS_CACHE_KEY = 'VSCode.ABExp.FeatureData'; + +export interface SdkTargetingValues { + readonly applicationVersion: string; + readonly build: string; + readonly clientId: string; + readonly language: string; +} + +function isCacheData(value: unknown): boolean { + if (typeof value !== 'object' || value === null) { + return false; + } + const cache = value as Record; + return ( + Array.isArray(cache.features) && + cache.features.every((feature) => typeof feature === 'string') && + typeof cache.assignmentContext === 'string' && + Array.isArray(cache.configs) && + cache.configs.every((config: unknown) => { + if (typeof config !== 'object' || config === null) { + return false; + } + const entry = config as Record; + return typeof entry.Id === 'string' && + typeof entry.Parameters === 'object' && entry.Parameters !== null && !Array.isArray(entry.Parameters); + }) + ); +} + +/** Scope globalState assignments by configuration and resolved targeting values, ignoring stopped SDK writes. */ +export class ExperimentationStorage implements Memento { + private readonly prefix: string; + private reportedInvalidCache = false; + + constructor( + private readonly storage: Memento, + configuration: ExperimentationConfiguration, + resolvedAssignmentParameters: ReadonlyMap, + sdkTargetingValues: SdkTargetingValues, + extensionVersion: string, + private readonly isActive: () => boolean, + ) { + const namespace = JSON.stringify([ + configuration.assignmentsEndpoint, + configuration.targetPopulation, + configuration.identityParameter, + Object.entries(configuration.assignmentParameters).sort(([a], [b]) => a.localeCompare(b)), + Array.from(resolvedAssignmentParameters.entries()).sort(([a], [b]) => a.localeCompare(b)), + Object.entries(sdkTargetingValues).sort(([a], [b]) => a.localeCompare(b)), + extensionVersion, + ]); + this.prefix = `python-envs.experimentation.${createHash('sha256').update(namespace).digest('hex')}.`; + } + + /** Check for a valid cached snapshot, including an empty one. */ + public hasCachedAssignments(): boolean { + return this.get(TAS_CACHE_KEY) !== undefined; + } + + public keys(): readonly string[] { + return this.storage.keys() + .filter((key) => key.startsWith(this.prefix)) + .map((key) => key.slice(this.prefix.length)); + } + + public get(key: string): T | undefined; + public get(key: string, defaultValue: T): T; + public get(key: string, defaultValue?: T): T | undefined { + const value = this.storage.get(`${this.prefix}${key}`); + if (key === TAS_CACHE_KEY && value !== undefined && !isCacheData(value)) { + if (!this.reportedInvalidCache) { + this.reportedInvalidCache = true; + traceWarn('[experimentation] Ignoring malformed cached assignments; awaiting a fresh fetch.'); + } + return defaultValue; + } + return value === undefined ? defaultValue : value; + } + + public async update(key: string, value: unknown): Promise { + if (!this.isActive()) { + return; + } + try { + await this.storage.update(`${this.prefix}${key}`, value); + } catch (error) { + traceWarn('[experimentation] Unable to persist assignments; they may not survive reload:', error); + } + } +} diff --git a/src/common/experimentation/telemetry.ts b/src/common/experimentation/telemetry.ts new file mode 100644 index 000000000..487a7e4b5 --- /dev/null +++ b/src/common/experimentation/telemetry.ts @@ -0,0 +1,110 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import type { IExperimentationTelemetry } from 'vscode-tas-client'; +import { ENVS_EXTENSION_ID } from '../constants'; +import { traceVerbose, traceWarn } from '../logging'; +import { EventNames, IEventNamePropertyMapping } from '../telemetry/constants'; +import { setSharedTelemetryProperty } from '../telemetry/reporter'; +import { sendTelemetryEvent } from '../telemetry/sender'; + +export const ASSIGNMENT_CONTEXT_PROPERTY = 'abexp.assignmentcontext'; +export type TasCall = IEventNamePropertyMapping[EventNames.EXPERIMENTATION_TAS_CALL]; + +function isOutcome(value: string | undefined): value is TasCall['outcome'] { + return value === 'Success' || value === 'ServerError' || value === 'NoResponse' || value === 'GenericError'; +} + +/** Forward classified SDK telemetry only while its instance is active. */ +export class ExperimentationTelemetry implements IExperimentationTelemetry { + private readonly ignoredEvents = new Set(); + + constructor( + private readonly isActive: () => boolean, + private readonly onCall: (call: TasCall) => void, + private readonly onContext: () => void = () => undefined, + ) {} + + /** Share the active SDK's assignment context. */ + public setSharedProperty(name: string, value: string): void { + if (!this.isActive()) { + return; + } + if (name !== ASSIGNMENT_CONTEXT_PROPERTY) { + this.warnOnce(`property:${name}`); + return; + } + setSharedTelemetryProperty(ASSIGNMENT_CONTEXT_PROPERTY, value || undefined); + this.onContext(); + } + + /** Forward allowlisted SDK events and properties. */ + public postEvent(eventName: string, properties: Map): void { + if (!this.isActive()) { + return; + } + switch (eventName) { + case EventNames.EXPERIMENTATION_QUERY: { + const feature = properties.get('ABExp.queriedFeature'); + if (!feature || !/^vscode\.[A-Za-z0-9_.-]+$/.test(feature)) { + this.warnOnce(eventName); + return; + } + sendTelemetryEvent(EventNames.EXPERIMENTATION_QUERY, undefined, { 'ABExp.queriedFeature': feature }); + return; + } + case EventNames.EXPERIMENTATION_TAS_CALL: { + const callType = properties.get('callType'); + const outcome = properties.get('outcome'); + if ((callType !== 'legacy' && callType !== 'assignments') || !isOutcome(outcome)) { + this.warnOnce(eventName); + return; + } + const call: TasCall = { + callType, + outcome, + extensionName: ENVS_EXTENSION_ID, + assignmentContext: properties.get('assignmentContext'), + }; + this.onCall(call); + traceVerbose(`[experimentation] ${callType} fetch: ${outcome}.`); + sendTelemetryEvent(EventNames.EXPERIMENTATION_TAS_CALL, undefined, call); + return; + } + case EventNames.EXPERIMENTATION_ASSIGNMENTS_VALIDATION: { + const measures: Record = {}; + for (const name of ['FeatureVariableCount', 'AssignedVariantCount']) { + const raw = properties.get(name); + const count = raw !== undefined && /^\d+$/.test(raw) ? Number(raw) : undefined; + if (count !== undefined && Number.isSafeInteger(count)) { + measures[name] = count; + } + } + sendTelemetryEvent(EventNames.EXPERIMENTATION_ASSIGNMENTS_VALIDATION, measures, { + DataVersion: properties.get('DataVersion'), + AssignmentContext: properties.get('AssignmentContext'), + }); + return; + } + case EventNames.EXPERIMENTATION_LEGACY_ERROR: + case EventNames.EXPERIMENTATION_ASSIGNMENTS_ERROR: { + const errorType = properties.get('ErrorType'); + if (!isOutcome(errorType) || errorType === 'Success') { + this.warnOnce(eventName); + return; + } + sendTelemetryEvent(eventName, undefined, { ErrorType: errorType }); + return; + } + default: + this.warnOnce(eventName); + } + } + + private warnOnce(key: string): void { + if (!this.ignoredEvents.has(key)) { + this.ignoredEvents.add(key); + traceWarn('[experimentation] Dropped an unclassified or malformed SDK telemetry payload.'); + } + } +} diff --git a/src/common/experimentation/transport.ts b/src/common/experimentation/transport.ts new file mode 100644 index 000000000..8ba511c2f --- /dev/null +++ b/src/common/experimentation/transport.ts @@ -0,0 +1,74 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import * as https from 'https'; +import type { FetchFn, IFetchResponse } from 'vscode-tas-client'; + +export const TAS_REQUEST_TIMEOUT_MS = 10_000; +const MAX_RESPONSE_BYTES = 2 * 1024 * 1024; + +/** Create a cancellable HTTPS transport with request and response bounds. */ +export function createExperimentationFetch( + signal: AbortSignal, + request: typeof https.request = https.request, +): FetchFn { + return async (url, init): Promise => { + if (signal.aborted) { + throw new Error('Experimentation request cancelled.'); + } + if (new URL(url).protocol !== 'https:') { + throw new Error('Experimentation requests require HTTPS.'); + } + + let timer: ReturnType | undefined; + try { + return await new Promise((resolve, reject) => { + let responseReceived = false; + const fail = (error: Error) => reject(Object.assign(error, { responseReceived })); + const req = request(url, { + method: init.method, + headers: { + ...init.headers, + ...(init.body === undefined ? {} : { 'Content-Length': String(Buffer.byteLength(init.body)) }), + }, + signal, + }, (response) => { + responseReceived = true; + const chunks: Buffer[] = []; + let size = 0; + response.on('data', (chunk: Buffer | string) => { + const buffer = typeof chunk === 'string' ? Buffer.from(chunk) : chunk; + size += buffer.length; + if (size > MAX_RESPONSE_BYTES) { + const error = new Error('Experimentation response exceeded the size limit.'); + fail(error); + req.destroy(error); + return; + } + chunks.push(buffer); + }); + response.on('error', fail); + response.on('aborted', () => fail(new Error('Experimentation response was interrupted.'))); + response.on('end', () => { + const text = Buffer.concat(chunks).toString('utf8'); + resolve({ + status: response.statusCode ?? 0, + json: async (): Promise => JSON.parse(text), + }); + }); + }); + req.on('error', fail); + timer = setTimeout( + () => req.destroy(new Error('Experimentation request timed out.')), + TAS_REQUEST_TIMEOUT_MS, + ); + timer.unref(); + req.end(init.body); + }); + } finally { + if (timer) { + clearTimeout(timer); + } + } + }; +} diff --git a/src/common/telemetry/constants.ts b/src/common/telemetry/constants.ts index e0b758cb3..e7645ebc7 100644 --- a/src/common/telemetry/constants.ts +++ b/src/common/telemetry/constants.ts @@ -1,6 +1,12 @@ export enum EventNames { EXTENSION_ACTIVATION_DURATION = 'EXTENSION.ACTIVATION_DURATION', EXTENSION_MANAGER_REGISTRATION_DURATION = 'EXTENSION.MANAGER_REGISTRATION_DURATION', + EXPERIMENTATION_INITIALIZATION = 'EXPERIMENTATION.INITIALIZATION', + EXPERIMENTATION_QUERY = 'query-expfeature', + EXPERIMENTATION_TAS_CALL = 'tas-call', + EXPERIMENTATION_ASSIGNMENTS_VALIDATION = 'assignments-validation', + EXPERIMENTATION_LEGACY_ERROR = 'call-tas-error', + EXPERIMENTATION_ASSIGNMENTS_ERROR = 'call-assignments-error', ENVIRONMENT_MANAGER_REGISTERED = 'ENVIRONMENT_MANAGER.REGISTERED', PACKAGE_MANAGER_REGISTERED = 'PACKAGE_MANAGER.REGISTERED', @@ -272,6 +278,67 @@ export type InlineScriptEnvErrorCategory = // Map all events to their properties export interface IEventNamePropertyMapping { + /* __GDPR__ + "experimentation.initialization": { + "result": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" }, + "cacheState": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" }, + "": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "isMeasurement": true, "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_INITIALIZATION]: { + result: 'notConfigured' | 'disabled' | 'cacheReady' | 'error' | 'timeout'; + cacheState: 'present' | 'absent' | 'unknown'; + }; + /* __GDPR__ + "query-expfeature": { + "ABExp.queriedFeature": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_QUERY]: { + 'ABExp.queriedFeature': string; + }; + /* __GDPR__ + "tas-call": { + "callType": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" }, + "outcome": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" }, + "extensionName": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "StellaHuang95" }, + "assignmentContext": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_TAS_CALL]: { + callType: 'legacy' | 'assignments'; + outcome: 'Success' | 'ServerError' | 'NoResponse' | 'GenericError'; + extensionName: string; + assignmentContext?: string; + }; + /* __GDPR__ + "assignments-validation": { + "FeatureVariableCount": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "isMeasurement": true, "owner": "StellaHuang95" }, + "AssignedVariantCount": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "isMeasurement": true, "owner": "StellaHuang95" }, + "DataVersion": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" }, + "AssignmentContext": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_ASSIGNMENTS_VALIDATION]: { + DataVersion?: string; + AssignmentContext?: string; + }; + /* __GDPR__ + "call-tas-error": { + "ErrorType": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_LEGACY_ERROR]: { + ErrorType: 'ServerError' | 'NoResponse' | 'GenericError'; + }; + /* __GDPR__ + "call-assignments-error": { + "ErrorType": { "classification": "SystemMetaData", "purpose": "PerformanceAndHealth", "owner": "StellaHuang95" } + } + */ + [EventNames.EXPERIMENTATION_ASSIGNMENTS_ERROR]: { + ErrorType: 'ServerError' | 'NoResponse' | 'GenericError'; + }; /* __GDPR__ "terminal.activation_outcome": { "operation": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "eleanorjboyd" }, diff --git a/src/common/telemetry/reporter.ts b/src/common/telemetry/reporter.ts index 251349d1a..3ea612c76 100644 --- a/src/common/telemetry/reporter.ts +++ b/src/common/telemetry/reporter.ts @@ -1,23 +1,147 @@ -import type TelemetryReporter from '@vscode/extension-telemetry'; +import type { + TelemetryEventMeasurements, + TelemetryEventProperties, + TelemetryReporter, +} from '@vscode/extension-telemetry'; +import { version as telemetryClientVersion } from '@vscode/extension-telemetry/package.json'; +import { arch, platform, release } from 'os'; +import type { Disposable, TelemetryLogger } from 'vscode'; +import { createTelemetryLogger } from '../env.apis'; +import { traceError } from '../logging'; + +/* __GDPR__COMMON__ + "abexp.assignmentcontext": { + "classification": "SystemMetaData", + "purpose": "FeatureInsight", + "owner": "StellaHuang95", + "comment": "SDK-provided experiment assignments associated with extension telemetry." + } +*/ +interface SharedTelemetryProperties { + 'abexp.assignmentcontext'?: string; +} + +const sharedProperties: SharedTelemetryProperties = {}; class ReporterImpl { private static telemetryReporter: TelemetryReporter | undefined; + private static automaticErrorLogger: TelemetryLogger | undefined; + private static registration: symbol | undefined; + + static register(): symbol { + const registration = Symbol(); + ReporterImpl.registration = registration; + delete sharedProperties['abexp.assignmentcontext']; + return registration; + } + static getTelemetryReporter() { - const tel = require('@vscode/extension-telemetry'); - const Reporter = tel.default as typeof TelemetryReporter; - ReporterImpl.telemetryReporter = new Reporter( - '0c6ae279ed8443289764825290e4f9e2-1a736e7c-1324-4338-be46-fc2a58ae4d14-7255', - [ - { - lookup: /(errorName|errorMessage|errorStack)/g, - }, - ], - ); + if (!ReporterImpl.registration) { + return undefined; + } + if (!ReporterImpl.telemetryReporter) { + const { TelemetryReporter: Reporter }: typeof import('@vscode/extension-telemetry') = + require('@vscode/extension-telemetry'); + const reporter = new Reporter( + '0c6ae279ed8443289764825290e4f9e2-1a736e7c-1324-4338-be46-fc2a58ae4d14-7255', + [ + { + lookup: /(errorName|errorMessage|errorStack)/g, + }, + ], + { ignoreUnhandledErrors: true }, + ); + try { + ReporterImpl.automaticErrorLogger = ReporterImpl.createAutomaticErrorLogger(reporter); + ReporterImpl.telemetryReporter = reporter; + } catch (error) { + void reporter.dispose().catch((disposeError) => + traceError('Failed to dispose an unregistered telemetry reporter:', disposeError), + ); + throw error; + } + } return ReporterImpl.telemetryReporter; } + + private static createAutomaticErrorLogger(reporter: TelemetryReporter): TelemetryLogger { + const forward = ( + eventName: string, + properties?: TelemetryEventProperties, + measurements?: TelemetryEventMeasurements, + ): void => { + if ( + !ReporterImpl.registration || + ReporterImpl.telemetryReporter !== reporter || + (reporter.telemetryLevel !== 'all' && reporter.telemetryLevel !== 'error') + ) { + return; + } + try { + // VS Code already checked consent and cleaned this data; avoid a second logger pass and name prefix. + reporter.sendDangerousTelemetryEvent( + eventName, { ...properties, ...getSharedTelemetryProperties() }, measurements, + ); + } catch (error) { + traceError('Failed to send automatic error telemetry:', error); + } + }; + return createTelemetryLogger({ + sendEventData: (eventName, data) => + forward(eventName, data?.properties ?? data, data?.measurements), + sendErrorData: (error, data) => forward('unhandlederror', { + ...(data?.properties ?? data), + name: error.name, + message: error.message, + stack: error.stack, + }, data?.measurements), + }, { + additionalCommonProperties: { + 'common.os': platform(), + 'common.nodeArch': arch(), + 'common.platformversion': release().replace(/^(\d+)(\.\d+)?(\.\d+)?(.*)/, '$1$2$3'), + 'common.telemetryclientversion': telemetryClientVersion, + }, + }); + } + + static async dispose(registration: symbol): Promise { + if (ReporterImpl.registration !== registration) { + return; + } + ReporterImpl.registration = undefined; + const reporter = ReporterImpl.telemetryReporter; + ReporterImpl.telemetryReporter = undefined; + const automaticErrorLogger = ReporterImpl.automaticErrorLogger; + ReporterImpl.automaticErrorLogger = undefined; + delete sharedProperties['abexp.assignmentcontext']; + automaticErrorLogger?.dispose(); + await reporter?.dispose(); + } } +/** Get the active registration's lazy reporter, or undefined outside its lifetime. */ export function getTelemetryReporter() { return ReporterImpl.getTelemetryReporter(); } + +/** Set a classified shared property, or remove it with undefined. */ +export function setSharedTelemetryProperty(name: keyof SharedTelemetryProperties, value: string | undefined): void { + if (value === undefined) { + delete sharedProperties[name]; + } else { + sharedProperties[name] = value; + } +} + +/** Copy shared properties for one event. */ +export function getSharedTelemetryProperties(): Readonly { + return { ...sharedProperties }; +} + +/** Register lazy telemetry ownership; stale disposers cannot end a newer registration. */ +export function registerTelemetryReporter(): Disposable { + const registration = ReporterImpl.register(); + return { dispose: () => ReporterImpl.dispose(registration) }; +} diff --git a/src/common/telemetry/sender.ts b/src/common/telemetry/sender.ts index 80874b16e..dabefd037 100644 --- a/src/common/telemetry/sender.ts +++ b/src/common/telemetry/sender.ts @@ -1,7 +1,8 @@ import type { IEventNamePropertyMapping } from './constants'; +import { traceError } from '../logging'; import { StopWatch } from '../stopWatch'; import { isTestExecution } from '../utils/testing'; -import { getTelemetryReporter } from './reporter'; +import { getSharedTelemetryProperties, getTelemetryReporter } from './reporter'; import { isPromise } from 'util/types'; type FailedEventType = { failed: true }; @@ -25,7 +26,6 @@ export function sendTelemetryEvent

{ + let context: ExperimentationContext & { subscriptions: Disposable[] }; + + setup(() => { + sinon.stub(sender, 'sendTelemetryEvent'); + context = { + globalState: new MockMemento(), + extension: { packageJSON: { version: '1.39.0' } }, + subscriptions: [], + }; + }); + + teardown(() => { + context.subscriptions.forEach((disposable) => disposable.dispose()); + sinon.restore(); + }); + + test('registers exactly one accessible service without awaiting a network request', () => { + const service = initializeExperimentation(context); + assert.strictEqual(getExperimentationService(), service); + assert.strictEqual(initializeExperimentation(context), service); + assert.strictEqual(context.subscriptions.length, 1); + assert.strictEqual(service.diagnostics.state, 'notConfigured'); + }); + + test('deactivation releases the service and allows a fresh activation', () => { + const first = initializeExperimentation(context); + context.subscriptions[0].dispose(); + assert.strictEqual(first.diagnostics.state, 'disposed'); + assert.strictEqual(getExperimentationService(), undefined); + const second = initializeExperimentation(context); + assert.notStrictEqual(second, first); + context.subscriptions[0].dispose(); + assert.strictEqual(getExperimentationService(), second, 'late old cleanup must not unset the new service'); + }); +}); diff --git a/src/test/common/experimentation/configuration.unit.test.ts b/src/test/common/experimentation/configuration.unit.test.ts new file mode 100644 index 000000000..63e09c0fd --- /dev/null +++ b/src/test/common/experimentation/configuration.unit.test.ts @@ -0,0 +1,101 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import { + EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + EXPERIMENTATION_IDENTITY_PARAMETER, + getExperimentationExtensionVersion, + readExperimentationConfiguration, +} from '../../../common/experimentation/configuration'; + +const configuration = { + targetPopulation: 'public', + assignmentParameters: { + approved_version: 'extensionVersion', + approved_language: 'language', + }, +}; + +suite('Experimentation configuration', () => { + test('an unconfigured build has no implicit endpoint or identity mapping', () => { + assert.strictEqual(readExperimentationConfiguration({ version: '1.0.0' }), undefined); + assert.strictEqual(readExperimentationConfiguration(undefined), undefined); + }); + + test('reads only the publisher manifest entry and copies approved bindings', () => { + const result = readExperimentationConfiguration({ experimentation: configuration }); + assert.deepStrictEqual(result, { + ...configuration, + assignmentsEndpoint: EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + identityParameter: EXPERIMENTATION_IDENTITY_PARAMETER, + assignmentParameters: { + devdeviceid: 'devDeviceId', + ...configuration.assignmentParameters, + }, + }); + assert.notStrictEqual(result?.assignmentParameters, configuration.assignmentParameters); + assert.ok(Object.isFrozen(result?.assignmentParameters)); + }); + + test('rejects publisher overrides of the platform-owned endpoint or identity parameter', () => { + for (const override of [ + { assignmentsEndpoint: 'https://other.example.invalid/api/v1/assignments' }, + { identityParameter: 'other_identity' }, + ]) { + assert.throws(() => readExperimentationConfiguration({ + experimentation: { ...configuration, ...override }, + })); + } + }); + + test('requires an explicit supported population', () => { + for (const targetPopulation of [undefined, 'insiders', 'unknown', true]) { + assert.throws(() => readExperimentationConfiguration({ + experimentation: { ...configuration, targetPopulation }, + })); + } + }); + + test('refuses legacy header names and overrides of generic SDK filters', () => { + for (const name of ['X-MSEdge-ClientId', 'x-client-id', 'extensionname', 'vscode_core_build', '../identity']) { + assert.throws(() => readExperimentationConfiguration({ + experimentation: { + ...configuration, + assignmentParameters: { [name]: 'devDeviceId' }, + }, + })); + } + }); + + test('adds the approved DevDeviceId identity and rejects publisher identity overrides', () => { + const identityOnly = readExperimentationConfiguration({ + experimentation: { targetPopulation: 'public' }, + }); + assert.deepStrictEqual(identityOnly?.assignmentParameters, { devdeviceid: 'devDeviceId' }); + + for (const assignmentParameters of [ + { approved_identity: 'DevDeviceId' }, + { devdeviceid: 'machineId' }, + { devdeviceid: 'devDeviceId', second_identity: 'devDeviceId' }, + { devdeviceid: 'devDeviceId', unapproved: 'some literal value' }, + ]) { + assert.throws(() => readExperimentationConfiguration({ + experimentation: { ...configuration, assignmentParameters }, + })); + } + }); + + test('does not treat a malformed configuration as an unconfigured build', () => { + for (const value of [false, null, [], 'enabled', {}]) { + assert.throws(() => readExperimentationConfiguration({ experimentation: value })); + } + }); + + test('validates the extension version', () => { + assert.strictEqual(getExperimentationExtensionVersion({ version: '1.39.0' }), '1.39.0'); + for (const manifest of [{}, { version: 1 }, { version: '' }, { version: ' ' }, undefined]) { + assert.throws(() => getExperimentationExtensionVersion(manifest)); + } + }); +}); diff --git a/src/test/common/experimentation/sdk.unit.test.ts b/src/test/common/experimentation/sdk.unit.test.ts new file mode 100644 index 000000000..7ea00aa35 --- /dev/null +++ b/src/test/common/experimentation/sdk.unit.test.ts @@ -0,0 +1,144 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import * as sinon from 'sinon'; +import { when } from 'ts-mockito'; +import { Disposable } from 'vscode'; +import type { FetchFn } from 'vscode-tas-client'; +import * as envApis from '../../../common/env.apis'; +import { EXPERIMENTATION_ASSIGNMENTS_ENDPOINT } from '../../../common/experimentation/configuration'; +import { ExperimentationService } from '../../../common/experimentation/service'; +import * as transport from '../../../common/experimentation/transport'; +import { getSharedTelemetryProperties } from '../../../common/telemetry/reporter'; +import * as sender from '../../../common/telemetry/sender'; +import { createDeferred } from '../../../common/utils/deferred'; +import { MockMemento } from '../../mocks/mementos'; +import { mockedVSCodeNamespaces } from '../../unittests'; + +suite('Experimentation SDK contract with fake transport', () => { + let service: ExperimentationService | undefined; + let calls: Parameters[]; + let legacyFails: boolean; + let state: MockMemento; + let responsesReleased: Promise; + let assignedFlag: string; + + setup(() => { + calls = []; + legacyFails = false; + state = new MockMemento(); + responsesReleased = Promise.resolve(); + assignedFlag = 'true'; + when(mockedVSCodeNamespaces.env!.isTelemetryEnabled).thenReturn(true); + when(mockedVSCodeNamespaces.env!.machineId).thenReturn('sdk-test-machine'); + when(mockedVSCodeNamespaces.env!.language).thenReturn('en'); + sinon.stub(envApis, 'isTelemetryEnabled').returns(true); + sinon.stub(envApis, 'onDidChangeTelemetryEnabled').returns(new Disposable(() => undefined)); + sinon.stub(envApis, 'getMachineId').returns('sdk-test-machine'); + sinon.stub(envApis, 'getLanguage').returns('en'); + sinon.stub(envApis, 'getVSCodeVersion').returns('1.110.0'); + sinon.stub(envApis, 'getAppName').returns('Insider'); + sinon.stub(sender, 'sendTelemetryEvent'); + const fetch: FetchFn = async (url, init) => { + calls.push([url, init]); + await responsesReleased; + if (legacyFails && init.method === 'GET') { + throw new Error('simulated legacy failure'); + } + return { + status: 200, + json: async () => init.method === 'GET' + ? { + Configs: [{ Id: 'vscode', Parameters: { genericFlag: false, legacyValue: 3 } }], + AssignmentContext: 'legacy;', + } + : { + featureVariables: { '/vscode/genericFlag': assignedFlag, '/vscode/mode': 'new' }, + assignedVariants: [], dataVersion: 1, assignmentContext: 'assignments;', + }, + }; + }; + sinon.stub(transport, 'createExperimentationFetch').returns(fetch); + }); + + teardown(() => { + service?.dispose(); + service = undefined; + when(mockedVSCodeNamespaces.env!.isTelemetryEnabled).thenReturn(false); + sinon.restore(); + }); + + async function start(waitForFetch = true): Promise { + service = new ExperimentationService( + { + globalState: state, + extension: { + packageJSON: { + version: '1.39.0', + experimentation: { + targetPopulation: 'public', + }, + }, + }, + }, + undefined, + () => 'sdk-test-dev-device', + ); + await service.initializePromise; + if (waitForFetch) { + await service.initialFetch; + } + return service; + } + + test('loads the installed SDK, uses both transports, and reads new assignments under bare names', async () => { + const initialized = await start(); + assert.strictEqual(initialized.diagnostics.state, 'ready'); + assert.strictEqual(initialized.diagnostics.assignmentsFetch, 'Success'); + assert.strictEqual(initialized.diagnostics.legacyFetch, 'Success'); + assert.deepStrictEqual(calls.map((call) => call[1].method).sort(), ['GET', 'POST']); + const post = calls.find((call) => call[1].method === 'POST'); + assert.ok(post?.[1].body); + const body = JSON.parse(post[1].body); + assert.strictEqual(post[0], EXPERIMENTATION_ASSIGNMENTS_ENDPOINT); + assert.strictEqual(body.userParams.devdeviceid, 'sdk-test-dev-device'); + assert.strictEqual(body.userParams.vscode_core_extensionname, 'ms-python.vscode-python-envs'); + assert.strictEqual(initialized.getTreatmentVariable('genericFlag', false), true); + assert.strictEqual(initialized.getTreatmentVariable('legacyValue', 0), 3); + assert.strictEqual(initialized.getTreatmentVariable('mode', 'fallback'), 'new'); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'legacy;assignments;' }); + }); + + test('a failed legacy endpoint cannot discard a successful assignments response', async () => { + legacyFails = true; + const initialized = await start(); + assert.strictEqual(initialized.diagnostics.assignmentsFetch, 'Success'); + assert.strictEqual(initialized.diagnostics.legacyFetch, 'GenericError'); + assert.strictEqual(initialized.getTreatmentVariable('genericFlag', false), true); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'assignments;' }); + }); + + test('reuses the persisted snapshot on restart and keeps a consumed decision while newer data is cached', async () => { + const first = await start(); + assert.strictEqual(first.getTreatmentVariable('genericFlag', false), true); + first.dispose(); + + const release = createDeferred(); + responsesReleased = release.promise; + assignedFlag = 'false'; + const second = await start(false); + assert.strictEqual(second.diagnostics.cacheState, 'present'); + assert.strictEqual(second.diagnostics.assignmentsFetch, 'notObserved'); + assert.strictEqual(second.getTreatmentVariable('genericFlag', false), true); + release.resolve(); + await second.initialFetch; + assert.strictEqual(second.diagnostics.assignmentsFetch, 'Success'); + assert.strictEqual(second.getTreatmentVariable('genericFlag', false), true, 'consumed decisions stay stable'); + second.dispose(); + + const third = await start(); + assert.strictEqual(third.diagnostics.cacheState, 'present'); + assert.strictEqual(third.getTreatmentVariable('genericFlag', true), false, 'the next session sees the new cache'); + }); +}); diff --git a/src/test/common/experimentation/service.unit.test.ts b/src/test/common/experimentation/service.unit.test.ts new file mode 100644 index 000000000..d571e95ba --- /dev/null +++ b/src/test/common/experimentation/service.unit.test.ts @@ -0,0 +1,578 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import * as sinon from 'sinon'; +import { EventEmitter } from 'vscode'; +import { ENVS_EXTENSION_ID } from '../../../common/constants'; +import * as envApis from '../../../common/env.apis'; +import { + EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + EXPERIMENTATION_IDENTITY_PARAMETER, + type ExperimentationConfiguration, +} from '../../../common/experimentation/configuration'; +import { + EXPERIMENTATION_INITIALIZATION_TIMEOUT_MS, + ExperimentationClient, + ExperimentationClientFactory, + ExperimentationClientOptions, + ExperimentationContext, + ExperimentationService, +} from '../../../common/experimentation/service'; +import { + ExperimentationStorage, + type SdkTargetingValues, + TAS_CACHE_KEY, +} from '../../../common/experimentation/storage'; +import { TasCall } from '../../../common/experimentation/telemetry'; +import * as logging from '../../../common/logging'; +import { EventNames } from '../../../common/telemetry/constants'; +import { getSharedTelemetryProperties } from '../../../common/telemetry/reporter'; +import * as sender from '../../../common/telemetry/sender'; +import { createDeferred } from '../../../common/utils/deferred'; +import { MockMemento } from '../../mocks/mementos'; + +const CONFIGURATION: ExperimentationConfiguration = { + assignmentsEndpoint: EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + targetPopulation: 'public', + identityParameter: EXPERIMENTATION_IDENTITY_PARAMETER, + assignmentParameters: { + devdeviceid: 'devDeviceId', + approved_version: 'extensionVersion', + approved_language: 'language', + }, +}; +const MANIFEST_CONFIGURATION = { + targetPopulation: CONFIGURATION.targetPopulation, + assignmentParameters: { + approved_version: 'extensionVersion', + approved_language: 'language', + }, +}; +const VERSION = '1.39.0'; +const DEV_DEVICE_ID = 'test-dev-device'; +const LEGACY_MACHINE_ID = 'test-machine'; +const VSCODE_VERSION = '1.110.0'; +const APP_NAME = 'Visual Studio Code'; +const LANGUAGE = 'en'; +const IDENTITY_ONLY_CONFIGURATION: ExperimentationConfiguration = { + ...CONFIGURATION, + assignmentParameters: { devdeviceid: 'devDeviceId' }, +}; +const IDENTITY_ONLY_MANIFEST = { + targetPopulation: IDENTITY_ONLY_CONFIGURATION.targetPopulation, +}; + +class FakeSdk implements ExperimentationClient { + readonly initialized = createDeferred(); + readonly fetched = createDeferred(); + readonly initializePromise = this.initialized.promise; + readonly initialFetch = this.fetched.promise; + readonly dispose = sinon.spy(); + readonly queried = sinon.spy(); + readonly values = new Map(); + + constructor(readonly options: ExperimentationClientOptions, initialize = true) { + const cached = options.memento.get<{ + assignmentContext: string; + configs: { Id: string; Parameters: Record }[]; + }>(TAS_CACHE_KEY); + if (cached) { + for (const [name, value] of Object.entries(cached.configs[0]?.Parameters ?? {})) { + this.values.set(name, value); + } + options.telemetry.setSharedProperty('abexp.assignmentcontext', cached.assignmentContext); + } + if (initialize) { + this.initialized.resolve(); + } + } + + getTreatmentVariable(configId: string, name: string): T | undefined { + this.queried(configId, name); + return this.values.get(name) as T | undefined; + } + + report(callType: TasCall['callType'], outcome: TasCall['outcome']): void { + this.options.telemetry.postEvent('tas-call', new Map([['callType', callType], ['outcome', outcome]])); + } + + async publish(values: Record, context = 'treatment;'): Promise { + Object.entries(values).forEach(([name, value]) => this.values.set(name, value)); + this.report('assignments', 'Success'); + this.options.telemetry.setSharedProperty('abexp.assignmentcontext', context); + await this.options.memento.update(TAS_CACHE_KEY, { + features: Object.keys(values), assignmentContext: context, + configs: [{ Id: 'vscode', Parameters: values }], + }); + this.fetched.resolve(); + } +} + +suite('Experimentation service', () => { + let clock: sinon.SinonFakeTimers; + let consent: boolean; + let changed: EventEmitter; + let state: MockMemento; + let context: ExperimentationContext; + let clients: FakeSdk[]; + let services: ExperimentationService[]; + let factory: sinon.SinonSpy<[ExperimentationClientOptions], FakeSdk>; + let events: sinon.SinonStub; + let warn: sinon.SinonStub; + let logError: sinon.SinonStub; + let getDevDeviceId: sinon.SinonStub<[], string | undefined>; + + setup(() => { + clock = sinon.useFakeTimers(); + consent = true; + changed = new EventEmitter(); + state = new MockMemento(); + context = { + globalState: state, + extension: { packageJSON: { version: VERSION, experimentation: MANIFEST_CONFIGURATION } }, + }; + clients = []; + services = []; + factory = sinon.spy((options: ExperimentationClientOptions) => { + const client = new FakeSdk(options); + clients.push(client); + return client; + }); + sinon.stub(envApis, 'isTelemetryEnabled').callsFake(() => consent); + sinon.stub(envApis, 'onDidChangeTelemetryEnabled').callsFake((listener) => changed.event(listener)); + sinon.stub(envApis, 'getMachineId').returns(LEGACY_MACHINE_ID); + sinon.stub(envApis, 'getLanguage').returns(LANGUAGE); + sinon.stub(envApis, 'getVSCodeVersion').returns(VSCODE_VERSION); + sinon.stub(envApis, 'getAppName').returns(APP_NAME); + getDevDeviceId = sinon.stub<[], string | undefined>().returns(DEV_DEVICE_ID); + events = sinon.stub(sender, 'sendTelemetryEvent'); + warn = sinon.stub(logging, 'traceWarn'); + logError = sinon.stub(logging, 'traceError'); + }); + + teardown(() => { + services.forEach((service) => service.dispose()); + changed.dispose(); + clock.restore(); + sinon.restore(); + }); + + function start(createClient: ExperimentationClientFactory = factory): ExperimentationService { + const service = new ExperimentationService(context, createClient, getDevDeviceId); + services.push(service); + return service; + } + + async function cache( + values: Record = { example: true }, + configuration = CONFIGURATION, + targetingOverrides: Partial = {}, + ): Promise { + const sdkTargetingValues: SdkTargetingValues = { + applicationVersion: VSCODE_VERSION, + build: APP_NAME, + clientId: LEGACY_MACHINE_ID, + language: LANGUAGE, + ...targetingOverrides, + }; + const sources = { + devDeviceId: DEV_DEVICE_ID, + extensionVersion: VERSION, + language: sdkTargetingValues.language, + }; + const parameters = new Map(); + for (const [name, source] of Object.entries(configuration.assignmentParameters)) { + parameters.set(name, sources[source]); + } + await new ExperimentationStorage( + state, configuration, parameters, sdkTargetingValues, VERSION, () => true, + ).update(TAS_CACHE_KEY, { + features: Object.keys(values), + assignmentContext: 'cached;', + configs: [{ Id: 'vscode', Parameters: values }], + }); + } + + function changeConsent(value: boolean): void { + consent = value; + changed.fire(value); + } + + test('an unconfigured build never creates an SDK, reads an identity, or fetches', async () => { + context = { ...context, extension: { packageJSON: { version: VERSION } } }; + const service = start(); + await service.initializePromise; + await service.initialFetch; + assert.strictEqual(service.diagnostics.state, 'notConfigured'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.notCalled(factory); + sinon.assert.notCalled(getDevDeviceId); + }); + + test('invalid configuration is reported and does not fall back to a legacy-only client', async () => { + context = { + ...context, + extension: { + packageJSON: { + version: VERSION, + experimentation: { + targetPopulation: CONFIGURATION.targetPopulation, + assignmentParameters: { devdeviceid: 'devDeviceId' }, + }, + }, + }, + }; + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'failed'); + sinon.assert.notCalled(factory); + sinon.assert.calledOnce(logError); + }); + + test('a configured build fails closed while the DevDeviceId provider is unavailable', async () => { + getDevDeviceId.returns(undefined); + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'failed'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.notCalled(factory); + sinon.assert.calledOnce(logError); + }); + + test('uses this extension identity, approved new parameter names, and a scoped global memento', async () => { + const service = start(); + await service.initializePromise; + const options = clients[0].options; + assert.strictEqual(options.extensionName, ENVS_EXTENSION_ID); + assert.strictEqual(options.extensionVersion, VERSION); + assert.strictEqual(options.targetPopulation, 'public'); + assert.strictEqual(options.assignmentsEndpoint, EXPERIMENTATION_ASSIGNMENTS_ENDPOINT); + assert.deepStrictEqual(options.assignmentsFilterProviders?.[0].getFilters(), new Map([ + ['devdeviceid', DEV_DEVICE_ID], ['approved_version', VERSION], ['approved_language', 'en'], + ])); + assert.ok(options.fetch, 'both endpoints use the lifetime-bound transport'); + assert.strictEqual(options.filterProviders, undefined, 'new parameters must not become legacy headers'); + }); + + test('cold initialization does not claim a successful fetch or consume an empty snapshot', async () => { + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'ready'); + assert.strictEqual(service.diagnostics.cacheState, 'absent'); + assert.strictEqual(service.diagnostics.initialFetch, 'pending'); + assert.strictEqual(service.diagnostics.assignmentsFetch, 'notObserved'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.notCalled(clients[0].queried); + const initialized = events.getCalls().find( + (call) => call.args[0] === EventNames.EXPERIMENTATION_INITIALIZATION, + ); + assert.strictEqual(initialized?.args[2].result, 'cacheReady'); + }); + + test('warm cache can be queried before networking completes', async () => { + await cache(); + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.cacheState, 'present'); + assert.strictEqual(service.diagnostics.initialFetch, 'pending'); + assert.strictEqual(service.diagnostics.assignmentsFetch, 'notObserved'); + assert.strictEqual(service.getTreatmentVariable('example', false), true); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'cached;' }); + }); + + test('does not reuse warm cache after a resolved targeting value changes', async () => { + await cache(); + (envApis.getLanguage as sinon.SinonStub).returns('fr'); + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.cacheState, 'absent'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + assert.strictEqual( + clients[0].options.assignmentsFilterProviders?.[0].getFilters().get('approved_language'), 'fr', + ); + }); + + for (const changedValue of [ + { + name: 'language', + update: () => (envApis.getLanguage as sinon.SinonStub).returns('fr'), + }, + { + name: 'VS Code version', + update: () => (envApis.getVSCodeVersion as sinon.SinonStub).returns('1.111.0'), + }, + { + name: 'application name', + update: () => (envApis.getAppName as sinon.SinonStub).returns('Visual Studio Code - Insiders'), + }, + ]) { + test(`does not reuse warm cache when the SDK ${changedValue.name} changes`, async () => { + context = { + ...context, + extension: { + packageJSON: { version: VERSION, experimentation: IDENTITY_ONLY_MANIFEST }, + }, + }; + await cache({ example: true }, IDENTITY_ONLY_CONFIGURATION); + changedValue.update(); + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.cacheState, 'absent'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + } + + test('serves newly fetched Boolean, number and string assignments with type-appropriate defaults', async () => { + const service = start(); + await service.initializePromise; + await clients[0].publish({ enabled: true, limit: 3, mode: 'treatment', control: false }); + await service.initialFetch; + assert.strictEqual(service.diagnostics.assignmentsFetch, 'Success'); + assert.strictEqual(service.getTreatmentVariable('enabled', false), true); + assert.strictEqual(service.getTreatmentVariable('limit', 1), 3); + assert.strictEqual(service.getTreatmentVariable('mode', 'control'), 'treatment'); + assert.strictEqual(service.getTreatmentVariable('control', true), false); + assert.strictEqual(service.getTreatmentVariable('missing', 7), 7); + sinon.assert.calledWithExactly(clients[0].queried, 'vscode', 'enabled'); + }); + + test('does not expose a successful provider response before the SDK commits its merged snapshot', async () => { + const service = start(); + await service.initializePromise; + clients[0].values.set('example', true); + clients[0].report('assignments', 'Success'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.notCalled(clients[0].queried); + await clients[0].publish({ example: true }); + assert.strictEqual(service.getTreatmentVariable('example', false), true); + }); + + test('retains a successful snapshot and attribution when later polling fails', async () => { + const service = start(); + await service.initializePromise; + await clients[0].publish({ example: true }); + await service.initialFetch; + clients[0].report('assignments', 'NoResponse'); + clients[0].report('legacy', 'NoResponse'); + assert.strictEqual(service.getTreatmentVariable('example', false), true); + assert.strictEqual(service.diagnostics.assignmentsFetch, 'NoResponse'); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'treatment;' }); + }); + + test('missing or wrong-typed assignments never enable a Boolean feature', async () => { + const service = start(); + await service.initializePromise; + await clients[0].publish({ stringBoolean: 'true', numberBoolean: 1, object: {}, invalidNumber: NaN }); + await service.initialFetch; + assert.strictEqual(service.getTreatmentVariable('missing', false), false); + assert.strictEqual(service.getTreatmentVariable('stringBoolean', false), false); + assert.strictEqual(service.getTreatmentVariable('numberBoolean', false), false); + assert.strictEqual(service.getTreatmentVariable('object', 'default'), 'default'); + assert.strictEqual(service.getTreatmentVariable('invalidNumber', 1), 1); + assert.strictEqual(warn.callCount, 4); + service.getTreatmentVariable('stringBoolean', false); + assert.strictEqual(warn.callCount, 4, 'the same bad treatment should not flood logs'); + }); + + test('rejects invalid variable names without asking the SDK', async () => { + await cache(); + const service = start(); + await service.initializePromise; + assert.strictEqual(service.getTreatmentVariable('/vscode/example', false), false); + assert.strictEqual(service.getTreatmentVariable('', false), false); + sinon.assert.notCalled(clients[0].queried); + assert.strictEqual(warn.callCount, 2); + }); + + test('a completed failed fetch is not reported as successful assignment', async () => { + const service = start(); + await service.initializePromise; + clients[0].report('assignments', 'NoResponse'); + clients[0].report('legacy', 'GenericError'); + clients[0].fetched.resolve(); + await service.initialFetch; + assert.strictEqual(service.diagnostics.initialFetch, 'completed'); + assert.strictEqual(service.diagnostics.assignmentsFetch, 'NoResponse'); + assert.strictEqual(service.diagnostics.hasUsableSnapshot, false); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.notCalled(clients[0].queried); + await clients[0].publish({ example: true }); + assert.strictEqual( + service.getTreatmentVariable('example', false), true, 'a later successful poll can recover', + ); + }); + + test('keeps warm-cache values when the first network attempt rejects', async () => { + await cache(); + const service = start(); + await service.initializePromise; + clients[0].fetched.reject(new Error('offline')); + await service.initialFetch; + assert.strictEqual(service.diagnostics.initialFetch, 'failed'); + assert.strictEqual(service.getTreatmentVariable('example', false), true); + assert.ok(warn.called); + }); + + test('a throwing SDK factory is nonfatal and reported', async () => { + const service = start(() => { throw new Error('module unavailable'); }); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'failed'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.calledOnce(logError); + }); + + test('a rejected cache initialization disposes the client and clears attribution', async () => { + const service = start((options) => { + const client = new FakeSdk(options, false); + clients.push(client); + options.telemetry.setSharedProperty('abexp.assignmentcontext', 'partial;'); + client.initialized.reject(new Error('bad cache')); + return client; + }); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'failed'); + sinon.assert.calledOnce(clients[0].dispose); + assert.strictEqual(service.diagnostics.initialFetch, 'failed'); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + + test('cache initialization has a deadline and ignores late completion', async () => { + const service = start((options) => { + const client = new FakeSdk(options, false); + clients.push(client); + return client; + }); + const pending = service.initializePromise; + await clock.tickAsync(EXPERIMENTATION_INITIALIZATION_TIMEOUT_MS); + await pending; + assert.strictEqual(service.diagnostics.state, 'failed'); + assert.strictEqual(service.diagnostics.initialFetch, 'timeout'); + clients[0].initialized.resolve(); + await clock.tickAsync(0); + assert.strictEqual(service.diagnostics.state, 'failed'); + sinon.assert.calledOnce(clients[0].dispose); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('disposes a client whose module finishes loading after initialization timed out', async () => { + const loaded = createDeferred(); + let options: ExperimentationClientOptions | undefined; + const service = start((value) => { options = value; return loaded.promise; }); + await clock.tickAsync(EXPERIMENTATION_INITIALIZATION_TIMEOUT_MS); + await service.initializePromise; + assert.ok(options); + const lateClient = new FakeSdk(options); + loaded.resolve(lateClient); + await clock.tickAsync(0); + sinon.assert.calledOnce(lateClient.dispose); + assert.strictEqual(service.diagnostics.state, 'failed'); + }); + + test('bounds a first-fetch promise that never settles', async () => { + const service = start(); + await service.initializePromise; + const pending = service.initialFetch; + await clock.tickAsync(15_000); + await pending; + assert.strictEqual(service.diagnostics.initialFetch, 'timeout'); + assert.strictEqual(service.diagnostics.state, 'failed'); + sinon.assert.calledOnce(clients[0].dispose); + }); + + test('telemetry disabled at startup makes no SDK or identity requests', async () => { + consent = false; + const service = start(); + await service.initializePromise; + assert.strictEqual(service.diagnostics.state, 'disabled'); + sinon.assert.notCalled(factory); + sinon.assert.notCalled(getDevDeviceId); + sinon.assert.notCalled(envApis.getMachineId as sinon.SinonStub); + }); + + test('consent revocation stops polling, clears attribution, blocks requests, and ignores old writes', async () => { + const service = start(); + await service.initializePromise; + await clients[0].publish({ example: true }); + const old = clients[0]; + const cached = old.options.memento.get(TAS_CACHE_KEY); + changeConsent(false); + assert.strictEqual(service.diagnostics.state, 'disabled'); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + sinon.assert.calledOnce(old.dispose); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + await assert.rejects(old.options.fetch!('https://unused.example.invalid', { method: 'GET', headers: {} })); + await old.publish({ example: false }, 'late;'); + assert.deepStrictEqual(old.options.memento.get(TAS_CACHE_KEY), cached); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + + test('consent restoration starts one replacement and stale callbacks cannot replace its state', async () => { + const service = start(); + await service.initializePromise; + const old = clients[0]; + changeConsent(false); + changeConsent(true); + await service.initializePromise; + assert.strictEqual(clients.length, 2); + await clients[1].publish({ example: true }, 'current;'); + old.report('assignments', 'NoResponse'); + old.options.telemetry.setSharedProperty('abexp.assignmentcontext', 'stale;'); + old.fetched.resolve(); + await clock.tickAsync(0); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'current;' }); + assert.strictEqual(service.diagnostics.assignmentsFetch, 'Success'); + changed.fire(true); + assert.strictEqual(clients.length, 2, 'a repeated consent value must not make another client'); + }); + + test('dispose is idempotent, releases waiting callers, and prevents future consent restarts', async () => { + const service = start(); + await service.initializePromise; + const fetch = service.initialFetch; + service.dispose(); + service.dispose(); + await fetch; + changeConsent(false); + changeConsent(true); + assert.strictEqual(service.diagnostics.state, 'disposed'); + assert.strictEqual(factory.callCount, 1); + sinon.assert.calledOnce(clients[0].dispose); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('automated extension hosts cannot construct a live SDK even with publisher configuration', () => { + const service = new ExperimentationService(context, factory, getDevDeviceId, true); + services.push(service); + assert.strictEqual(service.diagnostics.state, 'disabled'); + sinon.assert.notCalled(factory); + }); + + test('consent revocation during module loading cannot publish a late client', async () => { + const loaded = createDeferred(); + let options: ExperimentationClientOptions | undefined; + const service = start((value) => { options = value; return loaded.promise; }); + const initialized = service.initializePromise; + changeConsent(false); + await initialized; + assert.ok(options); + const lateClient = new FakeSdk(options); + loaded.resolve(lateClient); + await clock.tickAsync(0); + sinon.assert.calledOnce(lateClient.dispose); + assert.strictEqual(service.diagnostics.state, 'disabled'); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + + test('a thrown treatment query is contained without losing initialization', async () => { + await cache(); + const service = start(); + await service.initializePromise; + sinon.stub(clients[0], 'getTreatmentVariable').throws(new Error('query failed')); + assert.strictEqual(service.getTreatmentVariable('example', false), false); + assert.strictEqual(service.diagnostics.state, 'ready'); + sinon.assert.calledOnce(warn); + }); +}); diff --git a/src/test/common/experimentation/storage.unit.test.ts b/src/test/common/experimentation/storage.unit.test.ts new file mode 100644 index 000000000..30a3c7557 --- /dev/null +++ b/src/test/common/experimentation/storage.unit.test.ts @@ -0,0 +1,196 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import * as sinon from 'sinon'; +import { + EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + EXPERIMENTATION_IDENTITY_PARAMETER, + type ExperimentationConfiguration, +} from '../../../common/experimentation/configuration'; +import { + ExperimentationStorage, + type SdkTargetingValues, + TAS_CACHE_KEY, +} from '../../../common/experimentation/storage'; +import * as logging from '../../../common/logging'; +import { MockMemento } from '../../mocks/mementos'; + +const configuration: ExperimentationConfiguration = { + assignmentsEndpoint: EXPERIMENTATION_ASSIGNMENTS_ENDPOINT, + targetPopulation: 'public', + identityParameter: EXPERIMENTATION_IDENTITY_PARAMETER, + assignmentParameters: { + devdeviceid: 'devDeviceId', + approved_language: 'language', + }, +}; +const assignments = { + features: [], + assignmentContext: 'control;', + configs: [{ Id: 'vscode', Parameters: { example: false } }], +}; + +function resolvedParameters(identity: string, language = 'en'): ReadonlyMap { + return new Map([ + ['devdeviceid', identity], + ['approved_language', language], + ]); +} + +function sdkTargetingValues(overrides: Partial = {}): SdkTargetingValues { + return { + applicationVersion: '1.110.0', + build: 'Visual Studio Code', + clientId: 'machine-a', + language: 'en', + ...overrides, + }; +} + +suite('Experimentation storage', () => { + teardown(() => sinon.restore()); + + test('persists through globalState and reuses the same resolved parameters in any order', async () => { + const globalState = new MockMemento(); + const first = new ExperimentationStorage( + globalState, + configuration, + resolvedParameters('device-a'), + sdkTargetingValues(), + '1.0.0', + () => true, + ); + await globalState.update('unrelated', 1); + await first.update(TAS_CACHE_KEY, assignments); + const restarted = new ExperimentationStorage(globalState, configuration, new Map([ + ['approved_language', 'en'], + ['devdeviceid', 'device-a'], + ]), sdkTargetingValues(), '1.0.0', () => true); + assert.deepStrictEqual(restarted.get(TAS_CACHE_KEY), assignments); + assert.strictEqual(restarted.hasCachedAssignments(), true); + assert.deepStrictEqual(restarted.keys(), [TAS_CACHE_KEY]); + assert.strictEqual(globalState.get('unrelated'), 1); + assert.ok(globalState.keys().every((key) => !key.includes('device-a') && !key.includes('machine-a'))); + }); + + test('does not reuse another identity, targeting value, audience, endpoint or extension version', async () => { + const globalState = new MockMemento(); + const original = new ExperimentationStorage( + globalState, + configuration, + resolvedParameters('device-a'), + sdkTargetingValues({ clientId: 'machine-a' }), + '1', + () => true, + ); + await original.update(TAS_CACHE_KEY, assignments); + const others = [ + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-b'), + sdkTargetingValues({ clientId: 'machine-a' }), '1', () => true, + ), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a', 'fr'), + sdkTargetingValues({ clientId: 'machine-a' }), '1', () => true, + ), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a'), + sdkTargetingValues({ clientId: 'machine-a', language: 'fr' }), '1', () => true, + ), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a'), + sdkTargetingValues({ applicationVersion: '1.111.0', clientId: 'machine-a' }), '1', () => true, + ), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a'), + sdkTargetingValues({ build: 'Visual Studio Code - Insiders', clientId: 'machine-a' }), '1', + () => true, + ), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a'), + sdkTargetingValues({ clientId: 'machine-b' }), '1', () => true, + ), + new ExperimentationStorage( + globalState, + { ...configuration, targetPopulation: 'insider' }, + resolvedParameters('device-a'), + sdkTargetingValues({ clientId: 'machine-a' }), + '1', + () => true, + ), + new ExperimentationStorage(globalState, { + ...configuration, assignmentsEndpoint: 'https://other.example.invalid/api/v1/assignments', + }, resolvedParameters('device-a'), sdkTargetingValues({ clientId: 'machine-a' }), '1', () => true), + new ExperimentationStorage( + globalState, configuration, resolvedParameters('device-a'), + sdkTargetingValues({ clientId: 'machine-a' }), '2', () => true, + ), + ]; + assert.ok(others.every((storage) => !storage.hasCachedAssignments())); + }); + + test('an empty but valid assignment response is a cached snapshot', async () => { + const storage = new ExperimentationStorage( + new MockMemento(), + configuration, + resolvedParameters('a'), + sdkTargetingValues({ clientId: 'a' }), + '1', + () => true, + ); + await storage.update(TAS_CACHE_KEY, { features: [], assignmentContext: '', configs: [] }); + assert.strictEqual(storage.hasCachedAssignments(), true); + }); + + test('malformed cache is ignored and reported without logging its contents', async () => { + const warn = sinon.stub(logging, 'traceWarn'); + const storage = new ExperimentationStorage( + new MockMemento(), + configuration, + resolvedParameters('a'), + sdkTargetingValues({ clientId: 'a' }), + '1', + () => true, + ); + await storage.update(TAS_CACHE_KEY, 'malformed private payload'); + assert.strictEqual(storage.hasCachedAssignments(), false); + assert.strictEqual(storage.get(TAS_CACHE_KEY), undefined); + sinon.assert.calledOnce(warn); + assert.ok(!JSON.stringify(warn.args).includes('private payload')); + }); + + test('does not write after its SDK generation is stopped', async () => { + const globalState = new MockMemento(); + let active = true; + const storage = new ExperimentationStorage( + globalState, + configuration, + resolvedParameters('a'), + sdkTargetingValues({ clientId: 'a' }), + '1', + () => active, + ); + await storage.update(TAS_CACHE_KEY, assignments); + active = false; + await storage.update(TAS_CACHE_KEY, { features: [], assignmentContext: 'late;', configs: [] }); + assert.deepStrictEqual(storage.get(TAS_CACHE_KEY), assignments); + }); + + test('observes persistence failures even when the SDK does not await writes', async () => { + const globalState = new MockMemento(); + sinon.stub(globalState, 'update').rejects(new Error('storage unavailable')); + const warn = sinon.stub(logging, 'traceWarn'); + const storage = new ExperimentationStorage( + globalState, + configuration, + resolvedParameters('a'), + sdkTargetingValues({ clientId: 'a' }), + '1', + () => true, + ); + await assert.doesNotReject(storage.update(TAS_CACHE_KEY, assignments)); + sinon.assert.calledOnce(warn); + assert.strictEqual(storage.hasCachedAssignments(), false); + }); +}); diff --git a/src/test/common/experimentation/telemetry.unit.test.ts b/src/test/common/experimentation/telemetry.unit.test.ts new file mode 100644 index 000000000..be7cfe1ba --- /dev/null +++ b/src/test/common/experimentation/telemetry.unit.test.ts @@ -0,0 +1,121 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import * as sinon from 'sinon'; +import { ENVS_EXTENSION_ID } from '../../../common/constants'; +import { ExperimentationTelemetry } from '../../../common/experimentation/telemetry'; +import * as logging from '../../../common/logging'; +import { EventNames } from '../../../common/telemetry/constants'; +import { getSharedTelemetryProperties, setSharedTelemetryProperty } from '../../../common/telemetry/reporter'; +import * as sender from '../../../common/telemetry/sender'; + +suite('Experimentation telemetry adapter', () => { + let active: boolean; + let adapter: ExperimentationTelemetry; + let send: sinon.SinonStub; + let onCall: sinon.SinonSpy; + let warn: sinon.SinonStub; + + setup(() => { + active = true; + send = sinon.stub(sender, 'sendTelemetryEvent'); + warn = sinon.stub(logging, 'traceWarn'); + onCall = sinon.spy(); + adapter = new ExperimentationTelemetry(() => active, onCall); + }); + + teardown(() => { + setSharedTelemetryProperty('abexp.assignmentcontext', undefined); + sinon.restore(); + }); + + test('shares assignment context and clears it when the SDK has no assignments', () => { + adapter.setSharedProperty('abexp.assignmentcontext', 'control;'); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'control;' }); + adapter.setSharedProperty('abexp.assignmentcontext', ''); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + + test('forwards classified query names but not unrelated SDK properties', () => { + adapter.postEvent('query-expfeature', new Map([ + ['ABExp.queriedFeature', 'vscode.generalFeature'], + ['machineId', 'must-not-be-forwarded'], + ['path', 'must-not-be-forwarded'], + ])); + sinon.assert.calledOnceWithExactly(send, EventNames.EXPERIMENTATION_QUERY, undefined, { + 'ABExp.queriedFeature': 'vscode.generalFeature', + }); + }); + + test('reports assignments and legacy outcomes independently using the environments identity', () => { + for (const callType of ['legacy', 'assignments']) { + adapter.postEvent('tas-call', new Map([ + ['callType', callType], ['outcome', 'Success'], + ['extensionName', 'not-the-environments-extension'], ['assignmentContext', `${callType};`], + ])); + } + assert.strictEqual(onCall.callCount, 2); + assert.strictEqual(send.callCount, 2); + assert.deepStrictEqual(send.secondCall.args, [EventNames.EXPERIMENTATION_TAS_CALL, undefined, { + callType: 'assignments', outcome: 'Success', extensionName: ENVS_EXTENSION_ID, + assignmentContext: 'assignments;', + }]); + assert.deepStrictEqual( + getSharedTelemetryProperties(), {}, 'per-request context is not the merged SDK context', + ); + }); + + test('forwards only classified fetch failure categories', () => { + for (const event of ['call-tas-error', 'call-assignments-error']) { + adapter.postEvent(event, new Map([['ErrorType', 'NoResponse'], ['headers', 'private']])); + } + assert.deepStrictEqual(send.firstCall.args, ['call-tas-error', undefined, { ErrorType: 'NoResponse' }]); + assert.deepStrictEqual( + send.secondCall.args, ['call-assignments-error', undefined, { ErrorType: 'NoResponse' }], + ); + }); + + test('keeps diagnostic counts numeric and drops arbitrary extra properties', () => { + adapter.postEvent('assignments-validation', new Map([ + ['FeatureVariableCount', '3'], ['AssignedVariantCount', '1'], ['DataVersion', '2'], + ['AssignmentContext', 'a;'], ['request', 'private'], + ])); + sinon.assert.calledOnceWithExactly(send, EventNames.EXPERIMENTATION_ASSIGNMENTS_VALIDATION, { + FeatureVariableCount: 3, AssignedVariantCount: 1, + }, { DataVersion: '2', AssignmentContext: 'a;' }); + }); + + test('does not forward invalid counters as measurements', () => { + adapter.postEvent('assignments-validation', new Map([ + ['FeatureVariableCount', '-1'], ['AssignedVariantCount', 'not a number'], + ])); + assert.deepStrictEqual(send.firstCall.args[1], {}); + }); + + test('drops unknown events, raw headers, and unsupported shared properties with a bounded warning', () => { + adapter.postEvent('report-headers', new Map([['clientId', 'private']])); + adapter.postEvent('report-headers', new Map([['clientId', 'private']])); + adapter.setSharedProperty('unclassified', 'private'); + sinon.assert.notCalled(send); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + assert.strictEqual(warn.callCount, 2); + }); + + test('refuses malformed SDK outcomes and path-like query names', () => { + adapter.postEvent('tas-call', new Map([['callType', 'unknown'], ['outcome', 'Success']])); + adapter.postEvent('query-expfeature', new Map([['ABExp.queriedFeature', '/vscode/private/path']])); + adapter.postEvent('call-tas-error', new Map([['ErrorType', 'Success']])); + sinon.assert.notCalled(send); + sinon.assert.notCalled(onCall); + }); + + test('late callbacks after consent revocation or disposal cannot restore attribution or send events', () => { + active = false; + adapter.setSharedProperty('abexp.assignmentcontext', 'stale;'); + adapter.postEvent('tas-call', new Map([['callType', 'assignments'], ['outcome', 'Success']])); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + sinon.assert.notCalled(send); + sinon.assert.notCalled(onCall); + }); +}); diff --git a/src/test/common/experimentation/transport.unit.test.ts b/src/test/common/experimentation/transport.unit.test.ts new file mode 100644 index 000000000..57ed2d42f --- /dev/null +++ b/src/test/common/experimentation/transport.unit.test.ts @@ -0,0 +1,147 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import { EventEmitter } from 'node:events'; +import type { ClientRequest, IncomingMessage, RequestOptions } from 'node:http'; +import * as sinon from 'sinon'; +import { createExperimentationFetch, TAS_REQUEST_TIMEOUT_MS } from '../../../common/experimentation/transport'; + +suite('Experimentation transport', () => { + let clock: sinon.SinonFakeTimers; + let controller: AbortController; + let request: ClientRequest; + let response: IncomingMessage; + let requestStub: sinon.SinonStub; + let end: sinon.SinonStub; + let destroy: sinon.SinonStub; + let respond: (() => void) | undefined; + + setup(() => { + clock = sinon.useFakeTimers(); + controller = new AbortController(); + end = sinon.stub(); + destroy = sinon.stub(); + request = Object.assign(new EventEmitter(), { end, destroy }) as unknown as ClientRequest; + response = Object.assign(new EventEmitter(), { statusCode: 200 }) as unknown as IncomingMessage; + destroy.callsFake((error: Error) => { + request.emit('error', error); + return request; + }); + requestStub = sinon.stub().callsFake(( + _url: string, + options: RequestOptions, + callback: (message: IncomingMessage) => void, + ) => { + respond = () => callback(response); + options.signal?.addEventListener('abort', () => request.destroy(new Error('aborted')), { once: true }); + return request; + }); + }); + + teardown(() => { + clock.restore(); + sinon.restore(); + }); + + test('passes method, headers, body and cancellation through the HTTPS stack', async () => { + const fetch = createExperimentationFetch(controller.signal, requestStub); + const pending = fetch('https://assignments.example.invalid/api/v1/assignments', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{"value":true}', + }); + respond!(); + response.emit('data', Buffer.from('{"value":')); + response.emit('data', Buffer.from('true}')); + response.emit('end'); + const result = await pending; + assert.strictEqual(result.status, 200); + assert.deepStrictEqual(await result.json(), { value: true }); + sinon.assert.calledOnceWithExactly(end, '{"value":true}'); + assert.strictEqual(requestStub.firstCall.args[1].signal, controller.signal); + assert.strictEqual(requestStub.firstCall.args[1].headers['Content-Length'], '14'); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('preserves HTTP failure status without pretending malformed bodies are valid JSON', async () => { + response.statusCode = 503; + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + respond!(); + response.emit('data', Buffer.from('unavailable')); + response.emit('end'); + const result = await pending; + assert.strictEqual(result.status, 503); + await assert.rejects(result.json()); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('network failure is distinguishable from a server response', async () => { + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + const assertion = assert.rejects(pending, { responseReceived: false }); + request.emit('error', new Error('offline')); + await assertion; + assert.strictEqual(clock.countTimers(), 0); + }); + + test('a deadline destroys a request and releases its timer', async () => { + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + const assertion = assert.rejects(pending, /timed out/); + await clock.tickAsync(TAS_REQUEST_TIMEOUT_MS); + await assertion; + sinon.assert.calledOnce(destroy); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('revocation aborts an in-flight request', async () => { + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + const assertion = assert.rejects(pending, /aborted/); + controller.abort(); + await assertion; + sinon.assert.calledOnce(destroy); + assert.strictEqual(clock.countTimers(), 0); + }); + + test('never starts a request after its lifetime is cancelled', async () => { + controller.abort(); + await assert.rejects(createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + )); + sinon.assert.notCalled(requestStub); + }); + + test('rejects plaintext requests rather than sending identifiers over HTTP', async () => { + await assert.rejects(createExperimentationFetch(controller.signal, requestStub)( + 'http://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + )); + sinon.assert.notCalled(requestStub); + }); + + test('caps response bodies', async () => { + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + const assertion = assert.rejects(pending, /size limit/); + respond!(); + response.emit('data', Buffer.alloc(2 * 1024 * 1024 + 1)); + await assertion; + sinon.assert.calledOnce(destroy); + }); + + test('an interrupted response is a failure, not a successful empty body', async () => { + const pending = createExperimentationFetch(controller.signal, requestStub)( + 'https://assignments.example.invalid/api/v1/assignments', { method: 'GET', headers: {} }, + ); + const assertion = assert.rejects(pending, { responseReceived: true }); + respond!(); + response.emit('aborted'); + await assertion; + assert.strictEqual(clock.countTimers(), 0); + }); +}); diff --git a/src/test/common/telemetry/reporter.unit.test.ts b/src/test/common/telemetry/reporter.unit.test.ts new file mode 100644 index 000000000..6c20d69c2 --- /dev/null +++ b/src/test/common/telemetry/reporter.unit.test.ts @@ -0,0 +1,212 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +import assert from 'node:assert'; +import * as sinon from 'sinon'; +import type { Disposable, TelemetryLogger, TelemetrySender } from 'vscode'; +import * as envApis from '../../../common/env.apis'; +import * as logging from '../../../common/logging'; +import { + getSharedTelemetryProperties, + getTelemetryReporter, + registerTelemetryReporter, + setSharedTelemetryProperty, +} from '../../../common/telemetry/reporter'; +import { createDeferred } from '../../../common/utils/deferred'; +import { vscMockTelemetryReporter } from '../../mocks/vsc/telemetryReporter'; + +suite('Telemetry reporter lifecycle', () => { + let registrations: Disposable[]; + let automaticSenders: TelemetrySender[]; + let automaticLoggers: TelemetryLogger[]; + let createLogger: sinon.SinonStub; + + setup(() => { + registrations = []; + automaticSenders = []; + automaticLoggers = []; + createLogger = sinon.stub(envApis, 'createTelemetryLogger').callsFake((sender: TelemetrySender) => { + const logger: TelemetryLogger = { + isUsageEnabled: true, + isErrorsEnabled: true, + logUsage: sinon.spy(), + logError: sinon.spy(), + onDidChangeEnableStates: () => ({ dispose: () => undefined }), + dispose: sinon.spy(), + }; + automaticSenders.push(sender); + automaticLoggers.push(logger); + return logger; + }); + }); + + teardown(async () => { + await Promise.all(registrations.map((registration) => registration.dispose())); + sinon.restore(); + }); + + function register(): Disposable { + const registration = registerTelemetryReporter(); + registrations.push(registration); + return registration; + } + + test('does not create a reporter without an active registration', () => { + assert.strictEqual(getTelemetryReporter(), undefined); + sinon.assert.notCalled(createLogger); + }); + + test('reuses one lazy reporter until disposal', async () => { + const registration = register(); + const first = getTelemetryReporter(); + assert.ok(first); + assert.strictEqual(getTelemetryReporter(), first); + sinon.assert.calledOnce(createLogger); + const dispose = sinon.spy(vscMockTelemetryReporter.prototype, 'dispose'); + await registration.dispose(); + assert.strictEqual(dispose.callCount, 1); + sinon.assert.calledOnce(automaticLoggers[0].dispose as sinon.SinonSpy); + assert.strictEqual(getTelemetryReporter(), undefined); + register(); + assert.ok(getTelemetryReporter()); + assert.notStrictEqual(getTelemetryReporter(), first); + await registration.dispose(); + assert.strictEqual(dispose.callCount, 1, 'old cleanup must not dispose the next reporter'); + }); + + test('copies shared properties and removes assignment context on disposal', async () => { + const registration = register(); + setSharedTelemetryProperty('abexp.assignmentcontext', 'first;'); + const first = getSharedTelemetryProperties(); + setSharedTelemetryProperty('abexp.assignmentcontext', 'second;'); + assert.deepStrictEqual(first, { 'abexp.assignmentcontext': 'first;' }); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'second;' }); + await registration.dispose(); + assert.deepStrictEqual(getSharedTelemetryProperties(), {}); + }); + + test('an obsolete registration cannot dispose the current reporter or clear its context', async () => { + const obsolete = register(); + register(); + const current = getTelemetryReporter(); + assert.ok(current); + const dispose = sinon.spy(vscMockTelemetryReporter.prototype, 'dispose'); + setSharedTelemetryProperty('abexp.assignmentcontext', 'current;'); + await obsolete.dispose(); + assert.strictEqual(getTelemetryReporter(), current); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'current;' }); + sinon.assert.notCalled(dispose); + }); + + test('blocks creation during disposal and preserves a newer registration when flushing finishes', async () => { + const flush = createDeferred(); + sinon.stub(vscMockTelemetryReporter.prototype, 'dispose').returns(flush.promise); + const oldRegistration = register(); + const old = getTelemetryReporter(); + assert.ok(old); + const closing = oldRegistration.dispose(); + assert.strictEqual(getTelemetryReporter(), undefined); + + register(); + const current = getTelemetryReporter(); + assert.ok(current); + assert.notStrictEqual(current, old); + setSharedTelemetryProperty('abexp.assignmentcontext', 'current;'); + flush.resolve(); + await closing; + assert.strictEqual(getTelemetryReporter(), current); + assert.deepStrictEqual(getSharedTelemetryProperties(), { 'abexp.assignmentcontext': 'current;' }); + }); + + test('disables duplicate SDK exception collection and preserves automatic-error common properties', () => { + register(); + const reporter = getTelemetryReporter(); + assert.ok(reporter instanceof vscMockTelemetryReporter); + assert.strictEqual(reporter.initializationOptions?.ignoreUnhandledErrors, true); + const options = createLogger.firstCall.args[1]; + assert.strictEqual(options.ignoreUnhandledErrors, undefined, 'the owned logger handles automatic exceptions'); + assert.strictEqual(options.additionalCommonProperties['common.os'], process.platform); + assert.strictEqual(options.additionalCommonProperties['common.nodeArch'], process.arch); + assert.match(options.additionalCommonProperties['common.telemetryclientversion'], /^\d+\.\d+\.\d+/); + }); + + test('automatic exceptions preserve event names, measurements, and per-event assignment snapshots', () => { + register(); + const reporter = getTelemetryReporter(); + assert.ok(reporter); + const send = sinon.spy(vscMockTelemetryReporter.prototype, 'sendDangerousTelemetryEvent'); + const error = new Error('already cleaned by VS Code'); + const properties = { 'common.extname': 'test', 'abexp.assignmentcontext': 'wrong;' }; + setSharedTelemetryProperty('abexp.assignmentcontext', 'control;'); + automaticSenders[0].sendErrorData(error, { properties, measurements: { duration: 12 } }); + assert.deepStrictEqual(send.firstCall.args, ['unhandlederror', { + 'common.extname': 'test', + 'abexp.assignmentcontext': 'control;', + name: error.name, + message: error.message, + stack: error.stack, + }, { duration: 12 }]); + setSharedTelemetryProperty('abexp.assignmentcontext', 'treatment;'); + automaticSenders[0].sendErrorData(error); + assert.strictEqual(send.firstCall.args[1]?.['abexp.assignmentcontext'], 'control;'); + assert.strictEqual(send.secondCall.args[1]?.['abexp.assignmentcontext'], 'treatment;'); + assert.strictEqual(properties['abexp.assignmentcontext'], 'wrong;', 'input properties are not mutated'); + setSharedTelemetryProperty('abexp.assignmentcontext', undefined); + automaticSenders[0].sendErrorData(error); + assert.ok(!Object.prototype.hasOwnProperty.call(send.thirdCall.args[1], 'abexp.assignmentcontext')); + }); + + test('automatic exceptions independently enforce error consent', () => { + register(); + const reporter = getTelemetryReporter(); + assert.ok(reporter); + const send = sinon.spy(vscMockTelemetryReporter.prototype, 'sendDangerousTelemetryEvent'); + for (const level of ['off', 'crash'] as const) { + reporter.telemetryLevel = level; + automaticSenders[0].sendErrorData(new Error('disabled')); + } + sinon.assert.notCalled(send); + reporter.telemetryLevel = 'error'; + automaticSenders[0].sendErrorData(new Error('errors allowed')); + sinon.assert.calledOnce(send); + reporter.telemetryLevel = 'all'; + automaticSenders[0].sendErrorData(new Error('all allowed')); + sinon.assert.calledTwice(send); + }); + + test('automatic error callbacks cannot send after disposal or through an obsolete reporter', async () => { + const first = register(); + assert.ok(getTelemetryReporter()); + const oldSender = automaticSenders[0]; + const send = sinon.spy(vscMockTelemetryReporter.prototype, 'sendDangerousTelemetryEvent'); + await first.dispose(); + oldSender.sendErrorData(new Error('after disposal')); + assert.strictEqual(getTelemetryReporter(), undefined); + register(); + assert.ok(getTelemetryReporter()); + oldSender.sendErrorData(new Error('after replacement')); + sinon.assert.notCalled(send); + automaticSenders[1].sendErrorData(new Error('current')); + sinon.assert.calledOnce(send); + }); + + test('automatic error transport failures are contained and logged', () => { + register(); + assert.ok(getTelemetryReporter()); + sinon.stub(vscMockTelemetryReporter.prototype, 'sendDangerousTelemetryEvent').throws(new Error('unavailable')); + const log = sinon.stub(logging, 'traceError'); + assert.doesNotThrow(() => automaticSenders[0].sendErrorData(new Error('original'))); + sinon.assert.calledOnce(log); + }); + + test('failed automatic-error registration disposes its reporter and permits a fresh attempt', async () => { + register(); + createLogger.onFirstCall().throws(new Error('logger unavailable')); + const dispose = sinon.spy(vscMockTelemetryReporter.prototype, 'dispose'); + assert.throws(() => getTelemetryReporter(), /logger unavailable/); + sinon.assert.calledOnce(dispose); + const reporter = getTelemetryReporter(); + assert.ok(reporter); + assert.notStrictEqual(reporter, dispose.firstCall.thisValue); + }); +}); diff --git a/src/test/common/telemetry/sender.unit.test.ts b/src/test/common/telemetry/sender.unit.test.ts index 5de98e59e..f0719c30c 100644 --- a/src/test/common/telemetry/sender.unit.test.ts +++ b/src/test/common/telemetry/sender.unit.test.ts @@ -1,20 +1,33 @@ import assert from 'node:assert'; import * as sinon from 'sinon'; +import type { Disposable } from 'vscode'; +import type { EnvironmentManager } from '../../../api'; import { EventNames } from '../../../common/telemetry/constants'; import { sendTelemetryEvent } from '../../../common/telemetry/sender'; +import * as logging from '../../../common/logging'; +import { + getTelemetryReporter, + registerTelemetryReporter, + setSharedTelemetryProperty, +} from '../../../common/telemetry/reporter'; +import { createDeferred } from '../../../common/utils/deferred'; +import { InternalEnvironmentManager } from '../../../managers/common/registeredManagers'; import { vscMockTelemetryReporter } from '../../mocks/vsc/telemetryReporter'; suite('Telemetry sender', () => { let originalTestExecution: string | undefined; let sendTelemetryStub: sinon.SinonStub; + let registration: Disposable; setup(() => { originalTestExecution = process.env.VSC_PYTHON_CI_TEST; delete process.env.VSC_PYTHON_CI_TEST; + registration = registerTelemetryReporter(); sendTelemetryStub = sinon.stub(vscMockTelemetryReporter.prototype, 'sendTelemetryEvent'); }); - teardown(() => { + teardown(async () => { + await registration.dispose(); sinon.restore(); if (originalTestExecution === undefined) { delete process.env.VSC_PYTHON_CI_TEST; @@ -37,4 +50,121 @@ suite('Telemetry sender', () => { { duration: 120_000, stageDuration: 45_000 }, ]); }); + + test('attaches current assignments to ordinary events without changing measurements', () => { + setSharedTelemetryProperty('abexp.assignmentcontext', 'control;'); + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 25); + const first = sendTelemetryStub.firstCall.args; + const firstProperties = sendTelemetryStub.firstCall.args[1]; + assert.deepStrictEqual(first, [ + EventNames.EXTENSION_ACTIVATION_DURATION, { 'abexp.assignmentcontext': 'control;' }, { duration: 25 }, + ]); + setSharedTelemetryProperty('abexp.assignmentcontext', 'treatment;'); + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 30); + assert.strictEqual(firstProperties['abexp.assignmentcontext'], 'control;', 'sent events keep their snapshot'); + assert.strictEqual(sendTelemetryStub.secondCall.args[1]['abexp.assignmentcontext'], 'treatment;'); + }); + + test('attaches assignments to error events as well as normal events', () => { + const errors = sinon.stub(vscMockTelemetryReporter.prototype, 'sendTelemetryErrorEvent'); + setSharedTelemetryProperty('abexp.assignmentcontext', 'control;'); + const error = new Error('test failure'); + sendTelemetryEvent(EventNames.ENVIRONMENT_DISCOVERY, 5, { + managerId: 'test', result: 'error', errorType: 'unknown', + }, error); + assert.strictEqual(errors.callCount, 1); + const properties = errors.firstCall.args[1]; + assert.ok(properties); + assert.strictEqual(properties['abexp.assignmentcontext'], 'control;'); + assert.strictEqual(properties.errorName, 'Error'); + assert.deepStrictEqual(errors.firstCall.args[2], { duration: 5 }); + sinon.assert.notCalled(sendTelemetryStub); + }); + + test('shared attribution wins over colliding per-event properties', () => { + setSharedTelemetryProperty('abexp.assignmentcontext', 'sdk-owned;'); + const properties = { managerId: 'test', 'abexp.assignmentcontext': 'wrong;' }; + sendTelemetryEvent(EventNames.ENVIRONMENT_MANAGER_REGISTERED, undefined, properties); + assert.strictEqual(sendTelemetryStub.firstCall.args[1]['abexp.assignmentcontext'], 'sdk-owned;'); + assert.strictEqual(properties['abexp.assignmentcontext'], 'wrong;', 'caller properties are not mutated'); + }); + + test('cleared attribution is absent from subsequent events', () => { + setSharedTelemetryProperty('abexp.assignmentcontext', 'old;'); + setSharedTelemetryProperty('abexp.assignmentcontext', undefined); + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 1); + assert.deepStrictEqual(sendTelemetryStub.firstCall.args[1], {}); + }); + + test('test execution does not construct or send through the reporter', () => { + process.env.VSC_PYTHON_CI_TEST = '1'; + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 1); + sinon.assert.notCalled(sendTelemetryStub); + }); + + test('a reporter failure is logged without breaking the feature sending telemetry', () => { + const log = sinon.stub(logging, 'traceError'); + sendTelemetryStub.throws(new Error('reporter unavailable')); + assert.doesNotThrow(() => sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 1)); + sinon.assert.calledOnce(log); + }); + + test('drops normal and error events after disposal without creating another reporter', async () => { + const errors = sinon.stub(vscMockTelemetryReporter.prototype, 'sendTelemetryErrorEvent'); + const old = getTelemetryReporter(); + assert.ok(old); + const dispose = sinon.spy(vscMockTelemetryReporter.prototype, 'dispose'); + await registration.dispose(); + + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 1); + sendTelemetryEvent( + EventNames.ENVIRONMENT_DISCOVERY, 1, + { managerId: 'test', result: 'error' }, new Error('late failure'), + ); + + sinon.assert.notCalled(sendTelemetryStub); + sinon.assert.notCalled(errors); + sinon.assert.calledOnce(dispose); + assert.strictEqual(getTelemetryReporter(), undefined); + }); + + test('a delayed manager refresh completes without recreating telemetry after shutdown', async () => { + const completion = createDeferred(); + const manager: EnvironmentManager = { + name: 'test', + preferredPackageManagerId: 'test', + refresh: () => completion.promise, + getEnvironments: async () => [], + get: async () => undefined, + set: async () => undefined, + resolve: async () => undefined, + }; + const registered = new InternalEnvironmentManager('test:delayed', manager); + const old = getTelemetryReporter(); + assert.ok(old); + const refresh = registered.refresh(undefined); + await registration.dispose(); + completion.resolve(); + await refresh; + + sinon.assert.notCalled(sendTelemetryStub); + assert.strictEqual(getTelemetryReporter(), undefined); + }); + + test('a new registration restores sending with fresh attribution', async () => { + const old = registration; + const oldReporter = getTelemetryReporter(); + await old.dispose(); + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 1); + sinon.assert.notCalled(sendTelemetryStub); + + registration = registerTelemetryReporter(); + setSharedTelemetryProperty('abexp.assignmentcontext', 'new;'); + await old.dispose(); + sendTelemetryEvent(EventNames.EXTENSION_ACTIVATION_DURATION, 2); + + sinon.assert.calledOnce(sendTelemetryStub); + assert.notStrictEqual(sendTelemetryStub.firstCall.thisValue, oldReporter); + assert.strictEqual(sendTelemetryStub.firstCall.args[1]['abexp.assignmentcontext'], 'new;'); + }); }); diff --git a/src/test/mocks/vsc/telemetryReporter.ts b/src/test/mocks/vsc/telemetryReporter.ts index 02360e675..abca85b04 100644 --- a/src/test/mocks/vsc/telemetryReporter.ts +++ b/src/test/mocks/vsc/telemetryReporter.ts @@ -1,8 +1,42 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +import type { + ReplacementOption, + TelemetryEventMeasurements, + TelemetryEventProperties, + TelemetryReporter, +} from '@vscode/extension-telemetry'; +import type { TelemetryLoggerOptions } from 'vscode'; + export class vscMockTelemetryReporter { - public sendTelemetryEvent(): void { + public telemetryLevel: TelemetryReporter['telemetryLevel'] = 'all'; + + constructor( + _connectionString?: string, + _replacementOptions?: ReplacementOption[], + public readonly initializationOptions?: TelemetryLoggerOptions, + ) {} + + public sendTelemetryEvent( + _eventName: string, + _properties?: TelemetryEventProperties, + _measurements?: TelemetryEventMeasurements, + ): void { // Noop. } + + public sendTelemetryErrorEvent( + _eventName: string, + _properties?: TelemetryEventProperties, + _measurements?: TelemetryEventMeasurements, + ): void {} + + public sendDangerousTelemetryEvent( + _eventName: string, + _properties?: TelemetryEventProperties, + _measurements?: TelemetryEventMeasurements, + ): void {} + + public async dispose(): Promise {} } diff --git a/src/test/unittests.ts b/src/test/unittests.ts index 94a55469c..a340ed2dc 100644 --- a/src/test/unittests.ts +++ b/src/test/unittests.ts @@ -7,7 +7,7 @@ const Module = require('module'); type VSCode = typeof vscode; -const mockedVSCode: Partial = {}; +const mockedVSCode: Partial = { version: '1.110.0' }; export const mockedVSCodeNamespaces: { [P in keyof VSCode]?: VSCode[P] } = {}; const originalLoad = Module._load; @@ -42,6 +42,16 @@ export function initialize() { const clipboard = new MockClipboard(); when(mockedVSCodeNamespaces.env!.clipboard).thenReturn(clipboard); when(mockedVSCodeNamespaces.env!.appName).thenReturn('Insider'); + when(mockedVSCodeNamespaces.env!.createTelemetryLogger(anything(), anything())).thenCall( + (): vscode.TelemetryLogger => ({ + isUsageEnabled: true, + isErrorsEnabled: true, + logUsage: () => undefined, + logError: () => undefined, + onDidChangeEnableStates: () => new vscodeMocks.Disposable(() => undefined), + dispose: () => undefined, + }), + ); // This API is used in src/client/telemetry/telemetry.ts const extension = mock>(); @@ -59,7 +69,7 @@ export function initialize() { return mockedVSCode; } if (request === '@vscode/extension-telemetry') { - return { default: vscMockTelemetryReporter as any }; + return { TelemetryReporter: vscMockTelemetryReporter }; } // less files need to be in import statements to be converted to css // But we don't want to try to load them in the mock vscode