fix(server): honor an S3 endpoint scheme, admit storage destinations under destinations-only, scope DuckLake secrets to their prefix

[Sha-Bang]

Three fixes that stand between a bucket file and a Malloy source, plus the docs for that recipe.

1. S3 endpoint scheme

An s3 attachment's endpoint reached DuckDB verbatim, so http://localhost:9000 was requested as https://http://localhost%3A9000/…. The AWS SDK listing path had the opposite bug: a bare host:port threw Invalid URL. No endpoint form worked on both.

parseS3Endpoint (gcs_s3_utils.ts) splits an endpoint into host[:port][/path] and useSsl. A leading http(s):// (any case) and trailing slashes are accepted; http:// is the only thing that turns TLS off (USE_SSL false on the secret). A query, fragment, whitespace, userinfo, another scheme, or an empty host is refused at load: none of those ever produced a working query, and DuckDB and the SDK parse them differently. Bare hosts produce byte-identical SQL. The key-based secret arms now share one clause list with the chain arm, which also fixes endpoint + sessionToken silently dropping the token.

2. S3_CREDENTIAL_CHAIN_POLICY=destinations-only rejected the destinations it exists for

Only validateStorageDestinations told the gate a config was a storage destination. The load re-validation, the destination Malloy config rebuild, serve-time attach and the build session all used the default "connection" origin, so under the policy a credential_chain destination was dropped (the rebuild catch installs an empty config) or failed not permitted. The existing origin argument is now threaded through those paths. attachDuckLakeReadOnly (no callers) and the attachDuckLake wrapper are deleted.

"storage-destination" is only set on inputs from validated storageDestinations; no connection route reaches it. destinations-only still fences the connection routes. Environment create/update accepting destinations is the existing contract, not widened here.

3. Scope each DuckLake storage secret to its own prefix

Found while reviewing (2). A modeler's raw-SQL dimension (sql_number(...)) is carried verbatim into the storage-serve shape and runs on the destination's DuckDB, which held an unscoped credential_chain secret (default scope: all of s3://). On AWS that is the worker's role, and the managed tier bucket is shared across orgs under per-org prefixes, so such a dimension could read another org's prefix. This predates the PR; it touches the same function, so it is fixed here.

Every DuckLake storage secret (S3 chain, S3 key, GCS) now carries SCOPE '<bucketUrl>/', derived in attachDuckLakeWithMode and carried on CloudStorageCredentials so the expired-credential renewer re-emits it. Destination S3 sessions also run SET enable_global_s3_configuration=false: otherwise DuckDB signs unmatched reads with a legacy key seeded from AWS_* env, which bypasses the scope. Generic attachedDatabases stay unscoped.

Not closed by this: an author's SQL still runs unsandboxed in the destination session (SSRF, local-file reads). That is a separate design question (refuse sql_* in serve-shape refinements, or sandbox the session).

Tested

• Unit: specs for the validator, assembleEnvironmentConnections, an environment-level destination lookup, the build-session list/cleanup paths, the endpoint forms on both secret arms and the SDK client, and scope emission (trailing-slash normalization, GCS, renewer). Dropping each threaded origin in turn fails a spec for the policy reason (the empty-list fallback rebuild can't be proven: nothing to validate).
• compile_restriction.spec.ts "accepts every committed .malloy file" fails locally on a missing packages/skills/skills directory, also on main; unrelated.
• Live, endpoint: an S3-compatible server (RustFS) on plain HTTP, environment DuckDB connection with endpoint: "http://localhost:9000", lake.table('s3://bucket/x.csv') extend { measure: total is amt.sum() }.

	origin/main	this branch
query	Could not resolve hostname … 'https://http://localhost%3A9000/bucket/x.csv'	total = 60, count = 3

So conn.table('s3://…') compiles and runs through a DuckDB attachment.

• Live, scope: DuckLake with a Postgres catalog and RustFS data path, through attachDuckLakeReadWrite with a prefix bucketUrl. A 200k-row CREATE TABLE AS wrote parquet under the prefix and read back correctly; nothing touched the bare prefix. Reads of a sibling prefix and another bucket returned 403. With enable_global_s3_configuration left on and AWS_* set, the same reads succeeded, which is why the SET is there. Not run: a real AWS bucket with an IRSA role. Worth a read-only serve attach, a build and a read-back on the AWS dev cell before release.

Docs

connections.md gains "Data files in a bucket or at a link"; configuration.md documents S3_CREDENTIAL_CHAIN_POLICY.

🤖 Generated with Claude Code

[Sha-Bang]

@claude review