Skip to content

feat(s3): add S3-compatible object storage connections - #6582

Open
perundio wants to merge 4 commits into
t8y2:mainfrom
perundio:feat/s3-object-storage
Open

feat(s3): add S3-compatible object storage connections#6582
perundio wants to merge 4 commits into
t8y2:mainfrom
perundio:feat/s3-object-storage

Conversation

@perundio

@perundio perundio commented Aug 18, 2026

Copy link
Copy Markdown

Summary

  • Add native S3-compatible object storage connections for AWS S3, MinIO, and other SigV4-compatible endpoints across desktop and web.
  • Finish the S3 browser with bucket management, folder/prefix creation, paginated object listing, upload/download/preview/delete, and copy/move/rename workflows.
  • Harden the implementation for merge by moving temporary S3 session tokens into the existing secret/session lifecycle, fixing SigV4 path canonicalization for special-character keys, rebasing onto the latest main, and expanding targeted test coverage.

What’s Included

  • Connection dialog support for endpoint, region, addressing style, optional session token, and read-only mode.
  • Native S3 runtime integration in dbx-core, dbx-web, and Tauri commands.
  • S3 browser UI with:
    • list/create/delete bucket
    • create folder/prefix
    • paginated ListObjectsV2 browsing with NextContinuationToken
    • upload, download, preview, delete object
    • copy, move, and rename object
  • Updated S3 driver branding/icon.
  • Secret-handling fixes so temporary S3 session tokens do not persist in plaintext when save_password = false.
  • SigV4 fixes for already-escaped paths plus regression coverage for path-style and virtual-hosted URLs with spaces, Unicode, %, and ? in keys.

Test Plan

  • cargo test -p dbx-core s3:: --lib
  • cargo test -p dbx-core session_credentials::tests --lib
  • pnpm typecheck
  • pnpm vitest run apps/desktop/src/components/s3/__tests__/S3Browser.spec.ts
  • Manual smoke test against local MinIO / AWS-compatible endpoint
    • connect using endpoint + credentials
    • create/delete bucket
    • create prefix, upload/download/preview/delete object
    • copy/move/rename object
    • browse paginated folders with 200+ entries
    • verify read-only mode blocks write operations

Notes For Review

  • The PR head has been rebased onto the current main and the previous merge conflict state is resolved.
  • The review-requested security and signing fixes are included in the latest head commit.

@github-actions github-actions Bot added area/core Shared DBX core runtime area/desktop Desktop application or Tauri shell area/web Web backend or web API dependencies/backend Adds a backend dependency enhancement New feature or request ui-change Changes user-visible interface, text, or visual assets labels Aug 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🌐 I18n autofill completed

✅ Added translations for 33 new zh-CN i18n keys.

Locale Added entries
es 33
it 33
ja 33
pt-BR 33
zh-TW 33

@perundio
perundio force-pushed the feat/s3-object-storage branch from c8e82cb to 8305aee Compare August 18, 2026 10:18

@t8y2 t8y2 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm requesting changes for now. Three issues need fixing before this is safe to merge:

  1. ConnectionDialog.vue stores the temporary S3 session token in external_config. The existing persistence, no-save-password, and cloud-sync scrubbers do not treat this field as a secret, so it can remain in plaintext configuration and shared runtime state. Please move it through the same secret-store/session-isolation lifecycle as the secret access key.

  2. sigv4.rs passes the already escaped Url::path() through percent_encode_path again. Keys containing spaces, Unicode, %, ?, or similar characters are therefore signed with %25 escapes that differ from the transmitted path, causing SignatureDoesNotMatch. Canonicalize the raw key exactly once and add path-style and virtual-hosted regression tests with special-character keys.

    The url crate documents that Url::set_path percent-encodes its input, while AWS SigV4 requires the canonical URI to match the transmitted absolute path: https://docs.rs/url/2.5.8/url/struct.Url.html#method.set_path and https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html

  3. S3Browser.vue always requests 200 entries with a null continuation token and ignores isTruncated/nextContinuationToken, so larger folders silently hide every later object and common prefix. Add load-more or bounded page accumulation with a 201+ object/prefix test. AWS documents that truncated ListObjectsV2 results must continue with NextContinuationToken: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-NextContinuationToken

GitHub also currently reports this head as conflicting, while the main CI workflow stopped at action_required without running any jobs.

@perundio
perundio force-pushed the feat/s3-object-storage branch from 8305aee to abf0e6c Compare August 19, 2026 12:07
@perundio

Copy link
Copy Markdown
Author

Thanks for the detailed review and for calling out the risky parts so precisely.

I’ve addressed the requested changes in the latest head:

  • moved the temporary S3 session token into the existing secret-store / session-isolation flow so it no longer stays in plaintext config when save_password = false
  • fixed SigV4 canonical URI handling so already-escaped paths are signed exactly as transmitted, and added regression tests for path-style and virtual-hosted URLs with special-character keys
  • updated S3Browser.vue to continue truncated ListObjectsV2 results via NextContinuationToken, and added a 201+ object/prefix UI test
  • rebased the branch onto the latest main and resolved the previous conflict state

Also took the opportunity to finish the missing object-browser workflows in the same pass: bucket delete, folder/prefix creation, download, and copy/move/rename operations.

Thanks again for the careful review.

@perundio
perundio requested a review from t8y2 August 19, 2026 12:18
@ysicing

ysicing commented Aug 23, 2026

Copy link
Copy Markdown

@perundio Does this PR intend to include S3 deep-link support (dbx://connection/new?...)?

anutre and others added 4 commits August 23, 2026 15:29
Introduce a native SigV4 S3 client with bucket browser UI for MinIO and AWS,
unifying desktop and web on PoolKind::S3 and fixing query-string signing for
list-objects delimiters plus Range header support for previews.

Co-authored-by: Cursor <cursoragent@cursor.com>
Move temporary S3 session tokens through the existing secret lifecycle, fix SigV4 path handling, and finish the S3 browser with pagination plus bucket/object management flows so the connector is safe and review-ready.

Co-authored-by: Cursor <cursoragent@cursor.com>
Fill the new S3 browser strings in secondary locales so the autofill check no longer depends on a partial machine-translation response.

Co-authored-by: Cursor <cursoragent@cursor.com>
@perundio
perundio force-pushed the feat/s3-object-storage branch from 03b3eb5 to 927e1dd Compare August 23, 2026 12:31
@github-actions github-actions Bot added area/multiple Touches more than three repository areas and removed area/desktop Desktop application or Tauri shell area/core Shared DBX core runtime area/web Web backend or web API labels Aug 23, 2026
@perundio

Copy link
Copy Markdown
Author

@ysicing No — this PR does not intend to add S3 deep-link support (dbx://connection/new?...).

Scope here is the S3/MinIO connection type plus the object browser (list/upload/download/copy/move/rename/delete, session-token handling, etc.). Deep-link / connectionUrl scheme support for S3 would be a separate follow-up if we want it.

@antwa

antwa commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Hi @t8y2 (and @perundio) — I noticed this PR while working on a similar native S3-compatible object storage driver on my own fork, and wanted to check in on its status before continuing further.

Is this PR still actively being reviewed, or is there a rough timeline for merging it? I want to avoid duplicating effort if this lands soon — happy to hold off on my own implementation, or coordinate if there's anything useful 😄

here is my current implementation
image
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/multiple Touches more than three repository areas dependencies/backend Adds a backend dependency enhancement New feature or request ui-change Changes user-visible interface, text, or visual assets

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants