Skip to content

feat(python): add Python SDK for dfget - #87

Open
Chelics wants to merge 5 commits into
dragonflyoss:mainfrom
Chelics:feat/python-sdk
Open

Chelics wants to merge 5 commits into
dragonflyoss:mainfrom
Chelics:feat/python-sdk

Conversation

@Chelics

@Chelics Chelics commented Oct 29, 2026

Copy link
Copy Markdown

Implements dragonflyoss/dragonfly#4421.

Closes dragonflyoss/dragonfly#4421

What this adds

A synchronous Python SDK for dfget: dragonfly_sdk.Client talks to the local dfdaemon over its Unix domain socket and drives the DownloadTask gRPC service, so Python code can pull models and datasets through Dragonfly's P2P network.

from dragonfly_sdk import Client, HuggingFaceCredentials

with Client() as client:
    result = client.download(
        "hf://Qwen/Qwen2.5-7B/model.safetensors",
        "./model.safetensors",
        credentials=HuggingFaceCredentials(token="hf_..."),
        on_progress=lambda p: print(f"{p.percent:.1f}%"),
    )

Per the two-step approach discussed in the issue (Python gRPC bindings first — already published as dragonfly-api 2.3.8 — then the SDK), the SDK lives here alongside the existing Go and Rust client SDKs.

Scope (v1)

  • Single-file downloads from HTTP(S), S3-compatible object storage (s3/gs/abs/oss/obs/cos), HuggingFace (hf://), ModelScope (modelscope://) and OpenCSG (opencsg://).
  • Synchronous API, progress callback, typed exception hierarchy, context-manager lifecycle.
  • Request construction mirrors dfget defaults (priority 6, 2h timeout, the 30-item default filtered query params, optional proto fields only written when set) so the SDK and the CLI produce the same tasks in dfdaemon.
  • dfdaemon backend errors are decoded from grpc-status-details-bin into BackendError (message, headers, status_code).

Not in this PR: directory mode, --transfer-from-dfdaemon / piece content transfer, HDFS, asyncio. All are documented as limitations in python/README.md.

Layout

  • Built on the published dragonfly-api package, so no generated protobuf files are vendored.
  • python/pyproject.toml — hatchling, requires-python >= 3.9, PEP 639 license metadata, py.typed (PEP 561), PEP 621 extras (test, dev).
  • .github/workflows/ci-python.yml — ruff + mypy (strict) on 3.12, pytest on 3.9–3.13, wheel/sdist build + twine check + clean-venv import.
  • Root README.md, .gitignore, .github/dependabot.yml updated for the new package.

Testing

  • ruff check ., ruff format --check ., mypy src (strict) pass.
  • pytest -q -m "not integration": 144 passed on Python 3.9 and on 3.13. Unit tests run against an in-process fake dfdaemon over a temporary Unix socket.
  • python -m build + twine check dist/* pass; the wheel installs in a clean venv, imports, and ships py.typed.
  • An opt-in live test (DRAGONFLY_LIVE_TEST=1 pytest -q -m integration) covers the real dfdaemon path; it is skipped in CI.

Notes for reviewers

  • remote_ip is the one deliberate divergence from dfget: it depends on a local-IP probe, so the SDK leaves it unset.
  • Duplicate or unknown piece numbers are ignored rather than double-counted, and the output file is never deleted on failure.

Follow-ups (separate PRs)

  • PyPI publishing (trusted publishing plus a version/tag policy agreed with maintainers).
  • Directory mode and HDFS support.

🤖 Generated with Claude Code

Chelics and others added 5 commits October 29, 2026 10:21
Add Python cache/build ignores, a pip dependabot entry for /python, and a
Python row in the SDK table.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Chelics <1285873430@qq.com>
A synchronous client over dfdaemon's DownloadTask gRPC service that mirrors
dfget's request semantics for single-file downloads from HTTP(S), S3-compatible
object storage, HuggingFace, ModelScope and OpenCSG.

The package depends on the published dragonfly-api bindings, so no generated
protobuf code is vendored.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Chelics <1285873430@qq.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Chelics <1285873430@qq.com>
Unit tests run against an in-process fake dfdaemon over a temporary Unix
socket. The integration test is skipped unless DRAGONFLY_LIVE_TEST=1.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Chelics <1285873430@qq.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Chelics <1285873430@qq.com>
@Chelics

Chelics commented Oct 29, 2026

Copy link
Copy Markdown
Author

One note on CI: the Classify PR check is failing only because it requires exactly one of the bug / enhancement / documentation / dependencies labels, and contributors cannot add labels themselves. Could a maintainer add enhancement? Everything else (Lint, Test 3.9–3.13, Build, Check Size, DCO) is green — the failure is not a code issue.

DoubleCore pushed a commit to DoubleCore/dragonfly-sdk that referenced this pull request Nov 4, 2026
CLI-wrapper approach, complementary to dragonflyoss#87 (gRPC); mirrors python/dfcache_sdk from dragonflyoss#92.

Signed-off-by: DoubleCore <851678303@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement dfget's Python SDK

1 participant