Skip to content

cloud-archives.md

Latest commit

 

History

History
163 lines (119 loc) · 7.31 KB

cloud-archives.md

File metadata and controls

163 lines (119 loc) · 7.31 KB
Raw
title Cloud archives
nav_order 14
permalink /cloud-archives/

Cloud archives

{: .no_toc }

Use a separately deployed Worker service as an authenticated read and snapshot-publication boundary for a Gitcrawl archive. {: .fs-6 .fw-300 }

  1. TOC {:toc}

Responsibilities

Gitcrawl stores the Worker endpoint and archive identifier in its config and calls the remote service. The service itself is deployed separately from Gitcrawl.

Cloud mode does not replace the local or Git-backed portable-store workflows. A cloud-mode configuration sends supported read commands, including status and direct search, to the remote archive without creating a local SQLite database.

Configure an archive

gitcrawl init \
  --remote URL \
  --archive gitcrawl/openclaw__openclaw

Bearer-authenticated remote endpoints must use HTTPS. Plain HTTP is accepted only for loopback development endpoints.

Authenticate

gitcrawl remote login --endpoint URL --json
gitcrawl whoami --json
gitcrawl remote archives --json
gitcrawl remote status --json

remote login starts the service's GitHub OAuth flow, verifies organization and team membership server-side, and stores the returned signed bearer token in the operating-system keyring.

For non-browser bootstrap, name an environment variable that contains a GitHub token:

gitcrawl remote login \
  --endpoint URL \
  --github-token-env GITHUB_TOKEN \
  --json

The Worker verifies the GitHub token against the same organization and team policy. Gitcrawl stores only the remote session token.

Read from cloud mode

After cloud initialization and login, supported read commands use the configured archive:

gitcrawl status --json
gitcrawl search openclaw/openclaw --query "manifest cache" --json

Commands that require a writable local database fail in cloud mode instead of silently creating one.

Publish a local archive

Publication starts from a local SQLite archive:

gitcrawl cloud publish \
  --remote URL \
  --archive gitcrawl/openclaw__openclaw \
  --json

Before any upload or ingest, Gitcrawl checks the configured credential through the advertised /v1/whoami route and requires both publisher and reader roles. This also applies to stage-only publication.

The publisher freezes one local SQLite image and uses its SHA-256 digest as the snapshot identity. Repositories, threads, revisions, fingerprints, summaries, durable clusters, and pull request detail/file rows are exported from that same image. Gitcrawl negotiates the remote snapshot contract before changing R2, uploads a digest-scoped gzip bundle, and sends D1 data in row- and encoded-byte-bounded batches.

Gzip is lossless transport, not portable-store compaction. The cloud SQLite image retains complete canonical issue and pull request bodies, comments, revisions, review comments, and pull request patch text.

Staging and cutover

The remote must advertise gitcrawl.snapshot.staging.v1. Pass --stage-only to upload and validate an immutable snapshot without changing the archive served to unpinned readers:

gitcrawl cloud publish \
  --remote URL \
  --archive gitcrawl/openclaw__openclaw \
  --stage-only \
  --json

A later publish verifies the candidate through the publisher-only status projection. It skips repeated ingest only when the digest, source sync, schema, resolved publication profile, immutable warnings, generation timestamp, and coverage match.

Cutover requires reader-authenticated GET /sqlite. Gitcrawl validates the cutover acknowledgement, polls the scoped reader projection until its digest, profile, generation, and dataset coverage match, rechecks the publisher metadata, downloads the bound SQLite image, and verifies its hash before reporting success. Without --stage-only, a successful publish moves unpinned reads to the complete snapshot.

Incomplete local enrichment fails before remote mutation. --allow-incomplete is the explicit override. --observation-order publishes durable fetch ordering only after the remote operator fence is enabled.

Archive admission

To publish a raw archive with explicitly reported source and enrichment gaps:

gitcrawl cloud publish \
  --remote URL \
  --archive gitcrawl/openclaw__openclaw \
  --admission-policy=archive-v1 \
  --observation-order \
  --stage-only \
  --json

This opt-in policy requires the remote capabilities gitcrawl.archive-admission.v1 and gitcrawl.observation-order.v1. An older remote or a disabled observation fence fails preflight before upload. Do not combine the policy with --allow-incomplete. Omitting the policy retains the strict default.

Archive admission still requires SQLite integrity, compatible canonical tables, repositories, referential closure, full bodies, and the existing privacy scrub. Use the full runtime archive, not a lossy portable export. Native portable profiles that declare excerpts or excluded patches/history are rejected even when current bodies fit the excerpt limit. Empty patch rows in a full runtime are not proof of loss. The existing 4 GiB SQLite, 512 MiB gzip, eight-part, staging, and cutover limits still apply. Admission does not request summaries, embeddings, or other model work.

The JSON result includes typed admission evidence and stable warnings. Repository inventory observations distinguish unsupported, unknown, missing, partial, complete, and empty observations. Child observations also report stale reservations relative to the archived thread. Complete observations describe the recorded scope, not current GitHub state. Workflow freshness and current remote freshness remain unknown; export clocks never establish either. Source gaps, PR detail/file gaps, and all six enrichment metrics remain visible. Incomplete current revisions retain their actual coverage counts and complete: false; admission does not make stale enrichment current for readers.

The sanitized SQLite copy stores the same deterministic evidence in portable_metadata.cloud_admission_v1 before hashing. Existing source markers are not trusted, and strict exports remove them. The original database is not changed. Canonically sorted warnings are part of immutable snapshot identity, including stage-only replay and reader verification. Warnings alone never enable admission.

Privacy and retention

Digest-scoped bundles can contain private issue and pull request text. Bundle metadata declares both message-body and source-code sensitivity because patch text is retained.

Publication excludes raw API payloads, blob-backed payloads, local run diagnostics, source-code indexes, and machine-local paths. These are not part of the shared archive contract.

Gitcrawl intentionally has no remote deletion command. Operators should enable publication only when the remote deployment has bounded lifecycle rules for failed, superseded, and uncut staged bundles. --stage-only does not move retention responsibility back to the client.

Command summary

Command Purpose
gitcrawl init --remote URL --archive id Configure cloud mode
gitcrawl remote login Authenticate and store a remote session token
gitcrawl whoami Print the current remote identity
gitcrawl remote archives List archives visible to that identity
gitcrawl remote status Inspect the configured archive
gitcrawl cloud publish Publish a local snapshot to the Worker remote