Skip to content

bug(cloud): enrollment has no inverse and cloud status reports not configured while replication actively runs #1114

Description

@achangkuon

📋 Pre-flight Checks

Version: engram v2.0.0-rc.9 (Linux x86_64, WSL2)

📝 Bug Description

Two defects in the same surface, found on a store whose cloud backend was decommissioned months ago. sync_enrolled_projects held 4 projects enrolled since 2026-07-21; the server they replicate toward has been off since. Neither defect is the unbounded-queue problem of #1104 — that one assumes enrollment never happened. Here enrollment did happen, and the product offers no exit and reports the wrong state.


1. cloud enroll has no inverse

$ engram cloud
usage: engram cloud <subcommand> [options]
supported subcommands: status, enroll, config, serve, upgrade, repair, bootstrap

A project can be enrolled for cloud replication and can never be unenrolled. There is no unenroll, no disable, and no documented flag on any of the seven subcommands that reverses it.

The consequence is not cosmetic. While enrolled, mutations keep being enqueued against a dead target: 31,203 pending rows, 45.9 MB — larger than observations (25.5 MB), the data the queue exists to replicate. Growth continued at ~200–300 rows/day right up to the moment of writing.

The only way out was to delete rows from sync_enrolled_projects by hand, against the SQLite store. That is not a recovery path a user should have to find, and it silently bypasses whatever invariants the product maintains around enrollment.

Expected: engram cloud unenroll <project> (or an equivalent documented verb) that stops enqueueing for that project.

2. cloud status reports not configured while replication is actively running

$ engram cloud status
Cloud status: not configured

At the same moment, sync_state showed live push attempts against a real endpoint:

column value
target_key cloud
lifecycle pending
last_enqueued_seq 31203
last_acked_seq 21633
consecutive_failures 109
reason_code transport_failed
last_error push: transport push project "<redacted>": cloud: mutation push: Post "https://<redacted>/sync/mutations/push": context deadline exceeded
updated_at minutes before the status call

last_acked_seq = 21633 proves this target successfully replicated in the past — it is not an unconfigured store that never had a backend. Four additional targets exist (cloud:<redacted> ×4) with their own pending counters.

~/.engram/cloud.json is {"server_url": "", "token": ""}, which is presumably what status reads. But the enrollment rows and the sync targets are what actually drive the worker, and they disagree with the file.

Expected: cloud status reflects the state that governs behaviour — enrolled projects, live targets, pending backlog, consecutive failures and last error — rather than an empty config file. A user asking "is cloud on?" was told "no" while 109 failed pushes said otherwise.


Why they belong together

Both stem from enrollment being writable-once and invisible to the status surface. A user who enrolled, lost the backend, and asks the product what is going on gets not configured and no verb to stop it — while the queue quietly becomes the largest object in the database.

Impact

Silent unbounded growth toward a dead endpoint, with the diagnostic surface actively denying it is happening, and no supported remediation. Related to but not covered by #1104, whose store was never enrolled and whose status output was therefore correct.

Reproduction

  1. engram cloud config --server <url> and engram cloud enroll <project> against a reachable backend; let it sync so last_acked_seq > 0.
  2. Take the backend down permanently and clear cloud.json.
  3. Keep using engram normally.
  4. engram cloud statusnot configured, while sync_mutations keeps growing and sync_state.consecutive_failures climbs.
  5. Look for a way to unenroll. There is none.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions