Skip to content

Postmortem: coder.ddev.com drive failure and staging disruptions, 2026-11-16 #259

Open

Description

@rfay

Opened by Claude Code at @rfay's request.

Summary

On 2026-11-16 the system NVMe drive in coder.ddev.com failed. The server was down for about 7 hours 20 minutes, and Coder's database was lost because nothing backed it up. Workspace home directories survived on a separate drive. The same day, staging-coder.ddev.com had three smaller disruptions: Drupal 12 workspaces hung because of a Drush change, CI template pushes hit Coder's upload rate limit, and Hetzner swapped a worn workspace drive without warning.

All times are UTC.

coder.ddev.com: system drive failure

The production server lost its system drive and was rebuilt from the setup docs.

Timeline

Time Event
2026-11-16 07:10 The 1 TB system NVMe drive fails. Icinga checks start failing with execvpe ... Input/output error. Ping still works, but SSH, HTTP and HTTPS do not.
07:10–12:20 Icinga sends alerts. It is night in the maintainer's time zone, so nobody acts.
about 12:23 Recovery starts. A hardware reset from Hetzner Robot leaves the server unreachable. In the rescue system, the system drive is missing from PCIe. The two 512 GB drives pass SMART.
before 13:55 Hetzner replaces the failed drive. The NVMe device names change, so the workspace drive is no longer nvme1n1.
13:59 Ubuntu 24.04 is reinstalled on the new drive with installimage, keeping the workspace drive untouched.
14:10–14:49 Server set up again from the docs: TLS certificate, Coder, GitHub login, templates, workspace janitor, Discord notifications, Icinga agent.
about 14:30 Coder is usable again.
15:02, 15:20 Docker, Sysbox and Coder upgraded, then a reboot test of the new restart service (#247).
2026-11-17 02:38 First nightly encrypted backup to S3 succeeds (#249).

Impact

  • coder.ddev.com was unavailable for about 7 hours 20 minutes.
  • Coder's PostgreSQL database was lost: user accounts, templates, workspace records and API tokens. Users sign in again with GitHub and recreate their workspaces.
  • Workspace home directories survived, because they are on a separate drive. The maintainer's workspaces were reattached. The directories of two other users' workspaces were later deleted.
  • Docker's data and the registry mirror were lost and rebuilt.

Cause

The drive failed suddenly. In the two days before, Icinga showed no warning state changes, and nothing monitored drive health. The server had no RAID, so this one drive held the operating system, PostgreSQL and Docker. Nothing backed up the database (#198).

staging-coder.ddev.com

Staging had three separate problems on the same day.

Drupal 12 workspaces hung (from about 02:31)

Drush's 15.x branch replaced the drush command with a stub that called itself forever. Every Drupal 12 workspace stopped after "✓ Drush installed", and the Drupal 12 CI jobs failed. #250 pins drush/drush:<15, merged at 19:03. Drush fixed the stub upstream (drush-ops/drush#6682), but Drush 15 has no site:install, so the pin stays (#257).

CI template pushes rate-limited (19:03–19:05)

Three PRs were rebased while main's run was in progress, and about 30 CI jobs pushed templates in the same minute. Coder allows 12 file uploads per minute per user, so most jobs failed with You've been rate limited for sending more than 12 requests in 1m0s. #252 retries the push after a wait, merged at 20:34.

Workspace drive swapped without warning (19:53–about 20:15)

Both staging drives reported SMART "FAILED" with critical warning 0x04, at 134% wear and about 630 TB written. Hetzner explained that this warning only means the rated wear is exceeded. The available spare was 100%, with 0 media errors. Hetzner offered a replacement, and we asked for the workspace drive only. Hetzner shut the server down at 19:53 and swapped it with no time arranged.

  • The server came back at 19:58, but /coder-workspaces did not mount, because fstab named the old drive with nofail. Workspaces restarted with empty directories on the 40 GB root filesystem.
  • The new drive was formatted and mounted, its new UUID went into fstab, and workspaces were restarted. Staging was usable again at about 20:15.
  • Home directories of 8 user workspaces on staging were lost. CI jobs running between 19:52 and 20:15 failed with connection refused and were rerun.

Changes made

Change Reference
Nightly encrypted backups of the Coder database and host configuration to S3, with a restore test and an Icinga check of the backup's age #249
Restart running workspaces after a host reboot, so sudo works in them #247
Setup docs fixed for problems found during the rebuild, such as drive names that change and the installimage image name #248
Drush pinned below 15 for Drupal 12 #250, #257
CI retries template pushes that hit the rate limit #252
Other CI fixes made during the recovery #253, #254, #255
Icinga NVMe health check on both servers. It ignores the wear-only warning 0x04 while spare is above threshold and there are no media errors Server configuration
atop and sysstat logging on staging, to explain load alerts Server configuration
Docker 29.10, Sysbox 0.7.1 and Coder v2.39.4 on both servers. Sysbox 0.7.1 does not need "time-namespaces": false, tested on staging 2026-11-17 Server configuration

Follow-ups

  • Practice a database recovery: download the latest backup from S3 on a machine other than the server, using only the secrets in 1Password, and check its contents by hand. Repeat it regularly, and document the steps in the operations guide. Done 2026-11-17; steps in docs: add a backup recovery drill and fix the database restore steps #260.
  • RAID1 for the system drives on coder.ddev.com (Use RAID1 for the system drives on coder.ddev.com #258).
  • Back up workspace home directories, or tell users that workspaces have no backup.
  • Remove "time-namespaces": false from the setup docs and from coder.ddev.com.
  • Plan a replacement for staging's system drive, which is at 134% wear.
  • Ask Hetzner to schedule any future drive swap before doing it.

🤖 Generated with Claude Code

Activity

  1. rfay commented on Nov 17, 2026

    @rfay
    MemberAuthor

    Comment by Claude Code at @rfay's request.

    Practice recovery, 2026-11-17

    Recovered the coder.ddev.com backup on a Mac with Docker, using only the secrets from 1Password. The server wasn't used.

    • duplicity found 2 backup sets: a full backup from 2026-11-16 15:37 and an incremental backup ending 2026-11-17 02:38 (times as duplicity prints them).
    • The restored set contains coder.dump, /root/coder-pg-password, /etc/coder.d/coder.env, the Let's Encrypt certificate (valid until 2027-02-14), systemd units, Docker, Icinga and janitor configuration. /etc/coder-backup.env is left out on purpose, because its secrets come from 1Password.
    • The dump loaded into PostgreSQL 16 with 3 users, 4 templates and 2 workspaces. These match the live server, except for one user who signed up after the backup.

    #260 documents the drill with a script, scripts/coder-backup-verify.sh, and fixes the restore steps for a lost server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions