Repository navigation
Description
Opened by Claude Code at @rfay's request.
Summary
On 2026-11-16 the system NVMe drive in coder.ddev.com failed. The server was down for about 7 hours 20 minutes, and Coder's database was lost because nothing backed it up. Workspace home directories survived on a separate drive. The same day, staging-coder.ddev.com had three smaller disruptions: Drupal 12 workspaces hung because of a Drush change, CI template pushes hit Coder's upload rate limit, and Hetzner swapped a worn workspace drive without warning.
All times are UTC.
coder.ddev.com: system drive failure
The production server lost its system drive and was rebuilt from the setup docs.
Timeline
| Time | Event |
|---|---|
| 2026-11-16 07:10 | The 1 TB system NVMe drive fails. Icinga checks start failing with execvpe ... Input/output error. Ping still works, but SSH, HTTP and HTTPS do not. |
| 07:10–12:20 | Icinga sends alerts. It is night in the maintainer's time zone, so nobody acts. |
| about 12:23 | Recovery starts. A hardware reset from Hetzner Robot leaves the server unreachable. In the rescue system, the system drive is missing from PCIe. The two 512 GB drives pass SMART. |
| before 13:55 | Hetzner replaces the failed drive. The NVMe device names change, so the workspace drive is no longer nvme1n1. |
| 13:59 | Ubuntu 24.04 is reinstalled on the new drive with installimage, keeping the workspace drive untouched. |
| 14:10–14:49 | Server set up again from the docs: TLS certificate, Coder, GitHub login, templates, workspace janitor, Discord notifications, Icinga agent. |
| about 14:30 | Coder is usable again. |
| 15:02, 15:20 | Docker, Sysbox and Coder upgraded, then a reboot test of the new restart service (#247). |
| 2026-11-17 02:38 | First nightly encrypted backup to S3 succeeds (#249). |
Impact
- coder.ddev.com was unavailable for about 7 hours 20 minutes.
- Coder's PostgreSQL database was lost: user accounts, templates, workspace records and API tokens. Users sign in again with GitHub and recreate their workspaces.
- Workspace home directories survived, because they are on a separate drive. The maintainer's workspaces were reattached. The directories of two other users' workspaces were later deleted.
- Docker's data and the registry mirror were lost and rebuilt.
Cause
The drive failed suddenly. In the two days before, Icinga showed no warning state changes, and nothing monitored drive health. The server had no RAID, so this one drive held the operating system, PostgreSQL and Docker. Nothing backed up the database (#198).
staging-coder.ddev.com
Staging had three separate problems on the same day.
Drupal 12 workspaces hung (from about 02:31)
Drush's 15.x branch replaced the drush command with a stub that called itself forever. Every Drupal 12 workspace stopped after "✓ Drush installed", and the Drupal 12 CI jobs failed. #250 pins drush/drush:<15, merged at 19:03. Drush fixed the stub upstream (drush-ops/drush#6682), but Drush 15 has no site:install, so the pin stays (#257).
CI template pushes rate-limited (19:03–19:05)
Three PRs were rebased while main's run was in progress, and about 30 CI jobs pushed templates in the same minute. Coder allows 12 file uploads per minute per user, so most jobs failed with You've been rate limited for sending more than 12 requests in 1m0s. #252 retries the push after a wait, merged at 20:34.
Workspace drive swapped without warning (19:53–about 20:15)
Both staging drives reported SMART "FAILED" with critical warning 0x04, at 134% wear and about 630 TB written. Hetzner explained that this warning only means the rated wear is exceeded. The available spare was 100%, with 0 media errors. Hetzner offered a replacement, and we asked for the workspace drive only. Hetzner shut the server down at 19:53 and swapped it with no time arranged.
- The server came back at 19:58, but
/coder-workspacesdid not mount, because fstab named the old drive withnofail. Workspaces restarted with empty directories on the 40 GB root filesystem. - The new drive was formatted and mounted, its new UUID went into fstab, and workspaces were restarted. Staging was usable again at about 20:15.
- Home directories of 8 user workspaces on staging were lost. CI jobs running between 19:52 and 20:15 failed with
connection refusedand were rerun.
Changes made
| Change | Reference |
|---|---|
| Nightly encrypted backups of the Coder database and host configuration to S3, with a restore test and an Icinga check of the backup's age | #249 |
Restart running workspaces after a host reboot, so sudo works in them |
#247 |
| Setup docs fixed for problems found during the rebuild, such as drive names that change and the installimage image name | #248 |
| Drush pinned below 15 for Drupal 12 | #250, #257 |
| CI retries template pushes that hit the rate limit | #252 |
| Other CI fixes made during the recovery | #253, #254, #255 |
Icinga NVMe health check on both servers. It ignores the wear-only warning 0x04 while spare is above threshold and there are no media errors |
Server configuration |
atop and sysstat logging on staging, to explain load alerts |
Server configuration |
Docker 29.10, Sysbox 0.7.1 and Coder v2.39.4 on both servers. Sysbox 0.7.1 does not need "time-namespaces": false, tested on staging 2026-11-17 |
Server configuration |
Follow-ups
- Practice a database recovery: download the latest backup from S3 on a machine other than the server, using only the secrets in 1Password, and check its contents by hand. Repeat it regularly, and document the steps in the operations guide. Done 2026-11-17; steps in docs: add a backup recovery drill and fix the database restore steps #260.
- RAID1 for the system drives on coder.ddev.com (Use RAID1 for the system drives on coder.ddev.com #258).
- Back up workspace home directories, or tell users that workspaces have no backup.
- Remove
"time-namespaces": falsefrom the setup docs and from coder.ddev.com. - Plan a replacement for staging's system drive, which is at 134% wear.
- Ask Hetzner to schedule any future drive swap before doing it.
🤖 Generated with Claude Code
Activity
rfay commentedon Nov 17, 2026
Comment by Claude Code at @rfay's request.
Practice recovery, 2026-11-17
Recovered the coder.ddev.com backup on a Mac with Docker, using only the secrets from 1Password. The server wasn't used.
coder.dump,/root/coder-pg-password,/etc/coder.d/coder.env, the Let's Encrypt certificate (valid until 2027-02-14), systemd units, Docker, Icinga and janitor configuration./etc/coder-backup.envis left out on purpose, because its secrets come from 1Password.#260 documents the drill with a script,
scripts/coder-backup-verify.sh, and fixes the restore steps for a lost server.