docs: cloud deployment IaC design (OpenTofu + cloud-init + Hetzner)

[evangineer]

Summary

Proposes an Infrastructure-as-Code approach for deploying Holochain hApps using Edge Node on Hetzner Cloud + Cloudflare. Covers the full stack: Cloudflare Pages/Workers (joining service + log-collector), Hetzner VMs (edgenode + harvester), h2hc-linker, Caddy TLS.

What's included

• deploy/DESIGN.md — full design with architecture, DR, open questions, and spike findings (Appendix A)
• deploy/.env.example — secrets template
• deploy/spike/ — working OpenTofu + esbuild spike answering Cloudflare Worker deployment questions

Key design decisions

• Hetzner persistent volumes — hard requirement for lair keystore and conductor database survival; volumes are managed independently of VMs in OpenTofu
• cloud-init over Ansible — cloud-init is sufficient for current scope; Ansible deferred until operational complexity warrants it (rolling updates across many nodes, config drift correction)
• VM count — 2 edgenodes + 1 harvester for staging (3 VMs total); N edgenodes for production
• Harvester VM — runs conductor (Unyt hApp) + log-harvester together; conductor is required for zome-call signing and invoice creation via @holochain/client
• Harvester bootstrap — one-time operator step via a Docker bootstrap container (ghcr.io/holo-host/edgenode-bootstrap); generates agent key, whitelists it in the joining service, installs Unyt hApp; agent key is stable for the lifetime of the persistent volume
• Disaster recovery — daily Hetzner volume snapshots; RTO < 10 min (VM failure), < 30 min (volume restore); RPO < 24 hours

Spike findings (Appendix A)

• log-collector: deploys via OpenTofu. KV and D1 bindings work. Use Cloudflare provider v5.x.
• joining service: stays on wrangler deploy — @holo-host/lair → libsodium WASM can't be bundled with esbuild alone; wrangler handles it transparently
• Cross-directory imports: esbuild resolves them fine

Open questions (unresolved)

• Platform-track: self-registering harvester requires a machine-to-machine admin API on the joining service (Appendix B)
• @holo-host/lair libsodium WASM fix — needed before joining service can move to OpenTofu deployment

Test plan

• Review architecture, data persistence, and DR sections
• Confirm VM count and sizing for staging
• Review harvester bootstrap approach
• Forward libsodium WASM options to @holo-host/lair developer
• Forward platform self-registration requirement to joining service developer

🤖 Generated with Claude Code

[zippy]

Reading over the design doc it seem pretty reasonable, there's one problem, and I just have one questions:

Problem: it looks like the design show the log-harverster running on a node by itself which it can't. It needs a conductor running on the same node to which it connects to actually create the invoices. It does so via the js @holochain/client node-js package. So one of the deployment issues is how that node is authenticated to the network via the joining-services CLI interface, and also how it is signing it's zome-calls. For us this hasn't been decided I don't think, it could happen a few different ways.

Question: is this the right time to be adding ansible into the mix? It seems like it might be an extra layer that isn't actually needed right now because we could just be using a simple cloud-init file for deployment on Hetzner. On the other hand, it may be right to add it from the start. Just want to confirm the thinking.

[evangineer]

it looks like the design show the log-harverster running on a node by itself which it can't. It needs a conductor running on the same node to which it connects to actually create the invoices. It does so via the js @holochain/client node-js package.

FYI the harvester edgenode actually runs conductor + Unyt hApp + log-harvester.

The doc is mistaken and will be corrected.

[evangineer]

Thanks for the review @zippy — both concerns are addressed in the current branch HEAD:

Conductor requirement: The design already reflects this correctly. The harvester VM table shows conductor (Unyt hApp), log-harvester as co-located services, and open question #5 covers the bootstrap process — generating the agent key, whitelisting it in the joining service, and installing the Unyt hApp so the conductor can sign zome calls via lair. The earlier version of the doc was ambiguous; it's been corrected.

Ansible: The design explicitly defers Ansible in favour of cloud-init — there's a 'Why not Ansible (for now)' section covering the reasoning. For the current scope (a small number of VMs each running one container) cloud-init is sufficient and keeps the toolchain simpler. Ansible is noted as a candidate if operational needs grow (rolling updates across many nodes, config drift correction). The PR title and description have been updated to remove the Ansible reference.

Let me know if anything else needs clarification.