feat(hcp): manage HCP workspaces, project, VCS connection via the tfe provider #8

Context

The bootstrap arc finished in #1–#7. We now have two HCP Terraform workspaces (cloudflare, github-org) both VCS-linked, applying, and serving real infrastructure. Every workspace setting — VCS provider, path filter, speculative-plan toggles, fork-PR setting, exec mode — was clicked into the HCP UI by hand and is documented in docs/setup.md.

This issue closes that gap: bring the HCP layer itself under Terraform management via the hashicorp/tfe provider, so future workspace adds, setting changes, and team membership are PRs against this repo instead of UI clicks.

Goal

Add a new leaf terraform/hcp/ that manages:

• The perishdev org's infra project (tfe_project).
• All HCP workspaces under that project (tfe_workspace): cloudflare, github-org, and the new meta-workspace hcp-meta (self-managing — see below).
• The GitHub VCS connection (tfe_oauth_client — import the existing one).
• Non-sensitive Terraform variables on each workspace (e.g. github_owner on github-org).
• Per-workspace settings beyond what tfe_workspace exposes (tfe_workspace_settings), if needed.

After this lands, the existing two workspaces are unchanged in behavior but newly visible as code; adding a future workspace is a PR.

Design decisions (taken from the session conversation; flip if you disagree)

1. HCP API token type: Team API token

Create a Team named infra-meta-bot in HCP, give it Admin permissions on the infra project, generate a Team API token. Load that token into the hcp-meta workspace as an env var TFE_TOKEN, sensitive=true. The tfe provider picks it up automatically.

Why Team over User: doesn't break when a maintainer leaves; same setup effort.

2. The meta-workspace is self-managing

terraform/hcp/ includes a tfe_workspace block for hcp-meta itself, alongside cloudflare and github-org. Import on first apply.

Risk: a misconfiguration of hcp-meta's own block can lock the workspace out (it can't apply a change that breaks its own ability to apply). Mitigation: keep the meta-workspace's own block minimal and well-reviewed; the bootstrap docs explain how to manually reset settings in HCP UI if a bad apply happens.

3. Sensitive variable values are NOT managed by Terraform

tfe_variable resources are declared for non-sensitive vars only (e.g. github_owner = "perishdev"). Sensitive variable VALUES (Cloudflare token, GitHub App PEM, etc.) stay manually managed in the HCP UI. The trade-off:

• ✅ No secrets flow through Terraform state or code.
• ✅ No drift games on sensitive vars.
• ❌ Variable existence + sensitivity flag are NOT enforced by Terraform. Document in docs/setup.md which sensitive vars each workspace expects.

Alternative considered: declare with placeholder value + lifecycle { ignore_changes = [value] }. Rejected because the variable is briefly created with garbage, then drift is silently ignored forever — sneaky failure modes.

Bootstrap prerequisites (manual, do these BEFORE opening a PR)

1. Create the hcp-meta workspace in HCP UI:
   • Project: infra.
   • VCS provider: GitHub (perishdev/infra).
   • Working directory: terraform/hcp.
   • Trigger patterns: terraform/hcp/**.
   • Execution mode: Remote.
   • Auto-apply: off.
   • Automatic speculative plans: on.
   • Speculative plans on fork PRs: off.
2. Create Team API token:
   • HCP Settings → Teams → Create Team infra-meta-bot, project access: Admin on infra.
   • Generate Team API Token (org settings → team → token).
3. Load token into hcp-meta workspace:
   • Variables → add TFE_TOKEN, category Environment, sensitive=true.

Code shape

terraform/hcp/
  versions.tf            tfe provider ~> 0.60+, cloud block (org=perishdev, workspace=hcp-meta)
  providers.tf           provider "tfe" { hostname = "app.terraform.io" }  # token from env
  main.tf                tfe_project "infra" (import); locals for org name
  workspaces.tf          tfe_workspace × 3 + tfe_workspace_settings (import all)
  vcs.tf                 tfe_oauth_client (import); tfe_github_app_installation if applicable
  variables.tf           (none — no variable inputs needed)

Resources to import (do via import blocks, Terraform 1.5+)

• tfe_project.infra — find ID via curl /api/v2/organizations/perishdev/projects.
• tfe_workspace.cloudflare — ID ws-WWKeFPiCAjV4STNX.
• tfe_workspace.github_org — ID ws-iZFBsNEsUfJPRJ1Q.
• tfe_workspace.hcp_meta — ID from step 1 of the prereqs.
• tfe_oauth_client.github — find ID via curl /api/v2/organizations/perishdev/oauth-clients.
• tfe_variable.github_owner — find ID via curl /api/v2/workspaces/ws-iZFBsNEsUfJPRJ1Q/vars.

Hint: a terraform login deposits the HCP user API token at ~/.terraform.d/credentials.tfrc.json. See docs/state.md.

Branch protection impact

When the hcp-meta workspace starts running plans on PRs that touch terraform/hcp/**, HCP will post a new status check distinct from the aggregated rollup. Watch the PR's checks list — if a new Terraform Cloud/... check name appears, decide whether to add it to terraform/github/branch_protection.tf. The current aggregate roll-up name should still cover us, but worth verifying on the bootstrap PR.

Out of scope (defer to follow-up issues)

• Managing teams/users beyond the bootstrap Team API token's team. (Solo today.)
• Sentinel/OPA policy sets — Free tier doesn't include them anyway.
• Run triggers between workspaces.
• Notification configurations (Slack/email).
• Migrating sensitive variable values into Terraform-managed form. (See decision feat(cloudflare): import email DNS records for perish.dev #3 above.)

Acceptance criteria

• Manual prereqs done (workspace created, team + token, env var loaded).
• PR opened with terraform/hcp/ leaf.
• HCP speculative plan on the PR shows: imports only, zero adds/changes/destroys (or only the intentional updates documented in the PR description).
• HCP aggregated check rolls up to SUCCESS on the PR (branch protection passes).
• docs/setup.md updated to remove the manual workspace-creation steps that are now automated; add the hcp-meta bootstrap-once steps (Team API token, env var).
• docs/state.md updated to list hcp-meta in the workspaces table.
• terraform/README.md updated to show the new leaf.
• After merge, applying via HCP brings the existing two workspaces under management with no behavior change.
• Smoke test: open a no-op PR touching terraform/hcp/main.tf (e.g. a comment-only change) and confirm hcp-meta's own speculative plan runs cleanly.

Notes for the next session

• This repo uses always-PR + Conventional Commits/Branches/PR titles per ~/.claude/CLAUDE.md.
• HCP plan summaries are readable via the API trick documented in docs/state.md. Use that to verify imports before merging — much cleaner than reading the UI.
• The brittle HCP check name (Terraform Cloud/perishdev/repo-id-CffUfWW6H1x6Bauq) embedded in terraform/github/branch_protection.tf is unrelated to this work but worth knowing about.
• All terraform/*/ leaves run with the cloud {} block (not backend "remote"). Same for the new terraform/hcp/.