test: add integration suites that create real Droplets

.github/workflows/integration.yml

@@ -0,0 +1,151 @@
+---
+name: "Integration Tests"
+
+# These create real Droplets in a real account, so they never run automatically
+# on a pull request: secrets are not available to forks, and every run costs
+# money. Maintainers trigger them by hand, and they run weekly against main so
+# that a DigitalOcean-side change is noticed before a release rather than after.
+"on":
+  workflow_dispatch:
+    inputs:
+      suites:
+        description: "Regexp of suites to run, e.g. 'default|tags'. Empty runs all of them."
+        required: false
+        default: ""
+  schedule:
+    - cron: "0 4 * * 1"
+
+# Droplet and Droplet-name limits are per account. Two runs at once trip over
+# each other, so let an in-flight run finish rather than cancelling it.
+concurrency:
+  group: digitalocean-integration
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    # A fork gets neither the secret nor the bill.
+    if: github.repository == 'test-kitchen/kitchen-digitalocean'
+    env:
+      DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
+      KITCHEN_SSH_KEY: ${{ github.workspace }}/.ssh/integration
+      KITCHEN_RUN_ID: ${{ github.run_id }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Install Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.4"
+          bundler-cache: true
+
+      # Say why rather than failing forty lines later with a 401.
+      - name: Check the access token is available
+        run: |
+          if [ -z "${DIGITALOCEAN_ACCESS_TOKEN}" ]; then
+            echo "::error::DIGITALOCEAN_ACCESS_TOKEN is not set. See integration/README.md."
+            exit 1
+          fi
+
+      # A key pair per run, uploaded for the run and deleted after it. Nothing
+      # long lived is stored, and a leaked key expires with the run.
+      - name: Create and upload an SSH key
+        run: |
+          set -euo pipefail
+          mkdir -p "$(dirname "${KITCHEN_SSH_KEY}")"
+          ssh-keygen -t ed25519 -N "" -C "kitchen-digitalocean-${KITCHEN_RUN_ID}" -f "${KITCHEN_SSH_KEY}"
+
+          key_id="$(curl -sSf -X POST "https://api.digitalocean.com/v2/account/keys" \
+            -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" \
+            -H "Content-Type: application/json" \
+            -d "$(jq -n --arg name "kitchen-integration-${KITCHEN_RUN_ID}" \
+                        --arg key "$(cat "${KITCHEN_SSH_KEY}.pub")" \
+                        '{name: $name, public_key: $key}')" \
+            | jq -r '.ssh_key.id')"
+
+          test -n "${key_id}" && test "${key_id}" != "null"
+          echo "DIGITALOCEAN_SSH_KEY_IDS=${key_id}" >> "${GITHUB_ENV}"
+          echo "Uploaded SSH key ${key_id}"
+
+      # The firewalls suite needs a firewall that still lets SSH through, so it
+      # can tell "attached correctly" apart from "locked itself out".
+      - name: Create a cloud firewall
+        run: |
+          set -euo pipefail
+          firewall_id="$(curl -sSf -X POST "https://api.digitalocean.com/v2/firewalls" \
+            -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" \
+            -H "Content-Type: application/json" \
+            -d "$(jq -n --arg name "kitchen-integration-${KITCHEN_RUN_ID}" '{
+                  name: $name,
+                  inbound_rules: [
+                    {protocol: "tcp", ports: "22",
+                     sources: {addresses: ["0.0.0.0/0", "::/0"]}}
+                  ],
+                  outbound_rules: [
+                    {protocol: "tcp", ports: "all",
+                     destinations: {addresses: ["0.0.0.0/0", "::/0"]}},
+                    {protocol: "udp", ports: "all",
+                     destinations: {addresses: ["0.0.0.0/0", "::/0"]}},
+                    {protocol: "icmp",
+                     destinations: {addresses: ["0.0.0.0/0", "::/0"]}}
+                  ]
+                }')" \
+            | jq -r '.firewall.id')"
+
+          test -n "${firewall_id}" && test "${firewall_id}" != "null"
+          echo "KITCHEN_DO_FIREWALL_ID=${firewall_id}" >> "${GITHUB_ENV}"
+          echo "Created firewall ${firewall_id}"
+
+      # The regexp goes through the environment rather than being interpolated
+      # into the command, so a dispatch input cannot become shell.
+      - name: Run the integration suites
+        working-directory: integration
+        env:
+          SUITES: ${{ github.event.inputs.suites }}
+        run: bundle exec kitchen test ${SUITES} --concurrency 4
+
+      # Destroy runs whatever happened above. A suite that leaks Droplets on
+      # failure turns a red build into a recurring bill.
+      - name: Destroy everything
+        if: always()
+        working-directory: integration
+        run: bundle exec kitchen destroy --concurrency 4
+
+      # Belt and braces. If `kitchen destroy` could not run at all -- a cancelled
+      # job, a crashed runner -- the tag is still the way back to the Droplets.
+      - name: Sweep anything left behind
+        if: always()
+        run: |
+          curl -sS -X DELETE \
+            "https://api.digitalocean.com/v2/droplets?tag_name=run-${KITCHEN_RUN_ID}" \
+            -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" || true
+
+      - name: Delete the firewall and the SSH key
+        if: always()
+        run: |
+          if [ -n "${KITCHEN_DO_FIREWALL_ID:-}" ]; then
+            curl -sS -X DELETE "https://api.digitalocean.com/v2/firewalls/${KITCHEN_DO_FIREWALL_ID}" \
+              -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" || true
+          fi
+          if [ -n "${DIGITALOCEAN_SSH_KEY_IDS:-}" ]; then
+            curl -sS -X DELETE "https://api.digitalocean.com/v2/account/keys/${DIGITALOCEAN_SSH_KEY_IDS}" \
+              -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" || true
+          fi
+
+      - name: Upload logs
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: kitchen-logs
+          path: integration/.kitchen/logs/
+          retention-days: 7
+
+      - name: Warn about anything still running
+        if: failure()
+        run: |
+          echo "::warning::If the cleanup steps did not succeed, look for Droplets tagged run-${KITCHEN_RUN_ID}"

CONTRIBUTING.md

@@ -15,7 +15,8 @@ bundle install
 ```
 
 You do not need a DigitalOcean account or an API token to work on this driver.
-The test suite never talks to the network.
+The unit suite never talks to the network. The
+[integration suites](integration/README.md) do, and are the exception.
 
 ## Rake tasks
 
@@ -29,6 +30,8 @@ The test suite never talks to the network.
 | `bundle exec rake yard` | Build the API documentation into `doc/`. |
 | `bundle exec rake yard:stats` | Report which objects are missing documentation. |
 | `bundle exec rake yard:server` | Browse the documentation at `http://localhost:8808`. |
+| `bundle exec rake integration:test` | Run the integration suites against a real DigitalOcean account. Costs money; not part of `rake`. |
+| `bundle exec rake integration:destroy` | Destroy anything the integration suites left behind. |
 
 Run a single file or example while you iterate:
 
@@ -133,6 +136,24 @@ Coverage is opt-in so the default run stays dependency light. The suite holds
 both paths in a single process. Coverage is reported, not enforced in CI — treat
 a drop as a prompt to look, not as a target to game.
 
+## Integration tests
+
+Stubbing at the wire proves the driver sends the right request, not that
+DigitalOcean accepts it. The suites in [`integration/`](integration/README.md)
+close that gap: each one creates a real Droplet and asserts, on the Droplet,
+that the driver configured it as asked.
+
+```bash
+export DIGITALOCEAN_ACCESS_TOKEN=dop_v1_...
+export DIGITALOCEAN_SSH_KEY_IDS=12345678
+bundle exec rake integration:list
+bundle exec rake integration:test
+bundle exec rake integration:destroy   # after a failed run
+```
+
+They are not part of `rake default` — they cost money — and never run on a pull
+request. Maintainers run them on demand, and weekly against `main`.
+
 ## Documentation
 
 Every method, constant and module carries YARD tags, and `yard stats` reports

Rakefile

@@ -24,6 +24,25 @@ rescue LoadError
   puts "cookstyle/chefstyle is not available. (sudo) gem install cookstyle to do style checking."
 end
 
+namespace :integration do
+  # Deliberately not part of any default task, and not run on a pull request:
+  # these create real Droplets in a real account and cost real money.
+  desc "Run the integration suites against DigitalOcean (creates real Droplets)"
+  task :test do
+    Dir.chdir("integration") { sh "bundle exec kitchen test --concurrency 4" }
+  end
+
+  desc "Destroy anything the integration suites left behind"
+  task :destroy do
+    Dir.chdir("integration") { sh "bundle exec kitchen destroy --concurrency 4" }
+  end
+
+  desc "List the integration suites"
+  task :list do
+    Dir.chdir("integration") { sh "bundle exec kitchen list" }
+  end
+end
+
 # Documentation tasks are intentionally kept out of the `default` task and out
 # of CI: a missing YARD tag should never turn a pull request red.
 begin

integration/README.md

@@ -0,0 +1,119 @@
+# Integration suites
+
+The unit suite stubs HTTP at the wire, so it can prove the driver *sends* the
+right request but never that DigitalOcean accepts it. These suites close that
+gap: each one creates a real Droplet and asserts, on the Droplet, that the
+driver configured it the way the suite asked for.
+
+They are not part of `rake default`. They create real Droplets and cost real
+money.
+
+## What each suite covers
+
+| Suite | What it proves |
+| --- | --- |
+| `default` | Create, converge over SSH and destroy, and that `ubuntu-24` resolves to an image DigitalOcean still offers. |
+| `image-slug` | A platform name that is *not* in `PLATFORM_SLUG_MAP` reaches the API as a slug untouched. |
+| `server-name` | An explicit `server_name` survives to the Droplet unchanged. |
+| `tags` | `tags` written as a YAML list reach the Droplet. |
+| `tags-string` | `tags` written as a delimited string are split into separate tags, not sent as one tag with a comma in it. |
+| `user-data` | `user_data` reaches cloud-init verbatim and runs. |
+| `ipv6` | `ipv6: true` gets the Droplet a routable v6 address. |
+| `monitoring` | `monitoring: true` installs and starts the DigitalOcean metrics agent. |
+| `size-region` | A non-default `size` and `region` are accepted together and are the pair that was asked for. |
+| `firewalls` | `attach_firewalls` places the Droplet behind a cloud firewall without locking the transport out. |
+
+The assertions read the [DigitalOcean metadata
+service](https://docs.digitalocean.com/reference/api/metadata-api/) from the
+Droplet. That is the closest thing to reading back the request the driver sent,
+and it needs no credentials on the Droplet.
+
+## Running them
+
+You need an account, a personal access token with write scope, and an SSH key
+uploaded to DigitalOcean.
+
+```bash
+bundle install
+export DIGITALOCEAN_ACCESS_TOKEN=dop_v1_...
+export DIGITALOCEAN_SSH_KEY_IDS=12345678
+export KITCHEN_SSH_KEY=~/.ssh/id_kitchen_digitalocean   # the matching private key
+
+cd integration
+bundle exec kitchen list
+bundle exec kitchen test default-ubuntu-24
+```
+
+Or from the repository root:
+
+```bash
+bundle exec rake integration:test      # everything
+bundle exec rake integration:destroy   # clean up after a failed run
+```
+
+`kitchen test` destroys on success. It leaves the Droplet up on failure so you
+can log in and look, so **run `kitchen destroy` when you are done** — or
+`rake integration:destroy`, which does it for every suite.
+
+### Settings
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `DIGITALOCEAN_ACCESS_TOKEN` | *none* | Required. Token with write scope. |
+| `DIGITALOCEAN_SSH_KEY_IDS` | *none* | Required. ID or fingerprint of an uploaded key. |
+| `KITCHEN_SSH_KEY` | `~/.ssh/id_kitchen_digitalocean` | Private key the transport logs in with. |
+| `KITCHEN_DO_REGION` | `nyc3` | Region most suites deploy into. |
+| `KITCHEN_DO_ALT_REGION` | `sfo3` | Second region, used by `size-region`. |
+| `KITCHEN_DO_SIZE` | `s-1vcpu-1gb` | Droplet size. |
+| `KITCHEN_DO_FIREWALL_ID` | *none* | Cloud firewall for the `firewalls` suite. Without it that suite reports itself skipped. |
+| `KITCHEN_RUN_ID` | `local` | Tagged onto every Droplet as `run-<id>`, so a leaked one can be traced back. |
+
+## Cost and cleanup
+
+Ten Droplets, all `s-1vcpu-1gb` bar one, alive for a few minutes each. A full
+run is cents rather than dollars — but a Droplet that outlives the run is not,
+so every Droplet is tagged `kitchen-digitalocean-integration` and
+`run-<KITCHEN_RUN_ID>`. If a run is interrupted:
+
+```bash
+doctl compute droplet list --tag-name kitchen-digitalocean-integration
+doctl compute droplet delete --tag-name kitchen-digitalocean-integration
+```
+
+## In CI
+
+`.github/workflows/integration.yml` runs these weekly against `main`, and on
+demand through **Actions → Integration Tests → Run workflow**. It is never
+triggered by a pull request: secrets are not available to forks, and every run
+costs money.
+
+It needs one repository secret, `DIGITALOCEAN_ACCESS_TOKEN`. Everything else is
+created for the run and deleted afterwards — an ed25519 key pair uploaded to the
+account, and a cloud firewall for the `firewalls` suite — so no long-lived SSH
+key is stored anywhere.
+
+Three details matter more than they look:
+
+* **`Destroy everything` runs with `if: always()`.** A suite that leaks Droplets
+  on failure turns a red build into a recurring bill.
+* **A tag sweep runs after it, also with `if: always()`.** If `kitchen destroy`
+  could not run at all — a cancelled job, a crashed runner — the tag is the way
+  back to the Droplets.
+* **`concurrency: digitalocean-integration` with `cancel-in-progress: false`.**
+  Droplet limits are per account; two overlapping runs exhaust them and both
+  fail.
+
+## Adding a suite
+
+Add it to `kitchen.yml` with a script in `scripts/`. Assertions live in the
+**provisioner**, not a verifier: the script is transferred over the driver's own
+transport and executed on the Droplet, so reaching the machine at all is part of
+every assertion, and a non-zero exit fails the suite.
+
+The shell provisioner uploads only the one file it is pointed at, so each script
+repeats the same short preamble rather than sourcing a helper. Values that vary
+per run arrive as `arguments:`, since the provisioner has no way to pass
+environment variables.
+
+Keep each suite pointed at one behaviour — when it fails, its name should say
+what broke.