The situation we encountered was as follows:
- Start teardown process (
deploy --destroy)
- k8s resources removed
- Most cluster resources destroyed but one resource that failed to destroy
- Attempting to run the teardown process again failed because the k8s resources rely on remote state from the
cluster step which had already been destroyed.
The way to recover from this is to comment out the k8s step of the deploy process, but we should have some programmatic way of detecting this.
One potential solution is to allow destruction of the k8s resources to fail since destroying the cluster will destroy those resources anyways. This doesn't solve the whole problem but it does solve this specific case.
The situation we encountered was as follows:
deploy --destroy)clusterstep which had already been destroyed.The way to recover from this is to comment out the
k8sstep of the deploy process, but we should have some programmatic way of detecting this.One potential solution is to allow destruction of the k8s resources to fail since destroying the cluster will destroy those resources anyways. This doesn't solve the whole problem but it does solve this specific case.