Skip to content

[Draft] Fix cleanup of GRPCRoute resources in namespaces without psm-csm suffix#264

Draft
sreenithi wants to merge 1 commit into
grpc:mainfrom
sreenithi:fix-cleanup-route-resources
Draft

[Draft] Fix cleanup of GRPCRoute resources in namespaces without psm-csm suffix#264
sreenithi wants to merge 1 commit into
grpc:mainfrom
sreenithi:fix-cleanup-route-resources

Conversation

@sreenithi

Copy link
Copy Markdown
Contributor

[DO-NOT-MERGE] Yet to be tested.

Cleanup job was recently failing for some namespaces like:

I0712 14:44:50.416528 140554289934336 k8s_base_runner.py:943] Deleting namespace psm-interop-server-20260618-1647-r9cyq
E0712 14:49:51.754359 140554289934336 cleanup.py:535] Timeout waiting for namespace psm-interop-server-20260618-1647-r9cyq deletion. Failed resource status:

Resource name: psm-interop-server-20260618-1647-r9cyq
Resource kind: Namespace
Created: 2026-06-18 16:48:05+00:00; 23 days ago
Deletion requested: 2026-06-21 15:17:41+00:00; 20 days ago

---
conditions:
- last_transition_time: 2026-07-03 21:17:18+00:00
  message: All resources successfully discovered
  reason: ResourcesDiscovered
  status: 'False'
  type: NamespaceDeletionDiscoveryFailure
- last_transition_time: 2026-06-21 15:17:46+00:00
  message: All legacy kube types successfully parsed
  reason: ParsedGroupVersions
  status: 'False'
  type: NamespaceDeletionGroupVersionParsingFailure
- last_transition_time: 2026-06-21 15:17:46+00:00
  message: All content successfully deleted, may be waiting on finalization
  reason: ContentDeleted
  status: 'False'
  type: NamespaceDeletionContentFailure
- last_transition_time: 2026-06-21 15:17:56+00:00
  message: 'Some resources are remaining: servicenetworkendpointgroups.networking.gke.io
    has 1 resource instances'
  reason: SomeResourcesRemain
  status: 'True'
  type: NamespaceContentRemaining
- last_transition_time: 2026-06-21 15:17:56+00:00
  message: 'Some content in the namespace has finalizers remaining: [networking.gke.io/neg-finalizer](https://www.google.com/url?q=http://networking.gke.io/neg-finalizer&sa=D)
    in 1 resource instances'
  reason: SomeFinalizersRemain
  status: 'True'
  type: NamespaceFinalizersRemaining
phase: Terminating
...


E0712 14:49:51.754550 140554289934336 cleanup.py:644] ----- Skipped [cleanup timed out]: psm-interop-server-20260618-1647-r9cyq

When inspecting these name spaces manually, found that these have a pending backend service and a gRPC route attached to it

$ kubectl get servicenetworkendpointgroups.networking.gke.io -n psm-interop-server-20260618-1647-r9cyq
NAME                                                     AGE
psm-interop-server-20260618-1647-r9cyq-psm-grpc-server   24d

$ gcloud compute backend-services delete psm-interop-backend-service-20260618-1647-r9cyq   --region=us-west1   --project=grpc-testing
The following region backend services will be deleted:
 - [psm-interop-backend-service-20260618-1647-r9cyq] in [us-west1]

Do you want to continue (Y/n)?  y

ERROR: (gcloud.compute.backend-services.delete) Some requests did not succeed:
 - The resource cannot be deleted because it's being used by //networkservices.googleapis.com/projects/830293263384/locations/us-west1/grpcRoutes/psm-interop-grpc-route-20260618-1647-r9cyq

We ideally create a gRPC route only in gamma tests that should have a psm-csm suffix in the namespace. So Gamma servers cleanups that are triggered for psm-csm prefixed namespaces handle the grpc-route deletion correctly.
But looks like these non-Gamma namespaces also have a gRPC route attached with them at times and don't clean them correctly.
So, this PR just checks if such a route resource exists and delegates the cleanup to the Gamma servers cleanups instead.

Note:
However, the above problematic namespaces have been manually cleaned up already, so there is currently no way to test this change. Will open this up for review once tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant