helm uninstall gets stuck because node-cleanup hook job uses a ServiceAccount that is deleted too early

[Bianco95]

Description

Uninstall of the interlink chart can get stuck in uninstalling state because the cleanup Job created during deletion tries to use a ServiceAccount that has already been deleted.

As a consequence:

• helm uninstall never completes
• the release remains in uninstalling
• a subsequent helm install with the same release name fails with:

Error: INSTALLATION FAILED: cannot re-use a name that is still in use

Observed behavior

After uninstall, the release is still present in Helm history as and at the same time, the cleanup Job and related pods are still present:

job.batch/cleanup   Running   0/1   18h
pod/virtual-node-wg-node-7b75f99457-fj7f9       0/2   Unknown   1   18h
pod/virtual-node-wg-node-cleanup-s4zrh          1/1   Running   0   18h

And the pod events show the root problem:

Warning  FailedMount  pod/virtual-node-wg-node-7b75f99457-fj7f9
MountVolume.SetUp failed for volume "kube-api-access-zz64r" :
failed to fetch token: serviceaccounts "virtual-node-wg" not found

Warning  FailedMount  pod/virtual-node-wg-node-cleanup-s4zrh
MountVolume.SetUp failed for volume "kube-api-access-dcrrh" :
failed to fetch token: serviceaccounts "virtual-node-wg" not found

Expected behavior

helm uninstall should complete successfully and fully remove the release, so that the same release name can be installed again without manual cleanup.

Actual behavior

The uninstall hook / cleanup Job remains stuck because it references a ServiceAccount that is no longer available. Helm keeps the release in uninstalling, so the release name remains reserved.

Workaround

Manual cleanup is required to unblock the release, for example:

kubectl delete job -n <namespace> <job>-cleanup
helm uninstall -n <namespace> <chart_name> --no-hooks

[Bianco95]

The issue is still reproducible on my side.

In addition to the original ServiceAccount deletion problem, the cleanup pod now shows RBAC errors. The cleanup ServiceAccount exists, but it does not have enough cluster-scoped permissions on nodes.

Logs from the cleanup pod:

E0611 07:48:11.006147       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"
E0611 07:48:12.053939       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"
E0611 07:48:14.494603       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"
E0611 07:48:19.788953       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"
E0611 07:48:29.076423       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"
E0611 07:48:44.750157       7 reflector.go:227] "Failed to watch" err="failed to list *unstructured.Unstructured: nodes "vk-remote-dispatcher" is forbidden: User "system:serviceaccount:interlink:vk-remote-dispatcher-cleanup" cannot list resource "nodes" in API group "" at the cluster scope" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" type="*unstructured.Unstructured"

The current pre-delete-hook.yaml creates a ClusterRole for the cleanup hook with:

resources: ["nodes"]
verbs: ["get", "delete"]

but the cleanup process is trying to list / watch nodes, so the hook keeps failing with:

cannot list resource "nodes" in API group "" at the cluster scope

A possible fix would be to extend the cleanup ClusterRole permissions to include at least list and probably watch as well:

rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "delete"]

So the uninstall issue does not seem fully resolved yet: the cleanup hook still cannot complete because of missing cluster-scoped RBAC permissions for the cleanup ServiceAccount.