Finalizers preventing namespace termination

[rglyons]

I'm trying to delete a namespace in its entirety, however the arangodb resource finalizers are preventing that.

Say I have a namespace called example. Now say there is an ArangoDeployment running in this namespace. When I run kubectl delete ns example, The ArangoDeployment is triggered to delete. However, the finalizers on the ArangoDeployment are never removed, since the namespace deletion has already terminated the arangodb operator, because the operator has no finalizers. This means that the ArangoDeployment is never deleted. Also, the arangodb PVCs are never deleted, since they have finalizers that wait for the ArangoDeployment to tear down before they fully terminate. All in all, the namespace example is stuck terminating forever.

How can I safely tear down a namespace where an ArangoDeployment and arangodb operator are running? The only option I see is to delete the ArangoDeployment first, then wait for it to delete (along with its PVCs), then finally execute kubectl delete ns example. The problem I have with this workflow is that it requires scripting that would disrupt many of my existing workflows.

One idea that I had was to add finalizers to the arangodb operator that prevent it from fully terminating until all arangodb resources have fully terminated. This, however, would take away the ability to deploy arangodb on k8s using the operator, then delete the operator, which is a use case some might have, assuming they never want to update/scale arangodb on their cluster. Whether or not you find this use case valid as something you want to support determines the validity of this potential solution. I thought of this after coming across this comment.

Another potential solution is to add a config option to the ArangoDeployment object definition that excludes finalizers from ArangoDeployments and their PVCs, in case order of teardown doesn't matter, which is the case when deleting a namespace. This way, some flag like includeFinalizers: false can be passed into the ArangoDeployment yaml definition.

If not the above proposed solutions, what other solutions are there?

[Arabus]

Ran into the same issue twice now - only solution I found was to kill the whole cluster. I'd be grateful for a workaround when one has already deleted the operator. Is it possible to manually remove the finalizers?

[maierlars]

Yes you can just remove the finalizers.

[Arabus]

Apparently my question was a little misunderstandable so here is the answer I expected:

kubectl get ArangoDeployment <ArangoDeployment-instance-name> -o=json | \
jq '.metadata.finalizers = null' | kubectl apply -f -

This works the same with all other objects that have finalizers in them relying on the operator.

For a more permanent solution I recommend deploying the operator to a different namespace (You might need to adjust the rbac rules).

[maxzzze]

Reviving an old thread here but its the only one I can find. Ran into a similar issue. Tried draining a node and its stuck because of the arango prmr pods stuck terminating.

I tried the solution above to remove finalizers and also created a new operator on a new node but the pods are still stuck.

[SimonMisencik]

we also encountered a similar problem. Has anyone found a fix or workaround?

[Arabus]

See above, you can just patch out the finalizers. if the pods still don't die you might want to check for other issues. The finalizers are the only thing relating to the operator preventing the pod deleting.

Just guessing but the processes inside the containers also need to die to remove the pods, and there musn't be a pre-stop-hook intefering there. See also https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

[SimonMisencik]

Yes, the patch works but I want to make installation/uninstallation of helm chart which use arangodb operator as dependency "user-friendly" just with helm install and helm uninstall commands without the additional steps as the patch command which can be complicated for new users.