Namespace deletion is in stuck state + kubernetes ; kubernetes objects deletion is in stuck state how to delete it forcefully? How to remove finalizers manually to unblock namespace deletion?

Quick intro

As part of some cleanup on k8s cluster, I faced couple of issues related to deletion of few objects, mainly the NameSpace deletion. I was Googling for a solution and it took some time to get the proper solutions. So I planned to document it for the future reference as well as for the Engineers who all are in trouble because of this issue.

The issues that we are discussing today are:

• Why the object (I faced with k8s NameSpace & the deployment) went to terminating state infinitely?
• Why the deployment/s were not deleted even after deleting the k8s NameSpace?
• What are Finalizers and Owner reference in k8s?
• Solution for deleting the object which are in the terminating state.

So basically this happen because of the presence of Finalizers in your k8s object manifest. Finalizers are keys on resources that signal pre-delete operations. They control the garbage collection on resources, and are designed to alert controllers what cleanup operations to perform prior to removing a resource.

To demonstrate this I am creating a new k8s NameSpace and a sample deployment on it. In both NameSpace and the Deployment we add finalizers.

Test NameSpace: test-finalizers

Test deployment: nginx

Create the NameSpace and add the Finalizer

kubectl create ns test-finalizers

kubectl edit ns test-finalizers

Add following finalizer entry under metadata

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2022-01-07T18:20:09Z"
  labels:
    kubernetes.io/metadata.name: test-finalizers
  name: test-finalizers
  finalizers:
  - kubernetes

Create a test deployment and add Finalizer

kubectl create deployment nginx --image=nginx -n test-finalizers

kubectl edit deployment nginx -n test-finalizers

Add following finalizer entry under metadata

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2022-01-07T18:22:07Z"
  finalizers:
  - kubernetes
  generation: 1
  labels:
    app: nginx
  name: nginx

Now set for the demonstration. Here actually I had faced two issues and explaining both here. Both the case the cause is because of the Finalizers and fix is also same – that is removing the Finalizer from the object manifest.

Issue 1: The NameSpace was stuck in Terminating state

To illustrate this we’re going to delete the NameSpace that we created.

kubectl delete ns test-finalizers
namespace "test-finalizers" deleted

It will show the deleted message but won’t exit, because it is waiting at the Finalizers.

[arun@Zeta] [~] $ kubectl get ns test-finalizers
NAME              STATUS        AGE
test-finalizers   Terminating   9m36s

As it’s in terminating state we can not do any edit operation on this objects manifest, it is read only.

To fix this, somehow we need to remove those entries from the manifest. There are different ways, explaining those:

Method 1: Dump the current conf, modify and apply

kubectl get ns test-finalizers -o json > test-finalizers.json

Edit the json file and remove the Finalizer entry

cat test-finalizers.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2022-01-07T18:20:09Z",
        "deletionTimestamp": "2022-01-07T18:28:11Z",
        "finalizers": [
            "kubernetes"
        ],
        "labels": {
            "kubernetes.io/metadata.name": "test-finalizers"
        },
        "name": "test-finalizers",
        "resourceVersion": "725110",
        "uid": "b3ac4d03-2290-4096-b0d3-d0c7442b8e7b"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {

After modifying the file (remove finalizer array) execute the following:

kubectl replace --raw "/api/v1/namespaces/test-finalizers/finalize" -f ./test-finalizers.json

After running the above command, the namespace should now be absent from your k8s cluster.

Method 2: By using kubectl patch

You can simply use this command to remove the finalizer from the manifest.

kubectl patch ns/test-finalizers --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

Source -> https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/

Issue 2: Orphan deployments in deleted NameSpace

The deployments were not deleted even after the NameSpace deleted. But in normal case, deleting the NameSpace will do the other objects cleanup and if that fail because of some Finalizers, NameSpace will stay in Terminating state.

As that deployment was also in the terminating state (read only), we can not edit the manifest. Use the above method 2 (Kubectl patch)

Example

[arun@ddd] [~] $ kubectl get ns test-finalizers
Error from server (NotFound): namespaces "test-finalizers" not found
[arun@ddd] [~] $
[arun@ddd] [~] $ kubectl get all -n test-finalizers
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   0/1     0            0           59m

In this case we can not delete the deployment using kubectl delete. Use the following command:

kubectl patch deployment.apps/nginx -n test-finalizers --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

If you have more objects where we need to remove the finalizer, do it in a loop.

for i in `cat file.txt`; do kubectl patch $i -n test-finalizers --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'; done

That’s it!

What are Finalizers and Owner reference in k8s?

This is well explained in this documentation, please refer this https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/

Read more similar…

Kubernetes CNI, short explanations

Monitoring Kubernetes cluster using Prometheus and Grafana