The Top 5 Most Popular K8s Storage Troubleshooting Issues

Trying to configure persistent storage options using Kubernetes comes with its issues and challenges. Although the official storage docs are quite extensive, they only cover a fraction of the specific configurations you might need in order to successfully install and provision them.

From an operator’s point of view, you might encounter issues choosing a CSI and setting up a PV. From a developer’s point of view, on the other hand, you might be worried about claiming that PV and consuming it as a mounted drive.

This article aims to offer a detailed explanation of the top 5 most popular K8s storage troubleshooting issues as well as ways to resolve them.

Let’s get started.

1. Help! My Persistent Volume Claim Is Not Bound to the Persistent Volume

One of the most common issues when using PVCs is the inability to bind them to the specified PV. For example, you might receive this event when you try to apply a PVC:

PersistentVolumeClaim is not bound: “pv-demo”

To solve this problem, first make sure that you have created actual volumes, formatted them, and assigned a PV. For example, you can create disks with specific storage using the disk create command in GCloud.

$ gcloud compute disks create example-pv --size=10GB

Then, use the kubectl get pvc command to see if you get more information about their status. For example, you might get a message like this:

Waiting for the volume to be created, either by external provisioner or the system administrator.

In this case, the cluster may not support dynamic provisioning (such as a bare-metal cluster) and will therefore be unable to fulfil the request for a persistent volume and attach it to the container.

If the cluster does support dynamic provisioning, then you need to check the available storage and access modes. When you request a Persistent Volume Claim against a PV, you can specify a capacity and an access mode. If you request more storage than you have available in the PV, it won’t fulfill your PVC request.

resources:
    requests:
      storage: 10Gi

$ kubectl get storageclass

NAME                 PROVISIONER            AGE

standard (default)   kubernetes.io/gce-pd   10d

ReadWriteOnce
ReadOnlyMany

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: standard
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi

2. Cannot Apply RBAC Rules to Persistent Volumes

If you enable RBAC rules, then you will need to create specific cluster roles with list/watch permissions for PVs just like any other resource (like namespaces and nodes). For example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <cluster_role_name
rules:
- apiGroups: [""]
  resources:
  - nodes
  - persistentvolumes
  - namespaces
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses
  verbs: ["list", "watch"]

3. How to Get a List of Available CSI Drivers

You can get a list of the registered CSI drivers available in the cluster by issuing the following command:

$ kubectl get csidrivers.storage.k8s.io

NAME                    ATTACHREQUIRED   PODINFOONMOUNT   MODES        AGE

pd.csi.storage.gke.io   true             false            Persistent   11m

“ATTACHREQUIRED=true” means that K8s will skip any attach operations. It should call attach and wait for any attach operations to complete before mounting.

“PODINFOONMOUNT=false” means that the CSI volume driver does not require additional pod information (such as the pod name or UUID) during mount operations.

The modes field maps to the volumeLifecycleModes beta field, which means that it can be either persistent or ephemeral. You can learn more about the purpose of volumeLifecycleModes here.

Note that querying the CSI drivers is different from querying storage class provisioners. You won’t have a good way to query the latter, and you will get a “no volume plugin matched” message if you reference a provisioner that does not exist.

4. Can’t Delete PV or PVC

If you try to delete a PV or PVC and the command is stuck in “terminating” status, then you might check to see if the PV that the PVC is attached to is protected or if there are pods that use the PVC. This is documented in the “Storage Object in Use Protection” section of this doc.

You will see a “kubernetes.io/pv-protection” finalizer field for PVs and a “kubernetes.io/pvc-protection” field for PVCs, as follows:

$ kubectl describe pvc pvc-example -o json | jq -r '.Finalizers'

[kubernetes.io/pvc-protection]

$ kubectl get po -o json --all-namespaces | jq -j '.items[] | "\(.metadata.namespace), \(.metadata.name), \(.spec.volumes[].persistentVolumeClaim.claimName)\n"' | grep -v pvc-example

If that doesn’t work, you might want to update the PV or PVC manifest to remove the protection finalizers:

$ kubectl patch pvc pvc-example -p '{"metadata":{"finalizers":null}}'

5.  How to Share a PVC Among Containers

You cannot use a common PVC among pods, since each PVC is uniquely bound to a persistent volume (in a one-to-one association). The most straightforward way is to have an NFS provisioner in your cluster and share the folder among all pods using the same PVC.

Additionally, some provisioners like PortWorx enable easy installation of shared storage.

You can follow this guide to create an NFS server, expose it as a service, and create a PV that can be shared between pods.

Your PV also needs to support ReadWriteMany access modes in order to share volumes. You can create such disks using Google Filestore, since it’s a fully compliant NFS server:

gcloud filestore instances create nfs-server
    --project=[PROJECT_ID]
    --zone=[ZONE]
    --tier=STANDARD
    --file-share=name="export",capacity=100GB
    --network=name="default",reserved-ip-range="10.0.0.0/29"

nfs:
    path: /export
    server: <IP_ADDRESS>

Reading List to Further Your Understanding of How K8s Storage Works

There are several detailed articles that can help you further your understanding of how K8s storage works. For example, Jay Vyas’s “Kubernetes Storage: Dynamic Volumes and the Container Storage Interface” and Twain Taylor’s “Storage Considerations as You Migrate to Containers” are excellent resources that can advance your knowledge in this area. As we mentioned above, PortWorx abstracts the complexities of managing and provisioning persistent storage in K8s, and we recommend that you try them out. The official CSI documentation site also contains detailed information about the CSI and how to deploy new CSI drivers.