README.md

How to Set Up a Centralized NFS Server for Kubernetes Storage

This guide will walk you through setting up an NFS server to provide centralized storage for your Kubernetes cluster. We'll use a Docker container as the NFS server and configure it to work seamlessly with Kubernetes. This setup is very useful if you have an on-premise Kubernetes cluster and you're looking for a centralized storage solution.

Overview

1. What is NFS?

   • NFS (Network File System) allows you to share files between machines over a network. It's commonly used in Kubernetes to provide persistent storage that can be accessed by multiple pods.

2. Why Use This Setup?

   • This setup is lightweight, flexible, and easy to manage.
   • It works well for small to medium-sized Kubernetes clusters.

3. Requirements

   • A Docker host to run the NFS server container.
   • A Kubernetes cluster where you want to use the NFS storage.
   • Basic knowledge of Docker and Kubernetes.

---

Step 1: Prepare the NFS Server

Recommended Setup

• Create a separate partition or directory on your Docker host for the shared folder. For example:

  mkdir -p /shared
  chmod 777 /shared
  chown nobody:nogroup /shared

  Tip: Using a separate partition ensures better performance and isolation for your shared storage.

Start the NFS Server Container

Before starting the service, you will have to enable some kernel features in order for the server run properly.

modprobe nfs
modprobe nfsd

We'll use the erichough/nfs-server image to set up the NFS server. Below is the docker-compose.yml configuration file:

services:
  nfs-server:
    image: erichough/nfs-server
    container_name: nfs-server
    hostname: nfs-server
    cap_add:
      - SYS_ADMIN
    restart: always
    ports:
      # RPC portmapper (required for NFS)
      - "111:111"       # TCP
      - "111:111/udp"   # UDP

      # NFS service
      - "2049:2049"     # TCP
      - "2049:2049/udp" # UDP

      # rpc.statd (status monitor)
      - "32765:32765"   # TCP
      - "32765:32765/udp" # UDP

      # rpc.mountd (mount daemon)
      - "32767:32767"   # TCP
      - "32767:32767/udp" # UDP

    volumes:
      - /shared:/nfsshare   # Bind mount for the NFS share

    environment:
      - NFS_LOG_LEVEL=DEBUG
      - NFS_EXPORT_0=/nfsshare *(rw,sync,no_subtree_check,no_root_squash,insecure)

    healthcheck:
      test: ["CMD", "rpcinfo", "-p", "localhost"]
      interval: 30s
      timeout: 10s
      retries: 3

    deploy:
      resources:
        limits:
          cpus: "1"
          memory: "512M"

Explanation of Key Fields

• Ports: These are required for NFS to function. They include:
  • Port 111: Used by the RPC portmapper.
  • Port 2049: The main NFS service port.
  • Ports 32765 and 32767: Used by auxiliary services like rpc.statd and rpc.mountd.
• Volumes: Maps /shared on the Docker host to /nfsshare inside the container. This is the directory shared via NFS.
• Environment Variables:
  • NFS_EXPORT_0: Defines the NFS export rules. Replace * with specific client IPs if needed.
  • NFS_LOG_LEVEL: Sets the logging level to DEBUG for troubleshooting.

---

Step 2: Configure Kubernetes PersistentVolume (PV)

Now that the NFS server is running, we need to tell Kubernetes about it by creating a PersistentVolume (PV).

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-server
  labels:
    # Optional labels
    type: nfs
    location: city
    app: nfs-server
spec:
  capacity:
    storage: <size-you-want>
  accessModes:
    - ReadWriteOnce
  nfs:
    server: <your-server-ip>  # Replace with your NFS server IP
    path: /nfsshare           # Path exported by the NFS server

Notes

• Replace <size-you-want> with the actual size of the partition or size you want for your NFS server.
• Replace <your-server-ip> with the actual IP address of your NFS server.
• The path must match the directory you mounted in the NFS server container (/nfsshare).

---

Step 3: Create a PersistentVolumeClaim (PVC)

A PersistentVolumeClaim (PVC) is used to request storage from the PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  storageClassName: ""  # Leave empty if no default StorageClass exists
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Important Notes

• If your Kubernetes cluster has a default StorageClass, you must explicitly set storageClassName: "" in the PVC to avoid conflicts.
• The storage value (1Gi) should not exceed the capacity defined in the PV.

---

Step 4: Attach the PVC to a Pod

Finally, attach the PVC to a pod to use the NFS storage.

apiVersion: v1
kind: Pod
metadata:
  name: data-consumer
spec:
  containers:
    - name: busybox
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - mountPath: "/mnt/test"
          name: test-folder
  volumes:
    - name: test-folder
      persistentVolumeClaim:
        claimName: nfs-pvc

What Happens Here?

• The busybox container mounts the PVC (nfs-pvc) at /mnt/test.
• Any data written to /mnt/test will be stored on the NFS server.

---

Step 5: Verify the Setup

Run the following command to check the status of your PV and PVC:

kubectl get pv,pvc

You should see output similar to this:

NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/nfs-server   100Gi      RWO            Retain           Bound    default/nfs-pvc                  <unset>                          113m

NAME                            STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/nfs-pvc   Bound    nfs-server   100Gi      RWO                           <unset>                 113m

Checkpoints

• Both the PV and PVC should have the status Bound.
• If they are not bound, double-check the NFS server IP, paths, kernel modules, and permissions.

---

Tips for Success

1. Security:

   • Restrict NFS access to specific client IPs by modifying the NFS_EXPORT_0 environment variable.
   • Avoid using no_root_squash in production environments unless absolutely necessary.

2. Performance:

   • Use a dedicated partition or disk for /shared to improve I/O performance.
   • Monitor resource usage (CPU and memory) of the NFS server container.
   • High speed storage is recommended to the get better performance, but it is not mandatory.
   • If you use partition you could use RAID to improve speed and Data resilience.

3. Troubleshooting:

   • Check the logs of the NFS server container for errors:

     docker logs nfs-server

   • Check the logs of the pod, pv and pvc:

     kubectl describe pod <pod-name>
     kubectl describe pv <pv-name>
     kubectl describe pvc <pvc-name>

   • Ensure the required kernel modules (nfs, nfsd, etc.) are loaded on the Docker host.

---

Conclusion

You now have a fully functional NFS server integrated with your Kubernetes cluster! This setup provides centralized storage that can be shared across multiple pods. Customize the configuration as needed to suit your specific use case.

If you encounter any issues or have questions, feel free to ask!