Squid Labs

Kubernetes - Longhorn the distributed block storage

Kev's photo
Kev
·

4 min read

Long horn is an open source project by Rancer Labs, in its words offer reliable, lightweight, and user-friendly distributed block storage system for Kubernetes.

The Purpose of Longhorn

The principal aim of Longhorn is to make it as easy as possible to manage persistent storage for Kubernetes workloads.

Longhorn is designed to be platform-agnostic. This means it can work across bare-metal, on-premises, and cloud-based Kubernetes installations, providing a consistent experience no matter where your cluster is hosted.

The decision to choose Longhorn comes if it fits your criteria of

  • if you have bare metal clusters and no network storage, just attached directly, and you are running stateful workloads in your clusters.

  • Your environment requires some sort of storage and there is high no trust policy in place.

  • You have other options, like Portworx, Rook, or a wholesale move to like Robin. Longhorn competes with those. The difference is Longhorn is easy to manage and deploy

  • in AWS ebs volumes can't move across AZ, so if you have a POD that is created in AZ a and then, as result of a new deployment, or POD failure or node failure a new POD is scheduled in AZ b, the volume will not move and the POD will hang there

Installing Longhorn

You can perform kubectl, if you decide to change the replica amount, change it in the number of replica

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml

# Source: longhorn/templates/storageclass.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-storageclass
  namespace: longhorn-system
  labels:
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.7.2
data:
  storageclass.yaml: |
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: longhorn
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: driver.longhorn.io
    allowVolumeExpansion: true
    reclaimPolicy: "Delete"
    volumeBindingMode: Immediate
    parameters:
      numberOfReplicas: "2"
      staleReplicaTimeout: "30"
      fromBackup: ""
      fsType: "ext4"
      dataLocality: "disabled"
      unmapMarkSnapChainRemoved: "ignored"
      disableRevisionCounter: "true"
      dataEngine: "v1"
---

Once deployed if you plan to use it as Persistence Volume (PV) for your application, do this

Persistent Volume (PV)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: longhorn-pv
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteMany 
  persistentVolumeReclaimPolicy: Retain
  storageClassName: longhorn  
  csi:
    driver: driver.longhorn.io
    fsType: ext4
    volumeHandle: longhorn-volume-rwx

Persistent Volume Claim (PVC)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-pvc
spec:
  accessModes:
    - ReadWriteMany  
  resources:
    requests:
      storage: 30Gi
  storageClassName: longhorn

Pod uses Longhorn PVC

apiVersion: v1
kind: Pod
metadata:
  name: pod-1
spec:
  containers:
    - name: app
      image: nginx
      volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: longhorn-volume
  volumes:
    - name: longhorn-volume
      persistentVolumeClaim:
        claimName: longhorn-pvc
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-2
spec:
  containers:
    - name: app
      image: nginx
      volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: longhorn-volume
  volumes:
    - name: longhorn-volume
      persistentVolumeClaim:
        claimName: longhorn-pvc

1. How data is replicated :

  • Data Striping : When you create a volume in Longhorn, the data on that volume is split into replicas.

  • Distributed Replicas : These replicas are stored on different nodes in the Kubernetes cluster. Longhorn will ensure that the replicas are always in sync with each other.

  • Replica Manager : Each node has a Replica Manager to manage and monitor synchronization between replicas. When new data is written to one replica, it is automatically copied to other replicas to ensure consistency.

2. How many replicates should I keep?

The number of replicas depends on your availability and resource requirements:

  • Minimum 2 replicas : This is the baseline to ensure data is backed up and still exists if a node fails.

  • 3 replicas : This is a common and recommended configuration. With 3 replicas, the system can still operate normally even if one node fails, and you still have an extra copy of the data. When a replica is lost, Longhorn will automatically recreate the replica from the remaining copies on the other nodes.

    • If you use 3 replicas, the system still has the ability to self-heal even if one replica is lost or unavailable.

3. What happens if a node fails?

  • Automatic recovery : When a node fails, Longhorn automatically detects and disconnects the replica stored on the failed node.

  • Create a new replica : Longhorn will automatically recreate a new replica on another node from the remaining replicas. This process will happen in the background and will not affect your service.

    For example, if you have 3 replicas and one node fails, Longhorn will use the remaining 2 to recreate the 3rd replica on another node.

  • High availability : As long as there are enough active replicas (e.g. 2/3), your service continues to operate without interruption.

longhorn