Set up a Highly Available Kubernetes Cluster using kubeadm

Follow this documentation to set up a highly available Kubernetes cluster using Ubuntu 23.04. You can try this out in VMWare Workstation or VirtualBox

This documentation guides you in setting up a cluster with two master nodes, one worker node and a load balancer node using HAProxy.

Environment

If you want to try this in a virtualized environment on your workstation, please do the minimal installation using Ubuntu

• Virtualbox installed

• Host machine has atleast 8 cores

• Host machine has atleast 8G memory

In kmain1, kmain2, knode1, do this

sudo nano /etc/hosts

add kmain1 as the main cluster, which will be used later

192.168.1.101 dev-k8s-cluster.example.com

Set up load balancer node (go to Load Balancer machine)

Install Haproxy

apt update && apt install -y haproxy

Configure haproxy

Append the below lines to /etc/haproxy/haproxy.cfg

frontend kubernetes-frontend
    bind 192.168.1.100:6443
    mode tcp
    option tcplog
    default_backend kubernetes-backend

backend kubernetes-backend
    mode tcp
    option tcp-check
    balance roundrobin
    server kmain1 192.168.1.101:6443 check fall 3 rise 2
    server kmain2 192.168.1.102:6443 check fall 3 rise 2

Restart haproxy service

systemctl restart haproxy

On all kubernetes nodes (kmain1, kmain2, knode1)

1) Upgrade your Ubuntu servers

Provision the servers to be used in the deployment of Kubernetes on Ubuntu 22.04. The setup process will vary depending on the virtualization or cloud environment you’re using.

Once the servers are ready, update them.

sudo apt update
sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

2) Install kubelet, kubeadm and kubectl

Once the servers are rebooted, add Kubernetes repository for Ubuntu 22304 to all the servers.

sudo apt install curl apt-transport-https -y
curl -fsSL  https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/k8s.gpg
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Then install required packages.

sudo apt update
sudo apt install wget curl vimgit kubelet kubeadm kubectl -y
sudo apt-mark hold kubelet kubeadm kubectl

3) Disable Swap Space

Disable all swaps from /proc/swaps.

sudo swapoff -a

Now disable Linux swap space permanently in /etc/fstab. Search for a swap line and add # (hashtag) sign in front of the line.

$ sudo vim /etc/fstab
#/swap.img    none    swap    sw    0    0

Enable kernel modules and configure sysctl.

# Enable kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

4) Install Container runtime

To run containers in Pods, Kubernetes uses a container runtime. Supported container runtimes are:

Installing Containerd

You can also use containerd instead of Docker

# Configure persistent loading of modules
sudo tee /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF

# Load at runtime
sudo modprobe overlay
sudo modprobe br_netfilter

# Ensure sysctl params are set
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload configs
sudo sysctl --system

# Install required packages
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

# Add Docker repo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Install containerd
sudo apt update
sudo apt install -y containerd.io

# Configure containerd and start service
sudo su -
mkdir -p /etc/containerd
containerd config default>/etc/containerd/config.toml

# restart containerd
sudo systemctl restart containerd
sudo systemctl enable containerd
systemctl status containerd

To use the systemd cgroup driver, open vim/etc/containerd/config.toml, change the SystemdCgroup = true.

5) Initialize control plane (run on kmain1)

Login to the server to be used as master and make sure that the br_netfilter module is loaded:

$ lsmod | grep br_netfilter
br_netfilter           22256  0 
bridge                151336  2 br_netfilter,ebtable_broute

Enable kubelet service.

sudo systemctl enable kubelet

On kmain1

We now want to initialize the machine that will run the control plane components which includes etcd (the cluster database) and the API Server.

Pull container images:

$ sudo kubeadm config images pull

Initialize Kubernetes Cluster

$ sudo kubeadm init 
--pod-network-cidr=10.244.0.0/16 
--cri-socket unix://var/run/containerd/containerd.sock 
--upload-certs 
--control-plane-endpoint=dev-k8s-cluster.example.io

Once you see the initialization has completed, there is a long list of tasks to proceed, to which I touch below

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
#do this
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

On kmain2

You can add this for the new control plane for kmain2, replace the token and discover-token-ca-cert-hash and certificate-key it will appear on your screen on kmain1

  kubeadm join dev-k8s-cluster.example.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
    --discovery-token-ca-cert-hash sha256:1d2c7d5a14c5d717a676291ff5ac25e041386e6371d56bb5cc82d5907fdf62a1 \
    --control-plane  --certificate-key 540e822e3bb5094efac63be268d28496536c3b38be491e56bc093db5f5c0c80b

On knode1, replace the token and discover-token-ca-cert-hash

You can add more nodes along the way using kubeadm join

kubeadm join dev-k8s-cluster.example.com:6443 --token sr4l2l.2kvot0pfalh5o4ik \
    --discovery-token-ca-cert-hash sha256:c692fb047e15883b575bd6710779dc2c5af8073f7cab460abd181fd3ddb29a18

In kmain1, Deploy Calico network

kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml

In kmain1, verify the cluster once completed

kubectl cluster-info
kubectl get nodes
kubectl get cs

For further areas, you can setup
1. Dashboard https://computingforgeeks.com/how-to-install-kubernetes-dashboard-with-nodeport/

2. Metrics How To Deploy Metrics Server to Kubernetes Cluster

3. Deploy Prometheus and Grafana https://computingforgeeks.com/setup-prometheus-and-grafana-on-kubernetes/