Setup K3s/Kubernetes Cluster with Ubuntu 20.04 on Raspberry Pi 4s

/## Install K3s

Install the server version of Ubuntu on each Raspberry and assign a different hostname to each. In case you local router (wifi router etc.) does not resolve those hostnames add all to the /etc/hosts file on each Raspberry.

Update system on all Raspberry Pis

sudo apt-get update && sudo apt-get upgrade

Install will fail with error "Failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)" if you do not add a line to each Raspberry's cmdline.txt file and then reboot. So, run:

sudo sed -i '$ s/$/ cgroup_memory=1 cgroup_enable=memory/' /boot/firmware/cmdline.txt

Then reboot.

sudo reboot

Install K3s on master node with (see also https://rancher.com/docs/k3s/latest/en/quick-start/)

curl -sfL https://get.k3s.io | sh -s - server --tls-san <IP of master> --write-kubeconfig-mode 644

And replace with the Raspi's IP address (e.g. 192.168.1...)

If the master node does not start double check that you have run the side command above and reboot.

On the master node retrieve the node-token. You will need the token to add your worker nodes. Get it with:

sudo cat /var/lib/rancher/k3s/server/node-token

You will also need the hostname or IP that you have assigned to your master node for installing the workers.

Now install K3s on each worker with

curl -sfL https://get.k3s.io | K3S_URL=https://<hostname>:6443 K3S_TOKEN=<node-token> sh -

Replace with the hostname of your master node and with the token you retrieved above.

Kubernetes should be up and running now. On the Raspi running the master node run:

kubectl get nodes

This should list all you nodes (the master and the worker nodes).

Install Longhorn

See also https://www.jericdy.com/blog/installing-k3s-with-longhorn-and-usb-storage-on-raspberry-pi

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.2.2/deploy/longhorn.yaml

Setup Ingress for Longhorn

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: longhorn-system
  name: longhorn-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: longhorn.k3s.lan
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80

Add physical storage

Mount the disks
Create a mount point with sudo mkdir /media/hdd1, sudo mkdir /media/ssd1 etc. Find the disk's UUID with sudo blkid. Add entries to the Raspi's /etc/fstab file for each disk:

UUID=<partition uuid> /media/hdd1 ext4 defaults,noatime,nodiratime 0 2
UUID=<partition uuid> /media/ssd1 ext4 defaults,noatime,nodiratime 0 2

Now add the disk to Longhorn by using the Web GUI. Click on "Node"; Then on "Edit Node and Disks" (menu item is hidden under the "Operation" dropdown/button) and finally click on "Add Disk". Give the disk a name and then fill in the mount point previously created (e.g. "/media/hdd1"). Click on the "enable" radio button under scheduling and then on "Save".

To use the Longhorn volumes specify the storage class name "longhorn" in your persistent volume claim (PVC). For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: funky-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 500Gi

You also may want to change the default storage class to make Longhorn the new default and not the local-path that comes with k3s. The following command lists the storage classes:

kubectl get storageclass

The output should be something like:

NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  6h34m
longhorn               driver.longhorn.io      Delete          Immediate              true                   3h32m

Make the k3s local-path not the default:

kubectl patch storageClass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

And make cool longhorn storage class the new default:

kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Run the kubectl get storageclass command again to double check all is ok.

Security related configurations

Update the host os automatically

By default the unattended updates feature will automatically install security updates only. We will modify the configuration slightly to also install regular updates automatically.

The unattended update should already be installed on the Raspberry Ubuntu image. If not you can install it with:

sudo apt-get install unattended-upgrades

Activate it with:

sudo dpkg-reconfigure --priority=low unattended-upgrades

And confirm activation it with "".

Then edit /etc/apt/apt.conf.d/50unattended-upgrades (with sudo vi, sudo nano or your editor of choice) by uncommenting (removing the "//") of the line
"// "${distro_id}:${distro_codename}-updates";". It should then look like:

...
"${distro_id}:${distro_codename}-updates";
...

Leave the rest as is.

Repeat on all Raspberry Pis.

---

How about rolling out a Samba file server to your new cluster? Check out this post.