Skip to main content

Switching to 64 bit kernel

Introduction

The Hollyberry cluster has been running on Raspian's standard 32 bit kernel, which means we can only use armv7 images. The latest beta release however allows us to use a 64 bit kernel. Although the distribution itself will stay 32 bit for now, switching to 64 bt will allow us to use armv8 images next to armv7. To switch we'll update the firmware on the Raspberry pi's, (re)-enable some configurations and reboot. 

Upgrade the nodes

We will update the nodes one by one, while the cluster stays up. We are running K3S that needs to have legacy IP tables enabled and cgroups set.

Steps to follow per node:

  • Update firmware (rpi-update)
  • Enable 64bit kernel (going to 5.4.75-v8+ from 4.19.97-v7l+ 
  • Re-enable legacy ip tables
  • Enable cgroups memory on boot

On the first (master) node:

$ sudo rpi-update
$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
$ sudo vi /boot/config.txt #Adding arm_64bit=1
$ sudo vi /boot/cmdline.txt # Adding cgroup_memory=1 cgroup_enable=memory 
$ sudo reboot

Check that we are running the new kernel using uname -a

The other nodes will be upgraded using some ad-hoc ansible commands. For example updating pinode4:

$ ansible pinode4 -b -K -m shell -a "echo -n y | rpi-update"
$ ansible pinode4 -b -K -m shell -a "echo arm_64bit=1 >> /boot/config.txt"
$ ansible pinode4 -b -K -m shell -a "update-alternatives --set iptables /usr/sbin/iptables-legacy && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy"
$ ssh pinode4
pinode4 $ sudo vi /boot/cmdline.txt
	#Add "cgroup_memory=1 cgroup_enable=memory"
	#console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_memory=1 cgroup_enable=memory
pinode4 $ sudo reboot

While we are updating, we can check the status and kernel versions of the nodes using kubectl:

Every 2.0s: kubectl get nodes -o wide          
NAME      STATUS     ROLES    AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
pinode1   Ready      <none>   220d   v1.17.4+k3s1   192.168.1.101   <none>        Raspbian GNU/Linux 10 (buster)   5.4.75-v8+       containerd://1.3.3-k3s2
pinode0   Ready      master   220d   v1.17.4+k3s1   192.168.1.100   <none>        Raspbian GNU/Linux 10 (buster)   5.4.75-v8+       containerd://1.3.3-k3s2
pinode4   NotReady   <none>   219d   v1.17.4+k3s1   192.168.1.104   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     containerd://1.3.3-k3s2
pinode3   Ready      <none>   219d   v1.17.4+k3s1   192.168.1.103   <none>        Raspbian GNU/Linux 10 (buster)   5.4.75-v8+       containerd://1.3.3-k3s2
pinode5   Ready      <none>   219d   v1.17.4+k3s1   192.168.1.105   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     containerd://1.3.3-k3s2
pinode2   Ready      <none>   219d   v1.17.4+k3s1   192.168.1.102   <none>        Raspbian GNU/Linux 10 (buster)   5.4.75-v8+       containerd://1.3.3-k3s2

After the last node is updated, double check all kernels using ansible:

pi@pinode0:~ $ ansible picluster -m shell -a "uname -a"
pinode4 | CHANGED | rc=0 >>
Linux pinode4 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

pinode2 | CHANGED | rc=0 >>
Linux pinode2 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

pinode1 | CHANGED | rc=0 >>
Linux pinode1 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

pinode3 | CHANGED | rc=0 >>
Linux pinode3 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

pinode5 | CHANGED | rc=0 >>
Linux pinode5 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

pinode0 | CHANGED | rc=0 >>
Linux pinode0 5.4.75-v8+ #1367 SMP PREEMPT Mon Nov 9 15:11:16 GMT 2020 aarch64 GNU/Linux

Rancher

Adding a cluster to Rancher depends on an agent being deployed to the cluster, but there is no ready-made armv7 version at the moment. One of the reasons to upgrade to a 64 bit kernel was to be able to deploy the armv8 (aarch64) agent. Initial attempt showed that Rancher 2.5.2 doesn't have an armv8 image yet. Trying to install 2.5.0 for which an aarch64 image is available succeeded initially, but crashed on configuration.

  

   

 

Department