Skip to main content

Setting up a bare-metal cluster - part 1

Leda cluster

Introduction

We are going to setup a Kubernetes cluster on bare-metal. Specifically a HA instance of Rancher's k3s using etcd and a virtual ip on three small form factor Thinkcentre machines. The main purpose is to create a simple home lab to explore Kubernetes, do local development and also to host several in-house workloads. The underlying OS is Debian Buster. We'll also look at Ansible to do the initial provisioning of the servers. 

Versions used:

  • Etcd 3.4.13
  • K3s v1.19.3+k3s2
  • Rancher 2.5.2
  • Debian Buster

Overview of the final architecture

Leda cluster architecture

Initial setup of the nodes

The hardware for this cluster consists of three Lenovo Thinkcentre M93 SFF desktops with core i5 processors and 16GB memory. We start by installing Debian Buster from the net-install image, selecting a minimal install enabling only the SSHD service and base utilities. Make sure that there is a regular user with sudo rights, and assign static ip's and hostnames. We're going with; 

  • Helena (10.0.0.85) 
  • Castor (10.0.0.86)
  • Pollux (10.0.0.87)

The three nodes together will form a cluster named Leda. We'll set up a virtual IP 10.0.0.84 that will always point to an available node using Keepalive.  

Copy your ssh-key to all three hosts' default user so we have password-less access over ssh (ssh-copy-id). The rest of the provisioning will be done using ansible. 

Ansible provisioning

Ansible is a great way to administer and configure hosts, especially if you want to configure multiple machines in a cluster and if you may want to repeat the install on additional machines in future. An ansible playbook can take a list of hosts and apply any defined roles to the machines, either running an initial install or configuration, or updating it. You can run it against a host or hosts multiple times and it will only make changes when required.

Inventory

We begin by defining our hosts and groups:

####################
# Hosts            #
####################
pollux
castor
helena

####################
# Leda cluster     #
####################
[leda]
pollux
castor
helena

[primus]
helena

We define the leda cluster as consisting of all three hosts. We separately define helena as member of the primus group, since it will be the initial holder for the virtual IP address managed by Keepalive.        

Roles

Six roles are defined in the main playbook, although not all are needed for the cluster itself. Zabbix, Golang and Docker could be skipped e.g.

  roles:
      - base (apt-get install, setup user preferences, motd, etc)
      - zabbix (add the node to a Zabbix server for monitoring)
      - golang (install Golang)
      - docker (install and configure Docker)
      - keepalived (install and configure Keepalive for the virtual IP)
      - etcd (install and configure etcd in HA)

Keepalived role

Looking more closely at the keepalived role, there are a few specifics we ran into. We need to specify the interface on which the vip will live (Debian Buster uses eno1, rather than the historic eth0), and we'll make the Kubernetes health check conditional (since we won't start with Kubernetes installed.   

Starting with the variables:

    vip_iface: "eno1"
    vip_addr: "10.0.0.84"
    vip_password: "AStrongPassword"
    vip_check_kubernetes: False

The role will install the Keepalive service, make sure it is running and generate the configuration file for each host based on a template (src: keepalived.conf.j2 dest: /etc/keepalived/keepalived.conf). The template is setup to give a higher priority to the host Helena, so that will be the initial owner of the vip address.

keepalived.conf.j2

global_defs {
   # Name of VIP Instance
   router_id {{ cluster_name }}VIP
   enable_script_security
   script_user root
}

{% if vip_check_kubernetes %}
vrrp_script APIServerProbe {
    # Health check the Kubernetes API Server
    script "/usr/bin/curl -k https://{{ cluster_current_node_ip }}:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}
{% endif %}

vrrp_instance APIServerVIP {
    # Interface to bind to
    interface {{ vip_iface }}

    # Set host in group primus to master, rest as backup
    {% if ansible_fqdn in groups['primus'] %}
    state MASTER
    priority 100
    {% else %}
    state BACKUP
    priority 99
    {% endif %}

    # Address of this particular node
    mcast_src_ip {{ cluster_current_node_ip }}

    # A unique ID if more than one service is being defined
    virtual_router_id 61
    advert_int 1
    nopreempt

    # Authentication for keepalived to speak with one another
    authentication {
          auth_type PASS
          auth_pass {{ vip_password }}
    }

    # Kubernetes Virtual IP
    virtual_ipaddress {
        {{ vip_addr }}/24
    }

    {% if vip_check_kubernetes %}
    # Health check function, check Kubernetes api 
    track_script {
        APIServerProbe
    }
    {% else %}
    # No healthcheck required
    {% endif %}

}

So, initially, with vip_check_kubernetes set to false, we'll just rely on Keepalived's standard behaviour. Once we have Kubernetes installed and we have an API listening on port 6443, we'll be able to set vip_check_kubernetes to true and simply re-apply the playbook to enable the more specific health check. 

Etcd role

Etcd requires certificates to be generated and installed on all nodes, so rather than (re)creating them in the regular playbook, we have a createCertificates.yml that generates the certificates before running the main playbook.  

# Create certificates initially
$ ansible-playbook -i cluster/hosts createCertificates.yml
$ ls roles/etcd/artifacts
ca.crt  ca.csr  ca.key  castor.crt  castor.csr  castor.key  helena.crt  helena.csr  helena.key  pollux.crt  pollux.csr  pollux.key

The role itself will check the current version of etcd on the host and download the specified version if it is absent or current is older. Then it will install the configuration and service. The template builds the cluster definition by looping over the hosts in the inventory (so this results in initial-cluster: pollux=https://10.0.0.86:2380,castor=https://10.0.0.87:2380,helena=https://10.0.0.85:2380 in our case).

etcd.conf.yaml.j2

data-dir: /var/lib/etcd/{{ inventory_hostname }}.etcd
name: {{ inventory_hostname }}
initial-advertise-peer-urls: https://{{ hostvars[inventory_hostname]['ansible_facts'][vip_iface]['ipv4']['address'] }}:2380
listen-peer-urls: https://{{ hostvars[inventory_hostname]['ansible_facts'][vip_iface]['ipv4']['address'] }}:2380,https://127.0.0.1:2380
advertise-client-urls: https://{{ hostvars[inventory_hostname]['ansible_facts'][vip_iface]['ipv4']['address'] }}:2379
listen-client-urls: https://{{ hostvars[inventory_hostname]['ansible_facts'][vip_iface]['ipv4']['address'] }}:2379,https://127.0.0.1:2379
initial-cluster-state: new
initial-cluster: {% for host in groups['leda'] %}{{ hostvars[host]['ansible_facts']['hostname'] }}=https://{{ hostvars[host]['ansible_facts'][vip_iface]['ipv4']['address'] }}:2380{% if not loop.last %},{% endif %}{% endfor %}


client-transport-security:
  cert-file: /etc/etcd/ssl/server.crt
  key-file: /etc/etcd/ssl/server.key
  trusted-ca-file: /etc/etcd/ssl/ca.crt
peer-transport-security:
  cert-file: /etc/etcd/ssl/server.crt
  key-file: /etc/etcd/ssl/server.key
  trusted-ca-file: /etc/etcd/ssl/ca.crt

 

Applying the playbook

With everything set up, we can run the playbook against the nodes in the cluster. Several steps required sudo rights, so we will ask the user (-K) for the password before continuing:

#!/bin/bash
echo "Apply playbook to cluster - enter sudo password"
ansible-playbook provision_nodes.yml -K -i cluster/hosts

After Ansible has finished, we'll have all three nodes all configured, we'll have a virtual IP that will always point to a live host, and we will have a highly available etcd cluster that will be the basis of our HA k3s install.

We'll continue in part 2 by checking the services on the nodes and installing k3s.

 

 

 

Department