K3s - Kubernetes Simplified

Lightweight Kubernetes. Production ready, easy to install, half the memory, all in a binary less than 100 MB.

Great for:

• Edge
• IoT
• CI
• Development
• ARM
• Embedding k8s
• Situations where a PhD in k8s clusterology is infeasible

Extracted from https://github.com/k3s-io/k3s ... YMMV

What makes k3s special is that they have stripped out all the bloat-ware from upstream kubernetes that is not needed for small and bare-metal deployments. None of the so-called cloud features are present, and most of the CSI storage drivers are also gone.

As such, it is a simpler replacement for the kubeadm command, and it does a bit more for you:

• installs traefix as the default ingress controller -- it works ... especially for a test/dev cluster, but for production I'll use Contour/Envoy
• installs flannel as the CNI networking layer ... I had problems with flannel before, let's see if this works any better
• installs metrics-server to collect node/pod performance data
• provides a 'local path provisioner' ... not sure what this does ... yet ...

Basic Cluster Installation

They weren't kidding ... this is all it takes to create a simple cluster:

curl -sfL https://get.k3s.io | sh -

A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml and the service is automatically started or restarted. The install script will install K3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh, for example:

sudo kubectl get nodes

K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on your server. To install on worker nodes we should pass K3S_URL along with K3S_TOKEN or K3S_CLUSTER_SECRET environment variables, for example:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=XXX sh -

HA Cluster Installation

One thing that attracted me to this distribution (as it is a distribution, not a fork of kubernetes) is that it provides a very simple way to set up a HA cluster control plane. There are command options in the deployment that provide for the configuration of a common etcd server to facilitate a HA control plane. Even with that ... it can get a bit cumbersome. Enter k3sup.

k3sup

k3sup is a wrapper for the k3s installer that enables:

• installation of a complete cluster from a single login session using SSH
• combining multiple k3s parameters into single options to k3sup for simplicity (i.e. --cluster)

External LoadBalancer/Reverse Proxy

The one remaining item needed for a fully HA control plane is an external load balancer or reverse proxy to provide a single IP address for accessing the control API from inside or outside the cluster. Lacking something like a BigIron F5 load balancer or something equivalent, the poor-man's approach is to use a separate reverse proxy. This can be done with HAproxy or NGINX. HAproxy can support a HA reverse proxy if you need that level of redundancy, or a simple NGINX configuration like this will work:

stream {
  upstream k3s-servers {
    server 10.0.0.10:6443;
    server 10.0.0.74:6443;
    server 10.0.0.64:6443;
  }

  server {
    listen 6443;
    proxy_pass k3s-servers;
  }
}

This needs to be at the top level in the NGINX configuration -- either in the /etc/nginx/nginx.conf itself, or included at the top level. This process is more completely described in the K3S documentation:

 https://docs.k3s.io/datastore/cluster-loadbalancer

If setting up a reverse proxy is not an option, you can set up a round robin in the DNS. Problem is, the pfSense firewall doesn't do round robins by default, so a little arm twisting is needed. Put this in the 'custom options' in the DNS Resolver settings page:

server:
rrset-roundrobin: yes
local-zone: "kube-test.williams.localnet" transparent
local-data: "kube-test.williams.localnet A 10.0.0.10"
local-data: "kube-test.williams.localnet A 10.0.0.64"
local-data: "kube-test.williams.localnet A 10.0.0.74"

Change the name and IP addresses as needed, and then load the changes.

Using either approach somewhat complicates the deployment commands below, so pay attention ...

Get k3sup

Not surprisingly, there are multiple ways to do this. The author wants you to download it from a marketplace app (arkade):

curl -sLS https://dl.get-arkade.dev | sh
# Install the binary using the command given as output, such as: 
sudo cp arkade-darwin /usr/local/bin/arkade
# Or on Linux:
sudo cp arkade /usr/local/bin/arkade
arkade get k3sup

... or you can just download it directly ...

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/

You just need to put it somewhere in your path (or use a full path reference).

Install the master nodes

Install the first master node with these commands. This will put it on the node that the command is issued from ... if that is not the desire, remove the '--local' parameter

mkdir $HOME/.kube
export EXTRA_ARGS="--disable servicelb --disable traefik --disable local-storage --write-kubeconfig-mode=644 --flannel-backend host-gw" \
k3sup install --cluster --local --ip <server-ip-1> --k3s-channel stable --k3s-extra-args '$EXTRA_ARGS' \
              --local-path $HOME/.kube/config --tls-san <rr-dns-name>

Explanations:

• --cluster sets this up for a HA control plane
• --local says to do it on this system, instead of using SSH to the node -- but you still need to provide the IP address, otherwise it will put 127.0.0.1 as the API server address
• get the last stable version available from k3s (use 'latest' to get bleeding edge updates)
• we want to disable the simplified k3s loadbalancer ... it really is a crowbar solution and doesn't provide IP request capability
• disable the default ingress controller, as we really want something better
• disable the local storage driver (it's a pain if you're installing another storage driver)
• make the k3s.yaml file readable by the pods ... really should be by default
• (temporary workaround) force the flannel backend to use 'host-gw' mode since 'vxlan' mode seems to be broken with Debian 11
• the --local-path tells k3sup where to put the kubeconfig files ... this is the standard place, so we will put it there. The directory has to exist -- hence the 'mkdir' command ...
• by providing the name of the DNS round robin, it should be able to pass muster on TLS connections ...

Subsequent master nodes can be installed next:

k3sup join --server --host <new master server> --k3s-channel stable --server-host <original master server> --user ewilliam --k3s-extra-args "$EXTRA_ARGS"

Explanations:

• join as another server, not an agent/worker
• provide the hostname of the new master node to be joined
• specify the hostname of the first master created
• use the same EXTRA_ARGS provided to the first server

Repeat this for all the masters (minimum 3) providing the same server IP address of the first master. Note that if you need to add/remove more parameters to the k3s service, you need to do it on all server nodes -- just edit the k3s.service file and restart one server at a time.

Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the common name through the reverse proxy or DNS Round Robin instead of the IP of the first master:

sed -i -e s/<server-ip-1>/<common-name>/g $HOME/.kube/config

Install Worker Nodes

At this point, you should have an operational cluster. k3s enables masters to be used for normal pods by default, so this is the minimum operational cluster. Next, install the agent/worker nodes:

k3sup join --ip <agent-node-x> --k3s-channel stable --server-host <common-name>

Explanations:

• Joining as an agent, so no '--server' this time
• provide each agent/worker IP in turn
• instead of the '--server-ip' parameter, we are using the '--server-host' parameter so we can use the DNS RR to ensure that the nodes always can access a master

Automated Upgrades: System Upgrade Controller

Rancher has taken the simplicity of k3s to another level by providing a kubernetes controller that can update the cluster ``from within the cluster`` using the System Upgrade Controller.

Installing the controller is straightforward using a single manifest (also located in the git repo):

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.7.6/system-upgrade-controller.yaml

The controller looks for upgrade plans as custom resources (of type 'Plan', for some odd reason) to define its behavior. Two 'Plan's are required ... one for server/master nodes and one for agent/worker nodes. These plans can be set up to either upgrade to a specific k3s version or to monitor the release channel and automatically upgrade when a new release is available. The manifest looks like this:

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/master
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  #version: v1.20.4+k3s1
  channel: https://update.k3s.io/v1-release/channels/stable
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/master
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/k3s-upgrade #:v1.17.4-k3s1
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  #version: v1.20.4+k3s1
  channel: https://update.k3s.io/v1-release/channels/stable

Simply apply this manifest and the controller will identify nodes to upgrade and will take care of them as instructed. It's as simple as that. If you don't want to automatically upgrade, comment out the 'channel' parameter in each Plan, and uncomment the 'version' parameter and set it to the version to upgrade to and apply. The controller will see the new plan and act accordingly. If there is nothing to do, the controller just sits there waiting for a new plan with some work.

Cluster Access

The kubeconfig file stored at /etc/rancher/k3s/k3s.yaml is used to configure access to the Kubernetes cluster. If you have installed upstream Kubernetes command line tools such as kubectl or helm you will need to configure them with the correct kubeconfig path. This can be done by either exporting the KUBECONFIG environment variable or by invoking the --kubeconfig command line flag. Refer to the examples below for details.

Leverage the KUBECONFIG environment variable:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pods --all-namespaces
helm ls --all-namespaces

Or specify the location of the kubeconfig file in the command:

kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pods --all-namespaces
helm --kubeconfig /etc/rancher/k3s/k3s.yaml ls --all-namespaces

To access the kubernetes cluster from outside the cluster, copy

/etc/rancher/k3s/k3s.yaml

on your machine located outside the cluster as

~/.kube/config

Then replace the value of the server field with the common name of the K3s server cluster as described above. kubectl can now manage your K3s cluster.

Cluster Maintenance

As packages (particularly helm charts) are updated, old container images are left behind. Not deleting container images is not generally a bad thing since it enables pods to migrate between nodes in the cluster without waiting for the container to be downloaded. Upgrading packages, however leaves a lot of cruft behind that can eventually cause issues with disk space. Supposedly Kubernetes manages this garbage collection, but other related packages (such as ceph) complain about low disk space long before Kubernetes decides to do anything about it. Cleaning out the old containers is simple, fortunately:

/usr/local/bin/k3s crictl rmi -q

While this can be run at any time, running it too often will remove containers that are still valid, just not used at that time on that node -- resulting in longer migration times when nodes are drained or fail.

Finishing The Job

These instructions will create a simple or HA kubernetes cluster. The HA cluster can survive a single master failure, but two will take down the cluster.

At this point you jump back into the rest of the instructions for installing a Kubernetes Cluster, remembering that the network layer and metrics server have already been installed.