K3s - Kubernetes Simplified - Revision history

DrEdWilliams: /* External LoadBalancer/Reverse Proxy */

2025-03-29T19:32:26Z

External LoadBalancer/Reverse Proxy

← Older revision		Revision as of 19:32, 29 March 2025
Line 53:		Line 53:
	</pre>		</pre>

	This needs to be at the top level in the NGINX configuration -- either in the /etc/nginx/nginx.conf itself, or included at the top level.		This needs to be at the top level in the NGINX configuration -- either in the /etc/nginx/nginx.conf itself, or included at the top level. This process is more completely described in the K3S documentation:

			https://docs.k3s.io/datastore/cluster-loadbalancer

	If setting up a reverse proxy is not an option, you can set up a round robin in the DNS. Problem is, the pfSense firewall doesn't do round robins by default, so a little arm twisting is needed. Put this in the 'custom options' in the DNS Resolver settings page:		If setting up a reverse proxy is not an option, you can set up a round robin in the DNS. Problem is, the pfSense firewall doesn't do round robins by default, so a little arm twisting is needed. Put this in the 'custom options' in the DNS Resolver settings page:

DrEdWilliams at 19:30, 29 March 2025

2025-03-29T19:30:16Z

← Older revision		Revision as of 19:30, 29 March 2025
Line 111:		Line 111:
	Repeat this for all the masters (minimum 3) providing the same server IP address of the first master. Note that if you need to add/remove more parameters to the k3s service, you need to do it on all server nodes -- just edit the k3s.service file and restart one server at a time.		Repeat this for all the masters (minimum 3) providing the same server IP address of the first master. Note that if you need to add/remove more parameters to the k3s service, you need to do it on all server nodes -- just edit the k3s.service file and restart one server at a time.

	Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the DNS ~~RR name~~ instead of the IP of the first master:		Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the common name through the reverse proxy or DNS Round Robin instead of the IP of the first master:
	sed -i -e s/<server-ip-1>/<~~rr-dns~~-name>/g $HOME/.kube/config		sed -i -e s/<server-ip-1>/<common-name>/g $HOME/.kube/config

	=== Install Worker Nodes ===		=== Install Worker Nodes ===
	At this point, you should have an operational cluster. k3s enables masters to be used for normal pods by default, so this is the minimum operational cluster. Next, install the agent/worker nodes:		At this point, you should have an operational cluster. k3s enables masters to be used for normal pods by default, so this is the minimum operational cluster. Next, install the agent/worker nodes:
	k3sup join --ip <agent-node-x> --k3s-channel stable --server-host <~~rr-dns~~-name>		k3sup join --ip <agent-node-x> --k3s-channel stable --server-host <common-name>

	Explanations:		Explanations:
Line 192:		Line 192:
	helm --kubeconfig /etc/rancher/k3s/k3s.yaml ls --all-namespaces		helm --kubeconfig /etc/rancher/k3s/k3s.yaml ls --all-namespaces

	To access the kubernetes cluster from outside the cluster, copy <pre>/etc/rancher/k3s/k3s.yaml</pre> on your machine located outside the cluster as <pre>~/.kube/config</pre>Then replace the value of the server field with the ~~modified~~ name of the K3s server cluster as described above. kubectl can now manage your K3s cluster.		To access the kubernetes cluster from outside the cluster, copy <pre>/etc/rancher/k3s/k3s.yaml</pre> on your machine located outside the cluster as <pre>~/.kube/config</pre>Then replace the value of the server field with the common name of the K3s server cluster as described above. kubectl can now manage your K3s cluster.

	==Cluster Maintenance==		==Cluster Maintenance==

DrEdWilliams: /* External LoadBalancer/Reverse Proxy */

2025-03-29T19:25:53Z

External LoadBalancer/Reverse Proxy

← Older revision		Revision as of 19:25, 29 March 2025
Line 62:		Line 62:
	local-data: "kube-test.williams.localnet A 10.0.0.64"		local-data: "kube-test.williams.localnet A 10.0.0.64"
	local-data: "kube-test.williams.localnet A 10.0.0.74"		local-data: "kube-test.williams.localnet A 10.0.0.74"
	Change the name and IP addresses as needed, and then load the changes. ~~This~~ approach somewhat complicates the deployment commands below, so pay attention ...		Change the name and IP addresses as needed, and then load the changes.

			Using either approach somewhat complicates the deployment commands below, so pay attention ...

	=== Get k3sup ===		=== Get k3sup ===

DrEdWilliams: update to include nginx reverse proxy

2025-03-29T19:24:34Z

update to include nginx reverse proxy

@@ Line 36: / Line 36: @@
 * combining multiple k3s parameters into single options to k3sup for simplicity (i.e. '''--cluster''')
-=== External LoadBalancer (or DNS Round Robin) ===
+=== External LoadBalancer/Reverse Proxy ===
-The one remaining item needed for a fully HA control plane is an external load balancer to provide a single IP address for accessing the control API from inside or outside the cluster.  Lacking something like a BigIron F5 load balancer or something equivalent, the poor-man's approach is to use a round robin in the DNS.  Problem is, the pfSense firewall doesn't do round robins by default, so a little arm twisting is needed.  Put this in the 'custom options' in the DNS Resolver settings page:
+The one remaining item needed for a fully HA control plane is an external load balancer or reverse proxy to provide a single IP address for accessing the control API from inside or outside the cluster.  Lacking something like a BigIron F5 load balancer or something equivalent, the poor-man's approach is to use a separate reverse proxy.  This can be done with HAproxy or NGINX.  HAproxy can support a HA reverse proxy if you need that level of redundancy, or a simple NGINX configuration like this will work:
   server:
   rrset-roundrobin: yes

DrEdWilliams: /* External LoadBalancer (or DNS Round Robin) */

2025-03-29T12:44:35Z

External LoadBalancer (or DNS Round Robin)

← Older revision		Revision as of 12:44, 29 March 2025
Line 41:		Line 41:
	rrset-roundrobin: yes		rrset-roundrobin: yes
	local-zone: "kube-test.williams.localnet" transparent		local-zone: "kube-test.williams.localnet" transparent
	local-data: "kube-test.williams.localnet A 10.0.0.70"		local-data: "kube-test.williams.localnet A 10.0.0.10"
	local-data: "kube-test.williams.localnet A 10.0.0.71"		local-data: "kube-test.williams.localnet A 10.0.0.64"
	local-data: "kube-test.williams.localnet A 10.0.0.72"		local-data: "kube-test.williams.localnet A 10.0.0.74"
	Change the name and IP addresses as needed, and then load the changes. This approach somewhat complicates the deployment commands below, so pay attention ...		Change the name and IP addresses as needed, and then load the changes. This approach somewhat complicates the deployment commands below, so pay attention ...

10.0.0.10: added user to secondary master creation command and cleaned up typos

2024-09-02T16:13:50Z

added user to secondary master creation command and cleaned up typos

← Older revision		Revision as of 16:13, 2 September 2024
Line 64:		Line 64:
	Install the first master node with these commands. This will put it on the node that the command is issued from ... if that is not the desire, remove the '--local' parameter		Install the first master node with these commands. This will put it on the node that the command is issued from ... if that is not the desire, remove the '--local' parameter
	mkdir $HOME/.kube		mkdir $HOME/.kube
	EXTRA_ARGS="--disable servicelb --disable traefik --disable local-storage --write-kubeconfig-mode=644 --flannel-backend host-gw" \		export EXTRA_ARGS="--disable servicelb --disable traefik --disable local-storage --write-kubeconfig-mode=644 --flannel-backend host-gw" \
	k3sup install --cluster --local --ip <server-ip-1> --k3s-channel stable --k3s-extra-args '$EXTRA_ARGS' \		k3sup install --cluster --local --ip <server-ip-1> --k3s-channel stable --k3s-extra-args '$EXTRA_ARGS' \
	--local-path $HOME/.kube/config --tls-san <rr-dns-name>		--local-path $HOME/.kube/config --tls-san <rr-dns-name>
Line 81:		Line 81:

	Subsequent master nodes can be installed next:		Subsequent master nodes can be installed next:
	k3sup join --server --ip <server~~-ip-2~~> --k3s-channel stable --server-ip <server-ip-1> --k3s-extra-args "$EXTRA_ARGS"		k3sup join --server --host <new master server> --k3s-channel stable --server-host <original master server> --user ewilliam --k3s-extra-args "$EXTRA_ARGS"

	Explanations:		Explanations:
	* join as another server, not an agent/worker		* join as another server, not an agent/worker
	* provide the ~~IP address~~ of the node to be joined		* provide the hostname of the new master node to be joined
	* specify the IP of the first ~~mater~~ created		* specify the hostname of the first master created
	* use the same EXTRA_ARGS provided to the first server		* use the same EXTRA_ARGS provided to the first server

DrEdWilliams: added description on how to access the cluster from outside

2023-07-09T18:36:37Z

added description on how to access the cluster from outside

← Older revision		Revision as of 18:36, 9 July 2023
Line 159:		Line 159:
	Simply apply this manifest and the controller will identify nodes to upgrade and will take care of them as instructed. It's as simple as that. If you don't want to automatically upgrade, comment out the 'channel' parameter in each Plan, and uncomment the 'version' parameter and set it to the version to upgrade to and apply. The controller will see the new plan and act accordingly. If there is nothing to do, the controller just sits there waiting for a new plan with some work.		Simply apply this manifest and the controller will identify nodes to upgrade and will take care of them as instructed. It's as simple as that. If you don't want to automatically upgrade, comment out the 'channel' parameter in each Plan, and uncomment the 'version' parameter and set it to the version to upgrade to and apply. The controller will see the new plan and act accordingly. If there is nothing to do, the controller just sits there waiting for a new plan with some work.

	== Cluster Maintenance ==		== Cluster Access==
			The kubeconfig file stored at /etc/rancher/k3s/k3s.yaml is used to configure access to the Kubernetes cluster. If you have installed upstream Kubernetes command line tools such as kubectl or helm you will need to configure them with the correct kubeconfig path. This can be done by either exporting the KUBECONFIG environment variable or by invoking the --kubeconfig command line flag. Refer to the examples below for details.

			Leverage the KUBECONFIG environment variable:
			export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
			kubectl get pods --all-namespaces
			helm ls --all-namespaces

			Or specify the location of the kubeconfig file in the command:

			kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pods --all-namespaces
			helm --kubeconfig /etc/rancher/k3s/k3s.yaml ls --all-namespaces

			To access the kubernetes cluster from outside the cluster, copy <pre>/etc/rancher/k3s/k3s.yaml</pre> on your machine located outside the cluster as <pre>~/.kube/config</pre>Then replace the value of the server field with the modified name of the K3s server cluster as described above. kubectl can now manage your K3s cluster.

			==Cluster Maintenance==
	As packages (particularly helm charts) are updated, old container images are left behind. Not deleting container images is not generally a bad thing since it enables pods to migrate between nodes in the cluster without waiting for the container to be downloaded. Upgrading packages, however leaves a lot of cruft behind that can eventually cause issues with disk space. Supposedly Kubernetes manages this garbage collection, but other related packages (such as ceph) complain about low disk space long before Kubernetes decides to do anything about it. Cleaning out the old containers is simple, fortunately:		As packages (particularly helm charts) are updated, old container images are left behind. Not deleting container images is not generally a bad thing since it enables pods to migrate between nodes in the cluster without waiting for the container to be downloaded. Upgrading packages, however leaves a lot of cruft behind that can eventually cause issues with disk space. Supposedly Kubernetes manages this garbage collection, but other related packages (such as ceph) complain about low disk space long before Kubernetes decides to do anything about it. Cleaning out the old containers is simple, fortunately:
	/usr/local/bin/k3s crictl rmi -q		/usr/local/bin/k3s crictl rmi -q
	While this can be run at any time, running it too often will remove containers that are still valid, just not used at that time on that node -- resulting in longer migration times when nodes are drained or fail.		While this can be run at any time, running it too often will remove containers that are still valid, just not used at that time on that node -- resulting in longer migration times when nodes are drained or fail.

	== Finishing The Job ==		==Finishing The Job==
	These instructions will create a simple or HA kubernetes cluster. The HA cluster can survive a single master failure, but two will take down the cluster.		These instructions will create a simple or HA kubernetes cluster. The HA cluster can survive a single master failure, but two will take down the cluster.

	At this point you jump back into the rest of the instructions for installing a [[Kubernetes Cluster Installation\|Kubernetes Cluster]], remembering that the network layer and metrics server have already been installed.		At this point you jump back into the rest of the instructions for installing a [[Kubernetes Cluster Installation\|Kubernetes Cluster]], remembering that the network layer and metrics server have already been installed.

DrEdWilliams: added container cleanup instructions

2022-02-25T12:34:46Z

added container cleanup instructions

← Older revision		Revision as of 12:34, 25 February 2022
Line 158:		Line 158:

	Simply apply this manifest and the controller will identify nodes to upgrade and will take care of them as instructed. It's as simple as that. If you don't want to automatically upgrade, comment out the 'channel' parameter in each Plan, and uncomment the 'version' parameter and set it to the version to upgrade to and apply. The controller will see the new plan and act accordingly. If there is nothing to do, the controller just sits there waiting for a new plan with some work.		Simply apply this manifest and the controller will identify nodes to upgrade and will take care of them as instructed. It's as simple as that. If you don't want to automatically upgrade, comment out the 'channel' parameter in each Plan, and uncomment the 'version' parameter and set it to the version to upgrade to and apply. The controller will see the new plan and act accordingly. If there is nothing to do, the controller just sits there waiting for a new plan with some work.

			== Cluster Maintenance ==
			As packages (particularly helm charts) are updated, old container images are left behind. Not deleting container images is not generally a bad thing since it enables pods to migrate between nodes in the cluster without waiting for the container to be downloaded. Upgrading packages, however leaves a lot of cruft behind that can eventually cause issues with disk space. Supposedly Kubernetes manages this garbage collection, but other related packages (such as ceph) complain about low disk space long before Kubernetes decides to do anything about it. Cleaning out the old containers is simple, fortunately:
			/usr/local/bin/k3s crictl rmi -q
			While this can be run at any time, running it too often will remove containers that are still valid, just not used at that time on that node -- resulting in longer migration times when nodes are drained or fail.

	== Finishing The Job ==		== Finishing The Job ==

10.42.1.1: /* Install the master nodes */

2021-10-08T15:55:29Z

Install the master nodes

← Older revision		Revision as of 15:55, 8 October 2021
Line 92:		Line 92:

	Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the DNS RR name instead of the IP of the first master:		Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the DNS RR name instead of the IP of the first master:
	sed -ie s/<server-ip-1>/<rr-dns-name>/g $HOME/.kube/config		sed -i -e s/<server-ip-1>/<rr-dns-name>/g $HOME/.kube/config

	=== Install Worker Nodes ===		=== Install Worker Nodes ===

DrEdWilliams: updated server creation parameters

2021-09-26T19:30:28Z

updated server creation parameters

← Older revision		Revision as of 19:30, 26 September 2021
Line 64:		Line 64:
	Install the first master node with these commands. This will put it on the node that the command is issued from ... if that is not the desire, remove the '--local' parameter		Install the first master node with these commands. This will put it on the node that the command is issued from ... if that is not the desire, remove the '--local' parameter
	mkdir $HOME/.kube		mkdir $HOME/.kube
	k3sup install --cluster --local --ip <server-ip-1> --k3s-channel stable \		EXTRA_ARGS="--disable servicelb --disable traefik --disable local-storage --write-kubeconfig-mode=644 --flannel-backend host-gw" \
	--k3s-extra-args '~~--disable servicelb --write-kubeconfig-mode "0644" --flannel-backend=host-gw~~' \		k3sup install --cluster --local --ip <server-ip-1> --k3s-channel stable --k3s-extra-args '$EXTRA_ARGS' \
	--local-path $HOME/.kube/config --tls-san <rr-dns-name>		--local-path $HOME/.kube/config --tls-san <rr-dns-name>

Line 73:		Line 73:
	* get the last stable version available from k3s (use 'latest' to get bleeding edge updates)		* get the last stable version available from k3s (use 'latest' to get bleeding edge updates)
	* we want to disable the simplified k3s loadbalancer ... it really is a crowbar solution and doesn't provide IP request capability		* we want to disable the simplified k3s loadbalancer ... it really is a crowbar solution and doesn't provide IP request capability
			* disable the default ingress controller, as we really want something better
			* disable the local storage driver (it's a pain if you're installing another storage driver)
	* make the k3s.yaml file readable by the pods ... really should be by default		* make the k3s.yaml file readable by the pods ... really should be by default
	* (temporary workaround) force the flannel backend to use 'host-gw' mode since 'vxlan' mode seems to be broken with Debian 11		* (temporary workaround) force the flannel backend to use 'host-gw' mode since 'vxlan' mode seems to be broken with Debian 11
Line 79:		Line 81:

	Subsequent master nodes can be installed next:		Subsequent master nodes can be installed next:
	k3sup join --server --ip <server-ip-2> --k3s-channel stable --server-ip <server-ip-1>		k3sup join --server --ip <server-ip-2> --k3s-channel stable --server-ip <server-ip-1> --k3s-extra-args "$EXTRA_ARGS"

	Explanations:		Explanations:
Line 85:		Line 87:
	* provide the IP address of the node to be joined		* provide the IP address of the node to be joined
	* specify the IP of the first mater created		* specify the IP of the first mater created
			* use the same EXTRA_ARGS provided to the first server

	Repeat this for all the masters (minimum 3) providing the same server IP address of the first master. ~~After installing~~ the ~~additional master nodes~~, ~~verify that~~ all ~~the 'k3s~~-~~extra~~-~~args' have been added to~~ the k3s.service ~~module on the other nodes. They're supposed to transfer, but ..~~.		Repeat this for all the masters (minimum 3) providing the same server IP address of the first master. Note that if you need to add/remove more parameters to the k3s service, you need to do it on all server nodes -- just edit the k3s.service file and restart one server at a time.

	Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the DNS RR name instead of the IP of the first master:		Now that you have all the masters online, reset the kubeconfig file (~/.kube/config) to use the DNS RR name instead of the IP of the first master: