Setting up a Kubernetes Cluster on Proxmox (from Scratch!)

Table of Contents

• Introduction
• Prerequisites
• Setting up Virtual Machines in Proxmox
• Preliminary Tweaks
• Installing Kubernetes
• Creating a Node Template
• Initializing the Cluster
• Joining Nodes to the Cluster
• Running an Nginx Container and NodePort Service
• Conclusion

Introduction

This tutorial demonstrates setting up a Kubernetes cluster from scratch using Proxmox Virtual Environment. While Proxmox is used for the examples, the Kubernetes commands are applicable to other virtualization platforms. The tutorial uses Ubuntu Server 22.04.

Prerequisites

• Proxmox Virtual Environment (or another virtualization solution like VMware)
• At least two virtual machines capable of running Ubuntu Server 22.04. Recommended specs:
  • Controller: 2+ cores, 2+ GB RAM
  • Nodes: 1+ core, 1+ GB RAM (more nodes are recommended for load balancing)
• Basic understanding of virtualization and Linux command line.
• An Ubuntu Server 22.04 installation template in Proxmox (highly recommended, simplifies VM creation).

Setting up Virtual Machines in Proxmox

1. Clone the Template: Clone your Ubuntu 22.04 template at least twice. Name them appropriately (e.g., k8s-controller, k8s-node).
2. Configure VM Options:
   • Enable “Start at boot” for both controller and nodes.
   • Add a 15-second startup delay to the node VMs to ensure the controller starts first.
   • Enable the QEMU Guest Agent.
3. Start VMs: Start the controller and node VMs.
4. Install QEMU Guest Agent: Once the VMs boot, log in via SSH and run:

   sudo apt install qemu-guest-agent
   

   Repeat this on all VMs.

Preliminary Tweaks

These steps should be performed on both the controller and node VMs:

1. Install Updates:

   sudo apt update && sudo apt upgrade
   

2. Set Static IP (Controller): (Optional, but recommended). Edit /etc/netplan/*.yaml (the filename may vary), create a backup (cp /etc/netplan/*.yaml /etc/netplan/*.yaml.bak), and configure a static IP address, subnet mask, nameservers, and default gateway.

   The config file should look like this after editing:

   network:
    version: 2
    ethernets:
        eth0:
            addresses: [x.x.x.x/24]
            nameservers:
                addresses: [10.10.10.1]
            routes:
                - to: default
                  via: 10.10.10.1
   

   Test and apply the changes with:

   sudo netplan try
   sudo netplan apply
   

3. Check Hostname: Verify that the hostname is correctly set in /etc/hostname and /etc/hosts.
4. Install Container Runtime:

   sudo apt install containerd
   

5. Configure Containerd: Create the directory and configuration file:

   sudo mkdir -p /etc/containerd
   containerd config default | sudo tee /etc/containerd/config.toml
   

   Edit /etc/containerd/config.toml and set systemd_cgroup = true under [plugins."io.containerd.grpc.v1.cri".containerd.runc.options].

6. Disable Swap: If swap is enabled, comment out the relevant lines in /etc/fstab.
7. Enable IP Forwarding and Bridge Netfilter:
   • Uncomment net.ipv4.ip_forward=1 in /etc/sysctl.conf.
   • Create /etc/modules-load.d/k8s.conf and add br_netfilter.
8. Reboot: Reboot all VMs.

🛠 Installing Kubernetes (v1.30)

These commands should be run on both the controller and node VMs:

1. Install GPG Key:

   curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   

2. Add Kubernetes APT Repository:

   echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   

3. Update APT:

   sudo apt update
   

4. Install Kubernetes Packages:

   sudo apt install kubeadm kubectl kubelet
   

Creating a Node Template

1. Clean Cloud-init: On a node VM, run these commands to prepare it as a template:

   sudo cloud-init clean
   sudo rm -rf /var/lib/cloud/instances/*
   sudo truncate -s 0 /etc/machine-id
   sudo rm -rf /var/lib/dbus/machine-id
   sudo ln -sf /etc/machine-id /var/lib/dbus/machine-id
   

   This ensures that when you clone this VM as a new node, it gets a fresh identity and avoids conflicts.

2. Shutdown and Convert to Template: Shutdown the node VM and convert it to a template in Proxmox.

Initializing the Cluster

These commands are for the controller VM only:

1. Initialize Pod Network: Replace placeholders with your controller’s IP address and hostname.

   There are 2 options: Option 1: Standard (For Learning / Single Node Only) & Option 2: HA-Ready (Recommended). in this tutorial we are going forward with ✅ Option 2: HA-Ready

   sudo kubeadm init --control-plane-endpoint=controller-ip --node-name k8s-ctrlr --pod-network-cidr=10.244.0.0/16
   

   💡 Example:

   # Example:
   sudo kubeadm init --control-plane-endpoint=192.168.1.106 --node-name k8s-ctrlr --pod-network-cidr=10.244.0.0/16
   

2. Save Join Command: Copy the kubeadm join command from the output.
3. Configure Local User Access: Run the commands shown in the kubeadm init output to grant your user access to the cluster.
4. Install Flannel:

   curl -s https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml | kubectl apply -f -
   

5. Verify Pods: Check the status of pods:

   kubectl get pods --all-namespaces
   

   Wait until all pods are running.

Joining Nodes to the Cluster

1. Regenerate Join Command (if needed): If the join command from the initialization timed out, regenerate it on the controller:

   kubeadm token create --print-join-command
   

2. Join Nodes: On each node VM, run the kubeadm join command (with sudo).

Running an Nginx Container and NodePort Service

1. Create Nginx Pod YAML: Create a file named pod.yml with the following content: (Adapt the image source if needed).

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx-example
     labels:
       app: nginx
   spec:
     containers:
       - name: nginx
         image: nginx:latest # or a specific image from linuxserver.io
         ports:
           - containerPort: 80
             name: nginx-http
   

2. Apply Pod:

   kubectl apply -f pod.yml
   

3. Create NodePort Service YAML: Create a file named service-nodeport.yml with the following content:

   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-example
   spec:
     type: NodePort
     ports:
       - name: http
         port: 80
         targetPort: 80
         nodePort: 30080
     selector:
       app: nginx
   

4. Apply Service:

   kubectl apply -f service-nodeport.yml
   

5. Access Nginx: Access the Nginx container via a web browser using the IP address of any node and port 30080. for ex. http://node-ip:30080

Conclusion

You’ve successfully set up a multi-node Kubernetes cluster! Remember to explore further Kubernetes concepts and configurations to fully leverage its capabilities.