LXC + Docker on Proxmox

Proxmox server

Do the folling steps on the Proxmox server

Required configurations

Enable modules :

cat << 'EOF' >> /etc/modules
rbd
fuse
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
overlay
br_netfilter
iptable_nat
EOF

Enable forwarding

cat << 'EOF' > /etc/sysctl.d/k3s.conf
kernel.dmesg_restrict=0

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1

kernel.pid_max=100000
user.max_user_namespaces=15000
vm.compact_memory=1
vm.overcommit_memory=1
EOF

Install lxc containers

I choosed ubuntu-20.04-standard as the base container.

Create privileged lxc (uncheck unprivileged).

You have to activate 2 options for the container for Docker to work : Options -> Features -> keyctl and Nesting.

Change the lxc id below (999.conf) with your id :

cat << 'EOF' >> /etc/pve/lxc/999.conf
lxc.apparmor.profile = unconfined
lxc.cgroup2.devices.allow: c 1:11 rwm
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file
swap: 0
EOF

Install overlay and reboot :

  apt install fuse-overlayfs
  reboot

Install Docker in the lxc container

Start your lxc container and enter it

  pct start 999
  pct enter 999

Add this to root's .bashrc

cat << 'EOF' >> /root/.bashrc
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
EOF

Install using the repository

Before you install Docker Engine for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.

Add those packages

  sudo apt-get update

  sudo apt-get install apt-transport-https software-properties-common ca-certificates curl gnupg lsb-release

Add Docker’s official GPG key:

  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Use the following command to set up the stable repository. To add the nightly or test repository, add the word nightly or test (or both) after the word stable in the commands below. Learn about nightly and test channels.

  echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

This procedure works for Debian on x86_64 / amd64, armhf, arm64, and Raspbian.

Update the apt package index, and install the latest version of Docker Engine and containerd, or go to the next step to install a specific version:

  sudo apt-get update
  sudo apt-get install docker-ce docker-ce-cli containerd.io

Verify that Docker Engine is installed correctly by running the hello-world image.

  sudo docker run hello-world

K3s

Add the IP and hostname of each node in the /etc/hosts file of each host.

  vim /etc/hosts
    10.11.12.10 server01
    10.11.12.11 agent01
    10.11.12.12 agent02
    10.11.12.13 agent03

Setup the k3s Master server

  curl -sfL https://get.k3s.io | sh -s - --docker

  systemctl status k3s

  sudo kubectl get nodes -o wide

Setup the k3s Agent Nodes

# TOKEN comes from /var/lib/rancher/k3s/server/node-token on your server
curl -sfL https://get.k3s.io | K3S_TOKEN='K999...::server:999...' K3S_URL='https://10.11.12.13:6443' sh -s -