LXD Playground for Kubernetes

This article describes a way to build with LXD system containers usable to create Kubernetes servers and workers on a single host. I think that understanding the main steps is important not only for this task but also for future usages of LXD.

For a practical, scripted way, see the k-playground.sh script.

Test k0s and K3s using LXD shows a way to use this playground.

Using cloud-init allows for configuring users, IP addresses and SSH keys for those containers.

Here are the steps:

Create a project
Create a dedicate network
Add devices
Add the common cloud-init configuration to default profile
Create profiles to set static IP addresses to containers using cloud-init
Create the containers

For tests and troubleshooting I have used the k0s and K3s lightweight Kubernetes distributions.

Also, check the Notes and Info and links sections.

Create a project

lxc project create "K8sPlay" \
  -c features.images=false \
  -c features.profiles=true

Create a dedicated network

lxc network create "K8sPlayNet" --type=bridge \
    ipv4.address="10.11.12.1/24" \
    ipv4.dhcp.ranges="10.11.12.64-10.11.12.127" \
    ipv4.nat=true \
    ipv6.address=none

Add devices

# the root disk
lxc profile device add default root disk \
    path=/ pool=default --project "K8sPlay"

# a network interface
lxc profile device add default eth0 nic \
    name=eth0 \
    network="K8sPlayNet" --project "K8sPlay"

# /dev/kmsg is needed for Kubelet from K8s and derivatives
lxc profile device add default kmsg unix-char \
    source="/dev/kmsg" path="/dev/kmsg" --project "K8sPlay"

Add the common cloud-init configuration

# br_netfilter is needed by K8s network components
lxc profile set default --project "K8sPlay" linux.kernel_modules=br_netfilter

# for K8s the containers must be privileged
lxc profile set default --project "K8sPlay" security.privileged true

# drop the security, the containers need more rights then usual
cat << EOF | lxc profile set default --project "K8sPlay" raw.lxc -
lxc.apparmor.profile = unconfined
lxc.cgroup.devices.allow = a
lxc.cap.drop =
lxc.mount.auto = cgroup:mixed proc:rw sys:mixed
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file
EOF

# these are basic, "standard", settings
cat << EOF | lxc profile set default --project "K8sPlay" cloud-init.user-data -
#cloud-config
package_upgrade: true
packages:
  - openssh-server
ssh_pwauth: false
users:
- name: default
  gecos: System administrator
  groups: adm,netdev,sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  shell: /bin/bash
  lock_passwd: true
  ssh_authorized_keys:
  - "ssh-ed25519 AAAA___replace_this_with_ypur_public_key___"
EOF

Profiles to set static IP addresses using cloud-init

for (( idx=0; idx<5; idx++ ))
do
    lxc profile create "ip$idx" --project "K8sPlay"

    cat << EOF | lxc profile set "ip$idx" --project "K8sPlay" cloud-init.network-config -
version: 1
config:
  - type: physical
    name: eth0
    subnets:
      - type: static
        ipv4: true
        address: "10.11.12.$(( 10 + idx ))"
        netmask: 255.255.255.0
        gateway: "10.11.12.1"
        control: auto
  - type: nameserver
    address: "10.11.12.1"
EOF
done

Create the containers

for (( idx=0; idx<5; idx++ ))
do
    lxc launch ubuntu-minimal:22.04 "sc$idx" \
        --project "K8sPlay" \
        --profile default \
        --profile "ip$idx"
done

Notes

/dev/kmsg

The whole /dev/kmsg trouble may be avoided by creating a link to /dev/console:

[[ -e /dev/kmsg ]] || ln -s /dev/console /dev/kmsg

preferably with cloud-init.

lxc.mount.auto

I’ve made a test with this setting and worked:

lxc.mount.auto = cgroup:mixed proc:rw sys:mixed

Info and links

Of course, LXD documentation should be bookmarked.

I have sorted most of the settings and steps needed after many hours and tests … wish I have found, and read, these sooner: