LabFromZero Part 2 - Network, k8s, HSM

From Network to Cluster

This post condenses the foundational layers that turn bare hardware into a running, trusted Kubernetes cluster. It draws from the LabFromZero sections 1.1-networking, 1.2-k8s-nodes, 1.3-k8s-setup, and 1.4-YubiHSM.

Networking & DNS

The lab lives on its own subnet so it can be isolated from the rest of the LAN.

• External LAN: 10.0.0.0/16
• Lab network: 10.10.0.0/16
• Connectivity: static routes (chosen for simplicity over BGP or pure NAT)

Infra subnet plan (10.10.0.0/16)

Initial DNS records (labfromzero.internal)

control001.labfromzero.internal     A     10.10.0.11
worker001.labfromzero.internal      A     10.10.0.101
router.labfromzero.internal         A     10.10.0.1
gateway.labfromzero.internal        A     10.10.64.1

idp.labfromzero.internal            CNAME gateway.labfromzero.internal
headlamp.labfromzero.internal       CNAME gateway.labfromzero.internal
git.labfromzero.internal            CNAME gateway.labfromzero.internal
grafana.labfromzero.internal        CNAME gateway.labfromzero.internal
prometheus.labfromzero.internal     CNAME gateway.labfromzero.internal
openbao.labfromzero.internal        CNAME gateway.labfromzero.internal
hsm.labfromzero.internal            CNAME gateway.labfromzero.internal
flux-webhook.labfromzero.internal   CNAME gateway.labfromzero.internal
# ... additional CNAMEs as services appear

IaC note: These static mappings and the MikroTik route configuration are excellent candidates for Ansible or Terraform / OpenTofu.

Bare-Metal Nodes – LUKS + TPM2 and Storage

Control node (control001)

Install Ubuntu LTS 26.04 with the proper disk layout that uses LVM on LUKS. TPM2 auto-unlock is enrolled with PCR 7 (Secure Boot state).

After any firmware or dbx update, re-enroll the TPM slot!

Worker node (worker001)

• Root filesystem on LUKS (same TPM2 pattern).
• Data pool: 4×2 TB NVMe in ZFS RAID-Z2 double parity.
• Optional encrypted ZFS dataset for libvirt/VM storage, with the key sealed via systemd-creds and a small oneshot unit that loads it at boot.

IaC note: Node OS installation and the LUKS/TPM enrolment steps are still largely manual. Once the nodes are reachable, Ansible can lock down SSH, install packages, distribute the Root CA, and prepare kubeadm prerequisites.

Kubernetes Cluster Bootstrap

On the control node the kubernetes init configuration should look like:

NODE_IP="$(hostname -I | awk '{print $1}')"
NODE_NAME="$(hostname)"
cat > /root/kubeadm.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
skipPhases:
  - addon/kube-proxy
localAPIEndpoint:
  advertiseAddress: ${NODE_IP}
  bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
networking:
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
    - ${NODE_NAME}
    - ${NODE_IP}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
EOF
kubeadm init --config /root/kubeadm.yaml

Join the worker with the token and CA hash printed by kubeadm init. Because many platform services run on the control node, the control-plane taint is removed.

Install Gateway API CRDs and a local-path provisioner so workloads have storage. Later on consider deploying a proper distributed storage solution such as Longhorn, or an external storage appliance like NetApp / TrueNAS Scale / Synology NAS. This Lab environment features a 10 Gbit LAN so a network storage solution is feasible!

IaC note: The kubeadm configuration and the join process can be templated with Ansible or something like Kubespray can be used for the setup and day-2 operation!

YubiHSM and OpenBao Auto-Unseal

The YubiHSM 2 provides a hardware root of trust for OpenBao’s seal key.

Connector on the control node

• Install the YubiHSM SDK connector on the control node
• Place the TLS cert and key generated earlier under /etc/yubihsm/
• Configure /etc/yubihsm-connector.yaml to listen on the lab network and use the certificate
• Start and enable the yubihsm-connector service

Installation steps in detail

HSM key setup

Using the yubihsm-shell set up the YubiHSM:

• Create a new admin authentication key and delete the default weak key.
• Create a dedicated OpenBao authentication key (limited capabilities).
• Generate the RSA-4096 seal key that OpenBao will use for auto-unseal.

Setup steps in detail

Store the resulting PINs in KeePassXC and inject the OpenBao PIN into the cluster as a Kubernetes secret.

Later OpenBao is deployed via GitOps and configured with HSM auto-unseal so the unseal key never exists in software.

IaC note: The HSM ceremony is deliberately offline/manual. The subsequent OpenBao deployment and its HSM configuration belong in GitOps manifests.