LabFromZero Part 1 - Hardware, PKI, Security

LabFromZero Series Introduction

Series goal: A Practical guide to construct a self-hosted lab and private-cloud environment with security as a first-class concern. Targeting the audience who is looking for a sovereign self managed infrastructure and platfrom as an alternative to public cloud environments

This series is based on the public repository LabFromZero. The original material is organised as a collection of focused READMEs. This series is expanding it into a coherent blog series.

Scope

A realistic on-premises platform that can host:

• A private Kubernetes cluster (kubeadm, bare metal)
• 10GBit Ethernet Interconnect
• Identity provider, OIDC (Zitadel)
• Secrets management with hardware-backed unseal (OpenBao + YubiHSM)
• GitOps (ForgeJo + FluxCD)
• Observability (Prometheus + Grafana)
• Gateway (Cilium + Gateway API + Envoy Gateway)
• Supporting services (DNS, Headlamp, ODIC, etc.)

All of it sits behind a private PKI, full-disk encryption with TPM2 auto-unlock, and deliberate network segmentation.

Hardware Bill of Materials

Network edge

• Router: MikroTik Clmoud Core Router CCR2004
• 10GBase-T RJ45 SFP+ to Ethernet Adapters
• Patch cables, rack, UPS

Control plane node (control001)

• Lenovo ThinkCentre M75q Gen 5
• CPU: AMD Ryzen 7 PRO 8700GE (8c/16t)
• RAM: 32 GB
• SSD: 1 TB NVMe
• Role: control-plane + system services (IDP, HSM connector, ForgeJo, monitoring, FluxCD, etc.)

Worker node (worker001)

• Motherboard: ASRock TURIND8-2L2T
• CPU: AMD EPYC 9354P (32c/64t)
• RAM: 256 GB (4×64 GB Kingston)
• Storage: 5×2 TB Kingston Fury Renegade NVMe (ZFS RAID-Z2)
• Cooler / PSU / case: Arctic Freezer 4U-SP5, Seasonic Focus GX-1000,
• Case: 19" rackable Silverstone RM52
• Asrock TPM SPI module
• Role: Local AI/ML workloads, Build agents, various cloud services

This is an intentionally modest setup with 1 control and 1 worker node setup. The control node is hosting the core platform services. The worker is a single high-core, high-mem machine. Scale-out or HA can be added / hot-plugged dynamically later.

Security Considerations

Nodes run Ubuntu Server 26.04 with Secure Boot, TPM 2.0 enabled and Microsoft UEFI KEK CAs enabled. Make sure that the BIOS is updated.

Private PKI with offline Root CA

Trust for every TLS endpoint starts here. The Root CA key never touches the network, remains offline! Generate the PKI Infrastrucutre Store the PKI infra / Root Keys securely on a veracrypt drive or KeepassXC attachment

The root ca certificate needs to be distributed and trusted on all nodes, later configurations rely on the proper trust chain.

Intermediate CA and Service Certificates

Cert-manager requires a dedicated intermediate CA to issue gateway / ingress certificates.

YubiHSM needs a dedicated TLS server cert and key to protect the connector endpoint.

Full-disk encryption (LUKS) + TPM2 auto-unlock

During node installation make sure to ecrypt the disk partitions.

• Save the recovery passphrase into the offline / break-glass secret store (veracrypt / KeepassXC).
• Setup the PCR 7 (Secure Boot state) which should be the practical default.

Hardware Security Module (YubiHSM 2)

Network accessible HSM service via Yubico's YubiHSM.

• Hardware based strong security
• Secure key storage and operations
• Network shareable
• Extensive cryptographic capabilities: RSA, ECC, ECDSA (ed25519), SHA-2, AES
• M of N wrap key Backup and Restore
• Interfaces via YubiHSM KSP, PKCS#11, and native libraries
• Tamper evident Audit Logging
• Secure Random Generation
• Used for OpenBao seal-key operations so the unseal key stays secure and will not require manual unsealing.

Offline secret store / Breakglass access

KeePassXC (or equivalent) holds all break-glass credentials, CA private keys, HSM admin passwords, etc. Example store hierarchy

Network/
  |- Router: Mikrotik - CRRF2004
PKI/
  |- ROOT CA
  |- Cert Manager Intermediate CA
Nodes/
  |- WORKER: bearmetal001
  |- CONTROL: ctrl001
IAM - Zitadel/
  |- admin @ LabFromZero.idp.labfromzero.internal
  |- user @ LabFromZero.idp.labfromzero.internal
  |- Zitadel OIDC App - Platform: k8s-labfrom0
  |- Zitadel OIDC App - Platform: Headlamp
  |- Zitadel OIDC App - Platform: OpenBao
YubiHSM/
  |- admin-key
  |- openabao-key
Services/
  |- flux-deploy
  |- openbao-root
  |- k8s-image-puller-creds

Network segmentation

Dedicated lab subnet (10.10.0.0/16) with clear static ranges for nodes, load-balancers, and dynamic hosts.