LabFromZero Part 3 - Cluster Bootstraping and GitOps
Premise
With the network, PKI, encrypted nodes, Kubernetes cluster and YubiHSM foundation in place, the baseline is ready to build the declarative deployment on top. This post covers the GitOps bootstrap based on the LabFromZero repository.
The chicken-and-egg problems
There are many chicken and egg problems at this stage. The does not have:
- Networking so there is no pod communication, DNS, network policies, etc
- Storage capability so no services with persistence
- Cert-manager so no services are exp
- Ingress / Gateway to serve http traffic externally
- FluxCD controller
First Things First
Cilium CNI
Install Cilium CNI first, cluster networking is required
Install it with helm with the following values:
kubeProxyReplacement: true
k8sServiceHost: "10.10.0.11"
k8sServicePort: "6443"
gatewayAPI:
enabled: true
enableAlpn: true
externalIPs:
enabled: true
l2announcements:
enabled: true
operator:
replicas: 1
nodeSelector:
node-role.kubernetes.io/control-plane: ""
ipam:
operator:
clusterPoolIPv4PodCIDRList:
- 10.244.0.0/16
clusterPoolIPv4MaskSize: 24
bpf:
datapathMode: veth
Initial Gateway configuration
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: static-external
namespace: kube-system
spec:
gatewayClassName: cilium
addresses:
- type: IPAddress
value: 10.10.64.1
allowedListeners:
namespaces:
from: All
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: Same
Gateway API CRDs
Add the gateway api CRDs
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml
Local Path Storage
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
Cert Manager
Install cert-manager via helm with the following values
global:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
enabled: true
crds:
enabled: true
config:
gatewayAPI:
enabled: true
enableListenerSet: true
featureGates:
ListenerSets: true
Cert Manager Intermediate issuer
Add the cert-manager intermediate ca crt and key from the PKI.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: labfromzero-intermediate
spec:
ca:
secretName: cert-manager-intermediate-ca-key-pair
After these the cluster should be able to host and serve services externally!
FluxCD Controller
Install it via Helm
helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
--namespace flux-system \
--create-namespace
FluxCD Bootstrap
Prerequisites
FluxCD wants a Git repository, but the Git service (ForgeJo) itself needs to be deployed. The practical solution used in LabFromZero is:
- Install ForgeJo with Helm first
- Create the labfromzero ogranization with the bootstrap repository
- Create and add the Flux deployer SSH key to a dedicated technical user
Install ForgeJo with Helm
ForgeJo ListenerSet for the Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
name: forgejo
annotations:
cert-manager.io/cluster-issuer: labfromzero-intermediate
spec:
parentRef:
name: static-external
namespace: kube-system
kind: Gateway
group: gateway.networking.k8s.io
listeners:
- name: https
hostname: git.labfromzero.internal
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: forgejo-tls
kind: Secret
group: ""
allowedRoutes:
namespaces:
from: Same
- name: ssh
port: 22
protocol: TCP
allowedRoutes:
namespaces:
from: Same
ForgeJo Helm values:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
persistence:
enabled: true
storageClass: local-path
httpRoute:
enabled: true
hostnames:
- git.labfromzero.internal
parentRefs:
- name: forgejo
sectionName: https
kind: ListenerSet
group: gateway.networking.k8s.io
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: forgejo-http
port: 3000
extraVolumeMounts:
- name: host-ca
mountPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
extraVolumes:
- name: host-ca
hostPath:
path: /etc/ssl/certs/ca-certificates.crt
type: File
gitea:
config:
webhook:
ALLOWED_HOST_LIST: flux-webhook.labfromzero.internal
service:
DISABLE_REGISTRATION: true
ALLOW_ONLY_EXTERNAL_REGISTRATION: true
SHOW_REGISTRATION_BUTTON: false
Flux Bootstrap
Bootstrap the FluxCD GitOps Repo. Kube
flux bootstrap git \
--url=ssh://git@git.labfromzero.internal/LabFromZero/LabFromZero-gitops.git \
--branch=main \
--path=cluster \
--private-key-file=<flux-deploy-private-key>
After this point the cluster should be self-managable. Port the above manually installed services into the GitOps repo.
The GitOps repository layout typically looks like:
apps/
|- forgejo
|- headlamp
|- homepage
|- kube-prometheus-stack
|- metrics-server
|- openbap
|- zitadel
cluster/ # FluxCD Bootstrap Root
|- cert-manager
|- cilim
|- flux-system
|- k8s-oidc
|- <service_name>.yaml # Various customization files for the apps