[{"data":1,"prerenderedAt":735},["ShallowReactive",2],{"\u002Fblog\u002Flabfromzero-4":3,"\u002Fblog\u002Flabfromzero-4-surround":729},{"id":4,"title":5,"authors":6,"badge":7,"body":9,"date":718,"description":719,"extension":720,"image":721,"meta":723,"navigation":724,"path":725,"seo":726,"stem":727,"__hash__":728},"posts\u002F1.blog\u002F6.LabFromZero-4\u002Findex.md","LabFromZero Part 4 - IDP and Platform Services",null,{"label":8},"GitOps, IDP, Platform",{"type":10,"value":11,"toc":704},"minimark",[12,17,21,32,36,41,44,57,66,80,84,87,95,120,129,141,145,153,156,159,173,177,185,193,200,203,211,219,222,227,236,247,251,260,263,268,271,350,354,357,658,662,679,683,692,700],[13,14,16],"h2",{"id":15},"premise","Premise",[18,19,20],"p",{},"After all the infrastructure and GitOps prerequistes are available, it is now time to configure the platform services. This post covers the GitOps the core services (ForgeJo, FluxCD, Zitadel, OpenBao, Headlamp, monitoring) that turn the bare cluster into a usable private cloud.",[18,22,23,24,31],{},"This part extends ",[25,26,30],"a",{"href":27,"rel":28},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops",[29],"nofollow","GitOps repo"," and the configuration of various supporting platform services.",[13,33,35],{"id":34},"core-platform-services","Core platform services",[37,38,40],"h3",{"id":39},"cilium-and-gateway","Cilium and Gateway",[18,42,43],{},"Cilium is utilized for multiple purposes in this guide:",[45,46,47,51,54],"ul",{},[48,49,50],"li",{},"Kube proxy replacement",[48,52,53],{},"Gateway API controller",[48,55,56],{},"CNI",[18,58,59,60,65],{},"The ",[25,61,64],{"href":62,"rel":63},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fcluster\u002Fcilium",[29],"GitOps configuration"," installs various extra resources and configurations for a futureproof setup",[45,67,68,71,74,77],{},[48,69,70],{},"LoadBalancer IP Pool for the gateway services",[48,72,73],{},"L2 Announcement that makes the LB VIP visible and reachable on the local lab network",[48,75,76],{},"Gateway configuration for the main entry point for exosing services externally",[48,78,79],{},"HTTPS redirect route",[37,81,83],{"id":82},"identity-provider-zitadel","Identity Provider - Zitadel",[18,85,86],{},"A proper private cloud environment needs to have its own IDP, where users and groups are managed. A central location for authentication and authorization for the whole environment.",[18,88,89,94],{},[25,90,93],{"href":91,"rel":92},"https:\u002F\u002Fzitadel.com\u002F",[29],"Zitadel"," provides OIDC for:",[45,96,97,105,108,111,114,117],{},[48,98,99,100,104],{},"Kubernetes API access (",[101,102,103],"code",{},"kubectl oidc-login",")",[48,106,107],{},"Headlamp",[48,109,110],{},"OpenBao",[48,112,113],{},"ForgeJo",[48,115,116],{},"Grafana",[48,118,119],{},"And any future applications",[18,121,122,123,128],{},"The default admin user and the default Zitadel Organization is set as ",[25,124,127],{"href":125,"rel":126},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fzitadel\u002Fhelmrelease.yaml",[29],"helm overrides",".",[18,130,131,135,136,128],{},[132,133,134],"strong",{},"IaC note",": All the specific OIDC application and group configrations should be captured in ",[25,137,140],{"href":138,"rel":139},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fzitadel\u002Fterraform",[29],"terraform config",[37,142,144],{"id":143},"forgejo-git-and-docker-repo","ForgeJo (git and docker repo)",[18,146,147,152],{},[25,148,151],{"href":149,"rel":150},"https:\u002F\u002Fforgejo.org\u002F",[29],"Self-hosted Git"," that serves the GitOps repository. After the initial Helm install it is migrated under the FluxCD bootstrap repo so upgrades and configuration stay declarative. In this GitOps repository it is configured with the respective Zitadel OIDC client. Disabled registation so only federated users are allowed to log in.",[18,154,155],{},"ForgeJo is capable of hosting OCI packages as well. Later on we are going to use this functionality to store some custom images.",[37,157,107],{"id":158},"headlamp",[18,160,161,166,167,172],{},[25,162,165],{"href":163,"rel":164},"https:\u002F\u002Fheadlamp.dev\u002F",[29],"Kubernetes web UI",", ",[25,168,171],{"href":169,"rel":170},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fheadlamp",[29],"configured"," to use Zitadel OIDC so operators never need long-lived kubeconfig credentials on their workstations.",[37,174,176],{"id":175},"observability","Observability",[45,178,179,182],{},[48,180,181],{},"Prometheus (metrics)",[48,183,184],{},"Grafana (dashboards)",[18,186,187,188],{},"This is pretty much standard configuraition with ",[25,189,192],{"href":190,"rel":191},"https:\u002F\u002Fgithub.com\u002Fprometheus-community\u002Fhelm-charts\u002Ftree\u002Fmain\u002Fcharts\u002Fkube-prometheus-stack",[29],"kube-prometheus-stack",[18,194,195],{},[25,196,199],{"href":197,"rel":198},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fkube-prometheus-stack",[29],"Related Flux \u002F Helm Configuration",[37,201,110],{"id":202},"openbao",[18,204,205,206,210],{},"OpenBao with the YubiHSM unseal configuration requires a bit more preparation. ",[25,207,209],{"href":208},"\u002Fblog\u002Fopenbao-yubihsm","An earlier post explains some technicalities"," around this topic.",[18,212,213,214,128],{},"The interaction with the YubiHSM requres some specific PKCS#11 implementation libraries. These libs are required to be accessible in the image. ",[25,215,218],{"href":216,"rel":217},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F2.0-OCIs\u002Fopenbao-ubi-yubihsm\u002FContainerfile",[29],"The containerfile contains all the required installations.",[18,220,221],{},"This containerfile is then pushed into the forgejo package registry that should already be running fine in this lab environment:",[18,223,224],{},[101,225,226],{},"git.labfromzero.internal\u002FLabFromZero\u002Fopenbao-ubi-yubihsm:2.7.0",[18,228,229,230,235],{},"This custom image needs to be ",[25,231,234],{"href":232,"rel":233},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fopenbao\u002Fhelmrelease.yaml",[29],"overridden for the helm deployment"," with other specific details regarding the YubiHSM integration.",[18,237,238,240,241,246],{},[132,239,134],{},": All the specific OIDC authentication groups, policies and roles are ",[25,242,245],{"href":243,"rel":244},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fapps\u002Fopenbao\u002Fterraform",[29],"set up via terraform"," application that should be the foundation for any day-2 maintenance.",[37,248,250],{"id":249},"kubernetes-oidc-auth","Kubernetes OIDC Auth",[18,252,253,254,259],{},"Operators should not use the Kubernetes admin.conf to access the cluster via kubeclt. Since it support ODIC auth ",[25,255,258],{"href":256,"rel":257},"https:\u002F\u002Fcodeberg.org\u002Ftessellative\u002FLabFromZero\u002Fsrc\u002Fbranch\u002Fmaster\u002F3.0-gitops\u002Fcluster\u002Fk8s-oidc",[29],"the cluster can be configured"," to check the authorization of the users via OIDC.",[18,261,262],{},"The configuration is 2-fold. Both the apiserver and the operator clients need certain changes",[264,265,267],"h4",{"id":266},"oidc-kubernetes-api-server","OIDC Kubernetes API Server",[18,269,270],{},"The api server needs to have the respective OIDC config, so the following changes on all control nodes are required:",[272,273,278],"pre",{"className":274,"code":275,"language":276,"meta":277,"style":277},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# \u002Fetc\u002Fkubernetes\u002Fmanifests\u002Fkube-apiserver.yaml\n...\n- --oidc-issuer-url=https:\u002F\u002Fidp.labfromzero.internal\n- --oidc-client-id=39**********83 # k8s-LabFromZero app clientID\n- --oidc-username-claim=email\n- --oidc-groups-claim=groups\n- --oidc-ca-file=\u002Fetc\u002Fssl\u002Fcerts\u002Fca-certificates.crt # trusted CAs on debian \u002F ubuntu in this case certmanager intermediate CA is trusted by the host\n...\n","yaml","",[101,279,280,289,296,307,318,326,334,345],{"__ignoreMap":277},[281,282,285],"span",{"class":283,"line":284},"line",1,[281,286,288],{"class":287},"sHwdD","# \u002Fetc\u002Fkubernetes\u002Fmanifests\u002Fkube-apiserver.yaml\n",[281,290,292],{"class":283,"line":291},2,[281,293,295],{"class":294},"sBMFI","...\n",[281,297,299,303],{"class":283,"line":298},3,[281,300,302],{"class":301},"sMK4o","-",[281,304,306],{"class":305},"sfazB"," --oidc-issuer-url=https:\u002F\u002Fidp.labfromzero.internal\n",[281,308,310,312,315],{"class":283,"line":309},4,[281,311,302],{"class":301},[281,313,314],{"class":305}," --oidc-client-id=39**********83",[281,316,317],{"class":287}," # k8s-LabFromZero app clientID\n",[281,319,321,323],{"class":283,"line":320},5,[281,322,302],{"class":301},[281,324,325],{"class":305}," --oidc-username-claim=email\n",[281,327,329,331],{"class":283,"line":328},6,[281,330,302],{"class":301},[281,332,333],{"class":305}," --oidc-groups-claim=groups\n",[281,335,337,339,342],{"class":283,"line":336},7,[281,338,302],{"class":301},[281,340,341],{"class":305}," --oidc-ca-file=\u002Fetc\u002Fssl\u002Fcerts\u002Fca-certificates.crt",[281,343,344],{"class":287}," # trusted CAs on debian \u002F ubuntu in this case certmanager intermediate CA is trusted by the host\n",[281,346,348],{"class":283,"line":347},8,[281,349,295],{"class":294},[264,351,353],{"id":352},"kubectl-config","Kubectl Config",[18,355,356],{},"The local operator's kubectl OIDC configuration is in the ~\u002Fkube\u002Fconfig.yaml file. This way the auth flow goes through the Zitadel IDP seamlessly.",[272,358,360],{"className":274,"code":359,"language":276,"meta":277,"style":277},"# ~\u002F.kube\u002Fconfig\napiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUs******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\n    server: https:\u002F\u002Fcontrol001.labfromzero.internal:6443\n  name: LabFromZero\ncontexts:\n- context:\n    cluster: LabFromZero\n    user: oidc\n  name: LabFromZero\ncurrent-context: LabFromZero\nkind: Config\nusers:\n- name: oidc\n  user:\n    exec:\n      apiVersion: client.authentication.k8s.io\u002Fv1\n      args:\n      - oidc-login\n      - get-token\n      - --oidc-issuer-url=https:\u002F\u002Fidp.labfromzero.internal\n      - --oidc-client-id=39**********83\n      - --oidc-extra-scope=openid\n      - --oidc-extra-scope=email\n      - --oidc-extra-scope=profile\n      - --oidc-extra-scope=offline_access\n      command: kubectl\n      env: null\n      interactiveMode: IfAvailable\n      provideClusterInfo: false\n",[101,361,362,367,379,387,396,406,416,426,433,443,453,464,473,483,494,502,514,522,530,541,549,558,566,573,581,589,597,605,613,624,635,646],{"__ignoreMap":277},[281,363,364],{"class":283,"line":284},[281,365,366],{"class":287},"# ~\u002F.kube\u002Fconfig\n",[281,368,369,373,376],{"class":283,"line":291},[281,370,372],{"class":371},"swJcz","apiVersion",[281,374,375],{"class":301},":",[281,377,378],{"class":305}," v1\n",[281,380,381,384],{"class":283,"line":298},[281,382,383],{"class":371},"clusters",[281,385,386],{"class":301},":\n",[281,388,389,391,394],{"class":283,"line":309},[281,390,302],{"class":301},[281,392,393],{"class":371}," cluster",[281,395,386],{"class":301},[281,397,398,401,403],{"class":283,"line":320},[281,399,400],{"class":371},"    certificate-authority-data",[281,402,375],{"class":301},[281,404,405],{"class":305}," LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUs******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\n",[281,407,408,411,413],{"class":283,"line":328},[281,409,410],{"class":371},"    server",[281,412,375],{"class":301},[281,414,415],{"class":305}," https:\u002F\u002Fcontrol001.labfromzero.internal:6443\n",[281,417,418,421,423],{"class":283,"line":336},[281,419,420],{"class":371},"  name",[281,422,375],{"class":301},[281,424,425],{"class":305}," LabFromZero\n",[281,427,428,431],{"class":283,"line":347},[281,429,430],{"class":371},"contexts",[281,432,386],{"class":301},[281,434,436,438,441],{"class":283,"line":435},9,[281,437,302],{"class":301},[281,439,440],{"class":371}," context",[281,442,386],{"class":301},[281,444,446,449,451],{"class":283,"line":445},10,[281,447,448],{"class":371},"    cluster",[281,450,375],{"class":301},[281,452,425],{"class":305},[281,454,456,459,461],{"class":283,"line":455},11,[281,457,458],{"class":371},"    user",[281,460,375],{"class":301},[281,462,463],{"class":305}," oidc\n",[281,465,467,469,471],{"class":283,"line":466},12,[281,468,420],{"class":371},[281,470,375],{"class":301},[281,472,425],{"class":305},[281,474,476,479,481],{"class":283,"line":475},13,[281,477,478],{"class":371},"current-context",[281,480,375],{"class":301},[281,482,425],{"class":305},[281,484,486,489,491],{"class":283,"line":485},14,[281,487,488],{"class":371},"kind",[281,490,375],{"class":301},[281,492,493],{"class":305}," Config\n",[281,495,497,500],{"class":283,"line":496},15,[281,498,499],{"class":371},"users",[281,501,386],{"class":301},[281,503,505,507,510,512],{"class":283,"line":504},16,[281,506,302],{"class":301},[281,508,509],{"class":371}," name",[281,511,375],{"class":301},[281,513,463],{"class":305},[281,515,517,520],{"class":283,"line":516},17,[281,518,519],{"class":371},"  user",[281,521,386],{"class":301},[281,523,525,528],{"class":283,"line":524},18,[281,526,527],{"class":371},"    exec",[281,529,386],{"class":301},[281,531,533,536,538],{"class":283,"line":532},19,[281,534,535],{"class":371},"      apiVersion",[281,537,375],{"class":301},[281,539,540],{"class":305}," client.authentication.k8s.io\u002Fv1\n",[281,542,544,547],{"class":283,"line":543},20,[281,545,546],{"class":371},"      args",[281,548,386],{"class":301},[281,550,552,555],{"class":283,"line":551},21,[281,553,554],{"class":301},"      -",[281,556,557],{"class":305}," oidc-login\n",[281,559,561,563],{"class":283,"line":560},22,[281,562,554],{"class":301},[281,564,565],{"class":305}," get-token\n",[281,567,569,571],{"class":283,"line":568},23,[281,570,554],{"class":301},[281,572,306],{"class":305},[281,574,576,578],{"class":283,"line":575},24,[281,577,554],{"class":301},[281,579,580],{"class":305}," --oidc-client-id=39**********83\n",[281,582,584,586],{"class":283,"line":583},25,[281,585,554],{"class":301},[281,587,588],{"class":305}," --oidc-extra-scope=openid\n",[281,590,592,594],{"class":283,"line":591},26,[281,593,554],{"class":301},[281,595,596],{"class":305}," --oidc-extra-scope=email\n",[281,598,600,602],{"class":283,"line":599},27,[281,601,554],{"class":301},[281,603,604],{"class":305}," --oidc-extra-scope=profile\n",[281,606,608,610],{"class":283,"line":607},28,[281,609,554],{"class":301},[281,611,612],{"class":305}," --oidc-extra-scope=offline_access\n",[281,614,616,619,621],{"class":283,"line":615},29,[281,617,618],{"class":371},"      command",[281,620,375],{"class":301},[281,622,623],{"class":305}," kubectl\n",[281,625,627,630,632],{"class":283,"line":626},30,[281,628,629],{"class":371},"      env",[281,631,375],{"class":301},[281,633,634],{"class":301}," null\n",[281,636,638,641,643],{"class":283,"line":637},31,[281,639,640],{"class":371},"      interactiveMode",[281,642,375],{"class":301},[281,644,645],{"class":305}," IfAvailable\n",[281,647,649,652,654],{"class":283,"line":648},32,[281,650,651],{"class":371},"      provideClusterInfo",[281,653,375],{"class":301},[281,655,657],{"class":656},"sfNiH"," false\n",[13,659,661],{"id":660},"security-posture-of-the-platform-services-layer","Security posture of the platform services layer",[45,663,664,667,670,673,676],{},[48,665,666],{},"All service certificates come from the private PKI via the intermediate CA (cert-manager)",[48,668,669],{},"Authentication flows through Zitadel - no local accounts where avoidable",[48,671,672],{},"Secrets live in OpenBao - the unseal key never leaves the HSM",[48,674,675],{},"Break-glass credentials remain only in the offline password manager",[48,677,678],{},"Network policies and Gateway API routes limit exposure to the lab subnet",[13,680,682],{"id":681},"internal-link-manager-homepage","Internal Link Manager - Homepage",[18,684,685,686,691],{},"A proper link catalog is often missing for most platform teams. ",[25,687,690],{"href":688,"rel":689},"https:\u002F\u002Fgethomepage.dev\u002F",[29],"Homepage"," elegantly solves this problem for this internal lab environment:",[693,694],"img",{"alt":695,"className":696,"src":698,"width":699},"homepage",[697],"img-center","\u002Fblog\u002FLabFromZero\u002Fhomepage.webp",600,[701,702,703],"style",{},"html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .swJcz, html code.shiki .swJcz{--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178}html pre.shiki code .sfNiH, html code.shiki .sfNiH{--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC}",{"title":277,"searchDepth":291,"depth":291,"links":705},[706,707,716,717],{"id":15,"depth":291,"text":16},{"id":34,"depth":291,"text":35,"children":708},[709,710,711,712,713,714,715],{"id":39,"depth":298,"text":40},{"id":82,"depth":298,"text":83},{"id":143,"depth":298,"text":144},{"id":158,"depth":298,"text":107},{"id":175,"depth":298,"text":176},{"id":202,"depth":298,"text":110},{"id":249,"depth":298,"text":250},{"id":660,"depth":291,"text":661},{"id":681,"depth":291,"text":682},"2026-10-12","IDP and platform services setup","md",{"src":722},"\u002Fblog\u002FLabFromZero\u002Fcover_p4.webp",{},true,"\u002Fblog\u002Flabfromzero-4",{"title":5,"description":719},"1.blog\u002F6.LabFromZero-4\u002Findex","7waFS8sLOn4pUal4nBBMYowd0XsUc9sxbt1AIFad8ac",[730,6],{"title":731,"path":732,"stem":733,"description":734,"children":-1},"LabFromZero Part 3 - Cluster Bootstraping and GitOps","\u002Fblog\u002Flabfromzero-3","1.blog\u002F5.LabFromZero-3\u002Findex","Platform GitOps bootstrapping for the lab environment",1791905103952]