building homelab cluster part 2
Table of Content
building homelab cluster part 2¶
In this part, I am going to setup metallb which helps expose k8s service on LAN by l2 advertisement.
helm¶
Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application
There are many services available to install to kubernetes cluster using helm. Install helm on the same machine used to manage the kubernetes cluster. How to use helm when you are on gitops (you won't be using helm to directly install something to your cluster) will be covered along the way.
https://helm.sh/docs/intro/install/
https://github.com/helm/helm/releases
Go to release page and download desired version, and move it to appropriate path.
# download
cd ~/dnld
curl -LO https://get.helm.sh/helm-v3.14.2-linux-amd64.tar.gz
tar -zxvf helm-v3.14.2-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm version
helm help
metallb¶
https://metallb.universe.tf/installation/#installation-with-helm
I am first going to create a namespace. I want to manage the namespace in ./clusters/homelab/namespace so when I do trial and error to install helmrelease and then revert it, the namespace and other resources not managed by helm will remain unaffected.
---
apiVersion: v1
kind: Namespace
metadata:
name: metallb
labels:
service: metallb
type: infrastructure
Add metallb helm chart and prepare values.yaml file.
# add metallb helm repository
helm repo add metallb https://metallb.github.io/metallb
# confirm available charts and their version in the metallb helm repo
helm search repo metallb
# create directory for infrastructure
mkdir -p infrastructure/homelab/controllers
cd infrastructure/homelab/controllers
# place values file to edit and use
helm show values metallb/metallb > metallb-values.yaml
# in my case, I changed image repository of the metallb controller and speaker, and disabled frr
# prepare directory to store default values.yaml files for my reference
mkdir default-values
helm show values metallb/metallb > default-values/metallb-values.yaml
# edit ./infrastructure/controllers/metallb-values.yaml as necessary
# run diff to easily review the difference
diff metallb-values.yaml default-values/metallb-values.yaml
Prepare namespace, helmrepo, and helmrelease in ./infrastructure/controllers/metallb.yaml. Since helmrepo and helmrelease manifest can be generated using flux command, and helmrelease manifest must be updated everytime you make changes to the values.yaml, it's handy to prepare a script.
#!/bin/bash
# add flux helmrepo to the manifest
flux create source helm metallb \
--url=https://metallb.github.io/metallb \
--interval=1h0m0s \
--namespace=flux-system \
--export >> metallb.yaml
# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease metallb \
--source=HelmRepository/metallb \
--chart=metallb \
--chart-version=0.14.3 \
--values=./metallb-values.yaml \
--interval=10m \
--namespace=flux-system \
--target-namespace=metallb \
--export >> metallb.yaml
Set execution permission and run to generate manifest.
Here is how the generated manifest file looks like. .spec.values section of the HelmRelease is the reflection of how you modified the values file in metallb-values.yaml. It'd be easier to view and modify values.yaml file than the HelmRelease manifest directly, so change what you want in values file, and then run the script to generate the manifest.
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: metallb
namespace: flux-system
spec:
interval: 1h0m0s
url: https://metallb.github.io/metallb
---
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: metallb
namespace: flux-system
spec:
chart:
spec:
chart: metallb
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: metallb
interval: 10m0s
targetNamespace: metallb
values:
controller:
affinity: {}
enabled: true
extraContainers: []
### rest is omitted for brevity ###
Finally, edit ./infrastructure/homelab/controllers/kustomization.yaml by adding metallb.yaml which includes flux helmrepo and helmrelease manifests.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- metallb.yaml
infra-controllers¶
Now, the original directory bootstrapped was ./clusters/homelab/, and all the files added in ./infrastructure/homelab does not get processed by flux at all. I am going to add flux gitrepo for flux to watch this infrastructure directory manifests.
Here is the manifest. It adds the new flux kustomization named "infra-controllers". The existing flux-system kustomization looks after everything installed in ./clusters/homelab/, and now by adding this flux kustomization "infra-controllers" with path set to ./infrastructure/homelab/controllers, there will be a new kustomization created for flux to reconcile whatever placed in that path. This flux kustomization will find ./infrastructure/homelab/controllers/kustomization.yaml file and will run the kubernetes kustomization to include this resources manifest ./infrastructure/homelab/controllers/metallb.yaml.
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-controllers
namespace: flux-system
spec:
interval: 1m0s
path: ./infrastructure/homelab/controllers
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
what's implemented so far¶
2024-02-28T08:39:04.876Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:04.893Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:39:04.911Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 94.145944ms, next run in 1m0s
2024-02-28T08:39:49.812Z info GitRepository/flux-system.flux-system - stored artifact for commit 'add metallb manifest'
2024-02-28T08:39:49.877Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:49.921Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:39:49.938Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 113.169273ms, next run in 1m0s
2024-02-28T08:39:50.190Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:50.255Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-02-28T08:39:50.274Z info Kustomization/flux-system.flux-system - Reconciliation finished in 449.073186ms, next run in 10m0s
2024-02-28T08:39:50.695Z info HelmRelease/metallb.flux-system - Created HelmChart/flux-system/flux-system-metallb with SourceRef 'HelmRepository/flux-system/metallb'
2024-02-28T08:39:50.717Z info HelmRelease/metallb.flux-system - HelmChart 'flux-system/flux-system-metallb' is not ready: latest generation of object has not been reconciled
2024-02-28T08:39:51.050Z info HelmRepository/metallb.flux-system - stored fetched index of size 18.6kB from 'https://metallb.github.io/metallb'
2024-02-28T08:39:51.638Z info HelmChart/flux-system-metallb.flux-system - pulled 'metallb' chart with version '0.14.3'
2024-02-28T08:39:51.651Z info HelmChart/flux-system-metallb.flux-system - artifact up-to-date with remote revision: '0.14.3'
2024-02-28T08:39:51.656Z info HelmRelease/metallb.flux-system - HelmChart/flux-system/flux-system-metallb with SourceRef 'HelmRepository/flux-system/metallb' is in-sync
2024-02-28T08:39:51.664Z info HelmRelease/metallb.flux-system - release not installed: no release in storage for object
2024-02-28T08:39:51.683Z info HelmRelease/metallb.flux-system - running 'install' action with timeout of 5m0s
2024-02-28T08:40:04.712Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:40:04.735Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:40:04.748Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 232.203054ms, next run in 1m0s
2024-02-28T08:40:06.271Z info Kustomization/homelab-sops.flux-system - server-side apply completed
2024-02-28T08:40:06.355Z info Kustomization/homelab-sops.flux-system - Reconciliation finished in 314.22538ms, next run in 1m0s
2024-02-28T08:40:07.285Z info GitRepository/homelab-sops.flux-system - no changes since last reconcilation: observed revision 'main@sha1:202e03894e7025e5702eaeb47ffaab903d66ad9d'
2024-02-28T08:40:17.020Z info HelmRelease/metallb.flux-system - release in-sync with desired state
2024-02-28T08:40:47.788Z info GitRepository/flux-system.flux-system - garbage collected 1 artifacts
2024-02-28T08:40:47.826Z info GitRepository/flux-system.flux-system - no changes since last reconcilation: observed revision 'main@sha1:340bad8d073bceb13b92aec745b866e53b38a03f'
$ flux get all
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system main@sha1:340bad8d False True stored artifact for revision 'main@sha1:340bad8d'
gitrepository/homelab-sops main@sha1:202e0389 False True stored artifact for revision 'main@sha1:202e0389'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/metallb sha256:a13247d1 False True stored artifact: revision 'sha256:a13247d1'
NAME REVISION SUSPENDED READY MESSAGE
helmchart/flux-system-metallb 0.14.3 False True pulled 'metallb' chart with version '0.14.3'
NAME REVISION SUSPENDED READY MESSAGE
helmrelease/metallb 0.14.3 False True Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]
NAME REVISION SUSPENDED READY MESSAGE
kustomization/flux-system main@sha1:340bad8d False True Applied revision: main@sha1:340bad8d
kustomization/homelab-sops main@sha1:202e0389 False True Applied revision: main@sha1:202e0389
kustomization/infra-controllers main@sha1:340bad8d False True Applied revision: main@sha1:340bad8d
$ kubectl get ns
NAME STATUS AGE
calico-apiserver Active 4d23h
calico-system Active 4d23h
default Active 4d23h
flux-system Active 3d7h
kube-node-lease Active 4d23h
kube-public Active 4d23h
kube-system Active 4d23h
metallb Active 2m28s
tigera-operator Active 4d23h
$ kubectl get all -n metallb
NAME READY STATUS RESTARTS AGE
pod/metallb-metallb-controller-79cf4b46c-dhpmq 1/1 Running 0 2m33s
pod/metallb-metallb-speaker-4tz7h 1/1 Running 0 2m33s
pod/metallb-metallb-speaker-8fvjf 1/1 Running 0 2m33s
pod/metallb-metallb-speaker-b5qz8 1/1 Running 0 2m33s
pod/metallb-metallb-speaker-f9s2j 1/1 Running 0 2m33s
pod/metallb-metallb-speaker-jbkzk 1/1 Running 0 2m33s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/metallb-webhook-service ClusterIP 10.109.130.58 <none> 443/TCP 2m33s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/metallb-metallb-speaker 5 5 5 5 5 kubernetes.io/os=linux 2m33s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/metallb-metallb-controller 1/1 1 1 2m33s
NAME DESIRED CURRENT READY AGE
replicaset.apps/metallb-metallb-controller-79cf4b46c 1 1 1 2m33s
$ kubectl api-resources | grep metallb
bfdprofiles metallb.io/v1beta1 true BFDProfile
bgpadvertisements metallb.io/v1beta1 true BGPAdvertisement
bgppeers metallb.io/v1beta2 true BGPPeer
communities metallb.io/v1beta1 true Community
ipaddresspools metallb.io/v1beta1 true IPAddressPool
l2advertisements metallb.io/v1beta1 true L2Advertisement
.
|-clusters
| |-homelab
| | |-network-addon
| | |-flux-system
| | |-sops.yaml
| | |-nodes
| | |-infrastructure.yaml # flux gitrepo to watch and reconcile ./infrastructure/homelab/controllers resources
|-infrastructure
| |-homelab
| | |-controllers
| | | |-kustomization.yaml # ks resource metallb.yaml
| | | |-metallb.yaml # helmrepo and helmrelease
| | | |-metallb.sh # script to generate flux helmrepo and helmrelease manifest
| | | |-default-values
| | | | |-metallb-values.yaml # values file I just like to keep as a reference
| | | |-metallb-values.yaml # customized values file used to generate metallb.yaml
metallb configuration¶
https://metallb.universe.tf/configuration/#layer-2-configuration
I use metallb to L2-advertise IP addresses of the services. To do this I am going to prepare IPAddressPool and L2Advertisement manifests in .infrastructure/homelab/configs directory.
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: lan-addr-pool
namespace: metallb
spec:
addresses:
- 192.168.1.201-192.168.1.220
- 192.168.1.54/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2adv-addr-pool
namespace: metallb
And the kustomization.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- metallb-config.yaml
Following the official flux guide (and this repository), I will modify ./clusters/homelab/infrastructure.yaml flux gitrepo kustomization to include ./infrastructure/homelab/configs and set dependency to the infra-controllers.
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-controllers
namespace: flux-system
spec:
interval: 1m0s
path: ./infrastructure/homelab/controllers
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-configs
namespace: flux-system
spec:
dependsOn:
- name: infra-controllers
interval: 1h
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure/homelab/configs
prune: true
Here is the log.
$ flux get ks
NAME REVISION SUSPENDED READY MESSAGE
flux-system main@sha1:46462174 False True Applied revision: main@sha1:46462174
homelab-sops main@sha1:202e0389 False True Applied revision: main@sha1:202e0389
infra-configs main@sha1:46462174 False True Applied revision: main@sha1:46462174
infra-controllers main@sha1:46462174 False True Applied revision: main@sha1:46462174
$ kubectl get ipaddresspools,l2advertisements -n metallb
NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES
ipaddresspool.metallb.io/lan-addr-pool true false ["192.168.1.201-192.168.1.220","192.168.1.54/32"]
NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS INTERFACES
l2advertisement.metallb.io/l2adv-addr-pool
metallb demo¶
https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports
I am going to use sample manifest available on kubernetes official document to demonstrate metallb functionality.
So here is the manifest without metallb. One thing I added is the namespace. I'll just place it at ./clusters/homelab since it's temporary thing.
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
labels:
app.kubernetes.io/name: proxy
spec:
containers:
- name: nginx
image: nginx:stable
ports:
- containerPort: 80
name: http-web-svc
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: default
spec:
selector:
app.kubernetes.io/name: proxy
ports:
- name: name-of-service-port
protocol: TCP
port: 80
targetPort: http-web-svc
Here's what's created. You cannot access 10.106.254.8:80 from LAN, 192.168.1.0/24.
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx 1/1 Running 0 82s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d
service/nginx-service ClusterIP 10.106.254.8 <none> 80/TCP 82s
To use metallb to expose service on LAN, just choose LoadBalancer type for the service, and metallb will take care of the rest. Here is the modified manifest. Line 32 is all you need to add. The metallb annotation in line 22-23 is something you can use to let metallb use specific IP address from its IP address pool.
Now the demo service "nginx-service" has IP address of 192.168.1.229, and you can access it.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d
nginx-service LoadBalancer 10.96.236.229 192.168.1.219 80:31266/TCP 2m46s
$ curl -I 192.168.1.219
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Wed, 28 Feb 2024 09:53:03 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 11 Apr 2023 01:45:34 GMT
Connection: keep-alive
ETag: "6434bbbe-267"
Accept-Ranges: bytes
homelab gitops repository¶
Here is the homelab repository directory. The metallb demo file is not included.
.
|-clusters
| |-homelab
| | |-infrastructure.yaml # added infra-configs flux kustomization with dependency to infra-controllers
|-infrastructure
| |-homelab
| | |-configs
| | | |-kustomization.yaml # k8s kustomization to include metallb-config.yaml
| | | |-metallb-config.yaml # metallb ipaddress pool and l2 advertisement config
| | |-controllers
|-.git