Skip to content


beryl  cloudflare  cloudflare pages  wrangler  container  docker  registry  harbor  Dockerfile  kaniko  bytebase  gitops  mongodb  pymongo  mdbc  kubernetes  operator  os  debian  preseed  automation  git  github  credentials  gh  git tag  tips  gitlab  cicd  gitlab runner  glab  docker-compose  graphql  mutation  ollama  letta  memory  hlv3  ha  etcd  calico  flux  sops  gateway  nginx  nginx-gateway-fabric  certmanager  gitlab-runner  minio  s3  monitoring  kube-prometheus  prometheus  alertmanager  grafana  node-exporter  memcached  loki  promtail  keycloak  postgres  testbed  homelab  cilium  proxmox  cloud-init  fluxcd  cert-manager  encryption  longhorn  Kubernetes  index  dns  coredns  k9s  krew  storage  postgres-operator  hyper-v  helm  metallb  kubernetes gateway  ngf  directpv  logging  lab  reverse proxy  mfa  llm  open-webui  mcp  markdown  wip  mise  python  venv  go  node  npm  snmp  multicast  nvim  lazyvim  obsidian  ansible  poetry  regex  freenginx  mercurial  tls  tls passthrough  tls offload  bash  zsh  linux  ssg  hugo  mkdocs  tmux  tpm  pomodoro  vagrant  windows  virtualization  powershell  winget  windows terminal  windows 11 pro  oh-my-posh  nerdfonts  scoop 


building home lab part 2

#helm   #metallb  

Table of Content

building home lab part 2

Setting up kubernetes cluster, gitops using flux, and sops for secret encryption were covered in the previous section.

  • [x] prepare nodes (VMs on hyper-v in this series)
  • [x] setup kubernetes cluster
  • [x] bootstrap flux gitops
  • [x] setup sops for secret encryption
  • [ ] install helm
  • [ ] metallb to l2 advertise svc on LAN
  • [ ] gateway as well as nginx gateway fabric to setup https gateway
  • [ ] directpv to setup storage class, to serve persistent volume
  • [ ] minio as s3 storage
  • [ ] gitlab runner to execute CI/CD jobs on my gitlab
  • [ ] kube-prometheus for monitoring
  • [ ] loki for logging
  • [ ] cert manager to manage tls certificate
  • [ ] weave gitops dashboard
  • [ ] kube-dashboard
  • [ ] postgresql
  • [ ] mongodb
  • [ ] bytebase
  • [ ] my private apps

wip

helm

https://helm.sh/

Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application

There are many services available to install to kubernetes cluster using helm. Install helm on the same machine used to manage the kubernetes cluster. How to use helm when you are on gitops (you won't be using helm to directly install something to your cluster) will be covered along the way.

https://helm.sh/docs/intro/install/

https://github.com/helm/helm/releases

Go to release page and download desired version, and move it to appropriate path.

# download
cd ~/dnld
curl -LO https://get.helm.sh/helm-v3.14.2-linux-amd64.tar.gz
tar -zxvf helm-v3.14.2-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm version
helm help

metallb

https://metallb.universe.tf/

https://metallb.universe.tf/installation/#installation-with-helm

Add metallb helm chart and prepare values.yaml file.

# add metallb helm repository
helm repo add metallb https://metallb.github.io/metallb

# confirm available charts and their version in the metallb helm repo
helm search repo metallb

# create directory for metallb
cd {your repository}
cd infrastructure/controllers

# place values file to edit and use
helm show values metallb/metallb > metallb-values.yaml

# in my case, I changed image repository of the metallb controller and speaker, and disabled frr

# prepare directory to store default values.yaml files for my reference
mkdir default-values
helm show values metallb/metallb > default-values/metallb-values.yaml

# edit ./infrastructure/controllers/metallb-values.yaml as necessary
# run diff to easily review the difference
diff metallb-values.yaml default-values/metallb-values.yaml

Prepare namespace, helmrepo, and helmrelease in ./infrastructure/controllers/metallb.yaml. Since helmrepo and helmrelease manifest can be generated using flux command, and helmrelease manifest must be updated everytime you make changes to the values.yaml, it's handy to prepare a script.

./infrastructure/controllers/metallb.sh
#!/bin/bash

# add namespace to the manifest
cat > metallb.yaml<<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: metallb
  labels:
    service: metallb
    type: infrastructure
EOF

# add flux helmrepo to the manifest
flux create source helm metallb \
  --url=https://metallb.github.io/metallb \
  --interval=1h0m0s \
  --namespace=flux-system \
  --export >> metallb.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease metallb \
  --source=HelmRepository/metallb \
  --chart=metallb \
  --values=./metallb-values.yaml \
  --interval=10m \
  --namespace=flux-system \
  --target-namespace=metallb \
  --export >> metallb.yaml

Set execution permission and run to generate manifest.

chmod u+x metallb.sh
./metallb.sh

Here is how the generated manifest file looks like. .spec.values section of the HelmRelease is the reflection of how you modified the values file in metallb-values.yaml. It'd be easier to view and modify values.yaml file than the HelmRelease manifest directly, so change what you want in values file, and then run the script to generate the manifest.

metallb.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: metallb
  labels:
    service: metallb
    type: infrastructure
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: metallb
  namespace: flux-system
spec:
  interval: 1h0m0s
  url: https://metallb.github.io/metallb
---
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: metallb
  namespace: flux-system
spec:
  chart:
    spec:
      chart: metallb
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: metallb
  interval: 10m0s
  targetNamespace: metallb
  values:
    controller:
      affinity: {}
      enabled: true
      extraContainers: []
      ### rest is omitted for brevity ###

Finally, edit ./infrastructure/controllers/kustomization.yaml by adding metallb.yaml.

./infrastructure/controllers/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sops.yaml
  - metallb.yaml

what's implemented so far

flux logs --since 3m -f
2024-02-28T08:39:04.876Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:04.893Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:39:04.911Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 94.145944ms, next run in 1m0s
2024-02-28T08:39:49.812Z info GitRepository/flux-system.flux-system - stored artifact for commit 'add metallb manifest'
2024-02-28T08:39:49.877Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:49.921Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:39:49.938Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 113.169273ms, next run in 1m0s
2024-02-28T08:39:50.190Z info Kustomization/flux-system.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:39:50.255Z info Kustomization/flux-system.flux-system - server-side apply completed
2024-02-28T08:39:50.274Z info Kustomization/flux-system.flux-system - Reconciliation finished in 449.073186ms, next run in 10m0s
2024-02-28T08:39:50.695Z info HelmRelease/metallb.flux-system - Created HelmChart/flux-system/flux-system-metallb with SourceRef 'HelmRepository/flux-system/metallb'
2024-02-28T08:39:50.717Z info HelmRelease/metallb.flux-system - HelmChart 'flux-system/flux-system-metallb' is not ready: latest generation of object has not been reconciled
2024-02-28T08:39:51.050Z info HelmRepository/metallb.flux-system - stored fetched index of size 18.6kB from 'https://metallb.github.io/metallb'
2024-02-28T08:39:51.638Z info HelmChart/flux-system-metallb.flux-system - pulled 'metallb' chart with version '0.14.3'
2024-02-28T08:39:51.651Z info HelmChart/flux-system-metallb.flux-system - artifact up-to-date with remote revision: '0.14.3'
2024-02-28T08:39:51.656Z info HelmRelease/metallb.flux-system - HelmChart/flux-system/flux-system-metallb with SourceRef 'HelmRepository/flux-system/metallb' is in-sync
2024-02-28T08:39:51.664Z info HelmRelease/metallb.flux-system - release not installed: no release in storage for object
2024-02-28T08:39:51.683Z info HelmRelease/metallb.flux-system - running 'install' action with timeout of 5m0s
2024-02-28T08:40:04.712Z info Kustomization/infra-controllers.flux-system - server-side apply for cluster definitions completed
2024-02-28T08:40:04.735Z info Kustomization/infra-controllers.flux-system - server-side apply completed
2024-02-28T08:40:04.748Z info Kustomization/infra-controllers.flux-system - Reconciliation finished in 232.203054ms, next run in 1m0s
2024-02-28T08:40:06.271Z info Kustomization/homelab-sops.flux-system - server-side apply completed
2024-02-28T08:40:06.355Z info Kustomization/homelab-sops.flux-system - Reconciliation finished in 314.22538ms, next run in 1m0s
2024-02-28T08:40:07.285Z info GitRepository/homelab-sops.flux-system - no changes since last reconcilation: observed revision 'main@sha1:202e03894e7025e5702eaeb47ffaab903d66ad9d'
2024-02-28T08:40:17.020Z info HelmRelease/metallb.flux-system - release in-sync with desired state
2024-02-28T08:40:47.788Z info GitRepository/flux-system.flux-system - garbage collected 1 artifacts
2024-02-28T08:40:47.826Z info GitRepository/flux-system.flux-system - no changes since last reconcilation: observed revision 'main@sha1:340bad8d073bceb13b92aec745b866e53b38a03f'
flux get
$ flux get all
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:340bad8d      False           True    stored artifact for revision 'main@sha1:340bad8d'
gitrepository/homelab-sops      main@sha1:202e0389      False           True    stored artifact for revision 'main@sha1:202e0389'

NAME                    REVISION        SUSPENDED       READY   MESSAGE
helmrepository/metallb  sha256:a13247d1 False           True    stored artifact: revision 'sha256:a13247d1'

NAME                            REVISION        SUSPENDED       READY   MESSAGE
helmchart/flux-system-metallb   0.14.3          False           True    pulled 'metallb' chart with version '0.14.3'

NAME                    REVISION        SUSPENDED       READY   MESSAGE
helmrelease/metallb     0.14.3          False           True    Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]

NAME                            REVISION                SUSPENDED       READY   MESSAGE
kustomization/flux-system       main@sha1:340bad8d      False           True    Applied revision: main@sha1:340bad8d
kustomization/homelab-sops      main@sha1:202e0389      False           True    Applied revision: main@sha1:202e0389
kustomization/infra-controllers main@sha1:340bad8d      False           True    Applied revision: main@sha1:340bad8d
kubectl get
$ kubectl get ns
NAME               STATUS   AGE
calico-apiserver   Active   4d23h
calico-system      Active   4d23h
default            Active   4d23h
flux-system        Active   3d7h
kube-node-lease    Active   4d23h
kube-public        Active   4d23h
kube-system        Active   4d23h
metallb            Active   2m28s
tigera-operator    Active   4d23h

$ kubectl get all -n metallb
NAME                                             READY   STATUS    RESTARTS   AGE
pod/metallb-metallb-controller-79cf4b46c-dhpmq   1/1     Running   0          2m33s
pod/metallb-metallb-speaker-4tz7h                1/1     Running   0          2m33s
pod/metallb-metallb-speaker-8fvjf                1/1     Running   0          2m33s
pod/metallb-metallb-speaker-b5qz8                1/1     Running   0          2m33s
pod/metallb-metallb-speaker-f9s2j                1/1     Running   0          2m33s
pod/metallb-metallb-speaker-jbkzk                1/1     Running   0          2m33s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/metallb-webhook-service   ClusterIP   10.109.130.58   <none>        443/TCP   2m33s

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/metallb-metallb-speaker   5         5         5       5            5           kubernetes.io/os=linux   2m33s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/metallb-metallb-controller   1/1     1            1           2m33s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/metallb-metallb-controller-79cf4b46c   1         1         1       2m33s
metallb kinds
$ kubectl api-resources | grep metallb
bfdprofiles                                                                       metallb.io/v1beta1                       true         BFDProfile
bgpadvertisements                                                                 metallb.io/v1beta1                       true         BGPAdvertisement
bgppeers                                                                          metallb.io/v1beta2                       true         BGPPeer
communities                                                                       metallb.io/v1beta1                       true         Community
ipaddresspools                                                                    metallb.io/v1beta1                       true         IPAddressPool
l2advertisements                                                                  metallb.io/v1beta1                       true         L2Advertisement
homelab directory
.
 |-clusters
 | |-hyper-v
 | | |-infrastructure.yaml
 | | |-flux-system
 | | | |-kustomization.yaml
 | | | |-gotk-sync.yaml
 | | | |-gotk-components.yaml
 |-infrastructure
 | |-configs
 | |-controllers
 | | |-kustomization.yaml     # modified to newly include metallb.yaml
 | | |-metallb.yaml           # manifest with namespace, helmrepo, and hr (helmrelease) generated by metallb.sh
 | | |-metallb.sh             # script to run flux create commands, and simple cat > metallb.yaml<<EOF for the namespace manifest
 | | |-default-values         # not necessary. directory to store default values.yaml files for different services at the time of installation
 | | | |-metallb-values.yaml  # default metallb values.yaml file
 | | |-metallb-values.yaml    # modified metallb values.yaml file used by flux create helmrelease command inside metallb.sh
 | | |-sops.yaml
 |-.git

metallb configuration

https://metallb.universe.tf/configuration/#layer-2-configuration

I use metallb to L2-advertise IP addresses of the services. To do this I am going to prepare IPAddressPool and L2Advertisement manifests in .infrastructure/configs directory.

./infrastructure/configs/metallb-config.yaml
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan-addr-pool
  namespace: metallb
spec:
  addresses:
    - 192.168.1.221-192.168.1.230
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv-addr-pool
  namespace: metallb

And the kustomization.

./infrastructure/configs/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - metallb-config.yaml

Following the official flux guide (and this repository), I will modify ./clusters/hyper-v/infrastructure.yaml flux kustomization to include ./infrastructure/configs and set dependency to ./infrastructure/controllers. I have also added .spec.wait: true on infra-controllers.

./clusters/hyper-v/infrastructure.yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-controllers
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./infrastructure/controllers
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-configs
  namespace: flux-system
spec:
  dependsOn:
    - name: infra-controllers
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/configs
  prune: true

Here is the log.

$ flux get ks
NAME                    REVISION                SUSPENDED       READY   MESSAGE
flux-system             main@sha1:46462174      False           True    Applied revision: main@sha1:46462174
homelab-sops            main@sha1:202e0389      False           True    Applied revision: main@sha1:202e0389
infra-configs           main@sha1:46462174      False           True    Applied revision: main@sha1:46462174
infra-controllers       main@sha1:46462174      False           True    Applied revision: main@sha1:46462174

$ kubectl get ipaddresspools,l2advertisements -n metallb
NAME                                     AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
ipaddresspool.metallb.io/lan-addr-pool   true          false             ["192.168.1.221-192.168.1.230"]

NAME                                         IPADDRESSPOOLS   IPADDRESSPOOL SELECTORS   INTERFACES
l2advertisement.metallb.io/l2adv-addr-pool

metallb demo

https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports

I am going to use sample manifest available on kubernetes official document to demonstrate metallb functionality.

So here is the manifest without metallb. One thing I added is the namespace. I'll just place it at ./clusters/hyper-v since it's temporary thing.

./clusters/hyper-v/demo-svc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
  labels:
    app.kubernetes.io/name: proxy
spec:
  containers:
    - name: nginx
      image: nginx:stable
      ports:
        - containerPort: 80
          name: http-web-svc

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  selector:
    app.kubernetes.io/name: proxy
  ports:
    - name: name-of-service-port
      protocol: TCP
      port: 80
      targetPort: http-web-svc

Here's what's created. You cannot access 10.106.254.8:80 from LAN, 192.168.1.0/24.

$ kubectl get all
NAME        READY   STATUS    RESTARTS   AGE
pod/nginx   1/1     Running   0          82s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP   5d
service/nginx-service   ClusterIP   10.106.254.8   <none>        80/TCP    82s

To use metallb to expose service on LAN, just choose LoadBalancer type for the service, and metallb will take care of the rest. Here is the modified manifest. Line 32 is all you need to add. The metallb annotation in line 22-23 is something you can use to let metallb use specific IP address from its IP address pool.

./clusters/hyper-v/demo-svc.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
  labels:
    app.kubernetes.io/name: proxy
spec:
  containers:
    - name: nginx
      image: nginx:stable
      ports:
        - containerPort: 80
          name: http-web-svc

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
  annotations:
    metallb.universe.tf/loadBalancerIPs: 192.168.1.229
spec:
  selector:
    app.kubernetes.io/name: proxy
  ports:
    - name: name-of-service-port
      protocol: TCP
      port: 80
      targetPort: http-web-svc
  type: LoadBalancer

Now the demo service "nginx-service" has IP address of 192.168.1.229, and you can access it.

$ kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes      ClusterIP      10.96.0.1       <none>          443/TCP        5d
nginx-service   LoadBalancer   10.96.236.229   192.168.1.229   80:31266/TCP   2m46s

$ curl -I 192.168.1.229
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Wed, 28 Feb 2024 09:53:03 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 11 Apr 2023 01:45:34 GMT
Connection: keep-alive
ETag: "6434bbbe-267"
Accept-Ranges: bytes

homelab gitops repository

Here is the homelab repository directory. The metallb demo file is not included.

homelab repo directory structure
.
 |-clusters
 | |-hyper-v
 | | |-infrastructure.yaml    # added infra-configs flux kustomization with dependency to infra-controllers
 | | |-flux-system
 | | | |-kustomization.yaml
 | | | |-gotk-sync.yaml
 | | | |-gotk-components.yaml
 |-infrastructure
 | |-configs
 | | |-kustomization.yaml     # k8s kustomization of infra-configs
 | | |-metallb-config.yaml    # metallb ipaddr pool and l2adv config
 | |-controllers
 | | |-kustomization.yaml
 | | |-metallb.yaml
 | | |-metallb.sh
 | | |-default-values
 | | | |-metallb-values.yaml
 | | |-metallb-values.yaml
 | | |-sops.yaml
 |-.git