Skip to content




building home lab part 3

Here is the copy of todo list with update from part 2 which covered helm and metallb, and next up is the gateway to enable web access to the services on kubernetes cluster. One of the first to setup access to is going to be minio s3 which will be in the next part. This part will cover the procedure to install gateway.

  • [x] prepare nodes (VMs on hyper-v in this series)
  • [x] setup kubernetes cluster
  • [x] bootstrap flux gitops
  • [x] setup sops for secret encryption
  • [x] install helm
  • [x] metallb to l2 advertise svc on LAN
  • [ ] gateway as well as nginx gateway fabric to setup https gateway
  • [ ] directpv to setup storage class, to serve persistent volume
  • [ ] minio as s3 storage
  • [ ] gitlab runner to execute CI/CD jobs on my gitlab
  • [ ] kube-prometheus for monitoring
  • [ ] loki for logging
  • [ ] cert manager to manage tls certificate
  • [ ] weave gitops dashboard
  • [ ] kube-dashboard
  • [ ] postgresql
  • [ ] mongodb
  • [ ] bytebase
  • [ ] my private apps

gateway

https://kubernetes.io/docs/concepts/services-networking/gateway/

Here is the flow image in the official document.

gateway request flow

Introduction of the gateway API is on the kubernetes documentation site, and the detailed documentation is on a separate site.

https://gateway-api.sigs.k8s.io/

installation

https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api

# at the homelab repo root

# create a new directory to store custom resource definitions
mkdir infrastructure/controllers/crds
cd infrastructure/controllers/crds

# download the gateway crd manifest and save as gateway-{version}.yaml
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml -o gateway-v1.0.0.yaml

And I have edited infra-controllers kustomization manifest like this.

./infrastructure/controllers/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  # CRDs
  - crds/gateway-v1.0.0.yaml
  # infra-controllers
  - sops.yaml
  - metallb.yaml

Wait for the reconcilation and here it is.

$ kubectl api-resources | grep gateway
gatewayclasses                    gc                                              gateway.networking.k8s.io/v1             false        GatewayClass
gateways                          gtw                                             gateway.networking.k8s.io/v1             true         Gateway
httproutes                                                                        gateway.networking.k8s.io/v1             true         HTTPRoute
referencegrants                   refgrant                                        gateway.networking.k8s.io/v1beta1        true         ReferenceGrant

gateway controller

https://gateway-api.sigs.k8s.io/implementations/

There are many available gateway controller implementations, and NGINX Gateway Fabric is one of a few that's in GA status as of this time of writing. My main homelab cluster is running on NGF, and I am going to install the same for my hyper-v cluster.

NGINX Gateway Fabric

https://github.com/nginxinc/nginx-gateway-fabric

https://docs.nginx.com/nginx-gateway-fabric

helm values file

There is a page on NGF installation with helm. The NGF helm chart is on their OCI registry. As I did with metallb helm installation, I will first get a copy of the values file.

cd ./infrastructure/controllers/default-values
helm show values oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric > ngf-values.yaml
cp ngf-values.yaml ../.

# optionally, view chart, readme, and etc.
# helm show chart oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show readme oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show all oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric

# or even pull the chart locally and view however you like
# helm pull oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --untar && cd nginx-gateway-fabric

Before I prepare my values file and install NGF, I know the default NGF nginx image has default configuration which accept only a small request body size. It is not currently possible to edit nginx config for NGF like how you modify nginx.conf file, I am instead going to prepare a custom image.

custom image

There is an instruction page in the official NGF documentation website.

https://docs.nginx.com/nginx-gateway-fabric/installation/building-the-images/

I have GitLab and Harbor running outside kubernetes cluster, so I create a GitLab project with pipeline to build my custom NGF nginx image and push it to Harbor.

I might add a link to the page or repo here. For now, let me just proceed with my custom image created.

Here is the diff of the values file compared with the original one.

I have changed the image registry to my local registry proxy for both controller and nginx server, and as for nginx server image I point to my custom image. And the service gets created with LoadBalancer type by default so metallb will assign LAN IP address from its pool, and I specified the IP address to use, 192.168.1.79 here.

I have also added prestop and extended termination grace period as per the tip in the official document to accomplish zero-downtime upgrade.

$ diff ngf-values.yaml default-values/ngf-values.yaml
40c40
<     repository: registry.blink-1x52.net/cache-ghcr/nginxinc/nginx-gateway-fabric
---
>     repository: ghcr.io/nginxinc/nginx-gateway-fabric
49,55c49
<   lifecycle:
<     preStop:
<       exec:
<         command:
<           - /usr/bin/gateway
<           - sleep
<           - --duration=40s
---
>   lifecycle: {}
63c57
<     repository: registry.blink-1x52.net/gitlab/images/ngf-nginx
---
>     repository: ghcr.io/nginxinc/nginx-gateway-fabric/nginx
68,73c62
<   lifecycle:
<     preStop:
<       exec:
<         command:
<           - /bin/sleep
<           - "40"
---
>   lifecycle: {}
79c68
< terminationGracePeriodSeconds: 50
---
> terminationGracePeriodSeconds: 30
101,102c90
<   annotations:
<     metallb.universe.tf/loadBalancerIPs: 192.168.1.79
---
>   annotations: {}

This 192.168.1.79 IP address was not included in the pool, so I am also modifying metalb configuration like this.

./infrastructure/configs/metallb-config.yaml
diff --git a/infrastructure/configs/metallb-config.yaml b/infrastructure/configs/metallb-config.yaml
index a622bd6..c7def8e 100644
--- a/infrastructure/configs/metallb-config.yaml
+++ b/infrastructure/configs/metallb-config.yaml
@@ -7,6 +7,7 @@ metadata:
 spec:
   addresses:
     - 192.168.1.221-192.168.1.230
+    - 192.168.1.79/32
 ---
 apiVersion: metallb.io/v1beta1
 kind: L2Advertisement

I am going to commit and push metallb configuration change first, because I have set dependency between controller and config kustomizations where if I push everything at once, infrastructure and this new NGF will get processed first, but will find 192.168.1.79 IP address missing from metallb IP pool to allocate.

metallb ipaddresspools
$ kubectl get ipaddresspools -n metallb
NAME            AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
lan-addr-pool   true          false             ["192.168.1.221-192.168.1.230","192.168.1.79/32"]

I then re-use metallb.sh to prepare a script to generate manifest for NGF.

ngf.sh
#!/bin/bash

# create a namespace manifest
cat >ngf.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: ngf
  labels:
    service: nginx-gateway-fabric
    type: infrastructure
EOF

# add flux helmrepo to the manifest
flux create source helm ngf \
        --url=oci://ghcr.io/nginxinc/charts \
        --interval=1h0m0s \
        --export >>ngf.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease ngf \
        --interval=10m \
        --target-namespace=ngf \
        --source=HelmRepository/ngf \
        --chart=nginx-gateway-fabric \
        --chart-version=1.1.0 \
        --values=ngf-values.yaml \
        --export >>ngf.yaml

I run the ngf.sh script to generate ngf.yaml, and edit infra-controllers kustomization to include that.

diff --git a/infrastructure/controllers/kustomization.yaml b/infrastructure/controllers/kustomization.yaml
index 35383f2..5b7303d 100644
--- a/infrastructure/controllers/kustomization.yaml
+++ b/infrastructure/controllers/kustomization.yaml
@@ -6,3 +6,4 @@ resources:
   # infra-controllers
   - sops.yaml
   - metallb.yaml
+  - ngf.yaml

Here is the result on flux.

$ flux get source all
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:c54e5e7b      False           True    stored artifact for revision 'main@sha1:c54e5e7b'
gitrepository/homelab-sops      main@sha1:202e0389      False           True    stored artifact for revision 'main@sha1:202e0389'

NAME                    REVISION        SUSPENDED       READY   MESSAGE
helmrepository/metallb  sha256:a13247d1 False           True    stored artifact: revision 'sha256:a13247d1'
helmrepository/ngf                      False           True    Helm repository is Ready

NAME                            REVISION        SUSPENDED       READY   MESSAGE
helmchart/flux-system-metallb   0.14.3          False           True    pulled 'metallb' chart with version '0.14.3'
helmchart/flux-system-ngf       1.1.0           False           True    pulled 'nginx-gateway-fabric' chart with version '1.1.0'

$ flux get hr
NAME    REVISION        SUSPENDED       READY   MESSAGE
metallb 0.14.3          False           True    Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]
ngf     1.1.0           False           True    Helm install succeeded for release ngf/ngf-ngf.v1 with chart [email protected]

And on kubectl.

$ kubectl get all -n ngf
NAME                                                READY   STATUS    RESTARTS   AGE
pod/ngf-ngf-nginx-gateway-fabric-57b65fb766-95wnb   2/2     Running   0          19m

NAME                                   TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
service/ngf-ngf-nginx-gateway-fabric   LoadBalancer   10.97.228.153   192.168.1.79   80:32323/TCP,443:30210/TCP   19m

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ngf-ngf-nginx-gateway-fabric   1/1     1            1           19m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ngf-ngf-nginx-gateway-fabric-57b65fb766   1         1         1       19m

create gateway

I will go ahead and create gateway which uses ngf. I will start with plain http on port 80.

  • listens on tcp port 80
  • available for namespace with label "gateway-available=yes"
./infrastructure/configs/gateway.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: gateway
  labels:
    service: gateway
    type: infrastructure
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes

The gateway will be created using ngf.

$ kubectl get gtw -n gateway
NAME      CLASS   ADDRESS        PROGRAMMED   AGE
gateway   nginx   192.168.1.79   True         103s

directory structure of the repository so far

.
 |-clusters
 | |-hyper-v
 | | |-infrastructure.yaml
 | | |-flux-system
 | | | |-kustomization.yaml
 | | | |-gotk-sync.yaml
 | | | |-gotk-components.yaml
 |-infrastructure
 | |-configs
 | | |-kustomization.yaml
 | | |-metallb-config.yaml     # add 192.168.1.79/32 to the ipaddresspools
 | | |-gateway.yaml            # gateway using ngf listening on 80/tcp
 | |-controllers
 | | |-kustomization.yaml      # include ngf.yaml
 | | |-metallb.yaml
 | | |-metallb.sh
 | | |-default-values
 | | | |-ngf-values.yaml       # default values file from nginx-gateway-fabric v1.1.0 helm chart
 | | | |-metallb-values.yaml
 | | |-ngf-values.yaml         # ngf values file modified to use custom image, and other changes made around lifecycle and term grace period
 | | |-metallb-values.yaml
 | | |-crds
 | | | |-gateway-v1.0.0.yaml   # kubernetes gateway api v1.0.0
 | | |-sops.yaml
 | | |-ngf.yaml                # ngf manifest with namespace, helmrepo, and helmrelease
 | | |-ngf.sh                  # script to generate ngf.yaml using ngf-values.yaml
 |-.git