Skip to content

building homelab cluster part 3

I will setup gateway to setup web access to services running on the cluster. Metallb setup in the previous part will assign and advertise LAN IP address to the gateway and that will allow access to the cluster from the outside.


Here is the flow image in the official document.

gateway request flow

Introduction of the gateway API is on the kubernetes documentation site, and the detailed documentation is on a separate site.


# at the homelab repo root

# create a new directory to store custom resource definitions
mkdir infrastructure/homelab/controllers/crds
cd infrastructure/homelab/controllers/crds

# download the gateway crd manifest and save as gateway-{version}.yaml
curl -L -o gateway-v1.0.0.yaml

And I have edited infra-controllers kustomization manifest like this.

kind: Kustomization
  # CRDs
  - crds/gateway-v1.0.0.yaml
  # infra-controllers
  - metallb.yaml

Wait for the reconcilation and here it is.

$ kubectl api-resources | grep gateway
gatewayclasses                    gc                                                 false        GatewayClass
gateways                          gtw                                                true         Gateway
httproutes                                                                           true         HTTPRoute
referencegrants                   refgrant                                      true         ReferenceGrant

gateway controller

There are many available gateway controller implementations, and NGINX Gateway Fabric is one of a few that's in GA status as of this time of writing.

NGINX Gateway Fabric

helm values file

There is a page on NGF installation with helm you can find from the link above. The NGF helm chart is on their OCI registry. As I did with metallb helm installation, I will first get a copy of the values file.

cd ./infrastructure/homelab/controllers/default-values

# check the version
helm show chart oci://

# get the values file
helm show values oci:// > ngf-values.yaml
cp ngf-values.yaml ../.

# optionally, view chart, readme, and etc.
# helm show chart oci://
# helm show readme oci://
# helm show all oci://

# or even pull the chart locally and view however you like
# helm pull oci:// --untar && cd nginx-gateway-fabric

Before I prepare my values file and install NGF, I know the default NGF nginx image has default configuration which accept only a small request body size. It is not currently possible to edit nginx config for NGF like how you modify nginx.conf file, I am instead going to prepare a custom image.

custom image

There is an instruction page in the official NGF documentation website.

I have GitLab and Harbor running outside kubernetes cluster, so I create a GitLab project with pipeline to build my custom NGF nginx image and push it to Harbor.

I might add a link to the page or repo here. For now, let me just proceed with my custom image created.

Here is the diff of the values file compared with the original one.

I have changed the image registry to my local registry proxy for both controller and nginx server, and as for nginx server image I point to my custom image. And the service gets created with LoadBalancer type by default so metallb will assign LAN IP address from its pool, and I specified the IP address to use, here.

I have also added prestop and extended termination grace period as per the tip in the official document to accomplish zero-downtime upgrade.

$ diff ngf-values.yaml default-values/ngf-values.yaml
<     repository:
>     repository:
<   lifecycle:
<     preStop:
<       exec:
<         command:
<           - /usr/bin/gateway
<           - sleep
<           - --duration=40s
>   lifecycle: {}
<     repository:
>     repository:
<   lifecycle:
<     preStop:
<       exec:
<         command:
<           - /bin/sleep
<           - "40"
>   lifecycle: {}
< terminationGracePeriodSeconds: 50
> terminationGracePeriodSeconds: 30
<   annotations:
>   annotations: {}

I then re-use to prepare a script to generate manifest for NGF.

# add flux helmrepo to the manifest
flux create source helm ngf \
        --url=oci:// \
        --interval=1h0m0s \
        --export >>ngf.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease ngf \
        --interval=10m \
        --target-namespace=ngf \
        --source=HelmRepository/ngf \
        --chart=nginx-gateway-fabric \
        --chart-version=1.1.0 \
        --values=ngf-values.yaml \
        --export >>ngf.yaml

I run the script to generate ngf.yaml, and edit infra-controllers kustomization to include that.

diff --git a/infrastructure/controllers/kustomization.yaml b/infrastructure/controllers/kustomization.yaml
index 35383f2..5b7303d 100644
--- a/infrastructure/controllers/kustomization.yaml
+++ b/infrastructure/controllers/kustomization.yaml
@@ -6,3 +6,4 @@ resources:
   # infra-controllers
   - metallb.yaml
+  - ngf.yaml

And let's not forget, I am going to add the namespace too.

apiVersion: v1
kind: Namespace
  name: ngf
    service: nginx-gateway-fabric
    type: infrastructure

Here is the result on flux.

$ flux get source all
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:c54e5e7b      False           True    stored artifact for revision 'main@sha1:c54e5e7b'
gitrepository/homelab-sops      main@sha1:202e0389      False           True    stored artifact for revision 'main@sha1:202e0389'

NAME                    REVISION        SUSPENDED       READY   MESSAGE
helmrepository/metallb  sha256:a13247d1 False           True    stored artifact: revision 'sha256:a13247d1'
helmrepository/ngf                      False           True    Helm repository is Ready

NAME                            REVISION        SUSPENDED       READY   MESSAGE
helmchart/flux-system-metallb   0.14.3          False           True    pulled 'metallb' chart with version '0.14.3'
helmchart/flux-system-ngf       1.1.0           False           True    pulled 'nginx-gateway-fabric' chart with version '1.1.0'

$ flux get hr
metallb 0.14.3          False           True    Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]
ngf     1.1.0           False           True    Helm install succeeded for release ngf/ngf-ngf.v1 with chart [email protected]

And on kubectl.

$ kubectl get all -n ngf
NAME                                                READY   STATUS    RESTARTS   AGE
pod/ngf-ngf-nginx-gateway-fabric-57b65fb766-95wnb   2/2     Running   0          19m

NAME                                   TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
service/ngf-ngf-nginx-gateway-fabric   LoadBalancer   80:32323/TCP,443:30210/TCP   19m

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ngf-ngf-nginx-gateway-fabric   1/1     1            1           19m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ngf-ngf-nginx-gateway-fabric-57b65fb766   1         1         1       19m

create gateway

I will go ahead and create gateway which uses ngf. I will start with plain http on port 80.

  • listens on tcp port 80
  • available for namespace with label "gateway-available=yes"
kind: Gateway
  name: gateway
  namespace: gateway
  gatewayClassName: nginx
    - name: http
      port: 80
      protocol: HTTP
          from: Selector
              gateway-available: yes

I am going to skip the namespace manifest for gateway, but once both namespace and the gateway manifest is set, there will be a gateway resource.

$ kubectl get gtw -n gateway
gateway   nginx   True         103s

directory structure of the repository so far

homelab repo files changed
 | |-homelab
 | | |-infrastructure.yaml
 | | |-network-addon
 | | |-flux-system
 | | |-sops.yaml
 | | |-nodes
 | | |-namespace
 | | | |-metallb.yaml
 | | | |-gateway.yaml           # gateway namespace
 | | | |-ngf.yaml               # ngf namespace
 | |-homelab
 | | |-configs
 | | | |-kustomization.yaml
 | | | |-gateway.yaml           # Gateway resource using NGF listening on 80 tcp
 | | |-controllers
 | | | |-kustomization.yaml     # added gateway crds and ngf
 | | | |-default-values
 | | | | |-ngf-values.yaml      # keeping the default values just for reference
 | | | |-ngf-values.yaml        # customized ngf values file
 | | | |-crds
 | | | | |-gateway-v1.0.0.yaml  # installs k8s gateway api/crds
 | | | |-ngf.yaml               # ngf manifest, helmrepo and helmrelease
 | | | |                 # script to generate ngf.yaml