building homelab cluster part 3
Table of Content
building homelab cluster part 3¶
I will setup gateway to setup web access to services running on the cluster. Metallb setup in the previous part will assign and advertise LAN IP address to the gateway and that will allow access to the cluster from the outside.
gateway¶
https://kubernetes.io/docs/concepts/services-networking/gateway/
Here is the flow image in the official document.
Introduction of the gateway API is on the kubernetes documentation site, and the detailed documentation is on a separate site.
https://gateway-api.sigs.k8s.io/
installation¶
https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api
# at the homelab repo root
# create a new directory to store custom resource definitions
mkdir infrastructure/homelab/controllers/crds
cd infrastructure/homelab/controllers/crds
# download the gateway crd manifest and save as gateway-{version}.yaml
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml -o gateway-v1.0.0.yaml
And I have edited infra-controllers kustomization manifest like this.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# CRDs
- crds/gateway-v1.0.0.yaml
# infra-controllers
- metallb.yaml
Wait for the reconcilation and here it is.
$ kubectl api-resources | grep gateway
gatewayclasses gc gateway.networking.k8s.io/v1 false GatewayClass
gateways gtw gateway.networking.k8s.io/v1 true Gateway
httproutes gateway.networking.k8s.io/v1 true HTTPRoute
referencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant
gateway controller¶
https://gateway-api.sigs.k8s.io/implementations/
There are many available gateway controller implementations, and NGINX Gateway Fabric is one of a few that's in GA status as of this time of writing.
NGINX Gateway Fabric¶
https://github.com/nginxinc/nginx-gateway-fabric
https://docs.nginx.com/nginx-gateway-fabric
helm values file¶
There is a page on NGF installation with helm you can find from the link above. The NGF helm chart is on their OCI registry. As I did with metallb helm installation, I will first get a copy of the values file.
cd ./infrastructure/homelab/controllers/default-values
# check the version
helm show chart oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# get the values file
helm show values oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric > ngf-values.yaml
cp ngf-values.yaml ../.
# optionally, view chart, readme, and etc.
# helm show chart oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show readme oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show all oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# or even pull the chart locally and view however you like
# helm pull oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --untar && cd nginx-gateway-fabric
Before I prepare my values file and install NGF, I know the default NGF nginx image has default configuration which accept only a small request body size. It is not currently possible to edit nginx config for NGF like how you modify nginx.conf file, I am instead going to prepare a custom image.
custom image¶
There is an instruction page in the official NGF documentation website.
https://docs.nginx.com/nginx-gateway-fabric/installation/building-the-images/
I have GitLab and Harbor running outside kubernetes cluster, so I create a GitLab project with pipeline to build my custom NGF nginx image and push it to Harbor.
I might add a link to the page or repo here. For now, let me just proceed with my custom image created.
Here is the diff of the values file compared with the original one.
I have changed the image registry to my local registry proxy for both controller and nginx server, and as for nginx server image I point to my custom image. And the service gets created with LoadBalancer type by default so metallb will assign LAN IP address from its pool, and I specified the IP address to use, 192.168.1.54 here.
I have also added prestop and extended termination grace period as per the tip in the official document to accomplish zero-downtime upgrade.
$ diff ngf-values.yaml default-values/ngf-values.yaml
40c40
< repository: registry.blink-1x52.net/cache-ghcr/nginxinc/nginx-gateway-fabric
---
> repository: ghcr.io/nginxinc/nginx-gateway-fabric
49,55c49
< lifecycle:
< preStop:
< exec:
< command:
< - /usr/bin/gateway
< - sleep
< - --duration=40s
---
> lifecycle: {}
63c57
< repository: registry.blink-1x52.net/gitlab/images/ngf-nginx
---
> repository: ghcr.io/nginxinc/nginx-gateway-fabric/nginx
68,73c62
< lifecycle:
< preStop:
< exec:
< command:
< - /bin/sleep
< - "40"
---
> lifecycle: {}
79c68
< terminationGracePeriodSeconds: 50
---
> terminationGracePeriodSeconds: 30
101,102c90
< annotations:
< metallb.universe.tf/loadBalancerIPs: 192.168.1.54
---
> annotations: {}
I then re-use metallb.sh to prepare a script to generate manifest for NGF.
#!/bin/bash
# add flux helmrepo to the manifest
flux create source helm ngf \
--url=oci://ghcr.io/nginxinc/charts \
--interval=1h0m0s \
--export >>ngf.yaml
# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease ngf \
--interval=10m \
--target-namespace=ngf \
--source=HelmRepository/ngf \
--chart=nginx-gateway-fabric \
--chart-version=1.1.0 \
--values=ngf-values.yaml \
--export >>ngf.yaml
I run the ngf.sh script to generate ngf.yaml, and edit infra-controllers kustomization to include that.
diff --git a/infrastructure/controllers/kustomization.yaml b/infrastructure/controllers/kustomization.yaml
index 35383f2..5b7303d 100644
--- a/infrastructure/controllers/kustomization.yaml
+++ b/infrastructure/controllers/kustomization.yaml
@@ -6,3 +6,4 @@ resources:
# infra-controllers
- metallb.yaml
+ - ngf.yaml
And let's not forget, I am going to add the namespace too.
---
apiVersion: v1
kind: Namespace
metadata:
name: ngf
labels:
service: nginx-gateway-fabric
type: infrastructure
Here is the result on flux.
$ flux get source all
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system main@sha1:c54e5e7b False True stored artifact for revision 'main@sha1:c54e5e7b'
gitrepository/homelab-sops main@sha1:202e0389 False True stored artifact for revision 'main@sha1:202e0389'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/metallb sha256:a13247d1 False True stored artifact: revision 'sha256:a13247d1'
helmrepository/ngf False True Helm repository is Ready
NAME REVISION SUSPENDED READY MESSAGE
helmchart/flux-system-metallb 0.14.3 False True pulled 'metallb' chart with version '0.14.3'
helmchart/flux-system-ngf 1.1.0 False True pulled 'nginx-gateway-fabric' chart with version '1.1.0'
$ flux get hr
NAME REVISION SUSPENDED READY MESSAGE
metallb 0.14.3 False True Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]
ngf 1.1.0 False True Helm install succeeded for release ngf/ngf-ngf.v1 with chart [email protected]
And on kubectl.
$ kubectl get all -n ngf
NAME READY STATUS RESTARTS AGE
pod/ngf-ngf-nginx-gateway-fabric-57b65fb766-95wnb 2/2 Running 0 19m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ngf-ngf-nginx-gateway-fabric LoadBalancer 10.97.228.153 192.168.1.54 80:32323/TCP,443:30210/TCP 19m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ngf-ngf-nginx-gateway-fabric 1/1 1 1 19m
NAME DESIRED CURRENT READY AGE
replicaset.apps/ngf-ngf-nginx-gateway-fabric-57b65fb766 1 1 1 19m
create gateway¶
I will go ahead and create gateway which uses ngf. I will start with plain http on port 80.
- listens on tcp port 80
- available for namespace with label "gateway-available=yes"
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: gateway
namespace: gateway
spec:
gatewayClassName: nginx
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
gateway-available: yes
I am going to skip the namespace manifest for gateway, but once both namespace and the gateway manifest is set, there will be a gateway resource.
directory structure of the repository so far¶
.
|-clusters
| |-homelab
| | |-infrastructure.yaml
| | |-network-addon
| | |-flux-system
| | |-sops.yaml
| | |-nodes
| | |-namespace
| | | |-metallb.yaml
| | | |-gateway.yaml # gateway namespace
| | | |-ngf.yaml # ngf namespace
|-infrastructure
| |-homelab
| | |-configs
| | | |-kustomization.yaml
| | | |-gateway.yaml # Gateway resource using NGF listening on 80 tcp
| | |-controllers
| | | |-kustomization.yaml # added gateway crds and ngf
| | | |-default-values
| | | | |-ngf-values.yaml # keeping the default values just for reference
| | | |-ngf-values.yaml # customized ngf values file
| | | |-crds
| | | | |-gateway-v1.0.0.yaml # installs k8s gateway api/crds
| | | |-ngf.yaml # ngf manifest, helmrepo and helmrelease
| | | |-ngf.sh # script to generate ngf.yaml
|-.git