building home lab part 3
Table of Content
building home lab part 3¶
Here is the copy of todo list with update from part 2 which covered helm and metallb, and next up is the gateway to enable web access to the services on kubernetes cluster. One of the first to setup access to is going to be minio s3 which will be in the next part. This part will cover the procedure to install gateway.
- [x] prepare nodes (VMs on hyper-v in this series)
- [x] setup kubernetes cluster
- [x] bootstrap flux gitops
- [x] setup sops for secret encryption
- [x] install helm
- [x] metallb to l2 advertise svc on LAN
- [ ] gateway as well as nginx gateway fabric to setup https gateway
- [ ] directpv to setup storage class, to serve persistent volume
- [ ] minio as s3 storage
- [ ] gitlab runner to execute CI/CD jobs on my gitlab
- [ ] kube-prometheus for monitoring
- [ ] loki for logging
- [ ] cert manager to manage tls certificate
- [ ] weave gitops dashboard
- [ ] kube-dashboard
- [ ] postgresql
- [ ] mongodb
- [ ] bytebase
- [ ] my private apps
gateway¶
https://kubernetes.io/docs/concepts/services-networking/gateway/
Here is the flow image in the official document.
Introduction of the gateway API is on the kubernetes documentation site, and the detailed documentation is on a separate site.
https://gateway-api.sigs.k8s.io/
installation¶
https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api
# at the homelab repo root
# create a new directory to store custom resource definitions
mkdir infrastructure/controllers/crds
cd infrastructure/controllers/crds
# download the gateway crd manifest and save as gateway-{version}.yaml
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml -o gateway-v1.0.0.yaml
And I have edited infra-controllers kustomization manifest like this.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# CRDs
- crds/gateway-v1.0.0.yaml
# infra-controllers
- sops.yaml
- metallb.yaml
Wait for the reconcilation and here it is.
$ kubectl api-resources | grep gateway
gatewayclasses gc gateway.networking.k8s.io/v1 false GatewayClass
gateways gtw gateway.networking.k8s.io/v1 true Gateway
httproutes gateway.networking.k8s.io/v1 true HTTPRoute
referencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant
gateway controller¶
https://gateway-api.sigs.k8s.io/implementations/
There are many available gateway controller implementations, and NGINX Gateway Fabric is one of a few that's in GA status as of this time of writing. My main homelab cluster is running on NGF, and I am going to install the same for my hyper-v cluster.
NGINX Gateway Fabric¶
https://github.com/nginxinc/nginx-gateway-fabric
https://docs.nginx.com/nginx-gateway-fabric
helm values file¶
There is a page on NGF installation with helm. The NGF helm chart is on their OCI registry. As I did with metallb helm installation, I will first get a copy of the values file.
cd ./infrastructure/controllers/default-values
helm show values oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric > ngf-values.yaml
cp ngf-values.yaml ../.
# optionally, view chart, readme, and etc.
# helm show chart oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show readme oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# helm show all oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric
# or even pull the chart locally and view however you like
# helm pull oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --untar && cd nginx-gateway-fabric
Before I prepare my values file and install NGF, I know the default NGF nginx image has default configuration which accept only a small request body size. It is not currently possible to edit nginx config for NGF like how you modify nginx.conf file, I am instead going to prepare a custom image.
custom image¶
There is an instruction page in the official NGF documentation website.
https://docs.nginx.com/nginx-gateway-fabric/installation/building-the-images/
I have GitLab and Harbor running outside kubernetes cluster, so I create a GitLab project with pipeline to build my custom NGF nginx image and push it to Harbor.
I might add a link to the page or repo here. For now, let me just proceed with my custom image created.
Here is the diff of the values file compared with the original one.
I have changed the image registry to my local registry proxy for both controller and nginx server, and as for nginx server image I point to my custom image. And the service gets created with LoadBalancer type by default so metallb will assign LAN IP address from its pool, and I specified the IP address to use, 192.168.1.79 here.
I have also added prestop and extended termination grace period as per the tip in the official document to accomplish zero-downtime upgrade.
$ diff ngf-values.yaml default-values/ngf-values.yaml
40c40
< repository: registry.blink-1x52.net/cache-ghcr/nginxinc/nginx-gateway-fabric
---
> repository: ghcr.io/nginxinc/nginx-gateway-fabric
49,55c49
< lifecycle:
< preStop:
< exec:
< command:
< - /usr/bin/gateway
< - sleep
< - --duration=40s
---
> lifecycle: {}
63c57
< repository: registry.blink-1x52.net/gitlab/images/ngf-nginx
---
> repository: ghcr.io/nginxinc/nginx-gateway-fabric/nginx
68,73c62
< lifecycle:
< preStop:
< exec:
< command:
< - /bin/sleep
< - "40"
---
> lifecycle: {}
79c68
< terminationGracePeriodSeconds: 50
---
> terminationGracePeriodSeconds: 30
101,102c90
< annotations:
< metallb.universe.tf/loadBalancerIPs: 192.168.1.79
---
> annotations: {}
This 192.168.1.79 IP address was not included in the pool, so I am also modifying metalb configuration like this.
diff --git a/infrastructure/configs/metallb-config.yaml b/infrastructure/configs/metallb-config.yaml
index a622bd6..c7def8e 100644
--- a/infrastructure/configs/metallb-config.yaml
+++ b/infrastructure/configs/metallb-config.yaml
@@ -7,6 +7,7 @@ metadata:
spec:
addresses:
- 192.168.1.221-192.168.1.230
+ - 192.168.1.79/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
I am going to commit and push metallb configuration change first, because I have set dependency between controller and config kustomizations where if I push everything at once, infrastructure and this new NGF will get processed first, but will find 192.168.1.79 IP address missing from metallb IP pool to allocate.
$ kubectl get ipaddresspools -n metallb
NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES
lan-addr-pool true false ["192.168.1.221-192.168.1.230","192.168.1.79/32"]
I then re-use metallb.sh to prepare a script to generate manifest for NGF.
#!/bin/bash
# create a namespace manifest
cat >ngf.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: ngf
labels:
service: nginx-gateway-fabric
type: infrastructure
EOF
# add flux helmrepo to the manifest
flux create source helm ngf \
--url=oci://ghcr.io/nginxinc/charts \
--interval=1h0m0s \
--export >>ngf.yaml
# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease ngf \
--interval=10m \
--target-namespace=ngf \
--source=HelmRepository/ngf \
--chart=nginx-gateway-fabric \
--chart-version=1.1.0 \
--values=ngf-values.yaml \
--export >>ngf.yaml
I run the ngf.sh script to generate ngf.yaml, and edit infra-controllers kustomization to include that.
diff --git a/infrastructure/controllers/kustomization.yaml b/infrastructure/controllers/kustomization.yaml
index 35383f2..5b7303d 100644
--- a/infrastructure/controllers/kustomization.yaml
+++ b/infrastructure/controllers/kustomization.yaml
@@ -6,3 +6,4 @@ resources:
# infra-controllers
- sops.yaml
- metallb.yaml
+ - ngf.yaml
Here is the result on flux.
$ flux get source all
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system main@sha1:c54e5e7b False True stored artifact for revision 'main@sha1:c54e5e7b'
gitrepository/homelab-sops main@sha1:202e0389 False True stored artifact for revision 'main@sha1:202e0389'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/metallb sha256:a13247d1 False True stored artifact: revision 'sha256:a13247d1'
helmrepository/ngf False True Helm repository is Ready
NAME REVISION SUSPENDED READY MESSAGE
helmchart/flux-system-metallb 0.14.3 False True pulled 'metallb' chart with version '0.14.3'
helmchart/flux-system-ngf 1.1.0 False True pulled 'nginx-gateway-fabric' chart with version '1.1.0'
$ flux get hr
NAME REVISION SUSPENDED READY MESSAGE
metallb 0.14.3 False True Helm install succeeded for release metallb/metallb-metallb.v1 with chart [email protected]
ngf 1.1.0 False True Helm install succeeded for release ngf/ngf-ngf.v1 with chart [email protected]
And on kubectl.
$ kubectl get all -n ngf
NAME READY STATUS RESTARTS AGE
pod/ngf-ngf-nginx-gateway-fabric-57b65fb766-95wnb 2/2 Running 0 19m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ngf-ngf-nginx-gateway-fabric LoadBalancer 10.97.228.153 192.168.1.79 80:32323/TCP,443:30210/TCP 19m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ngf-ngf-nginx-gateway-fabric 1/1 1 1 19m
NAME DESIRED CURRENT READY AGE
replicaset.apps/ngf-ngf-nginx-gateway-fabric-57b65fb766 1 1 1 19m
create gateway¶
I will go ahead and create gateway which uses ngf. I will start with plain http on port 80.
- listens on tcp port 80
- available for namespace with label "gateway-available=yes"
---
kind: Namespace
apiVersion: v1
metadata:
name: gateway
labels:
service: gateway
type: infrastructure
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: gateway
namespace: gateway
spec:
gatewayClassName: nginx
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
gateway-available: yes
The gateway will be created using ngf.
directory structure of the repository so far¶
.
|-clusters
| |-hyper-v
| | |-infrastructure.yaml
| | |-flux-system
| | | |-kustomization.yaml
| | | |-gotk-sync.yaml
| | | |-gotk-components.yaml
|-infrastructure
| |-configs
| | |-kustomization.yaml
| | |-metallb-config.yaml # add 192.168.1.79/32 to the ipaddresspools
| | |-gateway.yaml # gateway using ngf listening on 80/tcp
| |-controllers
| | |-kustomization.yaml # include ngf.yaml
| | |-metallb.yaml
| | |-metallb.sh
| | |-default-values
| | | |-ngf-values.yaml # default values file from nginx-gateway-fabric v1.1.0 helm chart
| | | |-metallb-values.yaml
| | |-ngf-values.yaml # ngf values file modified to use custom image, and other changes made around lifecycle and term grace period
| | |-metallb-values.yaml
| | |-crds
| | | |-gateway-v1.0.0.yaml # kubernetes gateway api v1.0.0
| | |-sops.yaml
| | |-ngf.yaml # ngf manifest with namespace, helmrepo, and helmrelease
| | |-ngf.sh # script to generate ngf.yaml using ngf-values.yaml
|-.git