building homelab cluster part 5

Table of Content

building homelab cluster part 5¶

Metallb enabled L2 advertisement of service resources. Gateway and NGINX Gateway Fabric enabled web access to the services from outside kubernetes cluster. The first web access was setup for Minio, but on plain http.

In this part, I will setup https access for Minio.

cert-manager¶

https://cert-manager.io/

cert-manager is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

https://github.com/cert-manager/cert-manager

installation¶

https://cert-manager.io/docs/installation/

https://cert-manager.io/docs/installation/helm/

# add helm repo
helm repo add jetstack https://charts.jetstack.io

# confirm the version
helm show chart jetstack/cert-manager

# get a copy of values file
cd {gitops repo}/infrastructure/homelab/controllers/default-values
helm show values jetstack/cert-manager > cert-manager-values.yaml
cp cert-manager-values.yaml ../.

# download crds to install
cd ../crds
curl -L https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.crds.yaml -o cert-manager-v1.14.3.yaml

And as done many times with previous installations in this series, I prepare a script to generate namespace, flux helmrepo, and flux helmrelease manifests. At first I did not think I need to modify any settings, so I am not including custom values file in the helmrelease manifest, but I will come back to this point later.

./infrastructure/homelab/controllers/cert-manager.sh

#!/bin/bash

# add flux helmrepo to the manifest
flux create source helm cert-manager \
    --url=https://charts.jetstack.io \
    --interval=1h0m0s \
    --export >cert-manager.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease cert-manager \
    --interval=10m \
    --target-namespace=cert-manager \
    --source=HelmRepository/cert-manager \
    --chart=cert-manager \
    --chart-version=1.14.3 \
    --export >>cert-manager.yaml

Commit/push everything, let flux reconcilate, and here is the result.

$ kubectl get all -n cert-manager
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/cert-manager-cert-manager-784cddc4df-8ftfr              1/1     Running   0          67s
pod/cert-manager-cert-manager-cainjector-58bb9bfffc-rzv2m   1/1     Running   0          67s
pod/cert-manager-cert-manager-webhook-588db4b47f-8fq56      1/1     Running   0          67s

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager-cert-manager           ClusterIP   10.99.93.138    <none>        9402/TCP   69s
service/cert-manager-cert-manager-webhook   ClusterIP   10.109.17.212   <none>        443/TCP    69s

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager-cert-manager              1/1     1            1           69s
deployment.apps/cert-manager-cert-manager-cainjector   1/1     1            1           69s
deployment.apps/cert-manager-cert-manager-webhook      1/1     1            1           69s

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-cert-manager-784cddc4df              1         1         1       68s
replicaset.apps/cert-manager-cert-manager-cainjector-58bb9bfffc   1         1         1       68s
replicaset.apps/cert-manager-cert-manager-webhook-588db4b47f      1         1         1       68s

$ kubectl api-resources | grep cert
challenges                                                                        acme.cert-manager.io/v1                  true         Challenge
orders                                                                            acme.cert-manager.io/v1                  true         Order
certificaterequests               cr,crs                                          cert-manager.io/v1                       true         CertificateRequest
certificates                      cert,certs                                      cert-manager.io/v1                       true         Certificate
clusterissuers                                                                    cert-manager.io/v1                       false        ClusterIssuer
issuers                                                                           cert-manager.io/v1                       true         Issuer
certificatesigningrequests        csr                                             certificates.k8s.io/v1                   false        CertificateSigningRequest

certificate¶

https://cert-manager.io/docs/usage/

There are different ways to obtain certificate.

https://cert-manager.io/docs/usage/gateway/

I can add a little modification to my gateway resource manifest to have cert-manager generate valid certificates. The feature needs to be enabled by passing a feature flag as described in the link above, and here is the difference from the original values file.

./infrastructure/homelab/controllers/cert-manager-values.yaml

$ diff cert-manager-values.yaml default-values/cert-manager-values.yaml
240,241c240
< extraArgs:
<   - --feature-gates=ExperimentalGatewayAPISupport=true
---
> extraArgs: []

I have also revised the script to generate manifest. The only change is that it now has values option to use the custom values file.

./infrastructure/homelab/controllers/cert-manager.sh

#!/bin/bash

# add flux helmrepo to the manifest
flux create source helm cert-manager \
    --url=https://charts.jetstack.io \
    --interval=1h0m0s \
    --export >cert-manager.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease cert-manager \
    --interval=10m \
    --target-namespace=cert-manager \
    --source=HelmRepository/cert-manager \
    --chart=cert-manager \
    --chart-version=1.14.3 \
    --values=./cert-manager-values.yaml \
    --export >>cert-manager.yaml

issuer¶

https://cert-manager.io/docs/configuration/

Next, I need to configure Issuer or ClusterIssuer resources which represents CA to respond to CSR.

Cloudflare API token¶

https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/

I have my domain managed through Cloudflare, and I will use DNS01 challenge method.

Here is the required scopes to be set for the Cloudflare API token:

permissions
- zone, dns, edit
- zone, zone, read
zone resources
- include all zones

I will create a new api token with these scopes set, and create secret. Since I know I will be using gateway to ask for certificate, I will put the API token secret as well as issuer manifest in the gateway namespace.

api token secret

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: gateway
type: Opaque
stringData:
  api-token: <API Token>

I am placing secret yaml files like this in the sops repo.

gitops/homelab-sops

.
 |-clusters
 | |-homelab
 | | |-.sops.pub.asc
 | | |-gateway
 | | | |-cloudflare-api-token-secret.yaml  # secret with cloudflare api token for cert-manager issuer in gateway namespace
 | | |-.sops.yaml
 | | |-minio-tenant
 | | | |-myminio-env-configuration.yaml
 | | | |-minio-tenant-secret.yaml

issuer manifest¶

Here is my issuer manifest. The email address at .spec.acme.email is the one I usually use to get letsencrypt certificate. The other email address is the one used to login Cloudflare dashboard.

If the provided information is all valid, the acme account key will be stored in a secret with the name specified at .spec.acme.privateKeySecretRef.name.

And I will have two issuers, one in cert-manager and another in gateway namespace. This is because I use annotation on Gateway resource to auto-generate certificate.

./infrastructure/homelab/configs/issuer.yaml

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: issuer
  namespace: gateway
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - dns01:
          cloudflare:
            email: [email protected]
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token
        selector:
          dnsZones:
            - "blink-1x52.net"

And here is the result.

$ kubectl get issuers -n gateway
NAME     READY   AGE
issuer   True    28s

gateway¶

Let's make some changes to the gateway manifest file.

One is to add annotation, cert-manager.io/issuer. The value is the name of the issuer in the gateway namespace.

Another change is to add listener for tenant-mc.blink-1x52.net:443. TLS mode is set as "terminate" to TLS-offload, and the backend will actually receive plain http traffic. The "certificateREfs" name is going to be the name of Secret for the generated TLS certificate, so I can name it however.

./infrastructure/homelab/configs/gateway.yaml

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: issuer
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes

    - name: https-minio-tenant
      hostname: tenant-mc.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-tenant-mc-20240304
            namespace: gateway
            kind: Secret

Once this is reconcilated, I can see the "Certificate" resource working to generate the certificate, and "Challenge" resource working on the actual DNS01 challenge.

# Certificate resource working to get the certificate ready
$ kubectl describe certificate -n gateway
... omitted for brevity ...
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    34s   cert-manager-certificates-trigger          Issuing certificate as Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     34s   cert-manager-certificates-key-manager      Reusing private key stored in existing Secret resource "tls-tenant-mc-hyperv-20240304"
  Normal  Requested  34s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-tenant-mc-hyperv-20240304-1"

# Challenge resource working on DNS01 challenge to obtain signed certificate
$ kubectl describe challenges -n gateway
... omitted for brevity ...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for DNS-01 challenge propagation: DNS record for "tenant-mc.blink-1x52.net" not yet propagated
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    54s   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  50s   cert-manager-challenges  Presented challenge using DNS-01 challenge mechanism

# Certificate issued
$ kubectl describe certificate -n gateway
... omitted for brevity ...
Status:
  Conditions:
    Last Transition Time:  2024-03-04T08:18:16Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-06-02T07:18:14Z
  Not Before:              2024-03-04T07:18:15Z
  Renewal Time:            2024-05-03T07:18:14Z
  Revision:                1
Events:
  Type    Reason     Age    From                                       Message
  ----    ------     ----   ----                                       -------
  Normal  Issuing    2m32s  cert-manager-certificates-trigger          Issuing certificate as Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     2m32s  cert-manager-certificates-key-manager      Reusing private key stored in existing Secret resource "tls-tenant-mc-20240304"
  Normal  Requested  2m32s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-tenant-mc-20240304-1"
  Normal  Issuing    75s    cert-manager-certificates-issuing          The certificate has been successfully issued

I have also revised the HTTPRoutes manifest for minio tenant to use the https listener added to the gateway. Below is the section added. The difference from the plain http HTTPRoute is the name (.metadata.name) and the gateway sectionName, which is the name of the listener added to the gateway.

./infrastructure/configs/minio-tenant.yaml

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console-tls
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-minio-tenant
      namespace: gateway
  hostnames:
    - "tenant-mc.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090

test the connection¶

Here is the log and capture from my other lab.

$ openssl s_client -connect tenant-mc.hyperv.blink-1x52.net:443 </dev/null
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = tenant-mc.hyperv.blink-1x52.net
verify return:1
---
Certificate chain
 0 s:CN = tenant-mc.hyperv.blink-1x52.net
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  4 07:18:15 2024 GMT; NotAfter: Jun  2 07:18:14 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
---
...

minio-console-https-access

https for minio s3 access¶

I created http gateway for minio s3 access. Since I will not be using S3 over plain http, I will again revise minio config manifest and gateway manifest.

I have just changed the sectionName to https-s3. The backend port will remain the same, tcp port 80.

./infrastructure/homelab/configs/minio-tenant.yaml

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: http
      namespace: gateway
  hostnames:
    - "tenant-mc.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console-tls
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-minio-tenant
      namespace: gateway
  hostnames:
    - "tenant-mc.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-service
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-s3
      namespace: gateway
  hostnames:
    - "s3.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: minio
          port: 80

And for the gateway I added another listener named https-s3 with corresponding hostname.

./infrastructure/homelab/configs/gateway.yaml

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: issuer
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes

    - name: https-minio-tenant
      hostname: tenant-mc.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-tenant-mc-20240304
            namespace: gateway
            kind: Secret

    - name: https-s3
      hostname: s3.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-s3-20240304
            namespace: gateway
            kind: Secret

repository structure so far¶

Below is the structure and memo for the ones added/changed.

There are a few additions to the homelab-sops repo, adding Cloudflare API token secret for cert-manager and gateway namespaces. The former was not used but I will leave it so that I can test out different ways to request certificate.

gitops/homelab repository

.
 |-infrastructure
 | |-homelab
 | | |-configs
 | | | |-kustomization.yaml
 | | | |-issuer.yaml                    # issuer in the gateway namespace
 | | | |-gateway.yaml                   # edit metadata to enable cert-manager
 | | | |-minio-tenant.yaml              # HTTPRoutes modified to use https gateway listeners for tenant and s3 access
 | | |-controllers
 | | | |-kustomization.yaml             # include cert-manager crds and manifest file
 | | | |-cert-manager-values.yaml       # customized values file to enable gateway annotations experimental feature
 | | | |-cert-manager.yaml              # generated cert-manager manifest helmrepo and helmrelease
 | | | |-cert-manager.sh                # script to generate cert-manager
 | | | |-crds
 | | | | |-cert-manager-v1.14.3.yaml    # cert-manager crds