Skip to content


beryl  obsidian  cloudflare  cloudflare pages  wrangler  container  docker  registry  harbor  Dockerfile  kaniko  bytebase  gitops  mongodb  pymongo  mdbc  kubernetes  operator  os  debian  preseed  automation  git  github  credentials  gh  git tag  tips  gitlab  cicd  gitlab runner  glab  docker-compose  graphql  mutation  calico  flux  sops  gateway  nginx  nginx-gateway-fabric  certmanager  gitlab-runner  minio  s3  monitoring  kube-prometheus  prometheus  alertmanager  grafana  node-exporter  memcached  loki  promtail  keycloak  postgres  testbed  bastion  ssh  Kubernetes  index  dns  coredns  k9s  krew  storage  longhorn  postgres-operator  hyper-v  helm  metallb  kubernetes gateway  ngf  directpv  logging  lab  reverse proxy  mfa  markdown  wip  mise  python  venv  go  node  npm  snmp  multicast  nvim  lazyvim  ansible  poetry  regex  freenginx  mercurial  tls  tls passthrough  tls offload  bash  zsh  linux  ssg  hugo  mkdocs  tmux  tpm  pomodoro  vagrant  windows  virtualization  powershell  winget  windows terminal  windows 11 pro  oh-my-posh  nerdfonts  scoop 


building home lab part 5

#certmanager  

Table of Content

building home lab part 5

Now web access is available thanks to gateway and NGINX Gateway Fabric, and I setup web access to minio tenant console. It is currently only available with plain http. In this section, I am going to setup cert-manager, and add https access for the minio tenant access.

  • [x] prepare nodes (VMs on hyper-v in this series)
  • [x] setup kubernetes cluster
  • [x] bootstrap flux gitops
  • [x] setup sops for secret encryption
  • [x] install helm
  • [x] metallb to l2 advertise svc on LAN
  • [x] gateway as well as nginx gateway fabric to setup https gateway
  • [x] directpv to setup storage class, to serve persistent volume
  • [x] minio as s3 storage
  • [ ] cert manager to manage tls certificate
  • [ ] gitlab runner to execute CI/CD jobs on my gitlab
  • [ ] kube-prometheus for monitoring
  • [ ] loki for logging
  • [ ] weave gitops dashboard
  • [ ] kube-dashboard
  • [ ] postgresql
  • [ ] mongodb
  • [ ] bytebase
  • [ ] my private apps

cert-manager

https://cert-manager.io/

cert-manager is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

https://github.com/cert-manager/cert-manager

installation

https://cert-manager.io/docs/installation/

https://cert-manager.io/docs/installation/helm/

# add helm repo
helm repo add jetstack https://charts.jetstack.io

# get a copy of values file
cd {gitops repo}/infrastructure/controllers/default-values
helm show values jetstack/cert-manager > cert-manager-values.yaml

# download crds to install
cd ../crds
curl -L https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.crds.yaml -o cert-manager-v1.14.3.yaml

And as done many times with previous installations in this series, I prepare a script to generate namespace, flux helmrepo, and flux helmrelease manifests.

./infrastructure/controllers/cert-manager.sh
#!/bin/bash

# create a namespace manifest
cat >cert-manager.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    service: cert-manager
    type: infrastructure
EOF

# add flux helmrepo to the manifest
flux create source helm cert-manager \
    --url=https://charts.jetstack.io \
    --interval=1h0m0s \
    --export >>cert-manager.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease cert-manager \
    --interval=10m \
    --target-namespace=cert-manager \
    --source=HelmRepository/cert-manager \
    --chart=cert-manager \
    --chart-version=1.14.3 \
    --export >>cert-manager.yaml

Commit/push everything, let flux reconcilate, and here is the result.

$ kubectl get all -n cert-manager
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/cert-manager-cert-manager-784cddc4df-8ftfr              1/1     Running   0          67s
pod/cert-manager-cert-manager-cainjector-58bb9bfffc-rzv2m   1/1     Running   0          67s
pod/cert-manager-cert-manager-webhook-588db4b47f-8fq56      1/1     Running   0          67s

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager-cert-manager           ClusterIP   10.99.93.138    <none>        9402/TCP   69s
service/cert-manager-cert-manager-webhook   ClusterIP   10.109.17.212   <none>        443/TCP    69s

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager-cert-manager              1/1     1            1           69s
deployment.apps/cert-manager-cert-manager-cainjector   1/1     1            1           69s
deployment.apps/cert-manager-cert-manager-webhook      1/1     1            1           69s

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-cert-manager-784cddc4df              1         1         1       68s
replicaset.apps/cert-manager-cert-manager-cainjector-58bb9bfffc   1         1         1       68s
replicaset.apps/cert-manager-cert-manager-webhook-588db4b47f      1         1         1       68s

$ kubectl api-resources | grep cert
challenges                                                                        acme.cert-manager.io/v1                  true         Challenge
orders                                                                            acme.cert-manager.io/v1                  true         Order
certificaterequests               cr,crs                                          cert-manager.io/v1                       true         CertificateRequest
certificates                      cert,certs                                      cert-manager.io/v1                       true         Certificate
clusterissuers                                                                    cert-manager.io/v1                       false        ClusterIssuer
issuers                                                                           cert-manager.io/v1                       true         Issuer
certificatesigningrequests        csr                                             certificates.k8s.io/v1                   false        CertificateSigningRequest

issuer

https://cert-manager.io/docs/configuration/

Next, I need to configure Issuer or ClusterIssuer resources which represents CA to respond to CSR.

Cloudflare API token

https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/

I had my domain managed through Cloudflare, and I will use DNS01 challenge method.

Here is the required scopes to be set for the API token:

  • permissions
    • zone, dns, edit
    • zone, zone, read
  • zone resources
    • include all zones

I will create a new api token with these scopes set, and create secret.

api token example
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <API Token>

I am placing secret yaml files like this in the sops repo.

gitops/homelab-sops
.
 |-clusters
 | |-hyper-v
 | | |-.sops.pub.asc
 | | |-cert-manager
 | | | |-api-token.yaml
 | | |-myminio-env-configuration.yaml
 | | |-.sops.yaml
 |-.git

Encrypt the yaml file, commit/push and you'll have the secrets created.

$ kubectl get secret -n cert-manager
NAME                                   TYPE     DATA   AGE
cert-manager-cert-manager-webhook-ca   Opaque   3      2d19h
cloudflare-api-token-secret            Opaque   1      79s

# edit - namespace issuer also created in gateway namespace, so the token secret also must be here
$ kubectl get secret -n gateway
NAME                                  TYPE     DATA   AGE
cloudflare-api-token-secret           Opaque   1      6m36s
issuer-account-key                    Opaque   1      120m

issuer manifest

Here is my issuer manifest. The email address at .spec.acme.email is the one I usually use to get letsencrypt certificate. The other email address is the one used to login Cloudflare dashboard.

If the provided information is all valid, the acme account key will be stored in a secret with the name specified at .spec.acme.privateKeySecretRef.name.

And I will have two issuers, one in cert-manager and another in gateway namespace. This is because I use annotation on Gateway resource to auto-generate certificate.

./infrastructure/configs/issuer.yaml
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: issuer
  namespace: cert-manager
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - dns01:
          cloudflare:
            email: [email protected]
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token
        selector:
          dnsZones:
            - "blink-1x52.net"
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: issuer
  namespace: gateway
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - dns01:
          cloudflare:
            email: [email protected]
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token
        selector:
          dnsZones:
            - "blink-1x52.net"

And here is the result.

$ kubectl get issuers -n cert-manager
NAME     READY   AGE
issuer   True    28s

$ kubectl get secret -n cert-manager
NAME                                   TYPE     DATA   AGE
cert-manager-cert-manager-webhook-ca   Opaque   3      2d20h
cloudflare-api-token-secret            Opaque   1      60m
issuer-account-key                     Opaque   1      3m

certificate

https://cert-manager.io/docs/usage/

There are different ways to obtain certificate.

https://cert-manager.io/docs/usage/gateway/

I need to update cert-manager helm installation. The feature needs to be enabled by passing a feature flag as described in the link above, and here is the difference from the original values file.

./infrastructure/controllers/cert-manager-values.yaml
$ diff cert-manager-values.yaml default-values/cert-manager-values.yaml
240,241c240
< extraArgs:
<   - --feature-gates=ExperimentalGatewayAPISupport=true
---
> extraArgs: []

I have also revised the script to generate manifest. The only change is that it now has values option to use the custom values file.

./infrastructure/controllers/cert-manager.sh
#!/bin/bash

# create a namespace manifest
cat >cert-manager.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    service: cert-manager
    type: infrastructure
EOF

# add flux helmrepo to the manifest
flux create source helm cert-manager \
    --url=https://charts.jetstack.io \
    --interval=1h0m0s \
    --export >>cert-manager.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease cert-manager \
    --interval=10m \
    --target-namespace=cert-manager \
    --source=HelmRepository/cert-manager \
    --chart=cert-manager \
    --chart-version=1.14.3 \
    --values=./cert-manager-values.yaml \
    --export >>cert-manager.yaml

gateway

Let's make some changes to the gateway manifest file.

One is to add annotation, cert-manager.io/issuer. The value is the name of the issuer in the gateway namespace.

Another change is to add listener for tenant-mc.hyperv.blink-1x52.net:443. TLS mode is set as "terminate" to TLS-offload, and the backend will actually receive plain http traffic. The "certificateREfs" name is going to be the name of Secret for the generated TLS certificate.

x
---
kind: Namespace
apiVersion: v1
metadata:
  name: gateway
  labels:
    service: gateway
    type: infrastructure
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: issuer
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes

    - name: https-minio-tenant
      hostname: tenant-mc.hyperv.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-tenant-mc-hyperv-20240304
            namespace: gateway
            kind: Secret

Once this is reconcilated, I can see the "Certificate" resource working to generate the certificate, and "Challenge" resource working on the actual DNS01 challenge.

# Certificate resource working to get the certificate ready
$ kubectl describe certificate -n gateway
... omitted for brevity ...
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    34s   cert-manager-certificates-trigger          Issuing certificate as Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     34s   cert-manager-certificates-key-manager      Reusing private key stored in existing Secret resource "tls-tenant-mc-hyperv-20240304"
  Normal  Requested  34s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-tenant-mc-hyperv-20240304-1"

# Challenge resource working on DNS01 challenge to obtain signed certificate
$ kubectl describe challenges -n gateway
... omitted for brevity ...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for DNS-01 challenge propagation: DNS record for "tenant-mc.hyperv.blink-1x52.net" not yet propagated
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    54s   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  50s   cert-manager-challenges  Presented challenge using DNS-01 challenge mechanism

# Certificate issued
$ kubectl describe certificate -n gateway
... omitted for brevity ...
Status:
  Conditions:
    Last Transition Time:  2024-03-04T08:18:16Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-06-02T07:18:14Z
  Not Before:              2024-03-04T07:18:15Z
  Renewal Time:            2024-05-03T07:18:14Z
  Revision:                1
Events:
  Type    Reason     Age    From                                       Message
  ----    ------     ----   ----                                       -------
  Normal  Issuing    2m32s  cert-manager-certificates-trigger          Issuing certificate as Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     2m32s  cert-manager-certificates-key-manager      Reusing private key stored in existing Secret resource "tls-tenant-mc-hyperv-20240304"
  Normal  Requested  2m32s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-tenant-mc-hyperv-20240304-1"
  Normal  Issuing    75s    cert-manager-certificates-issuing          The certificate has been successfully issued

I have also revised the HTTPRoutes manifest for minio tenant to use the https listener added to the gateway. Below is the section added. The difference from the plain http HTTPRoute is the name (.metadata.name) and the gateway sectionName, which is the name of the listener added to the gateway.

./infrastructure/configs/minio-tenant.yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console-tls
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-minio-tenant
      namespace: gateway
  hostnames:
    - "tenant-mc.hyperv.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090

test the connection

$ openssl s_client -connect tenant-mc.hyperv.blink-1x52.net:443 </dev/null
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = tenant-mc.hyperv.blink-1x52.net
verify return:1
---
Certificate chain
 0 s:CN = tenant-mc.hyperv.blink-1x52.net
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  4 07:18:15 2024 GMT; NotAfter: Jun  2 07:18:14 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
---
...

minio-console-https-access

https for minio s3 access

I created http gateway for minio s3 access. Since I will not be using S3 over plain http, I will again revise minio config manifest and gateway manifest.

I have just changed the sectionName to https-s3. The backend port will remain the same, tcp port 80.

./infrastructure/configs/minio-tenant.yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: http
      namespace: gateway
  hostnames:
    - "tenant-mc.hyperv.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-console-tls
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-minio-tenant
      namespace: gateway
  hostnames:
    - "tenant-mc.hyperv.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: myminio-console
          port: 9090
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: minio-service
  namespace: minio-tenant
spec:
  parentRefs:
    - name: gateway
      sectionName: https-s3
      namespace: gateway
  hostnames:
    - "s3.hyperv.blink-1x52.net"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: minio
          port: 80

And for the gateway I added another listener named https-s3 with corresponding hostname.

./infrastructure/configs/gateway.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: gateway
  labels:
    service: gateway
    type: infrastructure
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: issuer
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes

    - name: https-minio-tenant
      hostname: tenant-mc.hyperv.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-tenant-mc-hyperv-20240304
            namespace: gateway
            kind: Secret

    - name: https-s3
      hostname: s3.hyperv.blink-1x52.net
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-available: yes
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-s3-hyperv-20240304
            namespace: gateway
            kind: Secret

repository structure so far

Below is the structure and memo for the ones added/changed.

There are a few additions to the homelab-sops repo, adding Cloudflare API token secret for cert-manager and gateway namespaces. The former was not used but I will leave it so that I can test out different ways to request certificate.

gitops/homelab repository
.
 |-clusters
 | |-hyper-v
 | | |-infrastructure.yaml
 | | |-flux-system
 | | | |-kustomization.yaml
 | | | |-gotk-sync.yaml
 | | | |-gotk-components.yaml
 | | |-node-vworker5-labels.yaml
 |-infrastructure
 | |-configs
 | | |-kustomization.yaml
 | | |-metallb-config.yaml
 | | |-issuer.yaml                 # cert-manager issuer in cert-manager and gateway namespace
 | | |-gateway.yaml                # https listeners added for tenant-mc.hyperv.blink-1x52.net and s3.hyperv.blink-1x52.net
 | | |-minio-tenant.yaml           # HTTPRoute manifests for https access to tenant-mc and s3 service
 | | |-directpv
 | | | |-drives.yaml
 | |-controllers
 | | |-kustomization.yaml
 | | |-minio-tenant-values.yaml
 | | |-metallb.yaml
 | | |-cert-manager-values.yaml    # cert-manager custom values file with gateway feature enabled
 | | |-cert-manager.yaml           # containing namespace, helmrepo, and helmrelease for cert-manager
 | | |-minio-tenant.sh
 | | |-metallb.sh
 | | |-minio-operator.yaml
 | | |-default-values
 | | | |-minio-tenant-values.yaml
 | | | |-cert-manager-values.yaml  # default cert-manager values file for reference
 | | | |-ngf-values.yaml
 | | | |-metallb-values.yaml
 | | |-ngf-values.yaml
 | | |-metallb-values.yaml
 | | |-cert-manager.sh              # script to generate cert-manager manifest file
 | | |-crds
 | | | |-cert-manager-v1.14.3.yaml  # crds for cert-manager
 | | | |-gateway-v1.0.0.yaml
 | | | |-directpv-v4.0.10.yaml
 | | |-minio-tenant.yaml
 | | |-sops.yaml
 | | |-ngf.yaml
 | | |-ngf.sh
 | | |-minio-operator.sh
 |-.git