Skip to content


beryl  kubernetes  calico  flux  sops  gitops  bytebase  gateway  nginx  nginx-gateway-fabric  certmanager  gitlab  gitlab-runner  cicd  minio  s3  monitoring  kube-prometheus  prometheus  alertmanager  grafana  node-exporter  memcached  loki  promtail  operator  keycloak  postgres  testbed  homelab  cilium  hlv3  ha  etcd  proxmox  cloud-init  fluxcd  cert-manager  encryption  longhorn  llm  open-webui  mcp  freenginx  mercurial  tls  tls passthrough  tls offload 


building homelab-v2 part 1

Table of Content

building homelab-v2 part 1

Here I am starting all over again with my homelab Kubernetes cluster using the same nodes as the previous one.

Part 1 just covers the same setup tasks done before, to setup GitOps using flux and add SOPS encrypt/decrypt flux kustomization.

node list

hostname role os arch core memory disk additional disk
rpi4bp control plane debian 12.5 arm64 4 4Gi 64GB n/a
livaz2 control plane debian 12.5 amd64 4 16Gi 128GB 6000GB, 5000Mbps
livaq2 worker node debian 12.5 amd64 4 4Gi 64GB n/a
ak3v worker node debian 12.5 amd64 2 8Gi 128GB 500GB, 5000Mbps
nb5 worker node debian 12.5 amd64 4 8Gi 128GB 500GB, 5000Mbps
venus worker node debian 12.5 amd64 12 16Gi 512GB n/a
gk41 docker, non-k8s debian 12.5 amd64 4 8Gi 128GB n/a
th80 docker, non-k8s debian 11.9 amd64 16 16Gi 512GB 500GB, 480Mbps
rpi4 ansible master, non-k8s debian 11.9 arm64 4 4Gi 32GB n/a

nb5 now in stockyard

venus joined on 2024-05-03

resetting the existing nodes

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#tear-down

All nodes were reset, and the main components such as cni, containerd, runc, kubelet, kubectl, and kubeadm were upgraded to the latest available on every kubernetes nodes.

My kubernetes cluster is now on v1.30.0, the latest available as of 2024-04-24.

kubectl get nodes -o wide
$ kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
ak3v     Ready    <none>          4h53m   v1.30.0   192.168.1.57    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64      containerd://1.7.15
livaq2   Ready    <none>          4h53m   v1.30.0   192.168.1.56    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64      containerd://1.7.15
livaz2   Ready    control-plane   4h54m   v1.30.0   192.168.1.52    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64      containerd://1.7.15
nb5      Ready    <none>          4h53m   v1.30.0   192.168.1.60    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-20-amd64      containerd://1.7.15
rpi4bp   Ready    control-plane   4h58m   v1.30.0   192.168.1.132   <none>        Debian GNU/Linux 12 (bookworm)   6.6.20+rpt-rpi-v8   containerd://1.7.15

Here is the version output from a amd64 node for cni, containerd, runc, and kube command line tools.

$ containerd --version
containerd github.com/containerd/containerd v1.7.15 926c9586fe4a6236699318391cd44976a98e31f1

$ /opt/cni/bin/dummy --version
CNI dummy plugin version unknown
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0

$ /usr/local/sbin/runc --version
runc version 1.1.12
commit: v1.1.12-0-g51d5e946
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.4

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0", GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-04-17T17:34:08Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}

$ kubelet --version
Kubernetes v1.30.0

$ kubectl version --client
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Here is the kubeadm init command used.

kubeadm init
sudo kubeadm init \
  --control-plane-endpoint=kube-endpoint.blink-1x52.net \
  --pod-network-cidr=10.244.0.0/16 \
  --service-dns-domain=hlv2.blink-1x52.net

The same network addon, calico, used in my previous homelab cluster was installed.

# confirm the latest version on cli
curl -sfL "api.github.com/repos/projectcalico/calico/releases/latest" | grep "tag_name"

# download v3.27.3
curl -LO https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/tigera-operator.yaml
curl -LO https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/custom-resources.yaml

# modify cidr in the custom-resources.yaml
# >> 10.244.0.0/16 to match what's specified in kubeadm init

# and then create
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml

# watch and wait for everything to be in RUNNING state
watch kubectl get pods -n calico-system -o wide

# allow control plane to work
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

# confirm
kubectl get nodes -o wide

setup gitops

gitops repository access token

  • create or go to the gitops repository on gitlab
  • create a project access token
    • name: hlv2
    • expiration date: default (1 month?)
    • role: owner
    • scope: api

flux bootstrap

export GITLAB_TOKEN={access_token_string_here}
export GITLAB_SERVER=cp.blink-1x52.net

flux bootstrap gitlab \
  --deploy-token-auth \
  --hostname="$GITLAB_SERVER" \
  --owner=gitops \
  --repository=homelab \
  --path=./clusters/hlv2 \
  --cluster-domain=hlv2.blink-1x52.net \
  --branch=main

setup sops

preparing gpg key
$ export KEY_NAME="hlv2.blink-1x52.net"
$ export KEY_COMMENT="flux sops secrets"

$ gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

gpg: key D29EC441F18C4B0A marked as ultimately trusted
gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/23C3E8985B27FF45689B2D8ED29EC441F18C4B0A.rev'

$ gpg --list-secret-keys "${KEY_NAME}"
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   6  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 6u
sec   rsa4096 2024-04-24 [SCEA]
      23C3E8985B27FF45689B2D8ED29EC441F18C4B0A
uid           [ultimate] hlv2.blink-1x52.net (flux sops secrets)
ssb   rsa4096 2024-04-24 [SEA]

$ export KEY_FP=23C3E8985B27FF45689B2D8ED29EC441F18C4B0A

$ gpg --export-secret-keys --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

secret/sops-gpg created

Now, go to the gitops repository and configure sops.

# gitops repository
cd {gitops repo}
mkdir -p sops/hlv2
cd sops/hlv2

# add sops configuration
cat <<EOF > .sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: ${KEY_FP}
EOF

# place public key so that others can import and use to encrypt
gpg --export --armor "${KEY_FP}" > .sops.pub.asc
# gpg --import .sops.pub.asc

# git commit and push

And finally add flux kustomization at ./clusters/hlv2/sops.yaml for this sops directory, ./sops/hlv2.

./clusters/hlv2/sops.yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: hlv2-sops
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
  interval: 1m0s
  path: ./sops/hlv2
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

testing sops

# still on the gitops repo, ./sops/hlv2 directory
kubectl -n default create secret generic basic-auth \
--from-literal=user=admin \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml

# confirm that the data are base64 encoded
cat basic-auth.yaml

# encrypt
sops -i --encrypt basic-auth.yaml

# confirm that the data are encrypted
cat basic-auth.yaml

# git commit and push

# confirm
kubectl -n default get secret basic-auth -o yaml