Skip to content

building homelab cluster part 1

I have been rebuilding my homelab kubernetes cluster from time to time. This series will be my record of rebuilding kubernetes cluster using baremetal machines.


I have a cluster of 5 nodes, 2 of them as control plane.

I have two more machines that I use to run docker containers. They run reverse proxy, MFA, GitLab, Harbor, and so on.

The last one, raspberry pi 4, is the ansible master to run maintenance playbooks.

hostname role os arch core memory disk additional disk
rpi4bp control plane debian 12.5 arm64 4 4Gi 64GB n/a
livaz2 control plane debian 12.5 amd64 4 16Gi 128GB 6000GB, 5000Mbps
livaq2 worker node debian 12.5 amd64 4 4Gi 64GB n/a
ak3v worker node debian 12.5 amd64 2 8Gi 128GB 500GB, 5000Mbps
nb5 worker node debian 12.5 amd64 4 8Gi 128GB 500GB, 5000Mbps
gk41 docker, non-k8s debian 12.5 amd64 4 8Gi 128GB n/a
th80 docker, non-k8s debian 11.9 amd64 16 16Gi 512GB 500GB, 480Mbps
rpi4 ansible master, non-k8s debian 11.9 arm64 4 4Gi 32GB n/a

k8s-ready nodes

I am skipping the part to build the cluster.

It's just about following the instruction on kubernetes official document, and I've also posted a blog post in Japanese here - Kubernetesクラスタ構築 - おうちでGitOpsシリーズ投稿2/6, however, I am planning to add a link to separate page or public repository to share the actual ansible playbooks used to configure nodes to be k8s-ready.

apply network addon to the cluster

And then I manually applied calico installation manifests to make the cluster ready for service.

# confirm the latest version on cli
curl -sfL "" | grep "tag_name"

# download v3.27.0
curl -LO
curl -LO

# modify custom-resources.yaml

# and then apply
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml

# watch and wait for everything to be ready
watch kubectl get pods -n calico-system -o wide

# allow control plane to work
kubectl taint nodes --all

Here is my calico custom-resources.yaml file. I just changed the cidr.

# This section includes base Calico installation configuration.
# For more information, see:
kind: Installation
  name: default
  # Configures Calico networking.
    # Note: The ipPools section cannot be modified post-install.
      - blockSize: 26
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()

# This section configures the Calico API server.
# For more information, see:
kind: APIServer
  name: default
spec: {}

And so here it is.

kubectl get nodes -o wide
ak3v     Ready    <none>          36m    v1.29.2    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64      containerd://1.7.13
livaq2   Ready    <none>          16m    v1.29.2    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64      containerd://1.7.13
livaz2   Ready    control-plane   2d2h   v1.29.2    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64      containerd://1.7.13
nb5      Ready    <none>          103m   v1.29.2    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64      containerd://1.7.13
rpi4bp   Ready    control-plane   2d3h   v1.29.2   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-rpi8-rpi-v8   containerd://1.7.13

installing calicoctl

curl -L -o kubectl-calico
chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/.

upgrading calico

Get the operator manifest and run kubectl replace -f {manifest}.

For example)

curl -O
kubectl replace -f tigera-operator.yaml

gitops - install and bootstrap flux

Here is the page I prepared apart from this lab building effort - flux

Required permissions

To bootstrap Flux, the person running the command must have cluster admin rights for the target Kubernetes cluster. It is also required that the person running the command to be the owner of the GitLab project, or to have admin rights of a GitLab group.

And here is what I actually do this time. Nothing special.

  • create a new repository gitops/homelab on my self-managed GitLab
  • create an access token with owner role and api access
  • install flux on whichever client machine with kubeconfig, management access to the control plane
  • bootstrap

creating project access token for flux bootstrap


install flux

curl -s | sudo bash
flux version --client


export GITLAB_TOKEN={token_string_here}
export GITLAB_SERVER={gitlab_server}

flux bootstrap gitlab \
  --deploy-token-auth \
  --hostname="$GITLAB_SERVER" \
  --owner=gitops \
  --repository=homelab \
  --path=./clusters/homelab \

Here is the log, not for the baremetals but for the hyper-v VM cluster.

$ flux bootstrap gitlab \
  --deploy-token-auth \
  --hostname="$GITLAB_SERVER" \
  --owner=gitops \
  --repository=homelab \
  --path=./clusters/hyper-v \
► connecting to
► cloning branch "main" from Git repository ""
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("0759d8f2ea8a74943868e932ebdd1d707b7c0c40")
► pushing component manifests to ""
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► checking to reconcile deploy token for source secret
✔ configured deploy token "flux-system-main-flux-system-./clusters/hyper-v" for ""
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("3ec779843915ffad20a2dc220f5a4e65d5bdb2fa")
► pushing sync manifests to ""
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

$ flux get all
NAME                            REVISION                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       main@sha1:3ec77984      False           True    stored artifact for revision 'main@sha1:3ec77984'

NAME                            REVISION                SUSPENDED       READY   MESSAGE
kustomization/flux-system       main@sha1:3ec77984      False           True    Applied revision: main@sha1:3ec77984

$ flux version
flux: v2.2.2
distribution: flux-v2.2.2
helm-controller: v0.37.2
kustomize-controller: v1.2.1
notification-controller: v1.2.3
source-controller: v1.2.3

gitops repository structure

I will try to follow the guide as much as possible.


I will make gitops/homelab-sops the repository to store encrypted secret.

First, I will generate a keypair, put it in the flux-system namespace as secret, and upload public key to the repository.

# install sops and gpg if not installed
# create gitops/homelab-sops repository on gitlab if not created yet

# set key name and comment
export KEY_NAME=""
export KEY_COMMENT="homelab flux secret"

# generate the keypair
gpg --batch --full-generate-key <<EOF
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}

# find the fingerprint of the generated keypair
# and export
gpg --list-secret-keys "${KEY_NAME}"
export KEY_FP={fingerprint of the generated keypair}

# create the secret using the keypair in the flux-system namespace
gpg --export-secret-keys --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg \
--namespace=flux-system \

# confirm that the secret is there
kubectl get secret -n flux-system sops-gpg

# clone and go to the homelab-sops project directory
git clone
cd homelab-sops

# create the directory for hyper-v cluster and place public key there
mkdir -p clusters/homelab
gpg --export --armor "${KEY_FP}" > ./clusters/homelab/
# commit and push so that going forward other member or yourself on other machine can import it to encrypt and place secret manifests in this repository
# to import...
# gpg --import ./clusters/homelab/

# delete the keypair from the local machine
gpg --delete-secret-keys "${KEY_FP}"

# configure encryption
cat <<EOF > ./clusters/hyper-v/.sops.yaml
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: ${KEY_FP}

And next, I will set additional git source for flux to watch on the original gitops/homelab repository.

Unless the gitops/homelab-sops is a public repository, flux needs something to be authorized to read the repository. Let's create a token with read access for flux.

Navigate to the gitops/homelab-sops repository, settings > repository > deploy tokens, and click add token to create one. I'd name it "homelab-sops" and set "read_repository" scope to this deploy token.


# create a secret from the deploy token to read the sops repository
flux create secret git deploy-token-homelab-sops \
  --namespace=flux-system \
  --username={deploy token name here} \
  --password={deploy token string here} \

This probably is the last secret to manually push to the kubernetes cluster, since the gitops by flux is in place and secret encryption will be ready after sops setup.

I move on to update the cluster setup in the original gitops/homelab repository to start using the other gitops/homelab-sops repository.

kind: GitRepository
  name: homelab-sops
  namespace: flux-system
  interval: 1m0s
    branch: main
    name: deploy-token-homelab-sops
kind: Kustomization
  name: homelab-sops
  namespace: flux-system
    provider: sops
      name: sops-gpg
  interval: 1m0s
  path: ./clusters/homelab
  prune: true
    kind: GitRepository
    name: homelab-sops

Now everything is set. The homelab-sops flux kustomization fails until there is some manifest. I will add test secret.

# on gitops/homelab-sops repository
cd clusters/homelab
kubectl -n default create secret generic basic-auth \
--from-literal=user=admin \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml
sops --encrypt --in-place basic-auth.yaml

# see the secret manifest file basic-auth.yaml is now encrypted
# git commit and push

# wait for flux reconcilation
# confirm the status
flux get ks homelab-sops

# confirm the secret created
kubectl get secret basic-auth -n default -o jsonpath='{.data.user}' | base64 -d

calico manifest files

The network addon calico was installed manually, but I would like to place the manifest on the repository. I copied the two files I used and placed them in ./clusters/homelab/network-addon/.

node labels

I will be installing directpv later, and I will go ahead and label nodes with disks.

apiVersion: v1
kind: Node
  labels: directpv
  name: livaz2
kubectl label --list nodes livaz2

repository structure so far

homelab repository structure
 | |-homelab
 | | |-network-addon
 | | | |-custom-resources.yaml   # calico config file
 | | | |-tigera-operator.yaml    # calico installation file
 | | | |-version.txt             # text file with version of the installed calico
 | | |-flux-system               # installed by flux
 | | | |-kustomization.yaml
 | | | |-gotk-sync.yaml
 | | | |-gotk-components.yaml
 | | |-sops.yaml                 # flux gitrepo and kustomization for sops repo
 | | |-nodes                     # node labels
 | | | |-node-livaz2-label.yaml
 | | | |-node-ak3v-label.yaml
 | | | |-node-nb5-label.yaml
homelab-sops repository structure
 | |-homelab
 | | |    # public key you can import and use to encrypt
 | | |-.sops.yaml       # encryption settings
 | | |-basic-auth.yaml  # test secret I will remove later