Skip to content


beryl  obsidian  cloudflare  cloudflare pages  wrangler  container  docker  registry  harbor  wip  Dockerfile  kaniko  bytebase  gitops  os  debian  preseed  automation  git  github  credentials  gh  git tag  tips  gitlab  cicd  gitlab runner  glab  kubernetes  calico  flux  sops  gateway  nginx  nginx-gateway-fabric  certmanager  gitlab-runner  minio  s3  monitoring  kube-prometheus  prometheus  alertmanager  grafana  node-exporter  memcached  loki  promtail  operator  keycloak  postgres  testbed  Kubernetes  index  dns  coredns  k9s  krew  postgres-operator  hyper-v  helm  metallb  kubernetes gateway  ngf  directpv  logging  lab  reverse proxy  mfa  markdown  mise  python  venv  go  node  npm  snmp  nvim  lazyvim  poetry  regex  freenginx  mercurial  tls  tls passthrough  tls offload  bash  zsh  linux  ssg  hugo  mkdocs  tmux  tpm  pomodoro  vagrant  windows  virtualization  ansible  winget  windows terminal  windows 11 pro  oh-my-posh  nerdfonts  scoop 


nginx gateway fabric

Table of Content

nginx gateway fabric

I have installed and setup nginx gateway fabric, NGF, as my kubernetes cluster gateway, and my notes from early 2024 is at [[building-home-lab-part-3]].

It is now late 2024 and it is about time I should do the upgrade of NGF. Current version running on my kubernetes cluster is 1.2.0 (and the page above is on installing 1.1.0, so there was one upgrade from there to 1.2.0), and the latest available is 1.4.0 which is the one I'm going for this time.

https://github.com/nginxinc/nginx-gateway-fabric

upgrading NGF

preparing helm values file

# see the release page of the repository and confirm which version you want

# confirm the chart
helm show chart oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 1.4.0

# get the copy of the values file
helm show values oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 1.4.0 > ngf-1.4.0-values.yaml

# have the backup of the values file currently running (version 1.2.0 in my case)

# edit the values file from the version 1.4.0
# notes: all the changes I had to make was the same from what I did for the previous versions

# need to apply Kubernetes Gateway API v1.1.0 for NGF 1.4.0 as per the tech apec
# https://github.com/nginxinc/nginx-gateway-fabric?tab=readme-ov-file#technical-specifications
# https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api
curl -L https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml -o gateway-api-v1.1.0-standard-install.yaml

Now, I keep the default values file in the different directory ./default-values so that I can diff and also start all over from there, and the values file under ./values is the one I edit and use for the helm deployment. The directory structure looks like this.

$ pwd
/home/{username}/repos/{repository}/gitops/homelab/infrastructure/{cluster_name}/controllers
$ tree
.
 |-kustomization.yaml       # flux kustomization for infra-controllers
 |-values
 | |-ngf-values.yaml        # to be, 1.4.0
 | |-ngf-values.yaml.bak    # running version 1.2.0
 |-default-values
 | |-ngf-1.4.0-values.yaml  # copy, just there as a record
 | |-ngf-1.2.0-values.yaml  # copy, just there as a record
 |-crds
 | |-gateway-api-v1.0.0-standard-install.yaml
 | |-gateway-api-v1.1.0-standard-install.yaml  # newer version to apply
 |-scripts
 | |-ngf.sh                 # a simple script to run gitops flux commands to generate flux helm source and helm repo manifests
 |-ngf.yaml                 # generated helm source and helm repo manifests to be included in the flux kustomize

preparing the custom nginx image

https://docs.nginx.com/nginx-gateway-fabric/installation/ngf-images/building-the-images/

I build custom image on my private gitlab repository pipeline using kaniko. The instruction to build the image on the official document is to clone the repo and run make, so I first inspect what will actually going to be executed.

# build instruction in the official document
# https://docs.nginx.com/nginx-gateway-fabric/installation/ngf-images/building-the-images/
git clone https://github.com/nginxinc/nginx-gateway-fabric.git --branch v1.4.0
cd nginx-gateway-fabric
make PREFIX=myregistry.example.com/nginx-gateway-fabric build-prod-nginx-image

This is what is in the ./Makefile for build-prod-nginx-image.

.PHONY: build-prod-nginx-image
build-prod-nginx-image: build-nginx-image ## Build the custom nginx image for production

.PHONY: build-nginx-image
build-nginx-image: check-for-docker ## Build the custom nginx image
    docker build --platform linux/$(GOARCH) $(strip $(NGINX_DOCKER_BUILD_OPTIONS)) -f $(SELF_DIR)build/Dockerfile.nginx -t $(strip $(NGINX_PREFIX)):$(strip $(TAG)) $(strip $(SELF_DIR))

So it is going to run docker build using ./build/Dockerfile.nginx, and I will prepare my gitlab repository to do the same thing.

Here is the Dockerfile with some changes to remove label and add my harbor registry proxy integrated with my gitlab.

gitlab doc on harbor integration

./Dockerfile
ARG HARBOR_HOST
FROM $HARBOR_HOST/cache-dockerhub/library/nginx:1.27.1-alpine-otel

ARG NJS_DIR
ARG NGINX_CONF_DIR

RUN apk add --no-cache libcap \
    && mkdir -p /var/lib/nginx /usr/lib/nginx/modules \
    && setcap 'cap_net_bind_service=+ep' /usr/sbin/nginx \
    && setcap -v 'cap_net_bind_service=+ep' /usr/sbin/nginx \
    && apk del libcap

COPY ${NJS_DIR}/httpmatches.js /usr/lib/nginx/modules/njs/httpmatches.js
COPY ${NGINX_CONF_DIR}/nginx.conf /etc/nginx/nginx.conf
COPY ${NGINX_CONF_DIR}/grpc-error-locations.conf /etc/nginx/grpc-error-locations.conf
COPY ${NGINX_CONF_DIR}/grpc-error-pages.conf /etc/nginx/grpc-error-pages.conf

RUN chown -R 101:1001 /etc/nginx /var/cache/nginx /var/lib/nginx

USER 101:1001

CMD ["sh", "-c", "rm -rf /var/run/nginx/*.sock && /docker-entrypoint.sh nginx -g 'daemon off;'"]

I need files required for COPY commands, so I will also prepare these four files. I copy from the original repository, and edit nginx.conf file however I need.

  • ./conf/nginx.conf
  • ./conf/grpc-error-locations.conf
  • ./conf/grpc-error-pages.conf
  • ./jssrc/httpmatches.js

My ./.gitlab-ci.yml looks like this. The two variables NGINX_CONF_DIR=conf and NJS_DIR=jssrc are specified here.

./.gitlab-ci.yml
variables:
  KANIKO_CACHE_DIR: "/cache"
  KANIKO_VERSION: "v1.23.2-debug"
  VERSION: "1.4.0"

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - "$KANIKO_CACHE_DIR"

stages:
  - build

build:
  stage: build
  retry: 2
  image:
    name: $HARBOR_HOST/cache-gcrio/kaniko-project/executor:$KANIKO_VERSION
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$HARBOR_HOST\":{\"auth\":\"$(echo -n ${HARBOR_USERNAME}:${HARBOR_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
      --build-arg "NJS_DIR=jssrc"
      --build-arg "NGINX_CONF_DIR=conf"
      --build-arg "HARBOR_HOST=$HARBOR_HOST"
      --cache
      --cache-dir /cache
      --cache-copy-layers
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}:${VERSION}"

Run this script to generate the flux helm source and helm repo manifests to let flux kustomization process them.

infrastructure/{cluster_name}/controllers/scripts/ngf.sh
#!/bin/bash

# add flux helmrepo to the manifest
flux create source helm ngf \
        --url=oci://ghcr.io/nginxinc/charts \
        --interval=1h0m0s \
        --export >../ngf.yaml

# add flux helm release to the manifest including the customized values.yaml file
flux create helmrelease ngf \
        --interval=10m \
        --target-namespace=ngf \
        --source=HelmRepository/ngf \
        --chart=nginx-gateway-fabric \
        --chart-version=1.4.0 \
        --values=../values/ngf-values.yaml \
        --export >>../ngf.yaml

Apply both Kubernetes Gateway API v1.1.0 CRDS and NGF 1.4.0 helm release.