Skip to content


beryl  obsidian  cloudflare  cloudflare pages  wrangler  container  docker  registry  harbor  wip  Dockerfile  kaniko  bytebase  gitops  os  debian  preseed  automation  git  github  credentials  gh  git tag  tips  gitlab  cicd  gitlab runner  glab  kubernetes  calico  flux  sops  gateway  nginx  nginx-gateway-fabric  certmanager  gitlab-runner  minio  s3  monitoring  kube-prometheus  prometheus  alertmanager  grafana  node-exporter  memcached  loki  promtail  operator  keycloak  postgres  testbed  Kubernetes  index  dns  coredns  k9s  krew  postgres-operator  hyper-v  helm  metallb  kubernetes gateway  ngf  directpv  logging  lab  reverse proxy  mfa  markdown  mise  python  venv  go  node  npm  nvim  lazyvim  poetry  regex  freenginx  mercurial  tls  tls passthrough  tls offload  bash  zsh  linux  ssg  hugo  mkdocs  tmux  tpm  pomodoro  vagrant  windows  virtualization  ansible  winget  windows terminal  windows 11 pro  oh-my-posh  nerdfonts  scoop 


nginx reverse proxy

#nginx   #tls   #tls passthrough   #tls offload  

Table of Content

nginx reverse proxy

TLS pass-through

In the beginning, I was using NGINX on docker to be the gateway to just proxy the traffic to other services with TLS offload.

When I started using kubernetes gateway, it was still fine at home as I could just point DNS record for the services to either docker NGINX or kubernetes gateway depending on where the service is hosted at. When I am outside however, since I only have one global IP address where the traffic gets forwarded to NGINX on docker only, I needed to have it pass-through the traffic towards kubernetes gateway for certain destination names.

Kroki

My /etc/nginx/nginx.conf file looks like this. I think it's just the copy of the default nginx.conf used in the offical docker image, I do not quite remember from which version this came from, and then I have modified server_names_hash_bucket_size so that the NGINX won't crash when using long server_name I rarely need for kubernetes. The stream block is of course not there by default. I added it to enable both TLS pass-through and TLS offload at the same time.

For http I just have it include the other .conf files in the other directory.

For stream I basically have it listening on port 443 with SSL preread feature to check the SNI, server name indicate, which is equivalent to the host header in plain http, to see and decide where should the traffic be forwarded to. So the https traffic on port 443 first hits this one. Then my options are the one labeled as k8s_cafe for kubernetes gateway at 192.168.1.54:443, and the rest on localhost:8443 where all the other reverse proxies are configured in the included .conf files in http section.

nginx.conf
user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '$request_time $upstream_connect_time $upstream_header_time $upstream_response_time $server_name';

    access_log  /var/log/nginx/access.log  main;
    error_log   /var/log/nginx/error.log   emerg;

    server_names_hash_bucket_size 128;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    # map
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
}


# stream
stream {

  # map sni and backend
  map $ssl_preread_server_name $name {
    hostnames;
    cafe.blink-1x52.net k8s_cafe;
    *.blink-1x52.net reverse_proxy;
  }

  # each backend/upstream
  upstream k8s_cafe {
    server 1.3.56.9:443 max_fails=3 fail_timeout=30s;
  }

  upstream reverse_proxy {
    server 127.0.0.1:8443 max_fails=3 fail_timeout=30s;
  }

  # listen
  server {
    listen 443;
    proxy_pass $name;
    ssl_preread on;
  }
}

preserving source IP address

When I only had http block, the basic http headers were sufficient for the services running behind the reverse proxy. For example, I have [[Authelia]] which is the open source authentication service, and I have it enabled for certain services I want to use when I'm outside and still keep it private to some extent. I of course do not want the additional MFA to run when I'm accessing the service at home, so I configured a simple IP address-based access control feature available on Authelia.

This easy IP address-based access control broke as soon as I implemented TLS pass-through as the source IP address reverse proxy instances see has changed to 127.0.0.1. The reverse proxies would add that IP address in the headers, and now the only IP address the services see is localhost IP address. You cannot discriminate if the access is from LAN or Internet anymore.

And so here is the additional feature I enabled to preserve the real source IP address of incoming traffics even with the stream processing in between.

https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/

I am using the official NGINX image available on Docker Hub, and it's built with stream_realip_module and http_realip_module, meeting the requirement to turn on this proxy protocol to preserve real source IP address.

$ docker exec nginx nginx -V
nginx version: nginx/1.25.1
built by gcc 12.2.0 (Debian 12.2.0-14)
built with OpenSSL 3.0.9 30 May 2023
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.25.1/debian/debuild-base/nginx-1.25.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

The changes I made on nginx.conf are highlighted below, proxy_protocol and set_real_ip_from lines.

https://nginx.org/en/docs/stream/ngx_stream_realip_module.html

Defines trusted addresses that are known to send correct replacement addresses

I have added rfc5737 private use and loopback address ranges.

nginx.conf stream block
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# stream
stream {

  # map sni and backend
  map $ssl_preread_server_name $name {
    hostnames;
    cafe.blink-1x52.net k8s_cafe;
    *.blink-1x52.net reverse_proxy;
  }

  # each backend/upstream
  upstream k8s_cafe {
    server 1.3.56.9:443 max_fails=3 fail_timeout=30s;
  }

  upstream reverse_proxy {
    server 127.0.0.1:8443 max_fails=3 fail_timeout=30s;
  }

  # listen
  server {
    listen 443;
    proxy_pass $name;
    proxy_protocol on;
    set_real_ip_from 10.0.0.0/8;
    set_real_ip_from 172.16.0.0/12;
    set_real_ip_from 192.168.0.0/16;
    set_real_ip_from 127.0.0.0/8;
    ssl_preread on;
  }
}

And then I have modified the http server config file as shown below for example. I have just added proxy_protocol.

service1.conf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
upstream service1 {
    server 1.2.3.45:8080;
    keepalive 90;
}

server {
    listen 8443 ssl proxy_protocol;
    http2 on;
    server_name service1.blink-1x52.net;

    # real ip
    real_ip_header proxy_protocol;

    ### omitted for brevity ###

And finally for any service which I want to pass the real source IP address to, I have revised the existing http header values to $proxy_protocol_addr in the config file.

    # proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Real-IP $proxy_protocol_addr;
    # proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_protocol_addr;

Same can be used for logging, and I have added $proxy_protocol_addr like this.

nginx.conf
    log_format  main  '$proxy_protocol_addr - $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '$request_time $upstream_connect_time $upstream_header_time $upstream_response_time $server_name';