Derek Neely

Nginx (Engine-X) is a very light-weight powerful web proxy server and load balancer. This is a quick how-to on configuring a weighted load balancer using Nginx. I’ll skip past the installation on your system. Mine is currently running on an OpenBSD firewall using Packet Filter, and is how I will be covering the configuration. Nginx is available and work on both Unix and Linux.

Configuring Nginx to start on system boot:

# vi /etc/rc.local

# start nginx
if [ -x /usr/local/sbin/nginx ]; then
   echo -n ' nginx'; /usr/local/sbin/nginx
fi

Load balance configuration (one public ip -> 3 private servers):

# vi /etc/nginx/nginx.conf

user  nobody;
worker_processes  2;
worker_rlimit_nofile 10240;

error_log  /var/log/nginx/error.log;

events {
   worker_connections  8192;
}

http {

   upstream  site {
        server 192.168.1.100:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:80 max_fails=2 fail_timeout=15 weight=1;
   }

   upstream  site_ssl {
        server 192.168.1.100:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:443 max_fails=2 fail_timeout=15 weight=1;
   }

   server {
        listen  AAA.BBB.CCC.DDD:80;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_pass http://site;
        }
   }

   server {
        listen  AAA.BBB.CCC.DDD:443;

        ssl on;
        ssl_certificate /etc/ssl/ssl.cert/site.com.crt;
        ssl_certificate_key /etc/ssl/ssl.key/site.com.key;

        server_name site.com;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X_FORWARDED_PROTO https;
             proxy_pass http://site_ssl;
        }
   }
}

Should you have a chain file you need to add it to the end of the certificate file as such:

# cat /path/to/chain.file >> /etc/ssl/ssl.cert/site.com.crt

Packet Filter Firewall Rules:

# vi /etc/pf.conf

site_ext="AAA.BBB.CCC.DDD"
site_int1="192.168.1.100"
site_int2="192.168.1.101"
site_int3="192.168.1.102"

pass in quick log on $ext_if inet proto tcp from any to $site_ext port 80 keep state
pass in quick log on $ext_if inet proto tcp from any to $corp_ext port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 443 keep state

Helpful Nginx server management commands:

Test Nginx Config file:

# nginx -t -c /etc/nginx/nginx.conf

Restart Nginx server:

# kill -HUP `cat /var/run/nginx.pid`

Stop Nginx server:

# kill -QUIT `cat /var/run/nginx.pid`

If your like me and want to keep tabs on the load balancing (as in ipvsadm for Linux ldirectord) you can add a label to the end of each of your firewall rules:

label “[some unique label for this rule]” (i.e. label “site-ext”)

Then using pftop -v label you will see:

RULE LABEL           PKTS     BYTES     STATES MAX  ACTION   DIR
....
16   site-ext        867991   630495K   11833       Pass     In  ...
17   site-ext-ssl    29427    13315230  847         Pass     In  ...
18   site-int-1      13761450 6842062   813483      Pass     Out ...
19   site-int-1-ssl  234678   12345     12344       Pass     Out ...
....