Derek Neely

Nginx (Engine-X) is a very light-weight powerful web proxy server and load balancer. This is a quick how-to on configuring a weighted load balancer using Nginx. I’ll skip past the installation on your system. Mine is currently running on an OpenBSD firewall using Packet Filter, and is how I will be covering the configuration. Nginx is available and work on both Unix and Linux.

Configuring Nginx to start on system boot:

# vi /etc/rc.local

# start nginx
if [ -x /usr/local/sbin/nginx ]; then
   echo -n ' nginx'; /usr/local/sbin/nginx
fi

Load balance configuration (one public ip -> 3 private servers):

# vi /etc/nginx/nginx.conf

user  nobody;
worker_processes  2;
worker_rlimit_nofile 10240;

error_log  /var/log/nginx/error.log;

events {
   worker_connections  8192;
}

http {

   upstream  site {
        server 192.168.1.100:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:80 max_fails=2 fail_timeout=15 weight=1;
   }

   upstream  site_ssl {
        server 192.168.1.100:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:443 max_fails=2 fail_timeout=15 weight=1;
   }

   server {
        listen  AAA.BBB.CCC.DDD:80;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_pass http://site;
        }
   }

   server {
        listen  AAA.BBB.CCC.DDD:443;

        ssl on;
        ssl_certificate /etc/ssl/ssl.cert/site.com.crt;
        ssl_certificate_key /etc/ssl/ssl.key/site.com.key;

        server_name site.com;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X_FORWARDED_PROTO https;
             proxy_pass http://site_ssl;
        }
   }
}

Should you have a chain file you need to add it to the end of the certificate file as such:

# cat /path/to/chain.file >> /etc/ssl/ssl.cert/site.com.crt

Packet Filter Firewall Rules:

# vi /etc/pf.conf

site_ext="AAA.BBB.CCC.DDD"
site_int1="192.168.1.100"
site_int2="192.168.1.101"
site_int3="192.168.1.102"

pass in quick log on $ext_if inet proto tcp from any to $site_ext port 80 keep state
pass in quick log on $ext_if inet proto tcp from any to $corp_ext port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 443 keep state

Helpful Nginx server management commands:

Test Nginx Config file:

# nginx -t -c /etc/nginx/nginx.conf

Restart Nginx server:

# kill -HUP `cat /var/run/nginx.pid`

Stop Nginx server:

# kill -QUIT `cat /var/run/nginx.pid`

If your like me and want to keep tabs on the load balancing (as in ipvsadm for Linux ldirectord) you can add a label to the end of each of your firewall rules:

label “[some unique label for this rule]” (i.e. label “site-ext”)

Then using pftop -v label you will see:

RULE LABEL           PKTS     BYTES     STATES MAX  ACTION   DIR
....
16   site-ext        867991   630495K   11833       Pass     In  ...
17   site-ext-ssl    29427    13315230  847         Pass     In  ...
18   site-int-1      13761450 6842062   813483      Pass     Out ...
19   site-int-1-ssl  234678   12345     12344       Pass     Out ...
....

At my job we are moving to Nginx for the load balancing of our sites. Nginx is a very powerful load balancing/proxy server tool. It allows weighting, ssl acceleration, among other functionality while remaining light weight and easy to configure.

In preperation for a large web services launch, I began to analyze some logs and keep an eye on the system. I noticed one of the sites that we’ve already deployed was hammering our error messages in /var/log/nginx/error.log reading:

2009/06/23 12:38:22 [crit] 808#0: *724154 open() “/var/nginx/tmp/proxy_temp/4/83/0000002834″ failed (13: Permission denied) while reading upstream, client: XXX.XXX.XXX.XXX, server: xxx.host.com, request: “GET /dir/page.php”, upstream: “http://backendserverip/dir/page.php”, host: “host.com”, referrer: “http://referrer.com/apage.php”

Upon reviewing the site I noticed some (not all) of the pages were only partially loading. The issue is exactly what the log says. Permission denied = Permission issue.

Check your /etc/nginx/nginx.conf (OpenBSD) file for the user nginx processes will run as:

user  nobody;

Or, do:

# ps aux | grep “nginx: worker process” | awk ‘{print $1}’
nobody

In both cases you see that I’m running the nginx worker process as user nobody. Now we need to check our permissions on: /var/nginx/tmp/proxy_temp

# ls -l /var/nginx/tmp/ | grep proxy_temp
drwxrwx—  12 nobody  _nginx  512 Jun 23 13:10 proxy_temp

Looks good. The directory is owned by nobody and is writeable by both nobody and the group _nginx. What could the issue be? Lets move up a level and check the permissions.

# ls -l /var/nginx | grep tmp
drwx——  5 _nginx  _nginx  512 May  7 11:54 tmp

Ah ha! The parent directory is owned my _nginx:_nginx and is only writeable for that user. Our user ‘nobody’ therefore does not have the permissions to write in here. So, we can do a few things. Either make the entire directory writeable by everyone or change the ownership.

# chmod 777 /var/nginx/tmp

or

# chown nobody:_nginx /var/nginx/tmp

This should cure your permissions issues and all pages should load completely (at least mine do!)