Nginx

Configuring Nginx to start on system boot:

# vi /etc/rc.local

# start nginx
if [ -x /usr/local/sbin/nginx ]; then
   echo -n ' nginx'; /usr/local/sbin/nginx
fi

Load balance configuration (one public ip -> 3 private servers):

# vi /etc/nginx/nginx.conf

user  nobody;
worker_processes  2;
worker_rlimit_nofile 10240;

error_log  /var/log/nginx/error.log;

events {
   worker_connections  8192;
}

http {

   upstream  site {
        server 192.168.1.100:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:80 max_fails=2 fail_timeout=15 weight=1;
   }

   upstream  site_ssl {
        server 192.168.1.100:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:443 max_fails=2 fail_timeout=15 weight=1;
   }

   server {
        listen  AAA.BBB.CCC.DDD:80;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_pass http://site;
        }
   }

   server {
        listen  AAA.BBB.CCC.DDD:443;

        ssl on;
        ssl_certificate /etc/ssl/ssl.cert/site.com.crt;
        ssl_certificate_key /etc/ssl/ssl.key/site.com.key;

        server_name site.com;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X_FORWARDED_PROTO https;
             proxy_pass http://site_ssl;
        }
   }
}

Should you have a chain file you need to add it to the end of the certificate file as such:

# cat /path/to/chain.file >> /etc/ssl/ssl.cert/site.com.crt

Packet Filter Firewall Rules:

# vi /etc/pf.conf

site_ext="AAA.BBB.CCC.DDD"
site_int1="192.168.1.100"
site_int2="192.168.1.101"
site_int3="192.168.1.102"

pass in quick log on $ext_if inet proto tcp from any to $site_ext port 80 keep state
pass in quick log on $ext_if inet proto tcp from any to $corp_ext port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 443 keep state

Helpful Nginx server management commands:

Test Nginx Config file:

# nginx -t -c /etc/nginx/nginx.conf

Restart Nginx server:

# kill -HUP `cat /var/run/nginx.pid`

Stop Nginx server:

# kill -QUIT `cat /var/run/nginx.pid`

If your like me and want to keep tabs on the load balancing (as in ipvsadm for Linux ldirectord) you can add a label to the end of each of your firewall rules:

label “[some unique label for this rule]” (i.e. label “site-ext”)

Then using pftop -v label you will see:

RULE LABEL           PKTS     BYTES     STATES MAX  ACTION   DIR
....
16   site-ext        867991   630495K   11833       Pass     In  ...
17   site-ext-ssl    29427    13315230  847         Pass     In  ...
18   site-int-1      13761450 6842062   813483      Pass     Out ...
19   site-int-1-ssl  234678   12345     12344       Pass     Out ...
....

I originally started with this tutorial from OpenBSD and was a good starting point but I felt it left out a few things.

First off. The tutorial above uses the ftp-proxy server on the box to allow proxying down to the ftp server. However, I didn’t exactly see any benefit of using it and didn’t seem to make anything easier. If I’m completely misunderstanding the usage of it please contact me and help me out. I DO want to do things right.

Here’s the overall layout of what is to happen:

[Client] <—> [PF Firewall] <—> [FTP Server]

Now, a few observations I’ve made while working with the firewall and maybe a bit clearer for others.

Here’s a great explanation of the overall FTP process from Tech Republic. Here’s the overall communication ports for the 2 different modes (as the PF firewall sees it – at least in my experience).

Active FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)

-Outbound-
— Data Transfer —
Outgoing: > 30000 (FTP Server Interface)
Outgoing: > 30000 (External interface – FW)

Passive FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)
— Data Transfer —
Incoming:  49000-51000 (External interface – FW)
Outgoing: 49000-50000 (Internal interface – FW)
Incoming: 49000-50000 (FTP Server Interface)

-Outbound-
Outgoing: 49000-50000 (FTP Server Interface)
Outgoing: 49000-50000 (External interface – FW)

Alright enough chat and on to the code. First we need to configure the ftp server. In the  below examples I’ll cover PureFTP and ProFTPd for just the primary pieces of passive port configuration (I’ll give a nice ProFTP advanced configuration in a post to come). We’ll also assume the default ‘listening’ port 21 of any standard FTP install/configuration. This is also on a Gentoo server so modify accordingly for your given distro.

PureFTPd:

MISC_OTHER=”[various options/flags] -p 49000:51000 -P [Public IP]“

ProFTPd:

PassivePorts 49000 51000
MasqueradeAddress [Public IP]

PF Firewall Rules: (Just the FTP Rules)

# Interface Definitions:
ftp_ext=”[Public IP]”
ftp_int=”[Private IP]”

# NAT Rules
nat on $ext_if from { $ftp_int } to any -> $ftp_ext

# Redirect Rules
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 21 -> $ftp_int port 21
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 49000:51000 -> $ftp_int

# Firewall Rules
pass out quick on $ext_if inet proto tcp from $ftp_ext to any port > 1024 keep state

pass out quick log on $int_if inet proto tcp from any to $ftp_int port 21 keep state tag FTP label “ftp”
pass out quick log on $int_if inet proto tcp from any to $ftp_int port 49000:51000 keep state tag FTP_PASV label “ftp-passive”

That should be all you need for the PF rules and configuration on the FTP servers to get everything working and passing through. Feel free to contact me should you have any additional fixes or have a good explanation of how the ftp-proxy works and benefits of. I’ve just gotta find the time to do some experimenting.

In preperation for a large web services launch, I began to analyze some logs and keep an eye on the system. I noticed one of the sites that we’ve already deployed was hammering our error messages in /var/log/nginx/error.log reading:

2009/06/23 12:38:22 [crit] 808#0: *724154 open() “/var/nginx/tmp/proxy_temp/4/83/0000002834″ failed (13: Permission denied) while reading upstream, client: XXX.XXX.XXX.XXX, server: xxx.host.com, request: “GET /dir/page.php”, upstream: “http://backendserverip/dir/page.php”, host: “host.com”, referrer: “http://referrer.com/apage.php”

Upon reviewing the site I noticed some (not all) of the pages were only partially loading. The issue is exactly what the log says. Permission denied = Permission issue.

Check your /etc/nginx/nginx.conf (OpenBSD) file for the user nginx processes will run as:

user  nobody;

Or, do:

# ps aux | grep “nginx: worker process” | awk ‘{print $1}’
nobody

In both cases you see that I’m running the nginx worker process as user nobody. Now we need to check our permissions on: /var/nginx/tmp/proxy_temp

# ls -l /var/nginx/tmp/ | grep proxy_temp
drwxrwx—  12 nobody  _nginx  512 Jun 23 13:10 proxy_temp

Looks good. The directory is owned by nobody and is writeable by both nobody and the group _nginx. What could the issue be? Lets move up a level and check the permissions.

# ls -l /var/nginx | grep tmp
drwx——  5 _nginx  _nginx  512 May  7 11:54 tmp

Ah ha! The parent directory is owned my _nginx:_nginx and is only writeable for that user. Our user ‘nobody’ therefore does not have the permissions to write in here. So, we can do a few things. Either make the entire directory writeable by everyone or change the ownership.

# chmod 777 /var/nginx/tmp

or

# chown nobody:_nginx /var/nginx/tmp

This should cure your permissions issues and all pages should load completely (at least mine do!)