Nginx

Configuring Nginx to start on system boot:

# vi /etc/rc.local

# start nginx
if [ -x /usr/local/sbin/nginx ]; then
   echo -n ' nginx'; /usr/local/sbin/nginx
fi

Load balance configuration (one public ip -> 3 private servers):

# vi /etc/nginx/nginx.conf

user  nobody;
worker_processes  2;
worker_rlimit_nofile 10240;

error_log  /var/log/nginx/error.log;

events {
   worker_connections  8192;
}

http {

   upstream  site {
        server 192.168.1.100:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:80 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:80 max_fails=2 fail_timeout=15 weight=1;
   }

   upstream  site_ssl {
        server 192.168.1.100:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.101:443 max_fails=2 fail_timeout=15 weight=3;
        server 192.168.1.102:443 max_fails=2 fail_timeout=15 weight=1;
   }

   server {
        listen  AAA.BBB.CCC.DDD:80;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_pass http://site;
        }
   }

   server {
        listen  AAA.BBB.CCC.DDD:443;

        ssl on;
        ssl_certificate /etc/ssl/ssl.cert/site.com.crt;
        ssl_certificate_key /etc/ssl/ssl.key/site.com.key;

        server_name site.com;

        location / {
             access_log off;
             proxy_connect_timeout 15;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X_FORWARDED_PROTO https;
             proxy_pass http://site_ssl;
        }
   }
}

Should you have a chain file you need to add it to the end of the certificate file as such:

# cat /path/to/chain.file >> /etc/ssl/ssl.cert/site.com.crt

Packet Filter Firewall Rules:

# vi /etc/pf.conf

site_ext="AAA.BBB.CCC.DDD"
site_int1="192.168.1.100"
site_int2="192.168.1.101"
site_int3="192.168.1.102"

pass in quick log on $ext_if inet proto tcp from any to $site_ext port 80 keep state
pass in quick log on $ext_if inet proto tcp from any to $corp_ext port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int1 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int2 port 443 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 80 keep state
pass out quick log on $int_if inet proto tcp from any to $site_int3 port 443 keep state

Helpful Nginx server management commands:

Test Nginx Config file:

# nginx -t -c /etc/nginx/nginx.conf

Restart Nginx server:

# kill -HUP `cat /var/run/nginx.pid`

Stop Nginx server:

# kill -QUIT `cat /var/run/nginx.pid`

If your like me and want to keep tabs on the load balancing (as in ipvsadm for Linux ldirectord) you can add a label to the end of each of your firewall rules:

label “[some unique label for this rule]” (i.e. label “site-ext”)

Then using pftop -v label you will see:

RULE LABEL           PKTS     BYTES     STATES MAX  ACTION   DIR
....
16   site-ext        867991   630495K   11833       Pass     In  ...
17   site-ext-ssl    29427    13315230  847         Pass     In  ...
18   site-int-1      13761450 6842062   813483      Pass     Out ...
19   site-int-1-ssl  234678   12345     12344       Pass     Out ...
....

I originally started with this tutorial from OpenBSD and was a good starting point but I felt it left out a few things.

First off. The tutorial above uses the ftp-proxy server on the box to allow proxying down to the ftp server. However, I didn’t exactly see any benefit of using it and didn’t seem to make anything easier. If I’m completely misunderstanding the usage of it please contact me and help me out. I DO want to do things right.

Here’s the overall layout of what is to happen:

[Client] <—> [PF Firewall] <—> [FTP Server]

Now, a few observations I’ve made while working with the firewall and maybe a bit clearer for others.

Here’s a great explanation of the overall FTP process from Tech Republic. Here’s the overall communication ports for the 2 different modes (as the PF firewall sees it – at least in my experience).

Active FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)

-Outbound-
— Data Transfer —
Outgoing: > 30000 (FTP Server Interface)
Outgoing: > 30000 (External interface – FW)

Passive FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)
— Data Transfer —
Incoming:  49000-51000 (External interface – FW)
Outgoing: 49000-50000 (Internal interface – FW)
Incoming: 49000-50000 (FTP Server Interface)

-Outbound-
Outgoing: 49000-50000 (FTP Server Interface)
Outgoing: 49000-50000 (External interface – FW)

Alright enough chat and on to the code. First we need to configure the ftp server. In the  below examples I’ll cover PureFTP and ProFTPd for just the primary pieces of passive port configuration (I’ll give a nice ProFTP advanced configuration in a post to come). We’ll also assume the default ‘listening’ port 21 of any standard FTP install/configuration. This is also on a Gentoo server so modify accordingly for your given distro.

PureFTPd:

MISC_OTHER=”[various options/flags] -p 49000:51000 -P [Public IP]“

ProFTPd:

PassivePorts 49000 51000
MasqueradeAddress [Public IP]

PF Firewall Rules: (Just the FTP Rules)

# Interface Definitions:
ftp_ext=”[Public IP]”
ftp_int=”[Private IP]”

# NAT Rules
nat on $ext_if from { $ftp_int } to any -> $ftp_ext

# Redirect Rules
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 21 -> $ftp_int port 21
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 49000:51000 -> $ftp_int

# Firewall Rules
pass out quick on $ext_if inet proto tcp from $ftp_ext to any port > 1024 keep state

pass out quick log on $int_if inet proto tcp from any to $ftp_int port 21 keep state tag FTP label “ftp”
pass out quick log on $int_if inet proto tcp from any to $ftp_int port 49000:51000 keep state tag FTP_PASV label “ftp-passive”

That should be all you need for the PF rules and configuration on the FTP servers to get everything working and passing through. Feel free to contact me should you have any additional fixes or have a good explanation of how the ftp-proxy works and benefits of. I’ve just gotta find the time to do some experimenting.

Install Pure-FTP with MySQL and/or PostgreSQL support:

# > vi /etc/make.conf
– add ‘postgres’ and/or ‘mysql’ to the USE flag or:

#> USE=”mysql postgres” emerge -av net-ftp/pure-ftpd

Now that Pure-FTP is installed with the various database support, we can configure Pure-FTP to authenticate off of a database. You may need to tailor the SQL queries to match the schema of your database.

PostgreSQL:

#>  vi /etc/pureftpd-pgsql.conf

# If PostgreSQL listens to a TCP socket
PGSQLServer localhost
PGSQLPort 5432
# *or* if PostgreSQL can only be reached through a local Unix socket
# PGSQLServer /tmp
# PGSQLPort .s.PGSQL.5432
# Mandatory : user to bind the server as.
PGSQLUser [pureftpd]
# Mandatory : user password. You *must* have a password.
PGSQLPassword [pureftpd_password]
# Mandatory : database to open.
PGSQLDatabase [pureftpd_database]
# Mandatory : how passwords are stored
# Valid values are : “cleartext”, “crypt”, “md5″ and “any”
#PGSQLCrypt cleartext
PGSQLCrypt crypt

# In the following directives, parts of the strings are replaced at
# run-time before performing queries :
#
# \L is replaced by the login of the user trying to authenticate.
# \I is replaced by the IP address the user connected to.
# \P is replaced by the port number the user connected to.
# \R is replaced by the IP address the user connected from.
# \D is replaced by the remote IP address, as a long decimal number.
#
# Very complex queries can be performed using these substitution strings,
# especially for virtual hosting.
# Query to execute in order to fetch the password
PGSQLGetPW SELECT password FROM ftp_users WHERE ftp_user=’\L’
# Query to execute in order to fetch the system user name or uid
PGSQLGetUID SELECT uid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default UID – if set this overrides PGSQLGetUID
#PGSQLDefaultUID 1000
# Query to execute in order to fetch the system user group or gid
PGSQLGetGID SELECT gid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default GID – if set this overrides PGSQLGetGID
#PGSQLDefaultGID 1000
# Query to execute in order to fetch the home directory
PGSQLGetDir SELECT dir FROM ftp_users WHERE ftp_user=’\L’
#########OPTIONAL SETTINGS#############
# Optional : query to get the maximal number of files
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTAFS SELECT QuotaFiles FROM users WHERE User=’\L’
# Optional : query to get the maximal disk usage (virtual quotas)
# The number should be in Megabytes.
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTASZ SELECT QuotaSize FROM users WHERE User=’\L’
# Optional : ratios. The server has to be compiled with ratio support.
PGSQLGetRatioUL SELECT ul_ratio FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetRatioDL SELECT dl_ratio FROM ftp_users WHERE ftp_user=’\L’
# Optional : bandwidth throttling.
# The server has to be compiled with throttling support.
# Values are in KB/s .
PGSQLGetBandwidthUL SELECT ul_bandwidth FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetBandwidthDL SELECT dl_bandwidth FROM ftp_users WHERE ftp_user=’\L’

Now we need to modify the pure-ftpd config file (keep in mind this is Gentoo)

#> vi /etc/conf.d/pure-ftpd

Look for the line: AUTH=”-l unix” and change to:

AUTH=”-l pgsql:/etc/pureftpd-pgsql.conf

#> /etc/init.d/pure-ftpd restart

This should conclude your intstallation of Pure-FTP with Postgres database support.

MySQL Config File:

Coming Soon!

A couple of little tweaks that I’ve had to use for some of the configurations.

If you want all users to go to the same directory and don’t have or want to store the directory information in the database you can change this line in the pureftpd-pgsql.conf:

PGSQLGetDir SELECT ‘/home/ftpdir’ FROM ftp_users WHERE ftp_user=’\L’

In preperation for a large web services launch, I began to analyze some logs and keep an eye on the system. I noticed one of the sites that we’ve already deployed was hammering our error messages in /var/log/nginx/error.log reading:

2009/06/23 12:38:22 [crit] 808#0: *724154 open() “/var/nginx/tmp/proxy_temp/4/83/0000002834″ failed (13: Permission denied) while reading upstream, client: XXX.XXX.XXX.XXX, server: xxx.host.com, request: “GET /dir/page.php”, upstream: “http://backendserverip/dir/page.php”, host: “host.com”, referrer: “http://referrer.com/apage.php”

Upon reviewing the site I noticed some (not all) of the pages were only partially loading. The issue is exactly what the log says. Permission denied = Permission issue.

Check your /etc/nginx/nginx.conf (OpenBSD) file for the user nginx processes will run as:

user  nobody;

Or, do:

# ps aux | grep “nginx: worker process” | awk ‘{print $1}’
nobody

In both cases you see that I’m running the nginx worker process as user nobody. Now we need to check our permissions on: /var/nginx/tmp/proxy_temp

# ls -l /var/nginx/tmp/ | grep proxy_temp
drwxrwx—  12 nobody  _nginx  512 Jun 23 13:10 proxy_temp

Looks good. The directory is owned by nobody and is writeable by both nobody and the group _nginx. What could the issue be? Lets move up a level and check the permissions.

# ls -l /var/nginx | grep tmp
drwx——  5 _nginx  _nginx  512 May  7 11:54 tmp

Ah ha! The parent directory is owned my _nginx:_nginx and is only writeable for that user. Our user ‘nobody’ therefore does not have the permissions to write in here. So, we can do a few things. Either make the entire directory writeable by everyone or change the ownership.

# chmod 777 /var/nginx/tmp

or

# chown nobody:_nginx /var/nginx/tmp

This should cure your permissions issues and all pages should load completely (at least mine do!)

Hardware Specs:

ASUS P3V4X Motherboard
Pentium Celeron 533MHz
2GB PC133 RAM (4 x 512MB)
6 NICs (3 x 100Mb 3COM – 3 x 100Mb Intel)
2 x 30GB Hard drives (RAID 1 intended)

Check out the gallery! Projects Home Page

We’re using OpenBSD for our firewalls and I was having a heck of a time getting the carp interfaces to work. The failover worked fine between the 2 VM load balancer/firewalls but the IPs on the carp interfaces were unreachable. This is apparently because VMs don’t support arbitrary MAC addresses and carp doesn’t support manually setting the MAC address. Long story short, the interfaces that are using carp, enable that network in ‘Promiscuous Mode’ in the VM management. Works like a charm now!

[Host Name] –> Configuration Tab –> Networking –> Properties –> [Network Name] –> Edit… –> Security

Original Source

What is SSH Proxying?
This is a means of setting up a Secure Shell (SSH) and then piping your various web requests across this pipe or tunnel.

I’ve got 2 different SSH Proxies that I use daily.

Web Traffic – SSH Tunnel/Proxy:

ssh -CqN -D 8080 [username]@[hostname]

For above tunnel I’m using the following:

-D: bind port – in this case 8080 locally
-C: enables compression
-q: quiet mode (suppresses any warnings)
-N: don’t execute any remote commands

The -CqN are just some bells and whistles I use for the connection but not required. Please see below on configuring your browser to use the newly established SSH Tunnel.

Various other traffic (IRC, VNC, Torrent, etc…) – SSH Port Forwarding

ssh -L 6667:irc.[hostname]:6667 [username]@[hostname]

In this example, I’m binding a local port (-L 6667) to a remote boxes port (6667) through the server I have SSH’ed into. You can also add some of the bells and whistles from the web proxy to this one as well. Please see below for using this port forward with and IRC client.

Configuring the Browser:
The general idea (for Firefox) is to go to: Preferences –> Advanced –> Network –> Connection –> Settings. Select ‘Manual proxy configuration’. Set SOCKS Host: localhost Port: 8080. Click OK/Save and you should be good to go.

Here’s a screen shot of my settings:

bwm-ng home page: http://www.gropp.org/?id=projects&sub=bwm-ng

On the Mac it works great with a little application called GeekTool (will cover more later) with the following options:

/Users/derek/Applications/bwm-ng/bin/bwm-ng -o plain -c 1