Derek Neely

Here at work we’ve recently been moving off of a Linux/IPTables firewall setup to an OpenBSD/PF firewall. Well, anyone who has worked with firewalls knows the fun time you have with getting FTP to work through them. This includes client and server side. Well, this is my little ‘how-to’ on doing just that with PF and PureFTP.

I originally started with this tutorial from OpenBSD and was a good starting point but I felt it left out a few things.

First off. The tutorial above uses the ftp-proxy server on the box to allow proxying down to the ftp server. However, I didn’t exactly see any benefit of using it and didn’t seem to make anything easier. If I’m completely misunderstanding the usage of it please contact me and help me out. I DO want to do things right.

Here’s the overall layout of what is to happen:

[Client] <—> [PF Firewall] <—> [FTP Server]

Now, a few observations I’ve made while working with the firewall and maybe a bit clearer for others.

Here’s a great explanation of the overall FTP process from Tech Republic. Here’s the overall communication ports for the 2 different modes (as the PF firewall sees it – at least in my experience).

Active FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)

-Outbound-
— Data Transfer —
Outgoing: > 30000 (FTP Server Interface)
Outgoing: > 30000 (External interface – FW)

Passive FTP:
-Inbound-
— Authentication/Commands —
Incoming:  21 (External interface – FW)
Outgoing: 21 (Internal interface – FW)
Incoming: 21 (FTP Server Interface)
— Data Transfer —
Incoming:  49000-51000 (External interface – FW)
Outgoing: 49000-50000 (Internal interface – FW)
Incoming: 49000-50000 (FTP Server Interface)

-Outbound-
Outgoing: 49000-50000 (FTP Server Interface)
Outgoing: 49000-50000 (External interface – FW)

Alright enough chat and on to the code. First we need to configure the ftp server. In the  below examples I’ll cover PureFTP and ProFTPd for just the primary pieces of passive port configuration (I’ll give a nice ProFTP advanced configuration in a post to come). We’ll also assume the default ‘listening’ port 21 of any standard FTP install/configuration. This is also on a Gentoo server so modify accordingly for your given distro.

PureFTPd:

MISC_OTHER=”[various options/flags] -p 49000:51000 -P [Public IP]“

ProFTPd:

PassivePorts 49000 51000
MasqueradeAddress [Public IP]

PF Firewall Rules: (Just the FTP Rules)

# Interface Definitions:
ftp_ext=”[Public IP]”
ftp_int=”[Private IP]”

# NAT Rules
nat on $ext_if from { $ftp_int } to any -> $ftp_ext

# Redirect Rules
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 21 -> $ftp_int port 21
rdr pass on $ext_if inet proto tcp from any to $ftp_ext port 49000:51000 -> $ftp_int

# Firewall Rules
pass out quick on $ext_if inet proto tcp from $ftp_ext to any port > 1024 keep state

pass out quick log on $int_if inet proto tcp from any to $ftp_int port 21 keep state tag FTP label “ftp”
pass out quick log on $int_if inet proto tcp from any to $ftp_int port 49000:51000 keep state tag FTP_PASV label “ftp-passive”

That should be all you need for the PF rules and configuration on the FTP servers to get everything working and passing through. Feel free to contact me should you have any additional fixes or have a good explanation of how the ftp-proxy works and benefits of. I’ve just gotta find the time to do some experimenting.

I recently upgraded to Snow Leopard (Mac OS X 10.6). I’m quite used to using MacFUSE to remotely mount (via SSH) development systems for ease of coding. However, I just noticed that MacFUSE now refuses to establish a connection to the remote server. So, here is a fix for just that.

1) Quit Macfusion

2) Open System Preferences and then open the MacFUSE pane. Check the “Show Beta Versions” box and click “Check For Updates”. Go ahead and update MacFUSE.

3) Open a terminal and copy and past the following:

rm /Applications/Macfusion.app/Contents/PlugIns/sshfs.mfplugin/Contents/Resources/sshnodelay.so

Enjoy!

Fix provided from: Racker Hacker

Pure-FTP is  a powerful ftp server. If you are like me however, you don’t like to create system accounts for each individual user. Well, this is where we are in luck! Pure-FTP has the ability to tie into both PostgreSQL and MySQL.  In this example I have tied it into a PostgreSQL database. I will also include the MySQL snippet of the configuration but it hasn’t been tested by me. Also note that this is a Gentoo based installation but the general configuration should be the same.

Install Pure-FTP with MySQL and/or PostgreSQL support:

# > vi /etc/make.conf
– add ‘postgres’ and/or ‘mysql’ to the USE flag or:

#> USE=”mysql postgres” emerge -av net-ftp/pure-ftpd

Now that Pure-FTP is installed with the various database support, we can configure Pure-FTP to authenticate off of a database. You may need to tailor the SQL queries to match the schema of your database.

PostgreSQL:

#>  vi /etc/pureftpd-pgsql.conf

# If PostgreSQL listens to a TCP socket
PGSQLServer localhost
PGSQLPort 5432
# *or* if PostgreSQL can only be reached through a local Unix socket
# PGSQLServer /tmp
# PGSQLPort .s.PGSQL.5432
# Mandatory : user to bind the server as.
PGSQLUser [pureftpd]
# Mandatory : user password. You *must* have a password.
PGSQLPassword [pureftpd_password]
# Mandatory : database to open.
PGSQLDatabase [pureftpd_database]
# Mandatory : how passwords are stored
# Valid values are : “cleartext”, “crypt”, “md5″ and “any”
#PGSQLCrypt cleartext
PGSQLCrypt crypt

# In the following directives, parts of the strings are replaced at
# run-time before performing queries :
#
# \L is replaced by the login of the user trying to authenticate.
# \I is replaced by the IP address the user connected to.
# \P is replaced by the port number the user connected to.
# \R is replaced by the IP address the user connected from.
# \D is replaced by the remote IP address, as a long decimal number.
#
# Very complex queries can be performed using these substitution strings,
# especially for virtual hosting.
# Query to execute in order to fetch the password
PGSQLGetPW SELECT password FROM ftp_users WHERE ftp_user=’\L’
# Query to execute in order to fetch the system user name or uid
PGSQLGetUID SELECT uid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default UID – if set this overrides PGSQLGetUID
#PGSQLDefaultUID 1000
# Query to execute in order to fetch the system user group or gid
PGSQLGetGID SELECT gid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default GID – if set this overrides PGSQLGetGID
#PGSQLDefaultGID 1000
# Query to execute in order to fetch the home directory
PGSQLGetDir SELECT dir FROM ftp_users WHERE ftp_user=’\L’
#########OPTIONAL SETTINGS#############
# Optional : query to get the maximal number of files
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTAFS SELECT QuotaFiles FROM users WHERE User=’\L’
# Optional : query to get the maximal disk usage (virtual quotas)
# The number should be in Megabytes.
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTASZ SELECT QuotaSize FROM users WHERE User=’\L’
# Optional : ratios. The server has to be compiled with ratio support.
PGSQLGetRatioUL SELECT ul_ratio FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetRatioDL SELECT dl_ratio FROM ftp_users WHERE ftp_user=’\L’
# Optional : bandwidth throttling.
# The server has to be compiled with throttling support.
# Values are in KB/s .
PGSQLGetBandwidthUL SELECT ul_bandwidth FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetBandwidthDL SELECT dl_bandwidth FROM ftp_users WHERE ftp_user=’\L’

Now we need to modify the pure-ftpd config file (keep in mind this is Gentoo)

#> vi /etc/conf.d/pure-ftpd

Look for the line: AUTH=”-l unix” and change to:

AUTH=”-l pgsql:/etc/pureftpd-pgsql.conf

#> /etc/init.d/pure-ftpd restart

This should conclude your intstallation of Pure-FTP with Postgres database support.

MySQL Config File:

Coming Soon!

A couple of little tweaks that I’ve had to use for some of the configurations.

If you want all users to go to the same directory and don’t have or want to store the directory information in the database you can change this line in the pureftpd-pgsql.conf:

PGSQLGetDir SELECT ‘/home/ftpdir’ FROM ftp_users WHERE ftp_user=’\L’

At my job we are moving to Nginx for the load balancing of our sites. Nginx is a very powerful load balancing/proxy server tool. It allows weighting, ssl acceleration, among other functionality while remaining light weight and easy to configure.

In preperation for a large web services launch, I began to analyze some logs and keep an eye on the system. I noticed one of the sites that we’ve already deployed was hammering our error messages in /var/log/nginx/error.log reading:

2009/06/23 12:38:22 [crit] 808#0: *724154 open() “/var/nginx/tmp/proxy_temp/4/83/0000002834″ failed (13: Permission denied) while reading upstream, client: XXX.XXX.XXX.XXX, server: xxx.host.com, request: “GET /dir/page.php”, upstream: “http://backendserverip/dir/page.php”, host: “host.com”, referrer: “http://referrer.com/apage.php”

Upon reviewing the site I noticed some (not all) of the pages were only partially loading. The issue is exactly what the log says. Permission denied = Permission issue.

Check your /etc/nginx/nginx.conf (OpenBSD) file for the user nginx processes will run as:

user  nobody;

Or, do:

# ps aux | grep “nginx: worker process” | awk ‘{print $1}’
nobody

In both cases you see that I’m running the nginx worker process as user nobody. Now we need to check our permissions on: /var/nginx/tmp/proxy_temp

# ls -l /var/nginx/tmp/ | grep proxy_temp
drwxrwx—  12 nobody  _nginx  512 Jun 23 13:10 proxy_temp

Looks good. The directory is owned by nobody and is writeable by both nobody and the group _nginx. What could the issue be? Lets move up a level and check the permissions.

# ls -l /var/nginx | grep tmp
drwx——  5 _nginx  _nginx  512 May  7 11:54 tmp

Ah ha! The parent directory is owned my _nginx:_nginx and is only writeable for that user. Our user ‘nobody’ therefore does not have the permissions to write in here. So, we can do a few things. Either make the entire directory writeable by everyone or change the ownership.

# chmod 777 /var/nginx/tmp

or

# chown nobody:_nginx /var/nginx/tmp

This should cure your permissions issues and all pages should load completely (at least mine do!)

Its often necessary to add bonding (NIC teaming – in windows world) to a linux config. This gives you 2 things: higher throughput on the interfaces and redundancy (if a card or switch happens to die). I made this post just as a quick reference for how to do such on a Gentoo system.

This article assumes that you have the proper kernel configuration to support bonded interfaces.

Device Driver => Network device support => <M> Bonding driver support

Load the necessary modules for bonding and install the necessary packages.

linux# modprobe bonding
linux# echo “bonding” >> /etc/modules.autoload.d/kernel-2.6
linux# emerge net-misc/ifenslave

Configure the interfaces: vi /etc/conf.d/net

config_eth0=( “null” )
config_eth1=( “null” )
slaves_bond0=”eth0 eth1″
config_bond0=( “10.10.10.69/24″ )
routes_bond0=( “default gw 10.10.10.1″ )

Start the interface and setup to start on boot.

linux# ln -s /etc/init.d/net.lo /etc/init.d/net.bond0
linux# rc-update add net.bond0 default
linux# /etc/init.d/net.bond0 start

If you have eth0 and/or eth1 setup to start on boot already you will need to delete those init scripts.
linux# rc-update del net.eth0; rc-update del net.eth1

Test

Constantly needing to edit files on remote systems? Find it tasking and annoying to edit localy and upload via ftp or ssh? Want to use your favorite editor to edit the files on the remote system rather than relying on the remote ‘vi’ or ‘nano’? MacFUSE and Macfusion can be a very powerful tool that can remedy all of the above. Below is a quick tutorial on how to setup and use it.

Download and install: MacFUSE and Macfusion

Enable Macfusion to run at startup and enable the menu item for quick mounting and remote file system access. Run Macfusion –> Preferences –> General –> ‘When I login, start’ –> ‘the macfusion agent’ and ‘the macfusion menu item’.

After reboot you will see a new Icon in your Menu Bar. If you select it, it will look something like…

Here you can see your various options and current mounted file systems. To setup a connection to a server, ‘Macfusion Menu Item’ –> ‘Open Configuration …’ –> ‘+’ –> Select connection type: SSHFS/FTPFS.

Configure the Connection Name, Hostname, User Name, Password.

Whalla! You should be all setup and able to edit and browse the files as if it was a local file system. Works great with your favorite editor (in this case TextMate).

TextMate tweak to prevent remote meta data files that can slowly make a mess out of the file system. Copy and paste the below line in the command line.

defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1

TextMate Manual on Saving Files. TextMate Manual

To view hidden files via finder add the following line on the command line.

defaults write com.apple.finder AppleShowAllFiles TRUE
killall Finder

Original Source: http://minimaldesign.net/articles/read/remote-textmate-projects

Part one of the OpenBSD Network Appliance is done. I’ve got all the hardware put together, everything is posting. RAM was seen. My biggest worry. A buddy of mine at work gave me 4 DIMMS of PC133 512MB. I was a little worried it wasn’t going to work. I thought the mo-bo used only PC100. Good news for me though! 2GB of RAM for this bad boy will be plenty!

Hardware Specs:

ASUS P3V4X Motherboard
Pentium Celeron 533MHz
2GB PC133 RAM (4 x 512MB)
6 NICs (3 x 100Mb 3COM – 3 x 100Mb Intel)
2 x 30GB Hard drives (RAID 1 intended)

Check out the gallery! Projects Home Page

A good friend of mine (Huggz) discovered a way of pulling your root password on your Mac. The oveall concept is that you are dumping the human readable text out of /dev/vm/sleepimage into another file in which you will find some interesting stuff including your root password. Kinda scary!

This takes a good while to dump all the content as that file can be upwards to 2GB. Search through the /var/vm/sleepimage-ascii file and you will see some goodies.

mac:~ me$ sudo su -
mac:~ root# cd /var/vm
mac:vm root# strings -n 4 sleepimage > sleepimage-ascii

I found my password all through this file. While most of these will be your password just on a line, and depending on your password, would be hard to know it was a password. However, one line I ran across blatenly says passwordXXXXusernameXXXX (line: 4186701 – for me). If you search throuh more you will see other lines that clearly displays hostname, username, password, home directory (lines: 7810286-7810293 – for me). If you keep digging you can find not only the login information for this particular machine but others for Samba mounts, ftp, web sites, etc. Good stuff!!!

There is also a more complete and detailed write up of other file dumps over at theInterW3bs.

Here at work we’ve setup a testing environment of our production systems. That is, we’re using this environment to setup a perfect world that we will migrate production into…eventually.

We’re using OpenBSD for our firewalls and I was having a heck of a time getting the carp interfaces to work. The failover worked fine between the 2 VM load balancer/firewalls but the IPs on the carp interfaces were unreachable. This is apparently because VMs don’t support arbitrary MAC addresses and carp doesn’t support manually setting the MAC address. Long story short, the interfaces that are using carp, enable that network in ‘Promiscuous Mode’ in the VM management. Works like a charm now!

[Host Name] –> Configuration Tab –> Networking –> Properties –> [Network Name] –> Edit… –> Security

Original Source

After some reading of consolidation options for Apache logs, I ran across mod_log_sql (we are hating spread) which will take Apache logs and log them off to a MySQL database. Sounded great! We could then run scripts to go through and parse the values and run statistics on. Twas perfect for our needs. RIGHT!!!

At my company we run about 12 vhosts over about 5-6 load balanced web servers. All of them were configured to log to our loganalysis server which is a pretty beefy machine. The logging all in all worked well with a few major exceptions listed below.

1) I used the directive:

LogSQLRequestIgnore .gif .jpg .css .ico .png .js

This directive is supposed to be used to ignore any pages ending with that extension. This did not work at all and I had to create a script to actually delete those before analyzing the logs. Bummer but not that big of a deal.

2) While working with the server to optimize the database, there were various times when I would need to restart the MySQL service and a few times I needed to reboot the server. During these periods of time, the web servers were unable to log to the database which brought them to their knees. The inability of the module to handle a database outage gracefully was a major deal breaker for us. I feel this issue is a result of intense disk IO when the database is down. The server is logging to its Apache logs, the the backup SQL logs, and to the Apache error logs for every failed request. This becomes emense with thousands of requests per second. Should this server die or needing maintenance would have ultimately brought our company to a hault.

Long story short, we’re scrapping mod_log_sql and going with an NFS mount out to all the web services which we can then parse and run statistics on using some custom scripts and/or AWStats or Splunk.