Anyhow, as of recently its been tasked upon us to work with some streaming video. Now, the hardware we will be using for our client are some nice IP/H.264 compatible cameras. Well, I personally do not have one but needed a solution by which I could test and work with. I have one of those little eye ball looking cameras and found some free software (WebCamXP) that would run a simple MJPEG stream. (Yea, yea…its windows but who doesn’t have a windows box to hack and slash on? Ha!) After this is up and running you have a basic MJPEG stream that is running and accessible via http://10.0.0.99:8080/webcamp_1.cgi (or whatever your internal IP is and the port you configured in the software)

** Note: This tutorial uses iPhone references as thats what I worked with first. I’ll update with the Android version later. I am also using Linux (Ubuntu) to do the encoding and serving of the new video stream. Also assuming you have Apache or another web server up and running.

Now, this .cgi page/script ‘can’ be placed as the source of a UIWebview and works ok.

NSString *urlAddress = @"http://10.0.0.99:8080/cam_1.cgi";
NSURL *url = [NSURL URLWithString:urlAddress];
NSURLRequest *requestObj = [NSURLRequest requestWithURL:url];
[streamWebView loadRequest:requestObj];

But, ultimately this is not what we’re looking for. We need to get a nice H.264 compatible stream rolling out so that we can play this directly to the iPhone (MPMoivePlayerController) or be able to hook into a streaming server (as we’re planning on doing) so that we can re-broadcast to several devices at once.

So…how might we convert this MJPEG (.cgi) into an H.264 compatible stream? VLC Media Player to the rescue.

On your Linux box install VLC and the x264 libraries.

# apt-get install vlc x264

A few of the tutorials you read on this all reference using the command: vlc. However, if you don’t have X11 installed on your system (as I do not) you will run into a few errors with regards to this. But, have no fear, VLC provided a command line option: cvlc.

Now what we need to do is fire up VLC with a bunch of options that will read in the MJPEG (.cgi) convert it to H.264 and then provide us with a means of accessing the stream (via the Linux server – 10.0.0.69). So, on the command prompt do something like:

cvlc --intf=rc http://10.0.0.99:8080/cam_1.cgi --sout '#transcode{fps=25,
vcodec=h264,venc=x264{aud,profile=baseline,level=30,keyint=30,
bframes=0,ref=1,nocabac},acodec=mp3,ab=56,audio-sync,deinterlace}
:standard{mux=ts,access=http,dst=10.0.0.69:8090/myStream.mp4}'

Now, on the command prompt you’ll see a bunch of stuff start scrolling by. This is what you want. Its reading in frames from the original MJPEG stream and converting them. But, you’re not done yet. In order to get this to play via MPMoviePlayerController on the iPhone, we need to setup the .m3u8 or the playlist file.

In your web server directory create a file myStream.m3u8 and put the following in and save:

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10, 

http://10.0.0.69:8090/myStream.mp4

Now you’re ready for some iPhone code to show the stream.

NSURL *url = [NSURL URLWithString:@"http://10.0.0.69/myStream.m3u8"];
MPMoviePlayerController *player = [[MPMoviePlayerController alloc] initWithContentURL:url];
player.controlStyle = MPMovieControlStyleNone;
[player.view setFrame:CGRectMake(0, 120, 320 , 200)];
[self.view addSubview:player.view];
[player play];

This should basically do it. I don’t believe I left out any steps. I’ll update if I run across any missing steps or you guys let me know. When you run the app, it can sometimes take a few seconds before the video starts playing. At one point I set a background color to the mpmovieplayercontroller’s view so that I could tell where it was being

With MPMoviePlayerController there are several other options you may be interested in using/implementing.

When setting up the player you can have your containing view controller listen for some of the playback events (can help with some debugging) with:

//Defined in the above MPMoviePlayerController allocation
[[NSNotificationCenter defaultCenter] addObserver:self  selector:@selector(moviePlayerPlaybackStateDidChange:)  name:MPMoviePlayerPlaybackStateDidChangeNotification  object:nil];

//NSNotification callback function
- (void)moviePlayerPlaybackStateDidChange:(NSNotification*)notification {
     NSLog(@"playbackDidChanged");
     MPMoviePlayerController *moviePlayer = notification.object;

     MPMoviePlaybackState playbackState = moviePlayer.playbackState;

     if(playbackState == MPMoviePlaybackStateStopped) {
          NSLog(@"MPMoviePlaybackStateStopped");
     } else if(playbackState == MPMoviePlaybackStatePlaying) {
          NSLog(@"MPMoviePlaybackStatePlaying");
     } else if(playbackState == MPMoviePlaybackStatePaused) {
          NSLog(@"MPMoviePlaybackStatePaused");
     } else if(playbackState == MPMoviePlaybackStateInterrupted) {
          NSLog(@"MPMoviePlaybackStateInterrupted");
     } else if(playbackState == MPMoviePlaybackStateSeekingForward) {
          NSLog(@"MPMoviePlaybackStateSeekingForward");
     } else if(playbackState == MPMoviePlaybackStateSeekingBackward) {
          NSLog(@"MPMoviePlaybackStateSeekingBackward");
     }
}

Please let me know if I’ve left out any steps or you have any issues and I’ll be happy to help and work through it with you. This was a quick and dirty how-to with a ways to go before I push anything out to production. Just wanted to share my research and discovery. Good luck!

References:
Apple – Reference: MPMoviePlayerController
StackOverflow – HTTP stream without extension in MPMoviePlayerController

I noticed after installation when I would go to start plugging in some of my external devices via USB the dang thing would lock up on me. That is, desktop completely frozen, unable to remotely ssh into the box, just plain dead.

So, I tried various things and what have you and nothing appeared to help.

My solution was:

apt-get remove modemmanager

Simple as that. I haven’t had any lock-ups, system has been up for several days and I’ve been able to add my alfa wireless and my magic jack to the box without any crashes. Hope this may help you too!

Install Pure-FTP with MySQL and/or PostgreSQL support:

# > vi /etc/make.conf
– add ‘postgres’ and/or ‘mysql’ to the USE flag or:

#> USE=”mysql postgres” emerge -av net-ftp/pure-ftpd

Now that Pure-FTP is installed with the various database support, we can configure Pure-FTP to authenticate off of a database. You may need to tailor the SQL queries to match the schema of your database.

PostgreSQL:

#>  vi /etc/pureftpd-pgsql.conf

# If PostgreSQL listens to a TCP socket
PGSQLServer localhost
PGSQLPort 5432
# *or* if PostgreSQL can only be reached through a local Unix socket
# PGSQLServer /tmp
# PGSQLPort .s.PGSQL.5432
# Mandatory : user to bind the server as.
PGSQLUser [pureftpd]
# Mandatory : user password. You *must* have a password.
PGSQLPassword [pureftpd_password]
# Mandatory : database to open.
PGSQLDatabase [pureftpd_database]
# Mandatory : how passwords are stored
# Valid values are : “cleartext”, “crypt”, “md5″ and “any”
#PGSQLCrypt cleartext
PGSQLCrypt crypt

# In the following directives, parts of the strings are replaced at
# run-time before performing queries :
#
# \L is replaced by the login of the user trying to authenticate.
# \I is replaced by the IP address the user connected to.
# \P is replaced by the port number the user connected to.
# \R is replaced by the IP address the user connected from.
# \D is replaced by the remote IP address, as a long decimal number.
#
# Very complex queries can be performed using these substitution strings,
# especially for virtual hosting.
# Query to execute in order to fetch the password
PGSQLGetPW SELECT password FROM ftp_users WHERE ftp_user=’\L’
# Query to execute in order to fetch the system user name or uid
PGSQLGetUID SELECT uid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default UID – if set this overrides PGSQLGetUID
#PGSQLDefaultUID 1000
# Query to execute in order to fetch the system user group or gid
PGSQLGetGID SELECT gid FROM ftp_users WHERE ftp_user=’\L’
# Optional : default GID – if set this overrides PGSQLGetGID
#PGSQLDefaultGID 1000
# Query to execute in order to fetch the home directory
PGSQLGetDir SELECT dir FROM ftp_users WHERE ftp_user=’\L’
#########OPTIONAL SETTINGS#############
# Optional : query to get the maximal number of files
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTAFS SELECT QuotaFiles FROM users WHERE User=’\L’
# Optional : query to get the maximal disk usage (virtual quotas)
# The number should be in Megabytes.
# Pure-FTPd must have been compiled with virtual quotas support.
# PGSQLGetQTASZ SELECT QuotaSize FROM users WHERE User=’\L’
# Optional : ratios. The server has to be compiled with ratio support.
PGSQLGetRatioUL SELECT ul_ratio FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetRatioDL SELECT dl_ratio FROM ftp_users WHERE ftp_user=’\L’
# Optional : bandwidth throttling.
# The server has to be compiled with throttling support.
# Values are in KB/s .
PGSQLGetBandwidthUL SELECT ul_bandwidth FROM ftp_users WHERE ftp_user=’\L’
PGSQLGetBandwidthDL SELECT dl_bandwidth FROM ftp_users WHERE ftp_user=’\L’

Now we need to modify the pure-ftpd config file (keep in mind this is Gentoo)

#> vi /etc/conf.d/pure-ftpd

Look for the line: AUTH=”-l unix” and change to:

AUTH=”-l pgsql:/etc/pureftpd-pgsql.conf

#> /etc/init.d/pure-ftpd restart

This should conclude your intstallation of Pure-FTP with Postgres database support.

MySQL Config File:

Coming Soon!

A couple of little tweaks that I’ve had to use for some of the configurations.

If you want all users to go to the same directory and don’t have or want to store the directory information in the database you can change this line in the pureftpd-pgsql.conf:

PGSQLGetDir SELECT ‘/home/ftpdir’ FROM ftp_users WHERE ftp_user=’\L’

This article assumes that you have the proper kernel configuration to support bonded interfaces.

Device Driver => Network device support => <M> Bonding driver support

Load the necessary modules for bonding and install the necessary packages.

linux# modprobe bonding
linux# echo “bonding” >> /etc/modules.autoload.d/kernel-2.6
linux# emerge net-misc/ifenslave

Configure the interfaces: vi /etc/conf.d/net

config_eth0=( “null” )
config_eth1=( “null” )
slaves_bond0=”eth0 eth1″
config_bond0=( “10.10.10.69/24″ )
routes_bond0=( “default gw 10.10.10.1″ )

Start the interface and setup to start on boot.

linux# ln -s /etc/init.d/net.lo /etc/init.d/net.bond0
linux# rc-update add net.bond0 default
linux# /etc/init.d/net.bond0 start

If you have eth0 and/or eth1 setup to start on boot already you will need to delete those init scripts.
linux# rc-update del net.eth0; rc-update del net.eth1

Test

At my company we run about 12 vhosts over about 5-6 load balanced web servers. All of them were configured to log to our loganalysis server which is a pretty beefy machine. The logging all in all worked well with a few major exceptions listed below.

1) I used the directive:

LogSQLRequestIgnore .gif .jpg .css .ico .png .js

This directive is supposed to be used to ignore any pages ending with that extension. This did not work at all and I had to create a script to actually delete those before analyzing the logs. Bummer but not that big of a deal.

2) While working with the server to optimize the database, there were various times when I would need to restart the MySQL service and a few times I needed to reboot the server. During these periods of time, the web servers were unable to log to the database which brought them to their knees. The inability of the module to handle a database outage gracefully was a major deal breaker for us. I feel this issue is a result of intense disk IO when the database is down. The server is logging to its Apache logs, the the backup SQL logs, and to the Apache error logs for every failed request. This becomes emense with thousands of requests per second. Should this server die or needing maintenance would have ultimately brought our company to a hault.

Long story short, we’re scrapping mod_log_sql and going with an NFS mount out to all the web services which we can then parse and run statistics on using some custom scripts and/or AWStats or Splunk.

I chose Squid for just this function. It was extremely easy to setup and has worked like a charm! Below are some of the basic steps for setting up and using your Squid proxy.

Install Squid:

Gentoo:
#> emerge squid

Red Hat/CentOS:
#> yum install squid

Ubuntu/Debian:
#> apt-get install squid

OpenBSD:
#> export PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/4.4/packages/i386/
#> pkg_add squid

Basic Configuration for Transparent Proxy:

#> vi /etc/squid/squid.conf

Look for the sections listed below and modify accordingly. This is a sample of my ACLs and configurations outside of some of the defaults.

#/etc/squid/squid.conf
acl all src 0.0.0.0/0.0.0.0
acl trusted_hosts 192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 443         # https
acl Safe_ports port 21         # ftp
acl CONNECT method CONNECT

# Deny requests to unknown ports
http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports

http_access allow localhost
http_access allow trusted_hosts
http_access deny all

# Listening port (default 3128)
http_port 3128

# Visible Hostname (may not be needed but doesn’t hurt)
visible_hostname yourserverhostname

This is the basic configuration that should get you going. More advanced configuration options will be coming soon.

Start up Squid:

#> /etc/init.d/squid start

Set Squid to start on-boot (optional)

Gentoo:
#> rc-update add squid default

Red Hat/CentOS:
#> chkconfig squid on

Ubuntu/Debian:
#> update-rc.d squid defaults

OpenBSD:
#> vi /etc/rc.local
Code coming soon…

Configure your Browser:
This is an example Firefox setup. Preferences –> Advanced –> Network –> Connection –> Settings. Select ‘Manual proxy configuration:’ Set the HTTP Proxy: value to your Squid server’s IP (public or private depending on how you are going to use it and based on the ACLs above). Then select ‘Use this proxy server for all protocols’ if you want to go ahead and use this same proxy for all connections.  Setup any exceptions to not proxy (i.e. – locally connected servers). Here’s a screen shot of my configuration.

SSL Configuration:
Coming soon…

Advanced Configurations:
Coming soon…

What is SSH Proxying?
This is a means of setting up a Secure Shell (SSH) and then piping your various web requests across this pipe or tunnel.

I’ve got 2 different SSH Proxies that I use daily.

Web Traffic – SSH Tunnel/Proxy:

ssh -CqN -D 8080 [username]@[hostname]

For above tunnel I’m using the following:

-D: bind port – in this case 8080 locally
-C: enables compression
-q: quiet mode (suppresses any warnings)
-N: don’t execute any remote commands

The -CqN are just some bells and whistles I use for the connection but not required. Please see below on configuring your browser to use the newly established SSH Tunnel.

Various other traffic (IRC, VNC, Torrent, etc…) – SSH Port Forwarding

ssh -L 6667:irc.[hostname]:6667 [username]@[hostname]

In this example, I’m binding a local port (-L 6667) to a remote boxes port (6667) through the server I have SSH’ed into. You can also add some of the bells and whistles from the web proxy to this one as well. Please see below for using this port forward with and IRC client.

Configuring the Browser:
The general idea (for Firefox) is to go to: Preferences –> Advanced –> Network –> Connection –> Settings. Select ‘Manual proxy configuration’. Set SOCKS Host: localhost Port: 8080. Click OK/Save and you should be good to go.

Here’s a screen shot of my settings:

Shortened file path

RewriteCond %{HTTP_HOST}    (.*)
RewriteRule ^images/(.*) sites/%1/files/images/$1 [L]

I used this one with a multi-site Drupal setup where the image paths wound up being something such as http://[domain]/sites/[domain]/files/images/image.jpg. Afterwards the URL was: http://[domain]/images/image.jpg.

References and Resources:
Apache Rewrite Guide and Examples
mod_rewrite Cheat Sheet

bwm-ng home page: http://www.gropp.org/?id=projects&sub=bwm-ng

On the Mac it works great with a little application called GeekTool (will cover more later) with the following options:

/Users/derek/Applications/bwm-ng/bin/bwm-ng -o plain -c 1

sys-libs/libstdc++-v3
virtual/libstdc++

I’m not sure which exactly did it, but I installed both and was able to get netpbm installed. Whoohoo!