Nginx – failed (13: Permission denied) while reading upstream
At my job we are moving to Nginx for the load balancing of our sites. Nginx is a very powerful load balancing/proxy server tool. It allows weighting, ssl acceleration, among other functionality while remaining light weight and easy to configure.
In preperation for a large web services launch, I began to analyze some logs and keep an eye on the system. I noticed one of the sites that we’ve already deployed was hammering our error messages in /var/log/nginx/error.log reading:
2009/06/23 12:38:22 [crit] 808#0: *724154 open() “/var/nginx/tmp/proxy_temp/4/83/0000002834″ failed (13: Permission denied) while reading upstream, client: XXX.XXX.XXX.XXX, server: xxx.host.com, request: “GET /dir/page.php”, upstream: “http://backendserverip/dir/page.php”, host: “host.com”, referrer: “http://referrer.com/apage.php”
Upon reviewing the site I noticed some (not all) of the pages were only partially loading. The issue is exactly what the log says. Permission denied = Permission issue.
Check your /etc/nginx/nginx.conf (OpenBSD) file for the user nginx processes will run as:
user nobody;
Or, do:
# ps aux | grep “nginx: worker process” | awk ‘{print $1}’
nobody
In both cases you see that I’m running the nginx worker process as user nobody. Now we need to check our permissions on: /var/nginx/tmp/proxy_temp
# ls -l /var/nginx/tmp/ | grep proxy_temp
drwxrwx— 12 nobody _nginx 512 Jun 23 13:10 proxy_temp
Looks good. The directory is owned by nobody and is writeable by both nobody and the group _nginx. What could the issue be? Lets move up a level and check the permissions.
# ls -l /var/nginx | grep tmp
drwx—— 5 _nginx _nginx 512 May 7 11:54 tmp
Ah ha! The parent directory is owned my _nginx:_nginx and is only writeable for that user. Our user ‘nobody’ therefore does not have the permissions to write in here. So, we can do a few things. Either make the entire directory writeable by everyone or change the ownership.
# chmod 777 /var/nginx/tmp
or
# chown nobody:_nginx /var/nginx/tmp
This should cure your permissions issues and all pages should load completely (at least mine do!)
Its often necessary to add bonding (NIC teaming – in windows world) to a linux config. This gives you 2 things: higher throughput on the interfaces and redundancy (if a card or switch happens to die). I made this post just as a quick reference for how to do such on a Gentoo system.
Constantly needing to edit files on remote systems? Find it tasking and annoying to edit localy and upload via ftp or ssh? Want to use your favorite editor to edit the files on the remote system rather than relying on the remote ‘vi’ or ‘nano’? MacFUSE and Macfusion can be a very powerful tool that can remedy all of the above. Below is a quick tutorial on how to setup and use it.
Part one of the OpenBSD Network Appliance is done. I’ve got all the hardware put together, everything is posting. RAM was seen. My biggest worry. A buddy of mine at work gave me 4 DIMMS of PC133 512MB. I was a little worried it wasn’t going to work. I thought the mo-bo used only PC100. Good news for me though! 2GB of RAM for this bad boy will be plenty!
A good friend of mine (