Derek Neely

...notes for thyself, but useful for all...

Hosting with CherryPy


Apr 14, 2017 by derek


At work we've been working with GAE quite a bit for deploying apps for ourselves and customers. In addition we do a lot with Python/Django for these apps. 

As of recent when deploying our Django app out to GAE we've noticed sporadic 500/502s. By default GAE uses and 'prefers' to use gunicorn. Out of pure frustration we decided to switch up our WSGI server. I've used uWSGI for a good while when deploying a basic Nginx/Django site and always had good luck with it. However, getting uWSGI up and working out on GAE was a bit problematic for us. So, one of our engineers decided to try Waitress and CherryPy. His Waitress config appeared pretty straight forward and worked well. He then found a compairson of the two and it seemed to show CherryPy being the faster of the two, so we gave that a shot and seemed to be pretty quick. So, for now, we're sticking with it. However, the documentation for the 'latest' of CherryPy didn't seem to cover things very well when it came to the server configs. So, here are some of the configs we wound up using to get everything up and going. This is for both GAE and if you just want to run it directly on a server or a Docker container.

If you're running a Django site on GAE you just need to get the application up and running. No need for additional mapping for static files and such. With GAE your static files and such will be out on a CDN somewhere so no mapping is needed. 

server.py

from myapp.wsgi import application

import cherrypy

server_config = {
    'global': {
        'server.socket_host': '0.0.0.0', 
        'server.socket_port': 8080, 
        'server.thread_pool': 30, 
        'server.max_request_body_size': 10485760, # Limit body size to 10M in bytes
        'server.max_request_header_size': 512000, # Limit header size to 512K in bytes
        'server.socket_timeout': 60, 
        'log.screen': True,
        'log.error_file': '/tmp/myapp.error.log',
        'log.access_file': '/tmp/myapp.access.log',
        'environment': 'production',
    }
}

if __name__ == '__main__':
    cherrypy.server.unsubscribe()
    cherrypy.config.update(server_config)
    cherrypy.tree.graft(application, '/')
    cherrypy.server.subscribe()
    cherrypy.engine.start()
    cherrypy.engine.block()

Now in the app.yaml file you'll set your entry point to entrypoint: python3 server.py

So, that should set you straight for throwing the app out on GAE using CherryPy instead of the default gunicorn.

I however wanted to take this a little further and move my own site onto CherryPy. But, I don't host on GAE but instead on a GCE instance with Nginx proxying back to a uWSGI socket. But, when we start using a new tech at work I like to try and deploy it for myself to build my knowledge and confidence in the technology. So, I worked up 2 different means of hosting the site: Direct CherryPy server running out of its virtualenv and a Docker container. As of this point/post the site is running out of the Docker container.

First off though lets cover my my server.py

from derekneely.wsgi import application
from django.conf import settings

import cherrypy
import os

server_config = {
    'global': {
        'server.socket_host': '0.0.0.0',
        'server.socket_port': 8080,
        'server.thread_pool': 10,
        'log.screen': True,
        'log.error_file': '/tmp/derekneely.com.error.log',
        'log.access_file': '/tmp/derekneely.com.access.log',
        'environment': 'production',
    }
}

application_config = {
    '/': {
        'tools.staticdir.root': os.path.dirname(__file__)
    },
}

# OPTIONAL CONFIG FOR HOSTING STATIC FILES
static_config = {
    '/': {
        'tools.staticdir.on': True,
        'tools.staticdir.dir': settings.STATIC_ROOT,
        'tools.expires.on': True,
        'tools.expires.secs': 86400
    }
}

# OPTIONAL CONFIG FOR HOSTING STATIC FILES
media_config = {
    '/': {
        'tools.staticdir.on': True,
        'tools.staticdir.dir': settings.MEDIA_ROOT,
        'tools.expires.on': True,
        'tools.expires.secs': 86400
    }
}

if __name__ == '__main__':
    cherrypy.server.unsubscribe()
    cherrypy.config.update(server_config)

    cherrypy.tree.mount(None, '', application_config)
    cherrypy.tree.mount(None, '/static', static_config) # OPTIONAL
    cherrypy.tree.mount(None, '/media', media_config) # OPTIONAL

    cherrypy.tree.graft(application, '/')

    cherrypy.server.subscribe()
    cherrypy.engine.start()
    cherrypy.engine.block()

The items noted as # OPTIONAL are just that. I maintained this code in place so I had it as a reference for hosting static files through CherryPy. However, I put Nginx in front of the CherryPy server so I use Nginx to server up content out of /static and /media. Now with all that said and done we just need to fire up the server and get it to run when the server boots. This 'application' uses a virtualenv for all of its dependencies. And, I did not know this until just recent but, turns out you can run your application in its virtualenv without having to actually activate the virtualenv (i.e. env/bin/activate). You can just run the version of python that was installed with your virtualenv and it all magically works! In my case it was something like: /sites/derekneely.com/env/bin/python3.4 /sites/derekneely.com/server.py &

And to get this up and going at boot I did something like:

rc.local

/sites/derekneely.com/env/bin/python3.4 /sites/derekneely.com/server.py > /var/log/cherrypy.log

There's probably some other better fancier way of doing this using a startup script and such but for now this worked for me. My end goal was to get this moved over to a Docker image anyhow. Let's cover that now.

So, not much changed regarding my server.py. I did remove the static/media stuff because I'll use Nginx to do that for me. There are numerous Python/Django Docker examples out there and all most likely would need to be tweaked or tuned for your particular needs. But, below is the Dockerfile that worked for me to get this up and going.

FROM python:3.4-slim

RUN apt-get update && apt-get install -y \
      gcc \
      gettext \
      mysql-client libmysqlclient-dev \
      postgresql-client libpq-dev \
      sqlite3 \
      python3-pip \
      python3-dev \
      libpython3-dev \
      libtiff-dev \
      libjpeg-dev \
      zlib1g-dev \
      libfreetype6-dev \
      liblcms2-dev \
      libwebp-dev \
      tcl8.5-dev \
      tk8.5-dev \
      python-tk \
   && rm -rf /var/lib/apt/lists/*

ENV PYTHONPATH /usr/local/lib/python3.4/dist-packages/
ENV PATH $PYTHONPATH:$PATH

RUN /usr/bin/pip3 install --upgrade pip setuptools

ADD . /app
RUN /usr/bin/pip3 install -r /app/requirements.txt

CMD /usr/bin/python3.4 /app/server.py
EXPOSE 8080

Many of the items I'm installing were due to the various packages I have in requirements.txt. You may not need all of them and in fact I'm still working to tune it down myself.

After you build this particular image you can get it on over to your server and stand it up as such:

docker run --name derekneelycom \
    -d \
    --restart=always \
    -p 8081:8080 \
    -v /path/to/my/static:/app/static \
    -v /path/to/my/media:/app/media \
    derekneelycom

If you see any issues with any of these or have any additional questions feel free to ping me on Twitter @derekneely

 

Python, Docker, Nginx

Social


Tweet Tweet Tweet


© 2016 Derek Neely