Getting Started

I have a domain name and an idea for a web application. This is not new, I have had them for a few months now. Since I have very little time to devote to personal projects and abundant initiative has not been my strong point as of late, the idea has not moved past that starting point. This is the main reason that I have started this blog. It is intended to keep the initiative going, document progress and share my experience in developing a web application.

Purpose
My day-to-day job involves working with, supporting and administering large enterprise applications. I rarely get the opportunity to work with the technology that I both admire and enjoy working with.

I’m referring to OSS or, Open Source Software.

I am also a manager and, increasingly, my day takes me further away from dealing directly with technology and this is my attempt to dive back in.

My rules for this project are simple, it can only contain OSS (and “cloud” solutions on the periphery) and as little of the tools I use daily at my existing job.

Caveat
I will be keeping both the site and idea private for now. This is not to prevent others from using it (as it is not that original) but to build suspense and provide a way out if efforts come to nought.

Technology
Enough pre-amble, on to business. This is a web application (I know, original huh?). It will require a domain name, hosting, a language/framework (backend and frontend), a database platform and version control.

As I am a regular reader of Hacker News, I’ve tried to select technologies that are both established and bleeding edge. Some may change as development progresses but, for now, this is what I’ve selected:

Hosting
I have evaluated a bunch of hosting options. From Google App Engine (GAE) and Amazon AWS/EC2 to Slicehost and Linode and eventually settled on Linode.

GAE and AWS are both excellent options and may be worth pursuing in the unlikely event that scalability becomes an issue later on, but the control and cost of VPS hosting options like Slicehost and Linode was the tipping point. Although the pricing models for Slicehost and Linode are similar, the latter was a few bucks cheaper per month which, given the nature of the project, was important.

On Linode, I plan on using a Ubuntu LTS VPS and crafting a Linode StackScript to both ease and standardize server deployment.

Language/Framework
This is a loaded conversation and one that I will expand on in later posts. As each selection also involves additional plugins and/or packages I will most likely have separate posts for each. For now, here is a brief rundown:

Backend – Python/Django
Front-end – HTML5/CoffeeScript/jQuery

Database Platform
I use relational database management systems all day, every day. Boring! This decision rested solely on my rule to avoid technology that I use daily and experiment with new and different tools. It may not work out, but I’m going to start with MongoDB and MongoEngine for Python/Django support.

More on this in a later post.

Version Control
This one is a no-brainer. All the cool kids are using Git, so I will jump right in.

That’s my starting point. Tomorrow, I will begin working on the StackScript to deploy my Linode. My post will contain a breakdown of all of the tools I plan on deploying on the server and links to source material that helped guide the way.

Hosting on Linode

Starting with yesterday’s post, I listed a number of requirements for getting a web application up and running. The first is finding a hosting provider.

As I mentioned, I compared a number of providers such as Google App Engine and Amazon Web Services to Virtual Private Server (VPS) providers such as Slicehost and Linode.

Due to cost, ability to configure any way you like and some fairly strong recommendations from other sources, I settled on Linode. I originally signed up with them over 7 months ago to handle this project and through fits and starts have redeployed a number of VPS’s with them in that time. Their dashboard is excellent, the StackScript functionality is great, and their uptime has been fantastic.

I should mention that I’ve chosen to house my VPS in their London hosting facility. This decision was made due to issues that users were reporting in their US hosting facilities that did impact uptime.

Deployment
There are a number of hosting options provided with Linode. For the sake of price, I have opted for the cheapest hosting option they provide — The Linode 512. This is offered for $19.95/mth and hardware, storage and bandwidth allocation is more than adequate for development purposes.

Linode (and I’m sure others) provides the ability to customize server deployments through StackScripts. Once you have established the hardware specifications through the plan you selected, you now have the ability to choose a particular Linux distribution. For this project, I am going to use a Ubuntu LTS release which is currently Ubuntu 10.04 LTS 64-bit.

StackScript
Once my StackScript is done, I will hopefully remember to update this post with a link to the published version.

As the root user is created as you are deploying your VPS, the StackScript will run under that user. From a unix security standpoint, it is highly recommended that you create an alternate user and deny SSH capabilities to the root user. First, add the alternate user to the sudoers file and then disable root login. More on that later.

It is helpful to include the first published StackScript. It contains a number of useful functions that you can call that will help setup your VPS. I will refer to those functions throughout this post.

source  # StackScript Bash Library
source  # Excellent helper script by nigma

system_update
goodstuff

In order for the installations below to complete, the universe source must be added to aptitude.

sed -i 's/^#\(.*deb.*\) universe/\1 universe/' /etc/apt/sources.list

Let’s start with some essential Linux tools. Since the StackScript is running as root, you will notice that sudo is conspicuously absent during most of the aptitude commands. It has to be installed first!

apt-get -y install build-essential gcc sudo screen htop locate logcheck logcheck-database logwatch

For security, the excellent fail2ban and Uncomplicated Firewall provide a strong foundation for analyzing web traffic and simplifying the firewall configuration on a Linux server. For a more in-depth source on securing a Ubuntu server, check out this article.

apt-get -y fail2ban ufw

Since Python is the language of choice, a number of additional packages and dependencies are required both for Django and other packages that will be installed. For instance, the software-properties dependency is a requirement for git. I am also favouring the pip package manager over easy_install as it will automatically handle dependency installation.

apt-get -y install python python-dev python-setuptools python-software-properties python-pip
...
pip install --upgrade pip
pip install --upgrade virtualenv

Mercurial is required for the Django non-relational database release (django-nonrel) so add it for the hg commands to work.

apt-get -y install mercurial meld

To compartmentalize the Django environment, I am going to deploy all of the python packages (except pip and virtualenv itself) in a virtualenv. The benefit to this approach is that it will enable testing different versions easier as each Django project can contain separate package versions.

Memcached is intended to alleviate some of the database dependencies for dynamic web applications. I am going to deploy it using apt and pip rather than the manual download and extraction process.

apt-get -y install memcached
...
pip install python-memcached

Rather than stick with the standard Apache/mod-wsgi route for hosting Django, I thought I would check out Nginx and Green Unicorn (gunicorn) as an alternative.

apt-get -y install nginx
...
pip install gunicorn

I have setup Nginx and gunicorn using the information available in this post. The only addition I made was based on re-loading project files when they have been modified.

This post recommended added a pid file argument to the gunicorn_django call in the script. So, this is the full shell script that I am using:

#!/bin/bash
set -e
cd /path/to/project
source ../bin/activate
exec /path/to/bin/gunicorn_django --workers 2 --log-level=debug --log-file=/var/log/gunicorn/site.log --pid=/tmp/gunicorn.pid --daemon

The –daemon argument is essential for Upstart to monitor the service, and the –pid argument allows you to reload the application via the following command:

sudo kill -HUP `cat /tmp/gunicorn.pid`

Version control is essential to any project. As I mentioned in my first post, I am opting for Git as it is what all the cool kids are using these days. Instructions on installing Git on Ubuntu is available in the official documentation).

apt-get -y install git-core gitosis

Now onto Django. As in the memcached example, a version for Django is specified as part of the StackScript deployment. This is to ensure that any future servers will always have the same version.

Since I am using the django-nonrel release, I am following these instructions on the installation of django-nonrel, djangotoolbox and the mongodb-engine packages.

hg clone http://bitbucket.org/wkornewald/django-nonrel
cd django-nonrel && python setup.py install

hg clone http://bitbucket.org/wkornewald/djangotoolbox
cd djangotoolbox && python setup.py install

git clone https://github.com/django-mongodb-engine/mongodb-engine
cd mongodb-engine && python setup.py install

One of the reasons I selected Django over other Python web frameworks (such as Flask, Bottle etc.) is the abundant packages that are available to extend core functionality. Walking through excellent resources such as Django Packages and others), I have opted to add the following as they will hopefully speed up the development process.

pip install south
pip install geopy
pip install django-social-auth
pip install django-easy-maps
pip install django-annoying
pip install PyCrypto
pip install Fabric

I ran into issues with django-debug-toolbar on pages not defined in the urls.py file. This was addressed in the latest version (0.9.0-dev at the time of writing) so rather than use pip, I cloned it from git.

git clone https://github.com/django-debug-toolbar/django-debug-toolbar
cd django-debug-toolbar && python setup.py install

I will elaborate on these specific packages in a later post.

Now on to the database components. As I said in my first post, I am going to try and use MongoDB with Django. This is probably one of the riskiest decisions that I have made for this project as I am not able to find a lot of success success stories on making Django and MongoDB work well together. I am particularly concerned about the potential loss of the admin tools in Django as it is one of the most compelling reasons for choosing it in the first place.

apt-get -y install mongodb

Clean up
I have focused mainly on the packages that I am installing as part of the VPS build. However, as I stated earlier, there are a lot of other steps to take when building a server that are essential.

Here are some suggestions (once I publish the StackScript, it will contain code to do most of the following):

  • Setup bash as the default shell
  • Set a hostname, proper timezone and locales on the server
  • Create an additional user and add them to the sudoers file
  • Disable root login via SSH
  • Use the Uncomplicated Firewall (ufw) to only allow the absolutely essential ports (80, 443, 22) and make sure the firewall is enabled
  • Finally, always update and patch your essential server components!

In my next post I am going to dive into the Nginx, gunicorn, Django and memcached configuration before starting to work on the project.

Using Git and Setting up Django

Now that the server is setup and Nginx is handling inbound requests and proxying them to gunicorn/Django it is high time to setup the first git repository and start configuring your Django environment.

Git
I have been using SVN for years and possess a passing familiarity with CVS but I decided to give Git a shot for this particular project. As I mentioned earlier, all the cool kids are using it and more often than not, I find myself browsing through OSS repositories on there all the time.

It is fairly straightforward to get going. Once I setup a paid account (the $7/mth plan), I followed the instructions on how to create the SSH key, setup Git on the server and create my first repository. Once this was done, I committed the root of the Django project folder (with README) into the repository.

Now onto configuring Django.

django-admin.py
I noticed fairly early on that the django-admin.py command was not working correctly. I did some searching around and found the following guide which had a helpful suggestion about adding the DJANGO_SETTINGS_MODULE environment variable to the activate script for virtualenv.

I ran the following commands in the project folder (outside of the virtualenv, replace myproject with your project name) to add this environment variable:

ln -s `pwd` ../lib/python2.6/site-packages/`basename \`pwd\``
export DJANGO_SETTINGS_MODULE=myproject.settings
echo "!!" >> ../bin/activate

Database
As I said in my first post, I am going to try and use MongoDB as the backend database for Django using django-nonrel, djangotoolbox and mongodb-engine.

Using this helpful tutorial as a guide,

Memcached
Using the helpful information provided in this post, I have added the following to the Django settings.py file:

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

And in the middleware classes section (at the top and bottom):

'django.middleware.cache.UpdateCacheMiddleware',
...
'django.middleware.cache.FetchFromCacheMiddleware',

South
Update: It looks like South does not work with either MongoDB or django-nonrel and may not be required anyway. As soon as I add it to the INSTALLED_APPS section Django will not start.

South improves on the shipped Django schema migration capabilities. Again, with MongoDB, I’m not sure how this particular component will work.

According to the installation guide, you just need to add ‘south’ to the INSTALLED_APPS section (at the very bottom).

django-debug-toolbar
Once I installed the 0.9.0-dev release, everything went fine. There is an issue with 0.8.5 wherein in will display an error on pages that are not defined in the urls.py file.

The django-debug-toolbar is an excellent tool for Django developers. It will display helpful information to a specified address (in the browser) that will help in debugging issues during development.

You need to add the following to the middleware section in your settings.py file:

'debug_toolbar.middleware.DebugToolbarMiddleware',

Just make sure that the memcached FetchFromCacheMiddleware class is left at the bottom of the list.

Then specify an INTERNAL_IPS directive that includes the IP addresses for all of the machines that you are using for development.

Finally, add ‘debug_toolbar’ to the INSTALLED_APPS section.

django-social-auth
The Django-SocialAuth plugin enables authentication via third party providers such as Facebook, Google, Twitter etc.

‘social_auth’ needs to be added to the INSTALLED_APPS section in settings.py.

AUTHENTICATION_BACKENDS need to be specified. I added all of the examples in the install documentation:

AUTHENTICATION_BACKENDS = (
    'social_auth.backends.twitter.TwitterBackend',
    'social_auth.backends.facebook.FacebookBackend',
    'social_auth.backends.google.GoogleOAuthBackend',
    'social_auth.backends.google.GoogleOAuth2Backend',
    'social_auth.backends.google.GoogleBackend',
    'social_auth.backends.yahoo.YahooBackend',
    'social_auth.backends.contrib.linkedin.LinkedinBackend',
    #'social_auth.backends.contrib.LiveJournalBackend',
    'social_auth.backends.contrib.orkut.OrkutBackend',
    'social_auth.backends.contrib.FoursquareBackend',
    'social_auth.backends.OpenIDBackend',
    'django.contrib.auth.backends.ModelBackend',
)

Watch out for issues in the documentation. I have to leave the LiveJournalBackend line commented out and remove the orkut reference in the FoursquareBackend line.

Again, using the documentation, I have added the following to the settings.py file as well (with the keys specified):

TWITTER_CONSUMER_KEY   = ''
TWITTER_CONSUMER_SECRET      = ''
FACEBOOK_APP_ID              = ''
FACEBOOK_API_SECRET          = ''
LINKEDIN_CONSUMER_KEY        = ''
LINKEDIN_CONSUMER_SECRET     = ''
ORKUT_CONSUMER_KEY           = ''
ORKUT_CONSUMER_SECRET        = ''
GOOGLE_CONSUMER_KEY          = ''
GOOGLE_CONSUMER_SECRET       = ''
GOOGLE_OAUTH2_CLIENT_KEY     = ''
GOOGLE_OAUTH2_CLIENT_SECRET  = ''
FOURSQUARE_CONSUMER_KEY      = ''
FOURSQUARE_CONSUMER_SECRET   = ''

LOGIN_URL          = '/login-form/'
LOGIN_REDIRECT_URL = '/logged-in/'
LOGIN_ERROR_URL    = '/login-error/'

SOCIAL_AUTH_ERROR_KEY = 'social_errors'
SOCIAL_AUTH_COMPLETE_URL_NAME  = 'complete'
SOCIAL_AUTH_ASSOCIATE_URL_NAME = 'associate_complete'
SOCIAL_AUTH_DEFAULT_USERNAME = 'new_social_auth_user'
SOCIAL_AUTH_EXTRA_DATA = False
SOCIAL_AUTH_EXPIRATION = 'expires'

Finally, add the following to the urls.py file:

url(r'', include('social_auth.urls')),

The Social_Auth section in the Django admin will fail with the following message when trying to access Associations, Nonces or User social auths.

Exception Type:	TemplateSyntaxError
Exception Value:
Caught DatabaseError while rendering: This query is not supported by the database.
Exception Location:	build/bdist.linux-i686/egg/djangotoolbox/db/basecompiler.py in check_query, line 272

Not surprising as it is most likely attempting a join behind the scenes. I have forked the social_auth repository on github for fun to see if I can figure it out. For now, I will leave it enabled to see if authentication works anyway.

django-easy-maps
django-easy-maps makes deploying Google Maps much easier in Django projects. As I plan on using them in this web application, I thought I would check out this plugin.

According to the documentation you need to add ‘easy_maps’ to INSTALLED_APPS and then specify your Google key in EASY_MAPS_GOOGLE_KEY in settings.py.

Admin, views.py and urls.py
Since we are using MongoDB, the SITE_ID in settings.py must be an ObjectID string or else the following error is issued:

AutoField (default primary key) values must be strings representing an ObjectId on MongoDB (got u’1′ instead). Please make sure your SITE_ID contains a valid ObjectId string.

I added the following to the INSTALLED_APPS sections:

'django_mongodb_engine',
'djangotoolbox',

Adding djangotoolbox here is not just recommended it is required if you plan on creating or editing users in the Django admin panel. The following command is supposed to return the ObjectID value:

python manage.py tellsiteid

Unfortunately, this did not work for me so I went over into the MongoDB engine and found the ObjectID with the following (dbname is the one specified in settings.py):

/usr/bin/mongo dbname
db.django_site.find()

This will return on ObjectID which you can add to your settings.py file by commenting out the existing value and replace it with:

SITE_ID = u'OBJECTIDVALUE'

Now, you will need to enable the admin module, create your first views.py file and make a few more changes to the urls.py file to have a working site.

To enable the admin module uncomment the following in settings.py:

'django.contrib.admin',

And uncomment the following in urls.py:

from django.contrib import admin
admin.autodiscover()
...
url(r'^admin/', include(admin.site.urls)),

Now, create a views.py file in the root of your project folder and add the following:

from django import forms
from django.shortcuts import render_to_response
from django.http import HttpResponse
from django.http import HttpResponseRedirect
from django.http import Http404
from django.core.mail import send_mail
from django.contrib import auth
from django.contrib.auth.decorators import login_required
from django.template import RequestContext
from django.core.urlresolvers import reverse

def home(request):
    return render_to_response('home.html', locals(), context_instance=RequestContext(request))

render_to_response is a Django shortcut which will render a defined template (home.html) and pass in context variables (locals() and RequestContext(request)). This is why I included the import statements above.

Now, where does it find home.html? You need to specify a templates folder in settings.py in the TEMPLATE_DIRS section. Make sure you use a fully qualified path (without trailing space). In this case, I created a templates folder under the project path (in the same folder as urls.py, views.py and settings.py). Create a home.html file in that folder.

That’s it! Reload gunicorn using the following command:

kill -HUP `cat /tmp/gunicorn.pid`

You should now see a functioning page!

Commit and syncdb
Finally, commit the recent changes to the Git repository and synchronize the Django database.

git add -A
git commit -m 'Enter comment here'

In order to run the syncdb command, you need to be in the virtualenv for the path to be setup correctly. So:

source bin/activate
python manage.py syncdb

Updates, Theme, and Authentication

Just a quick post since I have not had a chance to update in a couple of weeks.

Work has progressed, but a lot of bumps along the way. Particularly, with the decision to use MongoDB.

So, I’m going to go back and modify my earlier postings to reflect this particular change and others. Here is a quick summary:

Update
The excellent django-social-auth uses JOINs in the admin module so it will not work with MongoDB and django-nonrel. I intended to fork the project on github and see if I could make the necessary changes but I did not want to get off-track with the project.

It is on my list of to-dos for later.

Also, a lot of the django admin jobs (syncdb, validate etc.) were not working correctly. They were returning a lot of error messages and I have not even defined a model.

So, now I not abandoning MongoDB. Instead, I am going to try using the multiple database support in django. I will keep the user, groups, social-auth (and other packages) in SQLite and try to use MongoDB for everything I add to the project.

django-debug-toolbar
I was hellbent on using this tool but the current stable branch had a known issue with static pages and django errors were a plenty. I have updated the StackScript to download the latest dev version and the errors have disappeared.

There is probably more but I will leave it at that for now.

Theme
If you are anything like me (and not a designer) you end up spinning endlessly on the look and feel of a site. I have developed a number of other sites but the overall aesthetic and UX were never as good as I hoped.

Scratch all that. I hopped on over to themeforest and found a pretty slick theme for the site. I went in with a fairly well-formed idea on what I was looking for and the past week or so of using it has gone well.

My goal is to modify the theme as little as possible in the beginning in order to avoid going down the rabbit hole. It looks better than anything I could pull off on my own so I am going to leave it at that. The goal is functional site and a completed project not endless tweaking and frustration (sorry designers!).

django-social-auth
What a wonderful django package! It took a few days of tinkering to get all of the client keys/secrets etc. for the various social authentication sites (Twitter, Facebook, Google, LinkedIn) and figure out the example application, but it does what it says on the tin.

For anyone starting out with this package, start with the example application as it will get you up and running much faster.

That’s it for now! I’m working on a the multiple database support in django now and building out the models to attach to a few forms. I am hoping to have a site that actually does something within the next few days.

I know this post is a bit weak but I did not want to drop this along the way.

GridFS, Nginx and Python Imaging Library

After spending quite of bit of time on the server setup and beginning development, I realized that I missed a key component for the some of the basic site functionality.

Image uploads.

MongoDB provides this support through GridFS which is a specification for storing large objects in the database. While benchmark results show that serving images directly from the file system is faster, I decided to give it a shot anyway.

Also, there is an Nginx module that will allow you to serve content directly from GridFS.

In a previous post, I used aptitude to install Nginx from the universe repository. Since the addition of the nginx-gridfs module required recompiling Nginx, I decided to upgrade to the latest stable Nginx release (1.0.5).

It was a bit trickier than I expected, but this is my notes on compiling Nginx on Ubuntu 10.04 LTS and trying to mimic the install as it is delivered from the Ubuntu repositories (with some minor improvements).

In my earlier post I covered a lot of the initial server build that included most of the essential build components for compiling from source. The only addition is the PCRE libraries.

apt-get -y install libpcre3 libpcre3-dev

A few folders need to be manually created prior to the install. I would have assumed that they would be created during as they are specified in the configure statement, but they are not.

mkdir /var/lib/nginx
mkdir /var/lib/nginx/body
mkdir /var/lib/nginx/proxy
mkdir /var/lib/nginx/fastcgi

Now, grab the nginx-gridfs module. It is needed prior to compiling Nginx.

cd /usr/local/src/
git clone https://github.com/mdirolf/nginx-gridfs/
cd nginx-gridfs
git submodule init
git submodule update

That should leave you with a folder in /usr/local/src/ that contains the nginx-gridfs module. You will use this path during the configure step.

Let’s start the Nginx compilation process:

wget http://nginx.org/download/nginx-1.0.5.tar.gz
tar xzvf nginx-1.0.5.tar.gz
cd ../nginx-1.0.5
./configure \
--sbin-path=/usr/sbin \
--conf-path=/etc/nginx/nginx.conf \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--http-client-body-temp-path=/var/lib/nginx/body \
--http-proxy-temp-path=/var/lib/nginx/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
--with-debug \
--with-http_flv_module \
--with-http_ssl_module \
--with-http_dav_module \
--with-http_gzip_static_module \
--with-http_realip_module \
--with-mail \
--with-mail_ssl_module \
--with-ipv6 \
--add-module=/usr/local/src/nginx-gridfs/
make && make install

You should now have a working copy of Nginx in the /etc/nginx path and binary in /usr/sbin.

Part of the Nginx install from Ubuntu includes a pretty decent tie-in with the services management functionality in the OS. The base compilation of Nginx does not include any of this, so you need to add it.

There is an excellent init script on Github provided by JasonGiedymin. A few of the parameters need to be modified as the file and folder locations are different. Since these steps have been taken from my Linode StackScript, I am using the sed utility to make these changes.

cd /usr/local/src/
git clone https://github.com/JasonGiedymin/nginx-init-ubuntu/
cp ./nginx-init-ubuntu/nginx /etc/init.d/
sed -i 's#NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf#NGINX_CONF_FILE="/etc/nginx/nginx.conf#g' /etc/init.d/nginx
sed -i 's#DAEMON=/usr/local/sbin/nginx#DAEMON=/usr/sbin/nginx#g' /etc/init.d/nginx
sed -i 's#lockfile=/var/lock/subsys/nginx#lockfile=var/lock/nginx.lock#g' /etc/init.d/nginx
chmod +x /etc/init.d/nginx
/usr/sbin/update-rc.d -f nginx defaults

Now, we can easily start/stop Nginx as well as a number of other helpful commands.

service nginx start

Since it is installed, we need to configure it to work with gunicorn and Django.

mkdir /etc/nginx/conf.d
mkdir /etc/nginx/sites-available
mkdir /etc/nginx/sites-enabled
mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf-bak
touch /etc/nginx/nginx.conf

This is the Nginx conf file, located in /etc/nginx/nginx.conf. You will notice that a lot of the paths and files used in the configure statement above are used in this file.

user www-data;
worker_processes  1;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    access_log 	/var/log/nginx/access.log;
    sendfile        on;
    tcp_nopush     on;
    keepalive_timeout  65;
    tcp_nodelay        on;
    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

The /etc/nginx/sites-enabled folder contains the configuration files for each site you want to host. In this particular case, there is only one site hosted on the server but this is how the Ubuntu package is delivered.

Create the configuration file, setup a symlink between the sites-available and sites-enabled paths.

touch /etc/nginx/sites-available/site_name.conf
ln -s /etc/nginx/sites-available/site_name.conf /etc/nginx/sites-enabled/site_name.conf

Here is a sample configuration file (the site_name.conf provided above). Since this particular file is intended to proxy requests from Nginx to gunicorn.

server {
    listen   80;
    server_name site_name;
    access_log /var/log/nginx/site_name.access.log;
    error_log /var/log/nginx/site_name.error.log;
    location / {
        proxy_pass http://127.0.0.1:8000/;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        client_max_body_size 10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout 90;
        proxy_send_timeout 90;
        proxy_read_timeout 90;
        proxy_buffers 32 4k;
    }
    location /static {
        alias    /srv/site_name/public/static/;
        expires 24h;
    }
    location /gridfs/ {
        gridfs site_name;
    }
}

While I have not tested the gridfs proxy yet, the sample provided above has been taken from the module documentation.

Since we are intending to store images in MongoDB using the ImageField fieldtype provided as part of the Django model specification. This fieldtype requires the Python Imaging Library (PIL).

This is provided through aptitude but Django will not recognize it unless it is installed in the virtualenv. This is easy to solve by downloading the source and installing it once the virtualenv has been activated.

source bin/activate
wget http://effbot.org/downloads/Imaging-1.1.7.tar.gz
tar xzvf Imaging-1.1.7.tar.gz
cd Imaging-1.1.7
python setup.py install
cd .. && rm -r Imaging-1.1.7 && rm Imaging-1.1.7.tar.gz

Now that the PIL is installed, the database can be synchronized without throwing a error message about a missing PIL.

Hopefully, this is the last time I have to re-visit the server components as I should have everything required to complete the rest of the project. The next few posts should hopefully cover project development efforts.

Development Methodology and Django

It has been a few weeks. Work has been hectic and I have not had as much to devote to this project as I would like. I am hoping for some time off soon so I can knuckle down and get it done.

I have some experience in developing web applications as the need has arose in numerous ways over the years at my current job. Typically, a developer would meet with users, document requirements, build prototypes, perform A/B testing and iterate. As you might notice, this particular project falls inline with my original plan to use tools that I would never use in my day-to-day work.

So, I have opted for the Start in the Middle approach. I have an idea, I have focused on the primary objective of that idea and started the build. My goal is to build the minimum viable product, ship and then re-evaluate continued development efforts.

No question, it is a fun experience and I am learning some new skills so there is value regardless of whether it is successful or not. However, many a project is snagged by over-development, feature creep and lack of initiative. I was hoping to have this done by now but it always better to march on.

With that being said, onto Django.

What a wonderful framework. The community, the functionality and the extensibility are incredible. It really makes this project significantly easier and I applaud everyone who has contributed to the product. I only hope that one day I will have enough knowledge and some free time to help contribute.

This is supposed to be a technical blog but there is little to add about developing in Django that is not covered in the documentation. After that the wealth of knowledge being built over at Stack Overflow is beyond measure. One benefit of this project is that I have participated in a small way to the community.

As a continued trend, I will end this post with the intent that the next will contain more technical information about the project development. I have been working away at a few interesting pieces so I will try to wrap them up in a follow-up post soon.

Provided I can get some time off work first!

Sublime Text, iTerm, Cyberduck

I am still working away at the site. Hopefully, my next post will be a cumulative update on some of the interesting facets of Django, GridFS and MongoDB.

In an earlier post, I mentioned the django-easy-maps package, I have moved away from it. It is excellent, but it is limited when you want to specify multiple markers and attach event handlers to the map navigation.

Hopefully more on that later as well.

I thought I would use this entry to discuss the programming environment that I am using for the site.

I use a 2010 Macbook Air. It can be a bit slow at times, but it handles VMware Fusion well which allows me to run multiple OSes at the same time. Useful for quickly deploying a server, provide snapshot capabilities and flip back and forth between environments (using spaces).

I use Cyberduck for connecting to either local or remote servers (specifically Linode) and for text editing. It is an excellent application and I highly recommend donating to the project if you use the application.

For years, Textmate has been my editor of choice on OSX. I use vi while connected to servers (sorry emacs fans). However, I recently decided to move over to Sublime Text and I am hooked. I have made it my default editor for Cyberduck which automatically opens files in new tabs (a particular pet peeve with Textmate unless you launched it via terminal). The autocompletion and function list capabilities are excellent.

As for ssh access to servers, the base Terminal application is adequate but I prefer iTerm. It is useful for reloading gunicorn whenever non-template files are modified in Django.

Finally, the webkit developer tools provided in Chrome are excellent. I have used Firebug in the past in Firefox, but with the recent Firefox release acceleration and the fact that the lead developer on the Firebug plugin is working with Google, Chrome is the (IMO) the best browser for web development.

Add to that a comfy chair, footrest and side table (and associated alcoholic beverage) and you have my preferred development environment!

Django, Nginx and GridFS Con’t

Whew.

Took a little while longer than I anticipated but I have successfully managed to get uploaded files into MongoDB using GridFS and then served through nginx-gridfs in a nifty modal media viewer called Shadowbox.js.

In an earlier post I covered recompiling Nginx in Ubuntu to add the nginx-gridfs module. The steps outlined in that post are necessary before proceeding.

Django Model

As stated earlier, I am using the excellent Django MongoDB Engine which will allow base Django model FileField to upload files directly to GridFS as opposed to the local filesystem. Here is the models.py configuration that worked for me:

from django_mongodb_engine.fields import GridFSField
from django_mongodb_engine.storage import GridFSStorage

gridfs = GridFSStorage()
images = GridFSStorage(location='/gridfs')

Then in your specific model definition, define your FileField (or ImageField) as such:

class Foo(models.Model):
    ...
    file_1 = models.FileField(storage=gridfs, upload_to='gridfs')
    ...

Uploaded files will store the path and filename in the file_1 key/value store. So, bar.jpg would be stored as gridfs/bar.jpg. This is important later on.

Nginx

There are ways of serving files stored in GridFS directly out of Django but it is not recommended. The preferred approach is nginx-gridfs which is fairly easy to configure once you have defined your Django models.

The root_collection parameter is necessary as it assumes that the fs collection is used by default. Django MongoDB Engine appears to use storage.images/storage.files by default so you will need to specify either one in. Second, if you do not specify a field or type value, requests to example.com/gridfs/ will need to use the ObjectID (_id) to retrieve the specific files.

Since the Foo class will store the values using the path and filename, you will not know the ObjectID of the objects through a standard query.

location /gridfs/ {
    gridfs database root_collection=storage.images field=filename type=string;
    mongo 127.0.0.1:27017;
}

Once you restart Nginx, requests to example.com/gridfs/bar.jpg should return the uploaded file.