Simeon Franklin

Blog :: Sample fabfile

2 May 2009

Dan asked in the comments on my Baypiggies post if I could post a sample fabfile. I'll post the fabfile of the project I'm working on right now along with an explanation since it's doing a few different things... First the code - then the commentary:

import os

config(
    project = 'apple',
    fab_hosts = ['redacted.com'],
    fab_user = 'apple',
    django = '/opt/django1.0/django',
    package_file = "pyenv/lib/python2.5/site-packages/easy-install.pth",
    package_location = "pyenv/lib/python2.5/site-packages/",
    pth = """/usr/lib64/python2.3/site-packages
/usr/lib64/python2.3/site-packages/PIL
/home/$(fab_user)/django_site
/home/$(fab_user)/pyenv
/home/$(fab_user)/django_site/$(site)
/opt/django1.0
/opt/python-packages"""
    )

# Local convenience functions not related to deployment


def local_django():
    """ Link to django. Not virtualenv installed since shared install
    on server """
    local("ln -s /web/django_src/Django-1.0/django ./$(package_location)")


def syslibs():
    """ Link packages from local sitepackages I don't want to build
    via pip. Installed already on dev box, installed in global
    environment on the server"""

    for f in ["_mysql_exceptions.py", "_mysql.so", "MySQLdb"]:
        local("ln -s /var/lib/python-support/python2.5/%s $(package_location)" % (f, ),
              fail="ignore")
    local("ln -s /usr/lib/python2.5/site-packages/PIL $(package_location)", fail="ignore")


def setup():
    """ Assuming you copied another sites requirements file,
    initialises pyenv """
    local("pip -E ./pyenv install -r requirements.txt")


# Deployment commands
# Select test or production, build the package to transport, than deploy:
# $ fab production build deploy


def test():
    config.site="test_site"


def production():
    config.site="prod_site"


@requires('site', provided_by = ['test', 'production'])
def remote_env():
    """Not deploying virtualenvs - just setting up python env for shell via ~/.python/django.pth
    files."""

    run("mkdir /home/$(fab_user)/.python")
    lines = open(config.package_file).readlines()
    # limit to all the packages in the src dir and rewrite path for server
    pkg_lines = [l.replace('/web/', '/home/') for l in lines if "/src" in l]
    # We just manage server environment with ~/.python/django.pth file
    run("""echo "$(pth)\n%s" > /home/$(fab_user)/.python/django.pth""" % "".join(pkg_lines))


@requires('site', provided_by = ['test', 'production'])
def build():
    """Build tarballs for transfer of the "libraries" (external django apps)
    and the site (settings, media, template and site-specific django apps)"""

    #local("pip -E ./pyenv/ freeze requirements.txt") # Not using till bundles work better
    local("tar -czvf pyenv.tar.gz pyenv/src") # not wanting to checkout on server, just tar src dir
    local("cd django_site/mysite;bzr export ../../$(site).tar.gz")


def django():
    """Remotely link in appropriate django on the server"""
    run("ln -s $(django) /home/$(fab_user)/pyenv/src")


@requires('site', provided_by = ['test', 'production'])
def deploy_pluggables():

    # Hmm. Bundles are buggy w/ bzr, don't want to have to install from req (and check out)
    #put("requirements.txt", "requirements.txt")
    #run("pip -E ./pyenv install -r requirements.txt")
    # So just tar pyenv/src, transfer over and untar
    put("pyenv.tar.gz", "/home/$(fab_user)/")
    run("tar -xzf /home/$(fab_user)/pyenv.tar.gz")
    run("rm /home/$(fab_user)/pyenv.tar.gz")


@requires('site', provided_by = ['test', 'production'])
def local_settings():
    """Put the remote local_settings.py file (not the one in /django_site/mysite)"""
    put("django_site/local_settings.py", "/home/$(fab_user)/django_site/$(site)")


@requires('site', provided_by = ['test', 'production'])
def deploy_site():
    """Deploy the django project and custom apps in django_site/mysite"""

    run("mkdir /home/$(fab_user)/django_site")
    put("$(site).tar.gz", "/home/$(fab_user)/django_site")
    run("cd /home/$(fab_user)/django_site/; tar -xzf $(site).tar.gz")
    run("rm /home/$(fab_user)/django_site/$(site).tar.gz")
    # Put the local settings file for the remote server
    local_settings()


@requires('site', provided_by = ['test', 'production'])
def deploy():
    """Transfer pluggables and django site to remote server and verify
    that remote environment is prepared"""

    deploy_pluggables()
    deploy_site()
    remote_env()

Ok - what does all that do? Let me explain the flow for this project and then show how the fabfile supports it. In this case I'm developing locally with pip and virtualenv but not using either on the deployment server. The deployment server has a few libraries globally installed (MySql driver and PIL, primarily) but I want my 3rd party django apps and my custom app for the individual site to be in the virtualenv. On the server I'm building the environment by including ~/.python in the $PYTHONPATH environmental variable and adding paths to the django.pth file in ~/.python. Supervisor is in charge of running each process as the appropriate user - eventually I hope to migrate to mod_wsgi and use virtualenv on the server but I'm not there yet... I'm also not installing a virtual env copy of Django for this project - I'm trying to stick to releases for any substantial projects so I just symlink in the 1.0 release. Similarly on the server I've got several releases of Django in /opt and just symlink to the appropriate version for each project.

The first few functions in the fabfile after the call to config are just convenience for local development. I've found fabric a great place to stash frequently run shell commands and save on typing (instead of issuing a series of find calls to clear out compiled files, temporary files (.pyc, .py~, etc), for example, I could put several calls to local() in a function called cleanup and `fab cleanup` instead).

Starting at test() and production() I'm building and deploying my project. The test() and production() functions just pick my destination directory - I usually deploy to a test directory and run the built in server with sqlite to test. If everything checks out I deploy to the production directory and restart my django process in supervisorctl. Next remote_env() builds my remote environment as described by making sure ~/.python exists and writing a .pth file in it... The .pth file gets the hard coded libs from the config file plus everything listed in the easy_install file. This is pretty hacky - hence the desire to move to virtualenvs on the server...

The actual build process in build() just packages my django site's source directory to a tarball using bzr and tars up the virtualenv's /src directory for transport. The deploy commands transfer and untar the files and copy the remote site's local_settings.py file over. Breaking my "pluggables" (eg: possibly 3rd party django apps I'm not editing for this project) as a separate step from my site allows my "mysite" directory to only contain code I'm directly working on and lets me `fab build deploy_site` and only transfer the site specific code...

If this doesn't make sense and you haven't looked at the presentations in my previous post be sure to do so...