Simeon Franklin

Announcing the Modesto Scripting Language Meetup

Announcing the Modesto Scripting Languages Meetup. Central Valley Pythonistas, Rubyists, PHPers, etc all welcome.

Posted January 25th, 2012 in Programming Python (Comment)

Calling all PyCon Rejects

I recently received a very polite email from the PyCon program chair telling me what I already knew: my talk proposal didn't get accepted. My talk was rejected.

Let me explain how I already knew, why being rejected is not a bad thing, and why I'm challenging you to put your rejection to good use both personally and for the Python community.

Posted December 23rd, 2011 in Programming Python (Comment)

What is "Un-Pythonic"?

Tonight's Baypiggies topic is going to be What is Pythonic - Marilyn Davis is doing the presentation. I had a few thoughts but they're all in the opposite direction. I don't feel authoritative enough to define Pythonicness but as an instructor I frequently get to see code written by students new to Python that is obviously un-pythonic, for lack of a better term.

Posted December 15th, 2011 in Programming Python (Comment)

AHAH with Django and jQuery

I was recently asked about using AJAX via JQuery with Django and mentioned that I frequently use html fragments and a decorator to add Ajax functionality to existing views. Let's see how that works. (more after the jump).

Posted August 22nd, 2011 in Django Programming Python (Comment)

PDB Howto

Cool! See Max's awesome video editing skills render my pdb howto relatively stumble free! If you're curious about how to use the built in Python debugger this brief video tutorial should get you going.

Posted August 11th, 2011 in Programming Python (Comment)

The best way to OR a list of Django ORM Q objects

A co-worker asked me today about the best way to OR together a list of Q objects in the Django ORM. Usually you have a specific number of conditions and you can use Q objects and the bitwise OR operator to logical OR the query conditions together. The Q object is necessary because multiple arguments or successive calls to .filter are ANDed. But what if you have an arbitrary number of Q conditions?

One suggestion is to use the undocumented .add method of the Q object in a loop to add multiple query conditions together. I thought this might be a good use case for reduce and the operator module:



# Normal Usage with the | operator
from django.db.models import Q
qs = MyModel.objects.filter(Q(cond1=1) | Q(cond2="Y"))

#but given a list of Q conditions
from operator import __or__ as OR
lst = [Q(...), Q(...), Q(...)]
qs = MyModel.objects.filter(reduce(OR, lst))

Is this the most Pythonic approach?

Posted June 14th, 2011 in Programming (Comment)

Teaching Retrospective

As you can probably figure out from the last post, I have a new gig.

Thanks to the good folks at marakana.com I recently taught a Python Fundamentals course in San Francisco.

This was my first time teaching for Marakana and I enjoyed the experience immensely - as did my students judging by their class reviews at the end! I had a blast and am looking forwards to more python classes, including an under-development Pro Django course. More details to come in this space - and thanks to Jas, Brenda, Mike, Chris, and Robert for being great first-time students. I hope you all go on to Pythonic success.

Posted June 14th, 2011 in Programming (Comment)

Python Fundamentals Resources

Deprecated: see the Python Fundamentals Page instead of this post

This will just be a grab bag of extra resources and notes for my students taking the Python Fundamentals Course at Marakana. Don't forget to grab the labs.

Additional Resources

Python in a Nutshell
Dive into Python Online Book - somewhat dated but still good.
The Official Python Tutorial
Code like a Pythonista: read this and your code will be pythonic.
Doug Hellman's Python Module of the Week - a great way to get to know Python's vast stdlib.

Important Basics You Should Know

Python builtin functions: see the http://docs.python.org/library/functions.html. Or


import __builtin__
dir(__builtin__)

Keywords: see the docs http://docs.python.org/reference/lexical_analysis.html#keywords. Or


import keyword
print(keyword.kwlist)

Tools

IPython - a better python shell
Python Regex Testing Tool
virtualenv and virtualenvwrapper - tools for managing python environments. If you find yourself installing 3rd party python code you should be using these tools.
Stuck on windows? Use a replacement console

Additional Documentation Referred to in Class

Simple String Formatting
Built In Exception Types
Quick Doctest Overview - but I like to use
```
$ python -m doctest -v myfile.py
```
Don't forget the first dunderscore hack you'll need:
```
if __name__ == '__main__':
    main()
```
A great walkthrough of the stdlib logging module
str.format
JSON in stdlib
List of dunderscore methods

Don't forget the old labs - new labs and samples coming soon!

Posted May 30th, 2011 in Programming (Comment)

WSGI Mysteries

Recently I had a mysterious error cropping up on a Satchmo Store I set up for a client. Every so often I would have an HTTP 500 error with the error log indicating that a template could not be read because the specified charset wasn't available. I edited the Django template loader code to produce more information and only managed to push the exception down into the Python source code.


str = codecs.open(filepath, encoding="utf-8").read()
File "/opt/python2.6/lib/python2.6/codecs.py", line 865, in open
    file = __builtin__.open(filename, mode, buffering)
LookupError: unknown encoding: ANSI_X3.4-1968

The template that was being opened was valid UTF-8 and even stranger - as the same user and using the same virtualenv as the WSGI app I can open a file specifying an ANSI_X3.4-1968 encoding.

It occurred to me that it must be problem with the environment somehow so I spent some time reading the WSGI documentation. Eventually I changed one line in my wsgi config for the app and resolved my problem. The configuration:


    WSGIDaemonProcess sitename user=sitename threads=1 display-name=%{GROUP}
    WSGIProcessGroup sitename
    WSGIApplicationGroup sitename
    # Previous setting:
    # WSGIApplicationGroup %{GLOBAL}

If I understand correctly all my wsgi applications configured using %{GLOBAL} were sharing the same interpreter. Somehow another process must be messing up the state of the interpreter somehow - using WSGIApplicationGroup with a value forces this application to run in its own sub-interpreter and this cleared up my problem. What I don't have is any insight into why my original error was occuring. Any thought?

Posted April 28th, 2011 in Programming (Comment)

Hidden Django QuerySet Features II

Recently while reading a co-workers code I discovered that Django's Queryset can be OR'ed together - but this may not always be a good idea. Django's ORM overloads the bitwise OR operator to express logical OR and the documentation demonstrates this with a Q object. Let's query the ORM to find recently active users - users who joined this year or who have logged in this year:


>>> from django.db.models import Q
>>> from django.contrib.auth.models import User
>>> from datetime import date
>>> jan_1st = date(2011, 1, 1)
>>> recent = User.objects.filter(Q(last_login__gte=jan_1st) 
                             | Q(date_joined__gte=jan_1st))
>>> print(recent._as_sql())

('SELECT U0."id"
  FROM "auth_user" U0
  WHERE (U0."last_login" >= %s
          OR U0."date_joined" >= %s )',
 (u'2011-01-01 00:00:00', u'2011-01-01 00:00:00'))

The where clause in the generated SQL is exactly what we wanted. It turns out, however, that you can use the | operator on querysets directly to yield the same thing.


>>> from django.contrib.auth.models import User
>>> from datetime import date
>>> jan_1st = date(2011, 1, 1)
>>> recent_login = User.objects.filter(last_login__gte=jan_1st)
>>> recent_join = User.objects.filter(date_joined__gte=jan_1st)
>>> recent = recent_login | recent_join
>>> print(recent._as_sql())
('SELECT U0."id"
  FROM "auth_user" U0
  WHERE (U0."last_login" >= %s
          OR U0."date_joined" >= %s )',
 (u'2011-01-01 00:00:00', u'2011-01-01 00:00:00'))

Sweet! It's always bothered me that chained calls were AND'ed together and there was apparently no way to do a simple OR without importing an additional class. However this feature has to be used with care - you are OR'ing entire queries instead of clauses so careless usage might lead to expensive queries. For instance imagine that we want to only look at certain automatically created recent users whose usernames start with "applicant". OR'ing entire querysets works. Sort of.



>>> from django.contrib.auth.models import User
>>> from datetime import date
>>> jan_1st = date(2011, 1, 1)
>>> applicants = User.objects.filter(username__startswith="applicant")
>>> recent_login = applicants.filter(last_login__gte=jan_1st)
>>> recent_join = applicants.filter(date_joined__gte=jan_1st)
>>> recent = recent_login | recent_join
>>> print(recent._as_sql())
('SELECT U0."id"
  FROM "auth_user" U0
  WHERE
   ((U0."username"::text LIKE %s  AND U0."last_login" >= %s )
     OR
    (U0."username"::text LIKE %s  AND U0."date_joined" >= %s ))',
 (u'applicant%',
  u'2010-01-01 00:00:00',
  u'applicant%',
  u'2010-01-01 00:00:00'))

Note that this did exactly what we asked - OR'ed two querysets. This results in duplication in the where clause - we really only want to run the LIKE search on the username once and then filter the results by OR'ing the two date criterion. The results from the this query will be correct but the execution (depending on your DB backend) may be slower as the LIKE operator is run on the entire data set twice. In my case I noticed an excessively complicated query in the SQL results pane of my django-debug-toolbar. Due to directly OR'ing two complicated querysets four or five expensive operations were being duplicated on a table with a million records. Switching to a Q object produced a much shorter (and faster) query with the same results.

Posted February 20th, 2011 in Programming (Comment)

Hidden Django QuerySet Features I

I've been paying attention to the generated SQL the Django ORM provides lately. I've had to a bit of performance tuning on some apps I wrote so I started out with the django-debug-toolbar app which will show you all the queries run on a given page and their runtime. This is an indispensable tool in my daily toolkit but I wanted to play with creating queries in my console. How do I see the SQL that is generated?

Somewhere (probably just through introspection) I found the semi-private _as_sql() method.


>>> User.objects.filter(is_staff=True)._as_sql()
('SELECT U0."id" FROM "auth_user" U0 WHERE U0."is_staff" = %s ', (True,))

Notice that this doesn't return all the fields ("select id from..."). This method also won't work on value QuerySets and shouldn't be depended on as it is not a part of the public interface of the QuerySet class. It does helpfully returns a two part tuple of the query string with placeholders and a tuple containing the query parameters. This is useful if you want to tweak the parameters on the fly and paste them into a db console. A better method to get the sql actually being executed is to access the .query member of a QuerySet:


>>> users = User.objects.filter(is_staff=True)
>>> users.query
<django.db.models.sql.query.BaseQuery object at 0x9df63ac>
>>> str(users.query)
'SELECT "auth_user"."id", "auth_user"."username", "auth_user"."first_name", 
"auth_user"."last_name", "auth_user"."email", "auth_user"."password", "auth_user"."is_staff", 
"auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."last_login", 
"auth_user"."date_joined" FROM "auth_user" WHERE "auth_user"."is_staff" = True '

Finally you can see all the queries that have run by looking at the connection object.


>>> from django.db import connection
>>> print connection.queries
[{'sql': 'SELECT ....',
  'time': '0.009'},]

I redacted the actual query but this returns rows of dicts with the sql that was run and how long it took to run.

Posted February 14th, 2011 in Programming (Comment)

Python Code Quality

(I'm presenting the Newbie Nugget tonight @ Baypiggies. My topic is Python Code Quality - read on for the scoop.)

Posted February 25th, 2010 in Programming Python (Comment)

Pydelatt

Part of what I do for my clients is manage their software/hardware infrastructure. Most of my clients are not large enough to have dedicated sys-admin staff so in addition to wearing the software developer hat I sometimes get to wear the sysadmin hat as well. This is not always a good thing and sometimes I end up writing software (what I like to do) to fix a sys-admin style problem (the stuff I don't like to deal with).

So recently a colocated box I manage for a real estate company started to run low on disk space. The main culprit was the the mailbox accounts - realtors frequently mail large documents (pictures, contracts, flyers, etc) and most of the mail accounts had a gig or two of mail. I decided to set a policy of deleting old attachments and looked for a tool to accomplish this task.

No luck - Dan Born's Delatt looked like it would do what I wanted but I couldn't actually get it to work. This was probably my fault but trying to figure out what wasn't working meant debugging Perl. Not my favorite language, and more to the point my Perl chops are about a decade rusty now. So I wrote a tool in python to do what I want. Pydelatt accepts a maildir filename and strips out any attachments whose mime type is not text/*.

All the usual caveats apply (use at your own risk, attachments are deleted irrecoverably and user error may cause your hair to burst into flame) but I'm using it as a policy tool (`find -mtime 120 -size +3M | xargs -ix pydelatt.py 'x'`) tool and I've successfully run it on a couple hundred gigs of email without incident for a month now...

Posted December 21st, 2009 in Programming (Comment)

Off to present at Baypiggies again

I'm off to present again. My topic is Fixing Django with 3rd party apps and it's some best practices advice plus dev oriented apps I think are useful. The slides are here in s5 format (hit the spacebar to advance).

Update: The presentation went well - a few additional notes. The slides don't show it but I live demoed Rob Hudson's django-debug-toolbar and the command-extension runserver_plus/werkzeug debugger. My slide on South is non-informative because I followed Glen Jarvis' presentation on South... I had fun and I'll post links to the videos when they get posted.

I had follow up questions afterwords about finding cool 3rd party apps - and was trying to remember the recent blog post I saw that had a nice list. For anybody still looking for that check out Kevin Fricovsky's post on the apps that power mingus.

Posted October 22nd, 2009 in Programming (Comment)

Jutda Helpdesk

Jutda helpdesk rocks!

Posted May 14th, 2009 in Django Recommends (Comment)

Sample fabfile

As a follow-up to the Virtualen/Pip/Fabric presentations post, here's a sample fabfile from a project I'm working on right now...

Posted May 2nd, 2009 in Programming (Comment)

Baypiggies Presentations

Last night I participated in the Baypiggies Tools Night - I ended up in charge of the evening and listened to interesting presentations by Sandrine Ribeaux on Pylint, JJ on ... well ... random stuff in the Unix way, Drew demonstrating a bunch of different tools (depgraph makes cool pics like this out of your code's dependency graph, kcachegrind makes cool pics of your profiling output).

When all that (plus the newbie nugget on Big-O notation and python container types) was over we were almost out of time. I had three presentations prepared: one on using virtualenv to isolate python environments, one on using pip (Ian Bicking's easy_install replacement), and a presentation on fabric (the pythonic remote deployment tool). Due to the time limitations I did an abbreviated run through the first two and spent most of my time on fabric. I think a video of the audio and slides will be up at some point - in the mean time you can see my slides on virtualenv here, the pip slides here, and the fabric slides here - hit the space bar once the slides load to move through them.

I also ended up talking afterward about how I prepared my slides: I used the rst2s5 tool that's included in docutils to turn my slide's rst source into the html slides I used in my presentation. Any modern browsers will show a nice click through slide show using Eric Meyer's S5 slide format...

Posted March 28th, 2009 in Programming (Comment)

Supervisord 3.0?

I'm starting to run into a problem with the excellent supervisord. I currently use it to keep my Django processes alive on my VPS and now that I have a couple dozen managed processes I'm realising the shortcomings in the design of supervisord.

Supervisord is basically a friendly init system written in python. Rather than have to write init scripts in shell I just edit my supervisord.conf files, run supervisord as root, and all my long running processes (mostly Django instances) are started and managed by supervisord. This works well until I need to an additional process; currently reloading the config file means restarting the supervisor daemon which means restarting all the processes it controls (and a time wait/heavy server load while they all start simultaneously.)

I'm aware that there are some patches (twiddler) to allow you to dynamically add tasks without editing the .conf file. What I really want, however, is to be able to reload the conf file and only affect tasks that aren't already running (so adding a new process to the config file and reloading would only affect the newly added task.) It makes me very happy to see some discussion of this on the supervisor mailing list (see here, for example) towards the end of 2008. Of course now I'm just waiting anxiously for a 3.0 release - and wondering if I should stop complaining that my free ice-cream isn't being delivered fast enough and pitch in and help instead...

Posted March 5th, 2009 in Recommends (Comment)

Django Tree Menu

I plan to regularly highlight Django apps I've found useful. I know there are some pluggable app review sites springing up - but I think it's one way of thanking authors in a small way for sharing their code with the Django Community.

With that in mind - I recently switched from my own menu app to Django TreeMenus - mostly because they have a nice admin (I have to check out how they implemented the ordering buttons in the admin; it's very nice!) I do wish they'd use the indispensable mptt to add the tree management features. It would be nice to have one really polished reusable hierarchical tree app, instead of many custom re-implementations, but this is a small nit to pick. This is definitely worth your while if you want your menus to be adminable... It's just a `pip install -E env -e svn+http://django-treemenus.googlecode.com/svn/trunk#egg=treemenus` away :)