Profiling and Optimizing Python Code

All programmers have heard the advice: "Don't prematurely optimize code." What exactly does that mean? It means that you shouldn't guess about what is causing your code to run slowly. Chances are you'll guess wrong. Instead of guessing, use profiling and benchmarking tools to quickly and accurately identify the performance bottlenecks in your scripts.

Every now and then I run into a piece of Python code that just doesn't run as fast as I would like. I use 3 different Python tools to find and fix Python performance problems.

cProfile:

cProfile is a module that is included in the Python Standard Library. It logs function calls and execution times. There is also a pure-python version named 'profile'.

KCachegrind:

KCachegrind was written to visualize the output generated by Callgrind (a C profiler), but the function call logs from cProfile can be converted to the KCachegrind format with this script. KCachegrind uses the KDE framework. On Linux boxes, KCachegrind is usually bundled in a package with other development tools written for KDE. The package is named 'kdesdk' or something similar.

timeit:

timeit is a module included in the Python Standard Library that is used for measuring the execution time of arbitrary pieces of code.

Here is the code that needs to be improved:

class LameStringBuilder(object):
    def __init__(self, cols=40, rows=10000):
        self.cols = cols
        self.rows = rows

    def build(self, val):
        built = ''
        for i in range(self.rows):
            built += self.build_row(val) + "\n"
        return built

    def build_row(self, val):
        built = ''
        for i in range(self.cols - 1):
            built += val + ','
        built += val
        return built

Here is the code to execute a profile test and format the results into something that KCachegrind can understand:

import cProfile

# Use this module to convert profile data
# into the KCacheGrind format.
#
# The module is available here:
# http://www.gnome.org/~johan/lsprofcalltree.py
import lsprofcalltree

import lame_string_builder

def test():
    """Code to profile."""
    l = lame_string_builder.LameStringBuilder(40, 10000)
    l.build('foo')

if __name__ == "__main__":
    # Profile function
    p = cProfile.Profile()
    p.run('test()');

    # Get profile log in KCacheGrind format
    # and dump to file.
    k = lsprofcalltree.KCacheGrind(p)
    out = open('lame_profile.kgrind', 'w')
    k.output(out)
    out.close()

After running the test, launch KCachegrid and open the profile file to view the results.



The 'Flat Profile' and 'Callee Map' are the 2 most useful displays for finding the bottle necks in your code.

The 'Flat Profile' describes each function called in the test with the following columns:

• Incl. - The total percentage of time spent within a function.


• Self - The percentage of time spent within a function NOT including inner function calls.


• Called - The total number of times the function was called during the test.


• Function - The name of the function being described.

The 'Callee Map' represents the different functions called during the test. Each function is drawn as a rectangle. Inner function calls are drawn on top of their parent function's rectangle. The area used to draw each function is proportional to the execution time for each function call relative to the total execution time of the parent (also printed as a percentage in the graph).

In the example above it is easy to pick out where the bottle necks are by looking at the 'Callee Map'. Most of the test time was spent within the 'LameStringBuilder.build_row' method, so it is the main bottle neck. Other significant sources of time include calls to 'LameStringBuilder.build' and the built-in function 'range'.

After identifying the bottlenecks, I created a better version of the 'LameStringBuilder' class:

import lame_string_builder

class LessLameStringBuilder(lame_string_builder.LameStringBuilder):
    def build(self, val):
        return "\n".join([self.build_row(val) for i in xrange(self.rows)])

    def build_row(self, val):
        return ",".join([val for i in xrange(self.cols)])

After making changes to the code, Python's built-in 'timeit' module can be used to simply and accurately measure the differences in execution speed between the old and the new versions:

import timeit

import lame_string_builder
import less_lame_string_builder

def test_lame(val, cols, rows):
    l = lame_string_builder.LameStringBuilder(cols, rows)
    l.build(val)

def test_less_lame(val, cols, rows):
    l = less_lame_string_builder.LessLameStringBuilder(cols, rows)
    l.build(val)

def test_generic(repeat, name, args):
    print "%s (x%i):" % (name, repeat)
    t = timeit.Timer("%s(%s)" % (name, args), "from __main__ import %s" % name)
    print t.timeit(repeat)

def test_all(repeat, val, cols, rows):
    args = "'%s', %i, %i" % (val, cols, rows)   
    
    for name in ("test_lame", "test_less_lame"):
         test_generic(repeat, name, args)

if __name__ == "__main__":
    test_all(100, 'foo', 40, 10000)

The new code is significantly faster!

Role based security with Python

Most moderately complex enterprise applications require some sort of role based security and access control. For example, an employee should have access to fill-out her time card, but a manager should also be able to approve the time card. This post outlines how to create an 'access control list' (acl) for Python.

The acl is an object that can be queried to determine if a particular role has permission to access a resource. Each permission is also associated with a 'privilege'. For example, a manager may have both the 'read' and 'write' privileges for the weekly schedule, but an employee only has the 'read' privilege.

import acl

# Each user name should be associated with a role.
role = get_role(username)

# Query the acl to see if a role has access to a resource.
# params: role name, resource, privilege
acl = acl.Acl()
if not acl.check_access(role, 'time_card', 'write'):
    # User doesn't have access to this resource!!
    pass

The Acl class supports role inheritance. If the 'manager' role inherits the 'employee' role, then managers will have all of the same permissions as employees.

# Build acl list
acl = acl.Acl()

employee = acl.Role('employee')
resource = acl.Resource('time_card')
resource.set_privilege('write')
employee.set_resource(resource)
acl.set_role(employee)

manager = acl.Role('manager')
manager.set_parent(employee)
resource = acl.Resource('time_card')
resource.set_privilege('approve')
manager.set_resource(resource)
acl.set_role(manager)

if acl.check_access('manager', 'time_card', 'write'):
    # YES!
    pass

if acl.check_acces('manager', 'time_card', 'approve'):
    # YES!
    pass

if acl.check_access('employee', 'time_card', 'write'):
    # YES !
    pass

if acl.check_access('employess', 'time_card', 'approve');
    # NO!
    pass

I'm normally not a very big fan of XML, but in this particular case it is a good format to use if you prefer to store your acl in a configuration file. The Acl object's 'build_acl' method populates the object from a XML file with the following format:

<?xml version="1.0"?>
<!--
- This file configures the roles (user groups)
- and permissions for accessing the system.
-->
<config>
    <!--
    - Setup roles here.
    - Use the 'inheritFrom' tag to
    - inherit permissions from another role.
    -->
    <roleSet>   
        <role>
            <name>customer</name>
        </role>
        <role>
            <name>employee</name>
            <inheritFrom>customer</inheritFrom>
        </role>
        <role>
            <name>manager</name>
            <inheritFrom>employee</inheritFrom>
        </role>
    </roleSet>

    <!--
    - Set permissions for accessing application components here.
    - resource -> property being access controlled.
    - role -> group or user that can access resource.
    - privilege -> privilege that role can use with resource.
    -
    - Each permission tag can contain multiple
    - resources, roles, and privileges.
    -->
    <permissions>
        <permission>
            <resources>
                <resource>contact_details</resource>
                <resource>profile</resource>
            </resources>
            <roles>
                <role>customer</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>

        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>

            <roles>
                <role>employee</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>
        
        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>
            <roles>
                <role>manager</role>
            </roles>
            <privileges>
                <privilege>approve</privilege>
            </privileges>
        </permission>

    </permissions>
</config>

Here is the code for the acl.py module:

"""Role based security"""
from xml.dom.minidom import parse

class AccessError(Exception):
    pass

class Resource(object):
    """An Resource is an object that can be accessed by a Role."""
    def __init__(self, name=''):
        self.name = name
        self._privileges = {}

    def set_privilege(self, privilege, allowed=True):
        self._privileges[privilege] = allowed

    def has_access(self, privilege):
        if privilege in self._privileges:
            return self._privileges[privilege]
        return False
    
    def __str__(self):
        rpr = self.name + ': '
        for privilege, access in self._privileges.iteritems():
            rpr += "%s:%s " % (privilege, access)
        return rpr

class Role(object):
    def __init__(self, name=''):
        """An Acl role has access to resources with specific privileges."""
        self.name = name
        self._parents = {}
        self._resources = {}

    def set_parent(self, parent):
        self._parents[parent.name] = parent

    def set_resource(self, resource):
        self._resources[resource.name] = resource

    def has_access(self, attr_name, privilege):
        if attr_name in self._resources:
            if self._resources[attr_name].has_access(privilege):
                return True

        for parent in self._parents.values():
            if parent.has_access(attr_name, privilege):
                return True

        return False
    
    def __str__(self):
        rpr = self.name + ":\n"
        rpr += "parents:\n"
        for parent in self._parents.keys():
            rpr += "\t%s\n" % parent
        rpr += "resources:\n"
        for resource in self._resources.values():
            rpr += "\t%s\n" % resource.describe()
        return rpr

class Acl(object):
    """Manages roles and resources.
    
    Singleton class.
    """

    class __impl:
        """Implementation of the singleton interface"""
        
        def __init__(self):
            self._acl = {}

        def set_role(self, role):
            self._acl[role.name] = role
            
        def check_access(self, role_name, resource, privilege):
            """Check whether a role has access to a resource or not."""

            if not role_name in self._acl:
                raise AccessError('Role does not exist.')
            return self._acl[role_name].has_access(resource, privilege)

        def build_acl(self, file): 
            """Build acl from an XML file."""

            self._acl = {}
            roles_to_create = {}
            dom = parse(file)
            
            # Find roles to create
            roles_nodes = dom.getElementsByTagName('roleSet')
            for roles_node in roles_nodes:
                role_nodes = roles_node.getElementsByTagName('role')
                for role_node in role_nodes:
                    name_nodes = role_node.getElementsByTagName('name')
                    parent_nodes = role_node.getElementsByTagName('inheritFrom')
                    role_name = name_nodes[0].childNodes[0].data
                    roles_to_create[role_name] = []

                    # Find role parents
                    for parent_node in parent_nodes:
                        roles_to_create[role_name].append(parent_node.childNodes[0].data)

            # build inheritence chain
            for role, parents in roles_to_create.iteritems():
                self.set_role(self._create_role(role, roles_to_create))

            # assign permissions
            permissions = dom.getElementsByTagName('permissions')
            for permissions_node in permissions:
                permission_nodes = permissions_node.getElementsByTagName('permission')
                for permission_node in permission_nodes:
                    resource_nodes = permission_node.getElementsByTagName('resource')
                    role_nodes = permission_node.getElementsByTagName('role')
                    privilege_nodes = permission_node.getElementsByTagName('privilege')

                    for resource_node in resource_nodes:
                       resource = Resource()
                       resource.name = resource_node.childNodes[0].data
                       for privilege_node in privilege_nodes:
                           resource.set_privilege(privilege_node.childNodes[0].data)

                       for role_node in role_nodes:
                           try:
                               role = self._acl[role_node.childNodes[0].data]
                           except:
                               raise AccessError('Role in permission is not defined.')

                           role.set_resource(resource)

        def _create_role(self, role_name, roles_to_create):
            """Recursively create parent roles and then create child role."""
            
            if role_name in self._acl:
                role = self._acl[role_name]
            else:
                role = Role()
                role.name = role_name
                
            for parent_name in roles_to_create[role_name]:
                if parent_name in self._acl:
                    parent = self._acl[parent_name]
                else:
                    parent = self._create_role(parent_name, roles_to_create)
                    self.set_role(parent)
                role.set_parent(parent)
            return role
        
        def __str__(self):
            rpr = ''
            for role in self._acl.values():
                rpr += '----------\n'
                rpr += role.describe()
            return rpr

    __instance = None

    def __init__(self):
        """ Create singleton instance """
        
        # Check whether an instance already exists.
        # If not, create it.
        if Acl.__instance is None:
            Acl.__instance = Acl.__impl()

        self.__dict__['_Acl__instance'] = Acl.__instance

    def __getattr__(self, attr):
        """ Delegate get access to implementation """

        return getattr(self.__instance, attr)

    def __setattr__(self, attr, val):
        """ Delegate set access to implementation """

        return setattr(self.__instance, attr, val)

SQL Workshop

I've recently finished a SQL tutorial for a relational database workshop I'll be teaching. The tutorial covers basic SQL queries with SQLite and MySql, as well as basic database creation with SQLite. The tutorial is very basic, and the examples are geared toward biologists who need to use a database for performing research.

Facebook's Tornado

Today I got my first chance to play with Facebook's Python based web server and framework Tornado. The framework is single threaded and asynchronous, similar to how the Twisted framework operates, and Facebook uses the technology to provide data to the 'Friend Feed'.

Python thread creation and maintenance have relatively high overhead, so asynchronous solutions can provide better performance and scalability than more traditional threaded server implementations. This is especially true for servers with high numbers of concurrent connections, and for long-polling and streaming connections that spend most of their time waiting around for a message to be published to a client. Tornado's benchmarks look impressive against threaded Python web servers, but I wonder why they didn't include Twisted, their closest competitor, in the benchmarking.

My first impression is that Tornado is much easier to use than Twisted, but also much less powerful. Twisted is a monolithic package that can be used with many different networking protocols, but Tornado is focused only on http. Tornado provides built in support for templating, authentication, and signed cookies. One really cool feature Tornado provides is semi-automatic XSRF protection, which should save time for developers who would otherwise need to manually implement countermeasures.

The documentation for Tornado is very slim. For example, I couldn't find any information about how to interact directly with Tornado's main event loop (IOLoop). Fortunately, Tornado's code base is small and readable, so reading the source will quickly get you up to speed.

While implementing a Tornado channel for the AmFast project, I ran into a problem that I have also encountered with Twisted. Both Tornado and Twisted use callbacks to complete a request. As an example, if a long-poll client is waiting for a message, you call RequestHandler.finish() to send a response back to the client when a message is published.

The above method works well for a single server instance, but what about when you have multiple servers behind a proxy? A server process may publish a message for a client that is connected to an entirely different server process. The publishing process has no way to notify the subscribing process that a client has received a message.

This problem can be solved by polling a database table or a file that is accessible to all server processes, but that takes system resources, especially if you have many clients and your polling interval is short. One of my future goals is to figure out a more elegant way for the publishing server process to notify the subscribing server process when a client receives a message.

AmFast adds support for Google App Engine and Django

I just released version 0.4.0 beta of AmFast, a Flash remoting package for Python. The new release includes support for Google App Engine and the Django framework.

Browse the code here.


Check out the live Google App Engine demo.

To get the code:

svn checkout https://amfast.googlecode.com/svn/tags/0.4.0b

YAPT (Yet Another Python Tutorial)

The tutorial I created for the Python workshop is posted here. The tutorial is geared towards bioinformatics users, and includes molecular biology related exercises.

I created the tutorial with the Sphinx documentation package, which turned out to be the perfect tool for the job.

Learn Python Workshop

I'm going to be teaching an intro to Python workshop next month. It will consist of 3 3 hour days and will cover the basics of Python. Dates are June 15th, 17th, 19th. The workshop is free for all U of A students, staff, and faculty. I have never lead a workshop before, so it should be a lot of fun.

Register for the Python workshop