limscoder

AmFast 0.5.3 Released

2011-12-10T15:29:00.000-08:00

AmFast version 0.5.3 has been released. This release contains several important bug fixes. The code can be downloaded from PyPi or checked out from SVN.

ExtJs Component Config

2011-11-23T07:42:00.001-08:00

Cross post: ExtJs Component Config

Javascript taking a step backwards?

2011-10-24T14:33:00.000-07:00

I love Javascript, and believe its model of prototype inheritance is the future of programming. The prototype is elegant and eliminates many problems associated with traditional class-based OOP languages. I've used many Javascript libraries, from lite weight JQuery to everything's included libraries like ExtJS and Dojo. I'm excited by the capabilities of modern browsers, but the problem I see with Javascript, is that there are not very many tools available for creating large scale enterprise applications. Sure, a widget here, and a widget there are easy to build, but how about a real life application with hundreds of files and 10s of thousands of lines of code?

At the SenchaCon conference this morning the CEO of Sencha claimed that Javascript 'has won' and plugin based RIA technologies like Flex and Silverlight are being left behind. He is undeniably right, but at the same time I find it very odd that Javascript is still very far behind in some areas. For example, directly after the statement that Javascript has won, Sencha employees went on to debut several new features and products that have already been part of the Adobe/Flex ecosystem for many years. Messaging, a graphical component builder, an animation builder, a component market place, and an MVC framework are some of the features and products that are just starting to appear in the Javascript world.

These features have been around the Flex community for quite some time, and these new products seem rudimentary in comparison. Yet, there are no better options, because as far as I know, Sencha is the only company offering similar tools. Adobe is attempting to rework some of their existing tools to compete in the Javascript arena, but they aren't very developed yet. GWT also offers some similar tools, but why oh why would you want to write Javascript with Java? How long will it be before we have truly robust support for building large enterprise applications with Javascript?

Testing With Browser Mob

2011-10-08T08:08:00.000-07:00

The Project

I recently got the chance to work on a project using BrowserMob for automated testing. BrowserMob allows you to run Selenium test scripts "in the cloud". In my case I was not testing functionality, but instead testing performance of a Flex app. I needed to test latency and throughput of messages being dispatched through the Flex messaging system via http://code.google.com/p/amfast/. This proved very difficult to test locally, but was a snap with BrowserMob.

The Testing

I created a simple Flex client to send and receive Flex messages in a way that replicated a production environment. I also created a custom server component to replicate the production environment and to help log message data to be analyzed later. After getting the client and server running locally, I signed up for a BrowserMob account and launched several browsers with their web interface.

The whole process was simpler than it should have been, and I was very impressed with how well it worked. I highly recommend trying out but BrowserMob for performance and load testing applications, and I'm hoping to get a chance to try out running more full featured automated functional tests in the future.

Paired Programming

2011-08-30T17:30:00.000-07:00

Rally encourages its engineering staff to post to the company blog, so I decided to write a little about my initial experience with paired programming. TLDR: paired programming is better than I expected, increases code quality, and probably productivity.

tirtle: a Spring Web MVC project running on GAE

2011-07-12T19:51:00.000-07:00

I decided to put together a simple web app on my way to learning Java and Spring. Tirtle allows users to track daily numbers such as how many calories they eat in a day. The project uses the Spring Web MVC framework and is running on Google App Engine.

The most frustrating part of the project was just getting everything configured and getting a working server up and running. I feel there is a lack of documentation aimed at beginners, although part of my problems may have been related to jumping right in with Spring + GAE. I chose Spring Framework version 3.0 (the latest version), but there seemed to be more documentation and blog tutorials available for version 2.5. I found an official Web MVC tutorial on the Spring Source site, but it only covered version 2.5.

I had several problems figuring out the base configuration settings, and getting all the correct .jar files in my classpath. webmvc.jar was especially mysterious, because it is not included in the Spring Framework distribution. I ended up finding it via a Google search, but I have yet to find the official download from Spring Source.

Once I got a basic server running, the available documentation for how to actually use Spring seemed pretty good. The Web MVC framework works similar to every other MVC framework you have used. Classes are annotated to turn them into controllers, and controller methods are annotated to turn them into request handlers. You can use either JSPs (Java Server Pages: Java embedded in HTML similar to PHP) or a templating system for your view layer.

The Web MVC stuff is all built on top of Spring's dependency injection framework, so the simple MVC annotations you use are actually shortcuts to lots of complicated XML configuration. Spring's standard DI tools can be used to inject other non-mvc dependencies into your controllers. I used the DI features to configure a simple authentication object (didn't have time to figure out how to get Spring Security working) and also an ORM interface to Google's DataStore (objectify-appengine with the help of objectify-appengine-spring).

Unfortunately, the simple web app I developed didn't require much logic, so it didn't help too much in terms of learning how to code Java, but it was an excellent exercise in getting up and running with Spring. The code for the project is on github. Other new-to-Spring users may find Spring easier to learn by starting with a working app like this one, and learning by modification. I hope to have time to add additional features to the code as I learn how they are accomplished in the Java/Spring ecosystem (unit tests, Spring Security, Javascript framework integration, REST api, templating instead of JSPs).

Python Vs Java

2011-06-22T10:48:00.000-07:00

After using Python for the past several years, I'm going to be taking on a Java project. I am in the process of learning Java, and I thought I would write up a comparison of the features in each language.

Features Java has that Python doesn't:

Static typing
Strict access control (package, public, protected, private)
Traditional threading implementation wrapped in a decent API
Bytecode backwards compatibility

Features Python has that Java doesn't:

Dynamic objects
No explicit compile step
Properties (transparent getters/setters)
List comprehensions
Operator overloading
Generators (create iterators with the 'yield' statement)
Optional keyword arguments, *args, and **kwargs
pypi (the cheeseshop) and pip

Static Typing

Static typing has 2 main advantages. Type errors can be caught at compile time, and the compiler can make more optimizations. Statically typed code takes more time to write (creating explicit interface definitions), but you don't need to write manual type checking functions (duck typing) like you often need to in a dynamic language. Static typing also allows you to avoid runtime type errors that you would probably need a unit test to catch with a dynamic language. Javascript and Python both have decent JIT compilers available, so the speed difference between dynamic languages and static languages will continue to narrow.

Strong vs Weak Typing

While static typing may be helpful for some projects, I believe the biggest factor in type usability is not static vs dynamic but strong vs weak. Weakly typed languages allow you to cast objects from one type to another. If you've ever used Perl, PHP, of Javascript you've probably run into some hard-to-debug problems that were caused by implicit casting or automatic type coercion. Languages like these usually have confusing operators like '==='. C doesn't do any implicit casting, but it will let you manually cast in unsafe ways. Java also allows casting, but is safer than C, as it will throw a runtime exception if you try to cast to an incompatible type. On the other hand, Python is strongly typed. There is no such thing as a 'cast' in Python, and there are very few situations where automatic type conversion takes place (arithmetic with operands of different number types automatically convert all operands to the widest type used in the expression). Python's combination of strongly typed objects and dynamic objects with duck typed interfaces is a winner.

Access Controls

Python has no access control modifiers, and instead uses a convention of naming private attributes with a leading under score: '_private_method'. Client code is not 'supposed' to use attributes named with a leading underscore, but there is nothing technical stopping it. I must admit there have been several times when I wish Python had some equivalent construct. Java's 'final' modifier is particularly useful. Considering that many of Python's standard data types are immutable (string, unicode, tuple, frozenset), it's surprising that Python does not offer an easy way to define immutable objects. In Java it's as easy as adding 'private final'.

Threading

Unlike Python and the GIL, Java can utilize multiple cores when executing threads, and it's concurrency interface is wrapped in a nice API. However, I'm not sure how much I'll get to use the threading features. Networking code is increasingly being moved away from threaded implementations to asynchronous/non-blocking solutions such as NodeJS and Twisted, and computation is being distributed on a cluster or 'in the cloud', instead of being run on a single machine.

Other Features

Java lacks many useful features present in Python, and I'm sure I will miss many of them. I hope the list of useful Java constructs grows as I learn more about the language and start working on a production code base.

Javascript File Browser for Server-Side Files

2011-05-29T10:31:00.000-07:00

Problem - Galaxy

Galaxy is a web-based bioinformatics toolkit that allows users to create customized data analysis pipelines. It is becoming an extremely common tool, especially in the sequencing field. The project has one major flaw: it is difficult to get large data files into the system. Uploading 2G files (generated from sequencing runs) through a browser is not a workable solution.

Solution - iRods

iRods is a virtual file system commonly used to transfer and share files in the scientific community. It abstracts the storage details and provides tools for access control, sharing, metadata tracking, file type conversion, and high performance multi-threaded file transfer.

Integration

Myself and co-workers Fred Bevins and Susan Miller were tasked with integrating iRods and Galaxy during the iRods code sprint hosted by iPlant in April. My share of the work was a client-side Javascript file browser. The file browser allows users to browse and select server side files. The version I built talks to the iPlant Foundational API, which exposes iRods directories and files. Other back-ends could easily be added to allow the file browser to talk with a standard file system, or any other file repository. Fred and Susan worked on integrating the file browser into Galaxy, which allows Galaxy users to browse and select files in an iRods repository for use in an analysis. The javascript code is available now, and the Galaxy integration code should be available sometime soon.

Using Protovis to Create Simple Flow Charts

2011-04-04T11:31:00.000-07:00

Protovis is a Javascript library for creating SVG graphics to visualize datasets. The API is great, and I've been using it to visualize all sorts of data for a project I'm working on. I had a need to display a very simple (< 15 nodes) branching flow chart. The screen is simple enough that it doesn't justify a custom Protovis layout component or anything fancy like that.

I cooked up a scheme where the nodes are absolute positioned div elements that can be styled with CSS, and the edges are drawn with Protovis. I pass an object that defines the edge properties to a Javascript function that uses JQuery to find the exact positions of the nodes, and then I use Protovis to draw the edges.

Example:

HTML:

<div id="workflowContainer">
  <!--
    -- Draw Simple divs to represent workflow nodes, and connect them with Protovis.
    --
    -- Nodes are positioned absolutely.
    -- Node positions can be static and manually determined,
    -- or dynamic and determined by server-side or client-side
    -- code. This example uses hard coded node positions.
    -->
  
  <div id="workflowChart" >

    <!-- Clickable node -->  
    <a href=""><div id="startFlow" style="top 0; left: 440px;">Start</div></a>

    <!-- Foo branch -->

    <!-- Unclickable node -->  
    <div id="foo1Flow" style="top: 100px; left: 200px;">Foo 1</div>
  
    <a href=""><div id="foo2Flow"  style="top: 175px; left: 100px;">Foo 2</div></a>
  
    <div id="fooChoice1Flow"  style="top: 300px; left: 0px;">Foo Choice 1</div>
  
    <div id="fooChoice2Flow" class="inactive" style="top: 300px; left: 165px;">Foo Choice 2</div>
  
    <div id="fooChoice3Flow" class="inactive" style="top: 300px; left: 360px;">Foo Choice 3</div>
  
    <div id="fooOptionFlow"  style="top: 400px; left: 50px;">Foo Option</div>
  
    <a href=""><div id="fooCombineFlow"  style="top: 500px; left: 200px;">Foo Combine</div></a>
  
    <a href=""><div id="fooSplit1Flow"  style="top: 575px; left: 25px;">Foo Split 1</div></a>
  
    <a href=""><div id="fooSplit2Flow"  style="top: 575px; left: 250px;">Foo Split 2</div></a>
  
    <!-- bar branch -->
    <div id="barFlow" style="top: 100px; left: 700px;">Bar</div>
  
    <a href=""><div id="bar1Flow" class="inactive" style="top: 200px; left: 550px;">Bar 1</div></a>
  
    <a href=""><div id="bar2Flow" class="inactive" style="top: 200px; left: 825px;">Bar 2</div></a>
  </div>
</div>

CSS:

/* Contains both nodes and edges. */
#workflowChartContainer {
 position: relative;
 width: 1000px;
}

/* This is where the edges will be drawn by protovis. */
#workflowChartContainer span {
 position: absolute;
 top: 0;
 left: 0;
 background: transparent;
 z-index: 1000; /* SVG needs to be drawn on top of existing layout. */
}

#workflowChart {
 position: relative;
 top: 0;
 left: 0;
 height: 700px;
 width: 1000px;
}

#workflowChart div {
 border-color: #5b9bea;
 background-color: #b9cde5;
 position: absolute;
 margin: 0;
 padding: 4px;
 border: 2px solid #5b9bea;
 background: #b9cde5;
 border-radius: 4px;
 -moz-border-radius: 4px;
 -webkit-border-radius: 4px;
 color: #000;
 z-index: 10000; /* Needs to be drawn on top of SVG to be clickable. */
}

#workflowChart a {
 cursor: pointer;
}

#workflowChart a div {
 border-color: #f89c51;
 background: #fcd5b5;
}

#workflowChart div.inactive {
 border-color: #ccc;
 background-color: #eee;
 color: #ccc;
}

#workflowChart div:hover {
 border-color: #700000;
}

Javascript:

/* Initialize workflow screen. */
var initWorkflow = function() {
    // List HTML nodes to connect.
    //
    // The edges are hardcoded in this example,
    // but could easily be made dynamic.
    var edges = [
        {
            source: 'startFlow',
            target: 'foo1Flow'
        },
        {
            source: 'foo1Flow',
            target: 'foo2Flow'
        },
        {
            source: 'foo2Flow',
            target: 'fooChoice1Flow'
        },
        {
            source: 'foo2Flow',
            target: 'fooChoice2Flow'
        },
        {
            source: 'foo2Flow',
            target: 'fooChoice3Flow'
        },
        {
            source: 'fooChoice1Flow',
            target: 'fooOptionFlow'
        },
        {
            source: 'fooChoice2Flow',
            target: 'fooOptionFlow'
        },
        {
            source: 'fooOptionFlow',
            target: 'fooCombineFlow'
        },
        {
            source: 'fooChoice3Flow',
            target: 'fooCombineFlow'
        },
        {
            source: 'fooCombineFlow',
            target: 'fooSplit1Flow'
        },
        {
            source: 'fooCombineFlow',
            target: 'fooSplit2Flow'
        },
        {
            source: 'startFlow',
            target: 'barFlow'
        },
        {
            source: 'barFlow',
            target: 'bar1Flow'
        },
        {
            source: 'barFlow',
            target: 'bar2Flow'
        },
    ];
      
    // Us JQUery to set height and width equal to background div.
    var workflow = $('#workflowChart'),
        h = workflow.height(),
        w = workflow.width();
  
    // Create Protovis Panel used to render SVG.
    var vis = new pv.Panel()
        .width(w)
        .height(h)
        .antialias(false);
      
    // Attach Panel to dom
    vis.$dom = workflow[0];
      
    // Render connectors
    drawEdges(vis, edges);
    var test = vis.render();
 };
 
 /* Draw edges specified in input array. */
 var drawEdges = function(vis, edges) {
     // Direction indicators,
     var directions = []; 
 
     $.each(edges, function(idx, item){
         // Color of edges
         var color = '#000';
         
         // Arrow radius         
         var r = 5;
         
         // Use JQuery to get source and destination elements
         var source = $('#' + item.source);
         var target = $('#' + item.target);
         
         if (!(source.length && target.length)) {
             // One of the nodes is not present in the DOM; skip it.
             return;
         }
         
         var data = edgeCoords(source, target);
         if (item.sourceLOffset) {
             data[0].left += item.sourceLOffset;
         }
         if (item.targetLOffset) {
             data[1].left += item.targetLOffset;
         }
         
         if (source.hasClass('inactive') || target.hasClass('inactive')) {
             // If target is disabled, change the edge color.
             color = '#ccc';
         }
         
         // Use Protovis to draw edge line.
         vis.add(pv.Line)
             .data(data)
             .left(function(d) {return d.left;})
             .top(function(d) {
                 if (d.type === 'target') {
                     return d.top - (r * 2);
                 }
                 
                 return d.top;
              })
             .interpolate('linear')
             .segmented(false)
             .strokeStyle(color)
             .lineWidth(2);
         
         // Here you may want to calculate an angle
         // to twist the direction arrows to make the graph
         // prettier. I've left out the code to keep thing simple.
         var a = 0;
         
         // Add direction indicators to array.
         var d = data[1];
         directions.push({
             left: d.left,
             top: d.top - (r * 2),
             angle: a,
             color: color
         });
     });
     
     // Use Protovis to draw all direction indicators
     //
     // Here you may want to check and make
     // sure you're only drawing a single indicator
     // at each position, to avoid drawing multiple
     // indicators for targets that have multiple sources.
     // I've left out the code for simplicity.
     vis.add(pv.Dot)
         .data(directions)
         .left(function (d) {return d.left;})
         .top(function (d) {return d.top;})
         .radius(r)
         .angle(function (d) {return d.angle;})
         .shape("triangle")
         .strokeStyle(function (d) {return d.color;})
         .fillStyle(function (d) {return d.color;});
 };
 
 /* Returns the bottom-middle offset for a dom element. */
 var bottomMiddle = function(node) {
     var coords = node.position();
     coords.top += node.outerHeight();
     coords.left += node.width() / 2;
     return coords;
 };
 
 /* Returns the top-middle offset for a dom element. */
 var topMiddle = function(node) {
     var coords = node.position();
     coords.left += node.width() / 2;
     return coords;
 };
 
 /* Return start/end coordinates for an edge. */
 var edgeCoords = function(source, target) {
     var coords = [bottomMiddle(source), topMiddle(target)];
     coords[0].type = 'source';
     coords[1].type = 'target';
     return coords;
 };

PyCon 2011 Report

2011-03-16T14:41:00.000-07:00

Here is a presentation covering the status of Python and sessions I attended at PyCon 2011:

Pycon 2011

View more presentations from limscoder

SLM Presentation

2011-03-09T15:05:00.000-08:00

Presentation slides from UAWEBDEV presentation about SLM (Sample Lifecycle Manager).

SLM (Sample Lifecycle Manager)

View more presentations from limscoder

SLM (Sample Lifecycle Manager)

2011-02-18T08:08:00.000-08:00

SLM

We released the latest version of SLM (Sample Lifecycle Manager) on February 1st, and the site has been a resounding success so far. SLM supports life sciences laboratory services offered by UAGC including:

DNA extraction
Sanger sequencing
DNA fragment analysis (str/microsatellite)
Sequenom genotyping
Sequenom methylation analysis
Taqman genotyping
454 sequencing
Ion Torrent sequencing (coming soon)

EAGER

SLM is built with Eager, an application framework for developing custom LIMS. Eager is a collection of Django apps that provide common LIMS functionality including:

Workflow management with GLP compliant status logging
GLP compliant user and lab access control and management
Sample/tube/grid submission and management
Volume and concentration tracking
Automated sample and reagent dilution and 'cherry picking' transfers
Reagent lot tracking
Data management and collaboration
Integration with SOP management system
Environmental monitoring

The core features of Eager can be used 'out-of-the-box' for a complete LIMS solution with a generic sample tracking workflow, or can be customized to provide service specific workflows (such as Sequenom, 454, Ion Torrent, etc.) The framework includes tons of features, and additional workflows can be easily added by an experienced Django developer. Custom workflows are simply custom Django apps that hook into Eager's workflow definition system. All client-side code is written with the Dojo framework.

I am hoping to release the Eager framework on GitHub this spring or summer (it will be the first "open-source LIMS that doesn't suck"), but it currently needs to be reviewed by our IP/legal department first.

Django ORM Tools

2011-01-02T14:41:00.000-08:00

Django ORM Tools

The Django ORM is a great tool that makes it easy to work with simple data models, but it quickly shows its limitations as the complexity of the data model grows. The orm_tools module is an attempt to keep the simplicity of the Django ORM, while adding some extra features that make it much easier to work with complex object graphs. The code is available on Django Snippets.

Object Instances/Sessions

The Django ORM loads each object separately from the database. If different QuerySets select multiple objects with the same primary key, the resulting objects will all be different instances.


>>>MyModel.objects.get(pk=1) is MyModel.objects.get(pk=1)
False

The SQLAlchemy ORM solves this problem with sessions. orm_tools contains a Session class to provide similar functionality. Use the 'with' statement in combination with a Session instance to force QuerySets to retrieve cached object instances from the session.


>>>from orm_tools import Session
>>>with Session():
>>>    MyModel.objects.get(pk=1) is MyModel.objects.get(pk=1)
True

When QuerySet objects are executed inside of the 'with' block, all SQL queries are performed normally, but cached object instances are returned if an instance with an identical primary key already exists in the session. The session applies throughout any code called from within the 'with' block. Any objects inserted into the DB within the 'with' block are automatically added to the session.

Object Graphs

The Django ORM does not automatically save model object dependencies, so Django model instances must be saved one at a time.


>>>parent = MyModel()
>>>child = MyChild(parent=parent)
>>>child.save()
IntegrityError: app_mychild.parent_id may not be NULL

For simple data models, this problem is easily fixed by inserting the models into the database at the same time that they are created.


>>>parent = MyModel.objects.create()
>>>child = MyChild.objects.create(parent=parent)

However this is not always ideal for more complex data models, especially if the objects involved already exist in the database, and changes need to be persisted by updating existing rows. orm_tools contains a GraphSaver class that will save an entire object graph at once.


>>>from orm_tools import GraphSaver
>>>parent = MyModel()
>>>child = MyChild(parent=parent)
>>>saver = GraphSaver()
>>>saver.save(child)

When the 'save' method of the GraphSaver object is called, all dependencies will be detected and their 'save' methods will be called in the correct order, so that the entire object graph is saved. The GraphSaver's 'save' method works equally well for both inserts and updates, although updates can optionally be ignored by setting the 'update' argument to False. In the future, I hope to increase performance significantly by modifying the code to exeucte batched insert/update queries for databases that support it (postgres w/ psycopg 2).

Collections

The Django ORM supports one-to-many object relations. Objects on the 'many' side of a one-to-many relation cannot be attached to the 'one' unless the 'one' is already saved in the database. This causes some of the same problems as described in the 'Object Graphs' section. The orm_tools module contains a Collection class that enables 'many' objects to be added to a 'one' object, regardless of whether the 'one' object has been saved yet.


from django.db import models

from orm_tools import Collection

class One(models.Model):
    label = models.CharField(default='blank', max_length=20)

# Call the 'set_property' static method
# to create a collection object.
#
# Arguments
# ==========
#  * Model to add collection to
#  * Collection attribute name
#  * Many's foreign key attribute name
#  * One's 'many set' attribute name
Collection.set_property(One, 'children', 'parent', 'many_set')

class Many(models.Model):
    label = models.CharField(default='blank', max_length=20)
    parent = models.ForeignKey(One, null=False)


>>>one = One()
>>>one.children.add(Many())
>>>one.children.add(Many())
>>>saver = GraphSaver()
>>>saver.save(one)

The Collection object can be iterated through, indexed, and sliced regardless of whether the 'one' object and the 'many' objects have been saved yet. The GraphSaver's 'save' method will also automatically save all 'many' objects.

Apache, Virtual Hosts, and HTTPS

2010-10-16T09:07:00.000-07:00

Apache cannot use https with name-based virtual hosts due to the way the SSL handshake works. I've run across this problem several times in the past, and I always forget how to solve it. So I'll record it here for posterity.

To get things working, the Apache setup needs to be changed from name-based virtual hosting to ip-based virtual hosting. After configuring a separate ip for each vhost that requires https, the Apache config files (/etc/httpd/conf/ on RHEL, Apache 2.2) need to be updated to use ip-based virtual hosting.

If name-based vhosting was previously configured, it will need to be modified. If all vhosts are being converted to ip-based vhosting, then name-based vhosting can be completely turned off by commenting or deleting any 'NameVirtualHost' directives. However, it is also possible to continue to use name-based vhosting for vhosts that do not require https. Any existing 'NameVirtualHost' directives that contain wildcards ('NameVirtualHost *:80') will need to be modified. Replace the wildcard with the ip that will be shared by name-based vhosts.

Next, modify any existing 'VirtualHost' directives that contain wildcards in their definition ('VirtualHost *:80'). Replace the wildcard with the ip that the vhost will be using. Virtual hosts that do not require HTTPS can continue to use name-based virtual hosting, and can share the same ip, but all vhosts that require HTTPS must use a unique ip address.

Finally, configure a 'VirtualHost' directive for each ip-based vhost in the ssl section of the Apache configuration file ('/etc/httpd/conf.d/ssl.conf' on RHEL, Apache 2.2). Any name-based vhosts will continue to share the ssl config within the '_default_:80' 'VirtualHost' directive. Restart Apache for the changes to take affect.

Transactions for File Transfer

2010-09-04T12:05:00.000-07:00

Transactions

Database transactions are a convenient way to maintain consistent state during data processing functions. If an error occurs during processing, just rollback the transaction to avoid incomplete or incorrect data being stored.

Problem

I've worked on many problems where data processing involves retrieving a source file, performing some type of processing, and then writing to a destination file. These functions are tricky, because if a problem arises during the processing, you're left with an inconsistent, partially processed batch of files. This problem is especially pronounced if you're storing file metadata in a database. If you perform a rollback of your database transaction when an error occurs, then you've lost any updated metadata about the files that were processed correctly.

Solution

In an attempt to remedy this problem I've developed a somewhat naive implementation of a file transaction class that can be used to maintain consistent state during processing function involving many files. The transaction object keeps track of all files that have been created and all files that should be deleted. All files marked for deletion are deleted when a commit occurs. All files marked as created are removed when a rollback occurs. If a file needs to be moved, it is instead copied, and the source file is marked for deletion, and the destination file is marked as being created.

Implementation


import glob
import os
import shutil

class Transaction(object):
    """
    Manages transactions for file storage.

    Assumes each file is only being operated on by one person at a time.

    If multiple users try to operate on the same file, then the last
    to access gets an exception.
    """

    lock_postfix = 't_lock'

    def __init__(self):
        self._level = 0

    def _get_lock_path(self, path):
        """Return lock file path."""

        if path.endswith('/'):
            end = len(path) - 1
            path = path[:end]

        return path + '.%s' % self.lock_postfix

    def _set_files(self):
        """Resets file lists."""

        self._files_added = set()
        self._files_removed = set()
        self._dirs_added = set()
        self._dirs_removed = set()
        self._locked_files = set()

        # Unlike the other types,
        # move operations
        # must be ordered!!
        self._files_moved = []

    def _check_level(self):
        """Raises exception if level is not 1 or above."""

        if self._level < 1:
            raise exceptions.TransactionError('Transaction not active.')

    def _rm(self, file_paths, dir_paths):
        """Remove all files."""

        for dir_path in dir_paths:
            if os.path.exists(dir_path):
                shutil.rmtree(dir_path)

        for file_path in file_paths:
            if os.path.exists(file_path):
                os.unlink(file_path)

    def _rev_moves(self):
        """Reverse moved files."""

        for move in reversed(self._files_moved):
            shutil.move(move[1], move[0])

    def _acquire_lock(self, path):
        """Attempt to lock a file."""

        # Make sure transaction is started
        self._check_level()

        if path not in self._locked_files:
            # Create lock file on file system
            lock_path = self._get_lock_path(path)
            if os.path.exists(lock_path):
                # Multi-user access is not allowed!
                raise exceptions.TransactionError('File is locked.')
            out_file = open(lock_path, 'w')
            out_file.write('\n')
            out_file.close()
            self._locked_files.add(path)

    def _release_lock(self, path):
        """Release a lock file."""

        lock_path = self._get_lock_path(path)
        if os.path.exists(lock_path):
            os.unlink(lock_path)
        self._locked_files.discard(path)

    def _release_locks(self):
        """Release all locks."""

        locked_paths = self._locked_files.copy()
        for path in locked_paths:
            self._release_lock(path)

    def copy_file(self, src_path, dest_path, remove_existing=False, directory=False):
        """Copy a file. Set remove_existing to True to move file."""

        if directory is True:
            shutil.copytree(src_path, dest_path, symlinks=True)
        else:
            shutil.copyfile(src_path, dest_path)
        self.add_file(dest_path, directory=directory)

        if remove_existing is True:
            self.remove_file(src_path, directory=directory)

    def add_file(self, path, directory=None):
        """Add a file to the transaction."""

        self._check_level()

        self._acquire_lock(path)

        if directory is None:
            directory = os.path.isdir(path)

        if directory is True:
            self._dirs_added.add(path)
        else:
            self._files_added.add(path)

    def remove_file(self, path, directory=None):
        """Remove a file from the transaction."""

        self._check_level()

        self._acquire_lock(path)

        if directory is None:
            directory = os.path.isdir(path)

        if directory is True:
            self._dirs_removed.add(path)
        else:
            self._files_removed.add(path)

    def move_file(self, src_path, dest_path):
        """Move a file from one location to another."""

        self._check_level()

        self._acquire_lock(src_path)
        self._acquire_lock(dest_path)

        shutil.move(src_path, dest_path)
        self._files_moved.append((src_path, dest_path))

    def begin(self):
        """Begin transaction."""

        if self._level == 0:
            self._set_files()

        self._level += 1

    def commit(self):
        """Removes all 'removed' files and dirs."""

        self._check_level()

        self._level -= 1
        if self._level == 0:
            self._rm(self._files_removed, self._dirs_removed)
            self._release_locks()

    def rollback(self):
        """Removes all 'added' files and dirs."""

        self._check_level()

        self._level -= 1
        if self._level == 0:
            self._rm(self._files_added, self._dirs_added)
            self._rev_moves()
            self._release_locks()

Example


def process():
    transaction = Transaction()
    transaction.begin()
    try:
        # Mark a file as created
        transaction.add_file(new_file)

        # Mark a file as deleted
        transaction.remove_file(delete_file)

        # Copy a file
        transaction.copy_file(src_file, dest_file)

        # Move a file
        transaction.move_file(mov_src_file, mov_dest_file)
        transaction.commit()
    except:
        transaction.rollback()
        raise

Limitations

The class only works for single user environments. A lock file is created for every file added to a transaction. If a different transaction tries to acquire a lock for a file that is already locked, an exception is raised. Negotiating multi-user access would be quite tricky, especially in the case of delete files, where the file no longer exists after the lock is released.

AmFast 0.5.1 Released

2010-06-01T09:49:00.001-07:00

AmFast 0.5.1 has been released. This is a bug-fix release and can be downloaded from PyPi.

Python Workshop

2010-05-10T22:33:00.001-07:00

It's that time of the year again. Time to learn Python! I will be teaching a Python tutorial May 24-28. The tutorial will cover Python for users with little or no programming experience. Examples are geared toward biologists, but most concepts are applicable to all. Critiques and comments of the workshop tutorial material are greatly appreciated.

Register for the workshop.

Django + Dojo

2010-04-22T09:39:00.000-07:00

When I work on HTML projects, I usually use the Dojo Toolkit for my Javascript needs. Lately I've been spending some time playing around with the Python web framework Django. I did some internet searching and found Dojango, a project that integrates Django with Dojo. Dojango has features for automatically turning Django form fields into Dijits (Dojo UI widgets), but unfortunately Dojango uses Dojo's custom HTML attributes with Dojo's parseOnLoad option. I prefer to create Dijits programatically so that my markup stays clean. I decided to develop a Django app to meet my needs.

Code is available from SVN. Instructions are below.

Instructions:

Settings

# Setup app in settings.py

# Required attributes:

# The URL to get dojo.js from
DOJO_URL = MEDIA_URL + 'js/dojo'

# Optional attributes:

# Set to True to add Dojo setup to template
DOJO_ENABLED = True

# Set Dojo theme
DOJO_THEME = 'tundra' 

# Set the value of djconfig
DOJO_DJCONFIG = {'isDebug': False, 'parseOnLoad': False, 
                 'modulePaths': {'app': MEDIA_URL + 'js/app'}}

# More on this later
DOJO_FORM_FUNCTION = None

# Attach middleware
MIDDLEWARE_CLASSES = (
    'dojo.middleware.DojoMiddleware',
    ...
)

The dojo object

The middleware attaches a Dojo object to each request. You can access the object from your views, and use it to set Dojo parameters.

# The dojo object has several useful attributes and methods.

# The path to dojo.js (from settings.DOJO_URL), read-only
request.dojo.src

# Theme
request.dojo.theme = 'soria'

# DjConfig
request.dojo.dj_config['isDebug'] = True

# Convenience method to set module paths in dj_config
request.dojo.set_module_path('custom', 'url_to_custom_module')

# Add stylesheets
request.dojo.append_stylesheet('url_to_custom_stylesheet')

# Require modules
request.dojo.append_module('module.to.require')

# Set function to addOnLoad
request.dojo.append_aol('function() {do_something();}')

# Set function to addOnLoad before other already set
request.dojo.prepend_aol('function() {do_something();}'

Forms

Django forms can easily be 'dijitized'.

from dojo import models as dojo

class Register(forms.Form):
    username = forms.RegexField(
        label='Choose a username (letters and numbers only)',
        min_length=2,
        max_length=16,
        regex=r'^[\w]{2,16}$',
        error_messages={
            'invalid': 'Username must be 16 characters or shorter, and can only' 
                       ' contain letters, numbers, underscores and dashes.'
         }
    )

    # Use the dojo_field function to attach dijit
    # parameters to a Django form field.
    dojo.dojo_field(username, 'dijit.form.ValidationTextBox')



   """
   dojo_field arguments

   required
   ==========
    * field - Django field to attach dijit parameters to.
    * dojo_type - str, the qualified name of the dijit class to use

   keyword
   =========
    * attr_map - dict, Used to map Django field parameters to Dijit parameters.
                 Overrides the default values in dojo.models.default_attr_map.
                 The dict elements should be structured as follows:

                 Key == Django attribute name
                 Value == tuple with elements:
                     [0] == Dijit attribute name to map to
                     [1] == None, or callable to convert Django value to Dijit value

                 EXAMPLE:
                 {
                     'max_length': ('maxLength', None),
                     'regex': ('regExp', lambda a: '%s' % a.pattern)
                 }
    * dojo_attrs - dict, Attributes will be applied directly to dijit.
                   Key == dojo attribute name
                   Value == dojo attribute value
   """
}

After creating a form, it must be instrumented to create the Javascript required to create the dijits.



def my_view(request):

    my_form = Register()

    # This call generates all the necessary Javascript code
    request.dojo.dojo_form(my_form)

    # By default, the function code generated
    # is a string to be added in-line.
    #
    # If you prefer to call a pre-defined JS function,
    # just set the request.dojo.form_function attribute.
    #
    # The value of the attribute should be a tuple where:
    # [0] == qualified Dojo module name where function exists
    # [1] == function name
    #
    # request.dojo.form_function can also be set automatically
    # by setting DOJO_FORM_FUNCTION in settings.py

Template

Include the following tags within the 'head' tag of your HTML template:


{% load dojo %}
{% dojo request.dojo %}

The Dojo app also includes a script for creating a Dojo build:


python manage.py dojo_build

UI Design

2010-03-06T08:35:00.000-08:00

In the world of scientific software, UIs tend to be horrifically awful. When we started working on our next generation LIMS (called SLM: Sample Lifecycle Manager), one of our goals was to provide an intuitive UI. This post covers some of what we went through on the UI design front.

First Try

After evaluating several Javascript libraries, we decided that Flex would be a good choice, because it would be easy to create a desktop-like application to manage the large feature set. The main screen for our 1st alpha release looked something like this:

The design is similar to a complex desktop app such as Thunderbird or Eclipse. The app has a menu bar, a left panel with data/navigation, a bottom panel with several features, and a main tabbed panel with content.

Our users hated it. We quickly learned that our users don't want desktop-style apps with many features spread across multiple panels. Instead, they are looking for web-style apps where each function of the application can be performed on a single, simple screen, and links are used to navigate between feature screens.

Second Try

Our revised UI eliminated the menu bar. The left and bottom panels are still available for power users, but they are now hidden by default. These simple changes dramatically improved the user's acceptance of the app.

Forms

Flex's form system is great for building complex forms (as long as you can live with the standard theme). Flex has built-in validator classes that can provide real-time visual feed back to users as they enter information into the form. It is very important to customize your validators to provide useful messages to your users, so they know how to correct their input.

Always make sure your forms are fully navigable by keyboard, and make sure that by default, the form will be submitted when the user hits 'Enter'.

SLM has many features, and some of the forms rely on optionally selected fields. For example, if a user selects an option from a drop-down list, then other fields in the form become enabled or disabled. Many desktop-style apps 'grey-out' the disabled fields (this can be done using Flex UIComponent's 'enabled' property), but we found that un-usable fields that remain visible can be confusing for users. Instead, adding or removing the optional fields feels much more intuitive.

Visual Cues

Animations are not just eye-candy, they can also help users navigate through the system. For example, SLM has several drag-and-drop features. At first, some users were unsure about which items could be dragged into a drop-zone. To fix this problem we added a simple glow animation to highlight draggable items whenever the user's mouse is hovering over the drop-zone.

Trust

Customers use SLM to submit samples for testing at our facility. The submission process is similar to a shopping cart check-out on a retail site. It is extremely important to allow a user the ability to move forward and backward through the process. User's don't feel confident unless they trust the system to move back and forth between steps without munging any data they've already entered. If a user needs to change an option in step 2, it shouldn't affect the data they've already entered in step 3. Users must also be confident that nothing will be stored on the server until the final submit step.

For single step operations, the UI must always provide a 'Cancel' option or some other way for a user to back-out of the changes they've made.

Profiling and Optimizing Python Code

2009-11-24T15:16:00.000-08:00

All programmers have heard the advice: "Don't prematurely optimize code." What exactly does that mean? It means that you shouldn't guess about what is causing your code to run slowly. Chances are you'll guess wrong. Instead of guessing, use profiling and benchmarking tools to quickly and accurately identify the performance bottlenecks in your scripts.

Every now and then I run into a piece of Python code that just doesn't run as fast as I would like. I use 3 different Python tools to find and fix Python performance problems.

cProfile:

cProfile is a module that is included in the Python Standard Library. It logs function calls and execution times. There is also a pure-python version named 'profile'.

KCachegrind:

KCachegrind was written to visualize the output generated by Callgrind (a C profiler), but the function call logs from cProfile can be converted to the KCachegrind format with this script. KCachegrind uses the KDE framework. On Linux boxes, KCachegrind is usually bundled in a package with other development tools written for KDE. The package is named 'kdesdk' or something similar.

timeit:

timeit is a module included in the Python Standard Library that is used for measuring the execution time of arbitrary pieces of code.

Here is the code that needs to be improved:


class LameStringBuilder(object):
    def __init__(self, cols=40, rows=10000):
        self.cols = cols
        self.rows = rows

    def build(self, val):
        built = ''
        for i in range(self.rows):
            built += self.build_row(val) + "\n"
        return built

    def build_row(self, val):
        built = ''
        for i in range(self.cols - 1):
            built += val + ','
        built += val
        return built

Here is the code to execute a profile test and format the results into something that KCachegrind can understand:


import cProfile

# Use this module to convert profile data
# into the KCacheGrind format.
#
# The module is available here:
# http://www.gnome.org/~johan/lsprofcalltree.py
import lsprofcalltree

import lame_string_builder

def test():
    """Code to profile."""
    l = lame_string_builder.LameStringBuilder(40, 10000)
    l.build('foo')

if __name__ == "__main__":
    # Profile function
    p = cProfile.Profile()
    p.run('test()');

    # Get profile log in KCacheGrind format
    # and dump to file.
    k = lsprofcalltree.KCacheGrind(p)
    out = open('lame_profile.kgrind', 'w')
    k.output(out)
    out.close()

After running the test, launch KCachegrid and open the profile file to view the results.

The 'Flat Profile' and 'Callee Map' are the 2 most useful displays for finding the bottle necks in your code.

The 'Flat Profile' describes each function called in the test with the following columns:

Incl. - The total percentage of time spent within a function.

Self - The percentage of time spent within a function NOT including inner function calls.

Called - The total number of times the function was called during the test.

Function - The name of the function being described.

The 'Callee Map' represents the different functions called during the test. Each function is drawn as a rectangle. Inner function calls are drawn on top of their parent function's rectangle. The area used to draw each function is proportional to the execution time for each function call relative to the total execution time of the parent (also printed as a percentage in the graph).

In the example above it is easy to pick out where the bottle necks are by looking at the 'Callee Map'. Most of the test time was spent within the 'LameStringBuilder.build_row' method, so it is the main bottle neck. Other significant sources of time include calls to 'LameStringBuilder.build' and the built-in function 'range'.

After identifying the bottlenecks, I created a better version of the 'LameStringBuilder' class:


import lame_string_builder

class LessLameStringBuilder(lame_string_builder.LameStringBuilder):
    def build(self, val):
        return "\n".join([self.build_row(val) for i in xrange(self.rows)])

    def build_row(self, val):
        return ",".join([val for i in xrange(self.cols)])

After making changes to the code, Python's built-in 'timeit' module can be used to simply and accurately measure the differences in execution speed between the old and the new versions:


import timeit

import lame_string_builder
import less_lame_string_builder

def test_lame(val, cols, rows):
    l = lame_string_builder.LameStringBuilder(cols, rows)
    l.build(val)

def test_less_lame(val, cols, rows):
    l = less_lame_string_builder.LessLameStringBuilder(cols, rows)
    l.build(val)

def test_generic(repeat, name, args):
    print "%s (x%i):" % (name, repeat)
    t = timeit.Timer("%s(%s)" % (name, args), "from __main__ import %s" % name)
    print t.timeit(repeat)

def test_all(repeat, val, cols, rows):
    args = "'%s', %i, %i" % (val, cols, rows)   
    
    for name in ("test_lame", "test_less_lame"):
         test_generic(repeat, name, args)

if __name__ == "__main__":
    test_all(100, 'foo', 40, 10000)

The new code is significantly faster!

Role based security with Python

2009-11-02T17:30:00.000-08:00

Most moderately complex enterprise applications require some sort of role based security and access control. For example, an employee should have access to fill-out her time card, but a manager should also be able to approve the time card. This post outlines how to create an 'access control list' (acl) for Python.

The acl is an object that can be queried to determine if a particular role has permission to access a resource. Each permission is also associated with a 'privilege'. For example, a manager may have both the 'read' and 'write' privileges for the weekly schedule, but an employee only has the 'read' privilege.


import acl

# Each user name should be associated with a role.
role = get_role(username)

# Query the acl to see if a role has access to a resource.
# params: role name, resource, privilege
acl = acl.Acl()
if not acl.check_access(role, 'time_card', 'write'):
    # User doesn't have access to this resource!!
    pass

The Acl class supports role inheritance. If the 'manager' role inherits the 'employee' role, then managers will have all of the same permissions as employees.


# Build acl list
acl = acl.Acl()

employee = acl.Role('employee')
resource = acl.Resource('time_card')
resource.set_privilege('write')
employee.set_resource(resource)
acl.set_role(employee)

manager = acl.Role('manager')
manager.set_parent(employee)
resource = acl.Resource('time_card')
resource.set_privilege('approve')
manager.set_resource(resource)
acl.set_role(manager)

if acl.check_access('manager', 'time_card', 'write'):
    # YES!
    pass

if acl.check_acces('manager', 'time_card', 'approve'):
    # YES!
    pass

if acl.check_access('employee', 'time_card', 'write'):
    # YES !
    pass

if acl.check_access('employess', 'time_card', 'approve');
    # NO!
    pass

I'm normally not a very big fan of XML, but in this particular case it is a good format to use if you prefer to store your acl in a configuration file. The Acl object's 'build_acl' method populates the object from a XML file with the following format:


<?xml version="1.0"?>
<!--
- This file configures the roles (user groups)
- and permissions for accessing the system.
-->
<config>
    <!--
    - Setup roles here.
    - Use the 'inheritFrom' tag to
    - inherit permissions from another role.
    -->
    <roleSet>   
        <role>
            <name>customer</name>
        </role>
        <role>
            <name>employee</name>
            <inheritFrom>customer</inheritFrom>
        </role>
        <role>
            <name>manager</name>
            <inheritFrom>employee</inheritFrom>
        </role>
    </roleSet>

    <!--
    - Set permissions for accessing application components here.
    - resource -> property being access controlled.
    - role -> group or user that can access resource.
    - privilege -> privilege that role can use with resource.
    -
    - Each permission tag can contain multiple
    - resources, roles, and privileges.
    -->
    <permissions>
        <permission>
            <resources>
                <resource>contact_details</resource>
                <resource>profile</resource>
            </resources>
            <roles>
                <role>customer</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>

        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>

            <roles>
                <role>employee</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>
        
        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>
            <roles>
                <role>manager</role>
            </roles>
            <privileges>
                <privilege>approve</privilege>
            </privileges>
        </permission>

    </permissions>
</config>

Here is the code for the acl.py module:


"""Role based security"""
from xml.dom.minidom import parse

class AccessError(Exception):
    pass

class Resource(object):
    """An Resource is an object that can be accessed by a Role."""
    def __init__(self, name=''):
        self.name = name
        self._privileges = {}

    def set_privilege(self, privilege, allowed=True):
        self._privileges[privilege] = allowed

    def has_access(self, privilege):
        if privilege in self._privileges:
            return self._privileges[privilege]
        return False
    
    def __str__(self):
        rpr = self.name + ': '
        for privilege, access in self._privileges.iteritems():
            rpr += "%s:%s " % (privilege, access)
        return rpr

class Role(object):
    def __init__(self, name=''):
        """An Acl role has access to resources with specific privileges."""
        self.name = name
        self._parents = {}
        self._resources = {}

    def set_parent(self, parent):
        self._parents[parent.name] = parent

    def set_resource(self, resource):
        self._resources[resource.name] = resource

    def has_access(self, attr_name, privilege):
        if attr_name in self._resources:
            if self._resources[attr_name].has_access(privilege):
                return True

        for parent in self._parents.values():
            if parent.has_access(attr_name, privilege):
                return True

        return False
    
    def __str__(self):
        rpr = self.name + ":\n"
        rpr += "parents:\n"
        for parent in self._parents.keys():
            rpr += "\t%s\n" % parent
        rpr += "resources:\n"
        for resource in self._resources.values():
            rpr += "\t%s\n" % resource.describe()
        return rpr

class Acl(object):
    """Manages roles and resources.
    
    Singleton class.
    """

    class __impl:
        """Implementation of the singleton interface"""
        
        def __init__(self):
            self._acl = {}

        def set_role(self, role):
            self._acl[role.name] = role
            
        def check_access(self, role_name, resource, privilege):
            """Check whether a role has access to a resource or not."""

            if not role_name in self._acl:
                raise AccessError('Role does not exist.')
            return self._acl[role_name].has_access(resource, privilege)

        def build_acl(self, file): 
            """Build acl from an XML file."""

            self._acl = {}
            roles_to_create = {}
            dom = parse(file)
            
            # Find roles to create
            roles_nodes = dom.getElementsByTagName('roleSet')
            for roles_node in roles_nodes:
                role_nodes = roles_node.getElementsByTagName('role')
                for role_node in role_nodes:
                    name_nodes = role_node.getElementsByTagName('name')
                    parent_nodes = role_node.getElementsByTagName('inheritFrom')
                    role_name = name_nodes[0].childNodes[0].data
                    roles_to_create[role_name] = []

                    # Find role parents
                    for parent_node in parent_nodes:
                        roles_to_create[role_name].append(parent_node.childNodes[0].data)

            # build inheritence chain
            for role, parents in roles_to_create.iteritems():
                self.set_role(self._create_role(role, roles_to_create))

            # assign permissions
            permissions = dom.getElementsByTagName('permissions')
            for permissions_node in permissions:
                permission_nodes = permissions_node.getElementsByTagName('permission')
                for permission_node in permission_nodes:
                    resource_nodes = permission_node.getElementsByTagName('resource')
                    role_nodes = permission_node.getElementsByTagName('role')
                    privilege_nodes = permission_node.getElementsByTagName('privilege')

                    for resource_node in resource_nodes:
                       resource = Resource()
                       resource.name = resource_node.childNodes[0].data
                       for privilege_node in privilege_nodes:
                           resource.set_privilege(privilege_node.childNodes[0].data)

                       for role_node in role_nodes:
                           try:
                               role = self._acl[role_node.childNodes[0].data]
                           except:
                               raise AccessError('Role in permission is not defined.')

                           role.set_resource(resource)

        def _create_role(self, role_name, roles_to_create):
            """Recursively create parent roles and then create child role."""
            
            if role_name in self._acl:
                role = self._acl[role_name]
            else:
                role = Role()
                role.name = role_name
                
            for parent_name in roles_to_create[role_name]:
                if parent_name in self._acl:
                    parent = self._acl[parent_name]
                else:
                    parent = self._create_role(parent_name, roles_to_create)
                    self.set_role(parent)
                role.set_parent(parent)
            return role
        
        def __str__(self):
            rpr = ''
            for role in self._acl.values():
                rpr += '----------\n'
                rpr += role.describe()
            return rpr

    __instance = None

    def __init__(self):
        """ Create singleton instance """
        
        # Check whether an instance already exists.
        # If not, create it.
        if Acl.__instance is None:
            Acl.__instance = Acl.__impl()

        self.__dict__['_Acl__instance'] = Acl.__instance

    def __getattr__(self, attr):
        """ Delegate get access to implementation """

        return getattr(self.__instance, attr)

    def __setattr__(self, attr, val):
        """ Delegate set access to implementation """

        return setattr(self.__instance, attr, val)

SQL Workshop

2009-10-19T10:35:00.000-07:00

I've recently finished a SQL tutorial for a relational database workshop I'll be teaching. The tutorial covers basic SQL queries with SQLite and MySql, as well as basic database creation with SQLite. The tutorial is very basic, and the examples are geared toward biologists who need to use a database for performing research.

Facebook's Tornado

2009-09-20T22:16:00.000-07:00

Today I got my first chance to play with Facebook's Python based web server and framework Tornado. The framework is single threaded and asynchronous, similar to how the Twisted framework operates, and Facebook uses the technology to provide data to the 'Friend Feed'.

Python thread creation and maintenance have relatively high overhead, so asynchronous solutions can provide better performance and scalability than more traditional threaded server implementations. This is especially true for servers with high numbers of concurrent connections, and for long-polling and streaming connections that spend most of their time waiting around for a message to be published to a client. Tornado's benchmarks look impressive against threaded Python web servers, but I wonder why they didn't include Twisted, their closest competitor, in the benchmarking.

My first impression is that Tornado is much easier to use than Twisted, but also much less powerful. Twisted is a monolithic package that can be used with many different networking protocols, but Tornado is focused only on http. Tornado provides built in support for templating, authentication, and signed cookies. One really cool feature Tornado provides is semi-automatic XSRF protection, which should save time for developers who would otherwise need to manually implement countermeasures.

The documentation for Tornado is very slim. For example, I couldn't find any information about how to interact directly with Tornado's main event loop (IOLoop). Fortunately, Tornado's code base is small and readable, so reading the source will quickly get you up to speed.

While implementing a Tornado channel for the AmFast project, I ran into a problem that I have also encountered with Twisted. Both Tornado and Twisted use callbacks to complete a request. As an example, if a long-poll client is waiting for a message, you call RequestHandler.finish() to send a response back to the client when a message is published.

The above method works well for a single server instance, but what about when you have multiple servers behind a proxy? A server process may publish a message for a client that is connected to an entirely different server process. The publishing process has no way to notify the subscribing process that a client has received a message.

This problem can be solved by polling a database table or a file that is accessible to all server processes, but that takes system resources, especially if you have many clients and your polling interval is short. One of my future goals is to figure out a more elegant way for the publishing server process to notify the subscribing server process when a client receives a message.

AmFast adds support for Google App Engine and Django

2009-08-01T13:08:00.001-07:00

I just released version 0.4.0 beta of AmFast, a Flash remoting package for Python. The new release includes support for Google App Engine and the Django framework.

Browse the code here.

Check out the live Google App Engine demo.

To get the code:

svn checkout https://amfast.googlecode.com/svn/tags/0.4.0b

YAPT (Yet Another Python Tutorial)

2009-06-20T09:53:00.000-07:00

The tutorial I created for the Python workshop is posted here. The tutorial is geared towards bioinformatics users, and includes molecular biology related exercises.

I created the tutorial with the Sphinx documentation package, which turned out to be the perfect tool for the job.

Learn Python Workshop

2009-05-29T10:45:00.000-07:00

I'm going to be teaching an intro to Python workshop next month. It will consist of 3 3 hour days and will cover the basics of Python. Dates are June 15th, 17th, 19th. The workshop is free for all U of A students, staff, and faculty. I have never lead a workshop before, so it should be a lot of fun.

Register for the Python workshop

Real-time Flex messaging with AmFast

2009-05-20T21:49:00.000-07:00

This example is very similar to the previous, but we're going to use AmFast's HTTP streaming channel to implement real-time flex messaging. Get the complete example code here.

First let's setup the Producer and Consumer in Actionscript.



// Configure the ChannelSet that
// messages will be sent and received over.
import mx.messaging.ChannelSet;

// To use HTTP streaming, the
// StreamingAMFChannel class must be used.
import mx.messaging.channels.StreamingAMFChannel;
var channelSet:ChannelSet = new ChannelSet();
var channel:StreamingAMFChannel = new StreamingAMFChannel("channel-name", "server-url");
channelSet.addChannel(channel);

// Setup a Consumer to receive messages.
import mx.messaging.Consumer;
var consumer:Consumer = new Consumer();

// The consumer's destination is the 'topic'
// name that the consumer will be subscribed to.
// The consumer will receive all messages
// published to the topic.
consumer.destination = "topic";

// Use the ChannelSet that was already created.
consumer.channelSet = channelSet;

// This event listener will be called whenever
// the consumer receives a message from the server.
consumer.addEventListener(MessageEvent.MESSAGE, newMsgHandler);

// The consumer won't start receiving messages
// until it is subscribed.
consumer.subscribe();

// Setup a Producer to publish messages.
import mx.messaging.Producer;
var producer:Producer = new Producer();
producer.destination = "topic";
producer.channelSet = channelSet;

// Create an Async message and send it
// to all other clients subscribed to the topic.
import mx.messaging.messages.AsyncMessage;
var msg:AsyncMessage = new AsynMessage();

// Set the message's body attribute to the
// object that is going to be published.
//
// In this case the object contains
// the X and Y coordinates of a Sprite.
msg.body = {'x': 10, 'y': 10};
producer.send(msg);

Next we'll setup the server.


# Create a ChannelSet to serve messages.
from amfast.remoting.channel import ChannelSet
from amfast.remoting.wsgi_channel import StreamingWsgiChannel
channel_set = ChannelSet()

# Each individual ChannelSet can use
# one or more Channels. When messages
# are published through the ChannelSet,
# they will be published to all subscribed clients,
# regardless of which Channel the clients
# are connected through.

# Create a HTTP streaming channel.
#
# When a client connects to a streaming channel,
# the HTTP connection remains open until the client
# disconnects. When messages are published,
# they are immediately dispatched to clients connected
# to HTTP streaming channels.
stream_channel = StreamingWsgiChannel('stream-channel')

# WsgiChannels objects are wsgi applications
# that can be served with any Wsgi server.
# CherryPy is being used in this example.
cherrypy.tree.graft(stream_channel, '/amf')

# Start the server.
# App() is your root controller class
# for non-AMF functions.
cherrypy.quickstart(App(), '/')

# That's it, our server is up and running.
# It's that simple.

To test the example, download the full code from the repo. To run the example you'll need to install AmFast and CherryPy or Twisted 1st.


# To server the example with cherrypy
python cp_server.py

# To server the example with Twisted
twistd -noy twisted_server.tac

Open two browser windows and browse to 'http://localhost:8000'. Click the 'Subscribe' button in both windows. In the 1st window click on the 'Master' button. In the 1st window, click on the yellow circle, and drag the circle around the screen. As the circle is dragged around the screen, the position of the circle will be updated in the 2nd window. Pretty cool stuff.

Create a chat client and server with Flex and Python

2009-05-16T14:23:00.000-07:00

In this example we'll create a simple Flex chat client that uses Producer/Consumer messaging to send and receive messages through a Python server. The server will use AmFast's HTTP polling and long-polling channels. Get the complete example code here.

First let's setup the Producer and Consumer in Actionscript.


// Configure the ChannelSet that
// messages will be sent and received over.
import mx.messaging.ChannelSet;
import mx.messaging.channels.AMFChannel;
var channelSet:ChannelSet = new ChannelSet();
var channel:AMFChannel = new AMFChannel("channel-name", "server-url");
channelSet.addChannel(channel);

// Setup a Consumer to receive messages.
import mx.messaging.Consumer;
var consumer:Consumer = new Consumer();

// The consumer's destination is the 'topic'
// name that the consumer will be subscribed to.
// The consumer will receive all messages
// published to the topic.
consumer.destination = "topic";

// Use the ChannelSet that was already created.
consumer.channelSet = channelSet;

// This event listener will be called whenever
// the consumer receives a message from the server.
consumer.addEventListener(MessageEvent.MESSAGE, newMsgHandler);

// The consumer won't start receiving messages
// until it is subscribed.
consumer.subscribe();

// Setup a Producer to publish messages.
import mx.messaging.Producer;
var producer:Producer = new Producer();
producer.destination = "topic";
producer.channelSet = channelSet;

// Create an Async message and send it
// to all other clients subscribed to the topic.
import mx.messaging.messages.AsyncMessage;
var msg:AsyncMessage = new AsynMessage();

// Set the message's body attribute to the
// object that is going to be published.
//
// In this case the object is a String,
// but the message body can be any
// type of object.
msg.body = "This is being published!";
producer.send(msg);

Next we'll setup the server.


# Create a ChannelSet to serve messages.
from amfast.remoting.channel import ChannelSet
from amfast.remoting.wsgi_channel import WsgiChannel
channel_set = ChannelSet()

# Each individual ChannelSet can use
# one or more Channels. When messages
# are published through the ChannelSet,
# they will be published to all subscribed clients,
# regardless of which Channel the clients
# are connected through.

# Create s standard polling channel.
polling_channel = WsgiChannel('poll-channel')

# A long-polling channel is created similar
# to a normal channel, but the wait_interval
# argument must be set to -1.
long_channel = WsgiChannel('long-channel', wait_interval=-1)

# WsgiChannels objects are wsgi applications
# that can be served with any Wsgi server.
# CherryPy is being used in this example.
cherrypy.tree.graft(polling_channel, '/amf')
cherrypy.tree.graft(long_channel, '/longPoll')

# Start the server.
# App() is your root controller class
# for non-AMF functions.
cherrypy.quickstart(App(), '/')

# That's it, our server is up and running.
# It's that simple.

To test the example, download the full code from the repo. To run the example you'll need to install AmFast and CherryPy or Twisted 1st.


# To server the example with cherrypy
python cp_server.py

# To server the example with Twisted
twistd -noy twisted_server.tac

Open two browser windows and browse to 'http://localhost:8000'. In the 1st window set the messaging url to 'http://localhost:8000/amf' to use the polling channel. In the 2nd window set the messaging url to 'http://localhost:8000/longPoll' to use the long-polling channel. Click the 'Subscribe' button in both windows. Enter a message in the message entry box, and press the 'Publish' button to publish the message. The published message will appear in both clients.

'Pushing' messages to Flex clients with AmFast

2009-05-15T15:47:00.000-07:00

One of the coolest aspects of Flex development is the ability to 'push' messages from the server to clients. With the newest release of AmFast, you can finally utilize this capability with all of the goodness of Python. This will be the 1st of 3 posts describing how you can use the AmFast Flash remoting package to push messages to Flex clients from a Python server.

AmFast supports pushing messages to clients over HTTP using 3 different strategies: polling, long-polling, and streaming. Here is a good article describing the differences between those strategies. Go ahead and skip past the XML configuration sections, since you won't need any XML to configure AmFast :)

Streaming channels deliver messages in real time, but also consume the most server resources. Standard polling consumes the least amount of server resources, but is also the slowest of the three strategies. Long-polling sits in the middle and offers a nice compromise between latency and resource consumption.

AmFast has built-in support for using any of the 3 messaging strategies with either a multi-threaded WSGI server, or the Twisted web framework. One thing to think about when implementing a multi-threaded server is that any client using long-polling or streaming may consume a thread for a significant amount of time, even if all it's doing is waiting for a new message. I haven't done any tests, but I'm guessing that Twisted's single-threaded, asynchronous design can handle more concurrent connections then a multi-threaded implementation.

On the client side, the Actionscript Consumer class is used to receive messages from a server, and the Producer class is used to send messages to a server. Any type of object can be pushed to clients, so the same technology can be used to build many different types of applications.

In the next post I'll explain the details of using AmFast's HTTP polling and HTTP long-polling channels to create a simple chat client. In the following post I'll show an example of using HTTP streaming to create an app where a master client can manipulate a graphical entity on the screen, and changes are replicated to all other subscribed clients.

AmFast 0.3.0 Released!

2009-05-09T21:24:00.000-07:00

The latest version of AmFast is out. In addition to the changes listed below, the documentation has been updated, and there are several new examples.

remoting

Producer/Consumer messaging via HTTP polling.
Channels and ChannelSets allow resources to be exposed in multiple ways.
Built in Channels for CherryPy, Twisted Web, and straight-up WSGI.
Easy to configure authentication for NetConnection and RemoteObject.
Connection objects hold session attributes tied to Flex client id.
Customizable Endpoints to encode/decode messages with external encoding/decoding libraries.
PyAmfEndpoint to encode/decode messages with pure Python, with no C compiler dependencies.

encode/decode

Read and write to file-like-objects or strings (using strings is much faster).
Automatic type conversion.
amfast.class_def.as_types.AsProxy and amfast.class_def.as_types.AsNoProxy allow overriding 'use_collections' and 'use_proxies' settings for a individual objects.
readExternal and writeExternal methods of the IEXTERNALIZABLE interface accept different parameters to make custom encodings easier.
amfast.encoder.Encoder and amfast.decoder.Decoder: Wrappers around amfast.encode and amfast.decode, so you don't need to pass the same arguments every time you encode or decode.

Plater 0.1.3 maintenance release

2009-05-06T17:08:00.000-07:00

Plater 0.1.3 (laboratory information management package) maintenance release today. The release contains a bug fix that was affecting parsing of CSV plate maps.

AmFast 0.3 release is coming soon

2009-05-02T15:24:00.000-07:00

I've been working on AmFast 0.3 for a while now, and it looks like I should be able to release it next weekend. 0.3 has cool new features such as: Producer/Consumer messaging, pure-Python encoding and decoding using PyAmf, and a bunch of other cool stuff.

The coding is done, and I'm working on documentation, testing, and a couple of new example applications.

AmFast 0.2.2 Released!

2009-03-22T21:14:00.000-07:00

The latest version of AmFast is out. Improvements include an Actionscript code generator, an example using the Twisted framework, and several bug fixes.

Flash and Python Presentation

2009-03-21T15:31:00.000-07:00

On Friday I presented information about Flash and Python to several developers from iPlant.

The presentation covered the current state of Python tools designed to communicate with various flavors of Flash. I will be giving the same presentation to TuPLE this coming Tuesday.

Here are the slides.

AmFast 0.2 Released!

2009-03-11T23:00:00.000-07:00

Check out the latest version of AmFast. The 0.2 version implements AMF0, AMF3 and supports configurable remoting with NetConnection and RemoteObject.

It's fast, flexible, and easy to use.

AmFast Released!

2009-03-02T17:41:00.000-08:00

I released the alpha version of AmFast today. AmFast is an AMF3 encoder/decoder Python extension.

Some un-sophisticated testing shows that AmFast is around ~18x quicker than PyAmf. 10,000 runs through the test_complex_encode_decode_dict() unit test takes ~105 seconds on my machine using PyAmf, and ~5.8 seconds using AmFast.

If you want to try it out, you can download the package from PyPi, and take a look at the meager documentation.

AmFast supports all of the data types that PyAmf does, but there is not yet any functionality for implementing remoting, or working with AMF0. I'm hoping that AmFast will be able to integrate into PyAmf, so that you can optionally use either the existing pure Python AMF3 encoder/decoder, or AmFast within the PyAmf framework.

This was my first time working with the Python C API, and I think it's pretty cool. Besides having to remember to increment and decrement Python object reference counts in the right spots, it was pretty easy to learn. I'm not anywhere near a C expert, and I wrote this extension in my spare time over the course of a couple of weeks.

Plater Released!

2009-02-19T10:20:00.000-08:00

I released Plater 0.1 today.

Plater is a Python package that contains common LIMS classes that can be used to build a custom LIMS application.

Instead of starting your LIMS application from scratch, use the Plater package to represent common LIMS domain models, and piece them together in whatever configuration your application requires.

Take a look at the Plater project page for details on how to download/install/use Plater.

AMF vs. JSON

2009-02-10T20:14:00.000-08:00

You can find many blog posts comparing AJAX data transfer formats JSON and XML to AMF3, but they all seem to miss what I see as the 3 biggest advantages to AMF3.

1: AMF3 supports references for objects, strings, and class definitions. If you're encoding an object graph that contains references to the same object over and over, or even the same object type over and over, AMF3 will be significantly smaller over the wire, and once you get the data you won't need to do any manually reconciling of objects that are really the same thing.

2: AMF3 can decode typed objects that have methods, derived properties, and a static class definition. That saves time because you have to do less post-parsing processing of the data. It also makes development a lot quicker, especially if you're using an ORM on the server side. You can send objects retrieved from the database straight over to the client, the user can modify the objects with the GUI, and the objects can be sent straight back to the server to save changes. Your server-side AMF encoder/decoder library and ORM library take can care of all the CRUD grunt work, so all you have to do is figure out a decent data model and how to represent it in the client GUI. I use Python and PyAMF on the server side, which even has support for lazy-loading attributes of objects persisted with the SQLAlchemy ORM.

3: You can implement the IExternalizable interface on any AS3 class to fully customize the object encoding/decoding process. If you're brave and need optimization for large data sets, you can even write and read directly to/from the AMF3 byte stream and use your own custom encoding scheme suited for your data type

PyAMF SQLAlchemy Adapter

2009-01-22T08:45:00.001-08:00

PyAMF 0.4 was recently released, and the new code contains SQLAlchemy support by default, plus a bunch of other cool new features and bug fixes.

Check out my example Flex/PyAMF/SQLAlchemy project for ideas of how to get started with PyAMF and SQLAlchemy.

Using PyAMF and SQLAlchemy together

2008-09-21T13:44:00.000-07:00

This past week I recieved the go-ahead to re-code the backend of our LIMS application with Python (it's currently PHP). It's not a minute too late, as I don't think I can stand typing another '->' or '$' without throwing something at the wall. I could go on about why I'm really sick of PHP and why Python rocks, but I'm sure you can read all about that elsewhere.

We'll be using Python's SQLAlchemy ORM, which kicks some serious ass. SQLAlchemy makes it amazingly simple to model complex relationships like inheritance hierarchies and many-to-many relationships that I could only implement with gobs of gnarly custom code in PHP.

Since we're already re-writing the backend we decided that we may as well tweak the Flex client at the same time. We are going to be making the switch from JSON to AMF for server-client communication, which will be more performant and allow us to transfer richer objects back and forth.

Python is pretty awesome for web-development, but the project that aims to support AMF (PyAmf) is still new and immature (but it does encode and decode as advertised and is under active development). In fact, the current stable release of PyAmf will not properly encode objects that are being managed by SQLAlchemy. There is a SQLAlchemy 'adapter' in the development branch of the PyAmf project that can be used, but I found the code to be lacking.

I decided to take matters into my own hands, and I am currently working on my own SQLAlchemy adapter that will hopefully work a little better. The code for my SQLAlchemy adapter can be found on Google code. It is not production ready, but if you want to experiment with it (tested only with SA 0.5), you should be able to drop it in the 'adapters' directory of the PyAmf 4.* branch to use it.

The adapter has 2 main features. It overrides the pyamf.util.get_attrs function so that the pyamf encoder skips any attributes of an object that are used by SQLAlchemy (_sa_instance_state, _sa_manager, etc), and it also identifies and translates SQLAlchemy collection objects, such as InstrumentedList objects.

Flex JSON-RPC implementation

2008-08-25T20:23:00.000-07:00

So, you're checking out Flex and you're thinking to yourself, "Gee this e4x stuff is pretty rad, but it's 2008 and I could really use a good JSON implementation so I can integrate my bitchin' Flex component into my AJAX page." Amazingly, I was thinking the same thing, so I put together a couple of classes to implement a JSON-RPC v1.1 client.

The application code for this post is available on Google code. In addition to the JSON-RPC classes, the application contains a simple gui with controls for testing and logging JSON-RPC communication. To compile the code, you'll also need as3corelib in your compile path, which handles the JSON serialization/deserialization.

The 1st thing you'll need is a JSON-RPC server. If you don't already have one, it shouldn't be too difficult to put together. Follow this tutoria l to create a simple Python JSON-RPC server, or use any 1.1 compliant server.

Next, we need to extend mx.rpc.AbstractOperation. This class will send the method invokation to the server and decode the result. Two important methods are overriden. The first is 'send', which will encode the request according to the JSON-RPC v1.1 spec, create a HTTPRequestMessage, and then call the parent invoke() to send the message to the server. If JSON encoding fails, a FaultEvent must be dispatched. Use the parent method dispatchRpcEvent() to make sure the event is dispatched by both the service and the operation.

The other important method is 'processResult', which decodes the response from the server and checks for any errors defined by the server application. The 'processResult' method is in the mx_internal namespace. If you're not familiar with using mx_internal, you should probably read up on it. The parent class takes care of most of the heavy lifting, but you will need to dispatch a FaultEvent if the JSON decode fails, or if a server application error was defined. Set the _result variable equal to the decoded result, and you're good to go.

Finally, we need to extend mx.rpc.AbstractService. This class is used to instantiate mx.rpc.AbstractOperation instances and invoke remote method calls in our application code. The class's main purpose is to act as a proxy that intercepts method calls, so we will be able to invoke a remote method with the syntax: service.method(arguments) instead of operation.send(method, args). You will need to override the 'getOperation' method to instantiate the correct mx.rpc.AbstractOperation subclass.

Here's how the jsonrpc.JsonRpcService can be used in your application code to communicate with the server:


import mx.rpc.events.*;

import jsonrpc.JsonRpcService;

private function callFoo(fooArgs:Object):void
{
    // Create service object and
    // call remote method 'foo'.
    var jsonService:JsonRpcService = new JsonRpcService();
    jsonService.rootUrl = 'http://localhost';
    jsonService.url = 'test';
    jsonService.addEventListener(ResultEvent.RESULT, gotBar);
    jsonService.foo(fooArgs);
}

private function gotBar(event:ResultEvent):void
{
    event.target.removeEventListener(ResultEvent.RESULT, gotBar);
    var bar:Object = ResultEvent.result;
    // do something with bar
}

Flex360

2008-08-23T09:58:00.001-07:00

We've been using Adobe Flex for the past 6-months to develop SLiM, our RIA client to access our backend GAPSS LIMS server. Flex is the perfect fit for LIMS, because we can do really cool stuff, such as performant drag and drop plate maps right in the browser.

I got my first chance to interact with the larger Flex community this past week at Flex360 in San Jose. I was half-expecting to meet a gaggle of converted Flash developers of the 'creative' type, but I was pleasantly surprised by the number of attendees using Flex for enterprise level applications. Many big companies are developing internal apps with Flex, and most of the community seemed to be very high quality.

Overall, the conference re-enforced the decision to use Flex for our app. I learned a ton of new technical info at the sessions and I'm trying to prioritize what was coolest, so I can go back and re-code sections of SLiM to be better.

Lims Coder

2008-08-23T09:30:00.000-07:00

I'm starting this blog to discuss my experiences with the development of web-based LIMS. I'm drawing most of my content from experience developing applications for a university biotechnology core facility.

The topics will include server-side design, client-side design, lab equipment issues, code examples, and anything generally related to web-application development for life-sciences laboratories.