Upload Progress with mod_python

Now that Gmail has added a progress bar for attachment uploads, everybody wants to do the same including myself
So, with the same method that everybody uses consisting in an upload form with a hidden <iframe> as target, and then a periodic Ajax request to update the progress, I made a simple implementation using mod_python on the server side… This is a veeery simple example, like everything that you do with Python.
Additionally I’ve added an upload size limit control based in examples that can be found in the mod_python forum site. Use the source Luke!

See LIVE DEMO!

Libraries that I used for this example

The fancy JavaScript progress bar is from Bramus, I just changed the default options, renamed the main script file to progressbar.js and the images just because I like everything in lower case. See the copyright inside the file or visit the site for more information.

Prototype.js: this well known library to avoid cross-browser compatibility issues, is small and practical. I use it in almost all my web projects. Also Bramus requires this library. If you don’t like it for some reason you will need to replace many many many functions like $() which is an improved getElementById(), or the Hash class that I use to track in-progress uploads, JSON encoding, etc. etc. You don’t want to remove it, seriously.

simplejson: A python library to encode/decode JSON objects. I use this one for the messaging between the server and the page. In my opinion JSON is more efficient than XML because it generates more compact and still human-readable outputs. But you can use XML or any other method by modifying the example as you like.

Download example

Start by downloading the source code for this example. I decided to release the code under BSD license, so you can use it for whatever you want.

The source is organized like the following illustration:

Upload Progress source directory structure

/src/img: contains images for JavaScript Bramus progress bar

/src/js: contains JavaScript libraries Prototype.js and Bramus progress bar that I renamed to progressbar.js

/src/temp: directory where the uploaded files will be stored. IMPORTANT: in unix like environments the group and permission of this folder must be configured properly in order to Apache to have write access on it.

/src/_upload_limit.py: mod_python fixup handler that controls the file upload size limit, verifies if the file already exist in the server, stores file size and location information on the session and does the actual file upload after all the verifications.

/src/index.py: The main python publisher handler. Exposed URLs are the main / “index” page that loads the test.html and /get_upload_progress function that returns the current % progress of the files being uploaded.

/src/test.html: HTML page that contains all the JavaScript code used to create the upload forms, post the files, track the progress via Ajax, update the progress bar status, etc… This file is parsed by index.py as a PSP but does not contains any embedded python code currently.

/lib/simplejson: simplejson python library to encode / decode JSON messages.

Configuring Apache

I’m assuming that you already have mod_python installed on your Apache. Take a look to the sample configuration:

Alias / "/Users/fvicente/workspace/uprogress/src/"
<Directory "/Users/fvicente/workspace/uprogress/src/">
    SetHandler python-program
    PythonHandler mod_python.publisher
    PythonDebug On
    PythonFixupHandler _upload_limit
    <Files ~ "\.(gif|jpg|png|ico|js|css)$">
        SetHandler default-handler
    </Files>
    Order allow,deny
    Allow from all
</Directory>

mod_python manual has a lot of information about the configuration options, so I’m not going to describe this in too much depth, just remark that I’m using the very useful mod_python publisher, and more “unusual” option PythonFixupHandler pointing to the _upload_limit.py file.

Interesting parts of the code

_upload_limit.py

The fixuphandler function contained in this file does the checking for the size limit of the file being uploaded by reading the http header Content-Length sent by the browser allowing you to reject an upload even before starting the actual transfer… very clever!. I found _upload_limit.py example in the mod_python forum, and used it as a based adding the specific code for the upload progress handling. _upload_limit.py also stores the file size and name in the current session to be used later by the get_upload_progress function defined in index.py. Note that every time that the Session is used it will be accessed with the option lock=False thus using the session.lock() and session.unlock() methods manually when necessary, if you don’t do this that way all the parallel requests will be held until the upload finishes making impossible to monitor the progress via parallel Ajax request.

index.py

The exposed function get_upload_progress calculates the current upload percent by making a file-stat to see how much of the file has been stored so far. It is requested by the page test.html via Ajax in intervals of one seconds while the file is being uploaded. The expected parameter is a dictionary where the keys are “upload slots” that the JavaScript generates and understand, and the values are the name of the files to get the upload percent.

test.html

This one is 90% JavaScript code. But is not so bad, only 100 lines of code. In a very brief description you will see 4 functions:

addUploadSlot(): creates an HTML code dynamically with an “upload slot” which is a form with the file to upload, the hidden <iframe> that is the form target, the progress bar control, and the “Upload” button. It uses a global counter called slotCnt to give a unique id to each slot. This id will be the “key” in the get_upload_progress dictionary (see index.py description).

sendFile(slot): shows the progress bar for given slot, adds the file into a global Hash() table called “uploading”, starts the periodic Ajax progress monitor (I used JavaScript function setInterval() to do the periodic Ajax) and sends the form to the server. Finally it will disable the controls for that slot and add a new slot dynamically for other uploads.

updateProgressBars(): This is the function called by setInterval() while the files are uploading that does the actual Ajax.Request to /get_upload_progress in order to get the % progress for each file. The parameter sent with the request is a dictionary where the keys are the slots Id’s and the values the file names stored in the hash table.

uploadDone(slot): Will be called by the <iframe> when the upload finishes (successfully or not). Here I remove the file from the hash and stop the interval if no longer necessary.

Final comments

Remember that this is just an example and may contain errors. Use it at your own risk, I’m not responsable of any possible damage or data lost, blah, blah, etc. etc. etc.

The code is poorly commented, but at least is well formatted . I recommend you to read it, it is very short and you may find interesting chunks.

Enjoy it!

If you have a Mac OS X Leopard on an Intel platform, and you are interested in installing mod_python using your default apache setup and the python that comes with the Framework, even more if you want to give PyCAPTCHA a try together with mod_python, then check out this post! I’m covering also the installation of required libraries for PyCAPTCHA under Leopard (FreeType2, libjpeg, PIL)

mod_python

Compile mod_python

If you have Apple’s Xcode installed, compiling mod_python for Leopard to use with the system provided versions of httpd, python, etc. should be as simple as running these commands:


$ mkdir -p ~/mod_python
$ cd ~/mod_python
$ svn co http://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk mod_python-trunk
$ cd mod_python-trunk
$ ./configure --with-apxs=/usr/sbin/apxs
$ make
$ sudo make install
$ cd test/
$ python test.py

(original source: http://www.modpython.org/pipermail/mod_python/2008-March/025012.html)

Configure Apache to load mod_python

Open /private/etc/apache2/httpd.conf with your favorite editor and add the following line:


LoadModule python_module libexec/apache2/mod_python.so


Example of how to configure Apache to load a mod_python application

This is just an example, I recommend reading the mod_python manual for more information.

Open /private/etc/apache2/users/username.conf with your favorite editor and add:

Alias / "/Users/username/htdocs/"
<Directory "/Users/username/htdocs/">
    SetHandler python-program
    PythonHandler mod_python.publisher
    # Disable following line when in production
    PythonDebug On
    <Files ~ "\.(gif|jpg|png|ico|js|css)$">
        SetHandler default-handler
    </Files>
    Order allow,deny
    Allow from all
</Directory>

Restart Apache with the command below:

$ /usr/sbin/apachectl restart

PyCAPTCHA

In order to build and install PyCAPTCHA you’ll need to compile, or make sure that you have these required libraries:

• FreeType2
• libjpeg
• PIL (Python Imaging Library)

Let’s see how to set up these libraries under a Leopard / Intel environment.

FreeType2

This library is required for PyCAPTCHA, as far as I know it is already installed with the SDK, to check the directory where the library is installed use the command:

$ locate libfreetype.dylib

A version of this library should appear under /Developer/SDKs/MacOSX10.5.sdk/usr/X11/

libjpeg

Download jpeg source and uncompress it in any directory, e.g. ~/jpeg-6b then:


$ cd ~/jpeg-6b
$ chmod -R 777 *
$ ./configure


Edit Makefile and change the following line from:

    CFLAGS= -O2 -I$(srcdir)

to:

    CFLAGS= -arch ppc -arch i386 -arch ppc64 -arch x86_64 -O2 -I$(srcdir)

Finally, build and install:

$ make
$ sudo make install-lib

PIL

Download PIL source and uncompress it in any directory, e.g. ~/imaging-1.1.6 then:


$ cd ~/imaging-1.1.6


Edit setup.py and change the following lines from:

    FREETYPE_ROOT = None

to (note that I’m using the FreeType2 directory discovered previously with the locate command):

    FREETYPE_ROOT = libinclude("/Developer/SDKs/MacOSX10.5.sdk/usr/X11/")

and then following line from:

    JPEG_ROOT = None

to:

    JPEG_ROOT = libinclude("/usr/local")

Locate these lines:
from distutils import sysconfig
from distutils.core import Extension, setup
from distutils.command.build_ext import build_ext

And add the following lines (correctly indented from column 0):

OrigExtension = Extension
def Extension(*args, **kwargs):
    extra_args = ['-arch', 'ppc', '-arch', 'ppc64', '-arch', 'i386', '-arch', 'x86_64']
    kwargs['extra_compile_args'] = extra_args + kwargs.get('extra_compile_args', [])
    kwargs['extra_link_args'] = extra_args + kwargs.get('extra_link_args', [])
    return OrigExtension(*args, **kwargs)

Now proceed to compile and install the library:

$ sudo python setup.py install

Install PyCAPTCHA

Now you are ready to build and install PyCAPTCHA. Checkout the source code from this link and run the following command:

$ sudo python setup.py install

That’s it!

image source