Useful for sites that does not shows the upload progress like youtube.

This is my first add-on, so if you find something wrong please let me know.
I’ve submitted it to AMO but since it takes time to get released to the general public, I’ve decided to put it here in our blog if you want to give it a try.

Download it here, enjoy!

Tested in Linux, MacOS X and Windows, this small add-on is 100% JavaScript, no dlls no native XPCOMs. I’ve read in some forums that once upon a time (2004) Firefox already included an upload progress indicator but since then till today is broken. Today other modern browsers like Chrome already provides an upload percent indicator.

To create two bridged virtual serial ports use the following command:

socat -d -d pty,raw,echo=0 pty,raw,echo=0


The output will show you which are the virtual ports (or pseudo terminals) created, e.g.:

2010/02/19 16:16:33 socat[9662] N PTY is /dev/pts/3
2010/02/19 16:16:33 socat[9662] N PTY is /dev/pts/4
2010/02/19 16:16:33 socat[9662] N starting data transfer loop with FDs [3,3] and [5,5]


Note: if you are using Ubuntu and you do not have this command, try:

sudo apt-get install socat


Image source

The easiest way I found is dumping the binaries into text files using hexdump and then comparing them with your favourite program (diff, Meld, etc.). E.g.:

hexdump -C a.bin >a.txt
hexdump -C b.bin >b.txt
diff a.txt b.txt


Image source

from struct import unpack
from datetime import datetime

class FLVReader(dict):
    """
    Reads metadata from FLV files
    """

    # Tag types
    AUDIO = 8
    VIDEO = 9
    META = 18
    UNDEFINED = 0

    def __init__(self, filename):
        """
        Pass the filename of an flv file and it will return a dictionary of meta
        data.
        """
        # Lock on to the file
        self.file = open(filename, 'rb')
        self.signature = self.file.read(3)
        assert self.signature == 'FLV', 'Not an flv file'
        self.version = self.readbyte()
        self.typeFlags = self.readbyte()
        self.dataOffset = self.readint()
        extraDataLen = self.dataOffset - self.file.tell()
        self.extraData = self.file.read(extraDataLen)
        self.readtag()

    def readtag(self):
        unknown = self.readint()
        tagType = self.readbyte()
        dataSize = self.read24bit()
        timeStamp = self.read24bit()
        unknown = self.readint()
        if tagType == self.AUDIO:
            print "Can't handle audio tags yet"
        elif tagType == self.VIDEO:
            print "Can't handle video tags yet"
        elif tagType == self.META:
            endpos = self.file.tell() + dataSize
            self.event = self.readAMFData()
            metaData = self.readAMFData()
            # We got the meta data.
            # Our job is done.
            # We are complete
            self.update(metaData)
        elif tagType == self.UNDEFINED:
            print "Can't handle undefined tags yet"

    def readint(self):
      data = self.file.read(4)
      return unpack('>I', data)[0]

    def readshort(self):
      data = self.file.read(2)
      return unpack('>H', data)[0]

    def readbyte(self):
      data = self.file.read(1)
      return unpack('B', data)[0]

    def read24bit(self):
      b1, b2, b3 = unpack('3B', self.file.read(3))
      return (b1 << 16) + (b2 << 8 ) + b3

    def readAMFData(self, dataType=None):
        if dataType is None:
            dataType = self.readbyte()
        funcs = {
            0: self.readAMFDouble,
            1: self.readAMFBoolean,
            2: self.readAMFString,
            3: self.readAMFObject,
            8: self.readAMFMixedArray,
           10: self.readAMFArray,
           11: self.readAMFDate
        }
        func = funcs[dataType]
        if callable(func):
            return func()

    def readAMFDouble(self):
        return unpack('>d', self.file.read(8))[0]

    def readAMFBoolean(self):
        return self.readbyte() == 1

    def readAMFString(self):
        size = self.readshort()
        return self.file.read(size)

    def readAMFObject(self):
        result = {}
        data = True
        while data:
            size = self.readshort()
            key = self.file.read(size)
            dataType = self.readbyte()
            if not key and dataType == 9:
                break
            data = self.readAMFData(dataType)
            result[key] = data
        return result

    def readAMFMixedArray(self):
        size = self.readint()
        result = {}
        i = 0
        while i < size:
            key = self.readAMFString()
            dataType = self.readbyte()
            if not key and dataType == 9:
                break
            result[key] = self.readAMFData(dataType)
            i += 1
        return result

    def readAMFArray(self):
        size = self.readint()
        result = []
        i = 0
        while i < size:
            result.append(self.readAMFData())
            i += 1
        return result

    def readAMFDate(self):
        date = self.readAMFDouble()/1000
        localoffset = self.readshort()
        return datetime.fromtimestamp(date)

if __name__ == '__main__':
    import sys
    from pprint import pprint
    if len(sys.argv) == 1:
        print 'Usage: %s filename [filename]...' % sys.argv[0]
        print 'Where filename is a .flv file'
        print 'eg. %s myfile.flv' % sys.argv[0]
    for fn in sys.argv[1:]:
        x = FLVReader(fn)
        pprint(x)

Download this code snippet from here

<html>
<body>
<div style="color: blue;">
	<table>
	<tbody>
		<tr>
			<td>BLUE TEXT</td>
		</tr>
	</tbody>
	</table>
<div>
</body>
</html>

In the below example I would expect to see the “BLUE TEXT” in blue, but is not (at least in FireFox). So what the hell is wrong with my code? probably many things but because I’m not interested in following/reading the W3C recommendations for my uncle’s home page, here is one possible solution: Add the transitional DOCTYPE to your document, or the DOCTYPE that adequate better to your page (read more here). E.g.:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

So, now you know. Never forget the DOCTYPE it is very important… Why? read this… It has to do with poor standards, browsers and all that crap.

Simplified DES

This encryption algorithm is not secure at all, in fact it was made for educational purposes. It uses a 10-bit key that can be quickly broken by a brute force attack. But, who knows, it may be good for someone interested in some quick encryption where security is not that important, or for embedded devices were resources are limited, or just for lazy students.

So, here is my version of the S-DES algorithm in plain C. Differently from the other version in the web my implementation is more byte-oriented while the other (probably more didactic) is converting the input to a string with ’0′s and ’1′s and working over it. Also the other implementation uses some C++ functions while mine is written in plain C.

Download it and take a look to the code (it is less than 300 lines) — released under BSD license.

To build it with gcc type:

gcc sdes.c -lm -o sdes

Note that I’m using math library only because of the pow() function that is used only in debug mode.

And here is a copy of a document explaining the algorithm.

Image source

Tool Window

This JavaScript class wraps a given HTML element into a movable and resizable floating tool window.  Required libraries: scriptaculous and prototype.js

How to use the code:

Download full source (including scriptaculous and prototype). The pieces of code developed by me are released under BSD license, please verify third party libraries licenses before using them.

See the DEMO PAGE!

enjoy it!

Upload Progress with mod_python

Now that Gmail has added a progress bar for attachment uploads, everybody wants to do the same including myself
So, with the same method that everybody uses consisting in an upload form with a hidden <iframe> as target, and then a periodic Ajax request to update the progress, I made a simple implementation using mod_python on the server side… This is a veeery simple example, like everything that you do with Python.
Additionally I’ve added an upload size limit control based in examples that can be found in the mod_python forum site. Use the source Luke!

See LIVE DEMO!

Libraries that I used for this example

The fancy JavaScript progress bar is from Bramus, I just changed the default options, renamed the main script file to progressbar.js and the images just because I like everything in lower case. See the copyright inside the file or visit the site for more information.

Prototype.js: this well known library to avoid cross-browser compatibility issues, is small and practical. I use it in almost all my web projects. Also Bramus requires this library. If you don’t like it for some reason you will need to replace many many many functions like $() which is an improved getElementById(), or the Hash class that I use to track in-progress uploads, JSON encoding, etc. etc. You don’t want to remove it, seriously.

simplejson: A python library to encode/decode JSON objects. I use this one for the messaging between the server and the page. In my opinion JSON is more efficient than XML because it generates more compact and still human-readable outputs. But you can use XML or any other method by modifying the example as you like.

Download example

Start by downloading the source code for this example. I decided to release the code under BSD license, so you can use it for whatever you want.

The source is organized like the following illustration:

Upload Progress source directory structure

/src/img: contains images for JavaScript Bramus progress bar

/src/js: contains JavaScript libraries Prototype.js and Bramus progress bar that I renamed to progressbar.js

/src/temp: directory where the uploaded files will be stored. IMPORTANT: in unix like environments the group and permission of this folder must be configured properly in order to Apache to have write access on it.

/src/_upload_limit.py: mod_python fixup handler that controls the file upload size limit, verifies if the file already exist in the server, stores file size and location information on the session and does the actual file upload after all the verifications.

/src/index.py: The main python publisher handler. Exposed URLs are the main / “index” page that loads the test.html and /get_upload_progress function that returns the current % progress of the files being uploaded.

/src/test.html: HTML page that contains all the JavaScript code used to create the upload forms, post the files, track the progress via Ajax, update the progress bar status, etc… This file is parsed by index.py as a PSP but does not contains any embedded python code currently.

/lib/simplejson: simplejson python library to encode / decode JSON messages.

Configuring Apache

I’m assuming that you already have mod_python installed on your Apache. Take a look to the sample configuration:

Alias / "/Users/fvicente/workspace/uprogress/src/"
<Directory "/Users/fvicente/workspace/uprogress/src/">
    SetHandler python-program
    PythonHandler mod_python.publisher
    PythonDebug On
    PythonFixupHandler _upload_limit
    <Files ~ "\.(gif|jpg|png|ico|js|css)$">
        SetHandler default-handler
    </Files>
    Order allow,deny
    Allow from all
</Directory>

mod_python manual has a lot of information about the configuration options, so I’m not going to describe this in too much depth, just remark that I’m using the very useful mod_python publisher, and more “unusual” option PythonFixupHandler pointing to the _upload_limit.py file.

Interesting parts of the code

_upload_limit.py

The fixuphandler function contained in this file does the checking for the size limit of the file being uploaded by reading the http header Content-Length sent by the browser allowing you to reject an upload even before starting the actual transfer… very clever!. I found _upload_limit.py example in the mod_python forum, and used it as a based adding the specific code for the upload progress handling. _upload_limit.py also stores the file size and name in the current session to be used later by the get_upload_progress function defined in index.py. Note that every time that the Session is used it will be accessed with the option lock=False thus using the session.lock() and session.unlock() methods manually when necessary, if you don’t do this that way all the parallel requests will be held until the upload finishes making impossible to monitor the progress via parallel Ajax request.

index.py

The exposed function get_upload_progress calculates the current upload percent by making a file-stat to see how much of the file has been stored so far. It is requested by the page test.html via Ajax in intervals of one seconds while the file is being uploaded. The expected parameter is a dictionary where the keys are “upload slots” that the JavaScript generates and understand, and the values are the name of the files to get the upload percent.

test.html

This one is 90% JavaScript code. But is not so bad, only 100 lines of code. In a very brief description you will see 4 functions:

addUploadSlot(): creates an HTML code dynamically with an “upload slot” which is a form with the file to upload, the hidden <iframe> that is the form target, the progress bar control, and the “Upload” button. It uses a global counter called slotCnt to give a unique id to each slot. This id will be the “key” in the get_upload_progress dictionary (see index.py description).

sendFile(slot): shows the progress bar for given slot, adds the file into a global Hash() table called “uploading”, starts the periodic Ajax progress monitor (I used JavaScript function setInterval() to do the periodic Ajax) and sends the form to the server. Finally it will disable the controls for that slot and add a new slot dynamically for other uploads.

updateProgressBars(): This is the function called by setInterval() while the files are uploading that does the actual Ajax.Request to /get_upload_progress in order to get the % progress for each file. The parameter sent with the request is a dictionary where the keys are the slots Id’s and the values the file names stored in the hash table.

uploadDone(slot): Will be called by the <iframe> when the upload finishes (successfully or not). Here I remove the file from the hash and stop the interval if no longer necessary.

Final comments

Remember that this is just an example and may contain errors. Use it at your own risk, I’m not responsable of any possible damage or data lost, blah, blah, etc. etc. etc.

The code is poorly commented, but at least is well formatted . I recommend you to read it, it is very short and you may find interesting chunks.

Enjoy it!

If you have a Mac OS X Leopard on an Intel platform, and you are interested in installing mod_python using your default apache setup and the python that comes with the Framework, even more if you want to give PyCAPTCHA a try together with mod_python, then check out this post! I’m covering also the installation of required libraries for PyCAPTCHA under Leopard (FreeType2, libjpeg, PIL)

mod_python

Compile mod_python

If you have Apple’s Xcode installed, compiling mod_python for Leopard to use with the system provided versions of httpd, python, etc. should be as simple as running these commands:


$ mkdir -p ~/mod_python
$ cd ~/mod_python
$ svn co http://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk mod_python-trunk
$ cd mod_python-trunk
$ ./configure --with-apxs=/usr/sbin/apxs
$ make
$ sudo make install
$ cd test/
$ python test.py

(original source: http://www.modpython.org/pipermail/mod_python/2008-March/025012.html)

Configure Apache to load mod_python

Open /private/etc/apache2/httpd.conf with your favorite editor and add the following line:


LoadModule python_module libexec/apache2/mod_python.so


Example of how to configure Apache to load a mod_python application

This is just an example, I recommend reading the mod_python manual for more information.

Open /private/etc/apache2/users/username.conf with your favorite editor and add:

Alias / "/Users/username/htdocs/"
<Directory "/Users/username/htdocs/">
    SetHandler python-program
    PythonHandler mod_python.publisher
    # Disable following line when in production
    PythonDebug On
    <Files ~ "\.(gif|jpg|png|ico|js|css)$">
        SetHandler default-handler
    </Files>
    Order allow,deny
    Allow from all
</Directory>

Restart Apache with the command below:

$ /usr/sbin/apachectl restart

PyCAPTCHA

In order to build and install PyCAPTCHA you’ll need to compile, or make sure that you have these required libraries:

• FreeType2
• libjpeg
• PIL (Python Imaging Library)

Let’s see how to set up these libraries under a Leopard / Intel environment.

FreeType2

This library is required for PyCAPTCHA, as far as I know it is already installed with the SDK, to check the directory where the library is installed use the command:

$ locate libfreetype.dylib

A version of this library should appear under /Developer/SDKs/MacOSX10.5.sdk/usr/X11/

libjpeg

Download jpeg source and uncompress it in any directory, e.g. ~/jpeg-6b then:


$ cd ~/jpeg-6b
$ chmod -R 777 *
$ ./configure


Edit Makefile and change the following line from:

    CFLAGS= -O2 -I$(srcdir)

to:

    CFLAGS= -arch ppc -arch i386 -arch ppc64 -arch x86_64 -O2 -I$(srcdir)

Finally, build and install:

$ make
$ sudo make install-lib

PIL

Download PIL source and uncompress it in any directory, e.g. ~/imaging-1.1.6 then:


$ cd ~/imaging-1.1.6


Edit setup.py and change the following lines from:

    FREETYPE_ROOT = None

to (note that I’m using the FreeType2 directory discovered previously with the locate command):

    FREETYPE_ROOT = libinclude("/Developer/SDKs/MacOSX10.5.sdk/usr/X11/")

and then following line from:

    JPEG_ROOT = None

to:

    JPEG_ROOT = libinclude("/usr/local")

Locate these lines:
from distutils import sysconfig
from distutils.core import Extension, setup
from distutils.command.build_ext import build_ext

And add the following lines (correctly indented from column 0):

OrigExtension = Extension
def Extension(*args, **kwargs):
    extra_args = ['-arch', 'ppc', '-arch', 'ppc64', '-arch', 'i386', '-arch', 'x86_64']
    kwargs['extra_compile_args'] = extra_args + kwargs.get('extra_compile_args', [])
    kwargs['extra_link_args'] = extra_args + kwargs.get('extra_link_args', [])
    return OrigExtension(*args, **kwargs)

Now proceed to compile and install the library:

$ sudo python setup.py install

Install PyCAPTCHA

Now you are ready to build and install PyCAPTCHA. Checkout the source code from this link and run the following command:

$ sudo python setup.py install

That’s it!

image source

Today I’m introducing another utility script written in python that uses wxPython to edit your scanned images (JPEG, BMP, TIFF, etc.) obtain sub-images, rotate and save them.

This first version is very basic but effective, there is no necessity of learning complex commands or image handling programs, just:

• click in the upper-left corner
• click in the lower-right corner
• rotate if you wish
• close and save

The unfortunate name of this script is cutnrot and it is released under BSD license, download, modify and use it at your own risk, as usual. Here is the Google Code page for this tiny project.