For my main IDE on my windows box I use Eclipse with the PyDev module installed:

http://marketplace.eclipse.org/content/pydev-python-ide-eclipse

Since most of my development is on Linux, I have the Remote System Explorer plugin installed which lets me create and edit files on our linux server over sftp like they were local to my workstation:

http://help.eclipse.org/galileo/index.jsp?topic=/org.eclipse.rse.doc.user/gettingstarted/g1installing.html

Last but not least, I installed Eclipse Color Theme which is an easy way to install themes for eclipse:

http://marketplace.eclipse.org/content/eclipse-color-theme

I even when an made a theme (kind of Borland blue, not shown above) you can download here:

http://www.eclipsecolorthemes.org/?view=theme&id=4514

Being a good Pythonista, I followed the cardinal rules of:
Get it right.
Test it’s right.
Profile if slow.
Optimize.
Repeat from 2.
http://wiki.python.org/moin/PythonSpeed/PerformanceTips

The problem was, after a while I sort of hit a wall. Nothing I did could make this code appreciably faster (of course this was with my limited knowlegde. I’m sure that more experienced Python programmers could optimize this code a lot better then I can, rewrite the regex bottleneck in C, etc) but I was at the end of my rope.

On thing led to another and I remembered reading about Pypy. Pypy is implementation of Python using a JIT (Just In Time Compiler) and other things that have lost there meaning for me since I did systems programming in college. “What the heck”, I thought, “I’ll give it a try”. Pypy is supposed to be highly compatible with CPython (the regular python implementation) and my code didn’t use any exotic libraries.

So I dumped the tarball on my linux machine, unzipped and ran my unmodified code against it, and DAMN was it fast.

CPython run:

sean@linux1:~/code/python/hourly_alerts$ python alerter.py
Loaded regexes
Processing known_bad.log.gz
Searched 817051 lines in 119.398156166 seconds using filter

Pypy Run

sean@linux1:~/code/python/hourly_alerts$ /home/sean/bin/pypy-1.7/bin/pypy alerter.py
Loaded regexes
Processing known_bad.log.gz
Searched 817051 lines in 51.1275110245 seconds using filter

More then twice as fast! Now I know that it was a totally unscientific test and all, but its great to see such an improvement right away.

I my also try Cython, but that looks like it doesn’t have quite the drop-in functionality of Pypy.

Here are the link to Pypy : http://pypy.org/

import re
import gzip

searchName = "Example Regex"
compiledRegex = re.compile(r'example' re.I)

dataFile = gzip.open(fileName, "rb")
#read in the file to memory
fullData = dataFile.readlines()
regexName = searchName
regexSearch = compiledRegex .search
#Filter for just the lines matching the regex
hitList = filter(regexSearch, fullData)
#hitList now is just a list of our matches
for item in hitList:
    print "MATCH:%s" % regexName
    print "ENTRY:%s" % item

Obviously this code needs more to be at all usable, but this is the general idea and a pretty cool use of the filter function.

If you know of a better way, let me know in the comments.

References:
Python Regular Expressions (re) : http://docs.python.org/library/re.html
Filter Function : http://docs.python.org/library/functions.html#filter

[ED: Before we get into it though, regular expressions do have their place and are very powerful. None of this should be taken to mean "Don't use regular expressions EVER!" Use them where they make sense, but as always, try to write idiomatic python (http://jaynes.colorado.edu/PythonIdioms.html) when possible.]

Let’s look at a couple examples:

Here we have 2 strings:

myString = "This is a test string.  We'd like to match it."

myString2 = "We don't want to match this string."

stringList = [myString, myString2] #Just so we can iterate over them

If we wanted to find the string that started with “This” we could do this:

import re

for line in stringList:

      if (re.search("^This", line)):

            print "SUCCESS with line :" + line

      else:

            print "FAILURE with line :" + line

This gives:

SUCCESS with line :This is a test string.  We'd like to match it.

FAILURE with line :We don't want to match this string.

We could do the same thing using the standard methods on a string in a (at least to me) cleaner and more pythonic way:

for line in stringList:

      if line.startswith("This"):

            print "SUCCESS with line :" + line

      else:

            print "FAILURE with line :" + line

SUCCESS with line :This is a test string.  We'd like to match it.

FAILURE with line :We don't want to match this string.

Using regex to see if a certain substring is in the line:

import re

for line in stringList:

      if (re.search("test", line)):

            print "SUCCESS with line :" + line

      else:

            print "FAILURE with line :" + line

This gives:

SUCCESS with line :This is a test string.  We'd like to match it.

FAILURE with line :We don't want to match this string.

Now using the built in operations:

for line in stringList:

      if "This" in line:

            print "SUCCESS with line :" + line

      else:

            print "FAILURE with line :" + line

SUCCESS with line :This is a test string.  We'd like to match it.

FAILURE with line :We don't want to match this string.

Keeping it simple always seems to me to be the best policy.  Of course this doesn’t meant that you should write hugely complex code to avoid using regexes.  It just means that sometimes they might not be the right choice.  Use the right tool for the job.

Understanding Python Decorators - I’d never head of decorators until I saw this link and now can think of a bunch of uses for them

In Defense of the Internet Craftsman - “In Gutenberg’s era, the printmaker, not the machine, determined the subject matter of his work. No printing press could impose terms of service that dictated the language or content that could be printed. Instead, the craftsman was in full control of his speech. Yet these restrictions are being hardwired into modern technologies.”

Why Every Programmer Should Have A Tiddlywiki - I’ve used tiddlywiki’s before, but Eric shows a great way to start using a tiddlywiki to keep track of your projects and todo’s.

An Introduction to Asynchronous Programming and Twisted - This is not short, in fact it is probably book length and I have not read more then the beginning, but Twisted is something that I’ve always been really keen to learn and this looks like it might be a good introduction to it.

sys.argv Method
Using sys.argv is probably the first way everyone processes options and it is a great quick and dirty way to do it with not a lot of coding.

Take a simple example:

$ myprog.py somehost
Connecting to somehost...

In this trivial example, we pass a host to the program which (nominally) connects to the host (all it actually does is print connecting, but that is besides the point)

The code would be :

#!/usr/local/bin/python

import sys

host = sys.argv[1]
print "Connecting to" + host + "..."

What is happening here is that python takes all the parameters that we give them on a command line and shoves them into sys.argv. One thing to note, sys.argv[0] is always the name of the script.  So if we know the positions of the arguments or are only expecting a very limited argument set, sys.argv is a quick and easy way to access the command line parameters.  The problem with sys.argv is that it doesn’t handle more complex command line options well (or at least easily).

http://docs.python.org/library/sys.html#sys.argv

Using getopt
getopt should look pretty familiar to anyone that has worked with C before and allows us to do all the command line parsing that we expect in most CLI programs (ie pass values, handle long and short names, etc).

for example

$ myprog2.py --message "Message to print" -n 5
Message to print
Message to print
Message to print
Message to print
Message to print

This program takes 2 parameters, a message to print and  the number or times to print.  We can easily set to to recognize the options for the message to be either -m or –message and the number of times to print it as -n or –number with getopt as demonstrated below:

#!/usr/local/bin/python

import sys
import getopt

try:
    optsList, argsList = getopt.getopt(sys.argv[1:], "m:n:", ["message=", "number="])
except getopt.GetoptError, err:
    #Error handling for unknown or incorrect number of options
    print str(err)
    sys.exit(2)
for opt, arg in optsList:
    if opt in ('-m', '--message'):
        messageText = arg
    elif opt in ('-n', '--number'):
        numberToPrint = int(arg)
    else:
        print "Unhandled option"
        sys.exit(2)

i=0
while i in range(numberToPrint):
    print messageText
    i=i+1

For a more indepth looks, here is the documentation: http://docs.python.org/library/getopt.html

As you can see this is pretty flexible as well as powerful, but is not really that pythonic.  Two better options are optparse and argparse.

Using optparse
Optparse was definitely the way to go until Python 2.7 came out and still is if you are developing on systems with older (<2.7) versions of Python.  Optparse does all the neat things we’ve seen with getopt in a more pythonic (and to me, more intuitive (but that might be just that I’ve been drinking the Python Kool-Aid for so long)) way.

Since this can get a little confusing with trivial examples, I’m going to include a snippet of some code I’ve been working on:

import optparse

usage = "usage: %prog [options]"

#Handle the commands line options
parser = optparse.OptionParser(usage=usage)
parser.add_option("-d", "--date", dest="date",
             help="Process a specific date in the format %m%d", default="today")
parser.add_option("-f", "--force", action="store_true", dest="force",
             help="Force reindex of files - overwite existing files for date", default=False)
parser.add_option("-s", "--server", dest="server",
             help="Reindex all files for a specific server (for all available dates")

(options, args) = parser.parse_args()
print "Passed Date:%s" % options.date
print "Passed Force:%s" % options.force
print "Passed Server:%s" % options.server

Here we can see how we create an OptionParser object named parser and add our options to it.  We can set default values and even tag them with usage (“help”) messages. We can then access the discrete values directly with the label we’ve assigned (ie option.date).

One of the coolest things is that optparse autogenerates a usage() function.  Notice that we don’t really define a function called usage, we just pass it a usage string (“In a simple case like this, I don’t think that this is even necessary) and then tag our options with help messages.

So if we save the above snippet as myprog3.py and run it with –help (or -h) we get the following message:

$ python myprog3.py --help
Usage: myprog3.py [options]

Options:
-h, --help                 show this help message and exit
-d DATE, --date=DATE       Process a specific date in the format %m%d
-f, --force                Force reindex of files - overwite existing files for
                           date
-s SERVER, --server=SERVER Reindex all files for a specific server (for all
                           available dates

Pretty cool. But remember, this is a deprecated module and will not be updated going forward.  If you are using Python 2.7 or greater you should use argparse.

Obviously there is way more that you can do with it (ie positional options, requited arguments, etc) that you can find here:
http://docs.python.org/library/optparse.html  

Using argparse
Argparse (to me at least, I’ve not used it as much as optparse) looks to be a more streamlined version optparse, but the general gist is the same.  Usage function is handled by the library in  a similar way.  Rewriting the code that used optparse above with argparse would yeild (as myprog4.py):

import argparse

parser = argparse.ArgumentParser(description="This program process arguments")
parser.add_argument("-d", "--date", dest="date",
               help="Process a specific date in the format %%m%%d", default="today")
parser.add_argument("-f", "--force", action="store_true", dest="force",
               help="Force reindex of files - overwite existing files for date", default=False)
parser.add_argument("-s", "--server", dest="server",
               help="Reindex all files for a specific server (for all available dates")

args = parser.parse_args()
print "Passed Date:%s" % args.date
print "Passed Force:%s" % args.force
print "Passed Server:%s" % args.server

And running it we get:

H:\Code\python\examples>python myprog4.py --help
usage: myprog4.py [-h] [-d DATE] [-f] [-s SERVER]

This program process arguments

optional arguments:
-h, --help                 show this help message and exit
-d DATE, --date DATE       Process a specific date in the format %m%d
-f, --force                Force reindex of files - overwite existing files for
                           date
-s SERVER, --server SERVER Reindex all files for a specific server (for all
                           available dates)

As you can see, usage is handled very similarly, the options are handled in the same way, but down in the nitty gritty details there are a lot more options then existing in optparse.  Spend some time on in the python docs to get a better understanding of all that it can do.http://docs.python.org/library/argparse.html

Hopefully this gives you a couple more tools to use while processing command line option in Python.

So dutifully I cataloged my books for a couple years as I read them and was pretty content.  That is until earlier this week when I tried to add the last book I read (Day by Day Armageddon : Beyond Exile (which, while I love zombie apocalypse stories, was really rather meh)) I ran across this fun message:

Free memberships are limited to 200 books. Upgrade to a paid membership and catalog as many books as you like. A yearly membership costs only $10.

Now, I’m sure that this was listed somewhere in the Terms of Service for Librarything, but I was more then a little taken by surprise.

I have no problem paying for web services that I use and $10 dollars is really not a lot of money, but for basically keeping a spreadsheet of what I’ve read it seemed a bit excessive.  I really didn’t use any of the “social” functions of Librarything and didn’t especially like the records for individual books.  Also the 200 limit seemed pretty low since I only added 3 years worth of books that I was actually reading and hadn’t even loaded up books that I meant to read, or owned but haven’t gotten around to reading or whatever.

So I started going back over my choices.  I knew about Goodreads, but I really didn’t want to lose all the 200 books that I had cataloged and tagged in Librarything.  Luckily there exists an export function in Librarything and an import from a file in Goodreads.  I found that exporting and importing as a tab delimited file (xls) seemed to work better then exporting and importing as a comma separated value file (csv). I have noticed that some of the tags apprear to be messed up, but fixing that should be no big deal.

I also like the fact that Goodreads provides an Android App (Android Market Link)

So long story short, I’ve ported all my books over to Goodreads and that is where you can find me now.

My Goodreads profile http://www.goodreads.com/user/show/1401430-sean-lavelle

$ echo "This is the subject of my email" | mail - s "My Email Title" sean@example.com

(Of course you could do this though the python smtp library, but what we are interested in is the behavior of piping the echo command into the mail command).

Using the subprocess module,  we would would write it like this

import subprocess

emailAddress = 'sean@example.com'
title = 'My Email Title'
subject = 'This is the subject of my email'

p1 = subprocess.Popen(['echo', subject], stdout=subprocess.PIPE) #Set up the echo command and direct the output to a pipe
p2 = subprocess.Popen(['mail', '-s' ,title, emailAddress], stdin=p1.stdout #send p1's output to p2
p1.stdout.close() #make sure we close the output so p2 doesn't hang waiting for more input
output = p2.communicate()[0] #run our commands

Although it might seem a bit difficult at first, now that I’ve used this method for a while it make sense to me.  I’ve used it to rewrite shell scripts with six commands piping into each other and the method is the same as above.  And once you have these shell commands being run within python, you can do error handling, or more advanced processing which is difficult (at least for me) within shell.

As always the Python Documentation Library is your best friend for a more detailed explanation and other examples: http://docs.python.org/library/subprocess.html

$ python find_blocking_processes.py
Traceback (most recent call last):
File "find_blocking_processes.py", line 77, in <module>
find_blocking_processes(brokerConn)
File "find_blocking_processes.py", line 67, in find_blocking_processes
) x""").fetchall()
pyodbc.ProgrammingError: ('42000', '[42000] [FreeTDS][SQL Server]Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier. (4004) (SQLExecDirectW)')

After a bit of digging, it appears that you need to tell the ODBC driver which protocol to talk to a server in. Rectifying this was pretty straight forward:

In the file /etc/freetds.conf I added a line to the serverconfig  stating to use the version 8.0 protocol:

# A typical Microsoft SQL Server 2008 configuration
[DEVDATABASE]
host = 10.10.10.100
port = 1433
tds version = 8.0

Then in the file /etc/odbc.ini I added a line for the version as well:

[DEVDATABASE]
Driver          = /usr/lib64/libtdsodbc.so.0
Server          = 10.10.10.100
Port            = 1433
Trace           = Yes
TraceFile       = /tmp/freetdssql-foobar.log
tds_version     = 8.0

* Note the ‘_’ in the tds_version  variable in the odbc.ini

After this, the database call ran like a dream.

These were the links that pointed me in the right direction to solve this issue and contain a little more information on the causes of this:

• FreeTDS Error while trying to retrieve information from an ntext field – http://www.ubuntitis.com/?p=64
• PyODBC, UnixODBC, FreeTDS – config – http://kipb7.wordpress.com/2008/06/12/pyodbc-unixodbc-freetds-config/