TechOpinionation - Python

BetweenDict, a Python dict for value ranges

nospam@example.com (Joshua Kugler) — Thu, 31 Dec 2009 14:50:04 +0000

I was doing some programming today (no, really?) and had need of a data structure that would return a value based on the key falling within a given range. Kind of like a dict, but each key in the dict would be two values, between which the querying key would fall. Thus was born BetweenDict. It's short and sweet, and to the point. And works for what I need.

class BetweenDict(dict):
def __init__(self, d = {}):
for k,v in d.items():
self[k] = v

def __getitem__(self, key):
for k, v in self.items():
if k[0] <= key < k[1]:
return v
raise KeyError("Key '%s' is not between any values in the BetweenDict" % key)

def __setitem__(self, key, value):
try:
if len(key) == 2:
if key[0] < key[1]:
dict.__setitem__(self, (key[0], key[1]), value)
else:
raise RuntimeError('First element of a BetweenDict key '
'must be strictly less than the '
'second element')
else:
raise ValueError('Key of a BetweenDict must be an iterable '
'with length two')
except TypeError:
raise TypeError('Key of a BetweenDict must be an iterable '
'with length two')

def __contains__(self, key):
try:
return bool(self[key]) or True
except KeyError:
return False

Basie sounds good...but only SVN?

nospam@example.com (Joshua Kugler) — Wed, 23 Dec 2009 16:26:08 +0000

Started a comment on this post but it got a little long.

So, I follow Planet Python and have seen Greg Wilson's posts on the Basie project.

Basie is a web-based software project forge that integrates revision control, issue tracking, mailing lists, wikis, status dashboards, and other tools that developers need to work effectively in teams.

Basie uses Django and jQuery among other technologies to make a leaner, meaner, multi-project "forge."

I've read up a bit on Basie. Modern framework! jQuery! Multiple projects! Python! Only Subversion! What? Nearly every other* project tracking system out there (even Trac's shoe-horned support) has support for alternate VCS's: Git, Darcs, Bzr, Mercurial, etc. Are you serious? With countless hours invested, and probably 30+ people having worked on the code base, couldn't "multi VCS" have been a requirement from the get-go? Granted, I've never tried it, but as Trac hacks, and Redmine, have proven, it can be done.

I think it's great that we have a project tracking system that is based on modern web framework technologies, but I really think Basie is going to be at a disadvantage out of the gate because it does not support (granted, minority, but growing) VCS's. I hope they are able to rectify this soon: I'd love to see Basie grow into a viable competitor to Redmine. Side note: I love Redmine, but a system in Python allows us to use our intellectual investment to contribute to our VCS; there is very little Ruby knowledge in our shop, so it's harder to give back.

Something else that caught my eye on the Basie site:

Why Build Another Forge?
Because none of the others meet our needs.... Redmine provides many of the features we want, but is still immature;

Can it be clarified when this was written? We currently run Redmine 0.8.6 (with 0.9 around the corner). It has been VERY stable, very able, and everything we need in a PTS. Other than using Python, is there any reason you did not simply choose to invest the time of 30+ people in an existing PTS instead of starting your own from scratch?

Again, this isn't a rant, or to put down the project: I think it's great, and I hope it gets traction in the PTS world. It's just a design decision and your rationale of an "immature Redmine" that got me curious.

*"Every other" meaning "Every other that I am aware of." I'm sure there are plenty of PTS's that I'm not aware of.

Don't use strptime, use a regex

nospam@example.com (Joshua Kugler) — Wed, 23 Sep 2009 11:28:48 +0000

I am using Python to do some data file processing, converting data from a horrendously verbose, repetitive format to a nice, clean, CSV format. The date and time are in two different fields, and the date is in MM/DD/YYYY format, plus, the MM and DD might be one or two characters. That is, January is 1, not 01.

I am converting the timestamp to ISO format, so I was using time.strptime to extract the date/time and time.strftime to generate to proper ISO formatted date, like so:

return time.strftime("%Y-%m-%d %H:%M:%S", time.strptime(ts, "%m/%d/%Y %H:%M:%S"))

On the smallest of my data files, the processing was taking 13 to 15 seconds. I profiled it, and found that in a 13 second run, strptime was taking 8.755 seconds of that, and it was calling _getlang(), _parse_localename(), and the like very time.

So, thought I, regexes are pretty efficient, I wonder if that would reduce the run time any.

ts_re = re.compile('^(\d{1,2})/(\d{1,2})/(\d{4}) (\d{2}:\d{2}:\d{2})')
m = ts_re.match(ts).groups()
return ("%s-%02d-%02d %s") % (m[2], int(m[0]), int(m[1]), m[3])

(The re.compile() call is at the module level, outside the function, so it is only run once.)

My overall run time dropped to about 5 seconds, a little over 1/3 of the time it took previously. My convert_timestamp() function, which previously had consumed nearly 10 seconds, was only taking about 1.3 seconds now.

Sometimes regexes are the answer.

Building Python Eggs with C Extensions on Windows

nospam@example.com (Joshua Kugler) — Fri, 14 Nov 2008 15:13:31 +0000

I recently had the task of building Python Eggs on Windows that had C extensions. I did the usual googling, and found a few HOWTOs, but nothing I could find was very concise and straightforward. So, I present to you a very concise, very straightforward guide.

Setting up the Build Environment

We are assuming all installers are allowed to install to their default locations.

Download and install Python 2.5 and/or Python 2.4 from python.org. You may have to use Python 2.4.4 since that was the latest 2.4 series to have an installer at time of writing. Whichever you want associated with .py files, install last.

Download and install SetupTools for Python (both 2.4 and 2.5) here.

MingW files are here. Download and run the latest version of "Automated MinGW Installer." You only need to install g++, and maybe not even that. When it prompts you for old/current/preview version of the MinGW system, select current.

Download and run the current version of "MSYS Base System."

Open up the MinGW shell, and execute these commands:

cd /
mkdir mingw
mkdir code #Convenience, if you want to mount your code's dir at an "easy" spot
echo "c:/MinGW /mingw
c:/path/to/code /code" > /etc/fstab

Create the file c:\PythonNN\Lib\distutils\distutils.cfg and put the following in it, where NN is replaced with 24 or 25, depending on version::

[build]
compiler = mingw32

Do this for each version installed.

Building the Eggs

MinGW mounts your drives on the root directory, so your C: drive will be at /c, D: drive at /d, and so on. Open up a MinGW shell, and:

cd /code # or /c/path/to/code, if you didn't mount /code
# Depending on which package you're building, you'll either use setup.py
# or extended_setup.py. I'm using setup.py here as a placeholder. Replace as appropriate.
# Python 2.4:
/c/python24/python setup.py build_static # if needed
/c/python24/python setup.py bdist_egg

# Python 2.5:
/c/python25/python setup.py build_static # if needed
/c/python25/python setup.py bdist_egg

You'll now have eggs for your architecture and OS. Enjoy!