I was doing some programming today (no, really?) and had need of a data structure that would return a value based on the key falling within a given range. Kind of like a dict, but each key in the dict would be two values, between which the querying key would fall. Thus was born BetweenDict. It's short and sweet, and to the point. And works for what I need.

I am using Python to do some data file processing, converting data from a horrendously verbose, repetitive format to a nice, clean, CSV format. The date and time are in two different fields, and the date is in MM/DD/YYYY format, plus, the MM and DD might be one or two characters. That is, January is 1, not 01.

I am converting the timestamp to ISO format, so I was using time.strptime to extract the date/time and time.strftime to generate to proper ISO formatted date, like so:


On the smallest of my data files, the processing was taking 13 to 15 seconds. I profiled it, and found that in a 13 second run, strptime was taking 8.755 seconds of that, and it was calling _getlang(), _parse_localename(), and the like very time.

So, thought I, regexes are pretty efficient, I wonder if that would reduce the run time any.


(The re.compile() call is at the module level, outside the function, so it is only run once.)

My overall run time dropped to about 5 seconds, a little over 1/3 of the time it took previously. My convert_timestamp() function, which previously had consumed nearly 10 seconds, was only taking about 1.3 seconds now.

Sometimes regexes are the answer.

Had the need recently to generate a bunch of thumbnails for some PDFs we had. ImageMagick to the rescue.



The [0] tells it to only do the first page, and the quality setting of 50 created thumbnails that were only 2-4K, but still quite functional for the purpose.

If not already pulled in as a dependency, Ghostscript (/usr/bin/gs) will need to be installed too.

Dell sent me another one of their catalogs recently, which included this entry. You can get Windows Vista, or a "bonus" of an Windows XP Professional downgrade. I have nothing left to add.

Bob (from marketing): I have this great new idea for our product!
Joe (from engineering): Lay it on me!
Bob: Let's make our product easier to use! People will love it!
Joe: OK, we'll need three months to develop and test that feature.
Bob: Great! We'll start advertising the new feature immediately.
Joe: Uh.....

What happened next? The photo says it all.

I recently installed the Jaunty/KDE3.5 remix. I had some interesting issues in that I was trying to install on an existing LUKS/LVM setup. Whole-disk encryption on LVM rocks, but if the installer doesn't support it, you'll have to fiddle to get things working right.

It was up and working, but of course I was still fiddling around. At some point I ran mkswap on my swap partition. After this, I noticed I could no longer suspend to disk (hibernate), but I did not yet corelate the two events, as I had changed /etc/fstab to use the new UUID of the swap partition.

I then noticed that upon boot I now had two swap partitions active: my swap partition, and another one mounted on /dev/ramzswap0. I turned off that swap device, and behold, I could suspend to disk! BUT, when I resumed, it went through normal boot...but not resume.

So, I began looking through the files under /etc/initramfs-tools. I came across one in conf.d called "resume" which contained this line:

RESUME=UUID=some-uuid-that-wasn't-correct-anymore

AHA! I replaced the UUID with the correct one, ran

and now I can suspend to disk and resume.

Today I was trying to figure out how to ssh into one system, and then automatically ssh into another system. The logical way:


wasn't working. The logs on both inside and outside hosts would show a connection, but I would get no prompt. I tried typing a command, hit enter, and saw the output of the command! OK, that's odd.

So, I hit #openssh and posted my problem. User dmlloyd and I started dialoging and figured out that a TTY wasn't being allocated. OK, why not. AH! openssh's -t parameter to the rescue. So this does work:


NOTE: Yes, I know I could just set up port forwarding or DNAT firewall rules, but the inside hosts don't need any access from the outside at all, so the chained SSH method was much better.

So today I discovered that my first ever contribution to an Open Source project was accepted. Before I was let go from WordStream, my boss was having me work on a feature addition to Buildbot.

I was allowed to contribute that addition, under my own name, and today that change was merged in. It comprised these commits: 1, 2, 3

Needless to say, I'm pretty jazzed!

So, WordStream decided my skillset didn't fit what they wanted, and I am once again in the ranks of the unemployed. If you know of any programming or system administration positions you could point me to, that would be greatly appreciated!

My resume is linked over on the right side of this page. Feel free to pass it around.

I had this conversation this morning via a random AOL IM sent to me:

(10:09:04) AtypicalCoho: If you respond with an away message containing a phone number, I will laugh... then rickroll you by phone.
(10:10:08) Me: I am not available. If you need to contact me, please call 907-451-5100
(10:11:42) AtypicalCoho: who the h*** is this?
(10:12:08) Me: You tell me...you IM'ed me.

I can only assume he tried the number. It's for the Fairbanks office of the Alaska State Troopers. I don't think I'll be hearing from him again.

[The meat of this post was actually a reply to a UAF (Univeristy of Alaska Fairbanks) LUG (Linux User's Group) mailing-list thread a few months back. It's been reworked for this blog post. In addition, this post was moved from my person blog, jjncj.com where it was originally posted March 25 of last year.]

I was reminded of this topic by a post on CodeJustin that had a poll about whether or not you went to college for programming. For me the answer is yes and no. I went to college to learn many skills and much theory about computer systems, but 95%+ of what I use in my day-to-day job I've taught myself. However, that learned theory has formed a solid foundation for my learned skills.

This has been hashed out many, many times, but I'll jump into the fray again. If you want a foundation in computer science, and the ability to learn: get a theory-based CS degree. If you just want the skills you need for a job, take a class for it, read it on the web, or take a university course that is more geared toward certifications (for example: see the UAF Info Tech program).

But as to practical skills, the UAF CS program does offer many. If you want to program, take Operating Systems (321) and Assembly (301). Architecture (471) is a good one too. Those classes will make understanding programming so much easier, because you understand what the system is doing as your program executes. If you want to be a network guru, take CS 442 first (or the grad level 642). Will it teach you to set up a windows AD network and configure roaming profiles? No, but it will give you a base-line knowledge level that will make understanding how that network works, and troubleshooting that network, so much easier.

I've taken the full complement of CS classes to earn my BS, and almost my MS. There are classes I may never use again. But I'm glad I had CS 201/202 (I've used C/C++ since in job and school); 301 (Assembly language; understanding of a computer's operation); 331/631 (Compiler and language theory; better understanding of how compilers work, and the complexities thereof); 401 (senior project, better understanding of process and project management); Computer Architecture (gives me a good idea of how all the hardware fits together so I understand the system better when working on it); 311 (Algorithms and data structures; will I being doing heavy algorithm design, maybe, but I also know I can evaluate possible algorithms for efficiency and the load they will put on the system); 321 (Operating systems; especially helps when running on "sane" systems such as Linux or Mac OS X); 447 (software engineering: gave me so much insight into the proper ways to go about designing programs. Something I'm about to put into heavy use at my current job); and there are others.

Among the things I've learned on my own or via "on the job training": Python, Perl; SQL; Visual Basic; Linux administration; Apache administration; Postfix administration; a little Sendmail too; general system administration; network setup, with some routing (My friend Todd Medbury could still make my head hurt); hardware/software troubleshooting/assembly; Qt programming; CGI/web programming; Bind (DNS Server); HTML; VMWare Server; Bacula (backup server); as well as other skills I've probably failed to mention. I'm not bragging, I'm simply pointing out that taking classes in all those would have been prohibitively expensive, taken a LOT of time, and in the end I would have learned less than my on-the-job training taught me.

For another example of what all that theory got me, see this paper. Careful design and development led to a successful election with software that was designed, coded, and debugged (very little debugging, due to careful coding) in 80 hours.

My point is, with a CS degree from UAF (or another college that is more on the theory side), you will be able to drop into any job and pick up the skills quickly. With a purely skills-based degree, if you do not have the learn-on-your-own-itude that is needed in this industry, you will be totally
lost when faced with a new paradigm or language.

Bottom line: if one requires a university class to learn a job skill, then a CS degree won't do one much good in the real world, whether theoretical or practical.

Over on his blog, Raphael Pinson has a post about creating a package with "minimal dependencies." In short, make the dependencies of the package the "real" dependencies, not just the latest and greatest version of the package you rely on.

I have been waiting way too long for someone to say this. This has especially bitten me when trying to back port things like Python modules. Both debhelper and python-central versions are automatically bumped to the latest, when they will compile and install just fine with versions that are one or two distro releases old.

Then, a request for backports is bounced with the message "too many dependencies" when in reality there wouldn't be "too many" if the dependencies were really set to what they need to be and not just whatever version is in the latest release.

In the case of Ubuntu, I personally would like to see all packages test-compiled with the versions of software that comes with the current LTS releases. Then, if it truly cannot be compiled with those versions, but the dependency to what it needs to be, but not any higher.

Related gripe: I have seen bugs in Ubuntu closed when a fix is released for the latest version, but the LTS version of the package was ignored. Maintainers: LTS means "Long Term Support," and as long as the support window is open for that release, the package should be supported and all bug fixes backported to the package (without having to enable backports, incedentally).

Thank you Raphael for bringing this up! Hopefully this will spark discussion and make it easier in the future to backport the "latest and greatest" to the current LTS release.

Last January, coming home from my Boston trip, I was sitting in Chicago O'Hare aiport, desperately searching for free wifi, when I looked at my kernel log messages. But, what to my frustrated eyes should appear, than another example of the humor of Open Source developers, specifically, the developers of the NetworkManager framework:

NetworkManager: request_and_convert_scan_results(): card took too much time scanning. Get a better one.

Granted, this warning is probably warranted. I was using a 802.11b card that is several years old, plus the calls are going through NDIS Wrapper. It was time to get a new laptop. The one was using was a Dell Inspiron 4000: PIII-800, 384MB of RAM, no built LAN or WiFi.

I recently had the task of building Python Eggs on Windows that had C extensions. I did the usual googling, and found a few HOWTOs, but nothing I could find was very concise and straightforward. So, I present to you a very concise, very straightforward guide.

Setting up the Build Environment

We are assuming all installers are allowed to install to their default locations.

Download and install Python 2.5 and/or Python 2.4 from python.org. You may have to use Python 2.4.4 since that was the latest 2.4 series to have an installer at time of writing. Whichever you want associated with .py files, install last.

Download and install SetupTools for Python (both 2.4 and 2.5) here.

MingW files are here. Download and run the latest version of "Automated MinGW Installer." You only need to install g++, and maybe not even that. When it prompts you for old/current/preview version of the MinGW system, select current.

Download and run the current version of "MSYS Base System."

Open up the MinGW shell, and execute these commands:


Create the file c:\PythonNN\Lib\distutils\distutils.cfg and put the following in it, where NN is replaced with 24 or 25, depending on version::

[build]
compiler = mingw32

Do this for each version installed.

Building the Eggs

MinGW mounts your drives on the root directory, so your C: drive will be at /c, D: drive at /d, and so on. Open up a MinGW shell, and:


You'll now have eggs for your architecture and OS. Enjoy!

It's amazing the things that can cause a lump in your throat. Say, for instance, your computer not booting.

Tonight I was trouble-shooting a trackball I thought to be dead, and I went to plug it in to a USB port on the front of my work computer. I had not properly grounded myself first, and this being Alaska in the winter, and me wearing sweats, and my chair's cushions being very much made out of polyester, I had acquired quite the static potential.

As soon as the trackball's USB plug contacted the metal around the computer's USB port, there was a nice "pop" and the power on the computer dropped hard. No graceful shutdown, just went black.

Oops. So I pressed the power button. The fans started, but nothing else. Shut it off, tried again. Still nothing after a couple more attempts. Now, I'm a little nervous for a few reasons: 1) This is my work computer 2) The computer is on loan; 3) I have no spare; 4) I just started a new job three days prior and calling in with a sick computer the next day was not going to look good.

So after a little consternation, and some prayer (this is my livelihood we're talking about after all), I remembered a little peculiarity I'd run into in the past. Computers, being electronic devices, often contain capacitors, whether those capacitors be in the power supply, or on the motherboard. Since most computers these days never truly turn off (it's a "soft" power switch) those capacitors might not fully drain. Somehow, in certain situations, the system gets "wedged" due to various reasons (a static discharge, for instance).

So, the fix was thus: 1) unplug computer, 2) hold down power button for a while (I did a minute, I think I've heard 30 seconds will work). This drains the capacitors fully. 3) Plug computer back in and press the power button. It's Aliiiiiive! And working fine.

That's enough excitement for the night, thanks!

P.S. Anyone care to enlighten me as to why the capacitor drain trick works?

UPDATE: Some interesting discussion of this on Reddit

Note to xzxzzx: Sorry that you couldn't link. I thought I had those enabled.

After being out of full-time work for 17 business days, I'm starting up a full time job! I'm going to be doing, as I have been for the past 2.5 years, programming and system administration. My title is still in flux, but my duties will be a combination of two jobs I applied for with this company, which were "Linux Web Application Server Architect" and "Python/Adobe Flex Programmer." For the first couple months (or less) I'm basically on as a contractor. Then, if we like each other, we'll negotiate for a full-time hire.

The company in question is called WordStream, "a venture-backed startup engaged in providing search engine marketing software solutions for PPC/SEM [pay-per-click/search engine marketing] and SEO [search engine optimization]," among other things. Basically, they help advertisers get the best deal and exposure for their dollar.

They do have a product they've released, but they're still very much in startup mode, so I'm sure it will be fast and furious. Not quite the low-key environment I was used to with S&K Aerospace. Financially, they're quite sound, having recently landed USD4 million in Series A funding from Sigma Partners, a very strong and stable venture capitalist group which currently has over USD2 billion under management.

So, off we go. Sounds like it's going to be lots of fun!

Oh, yes, I'll still be working from home. No relocation required.

Cilk Arts is very close to releasing their multi-core programming library called Cilk++, and with it, a new license, the Cilk Arts Public License. Using this license, they attempt to close the "IDO [Internal Development Organizations] loophole" they believe exists with the current GPL. Namely: an organization can take a piece of GPL code, create a derivative work, and, if they do not distribute it outside their organization, they do not have to give it back to the community.

Cilk Arts' new license seeks to prevent this, basically, be redefining "distribution" to mean within the company as well:

If you are an IDO building applications for use by others but not "distributed" under the existing open source definitions (e.g., GPL) and you want to keep your Cilk++-based derivative work proprietary, then there is an impact. The CAPL requires you to make a fair exchange in order to use Cilk++. If you share your software with everyone, we share ours with you. If you do not wish to share, you can give back to the project by purchasing a commercial license.

I believe,that while a laudable goal, this will at best be ineffective, and at worst, stymie, or at least slow, the adoption of Cilk++; and yes, I have reasons.

There are, at least, three types of Open Source consumers:

1. Those who have full intention to give back
2. Those who have no intention to give back, and don't, despite the legal obligation to do so. See the archives of GPL Violations for plenty of examples.
3. Those who don't initially have an intention to give back, but after using the library, decide that they'd like to, or can, give back.

The first two types of consumers will not be affected by this license. The givers will give, and the keepers will keep. It is the third type of consumer where Cilk++ will lose out: the "potential givers" (PG) category.

When a PG is evaluating various libraries to use for their new multi-core program, their license and distribution model (even internally) may be in doubt. They may even have a (wrong headed) policy about giving back to open source programs. Thus, when they see "you can't distribute derivative works (even internally)" it may completely turn them off, and they'll go on to the next library. Which would be sad; I've not personally reviewed Cilk++, but as it is coming out of an MIT project, I would assume it to be created by some rather bright people. So, the PG will completely pass over the chance to use this library without any further evaluation. Cilk++ will lose in this case. If the PG would have decided not to give back, Cilk++ would have not lost or gained either way. However, if the PG would have decided to give back down the line (say, enough internal rumbling by developers, or management changing policy), then Cilk++ will have gained a contributor. But, because they say off the top that you have to give back, they will have lost everybody in the "potential giver" category. In addition, they may also lose out on a possible license sale by those who want to try the library, and then end up buying a commercial license because they wish to distribute their application as closed-source.

So, my feedback to Cilk Arts is: go with the standard GPL (or even BSD license). Those who want to give back will, those who don't, won't, and those who might will give you a try and possibly become a contributor (with code or money) when they might not have otherwise.

Your thoughts?

Yet another reason I will avoid PHP at all costs. The head developers of PHP have decided, after much discussion, to make the back-slash the name-space separator when referencing classes. Thats's right, this\is\a\tree\valid\variable. I'm sorry, a back-slash? This will create no end of confusion and frustration when it comes to parsers and syntax highlighters. What was wrong with the period? How about '::' as in: a::valid::namespace?

One of the (small) reasons I like Linux that the only reason I have to use a back-slash is for special control characters. Seeing back-slashes all over my code would be even more jarring than the supposed "white space" problem some people have with Python.

For more on the decision, read the RFC. For discussion, Reddit has the usual.

It boggles the mind.

So, due to lack of projects at my current employer, I might be getting laid off come Wednesday. Not the news one wants to hear.

If anyone has any leads for telecommuting tech jobs, please let me know. There is a link to my resume at the right of this page.

Thanks!

As part of yearly compliance, I have to go through some sort of training on sexual harassment. (Yes, even though I work at home and can't really harass anyone.) I don't particularly mind, it's not that onerous. What I do mind, however, is having to redo sections of the training that I've already done.

The Brightline courses use a flash application that walks you through the various modules. You listen to dialog, training, and such, and answer questions. It keeps track of which modules you've completed...kind of. That problem arises when the application can't connect to its server. Instead of retrying, it pops up a message saying "You seem to have lost connectivity. Instead of using the wireless connection, connect to the LAN using a wired connection before restarting the course." It then simply closes out the flash application. Which wouldn't be so bad, I suppose, but it doesn't always save your progress, so I am now in the process of redoing a 20 minute module that I've already done.

I'm not having any other bandwidth or connectivity issues today, so it would seem to be their servers, not my connection.

So, two suggestions for those doing online courses:

• Save state often! Every page of a module, or more


• When connectivity to the server seems to be lost, try again, several times, instead of just dumping the student out of the application, and forcing them to login again. At least consider the possibility that the problem may be on your end, and not the student's.

Edit: Then, after completing the entire course, and exiting as instructed, it logged me out. Upon logging back in, it told my I had not completed the last three modules. Sigh.

Passwords are a part of life. They're needed, but they're a pain. But their ubiquity often leads users to create short, or otherwise easily-guessable passwords. Not good.

I recently signed up for an account on oDesk, a site for connecting IT free-lancers and service-providers, and those looking for said services. While a lot of sites discourage the use of strong passwords by saying you can only use letters and numbers, or limiting your length, oDesk requires you to have at least one letter and at least one number or symbol, and they even tell you how strong they think your password is. They give you a little bar that turns different colors based on how good your password is. Looking at their page reveals this:



They believe best passwords are 14 characters or longer, and have a letter or special character in there. I'd have to say I agree.

Scott Westfall has a post about defects vs. features. He notes:

I like the generic term, “change request” for all changes in a system. But it’s very important to know whether it is a defect or a feature request. In my lexicon, a “defect” is something that doesn’t work as spec’ed; a feature request is a request to alter the intended behavior.

He goes on to point out that customers generally don't care which it is, they just want things changed. Programmers often care about the definition because if it's a defect, it might mean their code is flawed. Yes, programmers can have egos, often huge ones.

But what he doesn't answer seem to answer is: who, ultimately, decides whether a submitted "change request" is a defect or a feature request? And does unexpected behavior, even if it specified as such, or simply undefined, qualify as a bug?

I've run in to this exact issue on a "change request" for a product I use: the Adept package updater in Kubuntu. On May 10, 2006, I reported that Adept doesn't behave properly when the network is down, but simply says the transaction failed. I finally discovered that it didn't properly determine if the network was up before it attempted to start the upgrade process.

After submitting this change request as a bug, it was changed to a "wishlist" priority. After it was changed, I wrote:

I don't agree with this bug being a "wishlist" bug. If a user does not know their network is not up, and they try to update, they are immediately going to write for help saying "Updates aren't working," or they are simply going to give up on Ubuntu because it "doesn't work right." I know I would have given up if I hadn't known to go to the prompt and type "apt-get upgrade," which is how I found out my network was down.

The change back to whishlist was accompanied by this comment:

You don't have to agree. It is wishlist though. You are asking for a totally new feature. And missing features != bugs in my world. Please leave it on wishlist, there is absolutely no point in bloating the severity. Thanks.

Now, I'm not sure if, or where, the specification is for Adept, but if a network-aware program I was using didn't tell me the network was down, but simply failed with no explanation, I would qualify that as a bug. I think that view is justified, considering that 1) another user agreed with me, and 2) my original report now has three other reports attached to it as "duplicates."

So, I agree fully with Scott that not all change requests are defects. However, when a program violates expected (and/or reasonable) behavior, the programmers/project managers/whoever really need to take a hard look at either the specification (if the program is in fact following the specification), or they need to define the behavior for the given case, and possibly bring it in to alignment with users' expectation[1].

[1] Yes, I know user expectations can vary wildly. Maybe a usability study is in order, but that's a topic for another post entirely.

Update: Good additional discussion of the problem. With a good example of something that should be classified as a bug in Visual Studio 2008, but still has not been fixed; and some discussion of how the bug/feature request dichotomy is bad for software projects as a whole since it can create friction between users and developers.

A band for programmers tired of automata:

Rage Against the State Machine

Thanks, I'll be here all year.