Thursday, November 19, 2009

One last thought on Wavelet image sharpening

The GIMP's plugin documentation for the Wavelet sharpen plugin is interesting:

"The wavelet decomposition of an image results in multiple images with different frequency content. When amplifying the high frequency parts the recomposed image appears to be sharper than the original one. That way the frequency which should be amplified most can also be selected and a given unsharpness in the original image can be taken into account."

That matches with Tom Crimi's description of Wavelets as vaguely similar to the uniquitous FFT.

Wednesday, November 18, 2009

Sharpening photos like they do in CSI

There's a recurring joke on nerd hang-outs like reddit about the "enhanced photo" plot device on dramas like CSI: a grainy, low-resolution image is zoomed and sharpened to bring out impossibly small details -- finding the reflection of the killer in a distant chrome hub-cap, that sort of thing. It's obviously stupid.

But after playing around with The GIMP's "Wavelet enhance" filter, I'm really impressed -- it really seems to bring out fuzzy details:




(My best effort at cleaning the image up with contrast & brightness on the left; the wavlet-enhanced version on the right.) Clearly, it can't restore missing detail, but it seems to do a good job of improving the contrast around edges / features.

Monday, November 16, 2009

Firefox can't print, and what to do about it.

Say you've got a big, information-rich, scroll-heavy page, with annotations and doodads located a long scroll off the screen. But say it works: you can scroll around and find what you need; it's a solid app.

Firefox won't be able to print it -- not usefully, at least, b/c Firefox will print only to the right edge of the visible page, thereby limiting you to the width of your physical screen. Creating a new paper-type, some monstrously wide fictional ur-sheet, won't help things either. So what's a boy to do?

Enter CutyCapt, a WebKit-based command-line tool that renders to PNG. (Or PDF, or whatever file format your heart desires.) Better still, it will execute any Javascript on the page, so pages that are essentially blank until the JS runs will render just file.

You'll need Xvfb, the X virtual framebuffer, if you want to run it on a headless box, but Xvfb is a little miracle in itself.

Tuesday, October 20, 2009

Microsoft Outlook 2003: The Worst Program of All Time

I'm forced to use Outlook 2003 by a clueless IT department -- forced, in fact, to sit an entire separate ThinkPad next to my iMac just for email, since a Mac can't possibly sit on the trusted network. It's a security threat, after all. Unlike Windows. Which is secure.

Having used GMail for years, I'm shocked at the sheer unfriendliness of Outlook:

Searching takes minutes, and only barely works.

Perhaps due to way Exchange is configured, Outlook won't auto-complete contacts by first name, only last, and even then it only auto-completes names from my personal contact list, not names in the Exchange directory.

Threading. THREADING. I had threading in 1998 using Mutt, a command-line email client. GMail has had threading since day 1. An email and it's replies form a dialog, obviously, and Outlook's failure to group those emails into a thread makes it about as useful to me as my Nokia 3510's SMS functionality was.

Friday, July 31, 2009

On technolgical improvements in lighting

Two out of three light bulbs burned out in our new lamp, so off to the store we went. We wound up with two different bulbs: an LED ($9), and a compact fluorescent ($7). This lamp has 1 incandescent bulb, 1 CFL, and 1 LED; would anyone care to guess which is which?


Wikipedia's entry on LEDs has an enlightening explanation of the temperature of such white LEDs; the phosphor that converts the LED's blue light into white light doesn't do a very good job of it, and much of the blue leaks out -- this graph captures the effect nicely.

Friday, July 24, 2009

PostgreSQL and capitalization

The terminal-based PostgreSQL front-end psql, which ships standard, does not like capitalized table names.

ri=> \d KDplot_plotpoint
Did not find any relation named "KDplot_plotpoint"
ri=> \d "KDplot_plotpoint"
Table "public.KDplot_plotpoint"
Column | Type | Modifier
...

Double-quotes are your friends.

Tuesday, July 21, 2009

On ZFS and terror

I've had my main backup / nameless cruft repository on 3 x 1TB drives in a ZFS RAID-Z array for a while now, and finally found out what happens ZFS (or Linux ZFS-FUSE, at least) encounters bad blocks.

It loses it, completely.

I suspect this is largely a Linux problem: once a bad block is encountered, the Linux kernel (?) seems to go into an infinite-reread tailspin, flooding syslog with errors and generally making the SATA bus (and the machine at large) unusable. So it might not be ZFS' fault.

And a quick aside to the many posts claiming that "the OS should never see a bad block; the drive should silently remap the block to a spare and the OS will only be aware of it when you've run out of spares!": bullshit. I've had half a dozen drives turn up w/ bad blocks, and the SMART stats on each reported plenty of spares, and each drive was fine after a forced overwrite. (IE, I zero'ed out the drive.)

So I repeated my zero-out procedure on this drive, and the sequence of events gets fuzzy, but at some point I backed up /etc and at some later point I ran zpool export on the DEGRADED array. When the time came to put the newly-zero'ed drive back in the array, zpool wouldn't bring it back up: zpool import reported that "The pool can be imported despite missing or damaged devices," but zpool import tank ("tank" being the customary name for ZFS pools, I think) complained that it "cannot import 'tank': one or more devices is currently unavailable." zpool import -f tank yielded the same thing.

This when when I began to panic.

Despite having 2 fully functional drives, all my data would have been lost. But remember that backup of /etc, notably including /etc/zfs/zpool.cache? That was my salvation: restoring that brought me back to my pre-export, degraded state, which in turn let me replace the faulty drive.

My data is safe for now, but my confidence in ZFS is shaken. Why did exporting a degraded array make it un-importable? If I lose my zpool.cache, is all lost?

Wednesday, June 3, 2009

Developer preferences

Ubuntu still ships with GHC 6.8, released 12 December 2007. Macports offers 6.10.3, which has been around for about 3 weeks.

I might be reading into this too much, but methinks there's something there.

Friday, May 15, 2009

On the prescience of Infinite Jest

It's not a central plot point, but DFW's near-future Infinite Jest (1996) mentions Microsoft Pink, the operating system du jour.

Perhaps he has a fan in Redmond?

Wednesday, April 22, 2009

Are you sure you want to navigate away from this page?

Scenario: you've been tasked with creating a simple web-based time-sheet application, so employees can log what percentage of their time is devoted to which project. Now, this hardly qualifies as "Bioinformatics," a domain claimed by both your job title and personal aspirations, but it's a small company and they pay your rent.

So you build the little beasty, and your cube-neighbor sits down to fill out a time-sheet. She spends 7 minutes selecting options and punching in percentages, and then realizes she wants to trim one field from 15% to 10%, clicks her mouse 4 pixels too far to the right putting the page, and not the text input, into focus, and hits Backspace -- equivalent to the back button.

There go her 7 tedious minutes of form-filling -- this app isn't sophisticated enough to autosave a draft in real-time, after all.

Enter the onbeforeunload function.

I've borrowed from this guy a bit, but here's the code:
var confirm_exit = true;

window.onbeforeunload = function() {
if( confirm_exit )
return "";
return; };

$(document).ready( function() {
$("#id_theform").submit( function() { confirm_exit = false; } );
} );
Originally, I'd planned to skip the ugly global variable and simply re-bind the onbeforeunload function, but apparently you can't rebind like that.

Friday, April 10, 2009

A Beginner's Guide to Hating iMovie '08

Thank you, Apple. This is pretty much my first experience editing footage into a "movie," and with iMovie '08, you've made the process slow, painful, and constrained. What's more, I was lucky enough to get my computer weeks before the release of iMovie '09, so any of your stunning omissions fixed in the subsequent version will cost me $100. No thanks.

First off, titles. Adding chapter titles is a pretty common use for iMovie, I would think, but clips cut from one Project and pasted in another are stripped of their title. Thanks.

Oh, but wait: "chapter titles?" What's a chapter? iMovie isn't really meant to create movies for later burning onto a DVD -- who would ever want to distribute a DVD? So very 2003! The only way I've found to separate footage into chapters for iDVD is to put each chapter into its own Project. Annoying.

Thursday, April 9, 2009

The perils of laziness

I've begun cargo culting my way into Haskell, in the hopes that I'll eventually grasp such mathematical mysteries as anamorphisms. In the mean time, I'd like to write some code that actually works.

To this end, I've written a program to digest the C. elegans transcriptome in silico, then check the resulting fragments against data from a mass spectroscopy experiment (Multidimensional Protein Identification Technology --"MudPIT"--if you're curious). I use regular expressions to break up the protein sequences, store the fragments in a big Data.Map, and then test my MudPIT fragments against the Map.

I've greatly simplified my function to be a simple list-to-map conversion -- which would be stupid, since Data.Mat.fromList does this quite well -- but the real function involves plenty of off-topic complexity, so here's the svelt form:
  add_frags (i:l) m =
let m' = M.insert i () m in
add_frags l $! m'
add_frags [] m = m
The important part is the $!, in bold. Given some function f and parameter x, f $! x is equivalent to x `seq` f x; the seq function, meanwhile, forces the evaluation of the first thunk. So f $! x applies f to x AFTER x HAS BEEN EVALUATED.

Without that forced evaluation, ghc will simply add more thunks to the stack until the stack is blown ... hours after the run has begun. You can pretty easily imagine this as a series of substitutions, which is exactly what we're talking about:
  import qualified Data.Map as M
m = M.empty
add_frags ["ACDEF","GHI","KLMNO","PQRST",...] m
-> M.insert "ABCDEF" () (add_frags ["GHI","KLMNO","PQRST",...] m)
-> M.insert "ABCDEF" () (M.insert "GHI" () (add_frags ["KLMNO","PQRST",...] m))
And off the stack grows, until pop.

Monday, March 30, 2009

RAID 5 on an Leopard Xserve

This seems too trivial to merit a posting, but there are other posts that might lead one to believe that Leopard doesn't support RAID-5 on an Xserve, which it does, provided you've got a RAID card installed. When booting the installation DVD, just look for "RAID utility" under the "Utilities" menu. Just be sure to create the RAID before you pick an installation disk.

Update: Jashugan's right; I've updated appropriately. Though I still like the sound of "Xserver." One last thought: why ever does the RAID 5 volume build process take something like 2 hours / 100G? I understand the time required to rebuild parity after replacing a failed disk on a live array, but the initial build should be faster than that. Unless the OS is doing a bad block scan on the disks or somesuch, but ... why do that for a RAID and not under other circumstances?

Tuesday, March 24, 2009

Regarding encryption ...

Not encryption in general, but one very specific instance of it: Apache SSL on my Ubuntu/8.10 box.

Ubuntu ships with two SSL modules available -- the canonical, OpenSSL-based Apache mod_ssl, and the newer mod_gnults (pronounced "noodles?"). I decided to try mod_gnutls for no good reason, and it seemed to work -- I could serve content via an encrypted connection! Except for the silent redirects. I'd point my browser to /, get redirected to my SSL login page as expected ... and then wind up on the login page over vanilla HTTP.

In situations like this, HttpFox is your friend. As are netcat and OpenSSL itself, which, if you'll pardon the tangent, has it's own netcat-like behavior indispensable for debugging encrypted network services:
   openssl s_client -connect myserver.com:443
Via openssl, here's the offending HTTP transaction:
GET /ri/account/login?next=/ri/ HTTP/1.1
Host: myserver.com

HTTP/1.1 301 Moved Permanently
Date: Tue, 24 Mar 2009 15:26:47 GMT
Server: Apache/2.2.9 (Ubuntu) DAV/2 mod_gnutls/0.5.1 PHP/5.2.6-2ubuntu4 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2
Content-Type: text/html; charset=utf-8
Location: http://myserver.com/ri/account/login/?next=/ri/
Vary: Accept-Encoding
Content-Length: 0
I ask, over HTTPS, for the login page, and it redirects me back to the unencrypted version. And I never quite figured it out, either -- I just switched to mod_ssl and the problem went away.

I can't even say this is a GNUtls bug, but it's certainly some strange behavior.

Monday, March 23, 2009

python-ldap authentication and you

Or me, at least.

Here's how I managed to make my python client code running on a Debian/Ubuntu box authenticate against a Windows Active Directory LDAP server:
  1. Obtain the server's cert: openssl s_client -showcerts -connect myserver.com:636 > their_server.pem

  2. Edit the resulting file to isolate just the cert you want. In my case, the file had two certs embedded in it, plus some identifying cruft. Preceding the first, desired cert was a block that looked like this: 0 s:/C=US/ST=MyState/L=MyCity/O=My Company Inc./OU=Information Technology/OU=For Intranet Use Only/CN=my.host.com, followed by -----BEGIN CERTIFICATE-----. Next came VeriSign's cert. Delete everything but the CERTIFICATE block for your server, including the BEGIN and END lines, and save to /etc/ssl/certs/myserver.pem -- you can save to any location, really.


And here's the code:
#!/usr/bin/env python

import ldap
#ldap.set_option(ldap.OPT_DEBUG_LEVEL,4095)
ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
ldap.set_option(ldap.OPT_X_TLS_CACERTFILE,"/etc/ssl/certs/myserver.pem")
l = ldap.initialize("ldaps://myserver.com:636")
l.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
l.simple_bind_s("username@DOMAIN","password")

Wednesday, March 11, 2009

The trouble with DRM

Cory Doctorow's brilliant denunciation of DRM, as delivered at a Google Author talk a couple years ago:
If you deliver to an attacker the cipher-text, the cipher, and
the key, and rely on the attacker not combining those except
under circumstances you dictate, you're living in a fool's paradise.
I was reminded of this by Roger Gregory, who directed me to a youtube video about cold-boot attacks.

Thursday, March 5, 2009

Timing out an arbitrary Python function

Update:

The problem with the signal.alarm approach described below is that the alarm WILL happen, thus interrupting a hung function in the failure mode we're trying to account for. But it will ALSO SIGNAL if that function returned as expected, thus throwing an exception at some random point in later code.

The solution is to disable the alarm, as described below:


Occasionally, child processes die. If you don't want your script to hang, you do something like this:

TIMEOUT = 10
process = subprocess.Popen(...)
(o, _, _) = select.select( [process.stdout], [], [], TIMEOUT )
if o:
return process.stdout.readline()
else:
os.kill(process.pid, signal.SIGKILL)
os.waitpid(-1, os.WNOHANG)

This works perfectly well (and mimics the C version pretty closely), but what if the process is already wrapped in a python module and you don't have direct access to the process?

This guy has a good solution:
  def raiseTimeout(a,b):
raise Exception()
signal.signal(signal.SIGALRM, raiseTimeout) # bind a signal handler
signal.alarm(1) # raise the alarm sig in 1 s
some_hang_prone_function() # this is the problem function
signal.alarm(0) # disable the previously set alarm


My original version used a lambda in place of a proper function, but it turns out that lambda a,b: raise Exception() isn't proper syntax.

Saturday, February 28, 2009

Bay Area Kvetch: Pubilc Transit on the Peninsula

This really does seem to be the only public transit route from my job to SFO:

Wednesday, February 11, 2009

Apple's USB dongle

The Ethernet port on my 2006 Macbook died, so I went to Fry's and picked up Apple's USB Ethernet adapter -- you know, the one w/ the imperceptibly fine print on the back explaining "only for use in Macbook Air." That one.

And it worked at first, until I unplugged it for the night and plugged it back in. No amount of rebooting helped, but finally I worked out the procedure:
  1. Under System Preferences -> Network, select the USB Ethernet device and hit the (-) button at the bottom, removing it from the list. This apparently unloads the driver from the kernel.

  2. From the command line, run sudo kextload /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleUSBEthernet.kext

  3. Back under System Preferences -> Network, press the (+) button and choose "USB Ethernet," then "Apply"

That should do it.

Thursday, January 29, 2009

How Not to Create a Dialog Box

From Aquamacs on Mac OS X:



Or perhaps this was plain OS X Emacs, Aquamacs certainly has the same dialog. "Save this but no more," eh?

Wednesday, January 28, 2009

My computer is taunting me. Loudly.

I'm leaving Xoma, and arrived at work yesterday morning to a surprise: my computer is taunting me! And loudly, too.

There followed much perusal of ps lists and greping through the entire filesystem, but I couldn't find the source. Nithin finally suggested I find any file altered in the past few days, which led me to ... Nithin's .bash_history. Which led me to this, from "site-packages/date.py":

#!/usr/bin/env python
import os
import sys
import random as ran

taunts = [
'Wha' + 't ar' + 'e y' + 'ou do' + 'ing?',
'go' + 'od ridd' + 'ance',
'say' + 'anara! suc' + 'ker', 'y' + 'ou a' + 're a dir' + 'ty trai' + 'tor',
'I tho' + 'ught w' + 'e w' + 'ere frie' + 'nds',
'a ro' + 'ck wa' + 's mi' + 'spla' + 'ced, ' + 'Kie' + 'ran ha' + 's vani' + 'shed',
'won' + 'der b' + 'oi h' + 'as go' + 'ne t' + 'o t' + 'he da' + 'rk si' + 'de']

if __name__ == '__main__':
t_index = ran.randint(0, len(taunts)-1)
os.system('os' + 'ascr' + 'ipt -e "set Vol' + 'ume 6"')
os.system(('sa' + 'y "%s"') % taunts[t_index])

sys.exit()


This process was run every 15 min by a launchd plist called "com.apple.spaces.plist," which sounds pretty innocuous. Brilliant!

Xoma's lucky that Nithin Reddy and Jake Rothenbuhler possess more loyalty and general fortitude than I do.