See Jeff Run

Seeing My DNA

2011-05-20T14:14:00.002-06:00

There's a photomicrograph of my chromosomes on my living-room wall. I made it myself in a Human Cytogenetics course. When folks ask, I say it's a self-portrait.

I've been thinking about subscribing to 23andme, which analyzes your DNA, for several years, but the price was always too high. I knew it would drop over time.

Two things kicked me over the edge. One was posts by folks like Razib Khan about recent moves by the FDA to shut down or hyperregulate 23andme. The second was a trip to the Boulder Android Meetup, where I met G. Hussain Chinoy.

Chuck Coughlin and I went over to recruit an Android developer or two for a project we're working on, and Hussain sent me email saying he was interested. He wasn't a great fit for the particular slots we were trying to fill -- it happens -- but I told him I was very interested to see that he was a 23andme customer.

A few days later, Hussain dropped me a quick note to say that 23andme was having a "free day." Normally, the base price had been $199, but for that day, it was free.

That did it.

I'm now a happy customer.

I should say more about 23andme, but this post's long enough for now. Go look for yourself.

My company's still hiring, by the way -- contract and full-time positions.

Audiobooks, Sort Of

2011-05-15T21:00:00.002-06:00

I began looking for mp3 audiobooks to put in my Amazon Cloud Storage, so I could play them on my Android tablet, and ended up somewhere I hadn't guessed.

I've read that folks use their phones as MP3 players. Mine has its battery run down so fast when I use it as a phone that I can't imagine doing that. Perhaps I'm missing something.

But my NookieTab is a toy I'm exploring the boundaries of, so I bought some music, put it up in Amazon's free, Cloud Storage service, and tried playing it with their Cloud Player. Works fine.

Great.

How about an audiobook? Amazon doesn't sell many audiobooks in its MP3 store, because of DRM, but I saw Indian Tales, by Kipling and bought it. Kipling spins great yarns -- very listen-able.

Then I tried it. The book is machine-read. The synthesized voice has a British accent, which is a nice touch.

I went back and saw a book on Italian. This time, I sampled it before buying. Yup! It's lessons in synthesized English on how to speak Italian. Hearing is believing.

My Nookie Tablet

2011-05-14T06:52:00.005-06:00

The Web is filled with instructions on how to root and re-flash the Nook Color.

It has much richer hardware than, say, a Kindle, so although it's sold as a locked-down, e-book reader, a chunk of its market is folks turning it into something else.

Think pogoplug. Think LinkSys WRT54G. Think DEC PDP-7.

BestBuy had stacks of Kindles, but the shelf under the Nook was cleaned out. The salesman had to go to the stockroom to find me a box.

It comes as a locked-down reader, but the base OS is Android so you have two choices: root or re-flash. They'll both void your warranty, so it's a matter of personal taste.

If you re-flash it, there's plenty to choose from. Google will find you instructions for putting on anything from vanilla Froyo to Ubuntu. Someone probably has a port of Windows CE. To each, his own.

I opted for my usual route: I asked the nearest beautiful woman, "Could you help me turn this into an Android Tablet?"

She said, "Sure. It'll take about five minutes, unless you want to back it up first."

Five minutes was not hyperbole.

She had a micro-SD chip with the necessary files on her -- the OS, the Google Android apps, and an over-clocker. We sliced off the Nook's shrink wrap, took it out, popped open the SD port, slipped in her card, brought it up, reformatted the disk and loaded on the new software, and re-booted.

Never even un-boxed the instructions.

"Don't we need to charge it first?" I asked.

"I'm sure it comes with plenty of power," she said.

We looked after it was back up. The battery was 97% full.

It found someone's near-by, open WAP easily, so I configured it to use my Google account, checked my email, and downloaded a couple of apps from the Android marketplace.

I'm running Cyanogenmod's Gingerbread port (Android 2.3.4, with a 2.6.29 kernel).

The display is beautiful, but before I could watch a free movie, from Amazon Instant Video, I had to download Flash Player from the Android Marketplace. Their service does whine that I'm using an unsupported device, and the video's a little jerky, but I don't know whether that's the tablet or the format and bandwidth.

The rest is fiddling. For example, making the terminal usable meant using the menu key (it's all soft keys) to make the fonts bigger. Pairing with a blue-tooth headset required figuring out how to turn the blue-tooth headset on.

The earphone jack works. There's no built-in camera or microphone, and I don't think the hardware's there to turn it into a standard cellphone, but it seems like I should be able to make VOIP calls with the blue-tooth headset. I've poked at that one a little, but not succeeded. Yet.

I'd wiped out the Barnes & Noble book reader, so no Nook e-books for me; instead, Amazon's free, Kindle-for-Android app lets me download and read the books I buy for my Kindle.

I suppose this is a great time to plug the Kindle. Want a tablet computer? Lots of choices, including the Color Nook. Want an e-book reader? Buy a Kindle.

Sturdy, simple, beautiful, easy-to-read, dirt cheap. Connects free everywhere. Electronic paper that's as foolproof and power-thrifty as an Etch-a-Sketch. Unbeatable selection of books.

It is the C-47 of e-book readers. Jeremy Hinegardner tells me his grandpa's on his second one.

Toys

2011-05-14T05:32:00.000-06:00

Jealous of Steve Ballmer's buying Skype, I drove to Best Buy, Thursday afternoon, and bought a Color Nook.

The salesman offered me a protection package for $99. I said, "I'm going to void the warranty on this so fast it'll make your head spin."

The board has roughly the same chip set as the phone we're building at work. As a Nook, it was a locked-down, single-purpose reader. So far as I know, B&N won't even let you get to some of its hardware, like the bluetooth. Reflashed, I can now even load on whatever we write for our phone and play with it.

He said, "Next few hours?"

I said, "I'd guess the warranty on this will be void fifteen minutes after I leave the store."

I wasn't wrong. I now own a $250 Android tablet.

Before I went to bed, I slammed on Amazon's Kindle for Android app. Heh.

I'm waiting to see what Skype looks like after Ballmer re-flashes it.

RMBB!

2011-04-22T18:57:00.001-06:00

RMBB!

'bout time. See you there.

Free AWS Book Chapters

2011-04-14T08:46:00.002-06:00

In my email this morning, from Amazon:

*Free AWS Book Chapters*
Download a free PDF copy of the first three chapters of AWS Evangelist Jeff Barr's book, Host Your Website in the Cloud at http://www.sitepoint.com/books/cloud1/samplechapters.php. Jeff's book covers principal AWS services including EC2, S3, SimpleDB, CloudWatch, and RDS. The book makes use of the AWS SDK for PHP and includes plenty of working examples. Find the book on Amazon.com: http://www.amazon.com/Host-Your-Web-Site-Cloud/dp/0980576830

If you went to BLUG's AWS-Installfest-in-the-Cloud event, this ties right in. Good book, too.

Sort of Balkan Dancers

2011-04-11T08:31:00.002-06:00

Another forehead-smacking episode of "Why didn't I think of that?"

HT: Chuck Hipschman

23-and-me one-day sale

2011-04-11T08:13:00.002-06:00

23-and-me gives you information about your own DNA. I just became a customer. They're running a sale.

Normally, it's $199, plus a $9 a month subscription to updates for a year (after which you can continue to subscribe or cancel).

Today only: you still have to subscribe, but the $199 is waived, so it's about $100 instead of about $300. I'll find out whether the subscriptions are worth it over the next year.

Note that Federal bureaucrats want to shut them down because they think your DNA is something that only an M.D. should explain to you. Even without that, though, I'd have still bought it.

Handling Failure

2010-12-24T10:38:00.005-07:00

Here are two coding styles for handling failure. I read a lot of code that uses the first. I always use the second.

(1)

do-something;

if (success) {

do-this;
do-that;
...
[many lines of stuff]
do-some-other-thing;

}

else

{

print-error-message-and-exit;

}

(2)

do-something;

if (failure) {

print-error-message-and-exit;

}

do-this;

do-that;

...

[many lines of stuff]

do-some-other-thing;

I could go on at length to try to justify my strong preference, but I won't. In the end, it's just taste.

No, I take it back, I will rationalize a little. Here are some reasons I prefer #2.

It's shorter overall.
The short alternative comes first, so you don't overlook it in the noise.
There's one fewer nesting level to puzzle through. This also occasionally permits longer lines without word-wrap.
It feels more like exception handling, and saying what I mean: A problem? Let's bail.

In Perl, the solution is even more stereotyped and easy-to-read:

do-something OR die "it didn't work";
...

In bash, I use a shell function, die(), which I wrote to let me code in the same style:

do-something || die "It didn't work"

...

Process Whack-a-Mole: An Alternative Approach

2010-11-29T19:54:00.003-07:00

Gosh. I haven't blogged in, like, forever. Inspired, as is so often the case, by Hal Pomeranz and friends, here's something.

Over at his Command-Line Kung Fu blog, Hal has posted a solution to the problem of writing a process-whack-a-mole command, that watches for any new process and kills it.

(Hal's self-imposed constraint is always that his solutions really be command lines -- they can't be full-blown scripts -- and that they not use Perl, or other languages that are powerful enough that their use would feel like cheating.)

Nothing wrong with his approach at all, but here's another. (I mailed it to Hal, who said, "Post it somewhere, Dude.")

$ ps -e o pid,ppid,cmd | grep -v $$ > baseline; while : ; do
> sleep 5
> ps -e o pid,ppid,cmd | grep -v $$ > now
> join -v 2 baseline now
done

This is, at least, a single command line. :-)

It shows off a couple of things:

join -v
the output of ps is pre-sorted by process-number

The didactic downside? It doesn't show off arrays or "[[" . (I spent a minute in philosophical reverie, contemplating the elevated virtue of using files to store data for shell scripts, but then realized this is engineering and snapped out of it.)

Oh, and I completely love Hal's "| grep -v $$", which I never would have thought of.

On reflection, the only thing the join would normally see is the ps itself, which is gone by the time of the join, so for the real deal, this can be pared down a titch.

# ps -e o pid,cmd > baseline; while : ; do
> sleep 5
> ps -e o pid,cmd > now
> kill -9 $(join -v2 -o2.1 baseline now)
done 2>/dev/null &

Just kill everything new and ignore the error message from not being able to kill the ps:

"Kill them all and let The Kernel sort them out."

Just in case

2010-06-16T12:00:00.003-06:00

I just learned something new about shell syntax -- specifically, about the case statement.

The shell's big enough that I expect I'll be able to keep learning things about it for a long time, even though I've been writing shell scripts for ... lessee ... about 30 years.

Here's today's:

I'm used to writing this:

case $key in
whatever) do-something ;;
*) some-default-behavior ;;
esac

I just read a Fedora system script that looks like this, instead:

case $key in
(whatever) do-something ;;
(*) some-default-behavior ;;
esac

The leading paren is optional, but legal. And not just for bash, but for any POSIX shell. Amazing. Also, the last item doesn't have to have the semis, so it could even be this:

case $key in
(whatever) do-something ;;
(*) some-default-behavior
esac

Back Off Man, I'm a Scientist

2010-05-26T14:12:00.007-06:00

"I have no faith in anything short of actual measurement & the Rule of Three"

-- Charles Darwin

In an interminable thread about recycling computer parts, on the Boulder Linux Users' Group mailing list, Davide Del Vento injects a completely unsubstantiated and largely dubious claim:

The point of the discussion is what we agree is ok to "kill" or "destroy", and what is not. Is it ok to kill a cow to feed my kids? Most people would say "yes". Is it ok to kill Jeffrey Haemer to feed my kids? Most people would say "no".
Doing science means collecting data. The polls in the left sidebar test his claim.

We shall see, eh?

NCAR Software Manufacturing Presentation, the Movie

2010-05-14T11:04:00.003-06:00

Courtesy of Davide Del Vento, here's the video in H.264.

NCAR Manufacturing Presentation

2010-04-28T18:07:00.004-06:00

What I'll Say

How I'll Say It

Having The Exact Code You Need

Building What You Have

Testing What You've Built

Looking Closely at What You Tested

Sharing It After You've Looked At It Long Enough

"What Was He Talking About?"

"Who Was That Masked Man?"

Performance Tuning Shell Scripts? Why, yes.

2010-04-08T08:52:00.012-06:00

Can I write a command-line that lists open TCP ports as quickly as nmap? No way. But can I make one that's fast? Yes, indeedy.

The trick is to do it all in a single shell command.

I always read Hal Pomeranz's weekly, Command Line Kung Fu blog. Inevitably, either I learn something because he tells me, or I learn something because it gets me thinking about how I might do what he's done differently. (That's not an exclusive-or.)

This week, Hal writes a command-line that looks for open TCP ports.

Here's his command. (For a dramatic reading, see his post.)

for ((i=1; i<65535; i++)) ( echo > /dev/tcp/localhost/$i ) 2>/dev/null && echo $i; done

It isn't fast, and he ends with, "But really, if speed were a factor you'd be using nmap instead."

But can I get it to run faster? At least a little? Why, yes I can.

Replacing this:

for (( i=1; i<65535; i++ ))

by this:

for i in {1..65535}

and this:

echo > /dev/tcp/localhost/$i

by this:

> /dev/tcp/local/host/$i (or even < /dev/tcp/localhost/$i )

make minor improvements.

But a bigger win comes from getting rid of subshells.

The parens around the echo create a subshell, which requires a fork() and an exec(), each time through the loop.

By getting rid of those, and discarding error messages at the end of the loop, all the work takes place right in the parent shell.

How much does that improve things? A lot. Here are the numbers.

$ time nmap -p1-65535 --open localhost

real 0m1.366s

user 0m0.280s

sys 0m0.850s

$ time for (( i=1; i<65535; i++ )) ( echo > /dev/tcp/localhost/$i ) 2>/dev/null && echo $i; done

real 1m55.727s

user 0m12.640s

sys 1m28.200s

$ time for i in {1..65536} ; do >/dev/tcp/localhost/$i && echo $i; done 2>/dev/null

real 0m6.203s

user 0m3.290s

sys 0m2.780s

Tom Christiansen claims he can usually write Perl scripts that run within a factor of 'e' (2.718281828...) of the equivalent C program. Here, I'm only doing half that well, but that's not bad.

Even in the shell, sometimes a little tweak makes a big difference.

I'd offer extra points to the reader who knows an attribution for the quote "Make it work, then make it fast," but that would require readers.

It was, however, Frank Zappa who said, "Speed will turn you into your parents."

There's a Lesson Here, but I Can't Remember What

2010-03-29T17:24:00.008-06:00

Perhaps you are graced with a mind like a steel trap. I have always had a mind like a steel colander.

I frequently read stuff I wrote, and think, "That's clever. Too bad I can't remember ever knowing that, much less writing about it."

Just now, I was reading an RHCE-prep guide that was explaining pr. I thought, "pr? Geez. There's some ancient history. Next they'll be explaining FORTRAN line-printer-carriage-control codes."

This starts me reminiscing.

"There was an old, Software Tools filter, in RATFOR, that would interpret those codes, called asa (a reference to the American Standards Association, a progenitor of ANSI's). At some point, it was ported to C/Unix. I should see if it's on my Ubuntu desktop."

(As an aside, and before I forget to say it, Kernighan and Plauger's Software Tools is the best book ever written about software engineering.)

It's not there. I think, "Well, okay, I'll install it.

I try apt-cache search and don't find it. Rats.

I google for an Ubuntu version. Nothing. A Linux version? Nothing. Humph.

Well, surely it was in UNIX Version 7. I remember some work Tom Christiansen put in, collecting Perl implementations of old, V7 commands. Maybe he found an implementation of asa(1) that I can just port.

Except I can't remember what he called his collection. I go back to googling, this time for Tom's collection. After a bunch of failed tries, I finally get a hit. You guessed it: a column by Jeff Copeland and, um, me -- Software Ptools -- which I have no recollection of ever having written. How embarrassing.

I should have given up right there, while I was behind, but Noooo .... ("What would you pay? But wait! There's less!")

Had we provided the name of Tom's project? Sho 'nuff: "Perl Power Tools." Maddeningly, the link in our column has gone dead. The universe hates me and there's no beer in the fridge.

I scroll down, hoping for another link. Ooh! Look! There's code! We implement a V7 command, right there in the column, to contribute to PPT ourselves.

We implement asa(1)

Oh, ow.

(I've now learned that the entire Perl Power Tools project has been moved to the CPAN by Casey West.)

Estimating: The Envelope, Please.

2010-03-14T14:01:00.003-06:00

How much does it cost Amazon to ship me a Kindle book? About a nickel.

How much did it cost us to get letters saying we're going to get census forms? About $50 million.

How do I get these? Back-of-the-envelope calculations.

Back-of-the-envelope calculations are the quick calculations we do, from simple assumptions, to give us a sense of rough sizes. They may not let us tell whether the answer is 5 or 9, but they can let us see the answer isn't 5 billion -- a 5 followed by 9 zeros.

My sister, Jo, the Tattooed Lady, wondered out loud, this week, "... just how many millions of dollars it cost The US Commerce Dept (read 'us, the taxpayers'), to send everybody in the US a letter this week that says that they will be sending us a census report to fill out. 'Ooooooo. Look out!!!! Here it comes!!!' "

Let's do a back-of-the-envelope calculation. (No pun intended.) It's not hard.

How much does it cost to send a letter? A first-class stamp costs $0.44. The USPS loses money, which is why they want to cut back to 5-day-a-week delivery. So the real cost of processing and delivering a letter is something like $0.50. Could it be $0.30? Or $0.72? Maybe. But it's less than five bucks and more than a farthing.

What's the cost of producing each letter -- printing, stuffing, and so on? At Kinko's, they'd charge you somewhere between a nickel and a dime. Ditto for the public library. Real money, but we're still talking a total cost of around half a buck per letter.

They sent one to each household, and America has over 100 million of those.

We paid at least fifty million bucks for those letters. $50,000,000 . As Jo says, "Here it comes."

But what did we pay to draft the letter, translate it into a bunch of other languages, and get all that approved and processed through our Federal bureaucracy? Probably not even an extra fifty million.

Here's a second example: What sort of profit is Amazon making on Kindle books? I wondered this last year when I bought my Kindle.

Let's see .... Once they've payed the publisher for the book, they probably get a machine-readable version for next-to-nothing -- maybe free. Converting to the Kindle data format is probably done by a piece of software that they wrote once, and amortize across all their books, which means that probably doesn't contribute much either. Amazon's big cost is probably delivery -- what they pay Sprint to get it to us.

So how much is that? Hand me that envelope.

They'll sell me a subscription to a blog for about $2/month. The content is free if I have a browser, and I can't imagine they're trying to make a lot of money from these, either. The $2 is probably Amazon's delivery cost.

The kind of person who reads a blog on his Kindle is a junkie who, I'll guess, might read it three times a day. That's 30*3 = 90, or about a hundred deliveries a month: two cents a day. Books are bigger, but they come over so fast that I bet connection-set-up and -tear-down costs dominate the price.

Amazon's sells new releases at $9.99. This calculation says almost all of that is profit. Their delivery cost, I guessed, was under a nickel a copy.

How close did I come? In a January press release, Amazon revealed it was "less than six cents."

When the government can send us useless letters by Kindle (or email), they'll cost us far less.

"But what could the government do with its vast inventory of surplus envelopes?" the politicians will ask.

Two suggestions come to mind.

Collatz Conjecture

2010-03-11T14:00:00.002-07:00

I like this shell script, by Kyle Anderson.

I found out about it because Paul Hummer's created a Northern Colorado Linux Blog aggregator.

Thanks, Paul! And Kyle.

Generating Arbitrary Numbers

2010-03-09T17:30:00.007-07:00

Sometimes, "arbitrary" and "random" aren't synonyms. Here's an example of how to generate the former without their being the latter.

One nice thing about knowing people who make me think is that it gives me things to post about. For example, Hal Pomeranz, Ed Skoudis, Tim Medin, and Paul Asadoorian have a weekly blog, called Command Line Kung Fu, that compares and contrasts command-line tricks for different operating systems.

I only every use Linux, so I read Hal's stuff and skip the Windows and DOS stuff. Even with this, every week or two Hal's post makes me think, "Wait! Here's something he didn't mention!" (typically because it's slightly off-topic).

In this week's column they generate random time intervals.

Here's Hal's punchline:

[...] in larger enterprises you might have hundreds or thousands of machines that all need to do the same task at a regular interval. Often this task involves accessing some central server-- grabbing a config file or downloading virus updates for example. If a thousand machines all hit the server at exactly the same moment, you've got a big problem. So staggering the start times of these jobs across your enterprise by introducing a random delay is helpful. You could create a little shell script that just sleeps for a random time and then introduce it at the front of all your cron jobs like so:

0 * * * * /usr/local/bin/randsleeper; /path/to/regular/cronjob

(The column sketches how to implement 'randsleeper'.)

Yep. This works fine.

But as it stands, the cronjob could kick off one job at 9:59, and the next one at 10:00. What if I want to spread my machines across the hour, but want each machine to use a fixed timeslot, so the elapsed time between runs is a full hour for any given machine?

Here's one way:

Pick an arbitrary machine-specific number, like the IPV6 address, or the MAC address of the ethernet card,
Convert it to an integer.
Take it mod the time interval.
Use that number for the time to start the job.

Here's code to do that, which, as always, I grow, bit-by-bit, on the command line, by getting a little piece right, recalling that piece, and adding another step.

Step 1:

Get a unique, but arbitrary, machine-specific identifier (the MAC address of the first NIC).

$ ifconfig | awk '/HWaddr/ {print $NF; exit 0}'
00:1e:c9:3d:c0:0c

Step 2:

Strip the colons

$ ifconfig | awk '/HWaddr/ {print $NF; exit 0}' | sed 's/://g'
001ec93dc00c

And interpret the result as a hex number. (The shell requires hex numbers begin with "0x", so I'll just tack that on.)

$ echo $(( 0x$(ifconfig | awk '/HWaddr/ {print $NF; exit 0}' | sed 's/://g') ))
132225286156

Step 3:

Mod it by the number of seconds in an hour, to get an arbitrary second.

$ echo $(( 0x$(ifconfig | awk '/HWaddr/ {print $NF; exit 0}' | sed 's/://g') % (60*60) ))
556

Step 4:

Always sleep until that many seconds after the hour, then kick off the job.

$ crontab -l > Cronjobs
$ echo "0 * * * * sleep \$(( 0x\$(ifconfig | awk '/HWaddr/ {print \$NF; exit 0}' | sed 's/://g') % (60*60) )); /path/to/regular/cronjob" >> Cronjobs
$ crontab Cronjobs

Ta-da.

(For step four, I'd probably actually kick off cron -e and paste the line in; otherwise there are just too many ugly backslashes to get wrong.)

Warning: This will not work if your machines' MAC addresses cluster around the same value, mod 556. :-)

Better Safe Than Sorry: Writing Code that Writes Safer Code

2010-03-02T14:30:00.008-07:00

I write code that writes code. A lot. On the command line. It's safer.

Hal Pomeranz and co-conspirators have another fine post up about command-line programming. In it, they write a clever loop to rename a list of numbered attachments.

Here's Hal's code:

$ cat id-to-filename.txt | while read id file; do mv attachment.$id "$file"; done

(His input file is a two-column list, like this:

$ cat id-to-filename.txt
...
43567 sekrit plans.doc
44211 pizza-costs.xls
...

And, actually, Hal takes the list from stdin, with a less-than sign. Blogger whines and eats my posts when I use those -- it thinks I'm opening an unclosed HTML tag. What a pain.)

The quotes are there because without them the code tries to do this:

mv attachment.43567 sekrit plans.doc

which gets the mysterious message back

mv: target `plans.doc' is not a directory
$

Uh-oh.

When this happens, I usually don't know what the message means. Figuring it out eats time. Plus, with my luck, some files have been moved but others haven't. Recovering from that eats even more time.

Here's what I type instead:

First step: I write code that says what I'd like to do.

$ cat id-to-filename.txt | while read id file; do echo "mv attachment.$id $file"; done
...
mv attachment.43567 sekrit plans.doc
mv attachment.44211 pizza-costs.xls
...

Often, when I do this, I scan the output, notice something's going to go wrong, and fix it.

"Oh. Oops. I need quotes. I'm an idiot."

Note that no files were moved; my code's only echoing commands.
Next step: I recall my command-line, with an up-arrow, and add fixes. I keep doing that until the commands I see are the ones I actually want.
$ cat id-to-filename.txt | while read id file; do echo "mv attachment.$id '$file' "; done
...
mv attachment.43567 'sekrit plans.doc'
mv attachment.44211 'pizza-costs.xls'
...
Look okay? Yep.
Last step: I recall the previous command, one final time, and pipe it to a subshell, which executes the commands my code writes.
$ cat id-to-filename.txt | while read id file; do echo "mv attachment.$id '$file' "; done | bash
$
When I'm nervous about what I'm doing, I even try out the first line by itself, like this:
$ cat id-to-filename.txt | while read id file; do echo "mv attachment.$id '$file' "; done | head -1 | bash
$

I check the result, and if I've done the right thing I go ahead and run the rest.
$ cat id-to-filename.txt | while read id file; do echo "mv attachment.$id '$file' "; done | sed 1d | bash
$
"Never write code on the command line when you can write code that writes code on the command line," I always say.

NFS Made Easier

2010-03-01T15:51:00.003-07:00

Automounting disks is magic. Autodetecting what to automount is magic-er.

Last weekend I set up my pogoplug as an NFS server, and installed and configured autofs to look for specific directories on the pogoplug. This weekend, I revisited that configuration and learned I was working too hard.

In /etc/auto.master, the entry "/net -hosts" says, "When I type 'ls /net/foo', do these steps:"

look for a host named 'foo.com' ,
ask foo.com what filesystems it's exporting
mount them under /net/foo.com
now do the 'ls'

No modifying /etc/auto.misc every time you want to automount a new machine: the machine just appears. There is an /etc/auto.net, but it's a script that autofs uses to ask a remote host what it's exporting.

Using remote filesystems could be even easier and more transparent (I could, for example, imagine having upstart manage the whole process, and having autofs be installed by default) but not much.

The Magic SysRq Key

2010-02-23T05:48:00.002-07:00

My netbook hangs about once a day. I can now reboot it more easily, thanks to the magic SysRq key.

I'm not sure why it hangs, and I'm not very worried about it. Netbook releases are new, and I remember the same kinds of problems when Linux notebooks were new. I had one laptop that worked perfectly except for sound, the mouse, and networking. :-) The next release fixed all those. (Even now, I have a pair of old notebooks running Jaunty Jackalope because Karmic Koala won't run their fans. They overheat and shut themselves off within minutes.)

I even remember the problems when Unix PCs were new. I was in Dave Barach's living room, in 1983, when he got a call from our Maryland office, which had just received the editor Dave had sent them for our as-yet-unreleased PC/IX product, done under contract to IBM, which was to run on stock IBM PC/XTs.

I could only hear Dave's side of the conversation.

What do you mean, "The color's wrong"? What do you mean, "color"?

We'd never seen a PC with a color monitor. Weren't all displays monochromatic? Once we found out how they worked, Dave fixed the problem.

I am, however, discomfitted by having to power-cycle a box -- I worry about unsynced disks and stuff. Luckily, some neuron fired and I remembered the magic SysRq key. I have had to use it so little, over the years, that I didn't remember any details, but Wikipedia came to my rescue. It does the trick, and lets me do a soft re-boot from my keyboard.

(My keyboard doesn't actually have a key labelled "SysRq," but Alt-"Print Scrn"-k works just as well.)

No muss, no fuss, no bad blocks. Magic.

Thrupence and sixpence every day.

Space, The Final Frontier

2010-02-23T05:33:00.003-07:00

Last night, I gave my Dell netbook a terabyte drive.

A few weeks ago, Scott Mann talked me into buying a Pogoplug, an embedded-Linux device, about the size of my palm, with an ethernet port and four USB ports to hang external disks off of. Plug it in, plug in a drive (or four), and you're done: no muss, no fuss. Kristina gave me a terabyte drive for Christmas, so they're now up and running in one corner of a bookshelf.

Unfortunately, it's built to serve disks up through a web interface, so each byte goes out the door, off to their servers in San Jose or Boca Raton or Minneapolis or wherever they are, and back down to whereever it's wanted.

But wait! It's a Linux box.

The OpenPogo community has a repository of downloadable packages that you can use to customize it in a variety of ways. (It's a Debian-based distro, and the package manager is ipkg.) I turned it into an NFS server by installing unfs3, exported the disk, and it was instantly visible locally.

On my netbook, I then installed an automounter, autofs, and -- ta-daa! -- now the disk's there whenever I look at it.

Space. The final frontier. [ Cue Star Trek theme. ]

Where's My Jet Pack?

2010-02-16T06:28:00.001-07:00

At some point, I'll outgrow the wonder of new technology, but I hadn't by this weekend.

My bedside computer is a Dell mini-10 netbook. I've had it a week. Two gig of memory, 160 gig of hard disk, three pounds. The 6-cell battery lasts for hours. Whoof.

I bought it, on-line, from Wal-Mart, who delivered it in three days. Same week. I installed three different OS's on it, and settled on Ubuntu 9.10 (Karmic Koala).

Ubuntu is a lineal descendent of Unix, which I first installed on an IBM PC/XT, of which this Dell is, itself, a lineal descendent. The XT had an Intel 8088 processor, 256Kb of memory, and a 10Mb hard disk with another 10Mb expansion chassis to give it enough space to host multiple users. Those, and its cathode-ray-tube screen, took up a big chunk of a desk.

A megabyte, for those who don't rememeber them, is a milli-gig. A kilobyte is a milli-meg. "You had ones *and* zeroes?"

Kristina doubled the netbook's memory with a kit, also from Wal-Mart, helped by a call to a friendly Dell tech support guy, in Chennai, India.

Let me just pause to say that again: a phone call to Chennai, India. My mother's childhood phone number was 1. They had the first phone in Haynesville, Louisiana. To call my grandmother, we talked to operators. "I think Stella's down 't the beauty parlor. I'll ring down there."

This weekend, on my computer, I watched Spartacus, in bed. I downloaded and read a Kindle book, bought on-line from Amazon.com, with "Kindle for PC." I made a Skype video call to my sister, Jo, in Oregon. I did a software release at work from my living room while I was arranging for a barbershop quartet, from the Boulder Timberliners, to come seranade Kristina for Valentine's Day, at a restaurant I took her to.

I arranged it on my cell phone -- you know, the phone I carry in my pocket? The one running Linux? With the videocamera in it? That I get email on? That could give me turn-by-turn, voice directions to get to the restaurant? Which it could do by knowing where I was from the signals it was getting from the GPS satellite, in outer space?

My father helped open Vandenberg A.F.B., America's first operational missle base. He had computers with big cabinets that held tape drives with 7" reels. The computers and drives and disks took up big rooms and had their own air conditioning. They had bugs, too. We watched the first Atlas ICBMs launch, go astray, and then blow up. Made for cool sunsets.

Of course, we drove to the Valentine's Day dinner in a car. When he was a kid, my father's little sister was run over and killed, in Brooklyn, by a horse cart.

Time to get up and shower, so I can find out what new things I'll see today.

I feel like Duck Dodgers.

Tar Wars

2009-12-18T14:24:00.002-07:00

Here's a little, galactic-history lesson. When you finish it, you'll know both a new command and a portable way to copy directory hierarchies.

A long time ago, in a galaxy far, far away, there was a great struggle over which archiving tool the POSIX standard would choose: BSD's tar or AT&T's cpio. Who would rule the Empire? TAR2-D2 or 3Cpio? This mouse-and-frog battle was dubbed Tar Wars. (Cue John Williams' score.)

The winner was us. AT&T's Glenn Fowler, with Joerg Schilling, designed and implemented pax (Posix Archive eXchange), to supplant both. Pax would read and write either format, and is now on every POSIX-conforming system. Better still, on mine it's 30% smaller than tar, and less than a third the size of cpio.

Peace through unity.

A little-known-by-me feature of pax is that it will even copy directory hierarchies. Me, I use cp -a to do this job, but that's a GNU-specific idiom. Other versions of cp may lack the -a flag.

This portable command will copy olddir into newdir (newdir must exist), preserving ownerships and permissions:

$ sudo pax -pewr olddir newdir

Mnemonic: the copy is pure ("pewr").

Octal NUL

2009-12-09T09:28:00.003-07:00

"Well, here's another nice mess you've gotten me into." -- Oliver Hardy

It's disappointing, but not surprising, to see edge cases behave differently in different programs. It is surprising when they're inconsistent within one program: bash.

Oh, the behaviors are standards-conforming and well-documented. Still, watch this, keeping in mind that printf and echo are shell built-ins:

# within single quotes, the shell doesn't expand metacharacters
$ echo 'a\0000b' | od -c
0000000 a \ 0 0 0 0 b \n
0000010
# but echo -e interprets \n, \nn, \nnn, and \nnnn as octal characters
$ echo -e 'a\0000b' | od -c
0000000 a \0 b \n
0000004
# printf, however, only takes \n,\nn, and \nnn as octal
$ printf 'a\0000b' | od -c
0000000 a \0 0 b
0000004
# but $'...' is interpreted as a C string,
# and in C, \0 terminates a string
$ echo -e $'a\0000b' | od -c
0000000 a \n
0000002

In the last case, it's the shell interpreting the octal string before it even gets to echo.

Want proof?

$ cat <<< 'a\0000b' | od -c
0000000 a \ 0 0 0 0 b \n
0000010
$ cat <<< $'a\0000b' | od -c
0000000 a \n
0000002

Note also that much of this quirkiness only appears when you start using four-digit, octal representations and mess around with NUL (\0). Try keeping all those details in your head, bucko!

Me, I can't. Or won't. Good thing it's all documented in the man page.

"A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines." -- Emerson

Hat Tip: I started looking at this after puzzling over a line in Hal Pomeranz's latest Command-Line Kung Fu column.

Joinery

2009-12-04T10:59:00.002-07:00

The join command has some useful options.

Hal Pomeranz has a nice example of using join to combine the output of two different commands in this week's Command-Line Kung Fu column.

After some discussion, he ends up with this:

$ join -1 1 -2 2 <(openssl sha1 * | sed -r 's/SHA1$(.*)$= (.*)/\1 \2/') <(wc -c *) \

| awk '{print $2 " " $1 " " $3}'

One reason he uses openssl is to help teach that in process substitution, the contents of <( ) can be pipeline. If you relax his didactic requirement, join's options let you do the job with a lot less typing.

$ join -j2 -o 1.1,0,2.1 <(sha1sum *) <(wc -c *)

Note that sha1sum has the same output format as wc -c, which makes the join easier.

Non-Linux boxes might not have sha1sum, but if I didn't have it, I'd see if I had md5sum, which has the same output format. Their Wikipedia entries say these commands are widely available on lots of non-Linux OSs.

My Phone Starts Giving Me Things I Never Had

2009-11-24T08:38:00.002-07:00

This morning, my Android phone used Google's Turn-by-Turn navigation to coach me into work.

I'd wanted to play with voice navigation every since I got my Android phone, a few months ago, but all the available apps were pay services, so I waited. A few weeks ago, Google announced they'd provide it with Android 2.0, but my HTC Magic (T-Mobile myTouch) was still running 1.6.

Picture me, tapping my toe impatiently.

Yesterday, they back-ported the functionality to 1.6 and put it in the app store, and I grabbed it. This morning, I put in my destination, sat it in my lap, and set out.

Hearing it tell me directions as I drive is startling.

In one-quarter mile, turn right onto on-ramp, Highway 157, North.

Turn right onto on-ramp, Highway 157, North.

...

My father's Bell model 500 sits on my desk. When I got my first cell phone, I couldn't believe I was carrying a telephone in my pocket.

I shut off my land-line service, and jumped in with both feet. I quickly discovered I also had my watch, my alarm clock, and my pager in my pocket. My old pager was long gone, but I donated my alarm clock to the Salvation Army. My watch, a gift from an old girlfriend, went in a drawer.

The myTouch put my email reader in my pocket, too. To underscore this, Qwest, my DSL provider, had a service outage a week or so after I got it. No computer, but I could still read my email.

Having both persuaded me that I should take my Google Contacts seriously, so it now really does store all my email addresses and my phone numbers. The app that stores my library-card, my grocery-card, and other bar codes was also useful.

But all that was extensions to things I already had. The bubble level app was cute, but not important. Google Sky Maps was very, very cute, but not important.

Other stuff? Nice, if not quite ready for prime-time. Browser? Slooooow. Calendar? Too small to be useful. eBook reader? I'll let you know once there's a Kindle app.

Amazon Marketplace showed me the future of retail, but it's still mostly an "Oooh!"

The turn-by-turn directions though? Important.

I have a notebook full of maps I've printed to get places -- my own, personal atlas. No more. From now on, my phone can just tell me how to get where I want to go.

The phone in my pocket has replaced the GPS that I never owned.

Translate-As-You-Type

2009-11-17T12:46:00.002-07:00

I'm interested in translation tools. Google Translate now does translate-as-you-type, instead of waiting for you to finish typing and submitting the result.

It's claimed that it will also do phonetic translation of non-Roman scripts, but I'm not seeing that with Yiddish.

The "help" Command, And Other Aids

2009-11-12T13:39:00.002-07:00

In his latest Command-Line Kung Fu post, Hal Pomeranz talks about places to get help on Unix systems and their kin. (Technically, that's "their latest post," but his co-conspirators write about Microsoft stuff, which I don't care about.)

Hal concentrates on man(1), info(1), and apropos(1), and the --help flag. Good places to start, but there are other things worth mentioning.

So I will.

locate(1) is quite handy. I often find myself hunting for commands that don't have man entries of any kind, and are installed someplace bizarre. Once I find them, /path/to/command --help often tells me enough. If not, but they're in some special directory, there's sometimes a README or examples that do the trick.

Another frequent stop is the web. If command --help doesn't tell me what I want to know, the web's more likely to have a man page than my machine is. I'll now usually go there first.

If it's not an executable? How about type?

$ type ls
ls is aliased to `ls --color=auto'
$ type cdjob
cdjob is a function
cdjob ()
{
local d;
: ${1:?"usage $FUNCNAME %N"};
d=$(jobs $1 | perl -lane 'print "cd $1" if m/.*$wd: (.*)$.*/');
test "$d" && eval $d
}

And what's type?

$ type type
type is a shell builtin

For information about type, do we have to pour through the zillion-page, bash man page? On-line?

Nope. Watch:

$ man type
No manual entry for type
$ type type
type is a shell builtin
$ help type
type: type [-afptP] name [name ...]
Display information about command type.

For each NAME, indicate how it would be interpreted if used as a
command name.

Options:
-a display all locations containing an executable named NAME;
includes aliases, builtins, and functions, if and only if
the `-p' option is not also used
-f suppress shell function lookup
-P force a PATH search for each NAME, even if it is an alias,
builtin, or function, and returns the name of the disk file
that would be executed
-p returns either the name of the disk file that would be executed,
or nothing if `type -t NAME' would not return `file'.
-t output a single word which is one of `alias', `keyword',
`function', `builtin', `file' or `', if NAME is an alias, shell
reserved word, shell function, shell builtin, disk file, or not
found, respectively

Arguments:
NAME Command name to be interpreted.

Exit Status:
Returns success if all of the NAMEs are found; fails if any are not found.
typeset: typeset [-aAfFilrtux] [-p] name[=value] ...
Set variable values and attributes.

Obsolete. See `help declare'.

Yes, there's really a help command. It gives help about shell builtins.

If I don't even have the command? (If, for example, it's part of someone else's script I've pulled down from somewhere?)

Here's what I get out of bash when I invoke als(1) on my Ubuntu box.

$ als
The program 'als' is currently not installed. You can install it by typing:
sudo apt-get install atool
als: command not found

Not help with the command, okay, but help getting it so I can then get help with it.

What would you pay? But wait. There's more.

If I guess at a command but mis-type it?

$ las
No command 'las' found, did you mean:
Command 'als' from package 'atool' (universe)
Command 'ls' from package 'coreutils' (main)
Command 'lfs' from package 'lustre-utils' (universe)
Command 'lms' from package 'lms' (universe)
Command 'les' from package 'atm-tools' (universe)
Command 'last' from package 'sysvinit-utils' (main)
Command 'laps' from package 'epix1' (universe)
Command 'lvs' from package 'lvm2' (main)
Command 'cas' from package 'amule-adunanza-utils' (universe)
Command 'cas' from package 'amule-utils' (universe)
Command 'as' from package 'binutils' (main)
Command 'ras' from package 'ras' (universe)
Command 'kas' from package 'openafs-kpasswd' (universe)
Command 'lat' from package 'lat' (universe)
las: command not found

I think that's nice. And I can use apt-cache show to tell me what each package is.

Valour-IT Fundraiser

2009-11-11T06:03:00.005-07:00

Semper Fi.

If you're geeky enough to be reading this blog, you're geeky enough to appreciate the idea of a project that provides computers to wounded vets.

Checking Scripts for Syntax Errors

2009-11-09T10:17:00.002-07:00

If I'm maintaining a lot of shell scripts in a directory, I usually want to syntax-check them before I check them in.

The file(1) command will mark most of them as shell scripts. The rest are, typically, libraries of shell functions that I name with .sh suffixes.

All shell scripts that file can find:

$ file * | awk -F: '/shell/{print $1}'

plus the libraries

$ file * | awk -F: '/shell/{print $1}' ; echo *.sh

now syntax check them

$ for i in $(file * | awk -F: '/shell/{print $1}' ; echo *.sh); do bash -n $i || echo $i fails; done

Just with command-line recall and editing.

Google's Bulgarian Picture Dictionary

2009-10-23T11:15:00.004-06:00

Language translation is serious business, but playing with automatic translation tools is just fun.

I use Google Translate a lot, along with their translation bots. I've played with the Word Monkey gadget for iGoogle.

I've even built spreadsheets to do translation with Google's spreadsheet functions; I'd bet even beginning corpus-linguistics students could do some cool experiments with these.

Today, my pal, Kevin Cohen, put up this Google Chat status message:

My favorite Bulgarian word is хляб. What's yours?

I put it into Google Image Search, to see what would happen, and got this.

I think I can guess what хляб means.

A GUI Replacement for sshfs

2009-10-22T09:11:00.004-06:00

I've been using sshfs to remote-mount my home machine onto my work machine. Now, I've stumbled on an alternative that feels better integrated with my Gnome desktop: gvfs (gnome virtual file system) in Nautilus.

The short version:

$ nautilus sftp://test.com

brings up a file-browser window with test.com in it, which I can then browse and click around in. Also, test.com is mounted as ~/.gvfs/sftp on test.com/

Yes, it has all those blanks. Oh well. You can still get to it on the command line and in scripts.

If you already have a file-browser window up, just use the URI sftp://test.com

If you need to go in through a different port from 22, say 666, you'll first need to add a pair of lines to your .ssh/config file that look like like this:

Host test.com
Port 666

You can test whether they work or not by using ssh by hand.

$ ssh test.com

Bugs

2009-10-19T06:15:00.002-06:00

All programs have bugs. A "mature program" is a program with more obscure bugs.

When I first started programming, I was convinced that half my programs had uncovered bugs in the compiler.

They never were. "Oh. 'Missing semicolon on line 71' goes away if I put a semicolon in."

Because compilers are more mature than my code, the probability is higher that bugs are mine.

When a shell script I wrote last week failed, I began carving it down to see what I'd done wrong. This is usually fast and easy--chop pieces out of the script until you can get a very simplified statement that doesn't do what I thought it should do, then go back to read the man page to see why.

Here's the simplified test case I ended up with:

set -x # this is required
PS4='$(true)$ ' # must be a command in the prompt, but any command will do

false || A=3 # first expression must fail, second must be a variable assignment
echo $? # Should be 0, but isn't. Odd.

Much to my surprise, it's bug in bash, both versions 3 and 4.

Obscure? Sure. You have to have a lot of special things going on, and the symptom is a bad exit code. I reported it and used a workaround (if .. fi instead of the shortcut logical or).

Always remember: every program has bugs. They're only almost always yours.

Hello, World Again

2009-10-03T08:23:00.001-06:00

I'd like to try saying "We sometimes take the shell too much for granted," another way.

Advice from masters is often good advice.

Here's my favorite chunk of the greatest of all programming texts, Kernighan and Ritchie's The C Programming Language (Prentice-Hall, 1978).

1.1 Getting Started

The only way to learn a new programming language is by writing programs in it. The first program to write is the same for all languages:

Print the words

hello, world

This is the basic hurdle; to leap over it you have to be able to create the program text somewhere, compile it successfully, load it, run it, and find out where your output went. With these mechanical details mastered, everything else is comparatively easy.

In C, the program to print "hello, world" is

#include

main()
{
printf("hello, world\n");
}

Just how to run this program depends on the system you are using. As a specific example, on the UNIX operating system you must create the program in a file whose name ends in ".c", such as hello.c, then compile it with the command

cc hello.c

If you haven't botched anything, such as omitting a character or misspelling something, the compilation will proceed silently, and make an executable file called a.out. Running that by the command

a.out

will produce

hello, world

as its output. On other systems, the rules will be different; check with a local expert.

Exercise 1-1. Run this program on your system. Experiment with leaving out parts of the program, to see what error messages you get.

Fine advice. Let's do exercise 1-1, but in the shell.

Run the analogous program? Okay.

$ echo hello, world
hello, world

Leave out parts? Let's leave out a part of the string.

$ echo hell, world
hell, world

Or part of the command.

$ eco hello, world
bash: eco: command not found

How about whitespace? It's okay to leave it out of the string,

$ echo hello,world
hello,world

but you need some between a command and its arguments.

$ echohello, world
bash: echohello,: command not found

My point: Programing in the shell is quick and easy. You just type. There's no editing, no special naming, no compiling, no a.out file, no loading and running, no need to consult a local expert.

If you type something incomprehensible, the shell gives you an error message and lets you try again, right away.

The Shell Enters a Beauty Contest

2009-10-01T08:31:00.003-06:00

I'd never tout the shell as the be-all and end-all of programming languages, but it gets less attention and respect than it deserves.

For example, folks will remark, casually, that shell syntax is ugly. Who would design a language that doesn't even let you put spaces around the '=' in an assignment?

$ x=3
$ y = 3
-bash: y: command not found
$ z= 5
-bash: 5: command not found

Eeew. Real programs are in C. Or Perl. Or Python. Or Haskell. Or ...

Yep, the shell syntax has some design flaws all right. But let's run another beauty contest.

First, contestant #1:

#include <unistd.h>
#include <stdlib.h>

int main(void)
{
int     fd[2], nbytes;
pid_t   pid;

pipe(fd);

if ((pid = fork()) == 0) {
  dup2(fd[0], 0);
  close(fd[1]);
  execlp("/bin/grep", "/bin/grep", "^z", NULL);
} else {
  dup2(fd[1], 1);
  close(fd[0]);
  execlp("/bin/ls", "/bin/ls", "-1", "/bin", NULL);
  wait(NULL);
  exit(0);
}

return(0);
}

Next, contestant #2:

ls /bin | grep ^z

And contestant #1 doesn't even have normal error checking, which would make it much longer, uglier, and hard-to-follow.

Programmers get so used to the shell that they focus on its flaws, but take its virtues for granted.

Don't it always seem to go that you don't know what you've got till it's gone? -- Joni Mitchell

What tastes of paradise does the shell offer besides pipes? I/O redirection. Ease of process creation. Multi-process programming. Parallelism. Command-line editing. For that matter, the entire idea of a CLI, a "command-line interface."

Once I start listing things, it's hard to stop.

All this in a language you can use in scripts, or just by doing nothing harder than typing at a prompt.

Grobycs: Cyborgs in Reverse

2009-09-30T06:54:00.004-06:00

We've read about cyborgs, computer-enhanced humans, for decades. Misdirection? We should have been thinking about grobycs: human-enhanced computers. Cyborgs in reverse.

Consider these:

Facebook offers crowdsourced translations as an alternative to Google Translate.
Google gives me real-time traffic data collected from folks using their GPS-enabled smartphones to look at maps: in fact, at traffic data.
I can send Amazon a barcode scan , using my phone's camera. If they don't stock the item, they let me send a picture, from the same camera, and nearly instantly, offer me similar products. How? Amazon Mechanical Turk.
Spammers break into captcha-guarded sites by running a porn site. Every time they hit a captcha, they pass it through to the porn-site registration side. The next human who visits the site decodes it.

All use people as a piece of a collection of software. They're cyborgs in reverse: Grobycs.

The Six-Million-Dollar Man wanted her:

What do you think grobycs will lust after?

Variables Without Values

2009-09-23T18:40:00.002-06:00

Often, you only want to do something if a variable is unset.

I see code to handle such conditions that looks like this.

if [ "A$foo" = "A" ]
then
take-some-action
fi

It's either old code or code by old programmers.

The test is how you used to have to ask whether a variable was empty. Here's a newer idiom.

[ "$foo" ] || take-some-action

The test operator, [ ], comes in many flavors. For example

[ -d $X ] asks "Does $X name a directory?"

In its simplest form, though,

[ $X ]

asks "Is $X empty?" The test returns true if $X has something in it, false if it doesn't. The new code does the same thing as the old, but it's shorter and, arguably, easier to read.

But why not this?

[ $foo ] || take-some-action

Ah. Because if foo='1 2 3', then test complains that you've given it too many arguments.

[: 2: unary operator expected

One more trick: what if you're running with "set -u", which complains whenever it stumbles on an unset variable, with complaints like this

line 3: foo: unbound variable

Write the test like this:

["${foo-}" ] || take some action

If $foo is unset, ${foo-} has a null value. "Null" is a value; "unset" really means the variable was never set. So, with "${foo-}" the shell won't complain about an unset variable, but the test will still fail.

This is still shorter than the example we started with, but it's only more readable once you can read the idiom ${foo-} without stumbling.

On the other hand, the first example will also choke if $foo is unset, so it would also have to be modified like this:

if [ "A${foo-}" = "A" ]
then
take-some-action
fi

More Include Guards

2009-09-23T12:51:00.004-06:00

I've already written about how to make include guards for "shell libraries" (files full of shell functions or variables to be sourced).

This is the basic form of the statement, which requires tailoring for each library:

if [ ${_gripe_version:-0} -gt 0 ]
then
return 0
else
_gripe_version=1
fi

Here's a more generic statement that you can put at the top of a file to achieve the same goal:

[ "${_libs/${BASH_SOURCE[0]}}" = "${_libs=}" ] ||
return 0 && _libs+=" ${BASH_SOURCE[0]}"

A dramatic reading of this off-putting code is left as an exercise to the reader.

It has the added attraction that you can, at any point, find out which libraries you've already sourced with echo $_libs .

It has the added detraction that you'll have to cut-and-paste it. There's no way you'll ever keep it in your head and type it correctly.

As a good test, first try this infinite recursion without include guards. (You'll have to ^C out quickly, or you'll get stuck in a source-a-thon.)

$ echo 'source foo.sh' > foo.sh
$ chmod +x foo.sh
$ ./foo.sh # don't wait long to ^C

Next, edit foo.sh to add an include guard at the top and re-run it. This time, it will return immediately, without help.

Simplifying Loopy Code

2009-09-18T15:04:00.003-06:00

The determined Real Programmer can write FORTRAN programs in any language.

-- Ed Post, Real Programmers Don't Use PASCAL.

I just read through a friend's shell script. He doesn't program in the shell much, but he's a superb C programmer, so his script has loops that look like this:

file[0]=foo
file[1]=bar
file[2]=mumble
suffix[0]=.c
suffix[1]=.h
suffix[2]=.txt

nfiles=${#file[@]}
nsuffixes=${#suffix[@]}

i=0while [ $i -lt $nfiles ]
do
j=0
while [ $j -lt $nsuffixes ]
do
process ${file[$i]}${suffix[$j]}
(( j = j + 1 ))
done
(( i = i + 1 ))
done

Yep. That'll work. But this will, too.

for filename in {foo,bar,mumble}.{c,h,txt}
do
process $filename
done

Bash sees filenames as strings, and most shell commands accept space-separated lists of filenames. The shell handles anything that expands to a list of filenames, like globs or brace expansions, with real ease.

Use the Shell, Luke.

Call Me Now

2009-09-16T08:36:00.003-06:00

Click on the Google Call Me button, below, fill in your own name and phone number, and then click Connect.

You'll be connected to me immediately.

But How Random Is It?

2009-09-05T09:58:00.003-06:00

Folks use /dev/urandom and $RANDOM to generate uniformly-distributed random numbers. Are they random? Yes.

Let's test. Here's how.

Surprise! The easiest way to start is to ask a more general question: Suppose you have a couple of sets of numbers. Are they from the same distribution?

For example, if you have a couple of batches of 10,000, 8-digit numbers, from the two different distributions, are they really spread out from 0 to 9999 the same way, or do they have clumps in different places?

There are ways to attack this, even if you don't know (or care) what the original distribution was. Here's a step-by-step walkthrough:

Sort them together, smallest to biggest, coloring the first set red and the second, green.

Put your finger on the middle of a line--zero on the number line--and begin reading off colors. If the number's red, move your finger left. If it's green, move it right.

See how far away from the origin you wander.

Half the numbers are red, half green, so you'll end up back where you started.

How far away might you get along the way? Depends on the distribution of the reds and greens. If all the reds are smaller than all the greens, you'll go left to ten thousand, then turn around and come right back. If they're all bigger, then you'll get ten thousand away to the right before you snap back like a yo-yo. And except for these cases, you won't go as far before you return.

If the numbers are from the same distribution, then whether they're red or green is a coin-toss. The distance I expect to get from the the origin goes up as the square-root of the size of my collections.

(That's on average -- any single pair of batches could end up almost anywhere. "... the 'one chance in a million' will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be ...." -- R. A. Fisher)

There are lots of ways to get this result:

from the standard deviation of a binomial with p=0.5
from a two-sample Kolmogorov-Smirnov statistic with equal sample sizes
from looking at it as Brownian motion, using the root-mean-square (RMS) argument attributed to Einstein, in Feynman, volume 1, chapter 6

All roads lead to Rome.

Note: nothing I've said, so far, depends on the numbers being random--this just tests whether the red and green numbers come from the same distribution. In what follows, though, I'm going to ask whether batches spit out by random-number generators are uniformly-distributed.

Let's write code. And, since all code roads lead to Rome, too, I'll do it in the shell.

Start with two, random-number-generators, R1 and R2 that generate some fixed-size batch of random numbers.

Check that each one spits out the same sized batch.

$ R1 | wc -l ; R2 | wc -l

Mark each output with a second field, saying which way each number will move my finger

$ R1 | awk '$2=1'; R2 | awk '$2=-1'

Sort the two together, on the first column

$ sort -n <(R1 | awk '$2=1') <(R2 | awk '$2=-1')

Run through the output, keeping track, at each step, of how far I've gotten away from the origin

$ sort -n <(R1 | awk '$2=1') <(R2 | awk '$2=-1') | awk 'pos+=$2; d = ( pos <>

Notice how I'm just recalling the last command and editing it. All this is on the command line, where I can watch the effect of each step.

Print just the the biggest distance I get from the origin, for the whole trip

$ sort -n <(R1 | awk '$2=1') <(R2 | awk '$2=-1') | awk 'pos+=$2; d = ( pos > 0 ) ? -pos : pos; print d' | sort -n | tail -1

Easy enough. I'll try it first with a script that uses the Halgorithm:

#!/bin/bash

head -10000 /dev/urandom | tr -dc 0-9 | perl -pe 's/(.{8})/\1\n/g' | head -${1:-10000}

Whatever argument I give it on the command line tells it how many eight-digit integers to generate. [Default: 10,000]

Two runs should produce different batches, but the same distribution. I bring back the earlier command and do half a dozen runs with 100 numbers from each batch.

$ for i in {1..6}; do sort -n <(R1 100 | awk '$2=1') <(R1 100 | awk '$2=-1') | awk 'pos+=$2; d = ( pos > 0 ) ? -pos : pos; print d' | sort -n | tail -1 ; done
13
20
6
7
10
10

The average? 10.6 .

(Careful! I'm comparing the batch produced by one run of the generator with the run produced by a different batch. It tells me what to expect for different batches from the same distribution.)

More runs, a hundred instead of a dozen, just gives a more accurate average: 12.2

If it varies with the square of the size of the batches, batches of 10,000 should give numbers nearer to 120.

$ for i in {1..6}; do sort -n <(R1 | awk '">=1') <(R1 | awk '">=-1') | awk 'pos+=">; d = ( pos > 0 ) ? -pos : pos; print d' | sort -n | tail -1 ; done
90
97
71
91
105
127

Good. The average value, over 100 paired runs, is about 125.

This, finally, lets me compare Hal's method with the shell-only method from an earlier post, which uses $RANDOM? Does $RANDOM have a different distribution from /dev/urandom?

Nope. The average distance is, again, about 125. If they're non-uniform, they're non-uniform in the same way.

But maybe neither is random. What if I compare them with a batch of uniformly-distributed, true random integers, downloaded from the web?

About 135. A little higher, but not high enough to set off a panic alarm. $RANDOM and /dev/urandom both give pseudo-random-numbers that are, roughly, uniformly distributed.

Another Random Post

2009-09-03T06:28:00.004-06:00

Hal Pomeranz suggests this one-liner

head /dev/urandom | tr -dc 0-9 | sed -r 's/(.{8})/\1\n/g'

for generating big batches of random numbers. It's a winner: reasonable performance and easy-to-type. Timing information follows:

#!/bin/bash

echo == Shell arithmetic
(( R = 2**15-1, T = 10**8-1, C = T/R ))

readonly R T C

time for i in {0..10000}

do
printf "%8.8u\n" $(( RANDOM*C + ( RANDOM%C ) ))

done > /dev/null

echo; echo == Pipeline

time head -10000 /dev/urandom | tr -dc 0-9 | sed -r 's/(.{8})/\1\n/g' | head -10000 > /dev/null

Here's the numbers:

== Shell arithmetic

real 0m0.262s
user 0m0.256s

sys 0m0.004s

== Pipeline

real 0m0.421s
user 0m0.028s

sys 0m0.392s

Okay, the pipe's a little slower, but it's the same order of magnitude, and way easier to type.

The International Slide Rule Museum

2009-09-02T16:45:00.000-06:00

From Mike Rosenlof:

They have (who am I kidding! HE has..) an exhibit at the Louisville Public Library.

http://sliderulemuseum.com/

Another Challenge From Hal & Ed: Random Numbers In the Shell

2009-09-01T17:37:00.008-06:00

Hal Pomeranz and Ed Skoudis have a weekly blog, Command Line Kung Fu, where they set and solve interesting problems in both the Unix/Linux shell and Windows's CLI, COMMAND.COM.

This week's was how to generate random 8-digit integers.
Obligatory pedantic sidebar, which you can skip:

"Random" can mean a lot of things. If I flip a coin, and announce either "00000000" or "00000001", that's an 8-digit, random number. Ho hum. What I'm looking for here is integers uniformly distributed in the interval 0000000..99999999. I'll write this as U(0,10^8-1).

The problem is trickier than you might think. For example, the shell's random-number generator, $RANDOM, is U(0,2^15-1). Not big enough.

How about, say, summing calls to $RANDOM until the number is big enough?

Nope. Adding a bunch of uniformly-distributed numbers gives a random number, all right. But it isn't uniformly distributed anymore. It's a clumped-up, bell curve.

Hal explores several routes, culminating with a good one-liner:

$ head /dev/urandom | tr -dc 0-9 | cut -c1-8

But is there a way with the shell's own built-ins?

Yep. The answer's below. What's more, because this way doesn't call outside utilities or fork subshells, it should, in theory, be faster. And it is.

How much faster?

On my machine, the one-liner above generates 10,000 random numbers in just under 40 seconds. The code below does the same 10,000 in about a quarter of a second.

I've written it out as a script, with a dramatic reading in the comments, but if I were using it in something else, I'd ditch the big comments and put the code in-line.

#!/bin/bash
## Generate a uniformly distributed,
## 8-digit, random number: U(0,99999999)

## First, the logic

# (1) Get a uniformly-distributed random number,
# N = $RANDOM, up to R = 2^15-1 = 32767.
# That's U(0,R)

# (2) Stretch out the interval covered,
# up to nearly T = 10^8-1
# by multiplying by C = K/R.
# (We'll figure out K in a minute)
# This turns 0,1, 2, 3, ... into 0, C*1, C*2, ... K

# (3) Now add a random shift S = U(0,C-1)
# to turn these into 0, 0+1, 0+2, ... 0+C-1,
# C*1+0, C*1+1, ..., K+0, K+1, ... K+C-1.
# We can get S from another random number M = $RANDOM % C
# (% is the 'mod' operator)

# What's K? We want K+C-1 = T = 10^8-1, so we solve:
# K+C-1 = K+(K/R)-1 = 10^8-1; K[ (R+1)/R ] = 10^8;
# K = (10^8)*R/(R+1)

# For big R, that's so close to T
# that I'll just use K=T, C=T/R
# Our U(0,10^8-1) number is N*C + M

## Next, the code

# calculate the constants
(( R = 2**15-1, T = 10**8-1, C = T/R ))
readonly R T C

# do the calculation, and print the result out
# with 8, full digits.
printf "%8.8u\n" $(( RANDOM*C + ( RANDOM%C ) ))

Nothing up my sleeve ... Presto! Not bad, for a shell.

Update: Hal helpfully points out several things.

I misspelled his name.

Oof. Sorry. Fixed. Thanks.

There may be even faster solutions.

if the task were to generate 10K 8-digit
random numbers, I'd just suck 80K digits out of /dev/urandom and
chop them up into 8-digit chunks. That would be considerably
faster than running my command line 10K times in a row.

Maybe so .... Is there a way to do that in the shell? It's thought-provoking and an interesting challenge.

You couldn't store the 80K digits in the code, but you could keep from having to store them in a file, and the attendant I/O slowdown, by just providing them as output from a pipe.

(I'm told that in Windows, pipes are implemented with intermediate files. Not so in Unix and its offspring--the system really does hand data directly from one process to another.)

So, is there an easy way, in a shell script, to pull 8 digits at a time out of standard in?

Execution efficiency isn't why you use the shell.

If the task is to generate a single 8-digit number, I claim my solution
is better from a typing perspective. :-)

Just so. Hal's exactly right.

Hash Collisions in Git

2009-08-28T15:53:00.002-06:00

This post would be easier if I knew how to post formulas in Blogger. If you know, tell me.

In the Pro-Git Book, Scott Chacon notes that to have a probability of a SHA1-hash collision rise to 1/2, you need about 10^24 objects (more, he says, than the number of sand grains on earth).

Where's this come from? It's a classic birthday problem. If we have K people, with birthdays on any of 365 days, what are the odds of two or more people having the same birthday?

P(same) = 1-P(everyone's different).

To assign different birthdays to K different people, we can give the first person any of 365 birthdays, the next any of the remaining 364, and so on. The Kth person has a choice of 365-K+1 days.

Turning this into a probability means dividing by 365^K -- the number of different ways to assign birthdays to K folks if you don't care about collisions.

But what's that product? Just multiply it out:

X = 365(365-1)(365-2) ... (365-K+1) = 365^K - [1+2+...+(365-K+1)]365^(K-1) + something,

Where something is O(365^(K-2)) (That's big-Oh notation, if it's not clear.)

Dividing by 365^K gives, approximately, P(no collision) = 1-[K(K-1)/2]/365,

or P(collision) = (1/365)K(K-1)/2 .

For SHA1 sums, we replace 365 by the number of 160-bit SHA1 values: 2^160.

How big does K have to be for (1/2^160)K(K-1)/2 to be about 1/2? About

K ~ sqrt(2^160) = 2^80, or about 10^24!

Another quick applications of this idea comes from the fact that the SHA1 prefix git is now requiring to specify unique kernel objects is 10 hex digits long. We can, therefore, guess the database now has something on the order of 2^20, or a million, distinct objects.

And then there's the original birthday party question. If you have sqrt(365) people in the room -- that's 17 or 18 -- you have about a 50-50 chance that two will have the same birthday.

This would, however, been clearer if I could have stuck real formulae in the post.

Using Git Stash as a Snapshotter

2009-08-28T08:42:00.003-06:00

I use git stash to snapshot my work.

I even use it to snapshot my home directory. I have a cron job that periodically does, basically, this:

git stash
git apply

to squirrel away any work I haven't checked in yet.

Basically.

Until today, it did exactly that, but a few days ago, I started having a Makefile change get undone. My old version kept coming back up like bad Mexican food. Hmm.

The bad (old) version was in my stash. What would happen was that git stash would run, figure out I had no recent changes, and do nothing. Then, git apply would run, and restore the last un-checked-in code in the stash (my bad Makefile).

Hah.

Now, the code says this:

git stash | grep -q 'No local changes to save' ||
git stash apply

Remember When?

2009-08-26T13:45:00.003-06:00

My uncle used to send "Remember when?" email to giant lists of people.

You've seen them from folks who are feeling old. "Remember when penny candy cost a penny?" ... things like that. Mostly, my answer was, "No."

I was walking back from lunch with David Russell, today, when a woman stopped us to ask for directions. I said, "I'd look on Google Maps, but I left my phone at home."

Then I thought about that for a second, and said, "Remember when we always left all our phones at home?"

Remember when people used to send email to giant mailing lists instead of blogging or Tweeting? You're probably answering, "No."

Are We There, Yet? Using fuser() to See If The Job's Done.

2009-08-26T13:10:00.002-06:00

Ever write a script that ends before the things in it should?

The symptom is a file that gets partially written or partially copied. Sometimes the output files are zero-length. One example is the batch-mode sftp I wrote about last post.

Here's an illustrative example, followed by a trick to solve it.

In the empty-sftp program, empty spawns an sftp session, which runs in parallel, waiting for me (really empty, but it thinks it's me) to type ftp commands. If I only want to run one command

sftp> get filename

I can exit my script after issuing the command, but the pty will still be running. I can kill the pty before I exit the script, but if the file transfer hasn't finished, the kill will truncate the output.

The basic problem is that there is no process that I can watch that terminates after the file transfer ends; like any shell (which is really what it is), sftp just keeps running, waiting for the next command.

The trick:

When the transfer's over, sftp closes the output file. We can watch to see when that happens, like this:

while true
do

sleep 1
/sbin/fuser $localfile || break

done

As soon as it finishes, we kill the sftp session, and exit.

Empty Turns "Hard" Into "Not-So-Hard"

2009-08-26T09:20:00.005-06:00

Some things should be personal, even when you're at a computer.

Logging in, for example.

Good, Linux, command-line tools read from standard in and write to standard out, which are connected, by default, to the keyboard and screen. I try out programs by typing at them. When I'm convinced I understand how they work, I can redirect I/O, to give them input from a file and capture their output in a second.

The upside-down is true, too: I can automate things I typically do interactively, just by having the programs do I/O to files instead of terminals. I've written automated test suites for screen editors with this trick.

Logging-in is different. When the login program prompts you for your name and password, it puts the prompts directly on the screen and reads the answer from the keyboard. Stdin and stdout aren't part of the picture.

This not only makes it hard to automate, it's supposed to make it hard to automate. It's hard to make robots to batter down the system's doors, trying user name after user name and password after password for days on end.

It's a security feature. When a program seems not to be taking input from stdin, I think, "Is this like logging in?"

So, how do you automate those un-automatable jobs?

Don Libes, at NIST, in Gaithersburg, Maryland, gave us the first, good solution: Expect. Expect creates a pseudo-terminal whose virtual keyboard and screen are FIFOs that you can write to and read from. Expect programs are mostly lists of the strings you expect to see, like login:, paired with the responses you want Expect to send when it sees those strings.

It's wonderful. I've used Expect for everything from testing printers to creating a make replacement that drives builds on Tandem's archaic Guardian OS from Unix machines running Expect scripts.

Unfortunately, Don picked the wrong horse. Don built Expect on Tcl, a popular programming language in the early nineties, and used Tcl's syntax. Today, Tcl is no longer in fashion; for most folks, writing Expect is like writing Latin.

Nowadays, Perl, Java, and other languages, have Expect-like extensions, too, but they all feel fragile and hard to use.

Yesterday, I needed to automate an sftp session, without any way to install public keys on the target machine. I thought about Tcl, winced, then looked at empty, which is supposed to offer the same magic in shell scripts.

Installation was just this:

$ sudo apt-get install empty-expect

The program I wrote was short, simple, and it worked. The meat was these lines, trivial modifications of examples in the man page:

# set up a session
empty -f -i in.fifo -o out.fifo -p empty.pid -L $log sftp $username@$remote_machine
empty -w -i out.fifo -o in.fifo password: $password'\n'
# transfer the file
empty -w -i out.fifo -o in.fifo sftp "get $remotefile $localfile"'\n'

For things that should be personal but can't be, I recommend empty. At least, on your computer.

How Do I Find Missing Files in a Sequence?

2009-08-20T22:16:00.002-06:00

In this week's Command-Line Kung Fu, Hal Pomerantz & Ed Skoudis field this question from a reader:
I have about 1300 pictures that I am trying to organize.
They are numbered sequentially eg. ( 0001.jpg -> 1300.jpg )
The problem is that I seem to be missing some...

Is there a fast way to be able to scan the directory to see which ones I am missing? Other than to do it manually, which would take a long time.
While they provide good, loop-based solutions, like this one:

$ for i in $(seq -w 1 1300); do [ ! -f $i.jpg ] && echo $i.jpg; done

and this

for ((i=1; $i <= 1300; i++)); do file=$(printf "%04d.jpg" $i);[ ! -f $file ] && echo $file; done

they don't mention bash brace expansion. In bash 4.0 and above, this does the trick:

$ ls {0001..1300}.jpg >/dev/null

No loop. No subprocess. Quite readable.

Include Guards in Sourced Files

2009-08-20T15:13:00.002-06:00

How do I keep from sourcing the same files over and over?

My shell scripts often have lines like this:

source gripe.sh

Unfortunately, so do other sourced files. How do I keep from sourcing something more than once?

In C programs, my include files handle this with constructs like this:

// foo.h -- include file for foo definitions
#ifndef _FOO_H
#define _FOO_H
...
#endif

The first time foo.h is included, it grabs all the included information. If it's accidentally included a second time, though, _FOO_H is already defined, so the file is skipped.

Here's an analogous construct that I use at the top of shell "include" files.

# include guard
if [ ${_gripe_version:-0} -gt 0 ]
then
return 0
else
_gripe_version=1
fi
... # remainder of file

I could put the file contents inside the else block, and the fi at the end of the file, but I like just clumping the whole thing at the top.

Another Use For The Colon (:) Command

2009-08-10T10:09:00.002-06:00

I've already blogged about the shell's no-op command, when I use it, and why.

I forgot about another way I use it. Here 'tis.

Consider this loop:

for i in {1..100}
do
some-command-or-other $i
done

If you want to comment out the loop, you could comment out the whole thing by putting a comment character in front of each line, but that's tedious. Easier would be to comment out the one-line loop body, like this

for i in {1..100}
do
# some-command-or-other $i
done

Except, um, that's an empty loop body and a syntax error.

bash: syntax error near unexpected token `done'

This, however, is fine:

for i in {1..100}
do
: some-command-or-other $i
done

That loop body isn't empty, it just doesn't do anything. No fuss, no muss.

Amazondroid

2009-08-07T12:46:00.000-06:00

I have seen the future of retail and it is my phone.

Today, Amazon announced an Amazon Market app for the Android. (There's one for the iPhone, too.)

You point the camera at the barcode of an item and push a button. It scans the barcode, then looks to see whether Amazon stocks the item. If so, it puts the item onto your wish list, so you can buy it (or so someone else can buy it for you).

What would you pay? But wait. There's more.

If Amazon doesn't have your item, they offer another option. Select the other button, and your camera takes a picture of the item, sends it up to Amazon, and a minute or less later, Amazon tells you *similar* items that they stock. And puts them on your wish list.

How do they do this? Is it OCR? Is it image recognition? Is it Amazon Mechanical Turk? I have no idea.

And, frankly, who cares?

They make it easy to buy. From them. This is brilliant. Better even than the browser-based "Buy this together with ...." Better even than "You might also be interested in." Better even than "People who bought this item also bought ...."

I have seen the future of retail and it is my phone.

Linux Google Chrome, the Alpha Version

2009-07-27T16:42:00.005-06:00

I've been trying out the Linux alpha of Google Chrome. I started on Monday, June 8 09:40:37 MDT 2009 with version 3.0.183.1 . A month and a half later, it's at 3.0.195.1 . I haven't had any crashes. That's not bad.

I'm trying to switch to it, not because I'm unhappy with Firefox, but because it's the only way I ever learn new stuff -- by forcing myself to use it.

What's this Colon Doing in My Comments?

2009-07-26T13:19:00.005-06:00

Shell comments start with '#' -- except when they start with ':'.

Before 1981, they all started with ':', and they weren't really comments, they were commands. They still are.

$ type :
: is a shell builtin

':' is a no-op. Like true, it's a command that always succeeds, and does nothing.

Since it doesn't do anything, you can give it whatever arguments you want.

: This is a command that does nothing. And, a comment.

With one wrinkle: since it's a command, the shell processes its arguments. Side-effects? You bet.

Here's a surprisingly common idiom in shell scripts. (Look through the /etc/ directory for these.)

: ${FOO:=bar}

Huh? Okay, step at a time:

$FOO is the value of the variable FOO.
${FOO:-bar} has the value $FOO, unless FOO doesn't have a value. If FOO is unset or null, then the value of this typographic mess is the string, "bar".
${FOO:=bar} is like that, but it also assigns FOO the value bar as a side-effect if it didn't already have a value.
So this command

: ${FOO:=bar}

sets FOO to the default value "bar", if it hasn't already been set to something else. And then, it goes on to the next line in the script, because the command itself is a no-op.

The entire purpose of that null command is the side-effect produced when the shell evaluates its arguments -- it announces a default value for $FOO.

More verbose ways to say this include these:

$ FOO=${FOO:-bar}

$ [ "$FOO" ] || FOO=bar

or even

$ if [ "$FOO" = "" ]; then FOO=bar; fi

But if you want to understand the scripts in the /etc/ directory, you need to at least recognize:

${FOO:=bar}

and not be intimidated by it.

Plus, knowing this lets you make sense of shell scripts that are more than 29 years old. Since this is the 40th anniversary of Unix, that seems fitting.

File Access Times as Debugging Tools

2009-07-25T08:09:00.003-06:00

"Is that configuration file actually being read? I'm sure I changed it. Maybe I'm just not reading the data file I think I am."

"Is my command actually being executed? Is my initialization script getting executed first?"

I use the Linux "file access time" as a debugging tool.

If you want to know the last time a file was written, ls -l tells you the time. You knew that.

If you want to know the last time a file was read (or was run, if it's an executable), ls -ul tells you the time. Maybe you didn't know that.

Or the last time a particular app is getting invoked. The access time tells you when an executable was run because Linux has to read it to execute it.

Or, at least, it used to work this way. Newer Linux systems no longer do, but you can still get the information you need. I'll show you how.

On newer systems, the access time (the time ls -ul reports) is the last time a file was read after the last time it changed. You can get the same information, it just takes a little more setup work. Here's a script that shows how:

#!/bin/bash -x

PS4='$(date +%H:%M) $ '
: == Create a file at a well-defined time.
rm -f /tmp/foo
touch /tmp/foo
: == File write and access times are the same.
ls -l /tmp/foo; ls -ul /tmp/foo
: == Sleep, so the time changes, and read the file
sleep 60; cat /tmp/foo
: == The access time is now later than the write time.
ls -l /tmp/foo; ls -ul /tmp/foo
: == Do it again.
sleep 60; cat /tmp/foo
: == The access time DOES NOT change, because There was no intervening write.
ls -l /tmp/foo; ls -ul /tmp/foo
: == Now "write" the file.
touch /tmp/foo
: == Sleep so the time changes, and read it.
sleep 60; cat /tmp/foo
: == This time, the time does change.
ls -l /tmp/foo; ls -ul /tmp/foo

The meaning of access time was changed as an efficiency hack, so that inodes aren't re-written as often.

In the script, I'm using ':' as a comment character. I should explain why, but I'll leave that as a separate post. Or, you could run the script to see. :-)

Using Preferred Applications from the CLI

2009-07-02T11:38:00.003-06:00

I can use a command-line interface to invoke (not just "set," but "invoke") my GUI preferences.

System-> Preferences -> Preferred Applications lets me set things like my preferred email application and preferred browser. Works fine in the GUI, but how about from the command line?

The solution is XDG tools, from the Portland Project. Browse a url or edit a file?

xdg-open https://google.com
xdg-open /tmp/foo.txt

Email?

xdg-email

To Be Continued

2009-06-26T13:04:00.002-06:00

I hate backslashes. They make code harder to read.

In these four code snippets, I like the first two, but not the last two:

sleep 10 & date

sleep 10 &
date

sleep 10 & \
date

sleep 10 \
& date

So? So, for the same reason, I would re-write:

grep :0: /etc/passwd | \
wc -l

grep :0: /etc/passwd |
wc -l

and

grep -c jsh /etc/passwd \
|| echo jsh not found

grep jsh /etc/passwd ||
echo jsh not found

The symbols ', ", `, &, |, &&, and ||, at the ends of lines, say "I'm not done," and automatically continue to the next line (or, in the case of the quote, until the closing quote).

For shell symbols with parens, like $( and <(, the continuation is until the matching, closing paren.

Printing Strings in Shell Scripts

2009-06-25T13:08:00.002-06:00

echo's nice. printf is nicer.

I still see people use echo. Heck, I still use echo.

But POSIX shells have had printf as a built-in for twenty years. The syntax is like C's printf().

You don't always need fancy formatting, so echo's still good for simple strings, like this:

echo 'What are you -- nuts?'

but there's no reason to use echo -n when you can use printf. None.

Just the reverse. Experiments like this:

time for i in {1..10000}; do printf $i; done > /dev/null
time for i in {1..10000}; do echo -n $i; done > /dev/null

will show you that printf is about 10% faster.

Similarly, you can discard echo -e "foo\tbar" for printf "foo\tbar\n" . Why keep a bunch of flags to echo in your head if you're comfy with printf from other languages?

Truncating a File

2009-06-24T16:57:00.002-06:00

You can make truncating a file harder than it needs to be.

I'm walking through Hal Pomeranz's older Command-Line Kung Fu postings, both because I know there'll be useful stuff and because he has some fun challenges.

His Episode #19 attacks zeroing out an existing file. The solutions suggested are

cat /dev/null > my_file

and

cp /dev/null my_file

(which is both shorter and works even if noclobber is set, to prevent accidentally overwriting pre-existing files with > ).

Both of these are more typing than necessary. This works just as well:

> my_file

and never even invokes a command. It's all done by the shell.

What's that? You're running with set -o noclobber ?

>| my_file

If you want to do it in fewer characters, you can omit the space. :-)

Cut! It's a wrap.

TIMTOWTDI: There Is More Than One Way To Do It

2009-06-23T13:41:00.003-06:00

My favorite programming language is still the shell.

Hal Pomeranz, whom I finally met at the OS Bridge conference in Portland last week, has a blog called Command-line Kung Fu full of shell tricks and tips. He posts to it on Tuesdays.

In today's, a post on how to decide what tools to use and when, he recommends cut(1) for some tasks because "cut is the only tool that allows you to easily pull out ranges of characters."

Ah. A challenge.

Well, you can do his example:

grep -l spammer@example.com qf* | cut -c3- | xargs -I'{}' rm qf{} df{}

without cut at all.

grep -l spammer@example.com qf* | while read i; do echo ${i:2}; done | ...

I confess I'd do it differently, but that's because I like substitution better than cuts.

for f in $(grep -l spammer@example.com qf*); do rm $f ${f/q/d}; done

It's pronounced "Tim-toady." Hal emailed me saying he still likes his way better. YMMV.

For jobs like this, where I'm using a program to write commands on-the-fly, I usually play it safe. I start off using echo to see what the shell thinks I want to do

for f in $(grep -l spammer@example.com qf*); do echo rm $f ${f/q/d}; done

which spits out a list of the generated commands.

rm qf137965 df137965
rm qf200920 df200920
...

Then, when I think it's right, instead of removing the echo -- which risks a typo -- I recall the command, and tack on a pipe to a subshell, which executes the commands.

for f in $(grep -l spammer@example.com qf*); do echo rm $f ${f/q/d}; done | bash

Twitter, Again

2009-06-22T19:25:00.002-06:00

I'm giving Twitter another shot.

I tried Twitter a while back and couldn't get excited about it. I didn't know why I'd use it, or how.

I just got back from Open Source Bridge 2009, in Portland, OR, where it was all the rage. (The other things most attendees were into were Ruby and git.)

"Okay," I said, "let me find someone who gets it, and I'll just ask how to approach it."

Ms. Amye was willing to be my evangelista, so I logged in, set up my account, and am now following a few folks that I met at the conferences (#osb09 is the Twitter "hash code"). I've even tweeted.

I'm @goyishekop.

Beeees! And Kino to the rescue.

2009-06-15T05:46:00.003-06:00

This weekend, I put my first videos up on YouTube. Here's why.

Three Saturdays ago, May 30, Kristina and I were in King Soopers when my cell rang. It was Kristine Smock, a mandolin player from Lyons.

There's a swarm of bees in the alley behind my house. Do you want them?

We hurried home. Kristina grabbed a cardboard filing box, shoved in a frame with old comb, tossed it in the car, and we drove out to Lyons. After she'd gotten the swarm in the box, she put on the lid and duct-taped both handles shut. Once home, she put it on the deck, pulled off one piece of tape and -- presto! -- one new hive.

I'd gotten her an observation hive for Christmas, which was still unpopulated. By last Wednesday, it was set to go and she'd drilled a hole in the wall.

A flip videorecorder was waiting for me on the porch from Amazon, so its maiden voyage was taking a video of the installation.

Okay, I had it captured. My first, little, 15' home movie. Now what? It'd be nice, for example, to show it to Kristina's parents.

Can't mail it, it's half a gig. YouTube? Uh-oh. YouTube has a 10' limit. Other services? Turns out there's a reason YouTube is popular -- they're slow and hard to use.

By the next night, I could get someone at the BLUG meeting to show me enough rudimentary Kino to chop my video and post the pieces to YouTube.

The trickiest part was figuring out to download the jmpegtools package, so that Kino would export the files in a format that YouTube liked. Figuring it out required actually reading the error messages on the screen, which told me to install the jmpegtools package. Duh.

Here's the result.

And here's the first one, in embedded form:

I finally learn how to do realpath

2009-06-15T05:19:00.002-06:00

Kevin Fenzi said I should send my shell-level realpath off to the maintainer of an appropriate package -- coreutils, maybe -- for possible inclusion.

So, I wrote unit tests, a simple Makefile and documentation: man page, README, INSTALL, stuff like that.

Kevin, wiser than I, looked and found documentation that suggested readlink -f would do the trick. And it does. I could prove it to myself because I had unit tests.

One difference in functionality, which I can live with: if the file is missing, readlink -f succeeds and reports it's in the current directory.

So, to find the path to an executable, this does the trick:

path2 () {
local _cmd=$(type -p $1)
[ -f $_cmd ] && readlink -f $_cmd
}

iPods and Banshee

2009-06-09T05:36:00.000-06:00

Scott Mann posted a note to the Boulder Linux Users' Group Mailing list, asking for advice about iPods, MP3 players, and what software to use to manage them under Linux.

As you'd expect, much of the advice was hyperbolic. "Don't get an iPod. They're unuseable." Quick, someone tell Steve Jobs.

I replied with my recent experience.

A few months ago, I switched my girlfriend, Kristina, over to Linux; her old, G3, iBook was finally biting the dust. She's been completely happy with Ubuntu.

Last week, she bought an iPod Nano on eBay.

She's going to fill it with
her Dad's favorite music and give it to him.

When it arrived, she took it out of the box, turned to me, and said, "How do I put music on this?"

I took Scott Mann's advice and installed Banshee. It mostly just worked. It rips CDs to ogg by default, but I kept throwing gstreamer packages at it until it said it knew how to create mp3s, too.

To my mild surprise, it now also converts flac, ogg, wmv, and wv files to mp3s, on the fly, when I drag them onto the iPod. I found this out by accident. I mean, uh, by carefully reading all the documentation.

Still, after reviewing this thread, I started to wonder whether, instead, I should have traded Kristina in for a smarter girlfriend -- one who'd have bought an mp3 player with a much better interface and then re-flashed it with Linux or RockBox herself.

When I asked her if she thought so, too, she ignored the question.

Maybe she's smart enough.

What's the Absolute Path to a File?

2009-06-02T13:24:00.010-06:00

There's no standard command that gives you the absolute path to a file. That's insane. Here's an implementation.

Yes, it resolves symlinks and works on directories, too.

I couldn't figure out how to get Blogger to stop re-formatting my code, so I passed it through this, web-based, C#-formatting tool (!), and now it seems to look okay.

Argh.


// realpath - what's the real path to a file?
// Turn relative paths into absolute paths,
//   resolve '.' and '..',
//   follow symlinks

#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
  char path[_POSIX_PATH_MAX];

  if (argc != 2) {
    fprintf(stderr, "usage: %s filename\n", argv[0]);
    exit(EINVAL);
  }

  if (!realpath(argv[1], path)) {
    perror(NULL);
    exit(errno);
  }

  printf("%s\n", path);
  return(0);
}

Google Runs Rings Around M$ This Week

2009-05-29T11:35:00.003-06:00

Google seems to be in fine form this week.

Wednesday, they gave away Android phones to everyone at the Google I/O conference.
Yesterday, they announced Google Wave.
Today they have up the full video of the Google Wave talk. And the product will be Open Source.

And Microsoft? Their big announcement, yesterday, was bing, a new search engine to compete with Google. Which, at this point, even if it eventually works and comes out, is soooo 20th-century. Sorry.

It'll be interesting to see how Google ties this to ads.

Editing Parallel Trees

2009-05-28T08:11:00.003-06:00

Sometimes, I need to work in parallel trees -- trees with the same structure, but some important difference (different revision numbers, different compiler flags, ...). When I'm deep inside one tree, getting over to the homologous place in the other can be painful. Not only is it a lot of typing, but I'm often made aware that I should work at "Typos R Us."

The string substitution syntax ${FOO/bar/mumble}, which returns $FOO, but with mumble substituted for bar, solves my problem. For example, if I'm in the tree /home/jsh/tree1/a/b/c/d, and I want to pop over to /home/jsh/tree2/a/b/c/d, I just say

$ pushd ${PWD/tree1/tree2}

or even

$ cd ${PWD/1/2}

Presto! I'm there faster than you can say, "xyzzy." Or, at least as fast.

Not only am I in the new tree, but now pushd, with no arguments, (or cd - ) takes me back and forth.

How to Change File Extensions

2009-05-27T14:04:00.003-06:00

Changing file extensions is easier than people make it out to be. Even on sites that provide shell programming tips, I see recipes like this:

#!/bin/sh

for file in *.dat ;

do mv $file `echo $file | sed 's/\(.*\.\)dat/\1txt/'` ;

done

It's easier than that. Honest.

$ for file in *.dat; do mv $file ${file/.dat/.txt}; done

All in the same, parent shell. One process. No pipes, no sed, no magic regex metacharacters. So easy, you can type it on the command line.

I do it all the time.

If I need to be careful, I start like this:

      $ for file in *.dat; do echo mv $file ${file/.dat/.txt}; done

Then, if the output commands look like what I intended, I recall the loop, and tack a pipe of its output to bash on the end, like this:

      $ for file in *.dat; do echo mv $file ${file/.dat/.txt}; done | bash

Shell Programming Texts

2009-05-12T17:14:00.002-06:00

There are no brief but substantial shell-programming texts.

There are two, alternative models for programming textbooks: K&R, and almost any Java book you pick up. The Java books are boat anchors. They're comprehensive, verbose, detailed, and replete with extensive terminology. K&R is brief, to-the-point.

Shell-programming books seem to follow the Java model. What would a K&R-style shell book look like?

I can't write as well as Brian Kernighan or Dennis Ritchie, but let's take a shot at a few pieces.

Beginnings.

Unlike languages like Java or C, talking to the shell is programming in the shell. Just open a terminal window and type:

$ echo hello, world
hello, world

(Please try all the examples, even if you think you understand them. It's easy enough, so why not?)

You could type this a few times, for practice, but it's easier to recall the command than to retype it. Press the up-arrow once and you'll see the last command reappear. Press enter and the shell will execute the command.

$ ^
$ echo hello, world

The dollar sign is the prompt. It says the shell is ready to listen to your next command. Until you see the dollar sign, it's not. You can type while waiting for a prompt, and the terminal will buffer up your typing to give to the shell when it's finally done with what it's doing and ready for your next command.

A command will normally only be executed after you press ENTER, whether you've typed it or are just recalling it.

$ sleep 10
echo hello, world
[ten seconds elapses, then the next line appears, along with the execution of your command.]
$ echo hello, world
hello

You can press the up arrow a few times to look back through your history, or you can just type the command history.

$ history
1 echo hello, world
2 echo hello, world
3 sleep 10
4 echo hello, world

(Now that you've typed it, the command history is in your history.

$ history
1 echo hello, world
2 echo hello, world
3 sleep 10
4 echo hello, world
5 history

)

You can refer to and re-execute a command by its history number, but we'd like to discourage that. There are more powerful ways to use history, and you should build the habit of using them from the start.

Type control-R (^R), then, slowly, "hel" and watch what happens.

(reverse-i-search)`':
(reverse-i-search)`h': history
(reverse-i-search)`he': echo hello, world
(reverse-i-search)`hel': echo hello, world

The shell is looking, in reverse, through your history to find a command that matches the string you're giving it. This is incremental search. 'h' matched "history", so that's what it found first. As soon as you typed the 'e' , it looked further and found "echo hello, world" The "l" still matched, so it didn't look for a new line.

As soon as you've found the history line you want, press ENTER to execute it.

Exercise 1-1: Search for a command that isn't in your history.

If you find yourself marooned on a line that you do not want to execute, just press ^C to cancel it and get back to a prompt. Try this both while you're typing and while you're searching through your history. Notice, by examining your history when you do this, that canceled commands don't go into your history. Only executed commands are recorded.

Mounting USB Drives Under Ubuntu

2009-05-08T09:07:00.003-06:00

I backup daily to a USB drive, at work with git push. But the last couple of days, I've been home, sick.

I called in to ask Mike to stick in my USB drive, and he did, but it didn't automount. The automounting seems to be a Nautilus thing, and requires being logged in at the console. (I lock my screen when I leave at night.)

Humph.

I made him a login, so he could get in, and that got us farther, but when it mounted, it was owned by him.

I tried mount -o remount,uid=1000, but that didn't work. The mount command showed two uid= arguments -- the first, his. (If I did it again, there were three. The first one was still his, so he still was owner.) Okay.

I jotted down the name of the device, /dev/sd1, unmounted it, and remounted it by hand, as me. Ta-da.

Helping My Girlfriend with Google Docs Templates

2009-05-05T14:28:00.002-06:00

I was up past midnight beating out a first-cut resume for my girlfriend.

She's getting laid off, so she needs a new job. Her resume, built from an academic CV, is so big and impenetrable it needs to include bookmarks.

Sometimes, a job's so big that it's hard to start it yourself, so I told her I'd do it. I reasoned I could get it close enough that she could finish it up.

I started with Google Docs, which now has templates -- including, it turns out, resume templates.

A few hours playing, cutting, and pasting, and she has a first-cut that looks reasonable.

If you know someone who wants to hire a biologist, don't hesitate to post a comment.

Fixing the Default ISO App in Ubuntu Jaunty

2009-05-04T08:20:00.003-06:00

I just changed the default application for handling .iso files under Ubuntu Jaunty Jackalope back to what it was under Intrepid Ibex. With clicky stuff.

I've been making a deliberate effort to become more comfy with GUI tools. I've been a command-line guy forever, which works fine for me, but not so well if I'm trying to help my girlfriend, or other folks who are more used to GUI, user-friendly, point-and-click interfaces.

Every Monday morning, I burn a pair of ISO images to back some stuff up to DVDs. My scripts put them onto my desktop, and I just double-click on the Destktop icons, which launches a disk-burner. (See? GUI!).

Except that this morning, I double-clicked on an iso-image icon and ... Whoops! A behavior change. It mounted the ISO.

Can I fix this by clicking? (Not "Can one?" but "Can I?")

Yes.

Right click on the ISO to get a context menu. Click on "Properties," down at the bottom, then on the "Open With" tab. Sure enough, it's now set to Archive Mounter. Switch this to "Disc Burner." (And how did "disk" become "disc"? Is this a South African thing?)

Ta-da! The default for ISOs now is to burn them to disk, which is usually what I do with them.

I'm comfier with clicking all the time, which means I'm a better support guy for my friends.

Setting envelope-from on Ubuntu

2009-05-01T09:18:00.002-06:00

After my Jaunty Jackalope upgrade, at work, some of my command-line mail tests were bouncing. Why? My sending address didn't have a Fully-Qualified Domain Name (FQDN). Some sites bounce mail that doesn't have a valid domain name in the field.

Sure enough, dnsdomainname returned ... nothing. And uname -a gave me a machine name, but not an FQDN.

I use Gmail for almost all my routine email, at home and at work. Gmail doesn't complain.

I also use command-line email in shell scripts, but that stays inside my LAN, which doesn't care about FQDNs either. I could probably have lived with the problem, but I wanted it fixed so I started poking.

I set the domain in /etc/hosts. That fixed the output of dnsdomainname,
and the From: field in the email was set correctly.

But not the mail problem. Envelope-from was still wrong. The mail to my test, outside hosts still bounced with the same complaint.

I restored the old hosts file and got Dean, our sysadmin, to give my machine a domain name through DNS. That was a little more work but it worked ... and also didn't solve the mail problem.

Finally, I stumbled over /etc/mailname. Putting an FQDN in there does the trick. The man page provides details.

Cat Skinning, Made Easy

2009-04-16T13:20:00.005-06:00

[Click on the story, above, to enlarge it.]

My friend, Debi, wants to put a newsletter article about herself on her LinkedIn page. All she has is the pdf for the entire magazine issue. She doesn't even have a PDF writer on her Windows box.

Impossible?

Piece of cake.

She mails me the pdf.
I use Ubuntu's Document Viewer to bring up the PDF.
I print page 2 to a file, as PDF.
I use convert(1) to turn that pdf into a png.
I import the png into my blog.
We verify that she can see it, and that it suits her needs.
We use Blogger to create her a blog, put the same png into her own blog entry, and point to it on her LinkedIn page.

Google Voice

2009-03-19T15:52:00.002-06:00

Well, okey doke. Google's finally bringing GrandCentral back as Google Voice.

Let's see if their new call button works.

Grand Central

2009-01-11T21:39:00.008-07:00

I'm not posting regularly, but once in a while something is a must-post. This is one.

In the Marine Corps I was a radio operator, which required acquiring the skill "push to talk." Finally, this reflex returns in a way I wouldn't have predicted.

Jeff Bezos has a call-me button? Me, too.

Kevin Kempter

2008-08-15T06:35:00.005-06:00

I met Kevin Kempter at a CLUE Installfest.

Kevin was hauling around gear in a great, Marine Corps bag. I thought, "Hah! Another jarhead. Hey ... how can I get one of those bags?"

He'd had the same thought. On a visit to MCRD, he saw a guard with the same bag, and asked him where he could get one.

"They're only for active-duty Marines, so I'm not allowed to tell you that if you go down that way one block and over two, you'll find the PX. Sir."

"Interested in giving a BLUG talk?" I asked Kevin.

"Sure."

"What would you talk about?"

"Postgres."

Turns out, people are interested in Postgres.

Kevin played to a standing-room-only crowd last night.

He told us he has one client who's now pulling in a terabyte of data a day, and expects to expand this by a factor of 100.

Apache Status

2008-08-14T05:20:00.002-06:00

You can get your apache2 (and, I suspect, apache) server to display status information on prefab server-status and server-info pages by loading the status and info modules and tweaking the configuration files to pay attention to them.

Having done that, http://localhost/server-info and http://localhost/server-status
will tell you what modules are loaded and what they do. With little more work, you can also permit access to these pages by other boxes.

I set this up yesterday at work, but haven't done it at home because this laptop isn't serving up web pages at all.

Gitosis

2008-08-11T06:26:00.002-06:00

I spent part of the weekend playing with gitosis, a tool for administering remotely-accessible git repositories.

Access revolves around ssh and public/private key pairs. You give me your public key, and I can grant you selective permissions, repo-by-repo, if I'm the gitosis administrator. Once I do, it's painless to check stuff in and out, since all the permissions stuff is under-the-covers. Much nicer than Subversion's cacheing of passwords in plain text.

One of the coolest things is that the administrative data, including public keys, are kept in a git database. If I'm the (or an) administrator, I can administer access remotely, with git pull and git push -- including adding new users and their keys -- using gitosis itself for the repo access.

Error-handling, again.

2008-08-10T10:18:00.002-06:00

Here's a better version of my error-handling shell functions. Maybe I am educable, after all.

# common error-handling functions

if [ $_gripe ]; then return 0; else _gripe=1; fi # include guard

warn() { printf "$* at line %s file %s: $msg\n" $(caller) 1>&2 ; }
die() { printf "$* at line %s file %s: $msg\n" $(caller) 1>&2 ; exit -1; }

I have an include guard so I can include it in other shell functions that may be included with one another. I use if, rather than a [test] || alternative construct because I'm often running set -e, and don't want to die because of the test itself.

I have die() printing its own message, instead of calling warn(), so that it reports where it's invoked. Otherwise, warn() announces it's being called by die(). Duh.

It amazes me that after doing this for so long, I can still be fixing bugs in my own code -- and code as basic as this.

Making Makefiles Behave: Loops

2008-08-09T05:08:00.003-06:00

The last command in a shell for loop determines whether the loop succeeds or fails.

This succeeds:

$ for i in false true; do echo $i; $i; done
false
true
$ echo $?
0

This fails

$ for i in true false; do echo $i; $i; done
true
false
$ echo $?
1

When does this matter? In Makefiles. Suppose you have a make rule that looks like this:

foo:
for dir in $(SUBDIRS); do $(MAKE) -C $$dir; done

A failure of one of the subdirectory makes will not cause the make to fail!

How do you solve this? There are several sophisticated ways, but the dead-easy way is to start the Makefile like this:

SHELL := /bin/bash -e

This will cause all shell commands to be executed with the -e flag, which causes any simple command to terminate the shell.

$ ( set -e; for i in false true; do echo $i; $i; done )
false
$ echo $?
1

(I have to put the command and the set -e in parens here -- though not in the Makefile -- because any failure causes the command to fail and the shell to exit, so the following "echo $?" would never be reached.

CruiseControl

2008-08-05T06:29:00.002-06:00

For fun, I've set up CruiseControl on my laptop.

It's annoyingly hard to configure, and there seems not to be an rpm for Ubuntu, but it works.

I have it set up to watch a git repository, and build "hello, world." I can push to the repository from elsewhere and trigger a new build.

Ta-da!

Google Chat-Bots for Translation

2008-08-03T05:10:00.004-06:00

I use the Google chat-bots for quick translations. Here's an illustration from a note to a friend:

For whatever reason, I looked at the story you gave me, saw "laberintos," and thought, "Shouldn't that be 'labArintos'?"

So I popped up a chat window in gmail, invited < en2es@bot.talk.google.com> to chat, and typed in "labyrinth." It responded,
"Laberinto."

I said, "thank you," to which it replied "Gracias." :-)

Now that Google has group chat, you can invite a pair of simultaneous translators to join, and have chats with people you don't even share a language with. Mostly, though, I use them for quick translation: it's easier than going to a website.

Update: I just looked, and Google has added many more translation-bots.

Sort-of Techical Problems, Sort-of Technical Solutions

2008-07-28T19:12:00.003-06:00

I subscribe to the Boulder Linux Users' Group mailing list. It's my local LUG, it carries announcements of local events, and the posts are mostly technical.

Plus, when you know folks, it's fun to see what they think and how they write. I read the stuff Sean Reifschneider posts even when I have no clue what he's talking about.

The opposite's true, too. I started going to Dave Anselmi's Installfests just to meet him and Collins Ritchie. I'd been so impressed by their steady, helpful contributions to the mailing list that I wanted to see what they looked like.

Oh, sure, there's an occasional call to boycott a large company because they buy closed-source software.

In some cases, these move me to action. As soon as I learned Microsoft uses Microsoft products in-house, I stopped buying stuff from Microsoft.

In other cases, not so much. King Soops? I still shop there.

And, once in a while, some poster decides it's important we all read about their far-left political views. But what do you expect from the People's Republic of Boulder Linux Users' Group?

Mostly, though, the signal-to-noise ratio is high.

Just now, however, we've had a couple of back-to-back flamefests running that swamp out the parts I find useful.

At first, I thought, "Oh well. If this is going to turn into Slashdot, I'll unsubscribe."

Then I realized that I have a technical solution at hand. Gmail lets me, with a single keypress ('m'), mute threads I'm not interested in: they no longer show up in my mailbox. As long as no one hijacks the thread, by changing the subject line, I can go back to reading the stuff I'm interested in.

The thread, meanwhile, continues to go into my archives, so if I suddenly become passionate and want to flame, too, I can retrieve it.

I suppose there's a chance my mailbox will fill up with the stuff, but even if Google stopped giving me more space today, a quick back-of-the-envelope calculation says I'll be 72 before my mailbox fills up.

I can live with that.

I Need Google Gears, not Prism

2008-07-28T05:30:00.002-06:00

The Lifehacker list of favorite software and hardware is mostly non-Linux or things like Gmail that I already use. Prism, however, is neither.

I've looked and I don't want it.

Prism, from Mozilla Labs, is a single-app browser. For example, it lets you create something that will run only RememberTheMilk. They call this a "customized webapp."

The advantage? It's distraction-free. The disadvantage? It's still a browser.

Unfortunately, mostly what I want isn't freedom from distractions, it's performance. My major annoyances aren't the icon that says I have new mail -- I put that icon there, after all -- but network slowdowns.

Unfortunately, the best solution I've seen, Google Gears, often isn't available for the webapps I use. I mean, there's no Google Gears for Gmail!

And often, it's broken. Every new Firefox seems to break Gears.

If Mozilla Labs wants to help me, I hope they put some time into fixing the last of these problems.

Exporting HTML from Tomboy Notes

2008-07-27T10:42:00.002-06:00

I've now tried Tomboy Notes, so I know what it can do.

spent part of yesterday fussing with a little Java/Design Patterns project. I made some progress, but the most practical benefit was taking notes on what I was doing with Tomboy -- not because the notes helped, but because I'd never really tried Tomboy out.

One useful feature is its "Export to HTML" option. I spent a few hours noting what I was doing, as I was doing it, then exported the note. Here's the result, with no editing. After the export, it puts up a web page in the browser. From there, getting it here took three keystrokes: mark all (^A), cut (^C), and paste (^V).

HFP-July 26

Unpacked a bunch of stuff from the book's web site. It includes code for all examples.
Learned, from one of the downloaded textfiles, that I can run using a classpath with java -cp.
Hacked their ant build.xml file to works with my weatherstation data.
Verified that ant, like make, notices times.
Fixed an eclipse installation hiccup by paying attention to eclipse's output, which told me how to fix it.
Put weatherstation under git, and tried out git commit --interactive
Put all this into Tomboy Notes in a notebook called hfp-diary, for an excuse to play with TN. Tried out fonts, exporting to HTML, navigating menus with the keyboard.
Successfully, though unintentionally, removed my entire project, including the git repository. :-( After my hacking build.xml, ant clean wiped the entire directory.
Copy the code over from the book, and experiment with various things in git, to try to keep from wiping out the code again.
Get rid of the project lines, so that the sucker will compile and run.
Figure out how to get a run configuration set up. This is tedious, but I suppose I'll get used to it.

Character Mis-placement in Firefox

2008-07-26T05:39:00.002-06:00

Can anyone tell me why the diacriticals in Hebrew-character fonts, like the ones in this article, are misplaced, when I look at it in Firefox?

If you look at words containing alephs, you'll see that most of them have a little mark underneath the character to its left. The mark should be under the aleph itself. This makes the words hard to read, and is a bit like having the dots on every 'i' appear over the following character.

Background info, not directly relevant.

The text is in Yiddish, not Hebrew. Yiddish is a Germanic language, like English, but uses the Hebrew alphabet for the same reason we use the Latin alphabet: the people who started writing the language picked an alphabet they already knew. Hebrew-language text has the same problem.
The diacriticals are under the following character, not the preceding one. Hebrew script, like this, is written right-to-left, not left-to-right.
The diacriticals change the character -- just like drawing a horizontal line across the top of an 'l' turns it into a 'T'. One of the characters is silent, a second is an 'o', a third is an 'a'.

ReviewBoard

2008-07-25T05:16:00.002-06:00

ReviewBoard seems to be getting some interest as an open-sourced code-review tool. I like the slogan:

Code reviews are fun again! ...almost.

3 Things

2008-07-23T09:28:00.000-06:00

Get CC stuff reproducible.
Fix LF build.
Read through Randy's document.

makemake

2008-07-21T06:39:00.003-06:00

The newest "Plutoid" (almost a planet) is called makemake. The claim is that it's named after "a Polynesian creator of humanity and god of fertility."

Oh, nonsense.

Eclipse and Design Patterns

2008-07-21T05:09:00.003-06:00

I'm learning design patterns. Frank Stahl told me, about a similar subject, "You sure do know how to have fun."

Marcia Derr and I went to Caffe Sole yesterday, in the heat, and tried to slog through the very beginning of Head First Design Patterns by Eric Freeman, et al. (O'Reilly and Associates).

The book is a little verbose and cutesy for all our tastes, but it's what we had. Mostly, we just tried figuring out how to compile and run Java programs and how to use Eclipse.

In the end, we got an example to run. The second example, we're still stuck on; they seem to use a class they don't supply a definition of.

Still, it was a pleasant way to spend a Sunday afternoon. Frank Stahl told me, about a similar subject, "You sure do know how to have fun."

Command-Line Tricks

2008-07-20T06:20:00.002-06:00

I got diverted by a presentation by Simon Myers this morning and have spent time messing around with the shell.

I've used these before, and particularly like shell-expand-line for editing my path.

Put this into ~/.inputrc and arrange to re-read it (say, by starting a new shell).

$if Bash
Control-x: shell-expand-line
$endif

Now you get this behavior from the shell (which is giving you command-line editing throug readline(3) ).

$ PATH=$PATH
# now type control-x
$ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/jsh/bin/bash_functions:/home/jsh/bin:/usr/local/libexec/git-core:
# and the path is there, ready for editing

I can get a lot of his tricks to work, but I can't bind "yank-last-arg" in vi mode. Works fine in emacs mode (Meta-.), but I'm a vi guy. I'm doing something wrong, but I don't know what.

A Tool With Many Uses: The Local Yum Server

2008-07-18T05:07:00.003-06:00

I've set up a yum server, at work. This achieves several ends:

It lets us mirror the Fedora distro we're using locally, so we can pull and update packages faster.
It gives us a single place to stick third-party rpms. No more asking developers to "go out and get package X from web site Y."
It gives me a single distribution point for things I produce. I don't have to give developers a login on my box, or find a common server to ftp things to and from, I can just stick an rpm on the server and have them install it with yum.

Expanding Windows, a Mouse-Click at a Time

2008-07-17T05:07:00.003-06:00

I accidentally learned how to maximize my windows in just one direction: click a different mouse button.

At least in Gnome, clicking the "maximize window" decoration with the left mouse button expands it to the size of the screen (if the app permits).

However, clicking the same decoration with the right mouse button expands it horizontally, to the width of the screen, and clicking with the middle mouse button maximizes it vertically.

Paired, sequentially (right click, middle click) they act like a single left click. Indeed, they work the way you'd think in all combinations.

Cool.

Setting Defaults Properly

2008-07-16T08:50:00.002-06:00

Problem: I have a new developer who wants to check something into Subversion. He tries, and is mystified by svn's message, telling him he has to set the environment variable $EDITOR.

Solution: I tell him to set the environment variable.

(Okay, this should read "Problem: My software developers can't read." It's still a problem.)

git, in contrast, has this one figured out. From the git-config man page:

core.editor
Commands such as commit and tag that lets you edit messages by
launching an editor uses the value of this variable when it is set,
and the environment variable GIT_EDITOR is not set. The order of
preference is GIT_EDITOR environment, core.editor, VISUAL and
EDITOR environment variables and then finally vi.

Yes, it isn't quite grammatical. The author's Finnish. Give him a break. "... and then finally vi." This because every Posix-conforming system is required to have a text editor named vi that acts according to the IEEE spec. (It can be a superset, like gvim, but it has to respond correctly to the basic commands.)

Richard Stallman and the FSF were asked for an emacs spec to put into Posix (Stallman named Posix, by the way), but they refused, so there's no guarantee that a Posix-conforming system will have it, out of the box.

A Twisty Maze of Symlinks, All Different: readlink -f

2008-07-16T05:04:00.002-06:00

Which of the vi clones on your system are you using? Do you know?

In Linux, when more than one installed executable could reasonably have the same name a common solution is to make the executable in /bin or /usr/bin a symlink to an entry in /etc/alternatives.

The /etc/alternatives entry, in its turn, symlinks to the real executable. Or, maybe to another symlink to ...

Finding the ultimate executable in this twisty maze of symlinks, all different, can be an Adventure. The solution? readlink.

Here's an example, with java:

$ which java
/usr/bin/java
$ ls -l /usr/bin/java
lrwxrwxrwx 1 root root 22 2007-11-05 15:14 /usr/bin/java -> /etc/alternatives/java
$ ls -l /etc/alternatives/java
lrwxrwxrwx 1 root root 36 2008-04-26 07:10 /etc/alternatives/java -> /usr/lib/jvm/java-6-sun/jre/bin/java
$ ls -l /usr/lib/jvm/java-6-sun/jre/bin/java
-rwxr-xr-x 1 root root 47116 2008-03-25 03:01 /usr/lib/jvm/java-6-sun/jre/bin/java
$ readlink -f /usr/bin/java
/usr/lib/jvm/java-6-sun-1.6.0.06/jre/bin/java
$ readlink -f $(which java)
/usr/lib/jvm/java-6-sun-1.6.0.06/jre/bin/java

Beware: readlink won't warn you if something's not a symlink (okay) or doesn't exist (not okay):

$ ls -l /home/jsh/java
ls: cannot access /home/jsh/java: No such file or directory
$ readlink -f /home/jsh/java
/home/jsh/java
$ echo $?
0

Tom Tromey and Stormy Peters

2008-07-11T05:49:00.003-06:00

Tom Tromey did a fun BLUG talk about emacs last night, with slides he did in, well, you know, emacs.

As you can see from the pictures, it's all downhill from here.

He recommends the emacs wiki, which is created with EmacsWikiMode.

Tom's worked on a pile of stuff over the years, including the Gnu Java compiler, and both automake and autoconf.

Perhaps the video will be up soon. I hope it does as much for his career as her recent talk at BLUG did for Stormy Peters.

What's the hardest thing about switching from emacs? Getting the deposit back when you return a garage full of empty Alt-Meta-Cokebottles.

See Jeff Run

Seeing My DNA

Audiobooks, Sort Of

My Nookie Tablet

Toys

RMBB!

Free AWS Book Chapters

Sort of Balkan Dancers

23-and-me one-day sale

Handling Failure

Process Whack-a-Mole: An Alternative Approach

Just in case

Back Off Man, I'm a Scientist

NCAR Software Manufacturing Presentation, the Movie

NCAR Manufacturing Presentation

Performance Tuning Shell Scripts? Why, yes.

There's a Lesson Here, but I Can't Remember What

Estimating: The Envelope, Please.

Collatz Conjecture

Generating Arbitrary Numbers

Better Safe Than Sorry: Writing Code that Writes Safer Code

NFS Made Easier

The Magic SysRq Key

Space, The Final Frontier

Where's My Jet Pack?

Tar Wars

Octal NUL

Joinery

My Phone Starts Giving Me Things I Never Had

Translate-As-You-Type

The "help" Command, And Other Aids

Valour-IT Fundraiser

Checking Scripts for Syntax Errors

Google's Bulgarian Picture Dictionary

A GUI Replacement for sshfs

Bugs

Hello, World Again

The Shell Enters a Beauty Contest

Grobycs: Cyborgs in Reverse

Variables Without Values

More Include Guards

Simplifying Loopy Code

Call Me Now

But *How* Random Is It?

Another Random Post

The International Slide Rule Museum

Another Challenge From Hal & Ed: Random Numbers In the Shell

Hash Collisions in Git

Using Git Stash as a Snapshotter

Remember When?

Are We There, Yet? Using fuser() to See If The Job's Done.

Empty Turns "Hard" Into "Not-So-Hard"

How Do I Find Missing Files in a Sequence?

Include Guards in Sourced Files

Another Use For The Colon (:) Command

Amazondroid

Linux Google Chrome, the Alpha Version

What's this Colon Doing in My Comments?

File Access Times as Debugging Tools

Using Preferred Applications from the CLI

To Be Continued

Printing Strings in Shell Scripts

Truncating a File

TIMTOWTDI: There Is More Than One Way To Do It

Twitter, Again

Beeees! And Kino to the rescue.

I finally learn how to do realpath

iPods and Banshee

What's the Absolute Path to a File?

Google Runs Rings Around M$ This Week

Editing Parallel Trees

How to Change File Extensions

Shell Programming Texts

Mounting USB Drives Under Ubuntu

Helping My Girlfriend with Google Docs Templates

Fixing the Default ISO App in Ubuntu Jaunty

Setting envelope-from on Ubuntu

Cat Skinning, Made Easy

Google Voice

Grand Central

But How Random Is It?