Joel Franusic

Mystery Company #1

2009-03-18T15:16:12Z

Two months ago I hit the ground running at a new job with a company called NetBooks. They make accounting software for tiny businesses (not tiny computers or online book readers).

I'm having a great time getting to learn new technologies and learning from some very sharp people.

Ozymandias

2009-02-22T19:16:33Z

I met a traveller from an antique land
Who said: Two vast and trunkless legs of stone
Stand in the desert. Near them on the sand,
Half sunk, a shatter'd visage lies, whose frown
And wrinkled lip and sneer of cold command
Tell that its sculptor well those passions read
Which yet survive, stamp'd on these lifeless things,
The hand that mock'd them and the heart that fed.
And on the pedestal these words appear:
"My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!"
Nothing beside remains: round the decay
Of that colossal wreck, boundless and bare,
The lone and level sands stretch far away.

Equal pay for equal work?

2009-02-21T18:44:34Z

Earlier this year, I learned that my friends Mike and Kristina were expecting a second child...

Some data storage solutions that have inspired me

2009-02-20T10:07:07Z

In the course of my long term data storage research, I've run across some very neat storage solutions. While none of them meet my needs exactly, I've found all of the things listed below to be very inspirational in my quest to find or create a simple long term data storage solution.

Let's take a look at some of my favorites!

Clay tablets. The oldest known Clay tablets are over 6,000 years old. The Epic of Gilgamesh was written on clay tablets, the earliest versions of the epic date from 2150-2000 BCE!

"Photos". In the late 1950's IBM developed a few "Photostore" machines that would store data on photographs. Data was stored on "chips" like the one pictured above. Each chip was approximately 2.75 x 1.38 inches and could store about a half a megabyte of data.

Golden records. Probably the most famous of all golden records is the one that was launched into space as part of the Voyager spacecraft. It had controversial cartoon depictions of naked humans, and some great music.

Stainless steel tablets. When Scientologists return to earth from the distant future, Trementina Base will be there for them. Trementina houses the complete collection of L. Ron Hubbard's creative output, engraved on stainless steel tablets, stored in titanium capsules.

Salt mines. Naturally temperature and humidity stable, salt mines are perfect places to archive film and paper.

As I search for a simple long term storage solution, it has really helped to remember these methods of storage. I want to build on the past as much as possible. These solutions have also been extremely inspirational: The clay tablets for their longevity, the Voyager Golden Record for it's carefully selected contents, Trementina Base for its long term thinking, and the salt mines because they are a simple, elegant and non-obvious way to store human artifacts for very long periods of time.

But the most inspirational of all the things I've listed here is the IBM 1360 - the "Photostore". Much of my thinking regarding archiving data to paper has been inspired or influenced by the little know about this system. It does a lot of things that just make a lot of sense to me: "write once, read many", storage of data on non-electronic, non-magnetic, almost inert media, the ability to remove media from a running system for long term storage, the ability for a running system to request the re-insertion of media that is in long term storage.

As I look for ways to help people store data for long periods of time with little or no effort, it has been very encouraging to find similar things others have done in the past. I can't help but wonder what other technologies exist, hiding in the whispers of the past, that can help me find a solution.

The cop-out blog post

2009-02-18T07:30:38Z

I'm exhausted, need to sleep.

Did you know you can use your computer to travel back in time? It's true! Visit Russia from 100 years ago, in color.

On being an extra in a movie

2009-02-17T08:42:45Z

Being an extra isn't as fun or interesting as I thought it would be. From what I can tell, filming a movie is actually pretty tedious.

Here is the summary: The more scrappy the movie is, the more fun you'll have. Avoid being an extra unless you get paid to do so.

I've been an extra in a total of three films: a shoestring budget, student film for a buddy when I was in high-school; a big budget feature film for BBC Films; and an independent film for a web comicist named David Malki.

Of the three, the one where I had the most fun was the student film for my high-school buddy Joshua. As I remember it, it was a lot of goofing around with friends, with a few film takes here and there. (Joshua is still making movies, this makes me happy.)

Next is the David Malki movie, "Expendable". This was actually a lot of fun, probably due to the fact that I spent only a little bit of time as an extra. The rest of the time was spent as a grip assistant, and later a production design assistant - I'm listed in IMDB for this. I actually really enjoyed helping with production design, it's a busy job with tangible, immediate results. I list this second mainly because it was ... a job. Not a particularly hard job, but not a relaxing way to spend a weekend. One great plus of helping with this film was being able to explore an abandoned marine base!

Lastly is the big budget film. I was in this movie because I'm white and was in Rwanda at the time of the filming, we didn't look to be in this movie, they sought us out. The first few times were very fun, mostly because of the novelty. We got to dress in military uniform and walk around with rifles of questionable origin. After the first couple of times however, the only real reason to go was because we got paid, and for the food. They had amazing gourmet catered food on the set. Oh, and they gave us live ammunition to cary around.

Just like grandma used to make

2009-02-16T05:05:34Z

My favorite kind of burrito is the kind that my grandmother used to make:

"Un burrito de papas y huevos con chorizo"

This is one of those cases where using a foreign language allows one to be more precise and terse, in English the same thing would be "a burrito with Mexican diced and friend potatoes made with Mexican chorizo"

The nice thing about being fairly fluent in Spanish is that I can order this in most Mexican restaurants without getting any strange looks and get exactly what I want.

How I started programming

2009-02-15T09:27:14Z

(As some of you have noticed, I've been posting a lot lately. There is a reason for that. I'm part of a group of about 12 people who have made a "bloodpact" to write once a day for a month.)

"How did you get started with programming?" - I casually asked this question once while hanging out with my hacker friends. I expected that their answers would be short and fall into a small number of distinct categories. It came as a surprise then, that their answers to this question were neither short, nor easily categorizable.

My own story is below. I encourage you to write a blog post of your own, telling your story.

I learned how to program from my Dad over the course of many years. The events below are the most memorable.

The earliest memory I have of a specific computer was the Compaq Portable that my Dad had. I remember walking past my parents bedroom on my way back from a trip to the bathroom. It was very late at night, my Dad was awake and watching the computer give out results from a simulation that it was running.

For the next few years, the only thing I would use that computer for was to play Ernie's Big Splash. I think that this is also the computer that Dad used to teach me how to use the text editor that was part of of Microsoft QuickC.

My Dad eventually got a 386 "portable computer" from Dell. It is on this computer that I started learning how to program in C out of "K&R". It takes me a long time to finally feel like I understand C, from what I can tell my progress in mathematics correlated with my progress in learning C.

Time passes. I play a lot of Super Mario World with my brother on our SNES.

More time passes. My family moves to Austin, Texas. I spent practically 6 straight months learning DOS 6 from a book my dad bought for me, writing batch scripts on the Dell.

More time passes, my family moves to Sunnyvale, California. I figure out just enough BASIC to make new QBasic Nibbles levels for my cousin. My dad buys a 486 and we install RedHat Linux (kernel 1.2.13) on it. I start learning bash scripting.

My friend Chris wants to sell some of his SW:CCG cards online. eBay isn't out yet, so I decide to write an auction system for him. I try writing it using CGI and sed/awk. Luckily I find the "Learning Perl" book in the Mountain View public library and everything falls into place. I finish the project and Chris successfully uses the code to complete one auction. eBay comes on the scene. I lose track of the auction source code along with all the other digital artifacts I created in that time.

Finishing that auction script was a defining moment for me. From that time forward, it became easier and easier to move on to other platforms, languages and projects.

There is still much work to be done

2009-02-14T09:25:03Z

I never thought that this day would come: Microsoft is no longer relevant.

These changes didn't happen suddenly, it was an almost imperceptible change. One day I just realized that an overwhelming majority of my friends and peers use a variant of Unix on a daily basis. Many have not used Windows in years.

Linux users once felt like they needed to count themselves to prove that Linux mattered, now people are unaware they are running Linux. Eleven years ago, people were predicting the doom of Apple as their stock hit the bottom and Microsoft invested $150 million in them, today Microsoft is running advertisements attempting to save face from Apple's iconic "I'm a PC" advertisements. It's all pretty unbelievable when I stop and think about it.

So what changed? I'm not sure that I care about the "how" of the matter, I'll leave it up to the historians to figure that one out. I like where things are, I'm even more excited about where they are going.

Nobody has "won", not even close, there is still much work to be done. I get the distinct impression that the Free Software community has just finished laying down the groundwork upon which true innovation can be built - innovation born in the 1970's held hostage by incompetence until now ... that is a topic for another day.

There are still plenty of things to argue about in this area, there always will be. I just don't care to argue anymore, I don't have to. And there lies the crux of the matter: Free Software stands on its own merit.

Improving the fire alarm.

2009-02-13T08:24:19Z

Modern fire alarms are so stark, so robotic, so inhuman. I propose a better fire alarm. Ladies and Gentlemen. I introduce to you:

The Disco Fire Alarm

Why settle for the shirl and jarring beeping of an alarm, when you can have the shrill, jarring sounds of The Bee Gees:

Picture this, your house it filled with smoke, maybe it's just your next door neighbor barbecuing at midnight again, perhaps your roomate forgot to remove the plastic wrap from the pizza. Just as you are about to roll over and hope it all goes away, a disco ball drops from the fire alarm in your ceiling the lights turn on, "Staying Alive" starts to play and it's the 70's all over again.

This is not a drill: It's time to get out of your house as fast as you can, away from the smoke and fire, but more importantly, away from the disco.

Why I want to store data on paper

2009-02-12T08:55:24Z

In my continuing quest to find long term storage solutions, one medium keeps coming up: paper.

Why? With any new technology we can only make statistical guesses at the lifespan of that technology.

With paper, we have over 500 years of experience printing and storing it. Considering all that experience, and that there are 48 remaining copies of the original Gutenberg Bible, I don't think it is unfair to assume that there is a lot of research and guidelines for the long term preservation of paper.

With this in mind, what I'd like to have are two tools: The first tool would use existing research to simulate all the various types of ways in which paper can be damaged: rips, tears, fire damage, heat damage, water damage, and so on. The second tool would build off of the first, it would print and scan arbitrary binary data to and from paper using error correction sufficient to survive the common types of damage that paper experiences.

I can't say I've searched very hard for software to fill the role of the first tool, as such, I don't know of anything that fits in that domain.

As far as the second tool goes, Xerox has DataGlyphs, but that technology is proprietary and presumably expensive, Microsoft recently came out with a technology recently, but whatever. The only software Open Source software I've been able to find is one program Paperback - and it seems to be something of a joke. It seems like public domain PDF417 standard might be the closest existing solution in this area, if nothing else, it's probably a good place to start.

Since it seems that both of these tools are rather domain specific and ... esoteric. It looks like I'll be writing them myself.

I already have a name for the second tool. I'm going to call it "par".

Time to get hacking.

On Containerization

2009-02-11T06:54:56Z

One of the things I love about infrastructure is how it is practically invisible. When you go to get a glass of water, you don't need to understand anything about the complexities of the water supply network - the sources, dams, aqueducts, canals, pipelines, treatment plants, and so on - all you need to know is to turn the faucet. Containerization is one of those things.

In the last 50 years, our world has gone from a system that used ad-hoc containers to one that uses standard sizes (ISO 668). As a result, the prices of shipping have changed dramatically: "In 1956, most cargo was loaded and unloaded by hand by longshoremen. Hand loading a ship cost $5.86 a ton at that time. Using containers, it cost only 16 cents a ton to load a ship, a 36-fold savings." (source)

Additionally, because of the standardized sizes, every modern mode of transport is capable of handling ISO 668 shipping containers: Ships, Trains, Trucks. This is known as Intermodal freight transport.

This all basically constitutes a packet switched network for physical objects. A network which, if you don't care about latency, can be used for a high bandwidth link.

Keep your eye out for shipping containers, they're everywhere.

Unixtime version control

2009-02-10T08:01:46Z

Or: "How typing 10 characters can save you heartache".

Right now, it is "Mon Feb 9 23:21:07 PST 2009" or "1234250467" seconds since the Unix epoch.

When represented in base ten, the number of seconds since the epoch ("unixtime" henceforth) takes 11 code points ⁽¹⁾ to represent, possibly the most efficient way to represent time with second granularity (If we represented the time in another base, we could save even more code points).

How hard is it to find the current unixtime? Easy, use the command:
$ date +%s

This command should work on any system where date(1) makes use of a POSIX compliant strftime(3) implementation.

So what's the big deal?

I use unixtime frequently to avoid making mistakes. For example, I use this command to "delete" a file instead of rm(1):

$ mv $file $file.`date +%s`

I also use it when I want to run a command several times and easily find the differences between each invocation:

$ some_command > output.`date +%s`

And finally: I frequently come across files on systems where, for whatever reason, using a real version control system is impractical or impossible. In these cases, the `date +%s` command comes in very handy as a super ghetto version control system. Some examples:

To do a "checkin"
$ cp $file $file`date +%s`

To compare an old "revision" with "HEAD"
$ diff -u $file.$old_unixtime $file

To take a "snapshot" of your "repository"
$ tar -cf utvc.`date +%s`.tar ./*

At my last job, I had a button mapped to type `date +%s` when it was pressed, I miss that button.

Footnotes:
1: Until 2038, when it'll change to 12 code points. I use the term "code point" because I'm internationalized, yo.

Tools to help you survive data loss in the short term without thinking about it.

2009-02-09T09:24:24Z

Before I delve into my vision for a future of effortless backups, I'd like to cover some relatively inexpensive solutions that anybody can use right now to avoid dealing with accidental or unexpected data loss in the short term, which I'm defining as "less than 5 years".

I'm choosing my words and scope carefully, there are many ways to lose data . What I mean by "accidental or unexpected data loss" is what I think is the base case for most people: Those files you spent a lot of time creating are gone. Also known as: "I accidently ... the whole computer", "Why is my hard drive making grinding noises and where are my files?", etc. As I see the issue, this is the first step in keeping your data around for a long time. There are other things to consider, I'm going to try and cover them later.

In the course of my research, the following solutions have really stood out:

"Drobo" by Data Robotics
"Time Machine" by Apple Mac OS X 10.5 and above.
"Mozy" by Decho (an EMC company)
Jungle Disk, powered by Amazon's S3 service.
"vSafe" by Wells Fargo

I only have a passing knowledge of these tools, so please take these as suggestions for the start of your own investigation rather than concrete advice. I'm not an expert. Not yet.

Let me cover each of these solutions briefly, discussing why each one interests me.

"Drobo"

Summary: Put two or more drives in the Drobo, put your data on the Drobo, replace the drives when they fail.

What impresses me about the Drobo is its simplicity of design. I like how it doesn't require any special configuration for it to work, just plug in the drives and go. I like how you don't need to use any special software to administer the Drobo: when the light changes next to a drive, just replace the drive. I like that it isn't using RAID - more on RAID later.

"Time Machine"

Summary: Free automatic backup software built in to Mac OS X. I pity the fool who doesn't use it.

Unlike a lot of other backups solutions that I've read about, the internals of Time Machine are truly impressive. It keeps track of what files to backup by taking advantage of the indexing service already built into Mac OS X, it uses directory hard links to make compact, full disk backups. And finally, the Mac OS X install disks give you the option to restore your system from a Time Machine backup.

"Mozy"

Summary: Online backup that "just works" backed by one the storage giants.

Mozy is an online backup service backed by EMC (EMC purchased the company the created Mozy in 2007). EMC is one of the storage heavyweights, they are such a big company that I feel safe saying that the "No one ever got fired for choosing x" snowclone applies to them.

Because of the EMC backing, dual platform support, and awesome pricing ($5 per month per computer, "unlimited" storage), I think that Mozy stands out above the rest.

I'm also really impressed and amused by the "Alternatives to MozyHome" that are listed on the MozyHome page.

"Jungle Disk"

Summary: Software that makes it easy to backup your data your Amazon S3 data store.

Jungle Disk is software that automatically backs up your data to your personal Amazon S3 "bucket". Their software costs $20, Amazon charges you $0.15 per Gigabyte per month for storage. My personal issue with Jungle Disk, is that every time I look at it, I convince myself not to buy it because "I could code it myself", and then nothing happens.

"vSafe"

Summary: Woah, online backup from a bank. What?

A "digital safe deposit box" from Wells Fargo. This offering is impressive only because it is offered by a bank. One would assume, and hope, that they are good at keeping data around and secure for more than a decade.

This offering is the most expensive of all the others, their pricing starts at $5 per Gigabyte per month.

Final thoughts: There are a lot of options for storage out there, most of them are unappealing to me for various reasons (which I'd be happy to discuss). All the solutions above either give you full control of your data or are backed by organizations that are unlikely to disappear overnight.

Remember kids: "there are only two types of disk drives: those that have failed, and those that are about to do so".

EDIT: Changed title.

Why I'm obsessed with long term data storage.

2009-02-08T10:28:37Z

It all started with a question from a photo instructor at Cuesta College to me and my boss: "How do I keep my digital family photos around?".

The answer we ended up giving was along the lines of "you can't", "it's hard", or some combination of the two. This answer went against my intuition: How could that be? Digital data is pure, its "essence" isn't reduced, damaged, or reduced by copying. So why is it easier to keep a meatspace artifact around longer than a digital one? Or to put the question succinctly: Why is it so hard to keep data around for a long time?

I started looking for an answer, There had to be a better answer than constantly tending over backups. There had to be some sort of way to store arbitrary binary data for very long periods of time with little or no human intervention. I'm convinced there is an answer out there, but so far I haven't been able to find one answer.

As I talked about my search for a long term data solution with other people, I found that I had hit something of a raw nerve with them. It seems like everybody that I've talked to has a story of traumatic data loss. While I admit that I knew better, and that the reason why I have traumatic data loss stories of my own is due to negligence on my part, I don't think that it is reasonable or fair to place this blame on a non-technical person. With all of the progress that we've made in computing, long term storage of data for "anybody" should be a trivial and Solved problem.

My obsession has a goal: Find a way to trivially store arbitrary binary data for 10,000 years.

More on that later.

Towards an ideal bash environment.

2009-02-07T09:34:15Z

I've been using Linux for over 10 years, however it wasn't until I started getting paid to use Linux on a daily basis that I saw the value in keeping "dot files".

What I want is not a fancy prompt or tweaks to make a self-consistent system. I want to customize my environment so I can do more with less.

I'm still in the idealized design phase of this ideal environment. The core principles that I am going for are as follows:

The environment should work on any system with at least SSH and bash.
If part of my environment depends on an external tool, the environment should be able to attempt to get that tool onto the system.
If it is not possible to bring in an external tool, degrade gracefully.
Include as many custom configurations as possible.

One element of this environment that I'm excited about is an "infect" command. Given a remote host, username and password, this command will SSH to the remote host, install my SSH keys and then set up my environment. Part of this setup will be searching for external tool dependencies, if there are dependencies missing, I'm considering having the "infect" command attempt to compile the dependency on the remote host.

My end goal is this: No matter what host I am connected to, I want the following:

The name of the host I am on to be clear.
A core set of my commonly used tools.
Personal configurations for as many tools as possible.

The results of the idealized design I have done on this system so far are on my wiki: http://joel.franusic.com/Ideal-bash-environment

I'm tired but I'm working, yeah.

2009-01-20T02:59:16Z

Wednesday January 14th, 2009 approximately 10h20 PST: "T minus 0"

T minus 0hrs - I am laid off
T plus 3hrs - Post to Twitter and Facebook: "Laid off from PBwiki and looking for my next adventure! http://sargo.com/joel/resume.pdf"
I start getting phone calls and email from friends, old colleagues and acquaintances.
T plus 4hrs - I get a lead from an old colleague.
T plus 5hrs - I have an interview scheduled with Mystery Company #1.
T plus 24hrs - Mystery Company #2 contacts me for an interview.
T plus 25hrs - I interview at Mystery Company #1.
T plus 28hrs - I finish the interview with Mystery Company #1 and the lunch afterwards with my buddy.
T plus 31hrs - I get a phone call from the CTO of Mystery Company #1, we negotiate salary, he informs me that he intends to make me an offer.
T plus 46hrs - I have a phone interview with Mystery Company #2
T plus 49hrs - More interviews with Mystery Company #2
T plus 55hrs - I receive the offer letter from Mystery Company #1

I discuss my options and the results of the interviews with my family, mentors and advisors. I start working towards a decision.

T plus 101hrs - After much consideration, I accept the job offer with Mystery Company #1

SF Open Studios

2008-10-20T04:12:26Z

I visited a bunch of art studios that were part of the SF Open Studios event with Corrie today. Much of San Francisco's art talent seems to be in The Mission.

It was a very inspiring event overall. I'm now thinking about doing some art of my own... e-ink + context free perhaps?

Scaling

2008-09-08T23:15:55Z

I often need to explain to people why the network at their home is better than at the office.

Here's an analogy I just came up with:

"It's like baking a cake, it's easy to do at home, but difficult for 2000 people"

I have a sign in my cube

2008-09-04T22:29:07Z

This sign comes up quite often.

This is the sign.

Meow, MEOW, meow! Truck.

2008-09-01T00:47:10Z

"meow.pl":

#!/usr/bin/perl
# Runs from Mac OS X

my $n = 1;
my @voices = qw(bruce alex fred kathy vicki victoria);

$| = 1;


do {
    $voice = $voices[int(rand($#voices+1))];
    $out = "meow";
    $out =~ s/e/ee/ if (rand() < 0.2);
    $out .= '!' if (rand() < 0.4);
    $out = 'Truck' if (rand() < 0.05) && ($n > 3);
    system('/usr/bin/say', '-v', $voice, $out);
    $n++;
} while($n < 20);

system('/usr/bin/say', '-v', $voice, 'Truck');

How to print to a DYMO LaserWriter Twin Turbo from a Mac when you only have 2 hours to figure it out

2008-08-30T23:26:21Z

UPDATE: I've made lots of little updates to this.

I'm at SHDH26 right now. We now have a fancy login system that prints out name tags for you!

Tom Harrison, Ernesto S., and David S. had most of the system working last night, with the exception of the automatic label printer. This morning rndmcnlly and I got the printer working in about 2 hours!

Here is how we did it:

First we had to get the printer working on a Mac, this was pretty easy. We downloaded the drivers and set up the printer though the normal means.

After that we wrote the following two files:

"~/.enscriptrc":
Media: Label 162 288 10 10 152 278

Use the command enscript --list-media to make sure that your .enscriptrc is being parsed.
Use man enscript for more info
The numbers are in Postscript points!

"~/printer_server.py":


from twisted.web.resource import Resource
from twisted.web import server
from twisted.internet import reactor

from subprocess import *

class Launcher(Resource):
        isLeaf = True
        def render_POST(self, request):
                names = ['first_name', 'last_name']
                items = map((lambda l: request.args[l][0]), names)
                page = "\n".join(items)
                p = Popen(["enscript", "-r", "-B", "-f", "Helvetica35", "-MLabel"],stdin=PIPE)
                p.stdin.write(page)
                p.stdin.close()
                return page



reactor.listenTCP(1080, server.Site(Launcher()))
reactor.run()

(rndmcnlly wrote all the python, I just figured out the enscript syntax)

Once the files were written, we ran the printer server using the "python printer_server.py command.

The printer server above listens on port TCP 1080, we had to change the port number because Comcast sucks and blocks that port.

The server listens on port 1080 for a HTTP POST and prints the value "first_name" on the first line and "last_name" on the second line.

Labels are printed by the web application sending a HTTP POST to the port of the computer connected to the printer - we had to set up port forwarding to get this working.

Unfortunately we didn't have time to implement Lee Felsenstein's suggestion to let people "tag" themselves. We'll have that working by the next SHDH - maybe even just let people go freeform on their tags.

Tom's end goal for this is to eventually have a "DevHouse Appliance" that other people could easily set up to do the same thing!

Seattle: Day 1

2008-08-07T22:46:24Z

Woke up at 05h45, left my apartment at 06h00.

Arrived at SFO at approximately 07h15, not bad considering that the N Judah was delayed on the way to Civic Center station.

After clearing the checkpoint, I took some photographs of the brave men and women of the Department of Homeland Security. These men and women put their lives on the line every day to preserve our freedoms and protect America! As I was taking a few pictures, an officer informed me that taking pictures was not allowed, she walked up and tried to grab my camera. It's no wonder that these brave men and women are unappreciated, how can they be appreciated if we can't take pictures of them in action?

My flight was the first time I've flown Virgin America. What a fantastic airline! It doesn't really get better than playing video games while in flight.

I took a bus from SEA to Pioneer Square in Downtown Seattle. The Seattle Transit Tunnel is quite impressive! I didn't really know where I was or have any plans about what to do - I was reminded of my arrival in Paris, except I could read the signs.

I eventually found my way to Trabant Coffee, I was lured inside because of a sign outside that was advertising iced coffee made from a Clover coffee machine, I was intregued because of the article in Wired magazine about the Clover. The iced coffee didn't taste espically good, but I really appreciated being able to sit down and use the WiFi to plan out my day.

I was able to figure out that I was only a couple of blocks from the Seattle Underground and the Smith Tower. I also surprised to find that my friend Tom Music was only a few blocks away at a co-working space.

I decided to just show up at Giraffe Labs to visit Tom Music. Tom wasn't there when I arrived, but I did meet Anders Conbere, Theodore Nordsieck and Nathan Kaiser who are all very cool people. Even though I was in a new city and kept thinking that I should probably be doing some sightseeing, I ended up spending several hours at Giraffe Labs - it's the most comfortable co-working space I've been to yet.

I went to a local cafe that I found on Yelp called Zeitgeist. I had a fantastic sandwich, ran into Theo again, met up with Adam and Kathleen and went to the Seattle Underground tour.

The tour was far better than I had imagined. I was expecting the usual tacky tour used to push merchandising. Instead, we had a man named Rick Boetel as our tour guide. Rick had an astonishing working knowledge of the history of the underground that made the tour very intriguing. He was very open to questions and used them as excuses to delve into fascinating side stories.

After the tour was over, Tom, Adam, Kathleen and I went back to the Trabant cafe to sit around and talk. When we got to the cafe, we found that they were closed. One of their employees, a barista named Alex was preparing for a barista competition. However, instead of turning us away, we were invited into Trabant to sample the drinks that Alex was making. As Alex made drinks, we chatted with a friendly lady named Tatiana. It wasn't until after we left that I realized that Tatiana owns Trabant with her husband - I just assumed she was a co-worker who was sticking around to help Alex.

While Alex was making drinks, Tatiana gave us all a behind the counter demonstration of how their Clover worked. I had a hot coffee this time, it could be the hype, but it was one of the best cups of coffee I've ever tasted.

Tatiana and Alex were so friendly and amicable that we ended up spending over an hour chatting with them and tasting Alex's drinks - he had a specialty drink that was amazing, flavors like I've never tasted before.

After leaving Trabant, we walked Tom back to his car. Adam, Kathleen and I then went to an indian resturant for dinner then took a cab back to the house where Adam and Kathleen live.

I met the roommates, a cool group of people. I had a great time talking to one of the roommates named Aaron who is a Hadoop expert.

Adam and I were up until about 2am discussing Big Ideas.

It's far to late in the day, it's time for me to leave the house now.

joel @ 2008-07-23T16:59:00

2008-07-24T00:00:00Z

Now playing on the PBwiki office radio

A reducer for uniq -c

2008-07-03T23:39:39Z

If you think of "grep | sort | uniq -c" as a "map" operation, then I wrote a "reducer" for that. I do enough counting with grep | sort | uniq -c on really large files that I finally broke down and wrote a simple perl script that in some cases can dramatically speed up a query. For example:

Compare this:

$ time zcat /tmp/29M_of_logfiles | cut -d ' ' -f 1 | sort | uniq -c [results here] real 1m36.721s user 1m38.942s sys 0m1.600s

To this:

$ time zcat /tmp/29M_of_logfiles | cut -d ' ' -f 1 | uniq -c | perl uniq_sum.pl [exact same results here] real 0m17.146s user 0m15.437s sys 0m0.980s

Here's the source code, I wrote it at PBwiki, but I'm putting it online with permission from David Weekly.

#!/usr/bin/perl # uniq_sum.pl: a "reducer" for 'uniq -c' # # for example: you could do this: # $ grep THING * | sort | uniq -c # # but this would be faster (because you don't need to sort lots of lines); # $ (for file in *; do grep THING $file | uniq -c; done) | perl uniq_sum.pl # # Copyright 2008 PBwiki, Inc # # Author: Joel Franusic my $field_length = 0; my %hash; while(<>) { chomp; next unless($_ =~ /^(\s+\d+)\s(\S.+)$/); my @field = split '', $1; $field_length = $#field if($#field > $field_length); $hash{$2} += $1; } foreach my $key (sort keys %hash) { printf("%*d %s\n", $field_length, $hash{$key}, $key); }