Weather Prediction Error

I wrote a Python script to download the 1 through 5 day predictions for the weather in the top 50 American cities from the Weather Underground. I’ve got almost 2 years of data now. Here are the RMSE for predictions for all cities. I believe the RMSE for all cities is 2.24, 2.89, 3.68, 4.29, 4.92 degrees F for 1 through 5 days respectively. I put the data up here, but I don’t yet know how to add visualizations. Unfortunately I have forgotten what little statistics I’ve ever learned. I tried a naïve prediction: tomorrow will be the same as today. This scored a RMSE of 6.11, which is significantly worse than even the 5 day forecast. I’ll try some machine learning algorithms on the data. In addition to the temperature, I stored things like weather condition (rain, snow, cloudy, etc) which may help make better predictions. Overall, I guess the weather underground does ok.

Upgrading Ubuntu to 11.10

I run various versions of Ubuntu within VMware. Every single time I let it do a full upgrade to a new version it fails. This time an upgrade from 11.04 to 11.10 failed to reboot. Here’s the problem and solution. This is my first look at Ubuntu’s Unity. Generally I don’t care about the desktop because I live in Emacs. However, this is a colossally bad UI experience. Suddenly the firestorm of complaints about Unity make a lot more sense. I’ll keep trying it for a little while. If I still hate it in a week I’ll switch back to Gnome.

High Performance Counters

There doesn’t appear to be a high performance concurrent counter available in the .NET framework. I timed various simple implementations on my Core 2 Duo laptop:

1. A single thread can increment a static field 1.5 billion times in 4 seconds.
2. Two threads that update a static field runs slightly slower than 4 seconds probably due to the locking overhead.
3. Two threads that use Interlocked.Increment to update a shared field takes 52 seconds!! Why is it so slow?
4. Two threads that update a ThreadStatic field takes 2.5 seconds. Of course this isn’t a global counter, but it sets a baseline for performance.
5. Two threads that update separate fields takes 3.7 seconds. Why? Probably because of cache contention.
6. Two threads that update separate fields on different cache lines takes 2 seconds. A cache line is 64 bytes, or 8 long fields. So I added 7 long fields to force the 2 counter fields on different cache lines.
7. Same as #6, but use an array of counters rather than fields. Now takes 8 seconds. Is this because of bounds checking?
8. Same as #6, but use CompareExchange (CAS) to update fields, takes 22 seconds. There’s no contention and it always succeeds. Why is this operation so insanely expensive? It’s supposed to be a single machine op on x86.

Notice how small implementation differences result in giant performance gaps. Updating separate cache lines is important for speed, but I still need to sum the values to return the current count. Cliff Click has a clever implementation in Java here. However, it still relies on CAS to perform updates which will, on .NET at least, slow down performance by 10X. This post from Joe Duffy confirms that CAS operations are expensive. I suppose if these designs scale to a large number of threads then it’s worthwhile. But if it sucks for a few threads then it’s unusable for most programmers.

iOS 5 update ate my pictures

On vacation for the past month, I transferred pictures from my camera to my iPad using the camera connection kit. When I arrived home and synced, iTunes suggested I update to iOS 5. I thought the update process makes a backup anyway, so doing an update should be fine. Well, the update failed for some reason and no backup was made. My iPad is now in the restore state and the pictures are all lost.

Or are they? I assume the pictures are still on the drive somewhere. If I can somehow mount the drive I might be able to recover them. The jailbreak program greenpois0n published their exploit code called syringe. It says that the jailbreak can expose the filesystem. If I can mount the filesystem in Linux, perhaps using libimobiledevice, I can scan the drive and recover the pictures. There’s some technical information on iOS and jailbreaking available here. This is one of those rare moments when my CS degree may actually be useful.

PuSH with Twitter

PuSH (PubSubHubbub) is a sorta’ “real-time” notification protocol. Rather than poll for changes from a publisher (RSS), you can tell a PuSH hub to poll the publisher and receive a notification when there is a change. The notification is sent by POSTing to a URL on the subscribers machine. This sucks because it means the subscriber needs to run a client at all times to receive notifications.

Perhaps an alternative is to use Twitter as a PuSH message bus. When a PuSH hub detects an update it can send a direct message to the subscriber’s Twitter feed. Subscribers can use Twitter’s user streaming api to fetch and process these direct messages. The advantage is that Twitter queues the messages until you log in to grab them, and it has a decent streaming API for pushing lots of messages quickly to the subscribers. No need to reinvent the wheel when it’s available for free.

Bazaar Flow

Git flow is an implementation of a nice branching model for software development. I use Bazaar for private projects, but in a totally simplistic way. Here are notes to myself for using git flow with bzr. I’m quite certain I’m doing it wrong, so if anyone stumbles upon this post and has a better way of doing things I’d certainly appreciate the input.

To create lots of branches in bzr, it’s better to create a shared repository first. Go into that directory and make a branch that mirrors the main branch. Then make branches for develop and other features. After you’ve made some commits on a branch, go to the branch you want to merge the changes into. Then call merge from the feature branch into the main branch to add those commits. Fix any merge problems and call commit. This will not do a fast forward merge; instead, it preserves the branch info in the history.

  > bzr init-repo project
  > cd project
  > bzr branch /path/to/master
  > bzr branch master develop
  > bzr branch develop feature1
    ... hack on feature 1 ...
  > cd develop
  > bzr merge ../feature1
  > bzr commit

Confessions of a reddit addict

Reddit is addictive in much the same was as slot machines: the rewards are random. Though most of the comment threads on reddit are painfully stupid, there are enough stupendously hilarious threads to keep me buzzing on a dopamine rush. But it’s time to regain control of my life. To do so, I set my router to use OpenDNS, where you can block any site you want.  I run DD-WRT on my router, which is set to use DNSMasq as my local DNS proxy. However, you need to configure DNSMasq to use their dns servers, which I’ve added below. Of course I can turn this off anytime I want. But it’s enough of a speed bump to hopefully jolt me back to sanity. To ease the withdrawal symptoms, I’ll be slamming Red Bull continuously.

local=/home/
expand-hosts
no-resolv
strict-order
server=208.67.222.222
server=208.67.222.220

Spotify Review

I’ve had a chance to try Spotify for a week. For the narrow customer segment that I represent, it is probably not the right music subscription service. I am not a music nerd. I don’t really care whether a service has some obscure Dinosaur Jr. live performance from an old radio show. For me the most important features are music discovery and passive radio listening. The qualities that make Spotify popular have little appeal to me. Instant playback? I can wait a half second. Larger music catalog? I don’t listen to anything too obscure, nor do I care if a particular track is missing. Higher-quality audio? I’m fine with even 96kbps on my crappy computer speakers or headphones. Desktop app? It’s really for their benefit, not mine. At a recent talk their architect said peer-to-peer sharing handles 20-50% of the music delivery for them (depending on the market).

I actually like last.fm, but I do wish I could pick and choose tracks occasionally. I also would like offline access on both my laptop and iPods. Pandora is nice, but seems to cycle through the same songs and artists a bit too often. I’m going to try Rdio and Mog soon. Now I just need to pipe all this music through my stereo somehow.

Prolog and AWS

I gave a rambling short talk recently describing my project to use Prolog to administer deployments on Amazon Web Services (AWS). Here’s the source code. I completely failed to explain the reason for doing this. So let me try again in this space.

AWS can be programmatically controlled via APIs exposed as web services. There are many, many tools that offer easier access to these APIs. To do anything interesting, you have to write a complex script to accomplish a high-level goal, i.e. deploy new web servers, backup a database, expand a tier, run a new DB slave, etc. You will often end up with lots of scripts to do this or that because these scripts are imperative (a sequence of steps to achieve a goal). The scripts are imperative because the languages (Java, Python, Ruby) are imperative (the OO just makes it modular).

With Prolog I hope to build, in effect, a script generator. My Prolog code right now can query AWS and load in the initial state: all available instances, volumes, snapshots, security groups, etc. in your account. The programmer only needs to define the goal state: the way you want your AWS deployment to look. Given the appropriate rules, Prolog should be able to find a way to accomplish this goal by figuring out what set of AWS calls it should make. It is basically generating the scripts for you.

A simple example would be a goal state shutdown: no instances should be running. Let’s say right now there are 4 running servers. The Prolog system will see these 4 servers and call terminate-instance for each one. Simple, right?

A less simple example might be replicate-zone: copy the current deployment in zone A to a new availability zone B. Again the Prolog code will look at what you’ve got in zone A. Then it will try to launch the same instances in zone B.

A more complex example might be “run my website for under $100/month”. The system might explore deploying spot instances, launching 1 larger instance rather than 3 smaller ones, or whatever. It will somehow manage your resources to fit within a budget.

I’ll post my code soon, but it is a barely functional minimal prototype right now. There are lots of interesting issues to deal with like data modeling, expressing rules, inventing a DCG grammar. I’m confident it will work eventually.

Distributed Performance Tracing

I stumbled on this paper from Google describing Dapper, a distributed performance monitoring system (aka profiler) for their entire infrastructure. For any distributed system, even simple client/server systems, it is useful to measure performance across services & tiers. For example, in a distributed system service A may call B and C, B calls D and E, C calls F. All these calls form an acyclic graph. Most services return a value, though some may be asynchronous and some may be one-way calls (no return value).