Paul Graham wrote an essay a while back arguing that succinct programs (i.e. short) are more powerful, whatever that means. There seems to be correlation between languages we believe are more powerful (e.g. Lisp, Haskell) and shorter programs compared to mainstream languages (Java, C#, C++). Why is this so? Certainly, there is more messy boilerplate required by popular languages. In many cases, higher-order functions can simplify certain code patterns; however, C# and C++ support this fairly well now with lambdas, and Java is sorta’ close with anonymous classes. When I look at how much more succinct Factor (a concatenative language) programs are compared even to Lisp & Haskell, it dawned on me why some languages are more succinct than others: intermediate variables.

In procedural and OO languages, you spend a lot of time defining and using variables in your code, which makes your code verbose. In functional languages, you tend to use function composition a lot, which hides many intermediate variables. For example, “f(g(x))” is shorter than “y = g(x); return f(y);”. In Factor it would just be “g f” without any superfluous variables. Take a look at any chunk of code and imagine if all the variable definitions and uses were removed, leaving only the functions and control-flow. It would be a lot shorter, right? I think that explains a large part of why some languages are shorter than others.

I wrote a concatenative embedded DSL in Scheme to see if programs do become shorter. It doesn’t quite work like function composition nor F#’s pipeline operator. Both of those require that the number of outputs from one function match the number of inputs in the next (usually just 1 arg!). With concatenative languages, the excess arguments are stored on the stack. So if g produces 3 return values but f only needs 2, the extra value is left on the stack for someone else to consume later. In Scheme, the apply procedure is too rigid and multiple values is hopelessly broken. I had to work around both to produce a pitifully inefficient execution engine for this programming style. When I get higher-order words working I’ll post the code.

Of course, the central question remains, what does “power” mean for a language? Graham’s definition is circular: high-level languages are succinct and he thinks they are more “powerful”; therefore, succinct = powerful. I disagree. I don’t have a definition, but I think continuations are a good example of “power”. Languages without  continuations give you a handful of control flow operations. If you don’t like them you have to write tons of code to, in effect, simulate continuations. Haskell’s laziness is another example of “power”. Without it you have to jump through hoops to simulate it in another language. Nevertheless, shorter code is nice as long as it doesn’t compromise readability.

It been a while since I mentioned this idea, but here’s an implementation of a memcached client in Factor. It’s not completely robust, but it works well enough. Next up is a server that accepts Factor scripts to execute locally.

USING: kernel io io.crlf io.encodings.ascii io.sockets io.streams.duplex strings sequences splitting math.parser formatting ;
IN: memcached.client

! Based on protocol for memcached v1.2.2

<PRIVATE

: get-response-data ( int -- string )
read read-crlf drop ;

: get-response-header ( string -- int )
" " split fourth string>number ;

! I'm sure there's a better way to do this
: get-response-end ( -- string/f )
read-crlf [ "END" = not ] keep and ;

: get-response ( -- seq )
V{ } [ get-response-end dup ] [ get-response-header get-response-data suffix ] while drop ;

: store-command ( command key value -- string )
[ length ] keep "%s %s 0 0 %d\r\n%s\r\n" sprintf ;

: send ( cmd -- )
write flush ;

: store ( command key value stream -- string )
[ store-command send read-crlf ] with-stream* ;

: retrieve ( str stream -- data )
[ send get-response ] with-stream* ;

: remove ( command stream -- string )
[ send read-crlf ] with-stream* ;

PRIVATE>

! Handling connections

: connect ( ip port -- stream local )
<inet> ascii <client> ;

: disconnect ( stream -- )
drop ;

! Storage commands: set, add, replace

: set ( key data stream -- str )
[ "set" -rot ] dip store ;

: add ( key data stream -- str )
[ "add" -rot ] dip store ;

: replace ( key data stream -- str )
[ "replace" -rot ] dip store ;

! Retrieval commands: get

: get ( key stream -- data )
[ "get %s\r\n" sprintf ] dip retrieve ;

! Deletion commands: delete

: delete ( key stream -- str )
[ "delete %s\r\n" sprintf ] dip remove ;

! Increment/Decrement

: incr ( key val stream -- str )
[ "incr %s %d\r\n" sprintf send read-crlf ] with-stream* ;

: decr ( key val stream -- str )
[ "decr %s %d\r\n" sprintf send read-crlf ] with-stream* ;

The hysterical screaming over health care has reached outlandish new lows. One thing I’ve learned is that policy debates are exactly like political debates: misinformation, exaggeration, outright lies, and bitter hatred for the opposition. This isn’t a good way to transform a $2 trillion chunk of our economy. For posterity, here’s an outline of how to “fix health care”.

There are two problems with health care. First, the cost of the current system is growing so fast it will inevitably force the government, businesses and insurance companies to radically scale back medical services available to most people. Health care rationing already happens now, and it will increase in the future. The second problem is that millions of people are uninsured or underinsured. Those who are insured but fall gravely ill can be dropped from insurance. For most people, insurance is so important that it can prevent them from moving between jobs or starting a business, which makes our capitalist system less efficient. So the two problems are cost and access to health care.

The two sides of the cost issue are producers (hospitals, doctors, drug and tech companies, etc.) and consumers (people, businesses, insurance co. and government). All of these groups blame everyone else for the rise in health care costs. As with most things, everyone is a little bit right. Here’s my short list of causes ranked by significance, plus a quickie solution:

1. New treatments (drugs, technology, techniques) have extended our lifespan and are definitely worth the money. The latest and greatest stuff is very expensive, though. Solution: Use more generic drugs. Increase gov’t R&D to offset corporate R&D expense. Make FDA approval process much faster and cheaper.
2. Doctors and hospitals have figured out how to make lots of money by overusing and overcharging for procedures. Solution: Move away from pay-per-procedure in most cases. Issue guidelines for common illnesses. 
3. Long-term illnesses account for the bulk of medical costs. A big chunk of those illnesses are caused by poor personal behavior (smoking, drinking, obesity). Solution: Integrated, specialized facilities for long-term, high-cost illnesses. Much higher taxes for alcohol. Modestly higher insurance rates for people with poor health habits. Somehow do something about obesity.
4. Most hospitals are very inefficient. Not just their lack of tech (medical records and more), but their general management and structure. Solution: Hospitals need an incentive to become more efficient. If flat insurance payments are set fairly high then efficient hospitals pocket more as profit. If these payments decline steadily every year, then there’s pressure to improve over time. Publically publish cost metrics for every hospital and area in the country and audit those in the worst 10%.
5. Insurance companies raise rates quickly because they know businesses are reluctant to switch. (Insurance companies are very profitable. Look at ROE, not bottom-line profit numbers) Solution: Move to individual insurance rather than employer-paid insurance. Remove most state regulations that stifle competition. Most state markets are oligopolies.
6. Medical malpractice insurance rates are quite high. Malpractice suits cause doctors to perform “defensive medicine”, extra unnecessary tests and procedures to protect them from lawsuits. Solution: Again, not enough competition between insurance companies to lower premiums. Raise standards for legal liability, impose cap on “pain & suffering” payments. State licensing boards should go after doctors with too many lawsuits and remove bad ones promptly.

On the issue of access, the widely quoted numbers on the uninsured are probably wrong. A significant % qualify for Medicaid. Another chunk are young adults who choose not to spend money on insurance. Lower insurance rates due to lower health care costs will help, but there are still many people who can’t afford anything at all. Maybe letting more people into Medicaid (at a slightly higher fee) would reduce the number of “real” uninsured, while a fine structured as a tax credit might motivate those who can afford insurance to go get it.

More changes will be necessary to fix the health care market. Employer provided health insurance should be taxed as income (it is income!). Insurance companies can not turn away new customers, nor can they drop you when you get sick (called rescission, it happens a lot). The current system is so fantastically messed up that there are lots of easy ways to improve it without throwing the whole thing out. If the system doesn’t improve in 10 years, replace it with a European universal system and be done with it.

I used to read stacks of political magazines in the 90s. In 2004 The New Republic and The National Review formed a joint website called Opinion Duel, in which a writer from each magazine would debate a topic. All the debates quickly degenerated into flame wars, albeit with expensive Ivy League vocabularies. Both sides twisted facts, misrepresented their opponent’s arguments, made snide remarks, and were basically jerks. They made no attempt to actually understand the other side’s point-of-view. I finally understood that public policy debates are no better than political debates. Like trolls on Usenet, both are more interested in winning the argument than in finding the right answer.

This is my dog, Elvis. He’s a 2 year old Siberian Husky from a local animal rescue organization, which really consists of one frazzled woman with a dozen dogs packed in her house. I’ve had him for 5 months now and he’s turned out to be perfect: friendly, calm, gentle, clean, smart and playful. And when we’re walking, he scares people so they move out of our way.

I am lucky to have gotten this dog. Because I’ve never had a dog and I live in an apartment, the shelters I contacted from PetFinder either turned me down immediately or, in most cases, failed to respond. I’ve since met many people who also ran into this resistance from shelters and opted to get a dog from a breeder instead. The only reason I got Elvis is because the woman who runs this shelter was distracted by personal problems and allowed her much more reasonable and level-headed friends handle the adoption. They said it was common for shelters (including this one) to turn down lots of good homes, hoping to find a perfect home for their dogs. The people who run shelters think with their hearts, not with their brains.

The problem is that the number of surplus dogs & cats vastly exceeds the total capacity of all shelters to house and feed them. Consequently, the local animal control office is forced to euthanize thousands of animals. If a shelter can find a home for a dog, it frees up a spot to save one more dog. When these shelters hang on to their dogs, they are condemning another to its death. Worse, if a person grows frustrated and turns to a breeder, it sends an economic signal to continue breeding more dogs. The goal is to place as many dogs in good homes as possible. Obvious, yet it isn’t happening.

I’d like to build a website to help manage animal shelters. Sort of a “SAP for shelters”. It should track their inventory of pets and streamline communication with customers. It should manage volunteers and foster homes. It should monitor food, medicine, and supplies. It should keep track of finances, donations and lots of other stuff. But first I need to learn more about how shelters operate. If Once I get going, I hope to make this all open-source and encourage other, smarter web developers to fix my mistakes. This won’t solve all the problems faced by animal shelters, but it will help smooth the path to those solutions.

memcached is a hashtable with a simple API that allows for remote access. It is a major component of many large-scale web sites because it caches results from more expensive remote resources, i.e. DB access. The API consists of a several different storage methods (set, add, replace, append, prepend, cas), retrieval methods (get, gets), and a delete command (delete). Unfortunately, if you want to do something more complex than simple store/retrieve with memcached, then you have to make lots of expensive remote calls. For example, say I store a tree in the cache, where each node is “key, node key, node key”. To delete the entire tree, you have to recursively get the root node, delete it, then get both branches, delete them, etc. This is expensive.

A possible solution is to send Forth scripts to memcached to execute locally. By adding Factor, a fast implementation of Forth, as the front-end to memcached, clients can send more complex operations to execute locally. Forth is a good choice because the programs tend to be quite terse and easy to interpret quickly. For example, “key get [ “NOT_STORED” ] [ key value set ] if” implements the “add” command in memcached. Since that might be tedious, the user can load a script in Factor that implements that line as a new command: “: add ( key value — result ) “ + the previous code. Now clients can just send “key value add” as their command.

A better example is to delete a tree as described above. The Factor code would look something like this (I’m still learning the syntax). All the work would be done locally on the memcached node.

: delete-tree ( key -- )
     dup get each [ dup 0 = [ . ] [ delete-tree ] if ] delete

By adding an efficient processing language in front of memcached, people can write more powerful programs to utilize distributed caches. What would it look like to delete a tree that is scattered across 100 different nodes? As a matter of fact, LinkedIn (and Digg and Facebook) stores their entire social graph in-memory for very fast traversal. So people are building ad-hoc solutions for problems involving large distributed data structures stored in a cache. They put the logic in a separate program, but I think we can integrate the processing directly into the cache with Forth & memcached.

A commenter on my code sample for a concurrent skiplist correctly pointed out two errors in the program. I attempted to fix both, but further testing revealed another problem in Release mode (optimized, no debug). One of the thread would suddenly stop. After much screwing around, I discovered that the program was stuck in a spin wait: “while (!goAhead) ;”. I added a volatile declaration to that variable and then the program worked. It turns out the loop keeps reusing the value in the register, rather than reloading the variable when another thread says it’s OK to go.

I’ve put the code in Launchpad and removed it from the other post. You can download it like this using the Bazaar VC tool:

bzr branch lp:~suranap/+junk/concur.net

There’s still a problem with the program. When I run 2 threads, it is really quick. When I run 4 threads, there is so much contention that the “add” method has to retry a zillion times. Of course, my test case is probably extreme. I’ll compare the performance with Java’s implementation to see if it’s in the same league.

A new workshop at OOPSLA is aimed at finding ways to evaluate the effectiveness of programming languages and tools. Unfortunately, the entire conference rests on a single problem: how to measure programmer productivity. Except for pointy-haired bosses, most software professionals believe productivity can not be accurately measured. This workshop will likely be yet another outlet for HCI research with awful methodologies.

An ad-hoc embedded domain specific language, aka Fluent Interface, is an API that reads like an English sentence. LINQ does an excellent job of implementing this style so you can write long queries in normal dot notation rather than the query syntax. Someone at Google posted a collection library for Java that does something similar to LINQ, but it’s not as readable as LINQ’s extension methods. Someone wrapped the code with a fluent interface (from google cache) to make it more readable. This requires that all the methods return an Iterables interface so you can do proper method chaining. It doesn’t fit the normal OO-style, and can be tedious to wrap existing classes into this style.

A minor bit of syntactic sugar that I once thought would be useful for C# and Java is the ‘with’ expression, which allows you to write code like this:

with person {
    .FirstName = "Bob";
    .LastName = "Smith";
    .MoveToDepartment("HR");
};

Within the body of the ‘with’, any illegal use of the dot notation (“.Member”) is assumed to use the object “person”. The syntax is just “with Expr Body”, and it returns the Expr when it’s done. With this syntactic sugar, you can now write normal OO code (stateful changes on the same object) that reads like a Fluent Interface. It turns out that this syntax also subsumes the object initializer syntax in C#. Compare these two lines:

Person value1 = new Person { Name = "Chris Smith", Age = 31, CanCode = false};
Person value2 = with new Person() { .Name = "Chris Smith"; .Age = 31; .CanCode = false; };

Line #2 is only a few characters longer, but the syntax is more general than the object initializer in line #1. I can insert arbitrary code in the body, not just initialization code. And I can use the ‘with’ expression to write fluent code on existing OO APIs as shown above. Basically, if C#’s object initializer is a good idea, then my ‘with’ expression is a better idea, IMHO.

Because many programmers I respect are fond of Python, I’m forcing myself to learn it (again). On my previous attempts, I ran into some annoyances and dropped it. This time, I’m trying to solve some Facebook puzzles by gluing together existing libraries. I tried the Peak Traffic puzzle, which is to enumerate the maximal cliques in a directed graph. Some searching led me to an excellent library for graph problems called NetworkX. Some tinkering with the library hit the first problem: on very large complete graphs, I kept blowing the stack. That’s because v0.99 uses (perfectly normal) recursion to solve the problem but hits a stack limit in Python. In v1.0, the devs were forced to implement their own stack to get around this annoying limit. The second issue is that this is a branch & bound optimization problem, which can easily make use of many cores. Instead, the program maxes out at 50% cpu. This would have been trivial to parallelize in .NET 4.0 or Fortress with parallel blocks.

So I thought maybe Jython might solve both problems. Unfortunately, Jython (v2.5) is 4X slower than CPython running the exact same script on a very large input file. Next, Jython also sets the recursion limit to 1000 stack frames, so I ran into the same issue. I know I can raise it manually, but that doesn’t help when I write a program and don’t know how far I’m going to recurse. Why set an arbitrary limit? Just throw the same runtime exception when the platform runs out of space like every other language on the planet. Finally, I imported jsr166, an excellent threadpool library for Java. But on a simple fibonacci test, the performance was so much slower than the equivalent Java program that it simply wasn’t worth considering. It did, however, make use of more threads, though it rarely went over 80%. Maybe there’s a lock somewhere that’s slowing things down. In summary, Jython sucks and CPython is acceptable.