Lisp's major problem

So I've progressed quite a bit into my lisp book. And I'm impressed. There really is no limit to the level of abstraction that you can reach. But that's exactly the problem with lisp. Though programmers are generally quite intelligent, I'd say any individual is still pretty dumb anyhow. These days, it's hard to get anything done unless you're good at working in a team, and good at communicating ideas, abstractions, patterns, or whatever stuff there is that has a certain complexity to it. The major issue here is the speed with which one absorbs this kind of information.

And that's why languages like Java are popular and quite successful for large projects: They are limited/optimized to the kinds of abstractions that the majority of programmers out there can easily absorb and use productively. One lisp programmer may get more done than 5 Java programmers, but 25 Java programmers might get the better of 5 lisp programmers. And where Java programmers bumped into Java's abstraction limits (which you bump into pretty soon), they turn to well documented Design Patterns. And those patterns are, being well documented, easy to communicate.

I'll be honest and tell you that I don't know. I can't know because I haven't got enough lisp experience (obviously). It's just a hypothesis. I'll test it when I've achieved Lisp Guru Level ;)

Some I/O performance stats

I was a bit curious about the performance stats of my flash drive, especially with regard to Vista's ReadyBoost (which I haven't tried using yet). The idea is that a flash drive has much better random access performance than a harddrive, whereas a harddrive's sequential throughput is superior, but irrelevant for swap. So I tested it on my laptop's harddrive and a 2GB USB flash drive, using this simple but very useful tool called CrystalDiskMark. There's something I didn't expect: the random 4K write test was slower on the flashdrive.

How is that going to improve performance as a swap device? Is it filesystem related maybe? Still, the random read stats are quite telling. And the write performance might be ok if the system swaps well in advance.

Bram and I were discussing how to merge sorted lists...

import time
import heapq
from random import randint

def report(*arg): print "%-20s: sorted %d lists with a total of %d elements in %d milliseconds" % arg

domain, listsize, nlists = 100000, 5000, 500

lists = [ sorted([ randint(0, domain) for j in range(randint(1,listsize)) ]) for i in range(nlists) ]
indices = dict([ (i, 0) for i in range(nlists) ])

start_time = time.time()
result = sorted(sum(lists, []))
end_time = time.time()
report("primitive method", nlists, len(result), int((end_time - start_time) * 1000))

heap = [ (lists[list_index][0], list_index, 0) for list_index in range(nlists) ]
heapq.heapify(heap)

start_time = time.time()

result = []
while len(heap):
  min_value, min_list_index, min_value_index = heapq.heappop(heap)
  result.append(min_value)
  min_value_index += 1
  if min_value_index < len(lists[min_list_index]):
    heapq.heappush(heap, (lists[min_list_index][min_value_index], min_list_index, min_value_index))

end_time = time.time()
report("better method", nlists, len(result), int((end_time - start_time) * 1000))

Reading some of these "Road to lisp" stories (thx, Lars) does make me wonder... How well do people know a programming language before moving on to a new one? The stories do not always make sense. I mostly got upset by some people claiming ruby is better than python, whereas I *know* ruby, and I *know* it isn't. I would like people to demonstrate to me where ruby provides a truly higher level of abstraction than python. Please, enlighten me.

Anyway, I expect to raise to a new level of abstraction in lisp (I believe I maxed out in python a while ago). I also fear it won't come with as many "batteries included" as python.