Skyscrapers or mud huts

On habitability and the Curse of Lisp

Our last episode was with David Nolen. We talk about his development process, his origin, and his philosophy. The next episode is on Tuesday, April 22 with special guest Fogus. Please watch us live so you can ask questions.

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

---

Skyscrapers or mud huts

From 2005-2007, I lived in the West African country of Guinea. I loved the people, but their country had a sad story. After rejecting their French colonizers, they made a series of bad deals with other countries. While they started out as the hope of West Africa due to their wealth of natural resources, after 45 years of kleptocracy, the infrastructure had decayed. While Guinea’s neighbors saw increasing prosperity, theirs was declining.

One of the symptoms of their decline was that they were de-urbanizing. In other countries in the region, people were leaving the village, looking for a better life in the cities. But in Guinea, apparently the better life was in the village.

I would often ponder this on my infrequent trips to the capital, Conakry. Large portions of the streets were broken or missing, revealing mud that could swallow a small car during the rainy season. Utility poles were a rats-nest of spliced wires. Probably most of the wires were dead. Roofs leaked. Many inhabited ten-story highrises had no electricity or running water.

Contrast this to the village where I lived. Yes, they were poor, but I never saw squalor. What I came to discover was that the village was simply easier to maintain by people with few resources. You could make a house out of mud bricks and thatch it with straw you harvested from your field. If the roof leaked, you could climb up and patch it. If a wall crumbled, you could repair it.

In the city, the tall buildings require a huge number of resources to maintain. You need cranes and bulldozers and other diesel machines. You need cement and iron forges and other high-input materials. You need a functional supply chain to get the diesel and other inputs to you. And you need the highly skilled labor to do the repairs. Don’t forget a functioning economy to pay for those workers. In short, modern urban life, which is very attractive when it works, is hard to maintain.

Even if you aren’t talking about repairs, the cost of keeping the building working is expensive. You need to pump water up to the top floors. If the building is tall enough, you need to pump clean air up there as well. And you need to deal with all of the waste products of life. Things have to keep working or the top floors aren’t usable anymore.

When programming, sometimes I wonder if our high-power tools, like Clojure, cause us a similar issue. One person can create a wondrous indirection mechanism that solves the problem at hand. It’s like erecting a skyscraper. Sure, it might be easy to write the code—easier than construction. But there is a certain mental cost. The complexity of understanding it requires the mental equivalent of cranes.

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan

This quote from Kernighan (though I believe the sentiment comes from before his time). It expresses succinctly the idea I’m trying to convey. Clojure gives us the power to build things that are too clever to debug, just like a surge of political will can create a skyscraper that you don’t have the ability to maintain. Is maintaining a skyscraper twice as hard as building it?

I’ve worked on codebases where the cleverness was too much to maintain. The history of the project was a super smart programmer being brought onto the team to solve a tough problem. They built a brilliant solution. Then they left to use their brain somewhere more worthy of their mental capacity. The team enjoyed their skyscraper while it slowly decayed.

Richard Gabriel wrote a wonderful essay in Patterns of Software about the habitability of our code. In it, he argues that habitability is the important quality of good software. Programmers work inside the code, making adjustments, improvements, and expansions. We need to feel at home there—and like we can take care of it ourselves.

What is important is that it be easy for programmers to come up to speed with the code, to be able to navigate through it effectively, to be able to understand what changes to make, and to be able to make them safely and correctly. If the beauty of the code gets in the way, the program is not well written, just as an office building designed to win design awards is not well designed when the building later must undergo changes but those changes are too hard to make. — Richard Gabriel

A skyscraper has low habitability. It’s hard to adapt it to new purposes. It’s hard to adapt it as your needs change. But a mud hut is easy to change and expand. Our software needs that because we can’t plan it out ahead of time.

The genius’s plan for the software might have been good. But it was a master plan that only they could understand. They were the smartest programmer at the company and they used all of their capacity. How could anyone else keep up? Further, you can never fully communicate the “theory” behind the code. Just like Peter Naur recounted in Programming as Theory Building, it’s not just the raw intelligence. There’s also quite a lot of knowledge for how to build and extend the program:

In several major cases it turned out that the solutions suggested by group B [the extenders of the software] were found by group A [the original builders] to make no use of the facilities that were not only inherent in the structure of the existing compiler but were discussed at length in its documentation, and to be based instead on additions to that structure in the form of patches that effectively destroyed its power and simplicity. The members of group A were able to spot these cases instantly and could propose simple and effective solutions, framed entirely within the ex1 isting structure. This is an example of how the full program text and additional documentation is insufficient in conveying to even the highly motivated group B the deeper insight into the design, that theory which is immediately present to the members of group A. — Peter Naur

You have to learn to live in a skyscraper. And you have to learn to extend the software.

So we’ve got this triple whammy:

1. The mental capacity of the original “hero” developer.

2. The knowledge and plan they build while developing it (the “theory”).

3. The high-leverage Clojure programming language.

But here’s the thing: We can’t consider the power of Clojure to be a bad thing. That can’t make sense. If it were true, we’d want to move toward a much poorer programming language, maybe all the way back to assembly. I can’t believe that. Neither can we consider intelligence bad. Where would that lead us?

It seems to me that we have a problem of balancing power with wisdom. What I mean by wisdom is the skill of using half of one’s mental capacity to build the original software so that you have enough intelligence to debug it. It’s the ability to see that our plan is too ambitious, we need to grow the software more organically. It’s the ability to keep the theory simple so you can teach it to the rest of the team. Software can get complicated. The only solution seems to be humility.