The sources of software complexity

How identifying where a certain kind of complexity comes from can help you avoid or isolate it

Our last Apropos was with Alex Engelberg. Check it out. Our next episode is with Paula Gearon on March 11. Please watch us live so you can ask questions.

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

---

The sources of software complexity

In No Silver Bullet, Fred Brooks defines the line of discussion around software complexity. He identifies two broad classes of difficulty in software: essential and accidental. The essential is irreducible. You cannot get rid of it because it is inherently part of the task of writing software. The accidental is anything that could in principle be eliminated and you still get working software.

He also identified complexity as an essential difficulty in software. Brooks writes: As software grows in size, it “is not merely a repetition of the same elements in larger sizes, it is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly.”

Brooks also defines two similar conjectures which I call the strong and weak conjecture. The strong conjecture is that less than 90% of time is spent on accidental difficulties. Because of this, we will never see an order of magnitude improvement in software development speed.

The weak conjecture is that there won’t be a single improvement (a silver bullet) that alone accounts for an order of magnitude improvement in software development speed within a decade.

It’s always puzzled me why he made the weak conjecture. It’s weaker in two ways. First, there’s the deadline of one decade. If essence is less than 10%, it shouldn’t matter how long you give it. It’s in principle impossible to improve by an order of magnitude. Second, he limits it to a single improvement acting alone. This one allows him to weasel out of arguments about two improvements acting in concert, which he does repeatedly in the follow-up essay, ‘No Silver Bullet’ Refired.

I believe Brooks was wrong in the strong conjecture. He suffered from a lack of imagination about what we could do with the increases in computer power available to us. We turned a lot of what he called essential difficulties into non-problems. Dan Luu has a pretty insightful essay on how techniques like version control and software testing have done wonders for turning the essential into accidental.

As for the weak conjecture, I think it’s arbitrary and worthless. Who cares how you count improvements, and why a decade?

Check out two podcast episodes I did about the essay: 1 and 2.

In Out of the Tar Pit, Ben Mosely and Fred Marks argue that complexity is the major problem in software development. They redefine essential complexity as that due to the domain (instead of that inherent in software development). Accidental complexity is complexity due to something else, such as from bad language design or poor programming practices.

They identify two major sources of complexity: state and control. State leads to complexity because software systems can get into a bad state. And control leads to complexity because of the ordering of execution. They then try to show how using functional programming and the relational model can eliminate most accidental complexity. It’s a very good read that demonstrates a lot of first-principles thinking.

However, the division between essential and accidental is fraught with ambiguity. If you can eliminate it, it’s obviously accidental. But if you don’t know how to eliminate it, you can’t really tell if you haven’t found a way to eliminate it yet, or if it is truly essential. It’s a helpful concept, but not that helpful.

I much prefer to clearly delineate the sources of complexity. Here is an example list of sources of complexity:

• Domain complexity (ex: rocket science is just hard)

• Business complexity (ex: the business wants to bill clients based on usage)

• Architectural/platform complexity (ex: we have to deal with client/server issues on the web)

• Language complexity (ex: we need the visitor pattern in Java)

• Code complexity (ex: we introduced an unnecessary indirection)

If we identify a certain problem in our code as coming from the design of the Java language, we can deal with it in two ways. One, we can avoid it by changing languages. If we can’t do that, then we can isolate it. We build indirection to route around the bad design. We add linters to help us steer clear of the bad design. We increase the testing to make sure we aren’t bitten by the complexity.

It’s not necessary to come up with an exhaustive, indisputable list of sources. Instead, it is important to know that one can identify the source of complexity to the granularity you find useful. We can ask “Why is the browser always getting out of sync with data on the server? Oh, it’s because we’re on the web and that brings with it some complexities.” After identifying it, we can then isolate it. Add to the list if you need more resolution into your problem.

Complexity is real. And it’s a killer. We should be constantly on the lookout for how to avoid it. And when we can’t avoid it, we should isolate it. Having a list of sources of complexity can help you understand it better. I’ve presented the list I use. Next week I want to talk about why poor data modeling is a major source of complexity.

What sources do you typically use to classify complexity?