Compromised visions are superior

Real constraints force better products

My office hours from last week went well. I answered a question about seeking Clojure jobs. And I got into a nice discussion about the REPL getting out of sync with your code. I also learned that I should have more questions ready to fill time. There are no Office Hours this week. But I’ll pick it up next week. Please ask a question here.

---

Compromised visions are superior

The other day I pulled up The Muppet Show, the classic 1970s show by Jim Henson and his crew. I had never seen it, and I wanted to show it to my girls. I was blown away by the creative energy emanating from pieces of felt.

The show has clever writing and fun characters. But what blew me away was how alive the characters felt. They ooze life. I know that one of their practices was to make the puppets have motion multipliers. For example, a character’s hair might bounce around as the puppet moves. A small motion of the hand makes lots of motion on the screen.

After a few episodes, the show's limitations were obvious: it had a few sets, a few formulaic set pieces, and very restrictive shot angles for some puppets. In some shots, you could even see the puppeteers’ arms. You never forget that they’re puppets. But you watch them anyway.

The creative energy got me thinking about the role of constraints in creativity and how this relates to building software. Perhaps one of the reasons so many projects overrun with scope, go over budget and schedule, and spiral out of control with complexity is that there aren’t enough constraints.

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.

— Fred Brooks, The Mythical Man-Month

We think of puppets as a product of the imagination. But they’re physical. There are constraints everywhere. A person has to physically control it. They need to remain hidden. There is a tension created by the two constraints. Limited control. So you get clever to make it all work. And the constraints form a conduit to channel the creative energy.

There’s a story in the filmmaking world about Jaws, the shark movie. The crew had a big, animatronic shark that they would use to shoot. But it didn’t work. It didn’t look real. So they reworked the script and shot the movie so the shark was an unseen threat, which is what the movie became famous for. According to the director, Steven Spielgerg, the robot's failure made the movie better.

Computing used to be constrained as well. I love this story about the development of the Atari 2600 game Pitfall. Most of the game's design was a clever set of choices to get something fun working on the limited hardware designed to run Pong.

I want to make it clear that constraints don’t make it good on their own. You still need human ingenuity to find a good design. But constraints help humans focus their imaginations. Otherwise, imagination overruns. We all know the feeling of architectural astronomy.

I’ve done it myself. I’ve over-engineered solutions. And I don’t mean adding extra reinforcement to a bridge. I mean making something way more complicated than it needs to be. While working in Haskell, I made a little DSL to query our documents. It worked great and cleared up lots of nested maps and filters. But it was too complicated. No one—including me 6 weeks later—could understand it. I got carried away. The problem was interesting and challenging—a puzzle. And part of me thought we needed it. But I never stopped to consider the complexity of what I was creating.

This kind of complexity is well known. Engineers blame themselves for the unneeded complexity they add to code. But are we entirely to blame? As we gain experience, constrain ourselves. And we write better code for it. But who sets up constraints for the product itself? Many projects go off the rails because the people controlling the product have no limits. You can always add more desired features. Always modify an existing feature. Product complexity kills projects more than code complexity.

That’s why I’m intrigued by Shape Up, the software design and engineering process by 37 Signals. Instead of a backlog of features, the product owners “shape” a project that makes a bet on a product direction. That bet is given a time budget of a maximum of six weeks. Then, the developers do their best to make something shippable within that time. Instead of a hammer putting pressure on the developers to get through more features, the developers are given the hammer to bang down the scope. The feature’s only worth six weeks, so make it fit.

I like this idea because it aligns the authority and responsibility. The product owners set a constraint on the project. That constraint flows to those building it, who constrain their work to fit time and budget.

When I put it like that, the project management models I’ve worked in seem dysfunctional. They amount to unconstrained backlogs of features. Limited only by their imaginations, the product owners jot down feature ideas faster than programmers can finish them. And then they wonder what’s taking so long. The product is constrained only by the organization's limitless desires. They want it fast, cheap, well-designed, secure, efficient, documented, and long-lasting. What they get is the opposite because they can’t choose.

I didn’t write this to complain about product management (and all the lovely managers above them). Instead, I wanted to say we shouldn’t be so hard on ourselves about how complex our code is. Sure, we can overdo it. But there are usually systemic problems that lead to it. Here are some tips that help me set up constraints that help, because not all constraints aid creativity.

Set up negative constraints

Your constraints should be limiting. If they just make a hard job harder, they won’t work. Here’s how not to do it:

We need this feature by next week AND it needs to be secure.

While that is constraining, you should phrase it negatively:

If we can’t make it secure by next week, we don’t do it at all.

Limits should be hard

Your limits must be hard, otherwise you can always move the bar. While this might work:

We need to keep our AWS spending down.

Try this instead:

We have to keep our AWS bill under $500/month.

Constraints should force a tradeoff

When you give yourself constraints, something needs to give. This is not good:

We need to fit all of these settings onto one screen.

You might need to get rid of settings:

We will only have settings that fit on one screen.

Unfortunately, I don’t know how to convince others, especially those above me, that if I do less of the features they want, the product will be better, despite trying. Maybe you’ll have better luck. But do look past developer laziness or affinity for complexity when you’re diagnosing why things are spiraling out of control. It’s usually not any one actor’s fault. The system gets the software it deserves.