Solve problems by taking away

Our natural tendency is to add things to solve problems, but every thing you add can create more problems

A new chapter of Runnable Specifications has passed the “valuable enough to read” threshold. This time, it’s the Time Lens, which talks about build a notion of time into your model. As always, I appreciate questions and comments of all kinds. I’m not sensitive so you won’t hurt my feelings if you tell me you hate it! I’m especially looking for people to pick apart my reasoning. These ideas are at the edge of my abilities so I need all the help I can get.

I was on the latest episode of defn. We talked about the Clojure community. I have a cautiously optimistic take. However, there is one thing I’d like to expand on, which I didn’t explain within the context of the conversation, but (with hindsight) I should probably expand on.

I never wished to discourage anyone from the Scicloj community. It’s a great group of folks and they are doing amazing and important work into scientific computing resources for the Clojure ecosystem. Everything I said on the podcast was the truth: I did try some of the libraries and I did get stuck. I think that speaks more to myself than to the libraries. I wasn’t motivated to work through the learning curve, and I’m bad at reaching out for help. I would rather read docs than open up a chat.

Daniel Slutsky reached out to me and we had a nice talk about the resources available. They’re a lovely bunch of people, and if you’re looking for an active community, you should consider joining. They’ll help you get started in a friendly way.

I’m also impressed by the community’s output. They have study groups to join. And the London Clojurians group often visits Scicloj topics. There are many Scicloj events posted in the #events channel on Clojurians.

So, I apologize for not expanding this context in the podcast. As a somewhat public figure, I need to be more mindful of my messages.

---

Solve problems by taking away

I have a major flaw: I will read some really amazing idea and take it to heart so thoroughly that I can’t imagine how I thought before I discovered it. I can’t relate to others who haven’t discovered the same thing. Here’s a story about how that flaw manifests:

Many years ago and a few jobs ago, our whole team was on a meeting. I brought up a problem I noticed with the effectiveness of our code review process. Despite pairing and two phases of code review (with lots of back and forth and revisions), major bugs were still getting into our software.

I was self-conscious that my bringing up problems sounded like whining. On that team, that was usually the reaction. But this seemed very important. We were spending a lot of time and energy context switching between our own tickets and the two phases of code review every PR received. And it wasn’t working.

I was expecting to discuss options. My preferred option was to do away with at least one of the phases of code review. It seemed to have devolved into a checkbox item that nobody took seriously.

However, I was shocked by the response from the CTO: “Okay, let’s add a third phase of code review.”

I was so shocked that I didn’t know what to say. That’s not what I wanted at all! Looking back, I was hoping for two things:

1. Acknowledgment that it was an actual problem (not my imagination or something that only affected me).

2. Analyzing the systemic effects leading to bugs slipping through.

I think I managed to stop the conversation before the process was made any worse. But my flaw was the cause of both the problems: 1) Why I sounded like I was complaining and 2) why my approach was so different from the CTO’s.

I sounded like I was complaining because I actively refused to propose a solution, even when asked. There’s a common cliché piece of advice: "Don't complain about something unless you have a solution to offer." But I was also taught in school that the best way to solve a problem is to get a good definition of it before jumping into a solution. I assumed they would have learned the same thing and were ready to pick apart the problem into a decent shape before demanding a solution. I mentioned that we should define the problem better, but it fell on deaf ears.

My approach to analyzing the problem as a system was informed by my reading into systems thinking. It was obvious that there was some positive feedback loop at play that caused us to slack off on the code review and let bugs through. We just had to find it and address it.

I say it was obvious, but, again, that was my flaw. I’ve gotten kind of cynical about this flaw over time. Here’s how I see it: Imagine that your knowledge exists in a multidimensional space. The more you read, the further away from everyone you get. And when you speak with knowledge from a field with specialized vocabulary, you sound more and more like you’re making stuff up. I love knowledge, but it can alienate you.

What I’ve learned to do since then is to analyze the problem myself, then develop a solution in writing that I later present to the group. No need to use specialized vocabulary. Just show the goods. It usually works. And it never seems like complaining. But it’s more lonely. Wouldn’t it be great to do that as a group?

However, I do want to address that first tendency to add instead of subtract. I thought: This second phase of code review takes a lot of time and bugs still get through. Let’s can it.

The CTO thought: This second phase of code review is letting bugs through and code quality is important. Let’s add a third one.

I believe (but cannot prove) that a third phase would make the bugs worse. We would have even more context switching and more reason to do a shoddy review. Besides, if we’re doing a shoddy job on two phases, why would a third one be any better? And with all of the pressures on us to deliver software, where would the extra time come from?

Anyway, I see this tendency everywhere. Here are some additive solutions that I’ve lived with:

• Configuration for all of the services (×{dev, staging, prod}) was getting hard to manage. So let’s write 30,000 lines of Clojure to manage it.

• Dependency versions for all of our services are getting out of synch. Let’s write a templating system to generate deps.edn files with standardized versions.

• We’re having trouble understanding the big picture of our microservices system, so let’s add more and more detail to each ticket so programmers can get work done without the big picture.

• Our microservices system is having trouble synching configuration options and secrets. Let’s add a new secrets service.

These extra parts sometimes do solve the problem. But they gum up the works and have their own perverse interactions. But there’s another way. That way is to rearrange the existing parts of the system. And often, you’ll see parts that are unnecessary. Get rid of them! A system with fewer parts is easier to manage. The biggest problem is that it’s a lot more work (and less fun) to clean up the system you’ve got than to add a fun new thing.

And here’s the other thing I learned: Often, the people who say they hate process the most make the worst processes. They hate thinking about process and they hate talking about process—more than they hate process itself. What you really want to do is fall in love with process—so that you can refine it and tune it and make it sing. But that’s a topic for another issue.

Rock on!
Eric

PS What perverse additive solutions have you witnessed?