People over process

A series of mixed metaphors about leadership

I have had an item on my TODO list for about 5 years: Kill Agile.

It’s not one that I think I will ever accomplish. I don’t think anyone will or can. No, I am afraid Agile will live forever.

You see, Agile has conveniently branded itself as the opposite of Waterfall. Anything that is not Waterfall is, by definition, Agile. If you invent a new thing, and it’s not similar to Waterfall, Agile will assume it into its moist, smothering fold.

If your new thing gets any traction, Agile coaches will pick it up. They will teach it in workshops. They’ll issue certificates dubbing participants a practitioner. And they will mangle it beyond recognition to fit right into the panoply of “Agile methodologies, practices, and processes.” And soon, your boss will send you, the creator of this new thing, to that workshop where you, the inventor, get taught how to do it.

Does Agile have a will of its own? No. But like Richard Dawkins explained in The Selfish Gene, environmental forces shape the competitive environment of memes, and Agile is a very fit meme for its environment. Agile, like the common cold—and now Covid—is here to stay. Here is Kent Beck in an interview on Refactoring:

But Agile, everybody wants to be Agile. Nobody wants to be rigid or inflexible. So nobody's ever gonna say that they're not Agile. […] As a brand, it's not defensible, because everybody's going to co-opt it. And that's exactly what happened.

I had hope for Agile. In 2010, I started working for a startup. I was the second programmer hired. For a year, we worked with very little process. We wrote down tasks, but often, we would individually get calls from the CEO directly to do some tasks for him. What we learned too slowly was that he would wheedle all of us in turn until someone said yes. Some of his requests were good, but many contradicted his own plan or the good judgment of the other programmers. That’s just how he was—impulsive, demanding, and weasely. It wasn’t working. So we got together as the programming team, and we decided how we would work and, most importantly, how we would interface with the rest of the company.

We landed on a rather mundane sprint and ticket system. It had one major feature: We would promise to work on the priorities for the sprint, but nobody could change them during the sprint. Give us one-week blocks where we don’t have to replan on an impromptu call. That’s all we asked for.

At first, it was a miracle. We felt organized and focused. We knew what we committed to. And no one was interrupting us. But after a year, our own process was working against us. We had erected our own prison and toiled in the CEO's chain gang. The backlog kept growing. We were under enormous pressure to get through this sprint to start the new batch of tasks that were already overdue before they were even written down. We made guesses and assumptions and threw quality over the fence to testers to say we were done and come up for air. We’d have to revisit it, but at least we took a breath before sinking back into the whirlpool of work. (Sounds like a certain dysfunctional car factory in Fremont, California.)

Here’s what I learned: A process cannot shield you from a sociopath. Nor can it protect you from the myriad lesser disorders of micromanagement, task mastery, control freakiness, or power hunger. The only thing that can do that is force of will, deployed continuously. Kent Beck again:

There's a power structure at work. And that power structure defends its own power.

Along came Agile, and for a moment it looked like a threat. So the power structure co-opted it. And now it wasn't a threat anymore.

Now the people who are extracting more than their share of the value can still extract more than their share of the value. And the people who are working away for crumbs, you know, picking up crumbs on the floor, they still are picking up crumbs on the floor. So as far as the power structure is concerned, everything's just fine. We're back to the status quo.

Agile has been co-opted. The programmers all know it. I sometimes wonder if the managers do. If they know it, they’re really committed to the pretense. That has been my experience, anyway, at every job. I either watch it happen, or it happened long since I joined: The programmers have turned themselves into a black box. Tickets come in one end, written and prioritized by the business squad, trickled down through layers of management, and commits come out the other end, automatically deployed. The black box never makes features fast enough. The managers bang harder on the box to stimulate the adrenals of the code monkeys inside, hoping the jolt will speed them along on their sprint tasks.

Let me give you a secret: It’s not the process. There’s no extra field on the Jira ticket that would make it all better if only the business people would fill in. No cute hand signals for pointing tickets will save us. No arrangement of stand-ups, pairing, retros, TDD, or improv games will help us. Nope. It’s the people. Us. As people, we too easily slip into the archetypal relationships we are familiar with. The best we can hope for is finding better people and better archetypes. We need leadership.

Oh, no! I can already see the comments: “That’s what the Agile Manifesto said! Individuals and interactions over processes and tools.” Yes! I know! And it doesn’t help. The Manifesto has no sway. Never have I seen a manager, well-trained by an Agile workshop, be swayed by citing the Manifesto. No matter how ironic it is that Agile coaches teach processes and tools, the Manifesto has no power. That ship has sailed. Let it go.

I want to go over three development process methodologies that have inspired me. For each one, I will show how it’s not the methodology that was helpful but the leadership itself.

1. Extreme Programming

Kent Beck made magic when he invented Extreme Programming (XP) at Chrysler. But the practices were the least of his alchemy. The main wizardry was his leadership.

Here’s Kent Beck again in the same interview:

[XP] says, we're gonna take the value that we intend to deliver, and we're gonna turn it into tiny slices. And we're going to use those slices to build a relationship with people who currently don't trust us. Business doesn't trust programmers. We make promises. We don't keep them—all the time for decades. It's not just our fault. It's our, our parents fault and their parents fault.

That's built this dysfunctional relationship, but the way to rebuild that relationship is to deliver functionality that other people care about on a regular basis.

[…]

The biggest fear on the business side is we're gonna make all this investment, we're gonna get nothing for it, and we won't know until the last minute and we're gonna look stupid. That's a horrible position to put somebody in. So I want them out of that position as after 2% of the budget of the project has been spent

I believed in this. I really did. The distrust between programmers and business has been palpable everywhere I’ve worked. I thought that we could make the business folks happy by de-risking a project by giving them small bits of value each sprint. One week is a small amount to risk on the next features. When we ship features, they’ll thank us! But it’s never been true. We deliver, but it’s never enough.

I don’t understand why Beck can make such a world-weary statement (Agile was doomed to be coopted) and follow it with such a hopeful one (we can heal the decades-long distrust). The distrust stems back way before that. To me, it looks like the same distrust that is evident in other industries. It goes back hundreds if not thousands of years in our culture. I suggest it follows the archetype of the master/slave relationship.

So if the practices can’t save you, what can? Heroics. Taking risks. Facing the beast. Unity.

In another interview, Beck said:

The first time I was asked to lead a team, I asked them to do a little bit of the things I thought were sensible, like testing and reviews. The second time there was a lot more on the line. I thought, "Damn the torpedoes, at least this will make a good article," [and] asked the team to crank up all the knobs to 10 on the things I thought were essential and leave out everything else.

I’ll break it down: He bravely took a risk. He made judgments about how to work better. He helped the team with their practices. I believe the XP practices helped, but they won’t save a project. The salvation was that he helped his team improve.

I’ve never had a manager help my team work better. Have you? Is it rare? Or have I just been unlucky? My experience has been more like this story from Beck (previous interview):

I managed a team, my boss came to me and said, “well, when is it gonna be done?” I said, “well, nobody knows what it is, so I can't give you an answer.” And he said, “yeah, no, I understand. I appreciate the principles, but when it's gonna be done?Yeah, yeah, just give me an answer. I have to give an answer.”

And my heart goes out to him because he had people demanding of him questions he couldn't possibly answer. But I wasn't gonna pass that along to my team.

As soon as I say a date, I've got downside, and I got no upside. If I hit that date, [pat on the head]. If I don't hit that date, boom. So I'm not going to put that, I'm not going to create those incentives for my team.

But it was incredibly stressful to work in that environment. I ended up with migraines and arthritis and ended up quitting. I don't want to work in that kind of environment.

I applaud him for protecting his team. But even Kent Beck can’t work magic in all environments. It’s not just about the one manager; the entire management stack, all the way to the top, needs to want you to improve. Unfortunately, heroics won’t always work.

Let me say this again: The process is important. You should work on it. But the details of the process are not what matters. You can’t import someone else’s process and have it work for you. What’s important is that you try to improve it—and that the organization supports that work. The meta-process of improving is what you’re after. And that’s what I’m calling leadership.

The kind of management I’ve seen (the bad kind) is more like an accountability function. The manager must commit their resources to a timeline and be ready to report progress at any moment. If they’re falling behind, they must answer why. The layers of management create a cascading pressure to deliver that bottoms out at the worker bees at the bottom of the tree, who have no one to pass the pressure on to. Reports of progress flow up, pressure flows down.

Yes, this is a kind of management, but it is more related to the slave driver’s whip or the coxswain’s rhythm. Perhaps that’s an exaggeration. We think of Taylorist management, with its stopwatch and process optimization, as dehumanizing. However, even the Taylorist manager used the stopwatch to improve the work process. They were present, watching you work, teaching you to do it better. If you want to dehumanize, try the constant message, Work faster!

Kent Beck was a leader and any success I attribute to that.

2. Shape-up

Okay, I admit it: The Shape-up process sounds great. It’s from 37Signals. Instead of chopping a feature into a bunch of little tasks, then doling those out and hoping the programmers can see the big picture so they can make better decisions, give the programmer a high-level picture of what you want, give them a time budget, and let them figure it out. I appreciate the trust and autonomy it bestows on the designers and programmers. Part of me also likes that it’s a refreshing alternative to the Scrum model that is so common today. However, I think we’re looking at the wrong meta-level.

The process appeals to me, but more appealing is the idea of working at a company whose boss is so thoughtful that they came up with a process that balances so many forces. Again—it’s the leadership, not the process.

Here’s a quote from Ryan Singer about how he co-developed Shape-up:

As I got more between sort of the design and program, with one leg on each, it became natural for me to be kind of this person in the middle. Jason was the design lead, David the programming lead and then as the company started to hire a few more people, I was this person who could kind of speak both languages and understand the meeting point, you know? So then when it came into questions of how do we get stuff built? How do we manage the process of building a feature? I kind of understood both sides enough to help facilitate that more.

But now the question becomes, well, how do we know if the thing we’re making is the right thing? Those were the conversations that I was having with Jason and David that felt like they were the most important conversations, you know? So then it became more natural for me to put more and more of my time into that side of things.

What stands out? The leaders of the company are even having this conversation. They’re asking hard questions about how they want to work, implementing their ideas, and learning from them.

The opposite is what I felt at a recent job. The VP of Engineering gave a milquetoast presentation, urging us to focus on quality. Here was the takeaway: Quality is important. I expect people at the VP level to be PhD-level experts in whatever they espouse. But all this guy could manage to rally his troops was a stupid truism. He was a leader in title, but he did not lead. Give me someone with a plausible opinion about how to forge quality, and I’ll follow them.

3. Programmer anarchy

Fred George has a management philosophy called Programmer anarchy that doesn’t have as much marketing behind it as Shape-up, but nonetheless, it appeals to me. When he describes it, it almost sounds like getting rid of managers. Replace them with more specific roles like mentor, ambassador (liaison with the business team), etc. Give more autonomy to the programmers. It sounds reasonable.

But that’s not where the magic comes from. At the risk of repeating myself, the magic is in the fact that someone is working on the work (layers 2 & 3 in Wiring the Winning Organization). That is, it’s not the particular changes or even the result of those changes that matter. What matters is the fact that he is intelligently and bravely making changes.

Here’s a slide and quote from a talk by Fred George (slide credit):

I had a team. It was a very bright set of programmers. But they didn't understand these techniques.

So in that first part of the project we finished 23 stories in 15 days. That was a certain pace. I took them offline for one week. “So, okay, excuse me. We have to stop here. You guys don't know how to write code in this easy to change style. It's gonna cost us in the long run. So we held the class for one week, then we started back up again with the same people.

We recover the cost of the class within 11 days. 11 day payback period on the class. And we kept that aggressive slope from then on. By the way it goes up again. That's because we added another pair. We can actually measure the productivity increase of adding some people.

By the way this is one of my favorite management techniques for managing projects. Let's just chart it out and see what it looks like. And this is what I showed the sponsors and everybody else. This is how fast we're working. If you want us to work faster, give us more people.

Here’s my summary:

1. He measured baseline productivity.

2. He identified a possible improvement.

3. He staged an intervention.

4. He measured the improvement.

5. He used the measurement to help justify further interventions.

Has anything like that ever happened to you? This kind of thing makes me think that we have programmer productivity all wrong. Maybe we can measure programmer productivity! It’s just that managers don’t normally use the information well. They use it to crack the whip, not stage an intervention. Or they use it to compare two teams, not show improvement due to interventions.

I can hear the objection now: I don’t want anybody—certainly not a manager—telling me how to work. Well, it doesn’t have to be that way.

The following is from the NUMMI story I brought up two weeks ago. Here’s a quote from the This American Life episode by Jeffrey Liker, who studied Toyota:

Then, the biggest surprise was, when they had those problems, afterwards, somebody would come up to them and say, “What are your ideas for improvement so we don't have that problem again?”

They couldn't believe that responsiveness. I can't remember any time in my working life where anybody asked for my ideas to solve the problem. And they literally want to know. And when I tell them, they listen, and then suddenly they disappear, and somebody comes back with the tool that I just described. It's built, and they say try this.

The manager can ask you for improvement ideas! Amazing!

And John Shook, one of the trainers from Toyota:

And people were crying on both sides. You had union workers, grizzled old folks that had worked on the plant floor for 30 years, and they were hugging their Japanese counterparts, just absolutely in tears.

And it might sound flowery to say 25 years later, but they had such a powerful, emotional experience of learning a new way of working-- a way that people could actually work together collaboratively as a team.

I think I would cry, too, if a manager collaborated with me.

Re-quoting Kent Beck, what I learned from my last team leadership experience was this: Damn the torpedoes! You can kick me out as the leader, but as long as I’m leading, we’re going to:

1. Take every request from above with a huge grain of salt. Why take them seriously if we know they’re dysfunctional? Especially beware of status reports, including Jira ticket status, that will only be used against you.

2. Work on the things we think are important. Management has their priorities (filtered through their own self-interest). We have ours. The less trust there is, the less we listen to them. If they want us to work on something, they’ll have to negotiate it the old-fashioned way.

3. Work in ways that I know make sense. If you don’t like them, unseat me from the role.

I would probably cause a mutiny (from the team) or be fired (from management). But at least my soul wouldn’t have been crushed. I didn’t get anything done when I led the more timid way. And maybe, if I had done it this way, I would have accomplished something.

Rock on!
Eric