Let me first say I'm a big fan of your work and thank you for sharing it with us - I regularly think about the concepts in Grokking Simplicity.
I'll also add to the merits of "Ensemble programming". The company that I have been working for the past 3 years uses this for the release of new features and it works quite well. People frequently feel engaged in what they're doing, that they learn more and are overall more satisfied in their work.
For its negative sides, I'll reinforce the point that was made before that having a solid process is absolutely essential. Just gathering a bunch of people together and say "Deliver me this" won't work.
Besides the danger of personal conflicts and social fatigue I'll add that the opposite is also true. People can become too friendly with each other making it easier to forget disciple and work ethic.
Having said that, I want to question a couple of points.
1. Throughput
I agree with you that using throughput as a metric is a much better indicator of a productive team. But, it must be stressed that it means quality throughput. Not just churning out features for the sake of it.
I think you hinted at this for your next article but I believe quality is the metric that trumps every other.
Specially since in the Software industry, usually the problem isn't delivery time - teams output software like crazy when compared to other fields. I think the biggest problem affecting our industry is that most software is ridden with defects.
So yes, throughput is important but only if quality does not drop.
2. Documentation
I feel like you didn't give enough credit to the effect documentation can have in the productivity of a team. I worry that by saying "More and better documentation helps. But how are those efforts going for you?" will serve as a cop out for people to drop documentation altogether.
When done right, documentation is effectively "Ensemble programming" on steroids. The ensemble being the documents you've produced.
Now, I agree that as an industry, we don't know yet how to write precise and complete documentation that can be leveraged for huge productivity gains. But I believe it can be done.
We can look to Dave Parnas and its work in the Software Reduction Cost project for the US navy. 3 years later and the documents they produced were still being actively maintained, read and most important understood. This was in the 1970's. We could do much better now.
It is my hope that the industry starts to move towards a true engineering and good documentation is always and everywhere the hallmark of good engineering.
Once again, thank you for the articles and I'm looking forward for the new book.
Quality is super important. Thanks for bringing that up. I believe that quality is free in software, as it tends to be in manufacturing. That is, quality takes effort, but it's less than the cost of dealing with mistakes. Also, the DORA studies show that companies with high deployment frequency also have low defect rates.
As to the importance of documentation, I think it's really complex. I've never experienced working at a company with good docs for internal software. Not that we didn't try to have good documentation, it just never got to where we felt like it was good enough. Good documentation seems to be a legend. Perhaps you've had good experience with it.
I don't mean to give people a "cop out". But even that wording hints at the issue I see: No matter how much effort you put into documentation, it's never good enough. There's a tremendous amount of guilt around it. But the guilt is unwarranted.
I worked on a team that had rather good documentation as far as docs go, but I was terrible at using them. Not my fault--I was new. And not the documentation's fault. It was just the sheer quantity of things you had to know.
I remember very frequently searching through 5 different sources of documentation for multiple terms that people were asking about in our support channel. Finally, after 20 minutes of searching, I asked my colleagues and they told me yet another term to search and what doc to search in. Basically, they found it in 10 seconds. It was there. Well documented. Complete. It was just that you had to know the right word to find it. (Of course I added the terms I had thought to search for to the text so the next lucky person could get there faster.)
If we had been ensembling, we would have found the docs immediately and nobody would be interrupted. That's my point. Well, that, and the other idea that a complicated system needs more documentation. (See: https://ericnormand.substack.com/p/complexity-begets-complexity)
Now, on that same team, we did have a lot of luck diagraming our code together. We would all be on a call, sharing an excalidraw, and it was glorious. It was kind of like ensemble documentation, but the act of building it was more important than the artifact we built.
Are there case studies where ensemble programming has been studied and evaluated ? I didn’t find any with a web search, though there are books and articles on the topic.
seemed to have grown stale, the popular stories on Hacker News a >= 6 years old, more recent queries on Hacker New garner less than a dozen comments, I'm curious what the barrier to wide spread adoption is
First off, let me admit my bias. I would never work for a company doing mob programming. I have quit companies over required pair programming policies. I am a special snowflake. Emphasis on the flake. I don't exist solely for the companies benefit and efficiency. My like and dislkes matter. It might be the perfect fit for someone else.
Mob programming probably isn't for everyone, just as it might not be best for a particular company given its size and codebase. However, I still think it's more efficient and companies should consider it. If efficiency is important to them, they might prioritize ensemble over retention, especially in today's layoff-laden climate.
Regarding the selection bias, yes, it's there. I hope it was clear that it wasn't part of the main argument, which related to the efficiency and the fact that ToC can explain it.
And as to the three "Common Pitfalls", I'll address them here in case anyone doesn't want to read the linked article. They seem rather minor. The first and third state that individuals can sabotage the productivity if they don't like the ensemble method. That can happen with any method. And the second one is laughable: contagious disease from working in close proximity. Basically your whole team could get sick at the same time from breathing droplets and sharing a mouse.
Not that I don't think there are pitfalls, just that the article doesn't list any good ones.
'Here is the negatives we edited: " It was not all positive. Finding a smooth way of working took a long time, and we lost team members along the way, presumably at least partly because they did not enjoy this. Personality differences became painfully obvious, and a lot of time was spent talking openly and honestly to each other on how to stop rubbing each other the wrong way. ...'
But as you say individual conflict aren't unique to mob programming. (Again, another personal bias, I'm often amused by what I see in groups claiming to be some variant of AGILE (mob is featured at agilealliance.org) and, to my mind, oblivious of the first core value "Individuals and interactions over processes and tools" and get too focused on replicating some process)
If you do follow up, I'd be more interested in what is holding it back (if anything)
I'll lead with "if anything", since there seems to have been some branding effort away for "mob programming", so it might be flourishing under some name I didn't search Hacker News, but "Ensemble Programming" brought up even few search results than "mob programming"
It has been more than a half a dozen years since its most popular discussions.
?Is it to far ahead of its time? Tooling, infrastructure, familiarity lacking ?
?Are there structural issues? Are there corporate hierarchy structure of promotions and raises, operating separate from outcomes, working against it?
?Too many folks like me, stuck in a rut, unwilling to give up on the methodologies of the past?
?Is it tied to your Alan Kay article. What if you invent the future? What then, does, lack of a group of charismatic evangelists leave it unnoticed?
?Is popularity on Hacker News even a metric for concern, if so what is the right metric, is some measuring it? If no why not (Structural problems in the Academic Industrial Complex?)?
Without giving too much away about a future article I will write, I think our industry still doesn't know how best to work. We're flailing with new processes and practices. I look at it similar to how the Ford assembly line managed to transform the auto industry because of the huge efficiency gains. Then Taiichi Ohno (at Toyota) did it again with TPS. Both saw a huge step change in efficiency.
The Agile movement cleaned the slate (with its People over Process) but didn't replace it with anything concrete. You can't always rely on people to figure it out. Companies glommed onto whatever was being sold with an Agile label because they needed some sort of process. But it all sucks.
What we need is a strong leader--like Ford or Ohno--who takes a first-principles approach and makes it happen. And, BTW, that's not me! I'm no good at management. All I'm trying to do is to weave threads together that I see.
Thanks for the links! It really sounds interesting (and fun) and I would certainly try it in the right circumstances, - though I can imagine ways for it to go wrong too, with the wrong mix of people.
I wonder also if a more asynchronous and remote form of “ensemble” might also work. Eric used to offer weekly Clojure programming challenges, and when people started tweaking and refining each other’s code the result could be high quality, tight and readable code that all the contributors understood - a highly desirable outcome like that claimed by synchronous in-person ensemble.
Remote could definitely work. I've heard of people sharing a screen on teams. There are mechanics that need to be worked out, but the principle is the same remotely.
As to making it asynchronous, I think you'd lose a lot of the benefits. Part of the magic is in the immediate and low-cost communication.
Hi Eric & all,
Let me first say I'm a big fan of your work and thank you for sharing it with us - I regularly think about the concepts in Grokking Simplicity.
I'll also add to the merits of "Ensemble programming". The company that I have been working for the past 3 years uses this for the release of new features and it works quite well. People frequently feel engaged in what they're doing, that they learn more and are overall more satisfied in their work.
For its negative sides, I'll reinforce the point that was made before that having a solid process is absolutely essential. Just gathering a bunch of people together and say "Deliver me this" won't work.
Besides the danger of personal conflicts and social fatigue I'll add that the opposite is also true. People can become too friendly with each other making it easier to forget disciple and work ethic.
Having said that, I want to question a couple of points.
1. Throughput
I agree with you that using throughput as a metric is a much better indicator of a productive team. But, it must be stressed that it means quality throughput. Not just churning out features for the sake of it.
I think you hinted at this for your next article but I believe quality is the metric that trumps every other.
Specially since in the Software industry, usually the problem isn't delivery time - teams output software like crazy when compared to other fields. I think the biggest problem affecting our industry is that most software is ridden with defects.
So yes, throughput is important but only if quality does not drop.
2. Documentation
I feel like you didn't give enough credit to the effect documentation can have in the productivity of a team. I worry that by saying "More and better documentation helps. But how are those efforts going for you?" will serve as a cop out for people to drop documentation altogether.
When done right, documentation is effectively "Ensemble programming" on steroids. The ensemble being the documents you've produced.
Now, I agree that as an industry, we don't know yet how to write precise and complete documentation that can be leveraged for huge productivity gains. But I believe it can be done.
We can look to Dave Parnas and its work in the Software Reduction Cost project for the US navy. 3 years later and the documents they produced were still being actively maintained, read and most important understood. This was in the 1970's. We could do much better now.
It is my hope that the industry starts to move towards a true engineering and good documentation is always and everywhere the hallmark of good engineering.
Once again, thank you for the articles and I'm looking forward for the new book.
Cheers
Hey!
Thanks for the kind words.
Quality is super important. Thanks for bringing that up. I believe that quality is free in software, as it tends to be in manufacturing. That is, quality takes effort, but it's less than the cost of dealing with mistakes. Also, the DORA studies show that companies with high deployment frequency also have low defect rates.
As to the importance of documentation, I think it's really complex. I've never experienced working at a company with good docs for internal software. Not that we didn't try to have good documentation, it just never got to where we felt like it was good enough. Good documentation seems to be a legend. Perhaps you've had good experience with it.
I don't mean to give people a "cop out". But even that wording hints at the issue I see: No matter how much effort you put into documentation, it's never good enough. There's a tremendous amount of guilt around it. But the guilt is unwarranted.
I worked on a team that had rather good documentation as far as docs go, but I was terrible at using them. Not my fault--I was new. And not the documentation's fault. It was just the sheer quantity of things you had to know.
I remember very frequently searching through 5 different sources of documentation for multiple terms that people were asking about in our support channel. Finally, after 20 minutes of searching, I asked my colleagues and they told me yet another term to search and what doc to search in. Basically, they found it in 10 seconds. It was there. Well documented. Complete. It was just that you had to know the right word to find it. (Of course I added the terms I had thought to search for to the text so the next lucky person could get there faster.)
If we had been ensembling, we would have found the docs immediately and nobody would be interrupted. That's my point. Well, that, and the other idea that a complicated system needs more documentation. (See: https://ericnormand.substack.com/p/complexity-begets-complexity)
Now, on that same team, we did have a lot of luck diagraming our code together. We would all be on a call, sharing an excalidraw, and it was glorious. It was kind of like ensemble documentation, but the act of building it was more important than the artifact we built.
Anyway, thanks for the discussion!
Are there case studies where ensemble programming has been studied and evaluated ? I didn’t find any with a web search, though there are books and articles on the topic.
here is site with some more links to narratives of experiences.
https://www.remotemobprogramming.org/
still seems mainly subjective experiences, not studies evaluating metrics like costs/benefits/efficient gains.
Given the above sites has changed little in the last 4-5 year. and other like,
https://mobprogramming.org/ ,
seemed to have grown stale, the popular stories on Hacker News a >= 6 years old, more recent queries on Hacker New garner less than a dozen comments, I'm curious what the barrier to wide spread adoption is
First off, let me admit my bias. I would never work for a company doing mob programming. I have quit companies over required pair programming policies. I am a special snowflake. Emphasis on the flake. I don't exist solely for the companies benefit and efficiency. My like and dislkes matter. It might be the perfect fit for someone else.
The closest I've found to a case study is:
https://www.agilealliance.org/resources/experience-reports/mob-programming-agile2014/
Eric's newsletter article does not address "Common Pitfalls" listed here: https://www.agilealliance.org/glossary/mob-programming/
The selection bias in "There are companies out there ensemble programming. They like it!" is pretty obvious.
Hey Mark!
Thanks for all the great comments.
Mob programming probably isn't for everyone, just as it might not be best for a particular company given its size and codebase. However, I still think it's more efficient and companies should consider it. If efficiency is important to them, they might prioritize ensemble over retention, especially in today's layoff-laden climate.
Regarding the selection bias, yes, it's there. I hope it was clear that it wasn't part of the main argument, which related to the efficiency and the fact that ToC can explain it.
And as to the three "Common Pitfalls", I'll address them here in case anyone doesn't want to read the linked article. They seem rather minor. The first and third state that individuals can sabotage the productivity if they don't like the ensemble method. That can happen with any method. And the second one is laughable: contagious disease from working in close proximity. Basically your whole team could get sick at the same time from breathing droplets and sharing a mouse.
Not that I don't think there are pitfalls, just that the article doesn't list any good ones.
Rock on!
Eric
Perusing various Hacker News threads, I felt like I often encountered like:
https://news.ycombinator.com/item?id=11854541
'Here is the negatives we edited: " It was not all positive. Finding a smooth way of working took a long time, and we lost team members along the way, presumably at least partly because they did not enjoy this. Personality differences became painfully obvious, and a lot of time was spent talking openly and honestly to each other on how to stop rubbing each other the wrong way. ...'
But as you say individual conflict aren't unique to mob programming. (Again, another personal bias, I'm often amused by what I see in groups claiming to be some variant of AGILE (mob is featured at agilealliance.org) and, to my mind, oblivious of the first core value "Individuals and interactions over processes and tools" and get too focused on replicating some process)
If you do follow up, I'd be more interested in what is holding it back (if anything)
I'll lead with "if anything", since there seems to have been some branding effort away for "mob programming", so it might be flourishing under some name I didn't search Hacker News, but "Ensemble Programming" brought up even few search results than "mob programming"
It has been more than a half a dozen years since its most popular discussions.
?Is it to far ahead of its time? Tooling, infrastructure, familiarity lacking ?
?Are there structural issues? Are there corporate hierarchy structure of promotions and raises, operating separate from outcomes, working against it?
?Too many folks like me, stuck in a rut, unwilling to give up on the methodologies of the past?
?Is it tied to your Alan Kay article. What if you invent the future? What then, does, lack of a group of charismatic evangelists leave it unnoticed?
?Is popularity on Hacker News even a metric for concern, if so what is the right metric, is some measuring it? If no why not (Structural problems in the Academic Industrial Complex?)?
Without giving too much away about a future article I will write, I think our industry still doesn't know how best to work. We're flailing with new processes and practices. I look at it similar to how the Ford assembly line managed to transform the auto industry because of the huge efficiency gains. Then Taiichi Ohno (at Toyota) did it again with TPS. Both saw a huge step change in efficiency.
The Agile movement cleaned the slate (with its People over Process) but didn't replace it with anything concrete. You can't always rely on people to figure it out. Companies glommed onto whatever was being sold with an Agile label because they needed some sort of process. But it all sucks.
What we need is a strong leader--like Ford or Ohno--who takes a first-principles approach and makes it happen. And, BTW, that's not me! I'm no good at management. All I'm trying to do is to weave threads together that I see.
Thanks for the links! It really sounds interesting (and fun) and I would certainly try it in the right circumstances, - though I can imagine ways for it to go wrong too, with the wrong mix of people.
I wonder also if a more asynchronous and remote form of “ensemble” might also work. Eric used to offer weekly Clojure programming challenges, and when people started tweaking and refining each other’s code the result could be high quality, tight and readable code that all the contributors understood - a highly desirable outcome like that claimed by synchronous in-person ensemble.
Hey Mark!
Remote could definitely work. I've heard of people sharing a screen on teams. There are mechanics that need to be worked out, but the principle is the same remotely.
As to making it asynchronous, I think you'd lose a lot of the benefits. Part of the magic is in the immediate and low-cost communication.
Rock on!
Eric
this medium article has some links to academic research for supporting neurodiversity in pair programming. Seems like it would be a good first approximations for setting up for success a mob. https://sorrelharriet.medium.com/supporting-neurodiversity-in-paired-programming-8b250d2b5cab