27 Dec Agile 20 Years On : The Failed Rebellion
The Agile Manifesto turned 20 this year, and there are two facts that seem self-evident:
1. Agile, as a label, won; nobody wants to be called non-Agile.
2. Agile, as it is practiced, falls woefully short of the revolutionary ideas of its founders.
How did we get to this point? Everybody says they do Agile and yet almost nobody is Agile.
Hence The Manifesto
In February, 2001, a group of seventeen expert software practitioners met at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah. Over the course of a few days of discussion and debate, they collaboratively wrote the “Agile Software Development Manifesto”.
* I don’t know the personal history of every signatory, but of those I know, they were all either still actively writing code or had a long and recent history of it.
The first point to highlight is that these were practitioners. They weren’t project managers or CTOs or VP Engs. They were developers, programmers, scientists, and engineers. They were still slingin’ code* and working with their stakeholders to solve problems. This is important.
The other point is that the Agile Manifesto wasn’t created in a vacuum. Many of these people already had a methodology they had created and/or were proselytizing. I might have my timing slightly off, but I think all of these methodologies pre-existed “capital A Agile”: Extreme Programming, Scrum, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming. I know Schwaber and Sutherland spoke publicly about Scrum in 1995, and Beck and Jeffries started talking about Extreme Programming (XP) in 1996, I think.
Everyone in this group had deep experience writing software, and they were all looking for an alternative to documentation-driven, heavyweight software development processes that ruled the day. The heart of the manifesto was four statements of value:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more
A New Hope
From our vantage point in 2021, it’s easy to take so much of modern software development practice for granted, but in 2001, these ideas were wildly radical.
What do you mean you’re going to start building the software before you’ve gathered all of the requirements and estimated every piece of functionality? That’s insane!
The important piece that gets forgotten is that Agile was openly, militantly anti-management in the beginning. For example, Ken Schwaber was vocal and explicit about his goal to get rid of all project managers – not just get the people off his projects, eradicate the profession from our industry.
Agility & PMI
We have found that the role of the project manager is counterproductive in complex, creative work. The project manager’s thinking, as represented by the project plan, constrains the creativity and intelligence of everyone else on the project to that of the plan, rather than engaging everyone’s intelligence to best solve the problems.
Ken Schwaber, manifesto signatory and Scrum co-creator
Scrum Masters had almost no authority, no votes on issues. They were servant leaders, helping shield and unblock the team but not managing the team. XP was similar. If I recall correctly, XP originally had trackers and coaches, which had a similar facilitating, supportive vibe.
Alistair Cockburn, manifesto signatory and creator of the Crystal methodology and hexagonal architecture, had a marvelous, insightful thread on this recently, including this idea (paraphrased):
Scrum struck a magnificent bargain in hostile territory:
- Management got 12 times per year to change direction in any way they wanted, after each sprint.
- Teams got an entire month of total quiet time with no interruptions or changes of direction to do heavy thinking and working.
- Teams got to announce what they could and couldn’t do in the month, with no management interference in their bid.
No executives ever got a better deal.
No development team ever got a better deal.
I’m a certified Scrum Master, working on Agile teams for 15+ years, and I’ve read many of the popular books in the space. I’ve never seen anyone frame the idea so explicitly and succinctly (again paraphrasing Cockburn):
Scrum was invented to function in hostile environments. It’s a contract between hard-pushing managers and developers needing time to think and explore.
The Empire Strikes Back
In some ways, Agile was a grassroots labor movement. It certainly started with the practitioners on the ground and got pushed upwards into management. How did this ever succeed?
It’s partially due to developers growing in number and value to their businesses, gaining clout. But the biggest factor, in my view, is that the traditional waterfall approach simply wasn’t working. As software got more complicated and the pace of business accelerated and the sophistication of users rose, trying to plan everything up front became impossible. Embracing iterative development was logical, if a bit scary for managers used to planning everything.
I remember meetings in the mid-2000s where you could tell management wasn’t really buying it, but they were out of ideas.
What the hell, let’s try this crazy idea the engineers keep talking about. We’re not hitting deadlines now. How much worse can it get?
Then to their surprise, it started working, kind of, in fits and starts. Teams would thrash for a while and then slowly gain their legs, discovering what patterns worked for that individual team, picking up momentum. After a few sprints, you’d start to see the real power of prioritizing working software, collaboration, taking time to inspect and adapt, and all the rest.
In the course of about 5 years, Agile went from a methodology that you’d heard of but probably not used to something everybody did. In 2005, I was switching jobs, and I remember the fact that I knew a bit about Agile and TDD was a real differentiator. By 2010, it was assumed modern software teams were doing some flavor of Agile. At least, this was true for my bubble in the consulting world; large enterprises always move slower.
🎉🎉🎉 We did it! We won! Congratulations everybody! 🎉🎉🎉
And that’s the end of the story. You can go ahead and close the browser tab.🥳
Winning was easy, young man. Governing’s harder.
George Washington, as portrayed in Hamilton
Unfortunately, like many revolutions, the history of Agile didn’t unfold how the founders envisioned.
- It turns out that prioritizing individuals and interactions is a hard concept to sell. It’s much easier to sell processes and tools.
- It turns out that working software is harder to produce than unrealistic plans and reams of documentation.
- It turns out that collaborating with customers requires trust and vulnerability, not always present in a business setting.
- It turns out that responding to change often gets outweighed by executives wanting to feel in control and still legitimately needing to make long-term plans for their businesses.
“It Turns Out That Agile Done Poorly Often Feels Like Chaos”
That doesn’t mean the four values are wrong. It just means this whole thing takes some effort to get right, some courage to accept that software is inherently messy and chaotic at times. You have to understand and believe that if you keep learning and adapting and improving and shipping, you’ll eventually get to a much better place, a much more honest and realistic and productive place than you ever could with waterfall.
The Agile movement is not anti-methodology, in fact many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of XP or SCRUM or any of the other Agile Methodologies as “hackers” are ignorant of both the methodologies and the original definition of the term hacker.
Jim Highsmith, History: The Agile Manifesto
Those are important points. We still need to plan and document and have rigor in Agile. It’s about balance. However if your organization is struggling with an Agile transformation, drowning in chaos, you’re going to leap when someone offers you a lifeboat in the form of certifications and processes and tools. Executives understand processes and tools much more than they grok self-organizing teams.
Return of the Rogue One?
This is where my three-act structure breaks down a bit, because unfortunately, I don’t see the plucky rebels coming back on this one, at least not under the Agile label.
It’s been overrun with tool vendors and process consultants and experts making promises that can never be delivered. This is how we’ve ended up with SAFe and Scaled Scrum and all of the other enterprise Agile flavors. These frameworks weren’t created with malicious intent, and they probably even have some value in the right context. But I wouldn’t call them Agile. Trying to scale a methodology that focuses on individuals and interactions will inevitably lead to problems – and erode the original value of the methodology.
This is how we’ve ended up with this famous 2018 piece by Ron Jeffries, manifesto signatory and XP co-creator.
Developers Should Abandon Agile
When “Agile” ideas are applied poorly, they often lead to more interference with developers, less time to do the work, higher pressure, and demands to “go faster”. This is bad for the developers, and, ultimately, bad for the enterprise as well, because doing “Agile” poorly will result, more often than not, in far more defects and much slower progress than could be attained. Often, good developers leave such organizations, resulting in a less effective enterprise than prior to installing “Agile”.
This is how we’ve ended up with the famous 2014 piece by Dave Thomas, manifesto signatory and Pragmatic Programming co-creator.
Agile is Dead (Long Live Agility)
The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products… Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term.
So I think it is time to retire the word “Agile.”
Retro
The really sad part of this for me is seeing young developers denigrate Agile and think of it as a means for management to extract unrealistic promises and push dev teams to work crazy hours. I get it. The only Agile they’ve ever known has been a mechanism of control imposed on them, not a tool of self-empowerment they joyously embraced. But I hope kicking off some discussions around the history and original vision can help us remember how things were supposed to go.
The good news in all of this is that the principles of the Agile Manifesto are as wise and relevant today as they were 20 years ago. And even supposed apostates like Jeffries and Thomas still think that.
Jeffries in the article mentioned above, says, “However, the values and principles of the Manifesto for Agile Software Development still offer the best way I know to build software, and based on my long and varied experience, I’d follow those values and principles no matter what method the larger organization used.”
I agree.
It’s not hip or cool to talk about Agile now. Agile is boring. Everybody does Agile, right? Now is the perfect time to reflect on the last 20 years and ask ourselves a few questions:
What went right?
What went wrong?
What do we want to do differently next time?
A few of us at Simple Thread who lived through the revolution plan to reflect on each of the original 12 Agile Principles over the coming months, contextualize their original meaning, and consider their value in the current milieu of software development.
My hope is that by studying the founding principles, we can learn from the past, and in the words of Dave Thomas, we can retain our agility even if we choose to abandon “Agile”.
(https://www.simplethread.com/agile-at-20-the-failed-rebellion/)
IT project management success statistics
If you were to ask ten different researchers what percentage of IT projects were successful, you would get as many different answers. Yet, researchers can agree on one thing: the number of successful projects is lower than is ideal, but it’s possible to change this.
Only 0.5% of IT projects are truly ‘successful’
McKinsey found that, regardless of project size, 59% of all IT projects are completed within budget, 47% are completed on time, and 44% deliver the intended benefits. This is good, but it isn’t the full story.
Just one in every 200 IT projects meets all three of these measures of success, and only one in every 14 is delivered on time and on budget. Projects that fail to meet one or more measures exceeded their budgets by 75%, overran their schedules by 46%, and generated 39% less value than predicted, on average. Yikes.
While most project teams don’t experience colossal cost overruns, one in six IT projects looked at in one Harvard Business Review report had a cost overrun of 200%, on average. In one study on project success, the most extreme case saw a project come close to a 700% overrun.
IT projects with budgets over $15 million pose the largest risk
Businesses can run into trouble when their large IT projects — those with budgets over $15 million — fail. McKinsey and the University of Oxford found that, on average, these projects run 45% over budget and 7% over time — yet also deliver 56% less value than predicted.
Plus, the longer a project is scheduled to run, the more likely it’ll run over time and budget. Each additional year spent increases cost overruns by 15%.
These risks are not equally balanced across IT projects types. Software projects run a higher risk of cost and schedule overruns compared to nonsoftware projects, yet they see an average benefits shortfall of just 17% compared to 133%. (www.runn.io)
Why IT projects fail: project failure statistics
Why do so many IT projects fail? Project failure comes down to poor project management. McKinsey and the University of Oxford asked IT executives to identify the cause of their project cost overruns, distributing the 45% of failures as follows:
- Missing focus (13%)
- Content issues (9%)
- Skill issues (6%)
- Execution issues (11%)
- Unexplained causes (6%)
(www.runn.io)
At its heart, The Agile Manifesto is
… a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we would want to work.
To buy into the agile engineering processes also means that you buy into the cultural ideals that surround and encompass agile. The engineering processes of test often, ship working code, integrate continuously, etc. are only in service of a model that supports the kinds of engineering organizations where self-organizing teams thrive, people love the work they are doing, and trust in your fellow engineer is paramount.
Today, most software organizations would say they’ve adopted agile (or at least some bastardized form of agile). Yet, I would argue that the industry has missed the mark on adopting the true heart of agile.
We find ourselves in a very similar situation to the 2001 dot-com bubble collapse; increasing interest rates, massive droves of software talent being let go in some of the worst ways possible, priorities shifting away from collaboration and psychologically safe engineering organizations and moving more towards efficiently delivering products.
It seems that the industry is pushing back on the agile ideals.
And my worst fear is that the Agile Manifesto changed nothing with more and more of the sector being somewhere I wouldn’t really want to work.
I’ll be the first to admit: agile takes time, energy, and dedication. It’s not always easy. Retrospectives, planning meetings, user research (and so on) all take time and engineering resources. Time not spent coding or working directly on products. But if there’s anything the last 20 some years of this tech boom market have shown us, where agile was adopted by the industry very broadly, it’s that agile works. Happy engineers who love their work deliver amazing solutions and, in the long term, make for more sustainable organizations that can continue to ship stable, and innovative products that customers love. (John McBride 2023)
If you’ve been reading this so far and find yourself saying “Huh, the dot-com bubble sort of looks like the tech-market today”, that’s because it is. From an economic standpoint, culture perspective, and engineering process view. It seems when the economic goings get rough, engineering organizations get worse.
A primary, high profile example of this is the Elon Musk takeover of Twitter: most engineers have been laid off, there have been multiple lengthy outages, rumors of badly broken internal infrastructure systems, and a new “extremely hardcore” culture, all for the crusade to find profitability and ship software requirements.
But Elon is not solely to blame. The problems at Twitter existed long before the takeover:
Soon after joining Twitter in 2019, Dantley Davis gathered his staff in a conference room at the company’s San Francisco headquarters … He asked employees to go around the room, complimenting and critiquing one another. Tough criticism would help Twitter improve, he said. The barbs soon flew. Several attendees cried during the two-hour meeting, said three people who were there.
That sure doesn’t sound like “the type of organizational community in which we would want to work”.
This is significant because the tech market is a self feeding, always cannibalizing beast: whatever the big players do, typically companies like Google, Meta, and Amazon, the rest of the tech market will follow. These trends in engineering cultures, compensation, interviews, and so on will always trickle down to the rest of the industry. So, without you even knowing it, the Elon takeover of Twitter has probably already affected you.
Layoffs and downsizing in 2023 may not yet be over and many economists believe we are heading into a recession (if not already there) which could accelerate cultural and engineering organization changes at many companies. And like before, in 2001, when the dot-com bubble had its reckoning, the software and tech industry of today must face the economic music.
But my hope for the future is that engineering organizations and leadership recognize the history that is being repeated here, change course, and continue to focus on lean, agile processes that work for them. Otherwise, we may see more companies like Twitter with a failing business model, collapsing infrastructure, deplorable stability, and maybe worst of all, an engineering organization that no one wants to be apart of. (John McBride 2023)
Article written by Al Tenhundfeld 23rd July 2021
https://www.simplethread.com/agile-at-20-the-failed-rebellion/https://www.runn.io/blog/it-project-management-statistics
Article written by John McBride 23rd April 2023
https://onengineering.substack.com/p/how-the-agile-manifesto-changed-nothing