Is Agile a welfare program for developers?

How do you make developers comfortable?

There are free lunches, game rooms, and massages. Consultants can help you craft the most comprehensive perks program in town.

Or, you could implement Scrum.

In the paragraphs that follow, I will argue that Agile has become an overfunded welfare program for developers.

And since this is going to ruffle a few feathers, I will limit my indictment to Scrum projects in the context of enterprise software development. This represents the bulk of my professional experience. Please read the text below through this lens.

Also note that when I say Agile, I mean mostly Scrum.

A bit of history

Back in the Waterfall era, project managers ruled supreme.

They set deadlines, herded their flock of developers toward milestones, and had the CxOs on speed dial.

Did the projects run over time and budget? They sure did! But you know what? We got real work done!

One explanation is as follows. There's always tension between the client (the higher-ups) and the producer (developers). One side wants an X amount of work in a Y timeline while the other side can deliver a Z amount. And they struggle to assign values to these variables.

In the Waterfall environment, this power dynamics favored the client's side. Overtime used to be common as deadlines had to be met. The developers, when asked to estimate, would argue in man-days, and the project manager would inevitably challenge them. A compromise would be forged, in which the developers would often draw the shorter straw.

This was hard on developers. So, they invented Agile.

Agile: lofty and idealistic on paper, toothless in practice

Recall that Agile was invented by developers, for developers.

It was sold to the business folks differently, of course. As a way to deploy working software more quickly than the standard Waterfall allowed.

And while I know nothing about the state of mind of the Agile inventors, I don't have to. Just looking at how the power dynamics have changed tells the story.

Case in point

Consider any sufficiently ambitious program run by an enterprise so that it can finally enter the 21st century. Any "digitization" program would do.

Bonus points if it's a "startup" launched and wholly owned by the enterprise.

How is that program staffed? If the enterprise isn't a software house, chances are excellent that 90% of the staff are contractors. The company doesn't have enough developers in-house, and won't need them once the program finishes.

Which, of course, it never does. Because guess what? The methodology sold to the higher-ups as a surefire way of joining the 21st century was Scrum.

The enterprise has no agility in its DNA and the project runs as an isolated island embedded in the traditional hierarchy. The vision is handed down to contracted "product owners" who dutifully translate it to "user stories" and take it by the developers in the "refinement session."

There, the team evaluates each story and assigns it a point value on a scale between 1-13. "1" is like "yawn, I could do this now if I didn't have to be in this stupid meeting" and the "13" tells the PO, "eff you, break it down even further, there is too much unknown here."

What's the product owner to do? He could push back a little. But, the Scrum Master is there to ensure that The Process is adhered to, and in this process, the developers are always right. Always.

The result is that developers develop at their own speed, and if they so desire, they can drag the backlog items over several Sprints and nobody will punish them.

Now imagine you are paying someone to fix up your house, and they do this to you.

The counter-argument

Agile has also produced numerous benefits, and not just for developers.

It "normalized" relationships between developers and business folks. You don't need as many intermediaries as before (good-bye, business analysts!) and as developers got better at soft skills, they get to see the suits more often.

The 24/7 deployment story can also be attributed to Agile. You could argue that the technical progress in the CI/CD tooling (and the cloud) made it possible, and I won't disagree. What I see in the Agile manifestos is the desire to deploy as frequently as possible (i.e., at the end of the Sprint in Scrum), and we do that now.

Much better than once a quarter if you ask me.

The indictment

Agile has changed the power dynamics and given developers too much power.

My intent isn't to indict its inventors. I would like to inspire you to think about how we could course-correct instead.

A situation where the development team can dictate its tempo is not healthy. That they alone define the deliverables for the next iteration is not healthy. It's bad for the people paying the bills, and it creates incentives for the developers to become lazy.

Worse yet, it's a terrible fit for programs run by enterprises. Their internal processes have a huge impedance mismatch when confronted with Agile values, and so the implementation can only result in the most bastardized version possible.

How to evolve Agile

In my practice, I never use Scrum and have explained why (and what we do instead) on my agency blog. To be clear, the projects I run are nowhere near in size of the enterprise Scrum programs I talked about above.

However we address this, the key thing is to correct the power dynamics.

The Waterfall was bad because it made developers overworked cogs with no say about the process. It barely worked then and would be laughable now when you consider how long it took to ship anything.

Agile can be better but mostly isn't, because its implementation swayed the balance too much in the developers' favor.

Where is the ideal middle ground, and is there one?

I propose three areas for analysis:

1. adding more rigor to the planning
2. adding "skin in the game"
3. adding oversight

Planning with purpose

Agile favors working software over comprehensive documentation. And yet, you couldn't deliver anything useful if you had no inputs and no customers.

Having seen too many undercooked "user stories," I would argue that good, rigorously analyzed documentation makes developers happier. Developers hate uncertainty.

You don't have to go all the way back to the Waterfall days and try to describe the entire system up-front. I am in favor of iterative development. By all means, release something useful early and test it on customers.

Consider documenting your iteration rigorously before development, however.

By thinking through your use cases and documenting them, you can spot and resolve errors in your thinking early. Doing so on paper is cheaper than reworking already deployed code.

This creates the first condition to correct the power balance. It gives developers more clarity about what they are about to do.

Adding "skin in the game"

Agile is a little too optimistic about people's motivation and assumes everyone is equally interested in the outcome. I think adding carrots and sticks is necessary to make it so.

Whichever way you do it, aligning compensation with performance is necessary.

Before you object, yes, I am aware that measuring developer performance is not a trivial subject. It's also not specific to Agile, or any other methodology.

My point is to highlight the importance of "skin in the game" for the success of any endeavor, including software development projects.

Trading points may not suffice.

Whatever compensation scheme you have in place, I would consider not exempting your Agile teams from it. When people are incentivized to perform at their very best, you create a second condition to fix the power balance.

Oversight

This point is specific to enterprise projects that are largely staffed with contractors.

While there's no inherent guarantee that your employees have the best interests of your company at heart, contractors usually care less.

Giving contractors a blank check to run your project using Agile can lead to cost and deadline overruns that rival those of the Waterfall era.

I witnessed large programs where the organization didn't have even its own CTO! Everyone besides a few key product people was contracted. In such an environment, there is no oversight and no accountability.

Having core engineering staff is a must. They shouldn't be the "architecture astronauts" and prescribe solutions to the teams without having to deliver. On the contrary, I would put one or two internal employees on each team so that the technical decisions the team makes are aligned with the company's overall technical strategy and values.

In a broader sense, oversight and controls should be built into the process regardless of who is doing the work (employees or contractors).

Scrum has biweekly "reviews" and "retrospectives." I would argue that two weeks is too short a period - it's a microcosm of tactical decisions.

Your process should have checkpoints that review the process and outcomes on higher levels - maybe every 6 weeks to review how complete features are being (or not being) shipped, and once a quarter to check on major milestones. These are just examples - choose a frequency that best fits the project's timelines and your velocity.

Summary

I criticized Agile in light of how it's been implemented, especially in Scrum and specifically in the context of enterprise software development.

In the process of writing, I realized I chose a topic that's quite big to chew on; I admit that. Many themes covered above deserve analysis on a much deeper level. I've only touched the surface.

The first version went off the rails as I switched my focus to the evils of hourly billing and unsuccessfully tried to tie it together with the deficiencies of Agile as I see them. I removed it and will publish a rant against hourly billing separately.

All things considered, I am not against Agile in principle but think that we should evolve it pragmatically and that in the enterprise context, it needs more adult supervision than it often gets.

Published on 01 Dec 2022 , updated on 03 Dec 2022

What do you think? Sound off in the comments!