Monday, June 06, 2011

Beyond Agile Programming

After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history. In order to contribute my bit to addressing that problem, I've posted this essay—one that's sure to infuriate many of my readers, including some of my best friends. So first let me tell you how it came about.

While reformatting my book, Rethinking Systems Analysis and Design for e-booking, I noticed a few places that might have needed updating to present realities. The version I was using was more than 20 years old, from just after the peak of excitement about "structured programming." In particular, there was a whole section entitled, "Beyond Structured Programming." As I contemplated updating that section, it dawned on me that I could almost update completely by substituting the name of any more recent "movement" (or fad) for the word "structured.

 I also knew how smart most of my readers are, so I figured they would see the same possibility without my updating a thing. Instead of changing the book, I decided to update the section and publish it on this blog. Why? Because I think it shows an important pattern—a script where only the names have changed over at least five decades. So, here is the section with "agile" substituted for "structured," just as "structured" had been substituted for some other fad a generation earlier.

The Restructured Essay
Before I proceed further with the task of rethinking systems analysis and design, I'd like to express myself on the subject of another great "rethinking" in programming—the agile programming revolution. Although this essay was written a generation ago (now two generation), and the agile programming "revolution" is now an exhausted fad (for most programmers), most of what this essay says still applies—though to the next rethinking fad, and the next, and the next. I believe it will still apply long after I'm no longer writing new editions. Why? Because our industry seems to require a new fad every decade to keep itself from being bored. So, just apply the lessons to whatever fad happens to be dominating the computer press at the time you're reading this.

Before anyone becomes overly enthusiastic about what the rest of this book says, I want to take stock of what this great agile rethinking has done. I don't claim to be starting a new revolution of the magnitude most of the fads claim, so I'd like people to realize how slow and how small the agile programming movement has been, in case they think this book is going to make much difference.

My own personal stock-taking on the subject of agile programming is based on visits to some forty installations on two continents over the past ten years, plus a few hundred formal and informal discussions with programmers, analysts, managers, and users during the same period. Because of the conditions under which these visits and interviews took place, I would estimate the sample is quite heavily biased toward the more progressive organizations. By "progressive," I mean those organizations more likely to:
• Send staff to courses
• Hire outside consultants, other than in panic mode
• Encourage staff to belong to professional organizations, and to attend their meetings.
Consequently, my stock-taking is likely to be rather optimistic about the scope and quality of the effects of agile programming.

The first conclusion I can draw from my data is this:
Much less has been done than the press would have you believe.
I interpret the word "press" very loosely, including such sources as:
• Enthusiastic upper management
• The trade press
• The vendors and their advertising agencies
• The universities, their public relations staffs, and their journals
• The consulting trade.
Although this may be the most controversial of my observations, it is the most easily verified. All you need do is ask for examples of agile programming—not anecdotes, but actual examples of agile behavior and agile-produced code. If you're given any examples at all, you can peruse them for evidence of following the "rules" of agile programming. Generally, you will find:

a. Five percent can he considered thoroughly agile.

b. Twenty percent can be considered to follow agile practices sufficiently to represent an improvement over the average code of 1990.

c. Fifty percent will show some evidence of some attempt to follow some "agile rules," but without understanding and with little, if any, success.

d. Twenty-five percent will show no evidence of influence by any ideas about programming (not just agile) from the past twenty years.

Please remember: these percentages apply to the code and behavior you will actually see in response to your request. If you ask software organizations at random for "agile examples," about two-thirds will manage to avoid giving you anything. We can merely speculate what they do, and what their code contains.

My second conclusion:
There are rather many conceptions of what agile programming ought to look like, all of which are reasonably equivalent if followed consistently.
The operative clause in this observation seems to be "if followed consistently." Some of these conceptions are marketed in books and/or training courses. Some are purely local to a single installation, or even to one team in an installation. Most are mixtures of some "patented" method and local adaptations.

My third observation:
Methods representing thoughtful adaptations of "patented" and "local" ideas on agile programming are far more likely to be followed consistently.
In other words, programmers seem disinclined to follow an agile methodology when it is either:
1. Blind following of "universal rules"
2. Blind devotion to the concept: anything "not invented here" must be worthless.

My fourth observation:
I have other observations to make, but now I must pause and relate the effect these observations have on many readers, perhaps including you. I recall a story about a little boy who was playing in the schoolyard rather late one evening. A teacher who had been working late noticed the boy and asked if he knew what time it was.
"I'm not sure," the boy said, "but I know it isn't six o'clock yet."
"And how do you know that?" the teacher asked.
"Because I'm supposed to be home at six, and I'm not home."
When I make my first three observations about agile programming, I have a similar reaction—something like this:
"These can't be right, because if they were right, why would there be so much attention to agile programming?"

In spite of its naive tone, the question deserves answering. The answer can serve as my fourth observation:

Agile programming has received so much attention for the following reasons:
• The need is very great for some help in programming.
• To people who don't understand programming at all, it seems chaotic, so the term "agile" sounds awfully promising.

• The approach actually works, when it is successfully applied, so there are many people willing to give testimonials, even though their percentages among all programmers may not be great.

• The computer business has always been driven by marketing forces, and marketing forces are paid to be optimistic, and not to distinguish between an idea and its practical realization.

In other words, the phrase "agile programming" is similar to the phrase"our latest computer," because each phrase can be used interchangeably in statements such as these:

• "If you are having problems in information processing, you can solve them by installing our latest computer."

• "Our latest computer is more cost effective and easier to use."

• "Your people will love our latest computer, although you won't need so many people once our latest computer has been installed."

• Conversion? No problem! With our latest computer, you'll start to realize savings in a few weeks, at most."

So actually, the whole agile programming pitch was pre-adapted for the ease of professionals, who have always believed "problems" had "solutions" which could be mechanically applied.

My final observation is related to all of the others:
Those installations and individuals who have successfully realized the promised benefits of agile programming tend to be the ones who don't buy the typical hardware or software pitch, but who listen to the pitch and extract what they decide they need for solving their problems. They do their own thinking, which includes using the thoughts of others, if they're applicable. By and large, they were the most successful problem solvers before agile programming, and are now even more successful.

There's yet another lesson in all this that's much bigger than agile programming or any new hardware or software or process:

Our profession contains few, if any, easy solutions. Success in problem solving comes to those who don't put much faith in the latest "magic," but who are willing to try ideas out for themselves, even when those ideas are presented in a carnival of public relations blather.
Based on this lesson, I'd like to propose a new "programming religion," a religion based on the following articles of faith:

• There's no consistent substitute for a thorough understanding of your problem, though sometimes people get lucky.

• There's no solution applicable to every problem, and what may be the best approach in one circumstance may be precisely the worst in another.

• There are many useful approaches applicable to more than one problem, so it pays to become familiar with what has worked before.

• The trick to problem solving is not just "know-how," but "know-when"—which lets you adapt the solution method to the problem, and not vice versa.

• No matter how much you know how or know when, some problems won't yield to present knowledge, and some aspects of the problem nobody currently understands, so humility is always in order.

I realize writing a book is not the most humble thing a person can do, but it's what I do best, and how I earn my living. I'd be embarrassed if anyone took this book too seriously. We don't need another "movement" just now, unless it is something analogous to a bowel movement—something to flush our system clean of waste material we've accumulated over the years.

Where to read the original
If you want to check on my historical work, you can find the original essay (and many others) in Rethinking Systems Analysis and Design, which is an ebook on Smashwords (where you can probably see it in the free sample) and Kindle and Barnes and Noble.


Problem-Solving Leadership Workshop
Reminder: The second (and last) PSL Workshop for 2011 will take place in Albuquerque, New Mexico, USA, August 28-September 2, 2011. Only a few places left for participants, so for more information, see <http://www.estherderby.com/workshops/problem-solving-leadership-pslhttp://www.estherderby.com/workshops/problem-solving-leadership-psl>

20 comments:

OlafLewitz said...

Jerry,
Insightful as always...:-)
I like the style of this "experiment", and I agree that your observations match a certain, prominent, marketing view of "Agile". And, yet...
There is a difference, and it's substantial. Some new style of programming (OO, structured, functional, ...), any new practice or "the new computer" are intended to be specific solutions for specific challenges or problems. Correctly applied, they do the expected job, and they might lead to amazing results--if, and only if, you have the right people sharing a problem solving mindset.
Agile, or Lean, for that matter, are neither about solutions nor about programming. They are about new ways of thinking about human interactions in problem solving--values, principles and disciplines to better collaborate on a common goal.
This is new. Not as new as the latest computer, but new to management and employees of most enterprises. We need to view this on a larger scale, I think:
Our current education system is shaped by 18th century enlightenment and 19th century industrialisation thinking. Organisational structure and management are safely rooted in early 20th century business models.
Daniel Pink calls this century the conceptual age. We need a whole new mind, people want drive more than money, Linchpins make more of a difference than methods and practices.
It's only starting. What you and I observe are all early adopters. Being the first to HAVE an iPad does not mean you use it and change your information consumption habits. "Buying" Agile does not change anything, doing it, improving it, letting it change our thinking does.
So I'll continue on this path. To me, it's much more than a fad.
Thanks for inspiring me to write this down!
Take care
Olaf

Gerald M. Weinberg said...

Olaf, I totally agree with you about what we ought to be doing. My questions are about whether we are doing it--and if the "we" is a small set of energetic pioneers like you, then yes, we are doing it.

But will we affect the vast majority? So far, not yet, and agile is not the only attempt to do more than change the way people write programs. Part of the lesson from the past it the way people have forgotten the non-mechanical parts of that movement, and others before it whose very names have been forgotten.

So, what do we have to do so that the lofty goals of ours are not forgotten by the majority in twenty more years? That's the big issue that's facing those of us who want to "continue on this path."

I hope we get some answers. I'll try to put some ideas on this blog in the coming weeks and months. (And many such ideas are already in my books, as well as in the books of others. Now we must get them out of the books and into practice so they drive out those 20th century business models.)

Grant (PG) Rule said...

Thank you very much for expressing this, Gerry. Spot on! It does seem that those who ignore history are sentenced to repeat it.

However, I'd like to point out that the 'fashion-driven' nature of so-called 'progress' is not limited to IT.

Olaf suggests that "Agile, or Lean, for that matter… are about new ways of thinking…". Sorry, but hardly 'new'. The elements of 'flow production' (what we now call 'lean thinking') have been shown by Bob Emiliani to have been pretty well formulated in the 1920s by Frank G. Woollard of the Morris Engines Ltd.

What's more, some elements of lean thinking can be traced back to the shipwrights of the Carthaginian & Roman navies c. 242BC.

Mindsets take a long time to change. Individual mindset and group culture must relate and interact in subtle ways (which we don't yet understand, IMO). So whereas Henry Ford's concepts of mass production gained widespread acceptance, the principles of flow production (including 'agile practices') have yet to be truly appreciated or widely adopted.

I'm pleased to say I have been learning from you, Gerry, since the early 1970s. I continue to do so, and look forward to further inspiration.
Best regards,
Grant

J. B. Rainsberger said...

Jerry, I understand the mild cynicism inherent in this essay. It accounts for the reason I try not to sell agile as a pre-packaged solution. I know the power of agile-related thinking; it transformed my life allowing me to retire at 34. I recognise that the thinking itself wasn't particularly agile-specific, but agile methods inspired me. I can only hope it inspires other to expand their thinking.

Rini said...

Hi Jerry,

But still, though things look familiar, we are improving, aren't we?

You know this article: Life-cycle of a silver bullet? http://www.crosstalkonline.org/storage/issue-archives/2003/200307/200307-Sheard.pdf

Gr. Rini

Jason Little said...

You've hit the nail on the head with the definition of "progressive" companies. Agile, or not, companies that meet your "progressive" definition get it have a better chance of success than companies who rely on the latest fad.

Pat said...

> So, what do we have to do so
> that the lofty goals of ours are > not forgotten by the majority in > twenty more years? That's the
> big issue that's facing those of > us who want to "continue on this > path."

Roughly, that's my question. With profound respect for the wonderful books you've written (both Jerry and JB!), and that I frequently pass around, buy as gifts and otherwise recommend, I've noticed that the ideas therin are somewhat slow to pass in to practice, and that most organizations still find them somewhat new and unorthodox. Can this change? How can this change?

It might be that books aren't the whole picture... what else needs to be in place? My bias is to say 'relationships', where ideas are amplified by presence... strawberry jam, more than raspberry jelly. Agile, at its best, seems to focus on 'It's the team, not the technology' for answers to how to be better... something I know you, Jerry, care very much about.

Gerald M. Weinberg said...

"So, what do we have to do so that the lofty goals of ours are not forgotten by the majority in twenty more years?<

Excellent question. That why I (and my commentors) teach and consult and write books). For me, I teach the Problem Solving Leadership workshop (PSL) and the Organizational Change Shop and the AYE conference for the sort of people who will lead the changes and make them stick.

We call these people "change artists," and I have also written guides for them, such as "Becoming a Change Artist." The more change artists we can grow (rather than mere lecturers), the better our chances of pulling our profession up by its bootstraps.

Steven M Smith said...

Jerry,

It's uncanny how substituting a new subject for an old subject reincarnated the wisdom of the surrounding words. Nice work!

"No matter how much you know how or know when, some problems won't yield to present knowledge, and some aspects of the problem nobody currently understands, so humility is always in order."

Humility: a necessary ingredient for solving problems by individuals, teams and organizations.

Can humility be taught in a classroom? Or created by a mission statement?

Hugs,
Steve

Gerald M. Weinberg said...

Steve, your comment made me think of something I missed. The ability to substitute today's name for yesterday's suggest that we might try something different this time around—something that would increase the penetration of today's idea beyond that of yesterday's. So what things didn't we do in the past that might just push, say, agile practices over some tipping point, enabling them to become a standard throughout the industry. (Not the only standard, but one of several standards.)

What might that idea be, and where might we find the tipping point?

Ken said...

So it's the people, not the process, that makes good software. :)

Seems like that's the eternal truth that outlives different processes.

ken

Bruce Eckel said...

I also lived through the "structured" revolution but I would absolutely not equate the two. I immersed in structured and discovered the empty pieces; it really was a lot of sham.

Agile is a way of thinking about communicating around problem solving, not a particular set of blessed techniques as structured was.

Gerald M. Weinberg said...

Bruce,
I'm "equating the two" only in the sense of the dynamics of adoption and rejection. I'm warning the agile people that they're exposed to the same human dynamic, unless they change something.

As one tweeter reminded us, "Beware the Law of Raspberry Jam." (From The Secrets of Consulting)

Donald Cox said...

Great post and comments.

I feel as though the anniversary of the Agile Manifesto has occasioned at lot of Agile is Dead, Long Live Agile ebb and flow.

On the topic of grand change, I wonder if Kuhn's observation on change in scientific thinking/research applies - that it takes a generation for the change to occur, because it usually requires physical replacement of people for the change to occur.

Granted, one would expect the rate of change or turn-over to be greated in the commercial world than the academic. On the other hand, those who wield the most money/organizational power at this point are the ones who prospered during the structured programming revolution.

Best,
Don

Steven M Smith said...

> What might that idea be, and where might we find the tipping point (that enable effective practices to become standards)? <

Congruent management.

YvesHanoulle said...

hi Jerry,

One way I alway look at "the law of rasberry jam", was it was telling us, we can't avoid this. In other words: the fact that we spread it wider, it will be thinner.

After reading this, I'm thinking: how else can we look at this law?

Maybe it tells us if we want to spread agile better, we need more JAM. The question now is: what is jam in this case. And how can we have more quality jam.

AnthroTools said...

After seeing Bruce's name in the comment, I went and looked at his blog. He had a review of Seth Godin's Linchpin there. In that review he talks about the idea of slow change. That resonated for me. And it reminded me of a podcast that was an interview with David A. Black. One of the things he talked about was an excessive pace of change. He was talking about technical tools. I wonder if there is something about pacing change in non-technical areas as well. Perhaps part of the problem with Agile adoption is that people are trying to force change at a pace that is faster than can be sustained (for successful adoption).

My current thought is that the root phenomenon for Agile to really make deep and lasting penetration is to have an education system that fosters valuing one's own experience over the pronouncements of purported authorities and experts.

Expanding on Yves's exploration of the metaphor, I think getting more jam relies on understanding and respecting the underlying organic processes that generate your raw material.

Best,
Don

Richard said...

It's really hard to fight management, even from within management. Toyota, nudged by W. Edwards Deming, has more market cap than all other car companies in the world. I claim it's because they take the time to train (acculturate) their managers (executives, leaders, bosses) for literally years. See for example John Shook's "The A3 Process; Managing to Learn" where he shows a "trick" to be used in problem solving.

But even with successful Toyota and GM shared operation of NUMMI, GM couldn't roll out their findings to the rest of GM in two decades. See the NPR show from a few months ago in "This American Life" about NUMMI.

You, Jerry, and Tom GIlb, and Johanna Rothman, and other colleagues at AYE are Treasures to be shared and contemplated, and mined for inspiration. But then comes the hard part, application.

New ideas are dangerous. They seldom help much. When they do help, they cause much turmoil and many new problems. This is much like the random mutation which potentiates evolution. It almost never helps.

All humans suffer from ascorbic acid insufficiency because one of the four genes for the enzyme chain that converts glucose into ascorbic acid was accidentally broken millions of years ago. The basic tools needed to discover this are younger than either you or me.

We also suffer from the human reasoning system which was built not to find the truth, but instead to find the winning argument, quickly. It was enough to facilitate group hunting, and organized raiding, but it's not so good at introducing broader scale and longer term cooperation such as needed to cope with complex projects.

The technological tools to observe and repair such problems may possibly appear in the next few decades. If we're lucky. Meanwhile, we muddle through. Thanks for helping us muddle.

rukawa said...

Huh? Didn't structured programming succeed. I thought the whole idea was to eliminate goto statements. Hasn't that been completely achieved?

Gerald M. Weinberg said...

Trevor, there was a lot more to structured programming than eliminating gotos, though that was an important part, and it was partly successful. But, if you look at lots of code still being written today, you'll see that the goto is still around, just as there are still a lot of people around who think that if a cure doesn't hurt, it can't be working. Change doesn't happen in a binary way, either in one person or across an entire population. That's the way it's been for structured programming and other movements, and that's the way it will be (and is) for Agile programming.

Anyway, Trevor, I'm very glad you asked. Lots of people need to understand the way this works.