Beyond Agile Programming

After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history. In order to contribute my bit to addressing that problem, I've posted this essay—one that's sure to infuriate many of my readers, including some of my best friends. So first let me tell you how it came about.

While reformatting my book, Rethinking Systems Analysis and Design for e-booking, I noticed a few places that might have needed updating to present realities. The version I was using was more than 20 years old, from just after the peak of excitement about "structured programming." In particular, there was a whole section entitled, "Beyond Structured Programming." As I contemplated updating that section, it dawned on me that I could almost update completely by substituting the name of any more recent "movement" (or fad) for the word "structured.

 I also knew how smart most of my readers are, so I figured they would see the same possibility without my updating a thing. Instead of changing the book, I decided to update the section and publish it on this blog. Why? Because I think it shows an important pattern—a script where only the names have changed over at least five decades. So, here is the section with "agile" substituted for "structured," just as "structured" had been substituted for some other fad a generation earlier.

The Restructured Essay
Before I proceed further with the task of rethinking systems analysis and design, I'd like to express myself on the subject of another great "rethinking" in programming—the agile programming revolution. Although this essay was written a generation ago (now two generation), and the agile programming "revolution" is now an exhausted fad (for most programmers), most of what this essay says still applies—though to the next rethinking fad, and the next, and the next. I believe it will still apply long after I'm no longer writing new editions. Why? Because our industry seems to require a new fad every decade to keep itself from being bored. So, just apply the lessons to whatever fad happens to be dominating the computer press at the time you're reading this.

Before anyone becomes overly enthusiastic about what the rest of this book says, I want to take stock of what this great agile rethinking has done. I don't claim to be starting a new revolution of the magnitude most of the fads claim, so I'd like people to realize how slow and how small the agile programming movement has been, in case they think this book is going to make much difference.

My own personal stock-taking on the subject of agile programming is based on visits to some forty installations on two continents over the past ten years, plus a few hundred formal and informal discussions with programmers, analysts, managers, and users during the same period. Because of the conditions under which these visits and interviews took place, I would estimate the sample is quite heavily biased toward the more progressive organizations. By "progressive," I mean those organizations more likely to:
• Send staff to courses
• Hire outside consultants, other than in panic mode
• Encourage staff to belong to professional organizations, and to attend their meetings.
Consequently, my stock-taking is likely to be rather optimistic about the scope and quality of the effects of agile programming.

The first conclusion I can draw from my data is this:
Much less has been done than the press would have you believe.
I interpret the word "press" very loosely, including such sources as:
• Enthusiastic upper management
• The trade press
• The vendors and their advertising agencies
• The universities, their public relations staffs, and their journals
• The consulting trade.
Although this may be the most controversial of my observations, it is the most easily verified. All you need do is ask for examples of agile programming—not anecdotes, but actual examples of agile behavior and agile-produced code. If you're given any examples at all, you can peruse them for evidence of following the "rules" of agile programming. Generally, you will find:

a. Five percent can he considered thoroughly agile.

b. Twenty percent can be considered to follow agile practices sufficiently to represent an improvement over the average code of 1990.

c. Fifty percent will show some evidence of some attempt to follow some "agile rules," but without understanding and with little, if any, success.

d. Twenty-five percent will show no evidence of influence by any ideas about programming (not just agile) from the past twenty years.

Please remember: these percentages apply to the code and behavior you will actually see in response to your request. If you ask software organizations at random for "agile examples," about two-thirds will manage to avoid giving you anything. We can merely speculate what they do, and what their code contains.

My second conclusion:
There are rather many conceptions of what agile programming ought to look like, all of which are reasonably equivalent if followed consistently.
The operative clause in this observation seems to be "if followed consistently." Some of these conceptions are marketed in books and/or training courses. Some are purely local to a single installation, or even to one team in an installation. Most are mixtures of some "patented" method and local adaptations.

My third observation:
Methods representing thoughtful adaptations of "patented" and "local" ideas on agile programming are far more likely to be followed consistently.
In other words, programmers seem disinclined to follow an agile methodology when it is either:
1. Blind following of "universal rules"
2. Blind devotion to the concept: anything "not invented here" must be worthless.

My fourth observation:
I have other observations to make, but now I must pause and relate the effect these observations have on many readers, perhaps including you. I recall a story about a little boy who was playing in the schoolyard rather late one evening. A teacher who had been working late noticed the boy and asked if he knew what time it was.
"I'm not sure," the boy said, "but I know it isn't six o'clock yet."
"And how do you know that?" the teacher asked.
"Because I'm supposed to be home at six, and I'm not home."
When I make my first three observations about agile programming, I have a similar reaction—something like this:
"These can't be right, because if they were right, why would there be so much attention to agile programming?"

In spite of its naive tone, the question deserves answering. The answer can serve as my fourth observation:

Agile programming has received so much attention for the following reasons:
• The need is very great for some help in programming.
• To people who don't understand programming at all, it seems chaotic, so the term "agile" sounds awfully promising.

• The approach actually works, when it is successfully applied, so there are many people willing to give testimonials, even though their percentages among all programmers may not be great.

• The computer business has always been driven by marketing forces, and marketing forces are paid to be optimistic, and not to distinguish between an idea and its practical realization.

In other words, the phrase "agile programming" is similar to the phrase"our latest computer," because each phrase can be used interchangeably in statements such as these:

• "If you are having problems in information processing, you can solve them by installing our latest computer."

• "Our latest computer is more cost effective and easier to use."

• "Your people will love our latest computer, although you won't need so many people once our latest computer has been installed."

• Conversion? No problem! With our latest computer, you'll start to realize savings in a few weeks, at most."

So actually, the whole agile programming pitch was pre-adapted for the ease of professionals, who have always believed "problems" had "solutions" which could be mechanically applied.

My final observation is related to all of the others:
Those installations and individuals who have successfully realized the promised benefits of agile programming tend to be the ones who don't buy the typical hardware or software pitch, but who listen to the pitch and extract what they decide they need for solving their problems. They do their own thinking, which includes using the thoughts of others, if they're applicable. By and large, they were the most successful problem solvers before agile programming, and are now even more successful.

There's yet another lesson in all this that's much bigger than agile programming or any new hardware or software or process:

Our profession contains few, if any, easy solutions. Success in problem solving comes to those who don't put much faith in the latest "magic," but who are willing to try ideas out for themselves, even when those ideas are presented in a carnival of public relations blather.
Based on this lesson, I'd like to propose a new "programming religion," a religion based on the following articles of faith:

• There's no consistent substitute for a thorough understanding of your problem, though sometimes people get lucky.

• There's no solution applicable to every problem, and what may be the best approach in one circumstance may be precisely the worst in another.

• There are many useful approaches applicable to more than one problem, so it pays to become familiar with what has worked before.

• The trick to problem solving is not just "know-how," but "know-when"—which lets you adapt the solution method to the problem, and not vice versa.

• No matter how much you know how or know when, some problems won't yield to present knowledge, and some aspects of the problem nobody currently understands, so humility is always in order.

I realize writing a book is not the most humble thing a person can do, but it's what I do best, and how I earn my living. I'd be embarrassed if anyone took this book too seriously. We don't need another "movement" just now, unless it is something analogous to a bowel movement—something to flush our system clean of waste material we've accumulated over the years.

Where to read the original
If you want to check on my historical work, you can find the original essay (and many others) in Rethinking Systems Analysis and Design, which is an ebook on Smashwords (where you can probably see it in the free sample) and Kindle and Barnes and Noble.


Problem-Solving Leadership Workshop
Reminder: The second (and last) PSL Workshop for 2011 will take place in Albuquerque, New Mexico, USA, August 28-September 2, 2011. Only a few places left for participants, so for more information, see <http://www.estherderby.com/workshops/problem-solving-leadership-pslhttp://www.estherderby.com/workshops/problem-solving-leadership-psl>