Do programmers really know how to program?

I was asked, "Do programmers really know how to program?"

I believe this question is unproductive and  vague. What does it mean by “program”?

The person who asked this question seemed to think programmers were not really programming when all they did was copy some existing program, using it whole or perhaps pasting it in as part of a shell.

To me, programming a computer means instructing it to do something you want done, and to continue doing it as desired.

If that’s what we’re asking about, then yes, of course, some of us out here know how to program. (Some do not, of course.)

It is irrelevant how we do that. Whether we use genetic algorithms, cut-and-paste, or divine inspiration? Do we use Scrum or Agile or Waterfall? How about the programming language? C++, or Java, or Lisp, or Python, or APL? Well, none of those choices matters.

Then what does matter? How about, "Can we satisfy someone’s desires?" In other words, can we provide something that someone wants enough to pay what it costs, in time or money? That’s what counts, and we certainly know how do that—sometimes.

Sure, we fail at times, and probably too often. But no profession succeeds in satisfying its customers all the time. Did your teachers always succeed in teaching you something you wanted to know? Do surgeons know how to do surgery?

So what about using existing programs? To my mind, the first and foremost job of a programmer is knowing when not to write a program at all—either because the needed program already exists or because no program was needed in the first place.

In other words, not writing a program when no program is needed is the highest form of programming, and one of the marks of a true expert.

The Psychology of Computer Programming: Silver Anniversary eBook Edition 

or Kindle for the book in paper or ebook format

Is Waterfall the Opposite of Agile?

Agilists sometimes behave unreasonably by pummeling the Waterfall approach. Personally, I think such evangelists could better spend their effort producing solid, relevant code, but evidently they are on a crusade and need an enemy. If they need an enemy of Agile, however, they could have made a better choice.

There's nothing wrong with the Waterfall approach—if applied appropriately. For Waterfall to work well, you need to solve a problem that's not destined to change much, or, ideally, to change not at all. Some Agilists claim there is no such unchanging problem, but they're merely showing their lack of experience. I've seen a number of such invariant problems, and they yield very readily to a properly applied Waterfall development.

For example, Erik, a former student of mine, made a lucrative business of converting COBOL programs to new COBOL programs adapted to new versions of the COBOL compiler. Erik's customers wanted assurance that nothing would be changed in the conversion. This was a perfect situation for a simple Waterfall approach, one that Erik could perform profitably at a fixed price and schedule.

That said, the number of such invariant problems is not large, and it's usually difficult to know at the beginning if a problem will turn out to be so stable. In Erik's case, some of his customers would decide midway that they wanted a few "tiny" improvements in the converted application. Erik controlled that situation by charging outrageous fees for even the simplest modification. Most of us, however, try to control this situation by employing some Agile approach.

Let's be honest with ourselves: one consequence of an Agile approach is the loss of our ability to work to a fixed cost on a fixed schedule. Erik could do that, but only in a few carefully controlled situations. Many managers frustrate themselves over this lack of control and blame it on Agile. What's really to blame, however, is their inability to control the world in which they live. Things do change, and much of the time it's these very managers who instigate the change.

What do frustrated managers do? Quite often, they elevate their attempts to control the change by making rules. They may start using a pure Waterfall approach, but as their frustration with changes grows, they may add a Change Control Board, or change reviews, or a Change Tsar, or any of a number of other tactics. And, when those tactics fail to produce absolute predictability, they add more of the same kinds of rules and their supporting tactics.

After a while, these rules upon rules produce an approach that, though called "Waterfall," is actually something quite different—something for which we so far have no accurate name. This "something" is what Agile is responding to, so I suggest we name it.

What are these cobbled-together approaches like? First of all, they create a sad and dismal mood among those poor developers condemned to use them. When I visit a new client, I can generally detect the use of such an approach while I take a stroll through of the premises. I can even detect such approaches over the phone. How? Simple: the mood of my clients is mournful, gloomy, sad, unhappy, doleful, glum, melancholy, woeful, miserable, woebegone, forlorn, somber, solemn, serious, sorrowful, morose, dour, cheerless, joyless, and dismal.

That's quite a sobering list of adjectives, but that's what I can sense in many so-called "Waterfall" environments. Perhaps you recognize the list, but in any case, you can find that list in your dictionary as synonyms for the rare word, "lugubrious."

Perhaps the word "lugubrious" is unfamiliar, but that's good, because we won't often find it used in other contexts. Besides, it's a rather onomatopoetic word—a word that phonetically imitates, resembles or suggests the source of the sound or situation it describes. That's why Agile was invented—to replace those mournful, gloomy, rule-dominated approaches with brain-driven judgments of the actual builders.

So let's be truly Agile and stop bashing the true Waterfall approach. Instead, let's turn our contempt on every Lugubrious approach—or, to make a noun from the adjective, Lugubriousity. Maybe this more accurate name will help us defend our Agile projects from frustrated managers' attempts to smother us with yet more rules.

Or, to paraphrase DeMorgan, who in turn paraphrased Swift:

Great rules have little rules upon their backs to spite 'em,

And little rules have lesser rules, and so ad infinitum.

Never forget that's why we do Agile, not to dry out Waterfalls but to defeat Lugubriosity.

For more thoughs on the Agile approach, see

Agile Impressions

What's the most complex thing about software development?

What's the most complex thing about software development?

Interesting question.

So far, on Quora.com, there have been four excellent answers to this question: discussing 

- the confusing role of people, 

-the requirements problems,

-the interactions with the physical world.

Each of these factors certainly makes software development more complex, and processes such as Agile are designed to cope with this complexity. But, the ultimate complexity factor is software testing.

Why testing? In the software development literature, testing is not usually treated as a glamorous part of development, but when we're testing, we're up against the Second Law of Thermodynamics, which warns us that perfection is ultimately unobtainable.

So, even if we absolutely knew all the requirements (which we can't, of course), kept all the human factors under control (also impossible), and knew exactly all the physical properties of the real world (once more, impossible), we would still never be able to perform the infinite number of tests to cover all possible situations.

In other words, the software could still surprise us at any time. That's what I call complexity.

Of course, we can still work hard to solve these other problems. On requirements, for instance, see our Exploring Requirements books.

But no matter how hard you try, you'll still be faced with the testing problem. To understand this problem and what you can do to reduce (but not eliminate) it, take a look at Perfect Software and Other Illusions about Testing.

What are the most important software quality assurance techniques?

When this question was possed on a public form, the answers were predictably things like estimation, project management, and testing. All those things are important, but not the most important.

a) By far the most important Q/A technique is telling the truth about relevant Q/A issues in a way that can be understood by your audience. 

(see What Did You Say?: The Art of Giving and Receiving Feedback)

(see also, Are Your Lights On? How to Know What The Problem Really Is)

b) Of course, you must first be able to know what is the truth, and that requires observation skills of a high degree.

(see How to Observe Software Systems)

(see also, An Introduction to General Systems Thinking )

c) And, too, you must be able to know which truths are relevant. For that you must understand the processes practiced to produc the products whose quality you are supposed to be assuring.(see the Quality Software Series)

All specific techniques are dependent on these underlying techniques. If you lack a, b, or c, you will not be able to perform other techniques well.

If you wish to learn more about any of these techniques, or others, I’ve written many books on these subjects, not just the ones shown here. For a list of these books, see my author page on Leanpub.com

Iterative Development: Some History

As an old guy who's been around computing since 1950, I'm often asked about early history of computing. I appreciate efforts to capture some of our history, and try to contribute when my agin memory doesn't play tricks on me.

Back in 2003, Craig Larman and Victor R. Basili compiled an interesting article, Iterative and Incremental Development: A Brief History. I made several contributions to their history, but they did much, much more. Here's a small sample of what I told them:

We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale [at IBM's Service Bureau Corporation]. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell, indistinguishable from XP.

When much of the same team was reassembled in Washington, DC in 1958 to develop Project Mercury, we had our own machine and the new Share Operating System, whose symbolic modification and assembly allowed us to build the system incrementally, which we did, with great success. Project Mercury was the seed bed out of which grew the IBM Federal Systems Division. Thus, that division started with a history and tradition of incremental development.

All of us, as far as I can remember, thought waterfalling of a huge project was rather stupid, or at least ignorant of the realities… I think what the waterfall description did for us was make us realize that we were doing something else, something unnamed except for "software development."

Larman and Basili's article has a whole lot more to say, and as far as I know, is an accurate history. I strongly recommend that all in our profession give it a good read. We should all know these things about ourselves.

Medium-number Systems

A reader recently wrote asking about Medium Number systems. I thought other readers of An Introduction to General Systems Thinking might be interested in my answers to his questions:

Reader: I'm interested in learning more about the "Law of Medium Numbers" described in your book of An Introduction to General Systems Thinking. I feel the "Law of Medium Numbers" sheds light on many puzzles I have run into in dealing with nature and society. Thus I wonder whether I may ask you a few related questions? -

1. As I searched the literature trying to learn more about this subject and the "Law of Medium Numbers", I was surprised by how little I could find. I wonder whether I was not doing right searches or you may have more insights into this?

Jerry: It's not a popular subject with many scientists, because it shows how science—though amazingly successful in some areas—it's awfully limited in dealing with some of the most interesting problems. As someone pointed out years ago, "Physics is merely the study of those systems for which the methods of physics work."

Reader: 2. I wonder whether the "Law of Medium Numbers" might imply a kind of defiance of classical Western science and technology (which emphasize certainty and perfection)?

Jerry: That's well put. We'd like things to be different, but they aren't.

Reader: 3. Perhaps human individuals are medium number systems and thus may explain why we all have certain limitations (in other words, no one is perfect). So medium number systems may also be called imperfect systems. Am I right on this?

Jerry: Precisely right, except it's not the systems that are imperfect, but our understanding of them. You can see this idea worked out in detail in my book on software testing: Perfect Software and Other Illusions about Testing. It's extremely popular among software testers. They use it to explain to their customers what amounts to the Law of Medium Numbers applied to software. And, a few software developers have shown violent reactions to the claim that perfection is simply not possible.

Beyond Agile Programming

After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history. In order to contribute my bit to addressing that problem, I've posted this essay—one that's sure to infuriate many of my readers, including some of my best friends. So first let me tell you how it came about.

While reformatting my book, Rethinking Systems Analysis and Design for e-booking, I noticed a few places that might have needed updating to present realities. The version I was using was more than 20 years old, from just after the peak of excitement about "structured programming." In particular, there was a whole section entitled, "Beyond Structured Programming." As I contemplated updating that section, it dawned on me that I could almost update completely by substituting the name of any more recent "movement" (or fad) for the word "structured.

 I also knew how smart most of my readers are, so I figured they would see the same possibility without my updating a thing. Instead of changing the book, I decided to update the section and publish it on this blog. Why? Because I think it shows an important pattern—a script where only the names have changed over at least five decades. So, here is the section with "agile" substituted for "structured," just as "structured" had been substituted for some other fad a generation earlier.

The Restructured Essay
Before I proceed further with the task of rethinking systems analysis and design, I'd like to express myself on the subject of another great "rethinking" in programming—the agile programming revolution. Although this essay was written a generation ago (now two generation), and the agile programming "revolution" is now an exhausted fad (for most programmers), most of what this essay says still applies—though to the next rethinking fad, and the next, and the next. I believe it will still apply long after I'm no longer writing new editions. Why? Because our industry seems to require a new fad every decade to keep itself from being bored. So, just apply the lessons to whatever fad happens to be dominating the computer press at the time you're reading this.

Before anyone becomes overly enthusiastic about what the rest of this book says, I want to take stock of what this great agile rethinking has done. I don't claim to be starting a new revolution of the magnitude most of the fads claim, so I'd like people to realize how slow and how small the agile programming movement has been, in case they think this book is going to make much difference.

My own personal stock-taking on the subject of agile programming is based on visits to some forty installations on two continents over the past ten years, plus a few hundred formal and informal discussions with programmers, analysts, managers, and users during the same period. Because of the conditions under which these visits and interviews took place, I would estimate the sample is quite heavily biased toward the more progressive organizations. By "progressive," I mean those organizations more likely to:
• Send staff to courses
• Hire outside consultants, other than in panic mode
• Encourage staff to belong to professional organizations, and to attend their meetings.
Consequently, my stock-taking is likely to be rather optimistic about the scope and quality of the effects of agile programming.

The first conclusion I can draw from my data is this:
Much less has been done than the press would have you believe.
I interpret the word "press" very loosely, including such sources as:
• Enthusiastic upper management
• The trade press
• The vendors and their advertising agencies
• The universities, their public relations staffs, and their journals
• The consulting trade.
Although this may be the most controversial of my observations, it is the most easily verified. All you need do is ask for examples of agile programming—not anecdotes, but actual examples of agile behavior and agile-produced code. If you're given any examples at all, you can peruse them for evidence of following the "rules" of agile programming. Generally, you will find:

a. Five percent can he considered thoroughly agile.

b. Twenty percent can be considered to follow agile practices sufficiently to represent an improvement over the average code of 1990.

c. Fifty percent will show some evidence of some attempt to follow some "agile rules," but without understanding and with little, if any, success.

d. Twenty-five percent will show no evidence of influence by any ideas about programming (not just agile) from the past twenty years.

Please remember: these percentages apply to the code and behavior you will actually see in response to your request. If you ask software organizations at random for "agile examples," about two-thirds will manage to avoid giving you anything. We can merely speculate what they do, and what their code contains.

My second conclusion:
There are rather many conceptions of what agile programming ought to look like, all of which are reasonably equivalent if followed consistently.
The operative clause in this observation seems to be "if followed consistently." Some of these conceptions are marketed in books and/or training courses. Some are purely local to a single installation, or even to one team in an installation. Most are mixtures of some "patented" method and local adaptations.

My third observation:
Methods representing thoughtful adaptations of "patented" and "local" ideas on agile programming are far more likely to be followed consistently.
In other words, programmers seem disinclined to follow an agile methodology when it is either:
1. Blind following of "universal rules"
2. Blind devotion to the concept: anything "not invented here" must be worthless.

My fourth observation:
I have other observations to make, but now I must pause and relate the effect these observations have on many readers, perhaps including you. I recall a story about a little boy who was playing in the schoolyard rather late one evening. A teacher who had been working late noticed the boy and asked if he knew what time it was.
"I'm not sure," the boy said, "but I know it isn't six o'clock yet."
"And how do you know that?" the teacher asked.
"Because I'm supposed to be home at six, and I'm not home."
When I make my first three observations about agile programming, I have a similar reaction—something like this:
"These can't be right, because if they were right, why would there be so much attention to agile programming?"

In spite of its naive tone, the question deserves answering. The answer can serve as my fourth observation:

Agile programming has received so much attention for the following reasons:
• The need is very great for some help in programming.
• To people who don't understand programming at all, it seems chaotic, so the term "agile" sounds awfully promising.

• The approach actually works, when it is successfully applied, so there are many people willing to give testimonials, even though their percentages among all programmers may not be great.

• The computer business has always been driven by marketing forces, and marketing forces are paid to be optimistic, and not to distinguish between an idea and its practical realization.

In other words, the phrase "agile programming" is similar to the phrase"our latest computer," because each phrase can be used interchangeably in statements such as these:

• "If you are having problems in information processing, you can solve them by installing our latest computer."

• "Our latest computer is more cost effective and easier to use."

• "Your people will love our latest computer, although you won't need so many people once our latest computer has been installed."

• Conversion? No problem! With our latest computer, you'll start to realize savings in a few weeks, at most."

So actually, the whole agile programming pitch was pre-adapted for the ease of professionals, who have always believed "problems" had "solutions" which could be mechanically applied.

My final observation is related to all of the others:
Those installations and individuals who have successfully realized the promised benefits of agile programming tend to be the ones who don't buy the typical hardware or software pitch, but who listen to the pitch and extract what they decide they need for solving their problems. They do their own thinking, which includes using the thoughts of others, if they're applicable. By and large, they were the most successful problem solvers before agile programming, and are now even more successful.

There's yet another lesson in all this that's much bigger than agile programming or any new hardware or software or process:

Our profession contains few, if any, easy solutions. Success in problem solving comes to those who don't put much faith in the latest "magic," but who are willing to try ideas out for themselves, even when those ideas are presented in a carnival of public relations blather.
Based on this lesson, I'd like to propose a new "programming religion," a religion based on the following articles of faith:

• There's no consistent substitute for a thorough understanding of your problem, though sometimes people get lucky.

• There's no solution applicable to every problem, and what may be the best approach in one circumstance may be precisely the worst in another.

• There are many useful approaches applicable to more than one problem, so it pays to become familiar with what has worked before.

• The trick to problem solving is not just "know-how," but "know-when"—which lets you adapt the solution method to the problem, and not vice versa.

• No matter how much you know how or know when, some problems won't yield to present knowledge, and some aspects of the problem nobody currently understands, so humility is always in order.

I realize writing a book is not the most humble thing a person can do, but it's what I do best, and how I earn my living. I'd be embarrassed if anyone took this book too seriously. We don't need another "movement" just now, unless it is something analogous to a bowel movement—something to flush our system clean of waste material we've accumulated over the years.

Where to read the original
If you want to check on my historical work, you can find the original essay (and many others) in Rethinking Systems Analysis and Design, which is an ebook on Smashwords (where you can probably see it in the free sample) and Kindle and Barnes and Noble.


Problem-Solving Leadership Workshop
Reminder: The second (and last) PSL Workshop for 2011 will take place in Albuquerque, New Mexico, USA, August 28-September 2, 2011. Only a few places left for participants, so for more information, see <http://www.estherderby.com/workshops/problem-solving-leadership-pslhttp://www.estherderby.com/workshops/problem-solving-leadership-psl>

Was There Process Before Agile?

Reader June Kim recently wrote:

On p.328 of Volume 4 of your Quality Software Management series, the first line goes:

"Here's another example, mixing incremental development and hacking:"

In this Chapter 18, the process models you mention and describe are Waterfall, Cascade, Iterative Enhancement, and Prototyping (with Hacking and Rapid Prototyping as its variants). There isn't incremental development.

"incremental development" as you wrote on that page, doesn't appear before that line in the chapter.

Is Agile An Example of Incremental Development?

QUESTION: Which process model are you referring to when you wrote "incremental development"?
Is it one among the four process models that you mentioned earlier in the chapter, or Is it something different?

ANSWER: First, you have to remember that when QSM was written, there was no "Agile" development craze. We were doing various development process, some of which were given capital letters, some which were not, and some which were "owned" by certain advocates. I was trying to be descriptive then, not favor anybody's pet process.

There were some "owned" processes (using the hated word, "methodology," and charging tens of thousands of dollars for shelves full of notebooks which nobody read). I suppose some of them are still around, but most of the organizations I work with are now smarter than to fall for that fallacy. For the most part, every organization custom-tailored its own process (or, in most cases, processes, plural).

Of course, that's still true today. I don't find many organizations using some "pure" version of agile.

Where Do You Place Agile?

QUESTION: I am interested where you would put Agile process. I think Agile(XP and Scrum, for example) is closest to Rapid Prototyping as you described.

ANSWER: Historically, the people who first named "Agile" processes were borrowing the best of all these methods. You could also say that any agile process is a cascade (or iterative enhancement if you're actually putting each iteration's output into use). Agile is much more than these processes, making explicit many team practices to support the iterative nature of the development.

What Happens When the Customer Won't Participate?

QUESTION: If so, it has the same danger when the customer isn't willing to be the integral part of the process.

ANSWER: That's always the case, no matter what the method, if the customer is reluctant to participate. (Until the end, when they whine, "But that's not what we wanted.")

QUESTION: What could you do in this case? Drop Agile process?

ANSWER: It's not an Agile process if the customer (or surrogate) isn't participating. In fact, I would drop any customer who doesn't participate. That's the rule I use in all my consulting, too. I don't believe you can help people who aren't willing to help themselves.

QUESTION: Or, make the customer be the part of the process? Then how? This is still a hard question to me, even with 10 years of experience in agile.

ANSWER: It's definitely one of the hardest questions with Agile or any method. Hard for most technical leaders because they lack the training and skills to work with reluctant customers.

So, I train them in these skills (a major goal of the AYE Conference and the PSL workshop), but primarily everything starts by simply pausing the work unless and until the customer has been identified and persuaded to participate.