How do I become a programmer who gets stuff done?

The young man wanted to know, "How do I become a programmer who gets stuff done?" He received a number of good answers, like how to organize his work and schedule his working hours. Yet I had a different view of things, so I gave a different sort of answer, as follows.

Some good advice here, yet even if you do all these suggested things, you may be having a different problem. When you speak of getting stuff done, you may be speaking of “finishing” things.

Defining what “done” means has been a classic programmer problem for more than 50 years. Part of the problem is that programmers with different personalities have different ideas about what “done” means. For instance, look at the situation from the point of view of MBTI personality temperaments:

NTs tend to think a project is done when they have a clear description of the problem and a general approach to solving it. Someone else can work out the details.

SJs tend to think a project is done when it’s “code complete”—though research at Microsoft and other places seems to indicate that only about two-thirds of the eventual code has been written by that supposed benchmark. Perhaps this work isn't thought of real programming, but only "clean up."

NFs, such as me, tend to think that a project is done when everyone involved is satisfied that it’s done. Of course, because these "others" are of various personality types, our NF estimate of "done" isn't very reliable.

SPs tend to think a project is done when they are bored with doing it. They share this feeling with lots of other temperaments, too. Programmers in general don't tolerate boredom very well.

Not taking that last step to "clean up" is a curse of our profession, leaving many programs in a less-than wholesome state. Unclean, unfinished programs are the source of many maintenance problems, and of many errors shipped to customers.

Of course, this classification is only a rough model of non-finishers. Many good programmers of all temperaments are excellent finishers, having taught themselves to be aware of their tendencies and counter them with a variety of tactics. Primary among those tactics if the technical review by peers (which is an integral part of the Agile approach but  by no means is not confined to Agile).

So, you offer your “finished” project to your peers for review, and if they agree that it’s finished, you’ve truly gotten something “done.”

If they don’t agree that your work is done, they will give you a list of issues that you need to address before you’re “done.” You then go back to work and address these issues, then resubmit the newest version to another technical review.

Finally, you iterate in this reviewing process until your work passes the review. Then you know you’ve gotten something done.

For more detail on the review process, see, 

Handbook of Technical Reviews, 4th edition

Is Waterfall the Opposite of Agile?

Agilists sometimes behave unreasonably by pummeling the Waterfall approach. Personally, I think such evangelists could better spend their effort producing solid, relevant code, but evidently they are on a crusade and need an enemy. If they need an enemy of Agile, however, they could have made a better choice.

There's nothing wrong with the Waterfall approach—if applied appropriately. For Waterfall to work well, you need to solve a problem that's not destined to change much, or, ideally, to change not at all. Some Agilists claim there is no such unchanging problem, but they're merely showing their lack of experience. I've seen a number of such invariant problems, and they yield very readily to a properly applied Waterfall development.

For example, Erik, a former student of mine, made a lucrative business of converting COBOL programs to new COBOL programs adapted to new versions of the COBOL compiler. Erik's customers wanted assurance that nothing would be changed in the conversion. This was a perfect situation for a simple Waterfall approach, one that Erik could perform profitably at a fixed price and schedule.

That said, the number of such invariant problems is not large, and it's usually difficult to know at the beginning if a problem will turn out to be so stable. In Erik's case, some of his customers would decide midway that they wanted a few "tiny" improvements in the converted application. Erik controlled that situation by charging outrageous fees for even the simplest modification. Most of us, however, try to control this situation by employing some Agile approach.

Let's be honest with ourselves: one consequence of an Agile approach is the loss of our ability to work to a fixed cost on a fixed schedule. Erik could do that, but only in a few carefully controlled situations. Many managers frustrate themselves over this lack of control and blame it on Agile. What's really to blame, however, is their inability to control the world in which they live. Things do change, and much of the time it's these very managers who instigate the change.

What do frustrated managers do? Quite often, they elevate their attempts to control the change by making rules. They may start using a pure Waterfall approach, but as their frustration with changes grows, they may add a Change Control Board, or change reviews, or a Change Tsar, or any of a number of other tactics. And, when those tactics fail to produce absolute predictability, they add more of the same kinds of rules and their supporting tactics.

After a while, these rules upon rules produce an approach that, though called "Waterfall," is actually something quite different—something for which we so far have no accurate name. This "something" is what Agile is responding to, so I suggest we name it.

What are these cobbled-together approaches like? First of all, they create a sad and dismal mood among those poor developers condemned to use them. When I visit a new client, I can generally detect the use of such an approach while I take a stroll through of the premises. I can even detect such approaches over the phone. How? Simple: the mood of my clients is mournful, gloomy, sad, unhappy, doleful, glum, melancholy, woeful, miserable, woebegone, forlorn, somber, solemn, serious, sorrowful, morose, dour, cheerless, joyless, and dismal.

That's quite a sobering list of adjectives, but that's what I can sense in many so-called "Waterfall" environments. Perhaps you recognize the list, but in any case, you can find that list in your dictionary as synonyms for the rare word, "lugubrious."

Perhaps the word "lugubrious" is unfamiliar, but that's good, because we won't often find it used in other contexts. Besides, it's a rather onomatopoetic word—a word that phonetically imitates, resembles or suggests the source of the sound or situation it describes. That's why Agile was invented—to replace those mournful, gloomy, rule-dominated approaches with brain-driven judgments of the actual builders.

So let's be truly Agile and stop bashing the true Waterfall approach. Instead, let's turn our contempt on every Lugubrious approach—or, to make a noun from the adjective, Lugubriousity. Maybe this more accurate name will help us defend our Agile projects from frustrated managers' attempts to smother us with yet more rules.

Or, to paraphrase DeMorgan, who in turn paraphrased Swift:

Great rules have little rules upon their backs to spite 'em,

And little rules have lesser rules, and so ad infinitum.

Never forget that's why we do Agile, not to dry out Waterfalls but to defeat Lugubriosity.

For more thoughs on the Agile approach, see

Agile Impressions

The Order of Maria Theresa

Today's idea is embodied in a medal established in Austria. According to Wikipedia, the military order of Maria Theresa. "It was specifically given for 'successful military acts of essential impact to a campaign that were undertaken on [the officer's] own initiative, and might have been omitted by an honorable officer without reproach.' This gave rise to a popular myth that it was awarded for (successfully) acting against an explicit order."

The Order of Maria Theresa is a marvel of bureaucratic invention, but it's not unique. Every successful organization—nation, business, or neighborhood kite club—has rules for breaking its own rules. The only unusual aspect of the Order of Maria Theresa is that the rule was written down and officially recognized.

When Jefferson was drafting the United States Constitution, he naturally wrote an article concerning amendments. But when asked to write something granting the people the right to throw out the Constitution entirely and start afresh, Jefferson refused. He argued—correctly, I think—that the people had such a right whether or not it was written in the Constitution. It was a right superseding any government and any written rules of government. It was, in effect, a tautology, for without the consent of the governed, there is no government. A shadow, perhaps, but no government.

The same is true in any modern bureaucracy. Rules are not made to be broken, but neither are they made to be not broken. Rules are made so that the organization operates more effectively. The rule above all other rules is "Do what is necessary to operate effectively." You ultimately get punished for not operating effectively, but not for breaking the rules.

It seems to me the problem with bureaucracies is this: the obverse side of this coin isn't so shiny. If you obey the rules and things turn out badly, you don't get punished. In war, the problem may be easier, because if things turn out badly you may be dead and not have to face the Empress. In the XYZ Pork and Bean Factory, your life is not usually at risk—even if your customers are taking their lives in their hands every time they pick up a pork fork.

I'd be interested in pursuing the biographies of Maria Theresa winners after they won the medal. I know that in the United States—where the Medal of Honor is often given under similar circumstances—many winners wind up, as civilians, trying to get a few cents for their medals in a pawn shop. Even medal-winning is a short-lived glory in the best of circumstances.

When I started to write this essay, I hoped to conclude by recommending each organization install an Order of Maria Theresa to counteract the conformist tendencies infecting even the best-managed organizations. But as my thoughts developed, I realized medals are not the answer. If you're on top of a large organizational pyramid and want protection from your own mistaken orders, you're going to have to work harder than Maria Theresa.

You can start by ensuring nobody is punished merely for discussing the merits of a particular order of yours. Even if you don't punish discussion, you'll need a long time to overcome the fears people have learned throughout their long careers in other organizations. But the long wait will be worth it, for then you will be relieved of the burden of perfection—a burden no person and no nation can long endure.

You might think your next step would be to encourage people to disobey orders they think are wrong or foolish, but they won't need any encouragement if the climate for talking is right. At least some of them won't, and the others will be watching to see what happens to the pioneers. 

So your second problem comes when someone disobeys—and fails! Now you have the perfect opportunity to play "I told you so," but if you do, there won't be any further games. Instead, you must convince the person to tell you why the order was disobeyed. It might have been a stupid order, destined to fail even if it had been carried out. Or it might have been misunderstood—a most likely alternative.

And once you've understood the reasons for the disobedience, drop the whole matter! Everyone is entitled to make a mistake now and then. If people never make mistakes, it means they're never trying and never thinking, which is the most horrible fate a bureaucracy can contemplate. Only if the same person disobeys orders over and over will you have to take any action—and by then your course of action should be obvious.

But won't the repeated failures create a disaster? Perhaps, but then you can take comfort from the Austrian motto: The situation is hopeless but not serious.

In the programming business, we have the comfort of a large cushion of safety in what we do. We make many mistakes, but we have procedures designed to detect them and remove them before they cause too much harm. If those procedures are working well, it gives us some breathing room in which to make mistakes—the same room we need in order to learn. In such situations, we shouldn't need medals to keep us disobeying foolish orders.

This essay is adapted from a chapter in the book, Understanding the Professional Programmer. The book is actually a series of such essays, all aimed at the often difficult task mentioned in the title.

Switching Topics at Conference Presentations

My problem-solving letters seem to be very popular, so I'll continue this series whenever I have an interesting situation to discuss. Today, it starts with a letter from a writer colleague, let's call him Edgar.

Edgar's Letter
This weekend I'm presenting at a conference.

This morning I got a revamped schedule and suddenly I'm doing a break out session on "Coping with Foreign Withholding Taxes."

I'm in a bit of a panic because I've spent the last three months putting together a totally different presentation about "Working with Translators." Nothing like giving a presenter last minute notice. :}

Problem is, I have zero experience with foreign withholding taxes. I keep thinking I need to look into it, but have never had the time. If anyone has experience or thoughts that I could share at the conference I could really use some help.

Jerry's First Thoughts
This is an excellent example of a person offering a solution idea, rather than a problem statement. Edgar is asking for information on foreign taxes, presumably to help him cobble together a last-minute talk on the subject. In other words, he's already decided that he has to solve this problem by giving in to the organizers' totally unreasonable demands.

If he does that, his presentation will merely reveal him to be a fake, a pretender, not the real expert people at the conference would have a right to expect. In the long run, that's only going to tarnish Edgar's reputation as a professional, so I recommended a different approach altogether.

Jerry's Reply
Edgar, I sympathize with your predicament. In half a century of presenting at conferences, something similar has happened to me twice.

The first time, I didn't know how to handle it. I bungled around trying to wing it on the new topic. (I knew a bit about it, but probably not as much as some people in the audience, and in any case, I wasn't prepared.) I looked fake, and/or stupid, and news in our profession travels fast. Especially bad news.

Some years later, the same thing happened to me. This time, I knew what do do. I came to the session room a few minutes before the start time. As people arrived, I warned them that the announced topic had changed. But most people came just at the last minute, so they didn't hear my warning.

I gave them a few minutes to settle. Then I said, "The topic you see in your program is 'Coping with Foreign Withholding Taxes.' However, four months ago, when I agreed to do this session, I was told the topic was 'Working with Translators.' So, I have prepared on that topic for four months, but I haven't prepared at all for the Foreign Taxes topic. In fact, I know virtually nothing about that topic. So, if that's what you want to hear about, this isn't the place to hear it."

I went on to say, "If, however, some of you are interested in Working with Translators, stick around and I'll make my presentation."

Most of the people actually stuck around, and liked the presentation. If my topics had been like the two you mention, I might have said, "I assume you came here today because you're interested in doing business overseas. One way to help that happen is to handle the taxes wisely, but another way is to obtain good translations of your writing. After all, if you have no overseas business, you won't have any foreign taxes. So, this presentation could serve your purposes after all, especially since there are no other presentations on translations."

I may also have said (I don't remember, but I was younger and more impetuous then), "If you're unhappy about this last-minute switch, you might want to tell the conference organizers about your feelings. It won't help to tell me, as I had nothing to do with it."

References

I hope this story helps you a little. It's the best I have to offer.

If you'd like to learn more about my approach to solving problems such as this last-minute switch, you might want read one or more of my books, such as The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully; More Secrets of Consulting: The Consultant's Tool Kit; and Are Your Lights On: How to know what the problem really is.

Links to each of my books can be found on my website: http://www.geraldmweinberg.com

If you enjoyed this little essay, please sign up to come back for more.

Thanks,

Jerry

Why English Will Never Be 100% Automated: Example

One of the nice features of the Kindle eBook service is they way they copy-edit some of their better-selling books. This can be a particularly important service for print books that have been scanned to make eBooks. For instance, Amazon recently wrote about my Kindle book, An Introduction to General Systems Thinking:

Typo issues exist that may have been caused by an Optical Character Recognition (OCR) problem. Few examples are given below:

loc 1561 - "T call them" should be " I call them".
loc 2946 - "But 1 still can't" should be "But I still can't".
loc 3351 - "Shasta the liger" should be "Shasta the tiger".

Please look for the same kind of errors throughout the book.

The first two instances are common OCR (Optical Character Recognition) errors: "T" for "I" and "1" (the numeral) for "I" (the capital letter).

The third example is a not quite so common OCR error, "l" (the letter) for "t". And, in this case, it's not an error at all.

The sentence in question was:

We do not often have our excessively sharp view of the world challenged by phenomena like Shasta the liger at the Salt Lake City Zoo, whose father was an African lion and whose mother was a Bengal tiger.

Still, the sentence contains a more subtle error: the failure of the author (me) to account for the mental state of the typical reader, for whom the term "liger" may be unfamiliar. (Even though it is found in at least 16 on-line dictionaries.)

I corrected this much more subtle error by redrafting the questionable sentence as:

We do not often have our excessively sharp view of the world challenged by phenomena like Shasta the "liger" at the Salt Lake City Zoo, whose father was an African lion and whose mother was a Bengal tiger.

As I dealt with this situation, I kept sighing and thinking, "English will never be entirely automated."

And then I took a deep breath and thought, "I suppose I prefer it that way."

How about you? Would you like English to be entirely clear and logical?

An Introduction to General Systems Thinking

The Assumption of Fixed Requirements

Note 1: This post is extracted from Chapter 9 of my book, CHANGE: Planned and Unplanned.

Note 2: This book was written for project leaders in high-tech industries, but writers are also project leaders, and writing certainly requires great skill and precision. For writers, requirements may originate from publishers, agents, and for-hire customers—any of whom can cause unending grief by changing those requirements for a writer who has assumed they were fixed.

Until recently, the computing industry seems to have avoided the subject of requirements the way a debutante might avoid the subject of indigestion. We knew such things existed, but if we didn't think about them, perhaps they would simply take care of themselves.

Many of the classic early papers in software engineering were based on the position:

This is how we would design and build software (if we had unchanging requirements.)

[For writers: each of us has her/his own process for writing, but for many of us, that process is also based on having unchanging requirements. If someone changes our task, we may be thrown off our game, and into write-stopping turmoil.]

For instance, many of the early papers on structured programming were based on the Eight Queens Problem, a problem of fixed definition with no input whatsoever. Many papers on recursive programming were based on the Towers of Hanoi problem, another problem of fixed definition with no input whatsoever. The more recent Cleanroom methodology has the same basis: "The starting point for Cleanroom development is a document that states the user requirements." The following quotation from Parnas and Clemens shows how deeply this assumption runs, even in the most sophisticated process designers.

Usually, a requirements document is produced before coding starts and is never used again. However, that has not been the case for [the software requirements for the A-7E Aircraft]. The currently operational version of the software, which satisfies the requirements document, is still undergoing revision. The organization that has to test the software uses our document extensively to choose the tests that they do. When new changes are needed, the requirements document is used in describing what must be changed and what cannot be changed. Here we see that a document produced at the start of the ideal process is still in use many years after the software went into service. The clear message is that if documentation is produced with care, it will be useful for a long time. Conversely, if it is going to be extensively used, it is worth doing right.

Parnas and Clemens describe the benefits of returning after design to create a requirements document as if it had been present from the beginning.

In the light of all this literature, it's easy to understand why so many software engineering managers have made the mistake of believing they should have unchanging requirements before starting any project. This model or belief is what I call the Assumption of Fixed Requirements, an assumption that is a misreading of these classical works. These classics were not addressing the entire process of software engineering, but only selected parts of that process. What they are teaching us is how to translate requirements into code, once we have reliable requirements.

Translating requirements into code is an essential part of software engineering, and it is the part that has received the most research attention over the past decades. Because of that attention, however, it is no longer the most difficult part of the process. Many organizations know how to do this part quite well, but the quality of their products does not adequately reflect their coding prowess.

In recent internal studies of serious quality problems, three different clients of mine arrived at quite similar conclusions. They divided the sources of the most serious problems into a number of categories, including coding and gathering requirements. In all cases, coding contributed least of all categories to quality problems, and my clients would perhaps do better to work on the less glamorous job of improving their logistics processes. Perhaps these rather advanced organizations are not typical of all software engineering organizations. They still have a lot to learn about coding and especially design, but in each case, the majority of their serious problems stem from requirements.

Software engineers and their customers perceive quality differently, and this table accounts in large part for that discrepancy in perception. Over the past decades, engineers have seen coding faults drop dramatically as a result of their quality improvement efforts. The customers, however, have not seen a comparable decrease in the number of requirements problems, and so do not perceive the same increase in quality as the engineers, who are paying attention to their own priorities—which don't happen to coincide with their customers' priorities. The engineers need to learn that they will never become an Anticipating organization by getting better and better at coding—even though that was the improvement process that brought them as far as they've come.

The Exception is the Rule

Recently, I was trying to help a client (let me call them "StartupCompany") mired in conflicts, exceptions, errors, anomalies, lapses, modifications and other deviations from the norm. These annoying exceptions were playing tricks with my blood pressure, so I had to be wired to a wearable blood pressure computer for twenty-four hours. As if StartupCompany didn't have enough interruptions, now my wearable computer was inflating a blood pressure cuff at random intervals throughout the day.

Every time the cuff inflated, I petulantly asked myself: Why can't they run a project like real people living run-of-the-mill, low-blood-pressure lives?

That night, I was using the Yellow Pages, and in the A categories in the Yellow Pages index, I chanced to notice a curious pattern. Here are the first few items:

Abortion Services and Alternatives. These were the first two entries in the index. I decided to skip them both, so as not to take sides in the pro-choice/pro-life conflict. I had enough conflicts within StartupCompany.

Abuse - Men, Women, Children. I decided to continue my scan of the index, and this was the next entry. The normal process of family living involves people loving and respecting each other, communicating well, and behaving appropriately according to societal norms. But when people start behaving inappropriately, they need Abuse Services. In StartupCompany, people normally respected one another, communicated well, and behaved appropriately according to societal norms. But they sometimes didn't, and they lacked "abuse services" for coping.

Academies (including private schools and special education). When the formal education system doesn't provide special knowledge or handle special cases, private academies and special education are called for. People within StartupCompany often needed to know things they hadn't learned in the public schools, but StartupCompany had no provision for special education.

Accident Prevention. Accidents aren't "supposed" to happen, StartupCompany had accidents. In order to improve, they needed processes to prevent accidents and to mitigate their consequences.

Accordions. Despite what some people think, accordions are perfectly normal, though not everybody learns to play them or appreciate them. Still, StartupCompany could have used some entertainment to lighten the mood once in a while.

Accountants. Accounting is also normal, but, if everything always went according to plan, we wouldn't need to account for things so carefully. We have to protect our financial well-being from mistakes and misbehavior, and that's what accountants do - and also what they should have been doing in StartupCompany.

Acetylene Welding. Some welding is normal, and some is for repairing things that are not supposed to break - but do anyway. StartupCompany lacked a "welding team" to handle lots of stuff that broke.

Acrylic Nails. Most normal people have fingernails, so why is there a nail business? Oh, yes, it's the human interface, and StartupCompany had to cope with conflicting ideas of what made a system beautiful - but they had no special beauty experts to resolve the conflicts.

Acting Instruction. We all need to "put on an act" now and then when we're caught by surprise. StartupCompany's people certainly needed training in how to behave in improvisational situations, but there was no acting instruction.

Acupressure/Acupuncture. If we were all healthy all the time, we wouldn't need medical services, and if "normal" Western medical services worked all the time, we wouldn't need acupressure and acupuncture. So, there are not only abnormal services, but meta-abnormal services - the services when the normal abnormal services fail - certainly true in StartupCompany.

Addressing Service. Have you ever tried to maintain a mailing list? Almost all the work is not the mailing itself, but maintaining the addresses. It's even worse for email, because email services haven't yet evolved "normal" ways of dealing with changes. Gee, neither had StartupCompany.

Adjusters. Adjusters, of course, are an abnormal service from the get-go. Without accidents, we wouldn't need insurance, and if things stayed on course, StartupCompany wouldn't have needed risk analysis. But they did.

Adobe Materials and Contractors. Adobe materials may not be "normal" where you live, but here in New Mexico, adobe is a normal building method. StartupCompany, too, has its idiosyncratic processes that are not normal in other projects - and newcomers have to learn about them or pay the price. But StartupCompany had no special services to bring newcomers up to speed.

Adoption Services. Yes, sometimes people are not wanted by their parents, and StartupCompany certainly had some unwanted people. But, they lacked "adoption" services for moving unwanted people around.

Adult Supervisory Care. "Normal" adults can take care of themselves without supervision, and normal workers wouldn't need much managing at all. But StartupCompany had two adults who could not take proper care of themselves, and the managers spent an inordinate amount of time on these two out of a hundred.

I stopped there, sobered by my reading. It was now clear to me that StartupCompany, being a startup, had an overly simplistic picture of what it takes to run a company. I needed an adjustor to adjust my blood pressure - I needed to see that my job as their consultant was to teach them that deviations are normal, and that they (and I) could do what real people do:

• stop whining and deal with them

• create systems to deal with them

• create systems to prevent them

And, of course, I have to do these three things in my own company - like not whining about my blood pressure.