How do I fix a really difficult bug in programming?

Here was the question:

"How do I fix a really difficult bug in programming?"

Here was my first answer:

There is no such thing as a “difficult bug.”

I suspect my answer requires further explanation. First of all, I doubt that you have experienced actual bugs in your computer, the kind with 8 legs that bite and swarm. I have, a couple of times, but they are rare, and usually not difficult to eradicate.

Perhaps you are talking about errors, but using inaccurate language. In that case, I will assert “there is no such thing as a difficult error.” The same error might be handled easily by a different person. I have seen that circumstance often. For instance, I once spent a month trying to pinpoint a coding error. When I finally asked the help of a colleague, she found it in less than two minutes.

No, there are no difficult errors, but there are people who have difficulty with an error. We have all been there, and we tend to want to blame the error rather than ourselves.

So, the first thing you need to do to handle a “difficult bug” is to ask yourself,

“What is it about me that is making this error so difficult to handle?”

Perhaps you are having difficulty because you are impatient, or think failure to handle the error will make you look bad to your boss or colleagues.

Perhaps pressure to handle the error is throwing you off your center, distorting your thinking.

Perhaps you do not know enough about the system with the error, or the language in which the program is written.

Perhaps your mind is on other things in your life, things distracting you because they are more important to you than this darn “bug.”

Maybe you should discuss this error with a colleague or two, What is it about you that is keeping you from doing that?

Anyway, good luck in your quest for resolution.

Take a look at Errors: Bugs, Boo-Boos, and Blunders

Which code is more readable?

We were asked, "Which code is more readable, one that uses longer variable names or short ones?" 

Maybe some historical perspective will help answer this question.

In the very early days of computing (I was there), we used short variable names because:

* Programs were fairly short and simple, so scope wasn’t much of a problem.

• Memories were small, so programmers didn’t want to waste memory with long names.

• Compilers and assemblers were slow, and long names made them slower.

• Many compilers and assemblers wouldn’t allow names longer than a few characters, because of speed and memory limitations.

• We didn’t think much, if at all, about who would maintain a program once it left the hands of the original programmer.

As programs grew larger, one result of short naming was difficult maintenance, so the movement toward longer names grew stronger. It wasn’t helped by COBOL, which asserted that executives should be able to read code. Lots of COBOL code was littered with super-long names, but that didn't help executives read it.

The COBOL argument proved to be nonsense. Still, the maintenance argument for longer, more descriptive names made sense.

Unfortunately, like many movements, the long-name movement went too far, at least for my taste. It wasn’t because long names were harder to write. After all, a typical program is written oncem but read for modification and testing many, many times. So, if long names really made reading easier and more reliable, it was good.

But the length of a name is not really the issue. I’ve seen many programs with long, long names that were so similar that they were easily confused, one with another. For instance, we once wasted many days trying to find an error when the name radar_data_station_#46395_azimuth_reading was mistaken for radar_data_station_#46895_azimuth_reading. Psychologists and writers know well that items in the middle of long lists are frequently glossed over.

So, like lots of other things in software development, long versus short names becomes a tradeoff, a design decision for a programmer for which there is no “right” answer. Programmers must design their name-sets with the same kind of engineering thought they put into all their design decisions.

And, as maintainers modify a program, they must maintain the name-set, so as to avoid building up design debt as the program ages.

So, sorry, there’s no easy answer to this question, nothing a programmer can apply  mindlessly. Just as it’s always been, programmers who think will do a better job than those who blindly follow simplistic rules.

The Psychology of Computer Programming: Silver Anniversary eBook Edition

Terchnical Reviews and Organizational Renewal

We know that success can breed failured and doesn't automatically renew itself. I would like to offer some ideas on how this self-defeating tendency can be resisted.

One way toward renewal is through new perspectives gained from outside technical audits, but audits suffer from several serious disabilities. For one thing, audits are done intermittently. In between one audit and the next, the programmers don't stop programming, the analysts don't.stop analyzing, and the operators don't stop operating. Sometimes, though, the managers stop managing, And there's the rub.

A comparable situation exists when a firm has a system of personnel reviews mandated, say, every six months for each employee. Under such a system, managers tend to postpone difficult interactions with an employee until the next appraisal is forced upon them. A huge dossier may be accumulated, but then ignored in favor of the last, most conspicuous blunder or triumph. Good managers realize that the scheduled personnel review is, at best, a backup device—to catch those situations in which day-to-day management is breaking down.

In the same way, the outside technical audit merely backs up the day-to-day technical renewal processes within the shop. It may prevent utter disasters, but it's much better if we can establish technical renewal on a more continuous and continuing basis. One way to do this is through a technical team, such as an Agile team. For now, though, I want to introduce, or reintroduce, the concept of formal and informal technical reviews as tools for renewing the technical organization.

The Agile Manifesto requires "technical excellence" and "simplicity," and states: 

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. 

To achieve these and other goals, Agile teams conduct "walkthroughs" and "inspections," but these are only two very specific examples of technical review procedures used by Agile teams. In my reading and consulting, I've uncovered dozens of major variants of these two types of review, plus literally hundreds of minor variants that shops have introduced to make the review more suitable to their environments, whether or not they claim to be "Agile."

A more general definition of a technical review could be 

1. a review of technical material on the basis of content (this is what makes it a "technical" review, rather than, say, a financial or personnel review)

2. which is done openly by at least two persons (this is what distinguishes it from "desk checking")

3. who take lull responsibility for the quality of the review (not, please note, for the quality of the product)

Then we distinguish between the informal and formal review. A formal review produces a written report to management. That report is the formal part.

Informal reviews (that is, reviews which need not terminate with a written report to management) are also excellent devices for an organization's self-renewal. Informal reviews take all forms of technical reviews, and are practiced everywhere almost all the time. Indeed, they are an essential part  of the real world of programming work.

For instance, a team member passes a diagram to a teammate for an opinion on how to represent a particular design. Someone asks help from someone else in finding an error. A set of test cases is passed around to see if anyone can think of something that's been overlooked. One person reads another's user document to see if it's understandable.

Without a constant flow of such informal reviewing of one another's work, programming any meaningful project would be impossible. Formal reviewing, on the other hand, is not essential to programming. Many small projects are accomplished without formal reviewing, which is why some programmers don't appreciate the need for formal reviewing. 

As projects grow 'larger and more complex—as they are inclined to do in successful shops—the work of many people must be coordinated over a long period of

time. Such coordination requires management—though not necessarily managers—and such management requires reliable information. Formal reviews, essentially, are designed to provide reliable information about technical matters—particularly to non-technical people.

Clearly, a successful system of formal technical reviews—that is, a system that provides management with reliable information on technical matters—is essential to management participation in the organizational-renewal process. For the large shop, formal reviews provide information that the small shop manager gets "through the seat of the pants." Many, many failures of previously successful programming organizations an be traced directly to the breakdown of the earlier informal mechanisms for communicating reliable information about technical matters.

There may, of course, be other ways of getting this kind of information, and many smaller shops do an excellent job without any system of formal reviews. Even those shops, however, may benefit from an explicit system of reviews to supplement their implicit, or informal, existing system. 

Principally, the formal technical review provides reliable self-monitoring. An informal system may work just as well as a formal one, and if so, there are many reasons to prefer to keep the reviewing informal. In any case, there will always be an informal system at least supplementing the formal one, but we should really view the formal system as supplementing the informal. Its formality guards against creeping deterioration of the organization.

Regular reviews of specifications, designs, code, test plans, documentation, training materials, and other technical matter have many beneficial "side effects" on the health and success of an installation. Reviews have a very marked effect on maintenance—that quicksand in which a thousand successful shops have met an untimely end. A typical program that has been thoroughly reviewed during its development will

1. require less maintenance work per change

2. require fewer changes caused by poor specification, design, coding, or testing.

Instituting a program of technical reviews will not, of course, have any immediate effect on the existing burden of maintenance carried like an albatross from a sinful programming past. Indeed, when maintenance programmers participate in reviews of newer code, they may be further discouraged by the poor quality of the code with which they are burdened. But, the situation can improve rather quickly, for a variety of reasons:

1. Some new ideas can be applied even to patches to old programs, though naturally limited by poor existing structure and style.

2. Through mutual participation in reviews, the entire shop quickly obtains a realistic and sympathetic picture of the true dimensions of the maintenance situation.

3. The worst maintenance problems will, through frequent reviews, become exposed to fresh air and sunlight.

Quite frequently, installation of a review system is quickly followed by replacement of the most cantankerous old beasts left over from the Stone Age of Programming. The effect of even one such replacement on maintenance morale is a wonder to behold,

Other activities besides maintenance are similarly affected. In the long run, certainly, reviews have a remarkable effect on staff competence, perhaps the most important element in continuing success. We also see increased morale and professional attitude, reduced turnover, more reliable estimating and scheduling, and better appreciation of the management role in project success. (Usually, the technical staff has had no difficulty in appreciating the management role in project failure.)

At the same time, reviews provide management with a better appreciation for staff competence, both individually and collectively. The unsung hero who quietly saved a dozen shaky projects is now sung far and wide. The "genius programmer" who was always the darling of the executives has been exposed for the empty and obsolete shell the technical staff always knew, but couldn't expose to management.

No other factor is more depressing to a technical organization than misappraisal of technical competence by non-technical management. The openness of technical reviews marks an end to that misappraisal era. No longer will we all feel cheated by charlatans and incompetents.

As a consultant, I visited dozens of installations every year. The best way to summarize the effects of properly instituted reviews is to report that after five minutes in an installation, I can tell—without asking directly—to what extent there is an effective review-practice, formal or informal.

How do I tell? Metaphorically, I tell in the same way a government health inspector tells about a food processing plant—by the way it smells. It's hard to describe, but when you smell the difference, you-know it!

* * * * * *

Looking back over this essay, I sense its inadequacy to the task at hand. Those programmers and analysts who have experienced a shop with a properly functioning system of reviews will know all this without my giving any details. They've experienced it, and if they are professionals, they'll never agree to work in any other kind of environment.

But those who have never experienced such an environment of a self-renewing organization will not understand, or will misunderstand, what I've written. Some of them will have experienced a misguided attempt to institute reviews. Perhaps the attempt was in the form of a punitive management mandate. Perhaps it was merely a case of another programmer who read one article and blundered ahead with 99% enthusiasm and 1% information—and zero human feeling. To these people, the experience of "reviews" will have left a bitter taste, or a painful memory. They will not recognize their experience in what I've written.

In many ways, technical reviewing is like bicycling. Up until the time you first discover your balance in one miraculous instant, the whole idea seems impossible, contrary to nature, and a good way to break a leg. Indeed, if you'd never seen a cyclist before, and had the process described to you, you'd most certainly declare the whole procedure impossible. And, without instruction and encouragement from experienced cyclists, along with reliable "equipment," the most likely outcome would certainly be skinned knees, torn pants, and a few lumps on the head. And so it has been with technical reviews—until now—-so don't go off and try to learn the whole process on your own.

If you want to get started renewing the success of your own shop through a system of technical reviews, find an experienced shop, or a person from an experienced shop, to guide you. Listen to them, Read the literature. Then try a few simple reviews on an experimental basis, without much fanfare.

Adapt the "rules" to your own environment. Be forgiving and humane. Your rewards  will not be long in coming.

references: 

Handbook of Technical Reviews, 4th ed.

Do You Want To Be A (Better) Manager?

What's it like to rewrite a program from scratch?

This is an interesting question because so many programmers are so afraid of this task that they would never even ask it. Is this reluctance agile, or Agile?

But why would you want to do rewrite a program from scratch? The most important reason is to increase maintainability. In the initial writing, the focus is generally on merely getting the job done, without thinking of the program's future. Over their lifetime, many programs cost far more to maintain than to write originally, especially when the original program has become a thing of rags and patches.

A second, and often secondary reason to rewrite a program is efficiency. Newly constructed programs and highly patched old programs sometimes turn out to be slower than desired, but such inefficiency cannot erased by any amount of tweaking. Instead, increased efficiency requires a new approach, an approach that needs to be implemented from scratch.

But isn't rewriting expensive? Not usually. In fact, it’s generally far, far easier to rewrite a program from scratch than to write some brand-new program.

Why would a fresh start-over be cheaper than the original? Because in writing the original program, the original programmers answered so many difficult questions about what was really wanted. Requirements haven't changed, and most of the thought put into testing can be reused.

Those questions—requirements and test—usually make up more than half the total work put into a program. They have already been answered—but only if you resist the temptation to change those answers. If you don’t resist, then rewriting the program can be arbitrarily difficult.

I wish more programmers had to courage to rewrite some clumsy programs from scratch, rather than patch and patch and patch. And I wish their managers would encourage sensible rewriting, rather than force programmers to waste their skills, time, and energy keeping ancient programs on life support.

The Psychology of Computer Programming: Silver Anniversary eBook Edition 

My most challenging experience as a software developer

Here is my detailed answer to the question, "What is the most challenging experience you encountered as a software developer?:

We were developing the tracking system for Project Mercury, to put a person in space and bring them back alive. The “back alive” was the challenging part, but not the only one. Some other challenges were as follows:

- The system was based on a world-wide network of fairly unreliable teletype connections. 

- We had to determine the touchdown in the Pacific to within a small radius, which meant we needed accurate and perfectly synchronized clocks on the computer and space capsule.

- We also needed to knew exactly where our tracking stations were, but it turned out nobody knew where Australia's two stations were with sufficient precision. We had to create an entire sub-project to locate Australia.

- We needed information on the launch rocket, but because it was also a military rocket, that information was classified. We eventually found a way to work around that.

- Our computers were a pair of IBM 7090s, plus a 709 at a critical station in Bermuda. In those days, the computers were not built for on-line real-time work. For instance, there was no standard interrupt clock. We actually built our own for the Bermuda machine.

- Also, there were no disk drives yet, so everything had to be based on a tape drive system, but the tape drives were not sufficiently reliable for our specs. We beat this problem by building software error-correcting codes into the tape drive system.

We worked our way through all these problems and many more smaller ones, but the most challenging problem was the “back alive” requirement. Once we had the hardware and network reliability up to snuff, we still had the problem of software errors. To counter this problem, we created a special test group, something that had never been done before. Then we set a standard that any error detected by the test group and not explicitly corrected would stop any launch.

Our tests revealed that the system could crash for unknown reasons at random times, so it would be unable to bring down the astronaut safely at a known location. When the crash occurred in testing, the two on-line printers simultaneously printed a 120-character of random garbage. The line was identical on the two printers, indicating that this was not some kind of machine error on one of the 7090s. It could have been a hardware design error or a coding error. We had to investigate both possibilities, but the second possibility was far more likely.

We struggled to track down the source of the crash, but after a fruitless month, the project manager wanted to drop it as a “random event.” We all knew it wasn’t random, but he didn’t want to be accused of delaying the first launch.

To us, however, it was endangering the life of the astronaut, so we pleaded for time to continue trying to pinpoint the fault. “We should think more about this,” we said, to which he replied (standing under an IBM THINK sign), “Thinking is a luxury we can no longer afford.”

We believed (and still believe) that thinking is not a luxury for software developers, so we went underground. After much hard work, Marilyn pinpointed the fault and we corrected it just before the first launch. We may have saved an astronaut’s life, but we’ll never get any credit for it.

Moral: We may think that hardware and software errors are challenging, but nothing matches the difficulty of confronting human errors—especially when those humans are managers willing to hide errors in order to make schedules.

https://leanpub.com/errors

Must There Always Be Inferior Code?

Some people claim that when you learn high software standards you will never again develop in inferior ways. Is this true?

I think you can arrive at a meaningful answer by using an analogy:

Some people claim that when you learn high medical standards, a doctor or nurse will never again treat a patient in inferior ways. Is that true?

Seen in this light, the answer is obvious. Most doctors and nurses will not treat patients in inferior ways—unless it's an emergency, like an explosion or a fire in which many people need saving in a hurry. If that happens, the doctor or nurse will return to those patients when the emergency has calmed down. Same in software.

But there do exist a few medical professionals who don’t live up to such high standards. They are, after all, human beings. Yet in spite of their inferior practices, some of their patients do get better. Why? Because humans have built-in healing mechanisms—but software does not.

Software with sick code doesn’t heal itself. Those programmers who develop in inferior ways will eventually produce troublesome code. But the key word is "eventually."

The inferior programmer may not be around any longer when the code's trouble makes itself known, so some inferior programmers can get away with hacky ways for an entire career.

It’s a good manager's job to recognize these inferior programmers and replace them and their code before the true costs of their inferior work become evident.

Some managers overuse the tactic of forcing programmers to code in a hurry, as if there's always an emergency. Just as in medicine, emergency treatment of code tends to produce inferior results. Managers who care only about the short-term will not do anything about their inferior programmers, but they, too, may move out before the consequences of their inferior management become apparent.

That’s why inferior programming practices persist. And, as long as programmers and managers are human, inferior practices will always persist. But they don't have to persist in your world. It's up to you. \

Code in haste, debug forever.

Basic skills of a good programmer?

Many outstanding programmers were asked, "What are the basic skills required to be a good programmer?" Lots of good and useful answers were given to this question, such as, test before coding, use a particular tool, or use Agile methods.

For me, though, with more than 60 years of programming experience, the one thing that made me a better programmer was my ability and willingness to examine myself critically and do something about my shortcomings. And, after 60 years, I'm still doing that. You could say it's incremental development applied to myself.

I also examine my strengths (long-comings?) because I know that my greatest strengths can quickly become my greatest weaknesses.

For instance, one of my great strengths as a programmer was speed. If something had to be done quickly, I was the guy to do it. But the weakness in my speed was my tendency to omit the last few hours of testing that would make the project rock solid. I had to learn the importance of taking the time to do a precision job.

Many programmers do examine themselves critically, but then they work to improve their greatest strengths, to the exclusion of their weaknesses. That practice takes them a certain distance, but the nature of computers is to limit your ability, by highlighting your greatest weaknesses. 

A computer is like a mirror of your mind that brightly reflects all your poorest thinking. To become a better programmer, you have to look in that mirror with clear eyes and see what it's telling you about yourself.

Armed with that information about yourself, you can then select the most useful external things to work on. Those things will be different for you than for anyone else, because your shortcomings and strengths will be unique to you, so advice from others will often miss the mark.

Good programmers make good use of their best tools, and you are your best tool, so sharpen yourself.

See, for example, 

Becoming a Technical Leader 

The Psychology of Computer Programming

Survey: The Worst Error Message

I participated in a recent survey. The question was:

What's the Worse Error Message You've Ever Seen?

This was my contribution, which received thousands of up-votes:

A graduate student brought me an error message that said:

THIS IS AN IMPOSSIBLE ERROR. THIS ERROR CANNOT OCCUR.

IF THIS ERROR OCCURS, SEE YOUR IBM REPRESENTATIVE.

I told the student I wasn’t the right person to see, but he should see his IBM representative.

He said, “But I’m my IBM representative.”

—————————-

I'd like to collect some of these lousy error messages for my book, ERRORS.

If you've got a good (i.e. bad) one, please share it with us in a comment to this post.

Thanks for contributing.