Because of their psychological distance from the problems, external consultants are often able to see information that escapes the eyes and ears of their clients. Dani (Jerry's wife) tells this story about one of her consulting triumphs in the dog world:
======================================================
A woman with a Sheltie puppy was at her wits' end because "he always poops on the living room rug." She loved the little guy, so before giving up and taking him to the Humane Society, she came to Dani for a consultation. Dani listened to the woman describe the problem, then asked, "Are there any other behavior problems you've noticed?"
The woman thought about that for a while, then said, "Well, yes, there is one. He has this annoying habit of scratching on the front door and whining."
======================================================
It's easy to laugh at this woman's inability to see the connection between the two "problems," but that sort of blindness is typical of people who are too close, too emotionally involved, with a situation. Learning to recognize such free information is one of the secrets of successful testing. Using it, you can learn quickly about the quality of an organization's products or the quality of their information obtained from machine testing.
Here are some "Sheltie stories" from our consulting practices. Test yourself by seeing what information you can derive from them about the quality of a product or the quality of the information that's been obtained about the product through testing:
======================================================
1. Jerry is asked to help an organization assess its development processes, including testing. Jerry asks the test manager, "Do you use specs to test against?"
Test manager replies, "Yes, we certainly do."
Jerry: "May I see them?"
Test manager: "Sure, but we don’t know where they are."
======================================================
2. Jerry is called in to help a product development organization's testing group. He learns that they are testing a product that has about 40,000 lines of code.
"The problem," says the test manager, "is with our bug database."
"What's wrong with it," Jerry asks. "Is it buggy?"
"No, it's very reliable, but once it holds more than about 14,000 bug reports, its performance starts to degrade very rapidly. We need you to show us how to improve the performance so we can handle more bug reports."
======================================================
3. Bunny is asked to help improve the testing of a large product that has 22 components. The client identifies "the worst three components" by the high number of bugs found per line of code. Bunny asks which are the best components and is given the very low bugs per line of code figures for each.
She then examines the configuration management logs and discovers that for each of these three "best" components, more than 70 per cent of their code has not yet successfully been checked in for a build.
======================================================
4. Trudy is invited to help a development manager evaluate his testing department's work. She starts by looking at the reports he receives on the number and severity of bugs found each week. The reports are signed by the test manager, but Trudy notices that the severity counts have been whited out and written over.
"Those are corrections by the product development manager," the development manager explains. "Sometimes the testers don't assign the proper severity, so the development manager corrects them."
Using her fingernail, Trudy scratches off the whiteout. Under each highest severity count is a higher printed number. Under each lowest severity count is a lower printed number.
======================================================
5. Jerry is watching a tester running tests on one component of a software product. As the tester is navigating to the target component, Jerry notices an error in one of the menus. The tester curses and navigates around the error by using a direct branch.
Jerry compliments the tester on his ingenuity and resourcefulness, then asks how the error will be documented. "Oh, I don't have to document it," the tester says. "It's not in my component."
======================================================
6. Fanny watched one tester spend the better part of several hours testing the scroll bars on a web-based enterprise system. The scroll bars were, of course, part of web browser, not the system being tested.
======================================================
7. Jerry asked a development manager if the developers on her project unit tested their code. "Absolutely," she said. "We unit test almost all the code."
"Almost?" Jerry asked. "Which code don't you unit test?"
"Oh, some of the code is late, so of course we don't have time to unit test that or we'd have to delay the start of systems test."
======================================================
8. One of Christine's clients conducted an all-day Saturday BugFest, where developers were rewarded with cash for finding bugs in their latest release candidate. They found 282 bugs, which convinced them they were “close to shipping.”
They were so happy with the result that they did it again. This time, they found 343 new bugs - which convinced them they were “on the verge” of shipping.
======================================================
9. A general manager was on the carpet because a recently shipped product was proving terribly buggy in the hands of customers. Jerry asked him why he allowed the product to ship, and he said, "Because our tests proved that it was correct."
======================================================
10. Another manager claimed to Noreen that he knew that their product was ready to ship because "we've run 600,000 test cases and nothing crashed the system."
======================================================
11. When Jerry asked about performance testing, one of his clients said, "We've already done that."
"Really?" said Jerry. "What exactly have you done?"
"We ran the system with one user, and the response time was about ten milliseconds. Then we ran it with two users and the response time was twenty milliseconds. With three users, it was thirty milliseconds."
"Interesting, Jerry responded. "But the system is supposed to support at least a hundred simultaneous users. So what response time did you get when it was fully loaded?"
"Oh, that test would have been too hard to set up, and anyway, it's not necessary. Obviously, the response time would be one second - ten milliseconds per user times one-hundred users."
======================================================
12. Jerry's client calls an emergency meeting to find out "why testing is holding up our product shipment." In the meeting, the testers present 15 failed tests that show that the product did not even build correctly, so it couldn't be tested. They discuss each of the 15 problems with the developers, after which the development lead writes and email summary of the meeting reporting that there are only two "significant" problems.
The email goes to the development manager, the marketing manager, and all the developers who attended the meeting. None of the testers present are included in the cc-list, so none of them even know that the email was sent.
======================================================
13. Johnson watches a tester uncover five bugs in a component, but instead of logging the bugs in the bug database, the tester goes directly to the developer of the component and reports them orally. Johnson asks why the tester didn't record the bugs, and he replies, "If I do that, she (the developer) screams at me because it makes her look bad."
======================================================
14. Tim reviews a client's test plan and notices that there is no plan to test one of the components. When he asks the test manager (who is new to the company) why it's missing, he's told, "We don't need to worry about that. The development manager assures me that this developer is very careful and conscientious."
======================================================
So, were you able to extract meta-information from these ten examples? What did each of them tell you (or hint to you) about the quality of the information from testing - the accuracy, relevance, clarity, and comprehensiveness? Remember, though, that these are merely hints. Perhaps the Sheltie pooped on the rug because he had some medical problem that would need a veterinarian's help.
Perhaps there's a non-obvious explanation behind each of these Sheltie Situations, too, so you always have to follow-up on the hints they provide to validate your intuition or find another interpretation. And, even if your intuition is right on target, you probably won't have an easy time convincing others that you're correct, so you will have to gather other evidence, or other allies, to influence the people who can influence the situation.
When Dani asked the Sheltie's owner why she thought the pup was whining at the front door, the woman said, "I think I've spoiled him. He just wants to go out and play all the time, but I have too much housework to do - especially since I spend so much time cleaning up the mess on the rug."
When a problem persists in spite of how obvious the solution is to you, you aren't going to be able to convince others to solve the problem until you find out how they are rationalizing away the evidence that's so apparent to you. So we need to know how people immunize themselves against information, and what we can do about it.
(Adapted from Perfect Software: and Other Illusions about Testing)http://www.smashwords.com/books/view/25400
No comments:
Post a Comment