Monday, April 17, 2006

Estimating Projects: A History of Failure

When I lived in Crested Butte, Colorado, I sometimes hiked across the mountains to Aspen. There were five hiking trails that I knew of, all of them former toll roads from the old gold-mining days when the only way to reach Aspen was through Crested Butte. As I struggled to breathe going up these 13,000+ foot passes, I often wondered why there were five toll roads. But the time I'd hiked all five, I knew the answer. There were five toll roads because none of them were very good.

Consultants are frequently given the task of estimating, or helping to estimate, projects. To start with, they're asked to estimate how long it's going to take to solve their client's problem—which is usually undefined. Although there have been billions of words written on dozens of methods of estimating projects, most of them are, in my opinion, useless or nonsensical. Or both. Like toll roads to Aspen, there are so many because none of them are very good.

Why do I say they're not very good? Well, why were the toll roads to Apsen not very good? Mostly, I think, because if you're walking from Crested Butte to Aspen, you have to cross an extremely high barrier. We might want to be able to hike over a 13,000 foot pass without breathing hard, but it's impossible (at least for most of us).

And that's the primary reason that all these estimating methods are not very good is exactly the same: they are trying to do something that's impossible—predict the future. Sure, we'd all like to do this, but we can't.

People have been trying to predict the future as long as people have been around, so I thought it would be useful to read about methods from the past, so perhaps you could relate them to some of the failures of today. The following post is from dgc, a friend who has worked for a long time in the software industry, mostly as a contractor and consultant—long enough not to take himself too seriously, a good lesson for all of us consultants.

Some Estimating Methods

The fabulous Wizard of Oz
Retired from his racket because,
What with up-to-date science,
To most of his clients
He wasn’t the Wizard he was.
- Anonymous

Since estimation seems to have much in common with the mantic arts (i.e., the arts of divination), I turn to the writings of an expert in that field: Dreamas Gentilharte Cheynelokk, a nearly thousand-year-old wizard who writes about the mysteries of CMMI (Comprehensive Magistrate of Magical Integrity) certification.

For those unfamiliar with this form of CMMI, Dreamas briefly describes its five levels of magical competence as follows (the translation of this text is by Dr. Devin G. Kettenschloss in "Paduan Vellums, Volume VI, Item lvii- A Translation" following the seminal work of the late Dr. Blanche V. Foote-Falles):

"(1) Erotic [1]: animalistic, sensual, like kindling consumed in a flame, uncontrollable, never twice the same, exciting

(2) Telestic [2]: ritualistic, repeated, like an unbending oak planted in the earth, formal, delimited and limited capacities, routine

(3) Mantic [3]: prophetic, governable, like a ship sailing home on uncharted seas, divinatory, observed and proven by trial, directed

(4) Poetic: rhapsodic, meted, like a falcon riding steady the rolling air [4], quantifiable, sound and proportioned in structure, balanced

(5) Ecstatic [5]: euphoric, evolving, like the shining cosmic quintessence [6], heavenly transcendence married to earthly immanence, sustained."

Dreamas goes on to describe hundreds of mantic techniques useful across all aspects of software development. But for the topic of estimation, the following description of the art of "Alectryomancy" seems relevant when speaking about making time/schedule estimates for a project, especially since the technique used differs based on the CMMI level of the organization (again quoting from the previously cited translation).

"Alectryomancy [7]: Divination by the Actions of Poultry Pecking at Grain

So much software development goes awry for lack of sufficient numbers of dedicated top-breed poultry!

For those functioning at the CMMI Level 1 (Erotic), any hen or rooster of any kind will do. Simply put out a handful of grain and let the chicken loose for one minute. At the end of one minute, guess how many grains the chicken ate and multiply by 3. This calculation will yield the needed number of hours, days, weeks, months or years it will take to complete the project.

Those groups at CMMI Level 2 (Telestic) laugh at that naiveté. The wizards at this level realize it is not that easy. First, more chickens are needed, at least three but never more than nine. Second, before being let loose on the grain the chickens properly motivated with loud chanting and the waving of gleaming, sharpened axes. Third, the grain for each bird is placed in a line running east to west and the first grain eaten by each bird is the estimate. That is, if the 4th grain from the east is the first eaten by a hen, then the estimate from that hen is 4. Finally, after all chickens are done, the lowest estimate is picked with any multiplicative factor determined by the wizards using Capnomancy (divination by smoke) or Catoptromancy (divination by mirrors placed under water).

But groups at CMMI Level 3 (Mantic) realize estimation is a daring, daunting and dangerous undertaking. So in their practice, they only use roosters. Each rooster is allowed to make its estimate by pecking a grain in its own sealed off arena. A variety of techniques for deriving the numerical estimate can be employed (counting grains, counting volume, which grain is first picked, which is last picked, etc.) with skilled wizards selecting in advance the appropriate technique by means of Gyromancy (by whirling until dizzy). After each rooster has produced his estimate, all the roosters are placed in one arena for a cockfight {SIDE NOTE FROM 'dgc': I realize the cruelty of this technique may offend some people's sensibilities, but that is what the original text says} with the winning cock's estimate being adopted [8] and everyone being held to that estimate under threat of the hatchet.

Groups at CMMI Level 4 (Poetic) with more refined sensibilities bemoan the waste of possibly good estimators by the Mantic level wizards as well as the blatant discrimination against hens. Therefore, they eschew the wastefulness of cockfights and use both hens and roosters; but they also carefully record and plot age, breed, and accuracy of auguries over time. In addition, no matter what anyone says about chickens and lips, they firmly believe in the importance of using Labiomancy (divination by lip reading) as part of their process.

Finally, wizards at CMMI Level 5 (Ecstatic) realize that the work of estimation is never done. So they use the greatest number of poultry and have them estimating and reestimating often using dozens of techniques and inventing new ones whenever needed. At this level, the chicken coop and yard is a constant bustle of activity yielding untold numbers of eggs. After all, in the end, it is all about the real egg production."


Footnotes (From the translation)

[1] From the Greek meaning "sexual love" as associated with Eros, who is not the cute cupid of modern times but is rather the passionate young god, son of Venus, as best described in Apuleius' story of Psyche and Eros in "The Golden Ass" (Chapters 7-9).

[2] Very rare in modern English. Derives from the Greek "telestes," meaning a "hierophant or expounder of sacred mysteries."

[3] The Romans, especially Cicero in his treatise "De Divinatione," clearly preferred the word "divination," which derives from the Latin "divinus" and indicates a deific origin, to Plato's "mantic," which is equivalent to the Latin word "furor" meaning "madness, lunacy, or raving."

[4] Echoing Gerard Manly Hopkins' "The Windhover" (http://www.bartleby.com/122/12.html).

[5] There is a delicious irony in the CMMI master's choice of the word "ecstatic" for their highest and ideal level. From the writings of mystics throughout the ages, ecstasy is often associated with displays of frenzy and agitation bearing strong sexual overtones. See, for example, the writings of the Spanish mystics St. Teresa of Avila ("Interior Castles") or St. John of the Cross ("Dark Night of the Soul"). Given these strong sexual associations, it is possible it might be somewhat difficult to see how the Ecstatic level differs from the lowest level, Erotic.

[6] Here, Dreamas means the ancient and medieval concept of the perfect material of which the stars were made: an exact blend of all other types of matter (fire, earth, water, and air).

[7] An alternate spelling is "Alectromancy."

[8] Note how in many ways this is similar to the Delphi technique of estimation some speak of in software engineering.

7 comments:

paul said...

Wow! Thanks to dgc I now know that my estimation method is Erotic! That sounds pretty good until you realize that most of the time I'm a chicken pecking at grain and occaisionally the person who gets to say "and multiply that by 3". :-)

I have used Mantic techniques for estimating, architecture, and design. Eventually winning the cockfights felt as bad as loosing and I've been looking for better ways ever since.

paul said...
This comment has been removed by a blog administrator.
fish said...

Visualization is a tool that has been used for thousands of years by initiates of all the metaphysical schools. Today, it is incorporated into top athlete's daily routines and is used in business affairs frequently. It's use is wide-spread among highly successful people, either consciously or unconsciously, aware of its create power. So if it has stood the test of time and is still being used by high achievers we must come to the conclusion that it works! But has it ever worked for you?

If you answered 'yes' to the above question then you know how powerful this technique can be. If, on the other hand, you gave the more likely answer 'no' then take heart for I am about to reveal to you a sure fire way of reaching your objectives through this mostly misunderstood art.

The trouble with visualization is simple - its in its name!

When studying and contemplating the art of visualization most people have the impression that they must create visual images and make them real or life-like. Many people, in fact the majority, find this almost impossible to do. Even if they can formulate a solid picture of their objective they find it extremely difficult to sustain the image for any length of time. Either the image fades, changes or other intruding thoughts intervene.

This type of visualization is almost impossible to sustain and luckily it is not at all necessary. Why? Because it is in the subconscious mind that your visualization needs to be placed and there is good news. The subconscious mind does not know the difference between an imaginary event and a real one. Your visual image only needs to be a strong visually as any other imagined event. However, that is only half the story.

If all you had to do was just imagine stuff and your world automatically changed to reflect your imaginings this world would be full of chaos (not to mention all those creepy crawly bug-eyed monsters!). Therefore, there are a few more steps to complete before the visualization is passed to the subconscious for manifestation.

Let's try a little experiment. Remember a scene from your past that has a lot of good feelings around it. Any good memory will do, like the first time you heard the words "I love you" from your partner, an amazingly spectacular sunset, a great holiday event or your last birthday. Pick one and remember it. How clear is the image? Can you remember any sounds? What way did you feel? Is there any sense of touch, taste or smell? Identify how your memory works. Is it mostly visual, auditory, kinaesthetic or of a feeling nature?

Now we are going to create an imagined event in our lives that has the same strength and potency as that image. So relax and let's go.

Imagine something that you do everyday, something that you did yesterday, today and will do tomorrow. Let us take the example of waking up tomorrow morning. Don't try to add or take anything away, just think about it and analyse the scene. Is it dark or light? Are you lying next to someone in bed? Do you still feel tired? Has the alarm clock sounded? Are you irritable that you have to get up or full of joy at the dawn of a new day?

You will find that the imagined event is very similar to the memory with probably one key difference - your point of perspective. Is the memory behind you and the future event in front of you? Is one to the left and one to the right? Maybe they are both in front of you or the future seems to move in a clockwise direction. Whatever the perspective the thing to notice is that they are very similar in appearance.

Now imagine doing your future event a week from now, then a month from now, then six months from now. Where are those images placed? Are they moving further away, going clockwise, from left to right? This is your time-line and using it is important in visualization as you will see later.

Ok, let's imagine something that is very unlikely to happen and see where it differs from the last image.

Imagine you are sitting somewhere familiar which is extremely comfortable and relaxing to you. Now imagine that a person you know well comes up to where you are and says "hello". Imagine them telling you that they want to show you a new trick. All of a sudden they have three juggling balls. They throw them in the air and begin to juggle with ease. Then they begin to whistle one of your favourite tunes. You suddenly realize that there is a strong smell of flowers in the room and notice a vase of them just behind the juggler. Imagine laughing loudly at the scene and feeling joyful at the experience. Then the person juggling leans forward stands on leg and puts the other leg outstretched behind them. All the while still juggling and whistling. Then they begin to hop on their leg as a small bird flies over to perch on their head. Once you have the imagined event and stayed with it a few moments just let it fade.

Ok open your eyes. What was the difference between the two images? Can you spot any? Did you use more, less or roughly the same senses in your fantasy event as you did in the future one? Did you see them from different angles? Was the picture bigger in one than the other? Was the sound clearer, the feelings more acute or the smell stronger? Take some time and go back to each scene in your mind. How does the future event differ from the fantasy one? Are you looking at both from a different vantage point? Do you see yourself in the image of one but not the other? Analyse the scenes and see where they differ.

Have you identified how the future event differs from the fantasy one? If you have then its time to make visualization work for you! Take a goal that you have been working on or would like to achieve. Nothing too far-fetched at this point please! Pick something that is possible but at the moment seems a little impractical. Once you have it form a mental image of what it would be like to have, be or do that thing or be in that experience. Remember to form it the same way you do a memory. Give it the same strength visually, in sound, feeling, taste and touch - use your mind in its natural state. All you have to do is imagine the scene.

Ok how does it differ from the scene of waking in the morning? Can you identify the differences in perspective, sound, taste, touch, feelings and what you hear?

Now there will be one other key thing that differs in the images- it is very simple but often overlooked. You know that the future event is going to happen! This is reflected in the way we experience the image. So what we are going to do is fool your subconscious mind into thinking your goal is definitely going to happen by manipulating your goal image!

Once you know what the differences are in each image begin to change the goal image so that it is seen the same way as the future event in your imagination. Place the visualized scene in exactly the same position with the same perspective as your future event.

Place it in the correct position on your time-line. You may already begin to feel that the goal is more possible. Visualise in this way everyday and you will condition your subconscious mind to manifest the experiences necessary to make your goal attainment certain.

One more thing to remember: During the day think about your goal often. This reinforces the visualization and will begin to dispel doubt from your mind. personal development

Kamal said...

I have my say here: Software Project Estimation Techniques For Different CMMI Levels

Oscuridad said...

ok, so I'm at CMMI 1 level (Erotic estimation method). But, since my boss normally says "Oh, no! That's unacceptable. Divide it by 20. It doesn't matter that we don't have 20 developers!", I believe that the rooster fight method is more appropriate... even if my rooster has its legs tied and its eyes blinded!

willjud2009 said...

Yeah, Hi Gerry - its nice to hear from you. When I first started consulting, I read your book, 'The Secrets of Consulting'. That was, what - almost 20 years ago. I especially liked your rasberry jam theory, 'The wider you spread it, the thinner it gets!

I'll take your dialogue a little farther...the truth of the matter is, for the most part, deadlines and schedules are determined without project estimates, and the IT team is expected to meet ‘Time to Market’ dates, or when management thinks a project should be delivered. I’m actually in the midst of writing a book myself – keep your eyes open for ‘The Problems with IT - and Recommended Solutions’
Gerry, you’re a nice guy….but I’m going to be a little more blunt – hey kids – there is no Santa Claus. Project Estimation is more of an art than a science. Furthermore, Project Estimation is but one of the branches in your Project Management suite. What I discuss in my book is not just that project estimates are poor, but why they are often poor. For example, if your team is comprised of a Business Requirements team, a Data Team (DAs and/or DBAs), do you have a ‘symbiotic trio’ or a ‘dysfunctional family’?
But let’s talk about project estimation. The biggest problem with an estimate is threefold:

a) The team is requested to provide an estimate for something that they've never done before

b) The practice of re-estimating (once the Business Analysis and initial Design is done) is never practiced

c) There's a translation process that has to done to numbers provided by the

Allow me to discuss these issues separately—

a) When working on a project that involves an area of the business that the team is not intimately familiar, the team has to be honest with management. Let them know that the best you can do is provide a ‘framework timeframe’ for the first release of the project. Let them know that what exactly will be in a specific version of release of the project will be determined well in advance of the deadline. Just don’t be overly optimistic and promise things that you’re not sure you can deliver – not in the initial estimation phase. If your feet are put to the fire, talk to the Developers and figure out the things that are easiest to do (and make sure that they’ve at least done something similar) within the context of the target deliverables…only promise the items that are ‘slam-dunks’

b) Make sure your management is aware that the Business Area Analysis and initial System Design is intended to provide clarity pertaining to what is needed and how to go about meeting your requirements. If you perform your Business Area Analysis properly, you will have a good Data Architect and a good Business Analyst (or people that can do both). As the Business Area Analysis documentation (sometimes called Business Requirements Document) is being completed, the DA should be refining his Data Model to meet the requirements and should have a good handle on the data. Moreover, it is PARAMOUNT that the BA and the DA used the same terms, they should collectively be forging a Business Lexicon that is common to all. Once the model, BA doc, and initial draft of the System specs is published, provide management with a ‘re-estimation’ of the project. If your estimate has now changed, let your management know this ASAP. If your management balks, then let them know that you can still serve the meal, but it might not be ‘soup-to-nuts’…

c) Now I’m going to let all of you in on a secret….Over the last 20 years I’ve discovered something – call it ‘Will Estimating Corollary”. Developers speak in ‘dog-years’. If a Developer says it will take an hour to develop something, that doesn’t equate to it takes an hour to get it done. Basically, there are two estimate notions, the short task estimate (1-3 hours) , and the long task estimate (1 or more units of 8 hrs each….e.g., ‘5 days’). The Translation that you can probably feel safe to use as a guide are as follows-

Developer Provided Estimates and what you should put on your Project Plan:
1-3 Hours – anytime a developer gives you an estimate for something that they think will take more than an hour (an up to 3 hrs), you should put an estimate of 1 day for that task
4-8 Hour estimate  you might as well consider these as a ‘Day Estimate’
‘Day Estimates’ – I take each task that the Developer says will take 8 hrs and multiply is by 2.

Once I complete my plan, I provide the Developers with a the anticipated dates. I don’t let them know which tasks have been stretched from a time perspective.

Keep in mind that you should always be updating your plan. In addition to the uncertainties that I’ve mentioned above, there will be unplanned meetings, issues that will need to be resolved, and other stop gaps. If you encounter unanticipated external constraints, always make sure that you update your project plan. Remember, there should be no such thing as ‘make the time lost up by….’ – lost time is lost

Udi said...

I employ the "2 day rule" with the teams I consult.

If the estimate for any task is greater than 2 days, the task has to be broken down to subtasks. If that requires some architecture/design/business analysis work - so be it.

Quite frankly, any time a developer estimates something at 5 days or a week, they don't have a clue how to even begin let alone when it's going to end. The thing is nobody ever told them that "I don't know" is a valid answer when someone asks you for an estimate. They see everyone else in the room providing numeric estimates so they think they have to as well.

The "2 day rule" works to undo the damage of those misunderstandings.

You may also like this post on my blog:

Estimate invidually - fail globally