Estimating Projects: A History of Failure

When I lived in Crested Butte, Colorado, I sometimes hiked across the mountains to Aspen. There were five hiking trails that I knew of, all of them former toll roads from the old gold-mining days when the only way to reach Aspen was through Crested Butte. As I struggled to breathe going up these 13,000+ foot passes, I often wondered why there were five toll roads. But the time I'd hiked all five, I knew the answer. There were five toll roads because none of them were very good.

Consultants are frequently given the task of estimating, or helping to estimate, projects. To start with, they're asked to estimate how long it's going to take to solve their client's problem—which is usually undefined. Although there have been billions of words written on dozens of methods of estimating projects, most of them are, in my opinion, useless or nonsensical. Or both. Like toll roads to Aspen, there are so many because none of them are very good.

Why do I say they're not very good? Well, why were the toll roads to Apsen not very good? Mostly, I think, because if you're walking from Crested Butte to Aspen, you have to cross an extremely high barrier. We might want to be able to hike over a 13,000 foot pass without breathing hard, but it's impossible (at least for most of us).

And that's the primary reason that all these estimating methods are not very good is exactly the same: they are trying to do something that's impossible—predict the future. Sure, we'd all like to do this, but we can't.

People have been trying to predict the future as long as people have been around, so I thought it would be useful to read about methods from the past, so perhaps you could relate them to some of the failures of today. The following post is from dgc, a friend who has worked for a long time in the software industry, mostly as a contractor and consultant—long enough not to take himself too seriously, a good lesson for all of us consultants.

Some Estimating Methods

The fabulous Wizard of Oz
Retired from his racket because,
What with up-to-date science,
To most of his clients
He wasn’t the Wizard he was.
- Anonymous

Since estimation seems to have much in common with the mantic arts (i.e., the arts of divination), I turn to the writings of an expert in that field: Dreamas Gentilharte Cheynelokk, a nearly thousand-year-old wizard who writes about the mysteries of CMMI (Comprehensive Magistrate of Magical Integrity) certification.

For those unfamiliar with this form of CMMI, Dreamas briefly describes its five levels of magical competence as follows (the translation of this text is by Dr. Devin G. Kettenschloss in "Paduan Vellums, Volume VI, Item lvii- A Translation" following the seminal work of the late Dr. Blanche V. Foote-Falles):

"(1) Erotic [1]: animalistic, sensual, like kindling consumed in a flame, uncontrollable, never twice the same, exciting

(2) Telestic [2]: ritualistic, repeated, like an unbending oak planted in the earth, formal, delimited and limited capacities, routine

(3) Mantic [3]: prophetic, governable, like a ship sailing home on uncharted seas, divinatory, observed and proven by trial, directed

(4) Poetic: rhapsodic, meted, like a falcon riding steady the rolling air [4], quantifiable, sound and proportioned in structure, balanced

(5) Ecstatic [5]: euphoric, evolving, like the shining cosmic quintessence [6], heavenly transcendence married to earthly immanence, sustained."

Dreamas goes on to describe hundreds of mantic techniques useful across all aspects of software development. But for the topic of estimation, the following description of the art of "Alectryomancy" seems relevant when speaking about making time/schedule estimates for a project, especially since the technique used differs based on the CMMI level of the organization (again quoting from the previously cited translation).

"Alectryomancy [7]: Divination by the Actions of Poultry Pecking at Grain

So much software development goes awry for lack of sufficient numbers of dedicated top-breed poultry!

For those functioning at the CMMI Level 1 (Erotic), any hen or rooster of any kind will do. Simply put out a handful of grain and let the chicken loose for one minute. At the end of one minute, guess how many grains the chicken ate and multiply by 3. This calculation will yield the needed number of hours, days, weeks, months or years it will take to complete the project.

Those groups at CMMI Level 2 (Telestic) laugh at that naiveté. The wizards at this level realize it is not that easy. First, more chickens are needed, at least three but never more than nine. Second, before being let loose on the grain the chickens properly motivated with loud chanting and the waving of gleaming, sharpened axes. Third, the grain for each bird is placed in a line running east to west and the first grain eaten by each bird is the estimate. That is, if the 4th grain from the east is the first eaten by a hen, then the estimate from that hen is 4. Finally, after all chickens are done, the lowest estimate is picked with any multiplicative factor determined by the wizards using Capnomancy (divination by smoke) or Catoptromancy (divination by mirrors placed under water).

But groups at CMMI Level 3 (Mantic) realize estimation is a daring, daunting and dangerous undertaking. So in their practice, they only use roosters. Each rooster is allowed to make its estimate by pecking a grain in its own sealed off arena. A variety of techniques for deriving the numerical estimate can be employed (counting grains, counting volume, which grain is first picked, which is last picked, etc.) with skilled wizards selecting in advance the appropriate technique by means of Gyromancy (by whirling until dizzy). After each rooster has produced his estimate, all the roosters are placed in one arena for a cockfight {SIDE NOTE FROM 'dgc': I realize the cruelty of this technique may offend some people's sensibilities, but that is what the original text says} with the winning cock's estimate being adopted [8] and everyone being held to that estimate under threat of the hatchet.

Groups at CMMI Level 4 (Poetic) with more refined sensibilities bemoan the waste of possibly good estimators by the Mantic level wizards as well as the blatant discrimination against hens. Therefore, they eschew the wastefulness of cockfights and use both hens and roosters; but they also carefully record and plot age, breed, and accuracy of auguries over time. In addition, no matter what anyone says about chickens and lips, they firmly believe in the importance of using Labiomancy (divination by lip reading) as part of their process.

Finally, wizards at CMMI Level 5 (Ecstatic) realize that the work of estimation is never done. So they use the greatest number of poultry and have them estimating and reestimating often using dozens of techniques and inventing new ones whenever needed. At this level, the chicken coop and yard is a constant bustle of activity yielding untold numbers of eggs. After all, in the end, it is all about the real egg production."


Footnotes (From the translation)

[1] From the Greek meaning "sexual love" as associated with Eros, who is not the cute cupid of modern times but is rather the passionate young god, son of Venus, as best described in Apuleius' story of Psyche and Eros in "The Golden Ass" (Chapters 7-9).

[2] Very rare in modern English. Derives from the Greek "telestes," meaning a "hierophant or expounder of sacred mysteries."

[3] The Romans, especially Cicero in his treatise "De Divinatione," clearly preferred the word "divination," which derives from the Latin "divinus" and indicates a deific origin, to Plato's "mantic," which is equivalent to the Latin word "furor" meaning "madness, lunacy, or raving."

[4] Echoing Gerard Manly Hopkins' "The Windhover" (http://www.bartleby.com/122/12.html).

[5] There is a delicious irony in the CMMI master's choice of the word "ecstatic" for their highest and ideal level. From the writings of mystics throughout the ages, ecstasy is often associated with displays of frenzy and agitation bearing strong sexual overtones. See, for example, the writings of the Spanish mystics St. Teresa of Avila ("Interior Castles") or St. John of the Cross ("Dark Night of the Soul"). Given these strong sexual associations, it is possible it might be somewhat difficult to see how the Ecstatic level differs from the lowest level, Erotic.

[6] Here, Dreamas means the ancient and medieval concept of the perfect material of which the stars were made: an exact blend of all other types of matter (fire, earth, water, and air).

[7] An alternate spelling is "Alectromancy."

[8] Note how in many ways this is similar to the Delphi technique of estimation some speak of in software engineering.