Tuesday, November 22, 2011

Who is Right, and What is to Be Done About It?

A management consultant, whose client was an international manufacturer, was asked to evaluate an inventory management procedure that the client had used with stunning success in their French operations. As part of his study, he wanted to compare the performance of the French procedure with procedures used in other locations, using historical data from several countries. A programmer with a strong management science background was given the job of programming the simulation of the French procedure.

When the consultant received the results he could not reconcile them with the figures supplied by the French company. After extensive checking he initiated a series of long telephone calls to France, suggesting that perhaps the procedure had not actually performed as well as they had claimed. The French management took offense at the implication of incompetence.

The French manager complained to the manager who had hired the consultant. Tempers mounted and international relations were strained to the breaking point.

By sheer chance, someone examined the programmer's simulation program and noticed that one term was missing and a second term was negative rather than positive. These findings led to a full technical review of the formula as translated. The review showed that the programmer's formula did not match the formula supplied by the French.

The consultant, much relieved, took the program back to the programmer and showed him the error. "That's not an error," the programmer protested. "Actually, the formula was in error, so I corrected it. The formula I programmed is correct, whereas the original formula was simply wrong."

That's the end of Part 1.

Note to Readers

Now, for you readers, the question is this:

"If you were the consultant, how would you handle this situation going forward?"

If I receive a few comments, I'll publish the rest of the story—what actually happened.

And please note: I don't accept anonymous comments. They're automatically rejected. By all means, use a pseudonym, but don't waste your effort trying to post anonymous comments.

17 comments:

ringloss said...

I would talk to the programmer to understand what error they saw in the original formula. If we could determine that it was a problem with the programmer's understanding of the original formula, I would have the simulation run with the original formula. If we could determine that, to the best of our knowledge, there is no way the original formula could work, I would have the simulation run with the original formula.

So, I guess I would run the simulation with the original formula in any case, unless it was cost prohibitive to do so. Then I would compare the results to actual French results. Hopefully that would tell us whether the programmer's formula or the original French formula was in error.

Tobbe Ryber said...

Before doing anything else I would carefully consider my role in this. Not knowing all the details I still think that it was I as the management consultant that made a serious mistake which had some really bad consequences. I did what I considered careful checking but missed out on the, at least in hindsight, obvious part that the calculations may be wrong. Can the lost credibility be repaired or not. Will the parties trust me in the future or will they just ignore whatever I say? I may come to the conclusion that it is better to end it all right now.
However if I find future work may be fruitful then I would start by listing possible ways to repair the damage done before discussing any details of measurement. I would start by excusing myself for the misunderstandings and that we restart by looking into if it is possible to measure this kind of thing at all. There are, as always, at least three possibilities for the measurement:
1. It is possible to measure with this type of formula but the French got it wrong
2. It is possible to measure with this type of formula but the scientific programmer got it wrong
3. It is possible to measure with this type of formula but they both got it wrong. There is a better answer that YOU may be able to help them find and agree upon
4. The programmer may have a hidden agenda
5. They French may be well aware that they are wrong but need to keep their job and bonus
6. Nobody has a clue to what they are doing but thought it looked good to have some “facts on the table”
7. It is not possible to measure effectiveness with this type formula but with another type and that conclusion can be accepted by the parties
8. It is not possible to measure this type of work in this kind of detail, we need a higher level to get anything valuable out of the measures
9. It may be totally ridiculous to try to measure anything at all in this situation because there are so many factors that it is virtually impossible to create any kind of formula that takes all factors into account and can be understood by any of the humans involved in the work
Well this reasoning could go on for a while and it would probably be wise to start by asking open-ended questions until we know what the problem really is. I would take the list above and try to find out where we really are. What can we agree on as of right now, what is the purpose of the calculations we are doing? Should we be doing something else instead?

SueP said...

Ah come on.. You HAVE to tell us 'the rest of the story!'

Please? :-)

Personally, I guess my reaction would depend on who was actually correct about the formula.. Was the consultant deluded? Or were the French results not actually what they were claiming?

SueP said...

2nd comment -

Hmmm. I find it odd that the programmer did not alert the consultant when s/he found the discrepancy between the formula as given and "as correct.".

Seems like something "the boss" should know about...

Joe said...

If I were the consultant, I'd first attempt to confirm the programmer's claim that the French formula was incorrect.

If that claim is true and the formula is indeed correct, then I would inform my client of this bug and let them decide what they want to do about correcting it.

Then, I'd talk with the client and ask if they wanted to proceed with the comparison of the French procedure or not. Perhaps if the French procedure needs correction they want to stop, perhaps not.

Assuming they want to continue with the comparison, I would have the programmer change the formula to match the original French formula, and finish the comparison.

Jaap B said...

I definitely want to know what happened, so that's enough reason to comment in itself :)

First of all, the programmer should have told the consultant about the modifications he made to the formula.

But at this point I believe the consultant first of all has to find out why the programmer believes the formula is wrong, and what that error means in the real world.
Then he should go back to his original assignment, and evaluate the actual procedures used in several countries, correcting for faulty data where necessary. Naturally he uses data, standards and methods which are in the best interest of his client, and not necessarily those provided by the locations themselves.
He should report those results back to his client, and if his results don't match those of the national management he should not hide this but explain why he thinks his evaluation is more accurate.

Jaap B said...

I definitely want to know what happened, so that's enough reason to comment in itself :)

First of all, the programmer should have told the consultant about the modifications he made to the formula.

But at this point I believe the consultant first of all has to find out why the programmer believes the formula is wrong, and what that error means in the real world.
Then he should go back to his original assignment, and evaluate the actual procedures used in several countries, correcting for faulty data where necessary. Naturally he uses data, standards and methods which are in the best interest of his client, and not necessarily those provided by the locations themselves.
He should report those results back to his client, and if his results don't match those of the national management he should not hide this but explain why he thinks his evaluation is more accurate.

Neon Tapir said...

I think the management consultant should facilitate a dialog between the programmer and the entity that owns the formula (the French manager?)

The programmer should be given a voice in the process, so she can respectfully explain why she thought the formula was wrong. Blind adherence to poor requirements leads to poor software. It's her duty to educate the customer about the ramifications of the requirements.

Ultimately, the customer should have the final say because they are paying the bill. If they want the formula as written, the programmer should implement the "wrong" formula.

The management consultant's job here is to "raise the flag" and to make sure the issue gets addressed.

Mark Crowther said...

Hiya Jerry,

This reminds me of my days in manufacturing quality assurance!

First thing would have been to draw the teams together the moment some differences were observed. Try to understand the thinking of both the French team and the programmer and learn why they have a different view of the correct formula. Also, clarify what and how other teams are working.

Something that stands out, is the consultant just looking at data that’s been provided and modelling based solely on that data? It feels like they need to go to the manufacturing shop floor and walk/work the process. Sometimes the story that comes back from the shop floor is one version, the scrap bins or actual processes being used another story!

It sounds like our consultant is ping-ponging between accepting one view and then another, upsetting the parties involved and not having a deep understanding before expressing views via long and strained phone calls. There’s no need, I’d try and go for a bit of group-enlightenment before stating one team/person is right or wrong.

Mark.

windchill said...

Run the simulation twice, once with the French formula, once with the consultants formula.

Find out what the terms in the formula mean. How does this difference influence the results of the simulation.

Let's say the French formula was incorrect, and their results aren't as good as they thought. Deliver the bad news to your employer.

Gerald M. Weinberg said...

Wow, great comments.

I need to say, first of all, that I automatically don't accept comments from anonymous contributors. I should have said that on the original post, so I'm going back to change that before I address any of the other issues.

Jason Little said...

There are a few approaches I can think of taking. If I had to pick one, I'd arrange a meeting with the programmer, myself and the local stakeholders to discuss the differences in the formula.

The goal would be to validate which formula should be used for the calculation. The sensitivity of the situation is irrelevant to me. From what you describe, I've been brought in for a specific purpose and I will do what I need to do in order to provide objective (as objective as possible) data for my client to decide how to move forward.

Gerald M. Weinberg said...

Little by little, I'll release other parts to this story, in response to various of these terrific comments.

Perhaps the first thing to note about this situation is that the idea of "correct" and "incorrect" need not apply in this situation, which confused the participants (as well as some of those contributing comments).

That may be my fault in the way I wrote up the situation, but it's there, if you read the case carefully. This is the formula the French use (but as SueP and others suggest, we need to know if they actually use it. If they don't, then it's an entirely different story).

It's a formula for managing inventory, and there are many ways to do this. One way might be better than another, but it's hard to tell without a long history, since one might be better in the short run, but not in the long run.

So, as a consultant, I would work to eliminate the argument about right-vs-wrong—an argument that often traps programmers, who tend to believe in the "one right way."

Okay, so that's my first addition. Does it change your analysis in any way?

Markus Gärtner said...

In that situation I would have arranged an exchange between the two positions either face to face or via teleconferencing. I would have facilitated the discussion between the two, to get the differences between their two opinions, and help the two find a suitable formula - which could turn out to be a combination of the two, a complete new formula which both hadn't thought about, or either of the two formulas.

I believe that using examples to explore the requirements could have helped in that discussion. So I would get the two to discuss their differences using examples for the needed functionality.

Stephen Smoogen said...

I would try to figure out why the programmer thought something was incorrect. If it was a math issue then I would work with the French participants to see if the items match up. If it is a methodology issue then I would check to see if those assumptions are valid in this way. If it turns into an item where a methodology is legally valid in France/EU but not the US then I would point this out as problem with confirming/using it with my customer.

Gerald M. Weinberg said...

Several contributors suggested getting the parties together, something I would likely have suggested, too. However, the programmer refused to speak with the French, saying there was no point, as they were clearly wrong and he was right.

How does that affect your approach?

Tobbe Ryber said...

I try to put myself in the situation of the French. If I am in charge of an operation that has proved to be successful - and then suddenly some academic - living in the country where they renamed French fries to Freedom fries (?)- writes a formula - which is an attempt to model reality. This model, created by an ignorant person that cannot even speak French - I mean even the cildren are able to do that in France - suddenly "proves" that the success I have is NOT a success but a failure. How could this person using a formula that used in a simulation possibly know MORE about what is happening where I spend seven days a week to make this a success. Do you really think I care about a stupid formula? No, let´s start discussing success. If we cannot agree that what is happening right now is succesful or not how can we then expect to agree on anything. It is like I tell you - we are really succesful right now. And you answer we'll see, Let me create a model - and we know that all models are wrong but some are useful - and this model will tell me if your statement is correct or not!

So in short. Try to find common ground. We agree that we do not agree on the forula used. What elese can we agree on?