Project Metrics: The Double-Edged Sword

Metrics can swing both ways

Metrics can slice both ways!

We all know that metrics are critical to running a business. We need those leading and lagging indicators of performance, so we can change course where needed and keep doing what works.

Metrics, however, are a slippery beast. In addition to measuring the results of actions, they also motivate action. And this is where things can go either fabulously right, or terribly wrong. What’s worse is that we may not realize what outcome we are promoting until it is too late.

Here’s an example from early in my career. This experience seared onto my brain the awesome power of metrics to influence behavior. And with that power comes great responsibility to ensure that the outcome we motivate is the one that we want.

I was working at a large financial services firm as the relationship manager to the outsourced IT vendor. In this arrangement, the vendor would prepare financial estimates for each system change request from us, the customer. These financial estimates were then used to secure a budget to complete the work.

We had a huge problem. The financial estimates varied wildly from the actual project costs, with swings of plus-or-minus 75%. This caused heartburn for the business owners, because they couldn’t predict whether their funding requests would be wildly overstated or understated. I could understand the problem of being over budget, but I didn’t get why they cared so much if they came in under budget. That’s saving money right? WRONG! It was then I learned that coming in significantly UNDER budget is just as bad, if not worse, than coming in OVER budget—because you have tied up company dollars that could have been allocated elsewhere.

My division’s Vice President had a solution—or rather, a demand. We would impose a new metric immediately that required the vendor to deliver each project at no more than plus-or-minus 10% of the original estimate. Failure to meet this metric would result in a financial penalty to the vendor.

Within 3 months, we had a perfectly green scorecard. It was amazing! All we’d needed all along to keep that vendor sharp was a metric! Eureka!

Or not? Even though I was still early in my career, I smelled something funny. I wanted to understand this power of the metric, and how it had changed everything about the vendor’s performance. Were they just now being more careful? Did the old “What gets measured gets done” adage apply? I had to know.

I kept poking around about this measurement. Finally, I got the answer I sought from one of my vendor counterparts. His terse revelation went something like this.

“Well, we’ve always gotten really vague and crappy requirements from the business side, and we still do. We always tried to do our best on the estimates, given limited information and broad assumptions. Now that we have to meet this metric, here’s what we do. We pad all the estimates to make sure we won’t be over budget. And then if it turns out that we’re running more than 10% UNDER budget, we just pad more hours onto the project until we’re within the 10% tolerance.”

Ouch! We had gotten what we’d asked for. All projects were delivered within a 10% budget tolerance. Turns out that all projects had also become more expensive. Oops. We weren’t measuring that…

This lesson has stuck with me throughout my career, and I call on it whenever creating performance measures. It’s critical to analyze what behaviors will be driven by the measure, and how they may actually be counterproductive. It’s still true that “What gets measured, gets done.” The trick is to make sure that what gets done is what we really WANT to get done.

Advertisements

4 responses to “Project Metrics: The Double-Edged Sword

  1. Great insights Pam. You are absolutely right.

    Now, how do you propose we make sure what gets done is what we really want to get done? Or should that be another post entirely?

    In my field, the “monitoring & control” project phase is really underrated, especially for small to medium sized projects.

    We’ve seen through the years that managers that really pay attention to M&C are the ones that succeed. In fact, better M&C is the reason why our company exists.

    Thanks for the post!

    Cesar

    • Hi Cesar,

      Thank you for the comment. I completely agree that M&C is a critical part of the project phase. For me, though, it all starts with having a clear “Line of Sight” from the projects mission/value statement, to the outcomes, and then the activities that will drive those outcomes. If that can’t be mapped cleanly, then there’s a fair chance that the end result is going to be sub-par or off-base.

      I’m interested in your thoughts.

      Regards,
      Pam

  2. Hi Pam,

    Yes! Keeping your eyes on the prize, knowing where you’re coming from is the recipe for success, all the while communicating effectively with stakeholders.

    Thanks for the post/comment!

    Cesar

  3. Ha! Saw that one coming. I’m surprised your company was so reluctant to have projects come in under budget. Sounds like your steering committee wasn’t working particularly well.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s