Metrics: How to Improve Key Business Results Part 25

Standards and benchmarks, in the realm of metrics, are strongly interconnected. Standards, from the Industrial Age through today, are invaluable for providing a means for interoperability. Standards in the industrial world allow you to use a light bulb that you bought at Walmart in a lamp that you bought at a high-end designer furniture store. Standards allow you to get gas for your car from any station in the United States, without worrying if the gas pump nozzle will fit into your gas tank. From the ingredients label on a can of soup to the technology that allows you to tune your radio, standards give consistency and interoperability for manufacturers, distributors, builders, and customers alike.

Unlike the manufacturing industry, performance measures are more akin to an art than a science. The use of standards for how we measure things for improvement is arguable. What need is there? If our questions are unique, and thus our answers are unique, why do we need standards for our measures?

Since I advocate creating measures to answer your specific questions, I have trouble taking up the other side of the debate. Why indeed?

I would love to have standards for how you develop metrics; as in the use of expectations over targets, for example. Or for the definition of the data owner. But, standards for performance measures as a whole? Why?

Before I answer this question, let's look at benchmarks and why I think the two are interrelated.

Benchmarks: Best Used to Provide Meaningful Comparisons

Benchmarks are best used to provide meaningful comparisons for your metrics. Outside of defining expectations, you usually want to know how well you perform against your peers. If you're ambitious, you'll want to know how well you perform compared to the best-the best in your industry, the best in your country and, perhaps if you're really ambitious, the best in the world.

Benchmarks are a blessing and a curse.

Benchmarks are also useful for drawing a "line in the sand." You can establish a baseline from your own measures so that you can compare your present performance to your past performance. This is critical when your goal is to improve.

Establish Baselines.

One of my joys when working with clients on metrics is helping them establish a baseline; mostly because it forces them to put the metrics upfront in their improvement-process thinking. I almost always run into goals to improve effectiveness, improve efficiency, improve productivity, or improve customer satisfaction.

"Improve" is a lousy verb to use in a goal statement. You have to qualify it with more information-as in "how much" of an improvement? By a certain percentage? By a certain number?

My favorite recollection of the poor use of an "improve" goal was in my parish council. I was hoping to bring organizational development expertise to the council. I was teamed with a retired police officer, a successful business owner, a nurse, our priest, and a retired grandmother. The goal? Improve membership in the church. I wasn't perturbed because I had seen this type of goal (increase, decrease, etc.) many times before.

I said, "Improve membership-by how much?"

"What do you mean?" asked the ex-police officer.

"I mean, if it's our goal to improve membership, how will we know that we achieved it?"

The business owner said, "Oh, you're trying to get us to set a goal."

I countered, "I thought that was the intention-to come up with goals for the year?"

"Yes," said the business owner, "but you're trying to set us up for failure. We'll set a number and if we don't reach it, we will have failed."

Now I sat in stunned silence. I may have actually opened and closed my mouth once or twice. "Uh. Well. Would you be happy with just one more family joining?"

"Sure," said the nurse.

"Anything more than that is gravy!" said the ex-police officer.

I turned to the priest, still in shock. "You'll feel we've achieved this goal if we add just one family?"

He shook his head no. As the leader of the team, and our parish, his input carried the equivalent weight of a CEO.

"How many families join each year now?" I asked "Two or three a year," the priest answered.

"So if we do nothing, we'd achieve this goal?"

He nodded in the affirmative.

We eventually worked out a reasonable and measurable increase over the expected growth without making any changes. The purpose of the goal was to build up the parish. The purpose of the metric was to see if our efforts were successful. We had plans, ideas, and activities scheduled for the purpose of bringing in new members and bringing back parishioners who had fallen away. We needed to (1) set a goal to focus our efforts and ideas; and (2) set measures to tell us what worked and what didn't.

We also needed a benchmark. We could not determine if any of our efforts were producing the desired result if we didn't know the norm. Consider the benchmark in this case a "control group" or value. You have to know what you get if you do nothing different. Then, when you do new things in new ways, you can at least a.s.sume that any changes that you made caused the change to the outcome. Even if you implement so many changes that you can't determine what exactly worked or what exactly didn't work, you at least know whether the overall effort(s) worked.

So, benchmarks basically allow you to know where you are and, therefore, where you end up.

A benchmark is the starting line.

Even when a benchmark is used to compare you against your peers, it is essentially a starting line-a baseline to measure your progress against. The purest form of the benchmark is when you set it as an internal baseline (vs. an external comparison). This allows you to measure progress.

Set Benchmarks Responsibly.

Benchmarks falter from time to time when leaders want to use the comparison benchmark as the baseline. Most times, it starts with something like, "Can you get our compet.i.tors' average availability, response time, or customer satisfaction ratings so that we can compare ourselves to it?"

This requires that the performance of your compet.i.tors is a good starting point.

My simple and first argument against chasing this data is: "What if you are already better than your compet.i.tors? Does that mean you're doing well enough?"

And usually the leader that sent me after the data, who fully believes that the organization is woefully lagging behind compet.i.tors, is not ready for this question. I usually have to ask it twice.

"No, we still need to improve..."

So, while gathering information on another organization's performance can be enlightening, if your goal is to improve, it is not overly useful. If your goal is to be better than your peers, then, of course, this benchmark is essential. Even if your goal is to be better than your compet.i.tors, you'll need to know (1) whether your efforts are helping you improve; and (2) how far you are from the performance of your peers (if you're better than your peers, are you done?).

So, if you choose to look only internally at your performance, standards are not necessary. But, if (and when-because eventually you'll want to see how you compare to others) you decide to compare your performance to your peers or compet.i.tors, standards will be critical. You can't compare yourself to others when the methods of measurement are different.

Let's say you define the availability of the network as the amount of time without an outage divided by the total amount of time in a given period.

Availability = 1,440 minutes (number of minutes in one day (or 24 hours)) 20 minutes (of outage) divided by 1,440 minutes Availability = (1,440 20) 1,440 = 98.6% So far, so good. But, let's say your closest compet.i.tor (or peer) has a 100 percent availability rate for the same period. Are you going to step up your game a bit? Are you going to work harder? Is your compet.i.tor doing better than you?

Well, without standards, you can't tell if your compet.i.tor is doing a better job than you. What if you define an outage as any time span that your customers cannot use the network, but your compet.i.tors consider an outage as only those times when the network is unavailable due to unscheduled or unplanned downtime? In other words, let's say that during the same 24-hour period, your compet.i.tor had scheduled maintenance for four hours. If you define an outage as any time the service is unavailable (which is likely the way customers will interpret it), then the compet.i.tor's availability should be reported as follows: Compet.i.tor Availability: (1,440 240) 1,440 = 83.3% If you had all of the raw data for your compet.i.tor's reports, you could use your personal "standard" and determine how well you perform against your compet.i.tor. But this is highly unlikely to happen. What you will get, if you are extremely lucky, is the "score"-and even that can be difficult to get from your peers and compet.i.tors.

So, how you define an outage compared to the way your compet.i.tor defines an outage is critical to your use of their measures as a benchmark.

Standards Allow Comparison to Others.

Standards in performance measurement come down to providing the ability to compare measures between different organizations. Just as manufacturing standards allow you to use your products seamlessly with another organization's products, standards in performance measurement allow you to "use your measures" seamlessly with another organization's measures. If there were standardization of performance measures, you could "borrow" another unit's measures for your own purposes. If you had the same questions, you could trust that that metrics used by Company A could be used to measure the same things in your organization.

Without standards, it is hard to imagine how you could use the metrics of a different organization-even if you had the same exact root questions. And using measures produced by another organization as a benchmark is even more improbable.

Getting Good Data.

This problem with benchmarks and standards for performance measurement also creates problems for well-meaning organizations that seek to provide data warehouses of information. This information invariably is intended for your comparison. To make the data warehouse effective, it has to have a clear set of standards for the information provided by the different organizations.

HDI, a third-party survey company, offers benchmarking on customer satisfaction data by controlling data definitions. HDI administers the survey to your customers, collects the responses, and provides you with reports, a.n.a.lysis, and raw data. Since HDI standardizes its questions (the same questions are asked along with the same set of possible replies on a 5-point Likert scale), it is able to offer you comparisons against other organizations, including the following: Comparison of your scores (average, percentage satisfied, or other) against all other organizations who have used the HDI service Comparison of your scores with others in your industry (self-selected from a list) Comparison of your scores to the top nth percentile of others' customer base This provides you with a higher confidence in the comparability of the information. Of course, there are some drawbacks. Only organizations that use HDI's service are included in the comparison, and your main compet.i.tors or peers may not be among these organizations. Even if you compare your results against the entire customer base, this still may not reflect the pool you want to compare to. Another minor drawback is that you are forced to use one set of questions. No deviations. If you want to use the 10-point scale suggested by Reichheld in The Ultimate Question, you could not use HDI's service. And even if you could use the 10-point scale, you could not compare the data to those who used a 5-point scale.

The Goal: Reliable Industry Standards.

Industry standards for performance measures would make it possible to truly benchmark, rank, and compare peer organizations. It should be feasible to convince an industry (like higher education IT) to standardize performance measures before the chance of adopting universal standards. The tighter you can make the pool for standardization, the easier it should be to come to agreement. Higher education information technology is a pretty specific pool. If you started with information technology performance measures-your pool for coming to consensus on the standards is too large. If you narrow it to education information technology, you're doing better. Higher education tightens it a little more.

Consensus is required for success. Publishing a standard does not make it effective. You must have the majority of organizations (in your industry) using the standard to make it useful. Since you need high partic.i.p.ation in the use of the standards, it logically follows that you should involve as many of the target audience in the creation of the standard as possible I offer that the consortium structure is the best bet for creating standards for performance measures. The consortium creates, evaluates, reviews, and manages standards for an industry. The problem may be that the "industry" in this case is hard to define. Of course, if you do as I suggested and find a tighter definition of the target audience, you can make it happen. But, looking at performance measurement as an industry is obviously too large. So, as a performance measurement expert, you'll have to define your "industry" to build a consortium. If you have some standards for performance measurement to reference, you are ahead of the game.

Recap.

Standards are tools that allow for interoperability. In the case of performance measures, standards allow for comparison between organizations.

Benchmarks are either the starting line (baseline) for your improvement efforts or a goal for you to achieve. As a baseline, it helps you determine how far you have to progress to achieve your goals, how well you're getting there, and how far you have come. As a goal, it represents how good you want to become-"as good as Company A" or "better than the average."

To have real external benchmarks, you must have standards that are in agreement among the organizations you choose to compare to.

If you can find or develop standards for your performance measures, and your peers agree to them, you can compare measures.

START YOUR OWN CONSORTIUM.

Depending on your industry, there may be little to no standards for performance metrics. If you lack standards, you will also lack the ability to benchmark against your compet.i.tors or peers. Creating a consortium for developing, publishing, and using standards for your industry is a feasible answer to this need. I created the Consortium for the Establishment of Information Technology Performance Standards (CEITPS) in 2009 to address the lack of standards for performance measures in Higher Education IT. This effort has been more difficult than I antic.i.p.ated.

The population I have been working with-metrics a.n.a.lysts in peer inst.i.tutions-all agree that the ability to compare performance metrics is critical to organizational leaders' acceptance of metrics. They also readily agree that this is nearly impossible without standards for the measures that make up the metrics.

I'm not sure if the difficulty in getting anyone to draft standards is a result of a low priority for measures comparison or a fear of the future environment if standards are created and adopted. In some situations, standards can be "enforced"-especially if there are governing bodies with power to do so. If you are like most us, you'll have to build consensus, communicate the standards, market them, and then hope for adoption. There is a real reason to fear your own success in this endeavor. If you succeed and have standards adopted in your industry, you will then be asked to follow through with allowing comparison of your organization's performance with your peers. No excuses, no subjective "feelings" of how good you really are. You'll have real values to compare against. While most metric a.n.a.lysts won't "fear" this situation, many organizational leaders may.

I've had so little success in getting peer a.s.sistance in the development of standards (although the CEITPS has a healthy membership) that I have turned to alternative methods for drafting standards. At the next national conference, where most of these representatives will be present, I plan to lead a set of group sessions to create drafts for key measures. I will start with Effectiveness measures as defined in the Answer Key. My hope is that the consortium will be able to draft a full set of measures for our first standard: service availability based on outages.

Conclusion.

Standards and benchmarks are inextricably joined at the hip. Benchmarks are meaningless if there are no standards to ensure that definitions and measurements are consistent across organizations. This is particularly an issue with external benchmarks, where you are attempting to rank yourself against your peers and compet.i.tors.

Standards are unnecessary if you are only using internal benchmarks.

Benchmarks are better used for demonstrating progress against a baseline than they are for comparisons against outside organizations.

When are standards and benchmarks required?

When you are comparing your performance against compet.i.tors or peers. External benchmarks are required to determine the average performance of your comparison group the range of performance of your comparison group the top performers in your comparison group Standards are required to ensure you are comparing apples to apples with brand, type, color, and size. In other words, standards are required to ensure that you are comparing specifics.

When you are determining a realistic goal, tracking progress to a goal, or deciding if you have achieved a goal. Internal benchmarks are required to determine a norm to help set realistic goals for improvement a baseline or starting point to track progress from your current state a means for determining the achievement of a goal (when that goal was defined as achieving a specific level of improvement) Benchmarks are also useful in determining if your efforts in organizational development or process improvement are having a positive effect on your environment, services, or products. If you antic.i.p.ate your efforts (especially changes to processes) are going to have an effect on performance, you'll want baselines for any possible areas concerned. If you can also get historical data, it will help to determine if changes are due to your efforts or if they are normal fluctuations.

Respecting the Power of Metrics.