Facebook is in trouble with its advertisers.

According to a recent Wall Street Journal article, Facebook’s calculation of “Average Duration of Video Viewed” misled advertisers into thinking that their videos were being watched more than they were.  The problem had to do with the way that Facebook calculated the metric. They didn’t include videos that were viewed for less than three seconds.

Perhaps I’m an optimist but I don’t believe that Facebook was intentionally trying to distort the number or trick their advertisers.  This wasn’t an issue of manipulation.  It was miscommunication and misunderstanding.

Metrics are made by people.  Therefore, people make decisions about how a metric will be calculated. These decisions include what data will be represented by that metric. Often there are good reasons for those decisions.  I’m sure that Facebook’s decision to exclude those videos was logical.

The problem is that the person designing the metric and the person using the metric are usually not the same.  Unfortunately, information about the metric isn’t always passed along. When it is provided, the recipient often doesn’t take the time to understand it.

As a result, the consumer of the metric often makes assumptions about how that metric might have been calculated and what data may or may not be included.  I once had a client who couldn’t understand why their training expenses weren’t decreasing despite making a major shift to virtual training (significantly reducing their training-related travel expenses).  It turned out that the company didn’t account for expenses for training travel as part of training expenses.  Rather, they rolled it into a generic travel expense account.

As a consumer of data, it is your responsibility to ensure that you fully understand the assumptions, constraints, and context surrounding the information that you are using to make a decision.

Answering the following five questions will help to ensure that you have a working understanding of your metrics.

In plain language, what ‘yes’ or ‘no’ business question does this metric answer?

If you can’t explain your metrics in plain, simple language, you might not really understand them. Plain, simple language means that you can explain them without just rehashing their formula. You should be able to tell someone what the metric means to you and your business.

What’s the formula?

Can you write out and explain the formula for your metrics? Do you understand each component? Do you know if the numbers used in the calculation (or the output) are actual values, projected values, planned values, or annualized values? That makes a big difference as well.

Are there related data that are not included in calculation?

Don’t assume that the name of the metric captures all data associated with that name. Does “sales” mean all sales?  Does it include sales before returns or sales net returns? Facebook’s average duration of video viewed didn’t include all videos viewed.  My client’s training expense calculation didn’t include travel expenses.

What’s the possible range of values for this metric?

What’s the best and worst case scenario?  I once worked for a company that indexed a quality metric. Their metric was performing at around 3%. I asked why nobody was alarmed by that result.  They said, “It’s not great, but as long as it doesn’t go negative we are ok.” I replied, “That metric can’t go negative, its lowest possible value is 0”.  They freaked out!  They thought their metric had a range of -100% to +100%.

What are the implications of being above target on this metric? What are the implications of being below target?

This is another case where sometimes just looking at the name could be misleading.  Common sense would tell you that it would be better to come in higher on a quality target.  However, if your company defines quality as “errors per 1000 transactions” then you’d actually want to come in below.

Of course, you never want to be too far off of your targets in either direction.  There is such a thing as “too much of a good thing”.  Coming in too far under your expense targets could mean that you are cutting corners on quality or service.

However, in general you know that for expenses over is worse than under but with sales over is better than under. You’ll have a hard time reaching a decision if you want to increase a value while someone else wants to decrease it.

Every day, leaders use metrics to make decisions and drive actions. However, they often “fly blind” when it comes to actually understanding the metric. If you can’t answer all of these questions for the metrics you use, there is a good chance that you aren’t making the best decisions.

Download our free Metrics Literacy Worksheet to ensure that you and your team fully understand your metrics.

——————————-
Brad Kolar is an executive consultant, speaker, and author for Avail Advisors.  Avail’s Rethinking Data workshop will help you and your leaders become better at making data-driven decisions.  Check it out at www.availadvisors.com/rethinkingdata.  You can reach Brad at brad.kolar@availadvisors.com.