While once reserved for scientists and researchers, the term “statistical significance” has become part of the business lexicon. Unfortunately, among casual users of statistics, its meaning seems to be getting confused. Statistical significance doesn’t tell you if a change matters or not. It only tells you if the change is real.
I recently reviewed customer satisfaction data with a manager. His satisfaction scores rose 8% from the prior period. The manager was pleased. I commented that the change wasn’t really that big and probably didn’t represent a meaningful change in customer experience. His response was, “But it’s statistically significant.”
Part of the problem is with the word “significant”. The common meaning of significant is important or full of impact. A significant business deal probably has a high dollar value associated with it. A significant business setback is quite serious.
In statistical terms, significance also means important or noteworthy but for a different reason. It’s not a measure of impact. It’s a measure of certainty. It tells you the probability of whether what you are seeing occurred from random chance or whether it is indicative of a true change.
In the case of the satisfaction increase what it is saying is that there is a lot of certainly that your satisfaction increased by a very small amount.
Knowing that the customer satisfaction increase was significant can be an indication that your interventions are working. (Although to be absolutely sure you’d really need to measure those customers who experienced the intervention and those who didn’t…otherwise the increase could just be because of some other factor such as changes in pricing or competitor behavior.)
As a leader the next step is to take a step back and determine whether that significant change matters. Having customer satisfaction change from 60% being satisfied to 65%* being satisfied, even if significant still isn’t very good. One out of every three customers is still not happy. If you were still in school, you’d still be getting a D. You can have a statistically significant finding for a .0001% change or a 100% change. The significance doesn’t reflect the size or impact.
Statistically significant is not the same as important. Importance is a call that you as a leader must make. Don’t get fooled by statistically significant findings that tell you with high certainty that not much has changed.
————————-
* The metric was also an issue. 60% to 65% is an 8% increase. It is misleading because it doesn’t mean that 8% of your customer base is happier. The use of relative metrics is another practice that can misrepresent the actual situation.
Great post, Chief. We all need to learn more about the language and meaning of numbers. One thing I’d add is that statistical knowledge is relational and constructed as well, which means that those of us on the receiving end need to be vigilant about learning how its terms are defined and operationalized, and how data are collected. Because our society values quantification and prediction, statistical knowledge is considered sacrosanct. Nonsense, I say! It can be a tool for obscuring truth rather than revealing it. For example, you may have heard that anorexia nervosa has the highest death rate of any mental illness. What does that mean? How many people actually die of “mental illness” in the first place? How is this information defined? collected? Do coroners, for instance, include “anorexia nervosa” or “borderline personality disorder” as causes of death? No! Ask a few questions and the emperor is doing a pole dance in a thong. In other words, when data are collected in problematic ways, how can the knowledge generated be reliable? or meaningful? Personally, I prefer a naked emperor 🙂
Great comment – Your point is very well taken. Especially in business it seems that people have learned to distrust narrative or anecdotal evidence (which is a good thing). But, that same level of critical thinking isn’t applied to numbers, especially when those numbers support your claim.
Thanks for the post and ask the emperor to put on a robe.
These are great comments and lend a valuable perspective for me. However I will add a quotation that I think has been attributed to Albert Einstein:
“Not everything that is valuable can be measured and not everything that is measured is valuable.” I’m adding this in reference to my belief that narrative and anecdotal evidence does have a place somewhere here in the picture…