Quick math test. What is the percentage increase between “Strongly Agree” and “Agree”? How much more is “Very often” compared with “Neutral”? Is “Always” five times better than “Never”?
You can’t answer, can you? You can’t do math on labels. They aren’t quantities. We all know that. Yet, at one time or another, most of us probably have. That’s because sometimes labels look like numbers and that creates confusion.
We often use numbers as short-hand for labels. This is very common on surveys. One equals “Strongly disagree”, two equals “Disagree” and so on. When it comes time to summarize the data, we forget that those “numbers” aren’t really numbers and start running calculations on them. The answers look real and sound real but they are not real.
The problem is that the distance between points on a non-numeric continuum is not equal. The difference between agreeing and strongly agreeing with something can be much greater than the difference between being neutral and agreeing. The difference between never and rarely is not the same as the difference between rarely and sometimes. Yet, if each of those sets of responses were just one “number” apart, they’d be treated equally in your calculations.
Anyone who works with employee or customer satisfaction knows that there is almost always a diminishing return. At some point, increasing your score by even one tenth becomes quite difficult. Those diminishing returns also don’t get reflected in your calculations.
Performing standard calculations on numeric-labels won’t completely mislead you. If more people choose “five” in one instance compared to another and you take an average, the statistics will reflect that increase.
However, you won’t be getting a clear picture. The change from 1 to 2 is 100%. However, a person who has switched from being completely dissatisfied to just dissatisfied isn’t really 100% more satisfied. Depending on where you start, moving 20% of your people from a state of being neutral (3) to being satisfied (4) could result in a change to the average satisfaction score of less than 5%. That’s pretty misleading (although that has as much to do with the limitations of using averages as it does with doing calculations on numeric labels).
The picture is further distorted because in numeric calculations higher numbers carry more weight. Therefore, as you move up a “numeric” scale, each incremental change counts for less since the denominator is increasing. Going from 1 to 2 is a 100% change but going from 4 to 5 is only a 25% change. While that makes perfect sense mathematically, it doesn’t make sense when you are dealing with categories. Moving from four to five is actually much harder than moving from one to two but statistically you’d get less “credit”.
The biggest problem in running calculations on numeric labels is that it creates a false illusion of precision and understanding. The number 4.24 is very precise. The difference between 4.24 and 4.63 millimeters could be the difference between life and death in a surgical procedure. However, it’s much less clear what 4.24 units of agreement look like as compared to 4.63 units of agreement. Agreement is a concept not a unit of measure.
This illusion builds upon itself as more calculations are done. Averages get compared and differences are reported to the hundredths place. That causes us to believe that we are privy to the most subtle changes that exist in our organizations. What does a decrease of .17 units of agreement really mean? It means that there is a little less agreement. But “a little less” is not very precise. It’s definitely not as precise as .17 implies.
Finally, the illusion is cemented in our minds when the calculations are subject to tests of statistical significance. The magic “statistically significant” asterisk that is attached to a number creates a false perception of scientific validity and rigor in the analysis. The significance calculations themselves are rigorous. It’s the subject matter upon which they are based that is not. You can get useful information from numeric labels but that information is much rougher and more general than your statistics will lead you to believe.
Numeric labels are not numbers. They should not be treated like numbers. Instead, work with them the same way that you would with any set of categorical data. Use frequencies and distributions (that’s actually not a bad idea to do anyway, even with real numeric data). That will give you a much more accurate understanding of what is happening with your data.
You can’t perform math on a label. Don’t get confused by labels masquerading as numbers. If you do, you might miss an important part of the picture.
