*This article was updated 1/4/2023 to reflect the most up-to-date and accurate data.
At The Strawhecker Group, we live and breathe analytics. On the quest to make data comparable and accessible for our clients the goal and the challenge remain the same – what is the “correct” metric? Questioning data points to determine “what is meaningful” is necessary in order to improve and ultimately provide the best comparable.
In the age of online media, adding statistics or metrics can bolster any message, and this holds true for the payments industry. The concept of providing supporting metrics adds relevance to what would otherwise just be an opinion.
The intent of this article is to highlight a few questions for those that rely on metrics and data points to consider. While not all-inclusive, these questions are important for payments leaders to consider.
What is the calculation and time series used?
Take the two statements below:
The first example surely jumps off the page when compared to the latter (just a meager 5%). The truth is that neither are incorrect, but without context, which provides more confidence? That depends.
You may find that these questions are too in the weeds, but when presenting information to a colleague, executive management, or within the public domain, the goal is to be confident in the statements made. After all, if a business made a strategic decision off of a short-term 59% growth indicator without considering the long-term 5% change, they may be disappointed in the end.
Was there a normalization method?
There are a number of considerations and techniques available to normalize for known differences. The examples listed below are not an all-inclusive list of normalizations.
Time series-dependent metrics can be influenced by seasonality. For example, completing a 6-month look back of Golf Course foot traffic in June to January will likely show an overly inflated value that, out of context, could be misleading.
- In this example, a Year-over-Year calculation may be more appropriate.
Calendar Days – Monthly & Weekly
Especially for industries, that have higher daily spend on the weekend than weekdays, short-term metrics should consider the days of the week.
The same thought process applies to differences in the number of days within a month.
Let’s continue with the Public Golf Courses example, considering the uptick in golf course foot traffic on the weekends as compared to weekdays, non-normalized Month-over-Month comparisons could be misleading.
- Comparing June 2022, which has eight weekend days (Saturday + Sunday) to July 2022, which has ten weekend days.
- If spending was equal each Saturday and Sunday throughout June and July 2022, July would have 25.0% more weekend days (and associated weekend spending) simply due to the calendar.
- Additionally, the month of July has 31 calendar days compared to 30 in June.
Controlling for Skewed Populations
There is always a possibility of an over-represented (or under-represented) sample.
- When measuring total U.S. spending, and considering the significance and commoditized nature of gasoline, higher gas prices will result to higher U.S. spending levels.
- The question remains, what are you comparing against? Total U.S. spending? Consumer non-commodity spending?
- Depending on your target comparison, normalizing for skewed populations should be considered.
What is the sample and level of tolerance?
Let’s set an objective to derive a benchmark for merchant cost of acceptance using a random sample of 1,000 card-accepting merchants. This may not seem like a strong enough sample to make a data-driven decision. Without context, this is just a number.
- For example, if we are looking for a representative sample in Branson, MO that has 20,000 business establishments, just a 600 merchant sample provides a 95% Confidence interval with a 4% margin of error
- If we are looking for a representative sample in Los Angeles, CA that has +1M business establishments, just a 600 merchant sample provides a 95% Confidence interval with a 5% margin of error
Now the tolerance level needs to be established. I am reminded of the quote commonly attributed to Voltaire, “Perfect is the enemy of good.”
- In our Los Angeles, CA example (where we had a 600 merchant sample out of +1M business establishments), of course one would prefer to have a 100,000 merchant sample to get to a 99% CI, <1% margin of error. But if you are limited to a 600 merchant sample, is that good enough?
How does it compare?
The prior three questions provide context as to what a metric represents. Understanding and clarifying the previous questions gives an analyst a starting point.
Ultimately, analysts are beholden to primary, secondary, or survey data results.
Creating harmonized views gets us one step closer to standardized comparability. At TSG, within the Acquiring Industry Metrics platform, an extensive data harmonization exercise is utilized to compare across disparate data sources. By mapping data elements into “like-to-like” harmonized fields, comparable metrics are reliable and standardized.
Achieving the “correct” metric comparison is the goal and challenge across all industries. Scrutinizing internal or published comparables should be standard practice. I hope you consider the questions posed within this article next time you see a metric or indicator shared internally or within the public domain. Confidence is founded on understanding what metrics represent.
How confident are you? Contact us to discuss today!