Benchmarking is Lazy
High School Math and University Intro Classes Failed You
"“The heart of the problem is that simple averages, and the familiar standard deviation, are almost always meaningless and misleading when applied to complex systems.”" — Benoit Mandelbrot
“For every complex problem, there is a solution that is simple, neat, and wrong.” — H. L. Mencken
Children during their early school years learn about the measures of central tendency (the Mean, Median & Mode), as a way to understand data sets and draw conclusions. They are taught that most distributions are Gaussian (normal/bell curve), meaning more values converge towards the center (mean) and fewer observations happen the further you move to either extreme.
This leads to the inference, that something further from the middle, is better or worse. If you want to be good/great, just aim to be as far to the right of the distribution as possible. This heuristic powers most decision making in business, politics, academia and our personal lives.
What if I told you, this conventional thinking is (usually) overly simplistic, if not flawed and leading you astray? This article will explain why it’s time to move on from making decisions based on the mean/median, why benchmarking if often a lazy exercise that doesn’t help and what to do instead.
If this week’s article does not interest you, please check out some other recent ones:
The Most Obvious Secret to Success
The Latte Legislator (short story)
Robbing Banks is a Rich Man's Sport
Measures of Central Tendency aren’t that helpful (Intro to Statistics)
“Those who die, do so very early, while those who live go on living very long. Whenever there is asymmetry, the average survival has nothing to do with the median survival.”
― Nassim Nicholas Taleb, Fooled by Randomness
Nassim Nicholas Taleb, has been successful as a derivatives trader, best selling author, turned twitter provoker by understanding the limitations of basic statistics. His debut book Fooled by Randomness, was released in 2001 as the tech bubble was bursting.
Taleb had recently completed his PhD and moved back to New York when he wrote Fooled by Randomness. Due to his occupation as a trader, much of his criticism was centered around decision making as it pertains to finance. Fooled by Randomness accomplishes several things, one of which was to begin educating readers why they don’t understand probability. Part of the reason why is because of common interpretations of measures of central tendency: the mean and median.
The Gaussian Bell Curve Distribution. Fat in the center, thin at the edges
Most of us learnt how to interpret the Mean (average) and Median from school, and take these values to approximate the most likely outcome. Perhaps some will go a level further and say that most outcomes will happen within one (68%) or two (95%) standard deviations from this mean. Therefore if you know the mean and the standard deviation, you might believe you can forecast a likely range of outcomes.
Sounds simple enough right? Unfortunately when an activity does not follow the Bell Curve shape, this type of analysis far less useful. Taleb explains this by splitting different types of events into two categories: Mediocristan and Extremistan.
Mediocristan, refers to environments where outcomes are relatively stable, predictable, and bounded. Typical outcomes cluster around the mean, and no single observation has a dramatic effect on the overall picture. Examples include human height or weight, where deviations are relatively small and extreme outliers (e.g., someone growing 30 feet tall) are implausible. In such domains, it will follow the Bell curve so the mean and median are reliable enough to give a good understanding of the distribution. Other examples of Medicristan include: Test scores in standardized exams1 , wages in a unionized environment, blood pressure etc.
However, Taleb stresses that much of the real world operates in Extremistan, where rare and extreme events—“Black Swans”—occur and these outliers can disproportionately affect the mean. A few large outcomes can account for the majority of the results, rendering the average misleading or completely useless. Unlike in Mediocrastistan, events in Extremistan are highly volatile, unbounded (no structural max or minimum) and unpredictable. A few good examples are wealth generation, stock market returns2 , entrepreneurship, venture capital, book sales, and Pandemics.
Now that we’ve established that the measures of central tendency are only helpful in Mediocrastistan, and for an activity to belong there, it needs to be 1) stable, 2) predictable and 3) bounded. This means the bell curve applies to far fewer things than we thought, making the predictive power of the mean & median far less meaningful.
Now that this is settled, why as a society are we so fixated on means and medians, what should we be doing instead?
Reliance on Benchmarking
“Too often, people judge their progress on a relative basis instead of in absolute terms. We feel like we are good at something if are better than the people around us.” —Ben Saltiel, The Most Obvious Secret to Success
Benchmarking is a method that involves comparing current processes or performances against a wider population’s. The idea is that a company or individual can see where they stack up relative to the selected peer group, to understand if they are good, average or bad. This goes hand in hand with the measures of central tendency that we discussed above.
Nobody wants to be bad, so if we see an area that is below average or outside the top quartile/decile, the natural goal will be to focus improvement on that area until the desired level is reached. This is the logic that underpins most goal setting in corporate, political and personal environments.
Benchmarking gives a certain psychological safety, since there is reassurance and strength in numbers. If only one student in the class fails a test, it’s difficult for them to blame the teacher. If every student fails, it removes the blame from any one individual. It shifts the focus on just trying to be better than others, instead of just trying to be as good as possible.
In some cases it makes sense, if you are getting chased by a bear, you don’t need to be Usain Bolt, you just need to be faster than your friend. This is where relative performance matters more than absolute. This is why when planning a canoe trip, always select campmates you are confident you can outrun. While most of these cases are not life or death, most “rent seekers3” always try to be judged on a relative basis.
Rent seekers by definition, don’t care about making things better, they just want to extract wealth for themselves. This is why they can hide behind benchmarking to cloud their lack of performance or lack of added value. They can always argue that their performance is better than x% or in line with the top y% of their peer groups.
Mr. Burns often engages in rent seeking behavior.
Alternative to Benchmarking
Top performers don’t care about benchmarking. The fact of the matter is, they are the outliers and to become this way, they understood that their activity exists in Extremistan, where fat tails can drive a disproportionate amount of the results. Since they are at the top of the game, benchmarking against other competitors won’t help them. Top performers in Extremistan, aren’t slightly more successful than their peers, they are in a completely different universe.
Elon Musk has started several multi-billion dollar companies, in different industries. In the case of SpaceX, his approach was the antithesis of benchmarking. Elon realized that the production of rockets was outrageously expensive, industry wide. Doing some quick calculations, he estimated he could produce rockets at a much cheaper price. Instead of benchmarking, he devised an internal measure call The Idiot Index.
Idiot Index = Cost of finished product / Cost of raw materials
The higher the index, the more opportunity there is to design something more cheaply. He realized that the Idiot Index was high for many of the components of space rockets, which is why SpaceX started to building most of their parts in-house.
This runs contrary to what most bankers or consultants would do if they were advising Elon on his decision to start a company in that industry. They likely would have plotted the production costs of all the components performed by each manufacturer in the industry and told him his best case scenario was that he could hope to achieve. The difference in these approaches is that Elon understood that starting a company is a Extremistan activity, whereas consultants and bankers would evaluate this project as if Elon lived in Mediocrastistan.
Unfortunately, most people make the same mistakes as the bankers and consultants. They rely on benchmarking to assess how they perform relative to other participants, drastically limiting their potential to the local or global maximum. Instead of doing the difficult work, in raising the global maximum, they are settling for safety in the crowds.
In How to close anybody, I talked about the importance of building a business case or proving out an ROI to win over a skeptical CFO. What most sales people do, is make some meek claim such as “Customers of company X, saved on average 10% using our product”. A skeptical CFO, will tear this apart, for all kinds of reasons. A more diligent sales person or consultant, might first get collect some data from the prospect, input their data in a spreadsheet to benchmark their current performance against the competition. They will then show them how they are underperforming, and how their tool or method will get them up to the desired standard and how that would translate into more sales or lower costs. This is better but you might be drastically under or overselling your tool.
These are the most widely selected methods, mostly because they are the easiest to perform. They might help convince a CFO to sign off on your deal, but they are also part of the reason why most software purchases don’t live up to the ROI expected by buyers. This is because each company is unique, trying to plot their performance in Mediocrastistan does not make much sense. If a company is serious about getting proper value from their tool, they would probably have to do an analysis closer to what Elon did. This would likely result in them buying far less software.
Doing this level of analysis is difficult work. This is why people rarely do it and instead prefer to rely on benchmarking and a few simplifying assumptions to give them safety in crowds instead of trying to become an outlier themselves.
Instead of trying to treat every participant like they are the same, focus on factor endowments, or the unique value that you can bring. This is the only way you can hope to achieve outlier results. You can’t become great just trying to copying everybody else.
Employers or schools will regularly do this but it is for their benefit not yours. It’s easier for them to grade and rank students as if they were competing in a tournament than to asses individual greatness. This is why some Thiel Fellows have managed to start multi billion dollar companies in their 20s, while peers with similar grades might be working as accountants.
Similarly, employers will award you a salary based on what other people at the company make or what people with the same titles in the market get paid. This treats you as if you bring similar value or the same value as everybody else. Lebron James doesn’t earn meaningfully more than the 30th or 50th best player in the NBA despite making way more money for his franchise and impacting winning in a much greater degree. This is because the NBA has a salary cap and max salaries a player can earn. This is extreme but something similar goes on in all kinds of careers. When your employer tries to pay you in relation to what other people make, they are limiting your earnings potential. If they paid you in relation to how much value you provide to the organization, if you are one of the better employees, you might be worth being paid 2-20X the average employee.
(Do you even know who the person on the right is?)
This assumes your role can have a non-linear impact on the organization. If your role/responsibilities are fairly uniform with your colleagues than you work in in Mediocrastistan, where the best employee doesn’t drive much more value than an average one. However for many roles, the difference between an outlier performer and a good performer might be a factor of 10 or 100x.
This is why benchmarking is convenient for schools and employers but awful if you are trying to be great or drive unique value. You need to do the hard work to demonstrate the unique value that you bring unless your goal is to be average, than stick to benchmarking.
I hope you enjoyed today’s article. Please Subscribe or Share Some Feedback in the Comments
If you liked this article, here are some other Substack accounts you may enjoy:
One student getting a perfect score or zero won’t meaningfully change the average, since the outcomes are both bounded.
1 great trade can offset years of steady gains or losses. Taleb talks extensively about this in Fooled by Randomness and The Black Swan.
A rent seeker is an individual, business, or organization that seeks to increase their wealth without creating new value or contributing to productivity. Instead, rent seekers aim to gain financial benefits or economic advantage by manipulating the political or regulatory environment, often through lobbying, regulatory capture, or other non-productive activities. Rent-seeking behavior typically involves efforts to secure government favors, subsidies, special privileges, or market monopolies, which allow the rent seeker to earn income without contributing to the overall economic growth.