Nearly 20 years ago, Tim Ronak said he was frustrated with the way shops were being judged on KPIs.
“I saw an irrational behavior as a result of a purported truth when shops didn’t conform to a specific statistic or negatively performed when compared to that measurement. I felt that in many cases, the statistic benchmark didn’t have any validity,” said Ronak, senior services consultant for AkzoNobel. “Shops were frantically trying to move a number they weren’t able to move, and this is particularly true when discussing severity.”
As a result, about 10 years ago Ronak wrote an article titled “Does Severity Matter” as a tool to help people understand the statistics on which shops are being evaluated.
“You have a right to understand what they are and how to interpret it,” he said. “If it’s an arbitrary measurement, you need to call it what it is.”
During the SEMA 2017 Show in Las Vegas, Ronak asked a room full of collision repairers, “Have you ever been told your severity is too high? Absolutely---I know [that] everyone here who has run a shop has been told that at one point or another.”
What Ronak found over the years is that “typically, severity is too variable to actually be used to meaningfully measure performance.”
He helped SEMA attendees understand how severity is calculated and addressed what shops can do when told their severity is too high. The following information is from Ronak’s SEMA presentation “Severity—Why It Does Not Matter and What to Do About It!” The presentation was part of the SCRS Repairer Driven Education series.
Q: What is “severity?”
A: In the context of this discussion, severity is the cumulative costs associated with effecting repairs to a damaged vehicle to restore it to its pre-loss condition.
Q: What is important to understand in terms of severity?
A: Vehicles aren’t built the same way as they were in the past. Each one has its own unique set of circumstances when it comes to repairs. No longer is it one size fits all. According to Thatcham Research Centre, a think-tank study group funded by the insurance industry in Europe, OEMs must lower collision repair costs.
“The average repair bill has risen by 32 percent over the last three years,” said Thatcham CEO Peter Shaw in a statement. “This has been driven by the repairability of parts such as headlamps, increasing complexity of vehicle materials and technology and the rising cost of spare parts, influenced to some extent by currency fluctuations. Vehicle manufacturers must bring these costs under control.”
Thatcham is the United Kingdom’s version of the Insurance Institute for Highway Safety, and also researches and develops repair procedures.
If costs are rising and they are comparing your shop with historical data, what does the current data look like? How could you not have your severity too high in a rising environment? It’s mathematically impossible, yet it seems that everyone on DRPs are beaten into submission by a range of statistics they can’t directly change.
Frequency is up and severity is up. With both of those rising, we’re going to see higher severity amounts and even more claims. This will create more pressure to ‘manage’ to an ‘unmanageable’ target.
Q: When it comes to KPIs, is an “average” a meaningful performance measure?
A: In statistics, there are three measures of what is called central tendency---the typical value in the distribution that describes the way in which a group of data cluster around a central value. There is the mean, the arithmetic average of all values; the median, the midpoint of values; and the mode, the most frequently occurring value.
What I found from my research over the years is the mean is not a reliable or meaningful measurement. Think about the repair of a Maserati, a Cadillac and a Ford. As an industry, we have variability among the type of work that we do. Additionally, if we align manufacturer context, all Cadillacs do not run into the same tree at the same speed!
In a normal distribution, there is a concept of standard deviation, which measures the quality of an average. In a normal distribution, the standard deviation will be around 30-40 percent of the value of the mean, both above the mean and below the mean. The key point is that any random data set will vary around that central value both above and below until it contains the ‘majority’ (+68 percent) of the observed values. This ‘normal distribution’ typically only occurs when enough data is present (typically more than 1,000 observations) and the population the data is pulled from is completely random. In a financial environment where deductibles affect the number of submitted claims, the data loses this ‘random’ status and could be potentially skewed as a result. This affects the QUALITY of the arithmetic mean as a KPI measurement.
For example, if an insurer with a VERY large 5,000 claim data set for a current month in your immediate local area has a $3,000 average RO value for vehicles it purports to have paid repairs on, one-third (1 standard deviation) of that is about $1,000. This means if their data strictly followed a normally distributed pattern, 68 percent of all of the cars repaired would range between $2,000 and $4,000) following +/- 1 standard deviation of the normally distributed data. In this case with a perfect normally distributed dataset if an individual shop has an average severity of $3,600, you’re within one standard deviation above the calculated arithmetic mean. That’s 68 percent of all the data. The majority of the data would support the fact that $3,600 is not out of line, as it does not vary more than the 68 percent majority.
Statistics are only as good as the data you gather. Otherwise, you can’t use it as any kind of a measure.
Q: What is important for a shop to know in order to find out if the data is meaningful?
A: Sample size matters! The data analysis is dependent on the amount and quality of data that you have. Smaller sample sizes typically have wider variability. This reduces the validity of the data as a meaningful performance measure. For any arithmetic average to be meaningful, the sample sizes need to be larger than 1,000. For data sets where there are fewer than 1,000 individual values, you need to ‘test’ the data to determine the reliability of the calculated average mean as a representative KPI.
Q: How can KPI data be tested using standard deviation?
A: All of a shop’s data varies around a central value called the ‘mean.’ One test that can be used to assess the reliability of a particular KPI is a standard deviation calculation. This determines how far a particular KPI can vary above and below a particular central value while still including the majority of all that data---68 percent.
In other words, if we calculate the standard deviation, we are saying 68 percent of all the data that you have---if the sample size is large enough---should follow a random distribution of data and be contained within one standard deviation above and below the ‘mean.’
The purpose is to understand how to use this to your benefit. Unless you consider the variability of the actual data analyzed, a mean might not tell you the whole story and may be highly suspect as a measure of a central tendency for any specific data set from a performance benchmark standpoint.
In addition to severity, this can also apply to CSI, cycle time, aftermarket parts usage, type of part usage, rental car data and any other KPI used to evaluate your business activity.
Q: How far above/below a KPI is too far?
A: Accidents are random events. It is only when enough occur---more than 1,000---that they MAY begin to form a normal distribution, which will have the majority within 68.3 percent of the individual data set when you create a range +/- 1 standard deviation around that mean.
Smaller or ‘skewed’ sample sizes may create flatter distributions with larger variability and a wider numeric value range with all of the data.
Based on the idea that the data will naturally vary around a central point, it makes sense to consider where the majority---68 percent---of that data will fall. That happens to be one standard deviation away from the arithmetic mean. The standard deviation should be around 34 percent of the mean value. The wider or larger the magnitude of the standard deviation, the less reliable the mean is as a performance measure.
The point is that with this data, you’re being judged on something that you’re not [able to be judged] on. However, body shops will try to meet certain targets even if it doesn’t make mathematical sense. It’s important to understand the concept of sample size---how many cars your shop is being judged on.
I looked at the average RO data from 26 body shops over a 12-month period. The 2,858 ROs I reviewed represented every car repaired except for those that were parts-only with no labor, ROs that were detail only and all total losses. Every data set that was reviewed showed signs of significant skewness, which resulted in variability with a VERY large standard deviation value equaling 95—110 percent of the mean. This implies that for this data set, severity is a poor measure of central value.
Q: When a shop is told that their severity is too high, what can be done?
A: Establish a ceiling value that is reasonable! Use a calculation result of the purported ‘mean’ plus the standard deviation to identify an upper limit for the KPI. If the KPI monitor cannot provide you with a calculated standard deviation, simply calculate the dollar range yourself from YOUR actual data. Like it or not, KPIs are being used to make decisions about your business performance whether the KPIs are valid or not. Being able to discuss those variables in an intelligent way is crucial for you to be able to deflect a negative critique or unjustified demand, especially when it is beyond your control. I often ask shops if their DRP feels like they are playing whack-a-mole.
Q. What if I just work on high-end luxury cars? Does that affect the data?
A: Shops can identify the range that would include the 68 percent majority of the data of a particular data set. In addition to this, is the FACT that some cars are just more expensive to repair than others. You do need to adjust the stated “severity” value in any KPI discussion to reflect this disparity between brands and models. An excellent tool for this is the Highway Loss Data Institute (HLDI) online data that is the accumulation of virtually every insurer’s paid claim data and using a ‘base 100 scale’ clearly shows that some vehicle brands may in fact cost +150 percent to +200 percent more than others to repair. This would move the $3,000 average severity number discussed above to either $4,500 as an adjusted central number at 150 percent or $6,000 at 200 percent.
An important point with these more expensive luxury vehicles is that within the HLDI data, it indicates [that] the more expensive cars to repair with additional technology and safety systems seem to have significantly LESS occupant injury costs.
Ronak created a sortable Excel tool to determine the severity weighted value of a shop’s work mix percentage by vehicle brand. After entering the severity value supplied by the DRP, it can be adjusted to a shop’s specific work mix. The weighted value will represent where the severity value should be adjusted based on the type of work mix you have using the insurance-provided data.
Tim Ronak’s SEMA presentation “Severity—Why It Does Not Matter and What to Do About It!” can be accessed online. The presentation was part of the SCRS Repairer Driven Education series.
For more information or to obtain a copy of the sortable Excel tool, email Tim Ronak: timothy.ronak@akzonobel.com.
Action Plan for Shops
How to have a conversation with a vendor partner about your performance:
- Suggest that the number of cars repaired is insufficient to create any kind of comparable benchmark due to the variability.
- Discuss how the variability using the average value is a useless performance measure unless you include the value range that the majority (68 percent) of the data will fall into. This is a range of +/– 1 standard deviation. This implies that you need to know the standard deviation value for the DRP data as well as your own shop data.
- Ensure that any KPI central measure represents the TYPE of cars you actually work on. Use the available insurance company data available from the Insurance Institute for Highway Safety and the Highway Loss Data Institute (HLDI). A sortable Excel tool is available.
- Work to determine the quality of the data by requesting the standard deviation of the data set being used to evaluate your shop.
- Then use that value to assess the variability of the original data and define a range that 68 percent of the data falls into.
- If that is not forthcoming, calculate your own standard deviation and demonstrate that the sample is too small and too variable to be a meaningful measurement.
- If the debate continues, elevate the analysis of the KPI to include the actual provided data from the insurance industry regarding the paid-out claim severity data by OEM from HLDI.
- Use this data to weigh the DRP-provided average severity value to reflect the actual work mix completed by your facility.
- Imply that there still needs to be a range around this work mix adjusted severity value higher and lower due to variability.
Stacey Phillips Ronak