Forecast error can be consequential

Consider one item of many

• Product X costs $100 to make and nets $50 profit per unit.
• Sales of Product X will turn out to be 1,000/month over the next 12 months.
• Consider one item of many

What is the cost of forecast error?

• If the forecast is 10% high, end the year with $120,000 of excess inventory.
• 100 extra/month x 12 months x $100/unit
• If the forecast is 10% low, miss out on $60,000 of profit.
• 100 too few/month x 12 months x $50/unit

Three mistakes to avoid

1. Ignoring error.

• Unprofessional, dereliction of duty.
• Wishing will not make it so.
• Treat accuracy assessment as data science, not a blame game.

2. Tolerating more error than necessary.

• Statistical forecasting methods can improve accuracy at scale.
• Improving data inputs can help.
• Collecting and analyzing forecast error metrics can identify weak spots.

3. Wasting time and money going too far trying to eliminate error.

• Some product/market combinations are inherently more difficult to forecast. After a point, let them be (but be alert for new specialized forecasting methods).
• Sometimes steps meant to reduce error can backfire (e.g., adjustment).

RECENT POSTS

So how can you execute a real-world plan for JIT inventory amidst all this risk and uncertainty? The foundation of your response is your corporate data. Uncertainty has two sources: supply and demand. You need the facts for both.

On the supply side, exploit the data you have on recent supplier lead times, which reflect the current turbulence. Don’t use average values when you can use probability distributions that reflect the full range of contingencies. Consider this comparison. Supplier A is now reliably filling orders in exactly 10 days. Supplier B also averages 10 days but does with a 78%/22% mix of 7 and 21 days. Both A and B have an average replenishment delay of 10 days, but the operational results they provide will be very different. You can only recognize this if you use probability models of inventory performance.

On the demand side, similar considerations apply. First, recognize that there may have been a major shift in the character of item demand (statisticians call this a “regime change”), so purge from your analysis any data that represent the “good old days.” Then, again, stop thinking in terms of averages. While the average demand is important, it is not a sufficient descriptor of the problem you face. Equally important is the volatility of demand. Volatility is the reason you keep inventory in the first place. If demand were completely predictable, you would have neither stockouts nor excess inventory. Just as you need to estimate the full probability distribution of replenishment lead times, you need the full distribution of demand values.

Once you understand the range of variability in both supply and demand, probabilistic forecasting will allow you to account for disruptions and unusual events. Software will convert your data on demand and lead times into huge numbers of scenarios representing how your next planning period might play out. Given those scenarios, the software can determine how best to meet your goals for such metrics as inventory costs and stockout rates. Using solutions such as Smart Inventory Optimization , you will confidently plan based on your targeted stockout risk with minimal inventory carrying cost. You may also consider letting the solution prescribe optimal service level targets by assessing the costs of additional inventory vs. stockout cost.

In inventory planning, as in science, we cannot escape the reality of uncertainty and the impact of unusual events. We must plan accordingly: using inventory optimization software helps you identify the least-cost service level. This creates a coherent, company-wide effort that combines visibility into current operations with mathematically correct assessments of future risks and conditions.

Inventory planning has become more “interesting” and requires a greater degree of risk awareness and agility. The right software can help.

Four general types of error metrics 

1. Scale-dependent error

2. Percentage error

3. Relative error

4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

1. Scale-dependent error metrics

• Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
• Median Absolute Error (MdAE)
• Root Mean Square Error (RMSE)

• These metrics express the error in the original units of the data.
  • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
• Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
  • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
• To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

2. Percentage error metric

• Mean Absolute Percentage Error (MAPE)
  

• This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
• The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
• Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
• The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

3. Relative error metric

• Median Relative Absolute Error (MdRAE)

• Relative to what? To a benchmark forecast.
• What benchmark? Usually, the “naïve” forecast.
• What is the naïve forecast? Next forecast value = last actual value.
• Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

4. Scale-Free error metric

• Median Relative Scaled Error (MdRSE)
  

• This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
• The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
  • (This is the same as the error made by the naïve forecast.)
• How does this metric differ from the MdRAE above?
  • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
  • This matters because there are usually many more history values than there are forecasts.
  • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

The special problem of intermittent demand

• “Intermittent” demand has many zero demands mixed in with random non-zero demands.
• MAPE gets ruined when errors are divided by zero.
• MdRAE can also get ruined.
• MdSAE is less likely to get ruined.

Recap and remarks

• Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
• There are two major classes of metrics: absolute and relative.
• Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
• Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
• Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
• When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.

RECENT POSTS

Finally, as I have already noted, we need to accept that we can never eliminate all uncertainty. As a physicist, I have always been intrigued by the fact that, even at the most basic levels of reality as we understand it today, there is still uncertainty. Albert Einstein believed in certainty (determinism) in physical law.  If he were an inventory manager, he might have argued for JIT because he believed physical laws should allow perfect predictability. He famously said, “God does not play with dice.”  Or could it be possible that the universe we exist in was a “black swan” event in a prior “multi-verse” that produced a particular kind of universe that allowed us to exist.

In inventory planning, as in science, we cannot escape the reality of uncertainty and the impact of unusual events.  We must plan accordingly.

[1] https://www.bbc.com/news/business-56559073#:~:text=Looking%20at%20the%20bigger%20picture,0.2%20to%200.4%20percentage%20points.