The Smart Forecaster

 Pursuing best practices in demand planning,

forecasting and inventory optimization

Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving forecast accuracy by measuring forecast error. We begin by overviewing the various types of error metrics: scale-dependent error, percentage error, relative error, and scale-free error metrics. While some error is inevitable, there are ways to reduce it, and forecast metrics are necessary aids for monitoring and improving forecast accuracy. Then we will explain the special problem of intermittent demand and divide-by-zero problems. Tom concludes by explaining how to assess forecasts of multiple items and how it often makes sense to use weighted averages, weighting items differently by volume or revenue.

 

Four general types of error metrics 

1. Scale-dependent error
2. Percentage error
3. Relative error
4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

 

1. Scale-dependent error metrics

  • Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
  • Median Absolute Error (MdAE)
  • Root Mean Square Error (RMSE)
  • These metrics express the error in the original units of the data.
    • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
  • Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
    • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
  • To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

 

2. Percentage error metric

  • Mean Absolute Percentage Error (MAPE)
  • This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
  • The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
  • Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
  • The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

 

3. Relative error metric

  • Median Relative Absolute Error (MdRAE)
  • Relative to what? To a benchmark forecast.
  • What benchmark? Usually, the “naïve” forecast.
  • What is the naïve forecast? Next forecast value = last actual value.
  • Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

 

4. Scale-Free error metric

  • Median Relative Scaled Error (MdRSE)
  • This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
  • The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
    • (This is the same as the error made by the naïve forecast.)
  • How does this metric differ from the MdRAE above?
    • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
    • This matters because there are usually many more history values than there are forecasts.
    • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

 

Intermittent Demand Planning and Parts Forecasting

 

The special problem of intermittent demand

  • “Intermittent” demand has many zero demands mixed in with random non-zero demands.
  • MAPE gets ruined when errors are divided by zero.
  • MdRAE can also get ruined.
  • MdSAE is less likely to get ruined.

 

Recap and remarks

  • Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
  • There are two major classes of metrics: absolute and relative.
  • Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
  • Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
  • Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
  • When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.
Leave a Comment

RECENT POSTS

Uncover data facts and improve inventory performance

Uncover data facts and improve inventory performance

The best inventory planning processes rely on statistical analysis to uncover relevant facts about the data. When you have the facts and add your business knowledge, you can make more informed stocking decisions that will generate significant returns. You’ll also set proper expectations with internal and external stakeholders, ensuring there are fewer unwelcome surprises.

What Silicon Valley Bank Can Learn from Supply Chain Planning

What Silicon Valley Bank Can Learn from Supply Chain Planning

If you had your head up lately, you may have noticed some additional madness off the basketball court: The failure of Silicon Valley Bank. Those of us in the supply chain world may have dismissed the bank failure as somebody else’s problem, but that sorry episode holds a big lesson for us, too: The importance of stress testing done right.

Do your statistical forecasts suffer from the wiggle effect?

Do your statistical forecasts suffer from the wiggle effect?

What is the wiggle effect? It’s when your statistical forecast incorrectly predicts the ups and downs observed in your demand history when there really isn’t a pattern. It’s important to make sure your forecasts don’t wiggle unless there is a real pattern. Here is a transcript from a recent customer where this issue was discussed:

Recent Posts

  • professional technician engineer planning spare parts in industrial manufacturing factory,Prepare your spare parts planning for unexpected shocks
    In today's unpredictable business climate, we do have to worry about supply chain disruptions, long lead times, rising interest rates, and volatile demand. With all these challenges, it's never been more vital for organizations to accurately forecast parts usage, stocking levels, and to optimize replenishment policies such as reorder points, safety stocks, and order quantities. In this blog, we'll explore how companies can leverage innovative solutions like inventory optimization and parts forecasting software that utilize machine learning algorithms, probabilistic forecasting, and analytics to stay ahead of the curve and protect their supply chains from unexpected shocks. […]
  • Uncover data facts and improve inventory performanceUncover data facts and improve inventory performance
    The best inventory planning processes rely on statistical analysis to uncover relevant facts about the data. When you have the facts and add your business knowledge, you can make more informed stocking decisions that will generate significant returns. You'll also set proper expectations with internal and external stakeholders, ensuring there are fewer unwelcome surprises. […]
  • Electricity problems. Repairman is working indoors with Software for spare partsElectric Utilities’ Problems with Spare Parts
    Every organization that runs equipment needs spare parts. All of them must cope with issues that are generic no matter what their business. Some of the problems, however, are industry specific. This post discusses one universal problem that manifested in a nuclear plant and one that is especially acute for any electric utility. […]
  • Correlation vs Causation Relevant to your demand planning businessCorrelation vs Causation: Is This Relevant to Your Job?
    Outside of work, you may have heard the famous dictum “Correlation is not causation.” It may sound like a piece of theoretical fluff that, though involved in a recent Noble Prize in economics, isn’t relevant to your work as a demand planner. Is so, you may be only partially correct. […]
  • Downtown Miami skyline panorama and with software guided lights on at duskSmart Software Customer, Arizona Public Service to Present at USMA 2023
    Smart Software CEO and APS Inventory & Logistics Manager to present USMA 2023 Session on APS supply chain transformation project and the role of inventory optimization technology in their new process. […]

    Inventory Optimization for Manufacturers, Distributors, and MRO

    • professional technician engineer planning spare parts in industrial manufacturing factory,Prepare your spare parts planning for unexpected shocks
      In today's unpredictable business climate, we do have to worry about supply chain disruptions, long lead times, rising interest rates, and volatile demand. With all these challenges, it's never been more vital for organizations to accurately forecast parts usage, stocking levels, and to optimize replenishment policies such as reorder points, safety stocks, and order quantities. In this blog, we'll explore how companies can leverage innovative solutions like inventory optimization and parts forecasting software that utilize machine learning algorithms, probabilistic forecasting, and analytics to stay ahead of the curve and protect their supply chains from unexpected shocks. […]
    • Electricity problems. Repairman is working indoors with Software for spare partsElectric Utilities’ Problems with Spare Parts
      Every organization that runs equipment needs spare parts. All of them must cope with issues that are generic no matter what their business. Some of the problems, however, are industry specific. This post discusses one universal problem that manifested in a nuclear plant and one that is especially acute for any electric utility. […]
    • Worker maintenance industrial machine robotic Forecasting Spare PartsHow to Forecast Spare Parts with Low Usage
      What do you do when you are forecasting an intermittently demanded item, such as a spare part, with an average demand of less than one unit per month? Most of the time, the demand is zero, but the part is significant in a business sense; it can’t be ignored and must be forecasted to be sure you have adequate stock. […]
    • Spare Parts, Replacement Parts, Rotables, and Aftermarket PartsSpare Parts, Replacement Parts, Rotables, and Aftermarket Parts
      Those new to the parts planning game are often confused by the many variations in the names of parts. This blog points out distinctions that do or do not have operational significance for someone managing a fleet of spare parts and how those differences impact inventory planning. […]