Four Useful Ways to Measure Forecast Error

The Smart Forecaster

 Pursuing best practices in demand planning,

forecasting and inventory optimization

Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving forecast accuracy by measuring forecast error. We begin by overviewing the various types of error metrics: scale-dependent error, percentage error, relative error, and scale-free error metrics. While some error is inevitable, there are ways to reduce it, and forecast metrics are necessary aids for monitoring and improving forecast accuracy. Then we will explain the special problem of intermittent demand and divide-by-zero problems. Tom concludes by explaining how to assess forecasts of multiple items and how it often makes sense to use weighted averages, weighting items differently by volume or revenue.

 

Four general types of error metrics 

1. Scale-dependent error
2. Percentage error
3. Relative error
4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

 

1. Scale-dependent error metrics

  • Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
  • Median Absolute Error (MdAE)
  • Root Mean Square Error (RMSE)
  • These metrics express the error in the original units of the data.
    • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
  • Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
    • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
  • To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

 

2. Percentage error metric

  • Mean Absolute Percentage Error (MAPE)
  • This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
  • The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
  • Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
  • The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

 

3. Relative error metric

  • Median Relative Absolute Error (MdRAE)
  • Relative to what? To a benchmark forecast.
  • What benchmark? Usually, the “naïve” forecast.
  • What is the naïve forecast? Next forecast value = last actual value.
  • Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

 

4. Scale-Free error metric

  • Median Relative Scaled Error (MdRSE)
  • This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
  • The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
    • (This is the same as the error made by the naïve forecast.)
  • How does this metric differ from the MdRAE above?
    • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
    • This matters because there are usually many more history values than there are forecasts.
    • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

 

Intermittent Demand Planning and Parts Forecasting

 

The special problem of intermittent demand

  • “Intermittent” demand has many zero demands mixed in with random non-zero demands.
  • MAPE gets ruined when errors are divided by zero.
  • MdRAE can also get ruined.
  • MdSAE is less likely to get ruined.

 

Recap and remarks

  • Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
  • There are two major classes of metrics: absolute and relative.
  • Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
  • Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
  • Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
  • When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.
Leave a Comment

RECENT POSTS

Goldilocks Inventory Levels

Goldilocks Inventory Levels

You may remember the story of Goldilocks from your long-ago youth. Sometimes the porridge was too hot, sometimes it was too cold, but just once it was just right. Now that we are adults, we can translate that fairy tale into a professional principle for inventory planning: There can be too little or too much inventory, and there is some Goldilocks level that is “just right.” This blog is about finding that sweet spot.

Call an Audible to Proactively Counter Supply Chain Noise

Call an Audible to Proactively Counter Supply Chain Noise

You know the situation: You work out the best way to manage each inventory item by computing the proper reorder points and replenishment targets, then average demand increases or decreases, or demand volatility changes, or suppliers’ lead times change, or your own costs change.

An Example of Simulation-Based Multiechelon Inventory Optimization

An Example of Simulation-Based Multiechelon Inventory Optimization

Managing the inventory across multiple facilities arrayed in multiple echelons can be a huge challenge for any company. The complexity arises from the interactions among the echelons, with demands at the lower levels bubbling up and any shortages at the higher levels cascading down.

Recent Posts

  • Smart Software CEO to present at Epicor Insights 2022Smart Software to Present at Epicor Insights 2022
    Smart Software CEO will present at this year's Epicor Insights event in Nashville. If you plan to attend this year, please join us at booth #9 and #705, and learn more about Epicor Smart Inventory Planning and Optimization. […]
  • Smart Software and Arizona Public Service to Present at WERC 2022
    Smart Software CEO and APS Inventory Manager to present WERC 2022 Studio Session on implementing Smart IP&O in 90 Days and achieve significant savings by optimizing reorder points and order quantities for over 250,000 spare parts. […]

    Improve Forecast Accuracy by Managing Error

    The Smart Forecaster

     Pursuing best practices in demand planning,

    forecasting and inventory optimization

    Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

    In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving Forecast Accuracy by Managing Error. This video is the first in our series on effective methods to Improve Forecast Accuracy.  We begin by looking at how forecast error causes pain and the consequential cost related to it. Then we will explain the three most common mistakes to avoid that can help us increase revenue and prevent excess inventory. Tom concludes by reviewing the methods to improve Forecast Accuracy, the importance of measuring forecast error, and the technological opportunities to improve it.

     

    Forecast error can be consequential

    Consider one item of many

    • Product X costs $100 to make and nets $50 profit per unit.
    • Sales of Product X will turn out to be 1,000/month over the next 12 months.
    • Consider one item of many

    What is the cost of forecast error?

    • If the forecast is 10% high, end the year with $120,000 of excess inventory.
    • 100 extra/month x 12 months x $100/unit
    • If the forecast is 10% low, miss out on $60,000 of profit.
    • 100 too few/month x 12 months x $50/unit

     

    Three mistakes to avoid

    1. Ignoring error.

    • Unprofessional, dereliction of duty.
    • Wishing will not make it so.
    • Treat accuracy assessment as data science, not a blame game.

    2. Tolerating more error than necessary.

    • Statistical forecasting methods can improve accuracy at scale.
    • Improving data inputs can help.
    • Collecting and analyzing forecast error metrics can identify weak spots.

    3. Wasting time and money going too far trying to eliminate error.

    • Some product/market combinations are inherently more difficult to forecast. After a point, let them be (but be alert for new specialized forecasting methods).
    • Sometimes steps meant to reduce error can backfire (e.g., adjustment).
    Leave a Comment

    RECENT POSTS

    Goldilocks Inventory Levels

    Goldilocks Inventory Levels

    You may remember the story of Goldilocks from your long-ago youth. Sometimes the porridge was too hot, sometimes it was too cold, but just once it was just right. Now that we are adults, we can translate that fairy tale into a professional principle for inventory planning: There can be too little or too much inventory, and there is some Goldilocks level that is “just right.” This blog is about finding that sweet spot.

    Call an Audible to Proactively Counter Supply Chain Noise

    Call an Audible to Proactively Counter Supply Chain Noise

    You know the situation: You work out the best way to manage each inventory item by computing the proper reorder points and replenishment targets, then average demand increases or decreases, or demand volatility changes, or suppliers’ lead times change, or your own costs change.

    An Example of Simulation-Based Multiechelon Inventory Optimization

    An Example of Simulation-Based Multiechelon Inventory Optimization

    Managing the inventory across multiple facilities arrayed in multiple echelons can be a huge challenge for any company. The complexity arises from the interactions among the echelons, with demands at the lower levels bubbling up and any shortages at the higher levels cascading down.

    Recent Posts

    • Smart Software CEO to present at Epicor Insights 2022Smart Software to Present at Epicor Insights 2022
      Smart Software CEO will present at this year's Epicor Insights event in Nashville. If you plan to attend this year, please join us at booth #9 and #705, and learn more about Epicor Smart Inventory Planning and Optimization. […]
    • Smart Software and Arizona Public Service to Present at WERC 2022
      Smart Software CEO and APS Inventory Manager to present WERC 2022 Studio Session on implementing Smart IP&O in 90 Days and achieve significant savings by optimizing reorder points and order quantities for over 250,000 spare parts. […]

      Automatic Forecasting for Time Series Demand Projections

      The Smart Forecaster

       Pursuing best practices in demand planning,

      forecasting and inventory optimization

      Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

      In this video tutorial Dr. Thomas Willemain, co–Founder and SVP Research at Smart Software, presents Automatic Forecasting for Time Series Demand Projections, a specialized algorithmic tournament to determine an appropriate time series model and estimate the parameters to compute the best forecasts methods. Automatic forecasts of large numbers of time series are frequently used in business, some have trend either up or down, and some have seasonality so they are cyclic, and each of those specific patterns requires a suitable technical approach, and an appropriate statistical forecasting method.  Tom explains how the tournament computes the best forecasts methods and works through a practical example.

      AUTOMATIC FORECASTING COMPLETE-VIDEO-2
      Leave a Comment

      RECENT POSTS

      Goldilocks Inventory Levels

      Goldilocks Inventory Levels

      You may remember the story of Goldilocks from your long-ago youth. Sometimes the porridge was too hot, sometimes it was too cold, but just once it was just right. Now that we are adults, we can translate that fairy tale into a professional principle for inventory planning: There can be too little or too much inventory, and there is some Goldilocks level that is “just right.” This blog is about finding that sweet spot.

      Call an Audible to Proactively Counter Supply Chain Noise

      Call an Audible to Proactively Counter Supply Chain Noise

      You know the situation: You work out the best way to manage each inventory item by computing the proper reorder points and replenishment targets, then average demand increases or decreases, or demand volatility changes, or suppliers’ lead times change, or your own costs change.

      An Example of Simulation-Based Multiechelon Inventory Optimization

      An Example of Simulation-Based Multiechelon Inventory Optimization

      Managing the inventory across multiple facilities arrayed in multiple echelons can be a huge challenge for any company. The complexity arises from the interactions among the echelons, with demands at the lower levels bubbling up and any shortages at the higher levels cascading down.

      Recent Posts

      • Smart Software CEO to present at Epicor Insights 2022Smart Software to Present at Epicor Insights 2022
        Smart Software CEO will present at this year's Epicor Insights event in Nashville. If you plan to attend this year, please join us at booth #9 and #705, and learn more about Epicor Smart Inventory Planning and Optimization. […]
      • Smart Software and Arizona Public Service to Present at WERC 2022
        Smart Software CEO and APS Inventory Manager to present WERC 2022 Studio Session on implementing Smart IP&O in 90 Days and achieve significant savings by optimizing reorder points and order quantities for over 250,000 spare parts. […]

        Forecast Using Leading Indicators – Regression Analysis:

        The Smart Forecaster

         Pursuing best practices in demand planning,

        forecasting and inventory optimization

        Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

        In this video tutorial Dr. Thomas Willemain, co–Founder and SVP Research at Smart Software, presents Regression Analysis, a specialized statistical modeling technique to identify and harness leading indicators to achieve more accurate forecasts.  Regression analysis is a statistical procedure to estimate the relationship between a response variable and one or more predictor variables. Housing starts, for example, might be a good leading indicator of vinyl siding demand.  Tom explains how and when to use regression analysis and works through a practical example.

        Forecasting Techniques for a more profitable business
        Leave a Comment

        RECENT POSTS

        Goldilocks Inventory Levels

        Goldilocks Inventory Levels

        You may remember the story of Goldilocks from your long-ago youth. Sometimes the porridge was too hot, sometimes it was too cold, but just once it was just right. Now that we are adults, we can translate that fairy tale into a professional principle for inventory planning: There can be too little or too much inventory, and there is some Goldilocks level that is “just right.” This blog is about finding that sweet spot.

        Call an Audible to Proactively Counter Supply Chain Noise

        Call an Audible to Proactively Counter Supply Chain Noise

        You know the situation: You work out the best way to manage each inventory item by computing the proper reorder points and replenishment targets, then average demand increases or decreases, or demand volatility changes, or suppliers’ lead times change, or your own costs change.

        An Example of Simulation-Based Multiechelon Inventory Optimization

        An Example of Simulation-Based Multiechelon Inventory Optimization

        Managing the inventory across multiple facilities arrayed in multiple echelons can be a huge challenge for any company. The complexity arises from the interactions among the echelons, with demands at the lower levels bubbling up and any shortages at the higher levels cascading down.

        Recent Posts

        • Smart Software CEO to present at Epicor Insights 2022Smart Software to Present at Epicor Insights 2022
          Smart Software CEO will present at this year's Epicor Insights event in Nashville. If you plan to attend this year, please join us at booth #9 and #705, and learn more about Epicor Smart Inventory Planning and Optimization. […]
        • Smart Software and Arizona Public Service to Present at WERC 2022
          Smart Software CEO and APS Inventory Manager to present WERC 2022 Studio Session on implementing Smart IP&O in 90 Days and achieve significant savings by optimizing reorder points and order quantities for over 250,000 spare parts. […]

          Forecasting Techniques for a more profitable business

          The Smart Forecaster

           Pursuing best practices in demand planning,

          forecasting and inventory optimization

          Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

           

          In this Video Dr. Thomas Willemain, co–Founder and SVP Research, defines and compares the most useful Forecasting Techniques: Exponential Smoothing, Single Exponential Smoothing, Holt’s Method and Winter’s Method.  These videos explain the basic thinking under each technique as well as the math behind them, how they are used in practice and the tradeoff of each method.

          Forecasting Techniques for a more profitable business
          Leave a Comment

          RECENT POSTS

          Goldilocks Inventory Levels

          Goldilocks Inventory Levels

          You may remember the story of Goldilocks from your long-ago youth. Sometimes the porridge was too hot, sometimes it was too cold, but just once it was just right. Now that we are adults, we can translate that fairy tale into a professional principle for inventory planning: There can be too little or too much inventory, and there is some Goldilocks level that is “just right.” This blog is about finding that sweet spot.

          Call an Audible to Proactively Counter Supply Chain Noise

          Call an Audible to Proactively Counter Supply Chain Noise

          You know the situation: You work out the best way to manage each inventory item by computing the proper reorder points and replenishment targets, then average demand increases or decreases, or demand volatility changes, or suppliers’ lead times change, or your own costs change.

          An Example of Simulation-Based Multiechelon Inventory Optimization

          An Example of Simulation-Based Multiechelon Inventory Optimization

          Managing the inventory across multiple facilities arrayed in multiple echelons can be a huge challenge for any company. The complexity arises from the interactions among the echelons, with demands at the lower levels bubbling up and any shortages at the higher levels cascading down.

          Recent Posts

          • Smart Software CEO to present at Epicor Insights 2022Smart Software to Present at Epicor Insights 2022
            Smart Software CEO will present at this year's Epicor Insights event in Nashville. If you plan to attend this year, please join us at booth #9 and #705, and learn more about Epicor Smart Inventory Planning and Optimization. […]
          • Smart Software and Arizona Public Service to Present at WERC 2022
            Smart Software CEO and APS Inventory Manager to present WERC 2022 Studio Session on implementing Smart IP&O in 90 Days and achieve significant savings by optimizing reorder points and order quantities for over 250,000 spare parts. […]