Rethinking forecast accuracy: A shift from accuracy to error metrics

Measuring the accuracy of forecasts is an undeniably important part of the demand planning process. This forecasting scorecard could be built based on one of two contrasting viewpoints for computing metrics. The error viewpoint asks, “how far was the forecast from the actual?” The accuracy viewpoint asks, “how close was the forecast to the actual?” Both are valid, but error metrics provide more information.

Accuracy is represented as a percentage between zero and 100, while error percentages start at zero but have no upper limit. Reports of MAPE (mean absolute percent error) or other error metrics can be titled “forecast accuracy” reports, which blurs the distinction.  So, you may want to know how to convert from the error viewpoint to the accuracy viewpoint that your company espouses.  This blog describes how with some examples.

Accuracy metrics are computed such that when the actual equals the forecast then the accuracy is 100% and when the forecast is either double or half of the actual, then accuracy is 0%. Reports that compare the forecast to the actual often include the following:

  • The Actual
  • The Forecast
  • Unit Error = Forecast – Actual
  • Absolute Error = Absolute Value of Unit Error
  • Absolute % Error = Abs Error / Actual, as a %
  • Accuracy % = 100% – Absolute % Error

Look at a couple examples that illustrate the difference in the approaches. Say the Actual = 8 and the forecast is 10.

Unit Error is 10 – 8 = 2

Absolute % Error = 2 / 8, as a % = 0.25 * 100 = 25%

Accuracy = 100% – 25% = 75%.

Now let’s say the actual is 8 and the forecast is 24.

Unit Error is 24– 8 = 16

Absolute % Error = 16 / 8 as a % = 2 * 100 = 200%

Accuracy = 100% – 200% = negative is set to 0%.

In the first example, accuracy measurements provide the same information as error measurements since the forecast and actual are already relatively close. But when the error is more than double the actual, accuracy measurements bottom out at zero. It does correctly indicate the forecast was not at all accurate. But the second example is more accurate than a third, where the actual is 8 and the forecast is 200. That’s a distinction a 0 to 100% range of accuracy doesn’t register. In this final example:

Unit Error is 200 – 8 = 192

Absolute % Error = 192 / 8, as a % = 24 * 100 = 2,400%

Accuracy = 100% – 2,400% = negative is set to 0%.

Error metrics continue to provide information on how far the forecast is from the actual and arguably better represent forecast accuracy.

We encourage adopting the error viewpoint. You simply hope for a small error percentage to indicate the forecast was not far from the actual, instead of hoping for a large accuracy percentage to indicate the forecast was close to the actual.  This shift in mindset offers the same insights while eliminating distortions.

 

 

 

 

Improve Forecast Accuracy by Managing Error

The Smart Forecaster

 Pursuing best practices in demand planning,

forecasting and inventory optimization

Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving Forecast Accuracy by Managing Error. This video is the first in our series on effective methods to Improve Forecast Accuracy.  We begin by looking at how forecast error causes pain and the consequential cost related to it. Then we will explain the three most common mistakes to avoid that can help us increase revenue and prevent excess inventory. Tom concludes by reviewing the methods to improve Forecast Accuracy, the importance of measuring forecast error, and the technological opportunities to improve it.

 

Forecast error can be consequential

Consider one item of many

  • Product X costs $100 to make and nets $50 profit per unit.
  • Sales of Product X will turn out to be 1,000/month over the next 12 months.
  • Consider one item of many

What is the cost of forecast error?

  • If the forecast is 10% high, end the year with $120,000 of excess inventory.
  • 100 extra/month x 12 months x $100/unit
  • If the forecast is 10% low, miss out on $60,000 of profit.
  • 100 too few/month x 12 months x $50/unit

 

Three mistakes to avoid

1. Ignoring error.

  • Unprofessional, dereliction of duty.
  • Wishing will not make it so.
  • Treat accuracy assessment as data science, not a blame game.

2. Tolerating more error than necessary.

  • Statistical forecasting methods can improve accuracy at scale.
  • Improving data inputs can help.
  • Collecting and analyzing forecast error metrics can identify weak spots.

3. Wasting time and money going too far trying to eliminate error.

  • Some product/market combinations are inherently more difficult to forecast. After a point, let them be (but be alert for new specialized forecasting methods).
  • Sometimes steps meant to reduce error can backfire (e.g., adjustment).
Leave a Comment

RECENT POSTS

The Importance of Clear Service Level Definitions in Inventory Management

The Importance of Clear Service Level Definitions in Inventory Management

Inventory optimization software that supports what-if analysis will expose the tradeoff of stockouts vs. excess costs of varying service level targets. But first it is important to identify how “service levels” is interpreted, measured, and reported. This will avoid miscommunication and the false sense of security that can develop when less stringent definitions are used. Clearly defining how service level is calculated puts all stakeholders on the same page. This facilitates better decision-making.

The Cost of Spreadsheet Planning

The Cost of Spreadsheet Planning

Companies that depend on spreadsheets for demand planning, forecasting, and inventory management are often constrained by the spreadsheet’s inherent limitations. This post examines the drawbacks of traditional inventory management approaches caused by spreadsheets and their associated costs, contrasting these with the significant benefits gained from embracing state-of-the-art planning technologies.

Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

In this blog, we explore how leveraging Epicor Kinetic Planning BOMs with Smart IP&O can transform your approach to forecasting in a highly configurable manufacturing environment. Discover how Smart, a cutting-edge AI-driven demand planning and inventory optimization solution, can simplify the complexities of predicting finished goods demand, especially when dealing with interchangeable components. Learn how Planning BOMs and advanced forecasting techniques enable businesses to anticipate customer needs more accurately, ensuring operational efficiency and staying ahead in a competitive market.

Recent Posts

  • The Importance of Clear Service Level Definitions in Inventory ManagementThe Importance of Clear Service Level Definitions in Inventory Management
    Inventory optimization software that supports what-if analysis will expose the tradeoff of stockouts vs. excess costs of varying service level targets. But first it is important to identify how “service levels” is interpreted, measured, and reported. This will avoid miscommunication and the false sense of security that can develop when less stringent definitions are used. Clearly defining how service level is calculated puts all stakeholders on the same page. This facilitates better decision-making. […]
  • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
    Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
  • The Cost of Doing nothing with your inventory Planning SystemsThe Cost of Spreadsheet Planning
    Companies that depend on spreadsheets for demand planning, forecasting, and inventory management are often constrained by the spreadsheet’s inherent limitations. This post examines the drawbacks of traditional inventory management approaches caused by spreadsheets and their associated costs, contrasting these with the significant benefits gained from embracing state-of-the-art planning technologies. […]
  • Simple Inventory Optimization is Good Except When It Isn’t FHDSimple is Good, Except When It Isn’t
    In this blog, we are steering the conversation towards the transformative potential of technology in inventory management. The discussion centers around the limitations of simple thinking in managing inventory control processes and the necessity of adopting systematic software solutions. […]
  • Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately HDLeveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately
    In this blog, we explore how leveraging Epicor Kinetic Planning BOMs with Smart IP&O can transform your approach to forecasting in a highly configurable manufacturing environment. Discover how Smart, a cutting-edge AI-driven demand planning and inventory optimization solution, can simplify the complexities of predicting finished goods demand, especially when dealing with interchangeable components. Learn how Planning BOMs and advanced forecasting techniques enable businesses to anticipate customer needs more accurately, ensuring operational efficiency and staying ahead in a competitive market. […]

    Inventory Optimization for Manufacturers, Distributors, and MRO

    • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
      Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
    • Centering Act Spare Parts Timing Pricing and ReliabilityCentering Act: Spare Parts Timing, Pricing, and Reliability
      In this article, we'll walk you through the process of crafting a spare parts inventory plan that prioritizes availability metrics such as service levels and fill rates while ensuring cost efficiency. We'll focus on an approach to inventory planning called Service Level-Driven Inventory Optimization. Next, we'll discuss how to determine what parts you should include in your inventory and those that might not be necessary. Lastly, we'll explore ways to enhance your service-level-driven inventory plan consistently. […]
    • Why MRO Businesses Need Add-on Service Parts Planning & Inventory SoftwareWhy MRO Businesses Need Add-on Service Parts Planning & Inventory Software
      MRO organizations exist in a wide range of industries, including public transit, electrical utilities, wastewater, hydro power, aviation, and mining. To get their work done, MRO professionals use Enterprise Asset Management (EAM) and Enterprise Resource Planning (ERP) systems. These systems are designed to do a lot of jobs. Given their features, cost, and extensive implementation requirements, there is an assumption that EAM and ERP systems can do it all. In this post, we summarize the need for add-on software that addresses specialized analytics for inventory optimization, forecasting, and service parts planning. […]
    • Spare-parts-demand-forecasting-a-different-perspective-for-planning-service-partsThe Forecast Matters, but Maybe Not the Way You Think
      True or false: The forecast doesn't matter to spare parts inventory management. At first glance, this statement seems obviously false. After all, forecasts are crucial for planning stock levels, right? It depends on what you mean by a “forecast”. If you mean an old-school single-number forecast (“demand for item CX218b will be 3 units next week and 6 units the week after”), then no. If you broaden the meaning of forecast to include a probability distribution taking account of uncertainties in both demand and supply, then yes. […]

      Four Useful Ways to Measure Forecast Error

      The Smart Forecaster

       Pursuing best practices in demand planning,

      forecasting and inventory optimization

      Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

      In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving forecast accuracy by measuring forecast error. We begin by overviewing the various types of error metrics: scale-dependent error, percentage error, relative error, and scale-free error metrics. While some error is inevitable, there are ways to reduce it, and forecast metrics are necessary aids for monitoring and improving forecast accuracy. Then we will explain the special problem of intermittent demand and divide-by-zero problems. Tom concludes by explaining how to assess forecasts of multiple items and how it often makes sense to use weighted averages, weighting items differently by volume or revenue.

       

      Four general types of error metrics 

      1. Scale-dependent error
      2. Percentage error
      3. Relative error
      4 .Scale-free error

      Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

       

      1. Scale-dependent error metrics

      • Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
      • Median Absolute Error (MdAE)
      • Root Mean Square Error (RMSE)
      • These metrics express the error in the original units of the data.
        • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
      • Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
        • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
      • To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

       

      2. Percentage error metric

      • Mean Absolute Percentage Error (MAPE)
      • This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
      • The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
      • Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
      • The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

       

      3. Relative error metric

      • Median Relative Absolute Error (MdRAE)
      • Relative to what? To a benchmark forecast.
      • What benchmark? Usually, the “naïve” forecast.
      • What is the naïve forecast? Next forecast value = last actual value.
      • Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

       

      4. Scale-Free error metric

      • Median Relative Scaled Error (MdRSE)
      • This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
      • The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
        • (This is the same as the error made by the naïve forecast.)
      • How does this metric differ from the MdRAE above?
        • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
        • This matters because there are usually many more history values than there are forecasts.
        • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

       

      Intermittent Demand Planning and Parts Forecasting

       

      The special problem of intermittent demand

      • “Intermittent” demand has many zero demands mixed in with random non-zero demands.
      • MAPE gets ruined when errors are divided by zero.
      • MdRAE can also get ruined.
      • MdSAE is less likely to get ruined.

       

      Recap and remarks

      • Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
      • There are two major classes of metrics: absolute and relative.
      • Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
      • Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
      • Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
      • When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.
      Leave a Comment

      RECENT POSTS

      The Importance of Clear Service Level Definitions in Inventory Management

      The Importance of Clear Service Level Definitions in Inventory Management

      Inventory optimization software that supports what-if analysis will expose the tradeoff of stockouts vs. excess costs of varying service level targets. But first it is important to identify how “service levels” is interpreted, measured, and reported. This will avoid miscommunication and the false sense of security that can develop when less stringent definitions are used. Clearly defining how service level is calculated puts all stakeholders on the same page. This facilitates better decision-making.

      The Cost of Spreadsheet Planning

      The Cost of Spreadsheet Planning

      Companies that depend on spreadsheets for demand planning, forecasting, and inventory management are often constrained by the spreadsheet’s inherent limitations. This post examines the drawbacks of traditional inventory management approaches caused by spreadsheets and their associated costs, contrasting these with the significant benefits gained from embracing state-of-the-art planning technologies.

      Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

      Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

      In this blog, we explore how leveraging Epicor Kinetic Planning BOMs with Smart IP&O can transform your approach to forecasting in a highly configurable manufacturing environment. Discover how Smart, a cutting-edge AI-driven demand planning and inventory optimization solution, can simplify the complexities of predicting finished goods demand, especially when dealing with interchangeable components. Learn how Planning BOMs and advanced forecasting techniques enable businesses to anticipate customer needs more accurately, ensuring operational efficiency and staying ahead in a competitive market.

      Recent Posts

      • The Importance of Clear Service Level Definitions in Inventory ManagementThe Importance of Clear Service Level Definitions in Inventory Management
        Inventory optimization software that supports what-if analysis will expose the tradeoff of stockouts vs. excess costs of varying service level targets. But first it is important to identify how “service levels” is interpreted, measured, and reported. This will avoid miscommunication and the false sense of security that can develop when less stringent definitions are used. Clearly defining how service level is calculated puts all stakeholders on the same page. This facilitates better decision-making. […]
      • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
        Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
      • The Cost of Doing nothing with your inventory Planning SystemsThe Cost of Spreadsheet Planning
        Companies that depend on spreadsheets for demand planning, forecasting, and inventory management are often constrained by the spreadsheet’s inherent limitations. This post examines the drawbacks of traditional inventory management approaches caused by spreadsheets and their associated costs, contrasting these with the significant benefits gained from embracing state-of-the-art planning technologies. […]
      • Simple Inventory Optimization is Good Except When It Isn’t FHDSimple is Good, Except When It Isn’t
        In this blog, we are steering the conversation towards the transformative potential of technology in inventory management. The discussion centers around the limitations of simple thinking in managing inventory control processes and the necessity of adopting systematic software solutions. […]
      • Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately HDLeveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately
        In this blog, we explore how leveraging Epicor Kinetic Planning BOMs with Smart IP&O can transform your approach to forecasting in a highly configurable manufacturing environment. Discover how Smart, a cutting-edge AI-driven demand planning and inventory optimization solution, can simplify the complexities of predicting finished goods demand, especially when dealing with interchangeable components. Learn how Planning BOMs and advanced forecasting techniques enable businesses to anticipate customer needs more accurately, ensuring operational efficiency and staying ahead in a competitive market. […]

        Inventory Optimization for Manufacturers, Distributors, and MRO

        • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
          Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
        • Centering Act Spare Parts Timing Pricing and ReliabilityCentering Act: Spare Parts Timing, Pricing, and Reliability
          In this article, we'll walk you through the process of crafting a spare parts inventory plan that prioritizes availability metrics such as service levels and fill rates while ensuring cost efficiency. We'll focus on an approach to inventory planning called Service Level-Driven Inventory Optimization. Next, we'll discuss how to determine what parts you should include in your inventory and those that might not be necessary. Lastly, we'll explore ways to enhance your service-level-driven inventory plan consistently. […]
        • Why MRO Businesses Need Add-on Service Parts Planning & Inventory SoftwareWhy MRO Businesses Need Add-on Service Parts Planning & Inventory Software
          MRO organizations exist in a wide range of industries, including public transit, electrical utilities, wastewater, hydro power, aviation, and mining. To get their work done, MRO professionals use Enterprise Asset Management (EAM) and Enterprise Resource Planning (ERP) systems. These systems are designed to do a lot of jobs. Given their features, cost, and extensive implementation requirements, there is an assumption that EAM and ERP systems can do it all. In this post, we summarize the need for add-on software that addresses specialized analytics for inventory optimization, forecasting, and service parts planning. […]
        • Spare-parts-demand-forecasting-a-different-perspective-for-planning-service-partsThe Forecast Matters, but Maybe Not the Way You Think
          True or false: The forecast doesn't matter to spare parts inventory management. At first glance, this statement seems obviously false. After all, forecasts are crucial for planning stock levels, right? It depends on what you mean by a “forecast”. If you mean an old-school single-number forecast (“demand for item CX218b will be 3 units next week and 6 units the week after”), then no. If you broaden the meaning of forecast to include a probability distribution taking account of uncertainties in both demand and supply, then yes. […]