Four Useful Ways to Measure Forecast Error

The Smart Forecaster

 Pursuing best practices in demand planning,

forecasting and inventory optimization

Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving forecast accuracy by measuring forecast error. We begin by overviewing the various types of error metrics: scale-dependent error, percentage error, relative error, and scale-free error metrics. While some error is inevitable, there are ways to reduce it, and forecast metrics are necessary aids for monitoring and improving forecast accuracy. Then we will explain the special problem of intermittent demand and divide-by-zero problems. Tom concludes by explaining how to assess forecasts of multiple items and how it often makes sense to use weighted averages, weighting items differently by volume or revenue.

 

Four general types of error metrics 

1. Scale-dependent error
2. Percentage error
3. Relative error
4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

 

1. Scale-dependent error metrics

  • Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
  • Median Absolute Error (MdAE)
  • Root Mean Square Error (RMSE)
  • These metrics express the error in the original units of the data.
    • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
  • Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
    • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
  • To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

 

2. Percentage error metric

  • Mean Absolute Percentage Error (MAPE)
  • This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
  • The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
  • Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
  • The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

 

3. Relative error metric

  • Median Relative Absolute Error (MdRAE)
  • Relative to what? To a benchmark forecast.
  • What benchmark? Usually, the “naïve” forecast.
  • What is the naïve forecast? Next forecast value = last actual value.
  • Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

 

4. Scale-Free error metric

  • Median Relative Scaled Error (MdRSE)
  • This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
  • The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
    • (This is the same as the error made by the naïve forecast.)
  • How does this metric differ from the MdRAE above?
    • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
    • This matters because there are usually many more history values than there are forecasts.
    • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

 

Intermittent Demand Planning and Parts Forecasting

 

The special problem of intermittent demand

  • “Intermittent” demand has many zero demands mixed in with random non-zero demands.
  • MAPE gets ruined when errors are divided by zero.
  • MdRAE can also get ruined.
  • MdSAE is less likely to get ruined.

 

Recap and remarks

  • Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
  • There are two major classes of metrics: absolute and relative.
  • Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
  • Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
  • Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
  • When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.
Leave a Comment

RECENT POSTS

The Importance of Clear Service Level Definitions in Inventory Management

The Importance of Clear Service Level Definitions in Inventory Management

Inventory optimization software that supports what-if analysis will expose the tradeoff of stockouts vs. excess costs of varying service level targets. But first it is important to identify how “service levels” is interpreted, measured, and reported. This will avoid miscommunication and the false sense of security that can develop when less stringent definitions are used. Clearly defining how service level is calculated puts all stakeholders on the same page. This facilitates better decision-making.

The Cost of Spreadsheet Planning

The Cost of Spreadsheet Planning

Companies that depend on spreadsheets for demand planning, forecasting, and inventory management are often constrained by the spreadsheet’s inherent limitations. This post examines the drawbacks of traditional inventory management approaches caused by spreadsheets and their associated costs, contrasting these with the significant benefits gained from embracing state-of-the-art planning technologies.

Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

Leveraging Epicor Kinetic Planning BOMs with Smart IP&O to Forecast Accurately

In this blog, we explore how leveraging Epicor Kinetic Planning BOMs with Smart IP&O can transform your approach to forecasting in a highly configurable manufacturing environment. Discover how Smart, a cutting-edge AI-driven demand planning and inventory optimization solution, can simplify the complexities of predicting finished goods demand, especially when dealing with interchangeable components. Learn how Planning BOMs and advanced forecasting techniques enable businesses to anticipate customer needs more accurately, ensuring operational efficiency and staying ahead in a competitive market.

Recent Posts

  • Managing Spare Parts Inventory: Best PracticesManaging Spare Parts Inventory: Best Practices
    In this blog, we’ll explore several effective strategies for managing spare parts inventory, emphasizing the importance of optimizing stock levels, maintaining service levels, and using smart tools to aid in decision-making. Managing spare parts inventory is a critical component for businesses that depend on equipment uptime and service reliability. Unlike regular inventory items, spare parts often have unpredictable demand patterns, making them more challenging to manage effectively. An efficient spare parts inventory management system helps prevent stockouts that can lead to operational downtime and costly delays while also avoiding overstocking that unnecessarily ties up capital and increases holding costs. […]
  • 5 Ways to Improve Supply Chain Decision Speed5 Ways to Improve Supply Chain Decision Speed
    The promise of a digital supply chain has transformed how businesses operate. At its core, it can make rapid, data-driven decisions while ensuring quality and efficiency throughout operations. However, it's not just about having access to more data. Organizations need the right tools and platforms to turn that data into actionable insights. This is where decision-making becomes critical, especially in a landscape where new digital supply chain solutions and AI-driven platforms can support you in streamlining many processes within the decision matrix. […]
  • Two employees checking inventory in temporary storage in a distribution warehouse.12 Causes of Overstocking and Practical Solutions
    Managing inventory effectively is critical for maintaining a healthy balance sheet and ensuring that resources are optimally allocated. Here is an in-depth exploration of the main causes of overstocking, their implications, and possible solutions. […]
  • FAQ Mastering Smart IP&O for Better Inventory ManagementFAQ: Mastering Smart IP&O for Better Inventory Management.
    Effective supply chain and inventory management are essential for achieving operational efficiency and customer satisfaction. This blog provides clear and concise answers to some basic and other common questions from our Smart IP&O customers, offering practical insights to overcome typical challenges and enhance your inventory management practices. Focusing on these key areas, we help you transform complex inventory issues into strategic, manageable actions that reduce costs and improve overall performance with Smart IP&O. […]
  • 7 Key Demand Planning Trends Shaping the Future7 Key Demand Planning Trends Shaping the Future
    Demand planning goes beyond simply forecasting product needs; it's about ensuring your business meets customer demands with precision, efficiency, and cost-effectiveness. Latest demand planning technology addresses key challenges like forecast accuracy, inventory management, and market responsiveness. In this blog, we will introduce critical demand planning trends, including data-driven insights, probabilistic forecasting, consensus planning, predictive analytics, scenario modeling, real-time visibility, and multilevel forecasting. These trends will help you stay ahead of the curve, optimize your supply chain, reduce costs, and enhance customer satisfaction, positioning your business for long-term success. […]

    Inventory Optimization for Manufacturers, Distributors, and MRO

    • Managing Spare Parts Inventory: Best PracticesManaging Spare Parts Inventory: Best Practices
      In this blog, we’ll explore several effective strategies for managing spare parts inventory, emphasizing the importance of optimizing stock levels, maintaining service levels, and using smart tools to aid in decision-making. Managing spare parts inventory is a critical component for businesses that depend on equipment uptime and service reliability. Unlike regular inventory items, spare parts often have unpredictable demand patterns, making them more challenging to manage effectively. An efficient spare parts inventory management system helps prevent stockouts that can lead to operational downtime and costly delays while also avoiding overstocking that unnecessarily ties up capital and increases holding costs. […]
    • Innovating the OEM Aftermarket with AI-Driven Inventory Optimization XLInnovating the OEM Aftermarket with AI-Driven Inventory Optimization
      The aftermarket sector provides OEMs with a decisive advantage by offering a steady revenue stream and fostering customer loyalty through the reliable and timely delivery of service parts. However, managing inventory and forecasting demand in the aftermarket is fraught with challenges, including unpredictable demand patterns, vast product ranges, and the necessity for quick turnarounds. Traditional methods often fall short due to the complexity and variability of demand in the aftermarket. The latest technologies can analyze large datasets to predict future demand more accurately and optimize inventory levels, leading to better service and lower costs. […]
    • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
      Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
    • Centering Act Spare Parts Timing Pricing and ReliabilityCentering Act: Spare Parts Timing, Pricing, and Reliability
      In this article, we'll walk you through the process of crafting a spare parts inventory plan that prioritizes availability metrics such as service levels and fill rates while ensuring cost efficiency. We'll focus on an approach to inventory planning called Service Level-Driven Inventory Optimization. Next, we'll discuss how to determine what parts you should include in your inventory and those that might not be necessary. Lastly, we'll explore ways to enhance your service-level-driven inventory plan consistently. […]

      Leading Indicators can Foreshadow Demand

      The Smart Forecaster

      Pursuing best practices in demand planning,

      forecasting and inventory optimization

      Most statistical forecasting works in one direct flow from past data to forecast. Forecasting with leading indicators works a different way. A leading indicator is a second variable that may influence the one being forecasted. Applying testable human knowledge about the predictive power in the relationship between these different sets of data will sometimes provide superior accuracy.

      Most of the time, a forecast is based solely on the past history of the item being forecast. Let’s assume that the forecaster’s problem is to predict future unit sales of an important product. The process begins with gathering data on the product’s past sales. (Gregory Hartunian shares some practical advice on choosing the best available data in a previous post to the Smart Forecaster.) This data flows into forecasting software, which analyzes the sales record to measure the level of random variability and exploit any predictable aspects, such as trend or regular patterns of seasonal variability. The forecast is based entirely on the past behavior of the item being forecasted. Nothing that might have caused the wiggles and jiggles in the product’s sales graph is explicitly accounted for. This approach is fast, simple, self-contained and scalable, because software can zip through a huge number of forecasts automatically.

      But sometimes the forecaster can do better, at the cost of more work. If the forecaster can peer through the fog of randomness and identify a second variable that influences the one being forecasted, a leading indicator, more accurate predictions are possible.

      For example, suppose the product is window glass for houses. It may well be that increases or decreases in the number of construction permits for new houses will be reflected in corresponding increases or decreases in the number of sheets of glass ordered several months later. If the forecaster can distill this “lagged” or delayed relationship into an equation, that equation can be used to forecast glass sales several months hence using known values of the leading indicator. This equation is called a “regression equation” and has a form something like:

      Sales of glass in 3 months = 210.9 + 26.7 × Number of housing starts this month.

      Forecasting software can take the housing start and glass sales data and convert them into such a regression equation.

      Graph displaying a relationship between example figures for time-shifted building permits and demand for glass
      Leading indicators demonstrated
      However, unlike automatic statistical forecasting based on a product’s past sales, forecasting with a leading indicator faces the same problem as the proverbial recipe for rabbit stew: “First catch a rabbit”. Here the forecaster’s subject matter expertise is critical to success. The forecaster must be able to nominate one or more candidates for the job of leading indicator. After this crucial step, based on the forecaster’s knowledge, experience and intuition, then software can be used to verify that there really is a predictive, time-delayed relationship between the candidate leading indicator and the variable to be forecasted.

      This verification step is done using a “cross-correlation” analysis. The software essentially takes as input a sequence of values of the variable to be forecasted and another sequence of values of the supposed leading indicator. Then it slides the data from the forecast variable ahead by, successively, one, two, three, etc. time periods. At each slip in time (called a “lag”, because the leading indicator is lagging further and further behind the forecast variable), the software checks for a pattern of association between the two variables. If it finds a pattern that is too strong to be explained as a statistical accident, the forecaster’s hunch is confirmed.

      Obviously, forecasting with leading indicators is more work than forecasting using only an item’s own past values. The forecaster has to identify a leading indicator, starting with a list suggested by the forecaster’s subject matter expertise. This is a “hand-crafting” process that is not suited to mass production of forecasts. But it can be a successful approach for a smaller number of important items that are worth the extra effort. The role of forecasting software, such as our SmartForecasts system, is to help the forecaster authenticate the leading indicator and then exploit it.

      Thomas Willemain, PhD, co-founded Smart Software and currently serves as Senior Vice President for Research. Dr. Willemain also serves as Professor Emeritus of Industrial and Systems Engineering at Rensselaer Polytechnic Institute and as a member of the research staff at the Center for Computing Sciences, Institute for Defense Analyses.

      Leave a Comment

      Related Posts

      The Forecasting Process for Decision-Makers

      The Forecasting Process for Decision-Makers

      In almost every business and industry, decision-makers need reliable forecasts of critical variables, such as sales, revenues, product demand, inventory levels, market share, expenses, and industry trends.Many kinds of people make these forecasts. Some are sophisticated technical analysts such as business economists and statisticians. Many others regard forecasting as an important part of their overall work: general managers, production planners, inventory control specialists, financial analysts, strategic planners, market researchers, and product and sales managers. Still, others seldom think of themselves as forecasters but often have to make forecasts on an intuitive, judgmental basis.

      Leveraging ERP Planning BOMs with Smart IP&O to Forecast the Unforecastable

      Leveraging ERP Planning BOMs with Smart IP&O to Forecast the Unforecastable

      In a highly configurable manufacturing environment, forecasting finished goods can become a complex and daunting task. The number of possible finished products will skyrocket when many components are interchangeable. A traditional MRP would force us to forecast every single finished product which can be unrealistic or even impossible. Several leading ERP solutions introduce the concept of the “Planning BOM”, which allows the use of forecasts at a higher level in the manufacturing process. In this article, we will discuss this functionality in ERP, and how you can take advantage of it with Smart Inventory Planning and Optimization (Smart IP&O) to get ahead of your demand in the face of this complexity.

      The Forecast Matters, but Maybe Not the Way You Think

      The Forecast Matters, but Maybe Not the Way You Think

      True or false: The forecast doesn’t matter to spare parts inventory management. At first glance, this statement seems obviously false. After all, forecasts are crucial for planning stock levels, right? It depends on what you mean by a “forecast”. If you mean an old-school single-number forecast (“demand for item CX218b will be 3 units next week and 6 units the week after”), then no. If you broaden the meaning of forecast to include a probability distribution taking account of uncertainties in both demand and supply, then yes.

      Recent Posts

      • Managing Spare Parts Inventory: Best PracticesManaging Spare Parts Inventory: Best Practices
        In this blog, we’ll explore several effective strategies for managing spare parts inventory, emphasizing the importance of optimizing stock levels, maintaining service levels, and using smart tools to aid in decision-making. Managing spare parts inventory is a critical component for businesses that depend on equipment uptime and service reliability. Unlike regular inventory items, spare parts often have unpredictable demand patterns, making them more challenging to manage effectively. An efficient spare parts inventory management system helps prevent stockouts that can lead to operational downtime and costly delays while also avoiding overstocking that unnecessarily ties up capital and increases holding costs. […]
      • 5 Ways to Improve Supply Chain Decision Speed5 Ways to Improve Supply Chain Decision Speed
        The promise of a digital supply chain has transformed how businesses operate. At its core, it can make rapid, data-driven decisions while ensuring quality and efficiency throughout operations. However, it's not just about having access to more data. Organizations need the right tools and platforms to turn that data into actionable insights. This is where decision-making becomes critical, especially in a landscape where new digital supply chain solutions and AI-driven platforms can support you in streamlining many processes within the decision matrix. […]
      • Two employees checking inventory in temporary storage in a distribution warehouse.12 Causes of Overstocking and Practical Solutions
        Managing inventory effectively is critical for maintaining a healthy balance sheet and ensuring that resources are optimally allocated. Here is an in-depth exploration of the main causes of overstocking, their implications, and possible solutions. […]
      • FAQ Mastering Smart IP&O for Better Inventory ManagementFAQ: Mastering Smart IP&O for Better Inventory Management.
        Effective supply chain and inventory management are essential for achieving operational efficiency and customer satisfaction. This blog provides clear and concise answers to some basic and other common questions from our Smart IP&O customers, offering practical insights to overcome typical challenges and enhance your inventory management practices. Focusing on these key areas, we help you transform complex inventory issues into strategic, manageable actions that reduce costs and improve overall performance with Smart IP&O. […]
      • 7 Key Demand Planning Trends Shaping the Future7 Key Demand Planning Trends Shaping the Future
        Demand planning goes beyond simply forecasting product needs; it's about ensuring your business meets customer demands with precision, efficiency, and cost-effectiveness. Latest demand planning technology addresses key challenges like forecast accuracy, inventory management, and market responsiveness. In this blog, we will introduce critical demand planning trends, including data-driven insights, probabilistic forecasting, consensus planning, predictive analytics, scenario modeling, real-time visibility, and multilevel forecasting. These trends will help you stay ahead of the curve, optimize your supply chain, reduce costs, and enhance customer satisfaction, positioning your business for long-term success. […]

        Inventory Optimization for Manufacturers, Distributors, and MRO

        • Managing Spare Parts Inventory: Best PracticesManaging Spare Parts Inventory: Best Practices
          In this blog, we’ll explore several effective strategies for managing spare parts inventory, emphasizing the importance of optimizing stock levels, maintaining service levels, and using smart tools to aid in decision-making. Managing spare parts inventory is a critical component for businesses that depend on equipment uptime and service reliability. Unlike regular inventory items, spare parts often have unpredictable demand patterns, making them more challenging to manage effectively. An efficient spare parts inventory management system helps prevent stockouts that can lead to operational downtime and costly delays while also avoiding overstocking that unnecessarily ties up capital and increases holding costs. […]
        • Innovating the OEM Aftermarket with AI-Driven Inventory Optimization XLInnovating the OEM Aftermarket with AI-Driven Inventory Optimization
          The aftermarket sector provides OEMs with a decisive advantage by offering a steady revenue stream and fostering customer loyalty through the reliable and timely delivery of service parts. However, managing inventory and forecasting demand in the aftermarket is fraught with challenges, including unpredictable demand patterns, vast product ranges, and the necessity for quick turnarounds. Traditional methods often fall short due to the complexity and variability of demand in the aftermarket. The latest technologies can analyze large datasets to predict future demand more accurately and optimize inventory levels, leading to better service and lower costs. […]
        • Future-Proofing Utilities. Advanced Analytics for Supply Chain OptimizationFuture-Proofing Utilities: Advanced Analytics for Supply Chain Optimization
          Utilities in the electrical, natural gas, urban water, and telecommunications fields are all asset-intensive and reliant on physical infrastructure that must be properly maintained, updated, and upgraded over time. Maximizing asset uptime and the reliability of physical infrastructure demands effective inventory management, spare parts forecasting, and supplier management. A utility that executes these processes effectively will outperform its peers, provide better returns for its investors and higher service levels for its customers, while reducing its environmental impact. […]
        • Centering Act Spare Parts Timing Pricing and ReliabilityCentering Act: Spare Parts Timing, Pricing, and Reliability
          In this article, we'll walk you through the process of crafting a spare parts inventory plan that prioritizes availability metrics such as service levels and fill rates while ensuring cost efficiency. We'll focus on an approach to inventory planning called Service Level-Driven Inventory Optimization. Next, we'll discuss how to determine what parts you should include in your inventory and those that might not be necessary. Lastly, we'll explore ways to enhance your service-level-driven inventory plan consistently. […]

          Forecasting With the Right Data

          The Smart Forecaster

          Pursuing best practices in demand planning,

          forecasting and inventory optimization

          In order to reap the efficiency benefits of forecasting, you need the most accurate forecasts—forecasts built on the most appropriate historical data. Most discussions of this issue tend to focus on the merits of using demand vs. shipment history—and I’ll comment on this later. But first, let’s talk about the use of net vs. gross data.

          Net vs. Gross History

          Many planners are inclined to use net sales data to create their forecasts. Systems that track sales capture transactions as they occur and aggregate results into weekly or monthly periodic totals. In some cases, sales records account for returned purchases as negative sales and compute a net total. These net figures, which often mask real sales patterns, are fed into the forecasting system. The historical data used actually presents a false sense of what the customer wanted, and when they wanted it. This will carry forward into the forecast, with less than optimal results.

          (more…)