The Smart Forecaster

 Pursuing best practices in demand planning,

forecasting and inventory optimization

Improve Forecast Accuracy, Eliminate Excess Inventory, & Maximize Service Levels

In this video, Dr. Thomas Willemain, co-Founder and SVP Research, talks about improving forecast accuracy by measuring forecast error. We begin by overviewing the various types of error metrics: scale-dependent error, percentage error, relative error, and scale-free error metrics. While some error is inevitable, there are ways to reduce it, and forecast metrics are necessary aids for monitoring and improving forecast accuracy. Then we will explain the special problem of intermittent demand and divide-by-zero problems. Tom concludes by explaining how to assess forecasts of multiple items and how it often makes sense to use weighted averages, weighting items differently by volume or revenue.

 

Four general types of error metrics 

1. Scale-dependent error
2. Percentage error
3. Relative error
4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

 

1. Scale-dependent error metrics

  • Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
  • Median Absolute Error (MdAE)
  • Root Mean Square Error (RMSE)
  • These metrics express the error in the original units of the data.
    • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
  • Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
    • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
  • To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

 

2. Percentage error metric

  • Mean Absolute Percentage Error (MAPE)
  • This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
  • The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
  • Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
  • The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

 

3. Relative error metric

  • Median Relative Absolute Error (MdRAE)
  • Relative to what? To a benchmark forecast.
  • What benchmark? Usually, the “naïve” forecast.
  • What is the naïve forecast? Next forecast value = last actual value.
  • Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

 

4. Scale-Free error metric

  • Median Relative Scaled Error (MdRSE)
  • This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
  • The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
    • (This is the same as the error made by the naïve forecast.)
  • How does this metric differ from the MdRAE above?
    • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
    • This matters because there are usually many more history values than there are forecasts.
    • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

 

Intermittent Demand Planning and Parts Forecasting

 

The special problem of intermittent demand

  • “Intermittent” demand has many zero demands mixed in with random non-zero demands.
  • MAPE gets ruined when errors are divided by zero.
  • MdRAE can also get ruined.
  • MdSAE is less likely to get ruined.

 

Recap and remarks

  • Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
  • There are two major classes of metrics: absolute and relative.
  • Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
  • Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
  • Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
  • When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.
Leave a Comment

RECENT POSTS

Finding Your Spot on the Inventory Tradeoff Curve

Finding Your Spot on the Inventory Tradeoff Curve

This video blog holds essential insights for those working with the complexities of inventory management. The session focuses on striking the right balance within the inventory tradeoff curve, inviting viewers to understand the deep-seated importance of this equilibrium.

Why MRO Businesses Need Add-on Service Parts Planning & Inventory Software

Why MRO Businesses Need Add-on Service Parts Planning & Inventory Software

MRO organizations exist in a wide range of industries, including public transit, electrical utilities, wastewater, hydro power, aviation, and mining. To get their work done, MRO professionals use Enterprise Asset Management (EAM) and Enterprise Resource Planning (ERP) systems. These systems are designed to do a lot of jobs. Given their features, cost, and extensive implementation requirements, there is an assumption that EAM and ERP systems can do it all. In this post, we summarize the need for add-on software that addresses specialized analytics for inventory optimization, forecasting, and service parts planning.

Head to Head: Which Service Parts Inventory Policy is Best?

Head to Head: Which Service Parts Inventory Policy is Best?

Our customers have usually settled into one way to manage their service parts inventory. The professor in me would like to think that the chosen inventory policy was a reasoned choice among considered alternatives, but more likely it just sort of happened. Maybe the inventory honcho from long ago had a favorite and that choice stuck. Maybe somebody used an EAM or ERP system that offered only one choice. Perhaps there were some guesses made, based on the conditions at the time.

Recent Posts

  • Finding Your Spot on the Inventory Tradeoff CurveFinding Your Spot on the Inventory Tradeoff Curve
    This video blog holds essential insights for those working with the complexities of inventory management. The session focuses on striking the right balance within the inventory tradeoff curve, inviting viewers to understand the deep-seated importance of this equilibrium. […]
  • Why MRO Businesses Need Add-on Service Parts Planning & Inventory SoftwareWhy MRO Businesses Need Add-on Service Parts Planning & Inventory Software
    MRO organizations exist in a wide range of industries, including public transit, electrical utilities, wastewater, hydro power, aviation, and mining. To get their work done, MRO professionals use Enterprise Asset Management (EAM) and Enterprise Resource Planning (ERP) systems. These systems are designed to do a lot of jobs. Given their features, cost, and extensive implementation requirements, there is an assumption that EAM and ERP systems can do it all. In this post, we summarize the need for add-on software that addresses specialized analytics for inventory optimization, forecasting, and service parts planning. […]
  • Descriptive, Predictive, and Prescriptive AnalyticsThe Three Types of Supply Chain Analytics
    In this video blog, we explore the critical roles of Descriptive, Predictive, and Prescriptive Analytics in inventory management, highlighting their essential contributions to driving supply chain optimization through strategic foresight and insightful data analysis. […]
  • Warning Signs that You Have a Supply Chain Analytics GapWarning Signs that You Have a Supply Chain Analytics Gap
    “Business is war” may be an overdone metaphor but it’s not without validity. Like the “Bomber Gap” and the “Missile Gap,” worries about falling behind the competition, and the resulting threat of annihilation, always lurk in the minds of business executives, If they don’t, they should, because not all gaps are imaginary (the Bomber Gap and the Missile Gap were shown to not exist between the US and the USSR, but the 1980’s gap between Japanese and American productivity was all too real). The difference between paranoia and justified concern is converting fear into facts. This post is about organizing your attention toward possible gaps in your company’s supply chain analytics. […]
  • The Objectives in ForecastingThe Objectives in Forecasting
    A forecast is a prediction about the value of a time series variable at some time in the future. For instance, one might want to estimate next month’s sales or demand for a product item. A time series is a sequence of numbers recorded at equally spaced time intervals; for example, unit sales recorded every month. The objectives you pursue when you forecast depend on the nature of your job and your business. Every forecast is uncertain; in fact, there is a range of possible values for any variable you forecast. Values near the middle of this range have a higher likelihood of actually occurring, while values at the extremes of the range are less likely to occur. […]

    Inventory Optimization for Manufacturers, Distributors, and MRO

    • Why MRO Businesses Need Add-on Service Parts Planning & Inventory SoftwareWhy MRO Businesses Need Add-on Service Parts Planning & Inventory Software
      MRO organizations exist in a wide range of industries, including public transit, electrical utilities, wastewater, hydro power, aviation, and mining. To get their work done, MRO professionals use Enterprise Asset Management (EAM) and Enterprise Resource Planning (ERP) systems. These systems are designed to do a lot of jobs. Given their features, cost, and extensive implementation requirements, there is an assumption that EAM and ERP systems can do it all. In this post, we summarize the need for add-on software that addresses specialized analytics for inventory optimization, forecasting, and service parts planning. […]
    • Spare-parts-demand-forecasting-a-different-perspective-for-planning-service-partsThe Forecast Matters, but Maybe Not the Way You Think
      True or false: The forecast doesn't matter to spare parts inventory management. At first glance, this statement seems obviously false. After all, forecasts are crucial for planning stock levels, right? It depends on what you mean by a “forecast”. If you mean an old-school single-number forecast (“demand for item CX218b will be 3 units next week and 6 units the week after”), then no. If you broaden the meaning of forecast to include a probability distribution taking account of uncertainties in both demand and supply, then yes. […]
    • Whyt MRO Businesses Should Care about Excess InventoryWhy MRO Businesses Should Care About Excess Inventory
      Do MRO companies genuinely prioritize reducing excess spare parts inventory? From an organizational standpoint, our experience suggests not necessarily. Boardroom discussions typically revolve around expanding fleets, acquiring new customers, meeting service level agreements (SLAs), modernizing infrastructure, and maximizing uptime. In industries where assets supported by spare parts cost hundreds of millions or generate significant revenue (e.g., mining or oil & gas), the value of the inventory just doesn’t raise any eyebrows, and organizations tend to overlook massive amounts of excessive inventory. […]
    • Top Differences between Inventory Planning for Finished Goods and for MRO and Spare PartsTop Differences Between Inventory Planning for Finished Goods and for MRO and Spare Parts
      In today’s competitive business landscape, companies are constantly seeking ways to improve their operational efficiency and drive increased revenue. Optimizing service parts management is an often-overlooked aspect that can have a significant financial impact. Companies can improve overall efficiency and generate significant financial returns by effectively managing spare parts inventory. This article will explore the economic implications of optimized service parts management and how investing in Inventory Optimization and Demand Planning Software can provide a competitive advantage. […]