The Right Forecast Accuracy Metric for Inventory Planning

The Smart Forecaster

Pursuing best practices in demand planning, forecasting and inventory optimization

In a recent Supply Chain Shaman.com blog Trust, But Verify, Lora Cecere advocated for testing software solutions via a series of empirical competitions.

For forecasting/demand planning, she outlines a traditional “hold out” test in which 2012-2016 data are provided to software vendors and 2017 is held out for later comparison against forecasts provided by competing vendors. The company then measures forecast error and bias. This approach is advocated nearly universally for assessing forecast accuracy. It’s a good way to assess monthly or weekly forecast accuracy, but it is minimally useful if you have a different objective: Optimizing inventory.

In our last blog, we discussed how to pick a targeted service level. We indicated that just because you set a target (or a system recommends a target) doesn’t mean you’ll actually achieve the target. The right way to measure accuracy if you are interested in optimizing stock levels is to focus on the accuracy of the service level projection. This will account for both lead time demand and safety stock.

Setting a target service level is a strategic decision about inventory risk management. Inventory software does the tactical work by computing reorder points (a.k.a. mins) meant to achieve a user-defined target or that will achieve a system-calculated optimal target. But if the software uses the wrong demand model, the achieved service level will miss the target, sometimes significantly. The result of this error will be either shortages or inventory bloat, depending on the direction of the miss.

Forecasting is a means to an end. The end is to optimize inventory levels. Because demand is uncertain, companies that need to provide even moderate service levels must stock more than the forecast, often much more. But doesn’t low forecast error mean lower safety stock? The better my forecasts, the lower my inventory? Yes, true. But what matters when determining the required inventory are both accurate forecasts of the most likely demand and accurate estimates of the variability around the most likely demand.
Especially with with long tail, intermittent demand, traditional forecast accuracy assessments over a conventional 12 month forecast horizon miss the point three ways.

– First, the relevant time scale for inventory optimization is the replenishment lead time, which is usually much shorter than 12 months. Demand during lead times measured in days or weeks has volatility that gets averaged out over long forecast horizons. This is bad because factoring in the effect of volatility is essential to calculation of optimal reorder points.

– Second, forecast accuracy assessed over a multi-month forecast horizon focuses on the typical error in a typical month within the horizon. In contrast, inventory optimization requires a focus on cumulative demand, not period-by-period demand.

– Third, and most important is that forecast error metrics are focused on the middle of the demand distribution, aiming to estimate the most likely demand. But setting reorder points involves estimating high percentiles of the cumulative demand distribution over a lead time. Estimating the middle a bit better but having no clue about, say, the 95th percentile, is not helpful.

Consider this hypothetical example. If Vendor A forecasts 20 units with 110% error and Vendor B forecasts 22 units with 105% error, then Vendor B has an advantage in the forecasting game. But if you want a high service level and the demand is intermittent, you’ll have to stock a lot more than 20 or 22 units. Let’s assume you select Vendor B’s technology to plan stocking levels. You then notice that when planning reorder points to achieve a 95% service level, you often fall short – way more often that the expected 5% of the time. You come to realize that Vendor B’s approach completely underestimates the safety stock required to achieve the target service target. Focusing on vendors’ forecast error isn’t going to help. You will come to wish that you had verified Vendor A and B’s service level accuracy. Now you are stuck arbitrarily adjusting Vendor B’s service level targets to compensate for the shortfall.

So what’s needed in vendor competitions is assessment of their systems’ abilities to accurately forecast the inventory required to meet a given service level over an item’s replenishment lead time. Narrowly focusing on measuring forecast error is not appropriate if the mission is managing inventory. This is especially true for long tail items with intermittent demand or items that have medium to high volume but don’t have a demand distribution that looks like the classic “bell shaped curve” (Normal distribution).

The remainder of this blog explains how to test the accuracy of software’s service level calculations, so you can monitor the risk of missing your service level targets. We recommend this accuracy test over traditional “forecast versus actuals” tests because it provides much more insight into how reorder point recommendations will influence inventory levels and customer service.

Service Level Defined

Consider a single inventory item. When inventory drops to or below the reorder point, a replenishment order is generated. This starts a period of risk that lasts as long as the replenishment lead time. During the period of risk, there might be enough incoming demands to create backorders or lost sales. The service level is the probability that there are no backorders or stockouts during the replenishment lead time. Critical items might be given very high target service levels, say 99%, whereas other items might have more relaxed targets, such as 75%. Whatever the target service level, it is best to hit that target.

Calculating Service Level

The service level for an individual item can only be estimated by repeated comparison of observed lead time demand against the calculated reorder point. These estimates take a lot of time: at least dozens of lead times. But fleet-wise service level can be estimated using data compiled over a single lead time.

Let’s do an example. Suppose you have demand histories for 1,000 items over 365 days and that (for simplicity) all items have 45-day lead times. For each item, follow these steps to estimate the fleet-wise achieved service level:

Step 1: Step aside (“hold out”) the most recent 45 days of demand (or however many days is closest to your typical lead times). Compute their sum, which is the most recent value of the actual lead time demand. This is the ground truth to be used to estimate the achieved service level.

Step 2: Use the prior 320 days of demand history to forecast the required inventory to hit a range of service level targets, say 90%, 95%, 97%, and 99%.

Step 3: Check whether the observed lead time demand is less than or equal to the reorder point. If it is, count this as a win; otherwise, count it as a loss. For instance, if the reorder point is 15 units but the most recent lead time demand is 10 units, then this is a win, since the reorder point is high enough to cover a lead time demand of 10 without any shortage. However, if the most recent lead time demand is 18 units, there would be a stockout, and 3 units would either be backordered or counted as lost sales.

Step 4: Working across all items, and all service level targets, tally the percentage of tests for each service level target that resulted in a win. This is the achieved service level. If the target was 90% and 853 of the 1,000 units record a win, then the achieved service level is 85.3%.

Example

Consider a real-world example. The data are daily demand histories of 590 medical supply items used in an internationally famous clinic. For simplicity, we assume each item has a lead time of 45 days. We evaluate target service levels of 70%, 90%, 95% and 99%.
We compare two demand models. The “Normal” model assumes that daily demand has a Normal (“bell-shaped”) distribution. This is the classic assumption used in most introductory textbooks on inventory control and in many software products. Classic though it may be, it is often an inappropriate model of demand for spare parts or supplies. The “Probability Forecast” model takes explicit account of the intermittent nature of demand.

Exhibit 1 shows the results. Column J shows the actual demand over the final 45 observations. The computed reorder points for the Advanced Model are shown in columns L-O.  The computed reorder points for the Normal model are not displayed.  Columns Q-T and V-Y hold the results of the tests for whether the reorder points were high enough to handle the lead time demands in column J.

The final results (yellow cells) show a clear difference between the Normal and Probability (Advanced) demand models. Both did a good job of hitting the 70% service level target, but estimating higher service levels is a more delicate calculation, and the Probability model does a much better job. For instance, the Normal model’s supposed 99% service level turned out to be only 94.4%, while the Probability model hit the target with a 98.5% achieved service level.

Implications

Utilizing the more accurate method achieved the targeted service level, while the less accurate method did not. If the less accurate method is used then real and costly business decisions will be made on the false assumption that a higher service level will be achieved. For example, if a Service Level Agreement (SLA) is based on these results and a 99% service level is committed to, the supplier would actually be five times more likely to stock out than planned (service level promised = 99% or 1% stockout risk vs. service level achieved = 94.5% or 5.5% stock out risk)! This means financial penalties will be incurred five times more often than expected.

Suppose that planners knew the target service level would not be met but were stuck using an inaccurate model. They would still need a way to increase inventory and achieve the desired level of service. What might they choose to do? We have observed situations where the planner enters a higher service level target than needed in order to “trick” the system into delivering the required service level. In the above example, the Normal model needed to have a 99.99% service level entered before it could achieve a target service level of 99%. This change resulted in achieving a 99% service but more than doubled the inventory investment when compared to the Advanced model.

Implementing a Service Level Accuracy Test

At Smart Software, we’ve encouraged many of our customers to conduct the test of service level accuracy as a way for them to assess our and other vendors’ claims during the software selection process. Missing the service level target has extremely costly implications resulting in substantial over stocks or under stocks.  So, test service level accuracy before deploying a solution to identify situations when the modeling fails. Don’t assume that you will achieve the service level you decide to target (or that the system recommends). To request an Excel spreadsheet that serves as a template for a service level accuracy test, email your contact information to info@smartcorp.com and enter “Accuracy Template” in the subject line.

Leave a Comment

Related Posts

The Right Forecast Accuracy Metric for Inventory Planning

The Right Forecast Accuracy Metric for Inventory Planning

Traditional forecasting accuracy metrics aren’t applicable when the goal is to optimize inventory. It’s “service level accuracy” that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.

How to Choose a Target Service Level

How to Choose a Target Service Level

When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Inventory planning parameters such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Ensuring that these inputs are optimized regularly will dramatically improve customer service levels and will reduce the amount of unnecessary inventory spending.

Recent Posts

  • The Right Forecast Accuracy Metric for Inventory Planning
    Traditional forecasting accuracy metrics aren't applicable when the goal is to optimize inventory. It's "service level accuracy" that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.
  • How to Choose a Target Service Level
    When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.
How to Choose a Target Service Level

The Smart Forecaster

Pursuing best practices in demand planning, forecasting and inventory optimization

Summary

Setting a target service level or fill rate is a strategic decision about inventory risk management. Choosing service levels can be difficult. Relevant factors include current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position. Target setting is often best approached as a collaboration among operations, sales and finance. Inventory optimization software is an essential tool in the process.

Service Level Choices

Service level is the probability that no shortages occur between when you order more stock and when it arrives on the shelf. The reasonable range of service levels is from about 70% to 99%. Levels below 70% may signal that you don’t care about or can’t handle your customers. Levels of 100% are almost never appropriate and usually indicate a hugely bloated inventory.

Factors Influencing Choice of Service Level

Several factors influence the choice of service level for an inventory item. Here are some of the more important.

Current service levels:
A reasonable place to start is to find out what your current service levels are for each item and overall. If you are already in good shape, then the job becomes the easier one of tweaking an already-good solution. If you are in bad shape now, then setting service levels can be more difficult. Surprisingly few companies have data on this important metric across their whole fleet of inventory items. What often happens is that reorder points grow willy-nilly from choices made in corporate pre-history and are rarely, sometimes never, systematically reviewed and updated. Since reorder points are a major determinant of service levels, it follows that service levels “just happen”. Inventory optimization software can convert your current reorder points and lead times into solid estimates of your current service levels. This analysis often reveals subset of items with service levels either too high or too low, in which case you have guidance about which items to adjust down or up, respectively.

Replenishment lead times:
Some companies adjust service levels to match replenishment lead times. If it takes a long time to make or buy an item, then it takes a long time to recover from a shortage. Accordingly, they bump up service levels on long-lead-time items and reduce them on items for which backlogs will be brief.

Cost constraints:
Inventory optimization software can find the lowest-cost ways to hit high service level targets, but aggressive targets inevitably imply higher costs. You may find that costs constrain your choice of service level targets. Costs come in various flavors. “Inventory investment” is the dollar value of inventory. “Operating costs” include both holding costs and ordering costs. Constraints on inventory investment are often imposed on inventory executives and always imply ceilings on service level targets; software can make these relationships explicit but not take away the necessity of choice. It is less common to hear of ceilings on operating costs, but they are always at least a secondary factor arguing for lower service levels.

Shortage costs:
Shortage costs depend on whether your shortage policy calls for backorders or lost sales. In either case, shortage costs work counter to inventory investment and operating costs by arguing for higher service levels. These costs may not always be expressed in dollar terms, as in the case of medical/surgical supplies, where shortage costs are denominated in morbidity and mortality.

Competition:
The closer your company is to dominating its market, the more you can ease back on service levels to save money. However, easing back too far carries risks: It encourages potential customers to look elsewhere, and it encourages competitors. Conversely, high product availability can go far to bolstering the position of a minor player.

Collaborative Targeting

Inventory executives may be the ones tasked with setting service level targets, but it may be best to collaborate with other functions when making these calls. Finance can share any “red lines” early in the process, and they should be tasked with estimating holding and ordering costs. Sales can help with estimating shortage costs by explaining likely customer reactions to backlogs or lost sales.

The Role of Inventory Optimization and Planning Software

Without inventory optimization software, setting service level targets is pure guesswork: It is impossible to know how any given target will play out in terms of inventory investment, operating costs, shortage costs. The software can compute the detailed, quantitative tradeoff curves required to make informed choices or even recommend the target service level that results in the lowest overall cost considering holding costs, ordering costs, and stock out costs. However, not all software solutions are created equal. You might enter a user defined 99% service level into your inventory planning system or the system could recommend a target service – but it doesn’t mean you will actually hit that stated service level. In fact, you might not even come close to hitting it and achieve a much lower service level. We’ve observed situations where a targeted service level of 99% actually achieved a service level of just 82%! Any decisions made as a result of the target will result in unintended misallocation of inventory, very costly consequences, and lots of explaining to do. So be sure to check out our next blog article on how to measure the accuracy of your service level forecast so you don’t make this costly mistake.

Leave a Comment

Related Posts

The Right Forecast Accuracy Metric for Inventory Planning

The Right Forecast Accuracy Metric for Inventory Planning

Traditional forecasting accuracy metrics aren’t applicable when the goal is to optimize inventory. It’s “service level accuracy” that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.

How to Choose a Target Service Level

How to Choose a Target Service Level

When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Inventory planning parameters such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Ensuring that these inputs are optimized regularly will dramatically improve customer service levels and will reduce the amount of unnecessary inventory spending.

Recent Posts

  • The Right Forecast Accuracy Metric for Inventory Planning
    Traditional forecasting accuracy metrics aren't applicable when the goal is to optimize inventory. It's "service level accuracy" that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.
  • How to Choose a Target Service Level
    When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.
Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

The Smart Forecaster

Pursuing best practices in demand planning, forecasting and inventory optimization

Inventory planning parameters, such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Based on these parameter settings, your ERP system makes daily purchase order suggestions.

Ensuring that these inputs are understood and optimized regularly will substantially reduce wasteful inventory spending and dramatically improve customer service levels.

Given the importance of getting these planning parameters right, we spend a lot of time during our consultations asking (1) how these parameter values are calculated and (2) how often they are updated. Most often the methods for calculating the parameter values are rule of thumb. You can read about why using rule of thumb approaches is so problematic here  – Beware of Simple Rules of Thumb for Managing Inventory.

This blog will focus on the frequency of updates. When we interview companies and ask them how often they update planning parameters, the answer we nearly always hear is “every day!” A follow up question or two most often reveals that this just isn’t true. What “every day” actually means in practice is this: Every day, the ERP system suggests dozens to hundreds of purchase orders and/or production jobs. The planner, let’s call him Peter, reviews these orders daily and decides whether to release, modify, or cancel them. If the order suggestion doesn’t “feel right”, Peter reviews the planning inputs and modifies the order if necessary. For example, Peter may feel there is already enough inventory on hand. To “fix” the issue, he will reduce the reorder point and cancel the order. Or if he feels that the order should have been placed weeks ago, Peter may expedite the order and increase the reorder point and order quantity to ensure there will be plenty of stock the next time.

The principal flaws with this approach are that it is reactive and incomplete. Here is why:

Reactive

It only assesses the handful of items marked for replenishment on any given day but not others. The trigger for reviewing an item is when the ERP suggests an order, and that will only happen when the reorder point or Min is breached. If the Min is too high and breaches earlier than it should have, an unneeded order will be placed unless caught by the planner. If the Min is too low, well, it is too late to fix the error. No matter how large the order suggestion is, you still have to wait to be resupplied and since the order was suggested late, a stockout during the replenishment period is highly probable. Where is the “planning” in such a process? As one customer put it, “Our former process was, in hindsight, spent managing the outputs and not the inputs.”

Incomplete

What about the thousands of other items that have a Min/Max, safety Stock, Reorder Point, or other parameters that isn’t being reassessed given the updated demand and supply data. The planner isn’t reviewing any of these items which means problems aren’t being identified in advance. Compounding the problem is that when Peter does make a change he doesn’t have any tools to assess the quality of his changes. If he modifies the min/max settings he doesn’t know the specific impact this will have on inventory value, ordering costs, holding costs, stock outs, and service levels. He only knows that an increase in inventory will likely improve service and increase costs. He doesn’t know for example whether his inventory has reached a point of diminishing returns. When inventory decisions are made with only a very rough understanding of the trade offs it creates more problems downstream. You wouldn’t want your carpenter making rough estimates of their measurements yet it’s commonplace for inventory planning professionals to do so with millions of dollars in inventory spend at stake.

How Often Do Most Companies Update Parameters?

So how often do most companies make system-wide updates to their planning parameters such as reorder points, safety stocks, Min/Max settings, lead times, and order quantities? Typically, mass updates occur quarterly, annually, and in some cases never – the only times changes are made are when an order is triggered by ERP. Not exactly agile.

The biggest reason given for not intervening more often is that it takes too much time. Most companies set these key parameters using very unwieldy Excel programs or ERP applications that simply aren’t designed to conduct systemic inventory planning. This is where inventory optimization software can help.

Using inventory optimization software and probability forecasting to update key planning parameters frequently, say every week or month instead of quarterly or annually, enables you quickly respond to changes in your business. You can seize on cost saving opportunities, as when demand turns down and you can reduce reorder points and/or order quantities and possibly cancel outstanding orders. Or you can respond to problems, as when demand increases threaten your service level commitments to customers, or supplier lead times increase and require re-computation of reorder points.

How to do it Right

The key is establishing an agreed upon set of performance and inventory value metrics and letting the software monitor the state of play in the background and alert you to exceptional situations. This is simply one more way of saying that, once systems have been established, you want to go forward using management by exception. You can set ranges within which things can bubble along as they normally do, but once a critical parameter like “stock out risk exceeds a pre-defined level” or “inventory value or costs exceeds a pre-defined level,” the software can provide a daily alert and can also recommend a response, such as raising a reorder point. With this level of automated assistance, it becomes possible to keep your finger on the pulse of the inventory without being overwhelmed by the sheer volume of data.

For example, you may choose an initial set of inventory parameters as the policy because you could see from the software that it meets your service level goals within your inventory budget. You may let the system prescribe service level targets for you and be comfortable with the settings because inventory value is within the budget. However, if demand gets less predictable than historically, you won’t be able to achieve the same level of service without an increase in inventory. An exception report will identify this and enable you to make an informed decision on what to do. You can decide to modify the policy or keep it the same. If you keep it the same, you now know the additional risks and change in inventory costs. This can be communicated to all stake holders so that there aren’t any surprises.

Plan Don’t React

Rather than being constantly in reactive mode, you can handle what really needs to be handled and still have some time to do forward thinking. For instance, you can do “what if” analyses on such issues as which supplier lead times would yield the biggest payoff if reduced, or whether service level targets should be adjusted to account for shifts in customer criticality, or similar policy issues. After all, it’s not as if you are not going to end up with a full daily agenda, it’s just a question of whether you can elevate that agenda to a more strategic level. So if you are spending all of your “planning” time managing the outputs of your ERP instead of constructively reviewing and optimizing the inputs, it is time to reassess your inventory planning process.

 

 

Leave a Comment

Related Posts

The Right Forecast Accuracy Metric for Inventory Planning

The Right Forecast Accuracy Metric for Inventory Planning

Traditional forecasting accuracy metrics aren’t applicable when the goal is to optimize inventory. It’s “service level accuracy” that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.

How to Choose a Target Service Level

How to Choose a Target Service Level

When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Inventory planning parameters such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Ensuring that these inputs are optimized regularly will dramatically improve customer service levels and will reduce the amount of unnecessary inventory spending.

Recent Posts

  • The Right Forecast Accuracy Metric for Inventory Planning
    Traditional forecasting accuracy metrics aren't applicable when the goal is to optimize inventory. It's "service level accuracy" that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.
  • How to Choose a Target Service Level
    When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.
5 Demand Planning Tips for Calculating Forecast Uncertainty

The Smart Forecaster

Pursuing best practices in demand planning, forecasting and inventory optimization

Those who produce forecasts owe it to those who consume forecasts, and to themselves, to be aware of the uncertainty in their forecasts. This note is about how to estimate forecast uncertainty and use the estimates in your demand planning process. We focus on forecasts made in support of demand planning as well as forecasts inherent in optimizing inventory policies involving reorder points, safety stocks, and min/max levels.

Reading this, you will learn about:

-Criteria for assessing forecasts
-Sources of forecast error
-Calculating forecast error
-Converting forecast error into prediction intervals
-The relationship between demand forecasting and inventory optimization.
-Actions you can take to use these concepts to improve your company’s processes.

Criteria for Assessing Forecasts

Forecast error alone is not reason enough to reject forecasting as a management tool. To twist a famous aphorism by George Box, “All forecasts are wrong, but some are useful.” Of course, business professionals will always search for ways to make forecasts more useful. This usually involves work to reduce forecast error. But while forecast accuracy is the most obvious criterion by which to judge forecasts, but it is not the only one. Here’s a list of criteria for evaluating forecasts:

Accuracy: Forecasts of future values should, in retrospect, be very close to the actual values that eventually reveal themselves. But there may be diminishing returns to squeezing another half percent of accuracy out of forecasts otherwise good enough to use in decision making.

Timeliness: Fighter pilots refer to the OODA Loop (Observe, Orient, Decide, and Act) and the “need to get inside the enemy’s OODA loop” so they can shoot first. Businesses too have decision cycles. Delivering a perfectly accurate forecast the day after it was needed is not helpful. Better is a good forecast that arrives in time to be useful.

Cost: Forecasting data, models, processes and people all cost money.  A less expensive forecast might be fueled by data that are readily available; more expensive would be a forecast that runs on data that have to be collected in a special process outside the scope of a firm’s information infrastructure.  A classic, off-the-shelf forecasting technique will be less costly to acquire, feed and exploit than a complex, custom, consultant-supplied method. Forecasts could be mass-produced by software overseen by a single analyst, or they might emerge from a collaborative process requiring time and effort from large groups of people, such as district sales managers, production teams, and others. Technically advanced forecasting techniques often require hiring staff with specialized technical expertise, such as a master’s degree in statistics, who tend to cost more than staff with less advanced training.

Credibility: Ultimately, some executive has to accept and act on each forecast. Executives have a tendency to distrust or ignore recommendations that they can neither understand nor explain to the next person above them in the hierarchy. For many, believing in a “black box” is too severe a test of faith, and they reject the black box’s forecasts in favor of something more transparent.

All that said, we will focus now on forecast accuracy and its evil twin, forecast error.

Sources of Forecast Error

Those seeking to reduce error can look in three places to find trouble:
1. The data that goes into a forecasting model
2. The model itself
3. The context of the forecasting exercise

There are several ways in which data problems can lead to forecast error.

Gross errors: Wrong data produce wrong forecasts. We have seen an instance in which computer records of product demand were wrong by a factor of two! Those involved spotted that problem immediately, but a less egregious situation can easily slip through to poison the forecasting process. In fact, just organizing, acquiring and checking data is often the largest source of delay in the implementation of forecasting software. Many data problems seem to derive from the data having been unimportant until a forecasting project made them important.

Anomalies: Even with perfectly curated forecasting databases, there are often “needle in a haystack” type data problems. In these cases, it is not data errors but demand anomalies that contribute to forecast error. In a set of, say, 50,000 products, some number of items are likely to have odd details that can distort forecasts.

Holdout analysis is a simple but powerful method of analysis. To see how well a method forecasts, use it with older known data to forecast newer data, then see how it would have turned out! For instance, suppose you have 36 months of demand data and need to forecast 3 months ahead. You can simulate the forecasting process by holding out (i.e., hiding) the most recent 3 months of data, forecasting using only data from months 1 to 33, then comparing the forecasts for months 34-36 against the actual values in months 34-36. Sliding simulation merely repeats the holdout analysis, sliding along the demand history. The example above used the first 33 months of data to get 3 estimates of forecast error. Suppose we start the process by using the first 12 months to forecast the next 3. Then we slide forward and use the first 13 months to forecast the next 3. We continue until finally we use the first 35 months to forecast the last month, giving us one more estimate of the error we make when forecasting one month ahead. Summarizing all the 1-step ahead, 2-step ahead and 3-step ahead forecast errors provides a way to calculate prediction intervals.

Calculating Prediction Intervals

The final step in calculating prediction intervals is to convert the estimates of average absolute error into the upper and lower limits of the prediction interval. The prediction interval at any future time is computed as

Prediction interval = Forecast ± Multiplier x Average absolute error.

The final step is the choice of the multiplier. The typical approach is to imagine some probability distribution of error around the forecast, then estimate the ends of the prediction interval using appropriate percentiles of that distribution. Usually, the assumed distribution of error is the Normal distribution, also called the Gaussian distribution or the “bell-shaped curve”.

Use of Prediction Intervals
The most immediate, informal use of prediction intervals is to convey a sense of how “squishy” a forecast is. Prediction intervals that are wide compared to the size of the forecasts indicate high uncertainty.

There are two more formal uses in demand forecasting: Hedging your bets about future demand and guiding forecast adjustment.

Hedging your bets: The forecast values themselves approximate the most likely values of future demand. A more ominous way to say the same thing is that there is about a 50% chance that the actual value will be above (or below) the forecast. If the forecast is being used to plan future production (or raw materials purchase or hiring), you might want to build in a cushion to keep from being caught short if demand spikes (assuming that under-building is worse than over-building). If the forecast is converted from units to dollars for revenue projections, you might want to use a value below the forecast to be conservative in projecting cash flow. In either case, you first have to choose the coverage of the prediction interval. A 90% prediction interval is a range of values that covers 90% of the possibilities. This implies that there is a 5% chance of a value falling above the upper limit of the 90% prediction interval. In other words, the upper limit of a 90% prediction interval marks the 95th percentile of the distribution of predicted demand at that time period. Similarly, there is a 5% chance of falling below the lower limit, which marks the 5th percentile of the demand distribution.

Guiding forecast adjustment: It is quite common for statistical forecasts to be revised by some sort of collaborative process. These adjustments are based on information not recorded in an item’s demand history, such as intelligence about competitor actions. Sometimes they are based on a more vaporous source, such as sales force optimism. When the adjustments are made on-screen for all to see, the prediction intervals provide a useful reference: If someone wants to move the forecasts outside the prediction intervals, they are crossing a fact-based line and should have a good story to justify their argument that things will be really different in the future.

Prediction Intervals and Inventory Optimization

Finally, the concept behind prediction intervals play an essential role in a problem related to demand forecasting: Inventory Optimization.
The core analytic task in setting reorders points (also called Mins) is to forecast total demand over a replenishment lead time. This total is called the lead time demand. When on-hand inventory falls down to or below the reorder point, a replenishment order is triggered. If the reorder point is high enough, there will be an acceptably small risk of a stockout, i.e., of lead time demand driving inventory below zero and creating either lost sales or backorders.

The forecasting task is to determine all the possible values of cumulative demand over the lead time and their associated probabilities of occurring. In other words, the basic task is to determine a prediction interval for some future random variable. Suppose you have computed a 90% prediction interval for lead time demand. Then the upper end of the interval represents the 95th percentile of the distribution. Setting the reorder point at this level will accommodate 95% of the possible lead time demand values, meaning there will be only a 5% chance of stocking out before replenishment arrives to re-stock the shelves. Thus there is an intimate relationship between prediction intervals in demand forecasting and calculation of reorder points in inventory optimization.5

5 Recommendations for Practice

1. Set expectations about error: Sometimes managers have unreasonable expectations about reducing forecast error to zero. You can point out that error is only one of the dimensions on which a forecasting process must be judged; you may be doing fine on both timeliness and cost. Also point out that zero error is no more realistic a goal than 100% conversion of prospects into customers, perfect supplier performance, or zero stock price volatility.

2. Track down sources of error: Double check the accuracy of demand histories. Use statistical methods to identify outliers in demand histories and react appropriately, replacing verified anomalies with more typical values and omitting data from before major changes in the character of the demand. If you use a collaborative forecasting process, compare its accuracy against a purely statistical approach to identify items for which collaboration does not reduce error.

3. Evaluate the error of alternative statistical methods: There may be off-the-shelf techniques that do better than your current methods, or do better for some subsets of your items. The key is to be empirical, using the idea of holdout analysis. Gather your data and do a “bake off” between different methods to see which work better for you. If you are not already using statistical forecasting methods, compare them against whoever’s “golden gut” is your current standard. Use the naïve forecast as a benchmark in the comparisons.

4. Investigate the use of new data sources: Especially if you have items that are heavily promoted, test out statistical methods that incorporate promotional data into the forecasting process. Also check whether information from outside your company can be exploited; for instance, see whether macroeconomic indicators for your sector can be combined with company data to improve forecast accuracy (this is usually done using a method called multiple regression analysis).

5. Use prediction intervals: Plots of prediction intervals can improve your feel for the uncertainty in your forecasts, helping you select items for additional scrutiny. While it’s true that what you don’t know can hurt you, it’s also true that knowing what you don’t know can help you.

Leave a Comment

Related Posts

The Right Forecast Accuracy Metric for Inventory Planning

The Right Forecast Accuracy Metric for Inventory Planning

Traditional forecasting accuracy metrics aren’t applicable when the goal is to optimize inventory. It’s “service level accuracy” that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.

How to Choose a Target Service Level

How to Choose a Target Service Level

When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Inventory planning parameters such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Ensuring that these inputs are optimized regularly will dramatically improve customer service levels and will reduce the amount of unnecessary inventory spending.

Recent Posts

  • The Right Forecast Accuracy Metric for Inventory Planning
    Traditional forecasting accuracy metrics aren't applicable when the goal is to optimize inventory. It's "service level accuracy" that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.
  • How to Choose a Target Service Level
    When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.
Excess Inventory Hurts Customer Service!

The Smart Forecaster

Pursuing best practices in demand planning, forecasting and inventory optimization

Many companies adopt a philosophy of “it’s better to have it and not need it, then to need it and not have it.” Planning initiatives such as implementing inventory optimization software in order to optimize reorder points, safety stocks, and order quantities are often seen as narrowly focused on reducing inventory and not pursued. Stock-out costs may very well be extremely high. However, resources are finite. The opportunity cost of keeping too much of one product means less space, cash, and resources for another product. Overstocking on one item reduces the ability to provide adequate levels of service on other items. Justifying overstocks by stating it is good for the customer is a poor excuse at best that hurts the customer and ignores what inventory optimization is really about – properly reallocating inventory investments.

Diminishing Returns and Inventory

Each additional unit of inventory that you carry buys proportionally less service. Inventory optimization software can help you understand the exact stock out risk given a certain level of stock. For example, say your stock-out risk with 20 units of inventory is 10%. If you add another 10 units and carry 30 units, the stock out risk might get cut in half to 5%. If you then add an additional 10 for a total of 40 units, the stock-out risk may only drop to 4%. At some point, the additional inventory just isn’t worth the extra service it buys. This is especially so if the cash used to buy that extra 10 units to get a small service level bump on one item could have been spent on another equally important item for a larger increase in service.

Carrying more than you need means you aren’t efficiently managing assets, which costs money, which means you can’t offer the best price to your customer, which hurts your ability to beat the competition. It also means there is less money for investment in other items. This results in the common adage “We have too much of the stuff we don’t need and not enough of the stuff we do.”

Inventory Optimization is about reallocation

The example presented in the blog’s main image highlights the benefits of reallocating inventory.  We used probability forecasting to estimate the service levels and inventory costs that would result from the current stocking policy. We then conducted a “what-if” scenario by modifying the policy. In the benchmark shown in the first column, the current stock levels were forecasted to yield a 84.78% service level and required $1.67 Million in inventory. Nearly 12% of the items numbers had reached their point of diminishing return and were forecasted to achieve a 100% service level. By imposing a maximum service level of 99% and a minimum service level of 80%, we reallocated inventory.  As a result, the inventory investment dropped to $1.5 Million and service level increased by 3%!

The exact point of diminishing returns will differ depending on the item, the customers involved, and the company making the stocking decision. It is important to understand the inherent levels of stock-out risk that result from current inventory policies and how changes to current policies will impact risk and costs. This enables the reshaping of inventory so that service can be maximized at the minimum possible cost.

Download Smart Inventory Optimization product sheet here: https://smartcorp.com/inventory-optimization/

Leave a Comment

Related Posts

The Right Forecast Accuracy Metric for Inventory Planning

The Right Forecast Accuracy Metric for Inventory Planning

Traditional forecasting accuracy metrics aren’t applicable when the goal is to optimize inventory. It’s “service level accuracy” that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.

How to Choose a Target Service Level

How to Choose a Target Service Level

When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Infrequent Updates to Inventory Planning Parameters Costs Time, Money, and Hurts Service

Inventory planning parameters such as safety stock levels, reorder points, Min/Max settings, lead times, order quantities, and DDMRP buffers directly impact inventory spending and ability to meet customer demand. Ensuring that these inputs are optimized regularly will dramatically improve customer service levels and will reduce the amount of unnecessary inventory spending.

Recent Posts

  • The Right Forecast Accuracy Metric for Inventory Planning
    Traditional forecasting accuracy metrics aren't applicable when the goal is to optimize inventory. It's "service level accuracy" that matters because just setting a service target doesn’t mean you’ll actually achieve it. Poor accuracy here has extremely costly implications. The right way to measure accuracy for inventory planning is to focus on the accuracy of the service level projection. This blog explains why and details how to calculate the metric.
  • How to Choose a Target Service Level
    When setting a target service level, make sure to take into account factors like current service levels, replenishment lead times, cost constraints, the pain inflicted by shortages on you and your customers, and your competitive position.