The same approach can describe a more interesting random variable, like the daily demand for a spare part.  Figure 2 shows such a probability distribution. It was computed by compiling three years of daily demand data on a certain part used in a scientific instrument sold to hospitals.

Figure 1: The probability distribution of daily demand for a certain spare part

The distribution in Figure 1 can be thought of as a probabilistic forecast of demand in a single day. For this particular part, we see that the forecast is very likely to be zero (97% chance), but sometimes will be for a handful of units, and once in three years will be twenty units. Even though the most likely forecast is zero, you would want to keep a few on hand if this part were critical (“…for want of a nail…”)

Now let’s use this information to make a more complicated probabilistic forecast. Suppose you have three units on hand. How many days will it take for you to have none? There are many possible answers, ranging from a single day (if you immediately get a demand for three or more) up to a very large number (since 97% of days see no demand).  The analysis of this question is a bit complicated because of all the many ways this situation can play out, but the final answer that is most informative will be a probability distribution. It turns out that the number of days until there are no units left in stock has the distribution shown in Figure 2.

Figure 2: Distribution of the number of days until all three units are gone

The average number of days is 74, which would be a point forecast, but there is a lot of variation around the average. From the perspective of inventory management, it is notable that there is a 25% chance that all the units will be gone after 32 days. So if you decided to order more when you were down to only three on the shelf, it would be good to have the supplier get them to you before a month has passed. If they couldn’t, you’d have a 75% chance of stocking out – not good for a critical part.

The analysis behind Figure 2 involved making some assumptions that were convenient but not necessary if they were not true. The results came from a method called “Monte Carlo simulation”, in which we start with three units, pick a random demand from the distribution in Figure 1, subtract it from the current stock, and continue until the stock is gone, recording how many days went by before you ran out. Repeating this process 100,000 times produced Figure 2.

Applications of Monte Carlo simulation extend to problems of even larger scope than the “when do we run out” example above. Especially important are Monte Carlo forecasts of future demand. While the usual forecasting result is a set of point forecasts (e.g., expected unit demand over the next twelve months), we know that there are any number of ways that the actual demand could play out. Simulation could be used to produce, say, one thousand possible sets of 365 daily demand demands.

This set of demand scenarios would more fully expose the range of possible situations with which an inventory system would have to cope. This use of simulation is called “stress testing”, because it exposes a system to a range of varied but realistic scenarios, including some nasty ones. Those scenarios are then input to mathematical models of the system to see how well it will cope, as reflected in key performance indicators (KPI’s). For instance, in those thousand simulated years of operation, how many stockouts are there in the worst year? the average year? the best year? In fact, what is the full probability distribution of the number of stockouts in a year, and what is the distribution of their size?

Figures 3 and 4 illustrate probabilistic modeling of an inventory control system that converts stockouts to backorders. The system simulated uses a Min/Max control policy with Min = 10 units and Max = 20 units.

Figure 3 shows one simulated year of daily operations in four plots. The first plot shows a particular pattern of random daily demand in which average demand increases steadily from Monday to Friday but disappears on weekends. The second plot shows the number of units on hand each day. Note that there are a dozen times during this simulated year when inventory goes negative, indicating stockouts. The third plot shows the size and timing of replenishment orders. The fourth plot shows the size and timing of backorders.  The information in these plots can be translated into estimates of inventory investment, average units on hand, holding costs, ordering costs and shortage costs.

Figure 3: One simulated year of inventory system operation

Figure 3 shows one of one thousand simulated years. Each year will have different daily demands, resulting in different values of metrics like units on hand and the various components of operating cost. Figure 4 plots the distribution of 1,000 simulated values of four KPI’s. Simulating 1,000 years of imagined operation exposes the range of possible results so that planners can account not just for average results but also see best-case and worst-case values.

Figure 4: Distributions of four KPI’s based on 1,000 simulations

Monte Carlo simulation is a low-math/high-results approach to probabilistic forecasting: very practical and easy to explain. Advanced probabilistic forecasting methods employed by Smart Software expand upon standard Monte Carlo simulation, yielding extremely accurate estimates of required inventory levels.

Forecast error can be consequential

Consider one item of many

• Product X costs $100 to make and nets $50 profit per unit.
• Sales of Product X will turn out to be 1,000/month over the next 12 months.
• Consider one item of many

What is the cost of forecast error?

• If the forecast is 10% high, end the year with $120,000 of excess inventory.
• 100 extra/month x 12 months x $100/unit
• If the forecast is 10% low, miss out on $60,000 of profit.
• 100 too few/month x 12 months x $50/unit

Three mistakes to avoid

1. Ignoring error.

• Unprofessional, dereliction of duty.
• Wishing will not make it so.
• Treat accuracy assessment as data science, not a blame game.

2. Tolerating more error than necessary.

• Statistical forecasting methods can improve accuracy at scale.
• Improving data inputs can help.
• Collecting and analyzing forecast error metrics can identify weak spots.

3. Wasting time and money going too far trying to eliminate error.

• Some product/market combinations are inherently more difficult to forecast. After a point, let them be (but be alert for new specialized forecasting methods).
• Sometimes steps meant to reduce error can backfire (e.g., adjustment).

RECENT POSTS

Four general types of error metrics 

1. Scale-dependent error

2. Percentage error

3. Relative error

4 .Scale-free error

Remark: Scale-dependent metrics are expressed in the units of the forecasted variable. The other three are expresses as percentages.

1. Scale-dependent error metrics

• Mean Absolute Error (MAE) aka Mean Absolute Deviation (MAD)
• Median Absolute Error (MdAE)
• Root Mean Square Error (RMSE)

• These metrics express the error in the original units of the data.
  • Ex: units, cases, barrels, kilograms, dollars, liters, etc.
• Since forecasts can be too high or too low, the signs of the errors will be either positive or negative, allowing for unwanted cancellations.
  • Ex: You don’t want errors of +50 and -50 to cancel and show “no error”.
• To deal with the cancellation problem, these metrics take away negative signs by either squaring or using absolute value.

2. Percentage error metric

• Mean Absolute Percentage Error (MAPE)
  

• This metric expresses the size of the error as a percentage of the actual value of the forecasted variable.
• The advantage of this approach is that it immediately makes clear whether the error is a big deal or not.
• Ex: Suppose the MAE is 100 units. Is a typical error of 100 units horrible? ok? great?
• The answer depends on the size of the variable being forecasted. If the actual value is 100, then a MAE = 100 is as big as the thing being forecasted. But if the actual value is 10,000, then a MAE = 100 shows great accuracy, since the MAPE is only 1% of the actual.

3. Relative error metric

• Median Relative Absolute Error (MdRAE)

• Relative to what? To a benchmark forecast.
• What benchmark? Usually, the “naïve” forecast.
• What is the naïve forecast? Next forecast value = last actual value.
• Why use the naïve forecast? Because if you can’t beat that, you are in tough shape.

4. Scale-Free error metric

• Median Relative Scaled Error (MdRSE)
  

• This metric expresses the absolute forecast error as a percentage of the natural level of randomness (volatility) in the data.
• The volatility is measured by the average size of the change in the forecasted variable from one time period to the next.
  • (This is the same as the error made by the naïve forecast.)
• How does this metric differ from the MdRAE above?
  • They do both use the naïve forecast, but this metric uses errors in forecasting the demand history, while the MdRAE uses errors in forecasting future values.
  • This matters because there are usually many more history values than there are forecasts.
  • In turn, that matters because this metric would “blow up” if all the data were zero, which is less likely when using the demand history.

The special problem of intermittent demand

• “Intermittent” demand has many zero demands mixed in with random non-zero demands.
• MAPE gets ruined when errors are divided by zero.
• MdRAE can also get ruined.
• MdSAE is less likely to get ruined.

Recap and remarks

• Forecast metrics are necessary aids for monitoring and improving forecast accuracy.
• There are two major classes of metrics: absolute and relative.
• Absolute measures (MAE, MdAE, RMSE) are natural choices when assessing forecasts of one item.
• Relative measures (MAPE, MdRAE, MdSAE) are useful when comparing accuracy across items or between alternative forecasts of the same item or assessing accuracy relative to the natural variability of an item.
• Intermittent demand presents divide-by-zero problems which favor MdSAE over MAPE.
• When assessing forecasts of multiple items, it often makes sense to use weighted averages, weighting items differently by volume or revenue.

RECENT POSTS

RECENT POSTS

RECENT POSTS