No, not that kind of regime change: Nothing here about cruise missiles and stealth bombers. And no, we’re not talking about the other kind of regime change that hits closer to home: Shuffling the C-Suite at your company.

“Regime change” has a third meaning that is relevant to your profession as a demand planner or inventory manager. To researchers in economics and finance, regime change means sudden shifts in the very character of a time series of random observations. The random time series in question here is the sequence of daily (or weekly or monthly) demand counts for your products and inventory items.

Most forecasting software uses statistical algorithms to process historical demand. It may add additional steps, such as incorporating field intelligence from sales people, but everything starts with the demand history of whatever item you must manage.

The question raised by regime change is, which data do you use? The simple answer is “All of it”, because that leads to the most accurate forecasts — but only if your data world is stable. If your data world is turbulent, then using all the data means you are basing forecasts on bye-gone conditions. In turn, inputting obsolete data into your forecasting algorithms inevitably leads to reduced forecast accuracy.

Note that dealing with regime change is not the same as dealing with outliers. Outliers are usually one-off exceptions caused by transient events, such as a kink in your supply chain caused by a huge blizzard choking off all transit paths. In contrast, regime change persists over a longer period and is therefore capable of doing more damage to your forecasts. Here’s an analogy: Outliers are about weather, and regime change is about climate.

The most drastic forms of regime change are existential. Figure 1 shows an example of an existential change: There was no demand at all for a long time, then suddenly there was demand. If you had no demand for an item because it didn’t exist but you retain zero demand values in your database, and then the item goes live and you do have sales, the transition from nothing to something is an extreme regime change. Including all those zero demand values from before “Day One” is sure to bias statistical forecasts down below where they should be. The same thing happens if you kill off a product but keep recording zero demand: Including all those recent zeros degrades your demand forecasts.

In principle, careful record keeping should eliminate these problems. You should record only meaningful zero values. If you have a new item, start recording when it goes live. If you no longer have any demand for an item and expect none, purge it from your database, or at least forecast zero demand.

Unfortunately, there is a difference between principle and practice. We see many instances in which the data records for both new and dormant items are not properly kept, with “fake zeros” confounded with “real zeros”. This problem is not necessarily the result of incompetence: Usually, it is a byproduct of the scale of the problem, with too few people trying to keep track of too many items.

These existential regime changes are relatively easy to deal with compared to more subtle forms, which appear to afflict more items. Figure 2 shows two examples of regime changes in a pattern of ongoing sales. There are any number of factors that can change the demand for an item: salesforce performance, marketing and advertising efforts, competitor and supplier actions, new customers arising or old customers disappearing, etc. If demand for an item has been chugging along at a steady 1 unit per day but suddenly doubles (or vice versa), that’s a regime change. In the new world order, demand is 2 units/day and forecasts should reflect that. Instead, statistical forecasting algorithms will forecast too little demand if fed all the data, including that from before the regime change.

How do you protect yourself from regime change? The answer is the same for the cruelest dictator or the most innocent demand planner: Intelligence. And because threats are many, the intelligence is best automated. Modern software systems have the capability to screen tens of thousands of items for signs of regime change. Then the software can call your attention to the problematic items and prompt you to designate which recent data to use in calculations. Or the software can automatically detect and correct for regime change, working quickly at a scale that would easily defeat any busy person working “by hand”.

Quants and Financial Meltdowns

I spend much of my time developing new quantitative methods for statistical forecasting, demand forecasting and inventory optimization. For me, this is an engaging way to contribute to society. But I know that the most prudent way to do algorithm development is to stand a little to the side and cast a skeptical eye on my own work.

The need for this skepticism was highlighted for me recently as I read Scott Patterson’s book The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It (Crown Publishing, 2010). This book reviewed the “quants” whose complex financial models were largely responsible for the financial meltdown in 2007. As I read along and thought “What was wrong with these guys?” I began to wonder if we supply chain quants were guilty of some of the same sins.

Models versus Instincts

Generally, the supply chain field has lagged behind finance in terms of the use of statistical models. My university colleagues and I are chipping away at that, but we have a long way to go. Some supply chains are quite technically sophisticated, but many, perhaps more, are essentially managed as much by gut instinct as by the numbers. Is this avoidance of analytics safer than relying on models?

What makes gut instinct dangerous is that it is so amorphous. Everyone who works long in a job develops instincts, but longevity is not the same as wisdom. It is possible to learn all the wrong lessons over a long career. It is also possible to miss the chance to learn the right lessons because certain informative scenarios may never arise in one person’s career. It is also possible to have good days and bad days; even gurus can mess up. Gut instinct is also anti-productive, since all decisions have to pass through that one gut, which becomes an enterprise chokepoint. And Golden Guts eventually reach their Golden Years and take their Golden Watch and go off into a Golden Sunset; at that point, whatever expertise had been present has walked out the door.

In contrast, models have certain advantages. Relative to gut instinct, models are:

• Explicit: The theory of the supply chain operation is exposed for all to see.
• Adaptive: Because the theory is visible, it can be reviewed, critiqued, tested against data, and evolved.
• Consistent: Models may be more or less true, but they are not subject to day-to-day variability.
• Comprehensive: At least potentially, models can accumulate a wide range of empirical experience, including scenarios never encountered during any one person’s career.
• Instructive: Models are collections of relationships among variables. If the model’s “guts” are made visible, users can learn about those relationships.

Model Error

Nevertheless, despite all their virtues, models can also be wrong. In fact, that is a given. A constructive way to live with this is encoded in the famous aphorism by Dr. George Box, one of the best modelers of the last half century: “All models are wrong. Some are useful.”

The finance quants’ models were wrong by being oversimplified. They started with a quasi-religious belief in the efficiency of markets and developed statistical models that made certain assumptions that were more likely to be true of the physical world than the financial world. Among these were Normal distributions of changes in asset prices and independence of events across various corners of the market. They also assumed human rationality.

It should be a bit alarming that the Normal distribution and independence assumptions also underlie many of the models in supply chain software. In fact, there are alternative models of supply chain dynamics that do not require these simplifying assumptions, so this is an unnecessary risk being run by many, perhaps most, of the users of supply chain software.

But even with more robust and realistic model assumptions, there is no denying that model error is a constant risk. So, can you be victimized by your models? Of course you can.

Self-Protection: Look at the Data

Every supply chain professional who uses models, then, is subject to model risk. But unlike with decisions based on gut feel, decisions based on model calculations can be exposed and compared to real-world outcomes. Repeated checking is the best way to protect against model error, because it not only tests whether the model is realistic but also signals when it is time to update the model.

As noted above, a model is a set of functional relationships between key variables. Those relationships have parameters that tune the model to the current operating context. For instance, supply chain models often rely, in part, on estimates of demand volatility. Historical demand data are used to calculate numerical values for these parameters. If demand volatility changes, the model becomes obsolete and likely to produce inapt recommendations. Therefore, good practice demands frequent updates to model parameters.

Even when parameter values are current, there may still be trouble due to incorrect functional relationships. For example, consider the relationship between the mean and standard deviation of demand for spare parts. Generally speaking, the greater the average demand, the greater the demand volatility as measured by the standard deviation.

Now consider simplified “old school” models that describe spare part demand as a Poisson process. The Poisson process is widely useful and relatively simple, so it often shows up in Statistics 101 classes. Because of their relative simplicity, Poisson models are the white rats of supply chain analytics for spare parts, i.e., people do computer experiments and theory development based on the behavior of Poisson models of demand. For Poisson models, the standard deviation of demand equals the square root of the mean. However, when we look at our customers’ actual demand data, we discover that the actual relationship between the mean and standard deviation of demand is better described by a more general power-law relationship. Thus, the simple model may use accurate estimates of mean and standard deviation but still not accurately reflect their relationship. This in turn leads to incorrect recommendations about reorder points for spare parts. Checking real data is the best antidote to cavalier assumption-making.

What to Do Next

I do not sense that today’s supply chain models are on the brink of creating the kind of meltdown we saw in the start of the Great Recession. But those of us who are supply chain quants need to show more professional maturity than our financial colleagues. We need to not fall in love with our models, and we need to alert our customers to correct model hygiene.

So, model users, wash your hands frequently as we begin flu season, and wash your models thoroughly through hard data to be sure that the models you rely on are both up-to-date and grounded in reality. Both those steps will protect you from being victimized by your models and let you exploit their advantages over management by gut feel.

Appendix: Technical Tips

Supply chain analytics provide various types of outputs. In the realm of forecasting and demand planning, the obvious empirical check is to compare forecasts against the actual demand values that eventually reveal themselves. This same “forecast then check” approach can also be used in the generation of forecasts.  In the realm of inventory management, the models can build on forecasts to recommend policy choices, such as reorder points and order quantities or Min and Max values. There is a smart way to confirm the accuracy of recommendations of reorder points and Min’s.  See our blog The Right Forecast Accuracy Metric for Inventory Planning