We work with many customers in many industries to connect our advanced analytical, forecasting, and inventory planning software to their ERP systems. Despite the variety of situations we encounter, some data-related problems tend to crop up over and over. This blog lists ten tips that can help you avoid these common problems.

Once a customer is ready to implement software for demand planning and/or inventory optimization, they need to connect the analytics software to their corporate data stream. In our case, we mainline transaction data directly into the analytical software. This provides information on item demand and supplier lead times, among other things. We extract the rest of the data from the ERP system itself, which provides metadata such as each item’s location, unit cost, and product group.

These tips are important because it is not uncommon for implementation projects to start with great enthusiasm but then quickly bog down because of problems with the data that fuel for analytics. These delays can reduce team enthusiasm, embarrass project leaders, and delay (and thereby reduce) the ROI payoff that ultimately justified the implementation project in the first place.

The importance of connecting the analytics software to the corporate data stream

Here is the list of tips, grouped by the general themes of handling files safely, insuring data integrity, and dealing with exceptions.

Handling Files Safely

1. Have a test environment to use as a “sandbox.” Copy your current data to a test environment where you can safely experiment with the software without risking current operations. Besides helping users learn the ins-and-outs of the new software, having the latest data in the software allows end users to discover any problems with the data.

2. Protect your data extraction rules. If you aren’t utilizing a pre-built connector to your ERP system then you to need to ensure that you can create savable extract rules to move data from your ERP to a file.  Column orders, data types, date formats, etc. should not vary each time the same extract is re-executed.  Otherwise the project gets bogged down in manual errors or confusion in re-extracts after fixes to the data or when new data roll in. All data extraction rules should be saved and available to IT – we’ve encountered situations where files extracted were done so in ad hoc manner resulting in a slightly different formats with each new extract.  We’ve also seen customers work hard to develop a complex and accurate data extraction routine only to find all their work was lost when it was not properly archived.  Both situations led to confusion and project delays.

3. Don’t use Excel native file formats for data transfers. If your planning solution doesn’t have a direct integration to your ERP system, then export ERP data to a flat file format, such as comma delimited (.csv) or tab delimited text files.  Don’t use MS Excel formats such as .xls or .xlsx as the export file type because Excel auto-reformats field values in unexpected ways. Many users assume they need to use .xlsx files if they want to manually review them, not realizing that .csv or .txt files can be opened just as easily and don’t carry the risk of auto-reformats.

Insuring Data Integrity

Data Problems and solutions in Software Implementation. Here is the list of tips, grouped by the general themes of handling files safely, insuring data integrity, and dealing with exceptions.

4. Confirm the accuracy of your catalog data. Export your catalog data (i.e., list of products, list of customers, list of suppliers) and all their relevant attributes.  Check for wrong or suspicious values in the attributes (especially item lead times and costs).  Problematic values include blanks, zeros when you don’t expect zero as a data value, and text strings when you expect numeric values (or vice versa).  It can help to open each extract file in Excel and filter on each attribute field, looking at the unique values to see what jumps out as not like the others (e.g., “1”, “2”, “&&”, “3”…).

5. Confirm the accuracy of your grouping data. Another useful activity that can be done while viewing the product catalog data in Excel is to check major grouping/filtering fields like product family, category or class to make sure no products are assigned to the wrong category, class, or family.  Likewise check any product status/product lifecycle fields, e.g., make sure that you have correctly identified all discontinued products.

6. Check for spurious control characters within text fields. Check that there are no unusual characters extracted in your product descriptions, such as carriage returns or tabs within the description value itself.  If so, make sure you can extract that data using double quote enclosures around the description or else fix data entry errors in the ERP system directly.

7. Verify that data have a standard layout. Check that your extracts of transactional data (e.g., customer orders, customer shipments, purchase orders, supplier receipts) contain no duplicate rows.  If they do, either identify what fields need to be added to make the rows distinct or, if they are truly duplicates, remove the extra copies in the ERP database.

Dealing with Exceptions

8. Detect and react to exceptions. Identify any attributes of transactional data that would mean they should not be used, such as cancelled orders.  Understand the process around mistakenly entered orders or cancelled orders to ensure against counting, or double counting, these types of transactions.  Watch for other data attributes that would imply that attribute should not be used, such as drop shipping to the customer directly from a supplier rather than shipping it from your own company. 

9. Codify the handling of exceptional internal transfers. Define the idealized record of emergency internal stock transfers and then provide rules to edit any transactions done on an emergency basis that vary from the ideal pattern.  For example, if product P1 is supposed to be shipped out of location A, but there was an emergency shipment out of location B, the demand history for P1 at location A is hijacked and less than it should have been.  If possible, provide a rule on the preferred shipping location for each product so that the history can be corrected by the inventory optimization software for forecasting purposes.

10. Devise a procedure to handle supersession. Supersessions arise, for instance, when adopting a new ERP which re-indexes the products, or an old product is replaced by an updated version, or an entirely new product obsoletes and old one. If product identifiers changed within the past few years for any reason, identify a mapping from the old product ID to the new.  These rules should be available to the demand planning and forecasting system and editable within the application.

Failure to anticipate data problems is a major impediment to smooth implementation of new analytical software. No list can enumerate all the odd things that can go wrong in curating data, but this one highlights common problems and sensible responses.

Note: For more on how data problems can stymie the application of advanced analytical  software, see Sean Snapp’s excellent blog on how this issue is obstructing the application of artificial intelligence and machine learning.  https://www.brightworkresearch.com/demandplanning/2019/05/how-many-ai-projects-will-fail-due-to-a-lack-of-data/

In another blog we posed the question:  How can you be sure that you really have a policy for inventory planning and demand forecasting? We explained how an organization’s lack of understanding on the basics (how a forecast is created, how safety stock buffers are determined, and how/why these values are adjusted) contributes to poor forecast accuracy, misallocated inventory, and lack of trust in the whole process.

In this blog, we review 10 specific questions you can ask to uncover what’s really happening at your company. We detail the typical answers provided when a forecasting/inventory planning policy doesn’t really exist, explain how to interpret these answers, and offer some clear advice on what to do about it.

Always start with a simple hypothetical example. Focusing on a specific problem you just experienced is bound to provoke defensive answers that hide the full story. The goal is to uncover the actual approach used to plan inventory and forecasts that has been baked into the mental math or spreadsheets.   Here is an example:

Suppose you have 100 units on hand, the lead time to replenish is 3 months, and the average monthly demand is 20 units?   When should you order more?  How much would you order? How will your answer change if expected receipts of 10 per month were scheduled to arrive?  How will your answer change if the item is the item is an A, B, or C item, the cost of the item is high or low, lead time of the item is long or short?  Simply put, when you schedule a production job or place a new order with a supplier, why did you do it? What triggered the decision to get more?  What planning inputs were considered?

When getting answers to the above question, focus on uncovering answers to the following questions:

1. What is the underlying replenishment approach? This will typically be one of Min/Max, forecast/safety stock, Reorder Point/Order Quantity, Periodic Review/Order Up To or even some odd combination

2. How are the planning parameters, such as demand forecasts, reorder points, or Min/Max, actually calculated? It’s not enough to know that you use Min/Max.  You have to know exactly how these values are calculated. Answers such as “We use history” or “We use an average” are not specific enough.   You’ll need answers that clearly outline how history is used.  For example, “We take an average of the last 6 months, divide that by 30 to get a daily average, and then multiply that by the lead time in days.  For ‘A’ items we then multiply the lead time average by 2 and for ‘B’ items we use a multiplier of 1.5.” (While that is not an especially good technical approach, at least it has a clear logic.)

Once you have a policy well-defined, you can identify its weaknesses in order to improve it.  But if the answer provided doesn’t get much further past “We use history”, then you don’t have a policy to start with.   Answers will often reveal that different planners use history in different ways.  Some may only consider the most recent demand, others might stock according to the average of the highest demand periods, etc.  In other words, you may find that you actually have multiple ill-conceived “policies”.

3. Are forecasts used to drive replenishment planning and if so, how? Many companies will say they forecast, but their forecasts are calculated and used differently. Is the forecast used to predict what on hand inventory will be in the future, resulting in an order being triggered?  Or is it used to derive a reorder point but not to predict when to order (i.e. I predict we’ll sell 10 a week so to help protect against stock out, I’ll order more when on hand gets to 15)? Is it used as a guide for the planner to help subjectively determine when they should order more?  Is it used to set up blanket orders with suppliers?  Some use it to drive MRP. You’ll need to know these specifics.  A thorough answer to this question might look like this: “My forecast is 10 per week and my lead time is 3 weeks so I make my reorder point a multiple of that forecast, typically 2 x lead time demand or 60 unit for important items and I use a smaller multiple for less important items.  (Again, not a great technical approach, but clear.)

4.  What technique is actually used to generate the forecast? Is it an average, a trending model such as double exponential smoothing, a seasonal model? Does the choice of technique change depend on the type of demand data or when new demand data is available? (Spare parts and high-volume items have very different demand patterns.) How do you go about selecting the forecast model? Is this process automated?  How often is the choice of model reconsidered?  How often are the model parameters recomputed? What is the process used to reconsider your approach?  The answer here documents how the baseline forecasts are produced.  Once determined, you can conduct an analysis to identify whether other forecasting methods would improve forecast accuracy.  If you aren’t documenting forecast accuracy and conducting “forecast value add” analysis then you aren’t in a position to properly assess whether the forecasts being produced are the best that they can be.  You’ll miss out on opportunities to improve the process, increase forecast accuracy, and educate the business on what type of forecast error is normal and should be expected.

5. How do you use safety stock? Notice the question was not “Do you use safety stock?” In this context, and to keep it simple, the term “safety stock” means stock used to buffer inventory against supply and demand variability.  All companies use buffering approaches in some way.  There are some exceptions though.  Maybe you are a job shop manufacturer that procures all parts to order and your customers are completely fine waiting weeks or months for you to source material, manufacture, QA, and ship.  Or maybe you are high-volume manufacturer with tons of buying power so your suppliers set up local warehouses that are stocked full and ready to provide inventory to you almost immediately.  If these descriptions don’t describe your company, you will definitely have some sort of buffer to protect against demand and supply variability.  You may not use the “safety stock” field in your ERP but you are definitely buffering.

Answers might be provided such as “We don’t use safety stock because we forecast.”  Unfortunately, a good forecast will have a 50/50 chance of being over/under the actual demand.  This means you’ll incur a stock out 50% of the time without a safety stock buffer added to the forecast.  Forecasts are only perfect when there is no randomness. Since there is always randomness, you’ll need to buffer if you don’t want to have abysmal service levels.

If the answer isn’t revealed, you can probe a bit more into how the varying replenishment levers are used to add possible buffers which leads to questions 6 & 7.

6. Do you ever increase the lead time or order earlier than you truly need to?
In our hypothetical example, your supplier typically takes 4 weeks to deliver and is pretty consistent. But to protect against stockouts your buyer routinely orders 6 weeks out instead of 4 weeks.  The safety stock field in your ERP system might be set to zero because “we don’t use safety stock”, but in reality, the buyer’s ordering approach just added 2 weeks of buffer stock.

7. Do you pad the demand forecast?
In our example, the planner expects to consume 10 units per month but “just in case” enters a forecast of 20 per month.  The safety stock field in the MRP system is left blank but the now disguised buffer stock has been smuggled into the demand forecast.  This is a mistake that introduces “forecast bias.”  Not only will your forecasts be less accurate but if the bias isn’t accounted for and safety stock is added by other departments, you will overstock.

The ad-hoc nature of the above approaches compounds the problems by not considering the actual demand or supply variability of the item. For example, the planner might simply make a rule of thumb that doubles the lead time forecast for important items.  One-size doesn’t fit all when it comes to inventory management.  This approach will substantially overstock the predictable items while substantially understocking the intermittently demanded items. You can read “Beware of Simple Rules of Thumb for Managing Inventory” to learn more about why this type of approach is so costly.

The ad-hoc nature of the approaches also ignores what happens the company is faced with a huge overstock or stock out. When trying to understand what happened, the stated policies will be examined. In the case of an overstock, the system will show zero safety stock.  The business leaders will assume they aren’t carrying any safety stock, scratch their heads, and eventually just blame the forecast, declare “Our business can’t be forecasted” and stumble on. They may even blame the supplier for shipping too early and making them hold more than needed. In the case of a stock out, they will think they aren’t carrying enough and arbitrarily add more stock across many items not realizing there is in fact lots of extra safety stock baked into process.  This makes it more likely inventory will need to be written off in the future.

8. What is the exact inventory terminology used? Define what you mean by safety stock, Min, reorder point, EOQ, etc.  While there are standard technical definitions it’s possible that something differs, and miscommunication here will be problematic.  For example, some companies refer to Min as the amount of inventory needed to satisfy lead time demand while some may define Min as inclusive of both lead time demand and safety stock to buffer against demand variability. Others may mean the minimum order quantity.

9. Is on hand inventory consistent with the policy? When your detective work is done and everything is documented, open your spreadsheet or ERP system and look at the on-hand quantity. It should be more or less in line with your planning parameters (i.e. if Min/Max is 20/40 and typical lead time demand is 10, then you should have roughly 10 to 40 units on hand at any given point in time.  Surprisingly, for many companies there is often a huge inconsistency. We have observed situations where the Min/Max setting is 20/40 but the on-hand inventory is 300+.  This indicates that whatever policy has been prescribed just isn’t being followed.   That’s a bigger problem.

10. What are you going to do next?

Demand forecasting and inventory stocking policy need to be well-defined processes that are understood and accepted by everybody involved.  There should be zero mystery.

To do this right, the demand and supply variability must be analyzed and used to compute the proper levels of safety stock.   Adding buffers without an implicit understanding of what each additional unit of buffer stock is buying you in terms of service is like arbitrarily throwing a handful of ingredients into a cake recipe.  A small change in ingredients can have a huge impact on what comes out of the oven – one bite too sweet but the next too sour.  It is the same with inventory management.  A little extra here, a little less there, and pretty soon you find yourself with costly excess inventory in some areas, painful shortages in others, no idea how you got there, and with little guidance on how to make things better.

Modern inventory optimization and demand planning software with its advanced analytics and strong basis in forecast analysis can help a good deal with this problem. But even the best software won’t help if it is used inconsistently.

To test software solutions via a series of empirical competition can be a considerable option. For forecasting / demand planning, a traditional “hold out” test in which 2014-2018 data are provided to software vendors and 2019 is held out for later comparison against forecasts provided by competing vendors. The company then measures forecast error and bias. This approach is advocated nearly universally for assessing forecast accuracy. It’s a good way to assess monthly or weekly forecast accuracy, but it is minimally useful if you have a different objective: Optimizing inventory.

In our last blog, we discussed how to pick a targeted service level. We indicated that just because you set a target (or a system recommends a target) doesn’t mean you’ll actually achieve the target. The right way to measure accuracy if you are interested in optimizing stock levels is to focus on the accuracy of the service level projection. This will account for both lead time demand and safety stock.

Setting a target service level is a strategic decision about inventory risk management. Inventory software does the tactical work by computing reorder points (a.k.a. mins) meant to achieve a user-defined target or that will achieve a system-calculated optimal target. But if the software uses the wrong demand model, the achieved service level will miss the target, sometimes significantly. The result of this error will be either shortages or inventory bloat, depending on the direction of the miss.

Forecasting is a means to an end. The end is to optimize inventory levels. Because demand is uncertain, companies that need to provide even moderate service levels must stock more than the forecast, often much more. But doesn’t low forecast error mean lower safety stock? The better my forecasts, the lower my inventory? Yes, true. But what matters when determining the required inventory are both accurate forecasts of the most likely demand and accurate estimates of the variability around the most likely demand.

Especially with long tailed, intermittent demand, traditional forecast accuracy assessments over a conventional 12 month forecast horizon miss the point three ways.

– First, the relevant time scale for inventory optimization is the replenishment lead time, which is usually much shorter than 12 months. Demand during lead times measured in days or weeks has volatility that gets averaged out over long forecast horizons. This is bad because factoring in the effect of volatility is essential to calculation of optimal reorder points.

– Second, forecast accuracy assessed over a multi-month forecast horizon focuses on the typical error in a typical month within the horizon. In contrast, inventory optimization requires a focus on cumulative demand, not period-by-period demand.

– Third, and most important is that forecast error metrics are focused on the middle of the demand distribution, aiming to estimate the most likely demand. But setting reorder points involves estimating high percentiles of the cumulative demand distribution over a lead time. Estimating the middle a bit better but having no clue about, say, the 95th percentile, is not helpful.

Consider this hypothetical example. If Vendor A forecasts 20 units with 110% error and Vendor B forecasts 22 units with 105% error, then Vendor B has an advantage in the forecasting game. But if you want a high service level and the demand is intermittent, you’ll have to stock a lot more than 20 or 22 units. Let’s assume you select Vendor B’s technology to plan stocking levels. You then notice that when planning reorder points to achieve a 95% service level, you often fall short – way more often that the expected 5% of the time. You come to realize that Vendor B’s approach completely underestimates the safety stock required to achieve the target service target. Focusing on vendors’ forecast error isn’t going to help. You will come to wish that you had verified Vendor A and B’s service level accuracy. Now you are stuck arbitrarily adjusting Vendor B’s service level targets to compensate for the shortfall.

So what’s needed in vendor competitions is assessment of their systems’ abilities to accurately forecast the inventory required to meet a given service level over an item’s replenishment lead time. Narrowly focusing on measuring forecast error is not appropriate if the mission is managing inventory. This is especially true for long tail items with intermittent demand or items that have medium to high volume but don’t have a demand distribution that looks like the classic “bell shaped curve” (Normal distribution).

The remainder of this blog explains how to test the accuracy of software’s service level calculations, so you can monitor the risk of missing your service level targets. We recommend this accuracy test over traditional “forecast versus actuals” tests because it provides much more insight into how reorder point recommendations will influence inventory levels and customer service.

Office staff are analyzing The Right Forecast Accuracy Metric for Inventory Planning

Service Level Defined

Consider a single inventory item. When inventory drops to or below the reorder point, a replenishment order is generated. This starts a period of risk that lasts as long as the replenishment lead time. During the period of risk, there might be enough incoming demands to create backorders or lost sales. The service level is the probability that there are no backorders or stockouts during the replenishment lead time. Critical items might be given very high target service levels, say 99%, whereas other items might have more relaxed targets, such as 75%. Whatever the target service level, it is best to hit that target.

Calculating Service Level

The service level for an individual item can only be estimated by repeated comparison of observed lead time demand against the calculated reorder point. These estimates take a lot of time: at least dozens of lead times. But fleet-wise service level can be estimated using data compiled over a single lead time.

Let’s do an example. Suppose you have demand histories for 1,000 items over 365 days and that (for simplicity) all items have 45-day lead times. For each item, follow these steps to estimate the fleet-wise achieved service level:

Step 1: Step aside (“hold out”) the most recent 45 days of demand (or however many days is closest to your typical lead times). Compute their sum, which is the most recent value of the actual lead time demand. This is the ground truth to be used to estimate the achieved service level.

Step 2: Use the prior 320 days of demand history to forecast the required inventory to hit a range of service level targets, say 90%, 95%, 97%, and 99%.

Step 3: Check whether the observed lead time demand is less than or equal to the reorder point. If it is, count this as a win; otherwise, count it as a loss. For instance, if the reorder point is 15 units but the most recent lead time demand is 10 units, then this is a win, since the reorder point is high enough to cover a lead time demand of 10 without any shortage. However, if the most recent lead time demand is 18 units, there would be a stockout, and 3 units would either be backordered or counted as lost sales.

Step 4: Working across all items, and all service level targets, tally the percentage of tests for each service level target that resulted in a win. This is the achieved service level. If the target was 90% and 853 of the 1,000 units record a win, then the achieved service level is 85.3%.

Example

Consider a real-world example. The data are daily demand histories of 590 medical supply items used in an internationally famous clinic. For simplicity, we assume each item has a lead time of 45 days. We evaluate target service levels of 70%, 90%, 95% and 99%.
We compare two demand models. The “Normal” model assumes that daily demand has a Normal (“bell-shaped”) distribution. This is the classic assumption used in most introductory textbooks on inventory control and in many software products. Classic though it may be, it is often an inappropriate model of demand for spare parts or supplies. The “Probability Forecast” model takes explicit account of the intermittent nature of demand.

Exhibit 1 shows the results. Column J shows the actual demand over the final 45 observations. The computed reorder points for the Advanced Model are shown in columns L-O.  The computed reorder points for the Normal model are not displayed.  Columns Q-T and V-Y hold the results of the tests for whether the reorder points were high enough to handle the lead time demands in column J.

The final results (yellow cells) show a clear difference between the Normal and Probability (Advanced) demand models. Both did a good job of hitting the 70% service level target, but estimating higher service levels is a more delicate calculation, and the Probability model does a much better job. For instance, the Normal model’s supposed 99% service level turned out to be only 94.4%, while the Probability model hit the target with a 98.5% achieved service level.

Implications

Utilizing the more accurate method achieved the targeted service level, while the less accurate method did not. If the less accurate method is used then real and costly business decisions will be made on the false assumption that a higher service level will be achieved. For example, if a Service Level Agreement (SLA) is based on these results and a 99% service level is committed to, the supplier would actually be five times more likely to stock out than planned (service level promised = 99% or 1% stockout risk vs. service level achieved = 94.5% or 5.5% stock out risk)! This means financial penalties will be incurred five times more often than expected.

Suppose that planners knew the target service level would not be met but were stuck using an inaccurate model. They would still need a way to increase inventory and achieve the desired level of service. What might they choose to do? We have observed situations where the planner enters a higher service level target than needed in order to “trick” the system into delivering the required service level. In the above example, the Normal model needed to have a 99.99% service level entered before it could achieve a target service level of 99%. This change resulted in achieving a 99% service but more than doubled the inventory investment when compared to the Advanced model.

Implementing a Service Level Accuracy Test

At Smart Software, we’ve encouraged many of our customers to conduct the test of service level accuracy as a way for them to assess our and other vendors’ claims during the software selection process. Missing the service level target has extremely costly implications resulting in substantial over stocks or under stocks.  So, test service level accuracy before deploying a solution to identify situations when the modeling fails. Don’t assume that you will achieve the service level you decide to target (or that the system recommends). To request an Excel spreadsheet that serves as a template for a service level accuracy test, email your contact information to info@smartcorp.com and enter “Accuracy Template” in the subject line.