How Late Is Your Forecast to the Party? Measuring Trend-Detection Latency

In my last post, I argued that the real value of demand sensing is lead time on trend changes. A fair follow-up question is: how late is your current process today? Most, if not all, planners I ask this question have a feel for the answer (“we are usually a couple of months behind”) but very few have measured it. When did demand last shift meaningfully on one of your top items? And when did your forecast finally acknowledge the new level? One month later? Three? In this blog post, I will lay out a simple backtest that turns that feeling into a number. Rather than theorize, let us run the exercise on your own history.

Before we begin, a word on why one should bother. A number does two things that a feeling cannot. First, it gives you a baseline; if you later invest in weekly buckets or demand sensing, you can measure whether the latency actually shrank. Second, it can be dollarized. Budget approvers speak one language: money. That’s who decides whether your improvement project gets funded.

The Backtest, Step by Step

Step 1: Pick your items and your history. Pull 24-36 months of actuals and, importantly, the forecast of record for the same months. Start with 5-10 important items (think Pareto); there is no need to boil the ocean on the first pass. The forecast of record matters here: use the number the business actually planned against, warts and all. The statistical forecast that nobody looked at does not count.

Step 2: Mark the trend breaks in hindsight. Scan the actuals and flag every point where the level shifted by, say, 20+% and the new level persisted for 3+ months. This is easy to do looking backward; the exam answers are in the back of the book. The cutoffs of 20% and 3 months are somewhat arbitrary and you might want to use slightly different values; the point is to catch genuine level shifts and skip one-month blips. Do what works with your data.

Step 3: For each break, find the catch-up month. Walk forward from the break month and find the first month where the forecast of record was within, say, 10% of the new demand level. That is the month your process finally acknowledged the new reality.

Step 4: Compute the latency. Latency = catch-up month minus break month. For example, if demand for an item stepped up in April and the forecast first came within 10% of the new level in July, then the latency is July – April = 3 months. Do this for every break, then average.

All of this can be done in MS Excel in an afternoon. No new software required (imagine that).

A Worked Example

Let us make this concrete. The table below summarizes the results for a hypothetical exercise on 4 items.

Average latency = (2 + 3 + 1 + 4) / 4 = 10 / 4 = 2.5 months.

So, in this example, the process took 2.5 months on average to catch up to a level shift that, in hindsight, was plainly visible. In my experience, numbers in the 2-4 month range are common for monthly processes; your data may of course differ, which is precisely why one runs the exercise.

What Does 2.5 Months Cost?

Now, let us dollarize. Say item A sells 10,000 units per month at a margin of $10 per unit, and demand stepped up 30% (an extra 3,000 units per month). For the 2.5 months the plan was anchored to the old level, roughly 3,000 * 2.5 = 7,500 units of new demand were not planned for. If even a third to a half of that turned into lost or late sales, the margin impact is somewhere in the range of $25,000 to $37,500, for one item, on one break. Downward breaks cost you on the other side of the ledger, in excess inventory that, unlike a fine wine, does not improve with age. NOTE: these numbers are purely illustrative; the honest version of this calculation uses your margins, your service levels, and your actual lost-sales behavior.

A forecast that catches up 3 months after the shift is a guest arriving at the party after the food is gone. You are technically present; the moment has passed.

One Honest Complication

While you are in the data, also count the false alarms: how often did your process chase a change that evaporated a month or two later? A process that reacts instantly to everything will have a wonderful latency number and a terrible nervousness problem. Measuring the trade-off between the two properly requires a discussion of forecast bias and tracking signals, and that is the subject of the next post.

In summary:

• Pull 24-36 months of actuals plus the forecast of record for 5-10 important items.
• Mark level shifts of 20+% that persisted 3+ months (hindsight makes this easy).
• For each break, find the first month the forecast was within 10% of the new level.
• Latency = catch-up month minus break month; average across breaks.
• Dollarize the average latency to make the case for improving it.

How late is your process? If you run this exercise, I am interested in hearing what you find. We will continue the discussion in the next blog.