ML Systems

# Prophet | February 9, 2021

By John Hallman - February 9, 2021

This week, we discussed Prophet, an automatic, open-source, time-series forecasting tool by Facebook.

Materials

Why Prophet?

• Time-series forecasting is highly relevant to anomaly detection, which Sisu is interested in doing.
• Prophet focuses on business time-series similar to ones we face at Sisu.
• Prophet is highly automatic (requires little tuning by analysts/users), which is something we want in our products too.

Nuggets

• Model time series as $y(t) = g(t) + s(t) + h(t) + e_t$, with $g$ being trend, $s$ seasonality, $h$ holidays/outlier days, and $e$ Gaussian noise.
• Model $g$ as a piecewise-linear function, with a sparse (Laplace) prior on a large set of initial possible breakpoints.
• Model $s$ as a sum of Fourier series, with each coefficient having a Gaussian prior.
• Model $h$ by manually selecting a set of business-relevant holidays/special days with Gaussian priors.
• Assume a prior on each component model and train using maximum a posteriori estimation (MAP). This allows us to directly obtain a posteriori estimates of the probability of new observations.
• New evaluation procedure for time series models:
• Choose a business-problem appropriate loss $d(\hat{y}(t), y(t))$.
• Specify a set of time horizons of interest, for example $H \in \{ 30, 60, 90 \}$ days.
• Define sub-horizon loss $\phi(T, h) = d(\hat{y}(T+h|T), y(T+h))$ for each $h \in (0, H]$.
• Loop over time series dataset and make a length $H$ forecast every $H/2$ time steps.
• With these forecasts, compute sub-horizon losses $\xi(h) = \mathbb{E}[\phi(T, h)]$.
• In general, we expect $\xi$ to be smooth and nondecreasing. If this is not the case, consider using isotonic regression or similar to smooth out the results.
• General advice for how to evaluate time-series models:
• Compare to both other advanced models (eg. ETS, TBATS), as well as robust baselines (last value, sample mean).
• Validate over varying time horizons $H$.
• Look at both the graph of forecasted time-series as well as the error curves obtained from the evaluation procedure described above.
• In general, time-series models will always do poorly in some situations. It is important to set up a system for detecting when these scenarios occur. Advice on how to do this:
• Look at error relative to robust baseline models (if similar or greater then something is off).
• If all models perform really badly on certain data points, likely outliers.
• If error across horizons $H$ sharply increase at a certain cutoff, there may have been a change in the data generation process, or the model may be miscalibrated.

Raw notes

If you like applying these kinds of methods practical ML problems, join our team.