ML Systems

By John Hallman - February 9, 2021

This week, we discussed Prophet, an automatic, open-source, time-series forecasting tool by Facebook.

**Materials**

**Why Prophet?**

- Time-series forecasting is highly relevant to anomaly detection, which Sisu is interested in doing.
- Prophet focuses on business time-series similar to ones we face at Sisu.
- Prophet is highly automatic (requires little tuning by analysts/users), which is something we want in our products too.

**Nuggets**

- Model time series as $y(t) = g(t) + s(t) + h(t) + e_t$, with $g$ being trend, $s$ seasonality, $h$ holidays/outlier days, and $e$ Gaussian noise.
- Model $g$ as a piecewise-linear function, with a sparse (Laplace) prior on a large set of initial possible breakpoints.
- Model $s$ as a sum of Fourier series, with each coefficient having a Gaussian prior.
- Model $h$ by manually selecting a set of business-relevant holidays/special days with Gaussian priors.

- Assume a prior on each component model and train using maximum a posteriori estimation (MAP). This allows us to directly obtain a posteriori estimates of the probability of new observations.
- New evaluation procedure for time series models:
- Choose a business-problem appropriate loss $d(\hat{y}(t), y(t))$.
- Specify a set of time horizons of interest, for example $H \in \{ 30, 60, 90 \}$ days.
- Define sub-horizon loss $\phi(T, h) = d(\hat{y}(T+h|T), y(T+h))$ for each $h \in (0, H]$.
- Loop over time series dataset and make a length $H$ forecast every $H/2$ time steps.
- With these forecasts, compute sub-horizon losses $\xi(h) = \mathbb{E}[\phi(T, h)]$.
- In general, we expect $\xi$ to be smooth and nondecreasing. If this is not the case, consider using isotonic regression or similar to smooth out the results.

- General advice for how to evaluate time-series models:
- Compare to both other advanced models (eg. ETS, TBATS), as well as robust baselines (last value, sample mean).
- Validate over varying time horizons $H$.
- Look at both the graph of forecasted time-series as well as the error curves obtained from the evaluation procedure described above.
- In general, time-series models will always do poorly in some situations. It is important to set up a system for detecting when these scenarios occur. Advice on how to do this:
- Look at error relative to robust baseline models (if similar or greater then something is off).
- If all models perform really badly on certain data points, likely outliers.
- If error across horizons $H$ sharply increase at a certain cutoff, there may have been a change in the data generation process, or the model may be miscalibrated.

**Raw notes**

Presentation:

*If you like applying these kinds of methods practical ML problems, join our team.*