ML Systems

Prophet | February 9, 2021

By John Hallman - February 9, 2021

This week, we discussed Prophet, an automatic, open-source, time-series forecasting tool by Facebook.

Materials

Why Prophet?

  • Time-series forecasting is highly relevant to anomaly detection, which Sisu is interested in doing.
  • Prophet focuses on business time-series similar to ones we face at Sisu.
  • Prophet is highly automatic (requires little tuning by analysts/users), which is something we want in our products too.

Nuggets

  • Model time series as y(t)=g(t)+s(t)+h(t)+ety(t) = g(t) + s(t) + h(t) + e_t, with gg being trend, ss seasonality, hh holidays/outlier days, and ee Gaussian noise.
    • Model gg as a piecewise-linear function, with a sparse (Laplace) prior on a large set of initial possible breakpoints.
    • Model ss as a sum of Fourier series, with each coefficient having a Gaussian prior.
    • Model hh by manually selecting a set of business-relevant holidays/special days with Gaussian priors.
  • Assume a prior on each component model and train using maximum a posteriori estimation (MAP). This allows us to directly obtain a posteriori estimates of the probability of new observations.
  • New evaluation procedure for time series models:
    • Choose a business-problem appropriate loss d(y^(t),y(t))d(\hat{y}(t), y(t)).
    • Specify a set of time horizons of interest, for example H{30,60,90}H \in \{ 30, 60, 90 \} days.
    • Define sub-horizon loss ϕ(T,h)=d(y^(T+hT),y(T+h))\phi(T, h) = d(\hat{y}(T+h|T), y(T+h)) for each h(0,H]h \in (0, H].
    • Loop over time series dataset and make a length HH forecast every H/2H/2 time steps.
    • With these forecasts, compute sub-horizon losses ξ(h)=E[ϕ(T,h)]\xi(h) = \mathbb{E}[\phi(T, h)].
    • In general, we expect ξ\xi to be smooth and nondecreasing. If this is not the case, consider using isotonic regression or similar to smooth out the results.
  • General advice for how to evaluate time-series models:
    • Compare to both other advanced models (eg. ETS, TBATS), as well as robust baselines (last value, sample mean).
    • Validate over varying time horizons HH.
    • Look at both the graph of forecasted time-series as well as the error curves obtained from the evaluation procedure described above.
    • In general, time-series models will always do poorly in some situations. It is important to set up a system for detecting when these scenarios occur. Advice on how to do this:
      • Look at error relative to robust baseline models (if similar or greater then something is off).
      • If all models perform really badly on certain data points, likely outliers.
      • If error across horizons HH sharply increase at a certain cutoff, there may have been a change in the data generation process, or the model may be miscalibrated.

Raw notes

If you like applying these kinds of methods practical ML problems, join our team.


Read more

Stability Selection (Error Control) | January 25, 2021

A look at Stability Selection, an extremely general finite sample control technique for structure estimation. The main result we'll drive towards is an error control bound for the expected number of false discoveries.

Read more

SCANN | August 3, 2020

This week, we take a look at ScaNN (Scalable Nearest Neighbors), a method for efficient vector similarity search at scale.

Read more