ML Systems

Prophet | February 9, 2021

By John Hallman - February 9, 2021

This week, we discussed Prophet, an automatic, open-source, time-series forecasting tool by Facebook.

Materials

Why Prophet?

  • Time-series forecasting is highly relevant to anomaly detection, which Sisu is interested in doing.
  • Prophet focuses on business time-series similar to ones we face at Sisu.
  • Prophet is highly automatic (requires little tuning by analysts/users), which is something we want in our products too.

Nuggets

  • Model time series as y(t)=g(t)+s(t)+h(t)+ety(t) = g(t) + s(t) + h(t) + e_t, with gg being trend, ss seasonality, hh holidays/outlier days, and ee Gaussian noise.
    • Model gg as a piecewise-linear function, with a sparse (Laplace) prior on a large set of initial possible breakpoints.
    • Model ss as a sum of Fourier series, with each coefficient having a Gaussian prior.
    • Model hh by manually selecting a set of business-relevant holidays/special days with Gaussian priors.
  • Assume a prior on each component model and train using maximum a posteriori estimation (MAP). This allows us to directly obtain a posteriori estimates of the probability of new observations.
  • New evaluation procedure for time series models:
    • Choose a business-problem appropriate loss d(y^(t),y(t))d(\hat{y}(t), y(t)).
    • Specify a set of time horizons of interest, for example H{30,60,90}H \in \{ 30, 60, 90 \} days.
    • Define sub-horizon loss ϕ(T,h)=d(y^(T+hT),y(T+h))\phi(T, h) = d(\hat{y}(T+h|T), y(T+h)) for each h(0,H]h \in (0, H].
    • Loop over time series dataset and make a length HH forecast every H/2H/2 time steps.
    • With these forecasts, compute sub-horizon losses ξ(h)=E[ϕ(T,h)]\xi(h) = \mathbb{E}[\phi(T, h)].
    • In general, we expect ξ\xi to be smooth and nondecreasing. If this is not the case, consider using isotonic regression or similar to smooth out the results.
  • General advice for how to evaluate time-series models:
    • Compare to both other advanced models (eg. ETS, TBATS), as well as robust baselines (last value, sample mean).
    • Validate over varying time horizons HH.
    • Look at both the graph of forecasted time-series as well as the error curves obtained from the evaluation procedure described above.
    • In general, time-series models will always do poorly in some situations. It is important to set up a system for detecting when these scenarios occur. Advice on how to do this:
      • Look at error relative to robust baseline models (if similar or greater then something is off).
      • If all models perform really badly on certain data points, likely outliers.
      • If error across horizons HH sharply increase at a certain cutoff, there may have been a change in the data generation process, or the model may be miscalibrated.

Raw notes

If you like applying these kinds of methods practical ML problems, join our team.