Autoregression

Autoregression, at its core, is a statistical technique where the current value of a sequence (or “time series”) is predicted based on a linear combination of its own previous values. The term “auto” refers to “self,” meaning the regression is performed on the variable itself, using its past observations as predictors.

Here’s a breakdown of the key aspects:

• Self-Referential: The fundamental idea is that there’s a dependency between an observation at a given time and observations at earlier times within the same sequence. For example, today’s stock price might be influenced by yesterday’s price, the price from two days ago, and so on.
• Time Series Analysis: Autoregression is most commonly applied in time series analysis, where data points are collected sequentially over time (e.g., daily temperatures, monthly sales figures, stock prices).
• Lagged Values: The “previous values” used in the prediction are referred to as “lagged” values. An autoregressive model of order p, denoted as AR(p), means that the current value is predicted using the p immediately preceding values.
  • For instance, an AR(1) model uses only the immediately preceding value to predict the current one.
  • An AR(2) model uses the two preceding values.
• Linear Relationship: Autoregressive models typically assume a linear relationship between the current value and its past values. This relationship is quantified by coefficients, which indicate the strength and direction of the influence of each lagged value.
• Equation Form: A general autoregressive model of order p can be expressed as: $X^t=c+{\varphi }^1{X}^{t-1}+{\varphi }^2{X}^{t-2}+\dots +{\varphi }^p{X}^{t-p}+{ϵ}^t$ Where:
  • $X^t$ is the value at time t.
  • $c$ is a constant term.
  • ${\varphi }^1,{\varphi }^2,\dots ,{\varphi }^p$ are the autoregressive coefficients, representing the weights or influence of each lagged value.
  • $X^{t-1},X^{t-2},\dots ,X^{t-p}$ are the past values of the series at lags 1, 2, …, p.
  • ${ϵ}^t$ is a random error term (often assumed to be white noise).
• Applications: Autoregressive models are widely used for:
  • Forecasting: Predicting future values of a time series (e.g., predicting future stock prices, weather patterns, or economic indicators).
  • Understanding Temporal Dependencies: Identifying and quantifying how past events influence current and future events within a sequence.
  • Language Models: In the context of large language models (LLMs), “autoregressive” means they generate text token by token, sequentially predicting the next word or sub-word unit based on all the previously generated tokens in the sequence. This is why LLMs are often prone to “hallucinations” or repetitive text, as they don’t have a holistic view of the entire output until it’s generated.

In essence, autoregression leverages the inherent correlation within a sequence of data to make predictions, assuming that the past provides valuable information for understanding the present and forecasting the future.