Regression analysis

18 Feb 2018 on Math

1. ordinary Least Square

alt text

- linear regression analysis

$y_{i}=\beta _{1}x_{i1}+\beta _{2}x_{i2}+\cdots +\beta _{p}x_{ip}+\varepsilon _{i}$

- bias augmentation

$X = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1D} \\ x_{21} & x_{22} & \cdots & x_{2D} \\ \vdots & \vdots & \vdots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{ND} \\ \end{bmatrix} \rightarrow X_a = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1D} \\ 1 & x_{21} & x_{22} & \cdots & x_{2D} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & x_{N1} & x_{N2} & \cdots & x_{ND} \\ \end{bmatrix}$

$f(x) = w_0 + w_1 x_1 + w_2 x_2 + \cdots + w_D x_D = \begin{bmatrix} 1 & x_1 & x_2 & \cdots & x_D \end{bmatrix} \begin{bmatrix} w_0 \\ w_1 \\ w_2 \\ \vdots \\ w_D \end{bmatrix}$

$f(x) = x_a^T w_a = w_a^T x_a$

2. assumption

$\operatorname {Var} [\,{\tilde {\beta }}\mid X\,]-\operatorname {Var} [\,{\hat {\beta }}\mid X\,]\geq 0$

normalatiy

${\hat {\beta }}\ \sim \ {\mathcal {N}}{\big (}\beta ,\ \sigma ^{2}(X^{\mathrm {T} }X)^{-1}{\big )}$

$s^{2}\ \sim \ {\frac {\sigma ^{2}}{n-p}}\cdot \chi _{n-p}^{2}$

3. Residual sum of Square estimation

$\displaystyle {\hat y}=\operatorname {arg min} f(x)=\sum _{i=1}^{n}(y_{i}-x_{i}^{\mathrm {T} }b)^{2}=(y-Xb)^{\mathrm {T} }(y-Xb)$

4. ANOVA

${\rm {ESS}} = {\rm {TSS}} + {\rm {RSS}}$

- TSS (total sum of square)

$\text{TSS} = \sum_i (y_i-\bar{y})^2 = (y - \bar{y})^T(y - \bar{y} )$

- ESS (explained sum of square)

$\text{ESS}=\sum_i (\hat{y}_i -\bar{\hat{y}})^2 = (\hat{y} - \bar{\hat{y}})^T(\hat{y} - \bar{\hat{y}})$

- RSS (residual sum of square)

$\text{RSS}=\sum_i (y_i - \hat{y}_i)^2\ = e^Te$

- R square

$R^{2}={\frac {\sum ({\hat {y}}_{i}-{\overline {y}})^{2}}{\sum (y_{i}-{\overline {y}})^{2}}}={\frac {\rm {ESS}}{\rm {TSS}}}=1-{\frac {\rm {RSS}}{\rm {TSS}}}$

$0 \leq R^2 \leq 1$

- adjust R square

$R_{adj}^2 = 1 - \frac{n-1}{n-K}(1-R^2) = \dfrac{(n-1)R^2 +1-K}{n-K}$

Reference

ordinary least square - wikipedia

gauss markov theorem - wikipedia