Regression Analysis Regularization

23 Feb 2018 on Math

1. Ridge

L2 regularization

$\text{cost} &=& \sum e_i^2 + \lambda \sum w_i^2$

2. LASSO (Least Absolute Shrinkage and Selection Operator)

L1 regularization

$\text{cost} &=& \sum e_i^2 + \lambda \sum | w_i |$

3. ridge and lasso geographical understanding

- finding RSS as constraint satisfied

alt text

- finding RSS when gamma(constraint) changes

alt text

- linear regression vs ridge regression

alt text

4. Elastic Net

$\text{cost} &=& \sum e_i^2 + \lambda_1 \sum | w_i | + \lambda_2 \sum w_i^2$

alt text

Reference