Linear regression

See also slides for curve fitting, regression, colab link

Transclude of thoughts/university/twenty-four-twenty-five/sfwr-4ml3/ols_and_kls.py

curve fitting.

how do we fit a distribution of data over a curve?

Given a set of $n$ data points $S=\{(x^i,y^i){\}}_{n=1}^n$

• $x\in {\mathbb{R}}^d$
• $y\in \mathbb{R}$ (or ${\mathbb{R}}^k$)

ols.

Ordinary Least Squares (OLS)

Let $\widehat{y^i}$ be the prediction of a model $X$, $d^i=\Vert y^i-\widehat{y^i}\Vert$ is the error, minimize ${\sum }_{i=1}^n(y^i-\widehat{y^i}{)}^2$

In the case of 1-D ordinary least square, the problems equates find $a,b\in \mathbb{R}$ to minimize ${\text{MIN}}_{a,b}{\sum }_{i=1}^n(a{x}^i+b-y^i{)}^2$

Optimal solution

$$\begin{array}{rl}a& =\frac{\overline{xy}-\overline{x}\cdot \overline{y}}{\overline{x^2}-(\overline{x}{)}^2}=\frac{\text{COV}(x,y)}{\text{Var}(x)}\\ b& =\overline{y}-a\overline{x}\end{array}$$

where $\overline{x}=\frac{1}{N}\sum x^i$, $\overline{y}=\frac{1}{N}\sum y^i$, $\overline{xy}=\frac{1}{N}\sum x^i{y}^i$, $\overline{x^2}=\frac{1}{N}\sum (x^i{)}^2$

Hyperplane equation

$\hat{y}=w_0+{\sum }_{j=1}^d{w}_j{x}_j$ where $w_0$ is the $y$ intercept (bias)

Homogenous hyperplane:

$$\begin{array}{rl}{w}_0& =0\\ \hat{y}& =\sum _{j=1}^d{w}_j{x}_j=\text{textlangle}w,x\text{textrangle}\\ & =w^{T}x\end{array}$$ $$X_{n\times d}=\left(\begin{array}{ccc}{x}_1^1& \cdots & x_d^1\\ ⋮& \ddots & ⋮\\ x_1^n& \cdots & x_d^n\end{array}\right),Y_{n\times 1}=\left(\begin{array}{c}{y}^1\\ ⋮\\ y^n\end{array}\right),W_{d\times 1}=\left(\begin{array}{c}{w}_1\\ ⋮\\ w_d\end{array}\right)$$ $$\begin{array}{rl}\text{Obj}& :\sum _{i=1}^n({\hat{y}}^i-y^i{)}^2=\sum _{i=1}^n(⟨w,x^i⟩-y^i{)}^2\\ & \\ \text{Def}& :\Delta =\left(\begin{array}{c}{\Delta }_1\\ ⋮\\ {\Delta }_n\end{array}\right)=\left(\begin{array}{ccc}{x}_1^1& \cdots & x_d^1\\ ⋮& \ddots & ⋮\\ x_1^n& \cdots & x_d^n\end{array}\right)\left(\begin{array}{c}{w}_1\\ ⋮\\ w_d\end{array}\right)-\left(\begin{array}{c}{y}^1\\ ⋮\\ y^n\end{array}\right)=\left(\begin{array}{c}{\hat{y}}^1-y^1\\ ⋮\\ {\hat{y}}^n-y^n\end{array}\right)\end{array}$$

minimize $w$

${\min }_{W\in {\mathbb{R}}^{d\times 1}}\Vert XW-Y{\Vert }_2^2$

Thus we can find $W^{\text{LS}}=(X^{T}X{)}^{-1}{X}^{T}Y$