--- date: '2024-12-10' description: and what is she descending from, really? id: gradient descent modified: 2026-06-05 15:08:24 GMT-04:00 tags: - ml title: gradient descent created: '2024-12-10' published: '2024-12-10' pageLayout: default slug: thoughts/gradient-descent permalink: https://aarnphm.xyz/thoughts/gradient-descent.md generator: quartz: v4.6.0 hostedProvider: Cloudflare baseUrl: aarnphm.xyz full: https://aarnphm.xyz/llms-full.txt --- Let us define the standard [[thoughts/Vector calculus#gradient|gradient]] descent approach for minimizing a differentiable [[thoughts/Convex function|convex function]] In a sense, gradient of a differential function $f : \mathbb{R}^d \to \mathbb{R}$ at $w$ is the vector of partial derivatives: $$ \nabla f(w) = (\frac{\partial f(w)}{\partial w[1]},\ldots,\frac{\partial f(w)}{\partial w[d]}) $$ > \[!math\] 1. intuition > > $$ > x_{t+1} = x_t - \alpha \nabla f(x_t) > $$ > >

## idea - initialize $w^0$ - iteratively for each t=1: - $w^{t+1} = w^t - \alpha \nabla f(w^{(t)})$ intuition: It should convert to a local minimum depending on learning rate $\alpha$ > not necessarily global minimum But guaranteed global minimum for [[thoughts/Convex function|convex functions]] ## calculate the gradient $$ \begin{aligned} E(w) &= L(w) + \lambda \text{Reg}(w) \\[8pt] L(w) &= \sum_{i} l(f_w(x^i), y^i) \\[8pt] \nabla_w (L(w)) &= \sum_{i} \nabla_w (l(f_w(x^i), y^i)) --ready-check-timeout-sec \end{aligned} $$ trick: split into mini-batch of gradient $$ \begin{aligned} \nabla_w^j &= \sum_{(x,y) \in S_j} \nabla_W (l(f_W(x), y))\\[8pt] &= \sum_{j} \nabla_W^j \end{aligned} $$ ![[thoughts/university/twenty-four-twenty-five/sfwr-4ml3/Stochastic gradient descent|SGD]] ## analysis of GD for Convex-Lipschitz Functions