ml optimization

A list of optimization functions that can be used in ML training to reduce loss.

sigmoid

$$\text{sigmoid}(x)=\frac{1}{1+e^{-x}}$$

ReLU

$$\text{FFN}(x,W_1,W_2,b_1,b_2)=max(0,x{W}_1+b_1)W_2+b_2$$

A version in T5 without bias:

$${\text{FFN}}_{\text{ReLU}}(x,W_1,W_2)=max(x{W}_1,0)W_2$$

Swish

(Ramachandran et al., 2017) introduces an alternative to ReLU that works better on deeper models across different tasks.

$$\begin{gathered} f(x)=x\cdot \text{sigmoid}(\beta x) \\ \because \beta :\text{ constant parameter} \end{gathered}$$

Gated Linear Units and Variants

component-wise product of two linear transformations of the inputs, one of which is sigmoid-activated.

(Shazeer, 2020) introduces a few GELU activations to yield improvements in Transformers architecture.

$$\begin{array}{rl}\text{GLU}(x,W,V,b,c)& =\sigma (xW+b)\otimes (xV+c)\\ \text{Bilinear}(x,W,V,b,c)& =(xW+b)\otimes (xV+c)\end{array}$$

GLU in other variants:

$$\begin{array}{rl}\text{ReGLU}(x,W,V,b,c)& =\max (0,xW+b)\otimes (xV+c)\\ \text{GEGLU}(x,W,V,b,c)& =\text{GELU}(xW+b)\otimes (xV+c)\\ \text{SwiGLU}(x,W,V,b,c)& ={\text{Swish}}_{\beta }(xW+b)\otimes (xV+c)\end{array}$$

FFN for transformers layers would become:

$$\begin{array}{rl}{\text{FFN}}_{\text{GLU}}(x,W,V,W_2)& =(\sigma (xW)\otimes xV)W_2\\ {\text{FFN}}_{\text{Bilinear}}(x,W,V,W_2)& =(xW\otimes xV)W_2\\ {\text{FFN}}_{\text{ReGLU}}(x,W,V,W_2)& =(\max (0,xW)\otimes xV)W_2\\ {\text{FFN}}_{\text{GEGLU}}(x,W,V,W_2)& =(\text{GELU}(xW)\otimes xV)W_2\\ {\text{FFN}}_{\text{SwiGLU}}(x,W,V,W_2)& =({\text{Swish}}_{\beta }(xW)\otimes xV)W_2\end{array}$$

_note: reduce number of hidden units $d_{\text{ff}}$ (second dimension of $W$ and $V$ and the first dimension of $W_2$) by a factor of $\frac{2}{3}$ when comparing these layers

Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for Activation Functions. arXiv preprint arXiv:1710.05941 arxiv

Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202 arxiv