Activation Function Cheat Sheet

Activation Function Cheat Sheet

Common Functions

ReLU

$$ReLU(x) = max(0,x)$$


Sigmoid

$$Sigmoid(x) = \frac{1}{1+exp(-x)}$$


tanh

$$tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$$


Softmax

$$Softmax(x_{i}) = \frac{exp(x_{i})}{\sum_{j} exp(x_{j})}$$
Does not work with NLLLoss


LogSoftmax

$$Softmax(x_{i}) = log\bigg( \frac{exp(x_{i})}{\sum_{j} exp(x_{j})}\bigg)$$


Others Functions

ELU

$$ELU(x) = max(0,x) + min(0,\alpha * (exp(x)-1))$$


HardShrink

$$HardShrink(x)=\begin{cases}
x, & \text{if}\ x > \lambda \\
x, & \text{if}\ x < -\lambda \\ 0, & \text{otherwise} \end{cases}$$


HardTanh

$$HardTanh(x)=\begin{cases} 1, & \text{if}\ x > 1 \\
-1, & \text{if}\ x < -1 \\ x, & \text{otherwise} \end{cases}$$


LeakyReLU

$$LeakyReLU(x) = max(0,x) + negative\_slope * min(0,x)$$


LogSigmoid

$$LogSigmoid(x) = log\bigg(\frac{1}{1+exp(-x)}\bigg)$$


PReLU

$$PReLU(x) = max(0,x) + a * min(0,x)$$


ReLU6

$$ReLU6(x) = min(max(0,x),6)$$


SELU

$$SELU(x) = scale * (max(0,x) + min(0,\alpha * (exp(x)-1)))$$


Softplus

$$Softplus(x) = \frac{1}{\beta} * log(1 + exp(\beta * x))$$


Softshrink

$$SoftShrink(x)=\begin{cases} x – \lambda, & \text{if}\ x > \lambda \\
x + \lambda, & \text{if}\ x < -\lambda \\
0, & \text{otherwise}\end{cases}$$


Softsign

$$SoftSign(x) = \frac{x}{1+|x|}$$


Tanhshrink

$$Tanhshrink(x) = x – tanh(x)$$