Activation function definition

An activation function is a function that operates on a neuron in a neural network and is responsible for mapping the neuron's input to its output.

It enables the neural network model to have nonlinear differentiation capabilities.AnyApproximate any nonlinear function. This allows the neural network model to be applied to a wide range of nonlinear models.

Its existence enables the neural network model to distinguish nonlinear functions, which also allows the neural network model to be applied to many nonlinear models.

Activation function properties

Nonlinear： For linear activation functions, a two-layer neural network can satisfy almost all functions; but this condition does not hold for the identity activation function. If MLP uses the identity activation function, then the entire neural network is equivalent to a single-layer neural network.

Differentiability：It plays an important role in gradient-based optimization methods. (Differentiability means that the function can be differentiated in all directions, and differentiability is greater than or equal to differentiability)

Monotonicity： When the activation function is a monotonic function, the single-layer network appears as a convex function.

f(x)≈x: When the activation function satisfies this formula, the initialization parameter is a very small random value, then the training efficiency of the neural network will be improved; but when it does not satisfy this formula, it is necessary to set the initial value specifically.

Output value range: When the output value of the activation function is finite, the gradient-based optimization method will be more stable; when the output value of the activation function is infinite, the model training will be more efficient.

Common activation functions

• Sigmoid
• tanh
• ReLU
• Maxout
• ELU

Measurement Method

Usually, there are the following experimental methods to measure which activation function to use:

1. Can the gradient be effectively propagated?
2. Is the mean 0?
3. Is the computational cost high?

In a neural network, the activation function can determine whether a neuron is activated, whether the information is valuable, or whether it should be discarded.