Class Softplus

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Softplus activation function.

public class Softplus : ActivationFunction<float[,], float[,]>

Inheritance

object

Operation

Operation<float[,], float[,]>

ActivationFunction<float[,], float[,]>

Softplus

Inherited Members

Operation<float[,], float[,]>.Input

Operation<float[,], float[,]>.Output

Operation<float[,], float[,]>.Forward(float[,], bool)

Operation<float[,], float[,]>.Backward(float[,])

Operation<float[,], float[,]>.CalcOutput(bool)

Operation<float[,], float[,]>.CalcInputGradient(float[,])

Operation<float[,], float[,]>.Forward(object, bool)

Operation<float[,], float[,]>.Backward(object)

Operation<float[,], float[,]>.GetOutputType()

Operation<float[,], float[,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = log(1 + exp(x)) = ln(1 + e^x)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · exp(x)/(1 + exp(x)) = ∂L/∂y · σ(x)

Output Range: (0, ∞)

Description: Softplus is a smooth, differentiable approximation of the ReLU2D function. Unlike ReLU, which has a sharp corner at zero, Softplus is continuously differentiable everywhere. It asymptotically approaches zero for large negative values and linearly increases for large positive values (f(x) ≈ x when x >> 0). Often used when a smooth activation is preferred over the non-differentiability of ReLU.

Remarks: The derivative of Softplus is the sigmoid function: f'(x) = σ(x). For large positive x, Softplus approximates ReLU closely (f(x) ≈ x), but for x near zero, it provides smooth transitions. This can be beneficial for optimization but comes at the cost of computational expense due to the exponential and logarithm operations. Softplus is sometimes used in variational autoencoders (VAEs) and other probabilistic models. Note that for very large x, numerical stability must be considered to avoid overflow in exp(x).

Methods

CalcInputGradient(float[,])

Calculates input gradient.

protected override float[,] CalcInputGradient(float[,] outputGradient)

Parameters

outputGradient float[,]

Returns

float[,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.