Class Softsign

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Softsign activation function.

public class Softsign : ActivationFunction<float[,], float[,]>

Inheritance

object

Operation

Operation<float[,], float[,]>

ActivationFunction<float[,], float[,]>

Softsign

Inherited Members

Operation<float[,], float[,]>.Input

Operation<float[,], float[,]>.Output

Operation<float[,], float[,]>.Forward(float[,], bool)

Operation<float[,], float[,]>.Backward(float[,])

Operation<float[,], float[,]>.CalcOutput(bool)

Operation<float[,], float[,]>.CalcInputGradient(float[,])

Operation<float[,], float[,]>.Forward(object, bool)

Operation<float[,], float[,]>.Backward(object)

Operation<float[,], float[,]>.GetOutputType()

Operation<float[,], float[,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = x / (1 + |x|)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · 1 / (1 + |x|)²

Output Range: (-1, 1)

Description: Softsign is a smooth activation function similar to Tanh2D but with a different rate of convergence to its asymptotes. It maps inputs to the range (-1, 1) and is defined as x divided by (1 + |x|). Softsign approaches its limits more gradually than tanh, which can lead to better gradient flow in some scenarios.

Remarks: Compared to tanh, Softsign converges polynomially to its asymptotes (±1), while tanh converges exponentially. This means Softsign has a gentler slope near the extremes, potentially mitigating vanishing gradient issues better than tanh. However, it is computationally more expensive than ReLU and less commonly used in modern architectures. The absolute value operation makes it non-differentiable at x = 0, though in practice the gradient is typically defined as 1 at this point. Softsign can be useful in recurrent networks or when a bounded, smooth activation is desired.

Methods

CalcInputGradient(float[,])

Calculates input gradient.

protected override float[,] CalcInputGradient(float[,] outputGradient)

Parameters

outputGradient float[,]

Returns

float[,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.