Class TanhInputScaled2D

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Input-Scaled Hyperbolic Tangent activation function for 2D tensors.

public class TanhInputScaled2D : ActivationFunction<float[,], float[,]>

Inheritance

object

Operation

Operation<float[,], float[,]>

ActivationFunction<float[,], float[,]>

TanhInputScaled2D

Inherited Members

Operation<float[,], float[,]>.Input

Operation<float[,], float[,]>.Output

Operation<float[,], float[,]>.Forward(float[,], bool)

Operation<float[,], float[,]>.Backward(float[,])

Operation<float[,], float[,]>.CalcOutput(bool)

Operation<float[,], float[,]>.CalcInputGradient(float[,])

Operation<float[,], float[,]>.Forward(object, bool)

Operation<float[,], float[,]>.Backward(object)

Operation<float[,], float[,]>.GetOutputType()

Operation<float[,], float[,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = tanh(x / scale)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · (1 - f(x)²) / scale = ∂L/∂y · (1 - tanh²(x / scale)) / scale

Output Range: (-1, 1)

Description: A variant of the standard Tanh2D function that scales the input before applying the hyperbolic tangent. The scale parameter controls the "steepness" of the activation - larger scale values result in a smoother, more gradual transition, while smaller values create a sharper transition similar to a step function. This can help control gradient flow during training.

Remarks: The reciprocal of the scale is precomputed for efficiency, as multiplication is faster than division. Input scaling can help with gradient stability and allows for fine-tuning of the activation's sensitivity to input variations. When scale = 1, this reduces to the standard tanh function. Larger scale values (e.g., 2-10) can help prevent gradient saturation by keeping inputs in the more linear region of tanh.

Constructors

TanhInputScaled2D(float)

Input-Scaled Hyperbolic Tangent activation function for 2D tensors.

public TanhInputScaled2D(float scale)

Parameters

scale float

The scaling factor applied to the input before the tanh operation. Must be non-zero.

Remarks

Formula: f(x) = tanh(x / scale)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · (1 - f(x)²) / scale = ∂L/∂y · (1 - tanh²(x / scale)) / scale

Output Range: (-1, 1)

Description: A variant of the standard Tanh2D function that scales the input before applying the hyperbolic tangent. The scale parameter controls the "steepness" of the activation - larger scale values result in a smoother, more gradual transition, while smaller values create a sharper transition similar to a step function. This can help control gradient flow during training.

Remarks: The reciprocal of the scale is precomputed for efficiency, as multiplication is faster than division. Input scaling can help with gradient stability and allows for fine-tuning of the activation's sensitivity to input variations. When scale = 1, this reduces to the standard tanh function. Larger scale values (e.g., 2-10) can help prevent gradient saturation by keeping inputs in the more linear region of tanh.

Methods

CalcInputGradient(float[,])

Calculates input gradient.

protected override float[,] CalcInputGradient(float[,] outputGradient)

Parameters

outputGradient float[,]

Returns

float[,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.