Class LeakyReLU4D

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Leaky Rectified Linear Unit (Leaky ReLU) activation function for 4D tensors.

public class LeakyReLU4D : ActivationFunction<float[,,,], float[,,,]>

Inheritance

object

Operation

Operation<float[,,,], float[,,,]>

ActivationFunction<float[,,,], float[,,,]>

LeakyReLU4D

Inherited Members

Operation<float[,,,], float[,,,]>.Input

Operation<float[,,,], float[,,,]>.Output

Operation<float[,,,], float[,,,]>.Forward(float[,,,], bool)

Operation<float[,,,], float[,,,]>.Backward(float[,,,])

Operation<float[,,,], float[,,,]>.CalcOutput(bool)

Operation<float[,,,], float[,,,]>.CalcInputGradient(float[,,,])

Operation<float[,,,], float[,,,]>.Forward(object, bool)

Operation<float[,,,], float[,,,]>.Backward(object)

Operation<float[,,,], float[,,,]>.GetOutputType()

Operation<float[,,,], float[,,,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = max(alfa · x, beta · x) = beta · x if x > 0, else alfa · x

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else ∂L/∂y · alfa

Output Range: (-∞, ∞)

Description: Leaky ReLU addresses the "dying ReLU" problem by allowing a small, non-zero gradient when the input is negative. Instead of outputting zero for negative inputs, it outputs alfa · x (typically with alfa = 0.01). For positive inputs, it behaves like standard ReLU with scaling factor beta. This ensures that all neurons can continue to learn even if they receive negative inputs.

Remarks: The typical configuration uses alfa = 0.01 and beta = 1, resulting in f(x) = x for x > 0 and f(x) = 0.01x for x ≤ 0. This small negative slope prevents neurons from dying while maintaining most of ReLU's computational benefits. Variants include Parametric ReLU (PReLU) where alfa is learned during training, and Randomized ReLU (RReLU) where alfa is randomly sampled. Leaky ReLU generally performs better than standard ReLU in practice, especially in deeper networks.

Constructors

LeakyReLU4D(float, float)

Leaky Rectified Linear Unit (Leaky ReLU) activation function for 4D tensors.

public LeakyReLU4D(float alfa = 0.01, float beta = 1)

Parameters

alfa float

The slope for negative inputs. Default is 0.01.

beta float

The slope for positive inputs. Default is 1.

Remarks

Formula: f(x) = max(alfa · x, beta · x) = beta · x if x > 0, else alfa · x

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else ∂L/∂y · alfa

Output Range: (-∞, ∞)

Description: Leaky ReLU addresses the "dying ReLU" problem by allowing a small, non-zero gradient when the input is negative. Instead of outputting zero for negative inputs, it outputs alfa · x (typically with alfa = 0.01). For positive inputs, it behaves like standard ReLU with scaling factor beta. This ensures that all neurons can continue to learn even if they receive negative inputs.

Remarks: The typical configuration uses alfa = 0.01 and beta = 1, resulting in f(x) = x for x > 0 and f(x) = 0.01x for x ≤ 0. This small negative slope prevents neurons from dying while maintaining most of ReLU's computational benefits. Variants include Parametric ReLU (PReLU) where alfa is learned during training, and Randomized ReLU (RReLU) where alfa is randomly sampled. Leaky ReLU generally performs better than standard ReLU in practice, especially in deeper networks.

Methods

CalcInputGradient(float[,,,])

Calculates input gradient.

protected override float[,,,] CalcInputGradient(float[,,,] outputGradient)

Parameters

outputGradient float[,,,]

Returns

float[,,,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,,,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,,,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.