Class ReLU2D

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Rectified Linear Unit (ReLU) activation function for 2D tensors.

public class ReLU2D : ActivationFunction<float[,], float[,]>

Inheritance

object

Operation

Operation<float[,], float[,]>

ActivationFunction<float[,], float[,]>

ReLU2D

Inherited Members

Operation<float[,], float[,]>.Input

Operation<float[,], float[,]>.Output

Operation<float[,], float[,]>.Forward(float[,], bool)

Operation<float[,], float[,]>.Backward(float[,])

Operation<float[,], float[,]>.CalcOutput(bool)

Operation<float[,], float[,]>.CalcInputGradient(float[,])

Operation<float[,], float[,]>.Forward(object, bool)

Operation<float[,], float[,]>.Backward(object)

Operation<float[,], float[,]>.GetOutputType()

Operation<float[,], float[,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = max(0, beta · x)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else 0

Output Range: [0, ∞) when beta > 0

Description: ReLU is one of the most widely used activation functions in deep learning. It outputs zero for negative inputs and scales positive inputs by the beta parameter (typically 1). ReLU is computationally efficient and helps mitigate the vanishing gradient problem, allowing for faster training of deep networks.

Remarks: With beta = 1 (default), this is the standard ReLU. The beta parameter allows for scaling the positive region, though this is rarely modified in practice. ReLU can suffer from the "dying ReLU" problem where neurons can become permanently inactive if they always receive negative inputs. Despite this, ReLU and its variants remain the go-to activation for most hidden layers in modern architectures. ReLU is non-differentiable at x = 0, but in practice, the gradient is defined as 0 or 1 at this point.

Constructors

ReLU2D(float)

Rectified Linear Unit (ReLU) activation function for 2D tensors.

public ReLU2D(float beta = 1)

Parameters

beta float

The scaling factor for positive inputs. Default is 1.

Remarks

Formula: f(x) = max(0, beta · x)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else 0

Output Range: [0, ∞) when beta > 0

Description: ReLU is one of the most widely used activation functions in deep learning. It outputs zero for negative inputs and scales positive inputs by the beta parameter (typically 1). ReLU is computationally efficient and helps mitigate the vanishing gradient problem, allowing for faster training of deep networks.

Remarks: With beta = 1 (default), this is the standard ReLU. The beta parameter allows for scaling the positive region, though this is rarely modified in practice. ReLU can suffer from the "dying ReLU" problem where neurons can become permanently inactive if they always receive negative inputs. Despite this, ReLU and its variants remain the go-to activation for most hidden layers in modern architectures. ReLU is non-differentiable at x = 0, but in practice, the gradient is defined as 0 or 1 at this point.

Methods

CalcInputGradient(float[,])

Calculates input gradient.

protected override float[,] CalcInputGradient(float[,] outputGradient)

Parameters

outputGradient float[,]

Returns

float[,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.