Class ReLU4D

Namespace

NeuralNetworks.Operations.ActivationFunctions

Assembly

NeuralNetworks.dll

Rectified Linear Unit (ReLU) activation function for 4D tensors.

public class ReLU4D : ActivationFunction<float[,,,], float[,,,]>

Inheritance

object

Operation

Operation<float[,,,], float[,,,]>

ActivationFunction<float[,,,], float[,,,]>

ReLU4D

Inherited Members

Operation<float[,,,], float[,,,]>.Input

Operation<float[,,,], float[,,,]>.Output

Operation<float[,,,], float[,,,]>.Forward(float[,,,], bool)

Operation<float[,,,], float[,,,]>.Backward(float[,,,])

Operation<float[,,,], float[,,,]>.CalcOutput(bool)

Operation<float[,,,], float[,,,]>.CalcInputGradient(float[,,,])

Operation<float[,,,], float[,,,]>.Forward(object, bool)

Operation<float[,,,], float[,,,]>.Backward(object)

Operation<float[,,,], float[,,,]>.GetOutputType()

Operation<float[,,,], float[,,,]>.GetInputType()

Operation.SetRegistered()

Operation.GetOutputType()

Operation.GetInputType()

Operation.Forward(object, bool)

Operation.Backward(object)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Formula: f(x) = max(0, beta · x)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else 0

Output Range: [0, ∞) when beta > 0

Description: ReLU variant for 4D tensors, primarily used in convolutional neural networks (CNNs) where tensors typically represent [batch, channels, height, width]. Provides the same benefits as standard ReLU - computational efficiency, mitigation of vanishing gradients, and sparse activation - while supporting batch processing of multi-channel image data.

Remarks: Most commonly used in CNNs for computer vision tasks. The beta parameter (default 1) allows scaling of positive inputs, though the standard value is almost universally used. Like other ReLU variants, it can suffer from the dying ReLU problem, but alternatives like Leaky ReLU or ELU can address this if needed. The 4D implementation enables efficient batch processing typical in modern deep learning frameworks.

Constructors

ReLU4D(float)

Rectified Linear Unit (ReLU) activation function for 4D tensors.

public ReLU4D(float beta = 1)

Parameters

beta float

The scaling factor for positive inputs. Default is 1.

Remarks

Formula: f(x) = max(0, beta · x)

Input Gradient Formula: ∂L/∂x = ∂L/∂y · beta if x > 0, else 0

Output Range: [0, ∞) when beta > 0

Description: ReLU variant for 4D tensors, primarily used in convolutional neural networks (CNNs) where tensors typically represent [batch, channels, height, width]. Provides the same benefits as standard ReLU - computational efficiency, mitigation of vanishing gradients, and sparse activation - while supporting batch processing of multi-channel image data.

Remarks: Most commonly used in CNNs for computer vision tasks. The beta parameter (default 1) allows scaling of positive inputs, though the standard value is almost universally used. Like other ReLU variants, it can suffer from the dying ReLU problem, but alternatives like Leaky ReLU or ELU can address this if needed. The 4D implementation enables efficient batch processing typical in modern deep learning frameworks.

Methods

CalcInputGradient(float[,,,])

Calculates input gradient.

protected override float[,,,] CalcInputGradient(float[,,,] outputGradient)

Parameters

outputGradient float[,,,]

Returns

float[,,,]

Remarks

Based on outputGradient, calculates changes in input.

CalcOutput(bool)

Computes output.

protected override float[,,,] CalcOutput(bool inference)

Parameters

inference bool

Returns

float[,,,]

ToString()

Returns a string that represents the current operation.

public override string ToString()

Returns

string

A string that represents the current operation.

Remarks

It is used in a layer/model description.