Class SoftmaxCrossEntropyLoss

Namespace

NeuralNetworks.Losses

Assembly

NeuralNetworks.dll

Categorical Cross-Entropy loss combined with Softmax activation function.

public class SoftmaxCrossEntropyLoss : Loss<float[,]>

Inheritance

object

Loss<float[,]>

SoftmaxCrossEntropyLoss

Inherited Members

Loss<float[,]>.Prediction

Loss<float[,]>.Target

Loss<float[,]>.Forward(float[,], float[,])

Loss<float[,]>.Backward()

Loss<float[,]>.CalculateLoss()

Loss<float[,]>.CalculateLossGradient()

Loss<float[,]>.CloneBase()

Loss<float[,]>.Clone()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

Input: The predicted values (logits) and the target values (one-hot encoded) as 2D tensors.

Formula: L = -Σᵢ yᵢ · log(softmax(ŷᵢ)) = -Σᵢ yᵢ · log(exp(ŷᵢ) / Σⱼ exp(ŷⱼ))

Gradient Formula: ∂L/∂ŷᵢ = softmax(ŷᵢ) - yᵢ (remarkably simple due to combined softmax+cross-entropy derivative). This gradient formula is the same as in LogSoftmaxCrossEntropyLoss.

Description: This loss function combines the softmax activation with categorical cross-entropy loss, making it the standard choice for multi-class classification problems. It expects raw logits (unnormalized scores) as predictions and one-hot encoded vectors as targets. The softmax converts logits to a probability distribution, and the cross-entropy measures the dissimilarity between the predicted and target distributions.

Remarks: Combining softmax and cross-entropy into a single operation is numerically more stable than computing them separately, as it avoids potential numerical issues with log(0) or exp(large_number). The gradient simplifies beautifully to (softmax_output - target), which is both computationally efficient and numerically stable. This loss is widely used in classification tasks like MNIST digit recognition, ImageNet classification, and any multi-class classification problem. The eps (epsilon) parameter adds numerical stability to prevent log(0). For problems with many classes, label smoothing can be applied to the targets to improve generalization. This loss naturally handles class probabilities and encourages the model to be confident in its predictions.

Constructors

SoftmaxCrossEntropyLoss(float)

Categorical Cross-Entropy loss combined with Softmax activation function.

public SoftmaxCrossEntropyLoss(float eps = 1E-07)

Parameters

eps float

Small epsilon value added for numerical stability to prevent log(0). Default is 1e-7.

Remarks

Input: The predicted values (logits) and the target values (one-hot encoded) as 2D tensors.

Formula: L = -Σᵢ yᵢ · log(softmax(ŷᵢ)) = -Σᵢ yᵢ · log(exp(ŷᵢ) / Σⱼ exp(ŷⱼ))

Gradient Formula: ∂L/∂ŷᵢ = softmax(ŷᵢ) - yᵢ (remarkably simple due to combined softmax+cross-entropy derivative). This gradient formula is the same as in LogSoftmaxCrossEntropyLoss.

Description: This loss function combines the softmax activation with categorical cross-entropy loss, making it the standard choice for multi-class classification problems. It expects raw logits (unnormalized scores) as predictions and one-hot encoded vectors as targets. The softmax converts logits to a probability distribution, and the cross-entropy measures the dissimilarity between the predicted and target distributions.

Remarks: Combining softmax and cross-entropy into a single operation is numerically more stable than computing them separately, as it avoids potential numerical issues with log(0) or exp(large_number). The gradient simplifies beautifully to (softmax_output - target), which is both computationally efficient and numerically stable. This loss is widely used in classification tasks like MNIST digit recognition, ImageNet classification, and any multi-class classification problem. The eps (epsilon) parameter adds numerical stability to prevent log(0). For problems with many classes, label smoothing can be applied to the targets to improve generalization. This loss naturally handles class probabilities and encourages the model to be confident in its predictions.

Methods

CalculateLoss()

protected override float CalculateLoss()

Returns

float

CalculateLossGradient()

protected override float[,] CalculateLossGradient()

Returns

float[,]

ToString()

Returns a string that represents the current object.

public override string ToString()

Returns

string

A string that represents the current object.