Class LogSoftmaxCrossEntropyLoss
- Namespace
- NeuralNetworks.Losses
- Assembly
- NeuralNetworks.dll
Categorical Cross-Entropy loss with Log-Softmax activation (also known as Negative Log-Likelihood Loss).
public class LogSoftmaxCrossEntropyLoss : Loss<float[,]>- Inheritance
-
LogSoftmaxCrossEntropyLoss
- Inherited Members
Remarks
Input: The predicted values (logits) and the target values (one-hot encoded) as 2D tensors.
Formula: L = -Σᵢ yᵢ · log_softmax(ŷᵢ) = -Σᵢ yᵢ · (ŷᵢ - log(Σⱼ exp(ŷⱼ)))
Gradient Formula: ∂L/∂ŷᵢ = softmax(ŷᵢ) - yᵢ (identical to SoftmaxCrossEntropyLoss gradient)
Description: This loss function is mathematically equivalent to SoftmaxCrossEntropyLoss but uses a log-softmax formulation that can provide better numerical stability in some scenarios. It expects raw logits as predictions and one-hot encoded vectors as targets. Commonly used in multi-class classification tasks, particularly when numerical precision is critical or when working with very large class spaces.
Remarks: The log-softmax formulation: log_softmax(x) = x - log(Σ exp(x)) is often more numerically stable than computing softmax followed by log, especially when dealing with very large or very small logit values. The gradient computation remains identical to SoftmaxCrossEntropyLoss (softmax_output - target), demonstrating the mathematical equivalence of the two approaches. This loss is also known as Negative Log-Likelihood (NLL) loss when used with log-softmax outputs. It's particularly useful in frameworks where log-probabilities are preferred over probabilities (e.g., for computational efficiency in certain architectures). PyTorch's CrossEntropyLoss is similar, applying log-softmax internally. The choice between this and SoftmaxCrossEntropyLoss is often a matter of implementation preference and numerical considerations rather than mathematical differences.
Constructors
LogSoftmaxCrossEntropyLoss()
Categorical Cross-Entropy loss with Log-Softmax activation (also known as Negative Log-Likelihood Loss).
public LogSoftmaxCrossEntropyLoss()Remarks
Input: The predicted values (logits) and the target values (one-hot encoded) as 2D tensors.
Formula: L = -Σᵢ yᵢ · log_softmax(ŷᵢ) = -Σᵢ yᵢ · (ŷᵢ - log(Σⱼ exp(ŷⱼ)))
Gradient Formula: ∂L/∂ŷᵢ = softmax(ŷᵢ) - yᵢ (identical to SoftmaxCrossEntropyLoss gradient)
Description: This loss function is mathematically equivalent to SoftmaxCrossEntropyLoss but uses a log-softmax formulation that can provide better numerical stability in some scenarios. It expects raw logits as predictions and one-hot encoded vectors as targets. Commonly used in multi-class classification tasks, particularly when numerical precision is critical or when working with very large class spaces.
Remarks: The log-softmax formulation: log_softmax(x) = x - log(Σ exp(x)) is often more numerically stable than computing softmax followed by log, especially when dealing with very large or very small logit values. The gradient computation remains identical to SoftmaxCrossEntropyLoss (softmax_output - target), demonstrating the mathematical equivalence of the two approaches. This loss is also known as Negative Log-Likelihood (NLL) loss when used with log-softmax outputs. It's particularly useful in frameworks where log-probabilities are preferred over probabilities (e.g., for computational efficiency in certain architectures). PyTorch's CrossEntropyLoss is similar, applying log-softmax internally. The choice between this and SoftmaxCrossEntropyLoss is often a matter of implementation preference and numerical considerations rather than mathematical differences.
Methods
CalculateLoss()
protected override float CalculateLoss()Returns
CalculateLossGradient()
protected override float[,] CalculateLossGradient()Returns
- float[,]
ToString()
Returns a string that represents the current object.
public override string ToString()Returns
- string
A string that represents the current object.