What is cross-entropy loss?

Cross-entropy loss is a measure of performance for a classification model. If a classification model correctly predicts the class, the cross-entropy loss will be 0. And if the classification model deviates from predicting the class correctly, the cross-entropy loss value will be more.

For a binary classification problem, the cross-entropy loss can be given by the following formula:

Here, there are two classes 0 and 1. If the observation belongs to class 1, y is 1. Otherwise, y is 0. And p is the predicted probability that an observation belongs to class 1.

And, for a multiclass classification problem, the cross-entropy loss is given by the formula:

Here, N is the total number of observations. If the observation o is of class c, then yo,c is 1. Otherwise, yo,c is 0. po,c is the predicted probability that observation o is of class c.

How to calculate the cross-entropy loss using sklearn in Python?

We can use the following Python code to calculate the cross-entropy loss using sklearn. Please note that for a binary classification problem, the cross-entropy loss is also known as log loss.

from sklearn.metrics import log_loss


y_true = [0, 0, 1, 1]
y_pred = [[.9, .1], [.9, .1], [.15, .85], [.2, .8]]

log_loss = log_loss(y_true, y_pred)
print("Log Loss: ", log_loss)

Here, y_true indicates the true values of the target variable. y_pred indicates the predicted probabilities. For example, [.9, .1] indicates that the probability that the target variable is of class 0 is .9 and the probability that the target variable is of class 1 is 0.1.

The output of the given program will be:

Log Loss:  0.1490958780319093