To measure human log loss we need to ask people to predict a probability distribution for each image, e.g. (20%, 70%, 10%). So, whenever he predicted class 1, we assign it to the (3/7, 4/7, 0/7) probability distribution, for class 2 – (5/26, 18/26, 3/26) and for class 3 – (1/17, 4/17, 12/17). import numpy as np from sklearn.metrics import confusion_matrix # label - ground truth labels # predictions - prediction labels def entropy(x, epsilon=1e-6): # assumes x is normalized return (- x * np.log(x + epsilon)).sum() def conditional_entropy(mat): mat = mat / mat.sum() return entropy(mat) - entropy(mat.sum(axis=0)) print(conditional_entropy(confusion_matrix(label, prediction)))