netcal.metrics.MMCE¶
+-
+
-
+class
netcal.metrics.MMCE(detection: bool = False)¶
+ Bases:
+objectMaximum Mean Calibration Error (MMCE) 1. +A differentiable approximation to the Expected Calibration Error (ECE) using a +reproducing _kernel Hilbert space (RKHS). +Using a dataset \(\mathcal{D}\) of size \(N\) consisting of the ground truth labels \(\hat{y} \in \{1, ..., K \}\) +with input \(\hat{y} \in \mathcal{X}\), the MMCE is calculated by using a scoring classifier \(\hat{p}=h(x)\) +that returns the highest probability for a certain class in conjunction with the predicted label +information \(y \in \{1, ..., K \}\) and is defined by
++\[MMCE = \sqrt{\sum_{i, j \in \mathcal{D}} \frac{1}{N^2}(\mathbb{1}(\hat{y}_i = y_i) - \hat{p}_i) (\mathbb{1}(\hat{y}_j = y_j) - \hat{p}_j)k(\hat{p}_i, \hat{p}_j)} ,\]+with \(\mathbb{1}(*)\) as the indicator function and a Laplacian _kernel \(k\) defined by
++\[k(\hat{p}_i, \hat{p}_j) = \exp(-2.5 |\hat{p}_i - \hat{p}_j|) .\]+-
+
- Parameters +
detection (bool, default: False) – Detection mode is currently not supported for MMCE! +If False, the input array ‘X’ is treated as multi-class confidence input (softmax) +with shape (n_samples, [n_classes]). +If True, the input array ‘X’ is treated as a box predictions with several box features (at least +box confidence must be present) with shape (n_samples, [n_box_features]).
+
+
References
+-
+
- 1 +
Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain: +“Trainable calibration measures for neural networks from _kernel mean embeddings.” +International Conference on Machine Learning. 2018. +Get source online.
+
+
Methods
++
++ + ++ + __init__([detection])Constructor.
measure(X, y[, batched])Measure calibration by given predictions with confidence and the according ground truth.
-
+
-
+
measure(X: Union[Iterable[numpy.ndarray], numpy.ndarray], y: Union[Iterable[numpy.ndarray], numpy.ndarray], batched: bool = False)¶
+ Measure calibration by given predictions with confidence and the according ground truth.
+-
+
- Parameters +
-
+
X (iterable of np.ndarray, or np.ndarray of shape=(n_samples, [n_classes])) – NumPy array with confidence values for each prediction on classification with shapes +1-D for binary classification, 2-D for multi class (softmax). +If this is an iterable over multiple instances of np.ndarray and parameter batched=True, +interpret this parameter as multiple predictions that should be averaged.
+y (iterable of np.ndarray with same length as X or np.ndarray of shape=(n_samples, [n_classes])) – NumPy array with ground truth labels. +Either as label vector (1-D) or as one-hot encoded ground truth array (2-D). +If iterable over multiple instances of np.ndarray and parameter batched=True, +interpret this parameter as multiple predictions that should be averaged.
+batched (bool, optional, default: False) – Multiple predictions can be evaluated at once (e.g. cross-validation examinations) using batched-mode. +All predictions given by X and y are separately evaluated and their results are averaged afterwards +for visualization.
+
+- Returns +
Returns Maximum Mean Calibration Error.
+
+- Return type +
float
+
+