anomalib.utils.metrics

Custom anomaly evaluation metrics.

Submodules

Package Contents

Classes

AdaptiveThreshold

Optimal F1 Metric.

AnomalyScoreDistribution

Mean and standard deviation of the anomaly scores of normal training data.

AUPR

Area under the PR curve.

AUPRO

Area under per region overlap (AUPRO) Metric.

AUROC

Area under the ROC curve.

MinMax

Track the min and max values of the observations in each batch.

OptimalF1

Optimal F1 Metric.

PRO

Per-Region Overlap (PRO) Score.
class anomalib.utils.metrics.AdaptiveThreshold(default_value: float = 0.5, **kwargs)[source]

Bases: torchmetrics.PrecisionRecallCurve

Optimal F1 Metric.

Compute the optimal F1 score at the adaptive threshold, based on the F1 metric of the true labels and the predicted anomaly scores.

compute() → torch.Tensor

Compute the threshold that yields the optimal F1 score.

Compute the F1 scores while varying the threshold. Store the optimal threshold as attribute and return the maximum value of the F1 score.

Returns

Value of the F1 score at the optimal threshold.

class anomalib.utils.metrics.AnomalyScoreDistribution(**kwargs)[source]

Bases: torchmetrics.Metric

Mean and standard deviation of the anomaly scores of normal training data.

update(anomaly_scores: Optional[torch.Tensor] = None, anomaly_maps: Optional[torch.Tensor] = None) → None

Update the precision-recall curve metric.

compute() → Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]

Compute stats.

class anomalib.utils.metrics.AUPR[source]

Bases: torchmetrics.PrecisionRecallCurve

Area under the PR curve.

compute() → torch.Tensor

First compute PR curve, then compute area under the curve.

Returns

Value of the AUPR metric

update(preds: torch.Tensor, target: torch.Tensor) → None

Update state with new values.

Need to flatten new values as PrecicionRecallCurve expects them in this format for binary classification.

Parameters

  • preds (Tensor) – predictions of the model

  • target (Tensor) – ground truth targets

_compute() → Tuple[torch.Tensor, torch.Tensor]

Compute prec/rec value pairs.

Returns

Tuple containing Tensors for rec and prec

generate_figure() → Tuple[matplotlib.figure.Figure, str]

Generate a figure containing the PR curve as well as the random baseline and the AUC.

Returns

Tuple containing both the PR curve and the figure title to be used for logging

Return type

Tuple[Figure, str]

class anomalib.utils.metrics.AUPRO(compute_on_step: bool = True, dist_sync_on_step: bool = False, process_group: Optional[Any] = None, dist_sync_fn: Callable = None, fpr_limit: float = 0.3)[source]

Bases: torchmetrics.Metric

Area under per region overlap (AUPRO) Metric.

is_differentiable :bool = False
higher_is_better :Optional[bool]
full_state_update :bool = False
preds :List[torch.Tensor]
target :List[torch.Tensor]
update(preds: torch.Tensor, target: torch.Tensor) → None

Update state with new values.

Parameters

  • preds (Tensor) – predictions of the model

  • target (Tensor) – ground truth targets

_compute() → Tuple[torch.Tensor, torch.Tensor]

Compute the pro/fpr value-pairs until the fpr specified by self.fpr_limit.

It leverages the fact that the overlap corresponds to the tpr, and thus computes the overall PRO curve by aggregating per-region tpr/fpr values produced by ROC-construction.

Raises

ValueError – ValueError is raised if self.target doesn’t conform with requirements imposed by kornia for connected component analysis.

Returns

tuple containing final fpr and tpr values.

Return type

Tuple[Tensor, Tensor]

compute() → torch.Tensor

Fist compute PRO curve, then compute and scale area under the curve.

Returns

Value of the AUPRO metric

Return type

Tensor

generate_figure() → Tuple[matplotlib.figure.Figure, str]

Generate a figure containing the PRO curve and the AUPRO.

Returns

Tuple containing both the figure and the figure title to be used for logging

Return type

Tuple[Figure, str]

static interp1d(old_x: torch.Tensor, old_y: torch.Tensor, new_x: torch.Tensor) → torch.Tensor

Function to interpolate a 1D signal linearly to new sampling points.

Parameters

  • old_x (Tensor) – original 1-D x values (same size as y)

  • old_y (Tensor) – original 1-D y values (same size as x)

  • new_x (Tensor) – x-values where y should be interpolated at

Returns

y-values at corresponding new_x values.

Return type

Tensor

class anomalib.utils.metrics.AUROC[source]

Bases: torchmetrics.ROC

Area under the ROC curve.

compute() → torch.Tensor

First compute ROC curve, then compute area under the curve.

Returns

Value of the AUROC metric

Return type

Tensor

update(preds: torch.Tensor, target: torch.Tensor) → None

Update state with new values.

Need to flatten new values as ROC expects them in this format for binary classification.

Parameters

  • preds (Tensor) – predictions of the model

  • target (Tensor) – ground truth targets

_compute() → Tuple[torch.Tensor, torch.Tensor]

Compute fpr/tpr value pairs.

Returns

Tuple containing Tensors for fpr and tpr

generate_figure() → Tuple[matplotlib.figure.Figure, str]

Generate a figure containing the ROC curve, the baseline and the AUROC.

Returns

Tuple containing both the figure and the figure title to be used for logging

Return type

Tuple[Figure, str]

class anomalib.utils.metrics.MinMax(**kwargs)[source]

Bases: torchmetrics.Metric

Track the min and max values of the observations in each batch.

full_state_update :bool = True
update(predictions: torch.Tensor) → None

Update the min and max values.

compute() → Tuple[torch.Tensor, torch.Tensor]

Return min and max values.

class anomalib.utils.metrics.OptimalF1(num_classes: int, **kwargs)[source]

Bases: torchmetrics.Metric

Optimal F1 Metric.

Compute the optimal F1 score at the adaptive threshold, based on the F1 metric of the true labels and the predicted anomaly scores.

full_state_update :bool = False
update(preds: torch.Tensor, target: torch.Tensor) → None

Update the precision-recall curve metric.

compute() → torch.Tensor

Compute the value of the optimal F1 score.

Compute the F1 scores while varying the threshold. Store the optimal threshold as attribute and return the maximum value of the F1 score.

Returns

Value of the F1 score at the optimal threshold.

reset() → None

Reset the metric.

class anomalib.utils.metrics.PRO(threshold: float = 0.5, **kwargs)[source]

Bases: torchmetrics.Metric

Per-Region Overlap (PRO) Score.

target :List[torch.Tensor]
preds :List[torch.Tensor]
update(predictions: torch.Tensor, targets: torch.Tensor) → None

Compute the PRO score for the current batch.

compute() → torch.Tensor

Compute the macro average of the PRO score across all regions in all batches.