Evaluator#

class anomalib.metrics.Evaluator(val_metrics=None, test_metrics=None, compute_on_cpu=True)#

Bases: Module, Callback

Evaluator module for LightningModule.

The Evaluator module is a PyTorch module that computes and logs metrics during validation and test steps. Each AnomalibModule should have an Evaluator module as a submodule to compute and log metrics during validation and test steps. An Evaluation module can be passed to the AnomalibModule as a parameter during initialization. When no Evaluator module is provided, the AnomalibModule will use a default Evaluator module that logs a default set of metrics.

Parameters:
  • val_metrics (AnomalibMetric | Sequence[AnomalibMetric] | None) – Validation metrics. Defaults to [].

  • test_metrics (AnomalibMetric | Sequence[AnomalibMetric] | None) – Test metrics. Defaults to [].

  • compute_on_cpu (bool) – Whether to compute metrics on CPU. Defaults to True.

Examples

>>> from anomalib.metrics import F1Score, AUROC
>>> from anomalib.data import ImageBatch
>>> import torch
>>>
>>> f1_score = F1Score(fields=["pred_label", "gt_label"])
>>> auroc = AUROC(fields=["pred_score", "gt_label"])
>>>
>>> evaluator = Evaluator(test_metrics=[f1_score])
metrics_to_cpu(metrics)#

Set the compute_on_cpu attribute of the metrics to True.

Return type:

None

on_test_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)#

Update test metrics with the batch output.

Return type:

None

on_test_epoch_end(trainer, pl_module)#

Compute and log test metrics.

Return type:

None

on_validation_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)#

Update validation metrics with the batch output.

Return type:

None

on_validation_epoch_end(trainer, pl_module)#

Compute and log validation metrics.

Return type:

None

setup(trainer, pl_module, stage)#

Move metrics to cpu if num_devices == 1 and compute_on_cpu is set to True.

Return type:

None

static validate_metrics(metrics)#

Validate metrics.

Return type:

Sequence[AnomalibMetric]