niftynet.evaluation.segmentation_evaluations module

This module defines built-in evaluation functions for segmentation applications

Segmentations can be evaluated at several scales: ‘foreground’ refering to metrics computed once for a foreground label ‘label’ refering to metrics computed once for each label (including background) ‘cc’ referring to metrics computed once for each connected component set

Connected components are defined by one-or-more connected components on the reference segmentation and one-or-more connected components on the infered segmentation. These sets are defined by a cc_func. Currently this is hard coded to be union_of_seg_for_each_ref_cc which takes each connected component on the reference segmentation and all connected components on the infered segmentation with any overlap. This will eventually be made into a factory option for different cc set definitions

Overlap and distance measures can be computed at each of these levels by deriving from PerComponentEvaluation, which handles the logic of identifying which comparisons need to be done for each scale.

Overlap and distance measures are computed in two convenience functions (compute_many_overlap_metrics and compute_many_distance_metrics) and wrapped by Evaluation classes

class PerComponentEvaluation(reader, app_param, eval_param)

Bases: niftynet.evaluation.base_evaluations.CachedSubanalysisEvaluation

This class represents evaluations performed on binary segmentation components computed per label or per connected component. It encodes the generation of evaluation tasks. Derived classes should define the metric_name constant and the function metric_from_binarized()

subanalyses(subject_id, data)

This function defines the sub-analyses to run. All evaluations with matching sub-analyses will be run in sequence, before clearing the cache :param subject_id: subject identifier string :param data: a data dictionary as built by ImageReader :return: list of dictionaries, each containing information specifyng the analysis to run. Elements will be passed to layer_op one at a time in a cache friendly order

layer_op(subject_id, data, task)

Performs one sub-analysis

Parameters:

• subject_id – subject identifier string
• data – a data dictionary as built by ImageReader
• subanalysis – dictionary containing information specifying the

analysis to run :return: a list of pandas.DataFrame objects

metric_dict_from_binarized(seg, ref)

Computes a metric from a binarized mask :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: a dictionary of metric_name:metric_value

class PerComponentScalarEvaluation(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.PerComponentEvaluation

This class simplifies the implementation when the metric just returns a single scalar with the same name as the class name

metric_dict_from_binarized(seg, ref)

Wrap computed metric in dictionary for parent class

metric_from_binarized(seg, ref)

Computer scalar metric value :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: scalar metric value

get_aggregations()

Returns aggregations to compute for the metric. Each aggregation is a callable that computes a list of DataFrames from a dictionary of metric dataframes (index by the DataFrame index). See BaseEvaluator.ScalarAggregator for an example.

Returns:list of aggregation callables

class BuiltinOverlapEvaluation(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.PerComponentScalarEvaluation

Wrapper class to encode many similar overlap metrics that can be computed from a confusion matrix Metrics computed in compute_many_overlap_metrics can be wrapped by overriding self.metric_name

metric_from_binarized(seg, ref)

Computes a metric from a binarized mask by computing a confusion matrix and then delegating the metric computation :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: scalar metric value

metric_from_confusion_matrix(confusion_matrix)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_pos_ref(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_neg_ref(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_pos_seg(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_neg_seg(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class fp(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class fn(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class tp(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class tn(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_intersection(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class n_union(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class specificity(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class sensitivity(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class accuracy(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class false_positive_rate(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class positive_predictive_values(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class negative_predictive_values(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class dice(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

Dice

alias of niftynet.evaluation.segmentation_evaluations.dice

class jaccard(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

intersection_over_union

alias of niftynet.evaluation.segmentation_evaluations.jaccard

Jaccard

alias of niftynet.evaluation.segmentation_evaluations.jaccard

class informedness(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class markedness(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class vol_diff(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.BuiltinOverlapEvaluation

metric_from_confusion_matrix(M)

Compute metrics from a 2x2 confusion matrix :param confusion_matrix: 2x2 numpy array :return: scalar metric value

class average_distance(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.PerComponentScalarEvaluation

metric_from_binarized(seg, ref)

Computer scalar metric value :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: scalar metric value

class hausdorff_distance(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.PerComponentScalarEvaluation

metric_from_binarized(seg, ref)

Computer scalar metric value :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: scalar metric value

class hausdorff95_distance(*args, **kwargs)

Bases: niftynet.evaluation.segmentation_evaluations.PerComponentScalarEvaluation

metric_from_binarized(seg, ref)

Computer scalar metric value :param seg: numpy array with binary mask from inferred segmentation :param ref: numpy array with binary mask from reference segmentation :return: scalar metric value

union_of_seg_for_each_ref_cc(blobs_seg, blobs_ref)

Constructs connected component sets to compute metrics for. Each reference connected component is paired with the union of inferred segmentation connected components with any overlap :param blobs_seg: tuple (numbered_cc_array, number_of_ccs) :param blobs_ref: tuple (numbered_cc_array, number_of_ccs) :return: dictionary {label:(ref_label_list, seg_label_list)}