whylogs.experimental.performance_estimation.estimators#

Module Contents#

Classes#

PerformanceEstimator

Base class for performance estimators.

AccuracyEstimator

Estimates the accuracy of a model based on the confusion matrix of the reference result set and the target result set.

Attributes#

whylogs.experimental.performance_estimation.estimators.logger#
class whylogs.experimental.performance_estimation.estimators.PerformanceEstimator(reference_result_set: whylogs.api.logger.result_set.SegmentedResultSet)#

Bases: abc.ABC

Base class for performance estimators.

Parameters

reference_result_set (whylogs.api.logger.result_set.SegmentedResultSet) –

abstract estimate(target_result_set: whylogs.api.logger.result_set.SegmentedResultSet) whylogs.experimental.performance_estimation.estimation_results.EstimationResult#
Parameters

target_result_set (whylogs.api.logger.result_set.SegmentedResultSet) –

Return type

whylogs.experimental.performance_estimation.estimation_results.EstimationResult

class whylogs.experimental.performance_estimation.estimators.AccuracyEstimator(reference_result_set: whylogs.api.logger.result_set.SegmentedResultSet)#

Bases: PerformanceEstimator

Estimates the accuracy of a model based on the confusion matrix of the reference result set and the target result set.

The estimation is based on the individual performance of each segment of the reference result set. The reference accuracies for each segment are weighted by the number of records in each segment in the target result set. The estimated accuracy is the sum of the weighted accuracies of each segment.

Expects a single partition with mutually exclusive segments, and same set of segments in the reference and target result sets.

Supported for binary classification only.

Examples

Create target and reference dataframes:

from whylogs.experimental.performance_estimation import AccuracyEstimator
from whylogs.api.logger.result_set import SegmentedResultSet

estimator = AccuracyEstimator(reference_result_set: SegmentedResultSet = reference_results)

estimator_result = estimator.estimate(target_result_set: SegmentedResultSet = target_results)
print("Estimated accuracy: ", estimator_result.accuracy)
Parameters

reference_result_set (whylogs.api.logger.result_set.SegmentedResultSet) –

estimate(target: whylogs.api.logger.result_set.SegmentedResultSet) whylogs.experimental.performance_estimation.estimation_results.EstimationResult#

Estimate the overall accuracy of a target result set based on a reference result set.

Assumes the presence of a segmented column in the reference and target result sets.

returns: An EstimationResult object containing the estimated accuracy, partition’s id and the profiles’ dataset timestamp.

Parameters

target (whylogs.api.logger.result_set.SegmentedResultSet) –

Return type

whylogs.experimental.performance_estimation.estimation_results.EstimationResult