API Documentation

AbstractClassifier

An abstract type whose subtypes C<:AbstractClassifier must implement:

• Lighthouse.classes
• Lighthouse.train!
• Lighthouse.loss_and_prediction

Subtypes may additionally overload default implementations for:

• Lighthouse.onehot
• Lighthouse.onecold
• Lighthouse.is_early_stopping_exception

The AbstractClassifier interface is built upon the expectation that any multiclass label will be represented in one of two standardized forms:

• "soft label": a probability distribution vector where the ith element is the probability assigned to the ith class in classes(classifier).
• "hard label": the interger index of a corresponding class in classes(classifier).

Internally, Lighthouse converts hard labels to soft labels via onehot and soft labels to hard labels via onecold.

See also: learn!

Lighthouse.classes(classifier::AbstractClassifier)

Return a Vector or Tuple of class values for classifier.

This method must be implemented for each AbstractClassifier subtype.

Lighthouse.train!(classifier::AbstractClassifier, batches, logger::LearnLogger)

Train classifier on the iterable batches for a single epoch. This function is called once per epoch by learn!.

This method must be implemented for each AbstractClassifier subtype. Implementers should ensure that the training loss is properly logged to logger by calling Lighthouse.log_value!(logger, "train/loss_per_batch", batch_loss) for each batch in batches.

Lighthouse.loss_and_prediction(classifier::AbstractClassifier,
                               input_batch::AbstractArray,
                               args...)

Return (loss, soft_label_batch) given input_batch and any additional args provided by the caller; loss is a scalar, which soft_label_batch is a matrix with length(classes(classifier)) rows and size(input_batch).

Specifically, the ith column of soft_label_batch is classifier's soft label prediction for the ith sample in input_batch.

This method must be implemented for each AbstractClassifier subtype.

Lighthouse.onehot(classifier::AbstractClassifier, hard_label)

Return the one-hot encoded probability distribution vector corresponding to the given hard_label. hard_label must be an integer index in the range 1:length(classes(classifier)).

Lighthouse.onecold(classifier::AbstractClassifier, soft_label)

Return the hard label (integer index in the range 1:length(classes(classifier))) corresponding to the given soft_label (one-hot encoded probability distribution vector).

By default, this function returns argmax(soft_label).

Lighthouse.is_early_stopping_exception(classifier::AbstractClassifier, exception)

Return true if exception should be considered an "early-stopping exception" (e.g. Flux.Optimise.StopException), rather than rethrown from learn!.

This function returns false by default, but can be overloaded by subtypes of AbstractClassifier that employ exceptions as early-stopping mechanisms.

LearnLogger

A struct that wraps a TensorBoardLogger.TBLogger in order to enforce the following:

• all values logged to Tensorboard should be accessible to the post_epoch_callback argument to learn!
• all values that are cached during learn! should be logged to Tensorboard

To access values logged to a LearnLogger instance, inspect the instance's logged field.

learn!(model::AbstractClassifier, logger,
       get_train_batches, get_test_batches, votes,
       elected=majority.(eachrow(votes), (1:length(classes(model)),));
       epoch_limit=100, post_epoch_callback=(current_epoch -> nothing),
       optimal_threshold_class::Union{Nothing,Integer}=nothing)

Return model after optimizing its parameters across multiple epochs of training and test, logging Lighthouse's standardized suite of classifier performance metrics to logger throughout the optimization process.

The following phases are executed at each epoch (note: in the below lists of logged values, $resource takes the values of the field names of Lighthouse.ResourceInfo):

1. Train model by calling train!(model, get_train_batches(), logger). The following quantities are logged to logger during this phase:

   • train/loss_per_batch
   • any additional quantities logged by the relevant model/framework-specific implementation of train!.

2. Compute model's predictions on test set provided by get_test_batches() (see below for details). The following quantities are logged to logger during this phase:

   • test_set_prediction/loss_per_batch
   • test_set_prediction/mean_loss_per_epoch
   • test_set_prediction/$resource_per_batch

3. Compute a battery of metrics to evaluate model's performance on the test set based on the test set prediction phase. The following quantities are logged to logger during this phase:

   • test_set_evaluation/metrics_per_epoch
   • test_set_evaluation/$resource_per_epoch

4. Call post_epoch_callback(current_epoch).

Where...

• get_train_batches is a zero-argument function that returns an iterable of training set batches. Internally, learn! uses this function when it calls train!(model, get_train_batches(), logger).

• get_test_batches is a zero-argument function that returns an iterable of test set batches used during the current epoch's test phase. Each element of the iterable takes the form (batch, votes_locations). Internally, batch is passed to loss_and_prediction as loss_and_prediction(model, batch...), and votes_locations[i] is expected to yield the row index of votes that corresponds to the ith sample in batch.

• votes is a matrix of hard labels whose columns correspond to voters and whose rows correspond to the samples in the test set that have been voted on. If votes[sample, voter] is not a valid hard label for model, then voter will simply be considered to have not assigned a hard label to sample.

• elected is a vector of hard labels where the ith element is the hard label elected as "ground truth" out of votes[i, :].

• optimal_threshold_class is the class index (1 or 2) for which to calculate an optimal threshold for converting predicted_soft_labels to predicted_hard_labels. This is only a valid parameter when length(classes) == 2. If optimal_threshold_class is present, test set evaluation will be based on predicted hard labels calculated with this threshold; if optimal_threshold_class is nothing, predicted hard labels will be calculated via onecold(classifier, soft_label).

upon(logged::Dict{String,Any}, field::AbstractString; condition, initial)

Return a closure that can be called to check the most recent state of logger.logged[field] and trigger a caller-provided function when condition(recent_state, previously_chosen_state) is true.

For example:

upon_loss_decrease = upon(logger, "test_set_prediction/mean_loss_per_epoch";
                          condition=<, initial=Inf)

save_upon_loss_decrease = _ -> begin
    upon_loss_decrease(new_lowest_loss -> save_my_model(model, new_lowest_loss),
                       consecutive_failures -> consecutive_failures > 10 && Flux.stop())
end

learn!(model, logger, get_train_batches, get_test_batches, votes;
       post_epoch_callback=save_upon_loss_decrease)

Specifically, the form of the returned closure is f(on_true, on_false) where on_true(state) is called if condition(state, previously_chosen_state) is true. Otherwise, on_false(consecutive_falses) is called where consecutive_falses is the number of condition calls that have returned false since the last condition call returned true.

Note that the returned closure is a no-op if logger.logged[field] has not been updated since the most recent call.

evaluate!(predicted_hard_labels::AbstractVector,
          predicted_soft_labels::AbstractMatrix,
          elected_hard_labels::AbstractVector,
          classes, logger::LearnLogger;
          logger_prefix, logger_suffix,
          votes::Union{Nothing,AbstractMatrix}=nothing,
          thresholds=0.0:0.01:1.0,
          optimal_threshold_class::Union{Nothing,Integer}=nothing)

Return nothing after computing and logging a battery of classifier performance metrics that each compare predicted_soft_labels and/or predicted_hard_labels agaist elected_hard_labels.

The following quantities are logged to logger: - <logger_prefix>/metrics<logger_suffix> - <logger_prefix>/$resource<logger_suffix>

Where...

• predicted_soft_labels is a matrix of soft labels whose columns correspond to classes and whose rows correspond to samples in the evaluation set.

• predicted_hard_labels is a vector of hard labels where the ith element is the hard label predicted by the model for sample i in the evaulation set.

• elected_hard_labels is a vector of hard labels where the ith element is the hard label elected as "ground truth" for sample i in the evaulation set.

• thresholds are the range of thresholds used by metrics (e.g. PR curves) that are calculated on the predicted_soft_labels for a range of thresholds.

• votes is a matrix of hard labels whose columns correspond to voters and whose rows correspond to the samples in the test set that have been voted on. If votes[sample, voter] is not a valid hard label for model, then voter will simply be considered to have not assigned a hard label to sample.

• optimal_threshold_class is the class index (1 or 2) for which to calculate an optimal threshold for converting the predicted_soft_labels to predicted_hard_labels. If present, the input predicted_hard_labels will be ignored and new predicted_hard_labels will be recalculated from the new threshold. This is only a valid parameter when length(classes) == 2

predict!(model::AbstractClassifier,
         predicted_soft_labels::AbstractMatrix,
         batches, logger::LearnLogger;
         logger_prefix::AbstractString)

Return mean_loss of all batches after using model to predict their soft labels and storing those results in predicted_soft_labels.

The following quantities are logged to logger:

• <logger_prefix>/loss_per_batch
• <logger_prefix>/mean_loss_per_epoch
• <logger_prefix>/$resource_per_batch

Where...

• model is a model that outputs soft labels when called on a batch of batches, model(batch).

• predicted_soft_labels is a matrix whose columns correspond to classes and whose rows correspond to samples in batches, and which is filled in with soft-label predictions.

• batches is an iterable of batches, where each element of the iterable takes the form (batch, votes_locations). Internally, batch is passed to loss_and_prediction as loss_and_prediction(model, batch...).

forwarding_task = forward_logs(channel, logger::LearnLogger)

Forwards logs with values supported by TensorBoardLogger to logger::LearnLogger:

• string events of type AbstractString
• scalars of type Union{Real,Complex}
• plots that TensorBoardLogger can convert to raster images

returns the forwarding_task:::Task that does the forwarding. To cleanly stop forwarding, close(channel) and wait(forwarding_task).

outbox is a Channel or RemoteChannel of Pair{String, Any} field names starting with "plot" forward to TensorBoardLogger.log_image

_calculate_ea_kappas(predicted_hard_labels, elected_hard_labels, classes)

Return NamedTuple with keys :per_class, :multiclass containing the Cohen's Kappa per-class and over all classes, respectively. The value of output key :per_class is an Array such that item i is the Cohen's kappa calculated for class i.

Where...

• predicted_hard_labels is a vector of hard labels where the ith element is the hard label predicted by the model for sample i in the evaulation set.
• elected_hard_labels is a vector of hard labels where the ith element

is the hard label elected as "ground truth" for sample i in the evaulation set.

• class_count is the number of possible classes.

_calculate_ira_kappas(votes, classes)

Return NamedTuple with keys :per_class, :multiclass containing the Cohen's Kappa for inter-rater agreement (IRA) per-class and over all classes, respectively. The value of output key :per_class is an Array such that item i is the IRA kappa calculated for class i.

Where...

• votes is a matrix of hard labels whose columns correspond to voters and whose rows correspond to the samples in the test set that have been voted on. If votes[sample, voter] is not a valid hard label for model, then voter will simply be considered to have not assigned a hard label to sample.

• classes all possible classes voted on.

Returns nothing if votes has only a single voter (i.e., a single column) or if no two voters rated the same sample. Note that vote entries of 0 are taken to mean that the voter did not rate that sample.

_calculate_spearman_correlation(predicted_soft_labels, votes, classes)

Return NamedTuple with keys :ρ, :n, :ci_lower, and ci_upper that are the Spearman correlation constant ρ and its 95% confidence interval bounds. Only valid for binary classification problems (i.e., length(classes) == 2)

Where...

• predicted_soft_labels is a matrix of soft labels whose columns correspond to the two classes and whose rows correspond to the samples in the test set that have been classified. For a given sample, the two class column values must sum to 1 (i.e., softmax has been applied to the classification output).
• votes is a matrix of hard labels whose columns correspond to voters and whose rows correspond to the samples in the test set that have been voted on. If

votes[sample, voter] is not a valid hard label for model, then voter will simply be considered to have not assigned a hard label to sample. May contain a single voter (i.e., a single column).

• classes are the two classes voted on.

evaluation_metrics_plot(predicted_hard_labels::AbstractVector,
                        predicted_soft_labels::AbstractMatrix,
                        elected_hard_labels::AbstractVector,
                        classes,
                        thresholds=0.0:0.01:1.0;
                        votes::Union{Nothing,AbstractMatrix}=nothing,
                        strata::Union{Nothing,AbstractVector{Set{T}} where T}=nothing,
                        optimal_threshold_class::Union{Nothing,Integer}=nothing)

Return a plot and dictionary containing a battery of classifier performance metrics that each compare predicted_soft_labels and/or predicted_hard_labels agaist elected_hard_labels.

Where...

• predicted_soft_labels is a matrix of soft labels whose columns correspond to

classes and whose rows correspond to samples in the evaluation set.

• predicted_hard_labels is a vector of hard labels where the ith element

is the hard label predicted by the model for sample i in the evaulation set.

• elected_hard_labels is a vector of hard labels where the ith element

is the hard label elected as "ground truth" for sample i in the evaulation set.

• thresholds are the range of thresholds used by metrics (e.g. PR curves) that

are calculated on the predicted_soft_labels for a range of thresholds.

• votes is a matrix of hard labels whose columns correspond to voters and whose rows correspond to the samples in the test set that have been voted on. If votes[sample, voter] is not a valid hard label for model, then voter will simply be considered to have not assigned a hard label to sample.

• strata is a vector of sets of (arbitrarily typed) groups/strata for each sample in the evaluation set, or nothing. If not nothing, per-class and multiclass kappas will also be calculated per group/stratum.

• optimal_threshold_class is the class index (1 or 2) for which to calculate an optimal threshold for converting the predicted_soft_labels to predicted_hard_labels. If present, the input predicted_hard_labels will be ignored and new predicted_hard_labels will be recalculated from the new threshold. This is only a valid parameter when length(classes) == 2

confusion_matrix(class_count::Integer, hard_label_pairs = ())

Given the iterable hard_label_pairs whose kth element takes the form (first_classifiers_label_for_sample_k, second_classifiers_label_for_sample_k), return the corresponding confusion matrix where matrix[i, j] is the number of samples that the first classifier labeled i and the second classifier labeled j.

Note that the returned confusion matrix can be updated in-place with new labels via Lighthouse.increment_at!(matrix, more_hard_label_pairs).

accuracy(confusion::AbstractMatrix)

Returns the percentage of matching classifications out of total classifications, or missing if all(iszero, confusion).

Note that accuracy(confusion) is equivalent to overall percent agreement between confusion's row classifier and column classifier.

binary_statistics(confusion::AbstractMatrix, class_index)

Treating the rows of confusion as corresponding to predicted classifications and the columns as corresponding to true classifications, return a NamedTuple with the following fields for the given class_index:

• predicted_positives
• predicted_negatives
• actual_positives
• actual_negatives
• true_positives
• true_negatives
• false_positives
• false_negatives
• true_positive_rate
• true_negative_rate
• false_positive_rate
• false_negative_rate
• precision

cohens_kappa(class_count, hard_label_pairs)

Return (κ, p₀) where κ is Cohen's kappa and p₀ percent agreement given class_count and hard_label_pairs (these arguments take the same form as their equivalents in confusion_matrix).

calibration_curve(probabilities, bitmask; bin_count=10)

Given probabilities (the predicted probabilities of the positive class) and bitmask (a vector of Bools indicating whether or not the element actually belonged to the positive class), return (bins, fractions, totals, mean_squared_error) where:

• bins a vector with bin_count Pairs specifying the calibration curve's probability bins
• fractions: a vector where fractions[i] is the number of values in probabilities that falls within bin[i] over the total number of values within bin[i], or missing if the total number of values in bin[i] is zero.
• totals: a vector where totals[i] the total number of values within bin[i].
• mean_squared_error: The mean squared error of fractions vs. an ideal calibration curve.

This method is similar to the corresponding scikit-learn method:

https://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html

majority([rng::AbstractRNG=Random.GLOBAL_RNG], hard_labels, among::UnitRange)

Return the majority label within among out of hard_labels:

julia> majority([1, 2, 1, 3, 2, 2, 3], 1:3)
2

julia> majority([1, 2, 1, 3, 2, 2, 3, 4], 3:4)
3

In the event of a tie, a winner is randomly selected from the tied labels via rng.

area_under_curve(x, y)

Calculates the area under the curve specified by the x vector and y vector using the trapezoidal rule.

area_under_curve_unit_square(x, y)

Calculates the area under the curve specified by the x vector and y vector for a unit square, using the trapezoidal rule.