Package 'measures'

Description

Defined as: mean(response == truth)

Usage

ACC(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
ACC(truth, response)

Description

Defined as: 1 - (1 - rsq) * (p / (n - p - 1L)). Adjusted R-squared is only defined for normal linear regression.

Usage

ARSQ(truth, response, n, p)

Arguments

Examples

n = 20
p = 5
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
ARSQ(truth, response, n, p)

Description

Integral over the graph that results from computing fpr and tpr for many different thresholds.

Usage

AUC(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
AUC(probabilities, truth, negative, positive)

Description

Mean of true positive rate and true negative rate.

Usage

BAC(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
BAC(truth, response, negative, positive)

Description

Mean of misclassification error rates on all individual classes.

Usage

BER(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
BER(truth, response)

Description

The Brier score is defined as the quadratic difference between the probability and the value (1,0) for the class. That means we use the numeric representation 1 and 0 for our target classes. It is similiar to the mean squared error in regression. multiclass.brier is the sum over all one vs. all comparisons and for a binary classifcation 2 * brier.

Usage

Brier(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
Brier(probabilities, truth, negative, positive)

Description

Brier score scaled to [0,1], see http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/.

Usage

BrierScaled(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
BrierScaled(probabilities, truth, negative, positive)

Description

Similar to RSQ (R-squared). Defined as explained_sum_of_squares / total_sum_of_squares.

Usage

EXPVAR(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
EXPVAR(truth, response)

Description

Defined as: 2 * tp/ (sum(truth == positive) + sum(response == positive))

Usage

F1(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
F1(truth, response, positive)

Description

Defined as: fp / (tp + fp)

Usage

FDR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
FDR(truth, response, positive)

Description

Sum of misclassified observations in the negative class. Also called misses.

Usage

FN(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
FN(truth, response, negative)

Description

Percentage of misclassified observations in the negative class.

Usage

FNR(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
FNR(truth, response, negative, positive)

Description

Sum of misclassified observations in the positive class. Also called false alarms.

Usage

FP(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
FP(truth, response, positive)

Description

Percentage of misclassified observations in the positive class. Also called false alarm rate or fall-out.

Usage

FPR(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
FPR(truth, response, negative, positive)

Description

Geometric mean of recall and specificity.

Usage

GMEAN(truth, response, negative, positive)

Arguments

References

He, H. & Garcia, E. A. (2009) *Learning from Imbalanced Data.* IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9. pp. 1263-1284.

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
GMEAN(truth, response, negative, positive)

Description

Defined as: sqrt(ppv * tpr)

Usage

GPR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
GPR(truth, response, positive)

Description

Defined as: 1 - (1 - p0) / (1 - pe). With: p0 = 'observed frequency of agreement' and pe = 'expected agremeent frequency under independence

Usage

KAPPA(truth, response)

Arguments

Description

Defined as: Kendall's tau correlation between truth and response. Only looks at the order. See Rosset et al.: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.1398&rep=rep1&type=pdf.

Usage

KendallTau(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
KendallTau(truth, response)

Description

Lists all measures that are available in the package with their corresponding task.

Usage

listAllMeasures()

Value

Dataframe with all available measures and the correspoding task

Examples

listAllMeasures()

Description

Defined as: -mean(log(p_i)), where p_i is the predicted probability of the true class of observation i. Inspired by https://www.kaggle.com/wiki/MultiClassLogLoss.

Usage

Logloss(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
Logloss(probabilities, truth)

Description

Defined as: mean(log(p_i)), where p_i is the predicted probability of the true class of observation i. This scoring rule is the same as the negative logloss, self-information or surprisal. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

LSR(probabilities, truth)

Arguments

Description

Defined as: mean(abs(response - truth))

Usage

MAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MAE(truth, response)

Description

Defined as the abs(truth_i - response_i) / truth_i. Won't work if any truth value is equal to zero. In this case the output will be NA.

Usage

MAPE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MAPE(truth, response)

Description

Defined as (tp * tn - fp * fn) / sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)), denominator set to 1 if 0.

Usage

MCC(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
MCC(truth, response, negative, positive)

Description

Defined as: median(abs(response - truth)).

Usage

MEDAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MEDAE(truth, response)

Description

Defined as: median((response - truth)^2).

Usage

MEDSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MEDSE(truth, response)

Description

Defined as: mean(response != truth)

Usage

MMCE(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
MMCE(truth, response)

Description

Defined as: mean((response - truth)^2)

Usage

MSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MSE(truth, response)

Description

Defined as: mean((log(response + 1, exp(1)) - log(truth + 1, exp(1)))^2). This is mostly used for count data, note that all predicted and actual target values must be greater or equal '-1' to compute the mean squared logarithmic error.

Usage

MSLE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = abs(rnorm(n))
response = abs(rnorm(n))
MSLE(truth, response)

Description

Computes AUC of c(c - 1) binary classifiers while considering the a priori distribution of the classes. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AU1P(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AU1P(probabilities, truth)

Description

Computes AUC of c(c - 1) binary classifiers (all possible pairwise combinations) while considering uniform distribution of the classes. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AU1U(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AU1U(probabilities, truth)

Description

Computes the AUC treating a c-dimensional classifier as c two-dimensional classifiers, taking into account the prior probability of each class. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AUNP(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AUNP(probabilities, truth)

Description

Computes the AUC treating a c-dimensional classifier as c two-dimensional classifiers, where classes are assumed to have uniform distribution, in order to have a measure which is independent of class distribution change. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AUNU(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AUNU(probabilities, truth)

Description

Defined as: (1/n) sum_i sum_j (y_ij - p_ij)^2, where y_ij = 1 if observation i has class j (else 0), and p_ij is the predicted probability of observation i for class j. From http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf.

Usage

multiclass.Brier(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.Brier(probabilities, truth)

Description

Averaged proportion of correctly predicted labels with respect to the total number of labels for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are replaced with 1 before computing the average across all instances.

Usage

MultilabelACC(truth, response)

Arguments

Description

Harmonic mean of precision and recall on a per instance basis (Micro-F1), following the definition by Montanes et al.: http: / /www.sciencedirect.com / science / article / pii / S0031320313004019. Fractions where the denominator becomes 0 are replaced with 1 before computing the average across all instances.

Usage

MultilabelF1(truth, response)

Arguments

Description

Proportion of labels that are predicted incorrectly, following the definition by Charte and Charte: https://journal.r-project.org/archive/2015-2/charte-charte.pdf.

Usage

MultilabelHamloss(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
response = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
MultilabelHamloss(truth, response)

Description

Also called precision. Averaged ratio of correctly predicted labels for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are ignored in the average calculation.

Usage

MultilabelPPV(truth, response)

Arguments

Description

Proportion of observations where the complete multilabel set (all 0-1-labels) is predicted incorrectly, following the definition by Charte and Charte: https://journal.r-project.org/archive/2015-2/charte-charte.pdf.

Usage

MultilabelSubset01(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
response = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
MultilabelSubset01(truth, response)

Description

Also called recall. Averaged proportion of predicted labels which are relevant for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are ignored in the average calculation.

Usage

MultilabelTPR(truth, response)

Arguments

Description

Defined as: tn / (tn + fn).

Usage

NPV(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
NPV(truth, response, negative)

Description

Defined as: tp / (tp + fp). Also called precision. If the denominator is 0, PPV is set to be either 1 or 0 depending on whether the highest probability prediction is positive (1) or negative (0).

Usage

PPV(truth, response, positive, probabilities = NULL)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
PPV(truth, response, positive, probabilities = NULL)

Description

Defined as: 1 - (1/n) sum_i sum_j (y_ij - p_ij)^2, where y_ij = 1 if observation i has class j (else 0), and p_ij is the predicted probablity of observation i for class j. This scoring rule is the same as 1 - multiclass.brier. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

QSR(probabilities, truth)

Arguments

Description

Defined as sum_of_absolute_errors / mean_absolute_deviation. Undefined for single instances and when every truth value is identical. In this case the output will be NA.

Usage

RAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RAE(truth, response)

Description

The RMSE is aggregated as sqrt(mean(rmse.vals.on.test.sets^2))

Usage

RMSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RMSE(truth, response)

Description

Definition taken from: https: / /www.kaggle.com / wiki / RootMeanSquaredLogarithmicError. This is mostly used for count data, note that all predicted and actual target values must be greater or equal '-1' to compute the root mean squared logarithmic error.

Usage

RMSLE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = abs(rnorm(n))
response = abs(rnorm(n))
RMSLE(truth, response)

Description

Defined as sqrt (sum_of_squared_errors / total_sum_of_squares). Undefined for single instances and when every truth value is identical. In this case the output will be NA.

Usage

RRSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RRSE(truth, response)

Description

Also called R-squared, which is 1 - residual_sum_of_squares / total_sum_of_squares.

Usage

RSQ(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RSQ(truth, response)

Description

Defined as: sum(abs(response - truth))"

Usage

SAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SAE(truth, response)

Description

Defined as: Spearman's rho correlation between truth and response. Only looks at the order. See Rosset et al.: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.1398&rep=rep1&type=pdf.

Usage

SpearmanRho(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SpearmanRho(truth, response)

Description

Defined as: sum((response - truth)^2)

Usage

SSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SSE(truth, response)

Description

Defined as: mean(p_i(sum_j(p_ij))), where p_i is the predicted probability of the true class of observation i and p_ij is the predicted probablity of observation i for class j. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

SSR(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
SSR(probabilities, truth)

Description

Sum of correctly classified observations in the negative class. Also called correct rejections.

Usage

TN(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
TN(truth, response, negative)

Description

Percentage of correctly classified observations in the negative class. Also called specificity.

Usage

TNR(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
TNR(truth, response, negative)

Description

Sum of all correctly classified observations in the positive class.

Usage

TP(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
TP(truth, response, positive)

Description

Percentage of correctly classified observations in the positive class. Also called hit rate or recall or sensitivity.

Usage

TPR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
TPR(truth, response, positive)

Description

Defined as: 1 - sum(weights * conf.mat) / sum(weights * expected.mat), the weight matrix measures seriousness of disagreement with the squared euclidean metric.

Usage

WKAPPA(truth, response)

Arguments