Accuracy statistics in FFTrees

Conditional accuracy statistics

The first set of accuracy statistics are based on subsets of the data. These subsets result from focusing on particular cases of interest and computing conditional probabilities based on them. Given the 2x2 structure of the confusion table, measures can be conditional on either algorithm decisions (positive predictive vs. negative predictive values) or criterion values (sensitivity vs. specificity). In other words, these measures are conditional probabilities that are based on either the rows or columns of the confusion table:

Table 2: Conditional accuracy statistics based on either the rows or columns of a 2x2 confusion table.

The sensitivity (aka. hit-rate) is defined as sens = hi/(hi + mi) and represents the percentage of cases with positive criterion values that were correctly predicted by the algorithm. Similarly, specificity (aka. correct rejection rate, or the complement of the false alarm rate) is defined as spec = cr/(fa + cr) and represents the percentage of cases with negative criterion values correctly predicted by the algorithm.

The positive-predictive value ppv and negative predictive value npv are the flip-sides of sens and spec, as they are conditional accuracies based on decision outcomes (rather than on true criterion values).

Aggregate accuracy statistics

Additional accuracy statistics are based on all four cells in the confusion table:

Table 3: Aggregate accuracy statistics based on all four cells of a 2x2 confusion table.

Overall accuracy (acc) is defined as the overall percentage of correct decisions ignoring the difference between hits and correct rejections. The more specific measures bacc and wacc are averages of sensitivity and specificity, while bpv is an average of predictive values. The dprime measure is the difference in standardized (z-score) transformed sens and far (see Luan et al., 2011 for the relation between FFTs and signal detection theory, SDT).

Speed and frugality statistics

The next two statistics measure the speed and frugality of a fast-and-frugal tree (FFT). Unlike the accuracy statistics above, they are not based on the confusion table. Rather, they depend on how much information FFTs use to make their predictions or decisions.

Table 4: Measures to quantify the speed and frugality of FFTs.

To see exactly where these statistics come from, let’s look at the results for heart.fft (Tree #1):

heart.fft

#> FFTrees 
#> - Trees: 7 fast-and-frugal trees predicting diagnosis
#> - Cost of outcomes:  hi = 0,  fa = 1,  mi = 1,  cr = 0
#> - Cost of cues: 
#>      age      sex       cp trestbps     chol      fbs  restecg  thalach 
#>        1        1        1        1        1        1        1        1 
#>    exang  oldpeak    slope       ca     thal 
#>        1        1        1        1        1 
#> 
#> FFT #1: Definition
#> [1] If thal = {rd,fd}, decide True.
#> [2] If cp != {a}, decide False.
#> [3] If ca > 0, decide True, otherwise, decide False.
#> 
#> FFT #1: Training Accuracy
#> Training data: N = 303, Pos (+) = 139 (46%) 
#> 
#> |          | True + | True - | Totals:
#> |----------|--------|--------|
#> | Decide + | hi 118 | fa  37 |     155
#> | Decide - | mi  21 | cr 127 |     148
#> |----------|--------|--------|
#>   Totals:       139      164   N = 303
#> 
#> acc  = 80.9%   ppv  = 76.1%   npv  = 85.8%
#> bacc = 81.2%   sens = 84.9%   spec = 77.4%
#> 
#> FFT #1: Training Speed, Frugality, and Cost
#> mcu = 1.73,  pci = 0.87
#> cost_dec = 0.191,  cost_cue = 1.733,  cost = 1.924

According to this output, Tree #1 has mcu = 1.73 and pci = 0.87. We can easily calculate these measures directly from the x$levelout output of an FFTrees object. This object contains the tree level (i.e., node) at which each case was classified:

# A vector of levels/nodes at which each case was classified:
heart.fft$trees$decisions$train$tree_1$levelout

#>   [1] 1 3 1 2 2 2 3 3 1 1 1 2 1 1 1 2 1 3 2 2 2 2 2 1 1 2 2 2 3 1 2 1 2 1 2 3 1
#>  [38] 1 1 2 1 1 2 2 3 1 2 1 2 2 2 1 3 2 1 1 1 1 2 2 1 2 1 2 1 1 2 1 1 2 2 1 1 1
#>  [75] 3 2 1 2 2 1 3 3 2 1 2 2 2 2 3 2 3 1 1 2 2 1 1 1 2 3 3 2 3 2 1 1 1 1 1 1 1
#> [112] 3 1 1 1 1 2 3 1 1 1 1 2 1 2 2 1 1 2 3 1 1 2 3 2 2 1 1 1 2 2 1 2 1 1 2 1 2
#> [149] 2 2 1 3 1 1 3 3 1 1 1 1 1 3 2 3 2 1 2 2 1 2 1 1 3 3 1 1 1 1 2 2 1 1 2 1 3
#> [186] 2 1 1 1 1 2 1 1 3 2 3 2 3 2 2 3 3 1 1 1 1 1 1 2 3 2 1 2 1 3 1 2 3 3 3 2 2
#> [223] 2 1 3 2 3 2 3 3 2 3 2 2 2 3 1 1 2 2 2 2 3 2 2 3 1 3 1 2 1 1 1 2 3 2 3 2 2
#> [260] 1 2 2 2 2 3 1 3 1 1 2 1 1 1 3 2 1 2 2 2 3 1 2 1 2 1 1 1 1 1 2 1 2 1 1 3 2
#> [297] 1 1 1 1 1 2 2

Now, to calculate mcu (the mean number of cues used), we simply take the mean of this vector:

# Calculate the mean number or cues used (mcu): 
mean(heart.fft$trees$decisions$train$tree_1$levelout)

#> [1] 1.732673

Now that we know where mcu comes from, computing pci (i.e., the percentage of cues ignored) is just as simple — it’s just the total number of cues in the dataset minus mcu, divided by the total number of cues in the data:

# Calculate pci (percentage of cues ignored) as 
# (n.cues - mcu) / n.cues):
n.cues <- ncol(heartdisease) 
(n.cues - heart.fft$trees$stats$train$mcu[1]) / n.cues

#> [1] 0.8762376

Additional measures

There is a wide range of additional measures that can be used to quantify classification performance. Most of these can easily be computed from the numerical information provided in an FFTrees object. For alternative measures based on the frequency counts of a 2x2 matrix, see Table 3 of Neth et al. (2021).