Speed up classification_report

[fingoldo]

Describe the workflow you want to enable

I'm concerned with slow execution speed of the classification_report procedure which makes it barely suitable for production-grade workloads.
On a 8M sample it already takes 15 seconds whereas simple POC numba implementation takes 20 to 40 milliseconds. I understand that sklearn code is well covered with tests, has wide functionality, follows style guildlines and best practices, but there should be no excuse for the performace leap with simple POC of magnitude that big (x1000).

import numpy as np
from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2 ** 23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2 ** 20 : 2 ** 21])

print(classification_report(y_true=y_true, y_pred=y_pred, digits=10, output_dict=False, zero_division=0,))

          precision    recall  f1-score   support

       0  0.9373906697 0.9373906697 0.9373906697   4192570
       1  0.9374424159 0.9374424159 0.9374424159   4196038

accuracy                      0.9374165535   8388608
macro avg  0.9374165428 0.9374165428 0.9374165428   8388608
weighted avg  0.9374165535 0.9374165535 0.9374165535   8388608

time: 18.6 s (started: 2023-07-08 16:10:08 +03:00)

print(own_classification_report(y_true=y_true, y_pred=y_pred, zero_division=0))

(3930076, 3933544, 262494, 262494, 0.9374165534973145, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]))
time: 16 ms (started: 2023-07-08 16:11:18 +03:00)

Describe your proposed solution

import numpy as np
from numba import njit


@njit()
def own_classification_report(y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):

    correct = np.zeros(nclasses, dtype=np.int64)
    wrong = np.zeros(nclasses, dtype=np.int64)
    for truth, pred in zip(y_true, y_pred):
        if pred == truth:
            correct[pred] += 1
        else:
            wrong[pred] += 1
    tn, tp = correct
    fn, fp = wrong
    accuracy = (tn + tp) / len(y_true)

    groups = np.array([(tn + fn), (tp + fp)])
    supports = np.array([(tn + fp), (tp + fn)])

    precisions = np.array([tn, tp]) / groups
    recalls = np.array([tn, tp]) / supports
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)

    return tn, tp, fn, fp, accuracy, supports, precisions, recalls, f1s

Describe alternatives you've considered, if relevant

Can someone from original classification_report authors take a look please why it's taking so long?

Additional context

I have the latest versions of scikit-learn and numba to this date, an Intel CPU with AVX.

[fingoldo]

I have actually improved it slightly to have more concise variable names & handle multiclasses. The only things of sklearn's classification report it does not support are micro,sample averages and sample weights (not hard to add them as well). As an extra it has balanced accuracy though.

@njit()
def fast_classification_report(
    y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):
    """Custom classification report, proof of concept."""

    N_AVG_ARRAYS = 3  # precisions, recalls, f1s

    # storage inits
    weighted_averages = np.empty(N_AVG_ARRAYS, dtype=np.float64)
    macro_averages = np.empty(N_AVG_ARRAYS, dtype=np.float64)
    supports = np.zeros(nclasses, dtype=np.int64)
    allpreds = np.zeros(nclasses, dtype=np.int64)
    misses = np.zeros(nclasses, dtype=np.int64)
    hits = np.zeros(nclasses, dtype=np.int64)

    # count stats
    for true_class, predicted_class in zip(y_true, y_pred):
        supports[true_class] += 1
        allpreds[predicted_class] += 1
        if predicted_class == true_class:
            hits[predicted_class] += 1
        else:
            misses[predicted_class] += 1
    
    # main calcs
    accuracy = hits.sum() / len(y_true)
    balanced_accuracy = np.nan_to_num(hits / supports, copy=True, nan=zero_division).mean()

    recalls = hits / supports
    precisions = hits / allpreds
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    # fix nans & compute averages
    i=0
    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)
        weighted_averages[i] = (arr * supports).sum() / len(y_true)
        macro_averages[i] = arr.mean()
        i+=1
    return hits, misses, accuracy, balanced_accuracy, supports, precisions, recalls, f1s, macro_averages, weighted_averages

print(fast_classification_report_numba(y_true=y_true, y_pred=y_pred, zero_division=0))

(array([3930076, 3933544], dtype=int64), array([262494, 262494], dtype=int64), 0.9374165534973145, 0.9374165428009072, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93741654, 0.93741654, 0.93741654]), array([0.93741655, 0.93741655, 0.93741655]))
time: 31 ms (started: 2023-07-09 15:22:31 +03:00)

%timeit fast_classification_report_numba(y_true=y_true, y_pred=y_pred, zero_division=0)

29.9 ms ± 312 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[adrinjalali]

Could you please do a profiling and show where in the classification_report most of the time is spent? Then we can figure out what the best way improving it would be.

[fingoldo]

Could you please do a profiling and show where in the classification_report most of the time is spent? Then we can figure out what the best way improving it would be.

Well, %lprun -f classification_report classification_report(y_true=...) points at line 184 in _param_validation.py for some reason.
I don't know how to profile it correctly.

File: C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\_param_validation.py
Function: wrapper at line 180

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   180                                                   @functools.wraps(func)
   181                                                   def wrapper(*args, **kwargs):
   182         9       1953.0    217.0      0.0              global_skip_validation = get_config()["skip_parameter_validation"]
   183         9        233.0     25.9      0.0              if global_skip_validation:
   184         8  179797782.0 22474722.8     91.0                  return func(*args, **kwargs)

[adrinjalali]

So it turns out the culprit is pretty much mostly numpy's unique / unique_values which does a sort.

There's an open issue here: numpy/numpy#11136 about that

Even pandas' unique is twice as fast on even a bunch of random floats.

Note that you can profile your script with python -m cProfile -o script.prof script.py and then visualize the results nicely with snakeviz script.prof (needs snakeviz installed available via pip)

Looking at this table, 10.7s our of the 14.4s are spent in sort, and that's bad:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    970/1    0.019    0.000   14.388   14.388 {built-in method builtins.exec}
        1    0.000    0.000   14.388   14.388 1.py:1(<module>)
    65/57    0.004    0.000   13.788    0.242 _param_validation.py:180(wrapper)
        1    0.000    0.000   13.781   13.781 _classification.py:2405(classification_report)
  432/218    0.447    0.001   13.604    0.062 {built-in method numpy.core._multiarray_umath.implement_array_function}
       88    0.000    0.000   12.703    0.144 <__array_function__ internals>:177(unique)
       88    0.025    0.000   12.702    0.144 arraysetops.py:138(unique)
       88    0.694    0.008   12.677    0.144 arraysetops.py:323(_unique1d)
        4    0.000    0.000   12.632    3.158 _classification.py:1531(precision_recall_fscore_support)
       88   10.651    0.121   10.651    0.121 {method 'sort' of 'numpy.ndarray' objects}

We can maybe try to optimize the code, but the real fix would be to optimize numpy's unique.

@thomasjpfan was mentioning that pytorch doesn't sort on unique, so we might get that into numpy?

cc @rgommers

[rgommers]

Agreed, adding a keyword to disable sorting to numpy.unique & co would make sense to me.

[fingoldo]

Wow that's weird 'cause when I apply np.unique to either y_true or y_pred, it's never above 200 milliseconds on my 8M elements array.

Is it calling np. unique multiple times within one classification_report call, why would it do that?
Where are random floats coming from, we only have integer inputs, no?

What's making it even more sad is that I need to fit hundreds and thousands of models and then estimate their quality on a bunch of subsets like that, and I know there are only 2 classes possible 'cause it's a binary classification. Is there an option to pass nclasses directly to save on recomputing them each time?

[adrinjalali]

@fingoldo sounds like it would be a win-win for everybody if you could invest some time optimizing these routines then ;)

This is definitely something which could use some optimization, but we might not have the bandwidth to get to it ATM. However, we'd appreciate your contributions.

[fingoldo]

@fingoldo sounds like it would be a win-win for everybody if you could invest some time optimizing these routines then ;)

This is definitely something which could use some optimization, but we might not have the bandwidth to get to it ATM. However, we'd appreciate your contributions.

Great. Is optional numba allowed? Is optional passing nclasses directly allowed?