FEA Add Rand Index and pair confusion matrix

[ufmayer]

Reference Issues/PRs

Adding new functionality

What does this implement/fix? Explain your changes.

Two changes:

1. contingency_matrix() creates currently an int overflow for large data such as using adjusted_rand_score on large clusterings. This implementation adds an optional dtype parameter which can be used to store the counts in np.int64 to avoid overflows.
2. Adds computation of the unadjusted Rand Index (https://en.wikipedia.org/wiki/Rand_index). Computation is based on an efficient pair comparison helper method similar to the existing
   adjusted_rand_score method.

Any other comments?

Rand Index is essentially computing the non-chance-adjusted accuracy of a clustering, which is a standard measure for clusterings.

[jnothman]

Thanks for the PR.
The linter is unhappy.

At some point (I can't find the issue) I proposed that we have a function which turns a binary classification metric into a clustering metric via the pairwise transformation. In that framework, this would just be accuracy... Not sure if we want to go about implementing the generic solution. I know that in some spaces, F1 at least is calculated on pairs of clustered elements.

(Another generic class of clustering metrics is the max-sum assignment, which here we only implement as the consensus score for biclustering.)

[ufmayer]

Currently it passes the linter but it fails to fully build because the test code is looking for the new methods to be part of the package, but as they are not yet merged they are not found. That seems like a generic chicken-and-egg problem, but maybe I am misunderstanding something on how this is set up.

sklearn/tests/test_supervised.py:6: error: Module 'sklearn.metrics' has no attribute 'pair_confusion_matrix'
sklearn/tests/test_supervised.py:6: error: Module 'sklearn.metrics' has no attribute 'rand_score'
sklearn/metrics/cluster/tests/test_common.py:8: error: Module 'sklearn.metrics.cluster' has no attribute 'rand_score'

[ufmayer]

@jnothman I made the requested changes. Haven't seen any update on this for a week. I don't know how fast these things usually progress.l Is there something else that's waited for from me?

[cmarmo]

Hi @ufmayer, thanks for your work. Some checks didn't pass, do you mind having a look at them? Thanks!

[cmarmo]

Please, forgive me @ufmayer you already noticed the unsuccessful checks, my comment wasn't helpful.
The mypy check tells you that rand_score and pair_confusion_matrix should be defined in the sklearn/metrics/cluster/__init__.py to be imported correctly. Doing that will let the build know where the new module is.
Hope this would help to move forward.

[ufmayer]

@cmarmo Merci beaucoup. That was the information I needed. All checks have now passed. What's the next step?

[cmarmo]

@ufmayer, je vous en prie... :)

What's the next step?

'Waiting for reviewer' ... :) , pinging from time to time for the pull request don't get lost. Also it is useful to check if conflicts are generated by other merge, and fix them when they arise.
Last suggestion: you can add [MRG] to the PR title (as specified in the documentation), to specify that your PR is ready for review. Good luck!

[jnothman]

Sorry, my availability lately has been very scattered. I commented on this because I was interested in it, not because I had excess time for review! :D I'll try give it another shot.

[ufmayer]

@jnothman Thanks for saying you might find time to give this another shot. I think it's ready to go. How about it?

[ufmayer]

@jnothman @cmarmo It's been 3 months today since I filed this PR. Is there anything you could suggest to move this along?

[jnothman]

I've tagged it for the current release milestone, and hopefully it will be picked up by another reviewer as we work towards release over the next 6 weeks.

Sorry for the delays, but there's a lot to review!

[ufmayer]

@jnothman Thanks!

[glemaitre]

First round of reviews.

[glemaitre]

Suggested change

	n_samples = np.int64(labels_true.shape[0])
	n_samples = labels_true.shape[0]

[ufmayer]

I left this unchanged for now. The entire calculations below are using np.int64 and the return data type of the ndarray is np.int64, and this cast makes this explicitly clear. The cast will of course otherwise happen implicitly anyway but I think this makes it unambiguous. I am not opposed to the change, but if you don't feel strongly I would prefer to let it stand.

[glemaitre]

Some remaining comments.
But in general it looks good.

[glemaitre]

@ufmayer You can ignore the MacOS build failure. We are currently investigating it.
However, the other potential failure should be looked at :)

[ufmayer]

@glemaitre Thank you for the thorough review. I implemented the majority of your suggestions and resolved the corresponding auto-generated conversations in this PR. Please have a look at the remaining ones, I left brief comments on each that I didn't resolve.

[glemaitre]

There is the remaining comments. Just some nitpicks. Once done this is mergeable :)

[glemaitre]

Can we specify the following instead (if I don't make a mistake interpreting the note below).

Output dtype. Ignore if `eps` is not `None`.

[ufmayer]

@glemaitre Done.

[cmarmo]

@glemaitre, ready for a second approval? Thanks!

[glemaitre]

Thanks @ufmayer LGTM merging.