Some minor fixes for stats.wishart addition. #4313

rgommers · 2014-12-26T21:39:36Z

fixes to docstrings (references, see also, added in 0.16.0)
use np.linalg.slogdet where appropriate
group tests for multivariate distributions

ev-br · 2014-12-26T21:59:28Z

Might also want to mention it in the release notes already.

rgommers · 2014-12-26T22:08:52Z

good point, done.

coveralls · 2014-12-26T23:09:34Z

Coverage increased (+0.0%) when pulling ff92bb8 on rgommers:stats-cleanup into dc57092 on scipy:master.

… logdet(x) These are subtly different: In [46]: 2 * np.sum(np.log(linalg.cholesky(scale, lower=True).diagonal())) ... - np.log(np.linalg.det(scale)) Out[46]: 8.8817841970012523e-16 Use of ``np.linalg.slogdet`` is preferred. Related change in scipygh-4313.

rgommers · 2014-12-28T13:52:19Z

Added a couple more bug fixes.

argriffing · 2014-12-29T01:27:53Z

The commit rgommers@e3150fe may be a performance regression. It adds extra slogdet calls which are expensive. One of the points of using Cholesky decomposition is that it is faster than the LU decomposition used by slogdet.

rgommers · 2014-12-29T09:23:24Z

It's indeed slower:

In [16]: dim = 4

In [17]: scale = np.diag(np.arange(dim)+1)

In [18]: scale[np.tril_indices(dim, k=-1)] = np.arange(dim * (dim-1) // 2)

In [19]: scale = np.dot(scale.T, scale)

In [20]: scale
Out[20]: 
array([[11, 14, 18, 12],
       [14, 24, 26, 16],
       [18, 26, 34, 20],
       [12, 16, 20, 16]])

In [21]: 2 * np.sum(np.log(linalg.cholesky(scale, lower=True).diagonal()))
Out[21]: 6.3561076606958906

In [22]: %timeit 2 * np.sum(np.log(linalg.cholesky(scale, lower=True).diagonal()))
10000 loops, best of 3: 28.7 µs per loop

In [23]: %timeit np.linalg.slogdet(scale)
10000 loops, best of 3: 55.4 µs per loop

I don't think 30 us for something that is usually not called in inner loops is important though - I much prefer the maintainability/readability of slogdet. Note also that this particular computation is done on scale, which is never very large.

argriffing · 2014-12-29T16:25:48Z

I agree that 30 microseconds wouldn't be important except in an inner loop. In addition to the nitpick that the slowdown would be more like 3x than 2x because the Cholesky decomposition is computed anyway, I'd vote for log likelihood functions as candidates in the "most likely to be used in an inner loop" competition. Regarding the largeness of scale I'm not sure what sizes would be reasonable, but for large sizes this would be a bottleneck with no loop required. I'll ping @ChadFulton for his views on the cost/benefit of adding an slogdet call vs. using the diagonal of an already-computed Cholesky matrix.

ev-br · 2014-12-29T21:07:58Z

FWIW, I'm also in favor of reusing the Cholesky decomp here since it has been computed anyway. The readability cost is not too big IMO, and there is a cognitive burden of both "what is slogdet", and "why does it not use Cholesky, is there a reason for it"

ChadFulton · 2014-12-29T21:08:47Z

In principle, both maximum likelihood models and Metropolis-Hastings Bayesian computation of models fitting a multivariate normal may require a potentially large number of logpdf calls, where the size of the scale matrix would vary with the dimension of the normal.

I haven't personally used such a model (the models I am using take advantage of Gibbs Sampling methods, which requires lots of calls instead to rvs), but PyMC has the Wishart distribution available, and they use the Metropolis-Hastings approach. I looked at their PDF calculation, and they use log(det(scale)), where det eventually resolves to np.linalg.det, so maybe the performance difference isn't too critical.

On the other hand, given that the Cholesky is already being computed, the determinant calculation is essentially a constant (or grows very slowly) with respect to the size of scale, whereas slogdet will grow in some non-linear way.

Thanks to both of you for your work getting this merged.

rgommers · 2014-12-29T21:16:12Z

OK it seems you all agree. Then let's fix it by removing slogdet everywhere - the issue was that the test was doing assert_equal for something computed differently for the frozen dist as for the regular one. I'll update.

Fix docstring formatting and use np.linalg.slogdet where indicated by TODOs.

[ci skip]

… funcs. This raises DeprecationWarning's with recent numpy; should clearly be ints.

… logdet(x) These are subtly different: In [46]: 2 * np.sum(np.log(linalg.cholesky(scale, lower=True).diagonal())) ... - np.log(np.linalg.det(scale)) Out[46]: 8.8817841970012523e-16 Use of ``np.linalg.slogdet`` is preferred for readability/maintainability.

…gdet. This is faster, see discussion on scipygh-4313. Using it everywhere fixes the discrepancy between the normal and frozen distribution.

….vectorize Closes scipygh-4314.

rgommers · 2014-12-30T10:58:54Z

OK updated.

I think it would be useful at some point to either speed up np.linalg.slogdet or add an slogdet to scipy.linalg, because the last commit clearly doesn't help readability.

Also make all plots render properly.

coveralls · 2014-12-30T13:47:06Z

Coverage increased (+0.0%) when pulling a2a1aa3 on rgommers:stats-cleanup into 08238a5 on scipy:master.

argriffing · 2014-12-30T16:41:01Z

Looks good to me.

I think it would be useful at some point to either speed up np.linalg.slogdet

This speedup applies only to positive semidefinite matrices (and given the lack of singular Cholesky in numpy/scipy, only to positive definite matrices in practice). Stepping back, the larger issue is that although numpy and scipy are amazingly good at n-dimensional array hacking, interfacing with BLAS/LAPACK, and automatically dealing with dtypes and dispatching to the correct underlying functions (e.g. [sdcz]gemm), they have absolutely no dispatch scheme for matrix structure except for ad-hoc python function name differences or possibly one-off keyword arguments. In other words, if LAPACK functions are named like XYYZZZ where X is s/d/c/z indicating dtype, YY indicates matrix structure and ZZZ indicates the computation, then numpy/scipy automatically handles X via dtype but handles YYZZZ through ad-hoc means. Using this notation, the numpy logdet can deal with YY == 'GE', but the Cholesky logdet would require a structure more like YY == 'PO'.

rgommers · 2014-12-30T16:57:02Z

@argriffing thanks for the details, and interesting perspective on matrix structures.

argriffing · 2014-12-30T21:34:49Z

I guess this PR would conflict with #4318, I'm not sure the best way to deal with that.

rgommers · 2014-12-30T21:44:55Z

I'd merge this one first; it's good to go and the other one is easy to rebase.

ev-br · 2014-12-30T21:49:56Z

scipy/stats/_multivariate.py

-    >>> plt.contourf(x, y, rv.pdf(pos))
+    >>> fig2 = plt.figure()
+    >>> ax2 = fig2.add_subplot(111)
+    >>> ax2.contourf(x, y, rv.pdf(pos))


Any reason not to add plt.show(), so that the plot actually renders in the docs?

I get a plot in the HTML docs without including plt.show().

Interesting. What is the complete (correct) way of building the HTML docs you're using, what version of sphinx etc?

I use make html in the doc directory. The numpydoc extension in the numpy source code (i.e. numpy/doc/sphinxext) is in my python path. Sphinx version is 1.2.3.

I do the same, make html is the easiest. I don't have numpy/doc/sphinxext in my path but have simply installed numpydoc.

plt.show indeed isn't necessary on my local build, but for the ones on docs.scipy.org it does seem to be. May depend on the version of the plot-directive extension used.

plt.show() is used in most docstring examples, I'll add it.

I do make html-scipyorg and I do seem to need plt.plot(). Not sure what is different between the two.

This is surely a rather minor point, but all else being equal I'd rather have it implicit in docstrings (ditto for import matplotlib.pyplot as plt).

In any case, it's OT for this PR; my original comment is moot.

Hmm, I just checked and for all multivariate distributions adding plt.show() breaks generating the plots. So I suggest to leave it as is in this PR, and open a new issue for reconciling the different versions of building the docs (and/or a check on minimum Sphinx and numpydoc versions).

ev-br · 2014-12-31T14:32:37Z

Merging, thanks Ralf, all.

Some minor fixes for stats.wishart addition.

ev-br · 2014-12-31T14:41:06Z

The issue for tracking the docs build: #4346

rgommers added maintenance Items related to regular maintenance tasks scipy.stats labels Dec 26, 2014

rgommers mentioned this pull request Dec 26, 2014

ENH: Add Wishart and inverse Wishart distributions #4135

Merged

rgommers force-pushed the stats-cleanup branch from c65c8e6 to f93e8a3 Compare December 26, 2014 21:50

rgommers added this to the 0.16.0 milestone Dec 26, 2014

rgommers force-pushed the stats-cleanup branch from ff92bb8 to 0763413 Compare December 28, 2014 13:51

rgommers mentioned this pull request Dec 28, 2014

possible cleanup of np.vectorize usage after dropping numpy 1.5 #4314

Closed

WarrenWeckesser mentioned this pull request Dec 29, 2014

On one of the TravisCI configs, treat numpy deprecation warnings as errors or failures. #4327

Closed

Ralf Gommers added 7 commits December 30, 2014 11:50

MAINT: some fixes for stats.wishart added in scipygh-4135.

8b23555

Fix docstring formatting and use np.linalg.slogdet where indicated by TODOs.

TST: clean up stats._multivariate tests a bit.

2f0ae4f

DOC: add stats.wishart/invwishart to 0.16.0 release notes.

5dc6e13

[ci skip]

BUG: stats.wishart: fix issue with using floats for size arg to numpy…

4d79b3f

… funcs. This raises DeprecationWarning's with recent numpy; should clearly be ints.

MAINT: wishart: revert to using Cholesky decomposition instead of slo…

96a65af

…gdet. This is faster, see discussion on scipygh-4313. Using it everywhere fixes the discrepancy between the normal and frozen distribution.

MAINT: document in stats.distributions when to stop special casing np…

5047c4c

….vectorize Closes scipygh-4314.

rgommers force-pushed the stats-cleanup branch from 0763413 to 5047c4c Compare December 30, 2014 10:51

DOC: fix up some formatting issues and typos in stats.multivariate docs.

a2a1aa3

Also make all plots render properly.

ev-br reviewed Dec 30, 2014
View reviewed changes

ev-br added a commit that referenced this pull request Dec 31, 2014

Merge pull request #4313 from rgommers/stats-cleanup

75337bc

Some minor fixes for stats.wishart addition.

ev-br merged commit 75337bc into scipy:master Dec 31, 2014

ev-br mentioned this pull request Dec 31, 2014

different ways of building the docs are subtly different #4346

Closed

rgommers deleted the stats-cleanup branch December 31, 2014 14:41

Uh oh!

Some minor fixes for stats.wishart addition. #4313

Some minor fixes for stats.wishart addition. #4313

Uh oh!

Conversation

rgommers commented Dec 26, 2014

Uh oh!

ev-br commented Dec 26, 2014

Uh oh!

rgommers commented Dec 26, 2014

Uh oh!

coveralls commented Dec 26, 2014

Uh oh!

rgommers commented Dec 28, 2014

Uh oh!

argriffing commented Dec 29, 2014

Uh oh!

rgommers commented Dec 29, 2014

Uh oh!

argriffing commented Dec 29, 2014

Uh oh!

ev-br commented Dec 29, 2014

Uh oh!

ChadFulton commented Dec 29, 2014

Uh oh!

rgommers commented Dec 29, 2014

Uh oh!

rgommers commented Dec 30, 2014

Uh oh!

coveralls commented Dec 30, 2014

Uh oh!

argriffing commented Dec 30, 2014

Uh oh!

rgommers commented Dec 30, 2014

Uh oh!

argriffing commented Dec 30, 2014

Uh oh!

rgommers commented Dec 30, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ev-br commented Dec 31, 2014

Uh oh!

ev-br commented Dec 31, 2014

Uh oh!

Uh oh!