Skip to content

BUG: Fix packbits to correctly handle empty arrays #8327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 2, 2016

Conversation

iwiwi
Copy link
Contributor

@iwiwi iwiwi commented Nov 29, 2016

This small PR addresses several problems of numpy.packbits pointed out in Issue #8324.

As discussed in Issue #8324, numpy.packbits have problems with regard to empty input arrays. Specifically:

  • the output does not necessarily always have uint8 dtype,
  • the shape of the returned array is inconsistent when axis is specified, and
  • there is a leak of an object (new).

The fundamental reason for these problems is common: function pack_bits in compiled_base.c handles empty arrays separately, in a rough manner. I could not figure out a good reason why we should handle empty arrays separately. Rather, I found that this part is indeed unnecessary, i.e., without this, pack_bits can handle empty arrays correctly, as confirmed in my new tests. So, I propose to remove that part to fix these problems.

@juliantaylor
Copy link
Contributor

please squash the test and the fix into one commit, it is better to have the testsuite succeed on every commit for easier bisection.

@iwiwi iwiwi force-pushed the fix-packbits-empty branch from 330c4f9 to 41d10b5 Compare November 30, 2016 02:57
@iwiwi
Copy link
Contributor Author

iwiwi commented Nov 30, 2016

Hi @juliantaylor, thank you for your quick feedback. I squashed the two commits into one commit.

@@ -17,6 +17,22 @@ def test_packbits():
assert_raises(TypeError, np.packbits, np.array(a, dtype=float))


def test_packbits_empty():
for dtype in [np.bool, np.uint8, np.int]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why just these three types? And why just the last dimension zero? Tests should be exhaustive. Admittedly, most aren't but, strangely enough, when improved often reveal errors.

All integer types are available in np.typecodes['AllInteger'], the boolean type is separate. Or you could just use the string '?bBhHiIlLqQ' for typecodes. If boolean is acceptable I think the docstring need improvement also.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also not a good idea to shadow the numpy dtype variable. Use something like dt instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it checked anywhere that empty results unpack correctly?

@iwiwi iwiwi force-pushed the fix-packbits-empty branch from 41d10b5 to fe7448f Compare December 1, 2016 05:32
@iwiwi
Copy link
Contributor Author

iwiwi commented Dec 1, 2016

Hi @charris, thank you for the detailed and accurate feedback!

Why just these three types?
And why just the last dimension zero?
Also not a good idea to shadow the numpy dtype variable. Use something like dt instead.

According to your comments, I improved the tests in test_packbits.py.

If boolean is acceptable I think the docstring need improvement also.

I updated the statement in add_newdoc.py.

Is it checked anywhere that empty results unpack correctly?

You are correct --- I added tests for unpack against empty arrays, and indeed, I found a similar bug in unpack. So, I also updated function unpack_bits in compiled_base.c in the same way as pack_bits.

assert_array_equal(b, np.empty((0,)))


def test_packbits_empty_with_axis():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function has the wrong (duplicate) name, so only one of them runs

@iwiwi iwiwi force-pushed the fix-packbits-empty branch from fe7448f to 47c66c9 Compare December 2, 2016 01:48
@iwiwi
Copy link
Contributor Author

iwiwi commented Dec 2, 2016

Sorry for stupid mistake. fixed.

@charris charris merged commit d5ba9c7 into numpy:master Dec 2, 2016
@charris charris added this to the 1.12.0 release milestone Dec 2, 2016
@iwiwi iwiwi deleted the fix-packbits-empty branch December 3, 2016 00:38
@charris charris removed this from the 1.12.0 release milestone Dec 3, 2016
@juliantaylor
Copy link
Contributor

thanks, very nice pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants