BUG: Preserve identity of dtypes in make_mask_descr #8667

eric-wieser · 2017-02-22T11:09:10Z

Partially addresses #8666

mhvk

Looks good, with some nitpicks that really are a matter of taste, so fee free to ignore.

mhvk · 2017-02-22T16:12:05Z

numpy/ma/core.py

@@ -1296,14 +1296,23 @@ def _recursive_make_descr(datatype, newtype=bool_):
                # Prepend the title to the name
                name = (field[-1], name)
            descr.append((name, _recursive_make_descr(field[0], newtype)))
-        return descr
+        newtype = np.dtype(descr)


Maybe instead of calling this newtype (which is confusing as it is also a passed-in argument), just keep calling it descr?

mhvk · 2017-02-22T16:12:16Z

numpy/ma/core.py

    # Is this some kind of composite a la (np.float,2)
    elif datatype.subdtype:
        mdescr = list(datatype.subdtype)
        mdescr[0] = _recursive_make_descr(datatype.subdtype[0], newtype)
-        return tuple(mdescr)
+        newtype = np.dtype(tuple(mdescr))


mhvk · 2017-02-22T16:12:49Z

numpy/ma/core.py

    else:
-        return newtype


I think for this private function, we should be able to count on newtype being a dtype already, so I would leave the return newtype here. (I realise that means updating the two occurrences where "O" is passed in.)

Every top-level call-site in the file fails to pass in a dtype right now (either MaskType, 'O', or bool, I think), so this is absolutely required. Sure, we could change all the callsites, I guess...

I think that would be better, but it is a very mild preference at best, so also fine to leave it. For easy following of the flow of logic, I would still think it is best to not reuse the newtype name, though (it really threw me off, to convince me that the recursive calls could not pick that up).

If we do that, then I'd be tempted to change MaskType from np.bool_ to np.dtype(np.bool_)

mhvk · 2017-02-22T16:19:19Z

My feeling would be to have this for 1.13 rather than 1.12.1. What do you think?

mhvk · 2017-02-22T16:20:01Z

numpy/ma/core.py

+        newtype = np.dtype(newtype)
+
+    # preserve identity of dtypes
+    if newtype == datatype:


With the above, the np.dtype calls can be here, i.e.,

descr = np.dtype(descr) if descr == datatype: return datatype else: return descr

eric-wieser · 2017-02-22T16:25:52Z

Yeah, I agree - no pressure for 12.1 here

eric-wieser · 2017-02-23T00:35:29Z

@mhvk : Made the changes sort of like you requested

eric-wieser · 2017-02-23T00:35:49Z

numpy/ma/core.py

@@ -1285,25 +1285,48 @@ def __str__(self):
 ###############################################################################


-def _recursive_make_descr(datatype, newtype=bool_):
-    "Private function allowing recursion in make_descr."
+def _replace_dtype_fields_recursive(dtype, primitive_dtype):


_make_descr was a pretty bad name

eric-wieser · 2017-02-23T00:36:04Z

numpy/ma/core.py

+    """
+    dtype = np.dtype(dtype)
+    primitive_dtype = np.dtype(primitive_dtype)
+    return _replace_dtype_fields_recursive(dtype, primitive_dtype)


Now do the dtype conversions just once, so we don't need to again

eric-wieser · 2017-02-23T00:36:50Z

numpy/ma/core.py

+    return new_dtype
+
+
+def _replace_dtype_fields(dtype, primitive_dtype):


Possible scope for public visibility? I can see "convert all fields to object/bool" being marginally desirable elsewhere

eric-wieser · 2017-02-23T00:37:22Z

numpy/ma/core.py

@@ -3834,7 +3854,7 @@ def __str__(self):
                    res = data.astype("O")
                    res.view(ndarray)[mask] = f
                else:
-                    rdtype = _recursive_make_descr(self.dtype, "O")
+                    rdtype = _replace_dtype_fields(self.dtype, "O")


Implementation details of mask_make_descr shouldn't be used elsewhere!

eric-wieser · 2017-02-23T00:38:09Z

numpy/ma/core.py

+    """
+    dtype = np.dtype(dtype)
+    primitive_dtype = np.dtype(primitive_dtype)
+    return _replace_dtype_fields_recursive(dtype, primitive_dtype)


 def make_mask_descr(ndtype):


Ideally I'd rename this to make_mask_dtype or mask_dtype_for, but that's for another PR.

mhvk · 2017-02-23T00:44:44Z

I like this very much! My only comment is that with this change it is as much MAINT as it is BUG...

eric-wieser · 2017-02-23T00:54:20Z

Arguably BUG should be ENH here anyway, since identity comparison of dtypes is not really promised anywhere

ahaldane · 2017-03-03T02:54:54Z

numpy/ma/core.py

            if len(field) == 3:
                # Prepend the title to the name
                name = (field[-1], name)
-            descr.append((name, _recursive_make_descr(field[0], newtype)))
-        return descr
+            descr.append((name, _replace_dtype_fields_recursive(field[0], primitive_dtype)))


line is too long (more below too)

ahaldane · 2017-03-03T03:15:07Z

So if I understand, the main bug fix is the test for if new_dtype == dtype: new_dtype = dtype and the other changes are for style? That's fine, I just want to understand the motivation for the other changes.

eric-wieser · 2017-03-03T11:50:28Z

@ahaldane: Yep, that's correct. I'll rewrite this into two commits to emphasize that

Partially addresses numpy#8666

Cleans up make_mask_descr, and adds a private _replace_dtype_fields, which we need elsewhere to replace all dtypes with `object` instead of `bool`. This new version also removes repeated calls to `np.dtype`, doing the conversion only once.

eric-wieser · 2017-03-03T12:49:42Z

@ahaldane: Rebased, and long lines shortened. First commit is enough to make the tests pass

eric-wieser · 2017-03-03T12:51:12Z

numpy/ma/core.py

@@ -1334,13 +1359,10 @@ def make_mask_descr(ndtype):
    >>> ma.make_mask_descr(dtype)
    dtype([('foo', '|b1'), ('bar', '|b1')])
    >>> ma.make_mask_descr(np.float32)
-    <type 'numpy.bool_'>
+    dtype('bool')


This was already the behaviour before the PR - the docs were just wrong

mhvk · 2017-03-03T19:25:28Z

I'm now even happier with this, so @ahaldane, if you agree as well, I think this can just be merged.

ahaldane · 2017-03-06T16:57:51Z

Looks good, I'll go ahead with the merge then. Thanks @eric-wieser !

eric-wieser added 00 - Bug component: numpy.ma masked arrays labels Feb 22, 2017

eric-wieser force-pushed the reuse-ma-dtype branch 3 times, most recently from 53ed1c8 to 9defd18 Compare February 22, 2017 11:20

mhvk approved these changes Feb 22, 2017

View reviewed changes

mhvk reviewed Feb 22, 2017

View reviewed changes

mhvk added this to the 1.13.0 release milestone Feb 22, 2017

eric-wieser force-pushed the reuse-ma-dtype branch 2 times, most recently from ef5a1a3 to c6a2fe3 Compare February 22, 2017 22:44

eric-wieser commented Feb 23, 2017

View reviewed changes

ahaldane reviewed Mar 3, 2017

View reviewed changes

eric-wieser force-pushed the reuse-ma-dtype branch 2 times, most recently from 8115a12 to e13cf08 Compare March 3, 2017 12:45

eric-wieser added 2 commits March 3, 2017 12:47

BUG: Preserve identity of dtypes in make_mask_descr

02f3388

Partially addresses numpy#8666

eric-wieser force-pushed the reuse-ma-dtype branch from e13cf08 to b1a7057 Compare March 3, 2017 12:49

eric-wieser commented Mar 3, 2017

View reviewed changes

ahaldane merged commit 66f1b8a into numpy:master Mar 6, 2017

		return new_dtype


		def _replace_dtype_fields(dtype, primitive_dtype):

Uh oh!

BUG: Preserve identity of dtypes in make_mask_descr #8667

BUG: Preserve identity of dtypes in make_mask_descr #8667

Uh oh!

Conversation

eric-wieser commented Feb 22, 2017

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk commented Feb 22, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Feb 22, 2017

Uh oh!

eric-wieser commented Feb 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk commented Feb 23, 2017

Uh oh!

eric-wieser commented Feb 23, 2017

Uh oh!

ahaldane Mar 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahaldane commented Mar 3, 2017

Uh oh!

eric-wieser commented Mar 3, 2017

Uh oh!

eric-wieser commented Mar 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk commented Mar 3, 2017

Uh oh!

ahaldane commented Mar 6, 2017

Uh oh!

Uh oh!

ahaldane Mar 3, 2017 •

edited

Loading