Skip to content

BUG: np.array cleared errors occured in PyMemoryView_FromObject #13507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 28, 2019

Conversation

superbobry
Copy link
Contributor

This commit changes PyMemoryView_FromObject call sites to only
clear BufferError and TypeError. Any other error will be raised
to the caller of np.array.


I'd be happy to add a test for this, but it requires writing an extension class
with a custom buffer interface implementation. Should I do that?

@charris charris added 00 - Bug 09 - Backport-Candidate PRs tagged should be backported component: numpy._core Priority: low Valid, but not for immediate attention and removed Priority: low Valid, but not for immediate attention labels May 8, 2019
@eric-wieser
Copy link
Member

This change isn't enough - there are two calls to GetBuffer done in succession above - you need to insert a check for an error between them else you risk hitting a SystemError.

@superbobry
Copy link
Contributor Author

Done, apologies for the delay.

@superbobry superbobry force-pushed the pymemoryview-fromobject-errors branch from b0e2599 to de06bf6 Compare May 14, 2019 20:30
@mattip
Copy link
Member

mattip commented May 16, 2019

It does seem that this logic is sufficiently convoluted to justify a test, but it seems such a test would require a custom bf_getbuffer function and a lot of C code. If you are comfortable working with PyPy (which we test in the CI), it exposes bf_getbuffer as a __buffer__ attribute on pure python classes that inherit from __pypy__.bufferable, so theoretically you could write a pure-python test and mark it @pytest.skipif(not IS_PYPY, reason='only PyPy has bufferable class), write a ndarray subclass, override the __buffer__ attribute, and check that it asserts properly.

Let me know if you want help with that

pull bot pushed a commit to Rachelmorrell/tensorflow that referenced this pull request May 24, 2019
While this change makes EagerTensor.numpy __array__ redundant, it
does not remove __array__ due to numpy/numpy#13507.

Note also that unlike  __array__, the buffer interface does not lead
to a performance regression when np.array infers the dimensionality
of [tensor]. See tensorflow#27692 and numpy/numpy#8562 for details.

PiperOrigin-RevId: 249888195
This commit changes PyMemoryView_FromObject call sites to only
clear BufferError and TypeError. Any other error will be raised
to the caller of np.array.
@mattip mattip force-pushed the pymemoryview-fromobject-errors branch from de06bf6 to cfb5749 Compare May 28, 2019 08:33
@mattip mattip merged commit 51ebd80 into numpy:master May 28, 2019
@mattip
Copy link
Member

mattip commented May 28, 2019

Thanks @superbobry

@superbobry
Copy link
Contributor Author

@mattip sorry for not commenting earlier. I implemented a test but discovered more calls to PyObject_GetBuffer which need to be handled. I think it might be best to revert this PR until all of the calls are covered.

@mattip
Copy link
Member

mattip commented May 29, 2019

@superbobry is there harm in what I merged? It is fine to fix more in a follow-on, as long as the current state has no regressions from the previous one.

@mattip
Copy link
Member

mattip commented May 29, 2019

Fwiw, the xref to the pandas issue is wrong, that was due to #13399 and a fix proposed in #13663

@superbobry
Copy link
Contributor Author

No, no harm, but the commit description might be misleading :)

@charris charris added this to the 1.16.5 release milestone Jul 14, 2019
@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Jul 17, 2019
@charris charris removed this from the 1.16.5 release milestone Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants