MAINT: _lib: add __dealloc__ to MessageStream

[tirthasheshpatel]

Reference issue

N/A

What does this implement/fix?

Extension class MessageStream doesn't implement a __dealloc__ method which is required to free C level structures once the object goes out of scope. It is possible to do so with a call to the close() method but it is very inconvenient to call it every time one doesn't need the message stream anymore. Also, there could be a memory leak if one forgot to call the close method. Using __dealloc__, we can tell Cython to manage memory automatically for us.

Additional information

This is not in the public API but I was planning to use this in #14215 for error handling and it would help if this method was present.

[peterbell10]

Can't you just rename __del__ to __dealloc__? You might also have to change def close to cpdef close.

[tirthasheshpatel]

Can't you just rename __del__ to __dealloc__?

Yes, in fact, we should do this. From the documentation:

Note: There is no __del__() method for extension types

You might also have to change def close to cpdef close

That sounds right.

[tirthasheshpatel]

Done, thanks!

[rgommers]

There's still something that looks weird. As the Cython docs say, __dealloc__ is the counterpart to __cinit__, and should be used for explicit handling of C-level initialization. MessageStream doesn't have a __cinit__.

The first part of __init__ is:

        self._memstream_ptr = NULL
        self.handle = messagestream_open_memstream(&self._memstream_ptr,
                                                   &self._memstream_size)

Potential issues:

• self._memstream_size doesn't exist at all
• This bit should live in __cinit__ if I understood correctly

[tirthasheshpatel]

* `self._memstream_size` doesn't exist at all

I don't think that should happen because the variable _memstream_size is declared in messagestream.pxd as a C type which are always available by the time __init__ is called.

* This bit should live in `__cinit__` if I understood correctly

I think you are right here. According to Cython docs, its best to put all the C level initialization code in __cinit__ method. We can do that here by changing the __init__ method to __cinit__ because all the variables are declared as C objects in the .pxd file. Do you agree?

Update: I tried to do this and I get a RuntimeWarning during build and import: <frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject

[rgommers]

Update: I tried to do this and I get a RuntimeWarning during build and import:

Hmm interesting. I'm not entirely sure. Let's see if someone else who actually understands what Cython is doing under the hood will comment.

[tirthasheshpatel]

Hmm interesting. I'm not entirely sure. Let's see if someone else who actually understands what Cython is doing under the hood will comment.

Never mind the warning. I had to rebuild scipy from scratch. I think Cython was just picking up between the conflicting code generated by previous version of MessageStream that had the __init__ method. After rebuilding, tests pass locally without errors/warnings.

[tirthasheshpatel]

Hi @rgommers! Any thoughts on this? If you are happy with this, we can merge because this is required to resolve memory leaks in gh-14215.

[mckib2]

From the docs:

In particular, don’t call any other methods of the object or do anything which might cause the object to be resurrected. It’s best if you stick to just deallocating C data.

This is probably fine as close() is cpdef'd and thus as a C version is available even if the Python class is in an invalid state

[mckib2]

This looks like a good change to me and a win for low-level memory management safety. If there are no further comments, I'll plan on merging this later this evening

[tirthasheshpatel]

Thanks @mckib2 for the review!