Thread: [matplotlib-devel] [Fwd: [wxPython-users] Re: using wxImage in C++ python extension]

As a probably final installment in the thread about optimizing the wx 
back-end, here is a post from the wxPython list, in which someone posted 
SWIG code for making a PyBuffer from his data set, then using it to 
create a wx.Image without copying. A similar approach could be used for 
the wxAgg back-end.

-Chris

-------- Original Message --------
Subject: [wxPython-users]  Re: using wxImage in C++ python extension
Date: Fri, 18 Aug 2006 16:48:08 +0000 (UTC)
From: Andrew Murray <and...@st...>
Reply-To: wxP...@li...
To: wxp...@li...
References: <200...@uc...> 
<44E...@al...> <200...@uc...> 
<44E...@no...> <44E...@al...> 
<44E...@no...>

Hi all :)

I've followed the combined advice of Robin and Christopher in using a python
buffer object to instance a wx.Image in the python wrapper layer.  It 
all seems
to be working - and I don't think there are any 'data copies' going on ;)

To keep my underlying C++ python-free, I introduced the buffer (and 
creation of
the wx.Image) via my .i Swig file.  In case anyone who is reading is 
keen trying
to solve a similar problem, I've included the code that I added to my .i 
file in
order to implement the change below. (I'd appreciate any comments regarding
errors or shortcomings that anyone spots in my implementation.)

Sorry I didn't reply to the thread earlier.  A combination of being tied 
up with
more boring work, moving house and background research into using python 
buffers
with Swig means that I've only just got this far.

One remaining question I have is: Does the call to wx.EmptyImage that I make
cause any memory to be allocated?  I see from Robin's recent post that in my
case this call is now redundant (as I will soon be using 
ImageFromBuffer) - but
I'm curious to know the answer anyway ;)

Thanks for all the help.  The more I use wxPython the more I like it...

Andrew ;)


/* the RawImage C++ class that we are wrapping offers a method that
    returns a pointer to an internal 'unsigned char' buffer of display
    data.  To make the python class generated by SWIG a bit more
    friendly to the rest of our wxPython code we instruct SWIG to make
    two modifications during the wrapping process.  First we add a
    method to the C++ class that will return a version of the internal
    display buffer wrapped up as a 'python buffer object' (performing
    this in the C++ wrapper class keeps the wrapped C++ python-free)...
  */
%extend RawImage {
	PyObject* RawImage::getDisplayBuffer(void) {
		// return a new 'python buffer object' that intialised using
		// the memory address and length of our display buffer...
		void* pvBufferData = (void*)self->pcGetDisplayData();
		int iBufferSize = self->getDisplayBufferSize();
		return PyBuffer_FromMemory(pvBufferData, iBufferSize);
	}
}
/* ... we also add a method to the python wrapper class that will use
    the buffer offered by our new C++ method to create a wx.Image
    (neither of these methods actually copies the image data)...
  */
%extend RawImage {
%pythoncode %{
def getImage(self):
     # create an unintialised wx.Image of the correct size...
     wximage = wx.EmptyImage(self.getWidth(), self.getHeight(), clear=False)
     # then define the data content of the image...
     wximage.SetDataBuffer(self.getDisplayBuffer())
     return wximage
%}
}


---------------------------------------------------------------------
To unsubscribe, e-mail: wxP...@li...
For additional commands, e-mail: wxP...@li...


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Chris,

Thanks for the cross-post.  I'm not sure this approach will help  
speed up the wxAgg accelerator, but I'll put it on the list of things  
to look into.  The problem I foresee is that the Agg renderer's RGBA  
data has to be converted to RGB before a wxImage can be created by  
convert_agg2image().

Ken

On Aug 18, 2006, at 6:14 PM, Christopher Barker wrote:

> As a probably final installment in the thread about optimizing the wx
> back-end, here is a post from the wxPython list, in which someone  
> posted
> SWIG code for making a PyBuffer from his data set, then using it to
> create a wx.Image without copying. A similar approach could be used  
> for
> the wxAgg back-end.
>
> -Chris
>
> -------- Original Message --------
> Subject: [wxPython-users]  Re: using wxImage in C++ python extension
> Date: Fri, 18 Aug 2006 16:48:08 +0000 (UTC)
> From: Andrew Murray <and...@st...>
> Reply-To: wxP...@li...
> To: wxp...@li...
> References: <200...@uc...>
> <44E...@al...> <200...@uc...>
> <44E...@no...> <44E...@al...>
> <44E...@no...>
>
> Hi all :)
>
> I've followed the combined advice of Robin and Christopher in using  
> a python
> buffer object to instance a wx.Image in the python wrapper layer.  It
> all seems
> to be working - and I don't think there are any 'data copies' going  
> on ;)
>
> To keep my underlying C++ python-free, I introduced the buffer (and
> creation of
> the wx.Image) via my .i Swig file.  In case anyone who is reading is
> keen trying
> to solve a similar problem, I've included the code that I added to  
> my .i
> file in
> order to implement the change below. (I'd appreciate any comments  
> regarding
> errors or shortcomings that anyone spots in my implementation.)
>
> Sorry I didn't reply to the thread earlier.  A combination of being  
> tied
> up with
> more boring work, moving house and background research into using  
> python
> buffers
> with Swig means that I've only just got this far.
>
> One remaining question I have is: Does the call to wx.EmptyImage  
> that I make
> cause any memory to be allocated?  I see from Robin's recent post  
> that in my
> case this call is now redundant (as I will soon be using
> ImageFromBuffer) - but
> I'm curious to know the answer anyway ;)
>
> Thanks for all the help.  The more I use wxPython the more I like  
> it...
>
> Andrew ;)
>
>
> /* the RawImage C++ class that we are wrapping offers a method that
>     returns a pointer to an internal 'unsigned char' buffer of display
>     data.  To make the python class generated by SWIG a bit more
>     friendly to the rest of our wxPython code we instruct SWIG to make
>     two modifications during the wrapping process.  First we add a
>     method to the C++ class that will return a version of the internal
>     display buffer wrapped up as a 'python buffer object' (performing
>     this in the C++ wrapper class keeps the wrapped C++ python- 
> free)...
>   */
> %extend RawImage {
> 	PyObject* RawImage::getDisplayBuffer(void) {
> 		// return a new 'python buffer object' that intialised using
> 		// the memory address and length of our display buffer...
> 		void* pvBufferData = (void*)self->pcGetDisplayData();
> 		int iBufferSize = self->getDisplayBufferSize();
> 		return PyBuffer_FromMemory(pvBufferData, iBufferSize);
> 	}
> }
> /* ... we also add a method to the python wrapper class that will use
>     the buffer offered by our new C++ method to create a wx.Image
>     (neither of these methods actually copies the image data)...
>   */
> %extend RawImage {
> %pythoncode %{
> def getImage(self):
>      # create an unintialised wx.Image of the correct size...
>      wximage = wx.EmptyImage(self.getWidth(), self.getHeight(),  
> clear=False)
>      # then define the data content of the image...
>      wximage.SetDataBuffer(self.getDisplayBuffer())
>      return wximage
> %}
> }
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: wxP...@li...
> For additional commands, e-mail: wxPython-users- 
> he...@li...
>
>
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
>                                      		
> NOAA/OR&R/HAZMAT         (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chr...@no...
>
> ---------------------------------------------------------------------- 
> ---
> Using Tomcat but need to do more? Need to support web services,  
> security?
> Get stuff done quickly with pre-integrated technology to make your  
> job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache  
> Geronimo
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Ken McIvor wrote:
> I'll put it on the list of things to look into.

Great. I'm glad someone is working on this.

> The problem I foresee is that the Agg renderer's RGBA data has to 
> be converted to RGB before a wxImage can be created by convert_agg2image().

darn. I figured as much. wx is really due for an update to support alpha 
properly. But I guess you'll always have problems with different data 
formats.

 > I'm not sure this approach will help speed
 > up the wxAgg accelerator, but

Anyway, this thread started because people were having binary 
compatibility issues. Even if this doesn't speed up the accelerator, it 
may be possible to get the same performance without using 
wx-version-specific compiled code -- i.e. pure python.

I do have one question -- does the agg back-end really need to use an 
alpha channel for it's buffer? Isn't it the whole image anyway? What is 
is it going to get blended with?

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Ken McIvor wrote:
> The problem I foresee is that the Agg renderer's RGBA data has to 
> be converted to RGB before a wxImage can be created by convert_agg2image().

As if by magic, this from Robin Dunn today:
> You may want to take a look at my CVS commits for the last couple weeks.  I've now got some raw bitmap access code in place.  Both 2.6 and 2.7 will have wx.BitmapFromBuffer and wx.BitmapFromBufferRGBA factory functions which can copy from a buffer object directly into the bitmap's pixel buffer, and 2.7 will also have wx.NativePixelData and wx.AlphaPixelData which allow direct access to the pixel buffer from Python.  (The latter needed a bug fix that I'm not sure (yet) can be backported to 2.6...)  For example, I can now do this (in a PyShell):
> 
>  >>> import wx
>  >>> import numarray
>  >>> f = wx.Frame(None)
>  >>> p = wx.Panel(f)
>  >>> dc = wx.ClientDC(p)
>  >>> f.Show()
>  >>> dim=100
>  >>> R=0; G=1; B=2; A=3
>  >>> arr = numarray.array(shape=(dim, dim, 4), typecode='u1')
>  >>> for row in xrange(dim):
>  ...     for col in xrange(dim):
>  ...        arr[row,col,R] = 0
>  ...        arr[row,col,G] = 0
>  ...        arr[row,col,B] = 255
>  ...        arr[row,col,A] = int(col * 255.0 / dim)
>  ...
>  >>> bmp = wx.BitmapFromBufferRGBA(dim, dim, arr)
>  >>> dc.DrawBitmap(bmp, 20, 20, True)

This is looking pretty promising.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

On 08/28/06 12:19, Christopher Barker wrote:
>
> wx is really due for an update to support alpha properly. But I guess
> you'll always have problems with different data formats.

I think they added preliminary support for alpha channels in 2.5, in the form 
of wx.Image.HasAlpha().  My beef is that you have to convert everything to a 
wx.Bitmap before you can do anything useful with it.

> Anyway, this thread started because people were having binary 
> compatibility issues. Even if this doesn't speed up the accelerator, it 
> may be possible to get the same performance without using 
> wx-version-specific compiled code -- i.e. pure python.

As near as I can tell, the primary slowdown at this point is the way wxWidgets 
distinguishes from RGB image data (wx.Image) as opposed to displayed image 
data (wx.Bitmap).  Right now you cannot draw a wx.Image without first 
converting it into a wx.Bitmap, nor can you use a MemoryDC to blit or 
otherwise munge a wx.Image directly.  My impression is that this made sense 
when wxWindows was getting started (Win16 and Motif), but is more of an 
artificial distinction at this point.

I don't think we're going to be able to get performance similar to that of the 
accelerator using straight Python code unless something changes in the 
wxWidgets' Image/Bitmap/MemoryDC department.  That being said, I'd love to be 
proven wrong!  If you're interested in the gory details, you should check out 
the pure-Python implementation of the image conversion functions, at the end 
of `backend_wxagg.py'.

> I do have one question -- does the agg back-end really need to use an 
> alpha channel for it's buffer? Isn't it the whole image anyway? What is 
> is it going to get blended with?

I don't know enough about Agg to venture an educated guess.  My un-educated 
guess is that there's an RGBA buffer to support alpha in the drawing 
operations... how can Agg alpha-composite new pixels into the buffer when you 
draw something, unless you know the alpha values of the existing pixels?

Ken

Ken McIvor wrote:
> I think they added preliminary support for alpha channels in 2.5, in the 
> form of wx.Image.HasAlpha().

right, but it uses the old 24 bit RGB buffer, and separate Alpha buffer, 
it's kind of tacked on, rather than native.

 > My beef is that you have to convert
> everything to a wx.Bitmap before you can do anything useful with it.

And that DCs don't support alpha, even though the underlying device 
often does. This has been discuses a lot, but no one has done much about 
it yet -- wxTNG will have something better, but who knows how far out 
that is?

> As near as I can tell, the primary slowdown at this point is the way 
> wxWidgets distinguishes from RGB image data (wx.Image) as opposed to 
> displayed image data (wx.Bitmap).  Right now you cannot draw a wx.Image 
> without first converting it into a wx.Bitmap, nor can you use a MemoryDC 
> to blit or otherwise munge a wx.Image directly.  My impression is that 
> this made sense when wxWindows was getting started (Win16 and Motif), 
> but is more of an artificial distinction at this point.

a wxBitmap is the same format as the native rendering system. While most 
systems use 24b RGB (or 32b RGBA), people can still run displays at 
16bpp or whatever, so it's still needed. Also, I wouldn't be surprised 
if some less common systems use ARGB or something else weird.

> I don't think we're going to be able to get performance similar to that 
> of the accelerator using straight Python code

But whether it's Python or C++, you still need to do the Image->Bitmap 
conversion -- so if we can get rid of the data copying from Agg buffer 
to wxImage in Python, we don't need C++.

> unless something changes 
> in the wxWidgets' Image/Bitmap/MemoryDC department.

And it has. For wxPython 2.7 (and now in CVS) there are methods for 
dumping 32 bit RGBA data directly into a wxBitmap with no copying, if 
the data source is a Python Buffer object. I think I posted a note about 
this here yesterday.

> I'd love to be proven wrong!  If you're interested in the gory details, 
> you should check out the pure-Python implementation of the image 
> conversion functions, at the end of `backend_wxagg.py'.

I did, and I suggested some improvements a couple messages back. To 
really get it to work, the 24bit RGB Agg buffer needs to be a Python 
Buffer object -- is it now? I'm sorry I don't have the time to mess with 
this now -- maybe some day.

>> I do have one question -- does the agg back-end really need to use an 
>> alpha channel for it's buffer? Isn't it the whole image anyway? What 
>> is is it going to get blended with?
> 
> I don't know enough about Agg to venture an educated guess.  My 
> un-educated guess is that there's an RGBA buffer to support alpha in the 
> drawing operations... how can Agg alpha-composite new pixels into the 
> buffer when you draw something, unless you know the alpha values of the 
> existing pixels?

You can alpha composite into a non-alpha background. You just lose the 
alpha there, so that the background couldn't be alpha-composited onto 
anything else -- but does it ever need to be?

However, there is something to be said for just using alpha everywhere, 
and as we'll soon be able to dump RGBA data straight into a wx.Bitmap, 
this should work great.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

On 08/31/06 13:43, Christopher Barker wrote:
> Ken McIvor wrote:
> 
> a wxBitmap is the same format as the native rendering system. While most 
> systems use 24b RGB (or 32b RGBA), people can still run displays at 
> 16bpp or whatever, so it's still needed.

I can understand why it's still necessary, although it's nice to sometimes 
pretend that everyone's running 24-bit color displays.  I hope I didn't sound 
too judgemental!

>>I don't think we're going to be able to get performance similar to that 
>>of the accelerator using straight Python code
>
> But whether it's Python or C++, you still need to do the Image->Bitmap 
> conversion -- so if we can get rid of the data copying from Agg buffer 
> to wxImage in Python, we don't need C++.

I think we got some wires crossed at some point in the conversation, although 
it could be that I'm wearing the Stupid Hat today.  I was talking about the 
image-from-a-buffer business not helping us with WX 2.4/2.6 due to the RGBA to 
RGB conversion.

> And it has. For wxPython 2.7 (and now in CVS) there are methods for 
> dumping 32 bit RGBA data directly into a wxBitmap with no copying, if 
> the data source is a Python Buffer object. I think I posted a note about 
> this here yesterday.

Yes, you did mention it.  I agree completely with this analysis of the 
situation.  When I replied I wasn't thinking in terms of wxPython 2.7.

> To really get it to work, the 24bit RGB Agg buffer needs to be a Python 
> Buffer object -- is it now? I'm sorry I don't have the time to mess with 
> this now -- maybe some day.

I guess Guido lets John borrow his time machine, because RendererAgg appears 
to already have a buffer_rgba() method.

> You can alpha composite into a non-alpha background. You just lose the 
> alpha there, so that the background couldn't be alpha-composited onto 
> anything else -- but does it ever need to be?

I thought that the buffer's accumulated alpha played a role in compositing new 
pixels onto it, but I apparently misunderstood.  It must be time to read 
"Compositing Digital Images.  Anyway, if the buffer's alpha channel isn't 
used, then the whole situation does seem a bit odd.  Could the information be 
retained for PNGs or something?

> However, there is something to be said for just using alpha everywhere, 
> and as we'll soon be able to dump RGBA data straight into a wx.Bitmap, 
> this should work great.

Yes, it will be a great improvement over the current situation.

Ken

>>>>> "Ken" == Ken McIvor <mc...@ii...> writes:

    Ken> On 08/31/06 13:43, Christopher Barker wrote:
    >> Ken McIvor wrote:
    >> 
    >> a wxBitmap is the same format as the native rendering
    >> system. While most systems use 24b RGB (or 32b RGBA), people
    >> can still run displays at 16bpp or whatever, so it's still
    >> needed.

    Ken> I can understand why it's still necessary, although it's nice
    Ken> to sometimes pretend that everyone's running 24-bit color
    Ken> displays.  I hope I didn't sound too judgemental!

    >>> I don't think we're going to be able to get performance
    >>> similar to that of the accelerator using straight Python code
    >>  But whether it's Python or C++, you still need to do the
    >> Image->Bitmap conversion -- so if we can get rid of the data
    >> copying from Agg buffer to wxImage in Python, we don't need
    >> C++.

    Ken> I think we got some wires crossed at some point in the
    Ken> conversation, although it could be that I'm wearing the
    Ken> Stupid Hat today.  I was talking about the
    Ken> image-from-a-buffer business not helping us with WX 2.4/2.6
    Ken> due to the RGBA to RGB conversion.

    >> And it has. For wxPython 2.7 (and now in CVS) there are methods
    >> for dumping 32 bit RGBA data directly into a wxBitmap with no
    >> copying, if the data source is a Python Buffer object. I think
    >> I posted a note about this here yesterday.

    Ken> Yes, you did mention it.  I agree completely with this
    Ken> analysis of the situation.  When I replied I wasn't thinking
    Ken> in terms of wxPython 2.7.

    >> To really get it to work, the 24bit RGB Agg buffer needs to be
    >> a Python Buffer object -- is it now? I'm sorry I don't have the
    >> time to mess with this now -- maybe some day.

    Ken> I guess Guido lets John borrow his time machine, because
    Ken> RendererAgg appears to already have a buffer_rgba() method.

Guido has been very generous with us :-)

    >> You can alpha composite into a non-alpha background. You just
    >> lose the alpha there, so that the background couldn't be
    >> alpha-composited onto anything else -- but does it ever need to
    >> be?

    Ken> I thought that the buffer's accumulated alpha played a role
    Ken> in compositing new pixels onto it, but I apparently
    Ken> misunderstood. 

It does: here is agg's rgba pixel blending routing

        static AGG_INLINE void blend_pix(value_type* p, 
                                         unsigned cr, unsigned cg, unsigned cb,
                                         unsigned alpha, 
                                         unsigned cover=0)
        {
            calc_type r = p[Order::R];
            calc_type g = p[Order::G];
            calc_type b = p[Order::B];
            calc_type a = p[Order::A];
            p[Order::R] = (value_type)(((cr - r) * alpha + (r << base_shift)) >> base_shift);
            p[Order::G] = (value_type)(((cg - g) * alpha + (g << base_shift)) >> base_shift);
            p[Order::B] = (value_type)(((cb - b) * alpha + (b << base_shift)) >> base_shift);
            p[Order::A] = (value_type)((alpha + a) - ((alpha * a + base_mask) >> base_shift));
        }


    Ken> Images.  Anyway, if the buffer's alpha channel isn't used,
    Ken> then the whole situation does seem a bit odd.  Could the
    Ken> information be retained for PNGs or something?

It is useful to store the final pixel buffer (eg in a PNG) as RGBA
because some people like to have some parts of their figure
transparent to composite the figure with other images.

JDH

John Hunter wrote:
> It is useful to store the final pixel buffer (eg in a PNG) as RGBA 
> because some people like to have some parts of their figure 
> transparent to composite the figure with other images.

fair enough, and that's probably a really cool feature when you need it!

ken wrote:

> I was talking about the image-from-a-buffer business not helping us
> with WX 2.4/2.6 due to the RGBA to RGB conversion.

But it looks like RendererAgg has this a agg.tostring_rgb() method, so 
we should be able to do change :

image.SetData(agg.tostring_rgb())

to

image.SetDataBuffer(agg.tostring_rgb())

If we make sure to keepthe string around. I haven't looked at your C++ 
code, but does it do something faster than RendererAgg.tostring_rgb() ?

Another thing that would be nice (for all Agg back-ends, I imagine), is 
if we could replace this:

         # agg => rgb -> image => bitmap => clipped bitmap => image
         return wx.ImageFromBitmap(_clipped_image_as_bitmap(image, bbox))

with a RendererAgg._clipped_tostring_rgb(bbox)

So that we don't copy a bunch of RGB data we don't need.

even if we don't do that, I think

_clipped_image_as_bitmap()

could use wx.Image.GetSubImage(), rather than creating a bimtp of the 
whole thing and blitting. untested code:

def _clipped_image_as_bitmap(image, bbox):
     """
     Convert the region of a wx.Image described by bbox to a wx.Bitmap.
     """
     l, b, width, height = bbox.get_bounds()

     return wx.BitmapFromImage(image.GetSubImage(wxRect((l,b),(w,h))))


 > RendererAgg appears to already have a buffer_rgba() method.

So we're all set for wxPython 2.7 -- very nice! I hope it doesn't make a 
copy.

Is there a numpy_array_rgba method -- that could be nice, and would work 
as a buffer, too. Maybe when we are ready to dump Numeric and numarray.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...