matplotlib-devel Mailing List for matplotlib

Hi,

I was trying to make updating plot not cause to change zoom.
I have wxPython program with embedded plot and I can change some 
parameters what makes some changes on the plot. User should be able
to zoom and then change these parameters and see how it influences
plot, OTOH it should be easy to show whole plot, eg. using home button.
Perhaps I'm wrong, but I think its quite common requirement.

BTW pressing home button calls toolbar's draw() twice:

Index: lib/matplotlib/backend_bases.py
===================================================================
RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/backend_bases.py,v
retrieving revision 1.48
diff -u -u -1 -r1.48 backend_bases.py
--- lib/matplotlib/backend_bases.py	12 Apr 2005 16:07:12 -0000	1.48
+++ lib/matplotlib/backend_bases.py	28 Apr 2005 13:44:23 -0000
@@ -948,3 +948,2 @@
         self._update_view()
-        self.draw()
 

My plot update looks like this:
        # toolbar is NavigationToolbar2
        old_view = toolbar._views()
        toolbar.update() #clear zoom history
        if old_view: #don't change current view
            toolbar.push_current()
            toolbar._views.push(old_view)
        toolbar._update_view()

I'd like to hear what do you think about it. Is a simpler way to make
the same? I'm writting my first mini-program that uses MPL and don't
know it well. 

And why wx and gtk toolbars are different -- in GTK button descriptions
are in tooltips and x,y position is displayed in toolbar, and in wx
statusbar is used? Are they different by design? I prefer the first way,
without statusbar.

And last thing: what do you think about making toolbar2 more hmm.. 
interactive(?), i mean disabled back/forward buttons whan there is no
history, pressed pan/zoom buttons when in pan/zoom mode etc?
I don't know if I'll try to do it, but if someone would do it,
would it be included in MPL?

Marcin

-- 
Marcin Wojdyr  |  http://www.unipress.waw.pl/~wojdyr/

On Thu, 28 Apr 2005, John Hunter wrote:

> 
> I think it might be good to actually start using sourceforge's bug
> tracking system, since I find it easy to fall behind.  I have been

And the same about patches? 
I like SF tracker. 
It has "Send email on new submission to" option.
Perhaps it would be a good idea to send e-mail to devel list?

Marcin

-- 
Marcin Wojdyr  |  http://www.unipress.waw.pl/~wojdyr/

On Thursday 28 April 2005 10:03 am, John Hunter wrote:
> I think it might be good to actually start using sourceforge's bug
> tracking system, since I find it easy to fall behind.  
[...]
>
> Also, the ability to assign bugs to other people (you!) would be quite
> nice for me :-)
>
> What do you think?

I also think its a good idea. Although, I enjoy interacting with the users and 
devs, and hope that won't change too much.

I think it is a good idea too.

On Apr 28, 2005, at 10:03 AM, John Hunter wrote:

>
> I think it might be good to actually start using sourceforge's bug
> tracking system, since I find it easy to fall behind.  I have been
> using a combination of mailing list and a TODO file, but I think it
> might be better to report bugs ourself on the site, and encourage
> users to do so after they've posted to the site and we've determined
> that it is a non-trivial bug.  For trivial bugs, it suffices to fix
> them and forget them, but it would be nice to have a place to point
> people for status on the longer term, pesky bugs (axes aspect ratio,
> for example).  Also, a recent hard-drive crash with some unbacked up
> email and notes have convinced me that my computer may not be the
> ideal place to manage bugs.
>
> Also, the ability to assign bugs to other people (you!) would be quite
> nice for me :-)
>
> What do you think?
>
> JDH
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Tell us your software development plans!
> Take this survey and enter to win a one-year sub to SourceForge.net
> Plus IDC's 2005 look-ahead and a copy of this survey
> Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

I think it might be good to actually start using sourceforge's bug
tracking system, since I find it easy to fall behind.  I have been
using a combination of mailing list and a TODO file, but I think it
might be better to report bugs ourself on the site, and encourage
users to do so after they've posted to the site and we've determined
that it is a non-trivial bug.  For trivial bugs, it suffices to fix
them and forget them, but it would be nice to have a place to point
people for status on the longer term, pesky bugs (axes aspect ratio,
for example).  Also, a recent hard-drive crash with some unbacked up
email and notes have convinced me that my computer may not be the
ideal place to manage bugs.

Also, the ability to assign bugs to other people (you!) would be quite
nice for me :-)

What do you think?

JDH

I routinely plot fairly large datasets (~4million points) using imshow =
but on my machine this takes about 11 secs to complete.  I went through =
the code and made a couple of minor changes where the image data would =
get clipped by vmax and vmin only if the user supplied a vmax or vmin.  =
The clipping is unnecessary if the user does not supply these values =
since vmax and vmin default to the max and min in the image.  I also =
replaced two successive where(...) calls with a single clip(...) call =
and that seems to have helped a tiny bit as well.  This change has been =
tested on 0.80 compiled with Numeric, though I can't envision how this =
would break anything.  The profiler tells me that my plot time has =
decreased from 11.8 sec to 7.2 sec.  Hope this helps.

Scott.


zelakiew> diff colors.py =
/local/usr/src/matplotlib-0.80/lib/matplotlib/colors.py
34c34
<      zeros, asarray, sort, searchsorted, sometrue, ravel, divide, clip
---
>      zeros, asarray, sort, searchsorted, sometrue, ravel, divide
554,555c554
<       needs_clipping =3D True
<
---
>
562,563d560
<         if vmin is None and vmax is None:
<           needs_clipping =3D False
573,574c570,572
<             if needs_clipping:
<               val =3D clip(val,vmin, vmax)
---
>
>             val =3D where(val<vmin, vmin, val)
>             val =3D where(val>vmax, vmax, val)

sparkplot is a short Python module I wrote that uses matplotlib to
create sparklines. For details see:

http://agiletesting.blogspot.com/2005/04/sparkplot-creating-sparklines-with.html

Kudos to everybody working on matplotlib, of course first of all to
John Hunter. I was really impressed with the functionality and
versatility of matplotlib. I'll keep trying my hand at it while
enhancing sparkplot.

Grig

Hello,

I've been playing with the unicode rendering code that got added in
matplotlib 0.74. All tests have been done on Linux with the 0.74-1
debian package, lazy me... They should work on any platform, but you
will have to find out the gory details (!).

[ Most of these are examples, but there are a few remarks for the ps backend
   maintainers mostly, enclosed in square brackets in the text. No patches yet,
   but if you find the ideas useful, I can give it a try later. ]

To try this out, you may also need the rather complete FreeFont unicode font:
1) download the ttf from
    http://savannah.nongnu.org/download/freefont/freefont-ttf.tar.gz
2) put the .ttf files in your prefered Truetype fonts directory
3) remove ~/.ttffont.cache
4) restart matplotlib

Part I: Guess what I do for a living

Get your favorite interactive backend and display pretty plots

 >>> plot([0.3,0.01,-0.01,-0.01,-0.1,-0.1,-0.3,-0.01,0.01,0.01,0.1,0.1,0.3],\
... [105,100,98,90,92,101,105,100,98,90,92,101,105],'kD-')
 >>> ylim(85,110)
 >>> xlabel(u'\u03bc\u2080H(T)', name='FreeSans')
 >>> ylabel(u'R(\u03a9)', name='FreeSans')
 >>> ^D

or weird formulas

 >>> figtext(0.5,0.5,u'\u0127\u03c9 \u226a k\u0432T',name='FreeSerif',\
... size=30, ha='center', va='center', color='r')
 >>> ^D

You can also save to svg, and even to postscript (or eps) provided you set the
ps.useafm preference to False for now.

Part II: All work and no play...

 >>> plot([0.3,0.01,-0.01,-0.01,-0.1,-0.1,-0.3,-0.01,0.01,0.01,0.1,0.1,0.3],\
... [105,100,107,90,92,101,105,100,98,90,92,101,105],'kD-')
 >>> ylim(85,110)
 >>> text(-0.01,107,u'   \u261c booh! the ugly artifact!',name='FreeSerif',\
... size=20, va='top', ha='left')
 >>> ^D

;-)

Part III: Ugly, dirty and mean

Now it's time to produce a PDF. Run ps2pdf on one of the plots above,
at look at the ugly Type 3 fonts in your prefered PDF viewer.
The only way to get decent PDF is to set ps.useafm to True again.

For this to work, we have to provide the AFM files for FreeFont:
1) download the source of the font from:
    http://savannah.nongnu.org/download/freefont/freefont-sfd.tar.gz
2) download fontforge from fontforge.sourceforge.net
3) open each .sfd file in freefont, and run File\Generate Fonts;
    choose type 'PS Type 0'; this should produce a corresponding
    .afm and .ps file; save the .ps file for later.
4) move the afm file to a directory which is searched by matplotlib.
    Any subdirectory of /usr/share/fonts/ will do, provided said
    subdirectory is not a symlink
    [ is this a bug ? the implementation is in lib/font_manager.py,
      function x11FontDirectory(); os.path.walk() ignores symlinks ]
5) remove ~/.afmfont.cache
6) restart matplotlib
7) when you save the first .ps figure, the cache is rebuilt

Now that we have a postscript, we need to convince ghostscript to display it.
The first step is to provide the Type 0 fonts, like this:
1) move the .ps files we previously saved into a directory in ghostscript's
    path (try gs -h). A subdirectory won't work this time. Don't ask me why.
2) rename the font file to the name of the font, without extension, like
    $ mv FreeSans.ps FreeSans

If we try to look at our figure now, ghostscript will complain about
'/rangecheck yada yada' and fail to display the figure. This is because
postscript doesn't understand utf-8 encoding.
Postscript does however understand unicode hexa codes. So we have to replace
(R(\316\251))    [ octal representation of utf-8 characters ]
with
<0052002803a90029>    [ each 4 hexa figures are one character ]
For now, we have to do that manually in our favorite text editor. To compute
the hexa code in python, we do:
 >>> unistr=u'R(\u03a9)'
 >>> print ('<'+'%04x'*len(unistr)+'>') % tuple([ord(c) for c in unistr])
 >>> ^D

[ It looks that the ps backend should do just that with unicode strings if
   ps.useafm is True, as utf-8 is useless anyway.
   Using unicode hexa may also allow a much simpler implementation of
   draw_unicode() (in lib/backends/backend_ps.py) in the Type 42 case, by
   avoiding to position the glyphs one by one ]

I successfully tested .eps files produced with this procedure on both
a recent ghostscript and acrobat distiller; distiller or ps2pdf will produce
PDFs with nice embedded Type 1 fonts.

Part IV: Publish or perish

Producing pretty PDFs is well and nice, but most publishers will ask for
.eps with all fonts embedded. So we have to embed the fonts into the .eps
file. I could find no program to do this. DO NOT use gs -sDEVICE=pswrite
for that. Not only will ghostscript mangle the fonts, but also the plots (!).

Luckily, the FreeSans.ps from above is already a postscript with embedded
fonts, so we are golden. Just cat the font files together with the .eps
and merge the headers and footers by hand.

[ It would be nice to have a ps.embedfonts preference. For Type 0, this is
easy, as above; I don't know for Type 1. Also, it would be good to embed
only the needed glyphs, but I haven't looked at how to do it ]

Well, that's all for tonight. In conclusion, unicode support works already
very well, and allows lots of fun things. Thank you guys for the good
work.

BC

On Mon, 2005-04-18 at 20:28 -0700, matplotlib-devel-
re...@li... wrote:
> > Hi All,
> > I installed Matplotlib-0.80 on my linux machine with Freetype 2.1.7.I
> > didn't get any error in installation.
> > Now I am trying run some examples from Matplotlib and getting following
> > error
> > 
> > 
> > 
> > [ajve@cava matplotlib-0.80]$ python examples/logo.py
> > Traceback (most recent call last):
> >   File "examples/logo.py", line 3, in ?
> >     from pylab import *
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/pylab.py", line 1, in ?
> >     from matplotlib.pylab import *
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/matplotlib/pylab.py", line 196, in ?
> >     import backends
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/matplotlib/backends/__init__.py", line 19, in ?
> >     globals(),locals(),[backend_name])
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/matplotlib/backends/backend_gtkagg.py", line 10, in ?
> >     from backend_gtk import gtk, FigureManagerGTK, FigureCanvasGTK,\
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/matplotlib/backends/backend_gtk.py", line 20, in ?
> >     from backend_gdk import RendererGDK
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/matplotlib/backends/backend_gdk.py", line 31, in ?
> >     import gtk, pango
> >   File
> > "/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
> > es/gtk-2.0/gtk/__init__.py", line 43, in ?
> >     from _gtk import *
> > ImportError: /usr/lib/libpangoxft-1.0.so.0: undefined symbol:
> > FT_Seek_Stream
> > 
> > 
> > 
> > When I tried to see if thiss symbol is defined in Freetype 2.1.7 libs,I
> > found its not.But this symbol is defined in the previous version of
> > Freetype.But when I am trying to use old version of Freetyope,I get
> > "FT_CURVE_TAG_CUBIC" error while building Matplotlib which is not
> > defined in old version of Freetype.
> > 
> > Any help would be highly appreciated.
> > 
My guess is that you do not have a working pygtk installation. Try

$ python
>>> import pygtk
>>> pygtk.require('2.0')
>>> import gtk, pango

Does it produce the same error?

Steve

Hi All,
I installed Matplotlib-0.80 on my linux machine with Freetype 2.1.7.I
didn't get any error in installation.
Now I am trying run some examples from Matplotlib and getting following
error



[ajve@cava matplotlib-0.80]$ python examples/logo.py
Traceback (most recent call last):
  File "examples/logo.py", line 3, in ?
    from pylab import *
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/pylab.py", line 1, in ?
    from matplotlib.pylab import *
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/matplotlib/pylab.py", line 196, in ?
    import backends
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/matplotlib/backends/__init__.py", line 19, in ?
    globals(),locals(),[backend_name])
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/matplotlib/backends/backend_gtkagg.py", line 10, in ?
    from backend_gtk import gtk, FigureManagerGTK, FigureCanvasGTK,\
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/matplotlib/backends/backend_gtk.py", line 20, in ?
    from backend_gdk import RendererGDK
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/matplotlib/backends/backend_gdk.py", line 31, in ?
    import gtk, pango
  File
"/data/zion/intersect/local_linux/Python-2.3.4/lib/python2.3/site-packag
es/gtk-2.0/gtk/__init__.py", line 43, in ?
    from _gtk import *
ImportError: /usr/lib/libpangoxft-1.0.so.0: undefined symbol:
FT_Seek_Stream



When I tried to see if thiss symbol is defined in Freetype 2.1.7 libs,I
found its not.But this symbol is defined in the previous version of
Freetype.But when I am trying to use old version of Freetyope,I get
"FT_CURVE_TAG_CUBIC" error while building Matplotlib which is not
defined in old version of Freetype.

Any help would be highly appreciated.


Thanks
Ajay

On Fri, 2005-04-15 at 12:27 +0100, I wrote:
> My suggestion for improved performance would be to allow the Image class
> to work directly on the buffer passed to the generator function - I've
> started implementing this in c++.  Going with this approach should
> improve speed further and again conserve memory for very large images -
> and (without making other changes to the Image class) would require that
> rgba rather than rgb or intensity was used as an input.

Having tried this (patch attached) I get the following results on
running a slightly modified version of John Hunters test script
(attached).  In the original script it turned out that a significant
amount of time was being taken up with creating some of the test
environment - hence a smaller improvement was being shown.

Running with 1024x1024:
Starting array:
  Array set up: resident stack size:  39716, size:  10914
  Tests done: resident stack size:  43836, size:  11938
  Elapsed: 9.363
Starting buffer:
  Buffer set up: resident stack size:  15192, size:   4823
  Tests done: resident stack size:  15200, size:   4824
  Elapsed: 0.690
Fractional improvement: 12.577


Running with 2048x2048:
Starting array:
  Array set up: resident stack size: 146276, size:  37592
  Tests done: resident stack size: 158572, size:  40664
  Elapsed: 38.544
Starting buffer:
  Buffer set up: resident stack size:  39784, size:  10968
  Tests done: resident stack size:  39784, size:  10967
  Elapsed: 2.044
Fractional improvement: 17.855


Running with 4096x4096:
Starting array:
  Array set up: resident stack size: 564076, size: 142041
  Tests done: resident stack size: 613252, size: 154329
  Elapsed: 170.495
Starting buffer:
  Buffer set up: resident stack size:  67100, size:  35544
  Tests done: resident stack size: 133060, size:  35544
  Elapsed: 8.474
Fractional improvement: 19.120


As you can see - in both cases a big improvement.  In the case of large
data sets its a change from a noticeable delay 3.4s per plot to the very
usable 0.17s per plot (none of these plots required swapping - although
the set functions did).  If you don't have you data in the form of a
writable python buffer it's necessary to wrap the input buffer in a
Python array.array to get this performance (a read-only buffer is still
accepted but will be slightly slower).  Performance using a
non-modifiable buffer is slightly lower even with the overhead of an
additional Python function call (I guess it implements some more
intelligent buffer creation strategy).

> I made some alterations to these functions - but they are currently
> limited to my own situation.  I will make them general once I've played
> around with writable buffers a bit - this will be fairly easily but I
> don't want to spend time writing wrapper code until I'm happy with what
> I'm wrapping.

Changes in the patch I've attached (and were even simpler than I
imagined).

Nich

Hi,
I'm using code like this:
...
pylab.plot(..., label="something")
pylab.plot(..., label="something")
...
pylab.legend()

and i'd like to not include some plotted lines in legend.
I can specify it explicitly by legend(lines, labels), 
but it's more convenient for me to use:
pylab.plot(..., label=None)

I made a simple change in axes.py (attached patch).
I also have some questions about legend() function, that I put 
into the same patch.

Cheers,
Marcin

-- 
Marcin Wojdyr  |  http://www.unipress.waw.pl/~wojdyr/

On Thu, 2005-04-14 at 16:00 -0500, John Hunter wrote: 
> Thanks for the suggestions and patch.  I incorporated frombuffer and
> have been testing it.  I've been testing the performance of frombuffer
> vs fromarray, and have seen some 2-3x speedups but nothing like the
> numbers you are reporting.  [Also, I don't see any detectable memory
> leaks so I don't think you have any worries there]

That kind of speed up is probably more realistic - I probably made a
greater number of optimisations to the python buffer code than to the
numerix code (which I only remembered after posting my first message).
Performance do gains seem greater where memory is limited though as the
reduced memory consumption means less swapping, in cases where the
reduced memory consumption avoids swapping altogether they are, of
course, huge.

> Here is the test script I am using - does this look like a fair test?

It seems to be a fair test - I'd have created the string buffer outside
of the timing (as in reality you wouldn't go through that step) but as
it should be a fairly quick conversion it shouldn't matter too much.

> Another suggestion for Nicholas -- perhaps you want to support MxN,
> MxNx3 and MxNx4 arrays in your frombuffer function?

I could do - the main reason I don't particularly want to is that a
compiler should be able to more easily optimise a simple for(i,i<j,i++)
loop than a series of nested loops and other instructions.  As this code
is only really of use where performance is particularly important I'd
rather keep performance as high as possible - it's easy to generate
buffers in whatever format is necessary.

My suggestion for improved performance would be to allow the Image class
to work directly on the buffer passed to the generator function - I've
started implementing this in c++.  Going with this approach should
improve speed further and again conserve memory for very large images -
and (without making other changes to the Image class) would require that
rgba rather than rgb or intensity was used as an input.

> And a final question -- how are you getting your function into the
> matplotlib image pipeline.  Did you alter the image.py
> AxesImage.set_data function to test whether A is a buffer object?  If
> so, you might want to post these changes to the codebase as well.

I made some alterations to these functions - but they are currently
limited to my own situation.  I will make them general once I've played
around with writable buffers a bit - this will be fairly easily but I
don't want to spend time writing wrapper code until I'm happy with what
I'm wrapping.

Nicholas

>>>>> "Nicholas" == Nicholas Young <su...@su...> writes:

    Nicholas> I've attempted to implement this code myself (see
    Nicholas> attached patch to src/_image.cpp) but I'm not a regular
    Nicholas> c++ or even c programmer so it's fairly likely there
    Nicholas> will be memory leaks in the code. For a 1024x2048 array
    Nicholas> using the GTKAgg backend and with plenty of memory free
    Nicholas> this change results in show() taking <0.7s rather than
    Nicholas> >4.6s; if there is a memory shortage and swapping
    Nicholas> becomes involved the change is much more noticeable. I
    Nicholas> haven't made any decent Python wrapping code yet - but
    Nicholas> would be happy do do so if someone familiar with c++
    Nicholas> could tidy up my attachment.

Hi Nicholas, 

Thanks for the suggestions and patch.  I incorporated frombuffer and
have been testing it.  I've been testing the performance of frombuffer
vs fromarray, and have seen some 2-3x speedups but nothing like the
numbers you are reporting.  [Also, I don't see any detectable memory
leaks so I don't think you have any worries there]

Here is the test script I am using - does this look like a fair test?

You can uncomment report_memory on unix like systems to get a memory
report on each pass through the loop, and switch out fromarray vs
frombuffer to compare your function with mine

On a related note, below I'm pasting in a representative section the
code I am currently using in fromarray for MxNx3 and MxNx4 arrays --
any obvious performance gains to be had here numerix gurus?

Another suggestion for Nicholas -- perhaps you want to support MxN,
MxNx3 and MxNx4 arrays in your frombuffer function?

And a final question -- how are you getting your function into the
matplotlib image pipeline.  Did you alter the image.py
AxesImage.set_data function to test whether A is a buffer object?  If
so, you might want to post these changes to the codebase as well.


// some fromarray code

   //PyArrayObject *A = (PyArrayObject *) PyArray_ContiguousFromObject(x.ptr(), PyArray_DOUBLE, 2, 3); 
  PyArrayObject *A = (PyArrayObject *) PyArray_FromObject(x.ptr(), PyArray_DOUBLE, 2, 3); 

    int rgba = A->dimensions[2]==4;    
    double r,g,b,alpha;
    int offset =0;
    
    for (size_t rownum=0; rownum<imo->rowsIn; rownum++) {
      for (size_t colnum=0; colnum<imo->colsIn; colnum++) {
	offset = rownum*A->strides[0] + colnum*A->strides[1];
	r = *(double *)(A->data + offset);
	g = *(double *)(A->data + offset + A->strides[2] );
	b = *(double *)(A->data + offset + 2*A->strides[2] );
 	
	if (rgba) 
	  alpha = *(double *)(A->data + offset + 3*A->strides[2] );
	else
	  alpha = 1.0;
	
	*buffer++ = int(255*r);         // red
	*buffer++ = int(255*g);         // green
	*buffer++ = int(255*b);         // blue
	*buffer++ = int(255*alpha);     // alpha
	
      }
    }


## ... and here is the profile script ....

import sys, os, time, gc


from matplotlib._image import fromarray, fromarray2, frombuffer
from matplotlib.numerix.mlab import rand
from matplotlib.numerix import UInt8

def report_memory(i):
    pid = os.getpid()
    a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
    print i, '  ', a2[1],
    return int(a2[1].split()[1])

N = 1024
#X2 = rand(N,N)
#X3 = rand(N,N,3)
X4 = rand(N,N,4)

start = time.time()

b4 = (X4*255).astype(UInt8).tostring()
for i in range(50):

    im = fromarray(X4, 0)        
    #im = frombuffer(b4, N, N, 0)
    #val = report_memory(i)

end = time.time()
print 'elapsed: %1.3f'%(end-start)

Hi,

I'm a fairly heavy user of matplotlib (to plot results from plasma
physics simulations) and my use requires the display of fairly large
images.

Having done some testing I've discovered (after bypassing anything slow
from the python code) that for large images where there image size
approaches the available memory the main performance bar seems to be the
conversion of the raw data to the _image.Image class.  The way in which
the conversion takes place - with data being taken non-sequentially from
many points in a floating point source array and then converted to an 1
byte integer - is slow and if swapping becomes involved even slower.

To overcome this problem I suggest implementing c++ code to allow the
creation of the image from a buffer (with each rgba pixel as 4 bytes)
rather than a floating point array.  Where image data is being generated
elsewhere (in my case in Fortran code) it's trivial to output to a
different format and doing so means that the size of the input data can
be significantly smaller and that the data in the source array is
accessed sequentially (it's likely that a compiler will also be able to
optimise a copy of this data more effectively).  The image can then be
scaled and over plotted as with any existing image.

I've attempted to implement this code myself (see attached patch to
src/_image.cpp) but I'm not a regular c++ or even c programmer so it's
fairly likely there will be memory leaks in the code. For a 1024x2048
array using the GTKAgg backend and with plenty of memory free this
change results in show() taking <0.7s rather than >4.6s; if there is a
memory shortage and swapping becomes involved the change is much more
noticeable. I haven't made any decent Python wrapping code yet - but
would be happy do do so if someone familiar with c++ could tidy up my
attachment.

Hope this is useful to others,

Nicholas Young

On Wed, 13 Apr 2005, John Hunter wrote:

> I think your diff had improper line wrapping.  Could you resent it or
> the entire file as an attachment?

Here's the attachment.

--Michael

>>>>> "Michael" == Michael Brady <mb...@jp...> writes:

    Michael> Hi John, I'm one the developers at JPL who worked on the
    Michael> Qt backend.  Below are diffs for a couple of minor mods
    Michael> to the Qt backend.  They:

Hi Michael, 

I think your diff had improper line wrapping.  Could you resent it or
the entire file as an attachment?

Thanks,
JDH

Hi John,

I'm one the developers at JPL who worked on the Qt backend.  Below are 
diffs for a couple of minor mods to the Qt backend.  They:

1) fix a bug where keyboard input was grabbed by the figure
    and not released.
2) turn on cursor changes.
3) clean up a typo and commented-out print statement.

Please add them to backend_qt.py at your convenience.

Thanks,

Michael

========================================================================
  Michael Brady
  Jet Propulsion Laboratory (M/S 301-140L)
  4800 Oak Grove Drive
  Pasadena, CA  91109 
========================================================================

--- 
/home/mbrady/matplotlib-cvs/matplotlib/lib/matplotlib/backends/backend_qt.py 
2005-03-01 07:53:17.000000000 -0800
+++ backend_qt.py	2005-04-13 11:32:11.000000000 -0700
@@ -15,7 +15,7 @@
  from matplotlib.mathtext import math_parse_s_ft2font
  import qt

-backend_version = "0.9"
+backend_version = "0.9.1"
  def fn_name(): return sys._getframe(1).f_code.co_name

  DEBUG = False
@@ -88,10 +88,9 @@
          x = event.pos().x()
          # flipy so y=0 is bottom of canvas
          y = self.figure.bbox.height() - event.pos().y()
-        #print 'event.button()', event.button()
          button = self.buttond[event.button()]
          FigureCanvasBase.button_press_event( self, x, y, button )
-        if DEBUG: print 'button pressed'
+        if DEBUG: print 'button pressed:', event.button()

      def mouseMoveEvent( self, event ):
          x = event.x()
@@ -104,7 +103,7 @@
          x = event.x()
          # flipy so y=0 is bottom of canvas
          y = self.figure.bbox.height() - event.y()
-        button = button = self.buttond[event.button()]
+        button = self.buttond[event.button()]
          FigureCanvasBase.button_release_event( self, x, y, button )
          if DEBUG: print 'button released'
          self.draw()
@@ -147,7 +146,8 @@

          self.canvas.reparent( self.window, qt.QPoint( 0, 0 ) )
          # Give the keyboard focus to the figure instead of the manager
-        self.canvas.grabKeyboard()
+        self.canvas.setFocusPolicy( qt.QWidget.ClickFocus )
+        self.canvas.setFocus()
          self.window.setCaption( "Figure %d" % num )
          self.window.setCentralWidget( self.canvas )

@@ -227,8 +227,8 @@

      def set_cursor( self, cursor ):
          if DEBUG: print 'Set cursor' , cursor
-        #qt.QApplication.restoreOverrideCursor()
-        #qt.QApplication.setOverrideCursor( qt.QCursor( cursord[cursor] ) 
)
+        qt.QApplication.restoreOverrideCursor()
+        qt.QApplication.setOverrideCursor( qt.QCursor( cursord[cursor] ) 
)

      def draw_rubberband( self, event, x0, y0, x1, y1 ):
          height = self.canvas.figure.bbox.height()
@@ -242,12 +242,9 @@
          self.canvas.drawRectangle( rect )

      def save_figure( self ):
-        self.canvas.releaseKeyboard()
          fname = qt.QFileDialog.getSaveFileName()
-
          if fname:
              self.canvas.print_figure( fname.latin1() )
-        self.canvas.grabKeyboard()

  # set icon used when windows are minimized
  try:

>>>>> "Andrew" == Andrew Straw <str...@as...> writes:

    Andrew> Hi All, On my debian sarge (testing) box, I'm getting the
    Andrew> following error with the latest CVS code, WXAgg backend
    Andrew> and numerix = numarray.  I don't get a similar error with
    Andrew> GTKAgg, so I think it has to do with the WX backend.  My
    Andrew> thinking is that we should fix this before the 0.80
    Andrew> release.  Unfortunately, I don't have time to delve in
    Andrew> right now.

See the release notes <wink>

  Incorporated Werner's wx patch -- wx backend should be compatible
  with wxpython2.4 and recent versions of 2.5. Some early versions of
  wxpython 2.5 will not work because there was a temporary change in
  the dc API that was rolled back to make it 2.4 compliant

So basically my understanding is that early versions of wx 2.5 made
some changes to the 2.4 API, decided they were unwise, and unrolled
them to make them compatible with 2.4.

The error you are getting is one of these changes.

If you apt-get upgrade your wx on debian sarge, what wx version do you
get, and does the problem go away?

We do have the option of unrolling the namespace change Werner made
for a while, but this seems like a maintenance hassle to have to
reapply it later.  I open to suggestions.

JDH

Hi All,

On my debian sarge (testing) box, I'm getting the following error with 
the latest CVS code, WXAgg backend and numerix = numarray.  I don't get 
a similar error with GTKAgg, so I think it has to do with the WX 
backend.  My thinking is that we should fix this before the 0.80 
release.  Unfortunately, I don't have time to delve in right now.

$ python simple_plot.py
Traceback (most recent call last):
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wx.py", 
line 1067, in _onSize
    self.draw()
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wxagg.py", 
line 61, in draw
    self.gui_repaint()
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wx.py", 
line 897, in gui_repaint
    drawDC.DrawBitmap(self.bitmap, 0, 0)
  File 
"/home/ron/devel/debian/wxwindows/2.5.1.1-debs/wxwidgets2.5-2.5.1.1/wxPython/../debian/libwxgtk2.5-python/usr/lib/python2.3/site-packages/wx/gdi.py", 
line 2397, in DrawBitmap
TypeError: Expected a 2-tuple of integers or a wxPoint object.
Traceback (most recent call last):
  File "simple_plot.py", line 8, in ?
    plot(t, s)
  File "/usr/lib/python2.3/site-packages/matplotlib/pylab.py", line 
1900, in plot
    draw_if_interactive()
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wx.py", 
line 1161, in draw_if_interactive
    figManager.canvas.draw()
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wxagg.py", 
line 61, in draw
    self.gui_repaint()
  File 
"/usr/lib/python2.3/site-packages/matplotlib/backends/backend_wx.py", 
line 897, in gui_repaint
    drawDC.DrawBitmap(self.bitmap, 0, 0)
  File 
"/home/ron/devel/debian/wxwindows/2.5.1.1-debs/wxwidgets2.5-2.5.1.1/wxPython/../debian/libwxgtk2.5-python/usr/lib/python2.3/site-packages/wx/gdi.py", 
line 2397, in DrawBitmap
TypeError: Expected a 2-tuple of integers or a wxPoint object.

Every once in a while, when enough new features have been added and
the nasty bugs ironed out, I tick the major version number and post a
matplotlib release to the general python commuity.  These releases are
really just bug fix releases of the last release in disguise :-)

I just uploaded matplotlib-0.80 to the web site.  If you folks could
be kind enough to give this a test drive and let me know if you hit
any snags, I'd be much obliged.  If all goes well, I'll do the general
release tomorrow.

What's new in 0.80

kwargs to xlim, ylim, axis

  Applied a variant of Rick Muller's xlim/ylim/axis patch. These
  functions now take kwargs to let you selectively alter only the min
  or max if desired. Eg xlim(xmin=2) or axis(ymax=3). They always
  return the new lim. See, eg help(xlim). The same functionality is
  available in the API with ax.set_xlim and ax.set_ylim .

wx fixes

  Fixed a problem with wx app instantiation. Incorporated Werner's wx
  patch -- wx backend should be compatible with wxpython 2.4 and
  recent versions of 2.5. Some early versions of wxpython 2.5 will not
  work because there was a temporary change in the dc API that was
  rolled back to make it 2.4 compliant

Polygon editors

  Added some proof of concept code to show how to use matplotlib to
  interact with plot elements in a GUI neutral way. The editable
  polygon allows you to insert, delete and move vertices. See
  examples/poly_editor.py. The idea it to add interactor classes to
  support editable lines, text, polygons, etc. This could support a
  cross GUI colormap editor, or spline editor, for example.


http://matplotlib.sf.net


Thanks!
JDH

On Monday 04 April 2005 12:09 pm, John Hunter wrote:
>
> I agree ticks (and text in general) are too expensive.  In my
> experience, this is usually only starts a problem in animated plots
> (do you have another use case in mind?).  I think we might be able to
> work around this particular problem by supporting the drawing of only
> a subset of the artists in the scene.  
[...]
>
>
> I'm not opposed to a redesign of the Tick drawing if there are
> appreciable gains to be had, but my guess is we may get more bang for
> the buck in special casing the typical text layout (angle=0.0, no
> mathtext, no unicode) and handling dynamic updates more intelligently.

Data acquisition is a good example of where a new tick protocol would be 
useful. Supposing the user wants a plot in their gui that autoscales after 
the addition of each new point (which is not uncommon), the ticks would need 
to render as quickly as possible.

Everytime somebody I work with complains about the LabView program from 
National Instruments, I think about how nice it would be to do data 
acquisition with Python. I had hoped that Taco would mature into a solid 
library for interfacing with scientific instruments, but the project doesnt 
seem very active, judging by their webpage http://www.esrf.fr/taco/.

-- 

Darren

Hi John,

Hmm, could be.  Text is definitely slow, but my recollection is
that the Line2D drawing of the ticks was actually significant.
For example, the speed difference when turning on/off the right
and top ticks (which don't generally have text) was noticeable.

It's been awhile since I looked at this, and I'm not finding my
test scripts right now.  My conclusions at the time were that
agg rendering was dominating WXAgg time (so improving the WXAgg
icky get-rgb-image-then-render-as-bitmap was not so slow) and
that tick line rendering in Agg was much slower than I had
expected.  I'll try to reproduce this, but this week is sort of
full for me.  Currently, line plotting with WXAgg is fast enough
for me (I can reliably get better than 10 plots/sec on WinXP in
my app, for example).

Also, just to be clear: I owe you much more than doughnuts.
Thanks,

--Matt

On Mon, 4 Apr 2005, John Hunter wrote:

> >>>>> "Matt" == Matt Newville <new...@ca...> writes:
> 
>     Matt> I like Darren's and Paul's suggestion (set line properties
>     Matt> once, then have the ticks be a simple list of pen up / pen
>     Matt> down).  I believe major and minor ticks would need to have
>     Matt> different properties, but it's still only 2 set of
>     Matt> properties. I understand that this might mean a significant
>     Matt> redesign, but the performance boost might be worth it.
> 
> I would bet dollars to doughnuts (careful here, Perry still owes me a
> doughnut!) that almost all of the tick cost comes from laying out the
> text of the ticks and not in drawing the tick lines themselves -- Arnd
> posted some hotshot profile of this earlier, but I don't remember the
> exact results).
> 
> I agree ticks (and text in general) are too expensive.  In my
> experience, this is usually only starts a problem in animated plots
> (do you have another use case in mind?).  I think we might be able to
> work around this particular problem by supporting the drawing of only
> a subset of the artists in the scene.  I imagine something like the
> following is workable.
> 
>   line, = ax.plot(blah)
> 
>   dynamic = (line,)  # a list of artists to animate
>   # draws everything but artists in dynamic and caches Axes bbox to bitmap
>   ax.animate_prepare( dynamic) 
> 
>   while 1:
>      line.set_data(blah)
>      # blits the axes background cache and renders only the artists in dynamic
>      ax.animate()  
> 
> 
> I'm not opposed to a redesign of the Tick drawing if there are
> appreciable gains to be had, but my guess is we may get more bang for
> the buck in special casing the typical text layout (angle=0.0, no
> mathtext, no unicode) and handling dynamic updates more intelligently.
> 
> JDH
> 
> 
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
> 

>>>>> "Matt" == Matt Newville <new...@ca...> writes:

    Matt> I like Darren's and Paul's suggestion (set line properties
    Matt> once, then have the ticks be a simple list of pen up / pen
    Matt> down).  I believe major and minor ticks would need to have
    Matt> different properties, but it's still only 2 set of
    Matt> properties. I understand that this might mean a significant
    Matt> redesign, but the performance boost might be worth it.

I would bet dollars to doughnuts (careful here, Perry still owes me a
doughnut!) that almost all of the tick cost comes from laying out the
text of the ticks and not in drawing the tick lines themselves -- Arnd
posted some hotshot profile of this earlier, but I don't remember the
exact results).

I agree ticks (and text in general) are too expensive.  In my
experience, this is usually only starts a problem in animated plots
(do you have another use case in mind?).  I think we might be able to
work around this particular problem by supporting the drawing of only
a subset of the artists in the scene.  I imagine something like the
following is workable.

  line, = ax.plot(blah)

  dynamic = (line,)  # a list of artists to animate
  # draws everything but artists in dynamic and caches Axes bbox to bitmap
  ax.animate_prepare( dynamic) 

  while 1:
     line.set_data(blah)
     # blits the axes background cache and renders only the artists in dynamic
     ax.animate()  


I'm not opposed to a redesign of the Tick drawing if there are
appreciable gains to be had, but my guess is we may get more bang for
the buck in special casing the typical text layout (angle=0.0, no
mathtext, no unicode) and handling dynamic updates more intelligently.

JDH

Hi John, 

On Mon, 4 Apr 2005, John Hunter wrote:
> >>>>> "Darren" == Darren Dale <dd...@co...> writes:
> Darren> I think we could get a performance boost if all 
> Darren> similar ticks were passed together to draw_markers, 
> Darren> right now they a are passed independently.
>
> We could, but it would require some redesign.  Tick is a
> class, and the axis contains a list of ticks.  Thus it would
> take some top-level redesign.

I'd also encourage looking at how the Ticks are implemented. I
believe that for simple plots (say, simple_plot.py), the tick
drawing is what dominates rendering time, at least in the WxAgg
backend (which is dominated by the Agg rendering time). I
wouldn't be surprised if this was the case for most backends.

As far as I can tell, each tick mark is a separate Line2D with 2
points and have all the available properties of a Line2D. That
seems like a fine approach (certainly easy), but it's definitely
overkill.  My speed tests say that rendering one thousand lines
with two points is a lot slower than rendering two lines with
one thousand points (easy enough to test).  That means tick
drawing can easily be the performance bottleneck.

I like Darren's and Paul's suggestion (set line properties once,
then have the ticks be a simple list of pen up / pen down).  I
believe major and minor ticks would need to have different
properties, but it's still only 2 set of properties. I
understand that this might mean a significant redesign, but the
performance boost might be worth it.

Thanks,

--Matt

PS:  Someone might want tick marks to have all the flexibility
that they currently enjoy.  My guess is that this would be
unusual (I don't see any examples that use this flexibility),
and that such cases could just add custom lines themselves.