Thread: [Matplotlib-users] Interactive backends very (suprisingly?) slow for multiple subplots

Hi, I'm trying to plot a series of ~30-50 small plots, each of which contains
3 plots of ~10-20 points (one plot is data, one plot is errorbars, one plot
is a model fit).  I've tried using GtkAgg and Qt4Agg as backends, and both
are extremely slow - they take ~5-10 seconds for the first plot window and
then hang.   Sometimes they'll plot subsequent windows, but sometimes they
will not.

I tried the same series of plot commands using the SVG, PS, and PDF backends
and the whole series of 50 plots takes <~1s.

So, is there any way to speed up my plotting?  I've tried turning
interactive plotting off (ioff()), but that didn't help any.

Thanks,
Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23261074.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

keflavich <kef...@gm...> writes:

> I tried the same series of plot commands using the SVG, PS, and PDF backends
> and the whole series of 50 plots takes <~1s.

Did you produce any output with savefig? 50 plots per second sounds
pretty fast - at least on my computer, the matplotlib examples render
much slower than that, so I suspect your test script might not have
really caused the backends to render anything. Anyway, if you want to
compare the execution time of the interactive backends, I suppose the
best reference would be the Agg backend, which does the same rendering
without any of the interactiveness.

-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Jouni K. Seppänen wrote:
> 
> keflavich <kef...@gm...> writes:
> 
>> I tried the same series of plot commands using the SVG, PS, and PDF
>> backends
>> and the whole series of 50 plots takes <~1s.
> 
> Did you produce any output with savefig? 50 plots per second sounds
> pretty fast - at least on my computer, the matplotlib examples render
> much slower than that, so I suspect your test script might not have
> really caused the backends to render anything. Anyway, if you want to
> compare the execution time of the interactive backends, I suppose the
> best reference would be the Agg backend, which does the same rendering
> without any of the interactiveness.
> 

Yes, I'm using savefig.  I switched to the 'Paint' backend so I can save as
.png and use Mac's preview to view the files as they're updated; that's my
cheap workaround for the moment.

I tried 'agg' and it performs essentially the same as 'Paint' - pretty fast. 
Maybe closer to 2-3s than 1s.

Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23261638.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

On Mon, Apr 27, 2009 at 12:24 PM, keflavich <kef...@gm...> wrote:

>
> Hi, I'm trying to plot a series of ~30-50 small plots, each of which
> contains
> 3 plots of ~10-20 points (one plot is data, one plot is errorbars, one plot
> is a model fit).  I've tried using GtkAgg and Qt4Agg as backends, and both
> are extremely slow - they take ~5-10 seconds for the first plot window and
> then hang.   Sometimes they'll plot subsequent windows, but sometimes they
> will not.



That does sound exceedingly slow -- it looks like you are having some
problems with the GUI or environment and not just the mpl component.  How
are you running and profiling your script?  Can you post some free-standing
example code which exposes the problem?  Can you provide some of the
environment data detailed at
http://matplotlib.sourceforge.net/faq/troubleshooting_faq.html#reporting-problems

John Hunter-4 wrote:
> 
> That does sound exceedingly slow -- it looks like you are having some
> problems with the GUI or environment and not just the mpl component.  How
> are you running and profiling your script?  Can you post some
> free-standing
> example code which exposes the problem?  Can you provide some of the
> environment data detailed at
> http://matplotlib.sourceforge.net/faq/troubleshooting_faq.html#reporting-problems
> 

I haven't been doing any profiling; I've never been particularly comfortable
with timeit.

As for normal problem reporting details...
Darwin eta.colorado.edu 9.6.3 Darwin Kernel Version 9.6.3: Tue Jan 20
18:26:40 PST 2009; root:xnu-1228.10.33~1/RELEASE_I386 i386 i386
matplotlib version 0.98.3


My example standalone script (note that it takes the backend as a command
line argument and assumes you have not selected a backend / imported pylab):

import matplotlib
import sys
matplotlib.use(sys.argv[1])
from pylab import *

i=1; n=1; 
print "Working on figure 0"
figure(0); clf()
for j in xrange(36):
    if (i % 10)==0:
        print "Working on figure %i" % n
        figure(n); clf()
        i=1
        n+=1
    subplot(3,3,i)  
    plot([1,2,3],[3,2,1])
    plot([1,2,1],[1,2,3])
    plot([1,2,3],[1,2,3])
    i+=1


Results: 

In [1]: %run -t code/test.py 'Qt4Agg'

backend Qt4Agg version 0.9.1

IPython CPU timings (estimated):
  User  :  21.553853 s.
  System:        0.0 s.

In [1]: %run -t code/test.py 'pdf'

backend pdf version unknown

IPython CPU timings (estimated):
  User  :   1.056959 s.
  System:        0.0 s.

In [1]: %run -t code/test.py 'svg'
backend svg version 0.98.5.2

IPython CPU timings (estimated):
  User  :   1.056702 s.
  System:        0.0 s.


Curiously, if I add ioff() at the beginning of the script using the Qt4Agg
backend, I get the much nicer result:
  User  :   1.328496 s.

and show works quite rapidly.  This suggests to me that the placement of
ioff() is important, so I'm going to try playing with that a little...

Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23263295.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

So, as the Matplotlib help page suggests, working through a test problem
helped me narrow down my problem... but it still hasn't solved it.  If I set
ioff() at the main level, rather than in a function I call, it works. 
However, when I show() the plot, the code halts until I close it, which is
not helpful.  I feel like this is a problem I've resolved before, but
googling hasn't helped me so far - I think the closest I have come are
"show() hangs" threads that have no replies.  

I guess this changes my question to:
How can I draw all of my figures non-interactively, then show them and
return to the python command prompt?  

ion()
show()

should work, I think, but does not.

Thanks,
Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23263639.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

On Mon, Apr 27, 2009 at 2:28 PM, keflavich <kef...@gm...> wrote:

>
>
> John Hunter-4 wrote:
> >
> > That does sound exceedingly slow -- it looks like you are having some
> > problems with the GUI or environment and not just the mpl component.  How
> > are you running and profiling your script?  Can you post some
> > free-standing
> > example code which exposes the problem?  Can you provide some of the
> > environment data detailed at
> >
> http://matplotlib.sourceforge.net/faq/troubleshooting_faq.html#reporting-problems
> >
>
> I haven't been doing any profiling; I've never been particularly
> comfortable
> with timeit.
>
> As for normal problem reporting details...
> Darwin eta.colorado.edu 9.6.3 Darwin Kernel Version 9.6.3: Tue Jan 20
> 18:26:40 PST 2009; root:xnu-1228.10.33~1/RELEASE_I386 i386 i386
> matplotlib version 0.98.3
>
>
> My example standalone script (note that it takes the backend as a command
> line argument and assumes you have not selected a backend / imported
> pylab):
>
> import matplotlib
> import sys
> matplotlib.use(sys.argv[1])
> from pylab import *
>
> i=1; n=1;
> print "Working on figure 0"
> figure(0); clf()
> for j in xrange(36):
>    if (i % 10)==0:
>        print "Working on figure %i" % n
>        figure(n); clf()
>        i=1
>        n+=1
>    subplot(3,3,i)
>    plot([1,2,3],[3,2,1])
>    plot([1,2,1],[1,2,3])
>    plot([1,2,3],[1,2,3])
>    i+=1
>
>
> Results:
>
> In [1]: %run -t code/test.py 'Qt4Agg'



Ahh, mixing matplotlib.use from an interactive ipython session -- that is an
important detail :-)  What is your backend (import matplotlib; print
matplotlib.rcParams['backend'])   It is quite likely that you are getting
cross GUI / cross threading problems from trying to do this inside ipython,
which may be using one GUI backend based on rc at pylab startup, and mpl is
trying to use another one with the use directive.  Basically, "use" is not
supported for switching GUIs, it was designed to be the first line executed
in a session but when you do this interactively in ipython pylab mode, you
are asking for a world of pain.

Try running your example from a standard unix shell rather than from ipython

JDH

John Hunter-4 wrote:
> 
> Ahh, mixing matplotlib.use from an interactive ipython session -- that is
> an
> important detail :-)  What is your backend (import matplotlib; print
> matplotlib.rcParams['backend'])   It is quite likely that you are getting
> cross GUI / cross threading problems from trying to do this inside
> ipython,
> which may be using one GUI backend based on rc at pylab startup, and mpl
> is
> trying to use another one with the use directive.  Basically, "use" is not
> supported for switching GUIs, it was designed to be the first line
> executed
> in a session but when you do this interactively in ipython pylab mode, you
> are asking for a world of pain.
> 
> Try running your example from a standard unix shell rather than from
> ipython
> 
> JDH
> 

Sorry, I should have specified: I wrote this script to use separate from my
interactive sessions as a test.  I use Qt4Agg now, set in my matplotlibrc,
and I do not use matplotlib.use() at all (since it doesn't work anyway). 
The faster performance I reported was achieved by setting the backend in my
matplotlibrc, not on the command line.  So I don't think I'm running into
the issues you're talking about.

I don't really know how to run a script in which I expect plots to show up
from the command line.  If I do something simple like 'python test.py' it
pops up windows but never shows the plots I expect to see.

Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23264479.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

Since there don't seem to be any forthcoming answers, I have a somewhat
different question.  In the matplotlib FAQ, it states that using 'show()'
puts you in the GUI mainloop
(http://matplotlib.sourceforge.net/faq/howto_faq.html#use-show).  However,
using plot commands on the ipython command line does not shut down the
command line generally.  I gathered from some googling that this is because
ipython starts up the matplotlib graphics in a different 'thread', but I
don't understand how this is done and most of what I've seen says it is bad.

So, my question now: How can I exit the GUI mainloop without closing the
graphics windows?

Thanks,
Adam
-- 
View this message in context: http://www.nabble.com/Interactive-backends-very-%28suprisingly-%29-slow-for-multiple-subplots-tp23261074p23295883.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

On Wed, Apr 29, 2009 at 9:07 AM, keflavich <kef...@gm...> wrote:

>
> Since there don't seem to be any forthcoming answers, I have a somewhat
> different question.  In the matplotlib FAQ, it states that using 'show()'
> puts you in the GUI mainloop
> (http://matplotlib.sourceforge.net/faq/howto_faq.html#use-show).  However,
> using plot commands on the ipython command line does not shut down the
> command line generally.  I gathered from some googling that this is because
> ipython starts up the matplotlib graphics in a different 'thread', but I
> don't understand how this is done and most of what I've seen says it is
> bad.
>

most GUI mainloops are blocking, so after you start them you cannot issue
more commands from an interactive shell.  Either you need to run a GUI
shell, or in the case of ipython run the GUI in a separate thread.  One
exception to this is tkinter (tkagg), which plays nicely with a standard
python shell.  I understand that recent versions of pygtk work also w/o
running the mainloop in a separate thread, but I haven't dug into the
details.


>
> So, my question now: How can I exit the GUI mainloop without closing the
> graphics windows?
>

This question doesn't really make sense to me.  Perhaps you can clearly
describe your use case (what you need to do) rather than the proposed
solutions.

JDH

On Wed, Apr 29, 2009 at 8:29 AM, John Hunter <jd...@gm...> wrote:
>
>
> On Wed, Apr 29, 2009 at 9:07 AM, keflavich <kef...@gm...> wrote:
>>
>> Since there don't seem to be any forthcoming answers, I have a somewhat
>> different question.  In the matplotlib FAQ, it states that using 'show()'
>> puts you in the GUI mainloop
>> (http://matplotlib.sourceforge.net/faq/howto_faq.html#use-show).  However,
>> using plot commands on the ipython command line does not shut down the
>> command line generally.  I gathered from some googling that this is
>> because
>> ipython starts up the matplotlib graphics in a different 'thread', but I
>> don't understand how this is done and most of what I've seen says it is
>> bad.
>
> most GUI mainloops are blocking, so after you start them you cannot issue
> more commands from an interactive shell.  Either you need to run a GUI
> shell, or in the case of ipython run the GUI in a separate thread.  One
> exception to this is tkinter (tkagg), which plays nicely with a standard
> python shell.  I understand that recent versions of pygtk work also w/o
> running the mainloop in a separate thread, but I haven't dug into the
> details.

OK, that's very helpful.

>> So, my question now: How can I exit the GUI mainloop without closing the
>> graphics windows?
>
> This question doesn't really make sense to me.  Perhaps you can clearly
> describe your use case (what you need to do) rather than the proposed
> solutions.

I would like to have access to the command line while simultaneously
being able to interact with and/or display plots.  I think this is
what ipython does by default when you pass it a thread keyword
(-pylab, -qt4thread, etc.).  I had some trouble getting ipython to
work correctly, but I think that had to do with passing the thread
keyword before/after some other keywords.

Part of my question that I hope makes sense: Is there a way to unblock
the command line without closing the plot window when using an
interactive backend?

Thanks, and sorry about the misunderstandings / lack of clarity,
Adam

On Wed, Apr 29, 2009 at 9:22 AM, John Hunter <jd...@gm...> wrote:
>
>
> On Wed, Apr 29, 2009 at 9:50 AM, Adam <kef...@gm...> wrote:
>>
>> I would like to have access to the command line while simultaneously
>> being able to interact with and/or display plots.  I think this is
>> what ipython does by default when you pass it a thread keyword
>> (-pylab, -qt4thread, etc.).  I had some trouble getting ipython to
>> work correctly, but I think that had to do with passing the thread
>> keyword before/after some other keywords.
>
> From your previous posts, I think you may have been be using ipython
> incorrectly, ie mixing ipython -pylab with the matplotlib use directive.
> Start with a canconical simple script, eg::
>
>     import matplotlib.pyplot as plt
>
>     plt.figure(1)
>     plt.plot([1,2,3])
>
>     plt.figure(2)
>     plt.plot([4,5,6])
>
>     plt.show()
>
> and set your matplotlibrc to backend to TkAgg.  Start ipython with::
>
>   > ipython -pylab
>
> and run your test script with::
>
>   In [61]: run test.py
>
> If that works, close everything down and set your backend to QtAgg and try
> running it again in the same way and let us know what happens.  It should
> just work.  I'm suspecting in that as you were testing and trying a lot of
> things, you got yourself into a situation where multiple GUIs were competing
> for attention.
>
>> Part of my question that I hope makes sense: Is there a way to unblock
>> the command line without closing the plot window when using an
>> interactive backend?
>
> Yes, that makes sense, and basically you need to either use TkAgg from a
> regular python shell, use ipython in pylab mode with any supported backend,
> or use a GUI shell.  ipython also has support for embedding in GUI shells.
> See also
>
>   http://matplotlib.sourceforge.net/users/shell.html
>
> JDH
>
Thanks John.  I think you answered my questions completely now.

FWIW, I was not using matplotlib.use() with ipython, I was using it
when calling 'python test.py' on the command line.  My mistake with
ipython was using an import command before -pylab, i.e.:
ipython -i -c "import pyfits,matplotlib" -pylab

which does not work, whereas
ipython -pylab -i -c "import pyfits,matplotlib"

does.


Thanks for the help!