matplotlib-users Mailing List for matplotlib

> > I ran the example on my machine (which is a 64-bit Linux box with 8 GB
> of
> > RAM; Python 2.7, matplotlib 1.3.1) and it runs fine. However, it does
> use
> > around 2 GB of memory, perhaps slightly more. I think the memory usage
> > might be a problem for you if you are using 32-bit Windows. I'm not
> > familiar with the details but I believe the memory available to a single
> > 32-bit process on Win32 may be only 2 GB. I'm also not familiar with the
> > data you provided, but is it possible to reduce to number of points in
> > order to test if memory limitations are the underlying problemhere?
> 
> Nod, your suspicion is correct. The python interpreter bails out once the
> memory footprint reaches 2GBytes. That leaves us with the question if this
> is a quality of implementation issue - using up 2GBytes of main memory for
> 1 million node elements seems to be a bit excessive...
> 
> Thanks everybody for verifying anyways!

Just to round that issue up - I tried running this using Python 2.7 (64Bit) and it does not crash anymore. The memory requirement grows up to almost 4GByte. 

I will verify whether I can get the results I hope for and will report back.

Thanks again!
Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu

> 
> Regards Hartmut
> ---------------
> http://boost-spirit.com
> http://stellar.cct.lsu.edu
> 
> >
> >
> >
> > On 11 August 2014 14:54, Hartmut Kaiser <har...@gm...>
> wrote:
> > Ian,
> >
> > > I'm running into a crash while trying to construct a
> > > tri.LinearTriInterpolator. Here is the short version of the code:
> > >
> > >     import netCDF4
> > >     import matplotlib.tri as tri
> > >
> > >     var = netCDF4.Dataset('filename.cdf').variables
> > >     x = var['x'][:]
> > >     y = var['y'][:]
> > >     data = var['zeta_max'][:]
> > >     elems = var['element'][:, :]-1
> > >
> > >     triang = tri.Triangulation(x, y, triangles=elems)
> > >
> > >     # this crashes the python interpreter
> > >     interp = tri.LinearTriInterpolator(triang, data)
> > >
> > > The data arrays (x, y, data, elems) are fairly large (>1 mio
> elements),
> > > all
> > > represented as numpy arrays (as returned by netCDF4). The 'data' array
> > is
> > > a
> > > masked array and contains masked values.
> > >
> > > If somebody cares, I'd be able to post a link to the netCDF data file
> > > causing this.
> > >
> > > All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
> > >
> > > Any help would be highly appreciated!
> > > Regards Hartmut
> > >
> > > Hartmut,
> > > That is an excellent issue report; all the relevant information and
> > > nothing extraneous.  Hence the quick response.
> > > The second argument to TriLinearInterpolator (and other
> TriInterpolator
> > > classes), i.e. your 'data' array, is expected to be an array of the
> same
> > > size as the 'x' and 'y' arrays.  It is not expecting a masked
> array.  If
> > a
> > > masked array is used the mask will be ignored, and so the values
> behind
> > > the mask will be used as though they were real values.  If my memory
> of
> > > netCDF is correct, this will be whatever 'FillValue' is defined for
> the
> > > file, but it may depend on what is used to generate the netCDF file.
> > > I would normally expect the code to work but produce useless
> output.  A
> > > crash is possible though.  It would be best if you could post a link
> to
> > > the netCDF file and I will take a closer look to check there is not
> > > something else going wrong.
> > Thanks for the quick response!
> >
> > Here is the data file: http://tinyurl.com/ms7vzxw. I did some more
> > experiments. The picture stays unchanged, even if I fill the masked
> values
> > in the array with some real numbers (I'm not saying that this would give
> > me any sensible results...):
> >
> >     import netCDF4
> >     import matplotlib.tri as tri
> >     var = netCDF4.Dataset('maxele.63.nc').variables
> >     x = var['x'][:]
> >     y = var['y'][:]
> >     data = var['zeta_max'][:]
> >     elems = var['element'][:, :]-1
> >
> >     triang = tri.Triangulation(x, y, triangles=elems)
> >     data = data.filled(0.0)
> >
> >     # this still crashes the python interpreter
> >     interp = tri.LinearTriInterpolator(triang, data)
> >
> > Thanks again!
> > Regards Hartmut
> > ---------------
> > http://boost-spirit.com
> > http://stellar.cct.lsu.edu
> >
> >
> >
> > ------------------------------------------------------------------------
> --
> > ----
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
> >
> >
> > --
> > Dr Andrew Dawson
> > Atmospheric, Oceanic & Planetary Physics
> > Clarendon Laboratory
> > Parks Road
> > Oxford OX1 3PU, UK
> > Tel: +44 (0)1865 282438
> > Email: da...@at...
> > Web Site: http://www2.physics.ox.ac.uk/contacts/people/dawson

Andrew,

> I ran the example on my machine (which is a 64-bit Linux box with 8 GB of
> RAM; Python 2.7, matplotlib 1.3.1) and it runs fine. However, it does use
> around 2 GB of memory, perhaps slightly more. I think the memory usage
> might be a problem for you if you are using 32-bit Windows. I'm not
> familiar with the details but I believe the memory available to a single
> 32-bit process on Win32 may be only 2 GB. I'm also not familiar with the
> data you provided, but is it possible to reduce to number of points in
> order to test if memory limitations are the underlying problemhere?

Nod, your suspicion is correct. The python interpreter bails out once the memory footprint reaches 2GBytes. That leaves us with the question if this is a quality of implementation issue - using up 2GBytes of main memory for 1 million node elements seems to be a bit excessive...

Thanks everybody for verifying anyways!

Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu

> 
> 
> 
> On 11 August 2014 14:54, Hartmut Kaiser <har...@gm...> wrote:
> Ian,
> 
> > I'm running into a crash while trying to construct a
> > tri.LinearTriInterpolator. Here is the short version of the code:
> >
> >     import netCDF4
> >     import matplotlib.tri as tri
> >
> >     var = netCDF4.Dataset('filename.cdf').variables
> >     x = var['x'][:]
> >     y = var['y'][:]
> >     data = var['zeta_max'][:]
> >     elems = var['element'][:, :]-1
> >
> >     triang = tri.Triangulation(x, y, triangles=elems)
> >
> >     # this crashes the python interpreter
> >     interp = tri.LinearTriInterpolator(triang, data)
> >
> > The data arrays (x, y, data, elems) are fairly large (>1 mio elements),
> > all
> > represented as numpy arrays (as returned by netCDF4). The 'data' array
> is
> > a
> > masked array and contains masked values.
> >
> > If somebody cares, I'd be able to post a link to the netCDF data file
> > causing this.
> >
> > All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
> >
> > Any help would be highly appreciated!
> > Regards Hartmut
> >
> > Hartmut,
> > That is an excellent issue report; all the relevant information and
> > nothing extraneous.  Hence the quick response.
> > The second argument to TriLinearInterpolator (and other TriInterpolator
> > classes), i.e. your 'data' array, is expected to be an array of the same
> > size as the 'x' and 'y' arrays.  It is not expecting a masked array.  If
> a
> > masked array is used the mask will be ignored, and so the values behind
> > the mask will be used as though they were real values.  If my memory of
> > netCDF is correct, this will be whatever 'FillValue' is defined for the
> > file, but it may depend on what is used to generate the netCDF file.
> > I would normally expect the code to work but produce useless output.  A
> > crash is possible though.  It would be best if you could post a link to
> > the netCDF file and I will take a closer look to check there is not
> > something else going wrong.
> Thanks for the quick response!
> 
> Here is the data file: http://tinyurl.com/ms7vzxw. I did some more
> experiments. The picture stays unchanged, even if I fill the masked values
> in the array with some real numbers (I'm not saying that this would give
> me any sensible results...):
> 
>     import netCDF4
>     import matplotlib.tri as tri
>     var = netCDF4.Dataset('maxele.63.nc').variables
>     x = var['x'][:]
>     y = var['y'][:]
>     data = var['zeta_max'][:]
>     elems = var['element'][:, :]-1
> 
>     triang = tri.Triangulation(x, y, triangles=elems)
>     data = data.filled(0.0)
> 
>     # this still crashes the python interpreter
>     interp = tri.LinearTriInterpolator(triang, data)
> 
> Thanks again!
> Regards Hartmut
> ---------------
> http://boost-spirit.com
> http://stellar.cct.lsu.edu
> 
> 
> 
> --------------------------------------------------------------------------
> ----
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> 
> 
> 
> 
> --
> Dr Andrew Dawson
> Atmospheric, Oceanic & Planetary Physics
> Clarendon Laboratory
> Parks Road
> Oxford OX1 3PU, UK
> Tel: +44 (0)1865 282438
> Email: da...@at...
> Web Site: http://www2.physics.ox.ac.uk/contacts/people/dawson

Runs to completion without errors on my installation:

OS X  10.9.4
MacBook Air w/ 8GB of memory
Python 2.7 and matplotlib 1.3.1-1  lib 

-Dale


On Aug 10, 2014, at 13:43 , Hartmut Kaiser <har...@gm...> wrote:

> All,
> 
> I'm running into a crash while trying to construct a
> tri.LinearTriInterpolator. Here is the short version of the code:
> 
>    import netCDF4
>    import matplotlib.tri as tri
> 
>    var = netCDF4.Dataset('filename.cdf').variables
>    x = var['x'][:]
>    y = var['y'][:]
>    data = var['attrname'][:]     
>    elems = var['element'][:,:]-1
> 
>    triang = tri.Triangulation(x, y, triangles=elems)
> 
>    # this crashes the python interpreter
>    interp = tri.LinearTriInterpolator(triang, data)
> 
> The data arrays (x, y, data, elems) are fairly large (>1 mio elements), all
> represented as numpy arrays (as returned by netCDF4). The 'data' array is a
> masked array and contains masked values.
> 
> If somebody cares, I'd be able to post a link to the netCDF data file
> causing this.
> 
> All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
> 
> Any help would be highly appreciated!
> Regards Hartmut
> ---------------
> http://boost-spirit.com
> http://stellar.cct.lsu.edu
> 
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Hi Hartmut.

I ran the example on my machine (which is a 64-bit Linux box with 8 GB of
RAM; Python 2.7, matplotlib 1.3.1) and it runs fine. However, it does use
around 2 GB of memory, perhaps slightly more. I think the memory usage
might be a problem for you if you are using 32-bit Windows. I'm not
familiar with the details but I believe the memory available to a single
32-bit process on Win32 may be only 2 GB. I'm also not familiar with the
data you provided, but is it possible to reduce to number of points in
order to test if memory limitations are the underlying problemhere?




On 11 August 2014 14:54, Hartmut Kaiser <har...@gm...> wrote:

> Ian,
>
> > I'm running into a crash while trying to construct a
> > tri.LinearTriInterpolator. Here is the short version of the code:
> >
> >     import netCDF4
> >     import matplotlib.tri as tri
> >
> >     var = netCDF4.Dataset('filename.cdf').variables
> >     x = var['x'][:]
> >     y = var['y'][:]
> >     data = var['zeta_max'][:]
> >     elems = var['element'][:, :]-1
> >
> >     triang = tri.Triangulation(x, y, triangles=elems)
> >
> >     # this crashes the python interpreter
> >     interp = tri.LinearTriInterpolator(triang, data)
> >
> > The data arrays (x, y, data, elems) are fairly large (>1 mio elements),
> > all
> > represented as numpy arrays (as returned by netCDF4). The 'data' array is
> > a
> > masked array and contains masked values.
> >
> > If somebody cares, I'd be able to post a link to the netCDF data file
> > causing this.
> >
> > All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
> >
> > Any help would be highly appreciated!
> > Regards Hartmut
> >
> > Hartmut,
> > That is an excellent issue report; all the relevant information and
> > nothing extraneous.  Hence the quick response.
> > The second argument to TriLinearInterpolator (and other TriInterpolator
> > classes), i.e. your 'data' array, is expected to be an array of the same
> > size as the 'x' and 'y' arrays.  It is not expecting a masked array.  If
> a
> > masked array is used the mask will be ignored, and so the values behind
> > the mask will be used as though they were real values.  If my memory of
> > netCDF is correct, this will be whatever 'FillValue' is defined for the
> > file, but it may depend on what is used to generate the netCDF file.
> > I would normally expect the code to work but produce useless output.  A
> > crash is possible though.  It would be best if you could post a link to
> > the netCDF file and I will take a closer look to check there is not
> > something else going wrong.
>
> Thanks for the quick response!
>
> Here is the data file: http://tinyurl.com/ms7vzxw. I did some more
> experiments. The picture stays unchanged, even if I fill the masked values
> in the array with some real numbers (I'm not saying that this would give me
> any sensible results...):
>
>     import netCDF4
>     import matplotlib.tri as tri
>
>     var = netCDF4.Dataset('maxele.63.nc').variables
>     x = var['x'][:]
>     y = var['y'][:]
>     data = var['zeta_max'][:]
>     elems = var['element'][:, :]-1
>
>     triang = tri.Triangulation(x, y, triangles=elems)
>
>     data = data.filled(0.0)
>
>     # this still crashes the python interpreter
>     interp = tri.LinearTriInterpolator(triang, data)
>
> Thanks again!
> Regards Hartmut
> ---------------
> http://boost-spirit.com
> http://stellar.cct.lsu.edu
>
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>



-- 
Dr Andrew Dawson
Atmospheric, Oceanic & Planetary Physics
Clarendon Laboratory
Parks Road
Oxford OX1 3PU, UK
Tel: +44 (0)1865 282438
Email: da...@at...
Web Site: http://www2.physics.ox.ac.uk/contacts/people/dawson

Ian,

> I'm running into a crash while trying to construct a
> tri.LinearTriInterpolator. Here is the short version of the code:
> 
>     import netCDF4
>     import matplotlib.tri as tri
> 
>     var = netCDF4.Dataset('filename.cdf').variables
>     x = var['x'][:]
>     y = var['y'][:]
>     data = var['zeta_max'][:]
>     elems = var['element'][:, :]-1
> 
>     triang = tri.Triangulation(x, y, triangles=elems)
> 
>     # this crashes the python interpreter
>     interp = tri.LinearTriInterpolator(triang, data)
> 
> The data arrays (x, y, data, elems) are fairly large (>1 mio elements),
> all
> represented as numpy arrays (as returned by netCDF4). The 'data' array is
> a
> masked array and contains masked values.
> 
> If somebody cares, I'd be able to post a link to the netCDF data file
> causing this.
> 
> All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
> 
> Any help would be highly appreciated!
> Regards Hartmut
> 
> Hartmut,
> That is an excellent issue report; all the relevant information and
> nothing extraneous.  Hence the quick response.
> The second argument to TriLinearInterpolator (and other TriInterpolator
> classes), i.e. your 'data' array, is expected to be an array of the same
> size as the 'x' and 'y' arrays.  It is not expecting a masked array.  If a
> masked array is used the mask will be ignored, and so the values behind
> the mask will be used as though they were real values.  If my memory of
> netCDF is correct, this will be whatever 'FillValue' is defined for the
> file, but it may depend on what is used to generate the netCDF file.
> I would normally expect the code to work but produce useless output.  A
> crash is possible though.  It would be best if you could post a link to
> the netCDF file and I will take a closer look to check there is not
> something else going wrong.

Thanks for the quick response!

Here is the data file: http://tinyurl.com/ms7vzxw. I did some more experiments. The picture stays unchanged, even if I fill the masked values in the array with some real numbers (I'm not saying that this would give me any sensible results...):

    import netCDF4
    import matplotlib.tri as tri

    var = netCDF4.Dataset('maxele.63.nc').variables
    x = var['x'][:]
    y = var['y'][:]
    data = var['zeta_max'][:]
    elems = var['element'][:, :]-1

    triang = tri.Triangulation(x, y, triangles=elems)

    data = data.filled(0.0)

    # this still crashes the python interpreter
    interp = tri.LinearTriInterpolator(triang, data)

Thanks again!
Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu

On 10 August 2014 18:43, Hartmut Kaiser <har...@gm...> wrote:

> All,
>
> I'm running into a crash while trying to construct a
> tri.LinearTriInterpolator. Here is the short version of the code:
>
>     import netCDF4
>     import matplotlib.tri as tri
>
>     var = netCDF4.Dataset('filename.cdf').variables
>     x = var['x'][:]
>     y = var['y'][:]
>     data = var['attrname'][:]
>     elems = var['element'][:,:]-1
>
>     triang = tri.Triangulation(x, y, triangles=elems)
>
>     # this crashes the python interpreter
>     interp = tri.LinearTriInterpolator(triang, data)
>
> The data arrays (x, y, data, elems) are fairly large (>1 mio elements), all
> represented as numpy arrays (as returned by netCDF4). The 'data' array is a
> masked array and contains masked values.
>
> If somebody cares, I'd be able to post a link to the netCDF data file
> causing this.
>
> All this happens when using matplotlib 1.3.1, Win32, Python 2.7.
>
> Any help would be highly appreciated!
> Regards Hartmut
>

Hartmut,

That is an excellent issue report; all the relevant information and nothing
extraneous.  Hence the quick response.

The second argument to TriLinearInterpolator (and other TriInterpolator
classes), i.e. your 'data' array, is expected to be an array of the same
size as the 'x' and 'y' arrays.  It is not expecting a masked array.  If a
masked array is used the mask will be ignored, and so the values behind the
mask will be used as though they were real values.  If my memory of netCDF
is correct, this will be whatever 'FillValue' is defined for the file, but
it may depend on what is used to generate the netCDF file.

I would normally expect the code to work but produce useless output.  A
crash is possible though.  It would be best if you could post a link to the
netCDF file and I will take a closer look to check there is not something
else going wrong.

Ian Thomas

Exec. summary - I was having strange behavior with matshow in a loop and 
also with discrepancies between how iPython Notebook and Python via IDE 
displayed plots.

Solutions:
1) Using pause instead of show fixed matshow in a loop

2) Explicitly invoking %matplotblib qt or generally %matplotlib 
{backend} before  importing or using  matplotlib fixed various problems 
with plots in notebooks.  Now when I create a plot in a notebook, it 
appears, I can work with it, close it when appropriate, and 
simultaneously be able to do other work in notebook cells.

Thanks to everyone for the rapid responses.

JBB

On 7/30/14, 10:04 PM, JBB wrote:
> I've followed up on several suggestions and here is what I've done/found.
>
> (I know I don't use mlab or pylab but I pulled the import lines from
> another source and am leaving them in for the heck of it)


[ Woe/intrigue trimmed ]


>>      Is there a pointer to why this worked when my initial approach did not?
>>         I thought from the documentation/videos that preparing a plot with
>>      relevant commands then issuing the show() command was the preferred
>>      approach within Python/Matplotlib.
>>
>>      JBB
>>