Thread: [matplotlib-devel] scipy core update

I just committed some changes to fix the scipy core patch.  The
problem was that Daishi apparently had the full scipy installed, so
many of the imports that worked in his tests did not work for scipy
core.

Most things seem to be working, but there is a problem in the
colormapping code in colors.py.  The index into the LUT, which should
be an integer array, is coming out as a float array for scipy core.
I'm not sure why.  Perry wrote this section of the code so he has a
better understanding of it than I do (around line 240 in colors.py),
this line is failing

        #print 'types', typecode(self._lut), typecode(xa), xa.shape
        rgba = take(self._lut, xa)

There may be other problems, but this is the only one I know of
currently.  

A number of the symbols in numerix.mlab that Daishi originally defined
were from various scipy (non-core) proper modules and I could not find
these in the core.  Will scipy core not be providing the equivalent of
MLab.py?  There are a few of these functions used in matplotlib, and I
just redefined them manually for now in numerix.mlab.  Eg, mean, std,
hanning, and a few more.  Others, like cov, I buried the import into
the function that needs them, eg in matplotlib.mlab in the
detrend_linear function.  If these aren't going to be in the core,
what do people suggest we do for numerix.mlab.

If anyone has any suggestions for help on the colormap LUT problem
referenced above, that would be great.

Thanks,
JDH

John,

It looks like you are referring to bugs in the Colormap.__call__ method 
that were introduced when I added support for masked values 
out-of-bounds values.  I have committed a change that fixes the bugs I 
could find there (as tested with examples/image_masked.py), so that the 
colormap call produces exactly the same output with scipy as with 
Numeric or numarray--the same rgba values.  Unfortunately, although that 
demo now runs with all three, it produces different plots.  It appears 
that somewhere else in the code--I'm pretty sure it is not within 
colors.py--the floating point rgba values are getting rounded or, more 
likely, truncated, when using scipy.

Eric

John Hunter wrote:
> I just committed some changes to fix the scipy core patch.  The
> problem was that Daishi apparently had the full scipy installed, so
> many of the imports that worked in his tests did not work for scipy
> core.
> 
> Most things seem to be working, but there is a problem in the
> colormapping code in colors.py.  The index into the LUT, which should
> be an integer array, is coming out as a float array for scipy core.
> I'm not sure why.  Perry wrote this section of the code so he has a
> better understanding of it than I do (around line 240 in colors.py),
> this line is failing
> 
>         #print 'types', typecode(self._lut), typecode(xa), xa.shape
>         rgba = take(self._lut, xa)
> 
> There may be other problems, but this is the only one I know of
> currently.  
> 
> A number of the symbols in numerix.mlab that Daishi originally defined
> were from various scipy (non-core) proper modules and I could not find
> these in the core.  Will scipy core not be providing the equivalent of
> MLab.py?  There are a few of these functions used in matplotlib, and I
> just redefined them manually for now in numerix.mlab.  Eg, mean, std,
> hanning, and a few more.  Others, like cov, I buried the import into
> the function that needs them, eg in matplotlib.mlab in the
> detrend_linear function.  If these aren't going to be in the core,
> what do people suggest we do for numerix.mlab.
> 
> If anyone has any suggestions for help on the colormap LUT problem
> referenced above, that would be great.
> 
> Thanks,
> JDH
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

>>>>> "Eric" == Eric Firing <ef...@ha...> writes:

    Eric> John, It looks like you are referring to bugs in the
    Eric> Colormap.__call__ method that were introduced when I added
    Eric> support for masked values out-of-bounds values.  I have
    Eric> committed a change that fixes the bugs I could find there
    Eric> (as tested with examples/image_masked.py), so that the
    Eric> colormap call produces exactly the same output with scipy as
    Eric> with Numeric or numarray--the same rgba values.
    Eric> Unfortunately, although that demo now runs with all three,
    Eric> it produces different plots.  It appears that somewhere else
    Eric> in the code--I'm pretty sure it is not within colors.py--the
    Eric> floating point rgba values are getting rounded or, more
    Eric> likely, truncated, when using scipy.

Hey Eric,

Thanks for taking a look at this.  The only other logical place for a
problem to reside is in colors.normalize, since the pipleline is

  rgba = cmap(normalize(X))

The relevant bit of code is here -- if you or anyone else sees an
obvious candidate that might cause this truncation, let me know....  


class normalize:
    def __init__(self, vmin=None, vmax=None, clip = True):
        """
        Normalize a given value to the 0-1 range

        If vmin or vmax is not given, they are taken from the input's
        minimum and maximum value respectively.  If clip is True and
        the given value falls outside the range, the returned value
        will be 0 or 1, whichever is closer. Returns 0 if vmin==vmax.
        Works with scalars or arrays, including masked arrays.  If clip
        is True, masked values on input will be set to 1 on output; if
        clip is False, the mask will be propagated to the output.
        """
        self.vmin = vmin
        self.vmax = vmax
        self.clip = clip

    def __call__(self, value):

        if isinstance(value, (int, float)):
            vtype = 'scalar'
            val = ma.array([value])
        else:
            vtype = 'array'
            val = ma.asarray(value)

        self.autoscale(val)
        vmin, vmax = self.vmin, self.vmax
        if vmin > vmax:
            raise ValueError("minvalue must be less than or equal to maxvalue")
        elif vmin==vmax:
            return 0.*value
        else:
            if self.clip:
                val = clip(val.filled(vmax), vmin, vmax)
            result = (1.0/(vmax-vmin))*(val-vmin)
        if vtype == 'scalar':
            result = result[0]
        return result

    def autoscale(self, A):
        if not self.scaled():
            if self.vmin is None: self.vmin = ma.minimum(A)
            if self.vmax is None: self.vmax = ma.maximum(A)

    def scaled(self):
        'return true if vmin and vmax set'
        return (self.vmin is not None and self.vmax is not None)


JDH

John,

The problem is that a bunch of scipy support is missing.  I think I have 
most of it fixed; I will do a little more, commit, and send another 
message in a few minutes.

Eric

John Hunter wrote:
>>>>>>"Eric" == Eric Firing <ef...@ha...> writes:
> 
> 
>     Eric> John, It looks like you are referring to bugs in the
>     Eric> Colormap.__call__ method that were introduced when I added
>     Eric> support for masked values out-of-bounds values.  I have
>     Eric> committed a change that fixes the bugs I could find there
>     Eric> (as tested with examples/image_masked.py), so that the
>     Eric> colormap call produces exactly the same output with scipy as
>     Eric> with Numeric or numarray--the same rgba values.
>     Eric> Unfortunately, although that demo now runs with all three,
>     Eric> it produces different plots.  It appears that somewhere else
>     Eric> in the code--I'm pretty sure it is not within colors.py--the
>     Eric> floating point rgba values are getting rounded or, more
>     Eric> likely, truncated, when using scipy.
> 
> Hey Eric,
> 
> Thanks for taking a look at this.  The only other logical place for a
> problem to reside is in colors.normalize, since the pipleline is
> 
>   rgba = cmap(normalize(X))

> 
> The relevant bit of code is here -- if you or anyone else sees an
> obvious candidate that might cause this truncation, let me know....  
> 
> 
> class normalize:
>     def __init__(self, vmin=None, vmax=None, clip = True):
>         """
>         Normalize a given value to the 0-1 range
> 
>         If vmin or vmax is not given, they are taken from the input's
>         minimum and maximum value respectively.  If clip is True and
>         the given value falls outside the range, the returned value
>         will be 0 or 1, whichever is closer. Returns 0 if vmin==vmax.
>         Works with scalars or arrays, including masked arrays.  If clip
>         is True, masked values on input will be set to 1 on output; if
>         clip is False, the mask will be propagated to the output.
>         """
>         self.vmin = vmin
>         self.vmax = vmax
>         self.clip = clip
> 
>     def __call__(self, value):
> 
>         if isinstance(value, (int, float)):
>             vtype = 'scalar'
>             val = ma.array([value])
>         else:
>             vtype = 'array'
>             val = ma.asarray(value)
> 
>         self.autoscale(val)
>         vmin, vmax = self.vmin, self.vmax
>         if vmin > vmax:
>             raise ValueError("minvalue must be less than or equal to maxvalue")
>         elif vmin==vmax:
>             return 0.*value
>         else:
>             if self.clip:
>                 val = clip(val.filled(vmax), vmin, vmax)
>             result = (1.0/(vmax-vmin))*(val-vmin)
>         if vtype == 'scalar':
>             result = result[0]
>         return result
> 
>     def autoscale(self, A):
>         if not self.scaled():
>             if self.vmin is None: self.vmin = ma.minimum(A)
>             if self.vmax is None: self.vmax = ma.maximum(A)
> 
>     def scaled(self):
>         'return true if vmin and vmax set'
>         return (self.vmin is not None and self.vmax is not None)
> 
> 
> JDH

John,

I committed fixes to several files; looks OK now, at least for 
examples/image_masked.py.

Eric

John Hunter wrote:
>>>>>>"Eric" == Eric Firing <ef...@ha...> writes:
> 
> 
>     Eric> John, It looks like you are referring to bugs in the
>     Eric> Colormap.__call__ method that were introduced when I added
>     Eric> support for masked values out-of-bounds values.  I have
>     Eric> committed a change that fixes the bugs I could find there
>     Eric> (as tested with examples/image_masked.py), so that the
>     Eric> colormap call produces exactly the same output with scipy as
>     Eric> with Numeric or numarray--the same rgba values.
>     Eric> Unfortunately, although that demo now runs with all three,
>     Eric> it produces different plots.  It appears that somewhere else
>     Eric> in the code--I'm pretty sure it is not within colors.py--the
>     Eric> floating point rgba values are getting rounded or, more
>     Eric> likely, truncated, when using scipy.
> 
> Hey Eric,
> 
> Thanks for taking a look at this.  The only other logical place for a
> problem to reside is in colors.normalize, since the pipleline is
> 
>   rgba = cmap(normalize(X))
> 
> The relevant bit of code is here -- if you or anyone else sees an
> obvious candidate that might cause this truncation, let me know....  
> 
> 
> class normalize:
>     def __init__(self, vmin=None, vmax=None, clip = True):
>         """
>         Normalize a given value to the 0-1 range
> 
>         If vmin or vmax is not given, they are taken from the input's
>         minimum and maximum value respectively.  If clip is True and
>         the given value falls outside the range, the returned value
>         will be 0 or 1, whichever is closer. Returns 0 if vmin==vmax.
>         Works with scalars or arrays, including masked arrays.  If clip
>         is True, masked values on input will be set to 1 on output; if
>         clip is False, the mask will be propagated to the output.
>         """
>         self.vmin = vmin
>         self.vmax = vmax
>         self.clip = clip
> 
>     def __call__(self, value):
> 
>         if isinstance(value, (int, float)):
>             vtype = 'scalar'
>             val = ma.array([value])
>         else:
>             vtype = 'array'
>             val = ma.asarray(value)
> 
>         self.autoscale(val)
>         vmin, vmax = self.vmin, self.vmax
>         if vmin > vmax:
>             raise ValueError("minvalue must be less than or equal to maxvalue")
>         elif vmin==vmax:
>             return 0.*value
>         else:
>             if self.clip:
>                 val = clip(val.filled(vmax), vmin, vmax)
>             result = (1.0/(vmax-vmin))*(val-vmin)
>         if vtype == 'scalar':
>             result = result[0]
>         return result
> 
>     def autoscale(self, A):
>         if not self.scaled():
>             if self.vmin is None: self.vmin = ma.minimum(A)
>             if self.vmax is None: self.vmax = ma.maximum(A)
> 
>     def scaled(self):
>         'return true if vmin and vmax set'
>         return (self.vmin is not None and self.vmax is not None)
> 
> 
> JDH

John, Travis,

Now I find a genuine difference between scipy and num* that is causing a 
problem:  for a masked array, say "A", A.mask is a property in scipy but 
a function in num*.  I can work around it, but it seems like an anomaly 
in scipy.  What it means is that in matplotlib we can't use A.mask or 
A.mask(), but must instead use the ma.getmask() function.


Here is another anomaly that I needed to work around:

 >>> bb = scipy.array([1.1,2,2,3.3])
 >>> 1.1 in bb
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()
 >>> import numarray
 >>> nn = numarray.array([1.1,2.2,3.3])
 >>> 1.1 in nn
True

x in A behaves completely differently if A is a scipy ndarray than if it 
is a numarray array.  This seems to me like a bug in scipy; the num* 
behavior is certainly easier to use, in that it preserves the list-like 
behavior when used in a list-like context.

In any case, I have committed changes to make contour work; but 
examples/masked_demo.py still doesn't work with scipy.  It generates no 
exception, but the plot is completely wrong.  I can't spend any more 
time on this right now.

Eric

Eric Firing wrote:

I replied privately to Eric by mistake.

> John, Travis,
> 
> Now I find a genuine difference between scipy and num* that is causing a 
> problem:  for a masked array, say "A", A.mask is a property in scipy but 
> a function in num*.  I can work around it, but it seems like an anomaly 
> in scipy.  What it means is that in matplotlib we can't use A.mask or 
> A.mask(), but must instead use the ma.getmask() function.

Paul is the author of masked arrays.  He implemented the new behavior. 
I think mask as a property/attribute is right.

> 
> 
> Here is another anomaly that I needed to work around:
> 
>  >>> bb = scipy.array([1.1,2,2,3.3])
>  >>> 1.1 in bb
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: The truth value of an array with more than one element is 
> ambiguous. Use a.any() or a.all()
>  >>> import numarray
>  >>> nn = numarray.array([1.1,2.2,3.3])
>  >>> 1.1 in nn
> True

This is a bug in scipy core that has been fixed for several weeks.  It 
came about as an unintentional side-effect of raising an error for 
truth-testing of arrays.

Notice:


 >>> 1.1 in scipy.array([1.1,2.2,3.3])
True

-Travis