Thread: Re: [matplotlib-devel] Default matplotlib data path (HAS CHANGED)

     I just committed my changes.  The simplest approach would be to
specify the matplotlib module package_data, but the current cvs layout
doesn't tailor to that very well.  So I mimicked distutils install
command to determine where matplotlib is installed.  The datapath is
then defined as $platlib/matplotlib/mpl-data.
     Why this change?  If you take a look at
matplotlib._get_data_path() you will see.  This method has grown to
probably 100 lines of code to check for various cases, e.g. py2exe,
setuptools, embedding mpl, etc.  Now that the data is installed into
the matplotlib module you could pretty much reduce to 1 line:
"os.sep.join([os.path.dirname(__file__), 'mpl-data'])".  This now
handles all the cases mentioned above.  I left in the initial check
for the MATPLOTLIBDATA env key to still allow for some flexibility.
     I have tested on posix and w/wo setuptools.  I am going to check
windows right now, but pretty sure it should work.  Please check this
very carefully before next release as it is a pretty major change.=20
Let me know if anyone encounters a problem.

Thanks,
- Charlie

On 12/7/05, John Hunter <jdh...@ac...> wrote:
> >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
>
>     Charlie> Would it be considered cleaner to embed the mpl data into
>     Charlie> the matplotlib module?  This would make it easier to
>     Charlie> clean a mpl install.  The data path could be expressed
>     Charlie> fairly easily too, as a one-liner:
>
>     Charlie> os.sep.join([os.path.split(matplotlib.__file__)[0],
>     Charlie> 'matplotlib-data'])
>
> Yes, if you can engineer in a way that works with setup w/ and w/o a
> --prefix arg it would be preferable, in my view.
>
> JDH
>

FYI, built a bdist_wininst and it worked fine.

On 12/12/05, Charlie Moad <cw...@gm...> wrote:
>      I just committed my changes.  The simplest approach would be to
> specify the matplotlib module package_data, but the current cvs layout
> doesn't tailor to that very well.  So I mimicked distutils install
> command to determine where matplotlib is installed.  The datapath is
> then defined as $platlib/matplotlib/mpl-data.
>      Why this change?  If you take a look at
> matplotlib._get_data_path() you will see.  This method has grown to
> probably 100 lines of code to check for various cases, e.g. py2exe,
> setuptools, embedding mpl, etc.  Now that the data is installed into
> the matplotlib module you could pretty much reduce to 1 line:
> "os.sep.join([os.path.dirname(__file__), 'mpl-data'])".  This now
> handles all the cases mentioned above.  I left in the initial check
> for the MATPLOTLIBDATA env key to still allow for some flexibility.
>      I have tested on posix and w/wo setuptools.  I am going to check
> windows right now, but pretty sure it should work.  Please check this
> very carefully before next release as it is a pretty major change.
> Let me know if anyone encounters a problem.
>
> Thanks,
> - Charlie
>
> On 12/7/05, John Hunter <jdh...@ac...> wrote:
> > >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
> >
> >     Charlie> Would it be considered cleaner to embed the mpl data into
> >     Charlie> the matplotlib module?  This would make it easier to
> >     Charlie> clean a mpl install.  The data path could be expressed
> >     Charlie> fairly easily too, as a one-liner:
> >
> >     Charlie> os.sep.join([os.path.split(matplotlib.__file__)[0],
> >     Charlie> 'matplotlib-data'])
> >
> > Yes, if you can engineer in a way that works with setup w/ and w/o a
> > --prefix arg it would be preferable, in my view.
> >
> > JDH
> >
>

Charlie Moad wrote:

>FYI, built a bdist_wininst and it worked fine.
>
>
I just tested it too and it works for me with a non-setuptools install,
with a setuptools install, and with a non-setuptools install in which
setuptools was later installed.  I think it should be fine.

>>>>> "Charlie" == Charlie Moad <cw...@gm...> writes:

    Charlie>      I just committed my changes.  The simplest approach
    Charlie> would be to specify the matplotlib module package_data,
    Charlie> but the current cvs layout doesn't tailor to that very
    Charlie> well.  So I mimicked distutils install command to
    Charlie> determine where matplotlib is installed.  The datapath is
    Charlie> then defined as $platlib/matplotlib/mpl-data.  Why this
    Charlie> change?  If you take a look at
    Charlie> matplotlib._get_data_path() you will see.  This method
    Charlie> has grown to probably 100 lines of code to check for
    Charlie> various cases, e.g. py2exe, setuptools, embedding mpl,
    Charlie> etc.  Now that the data is installed into the matplotlib
    Charlie> module you could pretty much reduce to 1 line:
    Charlie> "os.sep.join([os.path.dirname(__file__), 'mpl-data'])".
    Charlie> This now handles all the cases mentioned above.  I left
    Charlie> in the initial check for the MATPLOTLIBDATA env key to
    Charlie> still allow for some flexibility.  I have tested on posix
    Charlie> and w/wo setuptools.  I am going to check windows right
    Charlie> now, but pretty sure it should work.  Please check this
    Charlie> very carefully before next release as it is a pretty
    Charlie> major change.  Let me know if anyone encounters a
    Charlie> problem.

I'm having some trouble with this on my system.  I don't know if
something is screwy with my setuptools because I have setuptools but I
don't have egg

peds-pc311:~/python/projects/matplotlib> python
Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
[GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> from setuptools.command import bdist_egg
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: cannot import name bdist_egg
>>>

and my install fails

peds-pc311:~/python/projects/matplotlib> sudo python setup.py install
installing data to ./matplotlib/mpl-data
running install
running build
running build_py
running build_ext
running install_lib
running install_data
copying matplotlibrc -> /usr/./matplotlib/mpl-data


So I replaced the has_setup check with this

try:
    from setuptools.command import bdist_egg    
    #from setuptools import setup # use setuptools if possible
    has_setuptools = True
except ImportError:
    from distutils.core import setup
    has_setuptools = False


which works on my system.  I hit the same bug with scipy, and the
moral seems to be that you should explicitly check for eggs rather
than just setuptools, because there are some versions of setuptools
floating around w/o eggs.

JDH

That seems a better approach.  Did you commit?

On 12/14/05, John Hunter <jdh...@ac...> wrote:
> >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
>
>     Charlie>      I just committed my changes.  The simplest approach
>     Charlie> would be to specify the matplotlib module package_data,
>     Charlie> but the current cvs layout doesn't tailor to that very
>     Charlie> well.  So I mimicked distutils install command to
>     Charlie> determine where matplotlib is installed.  The datapath is
>     Charlie> then defined as $platlib/matplotlib/mpl-data.  Why this
>     Charlie> change?  If you take a look at
>     Charlie> matplotlib._get_data_path() you will see.  This method
>     Charlie> has grown to probably 100 lines of code to check for
>     Charlie> various cases, e.g. py2exe, setuptools, embedding mpl,
>     Charlie> etc.  Now that the data is installed into the matplotlib
>     Charlie> module you could pretty much reduce to 1 line:
>     Charlie> "os.sep.join([os.path.dirname(__file__), 'mpl-data'])".
>     Charlie> This now handles all the cases mentioned above.  I left
>     Charlie> in the initial check for the MATPLOTLIBDATA env key to
>     Charlie> still allow for some flexibility.  I have tested on posix
>     Charlie> and w/wo setuptools.  I am going to check windows right
>     Charlie> now, but pretty sure it should work.  Please check this
>     Charlie> very carefully before next release as it is a pretty
>     Charlie> major change.  Let me know if anyone encounters a
>     Charlie> problem.
>
> I'm having some trouble with this on my system.  I don't know if
> something is screwy with my setuptools because I have setuptools but I
> don't have egg
>
> peds-pc311:~/python/projects/matplotlib> python
> Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
> [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import setuptools
> >>> from setuptools.command import bdist_egg
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ImportError: cannot import name bdist_egg
> >>>
>
> and my install fails
>
> peds-pc311:~/python/projects/matplotlib> sudo python setup.py install
> installing data to ./matplotlib/mpl-data
> running install
> running build
> running build_py
> running build_ext
> running install_lib
> running install_data
> copying matplotlibrc -> /usr/./matplotlib/mpl-data
>
>
> So I replaced the has_setup check with this
>
> try:
>     from setuptools.command import bdist_egg
>     #from setuptools import setup # use setuptools if possible
>     has_setuptools =3D True
> except ImportError:
>     from distutils.core import setup
>     has_setuptools =3D False
>
>
> which works on my system.  I hit the same bug with scipy, and the
> moral seems to be that you should explicitly check for eggs rather
> than just setuptools, because there are some versions of setuptools
> floating around w/o eggs.
>
> JDH
>

>>>>> "Charlie" == Charlie Moad <cw...@gm...> writes:

    Charlie> That seems a better approach.  Did you commit?

Not yet, but I will.

JDH

Btw, the best approach would to specify the data as package-data
(since that is what it is), and it would be the most compatible.=20
Unfortunately my attempt to use parent directory relative paths
failed.  Taking this approach would actually require moving all the
mpl data into the lib/matplotlib/mpl-data directory in cvs.  I thought
you might be opposed to that, hence I have the logic in the setup
file.

- Charlie

On 12/14/05, Charlie Moad <cw...@gm...> wrote:
> That seems a better approach.  Did you commit?
>
> On 12/14/05, John Hunter <jdh...@ac...> wrote:
> > >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
> >
> >     Charlie>      I just committed my changes.  The simplest approach
> >     Charlie> would be to specify the matplotlib module package_data,
> >     Charlie> but the current cvs layout doesn't tailor to that very
> >     Charlie> well.  So I mimicked distutils install command to
> >     Charlie> determine where matplotlib is installed.  The datapath is
> >     Charlie> then defined as $platlib/matplotlib/mpl-data.  Why this
> >     Charlie> change?  If you take a look at
> >     Charlie> matplotlib._get_data_path() you will see.  This method
> >     Charlie> has grown to probably 100 lines of code to check for
> >     Charlie> various cases, e.g. py2exe, setuptools, embedding mpl,
> >     Charlie> etc.  Now that the data is installed into the matplotlib
> >     Charlie> module you could pretty much reduce to 1 line:
> >     Charlie> "os.sep.join([os.path.dirname(__file__), 'mpl-data'])".
> >     Charlie> This now handles all the cases mentioned above.  I left
> >     Charlie> in the initial check for the MATPLOTLIBDATA env key to
> >     Charlie> still allow for some flexibility.  I have tested on posix
> >     Charlie> and w/wo setuptools.  I am going to check windows right
> >     Charlie> now, but pretty sure it should work.  Please check this
> >     Charlie> very carefully before next release as it is a pretty
> >     Charlie> major change.  Let me know if anyone encounters a
> >     Charlie> problem.
> >
> > I'm having some trouble with this on my system.  I don't know if
> > something is screwy with my setuptools because I have setuptools but I
> > don't have egg
> >
> > peds-pc311:~/python/projects/matplotlib> python
> > Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
> > [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import setuptools
> > >>> from setuptools.command import bdist_egg
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> > ImportError: cannot import name bdist_egg
> > >>>
> >
> > and my install fails
> >
> > peds-pc311:~/python/projects/matplotlib> sudo python setup.py install
> > installing data to ./matplotlib/mpl-data
> > running install
> > running build
> > running build_py
> > running build_ext
> > running install_lib
> > running install_data
> > copying matplotlibrc -> /usr/./matplotlib/mpl-data
> >
> >
> > So I replaced the has_setup check with this
> >
> > try:
> >     from setuptools.command import bdist_egg
> >     #from setuptools import setup # use setuptools if possible
> >     has_setuptools =3D True
> > except ImportError:
> >     from distutils.core import setup
> >     has_setuptools =3D False
> >
> >
> > which works on my system.  I hit the same bug with scipy, and the
> > moral seems to be that you should explicitly check for eggs rather
> > than just setuptools, because there are some versions of setuptools
> > floating around w/o eggs.
> >
> > JDH
> >
>

>>>>> "Charlie" == Charlie Moad <cw...@gm...> writes:

    Charlie> Btw, the best approach would to specify the data as
    Charlie> package-data (since that is what it is), and it would be
    Charlie> the most compatible.  Unfortunately my attempt to use
    Charlie> parent directory relative paths failed.  Taking this
    Charlie> approach would actually require moving all the mpl data
    Charlie> into the lib/matplotlib/mpl-data directory in cvs.  I
    Charlie> thought you might be opposed to that, hence I have the
    Charlie> logic in the setup file.


I'm happy to do it in the cleanest and best way so feel free to
reorganize as necessary, as long as we continue to test on the various
platforms as you have been doing.  You'll need to submit an admin
request to the sf developers to flush the unused CVS directories after
the reorganization.

JDH

Before I do it, you do know that there will be no folder heirarchy:
e.g. fonts/ttf/*, images...
All files will be in lib/matplotlib/mpl-data/.  I honestly don't think
there are terribly many files, so shouldn't be a huge deal.

- Charlie

On 12/14/05, John Hunter <jdh...@ac...> wrote:
> >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
>
>     Charlie> Btw, the best approach would to specify the data as
>     Charlie> package-data (since that is what it is), and it would be
>     Charlie> the most compatible.  Unfortunately my attempt to use
>     Charlie> parent directory relative paths failed.  Taking this
>     Charlie> approach would actually require moving all the mpl data
>     Charlie> into the lib/matplotlib/mpl-data directory in cvs.  I
>     Charlie> thought you might be opposed to that, hence I have the
>     Charlie> logic in the setup file.
>
>
> I'm happy to do it in the cleanest and best way so feel free to
> reorganize as necessary, as long as we continue to test on the various
> platforms as you have been doing.  You'll need to submit an admin
> request to the sf developers to flush the unused CVS directories after
> the reorganization.
>
> JDH
>

>>>>> "Charlie" == Charlie Moad <cw...@gm...> writes:

    Charlie> Before I do it, you do know that there will be no folder
    Charlie> heirarchy: e.g. fonts/ttf/*, images...  All files will be
    Charlie> in lib/matplotlib/mpl-data/.  I honestly don't think
    Charlie> there are terribly many files, so shouldn't be a huge
    Charlie> deal.

Is this the only way to do it (no folder hierarchy?).  If so, I don't
see it as a show stopper.  But if it's possible, some form of
organization would be cleaner...

JDH

http://www.python.org/doc/current/dist/node11.html

From this my impression is that the package data files will be placed
in the same way they are in the package.  I will look for a way to get
around this, and if anyone on the list knows how, please speak up.  As
I said previously, my attempts to use "../../images/*" seemed not to
work.

On 12/14/05, John Hunter <jdh...@ac...> wrote:
> >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
>
>     Charlie> Before I do it, you do know that there will be no folder
>     Charlie> heirarchy: e.g. fonts/ttf/*, images...  All files will be
>     Charlie> in lib/matplotlib/mpl-data/.  I honestly don't think
>     Charlie> there are terribly many files, so shouldn't be a huge
>     Charlie> deal.
>
> Is this the only way to do it (no folder hierarchy?).  If so, I don't
> see it as a show stopper.  But if it's possible, some form of
> organization would be cleaner...
>
> JDH
>

I missed the "New in version 2.4." note in the link I posted.  With
this I would say the current state is probably worth keeping.  Sorry
for the DoS attack of emails on the mailing list this morning. ;)

- Charlie

On 12/14/05, Charlie Moad <cw...@gm...> wrote:
> http://www.python.org/doc/current/dist/node11.html
>
> From this my impression is that the package data files will be placed
> in the same way they are in the package.  I will look for a way to get
> around this, and if anyone on the list knows how, please speak up.  As
> I said previously, my attempts to use "../../images/*" seemed not to
> work.
>
> On 12/14/05, John Hunter <jdh...@ac...> wrote:
> > >>>>> "Charlie" =3D=3D Charlie Moad <cw...@gm...> writes:
> >
> >     Charlie> Before I do it, you do know that there will be no folder
> >     Charlie> heirarchy: e.g. fonts/ttf/*, images...  All files will be
> >     Charlie> in lib/matplotlib/mpl-data/.  I honestly don't think
> >     Charlie> there are terribly many files, so shouldn't be a huge
> >     Charlie> deal.
> >
> > Is this the only way to do it (no folder hierarchy?).  If so, I don't
> > see it as a show stopper.  But if it's possible, some form of
> > organization would be cleaner...
> >
> > JDH
> >
>

On Dec 14, 2005, at 9:23 AM, Charlie Moad wrote:
> I missed the "New in version 2.4." note in the link I posted.  With
> this I would say the current state is probably worth keeping.

Attached is a script which I wrote to make setup.py's more  
declarative and to make using setuptools/py2exe/etc from one file  
less of a pain.  The important part is that it also backports  
"package_data" to Python 2.3, which could be ripped out and dropped  
into matplotlib's setup.py without too much trouble.

Ken

> try:
>     from setuptools.command import bdist_egg
>     #from setuptools import setup # use setuptools if possible
>     has_setuptools =3D True
> except ImportError:
>     from distutils.core import setup
>     has_setuptools =3D False
>

Won't setup not be defined if there is an ImportError? This instead?

try:
    from setuptools.command import bdist_egg
    has_setuptools =3D True
except ImportError:
    has_setuptools =3D False
from distutils.core import setup