matplotlib-devel Mailing List for matplotlib

On Wed, Jan 26, 2011 at 7:44 AM, Darren Dale <dsd...@gm...> wrote:
> Last night I noticed that, in the git repo, the commit messages
> produced by svnmerge.py still contain a lot of svn-specific
> information. Pauli's conversion script includes a step that filters
> out two lines at the end of each commit containing some svn metadata,
> but for svnmerge commits we still end up with:
>
>    Merged revisions 8933 via svnmerge from
>    https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
>
>    ........
>      r8933 | weathergod | 2011-01-22 10:35:26 -0600 (Sat, 22 Jan
> 2011) | 3 lines
>
>      Fixing problem where reversed colormaps of
> LinearSegmentedColormaps were not initialized properly.
>      Thanks to LittleBigBrain for reporting and Friedrich Romstedt
> for making the original patch.
>    ........
>
> Do we live with this, or try to replace references to svn commits with
> their git hash and references to svn branches with the git ones?

I think I just convinced myself that replacing svn references with git
hashes would be a Herculean task. One could generate a
svn_rev:git_hash mapping, but then changing the commit message
actually yields a new git hash, breaking the mapping. Its not an
impossible task, but it would be a lot of work and difficult to check.
Leaving the svn path and revision information will probably be more
reliable. "git log --all" will therefore yield enough information that
we can easily identify the provenance of a cherry pick and find it in
the git history:

commit adb1a7f67daf955a1af3a86d42b5181767c18819
Author: Ben Root <ben...@gm...>
Date:   Sat Jan 22 16:40:21 2011 +0000

    Merged revisions 8933 via svnmerge from
    https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_main

    ........
      r8933 | weathergod | 2011-01-22 10:35:26 -0600 (Sat, 22 Jan 2011) | 3 line

      Fixing problem where reversed colormaps of LinearSegmentedColormaps were n
      Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making t
    ........

    svn path=/trunk/matplotlib/; revision=8934

commit 2fa57c710607496a93000bfb3191bdd422f518cc
Author: Ben Root <ben...@gm...>
Date:   Sat Jan 22 16:35:26 2011 +0000

    Fixing problem where reversed colormaps of LinearSegmentedColormaps were not
    Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making the

    svn path=/branches/v1_0_maint/; revision=8933


Is this satisfactory?

Darren

On 01/25/2011 06:53 PM, Eric Firing wrote:
[...]
>
> Darren,
>
> It looks like at least some of the problem is the origin/unit_support:
>
> efiring@manini:~/test/matplotlib.git.ddale$ git checkout origin/unit_support
> Checking out files: 100% (4514/4514), done.
> Note: checking out 'origin/unit_support'.
> [...]
> HEAD is now at 8d705be... refactoring, moved units conversion to
> units.UnitsManager
> efiring@manini:~/test/matplotlib.git.ddale$ ls
> course  CVSROOT  htdocs  matplotlib  scipy06  toolkits  users_guide
>
> It appears to have branched trunk, not trunk/matplotlib.  Junk it!
>
> I think that losing some bits of history from the git repo is entirely
> acceptable.  The history is still in the svn repo, if anyone really
> needs to dig back into the earliest recorded origins of unit_support.
>
> Eric

Or, maybe a rule like this will work?

match /branches/unit_support/matplotlib/
  repository matplotlib
  branch unit_support
end match

(I don't know how significant the parentheses are; I am guessing they 
are for regular expressions, which are not needed here.)

Eric

On Wed, Jan 26, 2011 at 6:42 AM, Pauli Virtanen <pa...@ik...> wrote:
> Wed, 26 Jan 2011 20:35:03 +0900, Jae-Joon Lee wrote:
>> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen
>> <pa...@ik...> wrote:
>>> The complete magic stanza is:
>>>
>>> git reflog expire --expire=0 --all
>>> git prune
>>> git repack -f -a -d
>>> git gc --prune=0
>>
>> Wonderful!
>> With this, I get about 40 MB!
>
> Some additional compression can be obtained by passing also suitable
> values for the --window and --depth flags for git repack.
>
> http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/


Thank you all for the helpful feedback. I'll work on this again this evening.

Last night I noticed that, in the git repo, the commit messages
produced by svnmerge.py still contain a lot of svn-specific
information. Pauli's conversion script includes a step that filters
out two lines at the end of each commit containing some svn metadata,
but for svnmerge commits we still end up with:

    Merged revisions 8933 via svnmerge from
    https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint

    ........
      r8933 | weathergod | 2011-01-22 10:35:26 -0600 (Sat, 22 Jan
2011) | 3 lines

      Fixing problem where reversed colormaps of
LinearSegmentedColormaps were not initialized properly.
      Thanks to LittleBigBrain for reporting and Friedrich Romstedt
for making the original patch.
    ........

Do we live with this, or try to replace references to svn commits with
their git hash and references to svn branches with the git ones? I
tried merging the v.1.0.x branch into master, which applied pretty
cleanly, with only a few minor merge conflicts:

$ git merge origin/v1.0.x
Auto-merging CHANGELOG
CONFLICT (content): Merge conflict in CHANGELOG
Auto-merging examples/api/quad_bezier.py
CONFLICT (content): Merge conflict in examples/api/quad_bezier.py
Auto-merging lib/matplotlib/__init__.py
CONFLICT (content): Merge conflict in lib/matplotlib/__init__.py
Auto-merging lib/matplotlib/axes.py
Auto-merging lib/matplotlib/backends/backend_macosx.py
Auto-merging lib/matplotlib/backends/backend_ps.py
Auto-merging lib/matplotlib/backends/backend_qt4.py
Auto-merging lib/matplotlib/tests/test_axes.py
Auto-merging lib/matplotlib/text.py
Auto-merging lib/matplotlib/ticker.py
CONFLICT (content): Merge conflict in lib/matplotlib/ticker.py
Auto-merging src/_gtkagg.cpp
Auto-merging src/_macosx.m
CONFLICT (content): Merge conflict in src/_macosx.m


Here is the CHANGELOG:
<<<<<<< HEAD
2011-01-13 Added zdir and offset arguments to contourf3d to
           bring contourf3d in feature parity with contour3d. - BVR

2011-01-04 Tag 1.0.1 for release at r8896

2011-01-03 Added display of ticker offset to 3d plots. - BVR

2011-01-03 Turn off tick labeling on interior subplots for
           pyplots.subplots when sharex/sharey is True. - JDH

2010-12-29 Implment axes_divider.HBox and VBox. -JJL


=======
2011-01-04 Tag 1.0.1 for release at r8896

>>>>>>> origin/v1.0.x

Tue, 25 Jan 2011 18:12:35 -0500, Darren Dale wrote:
[clip]
> Pauli: could I trouble you to have a look at my rules file, maybe you
> will notice something I overlooked?
> (https://github.com/darrendale/mpl2git/blob/master/matplotlib.rules) Any
> other ideas?

As an additional check, I'd suggest checking that the history graph
in each repository is simple. Find all root (parentless) commits:

    git rev-list --parents master | egrep "^[a-f0-9]{40}$" # for master branch
    git rev-list --parents --all | egrep "^[a-f0-9]{40}$" # all branches

There should be only one root commit, unless there is a very good
reason to have several.

If there are multiple root commits, this may indicate that
the history was splintered into several parts (e.g. due to complicated 
SVN moves), which needs manual grafting to connect the parts.

For perfectionists, I'd also suggest eyeballing the "gitk --all" DAG.

-- 
Pauli Virtanen

Wed, 26 Jan 2011 20:35:03 +0900, Jae-Joon Lee wrote:
> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen
> <pa...@ik...> wrote:
>> The complete magic stanza is:
>>
>> git reflog expire --expire=0 --all
>> git prune
>> git repack -f -a -d
>> git gc --prune=0
> 
> Wonderful!
> With this, I get about 40 MB!

Some additional compression can be obtained by passing also suitable 
values for the --window and --depth flags for git repack.

http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/

	Pauli

On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen <pa...@ik...> wrote:
> The complete magic stanza is:
>
> git reflog expire --expire=0 --all
> git prune
> git repack -f -a -d
> git gc --prune=0

Wonderful!
With this, I get about 40 MB!

Regards,

-JJ

Wed, 26 Jan 2011 13:37:59 +0900, Jae-Joon Lee wrote:
[clip]
> I spend an hour to figure out how we can delete these unreachable
> objects. But it turned out that the answer seems to be simple.
> 
> $ git repack -ad

The complete magic stanza is:

git reflog expire --expire=0 --all
git prune
git repack -f -a -d
git gc --prune=0

On 01/25/2011 01:38 PM, Darren Dale wrote:
> On Tue, Jan 25, 2011 at 6:12 PM, Darren Dale<dsd...@gm...>  wrote:
>> There is still an outstanding issue that must be taken care of before
>> we migrate. The conversion routines create a basemap repository out of
>> trunk/toolkits/basemap, and matplotlib repository out of
>> trunk/matplotlib. Still, the matplotlib repo (at
>> github.com/darrendale/matplotlib) is over 200 MB. One can search the
>> objects in the large packfile, and find that there are still
>> references to basemap data in the matplotlib repo. I don't know how it
>> got in there, nor how to remove it.
>

Darren,

It looks like at least some of the problem is the origin/unit_support:

efiring@manini:~/test/matplotlib.git.ddale$ git checkout origin/unit_support
Checking out files: 100% (4514/4514), done.
Note: checking out 'origin/unit_support'.
[...]
HEAD is now at 8d705be... refactoring, moved units conversion to 
units.UnitsManager
efiring@manini:~/test/matplotlib.git.ddale$ ls
course  CVSROOT  htdocs  matplotlib  scipy06  toolkits  users_guide

It appears to have branched trunk, not trunk/matplotlib.  Junk it!

I think that losing some bits of history from the git repo is entirely 
acceptable.  The history is still in the svn repo, if anyone really 
needs to dig back into the earliest recorded origins of unit_support.

Eric

> I went through the exercise of identifying the largest blob, as
> described near the end of http://progit.org/book/ch9-7.html :
>
> $ git verify-pack -v
> objects/pack/pack-fa44ca56d7ec3964e562494f2fe08203143074bd.idx | sort
> -k 3 -n | tail -3
> 3b8b6c010f8ce59afac1e811b1bbc3efc21b770a blob   9154481 9089827 62749144
> 6328b70e665b58ed7f5aa1e110418cbb3facc07a blob   9331200 94297 156884507
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 blob   51399604 14333430 162328624
>
> $ git rev-list --objects --all | grep f784efc1518b10dff
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 toolkits/basemap/data/gshhs_h.txt
>
> This shell script is supposed to identify which commits have that blob
> in their tree (http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob):
>
> ---
> #!/bin/sh
> obj_name="$1"
> shift
> git log "$@" --pretty=format:'%T %h %s' \
> | while read tree commit subject ; do
>      if git ls-tree -r $tree | grep -q "$obj_name" ; then
>          echo $commit "$subject"
>      fi
> done
> ---
>
> but it comes up empty, so now I'm stuck. Any ideas would be greatly appreciated.
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

On Wed, Jan 26, 2011 at 8:38 AM, Darren Dale <dsd...@gm...> wrote:
> On Tue, Jan 25, 2011 at 6:12 PM, Darren Dale <dsd...@gm...> wrote:
>> There is still an outstanding issue that must be taken care of before
>> we migrate. The conversion routines create a basemap repository out of
>> trunk/toolkits/basemap, and matplotlib repository out of
>> trunk/matplotlib. Still, the matplotlib repo (at
>> github.com/darrendale/matplotlib) is over 200 MB. One can search the
>> objects in the large packfile, and find that there are still
>> references to basemap data in the matplotlib repo. I don't know how it
>> got in there, nor how to remove it.
>
> I went through the exercise of identifying the largest blob, as
> described near the end of http://progit.org/book/ch9-7.html :
>
> $ git verify-pack -v
> objects/pack/pack-fa44ca56d7ec3964e562494f2fe08203143074bd.idx | sort
> -k 3 -n | tail -3
> 3b8b6c010f8ce59afac1e811b1bbc3efc21b770a blob   9154481 9089827 62749144
> 6328b70e665b58ed7f5aa1e110418cbb3facc07a blob   9331200 94297 156884507
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 blob   51399604 14333430 162328624
>
> $ git rev-list --objects --all | grep f784efc1518b10dff
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 toolkits/basemap/data/gshhs_h.txt
>
> This shell script is supposed to identify which commits have that blob
> in their tree (http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob):
>
> ---
> #!/bin/sh
> obj_name="$1"
> shift
> git log "$@" --pretty=format:'%T %h %s' \
> | while read tree commit subject ; do
>    if git ls-tree -r $tree | grep -q "$obj_name" ; then
>        echo $commit "$subject"
>    fi
> done
> ---
>
> but it comes up empty, so now I'm stuck. Any ideas would be greatly appreciated.
>

First of all, I must clarify that I'm not a git expert by any means.

I suspected this could be some dangling objects within the repository,
which could be side effects of svn2git. After some googling, I found
that

$ git fsck --unreachable HEAD $(git for-each-ref
--format="%(objectname)" refs/heads)

This gave me 2774 objects which includes the blob of
"toolkits/basemap/data/gshhs_h.txt".
Since they are unreachable, I suppose that they can be simply removed.

I spend an hour to figure out how we can delete these unreachable
objects. But it turned out that the answer seems to be simple.

$ git repack -ad

Now there is no unreachable object reported and this seems to reduce
the total size down to ~140 MB.

Now the biggest blob is for "release/osx/matplotlib-0.98.5.tar.gz". and

$ git log -r -- release/osx/matplotlib-0.98.5.tar.gz

works as expected.

And some of the biggest blobs are associated with svg and pdf files.
It seems possible to remove these files (if needed) from the
repository using "git-filter-branch" to further reduce the size, but
I'm not sure if we need that.

IHTH,

-JJ




> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>