matplotlib-devel Mailing List for matplotlib

On 17 December 2012 14:04, campbell.kb <cam...@gm...> wrote:

> I am an environmental computer modeler. I've created scripts that use
> tripcolor() to do contour plots of various output variables (with values at
> the centers of triangles and at the corners of the triangles). The grids I
> am working on are rather large (I am involved in modeling the United
> States'
> Great Lakes). The issue is that tripcolor() is exceedingly slow to use
> because I am producing contour plots for 3 variables, for all 48 time steps
> in a 48 hour forecast. Is there any chance that the tripcolor() function
> can
> be sped up, or is there anything I should do (in generating the arguments
> passed to it) that would speed up its execution time?
>

There is probably something you can do to speed up your execution time, but
for us to help we will need to see exactly what you are doing.  Can you
post an example of one of your slow scripts with the appropriate data?  As
you are dealing with large datasets you may want to simplify the example
script/data you send.  If the data is still too large and/or you don't want
to post it to a public mailing list, you can email it to me directly.

Ian Thomas

I thought I would just throw this out there, and I'm not entirely sure it
would work, but here goes...

What about using something like Dropbox?  Create a matplotlib account, put
the binaries in a Public folder and then use that as the storage...I don't
know how this would work in practice, but I do know that people have been
using Dropbox to host things for websites in other venues (persontal blogs,
etc)

It's probably best to talk to Dropbox first, but since they are built on
Python, they might be willing to work with open source Python projects.


PTM

---
Patrick Marsh
Ph.D. Candidate / Liaison to the HWT
School of Meteorology / University of Oklahoma
Cooperative Institute for Mesoscale Meteorological Studies
National Severe Storms Laboratory
http://www.patricktmarsh.com



On Mon, Dec 17, 2012 at 7:33 AM, Maximilian Albert <
max...@gm...> wrote:

> 2012/12/16 Thomas Kluyver <th...@kl...>:
> > On 15 December 2012 23:38, Damon McDougall <dam...@gm...>
> > wrote:
> >>
> >> Maybe the best thing is to host the binaries on Sourceforge.
> >
> >
> > Having recently tried to do it, Sourceforge tries really hard to avoid
> > giving you a direct link that can repeatably be used to download a file
> > automatically, i.e. without a browser. In the case I was after it for, I
> > ended up downloading the file (a PyWin32 binary) with a browser, and
> storing
> > it on the CI server that I wanted to install it.
>
> I haven't followed this thread in detail, so not sure if it's really
> relevant: I agree that it's quite messy, but it's definitely possible
> to find stable download links for automated downloads from
> sourceforge. I sometimes need this because I use a ports-like system
> for installing certain packages from source on top of my regular
> Ubuntu system, and it does work quite well. So if this is what's
> keeping people from Sourceforge then there are definitely workarounds
> (give me a shout if you need more specific information; I'm likely to
> have very sporadic internet access over the next few weeks though, so
> replies may take a while). However, I also think Sourceforge's
> interface is really ugly so if there are better alternatives then
> that's great. :)
>
> Cheers,
> Max
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>

I am an environmental computer modeler. I've created scripts that use
tripcolor() to do contour plots of various output variables (with values at
the centers of triangles and at the corners of the triangles). The grids I
am working on are rather large (I am involved in modeling the United States'
Great Lakes). The issue is that tripcolor() is exceedingly slow to use
because I am producing contour plots for 3 variables, for all 48 time steps
in a 48 hour forecast. Is there any chance that the tripcolor() function can
be sped up, or is there anything I should do (in generating the arguments
passed to it) that would speed up its execution time?




--
View this message in context: http://matplotlib.1069221.n5.nabble.com/tripcolor-sluggish-execution-tp40054.html
Sent from the matplotlib - devel mailing list archive at Nabble.com.

2012/12/16 Thomas Kluyver <th...@kl...>:
> On 15 December 2012 23:38, Damon McDougall <dam...@gm...>
> wrote:
>>
>> Maybe the best thing is to host the binaries on Sourceforge.
>
>
> Having recently tried to do it, Sourceforge tries really hard to avoid
> giving you a direct link that can repeatably be used to download a file
> automatically, i.e. without a browser. In the case I was after it for, I
> ended up downloading the file (a PyWin32 binary) with a browser, and storing
> it on the CI server that I wanted to install it.

I haven't followed this thread in detail, so not sure if it's really
relevant: I agree that it's quite messy, but it's definitely possible
to find stable download links for automated downloads from
sourceforge. I sometimes need this because I use a ports-like system
for installing certain packages from source on top of my regular
Ubuntu system, and it does work quite well. So if this is what's
keeping people from Sourceforge then there are definitely workarounds
(give me a shout if you need more specific information; I'm likely to
have very sporadic internet access over the next few weeks though, so
replies may take a while). However, I also think Sourceforge's
interface is really ugly so if there are better alternatives then
that's great. :)

Cheers,
Max

On 12/16/2012 03:44 PM, Eric Firing wrote:
> On 2012/12/16 9:21 AM, Damon McDougall wrote:
>> On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>> <jas...@cr...> wrote:
>>> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>>>> sourceforge's horror of an interface.
>>> I'll second that.  Every time I go to Sourceforge, I have to figure out
>>> how in the world to download what I want (and I have to figure out which
>>> things *not* to click on too).
>> Ok sounds like there is a reasonable amount of resistance towards Sourceforge.
>>
>> Eric, when you suggest that NumFocus could 'provide hosting directly',
>> do you mean they would have the physical hardware to host the files,
>> or are you suggesting they provide the finances to seek hosting
>> elsewhere?
> I was thinking that perhaps NumFocus would be running a server that
> could provide the hosting.  Funding for an external service is also
> possible, though, and might make more sense.

I'll definitely walk down the hall and talk to my local Numfocus board 
member ;)

As for S3, their free tier is clearly insufficient -- I thought I'd get 
a hold on the cost of paying for it.

Our storage requirements are ~150MB per release (for all of the 
different binaries) (though likely to grow over time), so saying it's 
useful to have the last ~6 or so releases live at any given time, that's 
~900MB -- let's call it 1GB.

Our transfer requirements after a release seem to peak at around 
1.7TB/mo, about 67,000 requests, after a release (though presumably 
amortized over time, it's lower).  This is also likely to grow over 
time, of course.

So, I think we've got:

1,700 GB transfer @ $0.12/GB = $204.00
70,000 requests $0.01/10,000 = $0.07
1GB storage @ $0.095 = $0.10

That's a bit steep to cover with donations alone.

In any event, this is a useful assessment of what this is really worth 
(and what our eyeballs are worth to advertisers on Sourceforge ;)

Perhaps, as Skipper suggests, with a little investment in automation 
tools for SourceForge that may remain the best option.  The Sourceforge 
download experience isn't terrible if we provide direct links to the 
downloads on our own website.  wxPython has done this for years, and 
AFAIK they have not run in to any problems.  The links even work without 
a web browser (though, for example, wget and curl).

Cheers,
Mike

On 12/17/2012 07:39 AM, Michael Droettboom wrote:
> On 12/17/2012 07:36 AM, Michael Droettboom wrote:
>> On 12/16/2012 02:50 PM, Damon McDougall wrote:
>>> On Sun, Dec 16, 2012 at 1:38 PM, Todd<tod...@gm...>  wrote:
>>>> On Sun, Dec 16, 2012 at 8:21 PM, Damon McDougall<dam...@gm...>
>>>> wrote:
>>>>> On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>>>>> <jas...@cr...>  wrote:
>>>>>> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>>>>>>> sourceforge's horror of an interface.
>>>>>> I'll second that.  Every time I go to Sourceforge, I have to figure out
>>>>>> how in the world to download what I want (and I have to figure out which
>>>>>> things *not* to click on too).
>>>>> Ok sounds like there is a reasonable amount of resistance towards
>>>>> Sourceforge.
>>>>>
>>>>> Eric, when you suggest that NumFocus could 'provide hosting directly',
>>>>> do you mean they would have the physical hardware to host the files,
>>>>> or are you suggesting they provide the finances to seek hosting
>>>>> elsewhere?
>>>>>
>>>>> In the GitHub blog post, they suggest using S3. We could try that.
>>>>> It's fairly inexpensive and the first year is free (within monthly
>>>>> bandwidth limits). We could try it for a year and see how that pans
>>>>> out? I'm not entirely sure how the Amazon stuff works but I've heard
>>>>> good things about it.
>>>>>
>>>>>
>>>> Are you sure the monthly bandwidth limits are sufficient?
>>>>
>>>> Also, have you talked to the pypi people about making exceptions for really
>>>> popular projects?  If critical packages like numpy, scipy, and matplotlib
>>>> cannot use pypi, that seems like a major failing of the system.
>>> Here's the pricing:http://aws.amazon.com/s3/#pricing. The free tier
>>> programme limits are on there too. Unfortunately, I do not have the
>>> knowledge to be able to say whether we would hit that or not.
>>
>> Since Nov 3, when 1.2.0 was released, we've used 1.7 GB of transfer 
>> from the github download site.  The S3 "free tier" limit of 1.5 
>> GB/month is awfully close to that.
> Oops -- I totally misread the S3 requirements: it's 15GB/month, so 
> we're fine there, but as Eric pointed out, there's also a 20,000 
> request limit per month, which we're well over (we've have 67,500 
> requests since Nov 3's 1.2.0 release).
And once again, writing e-mails before coffee is a bad idea ;)

We've used about 1.7TB in the approx six weeks since the 1.2.0 release.

Mike

On 12/16/2012 02:38 PM, Todd wrote:
> On Sun, Dec 16, 2012 at 8:21 PM, Damon McDougall 
> <dam...@gm... <mailto:dam...@gm...>> wrote:
>
>     On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>     <jas...@cr... <mailto:jas...@cr...>>
>     wrote:
>     > On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>     >> sourceforge's horror of an interface.
>     >
>     > I'll second that.  Every time I go to Sourceforge, I have to
>     figure out
>     > how in the world to download what I want (and I have to figure
>     out which
>     > things *not* to click on too).
>
>     Ok sounds like there is a reasonable amount of resistance towards
>     Sourceforge.
>
>     Eric, when you suggest that NumFocus could 'provide hosting directly',
>     do you mean they would have the physical hardware to host the files,
>     or are you suggesting they provide the finances to seek hosting
>     elsewhere?
>
>     In the GitHub blog post, they suggest using S3. We could try that.
>     It's fairly inexpensive and the first year is free (within monthly
>     bandwidth limits). We could try it for a year and see how that pans
>     out? I'm not entirely sure how the Amazon stuff works but I've heard
>     good things about it.
>
>
> Are you sure the monthly bandwidth limits are sufficient?
>
> Also, have you talked to the pypi people about making exceptions for 
> really popular projects?  If critical packages like numpy, scipy, and 
> matplotlib cannot use pypi, that seems like a major failing of the system.
>
>
I don't know if this is still the case, but having talked to one of the 
people involved with PyPI a couple of years ago, my understanding is 
that their infrastructure simply doesn't support it.  They provide the 
ability to link to external files (which is what matplotlib does now), 
and that is the standard solution to that problem.

Cheers,
Mike

On 12/15/2012 09:25 PM, Jason Grout wrote:
> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>> sourceforge's horror of an interface.
> I'll second that.  Every time I go to Sourceforge, I have to figure out
> how in the world to download what I want (and I have to figure out which
> things *not* to click on too).
>
That's to say nothing of the upload interface, which is 10 times worse! :)

Mike

On 12/17/2012 07:36 AM, Michael Droettboom wrote:
> On 12/16/2012 02:50 PM, Damon McDougall wrote:
>> On Sun, Dec 16, 2012 at 1:38 PM, Todd<tod...@gm...>  wrote:
>>> On Sun, Dec 16, 2012 at 8:21 PM, Damon McDougall<dam...@gm...>
>>> wrote:
>>>> On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>>>> <jas...@cr...>  wrote:
>>>>> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>>>>>> sourceforge's horror of an interface.
>>>>> I'll second that.  Every time I go to Sourceforge, I have to figure out
>>>>> how in the world to download what I want (and I have to figure out which
>>>>> things *not* to click on too).
>>>> Ok sounds like there is a reasonable amount of resistance towards
>>>> Sourceforge.
>>>>
>>>> Eric, when you suggest that NumFocus could 'provide hosting directly',
>>>> do you mean they would have the physical hardware to host the files,
>>>> or are you suggesting they provide the finances to seek hosting
>>>> elsewhere?
>>>>
>>>> In the GitHub blog post, they suggest using S3. We could try that.
>>>> It's fairly inexpensive and the first year is free (within monthly
>>>> bandwidth limits). We could try it for a year and see how that pans
>>>> out? I'm not entirely sure how the Amazon stuff works but I've heard
>>>> good things about it.
>>>>
>>>>
>>> Are you sure the monthly bandwidth limits are sufficient?
>>>
>>> Also, have you talked to the pypi people about making exceptions for really
>>> popular projects?  If critical packages like numpy, scipy, and matplotlib
>>> cannot use pypi, that seems like a major failing of the system.
>> Here's the pricing:http://aws.amazon.com/s3/#pricing. The free tier
>> programme limits are on there too. Unfortunately, I do not have the
>> knowledge to be able to say whether we would hit that or not.
>
> Since Nov 3, when 1.2.0 was released, we've used 1.7 GB of transfer 
> from the github download site.  The S3 "free tier" limit of 1.5 
> GB/month is awfully close to that.
Oops -- I totally misread the S3 requirements: it's 15GB/month, so we're 
fine there, but as Eric pointed out, there's also a 20,000 request limit 
per month, which we're well over (we've have 67,500 requests since Nov 
3's 1.2.0 release).

Cheers,
Mike

On 12/16/2012 02:50 PM, Damon McDougall wrote:
> On Sun, Dec 16, 2012 at 1:38 PM, Todd <tod...@gm...> wrote:
>> On Sun, Dec 16, 2012 at 8:21 PM, Damon McDougall <dam...@gm...>
>> wrote:
>>> On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>>> <jas...@cr...> wrote:
>>>> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>>>>> sourceforge's horror of an interface.
>>>> I'll second that.  Every time I go to Sourceforge, I have to figure out
>>>> how in the world to download what I want (and I have to figure out which
>>>> things *not* to click on too).
>>> Ok sounds like there is a reasonable amount of resistance towards
>>> Sourceforge.
>>>
>>> Eric, when you suggest that NumFocus could 'provide hosting directly',
>>> do you mean they would have the physical hardware to host the files,
>>> or are you suggesting they provide the finances to seek hosting
>>> elsewhere?
>>>
>>> In the GitHub blog post, they suggest using S3. We could try that.
>>> It's fairly inexpensive and the first year is free (within monthly
>>> bandwidth limits). We could try it for a year and see how that pans
>>> out? I'm not entirely sure how the Amazon stuff works but I've heard
>>> good things about it.
>>>
>>>
>> Are you sure the monthly bandwidth limits are sufficient?
>>
>> Also, have you talked to the pypi people about making exceptions for really
>> popular projects?  If critical packages like numpy, scipy, and matplotlib
>> cannot use pypi, that seems like a major failing of the system.
> Here's the pricing: http://aws.amazon.com/s3/#pricing. The free tier
> programme limits are on there too. Unfortunately, I do not have the
> knowledge to be able to say whether we would hit that or not.

Since Nov 3, when 1.2.0 was released, we've used 1.7 GB of transfer from 
the github download site.  The S3 "free tier" limit of 1.5 GB/month is 
awfully close to that.

>
> Matt, have you had experienced comitting binaries to the gh-pages
> branch? Are there size limits?
>
There used to be limits on github-pages specifically, but they seem to 
have silently removed information about them.  As for any github 
repository, the limit is 1GB per repository.

https://help.github.com/articles/what-is-my-disk-quota

We already have 375MB in our documentation (pages) repository.  We use 
about 150MB for all of the binaries for each release.  So we'd be able 
to squeeze about 3-4 releases in there before needing to explicitly 
prune stuff.  Additionally, the link above seems to discourage hosting 
very large files in the pages repository -- I don't know if that means 
they intend to not support them.

Mike

On 12/15/2012 08:10 PM, Matt Newville wrote:
> Hi,
>
>> Github has removed the ability to host binaries.  They've removed this
>> feature without any apparent notification except on their blog saying
>> "it's gone today".  And the suggested alternative is to use paid services.
>>
>> https://github.com/blog/1302-goodbye-uploads
>>
>> I had planned to complete our set of 1.2.0 binaries with a Python 3.2
>> from Russell Owen in the near future.  So much for that.
>>
>> Any thoughts?  Do we go back to Sourceforge for our download hosting?
>> Is anyone familiar with any other services?  Do we try to piggy-back on
>> what other scipy projects are doing?
> Why not use a gh-pages branch to host both docs and binary files from github?
>
>
We are already over the size limits that the used to state for their 
documentation.  I have been personally assured that this won't remain a 
problem, but I do wonder how long until they start cracking down on that 
for free projects as well.  I'd rather not additionally put our binary 
downloads in that basket as well.

Mike

On Sun, Dec 16, 2012 at 2:44 PM, Eric Firing <ef...@ha...> wrote:
> On 2012/12/16 9:21 AM, Damon McDougall wrote:
>> On Sat, Dec 15, 2012 at 8:25 PM, Jason Grout
>> <jas...@cr...> wrote:
>>> On 12/14/12 10:55 AM, Nathaniel Smith wrote:
>>>> sourceforge's horror of an interface.
>>>
>>> I'll second that.  Every time I go to Sourceforge, I have to figure out
>>> how in the world to download what I want (and I have to figure out which
>>> things *not* to click on too).
>>
>> Ok sounds like there is a reasonable amount of resistance towards Sourceforge.
>>
>> Eric, when you suggest that NumFocus could 'provide hosting directly',
>> do you mean they would have the physical hardware to host the files,
>> or are you suggesting they provide the finances to seek hosting
>> elsewhere?
>
> I was thinking that perhaps NumFocus would be running a server that
> could provide the hosting.  Funding for an external service is also
> possible, though, and might make more sense.
>
>>
>> In the GitHub blog post, they suggest using S3. We could try that.
>> It's fairly inexpensive and the first year is free (within monthly
>> bandwidth limits). We could try it for a year and see how that pans
>> out? I'm not entirely sure how the Amazon stuff works but I've heard
>> good things about it.
>>
>
> The github page https://github.com/matplotlib/matplotlib/downloads shows
> 44,000 downloads for the 1.2 tarball, so I don't think the 20,000
> downloads per month limit of the free tier would work.

Note: that's 44,000 downloads for a gzipped bundle of the *source*,
which can still be downloaded via the "Tags" tab
(https://github.com/matplotlib/matplotlib/tags). Any time one of us
creates a tag, GitHub automagically tar/gzips it and makes it
downloadable. As far as I am aware, this is separate to the
"Downloads" section, which is for arbitrary files of any type, not
just source tarballs.

That said, not taking into account the downloads of the tarball, we're
still pretty close to the 20,000 mark.

-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229