DOC: Document release procedure with a walkthrough. #8106

charris · 2016-10-01T18:18:16Z

This is a walkthrough of a release. I have found it helpful to have the commands laid out in a step by step way so that I can go through line by line. This is preliminary, but does show parts of the procedure that could be further automated and provides a place for comments as well as a handy checklist. It is intended to supplement the expanded content in doc/HOWTO_RELEASE.

I'm hoping to foist some releases off on other folks, this should help.

charris · 2016-10-01T18:47:50Z

@rgommers @ev-br Comments would be helpful. I'm trying to make the actual release process easier to follow.

rgommers

Thanks Chuck, looks quite useful. My main comments are about the missing step for uploading docs to docs.scipy.org and moving away from SourceForge.

rgommers · 2016-10-01T21:32:05Z

doc/RELEASE_WALKTHROUGH.rst.txt

+
+    $ git tag -s v1.11.2rc1
+    $ python setup.py install --user
+    $ paver sdist


Maybe add a short comment that sdist starts with git clean -xdf, so this is OK (doesn't pick up the build artifacts from the last step)

rgommers · 2016-10-01T21:35:50Z

doc/RELEASE_WALKTHROUGH.rst.txt

+    $ python setup.py install --user
+    $ paver sdist
+    $ paver write_release_and_log
+    $ paver pdf


I think we should move away from SourceForge, so this is not needed anymore then. Now that we don't distribute .exe installers there's really no use for SF anymore, and it has a pretty bad reputation by now (deservedly).

What is missing is the procedure to upload pdf and html docs to docs.scipy.org, which would be good to add instead.

I'll propose dropping sourceforge on the list. We will need to edit some documentation and put a note up on sourceforge. I'll release 1.11.2 there as 1.11.2rc1 is already up.

I don't have permission to upload to docs.scipy.org, how should we deal with that?

we should get you access, but for now you can ping me to do the uploads. we're having a bit of an issue with that server at the moment; all the admins had misplaced their passwords. I have ssh access but no admin rights, so can't add you.

Instructions are:

$ make dist # check that html and pdf docs look right $ make upload USERNAME=<username> # username with ssh access to docs.scipy.org

Sorry, that's incorrect. Should be

$ make dist # check that html and pdf docs look right $ make upload USERNAME=<username> RELEASE=1.11.2 # username with ssh access to docs.scipy.org

Should be possible to put this into paver?

yep not difficult. The USERNAME=<username> still needs to be input also then, so won't get much shorter.

rgommers · 2016-10-01T21:38:46Z

doc/RELEASE_WALKTHROUGH.rst.txt

+the needed wheels have been build. There should be 17 of them at
+`<https://wheels.scipy.org>`_, 3 for Mac, 6 for Windows, and 8 for Linux. Here
+the wheelhouse directory is in the home directory. Then upload the
+wheels. You will need to sign every file separately in the process::


Just a note (text is fine): PyPI will likely lose support for signing soon, so this will become simpler.

Hmm, will there be anything to replace signing, or will it rely on permissions?

not sure yet. there was a long discussion about how signing was not very secure in practice anyway. haven't followed the details, but IIRC it'll go away with the switch from the current infrastructure to Warehouse

rgommers · 2016-10-01T21:39:18Z

doc/RELEASE_WALKTHROUGH.rst.txt

+the wheelhouse directory is in the home directory. Then upload the
+wheels. You will need to sign every file separately in the process::
+
+    $ cd ../terryfy.git/


should be cd ../terryfy?

I append .git to git repo directories. I can change that for this documentation. This walkthrough probably needs the layout it is written for documented at the beginning.

rgommers · 2016-10-01T21:40:22Z

doc/RELEASE_WALKTHROUGH.rst.txt

+
+The source files should be uploaded last to avoid problems::
+
+    $ cd ../numpy.git


same here, cd ../numpy

rgommers · 2016-10-01T21:42:28Z

doc/RELEASE_WALKTHROUGH.rst.txt

+    $ git checkout master
+    $ git pull upstream master
+    $ git checkout -b numpy-1.11.2rc1
+    $ gvim www/index.rst


add a comment # edit the News section

rgommers · 2016-10-01T21:43:03Z

doc/RELEASE_WALKTHROUGH.rst.txt

+Announce to the mailing list
+----------------------------
+
+The release should be announced on the numpy-devel, scipy-devel, and scipy-user mailing


numpy-discussion not numpy-devel. also, add python-announce

rgommers · 2016-10-01T21:43:41Z

doc/RELEASE_WALKTHROUGH.rst.txt

+
+This assumes that you have forked `<https://github.com/scipy/scipy.org>`_::
+
+    $ cd ../scipy.org.git


cd ../scipy.org

charris · 2016-10-01T22:47:38Z

@rgommers Looks like you have stuck yourself with some work by requesting changes, as they are made but apparently you need to sign off on them. Or maybe github can't deal with rewriting history.

rgommers · 2016-10-02T02:31:51Z

Looks like you have stuck yourself with some work by requesting changes, as they are made but apparently you need to sign off on them

Hit the button. Looks like a PR can be merged without that approval (which is good, otherwise there's a bottleneck on 1 person), it's just the color stays red until the reviewer who requested changes approved them.

Still getting used to the review functionality, but overall i like it.

rgommers · 2016-10-02T02:33:41Z

Still getting used to the review functionality, but overall i like it.

except that now comments that have been addressed aren't hidden on this conversation page....

charris · 2016-10-02T02:44:08Z

Could get ugly with comments...

I'm not ready to merge this yet, going to add a script for the announcement and a short list of dependencies. At some point should probably cover making the maintenance branch also. Trying to keep text at a minimum though.

ev-br · 2016-10-02T13:16:25Z

This looks useful!

I second the suggestion to drop Sourceforge in favor of GH releases.

Here's my rough draft of a checklist for the last scipy release: https://gist.github.com/ev-br/dd5878cb66aa52e8a548fa7eece55fec
The main difference seems to be that I checksum the wheels locally before uploading them to PyPI. The checksums end up in the README file on GH and in the email announcements. (Also cf some discussion in MacPython/terryfy#6)

rgommers · 2016-10-04T08:37:14Z

good point @ev-br, +1 for wheel checksums

charris · 2016-10-04T15:12:59Z

I'll revisit this after the 1.12.0 release. There appear to be two unresolved issues remaining

Signing/checksumming PyPI uploads
Documentation

Adding the checksums to the README.md could get cumbersome after a while, and it isn't the easiest thing to find. I suppose if the files aren't signed the checksum is needed to check the integrity of the upload.

rgommers · 2016-10-04T17:34:51Z

Just found another issue: tarballs need to be uploaded to GitHub Releases. Now we get the automatically created tarballs from the release tags, but those are not the same (no generated .c files from cython sources for example).

charris · 2016-11-04T23:15:11Z

@rgommers How does one upload tarballs to GitHub Releases?

rgommers · 2016-11-04T23:29:24Z

When you go to https://github.com/numpy/numpy/releases, there's a "draft a new release" button (right top). Click that, then you can edit the description (copy from release notes) and attached files in the UI.

charris · 2016-11-04T23:39:47Z

@rgommers Thanks. Has the password for documentation upload been found? Also, what documentation gets uploaded, HTML and PDF?

charris · 2016-11-04T23:47:24Z

@ev-br I don't see the checksums in the README.md on GH. Also, what do you use for the checksum, shasum -a 256, md5sum, ?

ev-br · 2016-11-05T00:07:20Z

Other README, here: https://github.com/scipy/scipy/releases
It"s generated by paver sdist. When the wheels are ready, I copy them to release/installers folder. Then paver write_release_and_log checksums all files in that folder.
This is scipy's pavement.py file, not sure how different numpy's is.
One trick is to download the wheels to some other location first: various paver commands like to clean the release/ folder.

charris · 2016-11-05T00:21:08Z

@ev-br Do you find the Changelog useful? I'm thinking of dropping it in favor of the PR links. I note the Source code links. Where do they come from?

rgommers · 2016-11-05T00:26:26Z

Has the password for documentation upload been found?

Yep, @pv has root access and I have ssh access again.

Also, what documentation gets uploaded, HTML and PDF?

Yes, those two. See `make distandmake upload``in``doc/Makefile``.

rgommers · 2016-11-05T00:27:03Z

This is scipy's pavement.py file, not sure how different numpy's is.

They're pretty much the same, and differences are probably unintentional.

ev-br · 2016-11-05T00:27:22Z

Source code links are I believe the useless autogenerated ones that Ralf mentioned. The tar.gz, zip and xz files are uploaded manuallyas binaries instead.
Changelog, no, I've no uses for it. I always thought it's something downstream integrators use for ... I dunno what :-).

rgommers · 2016-11-05T00:28:09Z

Changelog I think is just historically the way people did things. No one really looks at it AFAIK.

charris · 2016-11-05T00:33:11Z

@rgommers So should I get access or should I ping you for that?

rgommers · 2016-11-05T00:44:04Z

I'm happy to do the uploads (ping me when needed), but getting you access may also make sense. Send @pv your ssh key I guess.

charris · 2016-11-17T19:58:07Z

@rgommers Do we upload documents for pre-releases?

rgommers · 2016-11-17T20:20:12Z

no we don't

charris · 2016-11-17T21:29:22Z

@rgommers Could you document the changes needed in the index.rst file of docs.scipy.org? Also, would it be better to make that a normal pull request as opposed to simply pushing the result?

rgommers · 2016-11-18T10:24:17Z

doc/RELEASE_WALKTHROUGH.rst.txt

+
+    $ pushd doc
+    $ make dist
+    $ make upload USERNAME=<yourname> RELEASE=1.11.2rc1


remove rc1, we don't upload pre-release docs

rgommers · 2016-11-18T10:25:20Z

doc/RELEASE_WALKTHROUGH.rst.txt

+``docs.scipy.org`` front page::
+
+    $ cd ../docs.scipy.org
+    $ gvim index.rst 


Send a PR to https://github.com/scipy/docs.scipy.org with the changes that add links to the new docs. See scipy/docs.scipy.org#7 for an example

Also update the lists of release dependencies in doc/HOWTO_RELEASE.rst.txt. [ci skip]

ev-br · 2017-03-18T19:09:59Z

doc/RELEASE_WALKTHROUGH.rst.txt

+8 for Windows, and 10 for Linux. Here the wheelhouse directory is in the home
+directory. Then upload the wheels. You will need to sign every file separately
+in the process, keeping the pass phrase in the clipboard and pasting it in will
+make that easier::


IIRC twine -s signs them all, e.g. $ twine -s *macosx*whl only asks for the passphrase once.

I thought that also, but vaguely recall it not working. Probably need to test again. In any case, wheel-uploader doesn't do that and when I asked @matthew-brett about it he said they could not get it to work.

Might be a function of the twine version. I didn't realize that pip did not automatically upgrade, so was running version 1.6.5.

charris · 2017-03-26T17:29:45Z

I'm going to merge this now for reference. The documentation section will need revision if/when we move the documents to github.

charris added the 04 - Documentation label Oct 1, 2016

charris force-pushed the release-walkthrough branch from f784259 to 49dbd7d Compare October 1, 2016 18:21

charris force-pushed the release-walkthrough branch from 4e44300 to 3e9f324 Compare October 1, 2016 19:04

rgommers requested changes Oct 1, 2016

View reviewed changes

charris force-pushed the release-walkthrough branch 2 times, most recently from ce92adc to 8a5ab57 Compare October 1, 2016 22:41

rgommers approved these changes Oct 2, 2016

View reviewed changes

charris force-pushed the release-walkthrough branch 5 times, most recently from 981f9ef to 4c543e0 Compare October 4, 2016 03:08

charris force-pushed the release-walkthrough branch 2 times, most recently from 0ad0f86 to 0d814f0 Compare October 4, 2016 14:56

charris force-pushed the release-walkthrough branch from 0d814f0 to e0f1eda Compare November 4, 2016 23:32

charris force-pushed the release-walkthrough branch 2 times, most recently from 71d2767 to c8443d6 Compare November 17, 2016 19:57

charris force-pushed the release-walkthrough branch from c8443d6 to b2dd8c0 Compare November 18, 2016 00:33

rgommers reviewed Nov 18, 2016

View reviewed changes

charris force-pushed the release-walkthrough branch from b2dd8c0 to 4915ddd Compare January 21, 2017 17:35

charris force-pushed the release-walkthrough branch from 4915ddd to c48939b Compare March 18, 2017 18:09

DOC: Document release procedure with a walkthrough.

fc19d05

Also update the lists of release dependencies in doc/HOWTO_RELEASE.rst.txt. [ci skip]

charris force-pushed the release-walkthrough branch from c48939b to fc19d05 Compare March 18, 2017 18:27

ev-br reviewed Mar 18, 2017

View reviewed changes

charris merged commit d934718 into numpy:master Mar 26, 2017

charris deleted the release-walkthrough branch March 26, 2017 17:30

charris changed the title ~~WIP, DOC: Document release procedure with a walkthrough.~~ DOC: Document release procedure with a walkthrough. Mar 26, 2017


		The source files should be uploaded last to avoid problems::

		$ cd ../numpy.git


		This assumes that you have forked `<https://github.com/scipy/scipy.org>`_::

		$ cd ../scipy.org.git

Uh oh!

DOC: Document release procedure with a walkthrough. #8106

DOC: Document release procedure with a walkthrough. #8106

Uh oh!

Conversation

charris commented Oct 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charris commented Oct 1, 2016

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charris commented Oct 1, 2016

Uh oh!

rgommers commented Oct 2, 2016

Uh oh!

rgommers commented Oct 2, 2016

Uh oh!

charris commented Oct 2, 2016

Uh oh!

ev-br commented Oct 2, 2016

Uh oh!

rgommers commented Oct 4, 2016

Uh oh!

charris commented Oct 4, 2016

Uh oh!

rgommers commented Oct 4, 2016

Uh oh!

charris commented Nov 4, 2016

Uh oh!

rgommers commented Nov 4, 2016

Uh oh!

charris commented Nov 4, 2016

Uh oh!

charris commented Nov 4, 2016

Uh oh!

ev-br commented Nov 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charris commented Nov 5, 2016

Uh oh!

rgommers commented Nov 5, 2016

Uh oh!

charris commented Oct 1, 2016 •

edited

Loading

ev-br commented Nov 5, 2016 •

edited

Loading