Writing a gradient tutorial, focused on leaf vs non leaf tensors.

[JitheshPavan]

There is no tutorial that specifically talks about requires_grad, retain_grad, and leaf tensor/ non-leaf tensors and how they interact with each other. Can I write a tutorial specifically talking about this topic? This will be useful when gradients are used in unusual places, as is the case for the deep dream algorithm.

cc: @albanD

[albanD]

I think https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation contains the most in-depth discussion of this topic. I'm sure we could expand on it in a tutorial if there are enough questions we want to cover.
cc @soulitzer

[j-silv]

/assigntome

I'd love to give this a shot. It'd be great to have a little more guidance on what exactly we want to cover in the tutorial though. Just some dummy examples?

[soulitzer]

I'm not sure we necessarily need a separate tutorial, but this is indeed a gap

I think we'd benefit from beefing up the documentation for https://docs.pytorch.org/docs/stable/generated/torch.Tensor.retain_grad.html#torch.Tensor.retain_grad
by adding an example

The intro autograd tutorial already covers requires_grad https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial, but maybe worth linking/briefly to retains_grad.

[j-silv]

One note is that I don't think PyTorch has any tutorial on explicitly plotting/checking intermediate gradients.

Perhaps we could create a "visualizing gradients" tutorial, which seems like the natural place to include information about retain_grad, require_grad, and the concept of leaf vs non-leaf nodes. What do you think?

As a demo, we could investigate exploding/vanishing gradients and plot the gradient flow with something like this.

Edit 1: Andrej Karparthy did this in one of his zero-to-hero lectures when introducing batchnorm, and I think it was a valuable exercise. Note in the jupyter notebook he has to use retain_grad.

Edit 2: The original author of this issue mentioned the deep dream algorithm. That could be the application - "Visualizing Gradients - DeepDream" or something like that. It requires the intermediate gradients so that we can apply gradient ascent at some layer of the network, so there we could introduce the associated autograd features.

Edit 3: After doing some initial write-up, I've come to the conclusion that a "Visualizing Gradients" tutorial should probably be a stand-alone recipe which has a section on leaf/non-leaf tensors and require_grad/retain_grad parameters. The API docs for retain_grad, is_leaf, etc. could then point to that section. The recipe will include a section on visualizing gradients with the plot method I mentioned previously.

With that said, I still would like to create a separate tutorial for DeepDream, unless there are any objections. I think it's a really great way to show off the existing models within PyTorch (e.g. GoogLeNet) and how to hack one to visualize some pretty cool stuff. After work on the recipe, I will open up a PR for that.

Edit 4: Can't decide if the visualizing gradients should be a recipe or an advanced (maybe intermediate?) tutorial.

[j-silv]

PR is coming soon