Learning PyTorch 2.0: Experiment deep learning from basics to complex models using every potential capability of Pythonic PyTorch

Preface

This book is a comprehensive guide to understanding and utilizing PyTorch 2.0 for deep learning applications. It starts with an introduction to PyTorch, its various advantages over other deep learning frameworks, and its blend with CUDA for GPU acceleration. We delve into the heart of PyTorch – tensors, learning their different types, properties, and operations. Through step-by-step examples, the reader learns to perform basic arithmetic operations on tensors, manipulate them, and understand errors related to tensor shapes.

A substantial portion of the book is dedicated to illustrating how to build simple PyTorch models. This includes uploading and preparing datasets, defining the architecture, training, and predicting. It provides hands-on exercises with a real-world dataset. The book then segues into exploring PyTorch's nn module and gives a detailed comparison of different types of networks like Feedforward, RNN, GRU, CNN, and their combination.

Further, the book delves into understanding the training process and PyTorch's optim module. It explores the overview of optimization algorithms like Gradient Descent, SGD, Mini-batch Gradient Descent, Momentum, Adagrad, and Adam. A separate chapter focuses on advanced concepts in PyTorch 2.0, like model serialization, optimization, distributed training, and PyTorch Quantization API.

In the final chapters, the book discusses the differences between TensorFlow 2.0 and PyTorch 2.0 and the step-by-step process of migrating a TensorFlow model to PyTorch 2.0 using ONNX. It provides an overview of common issues encountered during this process and how to resolve them.

In this book you will learn:

A comprehensive introduction to PyTorch and CUDA for deep learning.

Detailed understanding and operations on PyTorch tensors.

Step-by-step guide to building simple PyTorch models.

Insight into PyTorch's nn module and comparison of various network types.

Overview of the training process and exploration of PyTorch's optim module.

Understanding advanced concepts in PyTorch like model serialization and optimization.

Knowledge on distributed training in PyTorch.

Practical guide to using PyTorch's Quantization API.

Differences between TensorFlow 2.0 and PyTorch 2.0.

Guidance on migrating TensorFlow models to PyTorch using ONNX.

The book equips the reader with comprehensive knowledge of PyTorch 2.0 and provides practical examples to apply this knowledge, making it an ideal read for aspiring network administrators.

Prologue

Dear reader, we are excited to take you on an exciting adventure into the fascinating world of PyTorch 2.0. This book has been painstakingly developed as a thorough reference, with the intention of illuminating the complex terrain of deep learning. Its goal is to change you from a curious enthusiast into a proficient practitioner by providing you with the theoretical understanding as well as the practical expertise necessary to properly use PyTorch. While we embark on this journey of discovery, we want you to participate completely by testing the validity of your previously held beliefs and broadening your perspective on the fascinating field of deep learning.

The first part of the book lays a strong basis in the fundamental principles that underpin PyTorch. Tensors, the fundamental building elements of PyTorch, are the subject of substantial discussion in the first few chapters. These first few chapters will walk you through the steps of comprehending tensors, as well as the process of building and manipulating them. This book will ensure that you have a solid foundation in the fundamentals before going on to more advanced ideas by gradually increasing the difficulty of the topics as you progress through it.

As we continue to delve further into the workings of PyTorch, the book begins to introduce the development of basic as well as more complex models. We break down the many parts that make up these models with the help of some examples to illustrate our points. You will gain knowledge on how to create and train a variety of neural networks, including feed-forward and recurrent neural networks, as well as GRUs and CNNs, which are more advanced models. You will be able to apply what you learn in the book to real-world situations since it combines theoretical explanations with real-world examples, which helps to drive home the many concepts and methods that are covered.

The knowledge of the training process comes next in the book, after an initial look at neural networks. Unraveling optimization algorithms, sometimes known as the engines that drive learning in neural networks, is one of the most important aspects of this endeavor. The book explains each algorithm in a manner that is both comprehensive and easy to understand, beginning with the fundamental Gradient Descent and progressing all the way up to the more complex Adaptive Moment Estimation. You will now have the ability to construct and train robust models using PyTorch thanks to the knowledge and the activities you have completed.

In the latter chapters of this book, we delve into the more sophisticated principles of the PyTorch programming language. You will acquire knowledge regarding the serialization and optimization of models, as well as the use of distributed training and the Quantization API provided by PyTorch. In addition, we investigate the connection between TensorFlow 2.0 and PyTorch 2.0, comparing and contrasting the advantages and disadvantages of both programs. In doing so, the book arms you with the knowledge necessary to select the framework that caters to your requirements in the most optimal manner.

As we embark on this adventure, we want to urge you to approach each chapter with an open heart and a mind that is full of curiosity. This book offers something to offer everyone, from someone who is just beginning out in deep learning to someone who is an experienced professional looking to develop their skills. We have high hopes that by the time you finish reading this book, you will not only have a comprehensive understanding of PyTorch 2.0 but also the self-assurance to put what you have learned into practice in the real world. Let's not waste any more time and go right into this thrilling adventure together, shall we? PyTorch 2.0 is where you want to be, so welcome!

Learning PyTorch 2.0

Experiment deep learning from basics to complex models using every potential capability of Pythonic PyTorch

Matthew Rosch

Copyright © 2023 by GitforGits.

All rights reserved. This book is protected under copyright laws and no part of it may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without the prior written permission of the publisher. Any unauthorized reproduction, distribution, or transmission of this work may result in civil and criminal penalties and will be dealt with in the respective jurisdiction at anywhere in India, in accordance with the applicable copyright laws.

Published by: GitforGits

Publisher: Sonal Dhandre

www.gitforgits.com

[email protected]

Printed in India

First Printing: July 2023

ISBN: 978-8196288372

Cover Design by: Kitten Publishing

For permission to use material from this book, please contact GitforGits at [email protected].

Content

Preface

Chapter 1: Introduction To PyTorch 2.0 and CUDA 11.8

Essentials of Neural Networks

Anatomy of a Neural Network

Learning in a Neural Network

Neural Network and Deep Learning

Introduction to PyTorch

PyTorch Overview

How PyTorch supports Deep Learning

PyTorch 2.0 and CUDA 11.8

PyTorch 2.0 Key Features

Enhanced TorchScript

Model Deployment and Serving

Improved Mobile Support

Enhanced Quantization Support

Named Tensors

Extended Ecosystem

Expanded ONNX Support

CUDA for Accelerated Deep Learning

A Primer on CUDA

Why is CUDA Necessary for Deep Learning?

Blend of CUDA and PyTorch

Setting up PyTorch 2.0 and CUDA 11.8 on Linux

Update Your System

Install Necessary Dependencies

Verify CUDA-capable GPU

Download and Install CUDA 11.8

Setup Environment Variables

Verify the Installation

Install PyTorch 2.0

Verify PyTorch and CUDA Integration

Summary

Chapter 2: Getting Started with Tensors

Diving into Tensors

Understanding Tensors

Different Tensors

Tensor Terminologies and Concepts

Scalar

Vector

Matrix

Tensor Operations

Broadcasting

Device

Sample Program to Create Tensors

Tensor Data Types

Standard Arithmetic Operations

Addition

Subtraction

Multiplication

Division

Tensor Manipulation

Reshaping Tensors

Slicing Tensors

Joining Tensors

Matrix Multiplication

Manage Tensor Shape Errors

Matrix Multiplication

Element-wise Operations

Reshaping Tensors

Aggregation Operations

Sum

Mean

Max

Min

Summary

Chapter 3: Advanced Tensor Operations

Advanced Tensor Operations Overview

Reshaping

Viewing

Stacking

Changing Shape of Tensor

Viewing Tensor Dimensions

Stacking Sequence of Tensors

Squeezing a Tensor

Un-squeezing a Tensor

Permuting a Tensor

Indexing and Selecting Data from Tensors

Indexing a Tensor

Slicing a Tensor

Selecting Specific Indices

Switching Access Between GPU and CPU

Necessity of Access of Choice

Summary

Chapter 4: Building Neural Networks with PyTorch 2.0

PyTorch NN Module Overview

Understanding Feedforward Networks

Overview

Advantages of Feedforward Networks

Build Multi-layer Perceptron using PyTorch

Understanding Recurrent Neural Networks

Overview

Creating RNNs in PyTorch

Exploring Gated Recurrent Units (GRUs)

Overview

Advantages of GRUs

Creating Gated Recurrent Units (GRUs)

Understanding Convolutional Neural Networks

Overview

Advantages of CNNs

Creating Convolutional Neural Networks (CNNs)

Combining RNN and CNN Together

Need of Designing Custom Layers

Design Custom Layers

Summary

Chapter 5. Training Neural Networks in PyTorch 2.0

Neural Network Training - Overview

PyTorch optim Module

Using Optimizer

Optimization Algorithms

Gradient Descent

Using Gradient Descent

Stochastic Gradient Descent

Using Stochastic Gradient Descent

Mini-batch Gradient Descent

Using Mini-batch Gradient Descent

Momentum

Using Momentum with SGD

Adaptive Gradient Algorithm

Using Adagrad Algorithm

Adaptive Moment Estimation

Summary

Chapter 6: PyTorch 2.0 Advanced

Overview

Model Serialization

Model Optimization

Perform Model Serialization

Serialize Models

Optimize Models

Distributed Training of Deep Learning

Methods of Distributed Training

Distributed Training in PyTorch

Using DataParallel