-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathautograd.html
1876 lines (1672 loc) · 397 KB
/
autograd.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Autograd mechanics — PyTorch 2.7 documentation</title>
<link rel="canonical" href="https://pytorch.org/docs/stable/notes/autograd.html"/>
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<!-- <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> -->
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/copybutton.css" type="text/css" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" type="text/css" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" type="text/css" />
<link rel="stylesheet" href="../_static/katex-math.css" type="text/css" />
<link rel="stylesheet" href="../_static/sphinx-dropdown.css" type="text/css" />
<link rel="stylesheet" href="../_static/panels-bootstrap.min.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/jit.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Broadcasting semantics" href="broadcasting.html" />
<link rel="prev" title="Automatic Mixed Precision examples" href="amp_examples.html" />
<!-- Google Tag Manager -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-T8XT4PS');</script>
<!-- End Google Tag Manager -->
<script src="../_static/js/modernizr.min.js"></script>
<!-- Preload the theme fonts -->
<link rel="preload" href="../_static/fonts/FreightSans/freight-sans-book.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="../_static/fonts/FreightSans/freight-sans-medium.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="../_static/fonts/IBMPlexMono/IBMPlexMono-Medium.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="../_static/fonts/FreightSans/freight-sans-bold.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="../_static/fonts/FreightSans/freight-sans-medium-italic.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="../_static/fonts/IBMPlexMono/IBMPlexMono-SemiBold.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<!-- Preload the katex fonts -->
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Math-Italic.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Main-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Main-Bold.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Size1-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Size4-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Size2-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Size3-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/fonts/KaTeX_Caligraphic-Regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.15.2/css/all.css" integrity="sha384-vSIIfh2YWi9wW0r9iZe7RJPrKwp6bG+s9QZMoITbCckVJqGCCRhc+ccxNcdpHuYu" crossorigin="anonymous">
</head>
<div class="container-fluid header-holder tutorials-header" id="header-holder">
<div class="container">
<div class="header-container">
<a class="header-logo" href="https://pytorch.org/" aria-label="PyTorch"></a>
<div class="main-menu">
<ul>
<li class="main-menu-item">
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
Learn
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/get-started">
<span class=dropdown-title>Get Started</span>
<p>Run PyTorch locally or get started quickly with one of the supported cloud platforms</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/tutorials">
<span class="dropdown-title">Tutorials</span>
<p>Whats new in PyTorch tutorials</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/tutorials/beginner/basics/intro.html">
<span class="dropdown-title">Learn the Basics</span>
<p>Familiarize yourself with PyTorch concepts and modules</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/tutorials/recipes/recipes_index.html">
<span class="dropdown-title">PyTorch Recipes</span>
<p>Bite-size, ready-to-deploy PyTorch code examples</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/tutorials/beginner/introyt.html">
<span class="dropdown-title">Intro to PyTorch - YouTube Series</span>
<p>Master PyTorch basics with our engaging YouTube tutorial series</p>
</a>
</div>
</div>
</li>
<li>
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
Ecosystem
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/ecosystem">
<span class="dropdown-title">Tools</span>
<p>Learn about the tools and frameworks in the PyTorch Ecosystem</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/#community-module">
<span class=dropdown-title>Community</span>
<p>Join the PyTorch developer community to contribute, learn, and get your questions answered</p>
</a>
<a class="nav-dropdown-item" href="https://discuss.pytorch.org/" target="_blank">
<span class=dropdown-title>Forums</span>
<p>A place to discuss PyTorch code, issues, install, research</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/resources">
<span class=dropdown-title>Developer Resources</span>
<p>Find resources and get questions answered</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/ecosystem/contributor-awards-2024">
<span class="dropdown-title">Contributor Awards - 2024</span>
<p>Award winners announced at this year's PyTorch Conference</p>
</a>
</div>
</div>
</li>
<li>
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
Edge
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/edge">
<span class="dropdown-title">About PyTorch Edge</span>
<p>Build innovative and privacy-aware AI experiences for edge devices</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/executorch-overview">
<span class="dropdown-title">ExecuTorch</span>
<p>End-to-end solution for enabling on-device inference capabilities across mobile and edge devices</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/executorch/stable/index.html">
<span class="dropdown-title">ExecuTorch Docs</span>
</a>
</div>
</div>
</li>
<li class="main-menu-item">
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
Docs
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/docs/stable/index.html">
<span class="dropdown-title">PyTorch</span>
<p>Explore the documentation for comprehensive guidance on how to use PyTorch</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/pytorch-domains">
<span class="dropdown-title">PyTorch Domains</span>
<p>Read the PyTorch Domains documentation to learn more about domain-specific libraries</p>
</a>
</div>
</div>
</li>
<li>
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
Blogs & News
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/blog/">
<span class="dropdown-title">PyTorch Blog</span>
<p>Catch up on the latest technical news and happenings</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/community-blog">
<span class="dropdown-title">Community Blog</span>
<p>Stories from the PyTorch ecosystem</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/videos">
<span class="dropdown-title">Videos</span>
<p>Learn about the latest PyTorch tutorials, new, and more </p>
<a class="nav-dropdown-item" href="https://pytorch.org/community-stories">
<span class="dropdown-title">Community Stories</span>
<p>Learn how our community solves real, everyday machine learning problems with PyTorch</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/events">
<span class="dropdown-title">Events</span>
<p>Find events, webinars, and podcasts</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/newsletter">
<span class="dropdown-title">Newsletter</span>
<p>Stay up-to-date with the latest updates</p>
</a>
</div>
</li>
<li>
<div id="resourcesDropdownButton" data-toggle="resources-dropdown" class="resources-dropdown">
<a class="with-down-arrow">
About
</a>
<div class="resources-dropdown-menu">
<a class="nav-dropdown-item" href="https://pytorch.org/foundation">
<span class="dropdown-title">PyTorch Foundation</span>
<p>Learn more about the PyTorch Foundation</p>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/governing-board">
<span class="dropdown-title">Governing Board</span>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/credits">
<span class="dropdown-title">Cloud Credit Program</span>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/tac">
<span class="dropdown-title">Technical Advisory Council</span>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/staff">
<span class="dropdown-title">Staff</span>
</a>
<a class="nav-dropdown-item" href="https://pytorch.org/contact-us">
<span class="dropdown-title">Contact Us</span>
</a>
</div>
</div>
</li>
<li class="main-menu-item">
<div class="no-dropdown">
<a href="https://pytorch.org/join" data-cta="join">
Become a Member
</a>
</div>
</li>
<li>
<div class="main-menu-item">
<a href="https://github.com/pytorch/pytorch" class="github-icon">
</a>
</div>
</li>
<!--- TODO: This block adds the search icon to the nav bar. We will enable it later.
<li>
<div class="main-menu-item">
<a href="https://github.com/pytorch/pytorch" class="search-icon">
</a>
</div>
</li>
--->
</ul>
</div>
<a class="main-menu-open-button" href="#" data-behavior="open-mobile-menu"></a>
</div>
</div>
</div>
<body class="pytorch-body">
<div class="table-of-contents-link-wrapper">
<span>Table of Contents</span>
<a href="#" class="toggle-table-of-contents" data-behavior="toggle-table-of-contents"></a>
</div>
<nav data-toggle="wy-nav-shift" class="pytorch-left-menu" id="pytorch-left-menu">
<div class="pytorch-side-scroll">
<div class="pytorch-menu pytorch-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<div class="pytorch-left-menu-search">
<div class="version">
<a href='https://pytorch.org/docs/versions.html'>2.7 ▼</a>
</div>
<div id="searchBox">
<div class="searchbox" id="googleSearchBox">
<script async src="https://cse.google.com/cse.js?cx=e65585f8c3ea1440e"></script>
<div class="gcse-search"></div>
</div>
<div id="sphinxSearchBox" style="display: none;">
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search Docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
</div>
<form id="searchForm">
<label style="margin-bottom: 1rem">
<input type="radio" name="searchType" value="google" checked>
Google Search
</label>
<label style="margin-bottom: 1rem">
<input type="radio" name="searchType" value="sphinx">
Classic Search
</label>
</form>
<script>
document.addEventListener('DOMContentLoaded', function() {
const searchForm = document.getElementById('searchForm');
const googleSearchBox = document.getElementById('googleSearchBox');
const sphinxSearchBox = document.getElementById('sphinxSearchBox');
// Function to toggle search box visibility
function toggleSearchBox(searchType) {
googleSearchBox.style.display = searchType === 'google' ? 'block' : 'none';
sphinxSearchBox.style.display = searchType === 'sphinx' ? 'block' : 'none';
}
// Determine the default search type
let defaultSearchType;
const currentUrl = window.location.href;
if (currentUrl.startsWith('https://pytorch.org/docs/stable')) {
// For the stable documentation, default to Google
defaultSearchType = localStorage.getItem('searchType') || 'google';
} else {
// For any other version, including docs-preview, default to Sphinx
defaultSearchType = 'sphinx';
}
// Set the default search type
document.querySelector(`input[name="searchType"][value="${defaultSearchType}"]`).checked = true;
toggleSearchBox(defaultSearchType);
// Event listener for changes in search type
searchForm.addEventListener('change', function(event) {
const selectedSearchType = event.target.value;
localStorage.setItem('searchType', selectedSearchType);
toggleSearchBox(selectedSearchType);
});
// Set placeholder text for Google search box
window.onload = function() {
var placeholderText = "Search Docs";
var googleSearchboxText = document.querySelector("#gsc-i-id1");
if (googleSearchboxText) {
googleSearchboxText.placeholder = placeholderText;
googleSearchboxText.style.fontFamily = 'FreightSans';
googleSearchboxText.style.fontSize = "1.2rem";
googleSearchboxText.style.color = '#262626';
}
};
});
</script>
</div>
<p class="caption" role="heading"><span class="caption-text">Community</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../community/build_ci_governance.html">PyTorch Governance | Build + CI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/contribution_guide.html">PyTorch Contribution Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/design.html">PyTorch Design Philosophy</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/governance.html">PyTorch Governance | Mechanics</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/persons_of_interest.html">PyTorch Governance | Maintainers</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Developer Notes</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="amp_examples.html">Automatic Mixed Precision examples</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Autograd mechanics</a></li>
<li class="toctree-l1"><a class="reference internal" href="broadcasting.html">Broadcasting semantics</a></li>
<li class="toctree-l1"><a class="reference internal" href="cpu_threading_torchscript_inference.html">CPU threading and TorchScript inference</a></li>
<li class="toctree-l1"><a class="reference internal" href="cuda.html">CUDA semantics</a></li>
<li class="toctree-l1"><a class="reference internal" href="custom_operators.html">PyTorch Custom Operators Landing Page</a></li>
<li class="toctree-l1"><a class="reference internal" href="ddp.html">Distributed Data Parallel</a></li>
<li class="toctree-l1"><a class="reference internal" href="extending.html">Extending PyTorch</a></li>
<li class="toctree-l1"><a class="reference internal" href="extending.func.html">Extending torch.func with autograd.Function</a></li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">Frequently Asked Questions</a></li>
<li class="toctree-l1"><a class="reference internal" href="fsdp.html">FSDP Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="get_start_xpu.html">Getting Started on Intel GPU</a></li>
<li class="toctree-l1"><a class="reference internal" href="gradcheck.html">Gradcheck mechanics</a></li>
<li class="toctree-l1"><a class="reference internal" href="hip.html">HIP (ROCm) semantics</a></li>
<li class="toctree-l1"><a class="reference internal" href="large_scale_deployments.html">Features for large-scale deployments</a></li>
<li class="toctree-l1"><a class="reference internal" href="libtorch_stable_abi.html">LibTorch Stable ABI</a></li>
<li class="toctree-l1"><a class="reference internal" href="modules.html">Modules</a></li>
<li class="toctree-l1"><a class="reference internal" href="mps.html">MPS backend</a></li>
<li class="toctree-l1"><a class="reference internal" href="multiprocessing.html">Multiprocessing best practices</a></li>
<li class="toctree-l1"><a class="reference internal" href="numerical_accuracy.html">Numerical accuracy</a></li>
<li class="toctree-l1"><a class="reference internal" href="randomness.html">Reproducibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="serialization.html">Serialization semantics</a></li>
<li class="toctree-l1"><a class="reference internal" href="windows.html">Windows FAQ</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Language Bindings</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../cpp_index.html">C++</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/javadoc/">Javadoc</a></li>
<li class="toctree-l1"><a class="reference internal" href="../deploy.html">torch::deploy</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Python API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../torch.html">torch</a></li>
<li class="toctree-l1"><a class="reference internal" href="../nn.html">torch.nn</a></li>
<li class="toctree-l1"><a class="reference internal" href="../nn.functional.html">torch.nn.functional</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tensors.html">torch.Tensor</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tensor_attributes.html">Tensor Attributes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tensor_view.html">Tensor Views</a></li>
<li class="toctree-l1"><a class="reference internal" href="../amp.html">torch.amp</a></li>
<li class="toctree-l1"><a class="reference internal" href="../autograd.html">torch.autograd</a></li>
<li class="toctree-l1"><a class="reference internal" href="../library.html">torch.library</a></li>
<li class="toctree-l1"><a class="reference internal" href="../accelerator.html">torch.accelerator</a></li>
<li class="toctree-l1"><a class="reference internal" href="../cpu.html">torch.cpu</a></li>
<li class="toctree-l1"><a class="reference internal" href="../cuda.html">torch.cuda</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch_cuda_memory.html">Understanding CUDA Memory Usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch_cuda_memory.html#generating-a-snapshot">Generating a Snapshot</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch_cuda_memory.html#using-the-visualizer">Using the visualizer</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch_cuda_memory.html#snapshot-api-reference">Snapshot API Reference</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mps.html">torch.mps</a></li>
<li class="toctree-l1"><a class="reference internal" href="../xpu.html">torch.xpu</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mtia.html">torch.mtia</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mtia.memory.html">torch.mtia.memory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../meta.html">Meta device</a></li>
<li class="toctree-l1"><a class="reference internal" href="../backends.html">torch.backends</a></li>
<li class="toctree-l1"><a class="reference internal" href="../export.html">torch.export</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.html">torch.distributed</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.tensor.html">torch.distributed.tensor</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.algorithms.join.html">torch.distributed.algorithms.join</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.elastic.html">torch.distributed.elastic</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fsdp.html">torch.distributed.fsdp</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.fsdp.fully_shard.html">torch.distributed.fsdp.fully_shard</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.tensor.parallel.html">torch.distributed.tensor.parallel</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.optim.html">torch.distributed.optim</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.pipelining.html">torch.distributed.pipelining</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributed.checkpoint.html">torch.distributed.checkpoint</a></li>
<li class="toctree-l1"><a class="reference internal" href="../distributions.html">torch.distributions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch.compiler.html">torch.compiler</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fft.html">torch.fft</a></li>
<li class="toctree-l1"><a class="reference internal" href="../func.html">torch.func</a></li>
<li class="toctree-l1"><a class="reference internal" href="../futures.html">torch.futures</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fx.html">torch.fx</a></li>
<li class="toctree-l1"><a class="reference internal" href="../fx.experimental.html">torch.fx.experimental</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hub.html">torch.hub</a></li>
<li class="toctree-l1"><a class="reference internal" href="../jit.html">torch.jit</a></li>
<li class="toctree-l1"><a class="reference internal" href="../linalg.html">torch.linalg</a></li>
<li class="toctree-l1"><a class="reference internal" href="../monitor.html">torch.monitor</a></li>
<li class="toctree-l1"><a class="reference internal" href="../signal.html">torch.signal</a></li>
<li class="toctree-l1"><a class="reference internal" href="../special.html">torch.special</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch.overrides.html">torch.overrides</a></li>
<li class="toctree-l1"><a class="reference internal" href="../package.html">torch.package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../profiler.html">torch.profiler</a></li>
<li class="toctree-l1"><a class="reference internal" href="../nn.init.html">torch.nn.init</a></li>
<li class="toctree-l1"><a class="reference internal" href="../nn.attention.html">torch.nn.attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="../onnx.html">torch.onnx</a></li>
<li class="toctree-l1"><a class="reference internal" href="../optim.html">torch.optim</a></li>
<li class="toctree-l1"><a class="reference internal" href="../complex_numbers.html">Complex Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../ddp_comm_hooks.html">DDP Communication Hooks</a></li>
<li class="toctree-l1"><a class="reference internal" href="../quantization.html">Quantization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../rpc.html">Distributed RPC Framework</a></li>
<li class="toctree-l1"><a class="reference internal" href="../random.html">torch.random</a></li>
<li class="toctree-l1"><a class="reference internal" href="../masked.html">torch.masked</a></li>
<li class="toctree-l1"><a class="reference internal" href="../nested.html">torch.nested</a></li>
<li class="toctree-l1"><a class="reference internal" href="../size.html">torch.Size</a></li>
<li class="toctree-l1"><a class="reference internal" href="../sparse.html">torch.sparse</a></li>
<li class="toctree-l1"><a class="reference internal" href="../storage.html">torch.Storage</a></li>
<li class="toctree-l1"><a class="reference internal" href="../testing.html">torch.testing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../utils.html">torch.utils</a></li>
<li class="toctree-l1"><a class="reference internal" href="../benchmark_utils.html">torch.utils.benchmark</a></li>
<li class="toctree-l1"><a class="reference internal" href="../bottleneck.html">torch.utils.bottleneck</a></li>
<li class="toctree-l1"><a class="reference internal" href="../checkpoint.html">torch.utils.checkpoint</a></li>
<li class="toctree-l1"><a class="reference internal" href="../cpp_extension.html">torch.utils.cpp_extension</a></li>
<li class="toctree-l1"><a class="reference internal" href="../data.html">torch.utils.data</a></li>
<li class="toctree-l1"><a class="reference internal" href="../deterministic.html">torch.utils.deterministic</a></li>
<li class="toctree-l1"><a class="reference internal" href="../jit_utils.html">torch.utils.jit</a></li>
<li class="toctree-l1"><a class="reference internal" href="../dlpack.html">torch.utils.dlpack</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mobile_optimizer.html">torch.utils.mobile_optimizer</a></li>
<li class="toctree-l1"><a class="reference internal" href="../model_zoo.html">torch.utils.model_zoo</a></li>
<li class="toctree-l1"><a class="reference internal" href="../tensorboard.html">torch.utils.tensorboard</a></li>
<li class="toctree-l1"><a class="reference internal" href="../module_tracker.html">torch.utils.module_tracker</a></li>
<li class="toctree-l1"><a class="reference internal" href="../type_info.html">Type Info</a></li>
<li class="toctree-l1"><a class="reference internal" href="../named_tensor.html">Named Tensors</a></li>
<li class="toctree-l1"><a class="reference internal" href="../name_inference.html">Named Tensors operator coverage</a></li>
<li class="toctree-l1"><a class="reference internal" href="../config_mod.html">torch.__config__</a></li>
<li class="toctree-l1"><a class="reference internal" href="../future_mod.html">torch.__future__</a></li>
<li class="toctree-l1"><a class="reference internal" href="../logging.html">torch._logging</a></li>
<li class="toctree-l1"><a class="reference internal" href="../torch_environment_variables.html">Torch Environment Variables</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Libraries</span></p>
<ul>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/audio/stable">torchaudio</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/data">TorchData</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/torchrec">TorchRec</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/serve">TorchServe</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/text/stable">torchtext</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/vision/stable">torchvision</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/xla/">PyTorch on XLA Devices</a></li>
<li class="toctree-l1"><a class="reference external" href="https://pytorch.org/ao">torchao</a></li>
</ul>
</div>
</div>
</nav>
<div class="pytorch-container">
<div class="pytorch-page-level-bar" id="pytorch-page-level-bar">
<div class="pytorch-breadcrumbs-wrapper">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="pytorch-breadcrumbs">
<li>
<a href="../index.html">
Docs
</a> >
</li>
<li>Autograd mechanics</li>
<li class="pytorch-breadcrumbs-aside">
<a href="../_sources/notes/autograd.rst.txt" rel="nofollow"><img src="../_static/images/view-page-source-icon.svg"></a>
</li>
</ul>
</div>
</div>
<div class="pytorch-shortcuts-wrapper" id="pytorch-shortcuts-wrapper">
Shortcuts
</div>
</div>
<section data-toggle="wy-nav-shift" id="pytorch-content-wrap" class="pytorch-content-wrap">
<div class="pytorch-content-left">
<!-- Google Tag Manager (noscript) -->
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-T8XT4PS"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) -->
<div class="rst-content">
<div role="main" class="main-content" itemscope="itemscope" itemtype="http://schema.org/Article">
<article itemprop="articleBody" id="pytorch-article" class="pytorch-article">
<div class="section" id="autograd-mechanics">
<span id="id1"></span><h1>Autograd mechanics<a class="headerlink" href="#autograd-mechanics" title="Permalink to this heading">¶</a></h1>
<p>This note will present an overview of how autograd works and records the
operations. It’s not strictly necessary to understand all this, but we recommend
getting familiar with it, as it will help you write more efficient, cleaner
programs, and can aid you in debugging.</p>
<div class="section" id="how-autograd-encodes-the-history">
<span id="how-autograd-encodes-history"></span><h2>How autograd encodes the history<a class="headerlink" href="#how-autograd-encodes-the-history" title="Permalink to this heading">¶</a></h2>
<p>Autograd is a reverse automatic differentiation system. Conceptually,
autograd records a graph recording all of the operations that created
the data as you execute operations, giving you a directed acyclic graph
whose leaves are the input tensors and roots are the output tensors.
By tracing this graph from roots to leaves, you can automatically
compute the gradients using the chain rule.</p>
<p>Internally, autograd represents this graph as a graph of
<code class="xref py py-class docutils literal notranslate"><span class="pre">Function</span></code> objects (really expressions), which can be
<code class="xref py py-meth docutils literal notranslate"><span class="pre">apply()</span></code> ed to compute the result of
evaluating the graph. When computing the forward pass, autograd
simultaneously performs the requested computations and builds up a graph
representing the function that computes the gradient (the <code class="docutils literal notranslate"><span class="pre">.grad_fn</span></code>
attribute of each <a class="reference internal" href="../tensors.html#torch.Tensor" title="torch.Tensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">torch.Tensor</span></code></a> is an entry point into this graph).
When the forward pass is completed, we evaluate this graph in the
backwards pass to compute the gradients.</p>
<p>An important thing to note is that the graph is recreated from scratch at every
iteration, and this is exactly what allows for using arbitrary Python control
flow statements, that can change the overall shape and size of the graph at
every iteration. You don’t have to encode all possible paths before you
launch the training - what you run is what you differentiate.</p>
<div class="section" id="saved-tensors">
<span id="saved-tensors-doc"></span><h3>Saved tensors<a class="headerlink" href="#saved-tensors" title="Permalink to this heading">¶</a></h3>
<p>Some operations need intermediary results to be saved during the forward pass
in order to execute the backward pass. For example, the function
<span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>↦</mo><msup><mi>x</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">x\mapsto x^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.522em;vertical-align:-0.011em;"></span><span class="mord mathnormal">x</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">↦</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span> saves the input <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">x</span></span></span></span></span> to compute the gradient.</p>
<p>When defining a custom Python <a class="reference internal" href="../autograd.html#torch.autograd.Function" title="torch.autograd.Function"><code class="xref py py-class docutils literal notranslate"><span class="pre">Function</span></code></a>, you can use
<code class="xref py py-func docutils literal notranslate"><span class="pre">save_for_backward()</span></code> to save
tensors during the forward pass and
<code class="xref py py-attr docutils literal notranslate"><span class="pre">saved_tensors</span></code> to retrieve them
during the backward pass. See <a class="reference internal" href="extending.html"><span class="doc">Extending PyTorch</span></a> for more information.</p>
<p>For operations that PyTorch defines (e.g. <a class="reference internal" href="../generated/torch.pow.html#torch.pow" title="torch.pow"><code class="xref py py-func docutils literal notranslate"><span class="pre">torch.pow()</span></code></a>), tensors are
automatically saved as needed. You can explore (for educational or debugging
purposes) which tensors are saved by a certain <code class="docutils literal notranslate"><span class="pre">grad_fn</span></code> by looking for its
attributes starting with the prefix <code class="docutils literal notranslate"><span class="pre">_saved</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">requires_grad</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">equal</span><span class="p">(</span><span class="n">y</span><span class="o">.</span><span class="n">grad_fn</span><span class="o">.</span><span class="n">_saved_self</span><span class="p">))</span> <span class="c1"># True</span>
<span class="nb">print</span><span class="p">(</span><span class="n">x</span> <span class="ow">is</span> <span class="n">y</span><span class="o">.</span><span class="n">grad_fn</span><span class="o">.</span><span class="n">_saved_self</span><span class="p">)</span> <span class="c1"># True</span>
</pre></div>
</div>
<p>In the previous code, <code class="docutils literal notranslate"><span class="pre">y.grad_fn._saved_self</span></code> refers to the same Tensor object as <cite>x</cite>.
But that may not always be the case. For instance:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">requires_grad</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">exp</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">y</span><span class="o">.</span><span class="n">equal</span><span class="p">(</span><span class="n">y</span><span class="o">.</span><span class="n">grad_fn</span><span class="o">.</span><span class="n">_saved_result</span><span class="p">))</span> <span class="c1"># True</span>
<span class="nb">print</span><span class="p">(</span><span class="n">y</span> <span class="ow">is</span> <span class="n">y</span><span class="o">.</span><span class="n">grad_fn</span><span class="o">.</span><span class="n">_saved_result</span><span class="p">)</span> <span class="c1"># False</span>
</pre></div>
</div>
<p>Under the hood, to prevent reference cycles, PyTorch has <em>packed</em> the tensor
upon saving and <em>unpacked</em> it into a different tensor for reading. Here, the
tensor you get from accessing <code class="docutils literal notranslate"><span class="pre">y.grad_fn._saved_result</span></code> is a different tensor
object than <code class="docutils literal notranslate"><span class="pre">y</span></code> (but they still share the same storage).</p>
<p>Whether a tensor will be packed into a different tensor object depends on
whether it is an output of its own <cite>grad_fn</cite>, which is an implementation detail
subject to change and that users should not rely on.</p>
<p>You can control how PyTorch does packing / unpacking with <a class="reference internal" href="#saved-tensors-hooks-doc"><span class="std std-ref">Hooks for saved tensors</span></a>.</p>
</div>
</div>
<div class="section" id="gradients-for-non-differentiable-functions">
<span id="non-differentiable-func-grad"></span><h2>Gradients for non-differentiable functions<a class="headerlink" href="#gradients-for-non-differentiable-functions" title="Permalink to this heading">¶</a></h2>
<p>The gradient computation using Automatic Differentiation is only valid when each elementary function being used is differentiable.
Unfortunately many of the functions we use in practice do not have this property (<code class="docutils literal notranslate"><span class="pre">relu</span></code> or <code class="docutils literal notranslate"><span class="pre">sqrt</span></code> at <code class="docutils literal notranslate"><span class="pre">0</span></code>, for example).
To try and reduce the impact of functions that are non-differentiable, we define the gradients of the elementary operations by applying the following rules in order:</p>
<ol class="arabic simple">
<li><p>If the function is differentiable and thus a gradient exists at the current point, use it.</p></li>
<li><p>If the function is convex (at least locally), use the sub-gradient of minimum norm (it is the steepest descent direction).</p></li>
<li><p>If the function is concave (at least locally), use the super-gradient of minimum norm (consider <cite>-f(x)</cite> and apply the previous point).</p></li>
<li><p>If the function is defined, define the gradient at the current point by continuity (note that <code class="docutils literal notranslate"><span class="pre">inf</span></code> is possible here, for example for <code class="docutils literal notranslate"><span class="pre">sqrt(0)</span></code>). If multiple values are possible, pick one arbitrarily.</p></li>
<li><p>If the function is not defined (<code class="docutils literal notranslate"><span class="pre">sqrt(-1)</span></code>, <code class="docutils literal notranslate"><span class="pre">log(-1)</span></code> or most functions when the input is <code class="docutils literal notranslate"><span class="pre">NaN</span></code>, for example) then the value used as the gradient is arbitrary (we might also raise an error but that is not guaranteed). Most functions will use <code class="docutils literal notranslate"><span class="pre">NaN</span></code> as the gradient, but for performance reasons, some functions will use other values (<code class="docutils literal notranslate"><span class="pre">log(-1)</span></code>, for example).</p></li>
<li><p>If the function is not a deterministic mapping (i.e. it is not a <a class="reference external" href="https://en.wikipedia.org/wiki/Function_(mathematics)">mathematical function</a>), it will be marked as non-differentiable. This will make it error out in the backward if used on tensors that require grad outside of a <code class="docutils literal notranslate"><span class="pre">no_grad</span></code> environment.</p></li>
</ol>
</div>
<div class="section" id="locally-disabling-gradient-computation">
<span id="locally-disable-grad-doc"></span><h2>Locally disabling gradient computation<a class="headerlink" href="#locally-disabling-gradient-computation" title="Permalink to this heading">¶</a></h2>
<p>There are several mechanisms available from Python to locally disable gradient
computation:</p>
<p>To disable gradients across entire blocks of code, there are context managers
like no-grad mode and inference mode.
For more fine-grained exclusion of subgraphs from gradient computation,
there is setting the <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> field of a tensor.</p>
<p>Below, in addition to discussing the mechanisms above, we also describe
evaluation mode (<code class="xref py py-meth docutils literal notranslate"><span class="pre">nn.Module.eval()</span></code>), a method that is not used
to disable gradient computation but, because of its name, is often mixed up with the three.</p>
<div class="section" id="setting-requires-grad">
<h3>Setting <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code><a class="headerlink" href="#setting-requires-grad" title="Permalink to this heading">¶</a></h3>
<p><code class="xref py py-attr docutils literal notranslate"><span class="pre">requires_grad</span></code> is a flag, defaulting to false <em>unless wrapped
in a</em> <code class="docutils literal notranslate"><span class="pre">nn.Parameter</span></code>, that allows for fine-grained exclusion of
subgraphs from gradient computation. It takes effect in both the
forward and backward passes:</p>
<p>During the forward pass, an operation is only recorded in the backward graph if
at least one of its input tensors require grad.
During the backward pass (<code class="docutils literal notranslate"><span class="pre">.backward()</span></code>), only leaf tensors with
<code class="docutils literal notranslate"><span class="pre">requires_grad=True</span></code> will have gradients accumulated into their <code class="docutils literal notranslate"><span class="pre">.grad</span></code>
fields.</p>
<p>It is important to note that even though every tensor has this flag,
<em>setting</em> it only makes sense for leaf tensors (tensors that do not have a
<code class="docutils literal notranslate"><span class="pre">grad_fn</span></code>, e.g., a <code class="docutils literal notranslate"><span class="pre">nn.Module</span></code>’s parameters).
Non-leaf tensors (tensors that do have <code class="docutils literal notranslate"><span class="pre">grad_fn</span></code>) are tensors that have a
backward graph associated with them. Thus their gradients will be needed
as an intermediary result to compute the gradient for a leaf tensor that
requires grad. From this definition, it is clear that all non-leaf tensors
will automatically have <code class="docutils literal notranslate"><span class="pre">require_grad=True</span></code>.</p>
<p>Setting <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> should be the main way you control which parts
of the model are part of the gradient computation, for example, if you need to
freeze parts of your pretrained model during model fine-tuning.</p>
<p>To freeze parts of your model, simply apply <code class="docutils literal notranslate"><span class="pre">.requires_grad_(False)</span></code> to
the parameters that you don’t want updated. And as described above,
since computations that use these parameters as inputs would not be recorded in
the forward pass, they won’t have their <code class="docutils literal notranslate"><span class="pre">.grad</span></code> fields updated in the backward
pass because they won’t be part of the backward graph in the first place, as
desired.</p>
<p>Because this is such a common pattern, <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> can also be set at
the module level with <code class="xref py py-meth docutils literal notranslate"><span class="pre">nn.Module.requires_grad_()</span></code>.
When applied to a module, <code class="docutils literal notranslate"><span class="pre">.requires_grad_()</span></code> takes effect on all
of the module’s parameters (which have <code class="docutils literal notranslate"><span class="pre">requires_grad=True</span></code> by default).</p>
</div>
<div class="section" id="grad-modes">
<h3>Grad Modes<a class="headerlink" href="#grad-modes" title="Permalink to this heading">¶</a></h3>
<p>Apart from setting <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> there are also three grad modes that can
be selected from Python that can affect how computations in PyTorch are
processed by autograd internally: default mode (grad mode), no-grad mode,
and inference mode, all of which can be togglable via context managers and
decorators.</p>
<table class="colwidths-given docutils colwidths-auto align-default">
<colgroup>
<col style="width: 20%" />
<col style="width: 20%" />
<col style="width: 20%" />
<col style="width: 20%" />
<col style="width: 20%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Mode</p></th>
<th class="head"><p>Excludes operations from being recorded in backward graph</p></th>
<th class="head"><p>Skips additional autograd tracking overhead</p></th>
<th class="head"><p>Tensors created while the mode is enabled can be used in grad-mode later</p></th>
<th class="head"><p>Examples</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>default</p></td>
<td></td>
<td></td>
<td><p>✓</p></td>
<td><p>Forward pass</p></td>
</tr>
<tr class="row-odd"><td><p>no-grad</p></td>
<td><p>✓</p></td>
<td></td>
<td><p>✓</p></td>
<td><p>Optimizer updates</p></td>
</tr>
<tr class="row-even"><td><p>inference</p></td>
<td><p>✓</p></td>
<td><p>✓</p></td>
<td></td>
<td><p>Data processing, model evaluation</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="default-mode-grad-mode">
<h3>Default Mode (Grad Mode)<a class="headerlink" href="#default-mode-grad-mode" title="Permalink to this heading">¶</a></h3>
<p>The “default mode” is the mode we are implicitly in when no other modes like
no-grad and inference mode are enabled. To be contrasted with
“no-grad mode” the default mode is also sometimes called “grad mode”.</p>
<p>The most important thing to know about the default mode is that it is the only
mode in which <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> takes effect. <code class="docutils literal notranslate"><span class="pre">requires_grad</span></code> is always overridden
to be <code class="docutils literal notranslate"><span class="pre">False</span></code> in both the two other modes.</p>
</div>
<div class="section" id="no-grad-mode">
<h3>No-grad Mode<a class="headerlink" href="#no-grad-mode" title="Permalink to this heading">¶</a></h3>
<p>Computations in no-grad mode behave as if none of the inputs require grad.
In other words, computations in no-grad mode are never recorded in the backward graph
even if there are inputs that have <code class="docutils literal notranslate"><span class="pre">require_grad=True</span></code>.</p>
<p>Enable no-grad mode when you need to perform operations that should not be
recorded by autograd, but you’d still like to use the outputs of these
computations in grad mode later. This context manager makes it convenient to
disable gradients for a block of code or function without
having to temporarily set tensors to have <code class="docutils literal notranslate"><span class="pre">requires_grad=False</span></code>, and then
back to <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
<p>For example, no-grad mode might be useful when writing an optimizer: when
performing the training update you’d like to update parameters
in-place without the update being recorded by autograd.
You also intend to use the updated parameters for computations in
grad mode in the next forward pass.</p>
<p>The implementations in <a class="reference internal" href="../nn.init.html#nn-init-doc"><span class="std std-ref">torch.nn.init</span></a> also
rely on no-grad mode when initializing the parameters as to avoid
autograd tracking when updating the initialized parameters in-place.</p>
</div>
<div class="section" id="inference-mode">
<h3>Inference Mode<a class="headerlink" href="#inference-mode" title="Permalink to this heading">¶</a></h3>
<p>Inference mode is the extreme version of no-grad mode. Just like in no-grad
mode, computations in inference mode are not recorded in the backward graph, but
enabling inference mode will allow PyTorch to speed up your model even more.
This better runtime comes with a drawback: tensors created in inference mode
will not be able to be used in computations to be recorded by autograd after
exiting inference mode.</p>
<p>Enable inference mode when you are performing computations that do not have
interactions with autograd, AND you don’t plan on using the tensors created
in inference mode in any computation that is to be recorded by autograd later.</p>
<p>It is recommended that you try out inference mode in the parts of your code
that do not require autograd tracking (e.g., data processing and model evaluation).
If it works out of the box
for your use case it’s a free performance win. If you run into errors after
enabling inference mode, check that you are not using tensors created in
inference mode in computations that are recorded by autograd after exiting inference
mode. If you cannot avoid such use in your case, you can always switch back
to no-grad mode.</p>
<p>For details on inference mode please see
<a class="reference external" href="https://pytorch.org/cppdocs/notes/inference_mode.html">Inference Mode</a>.</p>
<p>For implementation details of inference mode see
<a class="reference external" href="https://github.com/pytorch/rfcs/pull/17">RFC-0011-InferenceMode</a>.</p>
</div>
<div class="section" id="evaluation-mode-nn-module-eval">
<h3>Evaluation Mode (<code class="docutils literal notranslate"><span class="pre">nn.Module.eval()</span></code>)<a class="headerlink" href="#evaluation-mode-nn-module-eval" title="Permalink to this heading">¶</a></h3>
<p>Evaluation mode is not a mechanism to locally disable gradient computation.
It is included here anyway because it is sometimes confused to be such a mechanism.</p>
<p>Functionally, <code class="docutils literal notranslate"><span class="pre">module.eval()</span></code> (or equivalently <code class="docutils literal notranslate"><span class="pre">module.train(False)</span></code>) are completely
orthogonal to no-grad mode and inference mode. How <code class="docutils literal notranslate"><span class="pre">model.eval()</span></code> affects
your model depends entirely on the specific modules used in your model and
whether they define any training-mode specific behavior.</p>
<p>You are responsible for calling <code class="docutils literal notranslate"><span class="pre">model.eval()</span></code> and <code class="docutils literal notranslate"><span class="pre">model.train()</span></code> if your
model relies on modules such as <a class="reference internal" href="../generated/torch.nn.Dropout.html#torch.nn.Dropout" title="torch.nn.Dropout"><code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.Dropout</span></code></a> and
<a class="reference internal" href="../generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d" title="torch.nn.BatchNorm2d"><code class="xref py py-class docutils literal notranslate"><span class="pre">torch.nn.BatchNorm2d</span></code></a> that may behave
differently depending on training mode, for example, to avoid updating your
BatchNorm running statistics on validation data.</p>
<p>It is recommended that you always use <code class="docutils literal notranslate"><span class="pre">model.train()</span></code> when
training and <code class="docutils literal notranslate"><span class="pre">model.eval()</span></code> when evaluating your model (validation/testing) even
if you aren’t sure your model has training-mode specific behavior, because a
module you are using might be updated to behave differently in training and
eval modes.</p>
</div>
</div>
<div class="section" id="in-place-operations-with-autograd">
<h2>In-place operations with autograd<a class="headerlink" href="#in-place-operations-with-autograd" title="Permalink to this heading">¶</a></h2>
<p>Supporting in-place operations in autograd is a hard matter, and we discourage
their use in most cases. Autograd’s aggressive buffer freeing and reuse makes
it very efficient and there are very few occasions when in-place operations
lower memory usage by any significant amount. Unless you’re operating
under heavy memory pressure, you might never need to use them.</p>
<p>There are two main reasons that limit the applicability of in-place operations:</p>
<ol class="arabic simple">
<li><p>In-place operations can potentially overwrite values required to compute
gradients.</p></li>
<li><p>Every in-place operation requires the implementation to rewrite the
computational graph. Out-of-place versions simply allocate new objects and
keep references to the old graph, while in-place operations, require
changing the creator of all inputs to the <code class="xref py py-class docutils literal notranslate"><span class="pre">Function</span></code> representing
this operation. This can be tricky, especially if there are many Tensors
that reference the same storage (e.g. created by indexing or transposing),
and in-place functions will raise an error if the storage of
modified inputs is referenced by any other <code class="xref py py-class docutils literal notranslate"><span class="pre">Tensor</span></code>.</p></li>
</ol>
<div class="section" id="in-place-correctness-checks">
<h3>In-place correctness checks<a class="headerlink" href="#in-place-correctness-checks" title="Permalink to this heading">¶</a></h3>
<p>Every tensor keeps a version counter, that is incremented every time it is
marked dirty in any operation. When a Function saves any tensors for backward,
a version counter of their containing Tensor is saved as well. Once you access
<code class="docutils literal notranslate"><span class="pre">self.saved_tensors</span></code> it is checked, and if it is greater than the saved value
an error is raised. This ensures that if you’re using in-place
functions and not seeing any errors, you can be sure that the computed
gradients are correct.</p>
</div>
</div>
<div class="section" id="multithreaded-autograd">
<h2>Multithreaded Autograd<a class="headerlink" href="#multithreaded-autograd" title="Permalink to this heading">¶</a></h2>
<p>The autograd engine is responsible for running all the backward operations
necessary to compute the backward pass. This section will describe all the details
that can help you make the best use of it in a multithreaded environment. (This is
relevant only for PyTorch 1.6+ as the behavior in previous version was different.)</p>
<p>User could train their model with multithreading code (e.g. Hogwild training), and
does not block on the concurrent backward computations, example code could be:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># Define a train function to be used in different threads</span>
<span class="k">def</span> <span class="nf">train_fn</span><span class="p">():</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="n">requires_grad</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="c1"># forward</span>
<span class="n">y</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">3</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">4</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.5</span>
<span class="c1"># backward</span>
<span class="n">y</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span>
<span class="c1"># potential optimizer update</span>
<span class="c1"># User write their own threading code to drive the train_fn</span>
<span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">train_fn</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">())</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">threads</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>Note that some behaviors that user should be aware of:</p>
<div class="section" id="concurrency-on-cpu">
<h3>Concurrency on CPU<a class="headerlink" href="#concurrency-on-cpu" title="Permalink to this heading">¶</a></h3>
<p>When you run <code class="docutils literal notranslate"><span class="pre">backward()</span></code> or <code class="docutils literal notranslate"><span class="pre">grad()</span></code> via python or C++ API in multiple
threads on CPU, you are expecting to see extra concurrency instead of
serializing all the backward calls in a specific order during execution
(behavior before PyTorch 1.6).</p>
</div>
<div class="section" id="non-determinism">
<h3>Non-determinism<a class="headerlink" href="#non-determinism" title="Permalink to this heading">¶</a></h3>
<p>If you are calling <code class="docutils literal notranslate"><span class="pre">backward()</span></code> from multiple threads concurrently and have
shared inputs (i.e. Hogwild CPU training), then non-determinism should be expected.
This can occur because parameters are automatically shared across threads,
as such, multiple threads may access and try to accumulate the same <code class="docutils literal notranslate"><span class="pre">.grad</span></code>
attribute during gradient accumulation. This is technically not safe, and
it might result in race condition and the result might be invalid to use.</p>
<p>Users developing multithreaded models featuring shared parameters should have the
threading model in mind and should understand the issues described above.</p>
<p>The functional API <a class="reference internal" href="../generated/torch.autograd.grad.html#torch.autograd.grad" title="torch.autograd.grad"><code class="xref py py-func docutils literal notranslate"><span class="pre">torch.autograd.grad()</span></code></a> may be used to calculate the
gradients instead of <code class="docutils literal notranslate"><span class="pre">backward()</span></code> to avoid non-determinism.</p>
</div>
<div class="section" id="graph-retaining">
<h3>Graph retaining<a class="headerlink" href="#graph-retaining" title="Permalink to this heading">¶</a></h3>
<p>If part of the autograd graph is shared between threads, i.e. run first
part of forward single thread, then run second part in multiple threads,
then the first part of graph is shared. In this case different threads
execute <code class="docutils literal notranslate"><span class="pre">grad()</span></code> or <code class="docutils literal notranslate"><span class="pre">backward()</span></code> on the same graph might have issue of
destroying the graph on the fly of one thread, and the other thread will
crash in this case. Autograd will error out to the user similar to what call
<code class="docutils literal notranslate"><span class="pre">backward()</span></code> twice with out <code class="docutils literal notranslate"><span class="pre">retain_graph=True</span></code>, and let the user know
they should use <code class="docutils literal notranslate"><span class="pre">retain_graph=True</span></code>.</p>
</div>
<div class="section" id="thread-safety-on-autograd-node">
<h3>Thread Safety on Autograd Node<a class="headerlink" href="#thread-safety-on-autograd-node" title="Permalink to this heading">¶</a></h3>
<p>Since Autograd allows the caller thread to drive its backward execution for
potential parallelism, it’s important that we ensure thread safety on CPU with
parallel <code class="docutils literal notranslate"><span class="pre">backward()</span></code> calls that share part/whole of the GraphTask.</p>
<p>Custom Python <code class="docutils literal notranslate"><span class="pre">autograd.Function</span></code>s are automatically thread safe because of GIL.
For built-in C++ Autograd Nodes (e.g. AccumulateGrad, CopySlices) and custom
<code class="docutils literal notranslate"><span class="pre">autograd::Function</span></code>s, the Autograd Engine uses thread mutex locking to ensure
thread safety on autograd Nodes that might have state write/read.</p>
</div>
<div class="section" id="no-thread-safety-on-c-hooks">
<h3>No thread safety on C++ hooks<a class="headerlink" href="#no-thread-safety-on-c-hooks" title="Permalink to this heading">¶</a></h3>
<p>Autograd relies on the user to write thread safe C++ hooks. If you want the hook
to be correctly applied in multithreading environment, you will need to write
proper thread locking code to ensure the hooks are thread safe.</p>
</div>
</div>
<div class="section" id="autograd-for-complex-numbers">
<span id="complex-autograd-doc"></span><h2>Autograd for Complex Numbers<a class="headerlink" href="#autograd-for-complex-numbers" title="Permalink to this heading">¶</a></h2>
<p>The short version:</p>
<ul class="simple">
<li><p>When you use PyTorch to differentiate any function <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(z)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.04398em;">z</span><span class="mclose">)</span></span></span></span></span> with complex domain and/or codomain,
the gradients are computed under the assumption that the function is a part of a larger real-valued
loss function <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo stretchy="false">(</mo><mi>i</mi><mi>n</mi><mi>p</mi><mi>u</mi><mi>t</mi><mo stretchy="false">)</mo><mo>=</mo><mi>L</mi></mrow><annotation encoding="application/x-tex">g(input)=L</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">g</span><span class="mopen">(</span><span class="mord mathnormal">in</span><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal">L</span></span></span></span></span>. The gradient computed is <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>L</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msup><mi>z</mi><mo>∗</mo></msup></mrow></mfrac></mrow><annotation encoding="application/x-tex">\frac{\partial L}{\partial z^*}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2251em;vertical-align:-0.345em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8801em;"><span style="top:-2.655em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight" style="margin-right:0.05556em;">∂</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em;">z</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6183em;"><span style="top:-2.786em;margin-right:0.0714em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mbin mtight">∗</span></span></span></span></span></span></span></span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.394em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight" style="margin-right:0.05556em;">∂</span><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.345em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span>
(note the conjugation of z), the negative of which is precisely the direction of steepest descent
used in Gradient Descent algorithm. Thus, there is a viable path in making the existing optimizers
work out of the box with complex parameters.</p></li>
<li><p>This convention matches TensorFlow’s convention for complex
differentiation, but is different from JAX (which computes
<span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>L</mi></mrow><mrow><mi mathvariant="normal">∂</mi><mi>z</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">\frac{\partial L}{\partial z}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2251em;vertical-align:-0.345em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8801em;"><span style="top:-2.655em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight" style="margin-right:0.05556em;">∂</span><span class="mord mathnormal mtight" style="margin-right:0.04398em;">z</span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.394em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight" style="margin-right:0.05556em;">∂</span><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.345em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span>).</p></li>
<li><p>If you have a real-to-real function which internally uses complex
operations, the convention here doesn’t matter: you will always get
the same result that you would have gotten if it had been implemented
with only real operations.</p></li>
</ul>
<p>If you are curious about the mathematical details, or want to know how
to define complex derivatives in PyTorch, read on.</p>
<div class="section" id="what-are-complex-derivatives">
<h3>What are complex derivatives?<a class="headerlink" href="#what-are-complex-derivatives" title="Permalink to this heading">¶</a></h3>
<p>The mathematical definition of complex-differentiability takes the
limit definition of a derivative and generalizes it to operate on
complex numbers. Consider a function <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo>:</mo><mi mathvariant="normal">C</mi><mo>→</mo><mi mathvariant="normal">C</mi></mrow><annotation encoding="application/x-tex">f: ℂ → ℂ</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6889em;"></span><span class="mord amsrm">C</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">→</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6889em;"></span><span class="mord amsrm">C</span></span></span></span></span>,</p>
<blockquote>
<div><div class="math">
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo>=</mo><mi>x</mi><mo>+</mo><mi>y</mi><mi>j</mi><mo stretchy="false">)</mo><mo>=</mo><mi>u</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mo>+</mo><mi>v</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mi>j</mi></mrow><annotation encoding="application/x-tex">f(z=x+yj) = u(x, y) + v(x, y)j
</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.04398em;">z</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6667em;vertical-align:-0.0833em;"></span><span class="mord mathnormal">x</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">y</span><span class="mord mathnormal" style="margin-right:0.05724em;">j</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">u</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">y</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">y</span><span class="mclose">)</span><span class="mord mathnormal" style="margin-right:0.05724em;">j</span></span></span></span></span></div></div></blockquote>
<p>where <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span></span> and <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span></span></span></span></span> are two variable real valued functions
and <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.05724em;">j</span></span></span></span></span> is the imaginary unit.</p>
<p>Using the derivative definition, we can write:</p>
<blockquote>
<div><div class="math">
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>f</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mrow><mi>lim</mi><mo></mo></mrow><mrow><mi>h</mi><mo>→</mo><mn>0</mn><mo separator="true">,</mo><mi>h</mi><mo>∈</mo><mi>C</mi></mrow></munder><mfrac><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo>+</mo><mi>h</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><mi>h</mi></mfrac></mrow><annotation encoding="application/x-tex">f'(z) = \lim_{h \to 0, h \in C} \frac{f(z+h) - f(z)}{h}
</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0519em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.04398em;">z</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:2.3152em;vertical-align:-0.8882em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.6944em;"><span style="top:-2.3479em;margin-left:0em;"><span class="pstrut" style="height:3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">h</span><span class="mrel mtight">→</span><span class="mord mtight">0</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight">h</span><span class="mrel mtight">∈</span><span class="mord mathnormal mtight" style="margin-right:0.07153em;">C</span></span></span></span><span style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span><span class="mop">lim</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.8882em;"><span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.427em;"><span style="top:-2.314em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord mathnormal">h</span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.04398em;">z</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mord mathnormal">h</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.04398em;">z</span><span class="mclose">)</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.686em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></div></div></blockquote>
<p>In order for this limit to exist, not only must <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span></span> and <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.03588em;">v</span></span></span></span></span> must be
real differentiable, but <span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span></span></span></span></span> must also satisfy the Cauchy-Riemann <a class="reference external" href="https://en.wikipedia.org/wiki/Cauchy%E2%80%93Riemann_equations">equations</a>. In
other words: the limit computed for real and imaginary steps (<span class="math"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord mathnormal">h</span></span></span></span></span>)
must be equal. This is a more restrictive condition.</p>
<p>The complex differentiable functions are commonly known as holomorphic
functions. They are well behaved, have all the nice properties that
you’ve seen from real differentiable functions, but are practically of no
use in the optimization world. For optimization problems, only real valued objective
functions are used in the research community since complex numbers are not part of any
ordered field and so having complex valued loss does not make much sense.</p>
<p>It also turns out that no interesting real-valued objective fulfill the
Cauchy-Riemann equations. So the theory with holomorphic function cannot be
used for optimization and most people therefore use the Wirtinger calculus.</p>
</div>
<div class="section" id="wirtinger-calculus-comes-into-the-picture">
<h3>Wirtinger Calculus comes into the picture …<a class="headerlink" href="#wirtinger-calculus-comes-into-the-picture" title="Permalink to this heading">¶</a></h3>
<p>So, we have this great theory of complex differentiability and
holomorphic functions, and we can’t use any of it at all, because many
of the commonly used functions are not holomorphic. What’s a poor