|
| 1 | +--- |
| 2 | +layout: blog_detail |
| 3 | +title: "PyTorch 2.7 Release" |
| 4 | +--- |
| 5 | + |
| 6 | +We are excited to announce the release of PyTorch® 2.7 ([release notes](https://github.com/pytorch/pytorch/releases/tag/v2.7.0))! This release features: |
| 7 | + |
| 8 | +* support for the [NVIDIA Blackwell GPU architecture](https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/) and pre-built wheels for [CUDA 12.8](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) across Linux x86 and arm64 architectures. |
| 9 | +* *torch.compile* support for Torch Function Modes which enables users to override any *torch.** operation to implement custom user-defined behavior. |
| 10 | +* Mega Cache which allows users to have end-to-end portable caching for torch; |
| 11 | +* new features for FlexAttention - LLM first token processing, LLM throughput mode optimization and Flex Attention for Inference. |
| 12 | + |
| 13 | +This release is composed of 3262 commits from 457 contributors since PyTorch 2.6. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.7. More information about how to get started with the PyTorch 2-series can be found at our [Getting Started](https://pytorch.org/get-started/pytorch-2.0/) page. |
| 14 | + |
| 15 | + |
| 16 | +<table class="table table-bordered"> |
| 17 | + <tr> |
| 18 | + <td><strong>Beta</strong> |
| 19 | + </td> |
| 20 | + <td><strong>Prototype</strong> |
| 21 | + </td> |
| 22 | + </tr> |
| 23 | + <tr> |
| 24 | + <td>Torch.Compile support for Torch Function Modes |
| 25 | + </td> |
| 26 | + <td>NVIDIA Blackwell Architecture Support |
| 27 | + </td> |
| 28 | + </tr> |
| 29 | + <tr> |
| 30 | + <td>Mega Cache |
| 31 | + </td> |
| 32 | + <td>PyTorch Native Context Parallel |
| 33 | + </td> |
| 34 | + </tr> |
| 35 | + <tr> |
| 36 | + <td> |
| 37 | + </td> |
| 38 | + <td>Enhancing Intel GPU Acceleration |
| 39 | + </td> |
| 40 | + </tr> |
| 41 | + <tr> |
| 42 | + <td> |
| 43 | + </td> |
| 44 | + <td>FlexAttention LLM <span style="text-decoration:underline;">first token processing</span> on x86 CPUs |
| 45 | + </td> |
| 46 | + </tr> |
| 47 | + <tr> |
| 48 | + <td> |
| 49 | + </td> |
| 50 | + <td>FlexAttention LLM <span style="text-decoration:underline;">throughput mode optimization</span> on x86 CPUs |
| 51 | + </td> |
| 52 | + </tr> |
| 53 | + <tr> |
| 54 | + <td> |
| 55 | + </td> |
| 56 | + <td>Foreach Map |
| 57 | + </td> |
| 58 | + </tr> |
| 59 | + <tr> |
| 60 | + <td> |
| 61 | + </td> |
| 62 | + <td>Flex Attention for Inference |
| 63 | + </td> |
| 64 | + </tr> |
| 65 | + <tr> |
| 66 | + <td> |
| 67 | + </td> |
| 68 | + <td>Prologue Fusion Support in Inductor |
| 69 | + </td> |
| 70 | + </tr> |
| 71 | +</table> |
| 72 | + |
| 73 | + |
| 74 | +*To see a full list of public feature submissions click [here](https://docs.google.com/spreadsheets/d/1TzGkWuUMF1yTe88adz1dt2mzbIsZLd3PBasy588VWgk/edit?usp=sharing). |
| 75 | + |
| 76 | + |
| 77 | +## BETA FEATURES |
| 78 | + |
| 79 | + |
| 80 | +### [Beta] Torch.Compile support for Torch Function Modes |
| 81 | + |
| 82 | +This feature enables users to override any *torch.** operation to implement custom user-defined behavior. For example, ops can be rewritten to accommodate a specific backend. This is used in FlexAttention to re-write indexing ops. |
| 83 | + |
| 84 | +See the [tutorial](https://pytorch.org/tutorials/recipes/torch_compile_torch_function_modes.html) for more information. |
| 85 | + |
| 86 | + |
| 87 | +### [Beta] Mega Cache |
| 88 | + |
| 89 | +Mega Cache allows users to have end-to-end portable caching for torch. The intended use case is after compiling and executing a model, the user calls *torch.compiler.save_cache_artifacts()* which will return the compiler artifacts in a portable form. Later, potentially on a different machine, the user may call *torch.compiler.load_cache_artifacts()* with these artifacts to pre-populate the torch.compile caches in order to jump-start their cache. |
| 90 | + |
| 91 | +See the [tutorial](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html#torch-compile-end-to-end-caching-mega-cache) for more information. |
| 92 | + |
| 93 | + |
| 94 | +## PROTOTYPE FEATURES |
| 95 | + |
| 96 | + |
| 97 | +### [Prototype] NVIDIA Blackwell Architecture Support |
| 98 | + |
| 99 | +PyTorch 2.7 introduces support for NVIDIA's new Blackwell GPU architecture and ships pre-built wheels for CUDA 12.8. For more details on CUDA 12.8 see [CUDA Toolkit Release](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html). |
| 100 | + |
| 101 | + |
| 102 | + |
| 103 | +* Core components and libraries including cuDNN, NCCL, and CUTLASS have been upgraded to ensure compatibility with Blackwell platforms. |
| 104 | +* PyTorch 2.7 includes Triton 3.3, which adds support for the Blackwell architecture with torch.compile compatibility. |
| 105 | +* To utilize these new features, install PyTorch with CUDA 12.8 using: *pip install torch==2.7.0 --index-url https://download.pytorch.org/whl/cu128* |
| 106 | + |
| 107 | +More context can also be found [here](https://github.com/pytorch/pytorch/issues/145949). |
| 108 | + |
| 109 | + |
| 110 | +### [Prototype] PyTorch Native Context Parallel |
| 111 | + |
| 112 | +PyTorch Context Parallel API allows users to create a Python context so that every *torch.nn.functional.scaled_dot_product_attention() *call within will run with context parallelism. Currently, PyTorch Context Parallel supports 3 attention backends: 1. Flash attention; 2. Efficient attention; and 3. cuDNN attention. |
| 113 | + |
| 114 | +As an example, this is [used within TorchTitan as the Context Parallel solution for LLM training](https://discuss.pytorch.org/t/distributed-w-torchtitan-breaking-barriers-training-long-context-llms-with-1m-sequence-length-in-pytorch-using-context-parallel/215082). |
| 115 | + |
| 116 | +See [tutorial](https://pytorch.org/tutorials/prototype/context_parallel.html) here. |
| 117 | + |
| 118 | + |
| 119 | +### [Prototype] Enhancing Intel GPU Acceleration |
| 120 | + |
| 121 | +This latest release introduces enhanced performance optimizations for Intel GPU architectures. These improvements accelerate workloads across various Intel GPUs through the following key enhancements: |
| 122 | + |
| 123 | + |
| 124 | + |
| 125 | +* Enable torch.compile on Windows 11 for Intel GPUs, delivering the performance advantages over eager mode as on Linux. |
| 126 | +* Optimize the performance of PyTorch 2 Export Post Training Quantization (PT2E) on Intel GPU to provide a full graph mode quantization pipelines with enhanced computational efficiency. |
| 127 | +* Improve Scaled Dot-Product Attention (SDPA) inference performance with bfloat16 and float16 to accelerate attention-based models on Intel GPUs. |
| 128 | +* Enable AOTInuctor and torch.export on Linux to simplify deployment workflows. |
| 129 | +* Implement more Aten operators to enhance the continuity of operators execution on Intel GPU and increase the performance on Intel GPU in eager mode. |
| 130 | +* Enable profiler on both Windows and Linux to facilitate model performance analysis. |
| 131 | +* Expand the Intel GPUs support to [Intel® Core™ Ultra Series 2 with Intel® Arc™ Graphics](https://www.intel.com/content/www/us/en/products/details/processors/core-ultra.html), and [Intel® Arc™ B-Series graphics](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/desktop/b-series/overview.html) on both Windows and Linux. |
| 132 | + |
| 133 | +For more information regarding Intel GPU support, please refer to [Getting Started Guide](https://pytorch.org/docs/main/notes/get_start_xpu.html). |
| 134 | + |
| 135 | +See also the tutorials [here](https://pytorch.org/tutorials/prototype/inductor_windows.html) and [here](https://pytorch.org/tutorials/prototype/pt2e_quant_xpu_inductor.html). |
| 136 | + |
| 137 | + |
| 138 | +### [Prototype] FlexAttention LLM first token processing on x86 CPUs |
| 139 | + |
| 140 | +FlexAttention x86 CPU support was first introduced in PyTorch 2.6, offering optimized implementations — such as PageAttention, which is critical for LLM inference—via the TorchInductor C++ backend. In PyTorch 2.7, more attention variants for first token processing of LLMs are supported. With this feature, users can have a smoother experience running FlexAttention on x86 CPUs, replacing specific *scaled_dot_product_attention* operators with a unified FlexAttention API, and benefiting from general support and good performance when using torch.compile. |
| 141 | + |
| 142 | + |
| 143 | +### [Prototype] FlexAttention LLM throughput mode optimization |
| 144 | + |
| 145 | +The performance of FlexAttention on x86 CPUs for LLM inference throughput scenarios has been further improved by adopting the new C++ micro-GEMM template ability. This addresses the performance bottlenecks for large batch size scenarios present in PyTorch 2.6. With this enhancement, users can transparently benefit from better performance and a smoother experience when using FlexAttention APIs and torch.compile for LLM throughput serving on x86 CPUs. |
| 146 | + |
| 147 | + |
| 148 | +### [Prototype] Foreach Map |
| 149 | + |
| 150 | +This feature uses torch.compile to allow users to apply any pointwise or user-defined function (e.g. torch.add) to lists of tensors, akin to the existing *torch._foreach_** ops. The main advantage over the existing *torch._foreach_** ops is that any mix of scalars or lists of tensors can be supplied as arguments, and even user-defined python functions can be lifted to apply to lists of tensors. Torch.compile will automatically generate a horizontally fused kernel for optimal performance. |
| 151 | + |
| 152 | +See [tutorial](https://pytorch.org/tutorials/recipes/foreach_map.html) here. |
| 153 | + |
| 154 | + |
| 155 | +### [Prototype] Flex Attention for Inference |
| 156 | + |
| 157 | +In release 2.5.0, [FlexAttention](https://pytorch.org/blog/flexattention/)* torch.nn.attention.flex_attention* was introduced for ML researchers who’d like to customize their attention kernels without writing kernel code. This update introduces a decoding backend optimized for inference, supporting GQA and PagedAttention, along with feature updates including nested jagged tensor support, performance tuning guides and trainable biases support. |
| 158 | + |
| 159 | +### [Prototype] Prologue Fusion Support in Inductor |
| 160 | + |
| 161 | +Prologue fusion optimizes matrix multiplication (matmul) operations by fusing operations that come before the matmul into the matmul kernel itself, improving performance by reducing global memory bandwidth. |
0 commit comments