The Best GPU for Deep Learning

Critical Considerations for Large-Scale AI

Traditionally, the training phase of the deep learning To significantly reduce training time, you can
pipeline takes the longest to achieve. This is not only a use deep learning GPUs, which enable you to
timeconsuming process, but an expensive one. The most perform AI computing operations in parallel. When
valuable part of a deep learning pipeline is the human assessing GPUs, you need to consider the ability
element. Data scientists often wait for hours or days for to interconnect multiple GPUs, the supporting
training to complete, which hurts their productivity and software available, licensing, data parallelism, GPU
the time to bring new models to market. memory use and performance.

All rights reserved to Run:ai.

No part of this content may be used

1
without express permission of Run:ai. www.run.ai
In this guide, you will learn:
The importance of GPUs in deep learning 3

How to choose the best GPU for deep learning

3

Using consumer GPUs for deep learning 4

Best deep learning GPUs for data centers

5

DGX for deep learning at scale 6

Automated Deep Learning GPU Management With Run:ai 8

All rights reserved to Run:ai.

No part of this content may be used

2
without express permission of Run:ai. www.run.ai
Why Are GPUs Important GPU Factors to Consider
These factors affect the scalability and ease of use of
in Deep Learning? the GPUs you choose:

Ability to Interconnect GPUs

The longest and most resource intensive phase When choosing a GPU, you need to consider which
of most deep learning implementations is the units can be interconnected. Interconnecting GPUs is
training phase. This phase can be accomplished in a directly tied to the scalability of your implementation
reasonable amount of time for models with smaller and the ability to use multi-GPU and distributed
numbers of parameters but as your number increases, training strategies. Typically, consumer GPUs
your training time does as well. This has a dual cost; do not support interconnection (NVlink for GPU
your resources are occupied for longer and your team interconnects within a server, and Infiniband/RoCE for
is left waiting, wasting valuable time. linking GPUs across servers) and NVIDIA has removed
interconnections on GPUs below RTX 2080.
Graphical processing units (GPUs) can reduce these
Supporting Software
costs, enabling you to run models with massive
numbers of parameters quickly and efficiently. This is NVIDIA GPUs are the best supported in terms of
because GPUs enable you to parallelize your training machine learning libraries and integration with
tasks, distributing tasks over clusters of processors common frameworks, such as PyTorch or TensorFlow.
and performing compute operations simultaneously. The NVIDIA CUDA toolkit includes GPU-accelerated
libraries, a C and C++ compiler and runtime, and
GPUs are also optimized to perform target tasks, optimization and debugging tools. It enables you to
finishing computations faster than non-specialized get started right away without worrying about building
hardware. These processors enable you to process custom integrations.
the same tasks faster and free your CPUs for other
tasks. This eliminates bottlenecks created by compute Learn more in our guides about PyTorch GPUs, and
limitations. NVIDIA deep learning GPUs.

Licensing
Another factor to consider is NVIDIA’s guidance
How to Choose the Best regarding the use of certain chips in data centers. As
of a licensing update in 2018, there may be restrictions
GPU for Deep Learning? on use of CUDA software with consumer GPUs in
a data center. This may require organizations to
transition to production-grade GPUs.
Selecting the GPUs for your implementation has
significant budget and performance implications. You Algorithm Factors Affective GPU Use
need to select GPUs that can support your project In our experience helping organizations optimize large-
in the long run and have the ability to scale through scale deep learning workloads, the following are the
integration and clustering. For large-scale projects, this three key factors you should consider when scaling up
means selecting production-grade or data center GPUs. your algorithm across multiple GPUs.

All rights reserved to Run:ai.

No part of this content may be used

3
without express permission of Run:ai. www.run.ai
 Data Parallelism – Consider how much data your NVIDIA Titan V
algorithms need to process. If datasets are going to be
large, invest in GPUs capable of performing multi-GPU The Titan V is a PC GPU that was designed for use
training efficiently. For very large scale datasets, make by scientists and researchers. It is based on NVIDIA’s
sure that servers can communicate quickly with each
other and with storage components, using technology
Volta technology and includes Tensor Cores. The Titan
like Infiniband/RoCE, to enable efficient distributed V comes in Standard and CEO Editions.
training.

The Standard edition provides 12GB memory, 110

Memory Use – Are you going to deal with large data teraflops performance, a 4.5MB L2 cache, and 3,072-
inputs to model? For example, models processing bit memory bus. The CEO edition provides 32GB
medical images or long videos have very large training
sets, so you’d want to invest in GPUs with relatively memory and 125 teraflops performance, 6MB cache,
large memory. By contrast, tabular data such as text and 4,096-bit memory bus. The latter edition also uses
inputs for NLP models are typically small, and you can
the same 8-Hi HBM2 memory stacks that are used in
make do with less GPU memory.
the 32GB Tesla units.

Performance of the GPU – Consider if you’re going to NVIDIA Titan RTX

use GPUs for debugging and development. In this case
you won’t need the most powerful GPUs. For tuning
The Titan RTX is a PC GPU based on NVIDIA’s Turing
models in long runs, you need strong GPUs to GPU architecture that is designed for creative and
accelerate training time, to avoid waiting hours or days machine learning workloads. It includes Tensor Core
for models to run.
and RT Core technologies to enable ray tracing and
accelerated AI.

Using Consumer GPUs Each Titan RTX provides 130 teraflops, 24GB GDDR6
for Deep Learning memory, 6MB cache, and 11 GigaRays per second.
This is due to 72 Turing RT Cores and 576 multi
precision Turing Tensor Cores.
While consumer GPUs are not suitable for large-
scale deep learning projects, these processors NVIDIA GeForce RTX 2080 Ti
can provide a good entry point for deep learning. The GeForce RTX 2080 Ti is a PC GPU designed
Consumer GPUs can also be a cheaper supplement for enthusiasts. It is based on the TU102 graphics
for less complex tasks, such as model planning or processor. Each GeForce RTX 2080 Ti provides 11GB
low-level testing. However, as you scale up, you’ll of memory, a 352-bit memory bus, a 6MB cache, and
want to consider data center grade GPUs and roughly 120 teraflops of performance.
high-end deep learning systems like NVIDIA’s DGX
series (learn more in the following sections).
In particular, the Titan V has been shown to provide
performance similar to datacenter-grade GPUs
when it comes to Word RNNs. Additionally, its
performance for CNNs is only slightly below higher
tier options. The Titan RTX and RTX 2080 Ti aren’t
far behind.

All rights reserved to Run:ai.

No part of this content may be used

4
without express permission of Run:ai. www.run.ai