Julia High Performance: Optimizations, distributed computing, multithreading, and GPU programming with Julia 1.0 and beyond, 2nd Edition

Julia High Performance

Second Edition

Optimizations, distributed computing, multithreading, and GPU programming with Julia 1.0 and beyond

Avik Sengupta

BIRMINGHAM - MUMBAI

Julia High Performance Second Edition

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Aaron Lazar

Acquisition Editor: Denim Pinto

Content Development Editor: Akshita Billava

Technical Editor: Neha Pande

Copy Editor: Safis Editing

Project Coordinator: Vaidehi Sawant

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Coordinator: Jisha Chirayil

First published: April 2016

Second edition: June 2019

Production reference: 1070619

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78829-811-7

www.packtpub.com

To Vaishali and Ahan—for whom I do everything that I do. 

– Avik Sengupta

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Foreword

Those who are new to Julia often ask me what makes it so special, or how it achieves such high performance. Having to think on my feet in order to answer this question has proven challenging. This was the case, at least, until Avik Sengupta came along with the Julia High Performance book. Now all I have to do is tell the enquirers to read the book as it consists of just the right combination of details to answer their questions.

In an easy-to-read concise set of chapters - most of which contain words like performance and fast - Avik takes you through examples that you can run yourself in order to see how fast and easy  Julia is to use.

There are computer science words that have become Julia words. It is a pleasure to learn these in a manner that is easy to follow. Just a few examples of this are JIT, Multiple Dispatch, type system, generated functions, CUDA, and SIMD.

After learning about Julia's design, you will learn to measure performance. From there, you will appreciate Julia's type system. You will then master using arrays along with making fast function calls and fast numbers. Finally, you will learn to write parallel Julia programs.

With Julia High Performance, you'll pick up the key essentials of Julia in no time. You can then join the friendly, fast growing, online community of Julia programmers.

Welcome to the world of Julia! Read this book and you will soon join us in loving the Julia language.

Alan Edelman

Professor of Applied Mathematics

Computer Science & AI Labs Member

MIT

Co-creator, The Julia Language

Contributors

About the author

Avik Sengupta is the vice president of engineering at Julia Computing, a contributor to open source Julia, and the maintainer of several Julia packages. Avik is the co-founder of two start-ups in the financial services and AI sectors, and is a creator of large, complex trading systems for the world's leading investment banks. Prior to Julia Computing, Avik was co-founder and CTO at AlgoCircle and at Itellix, director at Lab49, and head of algorithmic solutions at Decimal Point Analytics. Avik earned his MS in computational finance at Carnegie Mellon and MBA Finance at the Indian Institute of Management in Bangalore. 

This book wouldn't exist, and my life would have been very different, if, almost over a decade ago, four people across the world did not imagine that scientific computing needed a new language. Thank you, Jeff, Alan, Viral, and Stefan, for launching a thousand ships. 

My colleagues at Julia Computing have been a source of immense support. Among others, Keno Fischer, Matt Bauman, Kristoffer Carlson, and Tanmay Mohapatra have helped clear my misconceptions, and have provided code and ideas that have seeped into this book. I'm also grateful to my reviewers, who've painstakingly provided detailed feedback on this book. All of them have made this book immeasurably better, but the responsibility for any errors, of course, remains my own. 

And finally, a word of thanks to the entire Julia community. The depth and breadth of knowledge and skill I have encountered are exceptional. Over the past few years, I've learned so much—things that I should have known, and things that I never thought I could know. Thank you for sharing your knowledge so generously.

About the reviewers

Lyndon White is a research engineer at Invenia Labs, living in Broadway Nedlands, Australia. His key research areas include natural language processing and computational linguistics, with a focus on the capturing of sentence meaning. More broadly, his interests include machine learning, data mining, pattern recognition, and artificial intelligence. His overarching goal is to develop methods to allow for better computational processing of vast amounts of data intended for manual (human) processing via media such as books, blogs, and newspapers. He gained a bachelor's degree in engineering (electrical and electronic) and a bachelor's degree in computer and mathematical science from the University of Western Australia in 2014, where he continued his PhD studies.

Zhuo Qingliang (also known as KDr2 online) is presently working at paodingai.com, a start-up FinTech company in China that is dedicated to improving the financial industry by using artificial intelligence technologies. He has over 10 years of experience in Linux, C, C++, Python, Perl, and Java development. He is interested in programming, doing consulting work, and participating in and contributing to the open source community (including the Julia community, of course).

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Julia High Performance Second Edition

Dedication

About Packt

Why subscribe?

Foreword

Contributors

About the author

About the reviewers

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Code in Action

Conventions used

Get in touch

Reviews

Julia is Fast

Julia – fast and dynamic

Designed for speed

JIT and LLVM

Types, type inference, and code specialization

How fast can Julia be?

Summary

Analyzing Performance

Timing Julia functions

The @time macro

Other time macros

The Julia profiler

Using the profiler

ProfileView

Using Juno for profiling

Using TimerOutputs

Analyzing memory allocation

Using the memory allocation tracker

Statistically accurate benchmarking

Using BenchmarkTools.jl

Summary

Types, Type Inference, and Stability

The Julia type system

Using types

Multiple dispatch

Abstract types

Julia's type hierarchy

Composite and immutable types

Type parameters

Type inference

Type-stability

Definitions

Fixing type instability

The performance pitfalls

Identifying type stability

Loop variables

Kernel methods and function barriers

Types in storage locations

Arrays

Composite types

Parametric composite types

Summary

Making Fast Function Calls

Using globals

The trouble with globals

Fixing performance issues with globals

Inlining

Default inlining

Controlling inlining

Disabling inlining

Constant propagation

Using macros for performance

The Julia compilation process

Using macros

Evaluating a polynomial

Horner's method

The Horner macro

Generated functions

Using generated functions

Using generated functions for performance

Using keyword arguments

Summary

Fast Numbers

Numbers in Julia, their layout, and storage

Integers

Integer overflow

BigInt

The floating point

Floating point accuracy

Unsigned integers

Trading performance for accuracy

The @fastmath macro

The K-B-N summation

Subnormal numbers

Subnormal numbers to zero

Summary

Using Arrays

Array internals in Julia

Array representation and storage

Column-wise storage

Adjoints

Array initialization

Bounds checking

Removing the cost of bounds checking

Configuring bound checks at startup

Allocations and in-place operations

Preallocating function output

sizehint!

Mutating functions

Broadcasting

Array views

SIMD parallelization (AVX2, AVX512)

SIMD.jl

Specialized array types

Static arrays

Structs of arrays

Yeppp!

Writing generic library functions with arrays

Summary

Accelerating Code with the GPU

Technical requirements

Getting started with GPUs

CUDA and Julia

CuArrays

Monte Carlo simulation on the GPU

Writing your own kernels

Measuring GPU performance

Performance tips

Scalar iteration

Combining kernels

Processing more data

Deep learning on the GPU

ArrayFire

Summary 

Concurrent Programming with Tasks

Tasks

Using tasks

The task life cycle

task_local_storage

Communicating between tasks

Task iteration

High-performance I/O

Port sharing for high-performance web serving

Summary

Threads

Threads

Measuring CPU cores

Hwloc

Starting threads

The @threads macro

Prefix sum

Thread safety and synchronization primitives

Multithreaded Monte Carlo simulation

Atomics

Synchronization primitives

Threads and GC

Threaded libraries

Over-subscription

The future of threading

Summary

Distributed Computing with Julia

Creating Julia clusters

Starting a cluster

Cluster managers

SSHManager

SLURM

Communication between Julia processes

Programming parallel tasks

The @everywhere macro

The @spawn macro

The @spawnat macro

Parallel for loops

Parallel map

Distributed Monte Carlo

Distributed arrays

Conway's Game of Life

Shared arrays

Parallel prefix sum with shared arrays

Summary

Licences

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

The Julia programming language has brought an innovative new approach to scientific computing, promising a combination of performance and productivity that is not usually available in the current set of languages that is commonly used. In solving the two-language problem, it has seen tremendous growth both in academia and industry. It has been used in domains from robotics, astronomy, and physics, to insurance and trading. It has particular relevance in the area of machine learning, with increasing use for the emerging field of differentiable computing. 

Most new developers are attracted to the language due to its promise of high performance. This book shows you how and why that is possible. We talk about the design choices of the language's creators that allow such a high-performance compiler to be built. We also show you the steps that you, as an application developer, can take to ensure the highest possible performance for your code. We also tell you the ways in which your code can work with the compiler and runtime to fully utilize your hardware to the greatest extent possible. 

Who this book is for

This book is for the beginner and intermediate Julia developer who wants to fully leverage Julia's promise of performance with productivity. We assume you are proficient with one or more programming languages and have some familiarity with Julia's syntax. We do not expect you to be expert Julia programmers yet but assume that you have written small Julia programs, or that you have taken an introductory course on the language.

What this book covers

Chapter 1, Julia is Fast, is your introduction to Julia's unique performance. Julia is a high-performance language, with the possibility to run code that is competitive in performance with code written in C. This chapter explains why Julia code is fast. It also provides context and sets the stage for the rest of the book.

Chapter 2, Analyzing Performance, shows you how to measure the speed of Julia programs and understand where the bottlenecks are. It also shows you how to measure the memory usage of Julia programs and the amount of time spent on garbage collection. 

Chapter 3, Types, Type Inference, and Stability, covers type information. One of the principal ways in which Julia achieves its performance goals is by using type information. This chapter describes how the Julia compiler uses type information to create fast machine code. It describes ways of writing Julia code to provide effective type information to the Julia compiler. 

Chapter 4, Making Fast Function Calls, explores functions. Functions are the primary artifacts for code organization in Julia, with multiple dispatch being the single most important design feature in the language. This chapter shows you how to use these facilities for fast code. 

Chapter 5, Fast Numbers, describes some internals of Julia's number types in relation to performance, and helps you understand the design decisions that were made to achieve that performance. 

Chapter 6, Using Arrays, focuses on arrays. Arrays are one of the most important data structures in scientific programming. This chapter shows you how to get the best performance out of your arrays—how to store them, and how to operate on them. 

Chapter 7, Accelerating Code with the GPU, covers the GPU. In recent years, the general-purpose GPU has turned out to be one of the best ways of running fast parallel computations. Julia provides a unique method for compiling high-level code to the GPU. This chapter shows you how to use the GPU with Julia. 

Chapter 8, Concurrent Programming with Tasks, looks at concurrent programming. Most programs in Julia run on a single thread, on a single processor core. However, certain concurrent primitives make it possible to run parallel, or seemingly parallel, operations, without the full complexities of shared memory multi-threading. In this chapter, we discuss how the concepts of tasks and asynchronous IO help create responsive programs. 

Chapter 9, Threads, moves on to look at how Julia now has new experimental support for shared memory multi-threading. In this chapter, we discuss the implementation details of this mode, and see how this is different from other languages. We see how to speed up our computations using threads, and learn some of the limitations that currently exist in this model.

Chapter 10, Distributed Computing with Julia, recognizes that there comes a time in every large computation's life when living on a single machine is not enough. There is either too much data to fit in the memory of a single machine, or computations need to be finished quicker than can be achieved on all the cores of a single processor. At that stage, computation moves from a single machine to many. Julia comes with advanced distributed computation facilities built in, which we describe in this chapter.

To get the most out of this book

This book has been written to be a practical guide to improving the performance of your Julia code. As such, we encourage you to run the code shown in this book yourself. Running the code and inspecting the output for yourself is the best way to learn the methods suggested here. All the code is available in machine-readable format (see the following for download instructions), so we