Understanding Real-World Concurrency Bugs in Go

Tengfei Tu∗ Xiaoyu Liu†

BUPT, Pennsylvania State University Purdue University
[email protected] [email protected]

Linhai Song Yiying Zhang

Pennsylvania State University Purdue University
[email protected] [email protected]

Abstract CCS Concepts • Computing methodologies → Con-

Go is a statically-typed programming language that aims current programming languages; • Software and its en-
to provide a simple, efficient, and safe way to build multi- gineering → Software testing and debugging.
threaded software. Since its creation in 2009, Go has ma- Keywords Go; Concurrency Bug; Bug Study
tured and gained significant adoption in production and
ACM Reference Format:
open-source software. Go advocates for the usage of mes-
Tengfei Tu, Xiaoyu Liu, Linhai Song, and Yiying Zhang. 2019. Un-
sage passing as the means of inter-thread communication derstanding Real-World Concurrency Bugs in Go . In Proceedings
and provides several new concurrency mechanisms and li- of 2019 Architectural Support for Programming Languages and Op-
braries to ease multi-threading programming. It is important erating Systems (ASPLOS’19). ACM, New York, NY, USA, 14 pages.
to understand the implication of these new proposals and the https://doi.org/http://dx.doi.org/10.1145/3297858.3304069
comparison of message passing and shared memory synchro-
nization in terms of program errors, or bugs. Unfortunately, 1 Introduction
as far as we know, there has been no study on Go’s concur- Go [20] is a statically typed language originally developed
rency bugs. by Google in 2009. Over the past few years, it has quickly
In this paper, we perform the first systematic study on gained attraction and is now adopted by many types of soft-
concurrency bugs in real Go programs. We studied six pop- ware in real production. These Go applications range from
ular Go software including Docker, Kubernetes, and gRPC. libraries [19] and high-level software [26] to cloud infrastruc-
We analyzed 171 concurrency bugs in total, with more than ture software like container systems [13, 36] and key-value
half of them caused by non-traditional, Go-specific problems. databases [10, 15].
Apart from root causes of these bugs, we also studied their A major design goal of Go is to improve traditional multi-
fixes, performed experiments to reproduce them, and eval- threaded programming languages and make concurrent pro-
uated them with two publicly-available Go bug detectors. gramming easier and less error-prone. For this purpose, Go
Overall, our study provides a better understanding on Go’s centers its multi-threading design around two principles:
concurrency models and can guide future researchers and 1) making threads (called goroutines) lightweight and easy
practitioners in writing better, more reliable Go software to create and 2) using explicit messaging (called channel)
and in developing debugging and diagnosis tools for Go. to communicate across threads. With these design princi-
ples, Go proposes not only a set of new primitives and new
libraries but also new implementation of existing semantics.
∗ The work was done when Tengfei Tu was a visiting student at Pennsylvania
It is crucial to understand how Go’s new concurrency prim-
State University.
† Xiaoyu Liu contributed equally with Tengfei Tu in this work.
itives and mechanisms impact concurrency bugs, the type of
bugs that is the most difficult to debug and the most widely
studied [40, 43, 45, 57, 61] in traditional multi-threaded pro-
gramming languages. Unfortunately, there has been no prior
Permission to make digital or hard copies of all or part of this work for work in studying Go concurrency bugs. As a result, to date,
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear it is still unclear if these concurrency mechanisms actually
this notice and the full citation on the first page. Copyrights for components make Go easier to program and less error-prone to concur-
of this work owned by others than ACM must be honored. Abstracting with rency bugs than traditional languages.
credit is permitted. To copy otherwise, or republish, to post on servers or to In this paper, we conduct the first empirical study on
redistribute to lists, requires prior specific permission and/or a fee. Request Go concurrency bugs using six open-source, production-
permissions from [email protected].
ASPLOS’19, April 13–17, 2019, Providence, RI, USA
grade Go applications: Docker [13] and Kubernetes [36],
© 2019 Association for Computing Machinery. two datacenter container systems, etcd [15], a distributed
ACM ISBN ISBN 978-1-4503-6240-5/19/04. . . $15.00 key-value store system, gRPC [19], an RPC library, and Cock-
https://doi.org/http://dx.doi.org/10.1145/3297858.3304069 roachDB [10] and BoltDB [6], two database systems.
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

In total, we have studied 171 concurrency bugs in these ap- 1 func finishReq(timeout time.Duration) r ob {
plications. We analyzed the root causes of them, performed 2 - ch := make(chan ob)
3 + ch := make(chan ob, 1)
experiments to reproduce them, and examined their fixing 4 go func() {
patches. Finally, we tested them with two existing Go con- 5 result := fn()
currency bug detectors (the only publicly available ones). 6 ch <- result // block
7 }
Our study focuses on a long-standing and fundamental 8 select {
question in concurrent programming: between message pass- 9 case result = <- ch:
ing [27, 37] and shared memory, which of these inter-thread 10 return result
11 case <- time.After(timeout):
communication mechanisms is less error-prone [2, 11, 48]. 12 return nil
Go is a perfect language to study this question, since it pro- 13 }
vides frameworks for both shared memory and message 14 }
passing. However, it encourages the use of channels over
Figure 1. A blocking bug caused by channel.
shared memory with the belief that explicit message passing
is less error-prone [1, 2, 21].
To understand Go concurrency bugs and the comparison Go proposes to ease the creation of goroutines, the usage
between message passing and shared memory, we propose of buffered vs. unbuffered channels, the non-determinism
to categorize concurrency bugs along two orthogonal dimen- of waiting for multiple channel operations using select,
sions: the cause of bugs and their behavior. Along the cause and the special library time. Although each of these fea-
dimension, we categorize bugs into those that are caused tures were designed to ease multi-threaded programming, in
by misuse of shared memory and those caused by misuse of reality, it is difficult to write correct Go programs with them.
message passing. Along the second dimension, we separate Overall, our study reveals new practices and new issues of
bugs into those that involve (any number of) goroutines that Go concurrent programming, and it sheds light on an answer
cannot proceed (we call them blocking bugs) and those that to the debate of message passing vs. shared memory accesses.
do not involve any blocking (non-blocking bugs). Our findings improve the understanding of Go concurrency
Surprisingly, our study shows that it is as easy to make con- and can provide valuable guidance for future tool design.
currency bugs with message passing as with shared memory, This paper makes the following key contributions.
sometimes even more. For example, around 58% of blocking • We performed the first empirical study of Go concur-
bugs are caused by message passing. In addition to the viola- rency bugs with six real-world, production-grade Go
tion of Go’s channel usage rules (e.g., waiting on a channel applications.
that no one sends data to or close), many concurrency bugs • We made nine high-level key observations of Go con-
are caused by the mixed usage of message passing and other currency bug causes, fixes, and detection. They can
new semantics and new libraries in Go, which can easily be be useful for Go programmers’ references. We further
overlooked but hard to detect. make eight insights into the implications of our study
To demonstrate errors in message passing, we use a block- results to guide future research in the development,
ing bug from Kubernetes in Figure 1. The finishReq func- testing, and bug detection of Go.
tion creates a child goroutine using an anonymous func- • We proposed new methods to categorize concurrency
tion at line 4 to handle a request—a common practice in bugs along two dimensions of bug causes and behav-
Go server programs. The child goroutine executes fn() and iors. This taxonomy methodology helped us to better
sends result back to the parent goroutine through channel compare different concurrency mechanisms and corre-
ch at line 6. The child will block at line 6 until the parent lations of bug causes and fixes. We believe other bug
pulls result from ch at line 9. Meanwhile, the parent will studies can utilize similar taxonomy methods as well.
block at select until either when the child sends result to ch
All our study results and studied commit logs can be found
(line 9) or when a timeout happens (line 11). If timeout hap-
at https://github.com/system-pclub/go-concurrency-bugs.
pens earlier or if Go runtime (non-deterministically) chooses
the case at line 11 when both cases are valid, the parent will
return from requestReq() at line 12, and no one else can 2 Background and Applications
pull result from ch any more, resulting in the child being Go is a statically-typed programming language that is de-
blocked forever. The fix is to change ch from an unbuffered signed for concurrent programming from day one [60]. Al-
channel to a buffered one, so that the child goroutine can most all major Go revisions include improvements in its con-
always send the result even when the parent has exit. currency packages [23]. This section gives a brief background
This bug demonstrates the complexity of using new fea- on Go’s concurrency mechanisms, including its thread model,
tures in Go and the difficulty in writing correct Go programs inter-thread communication methods, and thread synchro-
like this. Programmers have to have a clear understanding nization mechanisms. We also introduce the six Go applica-
of goroutine creation with anonymous function, a feature tions we chose for this study.
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

2.1 Goroutine The select statement allows a goroutine to wait on mul-

Go uses a concept called goroutine as its concurrency unit. tiple channel operations. A select will block until one of its
Goroutines are lightweight user-level threads that the Go cases can make progress or when it can execute a default
runtime library manages and maps to kernel-level threads branch. When more than one cases in a select are valid, Go
in an M-to-N way. A goroutine can be created by simply will randomly choose one to execute. This randomness can
adding the keyword go before a function call. cause concurrency bugs as will be discussed in Section 6.
To make goroutines easy to create, Go also supports creat- Go introduces several new semantics to ease the interac-
ing a new goroutine using an anonymous function, a function tion across multiple goroutines. For example, to assist the
definition that has no identifier, or “name”. All local variables programming model of serving a user request by spawn-
declared before an anonymous function are accessible to the ing a set of goroutines that work together, Go introduces
anonymous function, and are potentially shared between context to carry request-specific data or metadata across
a parent goroutine and a child goroutine created using the goroutines. As another example, Pipe is designed to stream
anonymous function, causing data race (Section 6). data between a Reader and a Writer. Both context and
Pipe are new forms of passing messages and misusing them
can create new types of concurrency bugs (Section 5).
2.2 Synchronization with Shared Memory Application Stars Commits Contributors LOC Dev History
Go supports traditional shared memory accesses across Docker 48975 35149 1767 786K 4.2 Years
Kubernetes 36581 65684 1679 2297K 3.9 Years
goroutines. It supports various traditional synchroniza- etcd 18417 14101 436 441K 4.9 Years
tion primitives like lock/unlock (Mutex), read/write lock CockroachDB 13461 29485 197 520k 4.2 Years
(RWMutex), condition variable (Cond), and atomic read/write gRPC* 5594 2528 148 53K 3.3 Years
BoltDB 8530 816 98 9K 4.4 Years
(atomic). Go’s implementation of RWMutex is different from
pthread_rwlock_t in C. Write lock requests in Go have a Table 1. Information of selected applications. The num-
higher privilege than read lock requests. ber of stars, commits, contributors on GitHub, total source lines of
As a new primitive introduced by Go, Once is designed code, and development history on GitHub. *: the gRPC version that is
to guarantee a function is only executed once. It has a Do written in Go.
method, with a function f as argument. When Once.Do(f)
is invoked many times, only for the first time, f is executed. 2.4 Go Applications
Once is widely used to ensure a shared variable only be Recent years have seen a quick increase in popularity and
initialized once by multiple goroutines. adoption of the Go language. Go was the 9th most popular
Similar to pthread_join in C, Go uses WaitGroup to al- language on GitHub in 2017 [18]. As of the time of writing,
low multiple goroutines to finish their shared variable ac- there are 187K GitHub repositories written in Go.
cesses before a waiting goroutine. Goroutines are added to In this study, we selected six representative, real-world
a WaitGroup by calling Add. Goroutines in a WaitGroup use software written in Go, including two container systems
Done to notify their completion, and a goroutine calls Wait (Docker and Kubernetes), one key-value store system (etcd),
to wait for the completion notification of all goroutines in two databases (CockroachDB and BoltDB), and one RPC
a WaitGroup. Misusing WaitGroup can cause both blocking library (gRPC-go1 ) (Table 1). These applications are open-
bugs (Section 5) and non-blocking bugs (Section 6). source projects that have gained wide usages in datacenter
environments. For example, Docker and Kubernetes are the
2.3 Synchronization with Message Passing top 2 most popular applications written in Go on GitHub,
with 48.9K and 36.5K stars (etcd is the 10th, and the rest are
Channel (chan) is a new concurrency primitive introduced
ranked in top 100). Our selected applications all have at least
by Go to send data and states across goroutines and to build
three years of development history and are actively main-
more complex functionalities [3, 50]. Go supports two types
tained by developers currently. All our selected applications
of channels: buffered and unbuffered. Sending data to (or
are of middle to large sizes, with lines of code ranging from 9
receiving data from) an unbuffered channel will block a gor-
thousand to more than 2 million. Among the six applications,
outine, until another goroutine receives data from (or sends
Kubernetes and gRPC are projects originally developed by
data to) the channel. Sending to a buffered channel will only
Google.
block, when the buffer is full. There are several underlying
rules in using channels and the violation of them can create 3 Go Concurrency Usage Patterns
concurrency bugs. For example, channel can only be used
after initialization, and sending data to (or receiving data Before studying Go concurrency bugs, it is important to first
from) a nil channel will block a goroutine forever. Sending understand how real-world Go concurrent programs are like.
data to a closed channel or close an already closed channel 1 We will use gRPC to represent the gRPC version that is written Go in the
can trigger a runtime panic. following paper, unless otherwise specified.
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

Application Normal F. Anonymous F. Total Per KLOC Goroutines/Threads Ave. Execution Time
Workload
Docker 33 112 145 0.18 client server client-Go server-Go
Kubernetes 301 233 534 0.23 g_sync_ping_pong 7.33 2.67 63.65% 76.97%
etcd 86 211 297 0.67 sync_ping_pong 7.33 4 63.23% 76.57%
CockroachDB 27 125 152 0.29 qps_unconstrained 201.46 6.36 91.05% 92.73%
gRPC-Go 14 30 44 0.83
BoltDB 2 0 2 0.22 Table 3. Dynamic information when executing RPC
gRPC-C 5 - 5 0.03 benchmarks. The ratio of goroutine number divided by thread
number and the average goroutine execution time normalized by the
Table 2. Number of goroutine/thread creation sites. The
whole application’s execution time.
number of goroutine/thread creation sites using normal functions and
Shared Memory Message
anonymous functions, total number of creation sites, and creation Application
Mutex atomic Once WaitGroup Cond chan Misc.
Total
sites per thousand lines of code. Docker 62.62% 1.06% 4.75% 1.70% 0.99% 27.87% 0.99% 1410
Kubernetes 70.34% 1.21% 6.13% 2.68% 0.96% 18.48% 0.20% 3951
This section presents our static and dynamic analysis results etcd 45.01% 0.63% 7.18% 3.95% 0.24% 42.99% 0 2075
CockroachDB 55.90% 0.49% 3.76% 8.57% 1.48% 28.23% 1.57% 3245
of goroutine usages and Go concurrency primitive usages in gRPC-Go 61.20% 1.15% 4.20% 7.00% 1.65% 23.03% 1.78% 786
our selected six applications. BoltDB 70.21% 2.13% 0 0 0 23.40% 4.26% 47

Table 4. Concurrency Primitive Usage. The Mutex column

3.1 Goroutine Usages
includes both Mutex and RWMutex.
To understand concurrency in Go, we should first under-
stand how goroutines are used in real-world Go programs. goroutine runtime durations and compare them to gRPC-C’s
One of the design philoshopies in Go is to make goroutines thread runtime durations. Since gRPC-Go and gRPC-C’s total
lightweight and easy to use. Thus, we ask “do real Go pro- execution time is different and it is meaningless to compare
grammers tend to write their code with many goroutines absolute goroutine/thread duration, we report and compare
(static)?” and “do real Go applications create a lot of gorou- the goroutine/thread duration relative to the total runtime
tines during runtime (dynamic)?” of gRPC-Go and gRPC-C. Specifically, we calculate average
To answer the first question, we collected the amount execution time of all goroutines/threads and normalize it
of goroutine creation sites (i.e., the source lines that create using the total execution time of the programs. We found all
goroutines). Table 2 summarizes the results. Overall, the threads in gRPC-C execute from the beginning to the end of
six applications use a large amount of goroutines. The av- the whole program (i.e., 100%) and thus only included the
erage creation sites per thousand source lines range from results of gRPC-Go in Table 3. For all workloads, the normal-
0.18 to 0.83. We further separate creation sites to those that ized execution time of goroutines is shorter than threads.
use normal functions to create goroutines and those that Observation 1: Goroutines are shorter but created more fre-
use anonymous functions. All the applications except for quently than C (both statically and at runtime).
Kubernetes and BoltDB use more anonymous functions.
To understand the difference between Go and traditional 3.2 Concurrency Primitive Usages
languages, we also analyzed another implementation of After a basic understanding of goroutine usages in real-world
gRPC, gRPC-C, which is implemented in C/C++. gRPC-C con- Go programs, we next study how goroutines communicate
tains 140K lines of code and is also maintained by Google’s and synchronize in these programs. Specifically, we calcu-
gRPC team. Compared to gRPC-Go, gRPC-C has surprisingly late the usages of different types of concurrency primitives
very few threads creation (only five creation sites and 0.03 in the six applications. Table 4 presents the total (absolute
sites per KLOC). amount of primitive usages) and the proportion of each type
We further study the runtime creation of goroutines. We of primitive over the total primitives. Shared memory syn-
ran gRPC-Go and gRPC-C to process three performance chronization operations are used more often than message
benchmarks that were designed to compare the performance passing, and Mutex is the most widely-used primitive across
of multiple gRPC versions written in different program- all applications. For message-passing primitives, chan is the
ming languages [22]. These benchmarks configure gRPC one used most frequently, ranging from 18.48% to 42.99%.
with different message formats, different numbers of con- We further compare the usages of concurrency primitives
nections, and synchronous vs. asynchronous RPC requests. in gRPC-C and in gRPC-Go. gRPC-C only uses lock, and it is
Since gRPC-C is faster than gRPC-Go [22], we ran gRPC-C used in 746 places (5.3 primitive usages per KLOC). gRPC-Go
and gRPC-Go to process the same amount of RPC requests, uses eight different types of primitives in 786 places (14.8
instead of the same amount of total time. primitive usages per KLOC). Clearly, gRPC-Go uses a larger
Table 3 shows the ratio of the number of goroutines cre- amount of and a larger variety of concurrency primitives
ated in gRPC-Go over the number of threads created in gRPC- than gRPC-C.
C when running the three workloads. More goroutines are Next, we study how the usages of concurrency primitives
created across different workloads for both the client side change over time. Figures 2 and 3 present the shared-memory
and the server side. Table 3 also presents our study results of and message-passing primitive usages in the six applications
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

Percentage of Studied Bugs

1 docker kubernetes etcd

Usage Proportion
1
Usage Proportion

0.8 cockroachdb grpc−go boltdb

0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
docker kubernetes etcd shared memory
0.2
cockroachdb grpc−go boltdb 0.2 message passing
0

.15−02

15−05

15−08

15−11

16−02

16−05

16−08

16−11

17−02

17−05

17−08

17−11

18−02

18−05
0
0
.15−02

15−05

15−08

15−11

16−02

16−05

16−08

16−11

17−02

17−05

17−08

17−11

18−02

18−05
0 100 200 300 400 500 600 700
Bug Life Time (Days)
Figure 3. Usages of Message-
Figure 2. Usages of Shared-Memory
Passing Primitives over Time. For Figure 4. Bug Life Time. The CDF of
Primitives over Time. For each appli-
each application, we calculate the proportion the life time of all shared-memory bugs and
cation, we calculate the proportion of shared-
of message-passing primitives over all all message-passing bugs.
memory primitives over all primitives.
primitives.
from Feb 2015 to May 2018. Overall, the usages tend to be Bug taxonomy. We propose a new method to categorize Go
stable over time, which also implies that our study results concurrency bugs according to two orthogonal dimensions.
will be valuable for future Go programmers. The first dimension is based on the behavior of bugs. If one or
Observation 2: Although traditional shared memory thread more goroutines are unintentionally stuck in their execution
communication and synchronization remains to be heavily and cannot move forward, we call such concurrency issues
used, Go programmers also use significant amount of message- blocking bugs. If instead all goroutines can finish their tasks
passing primitives. but their behaviors are not desired, we call them non-blocking
Implication 1: With heavier usages of goroutines and new ones. Most previous concurrency bug studies [24, 43, 45]
types of concurrency primitives, Go programs may potentially categorize bugs into deadlock bugs and non-deadlock bugs,
introduce more concurrency bugs. where deadlocks include situations where there is a circular
wait across multiple threads. Our definition of blocking is
4 Bug Study Methodology broader than deadlocks and include situations where there
This section discusses how we collected, categorized, and is no circular wait but one (or more) goroutines wait for
reproduced concurrency bugs in this study. resources that no other goroutines supply. As we will show
Collecting concurrency bugs. To collect concurrency in Section 5, quite a few Go concurrency bugs are of this
bugs, we first filtered GitHub commit histories of the six kind. We believe that with new programming habits and
applications by searching their commit logs for concurrency- semantics with new languages like Go, we should pay more
related keywords, including “race”, “deadlock”, “synchroniza- attention to these non-deadlock blocking bugs and extend
tion”, “concurrency”, “lock”, “mutex”, “atomic”, “compete”, the traditional concurrency bug categorization mechanism.
“context”, “once”, and “goroutine leak”. Some of these key- The second dimension is along the cause of concurrency
words are used in previous works to collect concurrency bugs. Concurrency bugs happen when multiple threads try to
bugs in other languages [40, 42, 45]. Some of them are re- communicate and errors happen during such communication.
lated to new concurrency primitives or libraries introduced Our idea is thus to categorize causes of concurrency bugs by
by Go, such as “once” and “context”. One of them, “goroutine how different goroutines communicate: by accessing shared
leak”, is related to a special problem in Go. In total, we found memory or by passing messages. This categorization can
3211 distinct commits that match our search criteria. help programmers and researchers choose better ways to
Behavior Cause perform inter-thread communication and to detect and avoid
Application
blocking non-blocking shared memory message passing potential errors when performing such communication.
Docker 21 23 28 16 According to our categorization method, there are a total
Kubernetes 17 17 20 14
etcd 21 16 18 19 of 85 blocking bugs and 86 non-blocking bugs, and there
CockroachDB 12 16 23 5 are a total of 105 bugs caused by wrong shared memory
gRPC 11 12 12 11
BoltDB 3 2 4 1
protection and 66 bugs caused by wrong message passing.
Total 85 86 105 66 Table 5 shows the detailed breakdown of bug categories
Table 5. Taxonomy. This table shows how our studied bugs dis- across each application.
tribute across different categories and applications. We further analyzed the life time of our studied bugs, i.e.,
We then randomly sampled the filtered commits, identified the time from when the buggy code was added (committed)
commits that fix concurrency bugs, and manually studied to the software to when it is being fixed in the software (a bug-
them. Many bug-related commit logs also mention the cor- fixing patch is committed). As shown in Figure 4, most bugs
responding bug reports, and we also study these reports for we study (both shared memory and message passing) have
our bug analysis. We studied 171 concurrency bugs in total. long life time. We also found the time when these bugs were
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

Shared Memory Message Passing

Application
Mutex RWMutex Wait Chan Chan w/ Lib
5.1 Root Causes of Blocking Bugs
Docker 9 0 3 5 2 2 Blocking bugs manifest when one or more goroutines con-
Kubernetes 6 2 0 3 6 0
etcd 5 0 0 10 5 1
duct operations that wait for resources, and these resources
CockroachDB 4 3 0 5 0 0 are never available. To detect and avoid blocking bugs, it is
gRPC 2 0 0 6 2 1 important to understand their root causes. We study block-
BoltDB 2 0 0 0 1 0
Total 28 5 3 29 16 4 ing bugs’ root causes by examining which operation blocks
a goroutine and why the operation is not unblocked by other
Table 6. Blocking Bug Causes. Wait includes both the Wait
goroutines. Using our second dimension of bug categoriza-
function in Cond and in WaitGroup. Chan indicates channel opera-
tion, we separate blocking bugs into those that are caused by
tions and Chan w/ means channel operations with other operations.
stuck operations that are intended to protect shared mem-
Lib stands for Go libraries related to message passing.
ory accesses and those that are caused by message passing
operations. Table 6 summarizes the root causes of all the
blocking bugs.
report to be close to when they were fixed. These results Overall, we found that there are around 42% blocking bugs
show that most of the bugs we study are not easy to be caused by errors in protecting shared memory, and 58% are
triggered or detected, but once they are, they got fixed very caused by errors in message passing. Considering that shared
soon. Thus, we believe these bugs are non-trivial and worth memory primitives are used more frequently than message
close examination. passing ones (Section 3.2), message passing operations are
Reproducing concurrency bugs. In order to evaluate the even more likely to cause blocking bugs.
built-in deadlock and data-race detection techniques, we Observation 3: Contrary to the common belief that message
reproduced 21 blocking bugs and 20 non-blocking bugs. To passing is less error-prone, more blocking bugs in our studied
reproduce a bug, we rolled the application back to the buggy Go applications are caused by wrong message passing than by
version, built the buggy version, and ran the built program wrong shared memory protection.
using the bug-triggering input described in the bug report.
We leveraged the symptom mentioned in the bug report
5.1.1 (mis)Protection of Shared Memory
to decide whether we have successfully reproduced a bug.
Due to their non-deterministic nature, concurrency bugs Shared memory accesses are notoriously hard to program
are difficult to reproduce. Sometimes, we needed to run a correctly and have always been one of the major focuses
buggy program a lot of times or manually add sleep to a on deadlock research [35, 51, 54]. They continue to cause
buggy program. For a bug that is not reproduced, it is either blocking bugs in Go, both with traditional patterns and new,
because we do not find some dependent libraries, or because Go-specific reasons.
we fail to observe the described symptom. Mutex 28 blocking bugs are caused by misusing locks
Threats to validity. Threats to the validity of our study (Mutex), including double locking, acquiring locks in conflict-
could come from many aspects. We selected six represen- ing orders, and forgetting to unlock. All bugs in this category
tative Go applications. There are many other applications are traditional bugs, and we believe traditional deadlock de-
implemented in Go and they may not share the same con- tection algorithms should be able to detect these bugs with
currency problems. We only studied concurrency bugs that static program analysis.
have been fixed. There could be other concurrency bugs RWMutex As explained in Section 2.2, Go’s write lock re-
that are rarely reproduced and are never fixed by developers. quests have a higher privilege than read lock requests. This
For some fixed concurrency bugs, there is too little infor- unique lock implementation can lead to a blocking bug when
mation provided, making them hard to understand. We do a goroutine (th-A) acquires one RWMutex twice with read
not include these bugs in our study. Despite these limita- locking, and these two read lock operations are interleaved
tions, we have made our best efforts in collecting real-world by a write lock operation from another goroutine (th-B).
Go concurrency bugs and in conducting a comprehensive When th-A’s first read lock operation succeeds, it will block
and unbiased study. We believe that our findings are general th-B’s write lock operation, since write locking is exclusive.
enough to motivate and guide future research on fighting However, th-B’s write lock operation will also block th-A’s
Go concurrency bugs. second read lock operation, since the write lock request has
a higher privilege in Go’s implementation. Neither th-A nor
th-B will be able to proceed.
5 Blocking Bugs Five blocking bugs are caused by this reason. Note that the
This section presents our study results on blocking bugs, same interleaving locking pattern will not cause blocking
including their root causes, fixes, and the effectiveness of the bugs for pthread_rwlock_t in C, since pthread_rwlock_t
built-in runtime Go deadlock detector on detecting blocking prioritize read lock requests under the default setting. The
situations. RWMutex blocking bug type implies that even when Go uses
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

1 var group sync.WaitGroup 1 - hctx, hcancel := context.WithCancel(ctx)

2 group.Add(len(pm.plugins)) 2 + var hctx context.Context
3 for _, p := range pm.plugins { 3 + var hcancel context.CancelFunc
4 go func(p *plugin) { 4 if timeout > 0 {
5 defer group.Done() 5 hctx, hcancel = context.WithTimeout(ctx, timeout)
6 } 6 + } else {
7 - group.Wait() 7 + hctx, hcancel = context.WithCancel(ctx)
8 } 8 }
9 + group.Wait()
Figure 6. A blocking bug caused by context.
Figure 5. A blocking bug caused by WaitGroup.
When combining with the usage of Go special libraries,
the channel creation and goroutine blocking may be buried
the same concurrency semantics as traditional languages,
inside library calls. As shown in Figure 6, a new context
there can still be new types of bugs because of Go’s new
object, hcancel, is created at line 1. A new goroutine is
implementation of the semantics.
created at the same time, and messages can be sent to the new
Wait Three blocking bugs are due to wait operations that
goroutine through the channel field of hcancel. If timeout
cannot proceed. Unlike Mutex and RWMutex related bugs,
is larger than 0 at line 4, another context object is created at
they do not involve circular wait. Two of these bugs happen
line 5, and hcancel is pointing to the new object. After that,
when Cond is used to protect shared memory accesses and
there is no way to send messages to or close the goroutine
one goroutine calls Cond.Wait(), but no other goroutines
attached to the old object. The patch is to avoid creating the
call Cond.Signal() (or Cond.Broadcast()) after that.
extra context object when timeout is larger than 0.
The third bug, Docker#25384, happens with the use of
a shared variable of type WaitGroup, as shown in Fig- 1 func goroutine1() {
ure 5. The Wait() at line 7 can only be unblocked, when 2 m.Lock()
3 - ch <- request //blocks 1 func goroutine2() {
Done() at line 5 is invoked len(pm.plugins) times, since 4 + select { 2 for {
len(pm.plugins) is used as parameter to call Add() at line 5 + case ch <- request 3 m.Lock() //blocks
2. However, the Wait() is called inside the loop, so that it 6 + default: 4 m.Unlock()
7 + } 5 request <- ch
blocks goroutine creation at line 4 in later iterations and it 8 m.Unlock() 6 }
blocks the invocation of Done() inside each created gorou- 9 } 7 }
tine. The fix of this bug is to move the invocation of Wait() (a) goroutine 1 (b) goroutine 2

out from the loop. Figure 7. A blocking bug caused by wrong usage of
Although conditional variable and thread group wait are channel with lock.
both traditional concurrency techniques, we suspect Go’s Channel and other blocking primitives For 16 blocking bugs,
new programming model to be one of the reasons why pro- one goroutine is blocked at a channel operation, and another
grammers made these concurrency bugs. For example, un- goroutine is blocked at lock or wait. For example, as shown
like pthread_join which is a function call that explicitly in Figure 7, goroutine1 is blocked at sending request to
waits on the completion of (named) threads, WaitGroup is a channel ch, while goroutine2 is blocked at m.Lock(). The
variable that can be shared across goroutines and its Wait fix is to add a select with default branch for goroutine1
function implicitly waits for the Done function. to make ch not blocking any more.
Observation 4: Most blocking bugs that are caused by shared Messaging libraries Go provides several libraries to pass data
memory synchronization have the same causes and same fixes or messages, like Pipe. These special library calls can also
as traditional languages. However, a few of them are different cause blocking bugs when not used correctly. For example,
from traditional languages either because of Go’s new im- similar to channel, if a Pipe is not closed, a goroutine can be
plementation of existing primitives or its new programming blocked when it tries to send data to or pull data from the
semantics. unclosed Pipe. There are 4 collected blocking bugs caused
by special Go message-passing library calls.
5.1.2 Misuse of Message Passing
Observation 5: All blocking bugs caused by message passing
We now discuss blocking bugs caused by errors in message are related to Go’s new message passing semantics like channel.
passing, which in the contrary of common belief are the They can be difficult to detect especially when message pass-
main type of blocking bugs in our studied applications. ing operations are used together with other synchronization
Channel Mistakes in using channel to pass messages across mechanisms.
goroutines cause 29 blocking bugs. Many of the channel- Implication 2: Contrary to common belief, message passing
related blocking bugs are caused by the missing of a send to can cause more blocking bugs than shared memory. We call for
(or receive from) a channel or closing a channel, which will attention to the potential danger in programming with message
result in the blocking of a goroutine that waits to receive passing and raise the research question of bug detection in this
from (or send to) the channel. One such example is Figure 1. area.
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

Adds Moves Changes Removes Misc. Root Cause # of Used Bugs # of Detected Bugs
Shared Memory Mutex 7 1
Mutex 9 7 2 8 2 Chan 8 0
Wait 0 1 0 1 1 Chan w/ 4 1
RWMutex 0 2 0 3 0 Messaging Libraries 2 0
Message Passing Total 21 2
Chan 15 1 5 4 4
Chan w/ 6 3 2 4 1 Table 8. Benchmarks and evaluation results of the
Messaging Lib 1 0 0 1 2
Total 31 14 9 21 10
deadlock detector.
Table 7. Fix strategies for blocking bugs. The subscript s between Chan and Adds is the second highest, with lift value
stands for synchronization. 1.42. All other categories that have more than 10 blocking
5.2 Fixes of Blocking Bugs bugs have lift values below 1.16, showing no strong correla-
After understanding the causes of blocking bugs in Go, we tion.
now analyze how Go programmers fixed these bugs in the We also analyzed the fixes of blocking bugs according to
real world. the type of concurrency primitives used in the patches. As
Eliminating the blocking cause of a hanging goroutine expected, most bugs whose causes are related to a certain
will unblock it and this is the general approach to fix block- type of primitive were also fixed by adjusting that primitive.
ing bugs. To achieve this goal, Go developers often adjust For example, all Mutex-related bugs were fixed by adjusting
synchronization operations, including adding missing ones, Mutex primitives.
moving or changing misplaced/misused ones, and removing The high correlation of bug causes and the primitives and
extra ones. Table 7 summarizes these fixes. strategies used to fix them, plus the limited types of syn-
Most blocking bugs caused by mistakenly protecting chronization primitives in Go, suggests fruitful revenue in
shared memory accesses were fixed by methods similar to investigating automatic correction of blocking bugs in Go.
traditional deadlock fixes. For example, among the 33 Mutex- We further find that the patch size of our studied blocking
or RWMutex-related bugs, 8 were fixed by adding a missing bugs is small, with an average of 6.8 lines of code. Around
unlock; 9 were fixed by moving lock or unlock operations 90% of studied blocking bugs are fixed by adjusting synchro-
to proper locations; and 11 were fixed by removing an extra nization primitives.
lock operation. Observation 6: Most blocking bugs in our study (both tradi-
11 blocking bugs caused by wrong message passing were tional shared-memory ones and message passing ones) can be
fixed by adding a missing message or closing operation to fixed with simple solutions and many fixes are correlated with
a channel (and on two occasions, to a pipe) on a goroutine bug causes.
different from the blocking one. 8 blocking bugs were fixed Implication 3: High correlation between causes and fixes in
by adding a select with a default option (e.g., Figure 7) Go blocking bugs and the simplicity in their fixes suggest that
or a case with operation on a different channel. Another it is promising to develop fully automated or semi-automated
common fix of channel-related blocking bugs is to replace an tools to fix blocking bugs in Go.
unbuffered channel with a buffered channel (e.g., Figure 1).
Other channel-related blocking bugs can be fixed by strate- 5.3 Detection of Blocking Bugs
gies such as moving a channel operation out of a critical Go provides a built-in deadlock detector that is implemented
section and replacing channel with shared variables. in the goroutine scheduler. The detector is always enabled
To understand the relationship between the cause of during Go runtime and it reports deadlock when no gorou-
a blocking bug and its fix, we apply a statistical metric tines in a running process can make progress. We tested all
called lift, following previous empirical studies on real-world our reproduced blocking bugs with Go’s built-in deadlock
P (AB)
bugs [29, 41]. lift is calculated as lift(A, B) = P (A)P (B) , where detector to evaluate what bugs it can find. For every tested
A denotes a root cause category, B denotes a fix strategy cate- bug, the blocking can be triggered deterministically in ev-
gory, P(AB) denotes the probability that a blocking is caused ery run. Therefore, for each bug, we only ran it once in this
by A and fixed by B. When lift value is equal to 1, A root experiment. Table 8 summarizes our test results.
cause is independent with B fix strategy. When lift value is The built-in deadlock detector can only detect two block-
larger than 1, A and B are positively correlated, which means ing bugs, BoltDB#392 and BoltDB#240, and fail in all other
if a blocking is caused by A, it is more likely to be fixed by B. cases (although the detector does not report any false posi-
When lift is smaller than 1, A and B are negatively correlated. tives [38, 39]). There are two reasons why the built-in detec-
Among all the bug categories that have more than 10 tor failed to detect other blocking bugs. First, it does not con-
blocking bugs (we omit categories that have less than 10 sider the monitored system as blocking when there are still
bugs because of their statistical insignificance), Mutex is the some running goroutines. Second, it only examines whether
category that has the strongest correlation to a type of fix—it or not goroutines are blocked at Go concurrency primitives
correlates with Moves with lift value 1.52. The correlation but does not consider goroutines that wait for other systems
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

Shared Memory Message Passing

Application 1 for i := 17; i <= 21; i++ { // write
traditional anon. waitgroup lib chan lib
2 - go func() { /* Create a new goroutine */
Docker 9 6 0 1 6 1
3 + go func(i int) {
Kubernetes 8 3 1 0 5 0
etcd 9 0 2 2 3 0 4 apiVersion := fmt.Sprintf("v1.%d", i) // read
CockroachDB 10 1 3 2 0 0 5 ...
gRPC 8 1 0 1 2 0 6 - }()
BoltDB 2 0 0 0 0 0 7 + }(i)
Total 46 11 6 6 16 1 8 }

Table 9. Root causes of non-blocking bugs. traditional: Figure 8. A data race caused by anonymous function.
traditional non-blocking bugs; anonymous function: non-blocking 1 func (p *peer) send() {
bugs caused by anonymous function; waitgroup: misusing WaitGroup; 2 p.mu.Lock()
3 defer p.mu.Unlock()
lib: Go library; chan: misusing channel.
4 switch p.status {
5 case idle:
resources. These two limitations were largely due to the de- 6 + p.wg.Add(1)
sign goal of the built-in detector—minimal runtime overhead. 7 go func() {
8 - p.wg.Add(1)
When implemented in the runtime scheduler, it is very hard 9 ...
1 func (p * peer) stop() {
for a detector to effectively identify complex blocking bugs 2 p.mu.Lock()
10 p.wg.Done()
3 p.status = stopped
without sacrificing performance. 11 }()
4 p.mu.Unlock()
12 case stopped:
Implication 4: Simple runtime deadlock detector is not ef- 13 }
5 p.wg.Wait()
6 }
fective in detecting Go blocking bugs. Future research should 14 }
(a) func1 (b) func2
focus on building novel blocking bug detection techniques, for
example, with a combination of static and dynamic blocking Figure 9. A non-blocking bug caused by misusing
pattern detection. WaitGroup.
Docker#22985 and CockroachDB#6111 are caused by data
6 Non-Blocking Bugs race on a shared variable whose reference is passed across
This section presents our study on non-blocking bugs. Simi- goroutines through a channel.
lar to what we did in Section 5, we studied the root causes Anonymous function Go designers make goroutine declara-
and fixes of non-blocking bugs and evaluated a built-in race tion similar to a regular function call (which does not even
detector of Go. need to have a “function name”) so as to ease the creation of
goroutines. All local variables declared before a Go anony-
6.1 Root Causes of Non-blocking Bugs mous function are accessible by the anonymous function.
Similar to blocking bugs, we also categorize our collected Unfortunately, this ease of programming can increase the
non-blocking bugs into those that were caused by failing chance of data-race bugs when goroutines are created with
to protect shared memory and those that have errors with anonymous functions, since developers may not pay enough
message passing (Table 9). attention to protect such shared local variables.
We found 11 bugs of this type, 9 of which are caused by a
6.1.1 Failing to Protect Shared Memory data race between a parent goroutine and a child goroutine
Previous work [8, 14, 16, 17, 46, 47, 52, 62–64] found that not created using an anonymous function. The other two are
protecting shared memory accesses or errors in such protec- caused by a data race between two child goroutines. One
tion are the main causes of data race and other non-deadlock example from Docker is shown in Figure 8. Local variable i
bugs. Similarly, we found around 80% of our collected non- is shared between the parent goroutine and the goroutines it
blocking bugs are due to un-protected or wrongly protected creates at line 2. The developer intends each child goroutine
shared memory accesses. However, not all of them share the uses a distinct i value to initialize string apiVersion at line 4.
same causes as non-blocking bugs in traditional languages. However, values of apiVersion are non-deterministic in the
Traditional bugs More than half of our collected non- buggy program. For example, if the child goroutines begin
blocking bugs are caused by traditional problems that also after the whole loop of the parent goroutine finishes, value of
happen in classic languages like C and Java, such as atomic- apiVersion are all equal to ‘v1.21’. The buggy program only
ity violation [8, 16, 46], order violation [17, 47, 62, 64], and produces desired result when each child goroutine initializes
data race [14, 52, 63]. This result shows that same mistakes string apiVersion immediately after its creation and before
are made by developers across different languages. It also i is assigned to a new value. Docker developers fixed this bug
indicates that it is promising to apply existing concurrency by making a copy of the shared variable i at every iteration
bug detection algorithms to look for new bugs in Go. and pass the copied value to the new goroutines.
Interestingly, we found seven non-blocking bugs whose Misusing WaitGroup There is an underlying rule when using
root causes are traditional but are largely caused by the lack WaitGroup, which is that Add has to be invoked before Wait.
of a clear understanding in new Go features. For example, The violation of this rule causes 6 non-blocking bugs. Figure 9
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

shows one such bug in etcd, where there is no guarantee 1 ticker := time.NewTicker()
that Add at line 8 of func1 happens before Wait at line 5 of 2 for {
3 + select {
func2. The fix is to move Add into a critical section, which 4 + case <- stopCh:
ensures that Add will either be executed before Wait or it 5 + return
will not be executed. 6 + default:
7 + }
Special libraries Go provides many new libraries, some of 8 f()
which use objects that are implicitly shared by multiple gor- 9 select {
outines. If they are not used correctly, data race may hap- 10 case <- stopCh:
11 return
pen. For example, the context object type is designed to 12 case <- ticker:
be accessed by multiple goroutines that are attached to the 13 }
context. etcd#7816 is a data-race bug caused by multiple 14 }

goroutines accessing the string field of a context object. Figure 11. A non-blocking bug caused by select and
Another example is the testing package which is de- channel.
signed to support automated testing. A testing function (iden-
tified by beginning the function name with “Test”) takes only 1 - timer := time.NewTimer(0)
one parameter of type testing.T, which is used to pass test- 2 + var timeout <- chan time.Time
3 if dur > 0 {
ing states such as error and log. Three data-race bugs are 4 - timer = time.NewTimer(dur)
caused by accesses to a testing.T variable from the gor- 5 + timeout = time.NewTimer(dur).C
outine running the testing function and other goroutines 6 }
7 select {
created inside the testing function. 8 - case <- timer.C:
Observation 7: About two-thirds of shared-memory non- 9 + case <- timeout:
blocking bugs are caused by traditional causes. Go’s new multi- 10 case <-ctx.Done():
11 return nil
thread semantics and new libraries contribute to the rest one- 12 }
third.
Implication 5: New programming models and new libraries Figure 12. A non-blocking bug caused by Timer.
that Go introduced to ease multi-thread programming can
themselves be the reasons of more concurrency bugs. be processed first. This non-determinism implementation of
select caused 3 bugs. Figure 11 shows one such example.
6.1.2 Errors during Message Passing The loop at line 2 executes a heavy function f() at line
Errors during message passing can also cause non-blocking 8 whenever a ticker ticks at line 12 (case 2) and stops its
bugs and they comprise around 20% of our collected non- execution when receiving a message from channel stopCh
blocking bugs. at line 10 (case 1). If receiving a message from stopCh and
Misusing channel As what we discussed in Section 2, there the ticker ticks at the same time, there is no guarantee which
are several rules when using channel, and violating them one will be chosen by select. If select chooses case 2, f()
can lead to non-blocking bugs in addition to blocking ones. will be executed unnecessarily one more time. The fix is to
There are 16 non-blocking bugs caused by misuse of channel. add another select at the beginning of the loop to handle
the unprocessed signal from stopCh.
1 - select {
2 - case <- c.closed:
Special libraries Some of Go’s special libraries use channels
3 - default: in a subtle way, which can also cause non-blocking bugs.
4 + Once.Do(func() { Figure 12 shows one such bug related to the time package
5 close(c.closed)
6 + }) which is designed for measuring time. Here, a timer is cre-
7 - } ated with timeout duration 0 at line 1. At the creation time
of a Timer object, Go runtime (implicitly) starts a library-
Figure 10. A bug caused by closing a channel twice. internal goroutine which starts timer countdown. The timer
As an example, Docker#24007 in Figure 10 is caused by the is set with a timeout value dur at line 4. Developers here
violation of the rule that a channel can only be closed once. intended to return from the current function only when dur
When multiple goroutines execute the piece of code, more is larger than 0 or when ctx.Done(). However, when dur is
than one of them can execute the default clause and try to not greater than 0, the library-internal goroutine will signal
close the channel at line 5, causing a runtime panic in Go. the timer.C channel as soon as the creation of the timer,
The fix is to use Once package to enforce that the channel is causing the function to return prematurely (line 8). The fix
only closed once. is to avoid the Timer creation at line 1.
Another type of concurrency bugs happen when using Observation 8: There are much fewer non-blocking bugs
channel and select together. In Go, when multiple messages caused by message passing than by shared memory accesses.
received by a select, there is no guarantee which one will Rules of channel and complexity of using channel with other
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

Timing Instruction Data Root Cause # of Used Bugs # of Detected Bugs

Misc.
Adds Moves Bypass Private Traditional Bugs 13 7
Shared Memory Anonymous Function 4 3
traditional 27 4 5 10 0 Lib 2 0
waitgroup 3 2 1 0 0 Misusing Channel 1 0
anonymous 5 2 0 4 0 Total 20 10
lib 1 2 1 0 2
Message Passing Table 12. Benchmarks and evaluation results of the
chan 6 7 3 0 0 data race detector. We consider a bug detected within 100 runs
lib 0 0 0 0 1
Total 42 17 10 14 3 as a detected bug.
Table 10. Fix strategies for non-blocking bugs. The sub-
goroutines and to replace shared variable to fix data race. It
script s stands for synchronization.
was also used to enforce the order between two operations
Mutex Channel Atomic WaitGroup Cond Misc. None in different goroutines. There are also bugs where channel is
Shared Memory not properly used and is fixed in the patch (e.g., Figure 10).
traditional 24 3 6 0 0 0 13
waitgroup 2 0 0 4 3 0 0 Interestingly, channels were not only used to fix message-
anonymous 3 2 3 0 0 0 3 passing bugs but also bugs caused by traditional shared mem-
lib 0 2 1 1 0 1 2
Message Passing ory synchronization. We suspect this is because some Go
chan 3 11 0 2 1 2 1 programmers view message passing as a more reliable way
lib 0 1 0 0 0 0 0
Total 32 19 10 7 4 3 19 or easier-to-program way of performing inter-thread com-
munication than shared memory synchronization.
Table 11. Synchronization primitives in patches of Finally, 24 bugs were fixed by other concurrency primi-
non-blocking bugs. tives and 19 bugs were fixed without using any concurrency
Go-specific semantics and libraries are the reasons why these primitives (e.g., Figure 8).
non-blocking bugs happen. Similar to our lift analysis in Section 5.2, we calculate lift
Implication 6: When used correctly, message passing can between causes and fix strategies and between causes and
be less prone to non-blocking bugs than shared memory ac- fix primitives for non-blocking bugs. Among bug categories
cesses. However, the intricate design of message passing in a with more than 10 bugs, the strongest correlation is between
language can cause these bugs to be especially hard to find the cause misusing channel and fix primitive channel, with a
when combining with other language-specific features. lift value of 2.7. The cause anonymous function and the fix
6.2 Fixes of Non-Blocking Bugs strategy data private has the second highest lift value of 2.23.
Next, Misusing channel is strongly correlated with Moves
Similar to our analysis of blocking bug fixes, we first ana-
with lift value 2.21.
lyze fixes of non-blocking bugs by their strategies. Table 10
Observation 9: Traditional shared memory synchronization
categorizes the fix strategies of our studied Go non-blocking
techniques remain to be the main fixes for non-blocking bugs in
bugs, in a similar way as a previous categorization of non-
Go, while channel is used widely to fix not only channel-related
blocking bug fixes in C/C++ [43].
bugs but also shared-memory bugs.
Around 69% of the non-blocking bugs were fixed by re-
Implication 7: While Go programmers continue to use tra-
stricting timing, either through adding synchronization prim-
itives like Mutex, or through moving existing primitives like ditional shared memory protection mechanisms to fix non-
moving Add in Figure 9. 10 non-blocking bugs were fixed blocking bugs, they prefer the use of message passing as a fix
by eliminating instructions accessing shared variables or by in certain cases possibly because they view message passing as
bypassing the instructions (e.g., Figure 10). 14 bugs were a safer way to communicate across threads.
fixed by making a private copy of the shared variable (e.g.,
Figure 8) and these bugs are all shared-memory ones. 6.3 Detection of Non-Blocking Bugs
To have a better understanding of non-blocking bug fixes Go provides a data race detector which uses the same happen-
and their relationship to bug causes, we further check what before algorithm as ThreadSanitizer [53]. It can be enabled
primitives are leveraged inside patches. Table 11 lists the by building a program using the ‘-race’ flag. During program
fixes according to the type of primitives used in the patches. execution, the race detector creates up to four shadow words
Similar to the results of a previous study on patches of con- for every memory object to store historical accesses of the
currency bugs in C/C++ [43], mutex is the most widely used object. It compares every new access with the stored shadow
primitive to enforce mutual exclusion and fix non-blocking word values to detect possible races.
bugs. Besides traditional bugs, mutex was also used to fix We use our 20 reproduced non-blocking bugs to evaluate
races caused by anonymous function and by WaitGroup and how many bugs the detector can detect. We ran each buggy
to replace misused channel. program 100 times with the race detector turned on. Table 12
As a new primitive, channel is the second most widely- summarizes the number of bugs detected under each root
used. Channel was leveraged to pass value between two cause category. The detector reports no false positives.
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

The data race detector successfully detected 7/13 tradi- in detecting bugs that are caused by the combination of
tional bugs and 3/4 bugs caused by anonymous functions. channel and locks, such as the one in Figure 7.
For six of these successes, the data race detector reported Misusing Go libraries can cause both blocking and non-
bugs on every run, while for the rest four, around 100 runs blocking bugs. We summarized several patterns about mis-
were needed before the detector reported a bug. using Go libraries in our study. Detectors can leverage the
There are three possible reasons why the data race detec- patterns we learned to reveal previously unknown bugs.
tor failed to report many non-blocking bugs. First, not all Our study also found the violation of rules Go enforces
non-blocking bugs are data races; the race detector was not with its concurrency primitives is one major reason for con-
designed to detect these other types. Second, the effective- currency bugs. A novel dynamic technique can try to enforce
ness of the underlying happen-before algorithm depends on such rules and detect violation at runtime.
the interleaving of concurrent goroutines. Finally, with only
four shadow words for each memory object, the detector
cannot keep a long history and may miss data races. 8 Related Works
Implication 8: Simple traditional data race detector cannot
Studying Real-World Bugs. There are many empirical
effectively detect all types of Go non-blocking bugs. Future studies on real-world bugs [9, 24, 25, 29, 40, 44, 45]. These
research can leverage our bug analysis to develop more infor- studies have successfully guided the design of various bug-
mative, Go-specific non-blocking bug detectors. combating techniques. To the best of our knowledge, our
7 Discussion and Future Work work is the first study focusing on concurrency bugs in Go
and the first to compare bugs caused by errors when access-
Go advocates for making thread creation easy and light-
ing shared memory and errors when passing messages.
weight and for using message passing over shared memory
Combating Blocking Bugs. As a traditional problem, there
for inter-thread communication. Indeed, we saw more gor-
are many research works fighting deadlocks in C and
outines created in Go programs than traditional threads and
Java [7, 28, 33–35, 51, 54, 55, 58, 59]. Although useful, our
there are significant usages of Go channel and other mes-
study shows that there are many non-deadlock blocking
sage passing mechanisms. However, our study show that
bugs in Go, which are not the goal of these techniques. Some
if not used correctly, these two programming practices can
techniques are proposed to detect blocking bugs caused by
potentially cause concurrency bugs.
misusing channel [38, 39, 49, 56]. However, blocking bugs
Shared memory vs. message passing. Our study found
can be caused by other primitives. Our study reveals many
that message passing does not necessarily make multi-
code patterns for blocking bugs that can serve the basis for
threaded programs less error-prone than shared memory.
future blocking bug detection techniques.
In fact, message passing is the main cause of blocking bugs.
Combating Non-Blocking Bugs. Many previous research
To make it worse, when combined with traditional synchro-
works are conducted to detect, diagnose and fix non-deadlock
nization primitives or with other new language features
bugs, caused by failing to synchronize shared memory ac-
and libraries, message passing can cause blocking bugs that
cesses [4, 5, 8, 14, 16, 17, 30–32, 43, 46, 47, 52, 62–64]. They
are very hard to detect. Message passing causes less non-
are promising to be applied to Go concurrency bugs. How-
blocking bugs than shared memory synchronization and sur-
ever, our study finds that there is a non-negligible portion of
prisingly, was even used to fix bugs that are caused by wrong
non-blocking bugs caused by errors during message passing,
shared memory synchronization. We believe that message
and these bugs are not covered by previous works. Our study
passing offers a clean form of inter-thread communication
emphasizes the need of new techniques to fight errors during
and can be useful in passing data and signals. But they are
message passing.
only useful if used correctly, which requires programmers
to not only understand message passing mechanisms well
but also other synchronization mechanisms of Go.
Implication on bug detection. Our study reveals many 9 Conclusion
buggy code patterns that can be leveraged to conduct con- As a programming language designed for concurrency, Go
currency bug detection. As a preliminary effort, we built a provides lightweight goroutines and channel-based message
detector targeting the non-blocking bugs caused by anony- passing between goroutines. Facing the increasing usage of
mous functions (e.g. Figure 8). Our detector has already dis- Go in various types of applications, this paper conducts the
covered a few new bugs, one of which has been confirmed first comprehensive, empirical study on 171 real-world Go
by real application developers [12]. concurrency bugs from two orthogonal dimensions. Many
More generally, we believe that static analysis plus pre- interesting findings and implications are provided in our
vious deadlock detection algorithms will still be useful in study. We expect our study to deepen the understanding
detecting most Go blocking bugs caused by errors in shared of Go concurrency bugs and bring more attention to Go
memory synchornization. Static technologies can also help concurrency bugs.
Understanding Real-World Concurrency Bugs in Go ASPLOS’19, April 13–17, 2019, Providence, RI, USA

References [23] Google. The Go Programming Language – Release History. URL:

[1] Principles of designing Go APIs with channels. URL: https://golang.org/doc/devel/release.html.
https://inconshreveable.com/07-08-2014/principles-of-designing-go- [24] Rui Gu, Guoliang Jin, Linhai Song, Linjie Zhu, and Shan Lu. What
apis-with-channels/. change history tells us about thread synchronization. In Proceedings
[2] The Go Blog: Share Memory By Communicating. URL: of the 2015 10th Joint Meeting on Foundations of Software Engineering,
https://blog.golang.org/share-memory-by-communicating. Bergamo, Italy, August 2015.
[3] Sameer Ajmani. Advanced Go Concurrency Patterns. URL: [25] Haryadi S. Gunawi, Mingzhe Hao, Tanakorn Leesatapornwongsa,
https://talks.golang.org/2013/advconc.slide. Tiratat Patana-anake, Thanh Do, Jeffry Adityatama, Kurnia J. Eliazar,
[4] Joy Arulraj, Po-Chun Chang, Guoliang Jin, and Shan Lu. Production- Agung Laksono, Jeffrey F. Lukman, Vincentius Martin, and Anang D.
run software failure diagnosis via hardware performance counters. Satria. What bugs live in the cloud? a study of 3000+ issues in cloud
In Proceedings of the 18th International Conference on Architectural systems. In Proceedings of the ACM Symposium on Cloud Computing
Support for Programming Languages and Operating Systems (ASPLOS (SOCC’ 14), Seattle, Washington, USA, November 2014.
’13), Houston, Texas, USA, March 2013. [26] Hectane. Lightweight SMTP client written in Go. URL:
[5] Joy Arulraj, Guoliang Jin, and Shan Lu. Leveraging the short-term https://github.com/hectane.
memory of hardware to diagnose production-run software failures. [27] C. A. R. Hoare. Communicating Sequential Processes. Communications
In Proceedings of the 19th International Conference on Architectural of the ACM, 21(8):666-677, 1978.
Support for Programming Languages and Operating Systems (ASPLOS [28] Omar Inverso, Truc L. Nguyen, Bernd Fischer, Salvatore La Torre, and
’14), Salt Lake City, Utah, USA, March 2014. Gennaro Parlato. Lazy-cseq: A context-bounded model checking tool
[6] boltdb. An embedded key/value database for Go. URL: for multi-threaded c-programs. In 30th IEEE/ACM International Confer-
https://github.com/boltdb/bolt. ence on Automated Software Engineering (ASE ’15), Lincoln, Nebraska,
[7] Yan Cai and W. K. Chan. Magiclock: Scalable detection of potential USA, November 2015.
deadlocks in large-scale multithreaded programs. IEEE Transactions [29] Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan
on Software Engineering, 40(3):266-281, 2014. Lu. Understanding and detecting real-world performance bugs. In
[8] Lee Chew and David Lie. Kivati: Fast detection and prevention of Proceedings of the 33rd ACM SIGPLAN Conference on Programming
atomicity violations. In Proceedings of the 5th European Conference on Language Design and Implementation (PLDI’ 12), Beijing, China, June
Computer systems (EuroSys ’10), Paris, France, April 2010. 2012.
[9] Andy Chou, Junfeng Yang, Benjamin Chelf, Seth Hallem, and Dawson [30] Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben Liblit. Au-
Engler. An empirical study of operating systems errors. In Proceedings tomated atomicity-violation fixing. In Proceedings of the 32nd ACM
of the 18th ACM symposium on Operating Systems Principles (SOSP ’01), SIGPLAN Conference on Programming Language Design and Implemen-
Banff, Alberta, Canada, October 2001. tation (PLDI’ 11), San Jose, California, USA, June 2011.
[10] Cockroach. CockroachDB is a cloud-native SQL database for build- [31] Guoliang Jin, Aditya V. Thakur, Ben Liblit, and Shan Lu. Instrumenta-
ing global, scalable cloud services that survive disasters. URL: tion and sampling strategies for cooperative concurrency bug isolation.
https://github.com/cockroachdb/cockroach. In Proceedings of the ACM International Conference on Object oriented
[11] Russ Cox. Bell Labs and CSP Threads. URL: programming systems languages and applications (OOPSLA ’10), Reno/-
http://swtch.com/ rsc/thread/. Tahoe, Nevada, USA, October 2010.
[12] Graphql Developers. A thread-safe way of appending errors into [32] Guoliang Jin, Wei Zhang, Dongdong Deng, Ben Liblit, and Shan Lu.
Result.Errors. URL: https://github.com/graphql-go/graphql/pull/434. Automated concurrency-bug fixing. In Proceedings of the 10th USENIX
[13] Docker. Docker - Build, Ship, and Run Any App, Anywhere. URL: Conference on Operating Systems Design and Implementation (OSDI’12),
https://www.docker.com/. Hollywood, California, USA, October 2012.
[14] John Erickson, Madanlal Musuvathi, Sebastian Burckhardt, and Kirk [33] Pallavi Joshi, Chang-Seo Park, Koushik Sen, and Mayur Naik. A ran-
Olynyk. Effective data-race detection for the kernel. In Proceedings domized dynamic program analysis technique for detecting real dead-
of the 9th USENIX Conference on Operating Systems Design and Imple- locks. In Proceedings of the 30th ACM SIGPLAN Conference on Program-
mentation (OSDI ’10), Vancouver, BC, Canada, October 2010. ming Language Design and Implementation (PLDI ’09), Dublin, Ireland,
[15] ETCD. A distributed, reliable key-value store for the most critical data June 2009.
of a distributed system. URL: https://github.com/coreos/etcd. [34] Horatiu Jula, Daniel Tralamazza, Cristian Zamfir, and George Candea.
[16] Cormac Flanagan and Stephen N Freund. Atomizer: A dynamic atomic- Deadlock immunity: Enabling systems to defend against deadlocks. In
ity checker for multithreaded programs. In Proceedings of the 31st ACM Proceedings of the 8th USENIX Conference on Operating systems design
SIGPLAN-SIGACT symposium on Principles of programming languages and implementation (OSDI ’08), San Diego, California, USA, December
(POPL ’04), Venice, Italy, January 2004. 2008.
[17] Qi Gao, Wenbin Zhang, Zhezhe Chen, Mai Zheng, and Feng Qin. 2nd- [35] Daniel Kroening, Daniel Poetzl, Peter Schrammel, and Björn Wachter.
strike: Toward manifesting hidden concurrency typestate bugs. In Sound static deadlock analysis for c/pthreads. In 31st IEEE/ACM In-
Proceedings of the 16th International Conference on Architectural Sup- ternational Conference on Automated Software Engineering (ASE ’16),
port for Programming Languages and Operating Systems (ASPLOS ’11), Singapore, Singapore, September 2016.
Newport Beach, California, USA, March 2011. [36] Kubernetes. Production-Grade Container Orchestration. URL:
[18] GitHub. The fifteen most popular languages on GitHub. URL: https://kubernetes.io/.
https://octoverse.github.com/. [37] Leslie Lamport. Concurrent Reading and Writing. Communications of
[19] Google. A high performance, open source, general RPC framework that the ACM, 20(11):806-811, 1977.
puts mobile and HTTP/2 first. URL: https://github.com/grpc/grpc-go. [38] Julien Lange, Nicholas Ng, Bernardo Toninho, and Nobuko Yoshida.
[20] Google. Effective Go. URL: https://golang.org/doc/effective_go.html. Fencing off go: Liveness and safety for channel-based programming.
[21] Google. Effective Go: Concurrency. URL: In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of
https://golang.org/doc/effective_go.html#concurrency. Programming Languages (POPL ’17), Paris, France, January 2017.
[22] Google. RPC Benchmarking. URL: [39] Julien Lange, Nicholas Ng, Bernardo Toninho, and Nobuko Yoshida.
https://grpc.io/docs/guides/benchmarking.html. A static verification framework for message passing in go using be-
havioural types. In IEEE/ACM 40th International Conference on Software
Engineering (ICSE ’18), Gothenburg, Sweden, June 2018.
ASPLOS’19, April 13–17, 2019, Providence, RI, USA Tengfei et al.

[40] Tanakorn Leesatapornwongsa, Jeffrey F. Lukman, Shan Lu, and multithreaded programs. ACM Transactions on Computer Systems,
Haryadi S. Gunawi. Taxdc: A taxonomy of non-deterministic concur- 15(4):391-411, 1997.
rency bugs in datacenter distributed systems. In Proceedings of the [53] Konstantin Serebryany and Timur Iskhodzhanov. Threadsanitizer:
21th International Conference on Architectural Support for Programming Data race detection in practice. In Proceedings of the Workshop on
Languages and Operating Systems (ASPLOS ’16), Atlanta, Georgia, USA, Binary Instrumentation and Applications (WBIA ’09), New York, USA,
April 2016. December 2009.
[41] Zhenmin Li, Lin Tan, Xuanhui Wang, Shan Lu, Yuanyuan Zhou, and [54] Vivek K Shanbhag. Deadlock-detection in java-library using static-
Chengxiang Zhai. Have things changed now?: An empirical study of analysis. In 15th Asia-Pacific Software Engineering Conference (APSEC
bug characteristics in modern open source software. In Proceedings ’08), Beijing, China, December 2008.
of the 1st workshop on Architectural and system support for improving [55] Francesco Sorrentino. Picklock: A deadlock prediction approach under
software dependability (ASID ’06), San Jose, California, USA, October nested locking. In Proceedings of the 22nd International Symposium on
2006. Model Checking Software (SPIN ’15), Stellenbosch, South Africa, August
[42] Ziyi Lin, Darko Marinov, Hao Zhong, Yuting Chen, and Jianjun Zhao. 2015.
Jacontebe: A benchmark suite of real-world java concurrency bugs. [56] Kai Stadtmüller, Martin Sulzmann, and Peter" Thiemann. Static trace-
In 30th IEEE/ACM International Conference on Automated Software based deadlock analysis for synchronous mini-go. In 14th Asian Sym-
Engineering (ASE ’15), Lincoln, Nebraska, USA, November 2015. posium on Programming Languages and Systems (APLAS ’16), Hanoi,
[43] Haopeng Liu, Yuxi Chen, and Shan Lu. Understanding and generating Vietnam, November 2016.
high quality patches for concurrency bugs. In Proceedings of the 2016 [57] Jie Wang, Wensheng Dou, Yu Gao, Chushu Gao, Feng Qin, Kang Yin,
24th ACM SIGSOFT International Symposium on Foundations of Software and Jun Wei. A comprehensive study on real world concurrency bugs in
Engineering (FSE ’16), Seattle, Washington, USA, November 2016. node.js. In Proceedings of the 32nd IEEE/ACM International Conference
[44] Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and on Automated Software Engineering (ASE ’17), Urbana-Champaign,
Shan Lu. A study of linux file system evolution. In Proceedings of the Illinois, USA, October 2017.
11th USENIX Conference on File and Storage Technologies (FAST ’13), [58] Yin Wang, Terence Kelly, Manjunath Kudlur, Stéphane Lafortune, and
San Jose, California, USA, February 2013. Scott A Mahlke. Gadara: Dynamic deadlock avoidance for multi-
[45] Shan Lu, Soyeon Park, Eunsoo Seo, and Yuanyuan Zhou. Learning threaded programs. In Proceedings of the 8th USENIX Conference on
from mistakes – a comprehensive study of real world concurrency Operating systems design and implementation (OSDI ’08), San Diego,
bug characteristics. In Proceedings of the 13th International Conference California, USA, December 2008.
on Architectural Support for Programming Languages and Operating [59] Yin Wang, Stéphane Lafortune, Terence Kelly, Manjunath Kudlur, and
Systems (ASPLOS ’08), Seattle, Washington, USA, March 2008. Scott A. Mahlke. The theory of deadlock avoidance via discrete control.
[46] Shan Lu, Joseph Tucek, Feng Qin, and Yuanyuan Zhou. Avio: Detecting In Proceedings of the 36th annual ACM SIGPLAN-SIGACT symposium
atomicity violations via access interleaving invariants. In Proceedings on Principles of programming languages (POPL ’09), Savannah, Georgia,
of the 12th International Conference on Architectural Support for Pro- USA, January 2009.
gramming Languages and Operating Systems (ASPLOS ’06), San Jose, [60] Wikipedia. Go (programming language). URL:
California, USA, October 2006. https://en.wikipedia.org/wiki/Go_(programming_language).
[47] Brandon Lucia and Luis Ceze. Finding concurrency bugs with context- [61] Weiwei Xiong, Soyeon Park, Jiaqi Zhang, Yuanyuan Zhou, and
aware communication graphs. In Proceedings of the 42nd Annual Zhiqiang Ma. Ad hoc synchronization considered harmful. In Pro-
IEEE/ACM International Symposium on Microarchitecture (MICRO ’09), ceedings of the 9th USENIX Conference on Operating systems design
New York, USA, December 2009. and implementation (OSDI ’10), Vancouver, British Columbia, Canada,
[48] Kedar S. Namjoshi. Are concurrent programs that are easier to write October 2010.
also easier to check? In Workshop on Exploiting Concurrency Efficiently [62] Jie Yu and Satish Narayanasamy. A case for an interleaving constrained
and Correctly, 2008. shared-memory multi-processor. In Proceedings of the 36th annual
[49] Nicholas Ng and Nobuko Yoshida. Static deadlock detection for con- International symposium on Computer architecture (ISCA ’09), Austin,
current go by global session graph synthesis. In Proceedings of the 25th Texas, USA, June 2009.
International Conference on Compiler Construction (CC ’16), Barcelona, [63] Yuan Yu, Tom Rodeheffer, and Wei Chen. Racetrack: Efficient detection
Spain, March 2016. of data race conditions via adaptive tracking. In Proceedings of the 20th
[50] Rob Pike. Go Concurrency Patterns. URL: ACM symposium on Operating systems principles (SOSP ’05), Brighton,
https://talks.golang.org/2012/concurrency.slide. United Kingdom, October 2005.
[51] Dawson R. Engler and Ken Ashcraft. Racerx: Effective, static detection [64] Wei Zhang, Chong Sun, and Shan Lu. Conmem: detecting severe
of race conditions and deadlocks. In Proceedings of the 19th ACM concurrency bugs through an effect-oriented approach. In Proceedings
symposium on Operating systems principles (SOSP ’03), Bolton Landing, of the 15th International Conference on Architectural Support for Pro-
New York, USA, October 2003. gramming Languages and Operating Systems (ASPLOS ’10), Pittsburgh,
[52] Stefan Savage, Michael Burrows, Greg Nelson, Patrick Sobalvarro, Pennsylvania, USA, March 2010.
and Thomas Anderson. Eraser: A dynamic data race detector for