Asynchronous Programming

in Rust
CS3211 Parallel and Concurrent Programming
Outline
• Non-blocking I/O
• Asynchronous programming paradigm
• From Futures to async/.await

CS3211 L11 - Asynchronous Programming 2

Last week
• Safety – due to ownership and borrowing
• Memory safety
• Thread safety
• Rust code does not compile if it is not safe

CS3211 L11 - Asynchronous Programming 3

Recap: ownership and borrowing
• Ownership
• Shared (immutable) borrow
• Mutable borrow

CS3211 L11 - Asynchronous Programming 4

Closures
• Can be called like a function
• But they are not just simple functions
• Can be passed as parameters and returned to and from functions

CS3211 L11 - Asynchronous Programming 5

Overheads of threads
• Context switching cost: threads give up the rest of the CPU time slice
when blocking functions are called, and a switch happens
• Registers get restored, virtual address space gets switched, cache gets
stepped on, etc.
• Big cost for high-performance situations (servers) with multiple threads, each
handling a connection
• Memory overhead
• Each thread has its own stack space that needs to get managed by the OS
• Trying to have 5000 concurrent connections?
• 5000 threads = 5000 stack segments = 40GB at 8MB/stack!

CS3211 L11 - Asynchronous Programming 6

Time slice and threads (and a bit of networking)

CS3211 L11 - Asynchronous Programming 7

Time slice and threads

CS3211 L11 - Asynchronous Programming 8

Time slice and threads

CS3211 L11 - Asynchronous Programming 9

Mitigate the disadvantages of threads
• Context switches are expensive
• Use lightweight threads (green threads)
• Usually they need a runtime (like in Go)

• Is there a way we can have concurrency with less penalties?

• Non-blocking I/O

CS3211 L11 - Asynchronous Programming 10

Introducing non-blocking I/O
• For example, the read() sys call would block if there is more data to
be read but not available
• Thread gets pulled off the CPU and it cannot do anything else in the
meantime
• Instead, we could have read() return a special error value instead
of blocking
• If the client hasn’t sent anything yet, the thread can do other useful work, e.g.
reading from other descriptors
• Non-blocking I/O enables concurrency with one thread!

CS3211 L11 - Asynchronous Programming 11

Non-blocking I/O
• epoll is a kernel-provided mechanism that
notifies us of what file descriptors are ready for
I/O
• Scenario: we are a server having conversations
with multiple clients at the same time (multiple
fds)
• epoll notifies the thread when a client said
something
• read() from each of those file descriptors,
continue those conversations
• Rinse and repeat  event loop

CS3211 L11 - Asynchronous Programming 12

State management
Non-blocking I/O is nice in theory, but
• Actual applications:
managing state seems hard
• Was I waiting for the client to send me
something, or was I in the middle of sending
something to the client?
• What was the client asking for before I got
distracted?
• Charlie the Client asked me for her emails, but I
needed to get them from Dan the Database
• Now Dan the Database responded with some
info, but I can’t remember what I was supposed
to do with it
• Managing one connection in each thread is
easy because each conversation is an
independent train of thought

CS3211 L11 - Asynchronous Programming 13

State management
• Key problem: manage the state
associated with each conversation
• Imagine trying to cook 10 dishes at
the same time. Need to remember…
• How long each thing has been on the
stove
• How long things have been in the oven
• How long things have been marinating
for
• What the next step is for each dish

CS3211 L11 - Asynchronous Programming 14

State management
• Rust (and a handful of other languages)
take state management to the next level
• Futures allow us to keep track of in-
progress operations along with associated
state, in one package
• Think of a future as a helper friend that
oversees each operation, remembering any
associated state

CS3211 L11 - Asynchronous Programming 15

Futures visualized

CS3211 L11 - Asynchronous Programming 16

Futures visualized

CS3211 L11 - Asynchronous Programming 17

Futures visualized

CS3211 L11 - Asynchronous Programming 18

Futures visualized

CS3211 L11 - Asynchronous Programming 19

Futures visualized

CS3211 L11 - Asynchronous Programming 20

Futures visualized

CS3211 L11 - Asynchronous Programming 21

Futures visualized

CS3211 L11 - Asynchronous Programming 22

Futures visualized

CS3211 L11 - Asynchronous Programming 23

Futures visualized

CS3211 L11 - Asynchronous Programming 24

Futures visualized

CS3211 L11 - Asynchronous Programming 25

Futures visualized

CS3211 L11 - Asynchronous Programming 26

Futures visualized

CS3211 L11 - Asynchronous Programming 27

Futures visualized

CS3211 L11 - Asynchronous Programming 28

Futures
• Represents a value that will exist sometime in the future
• Calculation that hasn’t happened yet
• Is probably going to happen at some point
• Just keep asking
• Event loop = runtime for Futures
• Keeps polling Future until it is ready
• Runs your code whenever it can be run
• User-space scheduler for futures

CS3211 L11 - Asynchronous Programming 31

Rust: zero-cost abstraction
• Code that you can’t write better by hand
• Abstraction layers disappear at compile-time

• Example: iterators API is a zero-cost abstraction

• in release mode, the machine code
• One can’t tell if it was done using iterators or implemented by hand
• Compiler optimize the cache locality that would be difficult to implement by hand

CS3211 L11 - Asynchronous Programming 32

Futures.rs – zero-cost abstraction for futures
• The code in the binary has no allocation and no runtime overhead in
comparison to writing it by hand (non-assembly code )
• Building async state machines
• The Future trait
• Event reactor

CS3211 L11 - Asynchronous Programming 33

The Future trait
• The executor thread should call poll() on the future to start it off
• It will run code until it can no longer progress

CS3211 L11 - Asynchronous Programming 34

The Future trait
• The executor thread should call poll() on the future to start it off
• It will run code until it can no longer progress
• If the future is complete, returns Poll::Ready(T)
• If the future needs to wait for some event, returns Poll::Pending, and
allows the single thread to work on another future

CS3211 L11 - Asynchronous Programming 35

The Future trait
• The executor thread should call poll() on the future to start it off
• It will run code until it can no longer progress
• If the future is complete, returns Poll::Ready(T)
• If the future needs to wait for some event, returns Poll::Pending, and
allows the single thread to work on another future

CS3211 L11 - Asynchronous Programming 36

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls

CS3211 L11 - Asynchronous Programming 37

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 38

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 39

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 40

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 41

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 42

The Future trait
• When poll() is called, Context structure passed in
• Context includes a wake() function
• Called when future can make progress again
• Implemented internally using system calls)
• After wake() is called, the executor can use Context to see which
Future can be polled to make new progress

CS3211 L11 - Asynchronous Programming 43

Executors
• An executor loops over futures that can currently make progress
• Calls poll() on them to give them attention until they need to wait again
• When no futures can make progress, the executor goes to sleep until one or
more futures calls wake()
• A popular executor in the Rust ecosystem is Tokio
• Wraps around mio.rs and futures.rs
• Executors can be single-threaded or multi-threaded
• Running on multiple cores machine
• Futures can truly run in parallel
• Need to protect shared data using synchronization primitives (although the
ownership model kind of already forces you to do this anyway)

CS3211 L11 - Asynchronous Programming 44

Workflow of a future

CS3211 L11 - Asynchronous Programming 45

Futures should not block!
• If code within a future causes the thread to sleep, the executor
running that code is going to sleep
• Asynchronous code needs to use non-blocking versions of everything,
including mutexes, system calls that would normally block, or
anything.
• Executor runtimes like Tokio provide these non-blocking
implementations for your favorite synchronization primitives

CS3211 L11 - Asynchronous Programming 46

Composition with futures
• Pretty much no one implements futures manually (unless you’re a
low-level library implementor)
• Instead, futures are combined with various combinators
• Sequential
• Concurrently

CS3211 L11 - Asynchronous Programming 47

Not great ergonomics
• This code works
• It’s better than manually dealing with callbacks and state machines as you
would in C/C++ with interfaces like epoll
• But can we do better?
• The syntax is a little clunky… too much typing for Rust
• Code becomes messier as complexity increases
• Sharing mutable data (e.g. in local variables) can be painful: if there can only
be one mutable reference at a time, only one closure can touch that data

CS3211 L11 - Asynchronous Programming 48

Example of poor ergonomics
• Lines 21, 24, 25, 29:
asynchronous functions
returning Futures
• Lines 27: synchronous
(normal) function

• Line 21: complicated syntax

CS3211 L11 - Asynchronous Programming 49

Example of poor ergonomics
• Lines 26: why does
get_recipient need to
take a message?
• To make this chain of futures
work, since the next futures
need both the message and
recipient as input
• Like a pipeline

CS3211 L11 - Asynchronous Programming 50

Syntactic sugar
• Line 41: async function
returns Future
• Line 44, 45, 47: .await waits
for a future and gets its
value
• Line 46: normal function
usage

CS3211 L11 - Asynchronous Programming 51

Async/.await
• An async function is a function that returns a Future
• Any Futures used in the function are chained together by the compiler
• .await waits for a future and gets its value
• .await can only be called in an async fn or block
• Structure of the code is similar to what we are used to!
• The Rust compiler transforms this code into a Future with a poll()
method
• Just as efficient as what you could implement by hand

CS3211 L11 - Asynchronous Programming 52

Example: a simple server
• Lines 78, 81, 82: convert any blocking functions to asynchronous
versions
• They return Futures

CS3211 L11 - Asynchronous Programming 53

Example: asynchronous programming
• Lines 78, 81, 82: .await the Futures
• .await can only be used in an async function or block
• The compiler will complain if you forget .await

CS3211 L11 - Asynchronous Programming 54

Example: asynchronous programming
• main() now returns a Future
• Futures don’t actually do anything unless an executor executes them
• Need to run main() and submit the returned Future to the executor
• Line 73: #[tokio::main] macro submits the future to the executor

CS3211 L11 - Asynchronous Programming 55

Async functions generate/return futures
• If you run this function, it will not actually do any work with any
messages
• This is still a function and you can still run it…
• But its purpose is now to produce a future that does the stuff that
was written inside the function

CS3211 L11 - Asynchronous Programming 56

Async functions generate/return futures
• If you do not run this function, it will not do any work with any
messages
• The purpose is now to produce a future that does the stuff that was
written inside the function

CS3211 L11 - Asynchronous Programming 57

Async functions generate/return futures
• If you run this function, it will not actually do any work with any
messages
• This is still a function and you can still run it…
• But its purpose is now to produce a future that does the stuff that
was written inside the function

CS3211 L11 - Asynchronous Programming 58

State management
• There are 5 places where we might be paused, not actively executing:
• Before anything has happened yet (i.e. Future has been created but not yet
poll()ed)
• Await-ing for loadMessage
• Await-ing for get_recipient
• Await-ing for addToInbox
• Future has completed
• Use an enum to store the state for these possibilities

CS3211 L11 - Asynchronous Programming 59

State management
• Attempting to implement poll() for this Future
• Look at the current state and execute the appropriate code from the async fn

CS3211 L11 - Asynchronous Programming 60

Conceptually: state management

CS3211 L11 - Asynchronous Programming 61

Implications
• Async functions have no stack
• Sometimes called stackless coroutines
• The executor thread still has a stack (used to run normal/synchronous functions), but
it isn’t used to store state when switching between async tasks
• All state is self-contained in the generated Future
• No recursion
• The Future returned by an async function needs to have a fixed size known at
compile time
• Rust async functions are nearly optimal in terms of memory usage and
allocations
• Low overhead: the performance is as good as (or possibly better) what you could get
tuning everything by hand

CS3211 L11 - Asynchronous Programming 62

Usage of async code
• Taking a step back - problems to solve:
• Memory usage from having so many stacks
• Unnecessary context switching cost
• Async code makes sense when…
• You need an extremely high degree of concurrency
• Not as much reason to use async if you don’t have that many threads
• Work is primarily I/O bound
• Context switching is expensive only if you’re using a tiny fraction of the time slice
• If you’re doing a lot of work on the CPU for an extended period of time, you might
prevent the executor from running other tasks

CS3211 L11 - Asynchronous Programming 63

Similar tools in other languages
• Rust lets us write asynchronous code in the synchronous style that
we’re used to
• Javascript: very similar toolbox with Promises and async/await
• Involves much more dynamic memory allocation, not as efficient
• Golang: goroutines are asynchronous tasks, but unlike Rust they are
not stackless
• Resizable stacks - possible because Go is garbage collected
• Runtime knows where all pointers are and can reallocate memory
• C++20 just got stackless coroutines
• Still lots of sharp edges, may want to wait for more libraries to make this
easier to use

CS3211 L11 - Asynchronous Programming 64

Summary
• Never block in async code!
• Asynchronous tasks are cooperative (not preemptive)
• You can only use await in async functions
• Rust won’t let you write async functions in traits
• You can use a crate called async-trait though
• References
• Lectures 18 and 19 from Stanford’s CS110L:
https://reberhardt.com/cs110l/spring-2021/
• Katharina Fey, 2018: https://www.youtube.com/watch?v=j0SIcN-Y-LA
• https://www.youtube.com/watch?v=g2PmsO_C4eY
CS3211 L11 - Asynchronous Programming 65