Fork

Mike Bostock · Feb 9, 2018

Code and data for humans. Founder @observablehq. Creator @d3. Former @nytgraphics.
Pronounced BOSS-tock.

Featured in Introduction

Introduction to Asynchronous Iteration

Image: Salvador Dali

Observable now supports asynchronous iteration, an exciting new feature of ES2018 that
was nalized last month by TC39! You’ll need a very recent browser to run this notebook,
such as Chrome 63+, Firefox 57+ or Safari Technology Preview 48.

Asynchronous programming (or async, for short) allows the browser to do multiple things
at the same time, such as to download les and perform complex calculations and
respond uidly to user interaction. Async iteration is a simple extension of this concept:
rather than a single async value, you have multiple async values and you want to step
through them one at a time.
t oug t e o e at a t e.

For instance, say you have several les you want to load into your notebook:

urls = [
"https://gist.githubusercontent.com/mbostock/2560c4da123c9d7bb5b2cb8da9f1f62f/raw/119a7
7019ee7b8b58f241a274081ad0cb1b105fb/2014-acs5-B01003-state-01.json",
"https://gist.githubusercontent.com/mbostock/2560c4da123c9d7bb5b2cb8da9f1f62f/raw/119a7
7019ee7b8b58f241a274081ad0cb1b105fb/2014-acs5-B01003-state-04.json",
"https://gist.githubusercontent.com/mbostock/2560c4da123c9d7bb5b2cb8da9f1f62f/raw/119a7
7019ee7b8b58f241a274081ad0cb1b105fb/2014-acs5-B01003-state-45.json"
]

▸Array(3) ["https://gist.githubusercontent.com/mbostock/2560c4…274081ad0cb1b105fb/2014-acs

These les contain population estimates by county for Alabama, Arizona and South
Carolina; the exact meaning isn’t important for this notebook. You can load these les in
parallel using array.map and Fetch (or here, a convenience wrapper for fetching and
parsing JSON from d3-fetch):

promises = urls.map(url => d3.json(url))

▸Array(3) [Promise, Promise, Promise]

datasets = Promise.all(promises)

▸Array(3) [Array(1182), Array(1527), Array(1104)]

You can wait for these fetches to nish using Promise.all. Either de ne the all-promise in
a separate cell for implicit await, as above, or use an explicit await, as below. Once the
les are loaded, you can use normal, synchronous iteration—a for loop—to iterate over
them and do something, like compute a sum.

{
let total = 0;
const datasets = await Promise.all(urls.map(url => d3.json(url)));
for (const dataset of datasets) {
for (let i = 1; i < dataset.length; ++i) {
total += +dataset[i][0];
}
}
return total;
}

16106467

But what if you don’t want to wait until all the data is loaded? What if you want to show
intermediate results as the les load? Observable uses generators to allow cells to yield a
value that changes over time. Wait for each le to load, then yield the incremental sum:
{
let total = 0;
for (const url of urls) {
const dataset = await d3.json(url);
for (let i = 1; i < dataset.length; ++i) {
total += +dataset[i][0];
yield total;
}
}
return total;
}

2423410

By using await and yield in the same cell, you’ve written an asynchronous generator.
Congrats! 🎉 But wait, there’s more!

The cell above loads the les serially; it waits for the sum to be computed for the previous
le before starting to fetch for the next le. If desired, you can instead start all the fetches
simultaneously, and then step through the les one-by-one to compute the sum. As long
as the server can handle the load, it’s typically faster to download multiple les
concurrently. And unlike Promise.all, we don’t have to wait for all the les to download—
just the next le in the list.

{
let total = 0;
for (const promise of urls.map(url => d3.json(url))) {
const dataset = await promise;
for (let i = 1; i < dataset.length; ++i) {
total += +dataset[i][0];
yield total;
}
}
return total;
}

2423410

This pattern—iterating over an array of promises, and waiting for each one to resolve
sequentially—can be expressed more succinctly using the new for-await-of loop:

{
let total = 0;
for await (const dataset of urls.map(url => d3.json(url))) {
for (let i = 1; i < dataset.length; ++i) {
total += +dataset[i][0];
yield total;
}
}
 return total;
}

2423410

So, async iteration is a way to consume (iterate over) async values, while an async
generator is a way to produce (yield) async values. The async generators above are implicit
—they are cells that both await and yield. But you can also make an explicit async
generator function in Observable using vanilla JavaScript:

async function* foo() {

for (let i = 1; i <= 1000; ++i) {
await Promises.delay(100);
yield i;
}
}

async ƒ* foo()

Calling foo returns an async generator that yields an incrementing number every 100
milliseconds. It starts at 1 and goes up to 1000. You can show these numbers in a
notebook cell by simply returning the generator:

foo()

100

Alternatively, you can read the yielded values “by hand” using the iterator protocol:

{
const generator = foo();
const iterator = generator[Symbol.asyncIterator]();
while (true) {
const {done, value} = await iterator.next();
if (done) return;
yield value;
}
}

100

Observable already supported asynchronous generators in a sense: you can also de ne a

(non-asynchronous) generator that yields promises. Observable waits until the previous
promise resolves before pulling the next one from the generator.

function* altFoo() {
for (let i = 0; i < 1000; ++i) {
yield Promises.delay(100, i);
}
}

ƒ* altFoo()

altFoo()

99

But async generators are a lot more exible that normal generators: you can both yield
and await! (Also, a normal generator must know synchronously whether it’s done,
whereas an asynchronous generator can decide whenever it wants to return.) For
example, async generators make it very easy to express animations as a sequence of
transitions between keyframes: rst yield the DOM element you want to display in the
notebook, and then repeatedly await for each transition to nish before starting the next.

{
const w = Math.min(640, width);
const h = 320;
const r = 20;
const t = 1500;
const svg = d3.select(DOM.svg(w, h));
const circle = svg.append("circle").attr("r", r).attr("cx", w / 4).attr("cy", h / 4);
yield svg.node();
while (true) {
circle.transition().duration(t).attr("cy", h * 3 / 4);
await Promises.delay(t);
circle.transition().duration(t).attr("cx", w * 3 / 4);
await Promises.delay(t);
circle.transition().duration(t).attr("cy", h * 1 / 4);
await Promises.delay(t);
circle.transition().duration(t).attr("cx", w * 1 / 4);
await Promises.delay(t);
}
}
And if transition.end returned a promise, this would be even easier!

▸Object {event: null, format: ƒ n(t), formatPrefix: ƒ formatPrefix(t, e), timeFormat: ƒ fo

Continue reading Introduction

PREV I O U S

How Saving Works

© 2018 Observable, Inc.

Terms Help Open Source