Mastering Arduinojson 6: Efficient Json Serialization For Embedded C++
Mastering Arduinojson 6: Efficient Json Serialization For Embedded C++
Mastering Arduinojson 6: Efficient Json Serialization For Embedded C++
CREATOR OF ARDUINOJSON
Mastering ArduinoJson 6
Efficient JSON serialization for embedded C++
THIRD EDITION
Contents
Contents iv
1 Introduction 1
1.1 About this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Code samples . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 What’s new in the third edition . . . . . . . . . . . . . . . . . . 3
1.2 Introduction to JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 What is JSON? . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 What is serialization? . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.3 What can you do with JSON? . . . . . . . . . . . . . . . . . . 5
1.2.4 History of JSON . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.5 Why is JSON so popular? . . . . . . . . . . . . . . . . . . . . . 8
1.2.6 The JSON syntax . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.7 Binary data in JSON . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.8 Comments in JSON . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3 Introduction to ArduinoJson . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.1 What ArduinoJson is . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.2 What ArduinoJson is not . . . . . . . . . . . . . . . . . . . . . 14
1.3.3 What makes ArduinoJson different? . . . . . . . . . . . . . . . 15
1.3.4 Does size matter? . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.3.5 What are the alternatives to ArduinoJson? . . . . . . . . . . . . 18
1.3.6 How to install ArduinoJson . . . . . . . . . . . . . . . . . . . . 20
1.3.7 The examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.3.3 Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.4 Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4.1 What is a pointer? . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4.2 Dereferencing a pointer . . . . . . . . . . . . . . . . . . . . . . 38
2.4.3 Pointers and arrays . . . . . . . . . . . . . . . . . . . . . . . . 39
2.4.4 Taking the address of a variable . . . . . . . . . . . . . . . . . 40
2.4.5 Pointer to class and struct . . . . . . . . . . . . . . . . . . . 40
2.4.6 Pointer to constant . . . . . . . . . . . . . . . . . . . . . . . . 41
2.4.7 The null pointer . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.4.8 Why use pointers? . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.5 Memory management . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.5.1 malloc() and free() . . . . . . . . . . . . . . . . . . . . . . . . 45
2.5.2 new and delete . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.5.3 Smart pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.5.4 RAII . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.6.1 What is a reference? . . . . . . . . . . . . . . . . . . . . . . . 49
2.6.2 Differences with pointers . . . . . . . . . . . . . . . . . . . . . 49
2.6.3 Reference to constant . . . . . . . . . . . . . . . . . . . . . . . 50
2.6.4 Rules of references . . . . . . . . . . . . . . . . . . . . . . . . 51
2.6.5 Common problems . . . . . . . . . . . . . . . . . . . . . . . . 51
2.6.6 Usage for references . . . . . . . . . . . . . . . . . . . . . . . . 52
2.7 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.7.1 How are the strings stored? . . . . . . . . . . . . . . . . . . . . 53
2.7.2 String literals in RAM . . . . . . . . . . . . . . . . . . . . . . . 53
2.7.3 String literals in Flash . . . . . . . . . . . . . . . . . . . . . . . 54
2.7.4 Pointer to the “globals” section . . . . . . . . . . . . . . . . . . 56
2.7.5 Mutable string in “globals” . . . . . . . . . . . . . . . . . . . . 56
2.7.6 A copy in the stack . . . . . . . . . . . . . . . . . . . . . . . . 57
2.7.7 A copy in the heap . . . . . . . . . . . . . . . . . . . . . . . . 58
2.7.8 A word about the String class . . . . . . . . . . . . . . . . . . 59
2.7.9 Pass strings to functions . . . . . . . . . . . . . . . . . . . . . 60
2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7 Troubleshooting 242
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
7.2 Program crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
7.2.1 Undefined Behaviors . . . . . . . . . . . . . . . . . . . . . . . . 244
7.2.2 A bug in ArduinoJson? . . . . . . . . . . . . . . . . . . . . . . 244
7.2.3 Null string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
7.2.4 Use after free . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
7.2.5 Return of stack variable address . . . . . . . . . . . . . . . . . 247
7.2.6 Buffer overflow . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.2.7 Stack overflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
7.2.8 How to diagnose these bugs? . . . . . . . . . . . . . . . . . . . 250
7.2.9 How to prevent these bugs? . . . . . . . . . . . . . . . . . . . . 253
7.3 Deserialization issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
7.3.1 EmptyInput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
7.3.2 IncompleteInput . . . . . . . . . . . . . . . . . . . . . . . . . . 256
7.3.3 InvalidInput . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
7.3.4 NoMemory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Contents xi
9 Conclusion 317
Index 318
Chapter 3
Deserialize with ArduinoJson
It is not the language that makes programs appear simple. It is the pro-
grammer that makes the language appear simple!
– Robert C. Martin, Clean Code: A Handbook of Agile Software
Craftsmanship
Chapter 3 Deserialize with ArduinoJson 66
Now that you’re familiar with JSON and C++, we’re going to learn how to use Arduino-
Json. This chapter explains everything there is to know about deserialization. As we’ve
seen, deserialization is the process of converting a sequence of bytes into a memory
representation. In our case, it means converting a JSON document to a hierarchy of
C++ structures and arrays.
In this chapter, we’ll use a JSON response from GitHub’s
API as an example. As you already know, GitHub is a
hosting service for source code; what you may not know,
however, is that GitHub provides a very powerful API that
allows you to interact with the platform.
We could do many things with GitHub’s API, but in this
chapter, we’ll only focus on a small part. We’ll get your ten
most popular repositories and display their names, numbers
of stars, and numbers of opened issues.
There are several versions of GitHub’s API; we’ll use the
latest one: the GraphQL API (or v4). We’ll use this one
because it allows us to get all the information we need
with only one query. It also returns much smaller responses
compared to v3, which is appreciable for embedded software.
If you want to run the example, you’ll need a user account on GitHub and a personal
access token. Don’t worry; we’ll see that later.
Because GitHub only allows secure connections, we need a microcontroller that supports
HTTPS. We’ll use the ESP8266 with the ESP8266HTTPClient as an example. If you
want to use ArduinoJson with EthernetClient, WiFiClient, or WiFiClientSecure, check
out the case studies in the last chapter.
Now that you know where we are going, we’ll back up a few steps and start with a
basic example. Then, we’ll progressively learn new things so that we’ll finally be able
to interact with GitHub by the end of the chapter.
Chapter 3 Deserialize with ArduinoJson 67
We’ll begin this tutorial with the simplest situation: a JSON document in memory.
More precisely, our JSON document resides in the stack in a writable location. This
fact is going to matter, as we will see later.
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 5246
},
"issues": {
"totalCount": 15
}
}
As you see, it’s a JSON object that contains two nested objects. It includes the name
of the repository, the number of stars, and the number of open issues.
In the previous chapter, we saw that this code creates a duplication of the string in
the stack. We know it’s a code smell in production code, but it’s a good example for
learning. This unusual construction creates a writable (i.e., not read-only) input string,
which is essential for your first contact with ArduinoJson.
Chapter 3 Deserialize with ArduinoJson 68
As we saw in the introduction, one of the unique features of ArduinoJson is its fixed
memory allocation strategy.
Here is how it works:
1. First, you create a JsonDocument to reserve a specified amount of memory.
2. Then, you deserialize the JSON document.
3. Finally, you destroy the JsonDocument, which releases the reserved memory.
The memory of the JsonDocument can be either in the stack or in the heap. The location
depends on the derived class you choose. If you use a StaticJsonDocument, it will be in
the stack; if you use a DynamicJsonDocument, it will be in the heap.
A JsonDocument is responsible for reserving and releasing the memory used by Arduino-
Json. It is an instance of the RAII idiom that we saw in the previous chapter.
When you create a JsonDocument, you must specify its capacity in bytes.
In the case of DynamicJsonDocument, you set the capacity via a constructor argument:
DynamicJsonDocument doc(capacity);
Since it’s a constructor parameter, you can use a regular variable whose value can
change at run-time.
In the case of a StaticJsonDocument, you set the capacity via a template parameter:
StaticJsonDocument<capacity> doc;
Chapter 3 Deserialize with ArduinoJson 69
As it’s a template parameter, you cannot use a variable. Instead, you must use a
constant expression, which means that the value must be computed at compile-time.
As we said in the previous chapter, the compiler manages the stack, so it needs to
know the size of each variable when it compiles the program; that’s why we must use
a constant expression here.
Now comes a tricky question for every new user of ArduinoJson: what should be the
capacity of my JsonDocument?
To answer this question, you need to know what ArduinoJson stores in the JsonDocument.
ArduinoJson needs to store a data structure that mirrors the hierarchy of objects in the
JSON document. In other words, the JsonDocument contains objects which relate to one
another the same way they do in the JSON document.
Therefore, the capacity of the JsonDocument highly depends on the complexity of the
JSON document. If it’s just one object with few members, like our example, a few
dozens of bytes are enough. If it’s a massive JSON document, like OpenWeatherMap’s
response, up to a hundred kilobytes are needed.
ArduinoJson provides macros for computing precisely the capacity of the JsonDocument.
The macro to compute the size of an object is JSON_OBJECT_SIZE(). It takes one argu-
ment: the number of members in the object.
Here is how to compute the capacity for our sample document:
Since our JsonDocument is small, we can keep it in the stack. Using the stack, we reduce
the executable size and improve the performance because we avoid the overhead due
to the management of the heap.
Here is our program so far:
Of course, if the JsonDocument were bigger, it would make sense to move it to the heap.
We’ll do that later.
Now that the JsonDocument is ready, we can parse the input with deserializeJson():
if (err) {
Serial.print(F("deserializeJson() failed with code "));
Serial.println(err.f_str());
}
In the “Troubleshooting” chapter, we’ll look at each error code and see what can cause
the error.
Chapter 3 Deserialize with ArduinoJson 72
There are multiple ways to extract the values from a JsonDocument; let’s start with the
simplest:
Not everyone likes implicit casts, mainly because they mess with overload resolution,
template parameter type deduction, and the auto keyword. That’s why ArduinoJson
offers an alternative syntax with explicit type conversion.
Implicit or explicit?
We saw two different syntaxes to do the same thing. They are all equivalent
and lead to the same executable.
I prefer the implicit version because it allows using the “or” operator, as
we’ll see. I use the explicit version only to solve ambiguities.
We saw how to extract values from an object, but we didn’t do error checking. Let’s
see what happens when a value is missing.
When you try to extract a value that is not present in the document, ArduinoJson
returns a default value. This value depends on the requested type:
The two last lines (JsonArray and JsonObject) happen when you extract a nested array
or object, we’ll see that in a later section.
No exceptions
ArduinoJson never throws exceptions. Exceptions are an excellent C++ fea-
ture, but they produce large executables, which is unacceptable for micro-
controllers.
Chapter 3 Deserialize with ArduinoJson 74
Sometimes, the default value from the table above is not what you want. In this
situation, you can use the operator | to change the default value. I call it the “or”
operator because it provides a replacement when the value is missing or incompatible.
Here is an example:
This feature is handy to specify default configuration values, like in the snippet above,
but it is even more helpful to prevent a null string from propagating.
Here is an example:
strlcpy(), a function that copies a source string to a destination string, crashes if the
source is null. Without the operator |, we would have to use the following code:
char hostname[32];
const char* configHostname = config["hostname"];
if (configHostname != nullptr)
strlcpy(hostname, configHostname, 32);
else
strcpy(hostname, "arduinojson.org");
We’ll see a complete example that uses this syntax in the case studies.
Chapter 3 Deserialize with ArduinoJson 75
In the previous section, we extracted the values from an object that we knew in advance.
Indeed, we knew that the JSON object had three members: a string named “name,” a
nested object named “stargazers,” and another nested object named “issues.” In this
section, we’ll see how to inspect an unknown object.
Now that we have a JsonObject, we can look at all the keys and their associated values.
In ArduinoJson, a key-value pair is represented by the type JsonPair.
We can enumerate all pairs with a simple for loop:
To know the actual type of the value in a JsonVariant, you need to call
JsonVariant::is<T>(), where T is the type you want to test.
// Is it a string?
if (p.value().is<const char*>()) {
// Yes!
// We can get the value via implicit cast:
const char* s = p.value();
// Or, via explicit method call:
auto s = p.value().as<const char*>();
}
If you use this with our sample document, you’ll see that only the member “name”
contains a string. The two others are objects, as is<JsonObject>() would confirm.
There are a limited number of types that a variant can use: boolean, integer, float,
string, array, and object. However, different C++ types can store the same JSON type;
for example, a JSON integer could be a short, an int, or a long in the C++ code.
The following table shows all the C++ types you can use as a parameter for
JsonVariant::is<T>() and JsonVariant::as<T>().
More on arduinojson.org
The complete list of types that you can use as a parameter for
JsonVariant::is<T>() can be found in the API Reference.
Chapter 3 Deserialize with ArduinoJson 78
If you have an object and want to know whether a key is present or not, you can call
JsonObject::containsKey().
Here is an example:
However, I don’t recommend using this function because you can avoid it most of the
time.
Here is an example where we can avoid containsKey():
The code above is not horrible, but it can be simplified and optimized if we just remove
the call to containsKey():
This code is faster and smaller because it only looks for the key “error” once, whereas
the previous code did it twice.
Chapter 3 Deserialize with ArduinoJson 79
We’ve seen how to parse a JSON object from GitHub’s response; it’s time to move up
a notch by parsing an array of objects. Indeed, our goal is to display the top 10 of
your repositories, so there will be up to 10 objects in the response. In this section, we’ll
suppose that there are only two repositories, but you and I know that it will be more in
the end.
Here is the new sample JSON document:
[
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 5246
},
"issues": {
"totalCount": 15
}
},
{
"name": "pdfium-binaries",
"stargazers": {
"totalCount": 249
},
"issues": {
"totalCount": 12
}
}
]
Let’s deserialize this array. You should now be familiar with the process:
Chapter 3 Deserialize with ArduinoJson 80
// Parse succeeded?
if (err) {
Serial.print(F("deserializeJson() returned "));
Serial.println(err.f_str());
return;
}
As said earlier, a hard-coded input like that would never happen in production code,
but it’s a good step for your learning process.
You can see that the expression for computing the capacity of the JsonDocument is quite
complicated:
• There is one array of two elements: JSON_ARRAY_SIZE(2)
• In this array, there are two objects with three members: 2*JSON_OBJECT_SIZE(3)
• In each object, there are two objects with one member: 4*JSON_OBJECT_SIZE(1)
Chapter 3 Deserialize with ArduinoJson 81
For complex JSON documents, the expression to compute the capacity of the
JsonDocument becomes impossible to write by hand. I did it above so that you un-
derstand the process, but in practice, we use a tool to do that.
This tool is the “ArduinoJson Assistant.” You can use it online at arduinojson.org/
assistant.
Don’t worry; the Assistant respects your privacy: it computes the expression locally in
the browser; it doesn’t send your JSON document to a web service.
Chapter 3 Deserialize with ArduinoJson 83
The process of extracting the values from an array is very similar to the one for objects.
The only difference is that arrays are indexed by an integer, whereas objects are indexed
by a string.
To get access to the repository information, we need to get the JsonObject from the
JsonDocument, except that, this time, we’ll pass an integer to the subscript opera-
tor ([]).
Of course, we could have inlined the repo0 variable (i.e., write doc[0]["name"] each
time), but it would cost an extra lookup for each access to the object.
It may not be obvious, but the program above uses implicit casts. Indeed, the subscript
operator ([]) returns a JsonVariant that is implicitly converted to a JsonObject.
Again, some programmers don’t like implicit casts, that is why ArduinoJson offers an
alternative syntax with as<T>(). For example:
All of this should sound very familiar because we’ve seen the same for objects.
Chapter 3 Deserialize with ArduinoJson 84
When we learned how to extract values from an object, we saw that if a member is
missing, a default value is returned (for example, 0 for an int). Similarly, ArduinoJson
returns a default value when you use an index that is out of the range of an array.
Let’s see what happens in our case:
The index 666 doesn’t exist in the array, so a special value is returned: a null JsonObject.
Remember that JsonObject is a reference to an object stored in the JsonDocument. In
this case, there is no object in the JsonDocument, so the JsonObject points to nothing:
it’s a null reference.
You can test if a reference is null by calling isNull():
if (repo666.isNull()) ...
Alternatively, you can compare to nullptr (but not NULL!), like so:
A null JsonObject evaluates to false, so you can check that it’s not null like so:
if (repo666) ...
A null JsonObject looks like an empty object, except that you cannot modify it. You
can safely call any function of a null JsonObject; it simply ignores the call and returns
a default value. Here is an example:
In the previous section, our example was very straightforward because we knew that the
JSON array had precisely two elements, and we knew the content of these elements. In
this section, we’ll see what tools are available when you don’t know the content of the
array.
Do you remember what we did when we wanted to enumerate the key-value pairs of an
object? We began by calling JsonDocument::as<JsonObject>() to get a reference to the
root object.
Similarly, if we want to enumerate all the elements of an array, the first thing we have
to do is to get a reference to it:
Again, JsonArray is a reference to an array stored in the JsonDocument; it’s not a copy of
the array. When you apply changes to the JsonArray, they affect the JsonDocument.
If you know absolutely nothing about the input (which is strange), you need to determine
a memory budget allowed for parsing the input. For example, you could decide that
10KB of heap memory is the maximum you accept to spend on JSON parsing.
This constraint looks terrible at first, especially if you’re a desktop or server application
developer; but, once you think about it, it makes complete sense. Indeed, your program
will run in a loop on dedicated hardware. Since the hardware doesn’t change, the
amount of memory is always the same. Having an elastic capacity would just produce
a larger and slower program with no additional value; it would also increase the heap
fragmentation, which we must avoid at all costs.
However, most of the time, you know a lot about your JSON document. Indeed, there
are usually a few possible variations in the input. For example, an array could have
between zero and four elements, or an object could have an optional member. In that
Chapter 3 Deserialize with ArduinoJson 87
case, use the ArduinoJson Assistant to compute the size of each variant and pick the
biggest.
The first thing you want to know about an array is the number of elements it contains.
This is the role of JsonArray::size():
As the name may be confusing, let me clarify: JsonArray::size() returns the number
of elements, not the memory consumption. If you want to know how many bytes of
memory are used, call JsonDocument::memoryUsage():
Note that there is also a JsonObject::size() that returns the number of key-value pairs
in an object, but it’s rarely helpful.
3.7.4 Iteration
Now that you have the size of the array, you probably want to write the following
code:
The code above works but is terribly slow. Indeed, ArduinoJson stores arrays as linked
lists, so accessing an element at a random location costs O(n); in other words, it takes
n iterations to get to the nth element. Moreover, the value of JsonArray::size() is not
cached, so it needs to walk the linked list too.
Chapter 3 Deserialize with ArduinoJson 88
That’s why it is essential to avoid arr[i] and arr.size() in a loop. Instead, you should
use the iteration feature of JsonArray, like so:
With this syntax, the internal linked list is walked only once, and it is as fast as it gets.
I used a JsonObject in the loop because I knew that the array contains objects. If it’s
not your case, you can use a JsonVariant instead.
We test the type of array elements the same way we did for object members: using
JsonVariant::is<T>().
Here is an example:
// Same in a loop
for (JsonVariant elem : arr) {
// Is the current element an object?
if (elem.is<JsonObject>()) {
JsonObject obj = elem;
// ...
}
}
There is nothing new here, as it’s exactly what we saw for object members.
Chapter 3 Deserialize with ArduinoJson 90
3.8.1 Definition
At the beginning of this chapter, we saw how to parse a JSON document from a
writable source. Indeed, the input variable was a char[] in the stack, and therefore, it
was writable. I told you that this fact would matter, and it’s time to explain.
ArduinoJson behaves differently with writable inputs and read-only inputs.
When the argument passed to deserializeJson() is of type char* or char[], ArduinoJson
uses a mode called “zero-copy.” It has this name because the parser never makes any
copy of the input; instead, it stores pointers pointing inside the input buffer.
In the zero-copy mode, when a program requests the content of a string member,
ArduinoJson returns a pointer to the beginning of the string in the input buffer. To
make it possible, ArduinoJson inserts null-terminators at the end of each string; it is
the reason why this mode requires the input to be writable.
3.8.2 An example
To illustrate how the zero-copy mode works, let’s have a look at a concrete example.
Suppose we have a JSON document that is just an array containing two strings:
["hip","hop"]
Chapter 3 Deserialize with ArduinoJson 91
And let’s says that the variable is a char[] at address 0x200 in memory:
After parsing the input, when the program requests the value of the first element,
ArduinoJson returns a pointer whose address is 0x202, which is the location of the
string in the input buffer:
deserializeJson(doc, input);
We naturally expect hip to be "hip" and not "hip\",\"hop\"]"; that’s why ArduinoJson
adds a null-terminator after the first p. Similarly, we expect hop to be "hop" and not
"hop\"]", so it adds a second terminator.
'[' '"' 'h' 'i' 'p' '"' ',' '"' 'h' 'o' 'p' '"' ']' 0
0x200
deserializeJson()
'[' '"' 'h' 'i' 'p' 0 ',' '"' 'h' 'o' 'p' 0 ']' 0
0x202 0x208
Adding null-terminators is not the only thing the parser modifies in the input buffer.
It also replaces escaped character sequences, like \n, by their corresponding ASCII
characters.
I hope this explanation gives you a clear understanding of what the zero-copy mode is
and why the input is modified. It is a bit of a simplified view, but the actual code is
very similar.
Chapter 3 Deserialize with ArduinoJson 92
As we saw, in the zero-copy mode, ArduinoJson returns pointers to the input buffer. This
can only work if the input buffer is still in memory when the pointer is dereferenced.
If a program dereferences the pointer after the destruction of the input buffer, it is very
likely to crash instantly, but it could also work for a while and crash later, or it could
have nasty side effects. In the C++ jargon, this is what we call an “Undefined Behavior”;
we’ll talk about that in Troubleshooting.
Here is an example:
// Declare a pointer
const char *hip;
// New scope
{
// Declare the input in the scope
char input[] = "[\"hip\",\"hop\"]";
// Parse input
deserializeJson(doc, input);
JsonArray arr = doc.as<JsonArray>();
// Save a pointer
hip = arr[0];
}
// input is destructed now
We saw how ArduinoJson behaves with a writable input and how the zero-copy mode
works. It’s time to see what happens when the input is read-only.
Let’s go back to our previous example except that, this time, we change its type from
char[] to const char*:
Previously, we had the whole string duplicated in the stack, but it’s not the case anymore.
Instead, the stack only contains the pointer input pointing to the beginning of the
string.
In the zero-copy mode, ArduinoJson stores pointers pointing inside the input buffer.
We saw that it has to replace some characters of the input with null-terminators.
With a read-only input, ArduinoJson cannot do that anymore, so it needs to make copies
of "hip" and "hop". Where do you think the copies would go? In the JsonDocument, of
course!
In this mode, the JsonDocument holds a copy of each string, so we need to increase its
capacity. Let’s do the computation for our example:
1. We still need to store an object with two elements, that’s JSON_ARRAY_SIZE(2).
2. We have to make a copy of the string "hip", that’s 4 bytes, including the null-
terminator.
3. We also need to copy the string "hop", that’s 4 bytes too.
The required capacity is:
In practice, you should not use the exact length of the strings. It’s safer to add a bit of
slack in case the input changes. My advice is to add 10% to the longest possible string,
which gives a reasonable margin.
3.9.3 Practice
Apart from the capacity of the JsonDocument, we don’t need to change anything to the
program.
Here is the complete hip-hop example with a read-only input:
// A read-only input
const char* input = "[\"hip\",\"hop\"]";
StaticJsonDocument<capacity> doc;
const char* is not the sole read-only input that ArduinoJson supports. For example,
you can also use a String:
It’s also possible to use a Flash string, but there is one caveat. As we said in the C++
course, ArduinoJson needs a way to figure out if the input string is in RAM or Flash.
To do that, it expects a Flash string to have the type const __FlashStringHelper*. If
you declare a char[] PROGMEM, ArduinoJson will not consider it as Flash string, unless
you cast it to const __FlashStringHelper*.
Alternatively, you can use the F() macro, which casts the pointer to the right type:
As we saw in the previous chapter, using F() and PROGMEM strings only makes sense on
Harvard architectures, such as AVR and ESP8266.
In the next section, we’ll see another kind of read-only input: streams.
Chapter 3 Deserialize with ArduinoJson 97
In the Arduino jargon, a stream is a volatile source of data, like the serial port or
a TCP connection. Contrary to a memory buffer, which allows reading any bytes at
any location (after all, that’s what the acronym “RAM” means), a stream only allows
reading one byte at a time and cannot rewind.
The Stream abstract class materializes this concept. Here are examples of classes derived
from Stream:
std::istream
b In the C++ Standard Library, an input stream is represented by the class
std::istream.
ArduinoJson can use both Stream and std::istream.
As an example, we’ll create a program that reads a JSON file stored on an SD card.
We suppose that this file contains the array we used as an example earlier.
The program will just read the file and print the information for each repository.
Chapter 3 Deserialize with ArduinoJson 98
// Open file
File file = SD.open("repos.txt");
Now is the time to parse the actual data coming from GitHub’s API!
As I said, we need a microcontroller that supports HTTPS, so we’ll use an ESP8266
with the library “ESP8266HTTPClient.” Don’t worry if you don’t have a compatible
board; we’ll see other configurations in the case studies.
Chapter 3 Deserialize with ArduinoJson 99
Access token
Before using this API, you need a GitHub account and a “personal access token.” This
token grants access to the GitHub API from your program; we might also call it an
“API key.” To create it, open GitHub in your browser and follow these steps:
1. Go to your personal settings.
2. Go in “Developer settings.”
3. Go in “Personal access token.”
4. Click on “Generate a new token.”
5. Enter a name, like “ArduinoJson tutorial.”
6. Check the scopes (i.e., the permissions); we only need “public_repo.”
7. Click on “Generate token.”
8. GitHub shows the token.
You can see each step in the picture below:
Chapter 3 Deserialize with ArduinoJson 100
GitHub won’t show the token again, so don’t waste any second and write it in the source
code:
With this token, our program can authenticate with GitHub’s API. All we need to do is
to add the following HTTP header to each request:
Certificate validation
Because I don’t want to make this example more complicated than necessary, I’ll disable
the SSL certificate validation, like so:
WiFiClientSecure client;
client.setInsecure();
What could be the consequence? Since the program doesn’t verify the certificate, it
cannot be sure of the server’s authenticity, so it could connect to a rogue serve that
pretends to be api.github.com. This is indeed a serious security breach because the
program would send your Personal Access Token to the rogue server. Fortunately, this
token has minimal permissions: it only provides access to public information. However,
in a different project, the consequences could be disastrous.
If your project presents any security or privacy risk, you must enable SSL certificate
validation. WiFiClientSecure provides several validation methods. For a simple solution,
use setFingerprint(), but you’ll have to update the fingerprint frequently. For a more
robust solution, use setTrustAnchors() and make sure your clock is set to the current
time and date.
The request
To interact with the new GraphQL API, we need to send a POST request (instead of the
more common GET request) to the URL https://api.github.com/graphql.
The body of the POST request is a JSON object that contains one string named “query.”
This string contains a GraphQL query. For example, if we want to get the name of the
Chapter 3 Deserialize with ArduinoJson 101
authenticated user, we need to send the following JSON document in the body of the
request:
{
"query": "{viewer{name}}"
}
The GraphQL syntax and the details of GitHub’s API are obviously out of the scope of
this book, so I’ll simply say that a GraphQL query allows you to select the information
you want within the universe of information that the API exposes.
In our case, we want to retrieve the names, numbers of stars, and numbers of opened is-
sues of your ten most popular repositories. Here is the corresponding GraphQL query:
{
viewer {
name
repositories(ownerAffiliations: OWNER,
orderBy: {
direction: DESC,
field: STARGAZERS
},
first: 10) {
nodes {
name
stargazers {
totalCount
}
issues(states: OPEN) {
totalCount
}
}
}
}
}
To find the correct query, I used the GraphQL API Explorer. With this tool, you can test
GraphQL queries in your browser. You can find it in GitHub’s API documentation.
We’ll reduce this query to a single line to save some space and bandwidth; then, we’ll
put it in the “query” string in the JSON object. Since we haven’t talked about JSON
Chapter 3 Deserialize with ArduinoJson 102
HTTPClient http;
http.begin(client, "https://api.github.com/graphql");
http.addHeader("Authorization", "bearer " GITHUB_TOKEN));
http.POST("{\"query\":\"{viewer{name,repositories(ownerAffiliations:...");
The response
As you see, we call getStream() to get the internal stream (we could have used client
directly). Unfortunately, when we do that, we bypass the part of ESP8266HTTPClient
that handles chunked transfer encoding. To make sure GitHub doesn’t return a chunked
response, we must set the protocol to HTTP 1.0:
Because the protocol version is part of the request, we must call useHTTP10() before
calling POST().
Now that we have the stream, we can pass it to deserializeJson():
Here, we used a DynamicJsonDocument because it is too big for the stack. As usual,
I used the ArduinoJson Assistant to compute the capacity.
Chapter 3 Deserialize with ArduinoJson 103
The body contains the JSON document that we want to deserialize. It’s a little more
complicated than what we saw earlier. Indeed, the JSON array is not at the root but
under data.viewer.repositories.nodes, as you can see below:
{
"data": {
"viewer": {
"name": "Benoît Blanchon",
"repositories": {
"nodes": [
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 5246
},
"issues": {
"totalCount": 15
}
},
{
"name": "pdfium-binaries",
"stargazers": {
"totalCount": 249
},
"issues": {
"totalCount": 12
}
},
...
]
}
}
}
}
So, compared to what we saw earlier, the only difference is that we’ll have to walk
several objects before getting the reference to the array. The following line will do:
The code
// Disconnect
http.end();
Chapter 3 Deserialize with ArduinoJson 105
If all works well, this program should print something like so:
You can find the complete source code of this example in the GitHub folder in the zip file
provided with the book. Compared to what is shown above, the source code handles the
connection to the WiFi network, check errors, and uses Flash strings when possible.
Chapter 3 Deserialize with ArduinoJson 106
3.11 Summary
In this chapter, we learned how to deserialize a JSON input with ArduinoJson. Here
are the key points to remember:
• JsonDocument:
– JsonDocument stores the memory representation of the document.
– StaticJsonDocument is a JsonDocument that resides in the stack.
– DynamicJsonDocument is a JsonDocument that resides in the heap.
– JsonDocument has a fixed capacity that you set at construction.
– You can use the ArduinoJson Assistant to compute the capacity.
• JsonArray and JsonObject:
– You can extract the value directly from the JsonDocument as long as there is
no ambiguity.
– To solve an ambiguity, you must call as<JsonArray>() or as<JsonObject>().
– JsonArray and JsonObject are references, not copies.
– The JsonDocument must remain in memory; otherwise, the JsonArray or the
JsonObject contains a dangling pointer.
• JsonVariant:
– JsonVariant is also a reference and supports several types: object, array,
integer, float, and boolean.
– JsonVariant differs from JsonDocument because it doesn’t own the memory;
it just points to it.
– JsonVariant supports implicit conversion, but you can also call as<T>().
• The two modes:
– The parser has two modes: zero-copy and classic.
– It uses the zero-copy mode when the input is a char*.
– It uses the classic mode with all other types.
Chapter 3 Deserialize with ArduinoJson 107
That was a free chapter from “Mastering ArduinoJson”; the book contains seven chap-
ters like this one. Here is what readers say:
I think the missing C++course and the troubleshooting chapter are worth
the money by itself. Very useful for C programming dinosaurs like myself.
— Doug Petican
The short C++section was a great refresher. The practical use of Arduino-
Json in small embedded processors was just what I needed for my home
automation work. Certainly worth having! Thank you for both the book
and the library. — Douglas S. Basberg
For a really reasonable price, not only you’ll learn new skills, but you’ll also be one of
the few people that contribute to sustainable open-source software. Yes, giving
money for free software is a political act!
The e-book comes in three formats: PDF, epub and mobi. If you purchase the e-book,
you get access to newer versions for free. A carefully edited paperback edition is
also available.
Ready to jump in?
Go to arduinojson.org/book and use the coupon code THIRTY to get a 30% discount.