Serialization Tutorial6 PDF
Serialization Tutorial6 PDF
Serialization Tutorial6 PDF
CREATOR OF ARDUINOJSON
Mastering ArduinoJson 6
Efficient JSON serialization for embedded C++
Contents
Contents iv
1 Introduction 1
1.1 About this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Introduction to JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 What is JSON? . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 What is serialization? . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 What can you do with JSON? . . . . . . . . . . . . . . . . . . 4
1.2.4 History of JSON . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.5 Why is JSON so popular? . . . . . . . . . . . . . . . . . . . . . 6
1.2.6 The JSON syntax . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.7 Binary data in JSON . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Introduction to ArduinoJson . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.1 What ArduinoJson is . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.2 What ArduinoJson is not . . . . . . . . . . . . . . . . . . . . . 12
1.3.3 What makes ArduinoJson different? . . . . . . . . . . . . . . . 13
1.3.4 Does size really matter? . . . . . . . . . . . . . . . . . . . . . . 15
1.3.5 What are the alternatives to ArduinoJson? . . . . . . . . . . . . 16
1.3.6 How to install ArduinoJson . . . . . . . . . . . . . . . . . . . . 17
1.3.7 The examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
6 Troubleshooting 186
6.1 Program crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
6.1.1 Undefined Behaviors . . . . . . . . . . . . . . . . . . . . . . . . 187
6.1.2 A bug in ArduinoJson? . . . . . . . . . . . . . . . . . . . . . . 187
6.1.3 Null string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
6.1.4 Use after free . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
6.1.5 Return of stack variable address . . . . . . . . . . . . . . . . . 190
6.1.6 Buffer overflow . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.1.7 Stack overflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Contents x
8 Conclusion 264
Index 265
Chapter 4
Serialize with ArduinoJson
ý Any fool can write code that a computer can understand. Good programmers
write code that humans can understand.
– Martin Fowler, Refactoring: Improving the Design of Existing Code
Chapter 4 Serialize with ArduinoJson 104
{
"value": 42,
"lat": 48.748010,
"lon": 2.293491
}
It’s a flat object, meaning that it has no nested object or array, and it contains the
following members:
1. “value” is an integer that we want to save in Adafruit IO.
2. “lat” is the latitude coordinate where the value was measured.
3. “lon” is the longitude coordinate where the value was measured.
Adafruit IO supports other optional members (like the elevation coordinate and the time
of measurement), but the three members above are sufficient for our example.
doc["value"] = 42;
doc["lat"] = 48.748010;
doc["lon"] = 2.293491;
The memory usage is now JSON_OBJECT_SIZE(3), so the JsonDocument is full. When the
JsonDocument is full, you cannot add more members, so don’t forget to increase the
capacity if you need.
With the syntax presented above, it’s not possible to tell whether the insertion suc-
ceeded. Let’s see another syntax:
doc["value"].set(42);
doc["lat"].set(48.748010);
doc["lon"].set(2.293491);
The compiler generates the same executable as with the previous syntax, except that
you can tell if the insertion succeeded. Indeed, JsonVariant::set() returns true for
success or false if the JsonDocument is full.
I never check if insertion succeeds in my programs. The reason is simple: the JSON
document is roughly the same for each iteration; if it works once, it always works. There
is no reason to bloat the code for a situation that cannot happen.
Chapter 4 Serialize with ArduinoJson 107
We just saw that the JsonDocument becomes an object as soon as you insert a member,
but what if you don’t have any member to add? What if you want to create an empty
object?
When you need an empty object, you cannot rely on the implicit conversion any-
more. Instead, you need to explicitly convert the JsonDocument to a JsonObject with
JsonDocument::to<JsonObject>():
This function clears the JsonDocument, so all existing references become invalid. Then,
it creates an empty object at the root of the document and returns a reference to this
object.
At this point, the JsonDocument is not empty anymore and JsonDocument::isNull()
returns false. If we serialize this document, the output is “{}.”
obj["value"] = 42;
obj["value"] = 43;
Chapter 4 Serialize with ArduinoJson 108
Most of the time, replacing a member doesn’t require a new allocation in the
JsonDocument. However, it can cause a memory leak if the old value has associated
memory, for example, if the old value is a string, an array, or an object.
o Memory leaks
Replacing and removing values produce a memory leak inside the
JsonDocument.
In practice, this problem only happens in programs that use a JsonDocument
to store the state of the application, which is not the purpose of ArduinoJson.
Let’s be clear, the sole purpose of ArduinoJson is to serialize and deserialize
JSON documents.
Be careful not to fall into this common anti-pattern and make sure you read
the case studies to see how ArduinoJson should be used.
Chapter 4 Serialize with ArduinoJson 109
[
{
"key": "a1",
"value": 12
},
{
"key": "a2",
"value": 34
}
]
The values 12 and 34 are just placeholder; in reality, we’ll use the result from
analogRead().
doc.add(1);
doc.add(2);
We saw that the JsonDocument becomes an array as soon as we add elements, but this
doesn’t allow creating an empty array. If we want to create an empty array, we need to
Chapter 4 Serialize with ArduinoJson 111
arr[0] = 666;
arr[1] = 667;
Most of the time, replacing the value doesn’t require a new allocation in the
JsonDocument. However, if there was memory held by the previous value, for exam-
ple, a JsonObject, this memory is not released. It’s a limitation of the library’s memory
allocator, as we’ll see in the next chapter.
As for objects, you can remove an element from the array, with JsonArray::remove():
arr.remove(0);
Again, remove() doesn’t release the memory from the JsonDocument, so you should never
call this function in a loop.
Chapter 4 Serialize with ArduinoJson 112
To conclude this section, let’s see how we can insert special values in the JSON docu-
ment.
The first special value is null, which is a legal token in a JSON. There are several ways
to add a null in a JsonDocument; here they are:
The other special value is a JSON string that is already formatted and that ArduinoJson
should not treat as a regular string.
You can do that by wrapping the string with a call to serialized():
// adds "[1,2]"
arr.add("[1,2]");
// adds [1,2]
arr.add(serialized("[1,2]"));
[
"[1,2]",
[1,2]
]
Chapter 4 Serialize with ArduinoJson 113
This feature is useful when a part of the document never changes, and you want to
optimize the code. It’s also useful to insert something you cannot generate with the
library.
Chapter 4 Serialize with ArduinoJson 114
We saw how to construct an array, and it’s time to serialize it into a JSON document.
There are several ways to do that. We’ll start with a JSON document in memory.
We could use a String but, as you know, I prefer avoiding dynamic memory allocation.
Instead, we’d use a good old char[]:
[{"key":"a1","value":12},{"key":"a2","value":34}]
As you see, there are neither space nor line breaks; it’s a “minified” JSON document.
If you’re a C programmer, you may have been surprised that I didn’t provide the size
of the buffer to serializeJson(). Indeed, there is an overload of serializeJson() that
takes a char* and a size:
However, that’s not the overload we called in the previous snippet. Instead, we
called a template method that infers the size of the buffer from its type (in this case
char[128]).
Of course, this shorter syntax only works because output is an array. If it were a char*
or a variable-length array, we would have had to specify the size.
` Variable-length array
A variable-length array, or VLA, is an array whose size is unknown at compile
time. Here is an example:
void f(int n) {
char buf[n];
// ...
}
C99 and C11 allow VLAs, but not C++. However, some compilers support
VLAs as an extension.
This feature is strongly detested in the C and C++ circles, but Arduino users
seem to love it. That’s why ArduinoJson supports VLAs in all functions that
accept a string.
The minified version is what you use to store or transmit a JSON document because
the size is optimal. However, it’s not very easy to read. Humans prefer “prettified”
JSON documents with spaces and line breaks.
To produce a prettified document, you just need to use serializeJsonPretty() instead
of serializeJson():
[
{
"key": "a1",
"value": 12
},
{
"key": "a2",
"value": 34
}
]
Of course, you need to make sure that the output buffer is big enough; otherwise, the
JSON document will be incomplete.
ArduinoJson allows computing the length of the JSON document before producing it.
This information is useful for:
1. allocating an output buffer,
2. reserving the size on disk, or
3. setting the Content-Length header.
There are two methods, depending on the type of document you want to produce:
The behavior is slightly different: the JSON document is appended to the String; it
doesn’t replace it. That means the above snippet sets the content of the output variable
to:
JSON = [{"key":"a1","value":12},{"key":"a2","value":34}]
Does that seem inconsistent? That’s because ArduinoJson treats String like a stream;
more on that later.
You should remember from the chapter on deserialization that we must cast JsonVariant
to the type we want to read.
It is also possible to cast a JsonVariant to a String. If the JsonVariant contains a
string, the return value is a copy of the string. However, if the JsonVariant contains
something else, the returned string is a serialization of the variant.
Chapter 4 Serialize with ArduinoJson 118
This trick works with JsonDocument and JsonVariant, but not with JsonArray and
JsonObject because they don’t have an as<T>() function.
Chapter 4 Serialize with ArduinoJson 119
For now, every JSON document we produced remained in memory, but that’s usually
not what we want. In many situations, it’s possible to send the JSON document directly
to its destination (whether it’s a file, a serial port, or a network connection) without
any copy in RAM.
We saw in the previous chapter what an “input stream” is, and we saw that Arduino
represents this concept with the Stream class. Similarly, there are “output streams,”
which are sinks of bytes. We can write to an output stream, but we cannot read. In
the Arduino land, an output stream is materialized by the Print class.
Here are examples of classes derived from Print:
c std::ostream
In the C++ Standard Library, an output stream is represented by the
std::ostream class.
ArduinoJson supports both Print and std::ostream.
Chapter 4 Serialize with ArduinoJson 120
You can see the result in the Arduino Serial Monitor, which is very handy for debug-
ging.
There are also other serial port implementations that you can use this way, for example,
SoftwareSerial and TwoWire.
Chapter 4 Serialize with ArduinoJson 121
You can find the complete source code for this example in the WriteSdCard folder of the
zip file.
You can apply the same technique to write a file on an ESP8266, as we’ll see in the
case studies.
We’re now reaching our goal of sending our measurements to Adafruit IO.
As said in the introduction, we’ll suppose that our program runs on an Arduino UNO
with an Ethernet shield. Because the Arduino UNO has only 2KB of RAM, we’ll not
use the heap at all.
If you want to run this program, you need an account on Adafruit IO (a free account is
sufficient). Then, you need to copy your user name and you “AIO key” to the source
code.
We’ll include the AIO key in an HTTP header, it will authenticate our program on
Adafruit’s server:
X-AIO-Key: baf4f21a32f6438eb82f83c3eed3f3b3
Chapter 4 Serialize with ArduinoJson 122
Finally, to run this program, you need to create a “group” named “arduinojson” in your
Adafruit IO account. In this group, you need to create two feeds “a1” and “a2.”
The request
To send our measured samples to Adafruit IO, we have to send a POST request to http://
io.adafruit.com/api/v2/bblanchon/groups/arduinojson/data, and include the following
JSON document in the body:
{
"location": {
"lat": 48.748010,
"lon": 2.293491
},
"feeds": [
{
"key": "a1",
"value": 42
},
{
"key": "a2",
"value": 43
}
]
}
As you see, it’s a little more complex than our previous sample because the array is not
at the root of the document. Instead, the array is nested in an object under the key
“feeds.”
Let’s review the complete request before jumping to the code:
{"location":{"lat":48.748010,"lon":2.293491},"feeds":[{"key":"a1","value"...
The code
OK, time for action! We’ll open a TCP connection to io.adafruit.com using an
EthernetClient; then we’ll send the above request. As far as ArduinoJson is concerned,
there are very few changes compared to the previous examples because we can pass the
EthernetClient as the target of serializeJson(). We’ll call measureJson() to set the
value of the Content-Length header.
Here is the code:
// Allocate JsonDocument
const int capacity = JSON_ARRAY_SIZE(2) + 4 * JSON_OBJECT_SIZE(2);
StaticJsonDocument<capacity> doc;
You can find the complete source code of this example in the AdafruitIo folder of the
zip file. This code includes the necessary error checking that I removed from the book
for clarity.
Below is a picture of a dashboard showing the data from this example.
Chapter 4 Serialize with ArduinoJson 125
When you add a string value to a JsonDocument, ArduinoJson either stores a pointer or
a copy of the string depending on its type. If the string is a const char*, it stores a
pointer; otherwise, it makes a copy.
As usual, the copy lives in the JsonDocument, so you may need to increase its capacity
depending on the type of string you use.
4.6.1 An example
They both produce the same JSON document, but the second one requires much more
memory because ArduinoJson copies the strings. If you run these programs on an
ATmega328, you’ll see 16 for the first one and 30 for the second.
In the example above, ArduinoJson copied the Strings because it needed to add them
to the JsonDocument. On the other hand, if you use a String to extract a value from a
JsonDocument, it doesn’t make a copy.
Here is an example:
I understand that it is disappointing that ArduinoJson copies Flash strings into the
JsonDocument. Unfortunately, there are several situations where it needs to have the
strings in RAM.
For example, if the user calls JsonVariant::as<char*>(), a pointer to the copy is re-
turned:
It is required for JsonPair too. If the string is a key in an object and the user iterates
through the object, the JsonPair contains a pointer to the copy:
Chapter 4 Serialize with ArduinoJson 127
However, retrieving a value using a Flash string as a key doesn’t cause a copy:
4.6.4 serialized()
We saw earlier in this chapter that the serialized() function marks a string as a JSON
fragment that should not be treated as a regular string value.
serialized() supports all the string types (char*, const char*, String and const
__FlashStringHelper*) and duplicates them as expected.
Chapter 4 Serialize with ArduinoJson 128
4.7 Summary
In this chapter, we saw how to serialize a JSON document with ArduinoJson. Here are
the key points to remember:
• Construct the document:
– To add a member to an object, use the subscript operator ([])
– To append an element to an array, call add()
– The first time you add a member to a JsonDocument, it automatically becomes
an object.
– The first time you append an element to a JsonDocument, it automatically
becomes an array.
– You can explicitly convert a JsonDocument with JsonDocument::to<T>().
– JsonDocument::to<T>() clears the JsonDocument, so it invalidates all previously
acquired references.
– JsonDocument::to<T>() return a reference to the root array or object.
– To create a nested array or object, call createNestedArray() or
createNestedObject().
In the next chapter, we’ll open the hood and look at ArduinoJson from the inside. We’ll
also see a few things that I couldn’t cover in this tutorial.
Continue reading...
That was a free chapter from “Mastering ArduinoJson”; the book contains seven chap-
ters like this one. Here is what readers say:
I think the missing C++course and the troubleshooting chapter are worth
the money by itself. Very useful for C programming dinosaurs like myself.
— Doug Petican
The short C++section was a great refresher. The practical use of Arduino-
Json in small embedded processors was just what I needed for my home
automation work. Certainly worth having! Thank you for both the book
and the library. — Douglas S. Basberg
For a really reasonable price, not only you’ll learn new skills, but you’ll also be one of
the few people that contribute to sustainable open-source software. Yes, giving
money for free software is a political act!
The e-book comes in three formats: PDF, epub and mobi. If you purchase the e-book,
you get access to newer versions for free. A carefully edited paperback edition is
also available.
Ready to jump in?
Go to arduinojson.org/book and use the coupon code THIRTY to get a 30% discount.