Mayer 2015
Mayer 2015
CloudThink Infrastructure
Gateway Server
Clients
Storage
Data Server
Cellular Operator
Fig. 2. A virtual dashboard that displays data from different sensors of a
Web-enabled car. The current location of the vehicle is shown on a map or
alternately in the form of a Google StreetView image.
CARduino
Fig. 1. Overview of the CloudThink platform. The CARduino hardware is are controllable only through specific, walled-garden
deployed on the car and communicates with the CloudThink back end via a companion applications. The access of vehicle data through
GSM link. In the back end, the data server parses, filters and synchronizes the OEM-provided interfaces like OnStar’s API or Ford’s
data and stores it in a database. The gateway server provides a REST API for
clients to conveniently access the data from a variety of devices.
OpenXC is limited in breadth, capable of accessing
standardized parameters and generic, but non-standardized
pieces of information. Output modulation is limited to simple
actuators, such as locks or other controls that act only when
and interoperable interface to these data. CloudThink thereby the vehicle is stationary,
aims to create an “App Store” for cars that hosts applications e.g. for car sharing applications. A system similar in nature
which can access and process car telemetry, and in cases, to CloudThink is required to facilitate the best application
control vehicle actuator function. The platform is distinguished development opportunities, by providing the richest possible
from similar projects in the automotive domain in that it vehicle data and actuation possibilities.
remains agnostic of the concrete type and make of vehicle
and therefore enables data sharing across the boundaries of car The main components of the CloudThink system are the
manufacturers, improving data generation by unsiloing data CARduino hardware, the data server that stores data uploaded
storage and allowing for rapid, crowdsourced data collection. from individual cars in a local database, and the gateway
Use cases for the CloudThink system stretch from enhancing server that enables third parties to access the stored data in a
vehicle security (for instance, a “virtual leash” for tracking of uniform way via a REST API (see Figure 1 on the previous
stolen vehicles) to reducing the environmental impact of cars page). Within the joint project, MIT is leading the hardware
by helping drivers to find out how they can better economize development (see Section I-A), while ETH’s main
fuel. Drivers could also use such a system to improve safety responsibility is the implementation and maintenance of the
by adapting their driving style [2], and the platform could back-end infrastructure (see Section I-B). On top of the
help set economic incentives to do so by enabling pay-as- system, the consortium has also created applications that
you-drive (PAYD) or usage based insurance schemes [3], [4]. provide better access to data stored on the platform in the form
Potentially, drivers could also use CloudThink to monetize of “virtual dashboards” (see Figure 2) that could be used, for
their driving data, for instance by sharing usage statistics and instance, by fleet managers, and has used telemetry data from
telemetry with the manufacturer of their car – or drivers could CloudThink to create an application that gives drivers more
provide the same data to non-profit organizations to achieve insight into their own driving behavior, especially with respect
better transparency of how specific vehicles fare in realistic to clues as to how they could improve their own fuel economy.
situations, for instance with respect to their fuel consumption,
maintenance effort, or to provide data leading up to a vehicle A. CARduino Hardware
recall. Another highly interesting application domain is using
real driving data for the improvement of vehicle Our CARduino hardware2 (see Figure 3) is responsible for
characteristics, for instance determining the optimal battery transmitting information from the vehicle to the CloudThink
size for electric cars [5]. On a larger scale, real-time feedback back-end infrastructure, where this data is stored in a SQL-
from vehicles could help improve road quality – and safety – based database. All information is sent via GSM connections
by enabling road operators to quickly react to problems that and encoded in a custom message format that is tuned to
range from potholes to developing traffic jams [6], [7], without minimize the volume of the transmitted data, thereby helping
the need for direct user reporting. to keep communication cost low. Each message that is sent
to the back end consists of the recording timestamp, the type
The CloudThink platform differs from contemporary of the transmitted data, the data itself, and a checksum to
OEM- backed application development systems. Interfaces ensure accurate data capture. Messages are received by the
proposed by auto manufacturers are limited in scope, with CloudThink data server that parses and checks each message
focus on intra- vehicle applications of data, such as within before persisting the contained data.
infotainment systems running locally on a car’s head unit. In 2
The CARduino hardware is open-source. It is manufactured by CarKnow
cases where direct, remote control of vehicle functions is LLC., a company that has been incorporated by one of our partners in the
allowed, these functions CloudThink project.
39
2015 5th International Conference on the Internet of Things (IoT)
(a)
Fig. 4. Example responses of the CloudThink API’s HTML interface to a
request for a minute of car telemetry (GPS coordinates, coolant temperature,
speed, and revolutions per minute).
to interact
41
2015 5th International Conference on the Internet of Things (IoT)
(a)
42
2015 5th International Conference on the Internet of Things (IoT)
1
2
@prefix
@prefix
:<ex#>.
ex:<http://example.org/#>.
to obtain the GPS latitude of the current location of a vehicle
3 @prefix http:<http://www.w3.org/2011/http#>. given its VIN: using Notation3, a superset of RDF, it specifies
4 @prefix dbpedia:<http://dbpedia.org/resource/>. that, given a specific VIN (line 7 in Listing 1), the car’s GPS
5
6 {
latitude can be obtained (line 15) by sending an HTTP GET
7 ?car a dbpedia:Car; ex:hasVin ?vin. request to the specified URL (lines 11-13) – more specifically,
8 } that this latitude value is conveyed in plain text within the
=>
9
10 {
body of the HTTP response. Note that the correct car’s URI is
11 _:request http:methodName "GET"; selected at runtime by substituting the ?vin parameter with
12 http:requestURI ("https://api.cloud-think.com/things the supplied VIN (line 12).
/" ?vin "/?parameters=latitude");
http:resp [ http:body ?gpslat ].
13
14
The CloudThink API contains a similar description for
15 ?gpslat a dbpedia:Latitude; ex:derivedFrom ?car. each parameter that can be accessed about its connected cars
16 }. (e.g., their velocity, coolant temperature, and engine
Listing 1. Part of the RESTdesc description of the CloudThink API that revolutions per minute). These descriptions can be used by
describes how the current GPS latitude of a car can be obtained given its VIN,
using an HTTP request to the CloudThink server. The server provides similar
semantic reasoners to integrate functionality across
descriptions for other information about the cars it has data about. heterogeneous Web-enabled devices and Web services – to
demonstrate that our system is capable of creating composite
services on top of data produced by vehicles and published via
CloudThink, we used its semantic descriptions together with
within the context of recognizing vehicles gives rise to several those of a service able to display arbitrary images on a display,
challenges, predominantly due to the visual similarity of and semantically annotated two mapping applications (Google
distinct vehicles: being based on visual object recognition Maps and OpenStreetMaps). Given this setup, users can
technologies, our systems cannot differentiate between formulate a semantic goal that expresses their desire to obtain
different cars that look alike, with the exception of cases where an image object that is also a map and is derived from the
unique features of the car are visible in the camera frame (and VIN of a specific car, and that a screen that is situated at
related training images). While our approach is thus able to the current location of the user should be displaying this
find and match visual features of vehicles, these are often image – this leads a semantic reasoner to propose that it
located on the rims or the number plate of a vehicle, on could display the current location of a car on a nearby screen,
custom stickers, inside the car, or on its reflective surfaces by mashing up the CloudThink back end with either Google
(see Figure 5(d)), which makes the robustness of our system Maps or OpenStreetMaps and the screen API (see Figure 7).
heavily dependent on the current lighting conditions. Our reasoning infrastructure, which is described in greater
Furthermore, especially in the case of “plain vanilla” cars detail in [12], then sends the appropriate HTTP requests to the
without any distinct features, our software is not able to client which can execute them and thereby achieve the desired
differentiate between cars of the same make, type, and color. user goal. Because this service mashup is not hard-coded but
Similar to humans, who often arbitrate between different cars rather derived on demand from the currently available semantic
that look alike by remembering their vehicle’s parking descriptions, the mashup seamlessly switches to using
location, we propose to partially overcome this challenge by alternatives when services become unavailable, e.g. due to
using the GPS location of a car (which may be obtained from network outages – in this specific case, when we blocked
the CloudThink API) in conjunction with the location of the access to one of the two mapping services, the mashup
user interface device as an additional context parameter to seamlessly switched to using the other, thus demonstrating the
uniquely identify a vehicle. However, while this technique can flexibility of our approach of using functional semantic
indeed help to achieve stable selection results, it introduces metadata for dynamic service composition.
another challenge: the (remote or local) application that helps
the user interface device arbitrate between different cars
requires access to the locations of cars that are registered to the C. Monitoring Car Interactions
CloudThink platform or, as a minimum, an interface to send We believe that keeping users informed about what their
geographically bounded queries for a car (i.e., “Is car C connected smart things are doing in an increasingly connected
currently located within 100 meters of the GPS coordinates Web of Things world – and why they are – will become
(Lat,Lon)”). Other types of context information might be increasingly important as many of our once isolated devices
used as well for this purpose, for instance ambient beacons start interacting with each other and with remote services [13],
(e.g., Google PhysicalWeb3), to ensure the minimum viable especially if distributed services are flexibly or even automati-
amount of data are shared with unlinked parties. cally combined. Indeed, Web-enabled cars and platforms such
as CloudThink represent an interesting domain where drivers
B. Functional Semantic Metadata for Smart Vehicles would want to monitor which services access their vehicles:
Apart from the direct interaction with connected automo- the ability to supervise and control interactions is relevant to
biles, we have used functional semantic metadata in the form protect drivers from attacks on their privacy such as location
described in [12] to capture the functionality of the profiling, undesired actuator control, or the misuse of private
CloudThink API with respect to providing access to vehicle data by authorities [14].
telemetry data of individual cars, where each car is uniquely To support users to stay in control of their connected cars,
identified using its Vehicle Identification Number (VIN). As we integrated a piece of software that logs HTTP requests
an example, Listing 1 shows the semantic description of the and responses with the CloudThink back end and makes
API that enables clients
the collected data accessible to users in real time, using an
43
2015 5th International Conference on the Internet of Things (IoT)
3
https://google.github.io/physical-web/ augmented reality interface on their handheld devices [13].
44
2015 5th International Conference on the Internet of Things (IoT)
(a) (b)
Fig. 6. Our application visualizes interactions with a Web-enabled automobile that are brokered by the CloudThink platform using a visual object recognition
algorithm and an augmented reality overlay on a handheld device. (a) Interactions of a client with a vehicle (and other endpoints). (b) Interaction of a handheld
device with a vehicle.
III. SUMMARY
The CloudThink platform, a joint effort of ETH, MIT, and
several additional international partner institutions, has been
created to make automobiles “first-class citizens” of the Web
GET and enable third-party applications to easily access and
GET process vehicle data while providing advanced services to
drivers, fleet managers, and other players in the transportation
PUT industry. We and others have implemented several such
applications on top of this platform, in particular in an effort to
support users and applications when interacting with
connected vehicles: using our systems, it is possible for users
to directly interact with sensors and actuators of connected
automobiles, monitor the communication of vehicles and
Fig. 7. By publishing semantic metadata that describes its interfaces,
functionality of the CloudThink API (i.e., access to sensor values and actuators clients in real time, and to seamlessly use functionality that is
of connected vehicles) can be flexibly integrated with other local and remote provided by cars within physical mashups.
Web services. In this example, a user instructs an interface device (in this case,
a smartphone) to display the location of his car on a map that is shown an a The future developments for the CloudThink project
nearby computer screen. A reasoner then finds the necessary HTTP requests include expanding the API to return interaction monitoring
to reach that goal and executes them. data for all available vehicles. Additionally, we plan to
implement the ability for third-party application builders to
run their own scripts and modify firmware behavior on the
CARduino within safe and controlled boundaries, using an
Using Java Agents which can modify the bytecode of Java embedded virtual machine of sorts to limit the risk of
classes and class transformers that can infuse any Web server malicious code execution and to protect sensitive vehicle
based on the Grizzly NIO framework4 with an HTTP request networks.
logger, the integration of the logging functionality with the
CloudThink gateway server proved straightforward, and
merely required a restart of the server with attached Acknowledgements. This work was supported by the
transformers. The combined system allows users to monitor Swiss National Science Foundation under grant number
interactions of clients with their vehicles using the Web 341627 and the SUTD-MIT International Design Center. The
interface of our visualization tool as well as displaying these authors thank Yassin N. Hassan and Ga´bor So¨ro¨s for their
interactions as a live overlay on handheld, or ultimately help with the object recognition and interactions visualization
wearable, devices (see Figure 6). For the live overlay view, we software.
again use our object recognition software to identify the
vehicle whose information should be displayed: the arcs REFERENCES
between endpoints represent HTTP connections and the dots [1] E. Wilhelm, J. Siegel, S. Mayer, L. Sadamori, S. Dsouza, C.-K. Chau,
traveling on these arcs represent actual pseudo-real time HTTP and S. Sarma, “CloudThink: A Scalable Secure Platform for Mirroring
requests (for the visualization, we increase the request Transportation Systems in the Cloud,” Transport, vol. 30, no. 3, 2015
(in press).
latencies to one second per request, however).
[2] T. Lotan and T. Toledo, “An In-Vehicle Data Recorder for Evaluation
of Driving Behavior and Safety,” Transportation Research Record, vol.
1953, pp. 112–119, 2006.
[3] J. Paefgen, T. Staake, and F. Thiesse, “Resolving the Misalignment
4 between Consumer Privacy Concerns and Ubiquitous IS Design: The
https://grizzly.java.net/ Case of Usage-based Insurance,” in Proceedings of the 33rd
45
2015 5th International Conference on the Internet of Things (IoT)
46
2015 5th International Conference on the Internet of Things (IoT)
47