EMPLOYING DEEP LEARNING TO ENABLE VISUAL EXPLORATION OF EARTH SCIENCE

EVENTS

Manil Maskey1, Rahul Ramachandran1, Iksha Gurung2, Muthukumaran Ramasubramanian2, Brian

Freitag2, Aaron Kaulfus2, Georgios Priftis2, Drew Bollinger3, Ricardo Mestre3,
Daniel da Silva3
1
NASA Marshall Space Flight Center
2
University of Alabama in Huntsville
3
Development Seed

ABSTRACT Advancements in deep learning techniques have produced

state-of-the-art image classification results, mostly using
Earth science data archives have significantly increased in convolutional neural networks (CNNs) [2]. However, very
size due to the number of advanced sensors and science few of these results have been transitioned to production
missions. In the meantime, Earth science data systems have applications using ML lifecycle [3]. This is mainly because
not taken advantage of data driven technologies to provide the scientists who develop the ML models are rarely equipped
advanced search capabilities. This paper discusses a machine to deploy and scale the models in production. Towards that
learning-based approach, an enabling data driven technology, end, a phenomena portal is developed as a production system
to detect Earth science events from image archives. The that utilizes deep learning-based real time Earth science event
automated event detection is cataloged in an event database detection with search and analysis interface to explore
that provides a novel way to explore large archives of data. cataloged Earth science events and complementary
In addition, a phenomena portal to visually explore events information.
and contextual information is discussed.
This paper presents several contributions: (i) the adaptation
Index Terms— Machine learning, neural network, Earth and application of CNN to identify several Earth science
science, labeled data, training data, portal, event detection phenomena, (ii) the development of an automated Earth
science event database, (iii) the development of automated
cloud based pipeline to transition ML models to production
1. INTRODUCTION environment, and (iv) the development of a phenomena portal
to visually explore the Earth science events.
Satellite remote sensing data is essential in advancing
research and applications in Earth science domain. Data from
the Earth observing platforms are produced at rates that 2. MACHINE LEARNING LIFECYCLE
challenge the scalability and efficiency of search systems.
Search based on traditional faceted filtering falls short of We adapt a typical ML lifecycle [3] to implement the
addressing advanced use cases of searching data based on phenomena portal. The adapted ML lifecycle (Figure 1) for
Earth science phenomena. An Earth science phenomenon is phenomena detection portal includes four phases: data
an observable occurrence of particular physical significance preparation, model development, model deployment, and
within Earth’s dynamics, for example, a hurricane. An Earth visual exploration.
science event is an instance of a phenomenon, for example,
hurricane Katrina. Currently, systems such as the Earth Data preparation: For most cases, data required to train the
Observatory Natural Event Tracker (EONET) [1] employ models are labeled using the image labeler tool that we
manual curation of Earth science events. Such an approach developed. The image labeler tool directly interfaces with
is time consuming and not scalable. Automated event satellite image databases and allows labeling of images in a
detection can provide a scalable solution to augment current distributed environment. Additionally, the image labeler tool
approaches. One way to allow Earth science phenomena- generates a machine learning ready data structure for training.
based search systems is to use machine learning (ML) In some cases, already available data were re-purposed for
techniques. ML is the field of study that involves creating training data.
development of algorithms to teach computers to learn from
data. Automated detection of Earth science events is possible Model development: For all event detection problems, we
using data driven approach enabled by ML techniques, use custom convolutional neural networks on labeled images
specifically, deep learning techniques. Deep learning to develop a model. Once trained, the model captures
consists of machine learning algorithms with multiple layers, knowledge about the events as present in the images which is
where each layer progressively learns features. expected to be consistent with domain experts’ understanding

978-1-7281-6374-1/20/$31.00 ©2020 IEEE 2248 IGARSS 2020

of the phenomena. The model training is done on-premise contour, minimum number of vertices is 4), and (vii) convert
using graphical processing units (GPUs). contours into shapes (stored as geojson in the database).
Smoke detection for 2019 California fire is shown in Figure
2 where, detected pixels are shown with yellow color.

Figure 1 ML lifecycle for phenomena portal (a production

Figure 2 Smoke detection from the California fire
system)
Transverse cirrus band detection: Transverse cirrus bands
Model deployment: Once the model is evaluated and
(TCBs) are ice clouds that often form in association with
optimized the model is deployed in Amazon Web Service
other weather phenomena such as mesoscale convective
(AWS) cloud. Once deployed, the system automatically
systems, hurricanes, and jet streaks [5]. TCBs are often
triggers a pipeline for new models to be executed. The results
associated with clear air turbulence.
of model execution are visualized on the portal.
A total of 5579 true color MODIS and VIIRS imagery were
Visual exploration: The deployed model is used for
obtained from the Global Imagery Browse Service (GIBS)
detection periodically and can be explored visually in a
and labeled with images having or not having TCB for
phenomena portal. The portal also allows users to send
training the model. The training data was augmented using
feedback of any misidentification of events. Those feedbacks
shearing, rotation, and random zooming. The Visual
are used in retraining of the model.
Geometry Group (VGG-16) [6] architecture was adapted and
used along with transfer learning of ImageNet dataset [7].
We have developed event detection models for three
After the final convolutional layer, a global average pooling
phenomena: smoke, transverse cirrus band, and high latitude
(GAP) layer was used instead of the fully connected layers
dust. Next, we describe the development of the models to
which was used to generate class activation maps (CAMs)
detect these phenomena and transition them to a production
[8]. CAM was used to understand the feature relevancy in
system.
classification, allowing us to interpret the model better. An
example classification and heatmap generated by CAM is
3. EVENT DETECTION MODELS
shown in Figure 3.
Smoke detection: An automated, deep learning-based model
capable of detecting smoke plumes from shortwave
reflectance for the Geostationary Operational Environmental
Satellite (GOES) R series of satellites was developed.
Training data was collected by first using a hand-labelled,
past instances of smoke plumes [4]. Then, the data was
quality controlled for spatiotemporal accuracy by a subject
expert. A custom CNN was trained on a number of smoke
events with varying optical thicknesses, geographical
locations and visually similar non smoke phenomena. The
model assigned a probability score of prediction for each Figure 3 Transverse cirrus band detection
pixel being a smoke pixel. To scale the model in production,
we developed following approach to construct a polygon of Once the image is detected with the presence of transverse
smoke plume from detected smoke pixels: (i) for each pixel, cirrus bands, a bounding box is constructed to localize the
construct a 14x14 neighborhood, (ii) get predictions for each part of the image consisting of the transverse cirrus bands.
pixel using the neighborhood, (iii) create a binary image We used following approach to construct the bounding box:
based on each pixel’s prediction, (iv) blur the binary image, (i) resize images into 224x224 px, (ii) divide images into
(v) draw contours on the blurred image, (vi) remove open and patches (25x25 px), (iii) run model for each patch, (iv) create
very small contours (based on number of vertices in the

2249
binary map from predictions, (v) construct contour around 4. ARCHITECTURE
predicted transverse cirrus bands, (vi) remove smaller
bounding box if it overlaps with a larger bounding box, and The architecture for the phenomena portal is shown in Figure
(vii) resize the predicted bounding box to respective original 5. To address the challenges in rapid model deployment,
image size. An example transverse cirrus band detection with transitioning to production environment, and subsequently
CAM heatmap is illustrated in Figure 3. triggering detection of Earth science events, we leverage our
extensive experience with Amazon Web Service (AWS)
High latitude dust detection: Dust has important cloud technologies to implement a specialized architecture.
implications in the climate as it regulates the radiation budget. AWS allows the system to auto scale with demands and
Dust that originates in the Tropics has received a lot of implement a system that is reliable. After the model is
attention, however dust in high latitudes, which appears to be developed on-premise and evaluated, the new model is
of equal importance, has only been studied in the last decade. simply dropped in the predefined AWS S3 bucket. Any new
High latitude dust originates in latitudes higher than 500N and prediction is performed with the new model. AWS Lambda
400S [9] and is produced by glacial and periglacial. Currently, functions are used to trigger daily event detections which in
dust detection is performed with thresholding methods, either turn retrieves relevant imageries for each phenomenon. Any
manually or statistically, over the tropics and sub-tropics. detected event is saved in AWS RDS database which is
Dust detection is also facilitated by ground-based served through an application program interface (API)
instrumentation, however at “cold” latitudes is more difficult created using the AWS API Gateway. The API has four
to maintain an operating network. Therefore, an automated primary endpoints, which accepts only GET requests except
method of detecting high latitude dust events, leveraging where noted: (i) phenomena: returns data on a collection of
satellite data, will help to closely monitor those events and phenomena which accepts a variety of query parameters to
create a database for further study of this phenomenon. filter the response according to user inputs, (ii)
phenomena/{id}: returns data on a single phenomenon. This
True color MODIS images of high latitude dust were endpoint can also accept PUT requests to update a single
identified since 2009 and 89 of them were chosen for training. phenomenon record. This functionality is generally used for
Polygons were drawn at the perimeter of the dust events using noting user feedback on the detection, (iii)
the image labeler. The images resulted in 94.24 million phenomena/histogram: this endpoint returns a collection of
pixels for training. phenomena but separated into different buckets to create a
data-based histogram, and (iv) metadata: returns database
A pixel-based CNN was developed that consists of two main metadata needed for correctly rendering the responses
components. First, a 2D-convolution with a max-pooling
layer was performed on 10x10 pixel size, followed by a
second convolution layer. In every layer, the output data are
passed into the rectified linear unit (ReLu) activation
function. Second, the data was flattened and 20% were
dropped out to prevent overfitting. The final output, consisted
of a probability that a pixel is classified as high latitude dust,
calculated using sigmoid function. An example detection
result is shown in Figure 4 where, image with high latitude
dust is on the left and detection result is on the right (pixels
with probability higher than 50% are indicated using pink to
yellow color scale).

Figure 5 Architecture components and information flow

5. PHENOMENA PORTAL

ML researchers closely collaborated with domain experts,

end user engagement team, UI experts, software engineers
and system architects to transition the developed model into
a real-time production system. The phenomena portal is a
map-based user interface (UI) to the production system which
Figure 4 High latitude dust detection incorporates several features to enhance the user experience
including visualization of events of the day, exploration of
events over time, collection of user feedback, and filtering.

2250
 detection. The event information is stored in a database and
The portal is built as a single-page application (SPA) using accessed via an API, which can be used for long-term
React and Redux. The maps and charts are rendered with analysis and also for consumption by other applications.
third-party libraries Mapbox and D3, respectively, using Additionally, this paper presents a framework for scaling
custom JavaScript code to create unique visualizations. One ML-based end-to-end Earth science applications using cloud
unique visualization, the home page globe, is created using technologies.
Three.js to allow for customization of three-dimensional
objects. The application uses the open API to query for
phenomenon data in response to user input and then display REFERENCES
it in the portal.
[1] Earth Observatory Natural Event Tracker,
https://eonet.sci.gsfc.nasa.gov/, Accessed January 10, 2020.

[2] Y. LeCun and Y. Bengio, “Convolutional networks for

images, speech, and time-series,” in The Handbook of Brain
Theory and Neural Net- works, M. A. Arbib, Ed.
Cambridge, MA, USA: MIT Press, 1998.

[3] M. Maskey et al., "Machine Learning Lifecycle for Earth

Science Application: A Practical Insight into Production
Deployment," IGARSS 2019, IEEE International
Geoscience and Remote Sensing Symposium, Yokohama,
Japan, 2019, pp. 10043-10046.

[4] M. Ramasubramanian, A. Kaulfus, M. Maskey, R.

Ramachandran, I. Gurung, B. Freitag, S. Christopher, "Pixel
level smoke detection model with deep neural network,"
Proc. SPIE 11155, Image and Signal Processing for Remote
Sensing XXV, 1115515 (7 October 2019);

[5] J. Knox, A. Bachmeier, W. Carter, J. Tarantino, L.

Paulik, E. Wilson, G. Bechdol, M. Mays, "Transverse cirrus
bands in weather systems: a grand tour of an enduring
enigma," Weather, 65: 35-41, 2010.

Figure 6 Phenomena portal [6] K. Simonyan and A. Zisserman, "Very Deep

Convolutional Networks for Large-Scale Image
Within the portal, user feedback is supported for each Recognition," Proc. International Conference on Learning
detected event. The feedback includes user agreement with Representations, 2015.
the detection. Initial response from the end user community
has been very encouraging. The resulting feedback is being [7] J. Deng, W. Dong, R. Socher, L. Li, Kai Li and Li Fei-
incorporated as part of our continuous improvement of the Fei, "ImageNet: A large-scale hierarchical image database,"
system. Screenshots of phenomena portal are shown in Figure 2009 IEEE Conference on Computer Vision and Pattern
6, where the overview of events over space and time is Recognition, Miami, FL, 2009, pp. 248-255.
illustrated at the top and a deep dive on a particular event is
shown at the bottom. [8] B. Zhou, A. Khosla, A. Oliva, and A. Torralba,
“Learning Deep Features for Discriminative Localization,”
6. CONCLUSION Proc. of the IEEE International Conference on Computer
Vision and Pattern Recognition, 2016, Las Vegas.
Most Earth science search systems lack functionalities to
support advanced use cases needed for case studies. This [9] J. Bullard, M. Baddock, T. Bradwell, J. Crusius, E.
paper introduces a ML-based event detection to augment data Darlington, D. Gaiero, S. Gassó, G. Gisladottir, R.
search based on Earth science events. The ML detected Hodgkins, R. McCulloch, C. McKenna-Neuman, T.
events are presented to users in a phenomena portal. The Mockford, H. Stewart, and T. Thorsteinsson, "High-latitude
portal allows users to visually explore the events with dust in the Earth system," Reviews of Geophysics, pp. 447-
spatiotemporal context and corresponding images used for 485, 2016.

2251