Study of GIS cloud based computation

system,Google Earth Engine, its capabilities and

computational abilities

Ahmad Haseeb Rabbani Muhammad Asgher Ali

Department of GIS & RS Department of GIS & RS
Institute of Space Technology Institute of Space Technology
Islamabad,Pakistan 42000 Islamabad,Pakistan 42000
Email: [email protected] Email: [email protected]

Dr. Rizwan Bulbul

Department of GIS & RS
Institute of Space Technology
Islamabad,Pakistan 42000
Email: [email protected]

3 Datasources available
Earth Engine database has a petabytes of widely used
geospatial datasets including complete archive of Land-
sat,Sentinel datasets as well as other geophysical and socio-
1 Introduction economic datasets. Following is the detailed list of datasets
available having global coverage in google earth engine
The Computer Science era is shifting from Standalone
database.
systems to cloud computing.Spatial Cloud computing ser-
vices are the new in this field, having petabytes of re-
mote sensing data from NASA,NOAA,USGS as well as 3.1 Satellite Imagery
from ESA. Many of the tools and system have been devel- Earth Engine database has a wide range of past and
oped to facilitate large scale processing of geospatial data current satellite imagery that is freely available from Opti-
e.g, NASA Worldwind, Amazon Web mapping services and cal to SAR data. Along with raw Imagery available, earth
google earth engine. The purpose of the following research engine also provides their products like NDWI, NDVI and
is to highlight the capabilities of google earth engine to han- many more. By having such a huge datasets of more than
dle and process large amount of geospatial datasets. Google 3 decades, google earth engine has proved to be useful for
Earth Engine is a clould based geospatial data processing multi-temporal time series analysis. Following are the some
main sources of Satellite Imagery available in google earth
system that can let users process current and archive of thou-
engine:
sands of remote sensing datasets available freely by multiple
providers. Google earth Engine put the multi-source geospa-
tial datasets on the google servers and let users process and
analyse the data without getting into pain of downloading Dataset Spatial Resolution Temporal Coverage
and handling too much large dataset.
Landsat 4-8 30m 1984-now
MODIS Aqua/Tera 250m-1000m 2000-now
Sentinel-1 10-40m 2014-now
2 Aims and Objectives
To study the capabilities and computational abilities of Sentinel-2 10m 2015-now
the google earth engine based on the following: 1)Data- Sentinel-3 300m 2016-now
sources available 2)System Architecture 3)Processing Capa-
bilities 4)Scientific Domain 5)Data Models Support 6)Data ASTER 15/30/90m 2000-now
Ingestion lag time 7)API support 8)Datasets upload limita-
tions 9)A case study to analyse the capabilities of the system Table 1. Main Satellite datasets available in Google Earth Engine
3.2 Topography 4 System Architecture
Earth Engine database has a wide range of Satellite de- A simplified system architecture is shown in Fig below.
rived Elevation products ranging from Global coverage of The Earth Engine Code Editor and third-party applications
SRTM to some local products having multiple resolutions. use client libraries to send interactive or batch queries to the
Following are the some global topography products available
system through a REST API. On-the-fly requests are handled
in google earth engine:
by Front End servers that forward complex sub-queries to
Compute Masters, which manage computation distribution
among a pool of Compute Servers.
Dataset Spatial Resolution
SRTM DEM 30m/90m
ASTER DEM 30m
ALOS DSM 30m
WWF HydroSHEDS 30m/90m
ETOPO1 1 arc minute
GTOPO30 1km

Table 2. Main Topographic datasets available in Earth Engine

3.3 Other landcover and Enviromental Datasets

Earth Engine database also gives some Landcover and
Environmental products from different research organiza-
tions like Global Forest change from Hansen and Oxford
Landcover datasets. Following are the some landcover and
environmental products available in google earth engine: Fig. 1. Google earth Engine System Architecture

Dataset Spatial Resolution Google Earth engine Workspace/ Explorer is an interac-

Global Forest Change 30m tive tool to visualize available datasets and their products.

Nighttime Lights 1km

Oxford EVI 5km
CHIRPS 5km
VIIRS Surface Reflectance 500m/1km
Oxford TCB 5km
AVHRR Aerosol Optical Thickness 11km
Oxford TCW 5km
MODIS Land Cover 500m
Oxford Landcover 5km Fig. 2. Google earth Engine Playground/ Code Editor

GlobCover 300m
Global Cropland and Watering 1km
Global Cropland and Crop Dominance 1km Google earth engine playground is based on javascript
NOAA Sea Surface temperature API and provides easy to use interface for earth engine cod-
ing with lot of code examples and functions help. With just
MODIS LST and emissivity the internet connection and without building the source of
earth engine, one can have full advantage of earth engine ca-
Table 3. Main Landcover & environmental datasets available pabilities through the earth engine playground / code editor.
 of datasets. Users can code their own algorithms using the
Python and JavaScript APIs for advanced functionalities and
also can upload their own datasets using the google fusion
tables or also through asset database.

7 Data Models Support

Google earth engine supports both vector and raster data
models. Google fusion tables are used to upload vector
Fig. 3. Google earth Engine Playground/ Code Editor datasets and similarly asset database can be used to upload
both vector and raster datasets. Google earth engine also pro-
vides tools for processing of both vector and raster datasets
and also provide conversion tools between the both.
5 Processing Capabilities
Functions in the google earth engine are mainly divided
into following categories: 1)Numerical operations 2)Array/- 8 Data Ingestion lag time
matrix operations 3)Machine learning algorithms 4)Kernel
The catalog is continuously updated at a rate of nearly
operations 5)Image Operations 6)Reducer functions 7)Ge-
6000 scenes per day from active missions, with a typical la-
ometry Operations 8)Table Operations 9)Vector/Raster Con-
tency of about 24 h from scene acquisition time. User can re-
versions. Some of the advanced functions are as follows:
quest for data ingestion or can add his/her own dataset to the
asset database through the code editor or through the com-
mand line tool.
Function Types

Bayes, CART,
Supervised classification
Random Forest, SVM, 9 API Support
and regression
Perceptron, Mahalanobis Google earth engine gives the API access of javascript
and python free of cost along with their public catalog ac-
Spectral operations Unmixing, HSV transform cess. User can access the public catalog and perform func-
Convolution Convolve, blur tionalities depending on the application and then fetch the
results using the rest API. Google javascript API can be ac-
Morphology Min, max, mean, distance cessed via Googel code editor available on the web. Sim-
Gaussian, Laplacian, ilarly, command-line tool for Python API is available for
Standard kernels download through github.
Roberts, Sobel

Edge detection Canny, Hough transform

Terrain Operations Slope, aspect, hillshade 10 Datasets upload limitation
User can add his/her own datasets on google earth
Resampling Bilinear, bicubic engine asset database via Google fusion tables and asset
Kendall, Spearman, database. Google stores users uploaded data in the google
Correlation drive linked with user’s account,so storage space of the
Pearson, Sen’s slope
google drive is the limitation in data uploading in earth en-
Regression Linear regression gine. In case of rasters, earth engine only supports GeoTIFF
Constructive Operations Intersection, union, difference format. So, one have to convert dataset into GeoTiff before
uploading it into earth engine asset database.
Spatial filtering Intersects, contains, within
Joins Simple, inner, grouping
11 Case Study - Water Surface Density plot of Indus
Spatial Interpolation Kriging, IDW interpolation River using Landsat time series data
The Purpose of the case study is to access the capabili-
Table 4. Main functionalities available in Earth Engine ties of time series analysis in google earth engine.

6 Scientific Domain 11.1 Task

Google earth engine is not domain specific.Its gives ba- Perform a time series analysis to study surface water oc-
sic and some advanced functionalities for raster and vector curance in Indus river for the past 10 years and to further
processing along with free archive of around 5+ petabyte study the river morphology.
11.2 Study area 11.5 Javascript Code
Downstream of tarbela Dam to shukkar barrage is taken
1 //Indus Bounds
as a study area. 2 var aoi = ee.FeatureCollection(’ft:1
eN0ZmcfzJPuIiZILlWYCRfMrwPBQF6aSQAUFI4ij’).
filter(ee.Filter.eq(’River Name’, ’Indus River
11.3 Datasets used ’));
Landsat-5 from 2008-2011 and Landsat-8 from 2013- 3 var quality_mask = function(image) {
4 var quality = image.select(’fmask’);
2017 Satellite Imagery are used for this study. A total of
5 var cloud = quality.eq(4);
around 800 scenes with a size of around 1TB are used. 6 var maskedImage = image.mask().and(cloud.not());
7 return image.mask(maskedImage);};
8 // NDWI Function L7/8
9 var ndwi = function(year){
Year Sensor No. of Scenes 10 var startdate=(year).toString()+’-03-01’;
11 var enddate=(year).toString()+’-06-30’;
2008 Landsat-5 48 12 var series = ((landsat.filterDate(startdate,
enddate)).median()).clip(aoi);
2009 Landsat-5 103 13 var series_unmasked = ((landsat_unmasked.
filterDate(startdate,enddate)).median()).
2010 Landsat-5 85
clip(aoi);
2011 Landsat-5 79 14 //cloud cover percentage
15 var origPixels = series_unmasked.select(0).
2013 Landsat-8 46 reduceRegion({reducer: "count", geometry:
aoi, scale: 30, bestEffort: true});
2014 Landsat-8 122 16 var maskedPixels = series.select(0).reduceRegion
({reducer: "count",geometry: aoi, scale: 30,
2015 Landsat-8 113 bestEffort: true});
17 var count1 = origPixels.get(’B2’).getInfo();
2016 Landsat-8 129 18 var count2 = maskedPixels.get(’B2’).getInfo();
19 print(((count1 - count2) / count1) * 100 + "%
2017 Landsat-8 78 cloudy");
20 var ndwi = series.normalizedDifference([’B2’, ’
Table 5. Datasets used to calculate Indus River Water occurance B4’]);
21 var ndwi_thres = (ndwi.gte(0)).toByte();
22 return (ndwi_thres); };
23 // Landsat-5 TOA with FMASK Collection
24 var l5 = ee.ImageCollection(’LANDSAT/
LT5_L1T_TOA_FMASK’).filterBounds(aoi)
25 .filterDate(’2008-01-01’,’2012-12-30’)
11.4 Methodology 26 var l5_umasked_select = l5.select([2,4],["B2","B4
"]);
27 var l5_new = l5.select([2,4],["B2","B4"]);
28 // Landsat-8 TOA with FMASK Collection
29 var lc8 = ee.ImageCollection(’LANDSAT/
LC8_L1T_TOA_FMASK’).filterBounds(aoi)
30 .filterDate(’2013-01-01’,’2017-12-30’)
31 var lc8_unmasked_select = lc8.select([3,5],["B2","
B4"]);
32 lc8 = lc8.map(quality_mask)
33 var lc8_new = lc8.select([3,5],["B2","B4"]);
34 // Landsat-5 & Landsat-8 collection merge unmask
35 var landsat_unmasked = ee.ImageCollection(
l5_umasked_select.merge(lc8_unmasked_select));
36 // Landsat-5 & Landsat-8 collection merge after
mask
37 var landsat = ee.ImageCollection(l5_new.merge(
lc8_new));
38 var densityViz = {"opacity":1,"min":1,"max":10,"
palette":["ffffff","0007ff"]}
39 var startYear = 2008; var endYear = 2017;
40 var density = ndwi(startYear);
41 for(var i =startYear+1;i<=endYear;i++){
42 if (i == 2012){}
43 else{
44 density = density.add(ndwi(i));}}
45 Map.addLayer(density,densityViz);

Listing 1. Google Earth Engine javaScript API code

Fig. 4. Methodology for water surface density
11.6 Processing time & memory 12 Conclusion
The Script takes around 17minutes to complete with 13 References
a CPU time of around 25000 and peak memory usage is
163MB. Google earth engine is a scale cloud processing sys-
tem that process only data equal to map extent in multiple
servers. Print and Export command is most costly as earth
engine cloud servers has to process all the available datasets
for a particular task. Google earth engine profiler displays in-
formation about CPU and memory usage (per algorithm and
asset) resulting from computations.

Fig. 5. Google earth Engine profiler

11.7 Results
Google earth engine script results can be exported to
google drive or can be directly downloaded or used in some
other application through earth engine API. Following is the
Water density plot displayed in QGIS software obtained as a
result from the case study:

Fig. 6. Google earth Engine result (water surface density)