GCP

Google Cloud
Professional Cloud
Database Engineer

© ANKIT MISTRY – GOOGLE CLOUD

Google Certified
Professional Cloud Database
Engineer

© ANKIT MISTRY – GOOGLE CLOUD

Database Concepts
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

Structure, Semi-structured &
unstructured Data
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

Structure Data
 Fixed Schema
 Stored in rows & column format - Table Book_id Book_name Author_id
 Relation exist between data 100 C 1
 Almost 20% of enterprise data 101 Java 1
 Stored in order
102 Python 2
 Require less storage
 SQL can be used to interact with data
Each row has same number of columns Author_id Author_name
 MySQL, Oracle SQL, PostgreSQL , MSSQL 1 John
 In GCP, Cloud SQL, Cloud Spanner 2 Alice

© ANKIT MISTRY – GOOGLE CLOUD

 Semi-structured Data

<employees>
<employee> {"employees":[
<firstName>John</firstName> { "firstName":"John", "lastName":"Doe" },
<lastName>Doe</lastName> { "firstName":"Anna", "lastName":"Smith" },
</employee> { "firstName":"Peter", "lastName":"Jones" }
<employee> ]}
<firstName>Anna</firstName>
<lastName>Smith</lastName>  No SQL Database
</employee>  JSON, XML, Document DB – No Fixed
<employee> schema
<firstName>Peter</firstName>
 MongoDB, Cassandra, Redis, Neo4j
<lastName>Jones</lastName>
</employee>  In GCP, BigTable, DataStore, memoryStore
</employees>
© ANKIT MISTRY – GOOGLE CLOUD
Unstructured Data
 No Fixed Schema
 No Structure within data
 occupies much more space for storage
 Almost 80% of data
 Video, Audio, Binary, Zip file kind of data are unstructured data
 Google Cloud Storage, File store inside GCP to store Unstructured
data

© ANKIT MISTRY – GOOGLE CLOUD

OLTP & OLAP

© ANKIT MISTRY – GOOGLE CLOUD

 OLTP
 OLTP – Online Transaction Processing
 Simple Query
 Large number of small transaction
 Traditional RDBMS
 Database modification
 Popular Database - MySQL, PostgreSQL, Oracle, MSSQL
 ERP, CRM, Banking application
 GCP - Cloud SQL, Cloud spanner

© ANKIT MISTRY – GOOGLE CLOUD

 OLAP
OLAP – Online Analytical Processing
 Data warehousing
 Data is collected from multiple sources
Complex Query
 Data analysis
 Google Cloud Big Query – Petabyte Data warehouse
 Reporting Application, Web click analysis, BI Dashboard app

© ANKIT MISTRY – GOOGLE CLOUD

Vertical – Horizontal Scaling

8 VCPU - 16 GB
8 VCPU - 16 GB
Horizontal
Vertical Scaling
Scaling

12 VCPU - 24 GB

© ANKIT MISTRY – GOOGLE CLOUD

 RTO & RPO
 Data loss : 13 hours

 System Downtime : 7 Hours

Downtime RTO

Disaster System
Last Backup happens Recovered

5th March 5th March 5th March

3:00 AM 4:00 PM 11:00 PM

Loss data RPO

© ANKIT MISTRY – GOOGLE CLOUD

 RTO & RPO
 RTO – Recovery Time objective
 Maximum time for which system can be down

 RPO - Recovery Point objective

 Maximum time for which organization can tolerate Dataloss

© ANKIT MISTRY – GOOGLE CLOUD

 RTO & RPO
 RTO – Recovery Time objective
 Maximum time for which system can be down

 RPO - Recovery Point objective

 Maximum time for which organization can tolerate Dataloss

© ANKIT MISTRY – GOOGLE CLOUD

Database Design Consideration
 Latency
 IOPS
 Types of Data
 Size of Data
 Schema (Fixed vs Flexible)

© ANKIT MISTRY – GOOGLE CLOUD

SQL & NOSQL Type
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

SQL Type
Relational - OLTP OLAP

© ANKIT MISTRY – GOOGLE CLOUD

NoSQL Type
Column Document Key-value Graph

Key1 Val1

Key2 Val2

Key3 Val3

Key4 Val4

© ANKIT MISTRY – GOOGLE CLOUD

Different GCP
Database Product
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

Different Database product
Structured
Semi-
Structured
Transactional Analytical

Big Table Data store

Cloud SQL Big Query (Row Key) Fire store

Cloud Spanner https://cloud.google.com/products/databases

© ANKIT MISTRY – GOOGLE CLOUD

 CHOOSING CORRECT
DATABASE

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
RDBMS in GCloud
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

 RDBMS in GCloud
 You need relational databases as backend for your application
 Which RDBMS system you want to use
 MySQL
 Postgress
 How will you setup in Google Cloud
 Install MySQL on Google Compute Engine
 But It will not be managed services
 What other solutions
 Managed MySQL services – Cloud SQL

© ANKIT MISTRY – GOOGLE CLOUD

 Google Cloud SQL
 Fully managed Relational database services for MySQL, PostgreSQL & SQL Server
 Lift & shift above database
 Regional Database with 99.95% SLA
 Storage up to 30 TB
 Vertical Scaling - Scale up to 96 core & 624 GB Memory
 Horizontal Scaling with Read Replicas - To transfer workload to other instance
 Data is encrypted with Google managed key or CMEK
 Cloud SQL can be accessed from anywhere like – App Engine, Compute Engine…
 Used for storing Transactional database
 Ecommerce, CRM kind application backend.

© ANKIT MISTRY – GOOGLE CLOUD

 Google Cloud SQL
 Due to managed Service No maintenance & auto update
 Back-up Database
 On-demand Backup
 Schedule backup
 Database migration service (DMS)
 migrate data from different SQL system to Cloud SQL
 Point-in Time Recovery

© ANKIT MISTRY – GOOGLE CLOUD

 Google Cloud SQL
 Create Google Cloud SQL Instances
 MySQL managed
 Postgre – Later
 Explore Cloud SQL Features & Properties

© ANKIT MISTRY – GOOGLE CLOUD

 Connect to Cloud SQL
 With Public IP – Easiest one
 With Cloud Proxy
 With Private IP address
 Adding Users
 Built-in users
 IAM users

 Connect with above two kind of users

 Secure Connect using SSL certificate
 Some Database Operations

© ANKIT MISTRY – GOOGLE CLOUD

 Lift-shift
Migrate from
 AWS RDS or Azure SQL or On-premises to Cloud SQL
 Database Migration or manual Migration

On-
premises
© ANKIT MISTRY – GOOGLE CLOUD
Zonal
Failover

© ANKIT MISTRY – GOOGLE CLOUD

 More Feature of Cloud SQL
 Backup demo & restore
 Export database
 Create Read replicas
 More instance operation start stop delete clone restart
 Cloud SQL IAM role
 Create Postgres Account & connect
 Cloud SQL Pricing calculator
 SQL instance management from command line gcloud

© ANKIT MISTRY – GOOGLE CLOUD

 Bare Metal
 What about database other than MySQL, Postgres, SQL Server
 Specialized Database like Oracle, SAP HANA, DB2
 GCP doesn’t provide any managed service for above Database

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
 Few Question
 Suppose you want to store 500 TB of Relational Database
 Want to write into Read Replica kind of Instances
 Want Horizontal Scalability
 Strong Consistency between all instance
 Distributed Global Scale Database
 Cloud Spanner is the Solution.

© ANKIT MISTRY – GOOGLE CLOUD

Google Cloud Spanner
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

 Google Cloud Spanner
 Distributed & scalable solution for RDBMS in GCP  Cloud native solution – specific to GCP
 Fully managed, Mission critical application  Lift & Shift not possible, Not recommended
 Horizontal Scalable  Spanner = Cloud SQL + Horizontal Scalable
 use when Data volume > 2 TB
 Scale to petabyte
 Costlier than Cloud SQL
 Regional/ Multi-region level instance can be
 Cloud SQL has just Read replicas,
 where as in cloud spanner horizontal read/write
created
across region
 Built on Google Cloud network
 Highly scalable, Petabyte scale
 Backup and Restore, point-in-time recovery
 Data is strongly typed.
 Must define schema database  Data export
 Datatype for each column of each table must be  can not export with gcloud
defined.
 Cloud Console or Cloud Dataflow Job
 99.999% availability
© ANKIT MISTRY – GOOGLE CLOUD
Spanner vs RDBMS-SQL

Spanner Cloud SQL

Availability High During failover little downtime
Scalable Horizontal vertical
Price Costly Cheaper than spanner
SQL/Schema Support yes yes
Replication High Only Read Replica

© ANKIT MISTRY – GOOGLE CLOUD

 Avoid HotSpots
 Choosing Primary is very important.
 https://cloud.google.com/spanner/docs/schema-design

© ANKIT MISTRY – GOOGLE CLOUD

Avoid HotSpots

© ANKIT MISTRY – GOOGLE CLOUD

Avoid HotSpots

© ANKIT MISTRY – GOOGLE CLOUD

 [Hands-on] Cloud Spanner
 Create Spanner Instance
 Create database lib_db
 Create 2 Table
 Author
 AId
 AName
 Book
 AId
 BId
 BTitle
 Insert records in Table
 Fetch same record with Python code

© ANKIT MISTRY – GOOGLE CLOUD

After Job Done
make sure to delete
Spanner Instance

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
AlloyDB for
PostgreSQL

© ANKIT MISTRY – GOOGLE CLOUD

 AlloyDB for PostgreSQL
 Fully managed PostgreSQL compatible database
 4X faster than standard PostgreSQL for transactional load
 100X faster analytical queries than standard PostgreSQL
 Works for OLTP & OLAP application
 Simplified management with machine learning-enabled autopilot systems
 AlloyDB offers a 99.99% uptime SLA
 Disaggregation of compute and storage
 In-built ML functionality
 https://cloud.google.com/blog/products/databases/alloydb-for-postgresql-intelligent-scalable-storage
 Let’s create AlloyDB instance

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
NoSQL In GCP

Datastore Firestore BigTable MemoryStore

© ANKIT MISTRY – GOOGLE CLOUD

Datastore & FireStore
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

History
 Datastore is one feature with Google App Engine in 2008
 Cloud Datastore was announced as a standalone product in 2013
 In October 2014, Firebase was acquired by Google – Web Mobile backend Database
 In October 2017, Firebase launched Cloud Firestore
 In 2018, the second-generation Firestore database was opened to general availability.

© ANKIT MISTRY – GOOGLE CLOUD

Naming in GCP
 In Single Project one can use either Datastore or Firestore
 You can not use both in single project
 Once you have selected mode + inserted data, you can not go back
 Still you want to use, need to create another project
Product
 Once region is selected, can not be changed later
Firestore
 In the future, all existing Datastore databases will be automatically
upgraded to Firestore in Datastore mode.

Native mode Datastore mode

Firestore Datastore

© ANKIT MISTRY – GOOGLE CLOUD

Cloud Datastore
 Highly scalable NoSQL database for web and mobile
Datastore RDBMS
 Firestore is the next generation of Datastore.
Kind Table
 Serverless Fully managed – No management required
Entity Row
 Select Region & start putting your data
 Schemaless Property Column

 ACID transactions support - so can be used for transactional Key Primary Key
app like Banking
 For Query - SQL-like queries, GQL
 Multiple indexes Support
 Nicely integrated with App Engine
Use case
 Session Info, Product catalog
© ANKIT MISTRY – GOOGLE CLOUD
[Hands-on] Cloud Datastore
 Select Region
 Create some kind & entity from Console
 Execute GQL Query on top
 Demo on indexing
 Access Datastore Operation from Python
 pip install google-cloud-Datastore
 https://cloud.google.com/docs/samples?language=python&product=datastore

© ANKIT MISTRY – GOOGLE CLOUD

 Cloud Firestore
 Firestore is the next generation of Datastore
 Highly scalable NoSQL database
Firestore RDBMS
 Collection & Document Model
Collection group Table
 Two mode
Document Row
 Native Mode
 Datastore mode Field Column

 Real-time updates Document ID Primary Key

 Mobile and Web client libraries
 Let’s see in Action

© ANKIT MISTRY – GOOGLE CLOUD

Datastore & Firestore
Pricing

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
Google Cloud BigTable
BY ANKIT MISTRY

© ANKIT MISTRY – GOOGLE CLOUD

 Cloud BigTable
 Wide column NOSQL database  No Multi column index
 Sparse Multi-dimensional array  Only Row key based indexing – key based
 Fully managed But Not serverless reading

 Scale horizontally with Multiple Node  Seamless integration with

 Warehouse – BigQuery, ML Product
 Scale to huge Volume of data - Petabyte scale
 Data stored at column wise  Equivalent Open Source DB - Cassandra, Hbase

 Column are grouped into column family  Some use case in

 Financial, IOT
 Low latency
 Time series Data
 Handles millions of request per second
 How to access
 cbt – command line (part of Google cloud SDK)
 Hbase API
© ANKIT MISTRY – GOOGLE CLOUD
Cloud BigTable Storage

Personal_data_cf Professional_data_cf
Row Key
name age salary designation company

Professional_data_cf:salary

© ANKIT MISTRY – GOOGLE CLOUD

BigTable Architecture

© ANKIT MISTRY – GOOGLE CLOUD

Row key Design
 Design Row Key is very important
 Row key is equivalent to primary key in Table in RDBMS
 No Join, No Foreign Key support
 Rows are sorted by rowkeys
 Design Row key by keeping in your mind
 which is your frequent query in application
 No Hot spotting
 Don’t use monotonically increasing key
 Low cardinality attributes
 Join multiple attribute for key
 Design such way data distributed evenly across node
 Bigtable uses Colossus file system for data storage
 Bigtable instance store only metadata

© ANKIT MISTRY – GOOGLE CLOUD

[Hands-on]
Google Cloud BigTable

© ANKIT MISTRY – GOOGLE CLOUD

BigTable Pricing

© ANKIT MISTRY – GOOGLE CLOUD

© ANKIT MISTRY – GOOGLE CLOUD
Google Cloud Memorystore

© ANKIT MISTRY – GOOGLE CLOUD

 Cloud Memorystore
 Fully managed In-memory database for Redis and Memcached
 sub-millisecond data access – very very low latency
 Two engine supported
 Redis
 Memcached
 zero code change require when you lift & shift to cloud
 Protected via Internal IP
 Highly scalable
 Highly available with 99.9% SLA
 Import/Export data from Cloud Storage to memory store
 Used when real-time responses required ,
 finance, telecom, Health care, Gaming, leaderboard
© ANKIT MISTRY – GOOGLE CLOUD
[Hands-on] Memorystore - Redis
 Create Memorystore – Redis instance
 Create Client VM to connect in same network
 Install Redis client
 sudo apt install redis-tools
 Redis command to store & retrieve data
 Export data to GCS
 Importing Data from GCP to another Memorystore Redis instance

© ANKIT MISTRY – GOOGLE CLOUD

Memorystore - Memcached
 Create Memorystore – Memcached instance
 Create Client VM to connect in same network
 Install telnet client
 sudo apt install telnet
 Memchached command to store & retrieve data
 Export & import data to GCS

© ANKIT MISTRY – GOOGLE CLOUD

Database Migration

© ANKIT MISTRY – GOOGLE CLOUD

THANK YOU

© ANKIT MISTRY – GOOGLE CLOUD