Google Cloud Platform for Developers: Build highly scalable cloud solutions with the power of Google Cloud Platform

Google Cloud Platform

for Developers

Build highly scalable cloud solutions with the power of Google Cloud Platform

Ted Hunter

Steven Porter

BIRMINGHAM - MUMBAI

Google Cloud Platform for Developers

Copyright © 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Vijin Boricha

Acquisition Editor: Shrilekha Inani

Content Development Editor: Abhishek Jadhav

Technical Editor: Aditya Khadye

Copy Editor: Safis Editing

Project Coordinator: Judie Jose

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Graphics: Tom Scaria

Production Coordinator: Aparna Bhagat

First published: July 2018

Production reference: 1280718

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78883-767-5

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

PacktPub.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Contributors

About the authors

Ted Hunter is a software engineering consultant working with fortune 500 companies to design cloud-native solutions and drive public cloud adoption, primarily within the Google ecosystem. He has a background in full stack development, DevOps transformation, and designing enterprise data solutions. He is currently a Solution Architect at Slalom Consulting, serving clients in the Southeastern United States.

Steven Porter is a consulting technology leader for Slalom Atlanta's Software Engineering practice, a Microsoft Regional Director, and a Google Certified Cloud Architect. His major focus for the past 5+ years has been IT modernization and cloud adoption with implementations across Microsoft Azure, Google Cloud Platform, AWS, and numerous hybrid/private cloud platforms.

Outside of work, Steve is an avid outdoorsman spending as much time as possible outside hiking, hunting, and fishing with his family of five.

About the reviewer

Sanket Thodge is an entrepreneur by profession based out of Pune, India. He is an author of the book Cloud Analytics with Google Cloud Platform. He is a founder of Pi R Square Digital Solutions Pvt Ltd. With expertise as Hadoop Developer, Sanket explored Cloud, IoT, Machine Learning, and Blockchain. He has also applied for a patent in IoT and has worked with numerous startups and MNCs in providing consultancy, architecture building, development, and corporate training across globe.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Google Cloud Platform for Developers

Packt Upsell

Why subscribe?

PacktPub.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Why GCP?

The public cloud landscape

Amazon Web Services

Microsoft Azure

Google Cloud Platform

Standing on the shoulders of giants

A world-class global presence

Choosing your own adventure

Leading the way for big data

The Open Cloud and innovation

Dedication to customer success

Bottom-up security

In good company

Summary

The Google Cloud Console

Getting started – Google Cloud projects

Architectural role of Google Cloud projects

Creating a project

Free trials on GCP

The Google Cloud Console

Understanding the Cloud Console dashboard

The Google Cloud Shell

Launching the Cloud Shell

Supporting multiple sessions

Features and integrations

File management

Web Preview

The Cloud Shell Code Editor

Opening in Cloud Shell

Trying it out

Installing additional tools

Boost mode

Repairing the Cloud Shell

Other tools

Mobile apps

Developer tool integrations

Summary

APIs, CLIs, IAM, and Billing

Google Cloud APIs

Managing APIs

Google APIs Explorer

Trying out the APIs Explorer

The Google Cloud SDK

Installing the Google Cloud SDK

The gcloud command-line tool

The basics of gcloud

Command groups

Root commands

Global flags

Initializing the Google Cloud SDK

Authentication

Managing your Google Cloud SDK

Updating and rollbacks

Alpha and beta channels

Configurations in the Google Cloud SDK

Modifying configuration properties

Multiple configurations

Other command-line tools

bq

gsutil

kubectl

Automating tasks with gcloud

Modifying output and behavior

Formatting attributes

Formatting projections

Filtering

Google Cloud IAM

How IAM works

IAM roles

The structure of IAM policies

Organization-level policies

Project-level policies

Resource-level policies

Cross-project access

Managing IAM

Service accounts

Billing on Google Cloud

Billing accounts

Billing accounts and IAM

Budgets and billing alerts

Google Cloud Platform Pricing Calculator

Creating an estimate

Summary

Google App Engine

Compute services on the GCP

Google Compute Engine

Google Kubernetes Engine (GKE)

Google App Engine

Google Cloud Functions

General considerations

Google App Engine

Features and benefits

Developer velocity

Visibility

Scalability

Simple integrations

Structure of a Google App Engine application

Architecture of an App Engine solution

Microservices

Batch work and task queues

App Engine locations

IAM on the Google App Engine

App Engine service accounts

The standard and flexible environments

Standard environment

Flexible environment

Setting up the App Engine

The App Engine standard environment

Language support

Developing for the App Engine standard environment

The Python runtime

WSGI and CGI

Getting started

The App Engine development server

The Go runtime

Running multiple services locally

The Java 8 runtime

Deploying App Engine standard services

Deployment behavior

Splitting network traffic

Instance classes

Pricing in the standard environment

Spending limits

The App Engine flexible environment

Benefits of the flexible environment

More control over the infrastructure

Application portability

Language support

Developing for the flexible environment

Deploying App Engine flexible apps

Container technologies

Google Container Builder

Google Container Registry

Custom runtimes

Building custom runtime services

Deploying a service to the flexible environment

Pricing in the flexible environment

App Engine resources and integrations

Task queues

Push and pull queues

Push queues

Named queues

Pull queues

Creating tasks

Structuring tasks queues

Scheduled tasks

Deploying a cron definition

Trying the App Engine cron service

Scaling App Engine services

Autoscaling

Basic and manual scaling

Externalizing configuration and managing secrets

Application configuration files

Compute Engine metadata server

Runtime Configurator

Cloud Key Management Service (KMS)

General considerations

Networking and security

The App Engine firewall

Cloud Endpoints

Google Cloud IAP

Virtual private networks 

Summary

Google Kubernetes Engine

Google Kubernetes Engine

When to choose GKE

GKE or App Engine Flex

Creating and maintaining a GKE cluster

Node pools

Multi-zonal and regional clusters

Container Registry

Deploying workloads to GKE

Rolling updates

Rolling back updates

Scaling deployments

Manually scaling deployments

Automatically scaling deployments

Exposing GKE Services

Exposing services within a cluster

Exposing services to external traffic

Managing secrets with GKE

Creating/Storing secrets

Using secrets

Billing

Summary

Google Cloud Functions

Functions as a Service

Google Cloud Functions

Advantages of Cloud Functions

Price

Scalability

Developer velocity

Considerations when using Cloud Functions

Invoking Cloud Functions

HTTP functions

Processing HTTP requests

Background functions

Cloud Pub/Sub functions

Cloud Storage functions

Background function retries and termination

Developing Cloud Functions

Using the Cloud Console

Local development

Debugging functions

Deploying Cloud Functions

Deploying from a local machine

Deploying from a source repository

Integrating with other Google services

IAM and billing

Cloud Functions and IAM

Frameworks and tooling

Summary

Google Compute Engine

Understanding Compute Engine

IaaS

Infrastructure as Code (IaC)

More than virtual machines

When to use Compute Engine

A straightforward migration path

Host anything

Building a robust global presence

Long running and resource intensive processes

Security and compliance

Virtual machines on Google Compute Engine (GCE)

Machine types

Standard machine types

High-memory machine types

Mega-memory machine types

High-CPU machine types

Shared-core machine types

Custom machine types

Extended memory

Other resources

Disk storage

GPUs

Images

Public images

Premium images

Community images

Container images

Managing Compute Engine instances

Creating instances

Remote access

SSH access

SCP access

Remote Desktop Protocol (RDP) access

Metadata server

Default metadata

Project-wide metadata

Instance-specific metadata

Setting and removing metadata

Querying metadata from within instances

Trying it out

Modifying API responses

Startup and shutdown scripts

Startup scripts

Shutdown Scripts

Windows machines

Updates and patches

Availability policies

Maintenance behavior

Restart behavior

Relocating an instance

Storage solutions

Persistent disks

Standard and solid-state drive (SSD) persistent disks

Persistent disk performance

Boot disks

Managing persistent disks

Persistent disk snapshots

Local SSDs

Creating scalable solutions with GCE

Custom images

Creating images from a persistent disk

Copying an image

Creating images from snapshots

Golden images

Security concerns

Managed instance group (MIG)

Instance templates

Creating MIGs

Built for resilience

Autoscaling

Autohealing

Change management

Performing a rolling update

IAM and service accounts

Administrative operations

General roles

Compute resource roles

Network and security resource roles

Compute instance IAM

Pricing on GCE

Instance discounts

Preemptible instances

Committed use discounts

Sustained use discounts

Other resource costs

Always-free tier

Summary

NoSQL with Datastore and Bigtable

NoSQL solutions on GCP

NoSQL technologies

Google Cloud Datastore

When to use Datastore

Getting started

Datastore locations

Managing entities in the Cloud Console

Datastore core concepts

The structure of Datastore data

Entities, kinds, and properties

Data types

Entity identifiers

Namespaces

Ancestry paths and keys

Entity groups and consistency

Entity groups

Consistency and queries

Working with entities

Queries with GQL

Using GQL in the Cloud Console

Indexes

Single property indexes

Composite indexes

Datastore under the hood

The entities table

Key

Entity group

Kind

Properties

Custom indexes

Index tables

EntitiesByKind

EntitiesByProperty

EntitesByCompositeProperty and Custom Indexes

Datastore management and integrations

Administrative tasks

The Datastore Admin Console

gcloud operations

Integrations with other GCP services

App Engine standard environment

Other GCP services

Datastore pricing and IAM

Permissions in Datastore

Google Cloud Firestore

Comparison to Datastore

A promising future

Google Bigtable

Core concepts

Structure of Bigtable data

Columns and column families

Column families

Scalable and intelligent

Bigtable under the hood

Building on other Google technologies

Tablets and servers

Creating and managing clusters

Instances, clusters, and nodes

Development instances

Bigtable locations

Create a development cluster

Using gcloud

Scaling clusters

Promoting development clusters

Deleting a cluster

Interacting with data on Bigtable

The cbt command-line interface

The Bigtable HBase Client

Platform integrations

BigQuery external tables

Dataflow Bigtable IO

Bigtable pricing and IAM

Permissions in Bigtable

Summary

Relational Data with Cloud SQL and Cloud Spanner

Google Cloud SQL

Configuring Cloud SQL instances

Creating a Cloud SQL instance

Database engines

MySQL generations

Machine and storage types

Choosing a machine type

Configuring storage

Cloud SQL locations

When to use multiple instances

Connecting to Cloud SQL

Authorized networks

Connecting with gcloud

SSL support

Establishing an SSL Connection

The Cloud SQL Proxy

Setting up the Cloud SQL Proxy

Authenticating with the Cloud SQL Proxy

Trying it out

Managing Cloud SQL instances

Maintenance operations

Importing data to Cloud SQL

Exporting data to cloud storage

Backups and recovery

Trying it out

Point-in-time recovery

Updates

Database flags

Database flags and SLAs

Replicas and high availability

Read-only replicas

External replicas

High availability

Forcing a failover

Scaling Cloud SQL instances

Scaling Storage

Scaling compute

Alerting on resource pressure

Horizontal scaling

Migrating databases to Cloud SQL

Cloud SQL IAM and users

IAM policies

Database users

Default and system users

Additional users

Changing user passwords

Cloud SQL Proxy users

Cloud SQL pricing

Google Cloud Spanner

Instances and instance configurations

Regional configurations

Multi-region configurations

Nodes, databases, and tables

Creating a Cloud Spanner instance

Importing data into Cloud Spanner

Performing a simple query

Understanding Cloud Spanner

Cloud Spanner and CAP theorem

Maintaining consistency

TrueTime and linearization

Paxos groups

Read operations

Write operations

Transactions

Database design and optimizations

Query execution plans

Primary keys

Data collocation and interleaving

Secondary indexes and index directives

Cloud Spanner administration

Cloud Spanner IAM Roles

Cloud Spanner prices

Summary

Google Cloud Storage

GCS basics

Buckets

Bucket names

Domain-named buckets

The global bucket namespace

Objects

Object data

Object metadata

Virtual file structures

Using gsutil

Creating and using a bucket

Uploading files to GCS

Storage classes and locations

Regional and Multi-Regional Storage

Standard and durable reduced availability

Nearline and Coldline Storage

Cloud Storage locations

Nearline and Coldline Storage locations

Choosing the right storage class

Cloud Storage pricing

Bucket and object storage classes

Automating object management

Monitoring lifecycle events

Object versioning

Data governance in Cloud Storage

Cloud Storage IAM

ACLs

Limitations of concentric access control

Customer supplied encryption keys

Signed URLs

Capabilities and integrations

Integrating with Google Cloud Functions

Static web content and Backend Buckets

Summary

Stackdriver

Lessons from SRE

Monitoring and alerting

Preparation for this chapter

Stackdriver basics

Stackdriver and GCP projects

Creating and linking a Stackdriver account

Stackdriver Logging

Filtering and searching

Basic filtering

Advanced filtering

Exporting Stackdriver logs

Exporting to Cloud Storage

Exporting to BigQuery and Cloud Pub/Sub

Monitoring and alerting

The Stackdriver Monitoring console

Exploring Stackdriver metrics

Creating dashboards

Stackdriver alerting policies

Policy conditions

Creating an alerting policy

Notifications and documentation

Stackdriver incidents

Other types of metrics

Error reporting

Investigating errors

Stackdriver APM

Stackdriver Trace

Investigating application latency

Stackdriver Debugger

Debugging the todos services

Logpoints

Stackdriver Profiler

Summary

Change Management

Preparing for this chapter

Google Cloud Source Repositories

Google Cloud Deployment Manager

Declarative configuration management

Basic configurations

Resource types and properties

Deployments

Deploying a simple configuration

Deployment manifests

Updating deployments

Create and delete policies

Maintaining deployment state

Remediation

Templates

Creating a template

Other template features

Cloud Launcher and Deployment Manager

Runtime Configurator

Watchers

Waiters

Google Cloud Container services

Google Container Registry – GCR

Container Builder

Build triggers

Continuous deployment in Google Cloud

Summary

GCP Networking for Developers

Networking fundamentals

Virtual private networks

Subnetworks

Configuring VPC networks

Networks and compute resources

Firewall rules

Components of a firewall rule

Action

Direction

Target

Source or destination

Protocol and port

Priority

Securing networks with firewall rules

Routes

IP addresses

Internal and external IP addresses

Ephemeral and static IP addresses

Global IP addresses

Google load balancers

Network load balancers

Target pools

Forwarding rules

Health checks

Failover ratio and backup pools

Creating a TCP network load balancer

Internal load balancing

Global load balancers

Components of global load balancers

Backend services

Target proxies

Global forwarding rules

SSL and TCP proxies

HTTP(S) load balancers

Autoscaling load balanced resources

Google Cloud DNS

Access control and API management

Google Cloud Endpoints

Services

API providers

Access and discovery

Identity-Aware Proxy

Cloud Armor

Summary

Messaging with Pub/Sub and IoT Core

Google Cloud Pub/Sub

Topics and subscriptions

Push and pull message delivery

Pull subscriptions

Push subscriptions

Choosing a subscription model

Message acknowledgment

Nacking messages

Designing for resilience

Message loss

Processing failures

Duplicate messages

Out-of-order messages

Google Cloud IoT Core

Device management and registries

Device authentication and security

Consuming device data

Summary

Integrating with Big Data Solutions on GCP

Big data and Google Cloud Platform

Cloud Dataflow

Evolution of data processing at Google

Pipelines

Collections

Transformations

Element-wise transforms

Aggregate transforms

Composite transforms

Sources and sinks

Creating and executing pipelines

Executing pipelines locally

Executing pipelines on Cloud Dataflow

Executing streaming pipelines

Pipeline templates

Google provided pipeline templates

Managing Cloud Dataflow jobs

Google BigQuery

How BigQuery executes queries

Integrating with BigQuery

BigQuery as a Cloud Dataflow Sink

Batch loading files from Cloud Storage

Streaming inserts

Exploring BigQuery data

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

Google Cloud Platform (GCP) provides autoscaling compute power and distributed in-memory cache, task queues, and datastores to write, build, and deploy Cloud-hosted applications.

With Google Cloud Platform for Developers, you will be able to develop and deploy scalable applications from scratch and make them globally available in almost any language. This book will guide you in designing, deploying, and managing applications running on Google Cloud. You'll start with App Engine and move on to work with Container Engine, compute engine, and cloud functions. You'll learn how to integrate your new applications with the various data solutions on GCP, including Cloud SQL, Bigtable, and Cloud Storage. This book will teach you how to streamline your workflow with tools such as Source Repositories, Container Builder, and StackDriver. Along the way, you'll see how to deploy and debug services with IntelliJ, implement continuous delivery pipelines, and configure robust monitoring and alerting for your production systems.

By the end of this book, you'll be well-versed with all the development tools of Google Cloud Platform, and you'll develop, deploy, and manage highly scalable and reliable applications.

Who this book is for

Google Cloud Platform for Developers is for application developers and DevOps engineers that wish to become familiar with the various service offerings available on Google Cloud Platform. This book will enable you to fully leverage the power of Google Cloud Platform to build resilient and intelligent software solutions.

What this book covers

Chapter 1, Why GCP?, this chapter introduces readers to the Google Cloud Platform. It provides an overview of cloud computing, a brief history of GCP, as well as a comparison to other public cloud providers.

Chapter 2, The Google Cloud Console, this chapter serves to familiarize readers with the primary user interfaces they will use when interacting with Google Cloud.

Chapter 3, APIs, CLIs, IAM, and Billing, in this chapter, readers will learn about the various command line tools provided by Google for managing cloud resources. Readers will also learn about the other tools that will enable them to manage their Google Cloud projects throughout the book.

Chapter 4, Google App Engine, this chapter will explain what Google App Engine (GAE) is, the driving philosophies behind it, and how to use it to run highly-scalable services.

Chapter 5, Google Kubernetes Engine, this chapter is about the Google Container Engine (GKE) platform for running and managing services on Google Cloud.

Chapter 6, Google Cloud Functions, this chapter is about creating and executing Cloud Functions using Google's serverless platform.

Chapter 7, Google Compute Engine, this chapter is about Google's IaaS offering: Google Compute Engine (GCE). This chapter will introduce readers to on-demand VMs and how they can be managed, scaled, and customized to the user's needs.

Chapter 8, NoSQL with Datastore and Bigtable, this chapter will introduce readers to the document based storage solutions offered by Google, including Datastore (plus the new Firestore), and Bigtable.

Chapter 9, Relational Data with Cloud SQL and Cloud Spanner, this chapter will cover Google's relational data storage solutions, including managed MySQL and PostgreSQL via Cloud SQL, as well as globally consistent relational data via Cloud Spanner.

Chapter 10, Google Cloud Storage, this chapter is about Google's unified object storage platform: Google Cloud Storage (GCS).

Chapter 11, Stackdriver, this chapter will cover Google's Stackdriver monitoring, logging, and diagnostics suite to drive application insights, availability, and fast incident resolution.

Chapter 12, Change Management, this chapter will introduce readers to the various platform tools Google offers around the developer/operations experience, including source control, building and deploying services.

Chapter 13, GCP Networking for Developers, this chapter will introduce readers to networking on Google Cloud, covering the products available and how to use them to build custom networking and security solutions. These topics will be presented in a manner appropriate for developers rather than networking professionals.

Chapter 14, Messaging with Pub/Sub and IoT Core, this chapter will introduce readers to the distributed messaging offerings on Google Cloud. Readers will learn how to leverage Google Cloud Pub/Sub for high-throughput messaging used both in service to service communications and Big Data ingestion pipelines, as well as Cloud IoT Core for widely distributed event-driven application architectures.

Chapter 15, Integrating with Big Data Solutions on GCP, this chapter will provide a high level overview of big data solutions on Google Cloud Platform. Users will learn how to build highly scalable, fully managed big data solutions with the power of Cloud Dataflow and BigQuery.

To get the most out of this book

This book is geared towards readers with a familiarity of basic application development and DevOps concepts. The exercises provided in this book include Java, Python, Node.js, Go, SQL, and shell scripting. These exercises are designed to be simple and easy to complete without prior knowledge of a specific language or framework.

All examples and exercises in this book can be completed directly within the Google Cloud Console and Google Cloud Shell, however many users will prefer to work within a local development environment. To do so, download and install the Google Cloud SDK available at https://cloud.google.com/sdk/.

Download the example code files

You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packtpub.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Google-Cloud-Platform-for-Developers. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/GoogleCloudPlatformforDevelopers_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Web Preview runs on port 8080 by default, but can be mapped to any port from 8080-8084 via the Web Preview settings

A block of code is set as follows:

{

    eventId: 27819225098479,

    timestamp: 2018-01-27T18:11:24.836Z,

    eventType: providers/cloud.pubsub/eventTypes/topic.publish,

    resource: projects//topics/,

    data: {

      @type: type.googleapis.com/google.pubsub.v1.PubsubMessage,