This document is part of a series that discusses architectural patterns that enterprises can use to optimize their cloud footprint at scale using Active Assist. The tutorial shows you how to build an automation pipeline for Active Assist recommendations that works with the GKE Enterprise toolchain. It's intended for people who are using Config Sync to manage their GKE Enterprise environments and Config Connector to manage Google Cloud resources. The other parts of the series are as follows:
- Patterns for using Active Assist at scale
- Using serverless pipelines with Active Assist
- Using the GKE Enterprise toolchain with Active Assist (this document)
The automation pipeline that you build in this tutorial can help you to achieve the following:
- Scaling the use of the Active Assist portfolio in your organization.
- Making Active Assist a part of your continuous integration and continuous delivery (CI/CD) pipeline.
- Controlling the review and actuation of Active Assist recommendations using constructs such as GitHub issues and pull requests.
This tutorial also uses kpt, an open source toolkit developed by Google to help you manage, manipulate, customize, and apply Kubernetes resource configuration data files.
Architecture
The following architectural diagram shows the components that you use in this tutorial.
The components are used in the following ways:
- A GitHub don't repeat yourself (DRY) repository, which is used for Config Connector templates that you deploy across projects in your Google Cloud organization.
- One or more GitHub repositories which are specific to a project or environment and hold hydrated configuration files. These hydrated repositories are for the environments that Config Sync manages. They use Config Connector to actuate and manage Google Cloud resources in the Google Cloud organization.
- An GKE cluster that uses Config Sync for version control and drift detection. This cluster also has Config Connector installed. Config Connector enables the cluster to manage Google Cloud resources across the Google Cloud organization.
- A Cloud Build trigger that triggers a build when a template is pushed into the GitHub DRY repository.
- A scheduled Cloud Build trigger that triggers a build periodically. The build job uses a kpt function. The function invokes the Active Assist Recommender APIs to fetch active recommendations. It reviews and parses recommendations to determine if the Google Cloud resources that Config Connector manages need to be resized or optimized. The kpt function creates a GitHub issue in the DRY repository with the details of the recommended change if the Config Connector-managed Google Cloud resources need to be resized or optimized.
The workflow for this architecture is as follows:
- Authorized teams with access to the DRY repository create and manage Config Connector templates in the repository.
- A Cloud Build job is triggered when a template is created or
modified and checked into the
main
branch. - The Cloud Build job hydrates the templates by invoking kpt setters. The job pushes the hydrated config files to the hydrated GitHub repository. Secret Manager is used to store GitHub deployment keys for the private repository.
- Config Sync monitors for changes to the hydrated repository and applies updates found in the repository to the managed cluster.
- Config Connector monitors for changes and actuates Google Cloud resources if any resources need to be created or updated as a result of the Kubernetes Resource Model (KRM) changes applied by the Config Sync.
- A scheduled Cloud Build trigger runs periodically to invoke the Recommender API to fetch active recommendations for the projects that Config Connector managed.
- The scheduled Cloud Build job executes a custom kpt function to invoke the Recommender API and fetch and parse active recommendations.
- The kpt function creates a GitHub issue that shows a comparison of the current resource configuration and the recommended configuration for the resource. With this approach, GitHub issues are created in the DRY repository, which makes it easier to keep track of repository changes.
Objectives
- Create the following sample GitHub repositories:
- A DRY repository for Config Connector KRMs.
- A repository to hold hydrated configuration files generated using kpt setters.
- Create an GKE cluster with Config Sync and Config Connector.
- Create a sample kpt function to retrieve Active Assist recommendations for projects managed by Config Connector.
- Create a Cloud Build trigger that gets triggered when a
template is pushed to the
main
branch of the DRY repository. - Create a scheduled Cloud Build job that runs periodically to retrieve available Active Assist recommendations for the resources being managed by Config Connector.
- Test the end to end pipeline with the stub recommendations provided in the GitHub repository for this tutorial.
Costs
In this document, you use the following billable components of Google Cloud:
- Cloud Build
- Cloud Run
- Firestore
- Pub/Sub
- Container Registry
- Cloud Scheduler
- Google Kubernetes Engine (GKE) Enterprise edition
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
-
In the Google Cloud console, go to the project selector page.
-
Select or create a Google Cloud project.
- Make a note of the Google Cloud project ID. You use this ID in the next section
when you set up your environment. This project is referred to throughout the
tutorial as the
build
project. -
Enable the Cloud Build, Firestore, App Engine, Pub/Sub, Cloud Run, Cloud Scheduler, and Cloud Source Repositories APIs.
You use the default application credentials for this tutorial. If you are prompted to create credentials on the Add credentials to your project page, click Cancel. -
Make sure that billing is enabled for your Google Cloud project.
Setting up your environment
In this tutorial, you run all of the commands in Cloud Shell.
In the Google Cloud console, activate Cloud Shell.
Set variables for the project ID and project number of the current
build
Google Cloud project:export RECO_MGR_PROJECT=PROJECT_ID gcloud config set project $RECO_MGR_PROJECT export RECO_MGR_PROJECT_NUMBER=$(gcloud projects describe $RECO_MGR_PROJECT --format='value(projectNumber)')
Replace
PROJECT_ID
with the project ID that you noted in the previous section.Set variables for the deployment region:
export REGION=us-central1 export ZONE=us-central1-a
Clone the repository that contains the code for the sample app used in this tutorial:
git clone https://github.com/GoogleCloudPlatform/activeassist-anthos-toolchain.git
Go to the project directory:
cd activeassist-anthos-toolchain
Build the pipeline
In this section, you create the components to build the pipeline. Active Assist recommendations are generated based on usage patterns and system metrics. Each recommendation category can use a different default window of time to analyze usage data and metrics based on which recommendations are generated. To test the end-to-end pipeline, the repository that you cloned in an earlier step provides sample recommendations (stubs) that you use to run the end-to-end pipeline.
Alternatively, if you are running the pipeline in a sample project that has existing resources and recommendations, you can make appropriate changes to the sample code and then run the pipeline.
Set up sample private GitHub repositories
In the following sections, you set up the sample GitHub repositories for this tutorial.
Set up a private DRY GitHub repository
Create a private GitHub repository for the DRY repository. Make a note of the name you give the repository.
In Cloud Shell, create an environment variable for your GitHub username and the name of the DRY repository:
export REPO_OWNER=YOUR_GITHUB_USERNAME export DRY_REPO_NAME=YOUR_PRIVATE_DRY_REPO
Replace the following:
YOUR_GITHUB_USERNAME
: your GitHub username.YOUR_PRIVATE_DRY_REPO
: the name of your DRY repository.
Create a personal access token (PAT) to create issues in this repository. The pipeline creates GitHub issues if there are Active Assist recommendations that need to be reviewed. For more information about creating PATs in GitHub, see the GitHub documentation.
When you set a scope for this token, select Full control of private repositories.
In Cloud Shell, create an environment variable for the PAT that you generated:
export GITHUB_TOKEN=YOUR_PERSONAL_ACCESS_TOKEN
Replace
YOUR_PERSONAL_ACCESS_TOKEN
with your own token.
Set up a private hydrated GitHub repository
Create a private GitHub repository for the hydrated repository. Make a note of the name you give the repository.
In Cloud Shell, set an environment variable for the hydrated repository:
export HYDRATED_REPO_NAME=YOUR_PRIVATE_HYDRATED_REPO export HYDRATED_REPO='[email protected]:$REPO_OWNER/$HYDRATED_REPO_NAME.git'
Replace
YOUR_PRIVATE_HYDRATED_REPO
with the name of your hydrated repository.Create a deploy key pair:
ssh-keygen -t rsa -b 4096 \ -C 'active-assist-robot' \ -N '' \ -f $(pwd)/active-assist-robot
A deploy key lets you deploy to your private GitHub repository when you run a Cloud Build job to hydrate config files.
Print the generated key:
cat $(pwd)/active-assist-robot.pub
Add the deploy key to the private GitHub repository. Make sure to select Allow write access when you add the deploy key. To learn how to add deploy keys to GitHub repositories, see the GitHub documentation for Managing deploy keys.
Upload GitHub keys to Secret Manager
In Cloud Shell, create a secret to store the private key from the deploy key pair:
gcloud secrets create github-ssh-key \ --data-file=$(pwd)/active-assist-robot
Create a secret to store the PAT:
echo $GITHUB_TOKEN | gcloud secrets create github-pat --data-file=-
Create an GKE cluster
In this section, you create a GKE cluster with the Config Connector add-on, create an identity, and configure Config Connector. You also configure Config Sync. You can use Config Sync to create a common configuration across all your infrastructure, including custom policies, and apply it both on-premises and in the cloud. Config Sync evaluates changes and rolls them out to all Kubernetes clusters so that your desired state is always reflected in your clusters.
In Cloud Shell, create a new GKE cluster with the Config Connector add-on enabled:
gcloud container clusters create sample-ops \ --machine-type n1-standard-4 \ --zone $ZONE \ --release-channel regular \ --addons ConfigConnector \ --workload-pool=$RECO_MGR_PROJECT.svc.id.goog \ --enable-stackdriver-kubernetes \ --enable-ip-alias
Complete the following sections in the Installing with the GKE add-on guide, to create an identity and configure Config Connector.
Install Config Sync in the GKE cluster that you created. When you configure Config Sync, you must do the following:
- Use a
token
to
grant Config Sync read-only access to Git.
Use the GitHub token that you
created when you set up a private DRY GitHub repository.
The token is available through the
$GITHUB_TOKEN
environment variable. - Configure Config Sync using gcloud.
Set the following settings:
- sourceFormat:
hierarchy
- syncRepo:
https://github.com/YOUR_GITHUB_USERNAME/YOUR_PRIVATE_HYDRATED_REPO
- syncBranch:
main
- secretType:
token
- policyDir: Don't fill in this option
- sourceFormat:
- Use a
token
to
grant Config Sync read-only access to Git.
Use the GitHub token that you
created when you set up a private DRY GitHub repository.
The token is available through the
Create a Cloud Build trigger to push to the hydrated repository
In the following sections, you create a Cloud Build trigger that is
triggered when templates are pushed to the main branch of your YOUR_PRIVATE_DRY_REPO
repository. This trigger runs the steps that hydrate the config-as-data KRM
templates in the YOUR_PRIVATE_DRY_REPO
repository and pushes the hydrated configuration files to your YOUR_PRIVATE_HYDRATED_REPO
repository.
Connect Cloud Build to your GitHub repositories
In this section, you
connect
the YOUR_PRIVATE_DRY_REPO
and YOUR_PRIVATE_HYDRATED_REPO
GitHub repositories to Cloud Build.
Go to the GitHub marketplace page for the Cloud Build app.
Click Setup with Google Cloud Build.
If prompted, sign in to GitHub.
Select Only select repositories.
Use the Select repositories drop-down to enable access to your
YOUR_PRIVATE_DRY_REPO
andYOUR_PRIVATE_HYDRATED_REPO
repositories through the Cloud Build app.Click Install.
Sign in to Google Cloud. The Authorization page is displayed and you are prompted to authorize the Google Cloud Build app to connect to Google Cloud.
Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.
Select your Google Cloud project.
Select the consent checkbox and click Next.
Click Install.
Sign in to Google Cloud. The Authorization page is displayed and you are prompted to authorize the Google Cloud Build app to connect to Google Cloud.
Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.
Select your Google Cloud project.
Select the consent checkbox and click Next.
In the Select repository page that appears, select the following GitHub repositories:
YOUR_PRIVATE_DRY_REPO
YOUR_PRIVATE_HYDRATED_REPO
Click Connect and then click Done.
Create a Cloud Build trigger for the DRY repository
In Cloud Shell, run the following command:
envsubst < cloudbuild.template.yaml > cloudbuild.yaml
The command generates a
cloudbuild.yaml
file.Create the trigger:
gcloud beta builds triggers create github \ --name ActiveAssistDemo \ --repo-name=$DRY_REPO_NAME \ --repo-owner=$REPO_OWNER \ --branch-pattern="main" \ --build-config=cloudbuild.yaml
Give the Cloud Build service account permission to access Secret Manager:
gcloud secrets add-iam-policy-binding github-ssh-key \ --member="serviceAccount:${RECO_MGR_PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \ --role="roles/secretmanager.secretAccessor" gcloud secrets add-iam-policy-binding github-pat \ --member="serviceAccount:${RECO_MGR_PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \ --role="roles/secretmanager.secretAccessor"
Create a scheduled Cloud Build trigger for Active Assist recommendations
In the following sections, you create a scheduled Cloud Build trigger
that runs periodically. This trigger fetches Active Assist
recommendations using a kpt function and determines if there are any active
recommendations for the resources in your
YOUR_PRIVATE_HYDRATED_REPO
repository. The kpt function also creates a GitHub issue in your
YOUR_PRIVATE_HYDRATED_REPO
repository if there are active recommendations for resource configuration that
need to be reviewed and actuated.
Generate a Cloud Build image
In this section, you generate a Cloud Build image that has kpt, gh, and Node components.
In Cloud Shell, build and push a Docker image to Container Registry:
gcloud auth configure-docker docker build -t gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1 ./recommender-kpt-function docker push gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1
Create a Cloud Build trigger for your hydrated repo
In Cloud Shell, create the configuration file needed to set up the scheduled Cloud Build trigger:
envsubst < cloudbuild-scheduled-recommendations.template.yaml > cloudbuild-scheduled-recommendations.yaml
Create the Cloud Build trigger:
gcloud beta builds triggers create github \ --name ActiveAssistScheduledRecommendations \ --repo-name=YOUR_PRIVATE_HYDRATED_REPO \ --repo-owner=$REPO_OWNER \ --branch-pattern="main" \ --build-config=cloudbuild-scheduled-recommendations.yaml
Get the ID of this trigger:
export TRIGGER_ID=`gcloud beta builds triggers describe \ ActiveAssistScheduledRecommendations \ --format="value(id)"`
Create a Cloud Scheduler job to invoke your trigger
In Cloud Shell, create a service account:
gcloud iam service-accounts create build-invoker \ --description "Service Account used by Cloud Scheduler to invoke the sample scheduled Cloud Build job" \ --display-name "recommender-scheduler-sa" \ --project $RECO_MGR_PROJECT
Cloud Scheduler jobs use this service account to run the
recommender-parser
service.Give the service account the permissions to invoke a Cloud Build job:
gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \ --member serviceAccount:build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com \ --role roles/cloudbuild.builds.editor \ --project $RECO_MGR_PROJECT gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \ --member serviceAccount:build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com \ --role roles/serviceusage.serviceUsageConsumer \ --project $RECO_MGR_PROJECT
Create a Cloud Scheduler job to invoke the trigger that you created in the previous step:
gcloud scheduler jobs create http scheduled-build \ --project $RECO_MGR_PROJECT \ --time-zone "America/Los_Angeles" \ --schedule="0 */3 * * *" \ --uri="https://cloudbuild.googleapis.com/v1/projects/${RECO_MGR_PROJECT}/triggers/${TRIGGER_ID}:run" \ --description="Scheduler job to invoke Cloud Build" \ --oauth-service-account-email="build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com" \ --headers="Content-Type=application/json" \ --http-method="POST" \
Select
Y
if your see the following message:There is no App Engine app in the project.
If you are prompted to choose the region where you want your App Engine application located, select the
us-central
region.
Commit and push the Cloud Build configuration files to GitHub
Push the two Cloud Build configuration files that you created to your
YOUR_PRIVATE_DRY_REPO
repository:
git remote add dry https://github.com/$REPO_OWNER/$DRY_REPO_NAME.git
git add cloudbuild.yaml
git add cloudbuild-scheduled-recommendations.yaml
git commit -m "Added cloudbuild configuration YAMLs"
git push dry main
You might be prompted to enter your GitHub credentials when you push to your private repository.
Review the outcome of the Cloud Build job
When you commit and push changes to your
YOUR_PRIVATE_DRY_REPO
repository, the
Cloud Build job is triggered. If the Cloud Build job runs
successfully, several resources are created. In this section, you verify whether
the resources are created after the Cloud Build job completes.
In Cloud Shell, in your
sample-ops
cluster, validate that you have a namespace calledactiveassist-kcc
:kubectl get ns | grep activeassist-kcc
Config Connector deploys a running sample Compute Engine instance to your
PROJECT_ID
project.Validate that the Compute Engine instance is in the project:
gcloud compute instances list | grep \ computeinstance-sample-cloudmachine
The
MACHINE_TYPE
type for this machine isn1-standard-1
.
Run end-to-end tests
To let you test the end-to-end pipeline, the repository that you cloned for this
tutorial provides sample recommendations (stubs). You use these stubs to run the
end-to-end pipeline. The stub imitates an Active Assist
recommendation payload and has a recommendation to change the machine type for
the compute engine instance that was deployed from the n1-standard-1
instance
type to the g1-small
instance type.
In this section, you invoke the scheduled Cloud Build trigger
manually to run the job that uses a kpt function to fetch
Active Assist recommendations. You also verify that a GitHub
issue is created in your YOUR_PRIVATE_DRY_REPO
repository.
Open the Build Triggers page in the Google Cloud console.
Select the
ActiveAssistScheduledRecommendations
trigger.To manually test the trigger, click Run on the entry for your trigger on the triggers list.
The trigger creates a GitHub issue in your
YOUR_PRIVATE_DRY_REPO
repository. The issue is similar to the following:gcloud auth configure-docker docker build -t gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1 ./recommender-kpt-function docker push gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1
In the sample issue, the kpt function output shows that the current
MACHINE_TYPE
type for the Compute Engine instance isn1-standard-1
type. The Active Assist recommendation is to change it to ag1-small
type.Version control reviewers in your enterprise can review automated GitHub issues and action them as appropriate for your enterprise.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
What's next
- Learn more about Google Cloud serverless technologies.
- Read about how to integrate Policy Intelligence recommendations into an Infrastructure as Code (IaC) pipeline.