Configure cluster notifications for third-party services

This tutorial shows you how to configure third-party messaging services to receive Google Kubernetes Engine (GKE) cluster notifications.

Services such as Slack offer incoming webhooks, a simple way to post messages from apps into Slack. Cloud Run functions is a lightweight Compute Engine solution to create single-purpose, stand-alone functions that respond to Google Cloud events, such as cluster notifications, without the need to manage a server or runtime environment. When GKE sends a cluster notification using Pub/Sub, a trigger responds by executing an action, such as sending a Slack notification.

There are many third-party services built with cross-application messaging functionality, such as IFTTT. You can use this tutorial as a template for connecting with these services.

In this tutorial, you use Cloud Run functions and Pub/Sub to send notifications about GKE cluster events to Slack.

Objectives

  • Deploy a Slack application to receive external notifications from GKE.
  • Write a Cloud Run function that sends Pub/Sub notifications to Slack.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the GKE, Cloud Run functions, Cloud Build, Eventarc and Pub/Sub APIs.

    Enable the APIs

  5. Install the Google Cloud CLI.

  6. To initialize the gcloud CLI, run the following command: 

    gcloud init

  7. Update and install gcloud components:

    gcloud components update
    gcloud components install alpha beta

  8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  9. Make sure that billing is enabled for your Google Cloud project.

  10. Enable the GKE, Cloud Run functions, Cloud Build, Eventarc and Pub/Sub APIs.

    Enable the APIs

  11. Install the Google Cloud CLI.

  12. To initialize the gcloud CLI, run the following command: 

    gcloud init

  13. Update and install gcloud components:

    gcloud components update
    gcloud components install alpha beta
  14. Enable GKE cluster notifications.

Required roles

To get the permissions that you need to set up Slack notifications for cluster notifications, ask your administrator to grant you the following IAM roles on your project:

To ensure that the Compute Engine default service account has the necessary permissions to invoke the Cloud Run function, ask your administrator to grant the Compute Engine default service account the Cloud Functions Invoker (roles/cloudfunctions.invoker) IAM role on your project.

Slack notifications

To set up Slack notifications, you must create a Slack application, activate Incoming Webhooks for the application, and install the application to a Slack workspace.

Create the Slack application

  1. Join a Slack workspace, either by registering with your email or by using an invitation sent by a Workspace Admin.

  2. Sign in to Slack using your workspace name and your Slack account credentials.

  3. Create a new Slack app

    1. In the Create an app dialog, click From scratch.
    2. Specify an App Name and choose your Slack workspace.
    3. Click Create App.
    4. Under Add features and functionality, click Incoming Webhooks.
    5. Click the Activate Incoming Webhooks toggle.
    6. In the Webhook URLs for Your Workspace section, click Add New Webhook to Workspace.
    7. On the authorization page that opens, select a channel to receive notifications.
    8. Click Allow.
    9. A webhook for your Slack application is displayed in the Webhook URLs for Your Workspace section. Save the URL for later.

Write the Cloud Run function

When GKE publishes a cluster notification to a Pub/Sub topic, the event triggers Cloud Run functions to send a Slack notification.

  1. Create a new directory named gke_slack and change directory into it:

    mkdir ~/gke_slack && cd $_

  2. Create the following files in the gke_slack directory:

    index.js 

    const functions = require('@google-cloud/functions-framework');
const { IncomingWebhook } = require('@slack/webhook');

const url = process.env.SLACK_WEBHOOK;

const webhook = new IncomingWebhook(url);

// Optionally filter what notification types to forward to Slack.
// If empty, all types will be allowed.
const allowedTypeURLs = [];

// Register a CloudEvent callback with the Functions Framework that will
// be executed when the Pub/Sub trigger topic receives a message.
functions.cloudEvent('slackNotifier', pubSubEvent => {
    const data = decode(pubSubEvent.data.message.data);

    // Send message to Slack.
    if (isAllowedType(pubSubEvent.data.message.attributes)) {
        const message = createSlackMessage(data, pubSubEvent.data.message.attributes);
        webhook.send(message);
    }
});

// decode decodes a pubsub event message from base64.
const decode = (data) => {
    return Buffer.from(data, 'base64').toString();
}

// isAllowedType can be used to filter out messages that don't match the
// allowed type URLs. If allowedTypeURLs is empty, it allows all types.
const isAllowedType = (attributes) => {
    if (allowedTypeURLs.length == 0) {
        return true;
    }
    for (var x in allowedTypeURLs) {
        if (attributes['type_url'] == allowedTypeURLs[x]) {
            return true;
        }
    }
    return false;
}

// createSlackMessage creates a message from a data object.
const createSlackMessage = (data, attributes) => {
    // Write the message data and attributes.
    text = `${data}`
    for (var key in attributes) {
        if (attributes.hasOwnProperty(key)) {
            text = text + `\n\t\`${key}: ${attributes[key]}\``
        }
    }
    const message = {
        text: text,
        mrkdwn: true,
    };
    return message;
}

    SLACK_WEBHOOK is a Cloud Run functions environment variable specifying the webhook URL created for your Slack application. You define the environment variable when you deploy the function.

    The webhook listens for and receives messages from Cloud Run functions. When GKE sends a cluster notification to Pub/Sub (the event), the function sends a message (the trigger) to the webhook URL, which in turn sends the message to the configured Slack workspace.

    You can expand the message in the createSlackMessage function to include much more, including text formatting and images. The isAllowedType function is provided to enable basic filtering of notifications by the type URL. You can specify what type URLs to allow in allowedTypeURLs. This function isn't necessary if you already filtered notifications, either in GKE or in your Pub/Sub subscription.

    package.json 

    {
  "name": "gke-slack",
  "version": "0.0.1",
  "description": "Slack integration for GKE, using Cloud Run functions",
  "main": "index.js",
  "dependencies": {
    "@slack/webhook": "6.1.0",
    "@google-cloud/functions-framework": "^3.0.0"
  }
}

    package.json describes the following attributes of the program:

    • Name, version, and description
    • Primary runtime file(s)
    • Dependencies

    You can add more dependencies, requirements, and other information as needed.

You should now have the index.js and package.json files in the gke_slack directory.

Deploy the Cloud Run function

You can deploy the Cloud Run function using either the Google Cloud CLI or the Google Cloud console.

gcloud

To deploy the function, run the following command in the gke_slack directory:

gcloud functions deploy slackNotifier \
  --gen2 \
  --trigger-topic=TOPIC_NAME \
  --runtime=nodejs14 \
  --entry-point=slackNotifier \
  --region=REGION \
  --source=. \
  --set-env-vars="SLACK_WEBHOOK=WEBHOOK_URL"

Replace the following:

The output is similar to the following:

Deploying function…
availableMemoryMb: 256
entryPoint: slackNotifier
environmentVariables:
  SLACK_WEBHOOK: https://hooks.slack.com/services/…
eventTrigger:
  eventType: google.pubsub.topic.publish
  failurePolicy: {}
  resource: projects/PROJECT_ID/topics/TOPIC_NAME
  service: pubsub.googleapis.com
labels:
  deployment-tool: cli-gcloud
name: projects/PROJECT_ID/locations/us-central1/functions/slackNotifier
runtime: nodejs10
serviceAccountEmail: PROJECT_ID@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/…
status: ACTIVE
timeout: 60s
updateTime: 'YYYY-MM-DDThh:mm:ssZ'
versionId: '1'

Console

  1. Go to the Cloud Run functions page in the Google Cloud console.

    Go to Cloud Functions

  2. Click Create function.

  3. On the Configuration page, perform the following steps:

    1. From the Environment drop-down list, select 2nd gen.
    2. For Function name, specify slackNotifier.
    3. For Region, specify a Compute Engine region.
    4. In the Trigger section, click ADD EVENTARC TRIGGER.
    5. In the opened window, check that the Event provider drop-down list, select Cloud Pub/Sub.
    6. Select the Pub/Sub topic you created when enabling cluster notifications.
    7. For Region, specify the same Compute Engine region as for the function.
    8. Click SAVE TRIGGER.
    9. Expand the Runtime, build, connections and security settings section.
    10. Under Runtime environment variables, click Add Variable.
    11. For Name, specify SLACK_WEBHOOK.
    12. For Value, specify the URL of the Internal Webhook created in Create the Slack application.
    13. Click Next.

  4. On the Code page, perform the following steps:

    1. Select Node.js 14 from the Runtime drop-down list.
    2. For Entry point, specify slackNotifier.
    3. In the navigation pane, select index.js and replace the code with the sample code in Write the Cloud Function.
    4. In the navigation pane, select package.json and replace the code with the sample code in Write the Cloud Function.
    5. Click Deploy.

After you've completed deployment of the Cloud Run function, you will receive a Slack notification whenever GKE sends a cluster notification.

Verify Slack notifications

If you used an Autopilot cluster, verify notifications as follows:

  1. Start a control plane upgrade.
  2. Wait for GKE to automatically upgrade your nodes to the new version. The time taken might vary based on configured maintenance windows and exclusions.
  3. After GKE upgrades your nodes, check Slack for a message.

If you used a Standard cluster, verify notifications as follows:

  1. Upgrade a specific node pool to a new version. If you don't want to change the GKE version on your nodes, you can upgrade to the same version currently on the nodes.

  2. After GKE upgrades your nodes, check Slack for a message.

The Slack notification is similar to the following:

Master is upgrading to version 1.20.10-gke.301.
    cluster_location: us-central1
    cluster_name: pubsub-cluster
    payload: {"resourceType":"MASTER", "operation":"operation-1632775054313-45128f4f", "operationStartTime":"2021-09-27T20:37:34.313742491Z", "currentVersion":"1.20.9-gke.1001", "targetVersion":"1.20.10-gke.301"}
    project_id: 729788050015
    type_url: type.googleapis.com/google.container.v1beta1.UpgradeEvent

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

  1. Delete the Cloud Run function you deployed in this tutorial:

    gcloud functions delete slackNotifier

    You can also delete Cloud Run functions from the Google Cloud console.

  2. Delete the Pub/Sub topic.

  3. Delete the Slack app.

What's next