06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards

Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium
< Go to the original

Protect your Infrastructure with Real-time

Notifications of AWS Console User
Changes
Step by step tutorial combining CloudTrail, Lambda, and SNS to get
alerts only when a user manually makes a change in the AWS
Console.
Matthew Harper
Follow

Towards Data Science androidstudio ~9 min read ·

February 20, 2020 (Updated: December 13, 2021) · Free: No

You've embraced infrastructure as code (IoC) and painstakingly

created Terraform or CloudFormation to model your EC2 Instances,
AutoScaling Groups, ECS clusters, and everything else. The devops
and development teams have committed to only modify and create

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 1/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

infrastructure using your chosen IoC tools — but, surprise, some

Freedium people just won't comply. Maybe someone gets lazy during feature
development or takes a shortcut during incident response. Maybe
you've outsourced your level one support and you don't entirely trust
that group yet. Whatever the reason, you would love to know when a
user logs in to the AWS Console web site and manually changes
things.

There are credible reasons for concern around this topic.

Environmental drift is the term often used to describe the difference
between the desired state (represented in your infrastructure as
code) and the actual, live state of your infrastructure. This drift can
eventually lead to failed deployments, scaling issues that impact
customers, security gaps, or worst of all, a data breach.

In this article, we're going to build a system to monitor your AWS

infrastructure and send notifications whenever a user makes a
change directly from the AWS Console. We'll leverage CloudTrail to
record the events and write them to S3. An S3 trigger will invoke a
Lamba function. That function will filter the events and post the
relevant ones to an SNS topic that your team can subscribe to. If that
sounds intimidating, don't worry; AWS does the heavy lifting and we
just need to connect the pieces. Each step is detailed below — let's
get started.

Enable CloudTrail
CloudTrail is the key AWS service in this project. You might be able
to build swap out the other components with functional equivalents,
but CloudTrail monitors your entire AWS account for changes and
publishes that data to log files in S3.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 2/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

CloudTrail is often used for security investigations, compliance, and

Freedium auditing purposes. and you can use its web interface to search the
events it logs. Need to know who launched that expensive instance
that is sitting idle? Check the CloudTrail logs. Track down who
modified a launch configuration to find out why? CloudTrail. Basic
search functionality for CloudTrail is available right on its AWS web
interface, and you can also use AWS Athena to execute SQL style
queries against the event logs.

By default, CloudTrail keeps an event log of the past 90 days of

activity in your account. But those logs are not accessible in S3; for
that, we need to create a new trail. AWS allows one trail of
management events to be created for free.

Navigate to the CloudTrail console and choose 'Create Trail.' Provide

a name, and select 'Write-only' for the event filter, because that's all
we're interested in.

Scroll down to the Storage Location section, and select or create an

S3 bucket to store your logs.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 3/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium

Now, to test this, go ahead and make some change in your AWS
environment — launch a t.2 micro instance, create an empty
Lambda function, anything will work. Then take a look at the S3
bucket that should contain our CloudTrail logs:

If you download a log file from this bucket, you'll see an array
("Records") of events. Each event has a ton of metadata attached to it
— here's a sample event (shortened) that is generated when you
launch a new EC2 instance:

Create a Lambda to Process CloudTrail Logs

Now that each modification to our AWS infrastructure logged to S3,
we need to write a function to parse the data. We'll do this in
Lambda because it's serverless and that fits our need to invoke the
function only when new events are recorded.
https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 4/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Navigate to the Lambda console and create a new function. Choose

Freedium empty function and Python 3.8 as the runtime environment.

The function needs permission to read from S3 and to publish

notifications to SNS. Select "Create a new role from AWS policy
templates," and choose both "S3 object read-only" and "SNS publish"
from the Policy template drop-down. Provide a name for your role
and then select Create Function.

After your function is created, scroll down to find the inline code
editor and a very basic Python function.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 5/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium

Go ahead and copy all of the code below and paste it in the editor,
replacing all the existing code:

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 6/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

1
2
import json
import urllib.parse

Freedium 3
4
import boto3
import io
5 import gzip
6 import re
7
8 s3 = boto3.client('s3')
9 sns = boto3.client('sns')
10 sns_arn = "arn:replace_me"
11
12 USER_AGENTS = {"console.amazonaws.com", "Coral/Jakarta", "Coral/Netty4"}
13 IGNORED_EVENTS = {"DownloadDBLogFilePortion", "TestScheduleExpression", "Test
14 "listDnssec", "Decrypt", "REST.GET.OBJECT_LOCK_CONFIGURATIO
15
16
17 def post_to_sns(user, event) -> None:
18 message = f'Manual AWS Changed Detected: {user} --> {event}'
19 sns_publish(message)
20
21
22 def post_to_sns_details(message) -> None:
23 message = {"Manual AWS Change Detected": message}
24 sns_publish(message)
25
26
27 def sns_publish(message) -> None:
28 sns.publish(
29 TargetArn=sns_arn,
30 Message=json.dumps({'default': json.dumps(message)}),
31 MessageStructure='json'
32 )
33
34
35 def check_regex(expr, txt) -> bool:
36 match = re.search(expr, txt)
37 return match is not None
38
39
40 def match_user_agent(txt) -> bool:
41 if txt in USER_AGENTS:
42 return True
43
44 expressions = (
45 "signin.amazonaws.com(.*)",
46 "^S3Console",
47 "^\[S3Console",
48 "^Mozilla/",
https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 7/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…
48 Mozilla/ ,
Advertise here and support our project! Reach out to us at [email protected]
49 "^console(.*)amazonaws.com(.*)",

Freedium
50 "^aws-internal(.*)AWSLambdaConsole(.*)",
51 )
52
53 for expresion in expressions:
54 if check_regex(expresion, txt):
55 return True
56
57 return False
58
59
60 def match_readonly_event_name(txt) -> bool:
61 # starts with
62 expressions = (
63 "^Get",
64 "^Describe",
65 "^List",
66 "^Head",
67 )
68 for expression in expressions:
69 if check_regex(expression, txt):
70 return True
71
72 return False
73
74
75 def match_ignored_events(event_name) -> bool:
76 return event_name in IGNORED_EVENTS
77
78
79 def filter_user_events(event) -> bool:
80 is_match = match_user_agent(event['userAgent'])
81 is_read_only = match_readonly_event_name(event['eventName'])
82 is_ignored_event = match_ignored_events(event['eventName'])
83 is_in_event = 'invokedBy' in event['userIdentity'] and event['userIdentit
84
85 status = is_match and not is_read_only and not is_ignored_event and not i
86
87 return status
88
89
90 def get_user_email(principal_id) -> str:
91 words = principal_id.split(':')
92 if len(words) > 1:
93 return words[1]
94 return principal_id
95
96
https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 8/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

96
97 def lambda_handler(event, context) -> None:

Freedium 98
99
"""
This functions processes CloudTrail logs from S3, filters events from the
100 :param event: List of S3 Events
101 :param context: AWS Lambda Context Object
102 :return: None
103 """
104 for record in event['Records']:
105 # Get the object from the event and show its content type
106 bucket = record['s3']['bucket']['name']
107 key = urllib.parse.unquote_plus(record['s3']['object']['key'], encodi
108 try:
109 response = s3.get_object(Bucket=bucket, Key=key)
110 content = response['Body'].read()
111
112 with gzip.GzipFile(fileobj=io.BytesIO(content), mode='rb') as fh:
113 event_json = json.load(fh)
114 output_dict = [record for record in event_json['Records'] if
115 if len(output_dict) > 0:
116 post_to_sns_details(output_dict)
117 for item in output_dict:
118 post_to_sns(get_user_email(item['userIdentity']['principa
119
120 return response['ContentType']
121 except Exception as e:
122 print(e)
123 message = f"""
124 Error getting object {key} from bucket {bucket}.
125 Make sure they exist and your bucket is in the same region as
126 """
127 print(message)
128 raise e
129
130
131 def unit_test() -> None:
132 with open('sample.txt') as json_file:
133 event_json = json.load(json_file)
134 output_dict = [record for record in event_json['Records'] if filter_u
135 for item in output_dict:
136 user_email = get_user_email(item['userIdentity']['principalId'])
137 print(f"{user_email} -- {item['eventName']}")
138 post_to_sns(get_user_email(item['userIdentity']['principalId']),
139 post_to_sns_details(item)
140
141
142 #unit_test()

lambda py hosted with ❤ by GitHub view raw

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 9/17
 lambda.py
06/07/2024, 02:58
hosted with ❤ by GitHub
Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Let's review the code. Scroll down to find lambda_handler — that's
Freedium
the entry point to our application. Because the Lambda is invoked by
an S3 trigger, the parameter 'event' contains a list of S3 Events. Each
event contains the key for an object that was created in the S3 bucket
that our CloudTrail logs are written to. We loop through the list, and
for each event, we download the file specified from S3 and unzip it.

Within each file, we have a list of CloudTrail events in JSON format.

We don't want to be notified of every CloudTrail event; the point of
this exercise is to catch only events triggered by users via the AWS
Console. Amazon does not provide an easy way to differentiate
manual user changes from those invoked by the SDK or command
line. Arkadiy Tetelma wrote a fantastic blog here in which he works
to solve the same issue. We're going to filter the CloudTrail events in
that manner that Arkadiy describes inside our Lamda.

To execute that filter, we use a Python list comprehension to run

filter_user_events on each item in the log and create output_dict
containing only the matches.

The filter_user_events function checks relevant fields in the event

against the filters we specify. First, it looks at the userAgent and tries
to match a series of regular expressions that mimic the AWS Console
URLs. Then it screens read-only events, like those that begin with
'Get' or 'Describe'. The third check is against a set of specific event
names that we ignore (like AWS Console logins.) The final check is
whether the event was invoked by 'AWS Internal.'

After we have the subset of events we're interested in, we want to

send the actual notification. We'll use Amazon's Simple Notification
Service (SNS), which is a very straightforward service that allows you
to create a topic and then publish/subscribe to it. The code to

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 10/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

publish to the SNS topic is in our sns_publish function, and there are

Freedium two wrapper methods (post_to_sns publishes the userName and

eventName only, while post_to_sns_details publishes the entire JSON
event.)

Note the sns_arn variable referenced here and defined near the top
of the file; you'll need to replace that with the ARN of the SNS topic
that you want to publish to. Click save near the upper right of the
screen to update your Lambda function with the code you pasted in;
this change will take effect immediately. Then navigate to the SNS
section of the AWS console, select Topics, and Create Topic.

Name the topic appropriately, and use the default setting for all of
the other sections. Optionally, you can edit the Access Policy
settings; the default policy is to allow only the topic owner to publish
and subscribe to this topic. That's fine for this demo, but if you want
to use this in a larger organization, you'll want to tailor that policy to
your liking. Select 'Create Topic' to complete the setup process.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 11/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium

From this screen, you can copy the ARN of your topic and head back
over to Lambda to update the code (replacing the value of the
sns_arn variable.) You'll also note that the topic has zero
subscriptions; select 'Create Subscription', choose Email as the
protocol, enter your email address, and select 'Create Subscription'
again.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 12/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium

Make sure you check your email and confirm the subscription,
otherwise you will not receive any of the messages published to the
topic.

Trigger Lambda on S3 Events

So — a quick recap. CloudTrail logging is enabled, an SNS topic exists
and we subscribed to it, our Lambda function is live (and the
https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 13/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

sns_arn variable is updated.) All that's left to do is to create the

Freedium trigger to invoke our Lambda function whenever a new CloudTrail

log is created in the S3 bucket.

Navigate back to the Lambda console, click through to your function

details, and select the 'Add Trigger' option near the top of the
designer window.

Select the S3 bucket where your CloudTrail logs are written to. For
Event Type, choose 'all object create events' to invoke our function
whenever new objects are created within that bucket. Make sure
'Enable Trigger' is selected to activate this trigger immediately, and
then click 'Add.'

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 14/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Freedium

You should see the trigger on the designer view after it is created.

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 15/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise here and support our project! Reach out to us at [email protected]

Test It Out
Freedium
Now to test it out! A simple (and free) way to cause an event that will
trigger our alert is to create a new Lambda function. If everything
worked, shortly after the function is created, you'll receive an email
like this:

Since the code is set up to post both the short and detailed versions
of the message to SNS, you should receive another email with the
entire JSON message. If everything is working, you're all set! Feel
free to change to the code or the settings to meet your needs. If you
make any improvements to the script, please send a PR to the
GitHub repo. If something's not quite right, check out the
troubleshooting tips below.

Troubleshooting
Check the CloudTrail S3 bucket; are logs being created?

Check the Cloudwatch logs for the Lambda function. Logs should
exist each time you expect the Lambda to be triggered. Look for
write messages in the logs.

Check the S3 trigger for the Lambda function; is it monitoring

the correct bucket?

Check the IAM role assigned to the Lambda; does it have the
proper permissions to read from S3 and publish to SNS?

Does the sns_arn variable in the Lambda Python code match

your SNS topic?

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 16/17
06/07/2024, 02:58 Protect your Infrastructure with Real-time Notifications of AWS Console User Changes | by Matthew Harper | in Towards Data Science - Fr…

Advertise
Did you here and support
subscribe to our
theproject! Reachwith
SNS topic out toyour
us atemail
[email protected]
address, and
confirm the subscription?
Freedium
If all else fails, you can uncomment and run the unit test in the
Python code on your local machine. You can also add more
extensive print() logging to the code and review the Cloudwatch
logs to get a better idea of what's happening while your function
executes.

Resources
Detecting Manual AWS Console Actions — Arkadiy Tetelman

PyCloudTrailProcessor GitHub Repo

#aws #programming #cloudtrail #infrastructure-as-code #web-development

https://freedium.cfd/towardsdatascience.com/protect-your-infrastructure-with-real-time-notifications-of-aws-console-user-changes-3144fd18c680 17/17