Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

AWS Log Collection into Splunk

Integrated with AWS Control
Tower
November 2020

1
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Notices
Customers are responsible for making their own independent assessment of the
information in this document. This document: (a) is for informational purposes only, (b)
represents current AWS product offerings and practices, which are subject to change
without notice, and (c) does not create any commitments or assurances from AWS and
its affiliates, suppliers or licensors. AWS products or services are provided “as is” without
warranties, representations, or conditions of any kind, whether express or implied. The
responsibilities and liabilities of AWS to its customers are controlled by AWS agreements,
and this document is not part of, nor does it modify, any agreement between AWS and
its customers.

© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved.

2
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Contents
Overview ..............................................................................................................................4
Considerations .....................................................................................................................6
Scenario ...............................................................................................................................7
Prerequisites ........................................................................................................................7
Shared VPCs for Forwarder Lambda Functions .............................................................7
Code Buckets for Forwarder Lambda Functions.............................................................8
IAM User for the Splunk User ..........................................................................................9
Runbook – Collection of logs from AWS into Splunk .......................................................10
Use Case ........................................................................................................................10
Architecture ........................................................................................................................10
How Each Log Input is Collected and Sent to Splunk ..................................................12
AWS Best Practices Recommended in the Architecture ..............................................14
Mechanism Implementation Guide ...................................................................................15
Automating Installation and Configuration Using CloudFormation StackSets .............20
Updating the Mechanism ...............................................................................................21
Splunk Installation and Configuration Steps .................................................................22
Removing the Mechanism .............................................................................................22
Future Possible Enhancements ........................................................................................24
Conclusion .........................................................................................................................24
Contributors .......................................................................................................................24
Document Revisions..........................................................................................................25

3
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

About this Guide

This guide shows how Amazon Web Services (AWS) logs can be collected into a
central account in AWS and sent to Splunk. Many enterprise customers use Splunk for
on-premises security and logging, and want to use the same tool for searching and
analyzing logs when the workloads migrate to AWS. This guide provides a fully
automated architecture which shows how AWS logs can be collected into a central
account in AWS and sent to a Security Information and Event Management (SIEM) tool;
in this case, Splunk.
A SIEM tool offers full visibility into activity within your network which empowers you to
respond to threats in real time. Many enterprise customers use Splunk for on-premises
security and logging, and want to use the same tool for searching and analyzing logs
when the workloads migrate to AWS.
The architecture is scalable, highly available, and has a fault-tolerant design.
This document is intended for multiple audiences:

• Customer cloud infrastructure teams who set up and implement changes to

the AWS CloudFormation StackSets, AWS Lambda functions, and the identity
access management (IAM) roles and policies created. In the future, these teams
can work on enhancing the capabilities of this mechanism to cover additional log
inputs.
• Customer security teams who need to validate the log collection in Splunk.
These teams provide data collection requirements, and may be capable of
creating resources necessary for testing.
• Customer DevOps and monitoring teams who will be the primary users of
Splunk visualizations for operations.

Overview
Splunk is a technology used for application management, security, and compliance, as
well as business and web analytics. Splunk works well to search for specific data in a
large volume of complex data.
The Splunk add-on for AWS enables you to collect:

• Configuration snapshots, configuration changes, and historical configuration data

from AWS Config.
• Metadata for your Amazon EC2 Autoscaling instances, reserved instances, and
Amazon Elastic Block Store (Amazon EBS) snapshots.

4
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

• Compliance details, compliance summary, and evaluation status of your AWS

Config rules.
• Assessment runs and findings data from Amazon Inspector.
• Management and change events from AWS CloudTrail.
• Amazon Virtual Private Cloud (Amazon VPC) Flow logs and other logs from
Amazon CloudWatch Logs.
• Performance and billing metrics from Amazon CloudWatch.
• Billing reports that you have configured in AWS.
• Amazon Simple Storage Service (Amazon S3), Amazon CloudFront, and Elastic
Load Balancing (ELB) access logs.
• Generic data from your Amazon S3 buckets.
• Generic data from your Amazon Kinesis streams.
• Generic data from Amazon Simple Queue Service (Amazon SQS).
This add-on provides modular inputs and Splunk Common Information Model (CIM)-
compatible knowledge to use with other Splunk apps, like the Splunk App for AWS,
Splunk Enterprise Security, and Splunk IT Service Intelligence.
This guide shows a central view of logs for the security teams in a multi-account AWS
environment. The log archive account in an AWS Control Tower landing zone is the
central log collector, and Amazon SQS queues are polled by Splunk to ingest logs. All
member accounts are configured to replicate their logs to the central account for
supported services.
The runbook included in this guide provides the mechanism and the scripts to
implement the central log collection. It also provides guidance on access and
permissions for cross-account replication of log data from member accounts into the
central Log-Archive account. The mechanism deploys the necessary permissions, AWS
Identity and Access Management (IAM) policies, Amazon S3 bucket policies, and the
required IAM roles.

The runbook provides steps for updating the AWS CloudFormation StackSet to deploy
the mechanism, or modify the configuration as necessary. It also provides removal
steps for the architecture. Where possible, the mechanism builds an automated
environment.
The runbook also provides guidance on possible enhancements for the mechanism, to
cover more log inputs at the workload level.

5
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

This runbook is meant to be a living document that evolves. As your organization

updates the mechanism to collect more log inputs, or updates the process to be more
cloud capable, it makes sense to update the runbook.

The rest of this document covers each of the prioritized use cases, and documents the
current process, architecture, and tooling used.

Considerations
Familiarity with the following AWS Services is highly recommended:

• Amazon CloudWatch (logs and metrics)

• Amazon GuardDuty
• Amazon S3
• Amazon VPC (flow logs)
• AWS CloudTrail
• AWS Config
• AWS Control Tower
• AWS Lambda
• AWS Managed Microsoft AD
• AWS Transit Gateway
• ELB
• IAM (roles and policies)
AWS CloudFormation gives developers and systems administrators a mechanism to
define, create, and manage a collection of related AWS resources as code, provisioning
and updating them in an orderly and predictable fashion. This document provides the
CloudFormation templates created for consistency with standard cloud automation
process and industry best practices.
AWS Lambda is a serverless compute service that runs your code in response to
events, and automatically manages the underlying compute resources for you. Lambda
runs your code on a high-availability compute infrastructure and performs all the
administration of the compute resources, including server and operating system
maintenance, capacity provisioning and automatic scaling, code and security patch
deployment, and code monitoring and logging. This document provides Lambda
functions created using Python and NodeJS.

6
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Scenario
An organization wants to deploy the mechanism in the Northern Hemisphere landing
zone, which is the customer’s region of operations in North America (NA) and Europe,
the Middle East and Africa (EMEA).

AWS regions Ohio (us-east-2), Virginia (us-east-1), Frankfurt (eu-central-1), and Ireland
(eu-west-1) have been selected. Ohio is the NA primary region with Virginia as backup,
and in EMEA Frankfurt is the primary with Ireland as backup.
This multi-account environment has:

• An AWS Managed Microsoft AD setup in an Amazon VPC in the AWS Control

Tower primary account.
• An Amazon GuardDuty administrator account.
• A shared IT support account.

Prerequisites
The following prerequisites are required to set up this mechanism.

Shared VPCs for Forwarder Lambda Functions

This step is required only if Splunk is on-premises.
To forward CloudWatch logs to an on-premises Splunk HTTP Event Collector (HEC),
Lambda functions must reside in an Amazon VPC which has a connection back to on-
premises. A shared VPC is in the IT support account in all four NA regions, and the
subnets are shared with the organization using AWS Resource Access Manager. This
allows the forwarder Lambda functions in the member accounts to use the shared
Amazon VPC to route traffic back to on-premises Splunk.
To create the shared Amazon VPC:

You will create a VPC with a Classless Inter-Domain Routing (CIDR) range as
x.x.8.0/22.
1. Create the shared subnets for the CIDRs using the third octets 8,9, and 10 as
/24 blocks.
2. Share the subnets with the organization using AWS Resource Access Manager
(AWS RAM).
3. Create the AWS Transit Gateway attachment for the Amazon VPC which allows
for connectivity on-premises.

7
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

4. Update the Amazon VPC route table to allow a route back to the organization’s
on-premises addresses.
5. Select range /22 for the Amazon VPC, anticipating the addition of accounts in
the AWS Control Tower environment, with added forwarder Lambda functions in
each account that reside in this shared Amazon VPC for each region.
Forwarder Lambda functions no longer require this shared Amazon VPC if the
organization is using Splunk cloud on AWS, and has the required network connectivity
for Lambda functions to talk to Splunk. They use the AWS backbone to communicate
with Splunk instances as they communicate with other AWS services.

Code Buckets for Forwarder Lambda Functions

Simple Lambda functions written in Node.js or Python can be declared inline in an AWS
CloudFormation template, so no zip files are necessary. The forwarder Lambda function
is written in Node.js, but requires a set of node modules and needs to be zipped and
stored in an S3 bucket. For this purpose, S3 code buckets are created in the IT support
account in all four NA regions. They have the following bucket policy so they can be
shared with the organization and be accessed by the forwarder Lambda function
deployment.
The bucket policy shows the permissions granted to the S3 code bucket in Frankfurt:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGetObject",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3::: s3-bucket-name-splunk-cust-
frankfurt/*",
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "org-id "
}
}
}
]
}

8
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

All the S3 code buckets are encrypted using the regional AWS Key Management
Service (AWS KMS) keys created in the same shared IT support account.

IAM User for the Splunk User

Create a Splunk IAM user to provide an access key and secret key pair for the Splunk
service to access AWS. This user should have permissions only to assume the Splunk
role in the various accounts, and that Splunk role should trust only assumption by that
Splunk IAM user. The Splunk role must have read access only, but part of that read
access includes S3 and log data.
Handling the access keys is part of cloud computing security, which is the customer’s
responsibility. See Best practices for managing AWS access keys, and a blog on how to
rotate access keys for IAM users.
The user is created in the Log-Archive account. Figure 1 displays the user’s
permissions.

Figure 1 – Permissions granted to the Splunk service user

The cust-AWS-Splunk-role has the permissions to be assumed by the Splunk user

to read the log data from each of the AWS accounts, so all member accounts have this
IAM role.
The cust-AWS-Splunk-role is created using the template cust-AWS-splunk-
role.yaml.

9
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Runbook – Collection of logs from AWS into

Splunk
Use Case
An organization has a requirement to export AWS log data to Splunk and store the logs
in AWS for 90 days. (The retention policy is configurable.)
The organization’s security team has a mandate to ingest these logs to Splunk for
further processing and analysis, and enforce logging standards when handling, storing,
and disposing of sensitive information. These standards are applicable to any person
accessing organization information, whether they are an employee, contractor, or third
party.
The runbook covers creating an IAM user for Splunk to pull the logs coming from all the
individual AWS accounts consolidated in a Log-Archive account of the AWS Control
Tower environment, It also includes creating IAM policies, bucket policies, and IAM
roles to capture the logs from individual accounts.

Architecture
As of the date of this publication, Splunk collects log inputs from the following 9 AWS
sources:

• Amazon VPC Flow logs

• AWS Config
• AWS Managed Microsoft Active Directory AD logs
• CloudTrail
• CloudWatch logs
• CloudWatch metrics
• ELB Access logs
• GuardDuty findings
• S3 Access Logs
The mechanism needs to be implemented in the Northern Hemisphere, which covers
the AWS Control Tower, including AWS regions for the organizations NA and EMEA
AWS regions (us-east-1, us-east-2, eu-central-1 and eu-west-1).

10
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Figure 2 illustrates the various mechanisms configured for Splunk to collect log data
from the AWS accounts.

Figure 2 —AWS log collection with AWS Control Tower into Splunk

The mechanisms are centrally deployed into each account, either by AWS
CloudFormation or AWS CloudFormation StackSets. The Log-Archive account is the
central collector. It aggregates logs from all the regions and member accounts. The logs
are stored in S3 buckets in the us-east-2 Region, which send notifications to SQS
queues which are polled by Splunk every five minutes for log data.
Each member account has local S3 buckets for collecting log data in each of the four
regions. These local buckets replicate log data to the central S3 buckets in the Log-
Archive account.

Log inputs such as CloudTrail, AWS Config, S3, ELB, Amazon VPC Flow Logs, and
GuardDuty are centrally collected into the Log-Archive in the us-east-2 Region.
CloudWatch metrics, CloudWatch logs, and AD logs are individually forwarded from
each AWS account in all the AWS Regions directly to a HEC.

11
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

How Each Log Input is Collected and Sent to Splunk

AWS CloudTrail
AWS Control Tower collects CloudTrail logs from all accounts and all AWSRegions, and
stores the log data in a central S3 bucket in Log-Archive.

Splunk pulls these logs using an SQS-based S3 input mechanism. New object
notifications are sent to SQS. Splunk polls the queue to discover new log items, then
ingests them from the S3 bucket.

AWS Config
AWS Control Tower collects AWS Config logs from all accounts in all four AWS regions,
and puts the log data in the central S3 bucket in the Log-Archive account. This is the
same bucket where CloudTrail logs are stored.
Splunk pulls these logs by using the same SQS-based S3 input mechanism.

CloudWatch Metrics
AWS accounts have Cloudwatch Metrics, which can be directly pulled by Splunk from
each individual account in all the AWS regions.

CloudWatch Logs
CloudWatch logs are subscribed by a subscriber Lambda function and sent to a
Splunk HEC by a forwarder Lambda function in each individual account, in all the
AWS Regions.

Because Lambda functions forward these logs to Splunk, no configuration is required on

the Splunk side. CloudWatch logs are sent to the Splunk index idx_aw for the Northern
Hemisphere.

S3 Access Logs
S3 access logs are collected locally in each region in each member account and
replicated to the central S3 log bucket in the Log-Archive account in the us-east-2
Region.
There is a Lambda function to enable S3 logging on newly created S3 buckets. The
target bucket is the local S3 logging bucket in the same Region.

12
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

ELB Logs
ELB logs are collected locally in each Region in each member account, and replicated
to the central S3 log bucket in the Log-Archive account in the us-east-2 Region.

There is a Lambda function to enable ELB logging on newly created ELBs. The
destination bucket is the local S3 logging bucket in the same Region.

Amazon VPC Flow Logs

Amazon VPC Flow logs can be sent directly to the central S3 log bucket in the Log-
Archive account in the us-east-2 Region.

Amazon VPC Flow logs generate a lot of log data, so we don’t recommend that you
generate them from each account and each region. GuardDuty analyzes Amazon VPC
Flow logs to create findings. There is a template to deploy in the necessary production
accounts, or for critical workloads to create Amazon VPC Flow logs.
As of the date of this publication, Amazon VPC Flow logs are created in the primary
account for AWS Managed Microsoft AD VPC and sent to the central Log-Archive in
us-east-2 from all the Regions.

Amazon GuardDuty Findings

GuardDuty supports exporting active findings to CloudWatch Events and, optionally, to
an S3 bucket. The new active findings that GuardDuty generates are automatically
exported within about five minutes after the finding is generated. You can set the
frequency for how often updated active findings are exported to CloudWatch Events and
your S3 bucket. The frequency that you select applies to exporting to both CloudWatch
Events and your S3 bucket, but only for updated findings.
As of the date of this publication, GuardDuty is enabled in us-east-1 and us-east-2. The
findings are exported every six hours to a central S3 bucket in the Log-Archive
account in the us-east-2 Region. The GuardDuty findings are exported from the
GuardDuty primary account.

AWS Managed Microsoft AD Logs

AWS Managed Microsoft AD has critical information, so CloudWatch logs are enabled
on the Managed Active Directory Service in the primary account in the four AWS
Regions. The CloudWatch logs are subscribed and forwarded to Splunk HEC.

13
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

AWS Best Practices Recommended in the

Architecture
Protection from Inside Threats

• Least privilege granted on IAM roles on resources (such as Lambda functions).

• Lambda VPCs can communicate only to on-premises network, limited to the
Splunk HEC as required.
Limited Access with One IAM User

• Although Splunk ingests AWS logs from all the accounts in the AWS Control
Tower, only one IAM user is created for the Splunk service, with a policy to
assume the AWS Splunk role in other accounts.
IAM Roles for Cross Account Access

• IAM roles enable you to delegate access with defined permissions to trusted
entities without having to share long-term access keys. You can use IAM roles to
delegate access to:
o IAM users managed within your account
o IAM users under a different AWS account
o An AWS service like Amazon EC2
• The AWS Splunk role in each account only has a trust-relationship with the
central Splunk user.
Data Protection

• Encryption enabled on logs in local S3 buckets using KMS keys

• Encryption enabled on logs replicated to central S3 buckets
• Bucket policies restricting access to least privilege
Design Pattern

• Scalable, highly available, and fault tolerant design

• Easy to deploy, update, and re-deploy infrastructure

14
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Mechanism Implementation Guide

The mechanism is implemented using AWS CloudFormation and AWS Lambda
functions. See best practices for using CloudFormation and Lambda functions. The
following AWS services are used:

• AWS Config
• CloudTrail
• CloudWatch
• Directory Service
• ELB
• GuardDuty
• IAM (user, roles and policies)
• KMS
• S3
• SQS
For networking, Amazon VPC, AWS Transit Gateway and VPN/DirectConnect are also
used.
Excepting IAM and S3, the previously mentioned services are regional, so the
necessary controls, processes, event rules, and infrastructure must be set up in every
region where AWS Services will leverage CloudWatch logging and monitoring.
One important aspect of the mechanism is that the logs from all the other regions and
individual AWS member accounts are collected in the central Log-Archive account in
the us-east-2 region. The exceptions are CloudWatch logs and metrics. CloudWatch
metrics are pulled directly from accounts, and CloudWatch logs are forwarded to the
Splunk HEC from the source accounts.

Code Repository
The code can be downloaded from this secure S3 bucket. The ZIP file contains a folder
called aws-splunk-whitepaper with the following structure:

src
• ELBLogger
o index.py

15
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

• Forwarder
o SplunkForwarderLambdaV44.zip
o SplunkForwarderLambdaV45.zip
• S3Logger
o index.py
• Subscriber
o index.py

architecture-diagrams
• AWS Splunk Architecture Diagram.png

templates
• aws-account-baseline
o aws-splunk-global-resources-cust-logging.yaml
o aws-splunk-local-resources-cust-logging-emea.yaml
o aws-splunk-local-resources-cust-logging-na.yaml
• log-archive
o cust-splunk-role.yaml
o kms-splunk-key.yaml
o splunk-queues-and-buckets.yaml
o splunk-vpcflow-guardduty-buckets-queues.yaml
o bucket-policy-gd-vpcflowlogs
vpc-flow-log-bucket-policy.json
guard-duty-bucket-policy.json
• vpcflowlogs
o flowlogs.yaml
• cwlogs
o cw-logs-ad-splunk_emea.yaml
o cw-logs-ad-splunk_na.yaml

16
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

o cw-logs-global-roles.yaml
The folder also contains a readme.MD file.

The code repository has the Lambda function code used inline in the CloudFormation
templates. For forwarder Lambda functions, the zip file is stored in the S3 code
bucket.
In the CloudFormation templates, look for cust. Replace cust with the customer name.
Replace S3bucket names with your bucket name Amazon Resource Names (ARNs) as
necessary. Update the KMS ARNs in the bucket policies as well. Use the Amazon VPC
and subnet IDs that you create to wrap the forwarder Lambda functions in a VPC.

Deployment
The mechanism can be deployed to customer Organizational Units (OUs) so it will
collect logs from all the child accounts.
The 6 inputs (Config, CloudTrail, S3, ELB, CloudWatch logs, CloudWatch Metrics) can
be deployed using CloudFormation StackSets deployed in the AWS Control Tower
primary account in the Region us-east-2.
The other three inputs (Microsoft AD, GuardDuty and Amazon VPC Flow logs) are
deployed in specific accounts using CloudFormation stacks:

• Microsoft AD logs are sent via CloudWatch logs to Splunk from the primary
account.
• Amazon VPC Flow logs are set up with a central bucket and queue in the Log-
Archive in the us-east-2 Region, but only generated for the Microsoft AD VPC in
the primary account.
• GuardDuty findings are collected only from the GuardDuty administrator account
and sent to a central bucket and queue in the Log-Archive account in the us-
east-2 Region.
The correct order of deployment is to create resources in the Log-Archive account
first, then deploy the global resources and the local resources in the member accounts.
For deploying the specific CloudFormation stacks in the core and member accounts,
you must use the Switch Role functionality to login with the
AWSControlTowerExecution role. The AWSControlTowerExecution role allows
AWS Control Tower to manage your individual accounts, and report information about
them to your audit and logging accounts.
AWSControlTowerExecution allows auditing by the AWS Control Tower audit
account, and helps you configure your organization’s logging so that all the logs for
every account are sent to the logging account.

17
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

To deploy this mechanism:

1. In the Log-Archive account, deploy the following Amazon CloudFormation
templates to create the central S3 buckets and SQS queues which are pulled by
Splunk for S3, ELB and CloudTrail (inclusive of AWS Config) logs using
CloudFormation stacks in the us-east-2 Region:
• cust-splunk-role.yaml
• kms-splunk-key.yaml
• splunk-queues-and-buckets.yaml
• splunk-vpcflow-guardduty-buckets-queues.yaml
For CloudTrail and AWS Config, the S3 bucket in the central account is created
by AWS Control Tower, so the SQS queue that sends S3 event notifications
must be created with the name CloudTrailSplunkSQS. More details on any
queue and its policy to provide permissions to the S3 log bucket to send
messages can be found in the template splunk-queues-and-buckets.yaml,
as all the queues for Splunk have similar policies and configurations.
After the GuardDuty bucket is created with the default policy defined in the
CloudFormation template, the bucket policy must be updated with the JSON file
in the following folder in the code-repository:
• templates
• log-archive
• bucket-policy-gd-vpcflowlogs
• guard-duty-bucket-policy.json
After the VPC Flow logs bucket is created with the default policy defined in the
CloudFormation template, update the bucket policy with the .json file uploaded in
the following folder in the code-repository:

• templates
• log-archive
• bucket-policy-gd-vpcflowlogs
• vpc-flow-log-bucket-policy.json
2. In all the member accounts, the following CloudFormation template needs to be
deployed for creation of global resources:
• aws-splunk-global-resources-cust-logging.yaml

18
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

To automate the deployment in all target accounts at once, use Amazon

CloudFormation service-managed StackSets. AWS CloudFormation Stacksets
enable you to deploy CloudFormation stacks to an OU and the child AWS
accounts within, in multiple regions, with just a few clicks. Future accounts
inherit the deployments and install the assigned stacks.
3. In all the member accounts, the following CloudFormation template needs to be
deployed for creation of local resources in the NA Regions us-east-1 and us-
east-2:
• aws-splunk-local-resources-cust-logging-na.yaml
4. In all the member accounts, the following CloudFormation template needs to be
deployed for creation of local resources in the EMEA Regions eu-central-1 and
eu-west-1:
• aws-splunk-local-resources-cust-logging-emea.yaml

Steps 3 and 4 can be combined to deploy local resources in all the AWS
NA regions and AWS EMEA regions at once. They are deployed
separately because, if there is a future need to provide different HEC
inputs or other parameters that need to vary for regions, they can simply
be updated by the StackSet deployed in those regions.

Steps 1-4 work to set up the central buckets and queues for all the log
inputs, and deploy local resources for the six log inputs. For the rest of the
inputs like GuardDuty, VPC Flow logs, and Managed AD logs, follow the
next steps.

5. To deploy Managed Microsoft AD logs, enable log forwarding in the AWS

Directory Service in the primary account in all four AWS regions. Once the
CloudWatch log groups are created, deploy the following stack in any one region
to create the roles required to forward CloudWatch logs to the Splunk HEC:
• cw-logs-global-roles.yaml
6. After the roles are successfully created, deploy the following CloudFormation
StackSet to create subscriber Lambda functions and forwarder Lambda
functions in the matching AWS Regions to forward CW logs:
• cw-logs-ad-splunk_emea.yaml
• cw-logs-ad-splunk_na.yaml

19
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

The reason you see two different stacks for the EMEA and NA regions is
because the S3 code bucket in the IT support account for forwarder
Lambda functions now has different code versions. It can provide different
input parameters, or mapping specific to the region for future use.

To enable VPC Flow logs in an account, deploy the following template in that
account and provide the Amazon VPC ID in the input parameters. Amazon VPC
flow logs are currently enabled in the primary account for the Amazon VPC in
which Managed Microsoft AD exists. Amazon VPC flow logs are enabled in all
regions and are sent to the central log-bucket in us-east-2 Region:

• flowlogs.yaml
7. For exporting GuardDuty findings, the only step required after creating the
GuardDuty buckets and queues in the Log-Archive account is to set up the
export configuration in all the required AWS Regions in the GuardDuty primary
account.

Automating Installation and Configuration Using

CloudFormation StackSets
Infrastructure as Code
The mechanism was implemented to provision infrastructure resources via AWS
CloudFormation, so you can manage the mechanism in the entire landing zone.

Updating Configuration
Any time there’s a need to change the configuration or update input parameters for a
particular region or account, it is recommended that you make changes using the
Update CloudFormation Stack or StackSet to avoid drift.

Retention Policies for Logs

• CloudWatch Logs forwarded to Splunk are configured for retention of 365 days.
• Log buckets in Log-Archive have a lifecycle policy of 90 days, except the S3
logging bucket, which has no retention policy.
• AWS Lambda functions are triggered every hour.
• Retention and lifecycle policies are configurable.

20
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Updating the Mechanism

When a New Account is Added
AWS CloudFormation StackSets are deployed in the primary account in the Ohio
region. They are enabled to deploy the mechanism every time a new account is added.
To update the settings:
1. Select the StackSet.
2. On the Actions menu, choose Edit automatic deployment.

Figure 3 — The Edit automatic deployment screen

3. Under Automatic deployment, choose Enabled.

4. Under Account removal behavior, choose Delete stacks.
5. Click Save.

When a New Region is Added

To add new regions:
1. Select the CloudFormation StackSet.
2. From the Actions menu, choose Add Stacks to Stackset.

21
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Splunk Installation and Configuration Steps

Installation
Use the Splunk Add-on for Amazon Web Services to collect CloudTrail logs,
performance, billing, and IT and security data on AWS products.
Install the Splunk Add-On for AWS following the directions at the linked site.

Configuring Permissions
Manage accounts and configure inputs for the Splunk Add-on for AWS.

Networking Endpoints
Ensure that the on-premises Splunk server is permitted through corporate firewalls to
access the AWS service endpoints, so it can poll for data. See more information about
the AWS service endpoints.

Configuring Inputs
See best practices and details on how to configure inputs on the Splunk Add-on for
AWS page.

Removing the Mechanism

Logs for Six Baselines Inputs
For the six baseline inputs including CloudTrail, AWS Config, S3, ELB, CloudWatch
Metrics, and CloudWatch logs, the mechanism is deployed using AWS CloudFormation
StackSets, so you will need to delete them.
See Delete stack instances from your stack set.

The templates in the aws-account-baseline folder in the repository are

deployed in member accounts in the customer OUs in the NA and EMEA regions
for these six baselines inputs.

• If you need to remove the mechanism from all accounts in all regions, after you
have deleted the stacks, delete the StackSets from the primary account.
• Make sure to empty S3 buckets before you delete them.

22
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

• Deleting the CloudFormation stacks or Stacksets won’t remove the log

subscriptions from the CloudWatch logs. Subscriptions must be removed
manually in each account in the regions where stacks are deployed.
Alternatively, you can write a Python script to remove the subscriptions.

GuardDuty Findings in GuardDuty Primary Account

The easiest way to stop receiving GuardDuty findings that are exported to Splunk and
collected in a central Log-Archive bucket is to disable the export in GuardDuty via
AWS Management Console in the GuardDuty primary account.

VPC Flow Logs in the Primary Account

To stop the VPC Flow logs, delete the CloudFormation stack flowlogs.yaml in the
Primary account created for Managed Microsoft AD VPC.

Managed Micosoft AD CloudWatch Logs in the Primary Account

To clean the CloudWatch logs and stop Microsoft AD from generating CloudWatch logs,
delete the CloudFormation stacks in the primary account in all the AWS regions:

• cw-logs-global-roles.yaml [in Ohio Region]

• cw-logs-ad-splunk_emea.yaml [in EMEA Regions]
• cw-logs-ad-splunk_na.yaml [in NA Regions]
Also remove the log subscriptions to forwarder Lambda functions, because
CloudFormation stack deletion does not delete resources created outside of
CloudFormation.

Cleaning the Central Resources in the Log-Archive Account

If you need to clean up logging for any or all inputs, you will need to delete the
resources created in the Log-Archive account as well.

• Delete the Log-Archive account resources after you delete local resources in
member accounts.
• Delete the stacks deployed in the Log-Archive account.
Steps to be performed manually:
1. Delete S3 code buckets in IT support account created for forwarder Lambda
functions.

23
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

2. After you remove the CloudWatch logs for subscriber Lambda functions, you can
delete the shared VPCs and the Resource Access Manager shares in the IT
support account.
3. Once everything else has been deleted, you can delete the IAM user for Splunk
in the Log-Archive account.
Remember to manually delete the resources created in the shared IT Support account.

Future Possible Enhancements

At present in the Northern Hemisphere, Splunk receives log data from nine log inputs. In
the future, it can be enhanced for more workload specific inputs such as:

• CloudFront logs
• AWS Web Application Firewall (AWS WAF) logs
• Amazon EC2 Auto Scaling logs
• Amazon RDS logs
Typically, these log inputs are workload specific and can be deployed in the necessary
production accounts using CloudFormation stacks, or in production OUs using
CloudFormation Stacksets.

Conclusion
We have provided an architecture to collect AWS logs from multiple AWS resources into
a central account in AWS and sent to the SIEM tool Splunk. The solution provided is a
fully automated mechanism leveraging an Infrastructure as Code (IaC) mechanism.
Leveraging the AWS CloudFormation Stacksets for provisioning and AWS Lambda for
its serverless benefits, the implementation is easy to deploy, configurable, and can be
updated as necessary. It can be extended for more log inputs, and will be added to the
new AWS Accounts and AWS Regions.

Contributors
Contributors to this document include:

• Deepa Pahuja, Cloud Architect

• Jason Cornick, Principal Cloud Architect

24
Amazon Web Services AWS Log Collection into Splunk Integrated with AWS Control Tower

Document Revisions
Date Description
November 2020 First publication

25