Log Analytics with Amazon
Kinesis and Amazon Elasticsearch
Service
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to do with a terabyte of logs?
Log analytics architecture
data source Amazon Kinesis Firehose Amazon Elasticsearch Kibana
Service
Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open source Elasticsearch for log
analytics, full-text search and more.
Amazon
Elasticsearch
Service
Amazon Elasticsearch Service benefits
Easy to use Scalable Highly available
Open-source Secure AWS integrated
compatible
Adobe Developer Platform (Adobe I/O)
1
Amazon Amazon
Data
Kinesis Spark Streaming Elasticsearch
Sources
Streams Service
PROBLEM SOLUTION BENEFITS
• Cost effective monitor • Log data is routed with • Management and
for XL amount of log Amazon Kinesis to operational simplicity
data Amazon Elasticsearch
• Over 200,000 API calls Service, then • Flexibility to try out
per second at peak - displayed using AES different cluster config
destinations, response Kibana during dev and test
times, bandwidth
• Adobe team can easily
• Integrate seamlessly see traffic patterns and
with other components
of AWS eco-system. error rates, quickly
identifying anomalies
and potential
challenges
McGraw Hill Education
PROBLEM SOLUTION BENEFITS
• Supporting a wide catalog • Search and analyze test • Confidence to scale
across multiple services in results, student/teacher throughout the school year.
multiple jurisdictions interaction, teacher From 0 to 32TB in 9 months
• Over 100 million learning effectiveness, student • Focus on their business, not
events each month progress their infrastructure
• Tests, quizzes, learning
modules begun / completed • Analytics of applications
/ abandoned and infrastructure are now
integrated to understand
operations in real time
Get set up right
Amazon ES overview
CloudTrail CloudWatch
Elasticsearch API
Elastic Load
IAM
Balancing
Amazon Route
53
Data pattern
One index per day Shard 1
logs_01.21.2017 Shard 2
logs_01.22.2017 Shard 3
host
logs_01.23.2017 Each index has ident
logs_01.24.2017 multiple shards auth
logs_01.25.2017 Each shard contains timestamp
a set of documents etc.
logs_01.26.2017
Each document contains
logs_01.27.2017
a set of fields and values
Amazon ES cluster
Deployment of indices to a cluster
Amazon ES cluster
• Index 1 Primary Replica
– Shard 1 1 1 1 3 2 1
– Shard 2 2 2
3 1 1 2
– Shard 3 3 3
Instance 1, Instance 2
Master
• Index 2 3 2
– Shard 1 1 1
2 3
– Shard 2 2 2
Instance 3
– Shard 3 3 3
How many instances?
The index size will be about the same as the
corpus of source documents
• Double this if you are deploying an index replica
Size based on storage requirements
• Either local storage or up to 1.5TB of EBS per
instance
• Example: 2TB corpus will need 4 instances
– Assuming a replica and using EBS
– Or with i2.2xlarge nodes (1.6TB ephemeral storage)
Instance type recommendations
Instance Workload
T2 Entry point. Dev and test.
M3, M4 Equal read and write volumes.
R3, R4 Read-heavy or workloads with high memory demands (e.g.,
aggregations).
C4 High concurrency/indexing workloads
I2 Up to 1.6 TB of SSD instance storage.
Cluster with no dedicated masters
Amazon ES cluster
1 3 2 1
3 1 1 2
Instance 1, Instance 2
Master
3 2
2 3
Instance 3
Cluster with dedicated masters
Amazon ES cluster
1 3 2 1
3 1 1 2
Instance 1 Instance 2
3 2
2 3
Dedicated master nodes Instance 3
Data nodes: queries and updates
Master node selection
• < 10 nodes - m3.medium, c4.large
• 11-20 nodes - m4.large, r4.large, m3.large, r3.large
• 21-40 nodes - c4.xlarge, m4.xlarge, r4.xlarge, m3.xlarge
Cluster with zone awareness
Amazon ES cluster
1 3 2 3 1 2
3 1 2 2 3 1
Instance 1 Instance 2 Instance 3 Instance 4
Availability Zone 1 Availability Zone 2
Small use cases
Application
Instance
• Logstash co-located on the • Up to 200GB of data
Application instance • m3.medium + 100G EBS
• SigV4 signing via provided data nodes
output plugin • 3x m3.medium master nodes
Large use cases
Amazon
DynamoDB
Amazon S3
bucket AWS
Lambda
Amazon
CloudWatch
• Data flows from instances • Up to 5TB of data
and applications via • r3.2xlarge + 512GB EBS
Lambda; CWL is implicit data nodes
• SigV4 signing via • 3x m3.medium master nodes
Lambda/roles
XL use cases
Amazon
Kinesis
Amazon
EMR
• Ingest supported through • Up to 60 TB of data
high-volume technologies • R3.8xlarge + 640GB data
like Spark or Kinesis nodes
• 3x m3.xlarge master nodes
Best practices
Data nodes = Storage needed/Storage per node
Use GP2 EBS volumes
Use 3 dedicated master nodes for production deployments
Enable Zone Awareness
Set indices.fielddata.cache.size = 40
Amazon Kinesis
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver, process streams on AWS
Amazon Kinesis Amazon Kinesis Amazon Kinesis
Streams Firehose Analytics
Amazon Kinesis Streams
• Easy administration
• Build real time applications with framework of choice
• Low cost
Amazon Kinesis Firehose
• Zero administration
• Direct-to-data store integration
• Seamless elasticity
Amazon Kinesis Analytics
• Interact with streaming data in real-time using SQL
• Build fully managed and elastic stream processing
applications that process data for real-time visualizations
and alarms
Amazon Kinesis - Firehose vs. Streams
Amazon Kinesis Streams is for use cases that require custom
processing, per incoming record, with sub-1 second processing
latency, and a choice of stream processing frameworks.
Amazon Kinesis Firehose is for use cases that require zero
administration, ability to use existing analytics tools based on
Amazon S3, Amazon Redshift and Amazon Elasticsearch, and a
data latency of 60 seconds or higher.
Kinesis Firehose overview
Delivery Stream: Underlying
AWS resource
Destination: Amazon ES,
Amazon Redshift, or Amazon
S3
Record: Put records in
streams to deliver to
destinations
Kinesis Firehose Data Transformation
• Firehose buffers up to 3MB of ingested data
• When buffer is full, automatically invokes Lambda function,
passing array of records to be processed
• Lambda function processes and returns array of transformed
records, with status of each record
• Transformed records are saved to configured destination
[{" [{
"recordId": "1234", "recordId": "1234",
"data": "encoded-data" "result": "Ok"
}, "data": "encoded-data"
{ },
"recordId": "1235", {
"data": "encoded-data" "recordId": "1235",
} "result": "Dropped"
] "data": "encoded-data"
}
]
Kinesis Firehose delivery architecture with
transformations
Data transformation
function
data source transformed
records
source records
Firehose Amazon Elasticsearch
delivery stream Service
S3 bucket
delivery failure
transformation failure
source records
Kinesis Firehose features for ingest
Serverless scale Error handling S3 Backup
Best practices
Use smaller buffer sizes to increase throughput, but be
careful of concurrency
Use index rotation based on sizing
Default: stream limits: 2,000 transactions/second, 5,000
records/second, and 5 MB/second
Log analysis with aggregations
Amazon ES aggregations
Buckets – a collection of documents meeting some criterion
Metrics – calculations on the content of buckets
Metric: count
Bucket: time
host:199.72.81.55 with <histogram of verb>
199.72.81.55
1, GET GET 5
4, GET POST 2
8, POST PUT 1
12, GET
30, PUT
42, GET
58, GET
100 POST
...
Look up Field data Buckets Counts
A more complicated aggregation
Bucket: ARN
Bucket: Region
Bucket: eventName
Metric: Count
Best practices
Make sure that your fields are not_analyzed
Visualizations are based on buckets/metrics
Use a histogram on the x-axis first, then sub-aggregate
Run Elasticsearch in the AWS cloud with Amazon
Elasticsearch Service
Use Kinesis Firehose to ingest data simply
Kibana for monitoring, Elasticsearch queries for
Amazon deeper analysis
Elasticsearch
Service
What to do next
Qwiklab:
https://qwiklabs.com/searches/lab?keywords=introduction
%20to%20amazon%20elasticsearch%20service
Centralized logging solution
https://aws.amazon.com/answers/logging/centralized-
logging/
Our overview page on AWS
https://aws.amazon.com/elasticsearch-service/
Q&A
Thank you for joining!
