AWS Prescriptive Guidance

Improve application performance

by using AWS Lambda event
filtering and partial batch responses
for Amazon SQS event sources
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources

AWS Prescriptive Guidance: Improve application performance by using

AWS Lambda event filtering and partial batch responses for Amazon
SQS event sources
Copyright © 2023 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not
Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or
discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may
or may not be affiliated with, connected to, or sponsored by Amazon.
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources

Table of Contents
Introduction ...................................................................................................................................... 1
Targeted business outcomes ........................................................................................................ 1
Benefits of using partial batch responses for Amazon SQS event sources ................................................... 2
Benefits of using Lambda event filters for Amazon SQS event sources ...................................................... 3
Example use cases for Lambda event filters ................................................................................... 3
Partial batch responses best practices ................................................................................................... 4
Avoiding snowball anti-patterns ................................................................................................... 4
Lambda event filtering best practices ................................................................................................... 6
Avoiding non-matching message accumulation .............................................................................. 6
FAQ .................................................................................................................................................. 7
Can I apply Lambda event filtering and partial batch responses to my existing Lambda and Amazon
SQS application? ........................................................................................................................ 7
Are the Lambda event filtering and partial batch responses features limited to specific programming
languages? ................................................................................................................................ 7
Can I use these features with the AWS Cloud Development Kit (AWS CDK)? ........................................ 7
Resources .......................................................................................................................................... 8
Document history .............................................................................................................................. 9

iii
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Targeted business outcomes

Improve application performance by

using AWS Lambda event filtering
and partial batch responses for
Amazon SQS event sources
Alan Oberto Jimenez, Amazon Web Services (AWS)

September 2022 (document history (p. 9))

This guide covers best practices for configuring AWS Lambda event filtering and partial batch responses
for Amazon Simple Queue Service (Amazon SQS) event sources. These controls address two common
performance pitfalls associated with Lambda’s default message batch processing behavior:

• Poison-pill messages that cause a Lambda function to retry entire Amazon SQS message batches
multiple times
• Irrelevant Amazon SQS events that cause a Lambda function to invoke unnecessarily

By setting up these controls, you can configure your Lambda functions to process Amazon SQS message
batches more efficiently and improve your serverless application’s performance.

Targeted business outcomes

This guide helps you do the following:

• Improve your serverless application’s user experience by reducing the time it takes for your Lambda
functions to process Amazon SQS message batches.
• Improve your serverless architecture’s operational capabilities and overall stability by isolating Amazon
SQS messages that your Lambda functions can’t process.
• Reduce the operational cost of your serverless application by lowering the number of unnecessary
Lambda function invocations.
• Simplify your Lambda functions’ code to improve their performance when processing Amazon SQS
message batches.

1
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources

Benefits of using partial batch

responses for Amazon SQS event
sources
Configuring partial batch responses gives your Lambda functions the ability to process partial Amazon
SQS message batches and retry only failed messages. This removes the need for repetitive data transfer
and increases throughput.

By default, if a Lambda functions fails to process one message in an Amazon SQS message batch, then
the entire batch returns to the queue. After the visibility timeout occurs, the Lambda function then
receives the message batch again. If the function fails to process valid messages multiple times, then
Amazon SQS sends the messages to your dead-letter queue, if you've configured one.

Because of this default batch processing behavior, a single failed (poison-pill) message can cause a
Lambda function to retry a message batch multiple times. These message batch retries can decrease an
application’s performance—even if your function code is idempotent and capable of handling messages
multiple times.

2
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Example use cases for Lambda event filters

Benefits of using Lambda event

filters for Amazon SQS event sources
Configuring Lambda event filters gives your functions the ability to be invoked only by the filtering
criteria that you specify. This helps reduce traffic to the functions and simplifies code, which can help
reduce cost.

If no event filters are configured, it’s possible for a Lambda function to receive irrelevant Amazon SQS
events that cause the function to invoke unnecessarily. These irrelevant messages can be caused by
message formatting issues (for example, missing fields) or fields that contain values that aren’t relevant
to the function.

Example use cases for Lambda event filters

Consider an application that notifies a customer care team when sales of more than $100 USD occur.
However, all sales of any amount ($5, $10, etc.) are recorded in the organization’s Amazon SQS queue.
Each time the application’s Lambda function polls the queue, it receives messages that aren’t relevant
to the application’s business logic. In this case, the processing fees for those irrelevant messages are
charged, which increases the operational cost of the application. Configuring Lambda event filters in this
situation would limit the number of messages that the Lambda function must process and reduce cost.

For a hotel-booking application that needs to process batches of hotel reservations, specific request
fields—like Username—are critical to the success of the operation. Without event filters, developers
would need to code a logic into the application that identifies whether or not the field is present in each
request. The application’s code would also need to be able to verify if each request included the correct
data type. By using Lambda event filters, all of this logic can be abstracted to Amazon SQS to reduce
development effort and code complexity.

3
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Avoiding snowball anti-patterns

Best practices for implementing

partial batch responses
The following are best practices for configuring partial batch responses for Amazon SQS event sources:

• Configure a dead-letter queue to avoid creating a snowball anti-pattern in your serverless application’s
architecture. For more information, see the Avoiding snowball anti-patterns (p. 4) section of this
guide.
• Configure your Lambda function event source mapping to make only the failed messages visible. To do
this, you must include the value ReportBatchItemFailures in the FunctionResponseTypes list when
configuring your event source mapping. For more information, see Reporting batch item failures in the
AWS Lambda Developer Guide.
• Define the number of times that you want a message delivered to the source queue before it’s moved
to the dead-letter queue. Make sure that the value you define fits your application’s use case by
identifying the most likely causes of failure and their estimated recovery times. To define the number
of retries, you must configure the maxReceiveCount value on the source queue’s RedrivePolicy. For
more information, see SetQueueAttributes in the Amazon SQS API Reference. Also, see Introducing
Amazon Simple Queue Service dead-letter queue redrive to source queues on the AWS Blog.
• Make sure that your Lambda function code is idempotent and capable of handling messages multiple
times. This prepares the function’s code to support individual jobs inside an Amazon SQS message
batch. A good starting point is incorporating ReportBatchItemFailures in your event source mapping
configuration. For more information, see Reporting batch item failures in the AWS Lambda Developer
Guide. Also, see How can I prevent an Amazon SQS message from invoking my Lambda function more
than once?
• Consider using tools like aws-embedded-metrics or AWS Lambda Power Tools for Python that are
available in GitHub. These tools help you incorporate business metrics in your function code to track
failed jobs and the details on those jobs.

Note
Code-level performance tracking is required to track the overall performance of an application
that uses partial batch processing. After partial batch processing is configured, Lambda function
invocations almost always succeed, no matter what the result of the batch processing is.

Avoiding snowball anti-patterns

Lambda and Amazon SQS can’t control the messages that upstream microservices write to an Amazon
SQS queue. If there are messages that can’t be processed, Lambda returns those unprocessed messages
to the source Amazon SQS queue, unless a separate dead-letter queue is configured. Those unprocessed
messages are then retried by the Lambda function in each following Amazon SQS message batch,
fail, and return to the queue to be retried. If no dead-letter queue exists, the number of unprocessed
messages returned to the Amazon SQS queue eventually outnumbers the valid messages in the queue.

This type of snowball anti-pattern—where each successive Lambda function invocation makes the
problem worse—can cause the following issues:

• Poor user experience because the jobs take much longer to process than usual, or don’t process at all
• Increased cost proportional to the exponentially increasing number of messages in the Amazon SQS
queue and message retries

4
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Avoiding snowball anti-patterns
• Reduced Lambda computing capacity for the application or entire AWS account if the function
doesn’t have a limit on its invocation requests

To avoid creating a snowball anti-pattern when configuring partial batch responses in Amazon SQS,
it’s a best practice to also create a dead-letter queue. This separate queue can store messages that
aren’t processed successfully and help you better manage the lifecycle of your application’s unprocessed
messages.

For instructions, see Configuring a dead-letter queue (console) in the Amazon SQS Developer Guide.

5
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Avoiding non-matching message accumulation

Best practices for implementing

Lambda event filtering
The following are best practices for configuring Lambda event filtering for Amazon SQS event sources:

• Familiarize yourself with the Lambda service’s filter rule syntax—specifically the filter rule syntax
Lambda uses for Amazon SQS. Then, design an event filter according to your business logic. The
effectiveness of any event filter depends on its ability to invoke a Lambda function only when it
should. To accomplish this, you must apply the correct syntax to your filter rules so that the rules
reflect the needs of your application’s business logic.
• In situations where the application’s business logic demands a more complex series of filter conditions,
make sure that you use multi-level filtering. For more information and example data objects, see
Multi-level filtering in the AWS Lambda Developer Guide.
• Limit how long you want Amazon SQS to retain unprocessed messages for by defining the
MessageRetentionPeriod value in your Amazon SQS request parameters. Make sure that the value
you define fits your application’s use case by identifying how long it typically takes a valid message to
process. For more information, see SetQueueAttributes in the Amazon SQS API Reference. Also, see the
Avoiding non-matching message accumulation (p. 6) section of this guide.
• Configure an Amazon SQS dead-letter queue in situations where Lambda function unavailability could
make it impossible to process valid messages. This extra Amazon SQS queue helps avoid creating
a snowball anti-pattern in your serverless application’s architecture. For more information, see the
Avoiding snowball anti-patterns (p. 4) section of this guide.
• Make sure that you activate the event filter feature on the Lambda service. The feature must be
activated when you initially create your Lambda function. For instructions, see Attaching filter criteria
to an event source mapping (console) or Attaching filter criteria to an event source mapping (AWS CLI)
in the AWS Lambda Developer Guide.

Avoiding non-matching message accumulation

If an event filter isn’t designed appropriately, a high volume of irrelevant messages can quickly
overwhelm a Lambda function’s Amazon SQS queue. In this scenario, a Lambda function processes all of
the messages in an initial message batch, even if few of those messages are relevant. While the function
takes longer to process the additional, irrelevant messages, more irrelevant messages are added to the
Amazon SQS queue. These new messages cause the function to take even longer to process the next
message batch, unless a MessageRetentionPeriod parameter is configured. Over time, the number of
irrelevant messages in the source queue outnumber the valid messages.

The MessageRetentionPeriod parameter limits how long an Amazon SQS queue retains unprocessed
messages for in seconds. For example, a MessageRetentionPeriod value of 3,600 seconds would remove
all of the unprocessed messages from a queue after one hour.

For instructions, see SetQueueAttributes in the Amazon SQS API Reference.

6
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources
Can I apply Lambda event filtering and
partial batch responses to my existing
Lambda and Amazon SQS application?

FAQ
Can I apply Lambda event filtering and partial
batch responses to my existing Lambda and
Amazon SQS application?
Yes, just make sure that you follow the implementation best practices outlined in this guide.

Are the Lambda event filtering and partial

batch responses features limited to specific
programming languages?
No, the features discussed in this guide are supported in any programming language that Lambda
currently supports.

Can I use these features with the AWS Cloud

Development Kit (AWS CDK)?
Yes, all of the features discussed in this guide are supported by the AWS CDK.

7
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources

Resources
FilterCriteria (AWS Serverless Application Model Developer Guide)

8
 AWS Prescriptive Guidance Improve application
performance by using AWS Lambda event filtering and
partial batch responses for Amazon SQS event sources

Document history
The following table describes significant changes to this guide. If you want to be notified about future
updates, you can subscribe to an RSS feed.

Change Description Date

Initial publication (p. 9) — September 30, 2022