Unleashing Control: Maximizing Concurrency in AWS Lambda with SQS
AWS Lambda now provides a way to control the maximum number of concurrent functions invoked by Amazon SQS as an event source. This new feature allows you to manage the concurrency of your Lambda functions processing messages in individual SQS queues, making your serverless applications even more efficient and scalable.
In this blog post, I will walk you through the process of setting the maximum concurrency of SQS triggers when using SQS as an event source with Lambda. I will also give you an overview of Lambda’s scaling behavior using this architectural pattern, the challenges this feature addresses, and a demo of the maximum concurrency feature.
First, let’s take a look at how Lambda uses event source mapping to process items from a stream or queue. The event source mapping reads from an event source, such as an SQS queue, filters messages, batches them, and invokes the mapped Lambda function. The scaling behavior for Lambda integration with SQS FIFO queues is straightforward — a single Lambda function processes batches of messages within a single message group to ensure that messages are processed in the order.
For SQS standard queues, the event source mapping polls the queue to consume incoming messages, starting with five concurrent batches and five functions at a time. As messages are added to the SQS queue, Lambda continues to scale out to meet demand, adding up to 60 functions per minute, up to 1,000 functions, to consume those messages.
One of the challenges that can arise when a large number of messages are in the SQS queue is that Lambda scales out, adding additional functions to process the messages. This can consume the concurrency quota in the account. To prevent this from happening, you can set reserved concurrency for individual Lambda functions. This ensures that the specified Lambda function can always scale to that much concurrency but cannot exceed this number.
The launch of maximum concurrency for SQS as an event source allows you to control Lambda function concurrency per source. You set the maximum concurrency on the event source mapping, not on the Lambda function. This feature does not change Lambda’s scaling or batching behavior with SQS. You can continue to batch messages with a customized batch size and window. It rather sets a limit on the maximum number of concurrent function invocations per SQS event source. Once Lambda scales and reaches the maximum concurrency configured on the event source, Lambda stops reading more messages from the queue. This feature also allows you to define the maximum concurrency for individual event sources when the Lambda function has multiple event sources.
In conclusion, the new maximum concurrency feature for SQS as an event source for AWS Lambda offers customers more control over concurrent invokes for each SQS event source. It eliminates the need for setting reserved concurrency limits, resulting in more robust and scalable event-driven applications on AWS Lambda. To learn more about this feature and how to get started, check out the official AWS blog post:
As always, I hope this blog post helps you understand the new maximum concurrency feature for AWS