Building a message handling service with Amazon SQS and AWS Lambda!

Journey: 📊 Community Builder 📊

Subject matter: Building on AWS

Task: Building a message handling service with Amazon SQS and AWS Lambda!

This project practices Scalability.

Using the 6 Pillars of the AWS Well-Architected Framework, Operational Excellence, Security, Reliability, Performance Efficiency, and Sustainability will be achieved in this build.

I’m really enjoying the serverless aspect of building in the cloud.

This week, I built a message-handling service with Amazon SQS and AWS Lambda!

I started by creating a new Amazon Simple Queue Service [SQS] queue, which is a highly scalable message queuing service, that acts as a buffer and queueing requests to be processed asynchronously.

Amazon SQS is a managed service and can scale as your services require it without manual intervention. You can also choose between two types of queue, Standard and FIFO. We’ll choose Standard for this project as we don’t mind what order the messages are polled in. However, if you need it, a FIFO queue configuration will process in a First In, First Out order and process a task exactly once.

After configuring the SQS queue, I then used Lambda to automatically manage the SQS queued events. This becomes a simple, effective fault-tolerant event-driven architecture.

Resource credit: This serverless architecture was created using guidance from Oğuzhan Hızıroğlu on Medium Here.

What did I use to build this environment?

• Amazon Simple Queue Service
• AWS Lambda
• Amazon CloudWatch
• AWS IAM

What is built?

• An SQS queue
• A Lambda Function which handles messages sent to the SQS queue
• A CloudWatch log stream to confirm Lambda has handled the queued data

Use case: Businesses use SQS and serverless event-driven architectures to collect datasets, such as sensor data which is then queued and then passed for onward processing and data handling. Additional services such as Lambda, S3, DynamoDB, or RDS and SNS can be utilised to house and analyse sensor data being passed to the SQS queue(s).

In this task, I created a new SQS queue in my desired region. I then created some test messages to be polled and received. After manually processing and deleting those queued messages, I created a new Lambda function.

To keep things simple, this function leveraged one of the existing blueprint offerings for SQS. After creating a new execution role, I pointed my Lambda function to my SQS queue and enabled the trigger. As I used the SQS queue blueprint, the Lambda code was created for me which made things easier!

After returning to the SQS queue and sending some new message requests to the queue, I observed that when polling for messages, my new message was not seen. This is good news as it meant that my Lambda function had successfully handled the request and cleared SQS of the queued message I sent. Great stuff!

I then opened Amazon CloudWatch and found the new log group stream waiting.

To further evidence the Lambda function doing its job correctly, I found the log entry and confirmed my message had been processed and deleted from the queued events!

Some of the highlights…

New SQS queue:

SQS test message sent:

Polling for messages, message received:

New Lambda function:

Blueprint-managed Lambda code:

SQS trigger enabled:

A new test message was sent, now handled by Lambda:

Polling for new messages:

CloudWatch log stream check:

CloudWatch indicating successfully handled message by Lambda:

My interpretation of the architecture:

I hope you have enjoyed the article!