In this tutorial we will create a standard SQS queue then write simple Python script( as AWS Lambda or Local) to send messages to SQS endpoint. Once it is done we create another lambda function, configure it to receive messages from SQS as events.
I am going straight in to the steps without further delay,
Let us create a standard SQS queue with the name “my-test-queue” or any other name of your choice. All I did in this step is to go to AWS Management Console, search for SQS, In the SQS homepage, click “Create queue” button, gave the name of the queue in the input box. I have not changed any default values. Make sure the queue type is “Standard“. My default region is US-East-1
Note the SQS queue url, we need this later.
Take a look at the Amazon SQS pricing page for more information https://aws.amazon.com/sqs/pricing/
Step 2: Send message to SQS using Python
Now that our SQS queue is ready, let us write a simple python code to push messages to the SQS queue. For that we need boto3 installed on your mac / windows or Linux Operating system you are running on.
In case If you don’t have boto3 package installed or would like to do some test before really installing this boto3 package on your machine there is an easy way out, you can try running the python code we are going to discuss here as a Lambda function.
At the time of writing this tutorial for Python 3.10 / 3.9 / 3.8 / 3.7 Lambda runtimes AWS has installed boto3 as a default package. More details https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
Note: You have to create an IAM user or need a role to push messages to SQS queue. IAM user should have sufficient permissions to SendMessage to SQS. Please take a look at the document link below to learn more about IAM policies with examples
I am assuming that you already have an IAM user ready for the Python Script.
import boto3
aws_id = ''
aws_secret = ''
queue_url='https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/my-test-queue'
def sqs_queue(data):
sqs = boto3.client('sqs',aws_access_key_id=aws_id, aws_secret_access_key=aws_secret)
response = sqs.send_message(QueueUrl=queue_url,MessageBody=(data))
print(response['MessageId'])
data="This is a test message"
sqs_queue(data)
If you wish to run it as a Lambda function then use this code, give some name for the Lambda function and create a simple test trigger to run the script. (I have selected Python runtime as 3.10).
Let us call this main lambda function as “send_message_to_SQS“
import boto3
aws_id = ''
aws_secret = ''
queue_url='https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/my-test-queue'
def sqs_queue(data):
sqs = boto3.client('sqs',aws_access_key_id=aws_id, aws_secret_access_key=aws_secret)
response = sqs.send_message(QueueUrl=queue_url,MessageBody=(data))
print(response['MessageId'])
data="This is a test message"
def lambda_handler(event, context):
sqs_queue(data)
# TODO implement
return {
'statusCode': 200,
'body': json.dumps('Message sent to SQS')
}
By executing the code you are just sending a message “This is a test message” to SQS queue. Feel free to modify this text in the code.
Test the function and you should see similar output
Step 3: Verify messages in SQS
Now that you have executed your lambda function successfully, let us go to the SQS console . On the header there will be a button with the name “Send and receive messages“.
Click on “Poll for messages” button to see the message we sent to the queue.
As you can see we have 1 message ending in my case message ID ending with 67385ea. Click on it to view its content.
Same message we sent from Lambda function i.e. “This is a test message“
Step 4: Configure lambda trigger on SQS queue
Awesome we have done a great job of sending message to SQS and viewing it via SQS console. Now how can we consume it programmatically by triggering another Lambda function that can read message from SQS.
Let us do that next, as you can see from the screenshot below. I have created a new Lambda function named trigger_from_SQS in Python (Selected Runtime as 3.10- latest version available at the time of writing this tutorial) . The purpose of this function is to read message from SQS.
Add the following code to the function, deploy and test it – of course you will receive an error message(ignore it). We are doing this to make sure that you have created CloudWatch log and configured test events for the specific function.
def lambda_handler(event, context):
# TODO implement
print(type(event))
for rec_sqs in event['Records']:
data = rec_sqs["body"]
print(type(data))
print(data)
return {
'statusCode': 200,
'body': json.dumps('Trigger from SQS')
}
If you run the function directly you may see this ( Expected error as we are not passing any test events with the key mapping it is looking for)
Response
{
"errorMessage": "'Records'",
"errorType": "KeyError",
"requestId": "ae55299a-a15b-4cba-8a4e-81e5224ce9ca",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 6, in lambda_handler\n for rec_sqs in event['Records']:\n"
]
}
After function is created, deploy and test run it . Then go to the Execution role details of the function and add the policy named AWSLambdaSQSQueueExecutionRole . This is mandatory to add function as a trigger for SQS otherwise it will not work. You can also create custom policy / role to call specific API according to your use case.
Execution Role name of Lambda function is in console under Configuration Tab > Permissions
Go back to the SQS console and under “Lambda Trigger” tab , click on the button that says “Configure Lambda function trigger“
Select the Lambda function we just created from the dropdown – trigger_from_SQS and hit Save button.
While saving the Lambda function from SQS as the trigger if you are getting similar error message like the one below, make sure execution Role for trigger_from_SQS in IAM has sufficient permission to receive messages from the SQS queue.
Error code: InvalidParameterValueException. Error message: The provided execution role does not have permissions to call ReceiveMessage on SQS
If everything is successful you should see the function with State as “Enabled“. It will be Creating state for a while before it ultimately switch to Enabled
Step 5: Receive messages from SQS to another Lambda
Execute the Main Lambda function named send_message_to_SQS again . Check whether the message reached trigger_from_SQS Lambda function (verify CloudWatch Logs to confirm the message delivery). You can also follow Step 3 to verify that message indeed reached SQS before checking the CloudWatch logs.
Since we are printing the event type and body you should see an output similar to the image below.
If you would like to see the entire event instead of body, add “print(event)” just before or after type(event) section .
print(type(event))
print(event)
This will show you the entire payload pushed by SQS to Lambda. It contains other details like eventSource, md5OfBody, ApproximateFirstReceiveTimestamp , SentTimestamp etc.,
admin