AWS SQS and Its Use-cases

Task 22: Create an article on case study of AWS SQS. Research on use cases of AWS SQS and create a blog, Article for the same.

NOTE:
1) It is a compulsory individual task.
2) If you have done while Contest Kindly Ignore.

When we start deploying multiple applications, they will inevitably need to communicate with each other. We have two types of communications.

1: Synchronous communication which is basically application to application through APIs.

2: Asynchronous / Event based which is an application to queue to application.

Synchronous communication between applications can be problematic if there are sudden increases in traffic.

• SQS: Queue model
• SNS: Pub(publish)/sub(subscribe) model
• Kinesis: Real-time streaming model

What is SQS

What is SQS service?

Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system.

Amazon SQS is a tool in the Message Queue category of a tech stack.

Why SQS?

SQS is used by a developer to reduce the complexity of managing and operating a service. Using this we can store, send, receive and manage messages between two services Instead of sending messages directly we can send through the intermediate queue.

Queues can be used to handle malfunctions in the scalability of services. when a service depends on another service that can not scale to the same level. Having a queue between A and B is useful in such situations because the queue can handle peak demand from A, and send messages at a slower rate with which B can cope. Where A will send a message directly to B this may cause an issue if B System gets failure, corrupt, or lost data during loading data to solve this we are gone using SQS Service where this acts as Intermediator which stores data and sends to B so even B gets failure until that system set up this will be stored in SQS than sent to B.

What Are the Main Benefits of Amazon SQS?

Security: You control who can send messages to and receive messages from an Amazon SQS queue.

Server-side encryption (SSE): It lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).

Durability: To ensure the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery, and FIFO queues support exactly-once message processing.

Availability: Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.

Types of Queue:

1 AWS SQS — Standard Queue

It’s fully managed, scales from 1 message per second to 10,000s per second.

Default retention of messages: 4 days, maximum of 14 days and no limit to how many messages can be in the queue.

2 AWS SQS — FIFO (First In First Out)

The most important features of FIFO queues are First-In-First-Out delivery and exactly-once processing. The order in which messages are sent and received is strictly preserved and a message is delivered once and remains until consumer processes delete it. Duplication is not allowed in FIFO and it is limited to 300 transactions/sec (TPS).

How to Produce Messaging: SQS

We define the body, add message attributes (metadata — optional), and provide delay delivery (optional). We Get back the message identifier and an MD5 hash of the body.

SQS — Consuming Messages

Poll SQS for messages (receive up to 10 messages at a time). Process the message within the visibility timeout, and delete the message using the message ID and receipt handle.

SQS — Visibility timeout

When a consumer polls a message from a queue, the message is “invisible” to other consumers for a defined period which is the Visibility Timeout.

We can set between 0 seconds and 12 hours (default 30 is seconds). If it is too high (15 minutes) and the consumer fails to process the message, you must wait a long time before processing the message again, and if it is too low (30 seconds) and the consumer needs time to process the message (2 minutes), another consumer will receive the message and the message will be processed more than once.

AWS SQS — Delay Queue

Delay queues let you postpone the delivery of new messages to a queue for a number of seconds. If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period. Delay a message (consumers don’t see it immediately) up to 15 minutes and the default is 0 seconds (message is available right away). We can set a default queue level and also can override the default using DelaySeconds parameter.

AWS SQS — Dead Letter Queue

If a consumer fails to process a message within the Visibility Timeout (as we discussed above), the message goes back to the queue.

We can set a threshold of how many times a message can go back to the queue which is called a “redrive policy”. After the threshold is exceeded, the message goes into a dead letter queue (DLQ). We have to create a DLQ first and then designate it dead letter queue. Make sure to process the messages in the DLQ before they expire!

AWS SQS — Long Polling

When a consumer requests a message from the queue, it can optionally “wait” for messages to arrive and if there is nothing in the queue, this is called Long Polling. Long Polling decreases the number of API calls made to SQS while increasing the efficiency and latency of your application. The wait time can be between 1 sec to 20 sec (20 sec preferable). Long Polling is preferable to Short Polling. Long polling can be enabled at the queue level or at the API level using WaitTimeSeconds.

AWS SQS Security

• Encryption in flight using the HTTPS endpoint.
• Enable SSE (Server Side Encryption) using KMS.
• IAM policy must allow usage of SQS.
• SQS queue access policy.
• No VPC Endpoint, must have internet access to access SQS.

The BMW Group is using AWS for its new connected-car application that collects sensor data from BMW 7 Series cars to give drivers dynamically updated map information.

BMW Group is one of the leading manufacturers of premium cars and mobility services in the world, with brands such as Rolls Royce, BMW, and Mini. BMW built its new car-as-a-sensor (CARASSO) service in only six months leveraging Amazon Simple Storage Service, Amazon SQS, Amazon DynamoDB, Amazon Relational Database Service (Amazon RDS), and AWS Elastic Beanstalk. By running on AWS, CARASSO can adapt to rapidly changing load requirements that can scale up and down by two orders of magnitude within 24 hours. By 2018 CARASSO is expected to process data collected by a fleet of 100,000 vehicles traveling more than eight billion kilometers.