Amazon SQS : use cases
What is Amazon SQS?
đź“Ť Amazon Simple Queue Service (Amazon SQS) is a distributed message queueing service introduced by Amazon.com in late 2004. It supports programmatic sending of messages via web service applications as a way to communicate over the Internet.
đź“Ť SQS is intended to provide a highly scalable hosted message queue that resolves issues arising from the common producer consumer problem or connectivity between producer and consumer.
đź“Ť It also enables you to decouple and scale microservices, distributed systems, and serverless applications.
Basic Amazon SQS architecture
Distributed queues
There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
In the following scenario, your system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
Message lifecycle
The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.
đź“Ť A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.
📍 When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn’t returned to subsequent receive requests for the duration of the visibility timeout.
đź“Ť The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.
Why Amazon Simple Queue Service?
Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system.
How does Amazon SQS identify messages?
All messages have a global unique ID that Amazon SQS returns when the message is delivered to the message queue. The ID isn’t required to perform any further actions on the message, but it is useful for tracking the receipt of a particular message in the message queue.
When you receive a message from the message queue, the response includes a receipt handle that you must provide when deleting the message.
Since its 2009 launch, Oyster has published more than one million high-quality digital images. When this massive volume of images became too cumbersome to handle in-house, the company decided to offload the content to a central repository on Amazon Simple Storage Service (Amazon S3). “We migrated to Amazon S3 in 2010,” says Eytan Seidman, Co-Founder and Vice President of Product. “We chose moving to the cloud and Amazon S3 because storing images in our data center would have been too costly. Amazon S3 was a more economical solution.”
Oyster reprocesses its entire collection of photographic images a few times each year to update the copyright year and, if necessary, to change the watermarks. Using their previous solution, reprocessing the entire collection of photographs required about 800 hours to complete. In addition, Oyster often recreated existing images in new formats and sizes for mobile and tablet devices. Resizing existing images and adding new ones was slowing down the rate at which the company was able to process the collection. “Our processes were slowing down,” says Seidman. “When the iPad with Retina display came out, for example, it took us more than a week to create new sizes specifically for that resolution.” Oyster considered purchasing additional hardware, but found the cost of new hardware and routine maintenance was too high, especially when the machines would sit idle most of the time.
Moreover, there were numerous software bugs in the multiprocessing solution that the company used, but since the solution didn’t scale, Oyster didn’t bother to fix them.
The Benefits
Oyster’s old system needed approximately 400 hours to process one million photos. By using AWS, the company can process the same number of photos in about 20 hours — a 95 percent improvement. “It took less time to rewrite the code and do a full processing job with AWS than it took to do a single run with the old method,” says Seidman. “It used to take close to a week to produce photos specifically for the iPad. With AWS, we can create the photos in just a few hours. The documentation is straightforward and the dashboards are incredibly helpful.”
Oyster has also been able to reduce in-house hardware expenses by repurposing two of its old servers, which were sitting idle more than 80 percent of the time. “We estimate that we saved roughly $10,000 in capital expenditures by moving to AWS, and reduced our operating expenses by an additional $10,000,” Seidman says. He believes that AWS is a perfect match for any company performing similar batch processing. “AWS lets us move faster without worrying about machine expenditures or maintenance, which frees us to focus on other things.”
“It took less time to rewrite the code and do a full processing job with AWS than it took to do a single run with the old method.” -Eytan Seidman,
Cofounder and Vice President of Product, Oyster
