Amazon SQS is a message buffer with a lease system on top. A producer writes a message to a queue, SQS stores it redundantly across multiple Availability Zones, and a consumer asks for it. Asking does not remove the message. It hides it for a while, a window called the visibility timeout, and if the consumer does not explicitly delete it before the window closes, the message becomes visible again and the next ReceiveMessage call can return it.

That one design decision explains most of the service. There is no acknowledgement socket and no consumer registration. There is only "receive" (take a lease), "delete" (end the lease for good), and a clock. Everything people complain about, the duplicates, the out-of-order delivery, the message that keeps coming back, is the clock doing its job.

SQS is a pull system. Nothing is pushed to your code, and there is no server for you to size. AWS operates the storage, the replication and the scaling. You operate the consumers, the timeout values and the decision about what happens to a message that never succeeds.

A queue holds messages. The default retention is 4 days, configurable from 60 seconds up to 14 days. A message body is up to 1 MiB (it was 256 KiB until AWS raised it for both queue types in 2025), and you can attach up to 10 message attributes. Larger payloads go through the SQS Extended Client Library, which parks the body in S3 and sends a pointer, up to 2 GB.

There are three queue flavors, and you pick one at creation time.

A standard queue gives you at-least-once delivery and best-effort ordering. The throughput is, in the words of the docs, nearly unlimited per API action. The price of that is that a message can be delivered more than once and can arrive out of order. The docs say so plainly: your application should be idempotent.

A FIFO queue gives you ordering within a message group and deduplication. If you retry a SendMessage with the same MessageDeduplicationId inside a 5-minute window, SQS does not enqueue a second copy. With content-based deduplication turned on, the ID is a SHA-256 hash of the body (attributes are not hashed). The throughput is 300 API calls per second per action without batching, 3,000 messages per second with batching of 10, and in high-throughput mode up to 70,000 transactions per second in US East (N. Virginia), US West (Oregon) and Europe (Ireland), with lower ceilings in other Regions (19,000 in Ohio and Frankfurt, 9,000 in Mumbai, Singapore, Sydney, Tokyo and Spain, 4,500 in London and Sao Paulo, 2,400 elsewhere). The queue name must end in .fifo.

A fair queue is not a third type, it is a behavior of a standard queue. If producers set a MessageGroupId on each message to identify a tenant, SQS watches how in-flight messages are spread across tenants and, when one tenant holds a disproportionate share, it prioritizes returning messages from the others. On a standard queue the group ID does not enforce ordering. It only names the tenant. There is no per-tenant rate limit: if consumers are idle, they will still happily drain the noisy tenant. What you get is that the quiet tenants stop waiting behind it. CloudWatch gains a metric called ApproximateNumberOfMessagesVisibleInQuietGroups so you can see the effect.

Three more primitives complete the model. Long polling (WaitTimeSeconds up to 20) makes ReceiveMessage wait for a message instead of returning empty, which cuts both latency and request count. A delay (up to 15 minutes) hides a message from consumers right after it is sent. A dead-letter queue (DLQ) is a second queue of the same type, in the same account and Region, that receives a message once it has been received maxReceiveCount times without being deleted.

New queues are encrypted at rest by default with SSE-SQS, the SQS-owned key. You can switch to a KMS key if you need key policy control, and from that point every send and receive involves KMS calls, which matters later.

People confuse SQS with three neighbors. The decision usually comes down to who owns the delivery and how many readers there are.

The honest rule: reach for SQS when you have work, not when you have news. Work is consumed once by someone and then it is gone. News is observed by many. If you find yourself wanting to read the same message twice from two different services, you wanted a topic or a stream.

Choose FIFO when order or deduplication is part of the correctness story and your throughput fits, not by default. A FIFO queue delivers in order within a message group, which means one failing message holds up every message behind it in the same group. That is a feature when you are applying account ledger entries and a trap when you are sending emails.

The pricing model is requests, not messages and not hours. AWS gives every account 1 million requests per month free, pooled across Regions (GovCloud excepted), not carried over, with no 12-month expiry. After that you pay per million requests. At the time of writing, third-party summaries of the us-east-1 list price put standard queues at $0.40 per million requests (with volume tiers at very high scale), FIFO at $0.50 per million, and fair-queue requests, those that carry a MessageGroupId, at the standard rate plus a $0.10 surcharge. Check the AWS pricing page before you budget, because AWS's own page renders the table dynamically and I could not retrieve the numbers from it directly.

The dimension that surprises people is the 64 KB chunk. Every 64 KB of payload counts as one request. A single request can carry 1 to 10 messages up to 1 MiB in total, so a full 1 MiB batch bills as 16 requests. A 65 KB message is 2 requests. Attributes count toward the payload. Teams who moved from the old 256 KiB limit to 1 MiB payloads often find their bill went up without a single extra message, because larger bodies now fit in the queue where they used to go to S3.

The other cost levers: every call counts, including empty ReceiveMessage calls (long polling is the cure), every DeleteMessage is a request (batch them, up to 10 per call), data transfer inside a Region is free while cross-Region and internet transfer is billed at the standard AWS rates, and SSE-KMS adds KMS request charges for each call SQS makes on your behalf. The Extended Client Library adds S3 storage and request charges for the parked payloads.

Following along in this article is well inside the free tier. The DynamoDB table we use for idempotency is on-demand and costs a fraction of a cent for a few dozen writes.

The in-flight limit is the one that turns up in incidents. A message is in flight from the moment a consumer receives it until it is deleted or its visibility timeout expires. For standard queues the limit is about 120,000 in-flight messages for most queues, depending on traffic and backlog. If you hit it using short polling, ReceiveMessage returns an OverLimit error. With long polling you get no error, just fewer messages than you expected, which is worse because nothing alerts. FIFO queues have the same 120,000 in-flight ceiling, with no error at the limit but degraded processing. The cause is almost always the same: consumers receive messages and do not delete them. Increases go through AWS Support.

The visibility timeout is 30 seconds by default and 12 hours at most. It is set per queue and can be overridden per receive call. If your job takes longer than the timeout, the message reappears while you are still working on it, and a second consumer starts the same job. This is the most common way to manufacture a duplicate on a standard queue and it has nothing to do with SQS misbehaving.

Batch calls take up to 10 entries. Queue names are limited to 80 characters (alphanumeric, hyphen and underscore). Policies are limited to 8,192 bytes. Tagging is rate-limited to 30 TPS per account.

FIFO has its own sharp edges. The deduplication interval is 5 minutes and it is about send attempts, not about whether the first copy is still in the queue. Throughput scales with the number of message groups, so a single hot group is a single lane of 300 calls per second. And a DLQ on a FIFO queue can reorder: the docs warn not to use one if you must preserve the exact order of operations, because moving a message out of the line changes its meaning.

One subtle DLQ rule deserves attention. On standard queues the enqueue timestamp is preserved when a message moves to the DLQ. If the source queue retention is 4 days and the DLQ retention is also 4 days, a message that spent 3 days in the source has one day left in the DLQ. On FIFO queues the timestamp resets. The docs recommend setting DLQ retention longer than the source retention, and you should treat that as mandatory.

We build a small order-processing pipeline: a standard queue with a DLQ, a producer that sends one real duplicate and one poison message on purpose, an idempotent consumer backed by DynamoDB, and a redrive at the end. Everything runs from your laptop with boto3.

Prerequisites. Python 3.10 or later, boto3 installed (pip install boto3), the AWS CLI configured, and a Region you can afford to experiment in. The examples use us-east-1. Export AWS_DEFAULT_REGION=us-east-1 or pass a region explicitly.

IAM. Attach this to the identity you run the scripts with. It is scoped to the names we create.

Step 1: create the queues and the table. The DLQ goes first because the main queue's redrive policy needs its ARN. Note the DLQ retention is 14 days, longer than the source's 4, as discussed above. The visibility timeout is 15 seconds so the demo finishes quickly. In production, set it to a comfortable multiple of your p99 processing time.

Run it and keep the two URLs. We will look them up by name in the later scripts so you do not have to paste them around.

Step 2: the producer. It sends five orders plus one poison message in a single batch call. Order 3 is sent twice, as two separate entries with different batch IDs, the way a retrying upstream client or an at-least-once webhook would send it. The sixth entry has a body that our consumer will refuse to parse. This is the whole point: SQS will faithfully deliver the duplicate, because as far as it can tell they are two different messages.

send_message_batch can partially fail, so the Failed list matters. In real code, retry those entries, and never assume a 200 means all six went in.

Step 3: the idempotent consumer. The key insight is the dedupe key. It must be a business identifier carried in the payload, not the SQS MessageId. A duplicate produced by a retrying client gets a brand new MessageId, so keying on MessageId would let it straight through.

The handler claims the order with a conditional write before doing the work, and marks it done afterwards. If the claim fails because the order is already done, we skip the work and delete the message. If it is claimed but not done (a previous consumer died mid-flight), we leave the message alone and let the visibility timeout bring it back, because we cannot tell whether the side effect happened.

Two details are worth a second look. Failed messages are simply not deleted, so SQS handles the retry through the visibility timeout and counts receives for us. And deletes are batched, which cuts the request count on the bill by up to ten times.

Step 4: verify. Run python produce.py and then python consume.py. Within about a minute you should see something like this: orders 1001, 1002, 1003 and 1004 processed once each, one line saying SKIP duplicate order 1003, and the poison message failing with receive counts 1, 2 and 3. The poison message reappears after each 15-second visibility timeout. On the third receive that is not followed by a delete, SQS moves it to the DLQ.

Confirm the DLQ holds exactly one message:

The count is approximate by design, and it can lag a few seconds behind reality, so rerun it if you see 0 right away.

Now the part that turns a DLQ from a graveyard into a tool. Pretend you fixed the bug in the parser. Redrive the dead message back to the source queue, throttled:

Set DLQ_URL and MAIN_URL from get-queue-url first. Omitting --destination-arn sends each message back to the queue it originally came from. The task is trackable with list-message-move-tasks and cancellable with cancel-message-move-task. The poison message will fail again, since we did not fix anything, and land back in the DLQ after three more receives. That is the correct outcome: redrive is a replay, not a repair, and the throttle is there so a replay of ten thousand fixed messages does not flatten the downstream system that was already unhappy.

You see duplicates and the dedupe table is empty. You keyed on MessageId. Key on a business identifier from the payload.

The same message is processed twice, minutes apart, and nothing failed. Your handler outlived the visibility timeout. Either raise the timeout, or call ChangeMessageVisibility from the worker to extend the lease while it is still working. The timeout can reach 12 hours, but extending in small steps while heartbeating is safer than one giant value that delays every real retry.

ReceiptHandleIsInvalid on delete. The receipt handle belongs to a specific receive. If the lease expired and another consumer received the message, your handle is stale. Treat this as a signal that you were too slow, not as a retryable error.

OverLimit on ReceiveMessage. About 120,000 messages are in flight. Find the consumer that receives and never deletes, or add queues. On long polling you will not even get this error, so alarm on the in-flight metric before it matters.

The DLQ message is gone before you looked. Retention. On a standard queue the message keeps its original enqueue time, so it ages in the DLQ from the day it was first sent. Set the DLQ retention to 14 days and alarm on ApproximateNumberOfMessagesVisible greater than 0 for the DLQ, because an unwatched DLQ is just slow data loss.

CreateQueue fails with a type mismatch on the redrive policy. A standard queue needs a standard DLQ, a FIFO queue needs a FIFO DLQ, and both must live in the same account and Region.

QueueDeletedRecently. After deleting a queue you must wait at least 60 seconds before creating one with the same name. It matters during cleanup and re-runs of this tutorial.

AccessDenied after turning on a KMS key. The producer needs permission to generate data keys and the consumer needs permission to decrypt. With SSE-KMS the failure shows up on send or receive, and the SQS policy alone is not enough.

A FIFO queue seems stuck. One message in a group keeps failing, and everything behind it in that group waits until it succeeds or reaches the DLQ. Set a sensible maxReceiveCount and a DLQ on every FIFO queue, and spread unrelated work across different message groups.

Delete both queues and the table, then confirm nothing is left.

The list calls should return no demo resources. A queue with no messages and no traffic costs nothing, but the deleted-table check is the one that matters for the bill. If you plan to run the scripts again, wait 60 seconds so SQS lets you reuse the queue names.