What it actually is
Amazon Kinesis Data Streams (KDS) is a managed, sharded, append-only log. Producers write records with a partition key, the key is hashed to a shard, and every record in a shard gets a sequence number in write order. Consumers read at their own pace, nothing is deleted when it is read, and records stay until the retention period expires: 24 hours by default, up to 8,760 hours (365 days).
The mental model the docs bury: a shard is not a server you rent, it is a throughput contract. One shard accepts 1 MB/s or 1,000 records per second of writes and serves 2 MB/s of reads to the consumers that share it. Everything else about Kinesis, from pricing to hot-key throttling to how many consumers you can attach, falls out of that one contract.
The second thing the docs bury is that "consuming a stream" used to be the hard part. A record does not leave the stream when you read it, so the reader has to remember where it is. For ten years that meant the Kinesis Client Library (KCL) with a DynamoDB lease table, or Lambda doing it for you, or your own checkpoint store. On November 17, 2026, AWS added queues to on-demand streams: a stream-side consumer that tracks per-record delivery state, hides in-flight records behind a visibility timeout, redelivers what you do not acknowledge, and moves repeated failures to a built-in dead-letter state. It is two days old as I write this, and it changes the default answer for a lot of workloads. Both reading styles are in the build below.
The model
Records, keys, shards. A record is a data blob (up to 10 MiB before base64 encoding since October 2025, configurable per stream with UpdateMaxRecordSize, with 1 to 10 MiB records treated as intermittent bursts), a partition key, and a sequence number assigned by Kinesis. The partition key is hashed with MD5 to a 128-bit hash key, and each shard owns a contiguous range of hash keys. Same key, same shard, same order. Different keys, no ordering guarantee.
Two capacity modes. In provisioned mode you choose the shard count and change it with UpdateShardCount, SplitShard and MergeShards. In on-demand mode Kinesis owns the shard count. A new on-demand stream starts at 4 MB/s of write and 8 MB/s of read capacity and accommodates up to double the peak write throughput it saw in the previous 30 days. It splits a shard within 15 minutes when traffic on it exceeds 500 KB/s. On-demand itself now comes in two billing flavors: On-demand Standard, with a per-stream hourly charge, and On-demand Advantage, an account-level setting launched in November 2025 with no stream-hour charge, prices at least 60% lower per GB, warm throughput you can pre-provision (and since July 2026, scale back down), and a commitment of 25 MiB/s ingest and 25 MiB/s retrieval across all on-demand streams in the Region.
Three ways to read. Shared-throughput consumers call GetShardIterator and then GetRecords in a loop; all of them split the shard's 2 MB/s and 5 read transactions per second. Enhanced fan-out (EFO) consumers register with the stream and get a dedicated 2 MB/s per shard pushed over HTTP/2 via SubscribeToShard. Queues (on-demand only) give you ReceiveQueueRecords, AcknowledgeQueueRecords and UpdateQueueRecordsVisibility, up to 10 records per call, with no shard awareness in your code at all. Queues and EFO consumers share the registered-consumer limit: 20 per stream on On-demand Standard and provisioned, 50 on On-demand Advantage.
What the queue does and does not promise. Delivery is at least once and ordering is best effort, so the per-key ordering that is the main reason to use a stream is not something a queue consumer gets. The default visibility timeout is 30 seconds (maximum 12 hours). Dead-letter handling is on by default with MaxReceiveCount 6; after that many receives without an acknowledgment the record becomes a dead-letter record that only ReceiveDeadLetterQueueRecords returns. And a queue does not extend retention: when the stream's retention expires a record, the queue stops delivering it, dead-lettered or not.
Who operates what. AWS runs the storage, replication across Availability Zones, shard placement, and in on-demand mode the scaling. You own the partition key design, producer retries, consumer checkpoints (unless you use a queue or Lambda), idempotency, and the alarms that tell you a consumer fell behind.
When to use it, when not to
You need | Use | Why not the others |
|---|---|---|
One worker per job, delete when done, no replay | Amazon SQS | A stream keeps everything for at least 24 hours and bills you for retrieval; SQS was built for this exact job |
Ordered, replayable stream read by several AWS-native consumers | Kinesis Data Streams | No brokers, IAM-native, Lambda and Firehose integrations out of the box |
Existing Kafka clients, Connect, Debezium, Kafka Streams | Amazon MSK | Kinesis does not speak the Kafka protocol |
Land a stream in S3 or Iceberg with zero consumer code | Data Firehose, or KDS streaming tables | Writing your own S3 sink is the thing these exist to delete |
A stream that several systems replay AND one worker pool that wants job semantics | KDS with a queue for the worker pool | Before November 2026 that second consumer meant KCL plus a DynamoDB lease table |
The honest version: if you have exactly one consumer and no need for replay, SQS is still cheaper and simpler. The SQS entry of this series covers it. If you already run Kafka, stay on Kafka; the MSK entry has the numbers, including a $547.50 a month idle floor for MSK Serverless that Kinesis does not have. Kinesis wins when you want a stream, you live in AWS, and you would rather pay per GB than run brokers.
A note on the new delivery options, because they compete with a hand-written consumer: since August 2026, an on-demand stream can deliver directly into Apache Iceberg tables on S3 Tables (streaming tables) or into a general purpose S3 bucket, through a "channel" resource with inline compaction. If your consumer's only job was "write it to S3 as Parquet", check that first. Firehose gets its own entry next.
What it costs
US East (N. Virginia) prices from the Kinesis Data Streams pricing page.
Provisioned: $0.015 per shard-hour ($10.95 per shard per 730-hour month) and $0.014 per million PUT payload units, where a unit is a 25 KB chunk of a record. Extended retention up to 7 days adds $0.020 per shard-hour; long-term retention beyond 7 days is $0.023 per GB-month, and reading data older than 7 days with GetRecords is $0.021 per GB. EFO is $0.015 per consumer-shard-hour plus $0.013 per GB retrieved.
On-demand Standard: $0.040 per stream-hour ($29.20 per 730-hour month, per stream, even when idle), $0.08 per GB ingested, $0.040 per GB retrieved, and $0.040 per GB received through a queue. EFO is billed per GB retrieved.
On-demand Advantage: no stream-hour charge, $0.032 per GB ingested, $0.016 per GB retrieved or received through a queue, no EFO premium, but the 25 MiB/s ingest and 25 MiB/s retrieval commitment is billed as a shortfall when you are under it. AWS's own example of an account below the minimum comes to $3,037.50 a month. Advantage is a volume discount, not a starter plan.
The surprise. AWS's on-demand Standard example is 1,000 records per second of 3 KB each with one consumer: $918.34 a month ($593.04 ingest, $296.50 retrieval, $28.80 stream-hours). The same workload in provisioned mode needs 3 shards for roughly 3 MB/s of writes: 3 x 720 x $0.015 = $32.40 in shard-hours, plus about 2,592 million PUT units at $0.014 per million, $36.29. Call it $69 a month, or under $100 with two shards of headroom. On-demand is the right default when traffic is spiky or unknown; for a steady, predictable workload it can cost an order of magnitude more. Look at your IncomingBytes for a month and do this arithmetic before you choose.
There is no Kinesis-specific free tier on the pricing page.
The limits that bite
One key, one shard. Every partition key maps to exactly one shard, so a single key can never exceed 1 MB/s or 1,000 records per second, in either mode. On-demand scaling does not save you: the docs say it does not isolate hot hash keys. Since September 2026, an on-demand stream can be switched to the AUTO record distribution strategy (UpdateStreamRecordDistributionStrategy), which spreads records evenly and ignores the partition key and ExplicitHashKey entirely. That fixes hot keys by giving up ordering, so only use it for stateless workloads.
Twice your previous peak. On-demand streams throttle writes when traffic rises to more than double the previous peak within 15 minutes, and you can see read or write exceptions for about 15 minutes. A launch-day spike on a quiet stream will be throttled unless you pre-warm (warm throughput, On-demand Advantage only) or retry.
Reads are shared. Each shard supports 5 GetRecords calls per second and 2 MB/s for all shared-throughput consumers combined. Three polling consumers on one stream are already fighting. If a call returns 10 MB, calls in the next 5 seconds throw ProvisionedThroughputExceededException. More than two independent readers is the point where you move to EFO or queues.
Iterators expire. A shard iterator is valid for 5 minutes. A consumer that pauses longer gets ExpiredIteratorException and must ask for a new one from its last checkpoint.
Retention is a deadline, not a buffer. Default retention is 24 hours. If your consumer is down for a day, the oldest records are gone, and a queue does not change that: it stops delivering records that the stream has expired. Extended retention is cheap insurance.
Account and control-plane limits. 50 on-demand streams per account by default. Provisioned shards default to 20,000 per account in N. Virginia, Oregon and Ireland, and 1,000 or 6,000 elsewhere. On-demand streams scale to 10 GB/s write in those three Regions and 200 MB/s elsewhere unless you ask. You can switch a stream between capacity modes only twice in 24 hours. CreateStream allows five streams in CREATING at once. PutRecords takes 500 records and 10 MiB per call.
Queue specifics. Queues need on-demand mode; you must delete every queue before you delete the stream or switch it to provisioned. You cannot change a queue's stream or starting position after creation. Receive and acknowledge work in batches of up to 10.
Build it
We will build one on-demand stream, orders-lab, and read it two ways. A producer writes 600 orders keyed by customer, three of them poison (an amount that is not a number). An "audit" reader walks every shard with GetRecords and prints MillisBehindLatest, the per-call version of the iterator-age metric. A "billing" worker reads the same records through a queue, acknowledges what it processes, releases what it cannot, and lets the queue move the poison records to the dead-letter state after 3 receives. Then we look at the dead letters, read the CloudWatch metrics, and tear it all down.
Prerequisites. Python 3.10+, boto3 1.44.6 or newer (older SDKs do not have the queue operations; check with python -c "import boto3; print(boto3.__version__)"), AWS credentials for an account where you can create Kinesis streams in us-east-1. Queues are new, so confirm they are offered in your Region before you start.
Estimated cost. One on-demand Standard stream for an hour is $0.04, the data is a few hundred KB (fractions of a cent at $0.08 per GB in and $0.04 per GB out), and the CloudWatch reads are a handful of API calls. Budget under $0.10 if you tear down the same day. Forget the teardown and the idle stream costs $29.20 a month.
IAM. Least privilege for the lab, with your account ID substituted:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:CreateStream", "kinesis:DeleteStream", "kinesis:DescribeStream",
"kinesis:DescribeStreamSummary", "kinesis:ListShards", "kinesis:PutRecords",
"kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:CreateQueue"
],
"Resource": "arn:aws:kinesis:us-east-1:111122223333:stream/orders-lab"
},
{
"Effect": "Allow",
"Action": [
"kinesis:CreateQueue", "kinesis:DescribeQueue", "kinesis:DeleteQueue",
"kinesis:ReceiveQueueRecords", "kinesis:AcknowledgeQueueRecords",
"kinesis:UpdateQueueRecordsVisibility", "kinesis:ReceiveDeadLetterQueueRecords",
"kinesis:AcknowledgeDeadLetterQueueRecords"
],
"Resource": "arn:aws:kinesis:us-east-1:111122223333:queue/*"
},
{
"Effect": "Allow",
"Action": ["kinesis:ListQueues", "cloudwatch:GetMetricStatistics"],
"Resource": "*"
}
]
}
Queue ARNs use a service-generated ID (queue/${QueueId}), not the queue name, which is why the lab scopes queue actions with a wildcard. In production, scope the worker's role to the one queue ARN that setup prints, and give it only receive, acknowledge and visibility.
The lab. One file, one subcommand per step. Save it as kds_lab.py:
"""kds_lab.py - Amazon Kinesis Data Streams lab: one on-demand stream, two ways to read it."""
import json, random, sys, time
from datetime import datetime, timedelta, timezone
import boto3
REGION = "us-east-1"
STREAM = "orders-lab"
QUEUE = "billing"
kinesis = boto3.client("kinesis", region_name=REGION)
cloudwatch = boto3.client("cloudwatch", region_name=REGION)
def stream_arn():
return kinesis.describe_stream_summary(StreamName=STREAM)["StreamDescriptionSummary"]["StreamARN"]
def queue_arn():
resp = kinesis.list_queues(StreamFilter=[{"StreamARN": stream_arn()}])
for q in resp["QueueSummaries"]:
if q["QueueName"] == QUEUE:
return q["QueueARN"]
raise SystemExit(f"queue {QUEUE} not found - run setup first")
def setup():
kinesis.create_stream(StreamName=STREAM, StreamModeDetails={"StreamMode": "ON_DEMAND"})
kinesis.get_waiter("stream_exists").wait(StreamName=STREAM)
s = kinesis.describe_stream_summary(StreamName=STREAM)["StreamDescriptionSummary"]
print(f"stream {s['StreamName']} {s['StreamStatus']} mode={s['StreamModeDetails']['StreamMode']} "
f"shards={s['OpenShardCount']} retention={s['RetentionPeriodHours']}h")
q = kinesis.create_queue(
QueueName=QUEUE,
StreamConfigurationList=[{"StreamARN": s["StreamARN"],
"StartingPosition": {"Type": "TRIM_HORIZON"}}],
QueueAttributes={"VisibilityTimeoutSeconds": 30,
"DeadLetterConfiguration": {"TargetType": "BUILT_IN", "MaxReceiveCount": 3}},
)["QueueDescription"]
while q["QueueStatus"] != "ACTIVE":
time.sleep(5)
q = kinesis.describe_queue(QueueARN=q["QueueARN"])["QueueDescription"]
print(f"queue {q['QueueName']} {q['QueueStatus']} {q['QueueARN']}")
def produce(n=600, poison=3):
arn = stream_arn()
orders = []
for i in range(n):
customer = f"cust-{random.randint(1, 40):03d}"
orders.append({"order_id": f"o-{i:05d}", "customer": customer,
"amount_cents": random.randint(500, 50000)})
for i in random.sample(range(n), poison):
orders[i]["amount_cents"] = "not-a-number" # the record your code cannot handle
pending = [{"Data": json.dumps(o).encode(), "PartitionKey": o["customer"]} for o in orders]
attempt = 0
while pending:
batch, pending = pending[:500], pending[500:]
resp = kinesis.put_records(StreamARN=arn, Records=batch)
if resp.get("FailedRecordCount", 0):
# PutRecords is NOT all-or-nothing: retry only the entries that carry an ErrorCode
failed = [rec for rec, res in zip(batch, resp["Records"]) if "ErrorCode" in res]
print(f" {len(failed)} throttled/failed, retrying")
pending = failed + pending
attempt += 1
time.sleep(min(2 ** attempt * 0.1, 5))
else:
attempt = 0
print(f"wrote {n} orders ({poison} poison) to {STREAM}")
def audit():
"""The classic way: one iterator per shard, you own the position."""
arn = stream_arn()
shards = kinesis.list_shards(StreamARN=arn)["Shards"]
total = 0
for sh in shards:
it = kinesis.get_shard_iterator(StreamARN=arn, ShardId=sh["ShardId"],
ShardIteratorType="TRIM_HORIZON")["ShardIterator"]
count, behind = 0, None
while it:
r = kinesis.get_records(StreamARN=arn, ShardIterator=it, Limit=1000)
count += len(r["Records"])
behind = r["MillisBehindLatest"]
it = r.get("NextShardIterator")
if behind == 0 and not r["Records"]:
break # caught up with the tip of this shard
time.sleep(0.2) # stay under 5 GetRecords/s per shard
total += count
print(f" {sh['ShardId']}: {count} records, MillisBehindLatest={behind}")
print(f"audit read {total} records across {len(shards)} shards")
def process(order):
return int(order["amount_cents"]) # raises ValueError on poison
def work(idle_rounds=3):
qarn = queue_arn()
done, failed_seen, idle = 0, 0, 0
while idle < idle_rounds:
recs = kinesis.receive_queue_records(QueueARN=qarn, Limit=10, WaitTimeSeconds=5)["QueueRecords"]
if not recs:
idle += 1
continue
idle = 0
ack, release = [], []
for r in recs:
try:
process(json.loads(r["Data"]))
ack.append({"ReceiptHandle": r["ReceiptHandle"]})
except (ValueError, KeyError):
failed_seen += 1
# give it back now instead of waiting out the 30 s visibility timeout
release.append({"ReceiptHandle": r["ReceiptHandle"], "VisibilityTimeoutSeconds": 0})
if ack:
resp = kinesis.acknowledge_queue_records(QueueARN=qarn, Entries=ack)
done += len(resp["Successful"])
for f in resp["Failed"]: # HTTP 200 does not mean every ack landed
print(f" ack failed: {f['ErrorCode']} {f.get('ErrorMessage', '')}")
if release:
kinesis.update_queue_records_visibility(QueueARN=qarn, Entries=release)
print(f"worker acknowledged {done} records, saw {failed_seen} failed deliveries")
def dlq():
qarn = queue_arn()
while True:
recs = kinesis.receive_dead_letter_queue_records(
QueueARN=qarn, Limit=10, WaitTimeSeconds=2)["QueueRecords"]
if not recs:
break
for r in recs:
print(f" DLQ {r['PartitionKey']} receives={r['ReceiveCount']} data={r['Data'][:80]!r}")
kinesis.acknowledge_dead_letter_queue_records(
QueueARN=qarn, Entries=[{"ReceiptHandle": r["ReceiptHandle"]} for r in recs])
print("dead-letter records inspected and discarded")
def metrics():
end = datetime.now(timezone.utc)
for name, stat in [("GetRecords.IteratorAgeMilliseconds", "Maximum"),
("WriteProvisionedThroughputExceeded", "Sum"),
("IncomingRecords", "Sum")]:
pts = cloudwatch.get_metric_statistics(
Namespace="AWS/Kinesis", MetricName=name,
Dimensions=[{"Name": "StreamName", "Value": STREAM}],
StartTime=end - timedelta(hours=1), EndTime=end, Period=300, Statistics=[stat],
)["Datapoints"]
vals = [p[stat] for p in sorted(pts, key=lambda p: p["Timestamp"])]
print(f" {name} ({stat}, 5 min buckets): {vals or 'no data yet'}")
def teardown():
try:
qarn = queue_arn()
kinesis.delete_queue(QueueARN=qarn)
while True: # a stream with queues cannot be deleted
try:
kinesis.describe_queue(QueueARN=qarn)
time.sleep(5)
except kinesis.exceptions.ResourceNotFoundException:
break
print("queue deleted")
except SystemExit:
pass
kinesis.delete_stream(StreamName=STREAM, EnforceConsumerDeletion=True)
kinesis.get_waiter("stream_not_exists").wait(StreamName=STREAM)
print("stream deleted")
if __name__ == "__main__":
{"setup": setup, "produce": produce, "audit": audit, "work": work,
"dlq": dlq, "metrics": metrics, "teardown": teardown}[sys.argv[1]]()
Step 1: create the stream and the queue. python kds_lab.py setup. The stream is on-demand with default retention; the script prints the open shard count AWS chose for the starting 4 MB/s of capacity. The queue starts at TRIM_HORIZON so it sees records written before the worker starts, and sets MaxReceiveCount to 3 instead of the default 6 so the dead-letter path triggers quickly.
Step 2: produce. python kds_lab.py produce. Two batches of PutRecords (500, then 100). The part of this function that matters in production is the retry loop: PutRecords returns HTTP 200 even when some entries were throttled. The response carries FailedRecordCount and an ErrorCode on each failed entry, and if you do not resend exactly those, you lose them silently. Forty customer keys means some shards get more records than others; that is the hot-key problem in miniature.
Step 3: audit the shards. python kds_lab.py audit. This is how every Kinesis consumer worked before queues: list shards, get an iterator per shard, loop GetRecords, sleep to stay under 5 reads per second per shard. Each response carries MillisBehindLatest, how far this reader is from the tip of the shard. Your output should show 600 records in total, spread unevenly across shards, ending at MillisBehindLatest=0. Notice what this code does not do: it does not save its position anywhere. A real consumer has to checkpoint (KCL uses a DynamoDB lease table), handle shard splits through ChildShards, and rebalance shards across workers. That is the work queues remove.
Step 4: run the worker. python kds_lab.py work. No shards in sight: receive up to 10, process, acknowledge. The three poison records raise ValueError; instead of waiting 30 seconds for their visibility timeout to expire, the worker sets their visibility to 0 so they come straight back. After the third receive, the queue marks each one as a dead-letter record and stops returning it. Expect worker acknowledged 597 records, saw 9 failed deliveries. To see horizontal scaling, rerun produce and start work in two terminals at once: both receive from the same queue with no shard assignment and no lease table, and in-flight records are hidden from the other worker.
Step 5: inspect the dead letters. python kds_lab.py dlq prints the three poison orders with their partition keys and receive counts, then acknowledges them, which discards them from the queue. In a real system you would fix the bug and call RestoreDeadLetterQueueRecords instead, which resets the receive count and puts them back on the normal path.
Verify. python kds_lab.py metrics reads GetRecords.IteratorAgeMilliseconds, WriteProvisionedThroughputExceeded and IncomingRecords for the last hour. CloudWatch can take a few minutes to show data. You should see an IncomingRecords sum of 600 (more if you reran produce) and an iterator age that returned to zero once audit caught up. Iterator age is the single most useful Kinesis alarm: it climbs whenever a consumer falls behind, and if it ever gets close to your retention period, you are about to lose data. Alarm on the Maximum statistic, well below retention. For the queue path, the equivalent signals are VisibleQueueRecords (scale workers on it), InFlightQueueRecords, and VisibleDeadLetterQueueRecords (alarm on anything above zero), published under AWS/Kinesis with QueueId and QueueName dimensions.
What I verified and what I did not. Every API call above was validated against the botocore 1.44.6 service model, and the script was run end to end against an in-memory simulation of the stream and queue (including a throttled PutRecords batch and the dead-letter transition). It was not run against a real AWS account, and queues are new enough that you should expect small behavior differences, especially in exactly when a record crosses into the dead-letter state.
When it breaks
ProvisionedThroughputExceededException on write. A shard is over 1 MB/s or 1,000 records per second. Check the shard-level WriteProvisionedThroughputExceeded metric (enable enhanced monitoring to get shard-level metrics): if one shard is hot, you have a hot key, and adding shards will not help. Fix the key, or switch an on-demand stream to AUTO distribution if you do not need ordering. If every shard throttles on a new on-demand stream, you doubled your previous peak inside 15 minutes; retry with backoff or pre-warm.
Records missing, no error. You ignored FailedRecordCount. PutRecords is per-entry, and the KPL hides this for you while a hand-written producer does not.
ProvisionedThroughputExceededException on read. Too many shared-throughput consumers, or a consumer calling GetRecords faster than 5 times per second per shard. Move consumers to EFO or a queue, or slow the poll loop.
ExpiredIteratorException. Your consumer held an iterator for more than 5 minutes, usually because processing a batch took too long. Get a new iterator with AFTER_SEQUENCE_NUMBER from your last checkpoint.
Iterator age climbing steadily. Consumers are slower than producers. In KCL, more workers than shards does not help, because a shard is processed by one worker at a time; you need more shards or faster processing. With a queue, add workers.
Acknowledged records come back. You read the HTTP status, not the Failed list of AcknowledgeQueueRecords, or your processing outlasted the visibility timeout and another worker got the record. Extend it with UpdateQueueRecordsVisibility for long jobs, and make processing idempotent, because delivery is at least once even when you do everything right.
Out-of-order processing on a queue. Expected. Queue ordering is best effort. If per-key order matters, use EFO or KCL on that consumer and keep the queue for order-insensitive work.
ResourceInUseException on DeleteStream. The stream still has queues, or registered EFO consumers without EnforceConsumerDeletion. Delete queues first; that flag does not apply to them.
Cleanup
python kds_lab.py teardown deletes the queue, waits until DescribeQueue returns ResourceNotFoundException, then deletes the stream with EnforceConsumerDeletion=True and waits for it to disappear. Confirm with aws kinesis list-streams --region us-east-1 that orders-lab is gone; an on-demand Standard stream left behind bills $0.04 every hour whether you use it or not. CloudWatch metrics age out on their own and cost nothing to leave. If you created an IAM role or user for the lab, delete it too.

