This website uses cookies

Read our Privacy policy and Terms of use for more information.

What it actually is

Amazon Data Firehose (called Amazon Kinesis Data Firehose until February 2024) is a managed delivery service. Producers send it records, it holds them in a buffer, optionally transforms them, and then writes them in batches to a destination: Amazon S3, Amazon Redshift, OpenSearch Service and OpenSearch Serverless, Splunk, Apache Iceberg tables, Snowflake, or an HTTP endpoint, including providers such as Datadog, New Relic, Dynatrace, MongoDB and Elastic. There are no shards, no consumers, no checkpoints and no servers. You create a "Firehose stream", point things at it, and objects appear in the destination.

The mental model the docs bury: Firehose is a buffer with a flush policy. Every record you send waits until either a size threshold or a time threshold is reached, and then the whole buffer is written as one object or one batch. Latency, file size, object count, and a good part of your bill are all the same knob seen from different angles. A five-minute buffer gives you large files and five minutes of delay. A one-minute buffer gives you fresher data and five times as many objects.

The second thing the docs bury is what dynamic partitioning does to that model. When Firehose writes records to S3 under keys computed from the record content (say event_type=checkout/dt=2026-11-20/), it does not keep one buffer per stream anymore. It keeps one buffer per active partition, and each one flushes on its own. Partition by a column with 4 values and you get 4 buffers. Partition by user ID and you get an error prefix full of rejected records, for reasons we will get to.

The model

Sources. A Firehose stream has exactly one source, fixed at creation. Direct PUT means your code (or an AWS service such as CloudWatch Logs, AWS WAF or Amazon EventBridge) calls PutRecord or PutRecordBatch. Alternatively the stream reads from an existing Kinesis data stream or an Amazon MSK topic, and Firehose runs the consumer for you.

The processing pipeline. Inside the stream, the order is fixed and documented: KPL (protobuf) deaggregation, JSON or delimiter deaggregation, Lambda processing, data partitioning, data format conversion, and S3 delivery. Every stage is optional. Lambda lets you rewrite records arbitrarily. Partitioning computes the S3 key per record, either with a built-in jq parser (jq 1.6 only) or with keys returned by your Lambda. Format conversion turns JSON into Apache Parquet or Apache ORC using a schema stored as an AWS Glue Data Catalog table.

Buffering hints. SizeInMBs ranges from 1 to 128 (default 5) and IntervalInSeconds from 0 to 900 (default 300). Whichever is reached first triggers delivery. Zero buffering exists for latency-sensitive destinations, but it is not available with dynamic partitioning. AWS calls these values hints, Firehose may pick different ones when it decides that is better, and because buffering happens in stages the end-to-end delay can reach about 1.5 times the configured interval.

Delivery semantics. If the destination is down, a Direct PUT stream keeps records for up to 24 hours while it retries. With a Kinesis data stream as the source, the retention of that stream is your safety net instead. Records that fail processing, partitioning or conversion are not dropped silently: they go to an ErrorOutputPrefix in your bucket, organized by error type. Retries mean a record can occasionally be delivered twice, so anything downstream that needs exactly-once has to deduplicate on a key you put in the record.

Who operates what. AWS runs capacity, scaling, buffering, retries, and the conversion and partitioning engines. You own the IAM role Firehose assumes to write to your destination, the Glue schema if you convert formats, the partition key design, producer retries for Direct PUT, and the shape of the files you end up with. That last one matters more than it sounds.

When to use it, when not to

You need

Use

Why

Land events in S3, Redshift, OpenSearch, Splunk or Snowflake with no consumer code

Data Firehose

Buffering, retries, partitioning and Parquet conversion are configuration, not code

Several independent consumers replaying the same ordered stream

Kinesis Data Streams or MSK

Firehose has no read API; it is write-in, deliver-out

An on-demand Kinesis stream whose only consumer writes Iceberg or S3

KDS streaming tables or S3 delivery (August 2026)

Delivery is now built into the stream itself; compare before adding a Firehose hop

Per-record transformation with business logic, joins or state

Managed Service for Apache Flink, or Lambda

Firehose transforms one record at a time and has no state

Sub-second delivery into a search index

Your own consumer, or Firehose with zero buffering

Partitioned S3 delivery cannot go below the buffer interval

The honest summary: Firehose is the right default when the destination is an analytics store and you would otherwise be writing an S3 sink yourself. It is the wrong tool when you need to read the data back out of the pipe, or when the transformation is the actual product. The August 2026 Kinesis Data Streams delivery features shrink its territory for one specific case (on-demand KDS to Iceberg or S3). They do not replace jq partitioning, Parquet conversion from Direct PUT, or the long list of non-S3 destinations.

What it costs

There is no free tier and no hourly charge for the stream itself. You pay per GB, and the per-GB rate depends on the source and the features you turn on. In us-east-1, as of the pricing page updated November 12, 2026:

Dimension

Price (us-east-1)

Ingestion, Direct PUT or Kinesis Data Streams source (first 500 TB/month)

$0.029 per GB, billed in 5 KB increments per record

Ingestion, Amazon MSK source

$0.055 per GB, no 5 KB rounding

Ingestion, vended logs (for example CloudWatch Logs)

$0.13 per GB, no 5 KB rounding

Format conversion to Parquet or ORC

$0.018 per GB ingested, 5 KB increments

Dynamic partitioning

$0.020 per GB processed + $0.005 per 1,000 S3 objects + $0.07 per jq processing hour

Delivery to a VPC destination

$0.01 per GB + $0.01 per AZ-hour

Delivery to Apache Iceberg tables (KDS source example)

$0.045 per GB

The one that surprises people is the 5 KB rule. Ingestion is billed per record rounded up to the next 5 KB (5,120 bytes). A 3 KB record is billed as 5 KB. A 200-byte clickstream event is also billed as 5 KB, which is 25 times its size. Send 100 million 200-byte events a day and the data itself is about 18.6 GB a day, roughly $16 a month at $0.029. The billed volume is about 477 GB a day, or roughly $415 a month, before conversion or partitioning, and conversion uses the same 5 KB rounding.

The fix is to send fewer, fuller records. Pack around 20 events into one newline-delimited record of about 4 KB and the same traffic bills at roughly $21 a month. If you partition, enable the JSON or delimiter deaggregation stage so Firehose splits the packed record back into events before the jq parser sees them. Do not confuse this with PutRecordBatch: a batch of 500 tiny records is still 500 records rounded up separately.

The second surprise is the object charge. Dynamic partitioning bills $0.005 per 1,000 objects, and S3 bills its own PUT requests on top. 500 active partitions flushing every 60 seconds is about 21.6 million objects a month, or $108 in Firehose object charges alone, plus a small-file problem for every query engine reading them. Larger buffers cost you freshness and save you money in two places.

The limits that bite

Direct PUT throughput per stream. In US East (N. Virginia), US West (Oregon) and Europe (Ireland) the default is 500,000 records per second, 2,000 requests per second and 5 MiB per second, combined across PutRecord and PutRecordBatch. In other Regions it is 100,000 records, 1,000 requests and 1 MiB per second. Firehose raises the limit automatically when it sees sustained throttling, but the throttling happens first, so producers must retry. None of this applies when the source is a Kinesis data stream; then the stream's shards are the limit.

Record and batch size. A record is at most 1,000 KiB before base64 encoding. PutRecordBatch takes at most 500 records or 4 MiB per call, whichever comes first, and that cannot be raised.

PutRecordBatch is not all-or-nothing. A call can succeed overall while some records inside it fail. The response carries FailedPutCount and a per-record ErrorCode. Ignore them and you lose data with a 200 response in hand.

Dynamic partitioning is a one-way door. You can enable it only when you create the stream, and once enabled it cannot be disabled. It only works with S3 as the destination, it requires an ErrorOutputPrefix, and its jq parser only reads JSON.

500 active partitions. A partition is active while it has data sitting in the buffer. The default quota is 500 per stream and can be raised to 2,500. Past the limit, records do not wait: they go to the error prefix with the activePartitionExceeded type. With a 60-second buffer, a key that produces 3 new partitions per second keeps about 180 partitions active. A key with 10,000 values per minute does not fit at all. Each active partition also tops out at 1 GB per second.

Prefix rules. Prefix and ErrorOutputPrefix are each capped at 512 characters after evaluation. !{firehose:error-output-type} is allowed only in ErrorOutputPrefix, and partition key expressions are not allowed there at all. Timestamps from !{timestamp:...} are arrival time in UTC unless you set CustomTimeZone, which is not the same thing as the time the event happened.

Format conversion is strict. Input must be concatenated JSON objects ({"a":1}{"a":2}), not a JSON array. The OpenX JSON SerDe accepts ISO-8601 style timestamps and epoch seconds or milliseconds; the Hive SerDe needs Joda-Time patterns and rejects column names with periods. The Glue table is the schema: a field your producer sends that is not in the table is silently not written, and a value that cannot be coerced to the column type sends the record to the error prefix.

Control plane. CreateDeliveryStream, DescribeDeliveryStream, UpdateDestination and friends are hard-limited to 5 calls per second each. Stream count is 5,000 per Region in the largest Regions and as low as 100 in the newest ones.

Build it

The goal: a Direct PUT Firehose stream that receives JSON events from Python, extracts two partition keys with jq (event_type and an event-time dt taken from the record's own timestamp, not the arrival time), converts every record to Snappy-compressed Parquet using a Glue table as the schema, and writes it to s3://<bucket>/events/event_type=<type>/dt=<date>/. Five deliberately broken records go in with the good ones so you can see the error prefix work.

Prerequisites. Python 3.10+, pip install boto3 pyarrow, AWS credentials for an account where you can create IAM roles, and Region us-east-1 (change REGION and the bucket creation call if you use another one).

IAM for you, the operator. The script needs sts:GetCallerIdentity; s3:CreateBucket, s3:ListBucket, s3:GetObject, s3:DeleteObject, s3:DeleteBucket; glue:CreateDatabase, glue:CreateTable, glue:DeleteTable, glue:DeleteDatabase; iam:CreateRole, iam:PutRolePolicy, iam:PassRole, iam:DeleteRolePolicy, iam:DeleteRole; logs:CreateLogGroup, logs:CreateLogStream, logs:DeleteLogGroup; firehose:CreateDeliveryStream, firehose:DescribeDeliveryStream, firehose:PutRecordBatch, firehose:DeleteDeliveryStream; and cloudwatch:GetMetricStatistics.

IAM for Firehose. The script creates a role trusted by firehose.amazonaws.com with the documented S3 delivery actions on the bucket, glue:GetTable, glue:GetTableVersion and glue:GetTableVersions on the catalog, database and table (plus glue:GetSchemaVersion, which the docs grant on *), and logs:PutLogEvents on its log group. No Kinesis or KMS statements, because this stream uses neither.

Cost of following along. About 2,000 records billed at 5 KB each is under 10 MiB of ingestion, conversion and partitioning, a few objects, and a few seconds of jq processing: well under one cent. The Glue database and table fit in the Data Catalog free tier, and the S3 storage is a few kilobytes. Run the cleanup and the steady-state cost is zero, because an idle Firehose stream has no hourly charge.

Save this as fh_demo.py:

"""Amazon Data Firehose demo: Direct PUT -> jq dynamic partitioning -> Parquet in S3.

Usage:
  python fh_demo.py setup     # bucket, Glue schema table, log group, IAM role, Firehose stream
  python fh_demo.py produce   # send 2,000 events (+5 malformed) with PutRecordBatch
  python fh_demo.py verify    # list partitions, read one Parquet file, print metrics
  python fh_demo.py cleanup   # delete everything
"""
import json
import random
import sys
import time
import uuid
from datetime import datetime, timedelta, timezone

import boto3
from botocore.exceptions import ClientError

REGION = "us-east-1"
NAME = "sl-firehose-demo"
DB, TABLE = "sl_firehose_demo", "events"
LOG_GROUP, LOG_STREAM = f"/aws/kinesisfirehose/{NAME}", "DestinationDelivery"

session = boto3.Session(region_name=REGION)
ACCOUNT = session.client("sts").get_caller_identity()["Account"]
BUCKET = f"{NAME}-{ACCOUNT}"
s3 = session.client("s3")
glue = session.client("glue")
iam = session.client("iam")
logs = session.client("logs")
fh = session.client("firehose")
cw = session.client("cloudwatch")


def setup():
    s3.create_bucket(Bucket=BUCKET)  # us-east-1: no LocationConstraint
    glue.create_database(DatabaseInput={"Name": DB})
    glue.create_table(
        DatabaseName=DB,
        TableInput={
            "Name": TABLE,
            "TableType": "EXTERNAL_TABLE",
            "StorageDescriptor": {
                "Columns": [
                    {"Name": "event_id", "Type": "string"},
                    {"Name": "event_type", "Type": "string"},
                    {"Name": "user_id", "Type": "string"},
                    {"Name": "amount", "Type": "double"},
                    {"Name": "ts", "Type": "timestamp"},
                ],
                "Location": f"s3://{BUCKET}/events/",
                "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
                "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
                "SerdeInfo": {"SerializationLibrary":
                              "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"},
            },
        },
    )
    logs.create_log_group(logGroupName=LOG_GROUP)
    logs.create_log_stream(logGroupName=LOG_GROUP, logStreamName=LOG_STREAM)

    trust = {"Version": "2012-10-17", "Statement": [{
        "Effect": "Allow", "Principal": {"Service": "firehose.amazonaws.com"},
        "Action": "sts:AssumeRole"}]}
    role_arn = iam.create_role(RoleName=NAME, AssumeRolePolicyDocument=json.dumps(trust))["Role"]["Arn"]
    policy = {"Version": "2012-10-17", "Statement": [
        {"Effect": "Allow",
         "Action": ["s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject",
                    "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject"],
         "Resource": [f"arn:aws:s3:::{BUCKET}", f"arn:aws:s3:::{BUCKET}/*"]},
        {"Effect": "Allow",
         "Action": ["glue:GetTable", "glue:GetTableVersion", "glue:GetTableVersions"],
         "Resource": [f"arn:aws:glue:{REGION}:{ACCOUNT}:catalog",
                      f"arn:aws:glue:{REGION}:{ACCOUNT}:database/{DB}",
                      f"arn:aws:glue:{REGION}:{ACCOUNT}:table/{DB}/{TABLE}"]},
        {"Effect": "Allow", "Action": ["glue:GetSchemaVersion"], "Resource": "*"},
        {"Effect": "Allow", "Action": ["logs:PutLogEvents"],
         "Resource": f"arn:aws:logs:{REGION}:{ACCOUNT}:log-group:{LOG_GROUP}:log-stream:*"},
    ]}
    iam.put_role_policy(RoleName=NAME, PolicyName="delivery", PolicyDocument=json.dumps(policy))

    s3_config = {
        "RoleARN": role_arn,
        "BucketARN": f"arn:aws:s3:::{BUCKET}",
        # Event-time partitions from the record itself, not arrival time.
        "Prefix": "events/event_type=!{partitionKeyFromQuery:event_type}/dt=!{partitionKeyFromQuery:dt}/",
        "ErrorOutputPrefix": "errors/!{firehose:error-output-type}/!{timestamp:yyyy-MM-dd}/",
        "BufferingHints": {"SizeInMBs": 64, "IntervalInSeconds": 60},
        "CompressionFormat": "UNCOMPRESSED",  # Parquet compresses internally (SNAPPY)
        "FileExtension": ".parquet",
        "CloudWatchLoggingOptions": {"Enabled": True, "LogGroupName": LOG_GROUP,
                                     "LogStreamName": LOG_STREAM},
        "DynamicPartitioningConfiguration": {"Enabled": True,
                                             "RetryOptions": {"DurationInSeconds": 300}},
        "ProcessingConfiguration": {"Enabled": True, "Processors": [{
            "Type": "MetadataExtraction",
            "Parameters": [
                {"ParameterName": "MetadataExtractionQuery",
                 "ParameterValue": '{event_type: .event_type, dt: (.ts | strftime("%Y-%m-%d"))}'},
                {"ParameterName": "JsonParsingEngine", "ParameterValue": "JQ-1.6"},
            ]}]},
        "DataFormatConversionConfiguration": {
            "Enabled": True,
            "InputFormatConfiguration": {"Deserializer": {"OpenXJsonSerDe": {}}},
            "OutputFormatConfiguration": {"Serializer": {"ParquetSerDe": {"Compression": "SNAPPY"}}},
            "SchemaConfiguration": {"RoleARN": role_arn, "DatabaseName": DB,
                                    "TableName": TABLE, "Region": REGION, "VersionId": "LATEST"},
        },
    }
    for attempt in range(10):  # new IAM roles take a few seconds to become assumable
        try:
            fh.create_delivery_stream(DeliveryStreamName=NAME,
                                      DeliveryStreamType="DirectPut",
                                      ExtendedS3DestinationConfiguration=s3_config)
            break
        except ClientError as e:
            if e.response["Error"]["Code"] != "InvalidArgumentException" or attempt == 9:
                raise
            print("role not assumable yet, retrying:", e.response["Error"]["Message"][:80])
            time.sleep(6)
    while True:
        status = fh.describe_delivery_stream(DeliveryStreamName=NAME)[
            "DeliveryStreamDescription"]["DeliveryStreamStatus"]
        print("stream status:", status)
        if status == "ACTIVE":
            return
        if status.endswith("FAILED"):
            sys.exit("stream creation failed, see DescribeDeliveryStream FailureDescription")
        time.sleep(10)


def make_event(now):
    ts = now - timedelta(days=random.choice([0, 1]), seconds=random.randint(0, 3600))
    return {"event_id": str(uuid.uuid4()),
            "event_type": random.choice(["page_view", "add_to_cart", "checkout", "refund"]),
            "user_id": f"u{random.randint(1, 300)}",
            "amount": round(random.uniform(1, 200), 2),
            "ts": int(ts.timestamp())}


def put_batch(records):
    """PutRecordBatch is not all-or-nothing: resend only the failed entries."""
    for attempt in range(6):
        resp = fh.put_record_batch(DeliveryStreamName=NAME, Records=records)
        if resp["FailedPutCount"] == 0:
            return
        records = [r for r, rr in zip(records, resp["RequestResponses"]) if "ErrorCode" in rr]
        print(f"retrying {len(records)} failed records")
        time.sleep(0.2 * 2 ** attempt)
    raise RuntimeError(f"{len(records)} records still failing")


def produce():
    now = datetime.now(timezone.utc)
    events = [make_event(now) for _ in range(2000)]
    payload = [{"Data": (json.dumps(e) + "\n").encode()} for e in events]
    payload += [{"Data": b"this is not json\n"} for _ in range(5)]
    random.shuffle(payload)
    for i in range(0, len(payload), 500):  # hard cap: 500 records or 4 MiB per call
        put_batch(payload[i:i + 500])
    print(f"sent {len(payload)} records; first objects appear after the 60s buffer flushes")


def verify():
    import io
    import pyarrow.parquet as pq
    keys = []
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET):
        keys += [o["Key"] for o in page.get("Contents", [])]
    data = sorted(k for k in keys if k.startswith("events/"))
    errors = sorted(k for k in keys if k.startswith("errors/"))
    print(f"{len(data)} data objects, {len(errors)} error objects")
    for k in data:
        print(" ", k)
    for k in errors:
        print(" ", k)
    if data:
        body = s3.get_object(Bucket=BUCKET, Key=data[0])["Body"].read()
        table = pq.read_table(io.BytesIO(body))
        print(table.schema)
        print(table.num_rows, "rows in", data[0])
    rows = sum(pq.read_metadata(io.BytesIO(
        s3.get_object(Bucket=BUCKET, Key=k)["Body"].read())).num_rows for k in data)
    print("total rows in Parquet:", rows)
    end = datetime.now(timezone.utc)
    for metric in ["IncomingRecords", "DeliveryToS3.Records", "SucceedConversion.Records",
                   "FailedConversion.Records", "PartitionCount", "DeliveryToS3.DataFreshness"]:
        pts = cw.get_metric_statistics(
            Namespace="AWS/Firehose", MetricName=metric,
            Dimensions=[{"Name": "DeliveryStreamName", "Value": NAME}],
            StartTime=end - timedelta(hours=1), EndTime=end, Period=3600,
            Statistics=["Sum", "Maximum"])["Datapoints"]
        print(f"{metric:28s}", pts[0] if pts else "no datapoints yet")


def cleanup():
    def quiet(fn, **kw):
        try:
            fn(**kw)
        except ClientError as e:
            print("skip:", e.response["Error"]["Code"])
    quiet(fh.delete_delivery_stream, DeliveryStreamName=NAME)
    while True:
        try:
            fh.describe_delivery_stream(DeliveryStreamName=NAME)
            time.sleep(5)
        except fh.exceptions.ResourceNotFoundException:
            break
    paginator = s3.get_paginator("list_objects_v2")
    for page in paginator.paginate(Bucket=BUCKET):
        objs = [{"Key": o["Key"]} for o in page.get("Contents", [])]
        if objs:
            s3.delete_objects(Bucket=BUCKET, Delete={"Objects": objs})
    quiet(s3.delete_bucket, Bucket=BUCKET)
    quiet(glue.delete_table, DatabaseName=DB, Name=TABLE)
    quiet(glue.delete_database, Name=DB)
    quiet(iam.delete_role_policy, RoleName=NAME, PolicyName="delivery")
    quiet(iam.delete_role, RoleName=NAME)
    quiet(logs.delete_log_group, logGroupName=LOG_GROUP)
    print("cleanup done")


if __name__ == "__main__":
    {"setup": setup, "produce": produce, "verify": verify, "cleanup": cleanup}[sys.argv[1]]()

Run it in order:

python fh_demo.py setup     # about 1-2 minutes, stream goes CREATING -> ACTIVE
python fh_demo.py produce
sleep 150                   # 60 s buffer hint, multi-stage buffering, S3 write
python fh_demo.py verify

Three choices in setup() are the point of the exercise. The jq query returns an object, and each key in it becomes a name you can reference as !{partitionKeyFromQuery:<key>} in the prefix. The dt key comes from strftime over the record's ts, so an event from yesterday that arrives today lands in yesterday's partition; the !{timestamp:...} namespace would have filed it under today. And the buffer is 64 MiB or 60 seconds: at this volume the interval always wins, so each of the 8 partitions (4 event types times 2 days) flushes its own object roughly once a minute.

Verification. verify lists the bucket, opens one Parquet object with pyarrow, counts the rows across all data objects, and prints six CloudWatch metrics for the stream. You should see data objects only under the 8 events/event_type=.../dt=.../ prefixes, a Parquet schema with event_id, event_type, user_id, amount and a timestamp ts, a total of 2,000 rows (more only if a retry produced a duplicate), and at least one object under errors/ holding the five records that were not JSON. PartitionCount should report 8 or fewer, and DeliveryToS3.DataFreshness tells you how old the oldest undelivered record got, which is the number to alarm on in production. Metrics can take a few minutes to show up; rerun verify if they read "no datapoints yet".

If you want to query it, create an Athena table over s3://<bucket>/events/ with event_type and dt as partition columns (partition projection avoids MSCK REPAIR), and a filter on dt will read only the matching prefixes.

When it breaks

InvalidArgumentException on create, mentioning the role. IAM is eventually consistent and Firehose tries to assume a role you created seconds ago. The script retries for a minute. If it never succeeds, the trust policy principal is wrong or your user lacks iam:PassRole.

Everything lands in errors/. Open one error object; each line carries the error code and message along with the raw record. format-conversion-failed means the record does not match the Glue table (a string where the column says double, an ISO timestamp the SerDe does not parse, a JSON array instead of objects). Partitioning failures mean the jq query errored or returned null for a key, usually a missing field or strftime on a string instead of a number.

activePartitionExceeded errors and PartitionCountExceeded = 1. Your key cardinality per buffer interval is above 500. Partition on something coarser (date and type, not user), lower the interval so partitions close faster, or request the raise to 2,500. Do not use high-cardinality IDs as partition keys; filter on them inside the files instead.

Thousands of tiny objects. DeliveryToS3.ObjectCount climbing with small DeliveryToS3.Bytes per object means your buffers flush on time, not size. Raise the interval, or reduce the number of partitions. Every object costs a Firehose object fee, an S3 PUT, and a file open in every query engine.

ServiceUnavailableException or a non-zero FailedPutCount. You hit the Direct PUT throughput quota. Back off and resend only the failed entries, as put_batch() does, watch ThrottledRecords, and let the automatic increase catch up or request one. Packing small events into fewer records also cuts the records-per-second count.

"I want to add partitioning to my existing stream." You cannot. Create a new stream with partitioning enabled, point producers at it, and delete the old one after its buffer drains.

Duplicate rows. Retries inside Firehose and in your producer can both deliver a record twice. Deduplicate on event_id in the query or in a compaction job.

Cleanup

python fh_demo.py cleanup

It deletes the Firehose stream first (so nothing writes into the bucket while you empty it) and waits until DescribeDeliveryStream returns ResourceNotFoundException, then deletes every object and the bucket, the Glue table and database, the IAM role and its inline policy, and the log group. Confirm with aws firehose list-delivery-streams, aws s3 ls | grep sl-firehose-demo and aws glue get-databases --query "DatabaseList[].Name". There is no idle charge for a Firehose stream, but there is no reason to keep one around either. If you created an Athena table, drop it too.

For the stream that usually sits upstream of a Firehose, see the previous entry in this series, SL#172 on Kinesis Data Streams, and for Iceberg on S3 as a destination, SL#117 on S3 Tables.

Sources