This website uses cookies

Read our Privacy policy and Terms of use for more information.

What it actually is

Amazon Managed Streaming for Apache Kafka (MSK) runs real Apache Kafka for you. Not a Kafka-flavored API, not a compatibility layer: your existing producers, consumers, Kafka Connect connectors, and Kafka Streams code speak the Kafka wire protocol to brokers that AWS provisions, patches, and replaces when they fail.

The mental model the docs bury is this: Kafka is a replicated, partitioned, append-only log that consumers read at their own pace. Nothing is deleted when it is read. A message stays until retention expires, and every consumer group keeps its own bookmark (the committed offset) per partition. That single design choice is why Kafka can replay last Tuesday, feed five independent systems from one stream, and keep per-key ordering, and it is also why Kafka is the wrong tool for "send this job to one worker and forget it". A queue deletes work when it is done. A log remembers everything and makes you track your position.

MSK sells that log in three shapes. MSK Provisioned with Standard brokers is classic Kafka on instances you size, with EBS volumes you size. MSK Provisioned with Express brokers is still a cluster of brokers you choose, but storage is elastic and managed, and AWS pre-configures and guards the Kafka settings that usually cause outages. MSK Serverless has no brokers you can see at all: you create a cluster, create topics, and pay per hour, per partition, and per GB.

2026 kept moving the product. Kafka 4.2 arrived on Express brokers in July. On July 30 Express clusters gained delivery of topics straight into Apache Iceberg tables on S3 Tables. On August 10 MSK added in-place migration from ZooKeeper to KRaft for provisioned clusters, which matters because Kafka 4.x supports only KRaft and 3.9 is the last version that supports ZooKeeper. On October 30 provisioned clusters got TLS 1.3 through opt-in security policies. None of that changes the model below.

The model

Topics and partitions. A topic is split into partitions. A partition is the unit of ordering, the unit of parallelism, and on Serverless the unit of billing. Records with the same key land in the same partition, so they stay in order relative to each other. Records with different keys have no ordering guarantee at all.

Replication and durability. Each partition has a leader and followers on brokers in different Availability Zones. A producer with acks=all gets an acknowledgment only after the in-sync replicas have the record. On Provisioned clusters you own the replication factor and min.insync.replicas; on Serverless you do not get to touch broker configuration at all.

Consumer groups. Consumers that share a group.id split the partitions of a topic between them. Each partition is read by at most one member of a group at a time, so six partitions means at most six useful consumers in one group. When a member joins or leaves, the group rebalances and partitions move. Each group commits its own offsets, which is how two different applications read the same topic independently.

Who operates what. AWS runs the brokers, the metadata quorum (ZooKeeper or KRaft), patching, broker replacement, and on Express and Serverless the storage. You own topic design (partition count, keys, retention), client configuration, IAM or ACL authorization, consumer lag, and every decision about what happens to a message your consumer cannot process. Kafka has no built-in dead-letter queue. That last part is the one that turns into an incident.

The three shapes, mechanically.

Standard brokers: you pick an instance type and broker count, and you provision EBS per broker (1 GiB to 16,384 GiB). The current Kafka versions without an end-of-support date are 3.8, 3.9 (the recommended one), 4.0 and 4.1; 4.2 is Express-only for now. You manage storage growth, partition rebalancing, and capacity headroom.

Express brokers: only offered in a three-AZ layout, storage is "virtually unlimited" and pay-as-you-go, AWS quotes up to 3x more throughput per broker than Standard (500 MBps of writes on an express.m7g.16xlarge against 153.8 MBps on the equivalent Standard broker), scaling and partition movement up to 20x faster, and no maintenance windows. Supported versions are 3.6, 3.8, 3.9 and 4.2. Two gaps to know: the Kafka Streams API is not yet fully supported, and KIP-932 (Queues for Kafka) is not supported.

Serverless: IAM authentication is mandatory ("Apache Kafka access control lists (ACLs) are not supported"), the cluster lives in your VPC subnets (at least two subnets per client VPC, up to five client VPCs), and capacity scales on its own within fixed per-cluster quotas.

When to use it, when not to

You need

Use

Why not the others

One worker should process each job, then it is gone

Amazon SQS

Kafka makes you manage offsets, partitions and retries for a problem a queue solves with a visibility timeout and a DLQ

Route events by content to many AWS targets, no client code

Amazon EventBridge

MSK has no rules engine; every consumer is code you run

Ordered, replayable stream read by several independent systems

Amazon MSK

SQS deletes on read and caps retention at 14 days; EventBridge replay is not a consumer-owned offset

An existing Kafka ecosystem: Connect, Debezium, Flink, Kafka clients in five languages

Amazon MSK

Nothing else speaks the Kafka protocol

AWS-native stream without Kafka clients, sized in shards

Amazon Kinesis Data Streams

Different API and scaling model; covered in the next article

The honest version of the SQS question: if your design has exactly one consumer per message, no replay requirement, and messages under SQS's 1 MiB limit, SQS is cheaper to run and nearly impossible to misconfigure. Teams pick Kafka for "scale" and then spend a quarter building retry topics, poison-message handling and lag alerts that SQS gives them as checkboxes. Kafka earns its keep when the same event must be read by several systems at different speeds, when you need to replay history into a new consumer, or when you already live in the Kafka ecosystem. I covered the queue side in the SQS entry of this series and content-based routing in the EventBridge entry.

What it costs

All prices below are the US East (N. Virginia) figures on the MSK pricing page, except where the page's own Serverless example uses US East (Ohio). Months are 730 hours unless I quote AWS's 31-day examples.

Serverless bills five dimensions: $0.75 per cluster-hour, $0.0015 per partition-hour, $0.10 per GB written, $0.05 per GB read, and $0.10 per GB-month of storage. The surprise is the first line. An idle Serverless cluster with zero partitions and zero traffic costs $0.75 x 730 = $547.50 a month. "Serverless" here means you never size brokers, not that idle is free. Partitions add up too: 2,400 partitions (the default per-cluster maximum) is $3.60 an hour on its own, about $2,628 a month.

AWS's own Serverless example (5 topics x 20 partitions, 100 GB written and 200 GB read per day, 24-hour retention, 31 days) totals $1,299.60: $558 cluster-hours, $111.60 partition-hours, $310 data in, $310 data out, $10 storage. Notice that data in and out are nearly half the bill ($620 of $1,299.60) at a modest 100 GB a day.

Express bills the broker hour ($0.408 for express.m7g.large), storage ($0.10 per GB-month), and $0.01 per GB written. AWS's example (three express.m7g.large, 1 TB ingested and stored for 31 days) is $1,020.66. The per-GB write price is a tenth of Serverless, which is why Express wins once traffic is steady and large.

Standard bills the broker hour ($0.204 for kafka.m7g.large) and provisioned EBS storage ($0.10 per GB-month), plus optional provisioned storage throughput ($0.08 per MB/s-month) and tiered storage ($0.06 per GB-month, $0.0015 per GB retrieved). Three kafka.m7g.large brokers are about $447 a month before storage, the cheapest floor of the three, and you pay for it in operations.

Replication traffic between brokers is free; standard AWS data transfer charges apply to traffic in and out of the cluster, and cross-Region replication pays cross-Region transfer. There is no MSK-specific free tier on the pricing page.

The rule of thumb that falls out: Serverless for spiky or small workloads where the $548 floor is acceptable and you never want to think about brokers; Express for steady high throughput; Standard when you need a configuration knob Express guards, or the lowest broker-hour price and you have the people to run it.

The limits that bite

Serverless throughput is per partition, not per cluster. The cluster tops out at 200 MBps in and 400 MBps out, but each partition is capped at 5 MBps in and 10 MBps out. A topic with 3 partitions can never take more than 15 MBps of writes, no matter how "serverless" the cluster is. Partition count is your throughput decision, and violations show up as slowdowns with a throttle duration in the response, not as errors.

Serverless partition and group caps. 2,400 partitions for non-compacted topics, only 120 for compacted topics, 500 consumer groups, 3,000 client connections, 100 connection attempts per second, and topic management APIs at 2 requests per second. Creating and deleting partitions is limited to 250 every 5 minutes. Ten Serverless clusters per account by default.

Serverless message size. Default max.message.bytes is 1,048,588 bytes; you can raise it per topic to 8 MiB, and nothing larger. cleanup.policy can only be set at topic creation time, and segment.bytes cannot be changed at all. Retention defaults to 7 days and 250 GiB per partition, and both can be set to unlimited.

Express per-partition ceiling. 15 MB/s per partition, and partition counts per broker are bounded by size: an express.m7g.large is recommended at 1,000 partitions (leaders plus followers) with a hard maximum of 1,500. Its sustained ingress is 15.6 MBps with a 23.4 MBps maximum quota.

IAM connection limits on provisioned brokers. 3,000 IAM-authenticated TCP connections per broker, and a connection creation rate of 100 per second on M5 and M7g sizes but only 4 per second on t3. A fleet of Lambda functions or pods reconnecting at once can exceed the t3 rate on a single deploy. AWS's advice is to set reconnect.backoff.ms (for example 1000) on the client.

Broker counts. 90 brokers per account and 60 per KRaft cluster (30 for ZooKeeper clusters) by default.

Version clock. Kafka 3.6 reached MSK end of support on June 1, 2026 and 3.7 on September 1, 2026. If your cluster is still on ZooKeeper, the path to 4.x is: upgrade to 3.9, run the in-place KRaft migration, then upgrade.

Build it

We will build the smallest honest Kafka system: a Serverless cluster with IAM auth, a six-partition orders topic with one-day retention, a keyed producer, two consumers in the billing group that split the partitions between them, and a lag check that reads the same numbers the SumOffsetLag metric reports.

MSK Serverless is reachable only from inside the VPC, so the client runs on a small EC2 instance in the same security group. You connect to it with Session Manager, so no SSH key and no open port 22.

Prerequisites. An AWS account with a default VPC in a Region that offers MSK Serverless (the docs list 14; this build uses us-east-1), Python 3.9+ with boto3 on your laptop, and the AWS CLI with the Session Manager plugin (or use EC2 console > Connect > Session Manager instead).

IAM for the person running setup. kafka:CreateClusterV2, kafka:DescribeClusterV2, kafka:GetBootstrapBrokers, kafka:DeleteCluster, the EC2 actions for security groups and instances, iam:CreateRole, iam:PutRolePolicy, iam:AttachRolePolicy, iam:CreateInstanceProfile, iam:AddRoleToInstanceProfile, iam:PassRole on the new role, their delete counterparts, and ssm:GetParameter. If your account has never used MSK, the first cluster creation also needs iam:CreateServiceLinkedRole for MSK's service-linked role.

What it costs to follow along. The cluster-hour dominates: $0.75 per hour, plus six partitions at $0.0015 each ($0.009 per hour) and a few megabytes of data. A t4g.micro and its public IPv4 address add a little on top. Budget about $2 for a two-hour session, and run the cleanup the moment you finish, because the cluster bills by the hour whether or not anything is connected.

Step 1: the cluster, the network, and the client. setup_msk.py finds the default VPC, takes one default subnet in each of three AZs, creates a security group that allows TCP 9098 only from itself, creates the Serverless cluster, waits for ACTIVE, and fetches the IAM bootstrap string. It then creates an instance role whose data-plane policy is scoped to one topic and one group on this cluster, and launches the client with the Kafka libraries pinned.

"""setup_msk.py - MSK Serverless cluster + one EC2 client in the default VPC. Run from your laptop."""
import json, time
import boto3

REGION = "us-east-1"
NAME = "sl-msk-lab"
session = boto3.Session(region_name=REGION)
ec2, kafka, iam, ssm = (session.client(s) for s in ("ec2", "kafka", "iam", "ssm"))
account = session.client("sts").get_caller_identity()["Account"]

# 1. Network: default VPC, one subnet per AZ (Serverless needs at least two)
vpc_id = ec2.describe_vpcs(Filters=[{"Name": "isDefault", "Values": ["true"]}])["Vpcs"][0]["VpcId"]
subnets = ec2.describe_subnets(Filters=[{"Name": "vpc-id", "Values": [vpc_id]},
                                        {"Name": "default-for-az", "Values": ["true"]}])["Subnets"]
subnet_ids = [s["SubnetId"] for s in sorted(subnets, key=lambda s: s["AvailabilityZone"])][:3]
sg_id = ec2.create_security_group(GroupName=NAME, Description="MSK lab", VpcId=vpc_id)["GroupId"]
ec2.authorize_security_group_ingress(GroupId=sg_id, IpPermissions=[{
    "IpProtocol": "tcp", "FromPort": 9098, "ToPort": 9098,
    "UserIdGroupPairs": [{"GroupId": sg_id}]}])          # only members of this SG reach port 9098

# 2. The cluster: no brokers, no storage, no Kafka version to pick
arn = kafka.create_cluster_v2(ClusterName=NAME, Serverless={
    "VpcConfigs": [{"SubnetIds": subnet_ids, "SecurityGroupIds": [sg_id]}],
    "ClientAuthentication": {"Sasl": {"Iam": {"Enabled": True}}}})["ClusterArn"]
while (state := kafka.describe_cluster_v2(ClusterArn=arn)["ClusterInfo"]["State"]) != "ACTIVE":
    print("cluster", state); time.sleep(30)
bootstrap = kafka.get_bootstrap_brokers(ClusterArn=arn)["BootstrapBrokerStringSaslIam"]

# 3. Data-plane permissions for the client, scoped to this cluster
uuid = arn.split("/")[-1]
base = f"arn:aws:kafka:{REGION}:{account}"
policy = {"Version": "2012-10-17", "Statement": [
    {"Effect": "Allow", "Action": ["kafka-cluster:Connect", "kafka-cluster:DescribeCluster"],
     "Resource": arn},
    {"Effect": "Allow", "Action": ["kafka-cluster:CreateTopic", "kafka-cluster:DescribeTopic",
                                   "kafka-cluster:WriteData", "kafka-cluster:ReadData"],
     "Resource": f"{base}:topic/{NAME}/{uuid}/orders"},
    {"Effect": "Allow", "Action": ["kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup"],
     "Resource": f"{base}:group/{NAME}/{uuid}/billing"}]}
trust = {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "sts:AssumeRole",
         "Principal": {"Service": "ec2.amazonaws.com"}}]}
iam.create_role(RoleName=NAME, AssumeRolePolicyDocument=json.dumps(trust))
iam.put_role_policy(RoleName=NAME, PolicyName="msk-data-plane", PolicyDocument=json.dumps(policy))
iam.attach_role_policy(RoleName=NAME, PolicyArn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore")
iam.create_instance_profile(InstanceProfileName=NAME)
iam.add_role_to_instance_profile(InstanceProfileName=NAME, RoleName=NAME)
time.sleep(15)                                            # IAM is eventually consistent

# 4. The client machine: same VPC, same SG, Python libs preinstalled
ami = ssm.get_parameter(Name="/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64")["Parameter"]["Value"]
user_data = f"""#!/bin/bash
dnf install -y python3-pip
pip3 install kafka-python==3.0.13 aws-msk-iam-sasl-signer-python==1.0.2
echo 'export BOOTSTRAP={bootstrap} AWS_REGION={REGION}' >> /etc/profile.d/msk.sh
"""
instance_id = ec2.run_instances(
    ImageId=ami, InstanceType="t4g.micro", MinCount=1, MaxCount=1,
    SubnetId=subnet_ids[0], SecurityGroupIds=[sg_id],
    IamInstanceProfile={"Name": NAME}, UserData=user_data,
    TagSpecifications=[{"ResourceType": "instance", "Tags": [{"Key": "Name", "Value": NAME}]}],
)["Instances"][0]["InstanceId"]

json.dump({"cluster_arn": arn, "sg_id": sg_id, "instance_id": instance_id, "bootstrap": bootstrap},
          open("msk_lab.json", "w"), indent=2)
print("bootstrap:", bootstrap)
print(f"connect: aws ssm start-session --target {instance_id} --region {REGION}")

Two details worth reading twice. The topic and group ARNs include the cluster UUID (topic/<cluster-name>/<cluster-uuid>/<topic>), so this role cannot touch any other cluster that happens to share the name. And WriteData requires DescribeTopic, while ReadData requires both DescribeTopic and AlterGroup; the MSK action table lists those dependencies, and missing one is the most common "authorization failed" you will see.

Creation takes a few minutes. Give the instance another two minutes after launch for the user data to finish installing packages.

Step 2: the client code. Open a session on the instance, start a login shell so BOOTSTRAP and AWS_REGION are set (bash -l), and save this as lab.py.

"""lab.py - topic, producer, consumer group and lag check for MSK Serverless (IAM auth)."""
import json, os, socket, sys, time
from kafka import KafkaAdminClient, KafkaConsumer, KafkaProducer, TopicPartition
from kafka.admin import NewTopic
from kafka.errors import TopicAlreadyExistsError
from kafka.net.sasl.oauth import AbstractTokenProvider
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

BOOTSTRAP = os.environ["BOOTSTRAP"]          # host:9098,host:9098 from get_bootstrap_brokers
REGION = os.environ.get("AWS_REGION", "us-east-1")
TOPIC, GROUP = "orders", "billing"


class MSKTokenProvider(AbstractTokenProvider):  # must subclass, see "When it breaks"
    def token(self):
        token, _expiry_ms = MSKAuthTokenProvider.generate_auth_token(REGION)
        return token


def conn():
    if os.environ.get("LOCAL"):                  # plain local broker for testing
        return {"bootstrap_servers": BOOTSTRAP}
    return {
        "bootstrap_servers": BOOTSTRAP,
        "security_protocol": "SASL_SSL",
        "sasl_mechanism": "OAUTHBEARER",
        "sasl_oauth_token_provider": MSKTokenProvider(),
        "client_id": socket.gethostname(),
    }


def create_topic():
    admin = KafkaAdminClient(**conn())
    try:
        admin.create_topics([NewTopic(TOPIC, num_partitions=6, replication_factor=-1,
                                      topic_configs={"retention.ms": str(24 * 3600 * 1000)})])
        print(f"created {TOPIC} with 6 partitions")
    except TopicAlreadyExistsError:
        print(f"{TOPIC} already exists")
    admin.close()


def produce(n):
    p = KafkaProducer(**conn(), acks="all", linger_ms=20,
                      key_serializer=str.encode,
                      value_serializer=lambda v: json.dumps(v).encode())
    for i in range(n):
        customer = f"customer-{i % 12}"             # same key -> same partition -> ordered
        p.send(TOPIC, key=customer, value={"order": i, "customer": customer, "ts": time.time()})
    p.flush()
    p.close()
    print(f"produced {n} records")


def consume(name, slow=0.0):
    c = KafkaConsumer(TOPIC, **conn(), group_id=GROUP, client_id=name,
                      auto_offset_reset="earliest", enable_auto_commit=False,
                      value_deserializer=lambda b: json.loads(b))
    seen = 0
    try:
        while True:
            batch = c.poll(timeout_ms=1000, max_records=100)
            for tp, records in batch.items():
                for r in records:
                    seen += 1
                    time.sleep(slow)
                print(f"[{name}] partition {tp.partition}: {len(records)} records, last offset {records[-1].offset}")
            if batch:
                c.commit()                          # commit only after the batch is processed
            mine = sorted(tp.partition for tp in c.assignment())
            print(f"[{name}] assigned {mine}, processed {seen}", end="\r")
    except KeyboardInterrupt:
        c.close()


def lag():
    # a consumer with the group id but no subscribe() reads offsets without joining the group
    probe = KafkaConsumer(**conn(), group_id=GROUP, enable_auto_commit=False)
    parts = [TopicPartition(TOPIC, p) for p in sorted(probe.partitions_for_topic(TOPIC))]
    end = probe.end_offsets(parts)
    total = 0
    for tp in parts:
        done = probe.committed(tp) or 0
        total += end[tp] - done
        print(f"partition {tp.partition}: end {end[tp]:>6} committed {done:>6} lag {end[tp] - done}")
    print(f"total lag for group {GROUP}: {total}")
    probe.close()


if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "topic": create_topic()
    elif cmd == "produce": produce(int(sys.argv[2]))
    elif cmd == "consume": consume(sys.argv[2], float(sys.argv[3]) if len(sys.argv) > 3 else 0.0)
    elif cmd == "lag": lag()

The auth piece is small: the AWS signer builds a SigV4-presigned token from the instance role's credentials locally, without calling any AWS endpoint, and Kafka's standard OAUTHBEARER mechanism carries it to the broker on port 9098. The consumer turns off auto-commit and commits after each processed batch, which gives you at-least-once processing: if it crashes mid-batch, the batch is read again. Your handler must be idempotent, exactly like the SQS consumer in the earlier article.

Step 3: run it.

python3 lab.py topic          # created orders with 6 partitions
python3 lab.py produce 5000   # produced 5000 records
python3 lab.py lag            # lag 5000 in total: nobody has consumed yet

Now open a second Session Manager session (also bash -l) so you have two terminals. Start the first consumer with a deliberate 5 ms delay per record so the group has time to show you what it does:

python3 lab.py consume c1 0.005    # terminal 1: assigned [0, 1, 2, 3, 4, 5]
python3 lab.py consume c2 0.005    # terminal 2: the group rebalances

When c2 joins, watch c1's assignment shrink from all six partitions to three while c2 picks up the other three. That is the consumer group protocol doing its one job. Stop c2 with Ctrl+C and c1 takes all six back after the rebalance. Start a third and fourth consumer if you like; start a seventh and it sits idle with an empty assignment, because six partitions means six useful consumers, never more.

Verification. With both consumers stopped, produce another batch and check lag from a third terminal:

python3 lab.py produce 2000
python3 lab.py lag            # every partition shows lag again, about 2000 in total
python3 lab.py consume c1     # drains it (no delay), Ctrl+C when it goes quiet
python3 lab.py lag            # total lag for group billing: 0

Every customer key maps to one partition, so customer-7's orders were processed in order even while the group rebalanced. In CloudWatch (namespace AWS/Kafka), the same picture shows up as SumOffsetLag, MaxOffsetLag and EstimatedMaxTimeLag with the dimensions Cluster Name, Consumer Group and Topic. Those metrics appear only after a consumer group has consumed from the topic, and the docs recommend the SUM statistic for a topic-level view. EstimatedMaxTimeLag is the one to alarm on: it converts offsets into "how many seconds behind", which is the number your on-call actually cares about.

When it breaks

KafkaConfigurationError: sasl_oauth_token_provider must implement kafka.net.sasl.oauth.AbstractTokenProvider. The MSK developer guide and the signer's README show a plain class with a token() method. kafka-python 3.x rejects anything that is not a subclass of its AbstractTokenProvider, and the import path moved to kafka.net.sasl.oauth. Subclass it, as lab.py does. I checked this against kafka-python 3.0.13, the current release.

The client hangs, then times out connecting to port 9098. A network problem, not an auth problem. The client is not in the cluster's VPC, or its security group is not allowed on 9098. Serverless has no public endpoint.

TopicAuthorizationFailedError or GroupAuthorizationFailedError. A missing kafka-cluster: action or a resource ARN without the cluster UUID. Remember the hidden dependencies: writing needs DescribeTopic, reading needs DescribeTopic and AlterGroup.

INVALID_REQUEST on produce. A record above the topic's max.message.bytes (1 MiB by default, 8 MiB at most on Serverless), or a fetch asking for more than 55 MB per request.

THROUGHPUT_QUOTA_EXCEEDED when creating topics. More than 250 partition creations or deletions in 5 minutes. Infrastructure-as-code that recreates topics on every deploy hits this first.

Producers slow down for no visible reason. You are being throttled at 5 MBps per partition on Serverless (or 15 MB/s per partition on Express). The fix is more partitions and a key with enough cardinality to spread across them, not a bigger client.

Lag grows on one partition only. A hot key. One customer, tenant or device produces far more than the rest, and because ordering is per key, its traffic cannot be split across consumers. Adding consumers will not help; changing the key will.

Constant rebalancing. A consumer that takes longer than max.poll.interval.ms between polls is evicted from the group, which triggers a rebalance, which delays everyone else. Lower max_records or move slow work off the poll thread.

Cleanup

From your laptop, in the folder that holds msk_lab.json:

"""teardown_msk.py - delete everything setup_msk.py created."""
import json, time
import boto3

REGION, NAME = "us-east-1", "sl-msk-lab"
s = json.load(open("msk_lab.json"))
session = boto3.Session(region_name=REGION)
ec2, kafka, iam = (session.client(x) for x in ("ec2", "kafka", "iam"))

ec2.terminate_instances(InstanceIds=[s["instance_id"]])
kafka.delete_cluster(ClusterArn=s["cluster_arn"])
ec2.get_waiter("instance_terminated").wait(InstanceIds=[s["instance_id"]])

iam.remove_role_from_instance_profile(InstanceProfileName=NAME, RoleName=NAME)
iam.delete_instance_profile(InstanceProfileName=NAME)
iam.delete_role_policy(RoleName=NAME, PolicyName="msk-data-plane")
iam.detach_role_policy(RoleName=NAME, PolicyArn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore")
iam.delete_role(RoleName=NAME)

while True:                                   # the cluster's ENIs hold the SG until it is gone
    try:
        kafka.describe_cluster_v2(ClusterArn=s["cluster_arn"]); time.sleep(30)
    except kafka.exceptions.NotFoundException:
        break
for attempt in range(10):
    try:
        ec2.delete_security_group(GroupId=s["sg_id"]); break
    except ec2.exceptions.ClientError as e:
        print("security group still in use, retrying:", e.response["Error"]["Code"]); time.sleep(30)
print("deleted; check: aws kafka list-clusters-v2 --region", REGION)

Confirm aws kafka list-clusters-v2 --region us-east-1 no longer lists sl-msk-lab, and check Cost Explorer the next day filtered on the Amazon MSK service. Deleting the cluster deletes the topic and its data; there is nothing else to empty.

Sources