Twenty-six episodes of this series leaned on Amazon Bedrock, and for good reason. Bedrock is serverless, you pay per token, and you never think about a GPU. But Bedrock only serves the models AWS chose to host, on the terms AWS set. The moment you need a specific open-weights checkpoint, a fine-tune you trained yourself, full control of the inference container, or a model in a Region Bedrock has not reached, you are off the managed-token path and onto managed-infrastructure. That is what SageMaker AI real-time inference is for.

In this tutorial we deploy an open-weights model (Llama 3.1 8B Instruct, though any JumpStart text-generation model works the same way) to a real-time SageMaker endpoint using the JumpStartModel class, which wires up the right container, model artifacts, and instance type for you. Then we attach an Application Auto Scaling target-tracking policy so the endpoint adds instances under load and removes them when traffic drops, instead of paying flat rate for a peak you hit twice a day. At the end you get a running HTTPS endpoint you can call from boto3, a scaling policy you can watch react, and a decision table that tells you when this architecture is the right one and when Bedrock still wins.

The non-obvious design choice: we do not hand-build a container or wrangle model weights. JumpStart resolves all of that from a model ID, and the same three lines that deploy an 8B model deploy a 120B one. The engineering that matters here is not the deploy call, it is everything around it: the instance sizing, the scaling policy, and the cost math.

You need an AWS account and an IAM principal that can create SageMaker endpoints and register Application Auto Scaling targets. If you run this from inside SageMaker Studio, the Studio execution role usually already has what you need. If you run it from your laptop, your role needs sagemaker:CreateModel, sagemaker:CreateEndpoint, sagemaker:CreateEndpointConfig, sagemaker:InvokeEndpoint, plus application-autoscaling:* on the SageMaker resource and iam:PassRole for the SageMaker execution role.

You need a service quota for a GPU instance. Llama 3.1 8B Instruct runs comfortably on ml.g5.2xlarge or ml.g6.xlarge. Brand-new accounts often have a zero quota for ml.g5 and ml.g6 endpoint usage, and the deploy will fail with ResourceLimitExceeded until you request an increase in the Service Quotas console under Amazon SageMaker. Request it before you start, because the increase can take a few hours to a day.

Assumed knowledge: comfortable with Python and boto3, you have run at least one thing on AWS before, and you understand that a GPU endpoint bills per second while it exists whether or not you call it. Local tooling: Python 3.10+ and pip.

Install the SageMaker Python SDK and boto3. Pin the SDK so this tutorial does not drift when the next release changes a default.

Now confirm your identity, Region, and execution role. If you are on SageMaker Studio, get_execution_role() returns the Studio role. If you are on a laptop, replace it with the ARN of a role that SageMaker can assume (it must trust sagemaker.amazonaws.com).

That print is the smoke test. If it shows a Region and a role ARN without throwing, your credentials and SDK are wired correctly and we can deploy. If get_execution_role() throws outside Studio, that is expected, fall through to the explicit ARN.

Rather than hardcode an instance type that might be wrong for your Region, ask JumpStart what the model wants. Every JumpStart model has a default deployment instance, and you can list the supported ones.

This matters because the AWS AI stack changes monthly. A model that defaulted to ml.g5.2xlarge last quarter might default to a ml.g6 instance now as newer GPU families roll out, and Regions do not all carry the same instances. Reading the default from the SDK instead of memory is the difference between a tutorial that works today and one that fails with an instance-not-available error.

Note the model_version="*" wildcard. It resolves to the latest version JumpStart has for that model. Pin a specific version in production so a silent model update does not change your outputs, but for a first deploy the latest is fine.

Three lines. The JumpStartModel class resolves the container image, the model artifact, and the default instance, and deploy creates the model, endpoint config, and endpoint. Open-weights models from Meta carry a EULA, so you must pass accept_eula=True explicitly, the same requirement AWS documents for the GPT OSS models.

This call blocks for roughly 8 to 12 minutes while SageMaker pulls the container, downloads the weights to the instance, and passes health checks. That wait is the real cost of leaving Bedrock: with Bedrock the model is already warm and shared across every customer, so your first token comes back in milliseconds. Here you are provisioning a dedicated GPU, and provisioning takes minutes. This is the single most important trade-off in the whole episode. You trade Bedrock's zero-provisioning, pay-per-token model for a dedicated instance you control and pay for by the second.

While it deploys, understand what you are paying for. An ml.g5.2xlarge real-time inference instance runs on the order of 1.50 US dollars per hour, billed per second from the moment the endpoint reaches InService until you delete it. There is no idle discount on a plain real-time endpoint. One instance left running for a month is over 1,000 dollars whether it served one request or one million.

Once deploy returns, the endpoint is live. The returned predictor speaks the model's native schema. For Llama text-generation models on JumpStart that is an inputs string plus a parameters object.

In production you will not always have the predictor object in scope, for example when a separate Lambda calls the endpoint. Any process with the right IAM permissions can invoke it directly through the SageMaker runtime, which is the call that actually matters operationally.

That invoke_endpoint call is your production API surface. It is a signed HTTPS request to a dedicated endpoint, not a shared token API, which is exactly why you would choose this path: the model, the container, and the instance are yours.

A single instance handles a single instance worth of traffic. The whole reason to run your own endpoint instead of one flat over-provisioned box is elasticity, and on SageMaker that comes from Application Auto Scaling, not from SageMaker directly. You register the endpoint variant as a scalable target, then attach a target-tracking policy.

Then the policy. Target tracking keeps a chosen metric near a target value, adding instances when it rises and removing them when it falls. The standard predefined metric for endpoints is SageMakerVariantInvocationsPerInstance, the average invocations per minute per instance. Set the target to the throughput one instance handles at acceptable latency, which you find by load testing, and SageMaker keeps you there.

Two numbers earn their keep here. ScaleOutCooldown is short (60s) because you want to add capacity fast when a spike hits. ScaleInCooldown is longer (300s) because removing an instance mid-spike causes thrashing, and the cost of holding an extra instance for five minutes is a rounding error next to a latency incident. Note the honest limitation: SageMakerVariantInvocationsPerInstance reacts on one-minute CloudWatch metrics, so a GPU endpoint that takes minutes to boot a new instance is slow to scale out. AWS shipped higher-resolution metrics emitted at 10-second intervals for generative workloads to cut that lag, and for spiky LLM traffic they are worth adopting over the classic predefined metric.

One more lever worth knowing: SageMaker supports scaling an endpoint down to zero instances when it uses inference components, so a dev or batch endpoint can cost nothing while idle and cold-start on the next request. That is the closest this architecture gets to Bedrock's pay-for-nothing-when-idle economics, at the price of a cold start.

You now have both tools in hand. Here is how to choose, distilled from the trade-offs above.

Reach for Bedrock when the model you want is in the Bedrock catalog, your traffic is spiky or low-volume, you never want to think about a GPU, and per-token pricing with zero idle cost is what you need. Bedrock wins the default case. Most applications should start there.

Reach for SageMaker AI real-time inference when you need a specific open-weights checkpoint Bedrock does not host, a model you fine-tuned yourself, control over the serving container or inference parameters, a Region Bedrock has not reached, or predictable high-volume throughput where a reserved GPU is cheaper per token than the metered API. The break-even is real: a busy endpoint saturated most of the day can beat per-token pricing, while the same endpoint at 3% utilization is pure waste.

The trap is treating this as a two-way door when it is closer to a one-way one. Bedrock is a Tuesday-afternoon decision you can reverse by changing a model ID. A SageMaker endpoint is infrastructure with a quota request, a scaling policy, a security group, and a bill that accrues whether or not anyone calls it. Default to Bedrock, and cross to SageMaker only when a concrete requirement forces you, not because owning the GPU feels more real.

Three checks confirm the whole stack. First, the endpoint is live:

Expected output: InService. Anything else (Creating, Failed) means wait or read the failure reason in the same describe output.

Second, inference returns text. Re-run the invoke_endpoint call from Step 3 and confirm you get a JSON body with generated text, not a ModelError. Third, the scaling target is registered:

Expected output: 1 4. If you want to watch scaling actually fire, drive load at the endpoint (a loop of invoke_endpoint calls from a few threads) and poll aws application-autoscaling describe-scaling-activities --service-namespace sagemaker. Within a couple of minutes of sustained load above your target you will see a scale-out activity, and the endpoint's instance count climb in describe-endpoint.

ResourceLimitExceeded on deploy. Your account has zero or too-low quota for the instance type. Open Service Quotas, choose Amazon SageMaker, find ml.g5.2xlarge for endpoint usage (or your instance), and request an increase. This is the single most common failure and it has nothing to do with your code.

ValueError about EULA or an access-denied on the model artifact. You did not pass accept_eula=True, or the model requires accepting terms you have not. Meta and OpenAI open-weights models on JumpStart gate on the EULA. Set the flag explicitly.

Endpoint reaches Failed during creation. Usually the instance is too small for the model to load into GPU memory, or the container ran out of disk pulling weights. Check the endpoint's CloudWatch logs under /aws/sagemaker/Endpoints/sl-openweights-llm. Move up an instance size before assuming a bug.

ModelError with a 424 on invoke. Your payload does not match the model's schema. Text-generation JumpStart models expect {"inputs": "...", "parameters": {...}}, not the OpenAI chat schema. If you deployed a different model family, check its model card for the exact input format, or call sagemaker.serializers.retrieve_options(model_id=...).

Autoscaling never scales out. Either you never crossed the target (one manual request per minute will not), the register_scalable_target call failed silently on a permissions gap, or you are watching the one-minute metric lag. Confirm the target exists with describe-scalable-targets, drive real sustained load, and give it two to three minutes.

First, the cheap win: convert the endpoint to inference components and set MinCapacity=0 so it scales to zero when idle. A dev endpoint that costs nothing overnight is worth the cold start, and it is a config change, not a rewrite.

Second, deploy a second model to the same endpoint as an additional inference component. Shared-endpoint hosting lets two or three smaller models share GPU and scale independently, which is how you get SageMaker's cost story to rival Bedrock's for a fleet of models rather than one.

Third, and hardest, wire this endpoint into the Bedrock-centric code from earlier episodes behind one interface, so a single generate() call routes to Bedrock or your SageMaker endpoint by config. Once both are one function call apart, the decision table in Step 5 stops being architecture astronomy and becomes a runtime flag. That is the real endgame: not choosing Bedrock or SageMaker forever, but making the choice cheap to change per model, per Region, per bill.