MLA 014 Hosting and Deploying ML in 2026: Managed APIs, Serverless GPUs, Spot, and Self-Hosting

Loading...

One thing to know up front. Gnothi made this episode. It's a tool I built: name a topic, and it researches the subject, writes chapters, and narrates them.

This is the infrastructure episode. It stands on its own, so you don't need to have heard anything else first, and it answers one question: you have a product, the product needs a model, where does the model run. Everything in this episode hangs off that. We're talking about hosting and deploying, the boring part that decides your margin.

Let me set the frame with the one decision that sorts everything else. Are you calling a model, or are you running one. Calling means someone else owns the GPU and you pay per token or per request. Running means a process you own loads weights into memory and answers requests, and you pay for the box, whether it's busy or not. Almost every other choice, which cloud, which server software, whether you need a queue, is downstream of that. So I'll spend the first stretch on that fork, then walk the actual menu of places to run things, then the software you'd run, then the operational stuff, queues, observability, evals, and a minute on infrastructure as code at the end.

So, calling versus running. The default in twenty twenty six, for a builder shipping a product, is to call a managed API, and the reason is arithmetic. When you call an API, your cost is proportional to usage and starts at zero. When you run a model, your cost is a floor: a GPU sitting there idle costs the same as a GPU sitting there busy, and idle is what most product traffic looks like most of the day. Until you have enough sustained volume to keep a box busy, you're paying for air.

There are four things that flip that default, and I want to name each one so you can check your own situation against them. The first is volume. At some sustained request rate, the per-token bill from an API crosses the fixed cost of a machine you keep warm, and it becomes cheaper to own the throughput. Where that line sits moves every quarter, so I'm not going to quote a number, but the shape of the comparison hasn't changed since Andreessen Horowitz wrote up the cost of AI compute back in twenty twenty three: renting wins for almost everyone because it avoids the capital and keeps you flexible while models and hardware churn underneath you, and owning only pays at large steady spend or when you need hardware nobody will rent you.

The second flip is privacy and data residency. If your contracts say customer data doesn't leave your account, or doesn't leave a region, then a hosted API from a model lab is a conversation with legal, and that conversation sometimes ends with you running open weights inside your own VPC. Note that the big clouds have built a middle path here, and I'll come back to it: models hosted by the cloud, in your region, under your IAM, with no training on your data. That satisfies a lot of the residency asks without you owning a GPU.

The third flip is latency shape. If you need the first token in tens of milliseconds, consistently, for a real-time interaction, a shared public API with rate limits and noisy neighbors may not give you that, and a dedicated deployment might. Conversely, if you can wait an hour, batch pricing is half off everywhere, and I'll get to that too.

The fourth is model availability. Sometimes the model you need is only open weights. A fine-tuned small model for your specific classification task, a text-to-speech model, an embedding model with the exact behavior you want, a vision model nobody hosts. Then the question isn't API versus self-host, it's just where do I self-host.

Here's what running open weights looks like now, because the options changed. Bedrock, Amazon's managed model service, went from an Anthropic-and-Amazon shop to a large multi-vendor catalog. As of this recording it lists Meta's Llama, Mistral's family, DeepSeek, Qwen, and OpenAI's open-weight g p t o s s models, alongside the closed frontier models. And it has a feature called Custom Model Import, where you upload safetensors weights to S3 for a supported architecture, Llama, Mistral, Qwen and a few others, and Bedrock serves them on demand with per-token billing and no instance to manage. That's the middle path I mentioned: open weights, your account, no GPU on your books. It has limits, on-demand only, no batch, size caps on the weights, but for the case of "I fine-tuned a Llama and I want it behind an endpoint that scales to zero," it's the least operational option on AWS.

So my decision procedure. Start on an API. Bedrock or Vertex or Azure if you're already in a cloud and want the IAM and billing consolidation, the lab's own API if you want new models the day they ship. Move a workload off the API when one of the four flips applies, volume, residency, latency, or availability, and not before. And when you do move it, move that one workload, not your whole stack.

That's the fork. Now the menu of where things actually run, starting with AWS because it's what I run and it's the most common answer for a serverless web builder.

AWS native has a clean split you should internalize: Lambda and Fargate have no GPUs. Still true as of this recording. Lambda's limits page has no accelerator dimension at all, and the Fargate FAQ still says use EC2 for GPU workloads. So on the serverless compute you already have, everything is CPU inference. That's not nothing. Lambda gives you up to ten gigabytes of memory and a fifteen-minute timeout, container images up to ten gigs, and SnapStart now covers Python, which matters because model loading is the cold start. Classic models, scikit-learn, XGBoost, ONNX-exported transformers for embeddings or classification, small quantized language models, those run fine on Lambda and Fargate. Fargate is what you reach for when the job runs longer than fifteen minutes or needs more memory, and Fargate Spot takes a large chunk off that price if the job can tolerate interruption.

Above that sit the GPU options, and there are three on AWS worth knowing. SageMaker AI endpoints are the managed one. Real-time endpoints run your container on an instance you choose, and since late twenty twenty four they can scale down to zero, but only when you deploy through what SageMaker calls inference components, which is their unit for packing a model and its compute onto an endpoint. Asynchronous inference is the one I'd point most builders at: you put the payload in S3, call the endpoint with a pointer, get back a job id, and SageMaker writes the result to S3 and pings SNS when it's done. Payloads up to a gigabyte, processing up to an hour, and it scales to zero on its own. That is exactly the shape of most product ML work, a user submits something, a result appears later. There's also a serverless inference tier that scales to zero per request, but it's CPU only, so mentally file it next to Lambda.

The second AWS GPU option is AWS Batch, and this is the one I use. Batch is a job scheduler with no surcharge; you pay for the EC2 underneath. You define a compute environment with the instance families you'll accept, a queue, and a job definition pointing at a container, and Batch launches instances when jobs arrive and scales them to zero when the queue empties. It supports NVIDIA GPU instances, and it supports Spot, and the combination is the cheapest way to run a GPU on AWS when the work is bursty. My own text-to-speech pipeline runs this way, though on CPU Spot rather than GPU, and I'll explain why in the quantization section.

The third is plain EC2, or ECS on EC2, or EKS with a GPU node group, which is "rent the box and run whatever you want." That's where you go when you have sustained load and want to run vLLM or SGLang yourself, and I'll treat that under bare cloud GPUs in a minute.

One more AWS piece, and the one most people skip: the vector database you already have. Aurora PostgreSQL ships pgvector, current at zero point eight point two, with HNSW indexes, and Aurora Serverless v2 can now pause to zero compute when idle and resume in about fifteen seconds. If you have a Postgres, you have a vector store, and Bedrock Knowledge Bases will even use that Aurora as its backing store for managed retrieval. I want to say this loudly because the reflex to add a dedicated vector database is strong and it's usually a second system you don't need yet. Aurora DSQL, Amazon's newer distributed Postgres, is a different animal, no extensions, no pgvector, so don't reach for it for this.

So that's AWS: CPU-only serverless, SageMaker if you want a managed endpoint, Batch on Spot if you want cheap bursty GPU, EC2 if you want a box. Now the platforms that exist because AWS has no serverless GPU.

The category is called serverless GPU, and the definition is: you give the platform a container or a Python function, it runs it on a GPU when a request arrives, bills you per second of GPU time, and scales to zero when nothing's happening. The two things that separate the players are cold start, how long between a request arriving on a cold endpoint and the model answering, and how they handle keeping things warm.

Modal is the one I'd point a Python developer at first. You decorate a function, tell it which GPU, and deploy; the platform builds the image, and containers boot in about a second according to their docs, with memory snapshots to restore a warmed process instead of reloading weights, and knobs for minimum warm containers and a scale-down window. Billing is per second. The GPU menu runs from T4 and L4 up through H100, H200, and Blackwell B200 and B300. It's very much a Python-native experience, the batch primitive is just mapping a function over inputs, and it fits the "I have a model and a script, make it an endpoint" case without you ever touching Kubernetes.

RunPod is the raw-GPU-marketplace version. Its Serverless product gives you queue-based or load-balanced endpoints that scale workers from zero, with a FlashBoot feature for faster cold starts, an async run endpoint you poll, a synchronous one, webhooks on completion, and per-second billing. The GPU list is wide, from sixteen-gigabyte economy cards through consumer RTX cards up to H100, H200, and B200 class. It tends to be the price-shopper's answer, and it's also a plain pods product if you just want a box by the hour.

Google Cloud Run GPU is the interesting one for people who like managed serverless containers. It's generally available, no driver installation, and as of this recording offers NVIDIA L4 with twenty-four gigabytes, and the RTX PRO 6000 Blackwell with ninety-six gigabytes. It scales to zero, but a GPU service has to use instance-based billing, so you're paying for the instance lifetime per GPU-second, not per request. If you already deploy web services to Cloud Run, adding a GPU service beside them is the least new-tooling path in this whole category.

Baseten positions as a production inference platform rather than a function runner. Their open-source packaging tool Truss wraps a model from any framework, and the platform gives you dedicated deployments for custom or fine-tuned models, pre-optimized model APIs, and the option to run their stack on your own cloud capacity. It's the "we have an ML team and we want someone else to operate the serving layer" tier.

Replicate is the model-catalog-as-API play, tens of thousands of public models you call by name, plus fine-tuning. And a status change you should know: Replicate joined Cloudflare in November twenty twenty five. That connects it to Cloudflare's Workers AI, which remains a curated catalog of open models you can call from the edge, but not a place to upload your own model without going through their custom requirements process. If you mainly want to call popular open models, both are good; if you want to run your own weights, neither is the answer.

fal is a generative media inference platform first, over a thousand image, video, audio and 3D models billed per output, with serverless GPUs for custom inference and dedicated clusters on the side. It comes up in the media pipeline episode; here I'll just say that if your workload is diffusion models, fal is optimized for exactly that and the generic platforms are not.

Two quick corrections to the map as of this recording. Lambda, the GPU cloud formerly Lambda Labs, is winding down its per-token inference API; their site says so and points people at GPU instances instead, so treat Lambda as a bare GPU provider, not an API. And Fly dot io's GPU Machines appear to be gone; their GPU docs pages return not found and the pricing page has no GPU line, though I couldn't find a formal sunset announcement, so I'll say "appears withdrawn" and leave it there.

So which one. If you write Python and want an endpoint this afternoon, Modal. If you want the cheapest GPU-second and don't mind a rougher edge, RunPod. If you're a Google Cloud shop, Cloud Run GPU. If you have a fine-tuned model and want an operated production deployment, Baseten. If you just want to call an open model by name, Replicate or Workers AI. If it's images and video, fal.

That's the serverless tier. Now bare GPUs, and the Spot side of the cost story.

Bare cloud GPUs means you rent an instance with a GPU attached and run your own server on it. The hyperscalers all sell current silicon; AWS has the P6 family on Blackwell, Google has A4 and A3 families on Blackwell and Hopper, Azure has ND and NC families up through GB300. And there's a tier the industry calls neoclouds, CoreWeave, Nebius, Lambda, Crusoe, which sell GPUs as the product rather than as one line in a catalog, with Kubernetes-native scheduling and cluster-scale networking. Vast dot ai is a marketplace on top of all of that, where prices float with supply and demand and you can rent an interruptible card cheaply. When does a bare box beat everything above it? When you have sustained load. If the GPU is busy most of the hour, per-second serverless billing is a premium for elasticity you aren't using, and a reserved instance, or a neocloud commitment at a discount, is cheaper per token by a wide margin.

Now Spot. AWS's headline is "save up to ninety percent versus on-demand," and that number is real for some instance types at some hours, and it's easy to misread. The mechanism is this. Spot is spare capacity. You bid to use it, AWS gives you a two-minute interruption notice when it wants the capacity back, and your instance disappears. In exchange, you pay a fraction of on-demand. The ninety percent is a ceiling, not a rate, and on my own Graviton Batch jobs the bill shows sixty to seventy percent off on-demand, which is still a large saving.

The price of the discount is that your work has to survive being killed. That means three things you have to build. First, checkpointing: your job saves progress to S3 or a database often enough that a two-minute warning doesn't cost you the hour before it. Second, retry: in AWS Batch you set the job definition's retry strategy to resubmit when the exit reason says the host was terminated, so a reclaimed instance turns into a re-queued job and nobody pages you. Third, pool diversity: you tell Batch you'll accept several instance families and sizes, and use the allocation strategy AWS recommends, spot price capacity optimized, which picks the pools least likely to be reclaimed at a good price. The more pools you accept, the less often you're interrupted. If your work can't be checkpointed and can't be retried, a real-time chat endpoint for instance, Spot is the wrong tool and you should stop feeling bad about paying on-demand for that one.

Let me put the cost shapes side by side. A managed API is linear in usage with zero floor. Batch mode on a managed API, and I'll detail it later, is the same line at half the slope with a delay. Serverless GPU is linear in seconds of GPU time, with a cold-start tax, and a floor of zero. A reserved box is a flat line regardless of usage. Spot is a flat line at a fraction of the height, with a chance it gets cut and you have to pick up where you left off. Draw your workload's traffic over a day and lay those lines on it, and the answer usually falls out.

Alright. That's where things run. Now the software you'd actually run on a GPU, because "self-host an LLM" in twenty twenty six means picking one of about five servers.

vLLM is the default. It came out of Berkeley, it's now hosted under the PyTorch Foundation, and its two foundational ideas are paged attention, which manages the key-value cache in pages like virtual memory so you don't waste GPU memory on padding, and continuous batching, which slots new requests into a running batch instead of waiting for the batch to finish. Its V1 engine rewrite made prefix caching essentially free. It exposes an OpenAI-compatible server, and now an Anthropic-compatible one too, which means your app code doesn't change when you move from a hosted API to your own box. It runs on NVIDIA and AMD GPUs, TPUs, Neuron, and CPUs.

SGLang is the other serious contender, hosted by LMSYS, the Chatbot Arena people. Its signature idea is RadixAttention, a radix tree over the key-value cache so that requests sharing a prefix, the same system prompt say, reuse the cached computation automatically. It cites production use at xAI, NVIDIA, Cursor and the big clouds, and it ships a release roughly every two weeks. It borrowed from vLLM and says so. Choose between them on your model and your benchmarks; they leapfrog each other constantly and the wrong move is picking on reputation.

The status change to know here: Hugging Face's Text Generation Inference server was archived in March twenty twenty six. It's in maintenance mode, and Hugging Face's own notice points people to vLLM and SGLang for servers, and to llama dot cpp or MLX locally. So if you see a tutorial that starts with TGI, that's the tell it's old.

TensorRT-LLM is NVIDIA's own library, NVIDIA GPUs only, with hand-tuned kernels, prefill and decode disaggregation, speculative decoding, and it's now built on PyTorch rather than a pure graph-compile flow, which makes it much less painful to customize than it used to be. It wins when you're NVIDIA-only and squeezing maximum throughput per dollar on recent silicon, and you have an engineer willing to own it. Above all of these sits NVIDIA Dynamo, which is not an engine but an orchestrator: it routes requests across pools of vLLM, SGLang or TensorRT-LLM workers with awareness of where the cache lives, separates prefill and decode onto different machines, and autoscales to a latency target. That's a many-GPU, many-node concern; if you have one box, ignore it.

Then there's the CPU and local family. llama dot cpp is a dependency-free C and C++ inference engine for the GGUF file format, with integer quantization from one and a half to eight bits and the widest hardware support of anything here, Apple Silicon, ARM with KleidiAI kernels, x86 with the AVX families, plus CUDA, Vulkan and more. It ships a server with an OpenAI-compatible API. Ollama started as a wrapper around llama dot cpp and still uses it as a backend, but since mid twenty twenty five it has its own engine for multimodal models built directly on the same GGML tensor library. Ollama is the pull-a-model-and-chat experience, with a model library and an OpenAI-compatible endpoint, and it's what I'd hand someone who wants a local model on a laptop in five minutes. For a server, llama dot cpp directly gives you more control over threads and memory.

Two more in one breath. NVIDIA Triton is the general multi-framework serving layer for classic models, TensorRT, PyTorch, ONNX, still maintained, and the right tool when your model isn't a language model. And Ray Serve is the composition layer when your endpoint is several models and some Python glue, a retrieval call, a validator, a reranker, wired as one app with autoscaling replicas. BentoML's OpenLLM wraps vLLM for a one-command deploy; LMDeploy and MLC-LLM are alternatives worth a search if your target is unusual hardware, MLC especially for the same model running on a phone and in a browser.

So, servers: vLLM or SGLang on a GPU, llama dot cpp or Ollama on a CPU or a laptop, TensorRT-LLM when you're all in on NVIDIA and need the last twenty percent, Triton for non-LLM models, Ray Serve for composition.

Now, running models on CPUs at all, which cuts against the reflex that inference means a GPU.

My text-to-speech pipeline runs a Qwen3 TTS model quantized to eight-bit integers, through a C port of the model, on Graviton ARM instances from AWS Batch, on Spot. No GPU anywhere. Why would anyone do that? Because the job is batch. A user submits a chapter, the audio appears later, and nobody is waiting on a first token. That means I don't need the latency a GPU buys me, I need throughput per dollar, and CPU Spot on ARM is very cheap per hour, comes in deep pools, and lets me run several model server processes side by side on one big box, each taking a slice of the memory. When Spot kills the instance, the job checkpoints its finished chunks to S3 and resubmits.

What makes this work is quantization, and specifically the fact that modern CPUs have integer matrix instructions. The mechanism: a model's weights are stored as floating point numbers, and quantizing them means storing each as a small integer plus a scale factor. Eight-bit integers cut memory by half against sixteen-bit floats and by three-quarters against thirty-two; four-bit cuts it further. Less memory means the weights fit in RAM and, more important for speed, less memory bandwidth per token, because CPU inference at batch size one is memory-bandwidth bound: every generated token has to stream the whole model through the core, so halving the bytes roughly halves the time. Then the integer matrix instructions, on ARM that's the dot-product and matrix-multiply-accumulate instructions in Graviton three and later, on Intel it's VNNI and AMX, let the chip do many eight-bit multiplies per cycle. Amazon's own Graviton guide recommends Graviton three and four for llama dot cpp precisely because of those instructions, and they publish a prebuilt container serving llama dot cpp's server on ARM.

The GGUF quant names you'll see, Q eight zero, Q four K M, and so on, encode the bits per weight, whether it's the k-quant scheme that mixes precisions per tensor, and a size variant. As one sourced data point, a Q four K M quant takes a Llama three point one eight billion parameter model from around thirty-two gigabytes to around five. That's the difference between "needs an eighty-gigabyte GPU" and "runs on a mid-size CPU instance." There's a quality cost, and it's non-linear: eight-bit is close to free, four-bit with the k-quants is usually fine for chat, two-bit is a science experiment. The rule I'd give is: quantize to eight bits without a second thought, quantize to four bits after you've run your evals on it, and don't go lower for anything you ship.

One caveat about CPU inference. It's for small models and batch work. Amazon's own demo of llama dot cpp on Graviton uses a half-billion-parameter model at four bits, which tells you the size they think is comfortable. A seven or eight billion parameter model at four bits runs on a big CPU instance, at a speed that's fine for a background job and painful for a chat box. For classic models, embeddings, classifiers, small vision models, ONNX Runtime with int8 quantization on a CPU is the standard tool, and their docs are blunt that the speedup only shows up on hardware with the integer instructions, which is one more reason to pick the current Graviton generation and not the cheapest thing in the list. And vLLM itself now has a CPU backend, x86 with AVX512 and ARM since late twenty twenty five, if you want the same server software on both.

So that's the quantization story: it's how you avoid renting a GPU for work that doesn't need one, and Spot on ARM is how you make that work nearly free. Now, queues and batch versus real-time, which is the design decision underneath all of that.

The question to ask about every model call in your product is: does a human wait for this. If no, it goes on a queue. A queue lets you run the work on Spot, batch multiple requests together for throughput, retry failures without a user seeing them, and cap concurrency so a traffic spike becomes a longer queue instead of an outage. On AWS the pattern is SQS with a Lambda or Batch consumer, and the two rules are idempotency, because a failed batch of messages returns to the queue and gets redelivered, and reporting partial batch failures so the successes in a batch aren't redone. SageMaker's asynchronous endpoint is the same idea with the queue built in. Modal's map and RunPod's async run endpoint with a webhook are the serverless-GPU equivalents.

If you're calling a managed API and a human isn't waiting, use the provider's batch mode. OpenAI's Batch API, Anthropic's Message Batches, and Bedrock's batch inference all price at fifty percent of on-demand, with a twenty-four-hour completion window, and in practice most batches finish well under that; Anthropic says most in under an hour. They also draw from separate rate limits, so your nightly summarization job doesn't starve your live chat. Half price for the same tokens is the cheapest optimization in this episode, and it's mostly a matter of how you structure the call.

And the other half of that: prompt caching. If your requests share a long prefix, a system prompt, a document, a tool schema, the providers will cache that prefix and bill cache reads at a large discount, in the neighborhood of a tenth of the input rate, with writes at a small premium. That only pays if the cached prefix gets read again within its time-to-live, which is minutes by default, so it's for chatty workloads with a shared head, not for a one-off call every hour. Put the stable content first in the prompt and the variable content last; that's all there is to it.

If a human is waiting, then you need a real-time path, and now cold starts matter. On serverless GPU that means keeping a minimum number of containers warm, which is where the "scales to zero" promise quietly becomes "scales to one." On a managed API it means you accept their latency and rate limits. On your own box it means the model is already loaded and the server does continuous batching so concurrent users don't serialize. Streaming tokens to the client is what makes a two-second answer feel instant, so do that everywhere it's a chat.

That covers the shape of the work. Now what you can't skip once it's live, observability and evals.

The minimum is traces. Every model call gets a span with the prompt, the response, the model, the token counts in and out, the latency, and the cost, tied to the user request that caused it. Without that you cannot answer the two questions you'll be asked every week, what did this cost and why did it say that. OpenTelemetry has generative AI semantic conventions now, moved into their own repository, and there are a handful of tools built on that standard. Langfuse is the open-source one I'd start with: tracing, prompt management, evals, datasets, self-hostable with Docker, and it became part of ClickHouse in early twenty twenty six, which is the database it was already built on. Arize Phoenix is the other open one, on OpenTelemetry and their OpenInference schema, though note its license is Elastic, not Apache. LangSmith is LangChain's hosted and self-hostable platform and works fine without LangChain. Weights and Biases Weave models agent sessions and turns as first-class objects. Braintrust is a commercial platform strong on evals and human annotation. Helicone takes the proxy route, change the base URL and you get tracing and cost accounting with no code change. Pick one; all of them beat printing to logs.

Evals in production are a different discipline from evals in a notebook, and the difference is that you don't have labels. What you have is traffic. So the working pattern is: sample production traces, score them with a rubric, and watch the score over time. Scoring by hand doesn't scale, and scoring with a model, the LLM-as-a-judge pattern, does. The paper that established it, Judging LLM as a Judge with MT-Bench and Chatbot Arena, found strong judge models agree with human preferences over eighty percent of the time, about as often as humans agree with each other, and it also named the biases to guard against: judges prefer the first answer shown, prefer longer answers, and prefer their own family's outputs. Randomize position, control for length, and use a judge from a different lab than the model under test.

For classic models the older word for this is drift, and the tooling consolidated hard. WhyLabs shut down, though it open-sourced its libraries; Evidently is alive and now covers both worlds, statistical drift tests on tabular features and a large set of LLM metrics. If you have a scikit-learn model in production, log its inputs and predictions to S3 and run a weekly distribution comparison; that's a cron job and a notebook, not a platform.

Two operational things that aren't tools. Set a cost budget per user action and alert on it; runaway agent loops and prompt-injection-driven token burns show up in cost before they show up anywhere else. And keep an eval set of a few hundred real cases you re-run before every model swap, because the providers deprecate models on a schedule and "we upgraded to the new version" is the most common way quality regresses in production.

Last real section, and I'll keep it short: infrastructure as code. All of the above should be declared in code, because a Batch compute environment, a SageMaker endpoint, an SQS queue and its dead-letter queue, and the IAM roles between them are exactly the kind of thing you'll misconfigure by hand at two in the morning. I use SST, which is a TypeScript framework on top of Pulumi; SST's own docs say Pulumi is the engine underneath and Terraform providers are bridged through it, so anything Pulumi or Terraform can provision, you can drop into an SST config next to your web app. Pulumi itself is the pick if you want real programming languages without SST's web opinions. Terraform moved to the Business Source License in twenty twenty three and OpenTofu is the open fork, both alive and both fine; HCL has the largest community, and most examples you'll find online are written in it. AWS CDK is the choice if you're all AWS and want CloudFormation underneath. Whichever you pick, the rule is the same: the ML infra lives in the same repo and the same deploy as the product, not in a separate ops repo, because the product is what changes it.

Now shout-outs, one sentence each, for the second tier. Groq and Cerebras are inference APIs on custom silicon, LPUs and wafer-scale chips respectively, whose pitch is raw tokens per second, and they're the answer when a hosted open model isn't fast enough. Together AI, Fireworks, and DeepInfra are the open-model inference clouds, per-token serverless plus dedicated endpoints plus, for Together, rentable GPU clusters. OpenRouter and Hugging Face's Inference Providers are aggregators, one API key and an OpenAI-compatible endpoint that routes across many providers with fallbacks; Hugging Face's version lets you ask for the fastest or the cheapest provider for a given model. Hugging Face Inference Endpoints is the managed dedicated deployment of a Hub model with vLLM or TGI underneath. AWS Trainium is Amazon's training silicon, in production at Anthropic and others; Inferentia is its inference cousin, and neither is a place a solo builder starts. And a plain warning: model hubs prune, SageMaker JumpStart delisted a batch of models in March twenty twenty six, so pin the weights you depend on in your own bucket.

Let me recap. One decision drives the rest: call a model or run one. Call by default; run when volume, residency, latency, or availability forces it, and move one workload at a time. On AWS, serverless compute is CPU-only; SageMaker async endpoints and Batch on Spot cover most bursty GPU needs, and Aurora pgvector is your vector store until it isn't. Serverless GPU platforms, Modal, RunPod, Cloud Run GPU, Baseten, fill the gap AWS left, each with a different cold-start and warm-pool story. Bare GPUs and neoclouds win on sustained load; Spot wins on interruptible load, and the discount is paid for in checkpointing, retries, and pool diversity. vLLM or SGLang on GPUs, llama dot cpp or Ollama on CPUs, and quantize to eight bits freely and four bits after evals. Queue everything a human doesn't wait for, use batch pricing and prompt caching, and put traces and a judge-scored eval loop in place before you think you need them. And declare all of it in code beside the product.

That's the infrastructure episode. If what you build on this infrastructure is agents, the next stop is AI Agents in 2026, and for the coding-agent workflow that ships this kind of code, the Vibe Coding sequence starting at episode twenty-two.