Quick answer: running an LLM locally converts a metered, per-token bill into a fixed infrastructure cost — and whether that's a good trade depends almost entirely on your volume and your model size. Small models handle a surprising amount of real business work on modest hardware; large models need serious GPUs and the cost climbs fast. The crossover point where local beats an API arrives sooner than most people expect for document-heavy workloads, and never arrives for light, occasional use. Here's how to actually work out which side of that line you're on.
The two cost models, and why they behave differently
An API bills per token — roughly, per unit of text in and out. That means your cost is directly proportional to usage: process twice as many documents and you pay twice as much, forever. The virtue of this model is that it starts at essentially zero and requires no capital or infrastructure. The vice is that it never stops scaling, and a workflow that becomes genuinely useful (which means people use it more) becomes progressively more expensive precisely because it's working.
Local inference bills for capacity, not usage. You buy or rent hardware capable of running your chosen model, and then the marginal cost of an additional request is effectively electricity. Ten requests or ten thousand, the monthly number barely moves. The virtue is predictability and, past sufficient volume, dramatically lower total cost. The vice is that you pay it whether you use the capacity or not, and you've taken on hardware and operational responsibility. These are genuinely different shapes, and the right choice follows from which shape fits your usage.
Hardware, by model size
Model size is the primary driver of hardware requirements, and it's where people most often over-buy. Small models — the compact open-weight models designed for efficiency — run acceptably on ordinary modern hardware, including a well-specified laptop or desktop and modest cloud instances, sometimes without a dedicated GPU. They're genuinely useful for classification, extraction, summarization of moderate-length text, and straightforward drafting, which covers a large share of practical business automation.
Mid-sized models deliver noticeably better reasoning and handle longer, more complex material, and they generally want a dedicated GPU with a reasonable amount of video memory — the tier where a purpose-built workstation or a GPU cloud instance becomes the sensible host. Large models, the ones that approach frontier proprietary capability, require substantially more GPU memory and often multiple GPUs, which moves you into meaningful capital expense or a significant monthly cloud bill. The practical lesson from deployments we've done: most business workloads are well served by small or mid-sized models, and starting there rather than at the top saves a great deal of money without meaningfully hurting outcomes.
The costs nobody puts in the spreadsheet
Hardware is the visible cost; the invisible ones decide whether the project succeeds. Operational time is the big one — someone has to deploy the model, keep the host patched and secure, monitor that it's actually running, handle backups, and troubleshoot when something breaks. That's modest ongoing effort, not a full-time job, but it's not zero, and pretending otherwise is how self-hosted systems quietly rot. Budget for it explicitly, whether that means internal time or a maintenance arrangement.
Idle capacity is the other one. If you buy hardware sized for your peak and your usage is spiky, you're paying for capacity that sits unused much of the time — which weakens the economics compared to an API that costs nothing when you're not using it. And there's opportunity cost: time spent standing up infrastructure is time not spent building the automation that actually produces value. None of these are reasons to avoid self-hosting; they're reasons to size honestly and to make sure the volume genuinely justifies it before you start.
Finding your crossover point
The practical way to decide is to estimate your realistic monthly token volume and price it both ways. Take the workload you actually intend to run — say, processing a few hundred documents a month, or powering an internal assistant used daily by a handful of staff — and estimate the tokens involved. Price that at your chosen API's rates to get a monthly API cost. Then price the hardware or cloud instance capable of running a model suited to that task, plus a realistic allowance for maintenance, to get a monthly self-hosted cost. Compare.
What you'll typically find: light, occasional use strongly favors the API — a few dollars a month versus a server is no contest. Document-heavy or high-frequency workloads favor local, often decisively, because token counts on long documents add up quickly and the fixed cost stops mattering as volume grows. And crucially, if confidentiality is a requirement, the cost comparison is secondary — you're not choosing the cheaper option, you're choosing the only option that lets you run the workload at all, which changes the calculation entirely.
The hybrid answer most businesses land on
In practice, the best architecture for most businesses isn't either/or. You run the sensitive and high-volume workloads on a local model where the economics and the data requirements both favor it, and you use an API for the occasional task that genuinely needs frontier capability. Building your automations so the model is a swappable component — rather than hard-coding one provider — is what makes this practical, and it also protects you from being stranded if pricing or terms change. That flexibility costs almost nothing to design in at the start and is expensive to retrofit later, which is why we build client systems that way by default.
FAQ
Can I run an LLM on a normal computer?
Smaller open-weight models can run on ordinary modern hardware — a well-specified desktop or laptop, sometimes without a dedicated GPU — and they're genuinely capable for classification, extraction, summarization, and routine drafting. Larger, stronger models need dedicated GPU hardware with substantial video memory. Whether "a normal computer" is enough depends entirely on which model you need, and most business tasks need less than people assume.
Is running an LLM locally cheaper than the API?
Past a certain volume, yes — often substantially. Local inference is a fixed cost that doesn't grow with usage, while API cost scales with every token forever. For light or occasional use the API is far cheaper and simpler. For document-heavy, high-frequency workloads the crossover arrives quickly and local wins thereafter. Estimate your realistic monthly volume and price it both ways; the answer is usually clear.
What GPU do I need for a local LLM?
It depends on model size. Small models may run without a dedicated GPU at all; mid-sized models generally want a GPU with a reasonable amount of video memory; large models need substantially more, sometimes multiple GPUs. The most common mistake is specifying for the largest model available rather than for the tasks you actually have — sizing to your real workload usually reduces hardware requirements considerably.
What are the hidden costs of self-hosting an LLM?
Mainly operational time — deployment, security patching, monitoring, backups, and troubleshooting — plus idle capacity you pay for whether or not you use it, and the opportunity cost of infrastructure work versus building automation. None are prohibitive, but leaving them out of the comparison makes self-hosting look better than it is. Budget for maintenance explicitly, either as internal time or a support arrangement.
Should I self-host or use an API?
Use an API for light, occasional, or frontier-capability work. Self-host for high-volume, document-heavy, or confidential workloads where the fixed cost beats metered pricing or where data simply cannot leave your infrastructure. Most businesses end up hybrid — and building your automations so the model is swappable lets you move workloads between the two without rebuilding anything.
Can I start with cloud GPU instead of buying hardware?
Yes, and it's often the smarter first step. Renting a GPU instance from a cloud provider you control gives you the data-control benefits of self-hosting without capital expenditure, and it lets you validate that a local model actually meets your quality bar before committing to hardware. If usage proves steady and substantial, buying can become cheaper than renting; if it doesn't, you've avoided a purchase you'd regret.
How do I estimate my token volume?
Take the workload you actually plan to run and work from documents rather than guesses: roughly how many documents or requests per month, and how long each is. A page of text is on the order of several hundred tokens, and both your input and the model's output count. Multiply through for a monthly figure, then price it at your API's published rates. It's a rough estimate, but it's usually accurate enough to show clearly which side of the crossover you're on.
Want help sizing this for your actual workload? A free audit prices your specific use case both ways honestly. Related: self-hosted AI explained and best open-source LLMs for business.