On-Premise AI: When to Run LLMs and Voice Agents on Your Own Infrastructure

Most companies start with hosted APIs, and for most workloads that is the right call. A smaller group reaches a point where APIs stop fitting: regulators require that customer data stays inside the company's perimeter, call volumes make per-minute speech pricing expensive, or a voice agent needs predictable response times that a shared public API cannot guarantee. For them, the question becomes how much hardware they need and whether it pays off.

Self-hosting AI pays off when data cannot leave the company's perimeter, when volume is high and steady enough to keep GPUs busy, or when latency has to be controlled end to end, and it rarely pays off on cost alone for small or irregular workloads.

This article explains when on-premise AI makes sense, how to size GPU memory for language models, how the costs compare with APIs and what a high-availability voice AI deployment looks like in practice. It does not cover training models from scratch, and it does not compare hardware vendors.

Key takeaways

• The strongest reasons to self-host are data residency, high steady volume and latency control. Cost alone rarely justifies it for small workloads.
• GPU memory needed for a model is roughly the parameter count multiplied by bytes per parameter, plus memory for the conversation context (the KV cache) and about 10–20% overhead.
• A 70-billion-parameter model needs about 140 GB for weights at 16-bit precision and about 35 GB at 4-bit. Open models such as gpt-oss-120b are designed to fit on one 80 GB GPU.
• For text workloads, hosted APIs are often cheaper. For speech at contact-center volume, self-hosted recognition can cost a fraction of per-minute API pricing.
• A voice agent has about two seconds to answer. Speech recognition, the language model, speech synthesis and the phone network each take a share of that budget.
• Production on-premise AI needs at least two nodes, automatic failover, monitoring and a plan for model updates.

Reasons to self-host, and reasons not to

Self-hosting tends to make sense when:

• Data must stay inside your perimeter. Telecom operators, hospitals, banks and public bodies often have rules or contracts that forbid sending recordings, health data or customer records to external processors.
• Volume is high and steady. GPUs bought or reserved for a year are cheapest when they are busy most of the day.
• Latency must be predictable. Voice agents and real-time assistants need consistent response times, without queues at a shared provider.
• You need a fixed model version. Some regulated processes require that the model producing a result does not change without validation.

APIs tend to be the better choice when:

• Volume is small or irregular. Paying per token is cheaper than paying for idle hardware.
• You need the strongest available models. The most capable frontier models are usually available only through APIs.
• The team has no infrastructure capacity. Running GPUs in production needs DevOps skills, monitoring and on-call coverage.

A hybrid setup is common. Sensitive steps, such as transcribing calls or reading health records, run on-premise, and tasks with anonymized or public data use hosted models. Our article on how to add AI to an existing app covers the integration patterns.

How much GPU memory a language model needs

GPU memory has three parts: the model weights, the KV cache that holds the context of active conversations, and runtime overhead.

Weights

Weights in gigabytes are approximately the number of parameters in billions multiplied by the bytes per parameter: 2 at 16-bit precision (FP16 or BF16), 1 at 8-bit and about 0.5 at 4-bit.

Mixture-of-experts models need memory for all their parameters, even though only a fraction is active for each token. According to its model card, gpt-oss-120b has 117 billion parameters with 5.1 billion active, and its 4-bit weights are designed to fit on a single 80 GB GPU such as an H100. The smaller gpt-oss-20b runs within 16 GB.

KV cache

The KV cache grows with the number of tokens in all active conversations. Per token, it equals 2 × layers × key-value heads × head dimension × bytes per value.

For a Llama-class 8B model with 32 layers, 8 key-value heads and a head dimension of 128, that is 128 KB per token at 16-bit precision. A comparable 70B model with 80 layers needs 320 KB per token.

This is why a model that fits on a GPU in a demo can run out of memory under real traffic. Inference servers such as vLLM manage the KV cache in pages and let you cap context length and concurrency, which keeps memory use predictable.

Overhead

Add 10–20% for activations, the inference runtime and memory fragmentation. A practical sizing rule: weights plus the KV cache for peak concurrency, plus 20%.

API or your own GPUs: a worked comparison

The comparison below uses a contact center handling 12,000 calls a day, with an average call of four minutes, every call transcribed and analyzed, and a voice agent that handles routine requests. All prices are illustrative assumptions for 2026.

Workload per month:

• Speech recognition: 12,000 calls × 4 minutes × 30 days = 1.44 million minutes.
• Language model: 8 turns per call, about 1,500 input tokens and 100 output tokens per turn, which comes to about 4.3 billion input tokens and 290 million output tokens.

Hosted APIs, per month:

• Speech recognition at $0.006–$0.024 per minute: about $8,600–$34,600.
• A mid-tier language model at about $0.40 per million input tokens and $1.60 per million output tokens: about $2,200.
• Total: about $10,800–$36,800.

Self-hosted, per month:

• Two servers with two 80 GB GPUs each, for high availability: about $160,000 for GPUs, servers and networking, or about $4,400 a month over three years. A single H100 card lists at roughly $25,000–$35,000.
• Power, cooling and rack space: about $1,500.
• Operations, about half of one DevOps engineer: about $5,000.
• Total: about $10,900.

The result is less obvious than vendors on either side suggest. The language model alone is cheaper through an API at this volume. Speech recognition changes the picture: at contact-center volume, per-minute pricing makes self-hosting break even at the low end of API prices and cost about a third as much at the high end. Add a requirement that recordings never leave the building, and the decision is made before the spreadsheet is finished.

Teams that only need text models and have no residency constraint should run the same calculation with their own volumes before buying hardware. Our breakdown of what it costs to run an LLM on a 250-page case file shows how API costs work per document.

Conversational AI voice agents: the two-second budget

Callers expect a reply within about two seconds of finishing a sentence. A voice agent built as a pipeline of speech recognition, a language model and speech synthesis has to fit every step into that window.

Three techniques keep the total under two seconds. Streaming recognition sends partial transcripts while the caller is still speaking. The reply is synthesized sentence by sentence, so audio starts before the full answer is generated. Common requests, such as balance checks, go to fast intent handlers instead of the language model.

On-premise deployment helps with this budget because the models sit on the same network as the phone system, and no request waits in a shared provider's queue.

A reference deployment: voice AI for a telecom operator

A regional telecom operator with over 12 million mobile subscribers, working with .wrk under NDA, ran customer contact through a legacy IVR and a distributed call-center network. Peak load exceeded 12,000 inbound calls a day. About 35% of routine requests, such as balance checks, tariff changes and SIM status, still needed a live agent, and quality teams reviewed fewer than 2% of calls by hand. Regulation required subscriber voice data to stay inside the operator's perimeter, so cloud speech APIs were ruled out.

Architecture

• Two-node high-availability cluster. Kubernetes on two nodes in the operator's data center, with automated failover tested before launch, so the service survives the loss of a node.
• Speech recognition. NVIDIA's Parakeet model for real-time transcription, speaker separation and post-call analytics on every recorded call. The current Parakeet TDT 0.6B v3 model has 600 million parameters and supports 25 European languages, so a single GPU can process large call volumes.
• Voice agent. A dialogue engine for the 15 most common subscriber requests, with speech synthesis and handoff to a live agent when the request falls outside its scope.
• Event bus. Kafka carries call events between the phone system, the speech pipeline, the analysis service and the CRM, so a slow component does not block calls.
• Integration. Connection to the operator's SIP trunk and PBX for inbound routing, overflow and after-hours calls, and webhooks that send call summaries, quality scores and escalation flags to the existing CRM and ticketing system.
• Observability. Prometheus and Grafana dashboards for latency, recognition quality, GPU load and service levels, reviewed weekly with contact-center supervisors.

Rollout

The team of a project manager, two AI engineers, two backend engineers and a DevOps engineer launched on two pilot contact-center sites. The agent ran in shadow mode first, then on limited traffic, then in full production on both sites. Within 12 weeks of go-live, the operator had every recorded call transcribed and scored, lower routine load on agents and full compliance with its data rules, without external speech APIs and without replacing its CRM. Read the on-premise voice AI case study.

What running on-premise AI involves after launch

• Model updates. New model versions are tested against a fixed set of real examples before they replace the old one, and the old version stays available for rollback.
• Capacity planning. GPU memory and utilization are tracked against peak traffic, with headroom for growth and for one node being offline.
• Security. Model servers sit in a restricted network segment, access is logged, and prompts and outputs that contain personal data follow the same retention rules as other records.
• Quality monitoring. Recognition accuracy, answer quality and escalation rates are measured continuously, because they drift as products, tariffs and customer language change.

Our guide to an AI governance framework covers the review and approval rules that apply to self-hosted models as much as to APIs.