# 6 tips: how companies run AI language models on their own hardware

Press release by the AI agency Iphos of 9 November 2026: memory decides whether a language model runs, the workflow how fast it is – six tips on model, machine, costs and location.

<https://searchit-agent.ai/unternehmen/pr/6-tipps-so-betreiben-unternehmen-ki-sprachmodelle-auf-eigener-hardware/>

*Vienna, 9 November 2026*

Memory decides whether a model runs – the workflow decides how fast it is

**(Vienna, 9 November 2026)** With the right hardware, companies, law firms and public authorities can run AI language models in-house. The question of running them yourself matters for two reasons: the data stays in-house, and with sustained utilisation the bill can turn out lower than when using a model via an application programming interface (API). Which machine is the right one depends less on the size of the model than on the workflow, and that is changing right now: a short chat question comprises a few words. An AI agent, by contrast, reads 10,000 to 30,000 tokens of search hits and document excerpts per work step; tokens are the text units in which language models compute. The following six tips show IT managers how to match model and machine, when buying pays off and where the device is best placed. They are based on published measurements, current prices and a worked example with 500,000 documents.

## 1. Choose the model first, then the machine

A language model consists of billions of learned numerical values, the parameters. For it to read texts and write answers, these values must sit entirely in fast memory directly next to the processing chip. On a workstation, that is the memory on the graphics card, the GPU memory (VRAM); on the Mac Studio and on AI boxes, the processor and the graphics unit share a common memory. If the model does not fit, it cannot be run in any meaningful way.

How much space is needed depends on size and storage format. A model with around 26 billion parameters occupies about 16 gigabytes of GPU memory in the 4-bit version, which stores each value compactly and somewhat less precisely, and about 26 gigabytes in the 8-bit version. In addition, every running request needs a cache in which the model keeps track of what it has already read. The longer the texts and the more requests run at the same time, the more space it takes up. On a graphics card with 96 gigabytes of GPU memory, around 70 gigabytes remain free for this cache alongside the 8-bit model; on a card with 48 gigabytes, around 20. Which models with open weights, i.e. whose model files are publicly available, run on which graphics card, and what obtaining them via an API would cost, is shown by a free comparison portal linked below.

“The question about GPU memory is the right first question, but it is only the first,” says Ing. Christoph Wendl, CEO of searchit. “Anyone who buys hardware for their own workflow rather than for the biggest model currently making the rounds gets more performance for their budget.”

## 2. Factor in the length of the inputs

A language model works in two steps, and each places different demands on the hardware. When reading in, it processes the entire input at once. That is a large computing task, so what counts here is the computing power of the graphics chip. When writing, it generates the answer token by token and reads its parameters from memory each time. What counts here is memory bandwidth, i.e. how fast data gets from memory to the chip.

For a short chat question, reading in takes fractions of a second; the waiting is mainly for the writing. With documents and agents, reading in becomes the bottleneck. For 20,000 tokens, the calculation gives around 0.6 seconds on an RTX PRO 6000, 1.2 seconds on an RTX PRO 5000 and roughly 3 to 12 seconds on a Mac Studio M5 Ultra. Since an agent takes several such steps per question, these times add up.

## 3. Plan for concurrent requests from the start

One person in a chat, ten colleagues with an agent and a batch of 500,000 documents are three different loads, and each calls for different hardware. For a few users with short questions, a Mac Studio (€6,599 with 96 gigabytes) or an AI box of the DGX Spark class (€4,900 to €6,900) is sufficient. Both sit quietly on the desk.

If many colleagues or several agents are to work at the same time, a workstation with an RTX PRO 6000 (around €22,200) plays to its strengths. There, the vLLM software bundles many requests and lets the graphics card process them together: 125 concurrent requests at 18 tokens per second each were measured. On the Mac, bundling, measured with the llama.cpp software, only helps when writing. Long inputs are read in there practically one after the other, so the requests wait for each other.

## 4. Check benchmarks against your own workflow

Benchmarks are standardised performance tests, and their values only apply to the conditions of the test. The widely used measuring tool llama-bench works by default with 512 tokens of input, 128 tokens of output and a single request. That says little about an agent that reads 20,000 tokens and runs alongside other requests. Meaningful values are those at your own input length and with as many concurrent requests as in operation.

The software version also matters: on a Mac Studio M3 Ultra, reading in 8,192 tokens dropped from around 2,106 to 1,183 tokens per second after an update of llama.cpp, with the same hardware and the same model. The most reliable approach therefore remains your own measurement with the planned model and typical inputs.

## 5. Calculate API and rental before buying

A language model does not have to run on your own hardware. Via an application programming interface it can also be used at a provider, billed by processed tokens. For a one-off large batch, that is almost always cheaper. An example with 500,000 documents, each with 7,500 tokens of input and 1,500 tokens of output, and the model Gemma 4 26B A4B shows this: with the cheapest offers processing in the EU, the run costs €600 to €850, a suitable workstation €13,500 to €22,200.

Your own hardware pays off when it works continuously, because it depreciates even when idle. At full utilisation over 36 months and an electricity price of €0.30 per kilowatt hour, 1,000 documents on an RTX PRO 6000 cost around €0.18 by calculation, via the API €1.20 to €1.70. If the workstation only runs eight hours a day, its cost per document roughly triples. Where data should not leave the building, for instance for professionals bound by confidentiality, own hardware is also the obvious choice. In between lies rental: European data centres offer an RTX PRO 6000 from around €1.25 per hour, without purchase and with the same advantages for concurrent requests.

## 6. Consider the location from the outset

Whatever power a graphics card draws, it gives off again as heat, and its fans have to dissipate this heat. That is why the question of noise and waste heat belongs before the purchase. An AI box with around 160 watts under load and a measured maximum of 37.5 dB(A) at a distance of 46 centimetres can stand at the workplace. A workstation with an RTX PRO 6000, by contrast, belongs in a side room with air supply: the card draws up to around 530 watts, the waste heat corresponds to a fan heater on a low setting, and the German workplace regulation ASR A3.7 specifies a planning value of 40 dB(A) for one- and two-person offices.

Running around the clock, electricity costs about €1,970 a year. The Max-Q variant manages with 300 instead of 600 watts, costs around €1,180 in electricity and, as measured, works 13 to 19 percent slower with language models. For runs lasting several days, an uninterruptible power supply is advisable so that a brief interruption in the power grid does not abort the entire run.

More information on the article “Hardware for local LLMs”: [www.searchit-enterprise-search.com/hardware-lokale-llm](https://www.searchit-enterprise-search.com/hardware-lokale-llm/)

More information on searchit Models – self-hosting: [models.searchit-agent.ai/selbst-betreiben](https://models.searchit-agent.ai/selbst-betreiben/), cost calculator: [models.searchit-agent.ai/werkzeuge/kostenrechner](https://models.searchit-agent.ai/werkzeuge/kostenrechner/)

More information on the searchit AI Agent: [searchit-agent.ai](https://searchit-agent.ai/)

## About searchit: AI agents for businesses

searchit turns an organisation's knowledge into the memory of its AI. The searchit AI Agent knows your data holdings (emails, documents, etc.) and works with them. This makes the agent a digital co-worker. The technology builds on searchit's long experience as an enterprise search platform used by numerous customers such as Samsung, TÜV, the Technisches Museum Wien and the Bavarian digital authority. searchit is made by Iphos IT Solutions GmbH, based in Vienna. Its in-house AI lab continuously develops the platform's AI functions. Ing. Christoph Wendl, together with Lyubomir Ivanov, runs the company, founded in 1998, as Chief Executive Officer.

## Media contact

- Ing. Christoph Wendl, Managing Director, Iphos IT Solutions GmbH
- Khekgasse 35, 1230 Vienna, Austria
- Phone: +43 1 869 84 00
- Email: [info@searchit.cc](mailto:info@searchit.cc)

- [Release as PDF (German)](https://searchit-agent.ai/unternehmen/pr/6-tipps-so-betreiben-unternehmen-ki-sprachmodelle-auf-eigener-hardware/20261109-presseaussendung-hardware-llm.pdf)

---

- searchit AI Agent: <https://searchit-agent.ai/>

This version is generated from the page's language files (scripts/gen-unternehmen.py). The page itself is authoritative.
