Almost every AI request your business makes leaves the building. You type a question, it travels to a data centre owned by somebody else, a model you cannot inspect produces an answer, and the answer comes back. That arrangement made sense while the only models worth using were far too large to run anywhere but a rented server rack.
That is changing quickly. Open-weight models have nearly caught up with the closed ones, a genuinely capable model now downloads in less space than a feature film, and the hardware to run it sits under a desk. Our view, after several years of tracking this closely, is that the end state is unremarkable: a small AI machine in the office, about as exciting as the Wi-Fi modem in the cupboard, doing most of the everyday work while the cloud handles the hard cases.
This article covers what a local AI model actually is, how much of the quality gap is left, what fits on hardware you can buy today, how to run one this afternoon, and the specific situations where running locally is the wrong call.
What a local AI model actually is
A local AI model is a language model whose weights are stored on hardware you control, so that inference, the process of generating an answer, happens on your own machine and no part of the prompt travels over the internet.
The weights are the model. A large language model is a very large file of numbers, called parameters, that encode everything the model learned during training. "Open weights" means the company that trained the model has published that file for anyone to download and run. Meta publishes Llama this way, Google publishes Gemma, DeepSeek publishes its reasoning models, Alibaba publishes Qwen, and Mistral publishes most of its range.
Two distinctions are worth getting right early, because they cause most of the confusion.
Open weights is not open source. The weights are published, but the training data and the training code usually are not. You can run the model, inspect its outputs, and adjust it. You cannot generally reproduce it from scratch or audit what went into it.
Running a model is not training one. Downloading Gemma and pointing it at your documents is a Tuesday afternoon. Training a comparable model from nothing costs tens of millions of dollars. The middle option, fine-tuning an existing open model on your own examples, is real and affordable, but it is a separate project from simply running one.
The gap that justified the cloud has nearly closed
For most of 2023 the argument against local models was straightforward: they were noticeably worse. That argument has largely expired.
Stanford's Institute for Human-Centered AI tracks the difference directly. In its 2025 AI Index, it reports that "in early January 2024, the leading closed-weight model outperformed the top open-weight model by 8.04% on the Chatbot Arena Leaderboard. By February 2025, this gap had narrowed to 1.70%."
Read that carefully, because it is easy to over-claim. It does not say open models are better. The very best frontier systems are still closed, still cloud-hosted, and still ahead on the hardest reasoning work. What it says is that the distance between the best model anyone can run and the best model money can rent has become small enough that, for the ordinary business tasks that make up most AI usage, the choice is now about control, cost and latency rather than capability.
Drafting a reply. Summarising a contract. Extracting the line items from a supplier invoice. Classifying an inbound enquiry. None of these need the frontier. They need a competent model that is reliably available and does not send your commercial data anywhere.
What actually fits on the hardware you can buy
This is where most people's mental model is wrong, usually by an order of magnitude in the pessimistic direction.
A model's file size is set by two things: how many parameters it has, and how many bits are used to store each one. Storing every parameter at full precision is wasteful for inference, so models are quantised, which means the numbers are compressed to fewer bits each with a small and usually acceptable loss of quality. Four bits per parameter is the common working default.
Google's Gemma 3 is a useful yardstick because Ollama publishes the exact download sizes:
| Variant | Parameters | Download size | Runs comfortably on |
|---|---|---|---|
| Gemma 3 270M | 270 million | 292 MB | Any laptop from the last five years |
| Gemma 3 4B | 4 billion | 3.3 GB | A 16 GB laptop, alongside your other work |
| Gemma 3 12B | 12 billion | 8.1 GB | A 24 GB machine |
| Gemma 3 27B | 27 billion | 17 GB | A 48 GB machine, or one consumer GPU |
Google describes the 27B variant as "the current, most capable model that runs on a single GPU". Seventeen gigabytes. That is smaller than a 4K film and it fits on a machine you can buy off the shelf.
Memory capacity is the constraint, not raw speed
Generating a token requires reading the model's weights out of memory. That makes inference bound mostly by memory, which is why the practical question is not "how fast is the processor" but "does the model fit in memory at all, and how quickly can it be read".
Apple's approach happens to suit this well, because the processor and graphics cores share one pool of memory rather than copying between separate ones. Apple's published Mac mini specifications list the M4 with 16 GB of unified memory, configurable to 24 GB, at 120 GB/s, and the M4 Pro with 24 GB configurable to 48 GB at 273 GB/s. A 48 GB desktop machine that draws a few watts at idle will hold a 27-billion-parameter model with room left for a long conversation.
A workable rule of thumb: budget the model's download size plus roughly a third again for the context window and overhead. A 17 GB model wants about 24 GB of headroom to be pleasant to use.
Run one this afternoon
The fastest way to replace an opinion with a fact is to do this yourself. It takes about fifteen minutes, most of which is downloading.
- 1.Install Ollama, which is free and available for macOS, Windows and Linux. It is the least fiddly of the local runtimes. LM Studio is the equivalent if you would rather have a graphical interface.
- 2.Open a terminal and run one command. Ollama downloads the 3.3 GB model, then drops you into a conversation with it.
ollama run gemma3:4b- 1.Ask it something about your business. Note the speed. On a recent laptop the reply starts within a second or two.
- 2.Now do the part that makes the point. Turn off your Wi-Fi and ask it another question. It answers. Nothing left the machine, because there was nowhere for it to go.
- 3.If you want to use it from your own software rather than the terminal, Ollama exposes a local API on port 11434 that behaves much like a hosted one:
curl http://localhost:11434/api/generate -d '{"model":"gemma3:4b","prompt":"Summarise this invoice","stream":false}'That last step is the one that matters commercially. Any automation you have already built against a hosted model can usually be pointed at a local one by changing a URL, which means the switch is reversible and you can test it against real work before committing.
What people get wrong about running models locally
"Local means worse." Mostly no longer true, and precisely quantified above. The honest version is narrower: local means a small step down from the frontier, which is invisible on routine work and occasionally noticeable on hard reasoning.
"Local means free." It means the cost moves rather than disappears. You stop paying per token and start paying for hardware, electricity and a little of your own time. That is a good trade at high volume and a poor one at low volume. A business making a handful of AI requests a week should stay on a hosted model.
"You need a rack of GPUs." You need enough memory. A single well-specified desktop covers the great majority of small-business workloads, and the machine can sit in a cupboard.
"Local is automatically private." This is the dangerous one. Keeping the model on your premises removes one risk, which is your data sitting in a third party's logs. It does nothing about the other risks: an unpatched machine, backups nobody encrypted, or an internal system reachable from the open internet. Local hosting is a component of a privacy posture, not a substitute for one. We cover the wider picture in security, privacy and trust in AI adoption.
When local wins, and when it does not
Here is the verdict, stated plainly.
| Situation | Recommendation |
|---|---|
| Sensitive client, legal, health or financial data | Run locally. The compliance argument usually settles it on its own. |
| High, steady request volume (thousands per day) | Run locally. Fixed hardware beats per-token pricing once volume is predictable. |
| Offline or unreliable connectivity (a site office, a regional depot) | Run locally. It is the only option that works. |
| Hardest reasoning, long research, frontier-quality writing | Use a hosted model. The remaining gap shows up exactly here. |
| Low or spiky volume, no in-house technical support | Use a hosted model. Owning hardware for a dozen requests a week is not worth it. |
| Anything experimental, before you know what you need | Use a hosted model first, then move the settled workloads local. |
Most businesses that get this right end up running both. The local machine handles the repetitive, sensitive, high-volume work. The hosted model gets called for the small number of tasks that genuinely need the frontier.
Why this argument lands differently in Australia
Trust, not capability, is what is actually holding Australian adoption back. The National AI Centre's quarterly adoption data for December 2025 to February 2026 found that 43% of Australian SMEs reported some level of AI adoption, and that "around 65% of non-adopting businesses cited either a distrust in AI decision-making or a strong preference to maintain human control over their business processes."
That is a control problem being solved with an abstinence policy. A model running on a machine in your own office is a direct answer to it: you can see where the data goes, because it does not go anywhere. For the accountants, clinics and law practices we speak to, this changes the conversation from "should we use AI at all" to "which of these tasks runs on our machine". Our wider read of the local data sits in AI adoption in Australian small business.
The Enki Approach
We build for reversibility, because the ground is still moving. That means treating the model as a component rather than a foundation: the automations and applications we build talk to a model through an interface we control, so swapping a hosted model for a local one (or the reverse) is a configuration change rather than a rebuild. Standards like the Model Context Protocol push in the same direction, giving the tools around the model a common connector regardless of where the model itself runs.
Our practical advice to clients today is to start hosted, instrument what you actually use, and move the workloads that are sensitive or high volume onto your own hardware once the pattern is clear. Buying a machine first is how businesses end up with an expensive box and no workflow to put on it. There is also a quieter benefit worth naming: models running on hardware you already own do not add to the water and electricity draw of a hyperscale data centre, which we looked at in AI sustainability and the hidden costs.
The modem analogy is the one we keep coming back to. Nobody deliberates about their Wi-Fi modem. It sits in a cupboard, it works, and the only time anyone thinks about it is when it stops. Local AI is heading for the same status, and the businesses that quietly get there first will spend the next few years paying less and worrying less about where their data went.