How to run AI locally for office work
A local model is the simplest answer to "where does our data go?" — it goes nowhere. Here is what it takes on an ordinary office computer.
Short answer
Install a model runner — Ollama or LM Studio — download an open-weight model sized for your computer's memory, and point your office tool at the runner's local address. A 16 GB laptop runs a 7–8 billion parameter model well enough for drafting, summarising and sorting text. For harder reasoning, use a bigger machine, or keep a cloud model for those tasks only.
What hardware do you need?
Memory decides which models you can run. Models are usually run "quantised" to 4 bits, which needs very roughly 0.6 GB per billion parameters plus a few gigabytes for the app and the conversation.
| Memory | Model size (4-bit) | Good enough for |
|---|---|---|
| 8 GB | 3–4B parameters | Short summaries, sorting and tagging text, simple rewrites |
| 16 GB | 7–8B, up to about 14B | Drafting routine letters, summarising, pulling fields out of text |
| 32 GB | 14–32B | Longer documents, better at following detailed instructions |
| 64 GB or more | 70B class | Closest to cloud quality for everyday work; slower |
These are rough guides. Apple Silicon Macs share memory between the processor and graphics, which suits local models well. On a Windows PC, a graphics card's own memory (VRAM) matters most; without one, models run on the processor, which works but is slower.
Step 1: install a model runner
A runner downloads models, loads them and answers requests on a local address that other apps can call.
- Ollama — MIT-licensed, for macOS, Windows and Linux. It runs in the background and serves an API on
http://localhost:11434, including an OpenAI-compatible endpoint at/v1. Best when other apps should use the model all day. - LM Studio — a desktop app for finding, downloading and trying models, free for home and work use but not open source. Its local server listens on
http://localhost:1234with OpenAI-compatible endpoints. Best for trying several models side by side.
Either works with OpenClerq. So does any other server that speaks the OpenAI chat format, such as llama.cpp's server or vLLM.
Step 2: choose a model
Open-weight model families you will see in both runners include Llama (Meta), Qwen (Alibaba), Gemma (Google), Mistral, DeepSeek and OpenAI's gpt-oss. Each comes in several sizes. Check the licence of the model as well as the runner — some allow any commercial use, others attach conditions.
# Ollama: download and try OpenClerq's default local model
ollama pull llama3.2
ollama run llama3.2 "Summarise this in two sentences: …"Leaderboards tell you little about your own work. Take 20 real tasks from last month — a summary, a letter, a classification — run them through two or three candidate models, and keep the smallest one whose answers you would accept.
Step 3: connect it to OpenClerq
OpenClerq reads its settings from ~/.clerq/.env (or .env in the repository if you run from source). For Ollama:
CLERQ_LLM_PROVIDER=ollama
CLERQ_MODEL=llama3.2
# only if Ollama is not on the default address
CLERQ_OLLAMA_URL=http://localhost:11434For LM Studio, set CLERQ_LLM_PROVIDER=lmstudio and CLERQ_MODEL to the model id LM Studio shows, then start its local server. Restart OpenClerq and the health check reports the model as local.
Then use the chat console's comparison mode: send the same message to your local model and, if you have one configured, a cloud model, and compare the answers, the time taken and the token counts side by side. It is the quickest way to find out which tasks the local model handles well.
Step 4: check that nothing leaves the machine
- Look before sending. OpenClerq's context preview shows the exact text the model will receive, without calling it.
- Check the health endpoint —
curl http://127.0.0.1:18790/healthreportsllm.modeaslocaland names the provider. - Test offline. Turn off Wi-Fi and run a task. If it answers, the model is local.
- Watch outbound connections with the firewall you already use; the runner may still check for updates, which you can usually switch off.
What local models are good and bad at
Good at: summarising, drafting routine correspondence, sorting and tagging, pulling fields out of text, rewriting in a house style, and explaining a rule in plain language.
Weaker at: long multi-step reasoning, very long documents (memory and context limits), anything needing current facts (no web access), and arithmetic — which no language model should be doing anyway. See how to stop AI getting numbers wrong.
Frequently asked questions
Can I run ChatGPT locally?
Not ChatGPT itself — it runs only on OpenAI's servers. OpenAI has released open-weight models, gpt-oss, which run locally through Ollama or LM Studio, and many other open-weight models cover the same everyday office tasks.
Is a local model as good as GPT or Claude?
The largest cloud models are still stronger at difficult reasoning. For routine office text — summaries, letters, classification — a mid-sized local model is often good enough. The only reliable test is your own tasks.
Does running AI locally need an internet connection?
Only to download the runner and the model. After that, the model runs offline.
Ollama or LM Studio?
Ollama if other apps should use the model in the background all day; it is also open source. LM Studio if you want a friendly window for browsing and comparing models. Both expose a local API that OpenClerq can use, and you can install both.