# Running AI Models Locally by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm) In [What Does Open Mean in Open AI Models?](https://www.kirupa.com/ai_fundamentals/what_open_means_in_ai_models.htm), Crumb & Co.'s owner learned that she could download a model's weights and run **Biscuit**, the bakery's help desk bot, on hardware the bakery controls. Before anybody buys a GPU server, though, Wren, the bakery's web developer, wants to try the idea on a laptop. Wren tinkers with Biscuit's instructions a lot, sometimes on a train where the Wi-Fi comes and goes, and would rather not send real customer questions to anyone else's servers while testing. So the plan is a prototype: Biscuit, running on a model that lives entirely on Wren's laptop. In this article, we'll look at what running a model locally gets us and what it costs, figure out whether a model will fit with some quick memory math, tour the tools people use at the time of writing, and then get a model running with Ollama and talk to it from a few lines of code. By the end, we'll know when local makes sense and when an API is still the better call. Onwards! ## Why Run a Model Yourself? Running a model locally means the model's weights sit on our own computer, and our own computer does the math for every token. Here is how that changes the trip a message takes: ![Two ways a message can travel. With a hosted model, the Biscuit prototype on a laptop sends each question over the internet to an AI provider, which runs the model and sends the reply back. With a local model, everything happens inside Wren's laptop: the Biscuit prototype sends the question to the qwen3.5:4b model on the same computer and gets the reply back with no internet involved.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_where_it_runs.png) Look at the bottom path. Wren's question goes from the prototype to a model on the same laptop and comes right back, so it never crosses the internet. Once the model is downloaded, that one change brings several nice things with it: - Messages stay private. [Ollama's FAQ](https://docs.ollama.com/faq) says it plainly: ***"We don't see your prompts or data when you run locally"***. - We can work offline, since no internet is needed after the download. [LM Studio's docs](https://lmstudio.ai/docs/app/offline) say it ***"can operate entirely offline, just make sure to get some model files first"***. - There's no per-token bill. We pay for the computer and its electricity, so Wren can run the same test a hundred times without watching a meter. - We're free to experiment. We can swap models, change settings, and keep using a model for as long as we like, since nobody can retire it out from under us. Now for the honest part. The models that fit on a laptop are small, and as we saw in [Small vs. Large Model Trade-offs](https://www.kirupa.com/ai_fundamentals/small_vs_large_models.htm), small models stumble sooner on long instructions and niche knowledge than the big frontier models behind the major APIs. Our hardware decides which models we can run at all and how fast they answer. And we're the ones installing, updating, and babysitting everything, which is work a provider normally does for us. The first of those limits to check is memory, so let's start there. ## Will It Fit? Memory Math A model's weights have to sit in fast memory the whole time it runs, so the first math is short: the number of parameters times the bytes per parameter. Here is Meta's Llama 3.1 8B, which has about 8 billion parameters, stored a few different ways: ![Llama 3.1 8B has about 8 billion parameters, and its weights take 8 billion times the bytes per parameter. At 16 bits, or 2 bytes each, that's 16 GB. At 8 bits, 1 byte each, it's 8 GB, and the real file is 8.54 GB. At 4 bits, half a byte each, it's 4 GB, and the real file is 4.92 GB, a little more than the math.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_weights.png) Models are usually shared at 16 bits (2 bytes) per number, which makes the weights 16 GB. Storing each number in fewer bits shrinks that, and that trick is called **quantization**. As we saw in [Why Training Takes a Data Center](https://www.kirupa.com/ai_fundamentals/data_centers_and_training.htm), it rounds every weight onto a coarser scale, so each one ends up a tiny bit wrong. At 8 bits, the weights take about 8 GB, and at 4 bits, about 4 GB. The real 4-bit file people download is [4.92 GB](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF), a little bigger than our math, because it also stores a scale for each small group of weights and keeps a few sensitive parts at higher precision. [Quantization](https://www.kirupa.com/ai_fundamentals/quantization.htm) gets its own article later, and it walks through exactly what gets lost. The weights aren't the whole story, though. While the model runs, it also needs working memory, and the biggest piece is the KV cache, the notes on every token that we met in [How Transformers and Attention Work](https://www.kirupa.com/ai_fundamentals/how_transformers_work.htm). Those notes grow with the length of the conversation: ![Memory for Llama 3.1 8B at 4 bits grows with the conversation. The 4.9 GB of weights stay the same, and the KV cache notes grow with the tokens: about 0.5 GB at 4K tokens for 5.5 GB in total, about 4.3 GB at 32K tokens for 9.2 GB, and about 17.2 GB at 128K tokens for 22.1 GB, far more than the weights themselves.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_working_memory.png) Look at the 128K bar. [Meta's published settings](https://arxiv.org/abs/2407.21783) give Llama 3.1 8B 32 layers, each with 8 sets of keys and values that hold 128 numbers apiece. At 2 bytes per number, our own arithmetic gives 128 KB of notes per token. A 4K-token conversation adds about half a gigabyte, but filling the model's full 128K context adds about 17 GB, which is much more than the 4-bit weights themselves! That's why local tools are careful about context length. At the time of writing, [Ollama's docs](https://docs.ollama.com/context-length) say it picks a default based on the GPU memory it finds: 4K tokens below 24 GB, 32K tokens from 24 to 48 GB, and 256K tokens above that. ## Unified Memory vs. GPU Memory Where that memory lives matters as much as how much there is. Most PCs with a graphics card have two separate pools of memory, while Apple silicon Macs have one: ![Two kinds of computer. A PC with a graphics card has two pools of memory: the CPU uses RAM on the main board, and the graphics card has its own VRAM next to the GPU, so data gets copied over the PCIe slot. A Mac with Apple silicon puts the CPU and GPU on one chip that shares a single pool of unified memory, with no copying between pools.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_two_pools.png) On a PC, the CPU uses the computer's main memory (RAM), and the graphics card has its own memory (VRAM). A model runs at full speed only if it fits in VRAM, which is 24 GB on an RTX 4090 and [32 GB on an RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/). If it doesn't fit, tools like Ollama can run part of the model on the CPU from regular RAM, and replies get slower. Apple's chips use what Apple calls [unified memory](https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/): the CPU and GPU share one pool, so the GPU can use a big share of the Mac's memory without copying anything between pools. At the time of writing, a [MacBook Air](https://www.apple.com/macbook-air/specs/) starts with 16 GB, a [MacBook Pro](https://www.apple.com/macbook-pro/specs/) goes up to 128 GB, and a [Mac Studio](https://www.apple.com/mac-studio/specs/) can be configured with up to 512 GB. Here's what that means for three sizes of Llama 3.1, using the 4-bit files from [Ollama's library](https://ollama.com/library/llama3.1/tags): ![Which 4-bit Llama 3.1 models fit in which memory. The 8B model, at 4.9 GB, fits on a 16 GB MacBook Air, a 24 GB RTX 4090, a 32 GB RTX 5090, a 128 GB MacBook Pro, and a 512 GB Mac Studio. The 70B model, at 43 GB, fits only on the MacBook Pro and the Mac Studio. The 405B model, at 243 GB, fits only on the Mac Studio.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_what_fits.png) Read it row by row. The 8B model fits on everything. The 70B model is too big for either graphics card, but it fits on a 128 GB MacBook Pro. Even the 405B model fits on the biggest Mac Studio. Fitting isn't the same as fast, though. As Quantization explains, a model streams its weights through memory for every token it writes, so how many gigabytes per second the memory can move sets a speed limit. That means a huge model can fit on a modest chip and still write each token very slowly. ## The Tools at the Time of Writing There's a whole toolbox for running models locally, and the pieces stack together in layers. Here is how they relate: ![The local AI toolbox in layers. At the top, our code, biscuit.mjs, sends OpenAI-style requests to any of three servers: Ollama on port 11434, LM Studio on port 1234, or vLLM on port 8000. Ollama and LM Studio sit on top of the engines llama.cpp, which runs GGUF files, and MLX, which runs on Apple silicon. vLLM has its own engine, built for GPU servers. At the bottom, Hugging Face holds the model files: GGUF, MLX, and safetensors.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_tools.png) Starting from the bottom, here is what each one does: - [Hugging Face](https://huggingface.co/models) is where most open models live. Every model has a [model card](https://huggingface.co/docs/hub/model-cards), a README that describes the model, its intended uses and limitations, its training data, its evaluation results, and its license. - [llama.cpp](https://github.com/ggml-org/llama.cpp) is an open source engine, written in C and C++, that runs models on everything from laptops to servers. Georgi Gerganov, who started llama.cpp, also created [GGUF](https://huggingface.co/docs/hub/gguf), the file format that packs a model's weights and its settings into one file. - [MLX](https://github.com/ml-explore/mlx) is Apple's framework for machine learning on Apple silicon, built around unified memory. Its [mlx-lm](https://github.com/ml-explore/mlx-lm) package can run thousands of models from Hugging Face with a single command, many of them prepared by the MLX Community. - [Ollama](https://ollama.com/) downloads, manages, and runs models with a single command, and serves them on a local API at `localhost:11434`. It runs on macOS, Windows, and Linux. - [LM Studio](https://lmstudio.ai/) is a desktop app with a chat window, a model search powered by Hugging Face, and a local server at `localhost:1234`. It runs GGUF models with llama.cpp everywhere, and MLX models on Apple silicon Macs. - [vLLM](https://docs.vllm.ai/), originally from UC Berkeley, serves models to lots of people at once on GPU servers. It manages the KV cache cleverly with a technique called PagedAttention and serves at `localhost:8000` by default. Notice that every server in that list speaks the same language. Ollama, LM Studio, llama.cpp, and vLLM all offer endpoints that follow the format of OpenAI's API, so code written for a hosted model can talk to a local one by changing an address. We'll lean on that in a moment. When we browse for a model file, its name usually tells us most of what we need to know. Here is the 4-bit Llama 3.1 8B file from our memory math, taken apart: ![The file name Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf taken apart. Meta-Llama-3.1 is Meta's Llama 3.1 family. 8B means 8 billion parameters. Instruct means it was tuned to follow instructions. Q4_K_M is llama.cpp's 4-bit mix, in its medium size. gguf is the GGUF file format.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_file_name.png) ***Instruct*** means the model went through the post-training we met in [What Exactly Is an LLM?](https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), so it follows instructions and chats instead of just continuing text. ***Q4_K_M*** is llama.cpp's medium-sized mix of 4-bit quantization. Before we build anything on a model, there's one more thing to check, and it's the license. As we saw in What Does Open Mean in Open AI Models?, open weights don't mean anything goes. The model card lists the license, and so does `ollama show` for a downloaded model. At the time of writing, [Qwen 3.5](https://huggingface.co/Qwen/Qwen3.5-4B) and [Gemma 4](https://huggingface.co/google/gemma-4-E4B-it) use Apache 2.0, which lets the bakery use and change them with very few strings, while [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) comes with Meta's own community license. ## Hands-On: Biscuit on a Laptop Let's get Wren's prototype running. We'll use Ollama, since it goes from nothing to a working model in the fewest steps. First, install it from [Ollama's download page](https://ollama.com/download), which has installers for macOS, Windows, and Linux. On macOS or Linux, that page also offers a one-line install from the terminal: ```bash curl -fsSL https://ollama.com/install.sh | sh ``` Next, we download a model. We'll use Alibaba's Qwen 3.5 in its 4-billion-parameter size. It's small enough for most laptops, with a download of 3.3 to 4 GB depending on your computer, and it uses the Apache 2.0 license: ```bash ollama pull qwen3.5:4b ``` Now we can chat with it. Qwen 3.5 likes to write out its thinking before it answers, which is the reasoning we met in [What Does It Mean for AI to Reason?](https://www.kirupa.com/ai_fundamentals/what_ai_reasoning_means.htm), and the `--think=false` flag turns that off so replies come back right away. Here is a short session, start to finish: ![A terminal session. The command ollama pull qwen3.5:4b downloads the model, showing pulling manifest, a 4.0 GB download at 100 percent, writing manifest, and success. Then ollama run qwen3.5:4b --think=false opens a chat. At the >>> prompt, the question In one sentence, what makes sourdough bread sour? gets the answer Sourdough bread is sour because the fermentation process carried out by wild yeast and lactic acid bacteria in the dough produces organic acids, primarily acetic acid and lactic acid. Typing /bye ends the chat.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_terminal.png) Once the download finished, nothing in that session needed the internet, and typing `/bye` ended the chat. While a model is loaded, `ollama ps` shows how much memory it's using and where it's running. Here is what it printed on the M4 Max MacBook Pro with 48 GB of memory that this article was tested on: ```text NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL qwen3.5:4b 7f3251fa6878 4.3 GB 100% GPU 32768 mlx 4 minutes from now ``` The model is using 4.3 GB, all of it on the GPU. The context is 32,768 tokens, which matches Ollama's rule for 24 to 48 GB of GPU memory, and the `mlx` runner means that on this Mac, Ollama ran the model with Apple's MLX. The last step is talking to the model from code. Ollama's API speaks the same format as OpenAI's, so we can use OpenAI's own JavaScript library and point it at our own computer. In an empty folder, run `npm install openai`, and then save this as `biscuit.mjs`: ```js import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:11434/v1/", apiKey: "ollama", // the library needs a key, and Ollama ignores it }); const response = await client.chat.completions.create({ model: "qwen3.5:4b", reasoning_effort: "none", messages: [ { role: "system", content: "You are Biscuit, the help desk assistant for the Crumb & Co. bakery. " + "Keep answers under 60 words. Bakery facts: rye bread today, 2 loaves left.", }, { role: "user", content: "Do you have rye bread today?" }, ], }); console.log(response.choices[0].message.content); ``` Two lines make this local. The `baseURL` points at Ollama on our own computer instead of at a provider, and `reasoning_effort: "none"` is the code version of `--think=false`. Everything else is the same code we'd write for a hosted model. When Wren runs `node biscuit.mjs` and wires the result into a local copy of Biscuit's chat window, here is one of the replies that comes back: ![A local copy of the Biscuit chat window, running on Wren's laptop. A note in the chat reads: Running on Wren's laptop. Wren asks, Do you have rye bread today? Biscuit replies, Yes! We are baking fresh rye bread today. However, only 2 loaves remain in our oven. Please grab them quickly before they sell out! A handwritten note flags that reply: Not in our facts!](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_chat_rye.png) With the model already loaded, replies took under a second on that MacBook Pro, and this one got the important part right: 2 loaves. Read it again, though. Nothing in the bakery facts says the rye is being baked fresh, and nothing says the loaves are sitting in an oven. The small model filled in plausible details on its own, and that's exactly what Wren would catch by testing it against real questions before any customer sees it. (Why models do this gets its own article, [Why AI Makes Things Up](https://www.kirupa.com/ai_fundamentals/why_ai_makes_things_up.htm).) ## When Local Makes Sense So should Biscuit run locally for real? It depends on the job, and here is how the trade usually shakes out: ![Two lists. Local makes sense when data must stay put, no internet is around, you test a lot, a small model is enough, or you own the hardware. An API is better when you need the best models, when traffic is big or spiky, when you'd rather have no servers to babysit, or when you need long contexts.](https://www.kirupa.com/ai_fundamentals/images/running_models_locally_decision.png) Hold the bakery up to those two lists. Wren's prototyping lands on the local side, since Wren tests a lot and wants customer questions to stay put. Answering real customers lands on the API side for now, so Biscuit keeps doing that through a hosted model. If the owner ever decides that running the real thing in-house is worth a GPU server, a serving tool like vLLM is built for exactly that job. ## Conclusion Running a model locally comes down to one idea: the weights live on our computer, and our computer does the math. That keeps messages on the machine, works without internet, and swaps the per-token bill for hardware we already have. In exchange, we're limited to models that fit, which starts with parameters times bytes per parameter plus room for the KV cache, and we're the ones keeping everything running. Ollama and LM Studio make the first try take minutes, llama.cpp and MLX do the heavy lifting underneath, Hugging Face is where the files and their licenses live, and vLLM takes over when it's time to serve lots of people. Browsing a model library also turns up names like ***gemma4:26b***, which [Ollama's page](https://ollama.com/library/gemma4) describes as a mixture of experts with only 4B active parameters. What does that mean, and why would one model have two sizes? That's where we're going next in [Types of AI Models](https://www.kirupa.com/ai_fundamentals/types_of_ai_models.htm).