How much GPU memory do you need to run a local AI model?

Updated November 3, 2026.

The GPU memory you need depends on the model's size, its quantization, and your context length, so check the exact model file before buying hardware for it. There is no single number that fits every model, because a small model and a large model can differ by tens of gigabytes, and a longer context window adds memory on top of the model weights themselves.

What actually uses GPU memory

Three things share a graphics card's memory while a model runs: the model's weights at the quantization you downloaded, the key-value cache that holds the context, and a small compute buffer that the runtime uses as working memory. Quantization is the precision of the weights: a 4-bit quantized model uses roughly half the memory per weight of an 8-bit model, and roughly a quarter of a 16-bit model, before overhead and context are added. That is why two downloads of the same model size can need very different amounts of memory: the file size, which reflects the quantization, is the number to check, not just the parameter count.

Context length matters because the key-value cache grows with the number of tokens in the conversation. A short chat uses a small cache, while a long document or a long conversation uses a much larger one. If a model fits on the card at a short context but runs out of memory at a long context, the runtime may offload part of the model to system memory, which is slower, or may fail to load. Start with a short context, confirm the model loads and responds, then increase the context in steps and watch memory use, rather than starting at the maximum context and guessing why it failed.

A practical way to size a model

A useful starting estimate is to take the model file size, add room for context and overhead, and compare that total with the dedicated memory on your card. As a rough guide from community sizing guides, a 7 to 8 billion parameter model at 4-bit quantization is often around 5 to 6 gigabytes, a 12 to 14 billion parameter model at the same quantization is often around 8 to 10 gigabytes, and a 70 billion parameter model at 4-bit is often around 40 gigabytes or more. Those figures are estimates from third-party guides, not specifications from a chip maker, and they vary with the exact quantization, the runtime, and the context length. Treat them as a starting point for choosing a model to test, not as a guarantee.

On Apple Silicon, there is no separate dedicated video memory number: the graphics processor shares unified memory with the system, so the usable budget is total memory minus what the operating system and other applications are using. That makes the same model file behave differently on a Mac with 16 gigabytes of unified memory than on a PC with a dedicated 16 gigabyte card, even though the numbers look the same. On a PC, check dedicated memory, not shared memory, because shared memory is system memory that the graphics processor can borrow, and it does not behave like dedicated memory for a large model.

If the model does not fit

If a model does not fit, the simplest fixes are to choose a smaller model, a lower quantization, or a shorter context, in that order. A smaller model that runs entirely on the graphics card will usually feel faster than a larger model that is partly offloaded to system memory, even if the larger model is more capable on paper. A lower quantization reduces memory use but can reduce quality, so test the smaller or more quantized model on your actual tasks, such as summarising a document or answering questions from your notes, before committing to it.

Check the runtime's own report of what is loaded, rather than guessing from file size alone. Ollama's command line tool, for example, can list which models are currently loaded into memory, and the project's frequently asked questions explain where models are stored and how to point the model directory elsewhere. Use those reports to confirm that the model you intended to run is the one actually loaded, and how much memory it is using, before you buy a larger card or a second card. Hardware is the last step, not the first, and a test run on the machine you already own will tell you more than a specification sheet.

Frequently asked questions

How much memory does a 7 billion parameter model need?

As a rough estimate, around 5 to 6 gigabytes at 4-bit quantization, plus room for context and overhead. Check the exact model file size and test at your intended context length, because the exact figure varies.

Does a longer context use more GPU memory?

Yes. The key-value cache grows with context length, so a model that fits at a short context may need more memory, or may offload to system memory, at a long context.

Is shared GPU memory the same as dedicated memory?

No. Shared memory is system memory the graphics processor can borrow, and it does not behave like dedicated memory for a large model. Compare model size with dedicated memory on a PC, or with available unified memory on Apple Silicon.

Sources

Sources checked November 3, 2026. Figures that depend on your exact model, runtime or machine are estimates; check the source or test on your own hardware.

Adults only (18+). Private uncensored AI · Terms · Privacy · All guides