How do you size a GPU for running a local AI model?

Updated November 2, 2026. Facts checked November 2, 2026.

Short answer

To size a GPU for running a local AI model, match the GPU's video memory to the model's size at your chosen quantisation, plus headroom for context, then check speed and compatibility before buying. A practical rule is that a 4-bit quantised model needs roughly half a gigabyte per billion parameters, plus 10 to 20 percent overhead. That gives about 6 gigabytes for an 8-billion-parameter model and about 40 gigabytes for a 70-billion-parameter model.

The memory math, step by step

Start with the parameter count, then multiply by the bytes per parameter at your quantisation. At 4-bit quantisation, each parameter takes about half a byte, so a 7-billion-parameter model needs roughly 3.5 gigabytes for weights alone, and in practice about 5 to 6 gigabytes once overhead is included. At 8-bit quantisation the figure roughly doubles, and at 16-bit precision it roughly quadruples, which is why quantisation choice dominates GPU sizing more than any other single decision.

Then add context memory. The key-value cache grows with context length and batch size, so a model that fits comfortably at a short context can run out of memory at a long context. Guides report that quantising the key-value cache can cut context memory by about half, but the exact saving depends on the runtime and settings. The safest check is the model's actual size on disk after download, since that is a good proxy for resident memory, plus a margin for the context you plan to use.

A sizing table for common model sizes

For 4-bit quantisation, published guides agree closely on these tiers. Models of 1 to 3 billion parameters need about 2 to 4 gigabytes and run on almost any modern GPU, including laptops. Models of 7 to 8 billion parameters need about 6 to 8 gigabytes and are comfortable on entry-level discrete GPUs. Models of 13 to 14 billion parameters need about 9 to 10 gigabytes. Models of 20 to 34 billion parameters need about 16 to 24 gigabytes, which points to high-end consumer cards. Models of 70 billion parameters need about 40 to 48 gigabytes, which generally means a professional card or multiple consumer cards.

These figures assume 4-bit quantisation. If you plan to run a model at higher quantisation, or with a very long context window, move up one tier. If the model does not fully fit in video memory, runtimes such as Ollama split layers between GPU and system memory, which still runs but at much lower speed, often 5 to 15 tokens per second instead of 40 to 120 tokens per second on a full GPU fit. A split model is usable for testing, but painful for daily work.

Beyond memory: speed, compatibility, and the CPU fallback

Memory decides whether a model runs well; the GPU itself decides how fast it generates. A high-end consumer card can generate roughly 80 to 120 tokens per second on a small 4-bit model, while a CPU-only machine may manage only a few tokens per second on the same model, which feels like seconds per word. Memory bandwidth, not just memory size, drives that speed, which is why two cards with the same memory can perform differently.

Compatibility matters too. Ollama supports NVIDIA cards through CUDA, AMD cards through ROCm, and Apple Silicon through Metal, with different levels of maturity, so check the runtime's GPU support for your exact card before buying. On Apple Silicon, there is no separate video memory: the unified memory pool serves as both system and GPU memory, so total system memory is the sizing number. A machine with 16 gigabytes of unified memory can run small models well, while 32 gigabytes or more opens up the mid-size range.

A buying decision in order

First, choose the model size that fits your actual task, not the largest model you can imagine, since larger models cost more memory, more power, and more waiting for every answer. Second, choose a quantisation, usually 4-bit for a first machine. Third, read the model's disk size and add headroom for context. Fourth, pick a GPU whose memory clears that total with room to spare, rather than exactly matching it. Fifth, confirm the runtime supports the card and that the power supply and cooling in the machine can handle it.

If the budget does not reach the GPU for the model you want, the honest alternatives are a smaller model, a lower quantisation, a CPU run for light use, or a machine with unified memory. What is unknown from guides alone is the exact tokens-per-second figure for your combination of card, model, quantisation, and prompt, since benchmarks vary widely and depend on settings. Treat any single benchmark number as an illustration, not a promise.

Frequently asked questions

How much VRAM do I need for a 7 billion parameter model?

About 5 to 6 gigabytes at 4-bit quantisation, or about 8 gigabytes of card memory as a comfortable buying target once context headroom is included.

Can I run a local model without a GPU?

Yes, for small models. Ollama and llama.cpp run on CPU and system memory, but generation is much slower, often a few tokens per second, so a GPU is strongly recommended for regular use.

Does more VRAM always mean faster generation?

No. VRAM decides which models fit, while memory bandwidth and the GPU itself decide speed. A card with more memory but lower bandwidth can be slower than a smaller, faster card on a model that fits both.

Sources

• Ollama — official site and documentation
• Ollama — source and documentation
• Hugging Face — documentation
• NVIDIA — documentation

This page is general information only, not purchasing or financial advice. Memory figures are approximate and vary with quantisation, context length, and runtime; check the model card and runtime documentation for the exact model you plan to run.