How do you size a GPU for a local AI model without overspending?

Updated November 2, 2026. Facts checked November 2, 2026.

Short answer

Size a GPU from the model's file size plus the memory needed for context, then choose the smallest card that holds both with headroom. A quantized 8-billion-parameter model needs roughly five gigabytes for its weights, while a 70-billion-parameter model needs roughly forty-three gigabytes, so the model choice decides the card, not the other way around. Buying a card first usually means paying for unused memory or accepting a slow GPU-CPU split.

Start with weights, then add context

A model's weights are the dominant memory cost. At a common 4-bit quantization, a model takes roughly half a gigabyte per billion parameters, plus overhead, which is why an 8-billion-parameter model is about five gigabytes and a 70-billion-parameter model is about forty-three gigabytes. Higher precision roughly doubles or quadruples that figure, so the quantization tag matters as much as the parameter count.

Context memory sits on top of the weights. The key-value cache grows with the context window and with the number of requests running at once, so a long context or several concurrent users can add gigabytes even for a small model. NVIDIA's sizing guidance for fine-tuning makes the same point in a different workload: the supported model size depends on the profile, the quantization, and the batch and sequence settings, not on the parameter count alone.

What happens when the model does not fit

When weights plus context exceed the card's memory, the runtime splits layers between the GPU and the CPU, and speed falls sharply. That split is the practical penalty for undersizing, and it is worse than a slightly slower card that holds the whole model. A card that holds the model plus a modest context, with about ten percent headroom, will usually feel faster than a larger card that is still too small for the next model size up.

Hugging Face's single-GPU inference guidance notes that quantized weights reduce memory sharply compared with full precision, and that 8-bit inference requires a GPU with tensor-core support. That guidance is a reminder to check the runtime's requirements, not just the card's memory: a card with enough memory but no supported kernels will not deliver the expected speed.

A practical sizing checklist

First, name the model and its quantization tag, and write down its file size. Second, name the context length you actually need and the number of concurrent requests you expect, and add an allowance for that cache. Third, add overhead for the runtime and the operating system, and round up to the next card tier. If the total lands just over a tier, consider a smaller quantization, a shorter context, or a smaller model before buying a larger card.

Finally, test with your own prompts before committing to a fleet. A sizing table can estimate memory, but it cannot tell you whether a quantized model answers your questions well enough, or whether your context length is really needed. One measured run on the actual card, with the actual model and context, is worth more than a second table.

Before you buy

Write down the model, its quantization tag, its file size, your context length, and your expected concurrent requests, and add them up before shopping. That one page of arithmetic is the sizing decision, and it prevents both overspending and a card that splits to the CPU.

Test the chosen model on the actual card with your own prompts and context before buying more cards. One measured run answers the speed and quality questions that a table cannot.

Frequently asked questions

How much VRAM does an 8-billion-parameter model need?

Roughly five gigabytes for the weights at a common 4-bit quantization, plus memory for context and overhead, so a card with 8 gigabytes is a common starting point for a single user.

Does a bigger GPU always make a local model faster?

Not by itself. Once the model and its context fit in memory, additional memory helps mainly with longer context or more concurrent requests; bandwidth and the runtime matter more for raw speed.

Should you size for the largest model you might run?

Only if you will actually run it. Sizing for a model you never load wastes money, while sizing for the model you run every day, plus headroom for context, is the practical rule.

Sources

• NVIDIA sizing guide — AI vWS Toolkit, fine-tuning large language models
• Hugging Face — efficient inference on a single GPU

This page is general information only, not legal, tax, or financial advice. Confirm the figures for a specific property or system with the official source or a qualified professional.