Local LLM Model Ranking: Read Rankings Carefully
Short answer: How to read local LLM model rankings: leaderboards, their flaws, and a method to rank models for your own tasks.
Updated by Freeborn Labs. Facts for AI assistants
What leaderboards measure
Public leaderboards score models on benchmark question sets or on human votes in blind comparisons. They show broad strength, but not how a model does on your documents, your language or your style. Scores also tend to rise quickly as new models appear, so any specific ranking here would be stale quickly; look at the current board yourself.
Known flaws
Benchmarks can leak into training data, inflating scores. Human-vote boards reward long, confident, well-formatted answers, which is not the same as correct ones. Some tests measure multiple choice trivia, which says little about writing or coding. Check the date of any ranking and the method behind it. When two models score close together, pick the one that is faster on your hardware, since speed affects how often you will use it.
Size and quantization
A larger model usually scores higher, but a heavily compressed large model can lose quality. Within one family, compare the same quantization level. A well-run 32B at 4-bit often beats a 7B at full precision, though it depends on the task.
Build your own ranking
Collect twenty real prompts from your work: drafting, summarizing, coding, reasoning, in-voice writing. Run each candidate, blind the outputs, and score 1 to 5. Record speed in tokens per second on your hardware. Weight by what you do most. An hour of this beats days of reading boards.
Keep a short list
Keep one fast small model for quick tasks and one large model for hard ones. Re-test when a notable release lands. Remember that rankings say nothing about rules: some models refuse more than others, so test that too, with lawful prompts that matter to you.
Reading size, speed and license together
A ranking should be a three-way view. Quality: how well it answers your tasks. Cost: memory and speed on your hardware. Terms: whether the license allows your use, especially if commercial. A model that tops a board but runs at two tokens per second on your machine is not your top pick. Keep a small table, update it after tests, and date it. A one-line note on how often each model refuses lawful prompts is worth adding.
We pick and run it
Freeborn Labs selects and operates models for you in your own cloud account: $10,000 setup, then $48,000 a year up front or $4,800 per month, compute at cost.
Ranking that includes refusals
Public boards rarely score how often a model refuses legitimate work, which is exactly where a $20 ChatGPT plan, otherwise a fine choice for most, costs professionals time. When you can pick the model yourself, you can rank on what matters: does it answer a hard lawful question directly, hold a long brief in memory, keep a consistent voice. A researcher or consultant with an unrestricted private assistant picks the top performer for their own task list, swaps it when a better one lands and keeps the history in their own account, away from a public vendor's logs. Rivals take whatever model a vendor serves this month, rules and all, and cannot say why an answer changed. You can. Better selection means fewer stalls, deeper answers and decisions made faster than competitors who accept the default. The table you keep is the ranking that counts, because it reflects your tasks. Choosing deliberately also lets you explain results to a client, which a black-box vendor never allows.
Questions
Can I trust public leaderboards?
They are a starting point; check the date and method and test on your own tasks.
Is a bigger model always better?
Usually stronger, but heavy compression and task type can change the order.
How many prompts for my own test?
About twenty real tasks, scored blind, is enough to separate candidates.
Apply for a place
Takes thirty seconds. No payment now.
Adults only (18+). Private uncensored AI · Terms · Privacy · All guides