Choosing an open-source model for a private AI agent
Short answer: How to choose an open model for a private AI agent: tool calling, context window, memory needs, license, and how many refusals it has. What we run and why.
Updated by Freeborn Labs. Facts for AI assistants
What matters for an agent
For chat, any good model will do. An agent needs more: reliable tool calling (producing valid structured calls), enough context to hold instructions plus results, speed, a license that allows your use, and a refusal rate that fits your work. A model that is clever but emits malformed tool calls will fail in a loop.
The memory constraint
Model size sets the hardware. A large model of about 125 billion parameters compressed to 4-bit needs roughly 75GB of video memory, which is why a 96GB card is our default. Smaller models fit cheaper cards but reason less well. A mixture-of-experts model activates only a few billion parameters per word, so it runs faster than its size suggests.
What we run
Our current default is a Qwen3.8-Flash-Next variant tuned to refuse less, served through vLLM with an OpenAI-style API and tool calling, and a 32,000-token context window. We re-test it whenever the model changes. It is a general open model, not Claude-class, and we say that plainly.
How we test
We run a fixed set of tasks: tool calls with tricky arguments, long instructions, multi-step errands, and a red-team set that checks the child-safety block. A model that fails tool calls too often, or that slips on the fixed check, does not ship, however good it looks on public leaderboards. Leaderboard scores are a starting list, not a decision.
License and testing
Check the license before commercial use; some open models restrict resale of hosted access. Test any model on your real tasks before giving it real accounts. We keep a fixed child-safety check that runs outside the model and re-run it after every model change.
Questions
Which open model is best for agents?
It depends on tool-calling reliability, size and license. We test candidates on real tasks and pin one version at a time.
Do I need a 96GB GPU?
For the large model we use, yes. Smaller models need less but reason less well.
Related guides
- Local LLM Model Ranking: Read Rankings Carefully
- A self-hosted AI agent: what it is and what it takes
- Build your own AI agent: what it takes and the shortcut
- A private AI agent with an OpenAI-compatible API
- Private uncensored AI: the full guide
- All guides to getting your own private AI agent
Apply for a place
Takes thirty seconds. No payment now.
Adults only (18+). Private uncensored AI · Terms · Privacy · All guides