This morning, I was working on an exploratory project where five AI models vote on whether resources requested by an agent qualify as legitimate build work.
I started with Claude's Opus 5, GPT-5.6-sol, GLM 5.2, and Kimi 3—models from Anthropic, OpenAI, Z.ai, and Moonshot.
I did not choose them to create a balanced sample of the AI industry. They are simply the models I turn to every day. I use them side by side for research, writing, strategy, and technical work.
Four voters created an obvious problem: they could split 2–2. So I asked Claude to recommend a non-Grok model for the fifth seat.
It chose Gemini 3.1 Pro.
The choice was reasonable. The rationale caught my attention:
“The roster is two US closed labs and two Chinese labs; Google is the only member of the big-three US labs absent.”
Claude went on to argue that adding another Chinese model would overrepresent one bloc, while Google would create a better balance. It even named two credible frontier models it rejected on those grounds—DeepSeek V4 Pro and Qwen 3.7 Max—because either “would make it three Chinese labs to two US.”
But I had never mentioned nationality.
I had not explained why I used those four models at all. Claude interpreted my everyday working set as intentional geopolitical composition.
That is what makes bias in AI so sneaky.
By bias, I do not mean that mentioning a company's national origin is inherently wrong. I mean that Claude introduced an unrequested frame, made that frame relevant to the decision, and then reasoned inside it.
The recommendation was not the surprising part
Gemini may be a perfectly good fifth voter. Claude also gave sensible non-geographic reasons for the choice: Google was a major vendor missing from the group, Gemini would not have its own competing arm to favor, and its cost was attractive.
The interesting behavior happened before the recommendation.
Claude could have asked why I chose the original four or compared candidates across several explicit dimensions. Instead, without access to my selection rationale, it supplied one. Nationality became a relevant category, geopolitical balance became an objective, and the recommendation followed logically from that frame.
Nothing in the final answer looked irrational. That is precisely the problem.
If I had recorded only “Claude recommends Gemini,” I would have missed the assumption underneath it. A plausible conclusion can conceal a different interpretation of the problem from the one its user had in mind.
One response does not prove that Claude has a generalized political bias. Nor does it show that considering geography is always inappropriate. It shows something narrower and more useful: when a model encounters missing context, it may fill the gap with categories and priorities of its own.
Bias can enter before the answer
We often look for AI bias in visible outcomes. Who received the opportunity? Which candidate was ranked first? What recommendation was made? Did the model produce an offensive or demonstrably false statement?
But bias can enter earlier, when a system decides:
- What kind of problem this is
- Which attributes are relevant
- Which comparisons deserve attention
- What “balanced,” “safe,” or “representative” means
- Which tradeoffs should carry the most weight
Once those choices are made, the reasoning downstream can be fluent, careful, and internally consistent.
This is not unique to AI. Humans also use mental shortcuts to fill in missing information. NIST's work on AI bias notes that implicit biases affect how people perceive information and fill unknowns, and that awareness by itself does not guarantee control over them. AI systems are built from human data, human choices, and human feedback, then placed back into human decision-making loops. The biases of the system and the biases of its users can compound rather than cancel each other out.
The practical danger is not only that an AI gives us a bad answer. It is that the answer arrives with an unstated framing that feels neutral because we never saw an alternative.
One fluent answer can feel like the answer
Every day, more of us use AI to help write proposals, evaluate options, interpret research, review code, make purchases, plan careers, and think through personal decisions.
The interface encourages a simple pattern: ask one question, receive one polished response.
That response may contain caveats, but it rarely arrives labeled as one model's perspective. It arrives as an answer.
Fluency makes this easy to overlook. When the reasoning is well organized and the conclusion sounds sensible, the model's framing can disappear into the prose. We may challenge a suspicious fact while accepting the categories used to organize the entire problem.
Relying on one model therefore does not give us a neutral perspective. It gives us one model's interpretation of the world, delivered so smoothly that it can feel like the interpretation.
Why AI models disagree
Different models are trained by different organizations, on different mixtures of data, with different feedback systems, model specifications, and product goals. Even when their capabilities look similar, they can prioritize different values when a prompt leaves room for interpretation.
Anthropic recently stress-tested twelve frontier models across more than 300,000 value-tradeoff scenarios. The researchers reported meaningful behavioral differences between at least one pair of models in more than 220,000 of them. They also found provider-level patterns: different model families tended to prioritize some values more heavily than others.
That does not mean every response represents a stable ideology. Outputs also vary with wording, context, tools, sampling, and the information a model can access. Some disagreement is meaningful; some is noise.
But disagreement tells us where to look.
Recent research has treated cross-model disagreement as a diagnostic signal rather than a defect to be averaged away. In one study of AI-assisted public-comment analysis, differences among models revealed genuine interpretive ambiguity—differences that ordinary accuracy checks could not see. Other research has found that a second model's disagreement can help identify cases where the first model is confidently wrong.
The point is not that the dissenting model must be correct. The point is that disagreement interrupts the illusion that the first answer was inevitable.
Two kinds of contrast
We have seen the value of contrast in another experiment too.
For ThinkRun's 40 Ways experiment, we gave GPT-5.6 Sol and Terra the same repository context and creative brief forty times across medium and high reasoning effort. Thirty-three runs produced a webpage. Seven produced nothing. Medium effort completed 19 of 20 runs; high effort completed 14 of 20, despite costing more overall.
The page I ultimately chose came from a high-effort Sol run. Completion rate did not choose it. Cost did not choose it. A human still had to decide which result best understood the work and was worth shipping.
That experiment exposed variation among repeated runs. The fifth-seat incident exposed variation in how models can frame the same decision. The two practices answer different questions:
- Repeating a prompt helps reveal how much an answer can vary even when the setup appears fixed.
- Comparing models helps reveal which assumptions, priorities, and categories are specific to one model's interpretation.
Our own measurement systems need the same scrutiny. Looking at work on this exploratory project, today's usage record shows activity across Claude, Codex, and Pi. It also has a few gaps, so it is not a complete measure of every model interaction.
A token-only chart can still make a multi-harness workflow look like a single-model workflow, especially when cache reads dominate the total or one provider's records are easier to measure. Missing telemetry is not zero usage. The categories and measurements we use to describe AI can introduce their own blind spots before the analysis begins.
Why I use multiple AI models
I have started running the same prompt through multiple models for important decisions and outputs.
To be precise, another model did not uncover the assumption in this particular incident. Claude made it visible by showing its rationale. The incident reinforced why I want more than one perspective available before I treat any model's framing as neutral.
The benefit is not that several models magically eliminate bias. They may share training data, cultural assumptions, safety conventions, and blind spots. Five correlated opinions are not the same as five independent ones.
The benefit is contrast.
When two strong models answer the same question differently, the gap gives me something concrete to inspect:
- Which assumptions did each model introduce?
- How did each model define the problem?
- Which facts did one emphasize while another ignored them?
- Did either model answer a different question from the one I asked?
- Is the disagreement factual, strategic, ethical, or merely stylistic?
- What evidence would resolve it?
I am not necessarily looking for a majority vote. Consensus can reflect shared training patterns just as easily as independent confirmation. And a lone dissenting model can be right—or simply confused.
The useful artifact is the disagreement itself.
It tells me where human judgment, better context, or primary-source verification is needed.
A practical multi-model workflow
For consequential work, my process is becoming:
1. Ask independently
Give the same question and relevant context to more than one model. Do not tell the second model what the first one concluded. That reduces the chance that it merely anchors on or politely critiques the first answer.
2. Compare the frames before the conclusions
Before deciding which answer is better, ask what each model assumed. Look for categories, stakeholders, risks, and objectives that appeared without being requested.
3. Isolate the disagreement
Turn a broad conflict into a specific question. Are the models relying on different facts? Applying different values? Predicting different consequences? Interpreting an ambiguous instruction differently?
4. Verify outside the models
For factual claims, go to primary sources. Multiple models can share the same outdated information or repeat the same widely circulated error. Cross-model agreement is a reason for confidence only when the models' claims survive external verification.
5. Keep the human decision visible
Models can surface tradeoffs, but they should not quietly decide which values matter on our behalf. When a disagreement is about priorities rather than facts, the resolution belongs to the person or institution accountable for the decision.
This approach is unnecessary for every email or routine task. It is most valuable when a decision is expensive, difficult to reverse, personally important, or dependent on ambiguous judgment.
Multiple models are not a cure for bias
There are limits to this approach.
Models are not fully independent witnesses. They learn from overlapping public information and are shaped by similar incentives to be helpful, safe, and persuasive. Asking more models can manufacture false confidence if we treat agreement as proof.
It can also manufacture false disagreement. Prompting one model to “be the contrarian” may produce objections because disagreement was requested, not because the evidence warrants it.
The goal is not to assemble an automated parliament that replaces judgment with a vote. It is to create enough perspective difference that hidden assumptions have a chance to become visible.
That is the opportunity this incident made clear.
Claude's answer did not simply tell me which model it preferred. It revealed how Claude had interpreted a set of choices I had made for completely different reasons.
As we rely more heavily on AI for our work and personal decisions, the question is not only whether a model's answer is accurate.
It is also:
What blind spots, assumptions, and perspectives are we adopting by default when we rely on only one model?
Questions about using multiple AI models
Why do AI models give different answers to the same question?
Models are trained by different organizations using different data mixtures, feedback systems, specifications, and product goals. Their answers can also change with wording, context, tools, and sampling. A disagreement may reveal a factual gap, an ambiguous prompt, a different value judgment, or ordinary output variability.
Does using multiple AI models eliminate bias?
No. Models can share data, assumptions, and blind spots. Using several models is valuable because it creates contrast: disagreement shows you where to inspect the framing, improve the context, or verify a claim outside the models.
How should I compare answers from multiple AI models?
Ask the models independently, compare their assumptions before choosing a conclusion, isolate the exact point of disagreement, and verify factual claims against primary sources. When the difference is about values rather than facts, keep the final human judgment explicit.
Sources
- NIST: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence
- Anthropic: Stress-testing model specs reveals character differences among language models
- When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
- Cross-Model Disagreement as a Label-Free Correctness Signal