AI Confidence Gap: Harvard Survey’s AI Literacy & Ethics 🔥
Discover Harvard’s AI confidence gap: AI literacy & AI ethics insights, AI gender gap findings, and how to improve AI in business decisions despite uncertainty.
AI Confidence Is Not the Same as AI Expertise
In AI conversations, the loudest certainty can win. That's handy when a team needs to decide quickly - but it turns into a problem when confidence gets treated like knowledge.
A survey of more than 5,400 Americans, discussed in a report from Harvard Kennedy School's Shorenstein Center, shows how that can play out. Some respondents said they recognized AI leaders who were fictional. Men were more likely than women to claim recognition, but the bigger takeaway is broader: confidence can shape who uses AI, who gets heard, and how organizations decide who counts as "expert."
None of this proves that one gender understands AI better, or that confidence reliably tracks ability. People who missed questions were wrong at similar rates. The difference showed up in something more human: how often they admitted they weren't sure. For employers, product teams, and anyone making AI-related decisions, that distinction matters. In this article, I'll walk through what the survey shows, what it can't establish, and a more practical way to evaluate AI knowledge without rewarding bluster over judgment.
What the survey found about AI confidence
The survey asked Americans about AI, including whether they recognized people associated with the field. The report says 73 percent of male respondents recognized OpenAI CEO Sam Altman. Just over half said they recognized Mira Murati, Susan Milner, and Craig Dalton.
But not all of those names are the same. Murati is a real technology executive - formerly OpenAI's chief technology officer, and (at the time of the report) CEO of Thinking Machines Lab. Milner and Dalton were fictional. So the survey didn't just measure familiarity with AI. It also revealed how willing some people were to claim recognition for names that don't exist.
This wasn't only a "men" issue. The report says 40 percent of women claimed they knew Milner and Dalton, while 38 percent said they recognized Murati. Men were more likely to claim recognition overall, but the pattern of mistaken certainty showed up across the sample.
The confidence gap is not the same as a knowledge gap
The most useful part comes when the researchers compare confidence with factual performance. Men reported using AI more often and, on average, answered more factual questions correctly. But men and women got wrong answers at roughly similar rates. One key difference was how often respondents chose uncertainty: women said they were unsure about an average of 3.3 questions out of 12, versus 2.3 for men.
That matters. Someone who says "I don't know" might be less familiar - or they might simply be careful. Someone who answers confidently might be right, or they might be guessing. A survey can capture the response. It can't tell, for each person, which explanation is behind it.
So the report's value isn't that it settles who knows more about AI. It points to a measurement problem: if confidence is treated like a stand-in for competence, people who are comfortable guessing can look more expert than people who calibrate their answers carefully.
What a survey can - and cannot - tell us
Self-reported recognition is a limited measure. It can hint at perceived familiarity and how willing people are to claim it. But it doesn't, by itself, show how well someone understands machine learning, evaluates model outputs, or uses AI safely at work.
The fictional names make the result easy to remember. That doesn't mean the names become a shortcut for judging people. These findings describe patterns in a survey sample, not the capabilities or behavior of every man or woman. They also don't show that confidence caused higher AI use - or that increased use led to better understanding.
For the full account of the survey and its results, see the WIRED report. The practical takeaway is narrower (and more actionable): when assessing AI knowledge, focus on what someone can explain or do, not just how certain they sound.
Why certainty can look like expertise
AI moves fast, and its vocabulary can make shallow familiarity sound impressive. It's easy for someone to talk smoothly about models, agents, prompts, or automation - even if they can't explain where systems fail, what data is involved, or when it's actually appropriate to use AI.
In a meeting, that kind of fluency can land as credibility. And under deadline pressure, it often goes unchallenged.
This isn't unique to AI. In many workplaces, admitting uncertainty comes with a social cost. People may worry that "I'm not sure" will be interpreted as incompetence - especially when the topic is technical and the language isn't familiar. That cost can be even higher for people who already have to work to establish their credibility.
The flip side is just as real. Confidence can become a performance. A firm answer can move a conversation forward - but without someone checking the assumptions behind it, a team can confuse decisiveness with evidence.
AI's pace makes false certainty more expensive
AI products change quickly, and claims about them often sound more certain than the evidence deserves. A feature might look great in a curated demo and behave differently with real customer data. A model can produce useful drafts and still make factual errors. A vendor's promised productivity may not match the workflow your team actually runs.
In practice, an AI decision usually involves multiple questions:
- Capability: Does the system perform the task accurately enough?
- Fit: Does it work with the organization's data, process, and users?
- Risk: What happens when it makes a mistake or exposes sensitive information?
- Value: Does it improve the workflow enough to justify cost and oversight?
Knowing the latest buzzwords won't answer those questions. Neither will being naturally optimistic or skeptical. What helps most is the ability to say what's known, what's uncertain, and what evidence would change the decision.
Uncertainty is part of competent judgment
AI discussions often frame the future in extremes: it'll transform everything, or it won't deliver meaningful value. Either stance can drown out the more practical question: what does this system reliably do in this situation, and what should you test before relying on it?
Saying "I'm not sure" isn't enough by itself. It becomes useful when it comes with a next step. For example: "I don't know whether this model handles our support tickets accurately. Let's test it on a representative sample, define an acceptable error rate, and review the failure cases before deployment." That's not indecision. It's a concrete plan for reducing uncertainty.
For teams, the goal shouldn't be to erase confidence. It should be to calibrate it. Good technical judgment means knowing when the evidence is strong enough to act - and when a claim needs checking.
How confidence shapes AI coverage and workplace decisions
The report also looks at who shows up in media coverage. Across a year of coverage reviewed by the researchers, men were named in AI headlines more than 14 times as often as women. When the analysis focused on people quoted as subject-matter experts, three-quarters were men. The report compares that share with an AI talent pool that is roughly 29 percent women.
These numbers don't explain why every editor chose a particular source, and they shouldn't be read as proof of a single cause. Still, they raise a practical question: whose views are easiest to find - and whose expertise gets treated as authoritative?
Visibility compounds over time
Media coverage shapes which voices audiences recognize. If the same kinds of executives and commentators keep showing up, their viewpoint can start to feel like the default account of the field. And AI debates aren't only about model performance. They include labor impacts, privacy, access, safety, product design, and who benefits versus who carries the risks.
If coverage relies on a narrow set of sources, it can miss relevant experience even when reporting is technically accurate. A broader range of qualified experts doesn't automatically produce better journalism or better policy. It does increase the chance that assumptions get challenged - and that different consequences become visible.
The same dynamic can happen inside a company. If people who speak most confidently are consistently treated as the most knowledgeable, they may end up driving decisions about tools, budgets, and deployment. Colleagues with relevant expertise may contribute less - especially if uncertainty is punished or if their input is only invited after a decision is already locked in.
The business risk is poor evaluation, not a confidence contest
A company doesn't need to turn a survey about gender into a hiring or promotion rule. That would be a bad use of the findings. What it needs is a reliable way to evaluate AI claims - so people affected by a decision can actually contribute to it.
When a team assesses an AI tool, it should be able to answer questions like:
- What specific task are we trying to improve?
- What evidence supports the vendor's performance claims?
- Who checks the outputs, and what happens when they're wrong?
- Which users or customers may see different outcomes?
- What information must not be entered into the system?
- What would make us stop, revise, or expand the pilot?
These questions shift the focus away from who sounds most certain and toward what the system needs to prove. They also make accountability clearer. If an AI tool fails in production, the organization needs more than a confident memory of a sales demo - it needs documented assumptions, test results, and someone accountable for the workflow.
A practical way to assess AI knowledge
If you manage a team, buy AI software, or make editorial decisions about AI, the approach below is more useful than trying to rank people by confidence. It treats knowledge as something you can see through explanations, tests, and decisions - not just through self-assessment.
1. Ask for a concrete use case
Start with the work, not the technology. "We should use AI" isn't a real business case. Ask what's happening today, where time or quality is slipping, and what a better outcome would look like.
For example, summarizing internal support tickets for a human agent is easier to evaluate than a vague goal like "automate customer service." You can define what counts as a useful summary, what must stay private, and which decisions remain with a person.
2. Separate facts, assumptions, and open questions
Have the project owner write down three categories:
- Known: What's been verified in the team's own environment?
- Assumed: What's being taken from a demo, vendor claim, or anecdote?
- Unknown: What needs testing before you commit?
This helps because confidence often grows when assumptions stay off the record. Once they're named, the team can decide which ones matter - and how to test them.
3. Use a small, representative evaluation
A pilot should reflect what the system will actually face. Testing only clean examples can create a false sense of reliability. Include routine cases, edge cases, and inputs that represent the variation in your data and users.
Agree on evaluation criteria before you look at results. Depending on the use case, that might mean factual accuracy, handling sensitive information, time saved after review, consistency, or how often human staff must correct outputs. The right metric depends on the job. A speed gain isn't useful if it creates extra review work - or unacceptable errors.
4. Reward calibrated answers
In reviews and planning meetings, make it normal to separate confidence from evidence. Ask people to explain what they know and what they would verify. A useful answer might include a confidence level, the basis for it, and the next test.
This isn't permission to hedge every statement. Teams still have to decide. The goal is to make uncertainty actionable, not a reason to delay forever. Assign a decision owner, define a test window, and set a threshold for proceeding. If the evidence meets the threshold, move forward with appropriate monitoring. If it doesn't, adjust the plan.
For organizations formalizing this process, the NIST AI Risk Management Framework provides a structured reference for identifying and managing AI risks. It won't make the business decision for you, but it can help teams think past whether a model looks impressive in a demo.
5. Broaden who gets asked - and how
When you select experts for a panel, an interview, or an internal review, look for people with relevant experience across technical, product, legal, operational, and user-facing roles. Don't assume the most visible person is the only qualified one. And don't treat representation as a substitute for expertise. The point is to gather people who can challenge the assumptions built into the decision.
Make participation useful by asking specific questions and giving people enough context to answer well. A broad request to "share thoughts on AI" can favor the fastest talker. Questions about limitations, deployment conditions, and likely failure modes create more room for demonstrated knowledge.
Conclusion: Make room for uncertainty
The survey's most useful lesson isn't that one group should be trusted over another. It's that self-assured answers are a weak substitute for demonstrated knowledge. The results point to differences in AI use, factual responses, and willingness to say "not sure" - and they also show how a narrow set of voices can dominate public discussion.
For businesses, the response is practical: evaluate AI systems against real tasks, make assumptions explicit, and give teams room to raise uncertainty early. For journalists and decision-makers, it means looking past the most visible - or most confident - voices when you're assessing expertise.
You don't need perfect certainty before acting. You do need a way to tell an informed decision from a confident guess. Start by picking one AI use case, defining what success and failure look like, and testing it with the people who will rely on it. That's a better basis for adoption than confidence alone.