Fantasy football is a pretty friendly test of prediction: you pick a squad, manage a budget, and hope your players actually perform. But building a good team still comes down to the same hard stuff that trips up real forecasting - injuries, shifting form, team news, upcoming fixtures, and all the uncertainty about what happens next.
WIRED reported an experiment where AI agents were put in charge of a Premier League fantasy team, to see whether a coordinated set of models could turn those signals into better decisions. The early suggestions weren't especially convincing. One version, for example, wanted to bench Erling Haaland. Adding agents that read news, analyze results, run models, and critique recommendations helped. Still, throwing in more than 100 agents didn't lead to any meaningful extra improvement.
It's useful that the experiment didn't hand out an easy win for AI. It shows where agent workflows can help, why "more agents" doesn't automatically mean "better forecasts," and where information processing still isn't the same thing as judgment. The same lessons apply to businesses looking at AI for forecasting, planning, or decisions where mistakes cost real money.
What did the AI fantasy football experiment actually test?
This wasn't a simple test of whether a language model could name famous players. The experiment tested whether an AI system could pull in current information, turn it into predictions, and then make practical lineup choices.
That difference matters. A model might know Haaland is a prolific striker, yet still bench him if its workflow misses the right evidence, uses the wrong scoring assumptions, or mishandles uncertainty. A prediction isn't just "what the model knows." It depends on what data it receives, what instructions guide it, and how you evaluate the output.
From a single model to a forecasting workflow
The system was built with Codex and asked to create a harness - basically a mix of instructions and code to organize the work. It initially gathered relevant information and ran a model to generate lineup suggestions. The researcher also provided research papers to shape the setup.
The first output was a good reminder that a process can look reasonable and still land on a bad decision. The system appeared to collect and analyze a lot of information, but early recommendations still included benching Haaland. More data and more analysis don't automatically make the final call make sense.
Next, the system was split into roles. Some agents read news and blog posts. Others analyzed recent results, ran models, or critiqued proposed selections. Suggestions improved. But once the agent count went past 100, results barely moved while API costs rose.
What the result does - and does not - show
This experiment ran over a limited period, so it's not proof that AI can consistently beat human fantasy managers. The team didn't outscore the average over the weeks described, and the researcher didn't judge the recommendations strong enough to rely on for a personal team.
That's still a meaningful takeaway. It separates a neat demonstration from something you could actually treat like a dependable decision system. Even if the roster looks coherent, the real test is whether it scores more points over time. To evaluate a forecasting system, you need a clear goal, a baseline to compare to, and enough repeat results to tell skill apart from luck.
For businesses, the parallel question isn't "can an AI produce a confident forecast?" It's "does a documented workflow improve on what we already do - and does the benefit outweigh the cost and the operational risk?"
Why did a team of agents improve the suggestions?
The improvement wasn't just "add more AI." The bigger change was splitting a broad task into smaller jobs and giving the system a way to challenge its own recommendations.
A single model that has to read news, interpret team form, estimate player output, and pick a roster has to juggle different kinds of reasoning at once. In a multi-agent setup, those steps can be separated. One agent can gather news, another can focus on recent performance, a modeling step can estimate likely outcomes, and a critic can flag missing assumptions or contradictions.
This also makes the workflow easier to inspect. If the final recommendation is wrong, you can dig into whether the information was stale, the analysis missed a factor, the model's assumptions were weak, or the critique didn't catch the problem. With a single opaque prompt, those failure points are much harder to isolate.
Why task separation can help
Different tasks need different evidence. Team news might point to injuries or a change in a player's likely role. Recent results can show short-term patterns, but they don't automatically predict the next match. A statistical model can add consistency, but only if its inputs and assumptions fit the decision. And a critique pass can catch a recommendation that doesn't match the evidence you actually have.
A useful agent workflow also needs to keep the evidence behind each conclusion instead of treating an agent's confident summary like proof. If an agent says a player is likely to start, the system should be able to trace that claim back to what it used - and when it pulled that information.
Why 100 agents were not better than a few
More agents can generate more output without adding useful signal. If multiple agents repeat the same analysis, they can amplify the same mistake instead of providing independent checks. And if each agent requires paid model calls, you also pay in costs, runtime, and coordination overhead.
This is a common scaling trap in AI projects. A system can look more sophisticated because it has more steps, but that complexity only matters if it reduces a known failure mode or improves measurable results. In this experiment, a relatively small set of specialized roles helped. Expanding beyond 100 agents didn't materially improve the suggestions.
The practical takeaway is simple: add an agent only if you can name the job it performs and the specific failure it's meant to catch. Then compare the outcome to a simpler workflow. If the extra role doesn't improve accuracy, calibration, coverage, or overall decision quality enough to justify the cost, cut it.
What do prediction benchmarks say about AI forecasting?
Fantasy football is just one example of a bigger question: can AI make reliable predictions about real-world events? Researchers are testing that idea with forecasting tasks and prediction-market simulations, where models have to assign probabilities to uncertain outcomes - not just explain what already happened.
Haifeng Xu, a computer scientist at the University of Chicago who studies AI in market settings, argues that newer models may forecast better partly because they can process more information and coordinate more complex systems. In the testing described in the source report, models sometimes did better than an average person without special forecasting instructions. That's promising, but it doesn't mean they consistently beat skilled experts or perform well in every market.
Benchmarks measure different things
Prophet Arena is one benchmark for forecasting ability. Researchers at Harvard and Arcada Labs also created Prediction Arena, where models were given $10,000 to wager on prediction markets over 57 days. In that experiment, the models lost money overall, though researchers observed improvement as the test went on.
Those results don't contradict each other. A system can improve during an experiment and still end up losing overall. It can do well on some questions and poorly on others. Benchmarks also vary in what questions they ask, how much time models get, what information they can access, and how predictions are scored.
So a model's ranking on one benchmark is evidence about how it performs under that benchmark's conditions - not a universal measure of forecasting skill. The source report lists GPT-5 as the strongest performer among models in Xu's testing, followed by Grok 4, Claude Sonnet, Gemini, and Llama. That ordering is tied to a particular evaluation, not a guarantee it holds for every task or future model release.
Humans still contribute context
AI systems can scan more information than a person can reasonably process. But strong forecasters often bring deeper judgment about how events connect - what sources are credible, what an announcement changes in practice, and which trend is likely to matter versus which is just noise.
This gap is especially important when evidence is ambiguous or incentives affect what people say publicly. A model may collect a lot of facts without knowing which ones actually shift the forecast. Human expertise can help separate the signal from the background.
The gap may be shrinking. The Metaculus Cup has shown that AI can sometimes outperform highly skilled human predictors, and ForecastBench is another effort to compare model forecasting with human performance. The sensible conclusion is neither "AI has solved prediction" nor "human judgment will always win." Performance depends on the task, the workflow, and the quality of feedback.
When should a business trust AI predictions?
A business should treat an AI forecast as an input to a decision system, not as a decision system by itself. That's true whether the output is about product demand, project timelines, churn, hiring needs, or a fantasy roster.
First, define what "good" means. Are you trying to predict an exact number, rank possible outcomes, spot a risk early, or improve decisions under uncertainty? A forecast can be statistically accurate and still be commercially useless if it arrives too late or doesn't change what the team does.
A practical evaluation framework
Before you put agents into a live workflow, run a short evaluation cycle:
- Set a baseline. Compare the AI to the current human process, a simple statistical model, or both. Without a baseline, a polished output can feel like progress even when it isn't.
- Define the outcome and time horizon. Decide what "correct" means, when the prediction will be checked, and how uncertainty gets scored.
- Log the evidence. Record inputs, timestamps, assumptions, model versions, and the agent outputs. If a forecast fails, you'll need this to figure out why.
- Test a small number of roles. Add agents for specific tasks - like gathering current information or checking for contradictions. Don't scale up agent count until you've measured what each role actually contributes.
- Track cost and operational impact. Include API usage, review time, latency, and the cost of acting on a bad prediction. More calls can make a workflow worse even if accuracy improves a little.
- Keep a human review point where errors matter. Humans should focus on consequential exceptions, unclear evidence, and decisions where context is critical - not manually re-check every routine output forever.
A decision rule for using forecasts
A forecast is worth using when it improves decisions enough to justify the cost and risk. In low-stakes settings, a modest gain might still be worthwhile. In high-stakes settings, you need stronger evidence, clearer accountability, and a way to challenge or override the recommendation.
This is also where governance stops being a buzzword. If a prediction affects customers, employees, or financial exposure, someone has to own the decision and understand the system's limits. Keeping an evidence trail helps with debugging and accountability, but it doesn't magically make an unreliable forecast reliable.
Fantasy football makes the trade-off easy to see. If the AI recommendation is wrong, you lose points and maybe some bragging rights. In business, the same kind of mistake can lead to missed sales, wasted inventory, or unfair treatment. The lesson is the same: test the workflow against the real consequences of being wrong.
Key takeaways
- A well-organized workflow matters more than agent count. Specialized roles helped, while scaling beyond 100 agents mainly raised costs.
- Information volume isn't the same as judgment. The system still made a questionable call about Haaland even after it gathered plenty of data.
- Benchmarks are evidence, not guarantees. Results vary by task, evaluation method, and what information is available.
- Measure against a baseline. Track forecast quality, decision impact, cost, and failure modes before using AI in production.
- Use human oversight where context or consequences demand it. The goal isn't to remove people from decisions - it's to apply judgment where it actually changes the outcome.
Conclusion
This fantasy football experiment gives a grounded view of AI forecasting: a small, specialized agent workflow can improve recommendations, but a larger system isn't automatically smarter - and smoother-looking analysis isn't the same as better results.
If your team is considering AI forecasts, the next step isn't to deploy a swarm. Define the decision, set a baseline, track evidence and cost, and test whether each component actually improves outcomes. Keep human judgment in the loop when context or consequences call for it. The aim shouldn't be to make prediction feel automated - it should be to make decisions more reliable.