Decision Models 🚀: Custom AI Models, Synthetic Data Now12 min read2,444 words

Decision Models 🚀: Custom AI Models, Synthetic Data Now

Discover how decision models, custom AI models, and synthetic training data beat benchmark scores—learn AI model evaluation with real-world calibration and testing.

decision modelscustom AI modelsAI model customizationsynthetic training dataAI model evaluation
Decision Models 🚀: Custom AI Models, Synthetic Data Now

Decision Models Are Growing Fast. Can Custom Data Set Them Apart?

AI teams are adding decision models to production systems at a surprisingly fast clip. Instead of generating paragraphs or code, these models pick between options: route a request, classify a case, select a tool, or decide when a more capable model (or a person) should step in. The appeal is simple: a fast, cheap decision can matter more than a long explanation.

Still, the category is getting crowded. After TypeSafe's Jev models started gaining traction, startups and major cloud providers began offering their own alternatives. When teams can swap models with relatively little effort, a general-purpose benchmark may not tell you which option will work best for your actual workload. The real question is whether a model can improve the data and outcomes that your business cares about.

This article breaks down what decision models do, why customization is starting to look like a competitive edge, and how to evaluate trade-offs before you put one into a workflow. It also draws a line where hype ends: customer data can improve fit, but it doesn't automatically make systems safer, more accurate, or cheaper.

What is a decision model, and why is adoption rising?

A decision model is built to select from a defined set of options, not generate a stream of tokens. It applies learned patterns to a choice and can include a confidence estimate with that choice. In an application, the options might look like "send to billing," "send to support," or "escalate to a human." In most real systems, it's not responsible for the entire interaction - it just handles one bounded decision inside a larger workflow.

That difference shows up in cost and latency. A conventional language model often has to produce a detailed response even when the app only needs a label or a routing decision. A decision model can handle that step more quickly and, in many setups, more cheaply. For teams processing lots of requests, even small savings per decision can add up - assuming the model is accurate enough for the consequences.

Why the timing matters

A recent data point shows how quickly usage is rising. In October, more than one in four requests on Vercel's AI model router went to decision models instead of LLMs. It's a sizable share for a category that many people still treat as new. That said, the numbers are about that router's traffic - they don't prove every task is a good fit, or that the same pattern holds across the industry.

TypeSafe's Jev family helped make the category noticeable. The report says Jev kept leading usage metrics available through services like OpenRouter, which reportedly handled more than 1.5 billion queries to TypeSafe's models in one week. But once a useful pattern is clear, competitors can move fast. The article also describes an Amazon engineer building a version over a weekend, with follow-on options from hyperscalers. Microsoft announced Microsoft-Decision-1.1 for broad Azure use.

Where they fit - and where they do not

Decision models work best when you need a frequent, bounded choice and you understand what happens when it's wrong. Common examples include routing requests, prioritizing a queue, choosing a tool, or deciding whether to invoke a larger model.

They're less compelling when the task needs open-ended synthesis, a user-facing explanation, or a decision that doesn't map cleanly to a set of alternatives.

In practice, teams often end up with a hybrid: use the smaller model for routine decisions, then escalate uncertain or high-impact cases to a stronger model or to a person. Confidence scores can help define that boundary - but only if they're actually calibrated on real examples. A score that looks precise but doesn't match real reliability can create false confidence, which is when things start going wrong.

Why could customer data be the next advantage?

As more providers offer decision models, the base architecture becomes less of a differentiator. The harder part is making those decisions match your specific task. TypeSafe CEO Diogo Almeida told TechCrunch that calibrating Jev's judgement with statistically adjusted synthetic data was among the hardest parts of building it. The takeaway is the same constraint in different words: how useful the model becomes depends on whether it learns decisions that behave the way you need.

Sqwish is betting that customer-specific post-training can address that. The UK-based startup, founded by Cambridge graduates Ushnish Sengupta and Federica Freddi, built smaller language models before adapting its approach to decision models. Its product, kettei fat, comes with tools meant to let customers use their own data to improve how the model judges the tasks they care about.

What customization can change

A general model may understand broad categories but miss the details that drive a company's preferred action. A support team, for instance, often has its own escalation rules, product terminology, and past examples of cases that were routed correctly. Training or tuning on a well-prepared set of examples could make the model reflect those choices instead of leaning on generic patterns.

Sqwish's pitch, according to its CEO, is that customers should keep uploading production data and continue improving their models over time. That's potentially valuable - customization isn't "one-and-done," it can evolve as a business's policies and cases change.

But "more data" isn't a plan by itself. Production data includes noise, outdated decisions, personal information, and inconsistent human judgement. If you reuse it without review, you can teach the model to repeat exactly the errors a company is trying to fix.

The governance work behind a data loop

If you're going to run a data loop, you need real controls - not just ingestion pipelines. Before tuning, teams should decide what data can be used, who approves it, how sensitive fields are handled, and how long examples are kept. For UK and EU businesses, GDPR obligations should be part of that work, not an afterthought. US organizations also have to account for the relevant state and sector rules that apply to them.

It also helps to separate training examples from evaluation data. If the same cases influence tuning and then show up as "proof" that the model improved, the evaluation isn't trustworthy. Keep a held-out set, record model and dataset versions, and review what changed before releasing. And when the model's decisions are consequential, keep a human review path and an audit trail. Customization can improve local fit, but it also adds ongoing operational responsibility.

How should teams evaluate decision models in production?

A good test isn't whether a model wins a single benchmark. It's whether it improves a specific workflow under the constraints that matter to your business. Sqwish says its kettei fat model does well on some benchmarks, but its founders argue you should test in the customer's own context. Microsoft engineering VP Achint Srivastava has made a similar point: models optimized too heavily for benchmark scoring can disappoint in real enterprise use.

That doesn't mean benchmarks are useless. They can help you screen candidates and provide a shared starting point. The mistake is treating a benchmark score like a deployment recommendation. A benchmark won't capture your request mix, your cost ceiling, your tolerance for false positives, or how failures propagate through your application.

Build a test around the actual decision

Start by writing down the decision the model will make and the available options. Then define what "correct" means, including cases where multiple answers are acceptable or where the right move is to abstain. Gather representative examples from normal operations, edge cases, and scenarios where mistakes would be especially expensive.

A useful evaluation usually compares several dimensions:

  • Accuracy by outcome: Which choices are right, and what kinds of cases produce errors?
  • Calibration: When the model reports confidence, does that confidence track real reliability?
  • Latency: Does response time meet the needs of the workflow, including at peak load?
  • Cost: What does each decision cost at realistic volume, including fallbacks (like another model) or human review?
  • Reliability: Does performance hold across customer segments, input formats, and less common cases?
  • Operational fit: Can the team inspect, version, and roll back changes without breaking the system?

The trade-offs matter. Microsoft-Decision-1.1 was described by Microsoft as balancing cost, latency, intelligence, and calibration. A model that's excellent on one measure but weak on another might still be the wrong production choice. For example: shaving a bit of latency isn't a win if it makes misrouting errors much more costly.

Run a controlled comparison

Before replacing an existing model, test both systems on the same held-out examples. If you can, do a shadow deployment: send real inputs to the candidate model, but don't let it control the live outcome. Compare its choices to the current system and to human-reviewed results. A limited rollout after that can surface issues that offline evaluation misses.

Set thresholds before you run the tests. Decide what error rate is acceptable, when the model must abstain, and which cases require escalation. Then track results after release - input patterns and business rules change. When customization is involved, compare the tuned model against both the untuned version and the current production baseline. Otherwise, it's easy to spend on training that doesn't actually move the needle.

The feedback loop is pretty direct: test, deploy cautiously, inspect outcomes, update data and rules, then test again. The model is one component in that loop, not a replacement for owning the decision process.

What does the decision-model market mean for businesses?

For buyers, a wider field of providers creates options and negotiation leverage. It also makes vendor claims harder to compare apples-to-apples. Sqwish is competing with companies that have much bigger distribution, including hyperscalers, while Microsoft's release suggests platform providers see decision models as part of their broader AI offerings. In a crowded market, "available everywhere" isn't the same thing as differentiation.

Sqwish's bet is that customers will value the ability to tune a model to their own data and keep improving it. That could give a startup a focused position even if larger providers can offer a general model at scale. But the pitch has to clear a simple bar: does customization produce better outcomes than a base model, at a total cost and risk level the customer can accept?

Distribution versus specialization

Large platforms can lean on existing cloud relationships, infrastructure, and tooling. A specialist can focus on a narrower workflow, a particular tuning approach, or more control over iteration. Neither approach automatically wins.

From a buyer's perspective, you should compare the whole operating cost: model usage, tuning, data preparation, evaluation, monitoring, compliance review, and the engineering time required to maintain the integration. A low per-call price can mislead if the model needs frequent human correction. Likewise, a more expensive option may still be cheaper in practice if it reduces downstream work without adding unacceptable risk.

System One and System Two as an architecture

Sqwish co-founder Federica Freddi described a future where "System One" intelligence - fast decisions - runs alongside "System Two" intelligence, like frontier LLMs. Treat that as an architectural metaphor, not a promise about how AI systems will evolve. The practical version is already easy to understand: use a quick model for routine choices, and call something stronger when uncertainty, complexity, or potential impact crosses a threshold.

Done well, that setup controls cost while preserving a path to deeper reasoning. It also adds complexity. Teams must decide how escalation works, make sure the handoff preserves the context, and evaluate the combined system rather than judging each model in isolation. And yes, routing errors matter: a small model can block the larger model from ever seeing a case, so routing itself needs careful evaluation.

Conclusion: treat customization as a capability to prove

Decision models are getting attention because many production systems need fast choices more often than they need another generated paragraph. As more providers enter the category, the model itself may be easier to swap. What's harder - and what you actually need to prove - is reliable performance on your company's decisions.

Custom data is a reasonable path to that edge, but it isn't proof by default. Start with a bounded workflow, protect and review the data, and compare tuned and untuned models on held-out examples. Keep a larger model or a human available for cases the fast path shouldn't handle. The right decision model is the one that improves the whole system under real operating conditions, not the one with the boldest claim.

Frequently Asked Questions

A decision model selects among predefined options instead of generating a sequence of text tokens. It can return a choice and a confidence estimate, making it useful for tasks such as routing, classification or selecting whether to escalate a request to another model or a person.
A large language model typically generates text or other output tokens, while a decision model is tuned to make a bounded choice. The distinction is about the task and output, not necessarily a completely separate underlying technology. A workflow can use both, assigning each the work it handles best.
It can improve performance on a specific task when the data is relevant, representative and correctly labelled. It can also reinforce outdated rules, inconsistent human decisions or biased patterns. Teams should review training data, preserve held-out evaluation examples and measure results against a real production baseline.
Test candidates on representative cases from the intended workflow. Compare accuracy, confidence calibration, latency, cost and reliability, then run a controlled rollout. A public benchmark can help narrow the field, but it cannot replace evidence from your own use case and operating constraints.

Related articles