On-Device AI Scam Detection: Google and GovTech
Explore how Google DeepMind and GovTech Singapore use on-device Gemma AI to detect conversational scams, protect privacy, and deliver real-time warnings.
Can On-Device AI Catch a Scam Before You Send the Money?
At first, scams can look pretty normal. A new contact promises an investment opportunity, a "recruiter" pushes your job chat into messaging apps, or someone claiming to represent a bank creates urgency around your account. The real danger usually shows up later - after a long back-and-forth - when the person on the other end nudges you to share credentials or send money. Catching that kind of manipulation is much harder than flagging a suspicious link.
Google DeepMind and Singapore's Government Technology Agency (GovTech Singapore) are trying a different approach: a lightweight Gemma model fine-tuned to spot local scam conversations, running directly on your device. The idea is to look for patterns in the conversation and warn you before you act - without sending the conversation to a cloud service for analysis.
That choice reflects a real trade-off. More context can improve detection, but collecting private messages raises privacy and trust issues. This article breaks down how the proposed two-stage system works, what the on-device setup can and can't do, and what product teams can take from the collaboration.
Why conversation-based scams are difficult to detect
Traditional security filters are good at known indicators: malicious links, suspicious domains, reported phone numbers, or message patterns that match an existing rule. Those controls still matter.
But scammers can adapt. They can swap a URL, use new accounts, or send messages that don't look "malicious" on their own.
The tougher cases rely on social engineering. The attacker builds credibility over multiple messages, responds to what you say, and gradually steers the conversation toward something risky. One message might be harmless. The danger usually comes from the sequence - switching platforms, promising unusually high returns, appealing to authority, or pressuring you to act quickly.
That sequence-based nature creates a detection problem. If a system only looks at a single message, it can miss the manipulation. If it analyzes a longer thread, it may need access to more sensitive data. In a messaging app built around end-to-end encryption, routing messages to a server for inspection can undermine the privacy model or conflict with what users think the app does.
Singapore's experience helps narrow the focus. The source announcement says scams - including investment fraud, fake job offers, and impersonation of public officials - make up more than half of reported crimes in Singapore and caused more than S$4 billion in losses since 2019. It also describes social-engineering scams as accounting for up to 80% of losses. These figures show why intent detection across a conversation is important. At the same time, the model's training and evaluation are grounded in local scam patterns, not a one-size-fits-all "universal scam" category.
Why local examples matter
Even if a model understands concepts like urgency or impersonation, it can still misread local names, institutions, common phrasing, or regional schemes. Fine-tuning Gemma on anonymised scam-related material is meant to help it tell local deception from everyday conversation.
This isn't a promise that it will catch every new scam. Scams evolve. A model can misclassify a legitimate chat as risky or miss an unusual attack. Local training should make results more relevant, not remove the need for evaluation, updates, and a plan for handling mistakes.
How the on-device detection pipeline works
Running a capable model on a smartphone isn't just "move the cloud model onto the device." Phones have limited compute and memory, and users can notice battery drain, lag, and interruptions. The project addresses that by using a two-stage pipeline, instead of asking the larger model to analyze every incoming message.
Stage one: screen efficiently
A lightweight classifier first screens inbound messages for possible threats. This pass is designed to use minimal power and flag messages (or patterns) that deserve closer attention. Think of it as a filter, not the final judgment.
The separation matters operationally. If every message triggered a heavier model, the system could burn more battery and waste compute. The whole point of a first pass is to narrow down what gets deeper review. And like any production classifier, it comes down to a threshold choice: set it too strict and you miss subtle scams; set it too low and you invite false alarms.
Stage two: use conversation context
When stage one raises a concern, the fine-tuned Gemma model analyzes a larger snapshot of the conversation. With that extra context, it can look for manipulative intent - either supporting a likely threat or dismissing a false alarm. If it detects risk, it can show a real-time warning on the device, urging the user to pause before sharing sensitive information or sending money.
This is a practical division of labor: fast, low-cost screening first, deeper context only when needed. It also fits a broader pattern in AI product design - models don't have to be the first and only component in a workflow. Routing, thresholds, and user-facing intervention often matter as much as raw model capability.
The warning itself is also a product choice. The system is described as advising caution, not automatically blocking a message or transaction. That keeps the user in control and reduces the cost of being wrong. Still, warnings can feel intrusive, so timing and wording need testing. If people see too many alerts that don't match their experience, they may start ignoring them altogether.
For teams building related systems, the key question isn't only "which model should we use?" It's also: what does each stage need, what happens after detection, and how can users recover if the warning is incorrect. For more on that kind of product framing, see our designing privacy-aware AI features.
What privacy-preserving means in practice
The central design choice is simple: the conversation analysis runs on the device. Unlike server-side scanning, the model isn't intended to upload users' conversations to Google or the Singapore Government for inference. Per the project announcement, neither organization can access users' conversations through the model's deployment.
That's a meaningful reduction in data exposure. Keeping message content on the endpoint avoids building a central store of conversations that could become a target, be misused, or require complex retention and access controls. It also means you can analyze quickly, without waiting for a server round trip.
Still, "on-device" isn't a complete privacy guarantee by itself. The real boundary depends on how the app integrates the feature, what permissions it asks for, what diagnostic tools it uses, and what data the app logs or sends for other purposes. A model can run locally while the application separately collects telemetry. So privacy claims should cover the full data flow - not just where inference happens.
Consent and application boundaries
The collaboration says the model will only access on-device conversations with the user's consent and in line with the application's terms of use. Rollout is described as phased and developed with the Singapore Government. That wording matters: this is a collaboration and deployment approach, not something already available in every messaging app on every phone.
Also, an on-device model generally can't inspect conversations inside unrelated apps just because it exists. Access depends on how the feature is implemented and what permissions the user grants. That distinction matters for expectations - this isn't presented as a universal system silently scanning all private messages across a device.
Training data governance
The project describes multiple controls during model development:
- A managed environment: Fine-tuning took place in a GovTech Singapore-managed Google Cloud Platform environment, with training data kept isolated.
- No citizen message data: The training process excluded private or personally identifiable information and used anonymised datasets derived from scam reports and intelligence feeds.
- Stress testing: The model was tested with synthetic dialogues and local corpora intended to cover different scam categories and changing impersonation tactics.
These controls address different risks. Local inference reduces exposure during use, while isolated training infrastructure and data minimisation address development risk. Neither makes the system risk-free, but splitting the controls makes the privacy model easier to inspect and manage.
For teams operating under regimes like the GDPR, the practical takeaway is to document purpose, access, retention, user consent, and model evaluation as one system. "The model runs on-device" is a design property - not a substitute for a full privacy assessment. For more on that, see our guide to AI data governance.
The product and operational trade-offs
On-device detection shifts some costs and risks away from servers and toward the device and the app interface. That can be a good trade, but it has to be evaluated as a system - not sold as "privacy" on its own.
Detection quality versus false alarms
A scam detector usually has two failure modes. A false negative misses a harmful conversation, which can create misplaced confidence if users assume the tool is comprehensive. A false positive flags normal conversation as suspicious, disrupting communication and eroding trust.
The right balance depends on what the alert does. A gentle prompt to pause can tolerate more uncertainty than an automatic block or account restriction. The project describes a warning that encourages caution, which is a reasonable fit for an early-stage detector. But it still needs testing in real usage - and it needs review as scammers change tactics.
Evaluation should go beyond a single accuracy number. Teams should look at performance by scam type, language, message style, and everyday conversation patterns. They should also measure how often the system triggers, whether users understand the warning, and whether it actually changes risky behavior. A model can score well on a test set and still fail as a product if the warning arrives too late or doesn't make sense to users.
Battery, latency, and device coverage
The two-stage architecture is meant to reduce unnecessary computation. That doesn't eliminate device constraints. Older phones may have less memory or processing power, and background processing rules vary across operating systems. Product teams need to test latency, battery impact, and reliability across the devices they plan to support - not just today's top-tier hardware.
There's also a business trade-off. Cloud services can centralize updates and use more compute, but they add network dependency and data-handling complexity. On-device inference can improve responsiveness and reduce data transfer, but it creates work around packaging, compatibility, updates, and support across a fragmented device ecosystem.
Updates, governance, and accountability
Scam patterns change quickly. A detector that works well at launch can degrade if criminals update scripts or target new services. Keeping it useful means refreshing relevant intelligence, evaluating changes, and monitoring for regressions - ideally without collecting private conversations by default. That takes an intentional feedback loop, not just a model release.
Governance is part of the product, too. The project's sandbox, anonymised training data, and stress testing are concrete safeguards. Deployment still needs clear ownership: who approves updates, how errors get reported, what happens when a user disagrees with a warning, and how the system's limits are communicated.
For businesses, the broader lesson isn't "every app should add a language model." It's that some security problems depend on local context, and sometimes inference can happen at the point where data is created. Before adopting this pattern, teams should weigh expected harm reduction against integration costs, support burden, ongoing model maintenance, and - crucially - user trust. A model that's technically elegant isn't helpful if the permission flow is confusing or if alerts turn into noise.
Conclusion: useful protection depends on the whole system
The Google DeepMind and GovTech Singapore project is a solid example of applying AI to a specific security problem: spotting manipulative intent in local scam conversations while keeping inference on the user's device. The two-stage design targets efficiency; the training and deployment controls address different privacy concerns; and the warning-based response leaves the final decision with the user.
The limits are equally important. On-device processing doesn't automatically make an app private, a model can't promise to catch every scam, and detection quality depends on continued evaluation and updates. What's promising here is that those constraints are treated as part of the system design, not as an afterthought.
If you're building an AI feature that handles sensitive user data, start by mapping the data flow and the actions the model is allowed to take. Then test whether on-device inference actually improves the product enough to justify the device, integration, and maintenance costs. Read the Google DeepMind announcement for the project details, and use the same questions to assess your own AI roadmap.