Urdu-English AI Voiceovers: The Switch-Point Problem

Why language switches matter in an ad

Code-switching means using more than one language in a conversation or even within a single utterance. In Pakistani advertising, Urdu-English mixing is common: Urdu often provides the grammar, while English words or phrases get inserted where they carry a specific tone.

That distinction matters because language choice is doing work. In a line like Naya smart TV sirf Rs. 85,000 me - free delivery ke saath!, free delivery is the kind of promise that pops out immediately to shoppers. Urdu carries the surrounding message in a familiar, conversational register. And a fluent speaker isn't just swapping vocab. They manage pronunciation, rhythm, emphasis, and audience expectations about how the line should sound.

The proposal describes this pattern using the Matrix Language Frame model, associated with linguist Carol Myers-Scotton. In simple terms, one language provides the grammatical structure, while elements from another language get embedded into it. For ad production, the practical challenge isn't the terminology - it's coordination. The English bits have to fit into the Urdu sentence without sounding like someone pasted in a separate recording.

This gets especially tricky around persuasive phrases. The speaker might lean into best, raise their pitch before order now, or quicken the pace as the offer turns "urgent." The switch point - the moment the languages transition - can change how the whole line lands. Even if the speech is technically understandable, it may still miss the intended emphasis.

So judging a multilingual ad voice only by "can listeners understand the words?" misses the point. A more useful evaluation looks at at least three things:

• Pronunciation: Are Urdu and English pronounced in ways listeners recognize as appropriate?
• Prosody: Do stress, pitch, rhythm, and pauses match how the phrase behaves in each language?
• Performance: Does the delivery support the ad's tone without sounding off or misleading?

They're related, but they don't automatically move together. Someone can pronounce everything clearly and still put the wrong stress on the wrong syllable. A lively pitch contour can sound expressive while the Urdu phonemes still feel awkward. In the end, listeners hear the whole performance - not an isolated pronunciation score.

Why standard AI voices can miss the moment

A lot of commercial TTS works best when the text and the chosen voice match the language and speaking style it was trained or configured for. If you feed a mixed Urdu-English script into an English voice, it may treat Urdu words like unfamiliar spellings. Then it can apply English sound patterns to them, stress the wrong syllables, or handle the transition like the language never really changed.

The voice can also flatten the delivery. The words are there, but the pitch and timing don't lift into the call to action or underline the offer. What you hear ends up closer to "a model reading tokens with its default settings" than "a bilingual person speaking naturally."

In the proposal, Usman points to a demonstration using the line Yeh product best hai, order now with a production English TTS voice. He reports English-style phonemization of Urdu function words, default English stress, and no persuasive pitch rise into the call to action. That's an illustrative test, not a controlled study showing how every commercial TTS product behaves. Systems differ, and settings - voice choice, language controls, text normalization, model versions - can all change the outcome.

More broadly, this isn't the first time researchers have run into issues with code-switched synthesis. The proposal mentions work on Hindi-English synthesis, including an earlier framework by Sitaram and Black, plus a 2023 study by Joshi and Garera on Hindi-English e-commerce voice bots. It also references Mandarin-English synthesis work and research that explicitly predicts log-F0 (a way to represent fundamental frequency for pitch modeling). Those examples show bilingual speech synthesis has been studied in other language pairs. They don't guarantee the same approach will transfer cleanly to Urdu-English ads.

One cited Chinese-English study reports more pitch variability for English words inside code-switched speech than in monolingual speech - average F0 standard deviation of 51.9 Hz versus 29.8 Hz. That's a useful reason to care about pitch. But it's not a direct measurement of Urdu-English advertising, and it shouldn't be treated like one.

Under the hood, the core engineering issue is a mismatch: what the system learned from its training data versus what the target utterance demands. A model trained or tuned on monolingual speech may not capture how bilingual speakers shift between languages - especially when an ad line depends on deliberate delivery. And it's not just a pronunciation problem. Better pronunciation alone won't fix timing and pitch. Likewise, more expressive prosody won't help if the pronunciation is still wrong.

What the proposed research would test

The proposal lays out three phases. The interesting part is that it separates the data issue, the acoustic measurement, and the listener response, instead of treating one "sounds nice" demo as proof.

The proposal's expected contributions are an acoustic characterization of Urdu-English switch-point prosody, a public annotated corpus, a quantified human-versus-synthetic prosody comparison, and a prosody-aware synthesis recipe evaluated against purchase intent. These are planned contributions, not completed results. It also notes that findings from Hindi-English or Mandarin-English systems might partly transfer - which could reduce novelty, but still leaves the need to study this language pair and this specific use case.

The business case - and the ethical limits

For a small business, synthetic voice can reduce both cost and turnaround time for producing multiple ad versions. A retailer could test different offers, formats, or calls to action without scheduling a new recording session every time. That's a real operational advantage - if the result sounds credible to the audience.

But it's not just "AI voice versus human voice." It's a trade-off between cost, speed, quality control, and how much the voice matters for trust. A low-budget test campaign might tolerate an imperfect synthetic read. A big brand campaign - where voice shapes trust - probably can't. And if every script needs manual pronunciation fixes and repeated generations, the savings can vanish in editing time.

A practical evaluation should go beyond one demo:

• Test with the audience you actually want to reach, not only your internal team.
• Compare human and synthetic reads using the same script and the same creative context.
• Check pronunciation, naturalness, and delivery of the key claims separately.
• Measure outcomes that matter, while remembering that a listening test isn't the same thing as a live sales experiment.
• Track editing and review time, not just per-minute generation cost.

There are also ethical and governance issues. Voice quality affects who gets represented well by AI tools. If a system handles some Urdu-English accents better than others, businesses may start treating one way of speaking as "correct" and other varieties as mistakes. A corpus of five speakers can't, by itself, establish broad coverage across Pakistan's linguistic diversity. Corpus design, consent, and transparent reporting of limits are technical work - not something to tack on later.

Persuasive voice generation also raises disclosure questions. An AI-generated voice can make a product claim sound confident and personal. The more convincing the delivery, the more important it is that the advertiser stays accountable for the claim - and doesn't use synthetic performance to hide material information or impersonate a real person without permission. Legal requirements vary by market and use case, but responsible production should include consent, rights checks, and human review.

For AI vendors, the opportunity isn't just adding "Urdu support" to a language list. A real product needs to handle the workflow: mixed-language text, pronunciation control, delivery that's flexible, and repeatable quality checks. The business value is reducing friction without offloading review and correction costs onto the customer.