Overview
A multimodal AI framework that localizes pre-recorded video advertisements by replacing spoken brand and location-specific entities while preserving the original speaker’s voice and visual identity. Integrates speech recognition, zero-shot voice cloning, lip synchronization, facial restoration, and multilingual content moderation to generate regionally tailored advertisements in Hindi, Marathi, Tamil, and Telugu.
Contributions
- Addressed the scalability and localization challenges of celebrity-led advertising by developing a framework that adapts a single brand campaign for different retailers, locations, and promotional offers.
- Designed a brand-to-retailer personalization workflow enabling MSMEs to generate localized advertisement variants without requiring a separate video shoot for every variation.
- Architected a two-stage multimodal AI pipeline integrating 5+ components across speech recognition, voice generation, lip synchronization, facial restoration, and content moderation.
- Developed spoken-entity personalization capabilities to replace brand names, location references, and promotional offers while preserving the original speaker’s voice identity and synchronized lip movements.
- Enabled multilingual advertisement localization with support for 22 Indic languages, targeting regionally tailored advertising across diverse linguistic markets.
- Engineered an evidence-based benchmarking framework using 640 speakers across two language families to evaluate model suitability for multilingual speech applications.
Recognition
- Colloquium 2026
Second Prize
Recognized with second prize at Colloquium 2026.