Generated by AI

Mar 20, 2024

Introducing SUTRA: The Next Frontier in Multilingual AI

Learn about our proprietary series of ultrafast and lightweight Generative AI models with multilingual conversation, search, and visual capabilities.

Abhijit Bendale, Michael Sapienza, Simon Gibbs, and Steven Ripplinger

In the evolving landscape of artificial intelligence, the ability of AI to understand and interact in multiple languages has remained a critical challenge. Traditional large language models (LLMs) have predominantly focused on English, leaving a significant gap in accessibility for billions of non-English speakers around the world. Recognizing this disparity, TWO is excited to introduce SUTRA, a massively multilingual language model that operates across 50+ languages, combining linguistic diversity with state-of-the-art performance.

Why SUTRA?

The advent of SUTRA is a response to the urgent need for more inclusive AI technologies. While language-specific models like HyperClova and OpenHaathi perform admirably in their niches (Korean and Hindi), they are limited in use on a global scale due to language inflexibility and significant training needs. On the other end of the spectrum, more general large language models (LLMs) such as BLOOM and Llama2 falter on multilingual tasks, struggling with understanding and language nuances such as grammar and formalities. SUTRA employs a state-of-the-art approach that enables multilinguality with unparalleled performance, redefining multilingual language models.

The Innovation Behind SUTRA

At its core, SUTRA takes inspiration from the human ability to separate concept understanding from language capabilities. This involves initially training a core concept model to grasp fundamental ideas within a limited language set. In parallel, specialized Neural Machine Translation (NMT) based encoders and decoders are utilized, ensuring consistent concept translation across languages, followed by a language alignment phase that merges conceptual understanding with linguistic fluency.

SUTRA's methodology leverages commonalities between similar languages like Hindi, Gujarati, and Bengali with shared semantics and grammar. This strategy significantly enhances linguistic proficiency and scalability to ensure its readiness to tackle the complexities of multilingual communication.

SUTRA employs a sparse Mixture of Experts architecture, which balances model efficiency and capacity to optimize computation distribution and resource utilization. This dynamic structure supports extensive scaling without exponentially increasing computational costs. The robust training of SUTRA is supported by a proprietary real-world conversational dataset encompassing over 10 million datapoints and leverages both real and synthetically translated paired data for effective cross-lingual training.

Benchmarking Excellence

In English, SUTRA is on par with GPT-3.5 and many other prominent models. On non-English languages like Korean, Hindi, and Gujarati, SUTRA truly shines, outperforming its peers by 30-50% in the MMLU benchmark. Notably, its purpose-built multilingual tokenizers efficiently reduce token usage by over 50% across languages, leading to significant savings during generation. Meanwhile, SUTRA's unparalleled speed, achieving up to 120 tokens per second, sets new benchmarks for efficiency.

These results underscore SUTRA’s robust understanding and processing capabilities, making it an optimal choice for developers and businesses seeking advanced multilingual AI solutions.

Online and Up-To-Date

Another critical advancement SUTRA introduces is its online connectivity. Unlike traditional LLMs, which can become quickly outdated from training data that is months or even years old, SUTRA models have real-time internet connectivity. This feature expands SUTRA’s knowledge base to the entire internet, ensuring SUTRA can provide up-to-date information, making it particularly useful for applications requiring timely and accurate data.

Beyond Text with SUTRA-Avatar

Moving beyond text, SUTRA-AVATAR is visual generative AI that can create photorealistic AI characters capable of realistic interactions. These models can produce natural-sounding speech, display a range of emotions, and perform gestures in real-time, offering new dimensions to AI-communication.

Looking Ahead with SUTRA

Our journey with SUTRA is only beginning. As we pave the way for further advancements, including the development of phonetic models (SUTRA-Dhvanim), our aim remains steadfast: to transcend linguistic barriers and build AI models and experiences aimed at the global community. We are excited to see what is built with SUTRA and invite partners to try it out for themselves at playground.two.ai.