Amazon releases 51-language dataset for language understanding

MASSIVE dataset and Massively Multilingual NLU (MMNLU-22) competition and workshop will help researchers scale natural-language-understanding technology to every language on Earth.

Imagine that all people around the world could use voice AI systems such as Alexa in their native tongues.

Multilingual Alexa.png
The MASSIVE dataset is a step toward the creation of multilingual natural-language-understanding models that can generalize easily to new languages.

One promising approach to realizing this vision is massively multilingual natural-language understanding (MMNLU), a paradigm in which a single machine learning model can parse and understand inputs from many typologically diverse languages. By learning a shared data representation that spans languages, the model can transfer knowledge from languages with abundant training data to those in which training data is scarce.

Today we are pleased to make three announcements related to MMNLU.

First, we are releasing a new dataset called MASSIVE, which is composed of one million labeled utterances spanning 51 languages, along with open-source code, which provides examples of how to perform massively multilingual NLU modeling and allows practitioners to re-create baseline results for intent classification and slot filling that are presented in our paper..

Related content
Neural text-to-speech enables new multilingual model to use the same voice for Spanish and English responses.

Second, we are launching a new competition using the MASSIVE dataset called Massively Multilingual NLU 2022 (MMNLU-22).

And third, we will cohost a workshop at EMNLP 2022 in Abu Dhabi and online, also called Massively Multilingual NLU 2022, which will highlight the results from the competition and include presentations from invited speakers and oral and poster sessions from submitted papers on multilingual natural-language processing (NLP).

“We are very excited to share this large multilingual dataset with the worldwide language research community,” says Prem Natarajan, vice president of Alexa AI Natural Understanding. “We hope that this dataset will enable researchers across the world to drive new advances in multilingual language understanding that expand the availability and reach of conversational-AI technologies.”

The MASSIVE dataset

MASSIVE is a parallel dataset, meaning that every utterance is given in all 51 languages. This enables models to learn shared representations of utterances with the same intents, regardless of language, facilitating cross-linguistic training on natural-language-understanding (NLU) tasks. It also allows for adaptation to other NLP tasks such as machine translation, multilingual paraphrasing, new linguistic analyses of imperative morphologies, and more.

Related content
In experiments, multilingual models outperform monolingual models.

NLU — a subdiscipline of NLP — is a machine's ability to understand the meaning of a text and identify the relevant entities. For instance, given the utterance “What is the temperature in New York?”, an NLU model might classify the intent as “weather_query” and recognize relevant entities as “weather_descriptor: temperature” and “place_name: new york.”

Our particular focus is on NLU as a component of spoken-language understanding (SLU), in which audio is converted to text before NLU is performed. Although SLU-based virtual assistants like Alexa have made major capability advances in the past decade, academic and industrial NLU efforts worldwide are still limited to a small subset of the world's 7,000+ languages. One difficulty in creating massively multilingual NLU models is the lack of labeled data for training and evaluation — particularly data that is realistic for a given task and natural for a given language. High naturalness typically requires human vetting, which is often costly.

MASSIVE — Multilingual Amazon SLURP (SLU resource package) for Slot Filling, Intent Classification, and Virtual-Assistant Evaluation — contains one million realistic, parallel, labeled virtual-assistant text utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize or translate the English-only SLURP dataset into 50 typologically diverse languages from 29 genera, including low-resource languages.

Name

# Lang

Utt/Lang

Domains

Intents

Slots

MASSIVE

51

19,521

18

60

55

SLURP (Bastianelli et al., 2020)

1

16,521

18

60

55

NLU Evaluation Data (Liu et al., 2019)

1

25,716

18

54

56

Airline Travel Information System (ATIS) (Price, 1990)

1

5,871

1

26

129

ATIS with Hindi and Turkish (Upadhyay et al., 2018)

3

1,315-5,871

1

26

129

MultiATIS++ (Xu et al., 2020)

9

1,422-5,897

1

21-26

99-140

Snips (Coucke et al., 2018)

1

14,484

-

7

53

Snips with French (Saade et al., 2019)

2

4,818

2

14-15

11-12

Task Oriented Parsing (TOP) (Gupta et al., 2018)

1

44,873

2

25

36

Multilingual Task-Oriented Semantic Parsing
(MTOP) (Li et al., 2021)

6

15,195-22,288

11

104-113

72-75

Cross-Lingual Multilingual Task Oriented Dialog
(Schuster et al., 2019)

3

5,083-43,323

3

12

11

Microsoft Dialog Challenge (Li et al., 2018)

1

38,276

3

11

29

Fluent Speech Commands (FSC)
(Lugosch et al., 2019)

1

30,043

-

31

-

Chinese Audio-Textual Spoken Language
Understanding (CATSLU) (Zhu et al., 2019)

1

16,258

4

-

94

We have released a paper describing the dataset and presenting baseline modeling results on XLM-R and mT5 models. Tools for the dataset, as well as the modeling code used for our baseline results, are available in our Github repository. MASSIVE is licensed under the CC BY 4.0 license, encouraging its broadest possible use across academia and industry.

MMNLU competition and workshop

The MASSIVE leaderboard and the Massively Multilingual NLU 2022 competition, hosted on eval.ai, are composed of two tasks. In the first, called MMNLU-22-Full, each competitor trains and tests a single model on all 51 languages of the full MASSIVE dataset. In the second task, called MMNLU-22-ZeroShot, each competitor fine-tunes a pretrained model only with English-labeled data and tests it on all 50 non-English languages.

Related content
As Alexa expands into new countries, she usually has to be trained on new languages. But sometimes, she has to be re-trained on languages she’s already learned. British English, American English, and Indian English, for instance, are different enough that for each of them, we trained a new machine learning model from scratch.

This assesses the model’s ability to generalize to new languages, an important consideration given the number of languages around the world for which there is little-to-no labeled data. Zero-shot learning is a key technology for scaling NLU technology to many more low-resource languages worldwide.

The permanent MASSIVE leaderboard has been launched, and on July 25 the Massively Multilingual NLU 2022 evaluation split will be released. Participants will then have until August 8 to perform inference on the evaluation set and submit their predictions, which will be used to determine the winners. Winners will be invited to give an oral presentation at the Massively Multilingual NLU 2022 workshop.

The Massively Multilingual NLU 2022 workshop is collocated with EMNLP 2022 and will take place on either December 7 or 8, both in person in Abu Dhabi and online. Paper submissions spanning the breadth of multilingualism in NLU are sought, and the first call for papers will be released soon. The workshop will feature speakers on various topics related to multilingualism and NLU, as well as talks from the top performers from the MMNLU-22 competition.

Related content
In a paper we’re presenting at this year’s Conference on Empirical Methods in Natural Language Processing, we describe experiments with a new data selection technique.

Let’s scale natural-language-understanding technology to every language on Earth. Come build with us!

Acknowledgments

Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan for core dataset contributions; Andrew Turner for product and program management; Anna-Karin Johansson for vendor management; Saleh Soltan for text-to-text modeling discussions; Anne Yoder, Zheng Xie, Adeetee Bhide, Misa Sunaga, Trang Doan, and Satyam Dwivedi for program management and language expertise; Wayne Blossom, Brendan Egan, Columbine Marshall, Todd Tieuli, and Augusta Niles for creating the hidden evaluation split of the dataset; Jack FitzGerald, Kay Rottmann, Julia Hirschberg, Anna Rumshisky, and Mohit Bansal for workshop organization; and Charith Peris and Jack FitzGerald for leaderboard and competition setup.

Research areas

Related content

US, NY, New York
We are seeking an Applied Scientist to develop and optimize Visual Inertial Odometry (VIO) and sensor fusion systems for our intelligent robots. In this role, you will design, implement, and deploy state estimation and tracking algorithms that enable robots to understand their position and motion in real time, even in challenging and dynamic environments. You will own the full pipeline from algorithm development through embedded deployment, ensuring that perception systems run efficiently on resource-constrained robotic hardware. You will also leverage modern machine learning approaches to push the boundaries of classical perception methods, combining learned representations with geometric techniques to achieve robust, real-time performance. This is a deeply hands-on role. You will work directly with sensors, hardware, and real-world data, while prototyping, testing, and iterating in physical environments. The ideal candidate has strong foundations in VIO and sensor fusion, practical experience optimizing algorithms for embedded platforms, and familiarity with how modern deep learning is transforming perception. Key job responsibilities - Design and implement Visual Inertial Odometry algorithms for robust real-time state estimation on robotic platforms like Sprout - Develop multi-sensor fusion pipelines integrating cameras, IMUs, and other sensing modalities for accurate pose tracking - Optimize perception and tracking algorithms for deployment on embedded hardware (e.g., ARM, GPU-accelerated edge devices) under strict latency and power constraints - Apply modern ML-based perception techniques (learned features, depth estimation, neural odometry) to complement and improve classical geometric approaches - Build and maintain calibration, evaluation, and benchmarking infrastructure for perception systems - Collaborate with hardware, controls, and navigation teams to integrate perception outputs into the robot’s autonomy stack - Lead technical projects from research prototyping through production deployment
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the limits. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. As an Applied Scientist on our team, you will focus on building state-of-the-art ML models for biology. Our team rewards curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the forefront of both academic and applied research in this product area, you have the opportunity to work together with a diverse and talented team of scientists, engineers, and product managers and collaborate with other teams. Key job responsibilities - Build, adapt and evaluate ML models for life sciences applications - Collaborate with a cross-functional team of ML scientists, biologists, software engineers and product managers
US, MA, Boston
MULTIPLE POSITIONS AVAILABLE Employer: AMAZON.COM SERVICES LLC Offered Position: Economist III Job Location: Boston, Massachusetts Job Number: AMZ9898444 Position Responsibilities: Mentor and guide the applied scientists and economists in our organization and hold us to a high standard of technical rigor and excellence in science. Design and lead roadmaps for complex science projects to help SP have a delightful selling experience while creating long term value for our shoppers. Work with our engineering partners and draw upon your experience to meet latency and other system constraints. Identify untapped, high-risk technical and scientific directions, and simulate new research directions that you will drive to completion and deliver. Be responsible for communicating our science innovations to the broader internal & external scientific community. Position Requirements: Ph.D. or foreign equivalent degree in Economics or a related field and two years of research or work experience in the job offered or a related occupation. Must have two years of research or work experience in the following skill(s): 1) experience in econometrics including experience with program evaluation, forecasting, time series, panel data, or high dimensional problems; 2) experience with economic theory and quantitative methods; and 3) coding in a scripting language such as R, Python, or similar. Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation. 40 hours / week, 8:00am-5:00pm, Salary Range $159,200/year to $215,300/year. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits.#0000
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues.
US, WA, Seattle
Amazon's Worldwide Pricing & Promotions organization is seeking a talented, hands-on Research Scientist to join the Pricing and Promotion Optimization Science (P2OS) team — the optimization "application layer" within Amazon's Pricing Sciences organization. Amazon adjusts prices on hundreds of millions of products daily across a global marketplace; P2OS is the team that makes those prices optimal. P2OS is a small, specialized unit with an outsized charter: develop and maintain the models that determine optimal prices and promotions across Amazon's catalog and merchant programs. We own the full optimization stack — from price prediction to promotion targeting to competitiveness guardrails — and we measure success in terms of accretive Gross Contribution and Customer Pricing Perception (GCCP). Our work spans Retail Core, Amazon Business, Fresh, Grocery, and international marketplaces, and we are continually investing in more extensible, generalizable science foundations to keep pace with a growing and evolving business. We are looking for an innovative, organized, and customer-focused scientist with exceptional machine learning and predictive modeling skills, causal and experimental evaluation experience, and the entrepreneurial spirit to apply state-of-the-art methods to some of the most impactful pricing problems in e-commerce. You should be comfortable with ambiguity, motivated by measurable business impact, and excited by the opportunity to work at Amazon-scale. Key job responsibilities * Innovate and build. Design, develop, and deploy machine learning models that set optimal prices and promotions across Amazon's global catalog. Own models end-to-end — from problem formulation and data analysis through offline evaluation, A/B testing, and production launch. * Build a generalizable science foundation. Develop models and evaluation frameworks designed to scale across merchant programs, product categories, and marketplaces — enabling cross-learning and reducing the time and cost of applying science to new business contexts. * Build and evolve optimization systems. Design and improve optimization systems — including reinforcement learning and multi-objective optimization approaches — that automate price and promotion decisions at scale across millions of products. * Apply generative AI and foundation models. Identify and pursue opportunities to leverage large language models, embeddings, and generative AI techniques in pricing science — from enriching product representations and extracting competitive signals from unstructured data, to building more capable and explainable pricing systems. * Experiment rigorously. Design and execute A/B tests and causal inference studies to measure the business and customer impact of pricing model changes. Translate findings into production-ready science improvements. * Stay at the frontier. Establish mechanisms to track the latest advances in reinforcement learning, causal ML, multi-objective optimization, generative AI, and demand modeling — and identify opportunities to apply them to Pricing & Promotions business problems. * See the big picture. Contribute to the long-term scientific vision for how Amazon sets competitive, perception-preserving prices — balancing profitability, customer trust, and marketplace health.
US, CA, San Francisco
Amazon is on a mission to redefine the future of automation — and we're looking for exceptional talent to help lead the way. We are building the next generation of advanced robotic systems that seamlessly blend cutting-edge AI, sophisticated control systems, and novel mechanical design to create adaptable, intelligent automation solutions capable of operating safely alongside humans in dynamic, real-world environments. At Amazon, we leverage the power of machine learning, artificial intelligence, and advanced robotics to solve some of the most complex operational challenges at a scale unlike anywhere else in the world. Our fleet of robots spans hundreds of facilities globally, working in sophisticated coordination to deliver on our promise of customer excellence — and we're just getting started. As a Sr. Scientist in Robot Navigation, you will be at the forefront of this transformation — architecting and delivering navigation systems that are intelligent, safe, and scalable. You will bring deep expertise in learning-based planning and control, a strong understanding of foundation models and their application to embodied agents, and as well as have in-depth understanding of control-theoretic approaches such as model predictive control (MPC)-based trajectory planning. You will develop navigation solutions that seamlessly blend data-driven intelligence with principled control-theoretic guarantees. Our vision is bold: to build navigation systems that allow robots to move fluidly and safely through dynamic environments — understanding context, anticipating change, and adapting in real time. You will lead research that bridges the gap between cutting-edge academic advances and production grade deployment, collaborating with world-class teams pushing the boundaries of robotic autonomy, manipulation, and human-robot interaction. Join us in building the next generation of intelligent navigation systems that will define the future of autonomous robotics at scale. Key job responsibilities - Design, develop, and deploy perception algorithms for robotics systems, including object detection, segmentation, tracking, depth estimation, and scene understanding - Lead research initiatives in computer vision, sensor fusion and 3D perception - Collaborate with cross-functional teams including robotics engineers, software engineers, and product managers to define and deliver perception capabilities - Drive end-to-end ownership of ML models — from data collection and labeling strategy to training, evaluation, and deployment - Mentor junior scientists and engineers; contribute to a culture of technical excellence - Define and track key metrics to measure perception system performance in real-world environments - Publish research findings in top-tier venues (CVPR, ICCV, ECCV, ICRA, NeurIPS, etc.) and contribute to patents A day in the life - Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment - Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations - Actively contribute to brainstorming sessions on adjacent topics, bringing fresh perspectives that help peers grow and succeed — and in doing so, build lasting trust across the team - Mentor team members while maintaining significant hands-on contribution to technical solutions About the team Our team is a group is a diverse group of scientists and engineers passionate about building intelligent machines. We value curiosity, rigor, and a bias for action. We believe in learning from failure and iterating quickly toward solutions that matter.
GB, London
Are you excited about using econometrics, experimentation, and machine learning to impact real-world business decisions? We are looking for an Economist II to work on challenging problems at the intersection of causal inference and machine learning for Prime Video Ads. You will design experiments, build econometric and ML models, and translate findings into decisions that shape how millions of customers experience advertising on Prime Video. If you have a deeply quantitative approach to problem-solving, enjoy building and implementing models end-to-end, and want to work on problems where rigorous economics meets production-scale ML, we want to talk to you. Key job responsibilities - Design, execute, and analyze experiments to measure the impact of ad policies on customer behavior and business outcomes - Develop causal inference models (experimental and observational) to estimate short- and long-term effects of strategic initiatives - Collaborate with scientists, engineers, and product teams to deliver measurable business impact - Influence business leaders based on empirical findings
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities The successful candidate will: - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues. A day in the life This is a unique and rare opportunity to get in early on a fast-growing segment of AWS and help shape the technology, product and the business. You will have a chance to utilize your deep technical experience within a fast moving, start-up environment and make a large business and customer impact. About the team Diverse Experiences Amazon Automated Reasoning values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying. Why Amazon Automated Reasoning? At Amazon, automated reasoning is central to maintaining customer trust and delivering delightful customer experiences. Our organization is responsible for creating and maintaining a high bar for automated reasoning across all of Amazon's products and services. We offer talented automated reasoning professionals the chance to accelerate their careers with opportunities to build experience in a wide variety of areas including cloud, devices, retail, entertainment, healthcare, operations, and physical stores. Inclusive Team Culture In Amazon Automated Reasoning, it's in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest automated reasoning challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. Training & Career Growth We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve.
US, WA, Seattle
Applied Scientists in AWS Automated Reasoning are dedicated to making AWS the best computing service in the world for customers who require advanced and rigorous solutions for automated reasoning, privacy, and sovereignty. Key job responsibilities The successful candidate will: - Solve large or significantly complex problems that require deep knowledge and understanding of your domain and scientific innovation. - Own strategic problem solving, and take the lead on the design, implementation, and delivery for solutions that have a long-term quantifiable impact. - Provide cross-organizational technical influence, increasing productivity and effectiveness by sharing your deep knowledge and experience. - Develop strategic plans to identify fundamentally new solutions for business problems. - Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues. A day in the life This is a unique and rare opportunity to get in early on a fast-growing segment of AWS and help shape the technology, product and the business. You will have a chance to utilize your deep technical experience within a fast moving, start-up environment and make a large business and customer impact. About the team Diverse Experiences Amazon Automated Reasoning values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying. Why Amazon Automated Reasoning? At Amazon, automated reasoning is central to maintaining customer trust and delivering delightful customer experiences. Our organization is responsible for creating and maintaining a high bar for automated reasoning across all of Amazon's products and services. We offer talented automated reasoning professionals the chance to accelerate their careers with opportunities to build experience in a wide variety of areas including cloud, devices, retail, entertainment, healthcare, operations, and physical stores. Inclusive Team Culture In Amazon Automated Reasoning, it's in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest automated reasoning challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. Training & Career Growth We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve.
US, NY, New York
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. About the team SPB Agent team's vision is to build a highly personalized and context-aware agentic advertiser guidance system that seamlessly integrates Large Language Models (LLMs) with sophisticated tooling, operating across all experiences. The SPB-Agent is the central agent that interfaces with advertisers across Ads Console, Selling Partner portals (Seller Central, KDP, Vendor Central), and internal Sales systems. We identify high-impact opportunities spanning from strategic product guidance to granular optimization and deliver them through personalized, scalable experiences grounded in state-of-the-art agent architectures, reasoning frameworks, sophisticated tool integration, and model customization approaches including fine-tuning, MCP, and preference optimization. This presents an exceptional opportunity to shape the future of e-commerce advertising through advanced AI technology at unprecedented scale, creating solutions that directly impact millions of advertisers.