New dataset for training household robots to follow human commands

Publicly released TEACh dataset contains more than 3,000 dialogues and associated visual data from a simulated environment.

Through smart-home devices and systems, customers can already instruct Alexa to do things like open garage doors, turn lights on and off, or start the dishwasher. But we envision a future in which AI assistants can help with a broader range of more-complex tasks, including performing day-to-day chores, such as preparing breakfast. 

To accomplish such tasks, AI assistants will need to interact with objects in the world, understand natural-language instructions to complete tasks, and engage in conversations with users to clarify ambiguous instructions.

Amazon launches new Alexa Prize SimBot Challenge

Today, Amazon also announced the Alexa Prize SimBot Challenge, a competition focused on helping develop next-generation virtual assistants that will assist humans in completing real-world tasks. One of the TEACh benchmarks will be the basis of the challenge's public-benchmark phase.

To aid in the development of such AI assistants, we have publicly released a new dataset called TEACh, for Task-driven Embodied Agents that Chat. TEACh contains over 3,000 simulated dialogues, in which a human instructs a robot in the completion of household tasks, and associated visual data from a simulated environment.

For each dialogue, the roles of human and robot were played by paid crowd workers. The worker playing the robot did not know what task needed to be completed but depended entirely on the other worker’s instructions. Each worker received a visual feed that reflected a first-person point of view on the simulated environment. Both workers could move freely through the environment, but only the robot could interact with objects. The workers needed to collaborate and communicate to successfully complete tasks.

The simulated home environment is based on the AI2-THOR simulator, which includes 30 variations on each of four types of rooms: kitchens, living rooms, bedrooms, and bathrooms. Each gameplay session in the dataset consists of the initial and final states of the simulated environment, a task defined in terms of object properties to be satisfied, and a sequence of actions taken by the crowd workers. 

Those actions could include movement through the environment, interactions with objects (the robot can pick and place objects, open and close cabinets, drawers, and appliances, toggle lights on and off, operate appliances and faucets, slice objects, and pour liquid out of one object into another).

teach_blog_post_fig_1_updated.png
A sample gameplay session for the Prepare Breakfast task, where the robot has to make coffee and a sandwich with lettuce. The user offers step-by-step instructions but occasionally provides the next step — for example, slicing bread — before the robot has finished the previous step. Occasionally, the user offers help too late, as when the robot finds the knife by searching for it because the user does not provide its location.

Data collection

To collect the dataset, we first developed a task definition language that let us specify what properties needed to be satisfied in the environment for a task to be considered complete.  For example, to check that coffee is made, we confirm that there exists a clean mug in the environment that is filled with coffee. We implement a framework to check the AI2-THOR simulator for the status of different tasks, and we provide natural-language prompts for the steps remaining to complete a task. 

We then pair two crowd workers using a web interface and place them in the same simulated room. The user can see the prompts describing what steps need to be completed and uses chat to communicate them to the robot. Additionally, the user can determine where important objects are by either clicking on the steps or searching the virtual space, so that, for example, the robot does not have to open every drawer in the kitchen to find a knife hidden in one of them. 

Task Definition Language and Progress Check.png
An example task definition from the dataset (left) and the views of the simulated environment (right) that let the crowd worker playing the role of the user monitor progress toward task completion.

We place no constraints on the chat interface used by the annotators, and as a result, users provide instructions with different levels of granularity. One might say, “First get a clean mug and prepare coffee,” while another might break this up into several steps — “Grab the dirty mug out of the fridge”, “go wash it in the sink”, “place mug in coffee maker” — waiting for the robot to complete each step before providing the next one.

A user might provide instructions too early — for example, asking the robot to slice bread before it has finished preparing coffee — or too late — telling the robot where the knife is only after it has found it and sliced the bread with it. The user might also help the robot correct mistakes or get unstuck — for example, asking the robot to clear out the sink before placing a new object in it.

In total, we collected 4,365 sessions, of which 3,320 were successful. Of those, we were able to successfully replay 3,047 on the AI2-THOR simulator, meaning that providing the same sequence of actions resulted in the same simulator state. TEACh sessions span all 30 kitchens in the simulator and most of the living rooms, bedrooms, and bathrooms. The successful TEACh sessions span 12 task types and consist of more than 45,000 utterances, with an average of 8.40 user and 5.25 robot utterances per session. 

Benchmarks

We propose three benchmark tasks that machine learning models can be trained to perform using our dataset: execution from dialogue history (EDH), trajectory from dialogue (TfD),  and two-agent task completion (TATC). 

In the EDH benchmark, the model receives some dialogue history, previous actions taken by the robot, and the corresponding first-person observations from a collected gameplay session. The model is expected to predict the next few actions the robot will take, receiving a first-person observation after each action. The model is judged on whether its actions yield the same result that the player’s actions did in the original gameplay session.

The EDH benchmark will also be the basis for the public-benchmark phase of the Alexa Prize SimBot Challenge, which we also announced today. The SimBot Challenge is focused on helping advance development of next-generation virtual assistants that will assist humans in completing real-world tasks by continuously learning and gaining the ability to perform commonsense reasoning.

In the TfD benchmark, a model receives the complete dialogue history and has to predict all the actions taken by the robot, receiving a first-person observation after each action. 

In the TATC benchmark, the designer needs to build two models, one for the user and one for the robot. The user model receives the same task information that the human worker did, as well as the state of the environment. It has to communicate with the robot model, which takes actions in the environment to complete tasks. 

We include baseline model performance on these benchmarks in a paper we’ve published to the arXiv, which we hope will be used as a reference for future work by other research groups.  

For the EDH and TfD benchmarks, we created “validation-seen” and “test-seen” splits, which evaluate the ability of models to generalize to new dialogues and execution paths in the rooms used for training, and “validation-unseen” and “test-unseen” splits, which evaluate the ability of models to generalize to dialogues and execution paths in rooms never previously seen. These splits are designed to enable easy model transfer to and from a related dataset, ALFRED, which also uses floorplans from AI2-THOR and splits the data similarly.

Acknowledgements: This project came together through the efforts and support of several people on the Alexa AI team. We would like to thank  Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, Dilek Hakkani-Tür, Ron Rezac, Shui Hu, Lucy Hu, Hangjie Shi, Nicole Chartier, Savanna Stiff, Ana Sanchez, Ben Kelk, Joel Sachar, Govind Thattai, Gaurav Sukhatme, Joel Chengottusseriyil, Tony Bissell, Qiaozi Gao, Kaixiang Lin, Karthik Gopalakrishnan, Alexandros Papangelis, Yang Liu, Mahdi Namazifar, Behnam Hedayatnia, Di Jin, and Seokhwan Kim for their contributions to the project.  

Latest news

The latest updates, stories, and more about Alexa Prize.
US, WA, Seattle
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies to deliver game changing value to our customers. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon’s Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon’s on-line retail business. As an economist on our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies with the potential to deliver game changing value to our customers. This is an opportunity for a high-energy individual to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals.
US, CA, San Francisco
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of statistical inference, experimentation design, economic theory and machine learning to design new methods and pricing strategies for assessing pricing innovations. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon's Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon's on-line retail business. As an economist on our team, you will will have the opportunity to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in experimentation design, applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of interns from previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US
The Amazon Supply Chain Optimization Technology (SCOT) organization is looking for an Intern in Economics to work on exciting and challenging problems related to Amazon's worldwide inventory planning. SCOT provides unique opportunities to both create and see the direct impact of your work on billions of dollars’ worth of inventory, in one of the world’s most advanced supply chains, and at massive scale. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Knowledge of econometrics and Stata/R/or Python is necessary, and experience with SQL, Hadoop, and Spark would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Seattle
The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. We are looking for an Intern Economist with excellent coding skills to design and develop rigorous models to assess the causal impact of fees on third party sellers’ behavior and business performance. As a Science Intern, you will have access to large datasets with billions of transactions and will translate ambiguous fee related business problems into rigorous scientific models. You will work on real world problems which will help to inform strategic direction and have the opportunity to make an impact for both Amazon and our Selling Partners.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Some knowledge of econometrics, as well as basic familiarity with Stata or R is necessary, and experience with SQL, Hadoop, Spark and Python would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking interns and co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it! Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, artificial intelligence, human-robot interaction, optimization and more.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking interns and co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects, including allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. We are seeking internship candidates with backgrounds in computer vision, machine learning, resource allocation, discrete optimization, search, and planning/scheduling. You will be challenged intellectually and have a good time while you are at it! Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, artificial intelligence, human-robot interaction, optimization and more.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, MA, Boston
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics, a wholly owned subsidiary of Amazon.com, empowers a smarter, faster, more consistent customer experience through automation. Amazon Robotics automates fulfillment center operations using various methods of robotic technology including autonomous mobile robots, sophisticated control software, language perception, power management, computer vision, depth sensing, machine learning, object recognition, and semantic understanding of commands. Amazon Robotics has a dedicated focus on research and development to continuously explore new opportunities to extend its product lines into new areas. AR is seeking uniquely talented and motivated data scientists to join our Global Services and Support (GSS) Tools Team. GSS Tools focuses on improving the supportability of the Amazon Robotics solutions through automation, with the explicit goal of simplifying issue resolution for our global network of Fulfillment Centers. The candidate will work closely with software engineers, Fulfillment Center operation teams, system engineers, and product managers in the development, qualification, documentation, and deployment of new - as well as enhancements to existing - operational models, metrics, and data driven dashboards. As such, this individual must possess the technical aptitude to pick-up new BI tools and programming languages to interface with different data access layers for metric computation, data mining, and data modeling. This role is a 6 month co-op to join AR full time (40 hours/week) from July – December 2023. The Co-op will be responsible for: Diving deep into operational data and metrics to identify and communicate trends used to drive development of new tools for supportability Translating operational metrics into functional requirements for BI-tools, models, and reporting Collaborating with cross functional teams to automate AR problem detection and diagnostics