How a universal model is helping one generation of Amazon robots train the next

New approach can cut the setup time required to develop vision-based machine learning solutions from between six to twelve months to one or two.

A fundamental theme at Amazon is movement. Obtaining a product ordered by a customer and moving that product as quickly and efficiently as possible from its source to the customer’s doorstep.

This video shows robots moving packages around an Amazon fulfillment center.

That journey will often take a package through multiple warehouses and include loadings, unloadings, sortings, and routings. Human associates are crucial to this process and so, increasingly, are robotic manipulators. A rising star in this department is the Robin robotic arm and the computer vision system that makes it possible.

Robin’s visual-perception algorithms can identify and locate packages on a conveyor belt below it, for example, and even distinguish individual packages and their type within a cluttered pile.

This perceptive ability is known as segmentation, and it is central to the development of flexible and adaptive robotic processes for Amazon fulfillment centers. That’s because packages vary enormously in their dimensions and physical characteristics, moving amid an ever-changing mix of packages and against varying backdrops.

Amazon's Robin robot arm is seen lifting packages
Robin’s visual-perception algorithms can identify and locate packages on a conveyor belt below it, for example, and even distinguish individual packages and their type within a cluttered pile.

Robin is a maturing technology, but there is a constant simmering of new ideas just below the surface at Amazon, with teams of scientists and engineers across the Amazon Robotics AI group and beyond collaborating to develop AI-powered robotic solutions to improve warehouse efficiency. A new modeling approach aims to serve them all.

An abundance of packages — but not data

The initial challenge for these early-stage collaborations is often the same.

“The biggest problem that new project teams usually face is data scarcity,” says Cassie Meeker, an Amazon Robotics AI applied scientist, based in Seattle. Obtaining images relevant to a warehouse process of interest takes time and resources, but that’s just the beginning.

Cassie Meeker, an Amazon Robotics AI applied scientist, is seen standing in front of a Robin robot arm
Cassie Meeker, an Amazon Robotics AI applied scientist, says she and her team started their quest to develop universal models by utilizing publicly available datasets to give their model basic classification skills.

“For some machine learning models, you must annotate each training image manually by drawing multiple polygons around the various packages in the picture,” Meeker explains. “It can take five minutes to annotate just one image if it’s cluttered.”

The lack of task-specific training data means teams might base their perceptual models on just a few hundred images, says Meeker: “If they're lucky, they have a thousand. But even a thousand images aren’t a lot for training a model.”

If new projects do not have sufficient variety in their training data, that’s a challenge.

“The production environment is typically very different to a prototyping environment, so when they go into the production phase on the warehouse floor, they will suddenly see all these things they've never seen before and that their perception system can’t identify,” says Meeker. “They could be setting themselves up for failure.”

This difficulty in obtaining data to train segmentation models is partly due to the very specific subject matter: packages. Many computer vision models are trained on enormous, publicly available datasets full of annotated imagery, including everything from aardvarks to zabaglione. A social media company might want to segment faces, or dogs or cats, because that’s what people have lots of pictures of.

“Many publicly available datasets are perfect for that,” says Meeker. “But at Amazon, we have such a specific application and annotation requirements. It just doesn’t translate well from cat pics.”

A ’universal model’ for packages

In short, building a dataset big enough to train a demanding machine learning model requires time and resources, with no guarantee that the novel robotic process you are working toward will prove successful. This became a recurring issue for Amazon Robotics AI. So this year, work began in earnest to address the data scarcity problem. The solution: a “universal model” able to generalize to virtually any package segmentation task.

To develop the model, Meeker and her colleagues first used publicly available datasets to give their model basic classification skills — being able to distinguish boxes or packages from other things, for example. Next, they honed the model, teaching it to distinguish between many types of packaging in warehouse settings — from plastic bags to padded mailers to cardboard boxes of varying appearance — using a trove of training data compiled by the Robin program and half a dozen other Amazon teams over the last few years. This dataset comprised almost half a million annotated images.

Meet the Amazon robot improving safety

Crucially, these images of packages were snapped from a variety of angles — not only straight down from above a conveyor belt — and against a variety of backgrounds. The sheer number and variation of images make the dataset useful in virtually any warehouse location that may benefit from robotic perception and manipulation.

Meeker estimates that starting a project with the universal model can slash the setup time required to develop vision-based ML solutions from between six to twelve months to just one or two. And it has been made available to other Amazon teams in a user-friendly form, so extensive machine learning expertise is not required.

The universal model has already demonstrated its prowess, courtesy of a project run by Amazon Robotics, called Cardinal. Cardinal is a prototype robotic arm-based system that perceives and picks up packages and places them neatly into large containers ready for transport on delivery trucks. Cardinal’s perception system was developed before the universal model was available, so the team spent a lot of time creating a bespoke training dataset for it, says Cardinal’s perception lead, Jeroen van Baar, an Amazon Robotics senior applied scientist, based in North Reading, Massachusetts.

This video shows Cardinal training itself to distinguish between package types.

“We trained the system using 25,000 annotated training images that we created ourselves. But those early training images were taken using a setup with a different appearance to our prototype Cardinal workstation,” van Baar says. “To achieve the performance that we initially desired, we had to fine-tune our model using a thousand new training images taken from that prototype setting.”

After being updated with only those new images, the universal model was as accurate for performing Cardinal’s task as the workstation’s own robust model.

“Had it been available sooner, I would only have captured data specific to our setup and fine-tuned the universal model from there,” says van Baar. “Being able to shorten training time so significantly is a major benefit.”

Related content
Company is testing a new class of robots that use artificial intelligence and computer vision to move freely throughout facilities.

And that’s the point. The universal model can quickly capitalize on any training data produced by a new-project team. This means that when new ideas are tested on the warehouse floor, or existing methods are transplanted to a new Amazon region where things are done slightly differently, the model will have enough data diversity to handle the differences.

Siddhartha Srinivasa, director of Robotics AI, thinks of the universal model as a supportive scaffold that you can use to build your house.

“We're not advocating that everybody live in the same house,” he says. “We're advocating that Amazon teams leverage the scaffolding we're providing to build whatever house they want, because it’s already very powerful, and it is getting better every day.”

Tipping point

Only recently has all this become possible.

“The Robotics AI program is young,” says Meeker. “In the beginning, there was no reason to use other teams’ data, because no one had very much.” But a tipping point has arrived. “We now have enough mature teams in production that we are seeing a real diversity and scaling of data. It is finally generalizable.”

Indeed, while the immediate focus of universal models is identifying and localizing various package types, diverse image data is now accumulating across a range of Amazon programs that cover more aspects of fulfillment centers.

Related content
Why detecting damage is so tricky at Amazon’s scale — and how researchers are training robots to help with that gargantuan task.

The universal model now includes images of unpackaged items, too, allowing it to perform segmentation across a greater diversity of warehouse processes. Initiatives such as multimodal identification, which aims to visually identify items without needing to see a barcode, and the automated damage detection program are accruing product-specific data that could be fed into the universal model, as well as images taken on the fulfillment center floor by the autonomous robots that carry crates of products.

“We’re moving towards a situation in which even data collected by small projects run by interns can be fed into the universal base model, incrementally improving the productivity of the entire robot fleet,” says Srinivasa.

We’re moving towards a situation in which even data collected by small projects run by interns can be fed into the universal base model, incrementally improving the productivity of the entire robot fleet.
Siddhartha Srinivasa

This diversity of data and its aggregation is particularly important for robotic perception within Amazon, especially given customers’ shifting needs, frequently novel Amazon packaging, and the company’s commitment to sustainability that means shipping more items in their own unique packaging.

All of this increases the visual variety of products and packages, making it harder for robots to identify from an image where one package ends and another begins.

Feeding the universal model in this way and having it available to new teams will accelerate the experimentation and deployment of future robotic processes. The use of the universal model is factored into Amazon’s immediate operational plans.

“We’re not doing this because it's cool — though it really is cool — but because it is inevitable,” says Srinivasa.

Related content

US, VA, Arlington
The People eXperience and Technology Central Science Team (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, wellbeing, and the value of work to Amazonians. We are an interdisciplinary team that combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal. We are looking for economists who are able to apply economic methods to address business problems. The ideal candidate will work with engineers and computer scientists to estimate models and algorithms on large scale data, design pilots and measure their impact, and transform successful prototypes into improved policies and programs at scale. We are looking for creative thinkers who can combine a strong technical economic toolbox with a desire to learn from other disciplines, and who know how to execute and deliver on big ideas as part of an interdisciplinary technical team. Ideal candidates will work in a team setting with individuals from diverse disciplines and backgrounds. They will work with teammates to develop scientific models and conduct the data analysis, modeling, and experimentation that is necessary for estimating and validating models. They will work closely with engineering teams to develop scalable data resources to support rapid insights, and take successful models and findings into production as new products and services. They will be customer-centric and will communicate scientific approaches and findings to business leaders, listening to and incorporate their feedback, and delivering successful scientific solutions. Key job responsibilities Use causal inference methods to evaluate the impact of policies on employee outcomes. Examine how external labor market and economic conditions impact Amazon's ability to hire and retain talent. Use scientifically rigorous methods to develop and recommend career paths for employees. A day in the life Work with teammates to apply economic methods to business problems. This might include identifying the appropriate research questions, writing code to implement a DID analysis or estimate a structural model, or writing and presenting a document with findings to business leaders. Our economists also collaborate with partner teams throughout the process, from understanding their challenges, to developing a research agenda that will address those challenges, to help them implement solutions. About the team We are a multidisciplinary team that combines the talents of science and engineering to develop innovative solutions to make Amazon Earth's Best Employer.
US, WA, Seattle
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Knowledge of econometrics, as well as basic familiarity with Python (or R, Matlab, or equivalent) is necessary, and experience with SQL would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Virtual Contact Center-WA
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. About the team The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. Within the Science team, our goal is to understand the impact of changing fees on Seller (supply) and Customers (demand) behavior (e.g. price changes, advertising strategy changes, introducing new selection etc.) as well as using this information to optimize our fee structure and maximizing our long term profitability.
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies to deliver game changing value to our customers. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon’s Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon’s on-line retail business. As an economist on our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies with the potential to deliver game changing value to our customers. This is an opportunity for a high-energy individual to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals.
US, CA, San Francisco
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of statistical inference, experimentation design, economic theory and machine learning to design new methods and pricing strategies for assessing pricing innovations. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon's Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon's on-line retail business. As an economist on our team, you will will have the opportunity to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in experimentation design, applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of interns from previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US
The Amazon Supply Chain Optimization Technology (SCOT) organization is looking for an Intern in Economics to work on exciting and challenging problems related to Amazon's worldwide inventory planning. SCOT provides unique opportunities to both create and see the direct impact of your work on billions of dollars’ worth of inventory, in one of the world’s most advanced supply chains, and at massive scale. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Knowledge of econometrics and Stata/R/or Python is necessary, and experience with SQL, Hadoop, and Spark would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Seattle
The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. We are looking for an Intern Economist with excellent coding skills to design and develop rigorous models to assess the causal impact of fees on third party sellers’ behavior and business performance. As a Science Intern, you will have access to large datasets with billions of transactions and will translate ambiguous fee related business problems into rigorous scientific models. You will work on real world problems which will help to inform strategic direction and have the opportunity to make an impact for both Amazon and our Selling Partners.