How a universal model is helping one generation of Amazon robots train the next

New approach can cut the setup time required to develop vision-based machine learning solutions from between six to twelve months to one or two.

A fundamental theme at Amazon is movement. Obtaining a product ordered by a customer and moving that product as quickly and efficiently as possible from its source to the customer’s doorstep.

This video shows robots moving packages around an Amazon fulfillment center.

That journey will often take a package through multiple warehouses and include loadings, unloadings, sortings, and routings. Human associates are crucial to this process and so, increasingly, are robotic manipulators. A rising star in this department is the Robin robotic arm and the computer vision system that makes it possible.

Robin’s visual-perception algorithms can identify and locate packages on a conveyor belt below it, for example, and even distinguish individual packages and their type within a cluttered pile.

This perceptive ability is known as segmentation, and it is central to the development of flexible and adaptive robotic processes for Amazon fulfillment centers. That’s because packages vary enormously in their dimensions and physical characteristics, moving amid an ever-changing mix of packages and against varying backdrops.

Amazon's Robin robot arm is seen lifting packages
Robin’s visual-perception algorithms can identify and locate packages on a conveyor belt below it, for example, and even distinguish individual packages and their type within a cluttered pile.

Robin is a maturing technology, but there is a constant simmering of new ideas just below the surface at Amazon, with teams of scientists and engineers across the Amazon Robotics AI group and beyond collaborating to develop AI-powered robotic solutions to improve warehouse efficiency. A new modeling approach aims to serve them all.

An abundance of packages — but not data

The initial challenge for these early-stage collaborations is often the same.

“The biggest problem that new project teams usually face is data scarcity,” says Cassie Meeker, an Amazon Robotics AI applied scientist, based in Seattle. Obtaining images relevant to a warehouse process of interest takes time and resources, but that’s just the beginning.

Cassie Meeker, an Amazon Robotics AI applied scientist, is seen standing in front of a Robin robot arm
Cassie Meeker, an Amazon Robotics AI applied scientist, says she and her team started their quest to develop universal models by utilizing publicly available datasets to give their model basic classification skills.

“For some machine learning models, you must annotate each training image manually by drawing multiple polygons around the various packages in the picture,” Meeker explains. “It can take five minutes to annotate just one image if it’s cluttered.”

The lack of task-specific training data means teams might base their perceptual models on just a few hundred images, says Meeker: “If they're lucky, they have a thousand. But even a thousand images aren’t a lot for training a model.”

If new projects do not have sufficient variety in their training data, that’s a challenge.

“The production environment is typically very different to a prototyping environment, so when they go into the production phase on the warehouse floor, they will suddenly see all these things they've never seen before and that their perception system can’t identify,” says Meeker. “They could be setting themselves up for failure.”

This difficulty in obtaining data to train segmentation models is partly due to the very specific subject matter: packages. Many computer vision models are trained on enormous, publicly available datasets full of annotated imagery, including everything from aardvarks to zabaglione. A social media company might want to segment faces, or dogs or cats, because that’s what people have lots of pictures of.

“Many publicly available datasets are perfect for that,” says Meeker. “But at Amazon, we have such a specific application and annotation requirements. It just doesn’t translate well from cat pics.”

A ’universal model’ for packages

In short, building a dataset big enough to train a demanding machine learning model requires time and resources, with no guarantee that the novel robotic process you are working toward will prove successful. This became a recurring issue for Amazon Robotics AI. So this year, work began in earnest to address the data scarcity problem. The solution: a “universal model” able to generalize to virtually any package segmentation task.

To develop the model, Meeker and her colleagues first used publicly available datasets to give their model basic classification skills — being able to distinguish boxes or packages from other things, for example. Next, they honed the model, teaching it to distinguish between many types of packaging in warehouse settings — from plastic bags to padded mailers to cardboard boxes of varying appearance — using a trove of training data compiled by the Robin program and half a dozen other Amazon teams over the last few years. This dataset comprised almost half a million annotated images.

Meet the Amazon robot improving safety

Crucially, these images of packages were snapped from a variety of angles — not only straight down from above a conveyor belt — and against a variety of backgrounds. The sheer number and variation of images make the dataset useful in virtually any warehouse location that may benefit from robotic perception and manipulation.

Meeker estimates that starting a project with the universal model can slash the setup time required to develop vision-based ML solutions from between six to twelve months to just one or two. And it has been made available to other Amazon teams in a user-friendly form, so extensive machine learning expertise is not required.

The universal model has already demonstrated its prowess, courtesy of a project run by Amazon Robotics, called Cardinal. Cardinal is a prototype robotic arm-based system that perceives and picks up packages and places them neatly into large containers ready for transport on delivery trucks. Cardinal’s perception system was developed before the universal model was available, so the team spent a lot of time creating a bespoke training dataset for it, says Cardinal’s perception lead, Jeroen van Baar, an Amazon Robotics senior applied scientist, based in North Reading, Massachusetts.

This video shows Cardinal training itself to distinguish between package types.

“We trained the system using 25,000 annotated training images that we created ourselves. But those early training images were taken using a setup with a different appearance to our prototype Cardinal workstation,” van Baar says. “To achieve the performance that we initially desired, we had to fine-tune our model using a thousand new training images taken from that prototype setting.”

After being updated with only those new images, the universal model was as accurate for performing Cardinal’s task as the workstation’s own robust model.

“Had it been available sooner, I would only have captured data specific to our setup and fine-tuned the universal model from there,” says van Baar. “Being able to shorten training time so significantly is a major benefit.”

Related content
Company is testing a new class of robots that use artificial intelligence and computer vision to move freely throughout facilities.

And that’s the point. The universal model can quickly capitalize on any training data produced by a new-project team. This means that when new ideas are tested on the warehouse floor, or existing methods are transplanted to a new Amazon region where things are done slightly differently, the model will have enough data diversity to handle the differences.

Siddhartha Srinivasa, director of Robotics AI, thinks of the universal model as a supportive scaffold that you can use to build your house.

“We're not advocating that everybody live in the same house,” he says. “We're advocating that Amazon teams leverage the scaffolding we're providing to build whatever house they want, because it’s already very powerful, and it is getting better every day.”

Tipping point

Only recently has all this become possible.

“The Robotics AI program is young,” says Meeker. “In the beginning, there was no reason to use other teams’ data, because no one had very much.” But a tipping point has arrived. “We now have enough mature teams in production that we are seeing a real diversity and scaling of data. It is finally generalizable.”

Indeed, while the immediate focus of universal models is identifying and localizing various package types, diverse image data is now accumulating across a range of Amazon programs that cover more aspects of fulfillment centers.

Related content
Why detecting damage is so tricky at Amazon’s scale — and how researchers are training robots to help with that gargantuan task.

The universal model now includes images of unpackaged items, too, allowing it to perform segmentation across a greater diversity of warehouse processes. Initiatives such as multimodal identification, which aims to visually identify items without needing to see a barcode, and the automated damage detection program are accruing product-specific data that could be fed into the universal model, as well as images taken on the fulfillment center floor by the autonomous robots that carry crates of products.

“We’re moving towards a situation in which even data collected by small projects run by interns can be fed into the universal base model, incrementally improving the productivity of the entire robot fleet,” says Srinivasa.

We’re moving towards a situation in which even data collected by small projects run by interns can be fed into the universal base model, incrementally improving the productivity of the entire robot fleet.
Siddhartha Srinivasa

This diversity of data and its aggregation is particularly important for robotic perception within Amazon, especially given customers’ shifting needs, frequently novel Amazon packaging, and the company’s commitment to sustainability that means shipping more items in their own unique packaging.

All of this increases the visual variety of products and packages, making it harder for robots to identify from an image where one package ends and another begins.

Feeding the universal model in this way and having it available to new teams will accelerate the experimentation and deployment of future robotic processes. The use of the universal model is factored into Amazon’s immediate operational plans.

“We’re not doing this because it's cool — though it really is cool — but because it is inevitable,” says Srinivasa.

Related content

LU, Luxembourg
The Decision, Science and Technology (DST) team part of the global Reliability Maintenance Engineering (RME) is looking for a Senior Operations Research Scientist interested in solving challenging optimization problems in the maintenance space. Our mission is to leverage the use of data, science, and technology to improve the efficiency of RME maintenance activities, reduce costs, increase safety and promote sustainability while creating frictionless customer experiences. As a Senior OR Scientist in DST you will be focused on leading the design and development of innovative approaches and solutions by leading technical work supporting RME’s Predictive Maintenance (PdM) and Spare Parts (SP) programs. You will connect with world leaders in your field and you will be tackling customer's natural language challenges by carrying out a systematic review of existing solutions. The appropriate choice of methods and their deployment into effective tools will be the key for the success in this role. The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail and outstanding ability in balancing technical leadership with strong business judgment to make the right decisions about model and method choices. Key job responsibilities • Provide technical expertise to support team strategies that will take EU RME towards World Class predictive maintenance practices and processes, driving better equipment up-time and lower repair costs with optimized spare parts inventory and placement • Implement an advanced maintenance framework utilizing Machine Learning technologies to drive equipment performance leading to reduced unplanned downtime • Provide technical expertise to support the development of long-term spares management strategies that will ensure spares availability at an optimal level for local sites and reduce the cost of spares A day in the life As a Senior OR Scientist in DST you will be focused on leading the design and development of innovative approaches and solutions by leading technical work supporting RME’s Predictive Maintenance (PdM) and Spare Parts (SP) programs. You will connect with world leaders in your field and you will be tackling customer's natural language challenges by carrying out a systematic review of existing solutions. The appropriate choice of methods and their deployment into effective tools will be the key for the success in this role. About the team Our mission is to leverage the use of data, science, and technology to improve the efficiency of RME maintenance activities, reduce costs, increase safety and promote sustainability while creating frictionless customer experiences. We are open to hiring candidates to work out of one of the following locations: Luxembourg, LUX
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, CA, Santa Clara
AWS AI Research and Engineering (AIRE) is looking for world class scientists and engineers to work on the development of autonomous AI agents. At AWS AI/ML you will invent, implement, and deploy state of the art machine learning algorithms and systems. You will build prototypes and innovate on new learning techniques. You will interact closely with our customers and with the academic and research communities. You will be at the heart of a growing and exciting focus area for AWS and work with other acclaimed engineers and world famous scientists. Large-scale foundation models have been the powerhouse in many of the recent advancements in computer vision, natural language processing, automatic speech recognition, recommendation systems, and time series modeling. Developing such models requires not only skillful modeling in individual modalities, but also understanding of how to seamlessly combine them, and how to scale the modeling methods to learn with huge models and on large datasets. We seek a strong technical leader with domain expertise in machine learning, large language models and multimodal models, reinforcement learning and setting up simulation environments to benchmark and evaluate. AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps services in AWS. Within AWS UC, Amazon Dedicated Cloud (ADC) roles engage with AWS customers who require specialized security solutions for their cloud services. About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Hybrid Work We value innovation and recognize this sometimes requires uninterrupted time to focus on a build. We also value in-person collaboration and time spent face-to-face. Our team affords employees options to work in the office every day or in a flexible, hybrid work model near one of our U.S. Amazon offices. We are open to hiring candidates to work out of one of the following locations: Santa Clara, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA