Scenario Diffusion helps Zoox vehicles navigate safety-critical situations

Generative AI supports the creation, at scale, of complex, realistic driving scenarios that can be directed to specific locations and environments.

Autonomous vehicles (AVs) such as the Zoox purpose-built robotaxi represent a new era in human mobility, but the deployment of AVs comes with many challenges. It’s essential to do extensive safety testing using simulation, which requires the creation of synthetic driving scenarios at scale. Particularly important is generating realistic safety-critical road scenarios, to test how AVs will react to a wide range of driving situations, including those that are relatively rare and potentially dangerous.

Zoox robotaxi.png
The Zoox robotaxi.

Traditional methods tend to produce scenarios of limited complexity and struggle to replicate many real-world situations. More recently, machine learning (ML) models have used deep learning to produce complex traffic scenarios based on specified map regions, but they offer limited means of shaping the resulting scenarios in terms of vehicle positionings, speeds, and trajectories. This makes it difficult to create specific safety-critical scenarios at scale. Designing huge numbers of such scenarios by hand, meanwhile, is impractical.

Related content
Leveraging a large vision-language foundation model enables state-of-the-art performance in remote-object grounding.

In a paper we presented at the 2023 Conference on Neural Information Processing Systems (NeurIPS), we address these challenges with a method we call Scenario Diffusion. Our system comprises a novel ML architecture based on latent diffusion, an ML technique used in image generation in which a model learns to convert random noise into detailed images.

Scenario Diffusion is able to output highly controllable and realistic traffic scenarios, at scale. It is controllable because the outputs of the Scenario Diffusion model are based not only on the map of the desired area but also on sets of easily produced descriptors that can specify the positioning and characteristics of some or all of the vehicles in a scene. These descriptors, which we call agent tokens, take the form of feature vectors. We can similarly specify global scene tokens, which indicate how busy the roads in a given scenario should be.

Directed scenario generation.png
Providing the Scenario Diffusion model with additional information about the desired scenario directs the generative process.

Combining a diffusion architecture with these token-based controls allows us to produce safety-critical driving scenarios at will, boosting our ability to validate the safety of our purpose-built robotaxi. We are excited to apply generative AI where it can have a big impact on the established practical challenge of AV safety.

Inside the Scenario Diffusion model

AV control software is typically divided into perception, prediction, and motion-planning modules. On the road, an AV’s cameras and other sensors perceive the road situation, which can be represented, for motion-planning purposes, as a simplified bird’s-eye-view image.

Related content
Company is testing a new class of robots that use artificial intelligence and computer vision to move freely throughout facilities.

Each of the vehicles (“agents”) in this multi-channelled image, including the AV itself, is represented as a “bounding box” that reflects the vehicle’s width, length, and position on the local map. The image also contains information on other characteristics of the vehicles, such as heading and trajectory. These characteristics and the map itself are the two key elements of a synthetic driving scenario that are required to validate motion planning in simulation.

The Scenario Diffusion model has two components. The first is an autoencoder, which projects complex driving scenarios into a more manageable representational space. The second component, the diffusion model, operates in this space.

Like all diffusion models, ours is trained by adding noise to real-world scenarios and asking the model to remove this noise. Once the model is trained, we can sample random noise and use the model to gradually convert this noise into a realistic driving scenario. For a detailed exploration of our training and inference processes and model architecture, dive into our paper.

We trained the model on both public and proprietary real-world datasets of driving logs containing millions of driving scenarios across a variety of geographical regions and settings.

Prior ML methods for generating driving scenarios typically place the bounding boxes of agents on a map — essentially a static snapshot, with no motion information. They then use object recognition to identify those boxes before applying heuristics or learned methods to decide on suitable trajectories for each agent. Such hybrid solutions can struggle to capture the nuances of real-world driving.

Related content
A combination of cutting-edge hardware, sensor technology, and bespoke machine learning approaches can predict trajectories of vehicles, people, and even animals, as far as 8 seconds into the future.

A key contribution of our work is that it achieves the simultaneous inference of agent placement and behavior. When our trained model generates a traffic scenario for a given map, every agent it positions in the scene has an associated feature vector that describes its characteristics, such as the dimensions, orientation, and trajectory of the vehicle. The driving scenario emerges fully formed.

Our feature vector approach not only provides more-realistic scenarios but also makes it very easy to add information to the model, making it highly adaptable. In the paper, we deal only with standard vehicles, but it would be straightforward to generate more-complex scenarios that include bikes, pedestrians, scooters, animals — anything previously encountered by a Zoox robotaxi in the real world.

Creating safety-critical “edge cases” on demand

If we simply want to create many thousands of realistic driving scenarios, with no particular situation in mind, we let Scenario Diffusion freely generate traffic on a particular map. This type of approach has been explored in prior research. But randomly generated scenarios are not an efficient way to validate how AV software deals with rare, safety-critical events.

Initial map.png
The model is provided with a map and a set of tokens that define the characteristics of an autonomous vehicle (agent A, red) and a bus (agent B, orange) turning right up ahead.
Scenario diffusion.gif
In the diffusion part of the process, the scenario undergoes multiple rounds of de-noising until a realistic scenario featuring the specified vehicles emerges.
Scenario diffusion bounding boxes.png
The final scenario shows trajectories that extend from two seconds in the past (pink) to two seconds into the future (blue).

Imagine we want to validate how an AV will behave in a safety-critical situation — such as a bus turning right in front of it — on a given map. Creating such scenarios is straightforward for Scenario Diffusion, thanks to its use of agent tokens and global scene tokens. Agent tokens can easily be computed from data in real-life driving logs or created by humans. Then they can be used to prompt the model to place vehicles with desired characteristics in specific locations. The model will include those vehicles in its generated scenarios while creating additional agents to fill out the rest of the scene in a realistic manner.

With just one GPU, it takes about one second to generate a novel scenario.

Successful generalization across regions

To evaluate our model’s ability to generalize across geographical regions, we trained separate models on data from each region of the Zoox dataset. A model trained solely on driving logs from, say, San Francisco did better at generating realistic driving scenarios for San Francisco than a model trained on data from Seattle. However, models trained on the full Zoox dataset of four regions come very close to the performance of region-specialized models. These findings suggest that, while there are unique aspects of each region, the fully trained model has sufficient capacity to capture this diversity.

The ability to generalize to other cities is good news for the future of AV validation as Zoox expands into new metropolitan areas. It will always be necessary to collect real-world driving logs in new locations, using AVs outfitted with our full sensor architecture and monitored by a safety driver. However, the ability to generate supplementary synthetic data will shorten the time it takes to validate our AV control system in new areas.

We plan to build on this research by making the model’s output increasingly rich and nuanced, with a greater diversity of vehicle and object types, to better match the complexity of real streets. For example, we could ultimately design a model to generate highly complex safety scenarios, such as driving by a school location at dismissal time, with crowds of kids and parents near or spilling onto the road.

It is this powerful combination of flexibility, controllability, and increasing realism that we believe will make our Scenario Diffusion approach foundational to the future of safety validation for autonomous vehicles.

Acknowledgments: Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Nicholas Roy.

Research areas

Related content

US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
LU, Luxembourg
The Decision, Science and Technology (DST) team part of the global Reliability Maintenance Engineering (RME) is looking for a Senior Operations Research Scientist interested in solving challenging optimization problems in the maintenance space. Our mission is to leverage the use of data, science, and technology to improve the efficiency of RME maintenance activities, reduce costs, increase safety and promote sustainability while creating frictionless customer experiences. As a Senior OR Scientist in DST you will be focused on leading the design and development of innovative approaches and solutions by leading technical work supporting RME’s Predictive Maintenance (PdM) and Spare Parts (SP) programs. You will connect with world leaders in your field and you will be tackling customer's natural language challenges by carrying out a systematic review of existing solutions. The appropriate choice of methods and their deployment into effective tools will be the key for the success in this role. The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail and outstanding ability in balancing technical leadership with strong business judgment to make the right decisions about model and method choices. Key job responsibilities • Provide technical expertise to support team strategies that will take EU RME towards World Class predictive maintenance practices and processes, driving better equipment up-time and lower repair costs with optimized spare parts inventory and placement • Implement an advanced maintenance framework utilizing Machine Learning technologies to drive equipment performance leading to reduced unplanned downtime • Provide technical expertise to support the development of long-term spares management strategies that will ensure spares availability at an optimal level for local sites and reduce the cost of spares A day in the life As a Senior OR Scientist in DST you will be focused on leading the design and development of innovative approaches and solutions by leading technical work supporting RME’s Predictive Maintenance (PdM) and Spare Parts (SP) programs. You will connect with world leaders in your field and you will be tackling customer's natural language challenges by carrying out a systematic review of existing solutions. The appropriate choice of methods and their deployment into effective tools will be the key for the success in this role. About the team Our mission is to leverage the use of data, science, and technology to improve the efficiency of RME maintenance activities, reduce costs, increase safety and promote sustainability while creating frictionless customer experiences. We are open to hiring candidates to work out of one of the following locations: Luxembourg, LUX
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Economists in the Forecasting, Macroeconomics & Finance field document, interpret and forecast Amazon business dynamics. This track is well suited for economists adept at combining cutting edge times-series statistical methods with strong economic analysis and intuition. This track could be a good fit for candidates with research experience in: macroeconometrics and/or empirical macroeconomics; international macroeconomics; time-series econometrics; forecasting; financial econometrics and/or empirical finance; and the use of micro and panel data to improve and validate traditional aggregate models. Economists at Amazon are expected to work directly with our senior management and scientists from other fields on key business problems faced across Amazon, including retail, cloud computing, third party merchants, search, Kindle, streaming video, and operations. The Forecasting, Macroeconomics & Finance field utilizes methods at the frontier of economics to develop formal models to understand the past and the present, predict the future, and identify relevant risks and opportunities. For example, we analyze the internal and external drivers of growth and profitability and how these drivers interact with the customer experience in the short, medium and long-term. We build econometric models of dynamic systems, using our world class data tools, formalizing problems using rigorous science to solve business issues and further delight customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Amazon.com strives to be Earth's most customer-centric company where customers can shop in our stores to find and discover anything they want to buy. We hire the world's brightest minds, offering them a fast paced, technologically sophisticated and friendly work environment. Economists at Amazon partner closely with senior management, business stakeholders, scientist and engineers, and economist leadership to solve key business problems ranging from Amazon Web Services, Kindle, Prime, inventory planning, international retail, third party merchants, search, pricing, labor and employment planning, effective benefits (health, retirement, etc.) and beyond. Amazon Economists build econometric models using our world class data systems and apply approaches from a variety of skillsets – applied macro/time series, applied micro, econometric theory, empirical IO, empirical health, labor, public economics and related fields are all highly valued skillsets at Amazon. You will work in a fast moving environment to solve business problems as a member of either a cross-functional team embedded within a business unit or a central science and economics organization. You will be expected to develop techniques that apply econometrics to large data sets, address quantitative problems, and contribute to the design of automated systems around the company. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Bellevue, WA, USA | Boston, MA, USA | Los Angeles, CA, USA | New York, NY, USA | San Francisco, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA