In June 2020, Amazon acquired Zoox, a then six-year-old California-based startup focused on “creating autonomous mobility from the ground up.”
Six months later, Zoox, now an independent Amazon subsidiary, shared publicly for the first time a look at its electric, autonomous vehicle created for dense, urban environments. The vehicle reveal marked a key milestone toward the organization’s vision of creating an autonomous robotaxi fleet and ride-hailing service designed with passengers in mind.
At its unveiling in December 2020, Zoox CEO Aicha Evans said her team is transforming the rider experience to provide superior “mobility-as-a-service” for customers. Moreover, she added, given the current data related to carbon emissions and traffic accidents, “It’s more important than ever that we build a sustainable, safe solution that allows riders to get from point A to point B.”
Jesse Levinson, co-founder and chief technology officer of Zoox, guides the company’s technology roadmap and execution to turn its mobility-as-a-service vision into reality. After graduating summa cum laude from Princeton, he completed his PhD and postdoc under Sebastian Thrun at Stanford. There, he developed algorithms for Stanford’s successful entry in the 2007 DARPA Urban Challenge and went on to lead the self-driving team’s research and development efforts.
Amazon Science asked Levinson about the challenges of developing self-driving vehicles and why he’s excited about Zoox’s approach.
Q. You were one of the authors on the 2008 paper, Junior: The Stanford Entry in the Urban Challenge. That race was a closed-course competition, and not quite representative of real-world challenges. But what key observations did you take away from that experience?
Probably the most important realization after the race was the dichotomy of how much there was still left to solve and the fact that it was actually all going to be solvable. It’s quite easy to get enchanted with one or the other of those observations; either that the problem is practically impossible because of all the things that still aren’t perfect, or that it must be almost solved because of some super cool demo or milestone that seems incredibly impressive. The reality is in between, and for whatever reason, it’s surprisingly hard for people to maintain a nuanced appreciation of that balance.
Achieving a world with ubiquitous autonomous vehicles will be an incremental process that advances every year — and remember, the alternative is the bar of human performance that stays nearly stagnant.
In 2004, DARPA held its first Grand Challenge: a 125-mile race in the desert. Of the 20 teams that entered, none completed the race, and the best vehicle only completed about six miles. The industry (and the media) widely regarded the outcome as an abysmal failure of AI. Yet it was not a failure, but an incredible feat of engineering. If an autonomous vehicle can drive six miles in the desert all by itself, then it doesn’t take an incredible imagination to foresee it driving 125 miles.
Lo and behold, the very next year, six vehicles finished the full 125-mile course. It was a promising step towards the future, and a year later, in 2006, DARPA announced the Urban Challenge, which several teams completed successfully. Our entry at Stanford came in second place. Excited by the results, many people made overly optimistic predictions on the mass-adoption of self-driving cars, which were subsequently deflated by various challenges we’ve seen in the industry since that time.
It has been eye-opening to watch the public's reaction to self-driving cars over time. I have always tried my best to be upfront, honest, and realistic about where the technology is — and while I’ve certainly not nailed all of my predictions, I do think I’ve managed to be fairly balanced overall. As technologists, when we are overly optimistic or pessimistic, we do a disservice to ourselves, the industry, and our technology. Achieving a world with ubiquitous autonomous vehicles will be an incremental process that advances every year — and remember, the alternative is the bar of human performance that stays nearly stagnant. It’s the opportunity of a lifetime to participate in the journey of making autonomous driving technology relentlessly better. Soon, it will reach a crossover point where the public begins to adopt it at scale, which will be a transformative win for society at large.
Q. Following up on your answer, what did you learn from that experience that you apply to your current role at Zoox? Has your approach changed since that challenge or remained largely the same?
So much! I’m grateful for that experience because it was formative in the early approach of Zoox. Here’s some of the lessons I took away from it:
First, teaching cars to drive will not take as long as we thought. In the early 2000s, we all thought it would be many, many decades before self-driving cars would be a reality. The DARPA challenge changed that. To build a vehicle that could navigate many realistic traffic scenarios only took about a year for a small team. Of course, there’s a huge difference between that and what’s required to operate an autonomous vehicle on public roads. But it was an important milestone that highlighted that autonomous driving technology could be a reality within a couple of decades.
Second, system integration and wide-scale testing is critical. No amount of knowledge about artificial intelligence, or anything else for that matter, will lead a mythical genius to intellectually divine a perfect solution. We need to combine and integrate many different complex systems and then see what works and fails through simulations, then closed courses, then public roads (with safety drivers). We have to test and experiment and iterate with massive data and scale, as opposed to trying to reason our way to a perfect solution.
On the other hand, blindly searching for progress without having any vision or architectural insights is also a bad idea; that’s one of the reasons why we identified the benefits of 270-degree sensing on all four corners of our ground-up vehicle at Zoox way back in 2014, a few years before we could drive autonomously in cities — because we knew from first principles that it was the right way to perceive the world.
Last, we have to test the various software and hardware components collectively to see how they respond to errors and uncertainty. By building a robust system that handles a cascading series of errors and ambiguities, you can explicitly track uncertainty and represent the state of the world more thoroughly. The proper representation of the world is not a singular, perfect model, but rather a distribution of probabilities and uncertainties. If you can design your system to be robust to imperfect sensor data, unpredictable agents, and unusual environments, you have a real shot at solving the problem in a world that’s not always the way you want it to be. It’s actually what humans do really well all the time, even though we’re rarely conscious that we’re doing it.
Q. You’ve said that safety is the foundation of everything Zoox does, and that the experience of building Zoox’s robotaxi has given you the opportunity to reimagine passenger safety. Can you give us insight into some of the systems you’ve developed for passenger safety, particularly the AI stack that underpins these efforts?
Yes, that’s right: safety is absolutely fundamental to the Zoox mission. With apologies for using an overused phrase, autonomous mobility allows for a paradigm shift (sorry!) in safety — from reactive to proactive. It’s an important point: automotive safety has always been reactive, focused on protecting vehicle occupants in crashes, which are seen to be inevitable. By building an autonomous vehicle from the ground-up, we can add a layer of proactive crash prevention that simply does not exist in today’s human-driven cars, and a focus on preventing crashes from occurring in the first place. We have more than a hundred safety innovations that do not exist in conventional cars today.
We are also developing the AI, vehicle, and service all together. Integrating the software, sensor, and vehicle subsystems is a complex challenge that requires tight, cross-functional collaboration. It would be difficult to create this level of system integration across multiple companies with divergent commercial interests. Building a ground-up vehicle has allowed us to design and choose our own sensor suite to best solve self-driving. We’ve outfitted our Toyota Highlander fleet with this same sensor architecture as our ground-up vehicle so that we can gather large amounts of data and test in environments like San Francisco and Las Vegas while our in-house vehicle is still under development.
Our software stack includes mapping, localization, sensor calibration, perception, prediction, path planning, vehicle control, infrastructure, firmware, diagnostics/messaging/monitoring/logging, and simulation. All of this software is continuously improving, with additions of new features and iterative software updates that are put through rigorous offline validations and on-vehicle structured testing.
Our vehicles also use a variety of advanced sensors, including LIDAR, cameras, and radar, to see objects on all sides of the vehicle. And because of the geometrical configuration of these sensors, we can almost always see around and behind the objects nearest to us, which is particularly helpful in dense urban environments. Our software then uses a combination of machine learning and geometric reasoning to understand the sensor data, make sense of the scene unfolding around the vehicle, and effectively navigate the roads.
We’re excited to launch our first commercial driverless service, but we won’t do so until we’re ready to operate on public roads at safety levels that meaningfully surpass that of humans.
For example, in a busy downtown intersection, our vehicle might be identifying a construction zone based on road cones and signs, while also detecting, tracking, and predicting the motion of hundreds of other agents (vehicles, pedestrians, bicyclists, etc.) around it. Once the perception system understands the environment and can predict how surrounding agents will move, the planner uses that information and context to adapt its driving behavior to the dynamic road conditions. The planner normally tries to maintain a certain lateral distance between itself and other vehicles, but it could decide to slightly reduce that distance in order to avoid a cone in the road ahead.
By combining both the hardware and software design, we are able to reimagine passenger safety. We are confident in our sensors’ abilities to detect activity in the environment around the vehicle, but that has to be validated in a wide range of scenarios. And our vehicle has performed extremely well in crash testing, which is still important, because no matter how sophisticated the AI is, we can’t guarantee that nothing will ever hit us. We’re excited to launch our first commercial driverless service, but we won’t do so until we’re ready to operate on public roads at safety levels that meaningfully surpass that of humans.
