“You’re trying to predict the unpredictable”

Amazon scientist Dean Foster and coauthor receive “test of time” award for paper authored 23 years ago.

Dean Foster is in the forecasting business. More specifically, he is in the business of ensuring the forecasts Amazon makes for its supply chain are as accurate as possible.

Foster, a research scientist, works in the company’s Supply Chain Optimization Technologies (SCOT) organization. “Our main focus is trying to predict what customers will buy before they buy it,” he said. “We need to make sure that we know what people want so we can find it, get it moved across the country, and have it sitting there waiting when a customer places an order.”

Dean Foster, an economist who works on optimizing Amazon's supply chain technologies, is seen holding an Amazon package he ordered from the middle of a forest in Japan.
Dean Foster, an economist who works on optimizing Amazon's supply chain technologies, is seen holding an Amazon package he ordered from the middle of a forest in Japan.
Courtesy Dean Foster

Forecasting what customers might want at any given time, at scale, is inherently complex. One of the ways those forecasts are strengthened is by a concept known as calibration — a topic that Dean has researched extensively. In fact, a paper he co-authored with Rakesh Vohra 23 years ago, “Calibrated Learning and Correlated Equilibrium”, was honored this week with the Test of Time Award at the 21st ACM Conference on Economics and Computation.

The award, presented by the conference’s award committee, “recognizes the author or authors of an influential paper or series of papers published between ten and twenty-five years ago that has significantly impacted research or applications exemplifying the interplay of economics and computation.”

Foster and Vohra’s paper “spurred a sizeable theoretical literature,” notes Steve Tadelis, an Amazon Scholar and economist. It has also won praise for its influence on games played by learning agents, addressing a question emanating from an idea proposed by the famed mathematician John Forbes Nash Jr.

“If we have two different agents learning to play each other, they learn to play an equilibrium,” Foster said. “Nash came up with a fixed point and argued equilibriums exist. But the question, ‘Why would people play them?’ was open.” In other words, the actions of human beings are neither neat nor uniformly predictable. “You are not a simple creature, so modeling your behavior as if you're going to do the exact same thing today as you've done every other day of your life is just wrong,” he says.

Foster and his coauthor, Rakesh Vohra, now a professor at the University of Pennsylvania, set out to account for some of that complexity by including arbitrary sequences when utilizing calibration.

Calibration, in this context, involves comparing a prediction against its actual outcome, measuring the difference between the two and then adjusting as needed. By learning from previous comparisons, prediction models can be calibrated to more accurately match outcomes.

“Not being calibrated is an embarrassment,” Foster said, “You should fix it! If I were trying to predict what nature was going to do, say rain or shine tomorrow, and suppose nature only has one goal in life—make me look stupid—in spite of that, I can still use calibration to figure out an accurate forecast.”

Foster notes that while that explanation may seem hyperbolic, it is particularly relevant to machine learning.

“That idea of calibration and doing predictions when the world's out to get you, is now relatively standard in machine learning. It grew out of connecting a lot of computer science,” he explains. “In computer science, for most things that you can prove, you can show the worst case and the average case are about the same. So the worst possible data for a sorting algorithm is about as hard to sort as a typical problem for sorting. We took that model and said, ‘Well, is it true in statistics?’ And that's where this idea came from. That you can make these predictions that are every bit as good, even when nature is out there trying to fool you.”

That idea has roots in game theory, an area in which calibration is particularly useful. Game theory assumes your opponent is attempting to fool you, for example, a chess player who wants you to think your queen is in danger when it really isn’t. Making errant predictions (or forecasts) means you will lose more frequently. Alternately, calibrated predictions can help you win more often.

“With a calibrated forecast, if I believe someone will take some action two-thirds of the time and two-thirds of the time they actually do that, I can now know two-thirds is the right answer,” he says. “And I can trust that I won't have to go back and say, ‘Well, most of time, it landed the other way...’ It's landing the way I thought it was going to land.”

Foster noted that calibration also helps ensure that forecasts aren’t tripped up by things like sample sizes. “A way to describe calibration for our forecast is that, after we announce the forecast, someone else shouldn't be able to come along and say, ‘Hey, if you made that forecast 20% larger it would be more accurate.’ Calibration is a check to make sure that you haven't left an easy, low-hanging fruit modification around.”

As to the future of forecasting, when it comes to supply chain optimization, Foster sees lots of potential. He is particularly excited about Amazon’s expanded usage of reinforcement learning, a machine learning approach focused on maximizing cumulative reward.

“I've been working on how to take a more economic viewpoint,” Foster said. “How do we connect the forecast to the economic decisions that we make so there’s a more integrated approach? We're trying to do this by applying more reinforcement learning techniques.”

For all of the citations and accolades his paper has earned, it didn’t start smoothly. “When we first tried to publish the paper, it was a shock enough in statistics journals that it was rejected several times,” Foster recalled. “One referee said, ‘You're trying to predict the unpredictable.’” In a way, he still is.

Research areas

US, WA, Seattle
Job summaryWork at the intersection of data science and economics.The DAC AdsEcon Team is looking for a Data Scientist II to help and be part of a team to put cutting edge economic and data science advertising research into production. We are looking for a unique individual to help us build a prototype that will have a profound impact in our advertising businesses.Advertising is used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team sits in the Business/Corporate Development, and our charter is to use econometrics, machine learning, and data science to build disruptive products that move the needle in our multiple Amazon Advertising businesses. We also generate insights to guide Amazon Advertising strategy, providing direct support to the high level leaders.If you have a background in economics, computer science, statistics, or mathematics and have a passion for solving large, and impactful problems, this is the job for you. Key responsibilities of Data Scientist include the following:· Partnering with economists and senior team members to drive science improvements and implement technical solutions at the cutting edge of machine learning and econometrics· Helping build data systems that leverage diverse data sources to understand how different advertiser’s decisions impact their performance across multiple advertising products.· Build interpretable statistical models and analyze experiment results to answer questions that will drive high impact decisions across Amazon.About Amazon's Advertising business:Amazon is investing heavily in building a world class advertising business and we are responsible for defining and delivering a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities.
US, NJ, Newark
Job summaryGood storytelling starts with great listening. At Audible, that means each role and every project has our audience in mind. Because the same people who design, develop, and deploy our products also happen to use them. To us, that speaks volumes.ABOUT THIS ROLEAudible is searching for an exceptional data scientist to join our economics team and drive the development of models at the intersection of machine learning and econometrics at scale. The Audible economics organization works across the business to measure and maximize the value Audible delivers to customers, creators, and communities globally. In this role, there will be a focus on partnering with our content and product teams to build a groundbreaking catalog of audiobooks and spoken-word entertainment, develop innovative tools to generate value for creators, and optimize content distribution and monetization.We are looking for someone experienced in building ML models at scale for complex prediction and optimization problems, who also has a background (or burgeoning interest!) in causal inference or interpretable machine learning. In addition to working with our staff economists and data scientists, you will also collaborate closely with scientists across Audible and partner teams at Amazon on problems pertinent to subscription businesses and the production of original media content.As a Data Scientist, you will...· Work with leadership in our content and product organizations to identify key analytical problems and opportunities – your work is expected to be a key input to our future content strategy.· Develop and maintain scalable, innovative data science and machine learning models that deliver actionable insights and results.· Collaborate with other data scientists, economists, and analysts at Audible to build data-driven solutions to key business problems.
US, NJ, Newark
Job summaryGood storytelling starts with great listening. At Audible, that means each role and every project has our audience in mind. Because the same people who design, develop, and deploy our products also happen to use them. To us, that speaks volumes.ABOUT THIS ROLEAudible seeks a Data Scientist who will help our marketing team improve paid marketing efficiency and performance. In this role, you will make the best of your skillset in modeling and general analytics. Modelling: use your knowledge of (un-) supervised learning, reinforcement learning, and simulation to explain, quantify, predict and prescribe. Analytics: use your knowledge of marketing and paid media to translate business and financial goals into insights and influence action. Overall: you will seek to create value for both stakeholders and customers and will convey results in a clear, actionable way to managers and senior leaders.As a Data Scientist, you will...· Will build analytical products end-to-end (decks, dashboards, data science models, simulations) at scale and at speed, from ideation and data extraction to presenting results to stakeholders (from manager to VP level).· Support development of models to optimize the Who, When, Where and How of all our conversations with customers and specifically to measure and optimize paid media.· Develop, maintain, and iterate on Amazon-scale data engineering and modelling pipelines.· Imagine and invent before the business asks, and create groundbreaking applications using cutting-edge approaches.· Contribute to the growth of the Audible Global Insights and Data Science team by sharing your ideas, intellectual property and learning from others.· Work closely with Audible stakeholders to drive the business forward, and deliver impactful models and analyses based on robust economic, financial, and statistical analysis.
US, MA, North Reading
Job summaryAre you an MS or PhD student interested in Robotics, Manipulation, Computer Vision, or Machine Learning? Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?At Amazon Robotics, we strive to push boundaries in order to provide the best possible experience for our customers. We are looking for scientists striving to use their domain expertise to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. As an Applied Scientist intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.As an Applied Scientist intern, you will work from concept through to execution. This role will give you the opportunity to build tools and support structures needed to analyze data, dive deep to resolve root cause of systems errors and changes, and present findings to business partners to drive improvements.Come build the future with us. Amazon internships are full-time (40 hours/week) for 12 or more consecutive weeks with start dates between May and June 2022.Amazon Robotics intern opportunities will be based in the Greater Boston Area, in our two state-of-the-art facilities in Westborough and North Reading, MA. Both campuses provide a unique opportunity for co-ops to have direct access to robotics testing labs and manufacturing facilities!
US, WA, Seattle
Job summaryAmazon’s Shipping and Delivery Support (SDS) team is a part of Amazon World Wide Customer Service dedicated to support successful package deliveries to Amazon Customers. As a Data Scientist on our team, you’ll use Amazon’s wealth of data to help answer tough questions like where and when preemptively intervening with a problem is most likely to result in a successful delivery, which signals should alert us that a delivery is at risk of missing its estimate, and what is the relative value of a specific set of support associate actions as they relate to delivery success. You will also leverage Amazon's rich datasets and machine learning techniques to understand customer urgency, and build algorithms to recommend treatment actions to optimize delivery outcome. This role will be a key member of the Shipping and Delivery Support Science Team.The Senior Data Scientist will work closely with Business Intelligence Engineers, Data Engineers, Product Managers, Software Engineers, and Program Managers to develop statistical and machinelearning models, design and run experiments, and find new ways to improve support experience to optimize the customer experience and Amazon’s on-time deliveries. The Scientist will collaborate with technology and product leaders to solve business and technology problems using scientific approaches to build new services that surprise and delight Amazon drivers and our customers. Science at Amazon is a highly experimental activity, although theoretical analysis and innovation are also welcome. Our scientists work closely with software engineers to put algorithms into practice. They also work on cross-disciplinary efforts with other scientists within Amazon.The key strategic objectives for this role include:· Understanding drivers, impacts, and key influences on delivery success and support contacts.· Optimizing support processes to improve the Customer experience and Amazon’s on time delivery.· Automating feedback loops for algorithms in production.· Collaborate with researchers, software developers, and business leaders to define product requirements and provide analytical support.· Utilizing Amazon systems and tools to effectively work with terabytes of data.· Communicating verbally and in writing to business customers and leadership team with various levels of technical knowledge, educating them about our systems, as well as sharing insights and recommendations
US, NY, New York
Job summaryCalling all inventors! Are you excited about Advertising technology? Love to work at the intersection between Machine Learning, Customer Experience, and Revenue Growth? Keep reading.Amazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our products are strategically important to our businesses driving long term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day!As a Senior Data Scientist for Amazon Business Advertising, you will help build a new Business-to-Business (B2B) advertising experience from the ground up and create and launch features for both advertisers and business shoppers globally. Our team owns end-to-end advertising experience including placements, ad relevance, creative, ad serving, advertiser experience, and marketing. You will work on complex science, engineering, optimization, econometric, and user-experience problems in order to deliver relevant Amazon Business ads on Amazon search and detail pages world-wide. Leveraging Amazon's massive data repository, you will develop experiments, insights and optimizations that enable the monetization of Amazon online and mobile search properties while enhancing the experience of Amazon shoppers.As a Senior Data Scientist on this team you will:* Lead full life-cycle Data Science solutions from beginning to end.* Deliver with independence on challenging large-scale problems with complexity and ambiguity.* Write code (Python, R, Scala, SQL, etc.) to obtain, manipulate, and analyze data to better understand Amazon Business customer shopping journey so as to help enhance their experience at Amazon.* Build Machine Learning and statistical models to solve business problem you define* Conduct written and verbal presentations to share insights to audiences of varying levels of technical sophistication.*Why you will love this opportunity*: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate.*Impact and Career Growth:* You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.Team video ~ https://youtu.be/zD_6Lzw8raEAbout the teamAmazon is building a world class advertising business and defining and delivering a collection of self-service performance advertising products that drive discovery and sales of merchandise. Our products are strategically important to our Retail and Marketplace businesses driving long term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products.Amazon Business Advertising team's mandate is to build a new Business-to-Business (B2B) advertising experience from the ground up as it requires a differentiated customer and advertiser experience. We are investing in a deep science and technical team to pursue a transformation opportunity.#adptjobs#adptvs#vertcat#amazonbusinessads
US, WA, Seattle
Job summaryAmazon brings buyers and sellers together. Our retail customers depend on us to give them access to every product at the best possible price. Our sellers depend on us to give them a platform to launch their business into every home and marketplace. Making this happen is the mission of every engineer in Amazon's North America Consumer (NAC) organization.To this end, the Science team is tasked with:· Organizing available data sources, and creating detailed dictionaries of data that can be used in future analyses.· Partnering with product teams in evaluating the financial and operational impact of new product offerings.· Conducting research into optimization and machine learning algorithms which can be applied to solve business problems.· Partnering with other scientists in evaluating algorithms and suggestions from a business view point.· Carrying out independent data-backed initiatives that can be leveraged later on in the fields of network organization, costing and financial modeling of processes.In order to execute the above mandate we are on the look out for smart and qualified Data Scientists who will own projects in partnership with product and research teams as well as operate autonomously on independent initiatives that are expected to unlock benefits in the future. A past background in Statistics is necessary, along with advanced proficiency in languages such as Python and R.Key job responsibilitiesAs a Data Scientist, you are able to use a range of advanced analytical methodologies to solve challenging business problems when the solution is unclear. You have a combination of business acumen, broad knowledge of statistics, deep understanding of ML algorithms, and an analytical mindset. You thrive in a collaborative environment, and are passionate about learning. Our team utilizes a variety of AWS tools such as Redshift, Sagemaker, Lambda, S3, and EC2 with a variety of skillsets in Linear and Discrete Optimization, ML, NLP, Forecasting, Probabilistic ML and Causal ML. You will bring knowledge in many of these domains along with your own specialties and skillsets.
US, CA, Pasadena
Job summaryThe Amazon Web Services (AWS) Center for Quantum Computing in Pasadena, CA, is hiring a Quantum Research Scientist to join a multi-disciplinary, fast-paced team of theoretical and experimental physicists, materials scientists, and hardware and software engineers pushing the forefront of quantum computing. The candidate should demonstrate a thorough knowledge of experimental measurement techniques as well as quantum mechanics theory.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.Key job responsibilities* Contribute to fast-paced and agile research to help close the many orders of magnitude gap in gate error rates required for fault tolerant quantum computation* Design and perform experiments to characterize quantum devices in close collaboration with software and engineering teams* Develop models to understand and improve device performance* Effectively document results and communicate to a broad audience* Create robust software for implementation, automation, and analysis of measurements* Specify technical requirements in a cross-team collaboration using analytical arguments derived from physics theoryA day in the life* Analyze experimental data* Develop software to test and run new experiments on existing devices; collaborate with software engineers to achieve high code standard* Debug test setups to achieve high-quality data* Present results and cross-collaborate with others’ work* Perform code review for a colleague’s merge request
US, CA, Pasadena
Job summaryThe Amazon Web Services (AWS) Center for Quantum Computing in Pasadena, CA, is looking to hire a Quantum Research Scientist in the Test and Measurement group. You will join a multi-disciplinary team of theoretical and experimental physicists, materials scientists, and hardware and software engineers working at the forefront of quantum computing. You should have a deep and broad knowledge of experimental measurement techniques.Candidates with a track record of original scientific contributions will be preferred. We are looking for candidates with strong engineering principles, resourcefulness and a bias for action, superior problem solving, and excellent communication skills. Working effectively within a team environment is essential. As a research scientist you will be expected to work on new ideas and stay abreast of the field of experimental quantum computation.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.Key job responsibilitiesIn this role, you will drive improvements in qubit performance by characterizing the impact of environmental and material noise on qubit dynamics. This will require designing experiments to assess the role of specific noise sources, ensuring the collection of statistically significant data, analyzing the results, and preparing clear summaries for the team. Finally, you will work with hardware engineers, material scientists, and circuit designers to implement changes which mitigate the impact of the most significant noise sources.
US, MA, Cambridge
Job summaryThe Alexa Artificial Intelligence (AI) team is looking for a passionate, talented, and inventive Applied Scientist with a strong machine learning background, to help build industry-leading Speech and Language technology.Key job responsibilitiesAs an Applied Scientist with the Alexa AI team, you will work with talented peers to develop novel algorithms and modeling techniques to advance the state of the art in spoken language understanding. Your work will directly impact our customers in the form of products and services that make use of speech and language technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in spoken language understanding.About the teamThe Alexa AI team has a mission to push the envelope in Natural Language Understanding (NLU). Specifically, we focus on incremental learning, continual learning and fairness, in order to provide the best-possible experience for our customers.