Aerial photo of the San Diego waterfront on an overcast day
Aerial photo of the San Diego waterfront on an overcast day
Credit: Jerry Uomala / Getty Images / iStockphoto

Amazon at AEA: The crossroads of economics and AI

Pat Bajari, VP and chief economist for Amazon's Core AI group, on his team's new research and what it says about economists' role at Amazon.

The 2020 meeting of the American Economic Association begins on January 3 in San Diego, and among the Amazon economists attending will be Pat Bajari, VP and chief economist for Amazon’s Core AI group, who is a coauthor on two papers accepted to the conference.

Economic research at Amazon, Bajari explains, is distinctive in the way it crosses disciplinary boundaries. “These disciplines are like their own worlds,” he says. “It’s easy to get siloed doing engineering, machine learning, natural-language processing, computer vision, stats, operational research, economics, and so on. But when these disciplines interact, you get more interesting and useful results.”

Apples to apples

One of Bajari’s two papers at AEA is a case in point. Titled “New Goods, Productivity and the Measurement of Inflation: Using Machine Learning to Improve Quality Adjustments,” it applies new AI techniques to an old problem in the calculation of inflation rates.

Pat Bajari
Pat Bajari, Amazon vice president and chief economist
Carl Clark, Amazon Imaging Studio

“If you look at a product line, over the course of a year, 80% of the products might vanish,” Bajari explains. “When you calculate the rate of inflation, you’re usually doing an annual measure of price changes. But if 80% of products are gone, that measurement can be inaccurate.”

A famous example, Bajari explains, is personal computers in the late ’90s. At the time, he says, 95% of computers would sell out in the course of a year. The computers on the shelves one January could have very different technical specifications from those on the shelves a year later, making direct price comparison misleading.

Consequently, the standard method of calculating inflation indicated little change in the price of personal computers, even though the price of computational power was plummeting. The classical solution to this problem is so-called hedonic pricing, in which the price of a product is factored into several components, which can be compared independently.

So, for instance, late-’90s computers could be compared according to their price per megahertz of processing speed, per megabyte of random-access memory, per megabyte of storage, and so on. Bajari’s first AEA paper updates hedonic pricing for the age of deep learning. On the paper, he joins Victor Chernozhukov, a professor of economics at MIT and a senior principal economist in Amazon’s Core AI group; Ramon Huerta, a research scientist at the University of California, San Diego, and a principal applied scientist in the Amazon North American Consumer group; George Monokroussos, a former senior economist at Amazon; and three other members of Core AI: Zhihao Cen, a senior applied scientist Junbo Li; a senior software engineer; and Manoj Manukonda, a senior data engineer.

Instead of factorizing product prices themselves, the researchers trained a machine learning model to identify correlations between product features and prices. If the model is trained on data from one year but fed descriptions of products on the shelves a year later, it will spit out the products’ prices according to the earlier valuation. Comparing the predicted and actual prices provides a measure of inflation.

Hedonic-pricing model
To predict a product's price, a new machine learning model factors in numeric data such as number of reviews and average star rating, textual data such as product descriptions and titles, and even visual data such as product shots.
Stacy Reilly

Internally, Amazon can use this type of model to analyze business trends. But if central bankers applied a similar model to products representative of the economy as a whole, they could observe inflation rate variations in real time.

“If central bankers have a view with a one-day latency, it could give them signals about whether monetary policy is too loose or too tight,” Bajari explains.

Feedback loops

Bajari’s other AEA paper examines the design of randomized experiments. It reports work done in collaboration with Guido Imbens, a member of Core AI and the Applied Econometrics Professor and professor of economics at Stanford Business School; Thomas Richardson, a professor of statistics at the University of Washington and an Amazon Scholar; Brian Burdick, the director of Core AI; Ido Rosen, a principal software engineer in Burdick’s group; and James McQueen, a senior applied scientist in Amazon’s Customer Behavior Analytics group.

The most familiar example of a randomized experiment is a drug trial, where some subjects receive an experimental drug, some receive a placebo, and their outcomes are compared. But randomized experiments are also common in industry.

Suppose, for instance, that Amazon researchers develop a new algorithm for calculating how much of a product to restock at a fulfillment center as a function of recent sales rates and supply on hand. In simulations, the algorithm promises more reliable delivery and greater customer satisfaction, but there’s a question about whether those theoretical gains will translate into practice.

Amazon might conduct a randomized experiment in which some fulfillment centers use the new algorithm, some use the old algorithm, and the average results are compared. Such experiments, however, are liable to so-called spillover effects, where the “treatment” — in this case, the deployment of the new algorithm — ends up having consequences for the control group — in this case, the fulfillment centers using the old algorithm.

Suppose that the treatment results in faster delivery of certain products, and consequently, those products grow in popularity. Amazon’s recommendation engine begins recommending those products more frequently, even to customers served by fulfillment centers using the old restock algorithm. Demand for the products spikes, and the control group starts selling through its stocks — a negative outcome, in terms of the experimental design. When the results of the experiment are tallied, the control group’s performance is artificially depreciated because of the treatment.

“This type of spillover does not happen in standard medical-drug trials, because one individual taking the new drug does not affect the outcome for another individual taking the placebo,” Imbens says. “But it is a feature of many experiments at Amazon and similar companies, where we have complex feedback loops.”

Exerting controls

One way to identify such spillover effects would be to ensure that, for every product that receives the treatment, there’s a related product that doesn’t, regardless of where it’s stored. That would make it possible to determine whether demand spikes are affecting product classes as a whole or are limited to treated products. But it complicates the experimental design.

The researchers’ paper presents an ambitious blueprint for performing such complex experiments. It describes how to simultaneously measure average effects and identify spillovers within a single experiment — by, for instance, systematically varying the treatment’s application to pairs of fulfillment centers and products. It also presents statistical techniques for analyzing the results of such experiments.

The researchers’ blueprint could be applied in a host of different contexts — movie recommendations, rideshare services, short-term-property-rental sites, homebuying sites, retail sites, job search sites, and the like. It also generalizes from double randomization — a given product can receive different treatments at different fulfillment centers, and a given fulfillment center can treat some products and not others — to higher-dimensional randomization — varying treatment according to season, delivery destination, vendor, and so on.

“When people do these kinds of experiments, they usually randomize only one variable at a time,” Bajari explains. “We want to go further with this idea, where we use multiple randomizations to learn supply responses, demand responses, equilibria — all with the goal to keep improving the customer experience.”

Helping identify the causal relationships that underlie the data, Bajari says, is one of the ways in which the economic perspective is useful. But another is in deciding what to measure, across what time frame.

“Usually, ML and AI are tools for making decisions,” Bajari says. “If you have a particular product, how much should you stock of it? You want to make that decision in a present-value-maximizing way. You don’t want to sacrifice long-term success for short-term gains. If you only looked at short-term numbers, we would cut safety stock by half. Then customers would be more apt to find products out of stock, which means they might be less likely to shop on Amazon, which in turn could hurt growth.

“If you want to use ML and AI to make decisions in a rational way, you need a way to trade off long-term and short-term results. This is a place where economists help. What should a firm rationally optimize for? That’s just squarely in economics. That’s what we do.”

Amazon's involvement at AEA/ASSA

Paper and presentation schedule

Friday, 1/3 | 2:30 pm - 4:30 pm | Marriott Marquee San Diego | San Diego Ballroom A

"GDPR and the Home Bias of Venture Investment"

Jian Jia (Illinois Institute of Technology) · Ginger Jin (University of Maryland/Amazon Scholar) · Liad Wagman (Illinois Institute of Technology)

"New Goods, Productivity and the Measurement of Inflation: Using Machine Learning to Improve Quality Adjustments"

Pat Bajari (Amazon) · Zhihao Cen (Amazon) · Victor Chernozhukov (MIT/Amazon) · Ramon Huerta (UCSD/Amazon) · Junbo Li (Amazon) · Manoj Manukonda (Amazon) · George Monokroussos (Wayfair)

"Double Randomized Online Experiments"

Pat Bajari (Amazon) · Brian Burdick (Amazon) · Guido Imbens (Stanford Graduate School of Business/Amazon) · James McQueen (Amazon) · Thomas Richardson (University of Washington/Amazon Scholar) · Ido Rosen (Amazon)

Saturday, 1/4 | 2:30 pm - 4:30 pm | Marriott Marquis San Diego | Del Mar

"Sustained Credit Card Borrowing"

Sergei Koulayev (Amazon) · Daniel Grodzicki (Pennsylvania State University/Consumer Financial Protection Bureau)

Workshops

Econometrica Session: New Developments in Econometrics

Chair: Guido Imbens

Research areas

Related content

US, CA, San Diego
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to help Amazon provide the best customer experience by preventing eCommerce fraud? Are you excited by the prospect of analyzing and modeling terabytes of data and creating state-of-the-art algorithms to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you enjoy collaborating in a diverse team environment? If yes, then you may be a great fit to join the Amazon Buyer Risk Prevention (BRP) Machine Learning group. We are looking for a talented scientist who is passionate to build advanced algorithmic systems that help manage safety of millions of transactions every day. Key job responsibilities Use machine learning and statistical techniques to create scalable risk management systems Learning and understanding large amounts of Amazon’s historical business data for specific instances of risk or broader risk trends Design, development and evaluation of highly innovative models for risk management Working closely with software engineering teams to drive real-time model implementations and new feature creations Working closely with operations staff to optimize risk management operations, Establishing scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation Tracking general business activity and providing clear, compelling management reporting on a regular basis Research and implement novel machine learning and statistical approaches
GB, Cambridge
The Artificial General Intelligence team (AGI) has an exciting position for an Applied Scientist with a strong background NLP and Large Language Models to help us develop state-of-the-art conversational systems. As part of this team, you will collaborate with talented scientists and software engineers to enable conversational assistants capabilities to support the use of external tools and sources of information, and develop novel reasoning capabilities to revolutionise the user experience for millions of Alexa customers. Key job responsibilities As an Applied Scientist, you will develop innovative solutions to complex problems to extend the functionalities of conversational assistants . You will use your technical expertise to research and implement novel algorithms and modelling solutions in collaboration with other scientists and engineers. You will analyse customer behaviours and define metrics to enable the identification of actionable insights and measure improvements in customer experience. You will communicate results and insights to both technical and non-technical audiences through written reports, presentations and external publications.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a highly skilled and experienced Senior Applied Scientist, to lead the development and implementation of cutting-edge algorithms and models for supervised fine-tuning and reinforcement learning through human feedback; with a focus across text, image, and video modalities. As a Senior Applied Scientist, you will play a critical role in driving the development of Generative AI (GenAI) technologies that can handle Amazon-scale use cases and have a significant impact on our customers' experiences. Key job responsibilities - Collaborate with cross-functional teams of engineers, product managers, and scientists to identify and solve complex problems in GenAI - Design and execute experiments to evaluate the performance of different algorithms and models, and iterate quickly to improve results - Think big about the arc of development of GenAI over a multi-year horizon, and identify new opportunities to apply these technologies to solve real-world problems - Communicate results and insights to both technical and non-technical audiences, including through presentations and written reports - Mentor and guide junior scientists and engineers, and contribute to the overall growth and development of the team
US, WA, Bellevue
Amazon’s Last Mile Team is looking for a passionate individual with strong optimization and analytical skills to join its Last Mile Science team in the endeavor of designing and improving the most complex planning of delivery network in the world. Last Mile builds global solutions that enable Amazon to attract an elastic supply of drivers, companies, and assets needed to deliver Amazon's and other shippers' volumes at the lowest cost and with the best customer delivery experience. Last Mile Science team owns the core decision models in the space of jurisdiction planning, delivery channel and modes network design, capacity planning for on the road and at delivery stations, routing inputs estimation and optimization. Our research has direct impact on customer experience, driver and station associate experience, Delivery Service Partner (DSP)’s success and the sustainable growth of Amazon. Optimizing the last mile delivery requires deep understanding of transportation, supply chain management, pricing strategies and forecasting. Only through innovative and strategic thinking, we will make the right capital investments in technology, assets and infrastructures that allows for long-term success. Our team members have an opportunity to be on the forefront of supply chain thought leadership by working on some of the most difficult problems in the industry with some of the best product managers, scientists, and software engineers in the industry. Key job responsibilities Candidates will be responsible for developing solutions to better manage and optimize delivery capacity in the last mile network. The successful candidate should have solid research experience in one or more technical areas of Operations Research or Machine Learning. These positions will focus on identifying and analyzing opportunities to improve existing algorithms and also on optimizing the system policies across the management of external delivery service providers and internal planning strategies. They require superior logical thinkers who are able to quickly approach large ambiguous problems, turn high-level business requirements into mathematical models, identify the right solution approach, and contribute to the software development for production systems. To support their proposals, candidates should be able to independently mine and analyze data, and be able to use any necessary programming and statistical analysis software to do so. Successful candidates must thrive in fast-paced environments, which encourage collaborative and creative problem solving, be able to measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs.
IN, KA, Bangalore
Alexa is the voice activated digital assistant powering devices like Amazon Echo, Echo Dot, Echo Show, and Fire TV, which are at the forefront of this latest technology wave. To preserve our customers’ experience and trust, the Alexa Sensitive Content Intelligence (ASCI) team creates policies and builds services and tools through Machine Learning techniques to detect and mitigate sensitive content across Alexa. We are looking for an experienced Applied Science Manager to lead a team to build industry-leading technologies in attribute extraction and sensitive content detection across all languages and countries. A Manager, Applied Science will be a tech leader for a team of exceptional scientists to develop novel algorithms and modeling techniques to advance the state of the art in NLP or CV related tasks. You will work in a hybrid, fast-paced organization where scientists, engineers, and product managers work together to build customer facing experiences. You will collaborate with and mentor other scientists to raise the bar of scientific research in Amazon. Your work will directly impact our customers in the form of products and services that make use of speech, language, and computer vision technologies. We are looking for a leader with strong technical experiences a passion for building scientific driven solutions in a fast-paced environment. You should have good understanding of NLP models (e.g. LSTM, transformer based models) or CV models (e.g. CNN, AlexNet, ResNet) and where to apply them in different business cases. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience of building large-scale distributed systems to creating reliable, scalable, and high-performance products. In addition to technical depth, you must possess exceptional communication skills and understand how to influence key stakeholders. You will be joining a select group of people making history producing one of the most highly rated products in Amazon's history, so if you are looking for a challenging and innovative role where you can solve important problems while growing as a leader, this may be the place for you. Key job responsibilities You'll lead and manage the science driven solution development including design, run experiments, research new algorithms, and find new ways of optimizing customer experience. You set examples for the team on good science practice and standards. Besides theoretical analysis and innovation, you will work closely with talented engineers and ML scientists to put your algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally. You contribute directly to our growth by hiring smart and motivated Scientists to establish teams that can deliver swiftly and predictably, adjusting in an agile fashion to deliver what our customers need. A day in the life You will be working with a group of talented scientists as well as stakeholder from different functional areas (e.g. product, engineering) on researching algorithm and running experiments to test scientific proposal/solutions to improve our sensitive contents detection and mitigation. This will involve collaboration with partner teams including engineering, PMs, data annotators, and other scientists to discuss data quality, policy, and model development. You will mentor other scientists, review and guide their work, help develop roadmaps for the team. You work closely with partner teams across Alexa to deliver platform features that require cross-team leadership. About the team The mission of the Alexa Sensitive Content Intelligence (ASCI) team is to (1) minimize negative surprises to customers caused by sensitive content, (2) detect and prevent potential brand-damaging interactions, and (3) build customer trust through appropriate interactions on sensitive topics. The term “sensitive content” includes within its scope a wide range of categories of content such as offensive content (e.g., hate speech, racist speech), profanity, content that is suitable only for certain age groups, politically polarizing content, and religiously polarizing content. The term “content” refers to any material that is exposed to customers by Alexa (including both 1P and 3P experiences) and includes text, speech, audio, and video.
US, WA, Seattle
Are you passionate about leveraging data to deliver actionable insights that impact daily marketing activities at Amazon? The Customer Targeting team in Amazon is seeking an Applied Scientist to join our team to develop models for optimizing the performance of Amazon’s marketing initiatives across channels and advertising formats. You have experience applying modern machine learning methods to answer key business questions, make strategic and tactical recommendations for change, and work with business leaders to drive these to production. You are entrepreneurial and able to work in a highly collaborative environment. This role requires an individual with strong quantitative modeling skills and experience using statistical methods. The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail, an ability to work in a fast-paced and ever-changing environment. You will be expected to: - Leverage knowledge of statistics and optimization to frame decision-making problems for determining marketing spends across channels. - Predict future customer behavior and business conditions through machine learning and predictive modeling. - Use analytical and predictive techniques to build models for optimizing targeting. - Present proposals and results in a clear manner backed by data and coupled with actionable conclusions.
US, CA, Pasadena
The Amazon Web Services (AWS) Center for Quantum Computing (CQC) is a multi-disciplinary team of scientists, engineers, and technicians, on a mission to develop a fault-tolerant quantum computer. We are looking to hire a Research Scientist with fabrication and data analysis experience working on all elements of a superconducting circuit. The position is on-site at our lab, located on the in Pasadena, CA. The ideal candidate will have had prior experience building software tools for data analysis and visualization to enable deep diving into fabrication details, electrical test data. We are looking for candidates with strong engineering principles, resourcefulness and data science experience. Organization and communication skills are essential. Key job responsibilities * Develop and automate data pipeline pertinent to superconducting device fabrication. * Develop analytical tools to uncover new information about established and new processes. * Develop new or contribute to modifying existing data visualization tools. * Utilize machine learning to enable better deeper dives into fabrication and related data. * Interface with various software, design, fabrication and electrical test teams to enable new functionalities. A day in the life The role will be vital to the fabrication team and quantum computing device integration mechanism. The candidate will develop software based analytical tools to enable data driven decisions across projects related to fabrication and supporting infrastructure. Each fabrication run delivers additional data. The candidate will stay close to the details of fabrication providing data analysis and quick feedback to key stakeholders. At the end of fabrication runs custom and standardized reports will be generated by the candidate to provide insights into data generated from the run. This position may require occasional weekend work. About the team AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Hybrid Work We value innovation and recognize this sometimes requires uninterrupted time to focus on a build. We also value in-person collaboration and time spent face-to-face. Our team affords employees options to work in the office every day or in a flexible, hybrid work model near one of our U.S. Amazon offices.
US, WA, Bellevue
Conversational AI ModEling and Learning (CAMEL) team is part of Amazon Devices organization where our mission is to build a best-in-class Conversational AI that is intuitive, intelligent, and responsive, by developing superior Large Language Models (LLM) solutions and services which increase the capabilities built into the model and which enable utilizing thousands of APIs and external knowledge sources to provide the best experience for each request across millions of customers and endpoints. We are looking for a passionate, talented, and resourceful Senior Applied Scientist in the field of LLM, Artificial Intelligence (AI), Natural Language Processing (NLP), Recommender Systems and/or Information Retrieval, to invent and build scalable solutions for a state-of-the-art context-aware conversational AI. A successful candidate will have strong machine learning background and a desire to push the envelope in one or more of the above areas. The ideal candidate would also have hands-on experiences in building Generative AI solutions with LLMs, enjoy operating in dynamic environments, be self-motivated to take on challenging problems to deliver big customer impact, moving fast to ship solutions and then iterating on user feedback and interactions. Key job responsibilities As a Senior Applied Scientist, you will leverage your technical expertise and experience to demonstrate leadership in tackling large complex problems, setting the direction and collaborating with other talented applied scientists and engineers to research and develop LLM modeling and engineering techniques to reduce friction and enable natural and contextual conversations. You will analyze, understand and improve user experiences by leveraging Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in artificial intelligence. You will work on core LLM technologies, including Prompt Engineering, Model Fine-Tuning, Reinforcement Learning from Human Feedback (RLHF), Evaluation, etc. Your work will directly impact our customers in the form of novel products and services .
US, WA, Seattle
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to help Amazon provide the best customer experience by preventing eCommerce fraud? Are you excited by the prospect of analyzing and modeling terabytes of data and creating state-of-the-art algorithms to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you enjoy collaborating in a diverse team environment? If yes, then you may be a great fit to join the Amazon Buyer Risk Prevention (BRP) Machine Learning group. We are looking for a talented scientist who is passionate to build advanced algorithmic systems that help manage safety of millions of transactions every day. Key job responsibilities Use machine learning and statistical techniques to create scalable risk management systems Learning and understanding large amounts of Amazon’s historical business data for specific instances of risk or broader risk trends Design, development and evaluation of highly innovative models for risk management Working closely with software engineering teams to drive real-time model implementations and new feature creations Working closely with operations staff to optimize risk management operations, Establishing scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation Tracking general business activity and providing clear, compelling management reporting on a regular basis Research and implement novel machine learning and statistical approaches
US, CA, Palo Alto
We are reimagining Amazon Search by introducing an interactive conversational experience that makes finding the perfect product easier than ever. With our state-of-the-art Large Language Model (LLM) innovations, you can now ask product-related questions, compare products, receive personalized suggestions, and more—all through a fast and reliable natural language conversation. This is just the beginning of a new era in online shopping, and the future is yours to shape. We're searching for pioneers who are passionate about technology, innovation, and customer experience, and who are ready to make a lasting impact on the industry. You'll be working with talented scientists, and engineers to innovate on behalf of our customers. If you're fired up about being part of a dynamic, driven team, then this is your moment to join us on this exciting journey and change the world of eCommerce forever!