The science of price experiments in the Amazon Store

The requirement that at any given time, all customers see the same prices for the same products necessitates innovation in the design of A/B experiments.

The prices of products in the Amazon Store reflect a range of factors, such as demand, seasonality, and general economic trends. Pricing policies typically involve formulas that take such factors into account; newer pricing policies usually rely on machine learning models.

With the Amazon Pricing Labs, we can conduct a range of online A/B experiments to evaluate new pricing policies. Because we practice nondiscriminatory pricing — all visitors to the Amazon Store at the same time see the same prices for all products — we need to apply experimental treatments to product prices over time, rather than testing different price points simultaneously on different customers. This complicates the experimental design.

Related content
Amazon Scholar David Card on the revolution in economic research that he helped launch and its consequences for industry.

In a paper we published in the Journal of Business Economics in March and presented at the American Economics Association’s annual conference in January (AEA), we described some of the experiments we can conduct to prevent spillovers, improve precision, and control for demand trends and differences in treatment groups when evaluating new pricing policies.

The simplest type of experiment we can perform is a time-bound experiment, in which we apply a treatment to some products in a particular class, while leaving other products in the class untreated, as controls.

Time-bound experiment.png
A time-bound experiment, which begins at day eight, with treatments in red and controls in white.

One potential source of noise in this type of experiment is that an external event — say, a temporary discount on the same product at a different store — can influence treatment effects. If we can define these types of events in advance, we can conduct triggered interventions, in which we time the starts of our treatment and control periods to the occurrence of the events. This can result in staggered start times for experiments on different products.

Triggered interventions.png
The design of a triggered experiment. Red indicates treatment groups, and green indicates control groups. The start of each experiment is triggered by an external event.

If the demand curves for the products are similar enough, and the difference in results between the treatment group and the control group are dramatic enough, time-bound and triggered experiments may be adequate. But for more precise evaluation of a pricing policy, it may be necessary to run treatment and control experiments on the same product, as would be the case with typical A/B testing. That requires a switchback experiment.

Related content
Context vectors that capture “side information” can make experiments more informative.

The most straightforward switchback experiment is the random-days experiments, in which, each day, each product is randomly assigned to either the control group or the treatment group. Our analyses indicate that random days can reduce the standard error of our experimental results — that is, the extent to which the statistics of our observations differ, on average, from the true statistics of the intervention — by 60%.

Random days.png
A random-days experiment. The experiment begins on day 8; red represents treatment, white control.

One of the drawbacks with any switchback experiment, however, is the risk of carryover, in which the effects of a treatment carry over from the treatment phase of the experiment to the control phase. For instance, if treatment increases a product’s sales, recommendation algorithms may recommend that product more often. That could artificially boost the product’s sales even during control periods.

Related content
Pat Bajari, VP and chief economist for Amazon's Core AI group, on his team's new research and what it says about economists' role at Amazon.

We can combat carryover by instituting blackout periods during transitions to treatment and control phases. In a crossover experiment, for instance, we might apply a treatment to some products in a group, leaving the others as controls, but toss out the first week’s data for both groups. Then, after collecting enough data — say, two weeks’ worth — we remove the treatment from the former treatment group and apply it to the former control group. Once again, we throw out the first week’s data, to let the carryover effect die down.

Crossover experiment.png
A crossover experiment, with blackout periods at the beginning of each phase of the experiment. In week 7, the treatment (red) has been applied to products A, D, F, G, and J, but the data is thrown out. In week 10, the first treatment and control groups switch roles, but again, the first week’s data is thrown out.

Crossover experiments can reduce the standard error of our results measurements by 40% to 50%. That’s not quite as good as random days, but carryover effects are mitigated.

Heterogeneous panel treatment effect

The Amazon Pricing Labs also offers two more sophisticated means of evaluating pricing policies. The first of these is the heterogeneous panel treatment effect, or HPTE.

HPTE is a four-step process:

  1. Estimate product-level first difference from detrended data.
  2. Filter outliers.
  3. Estimate second difference from grouped products using causal forest.
  4. Bootstrap data to estimate noise.

Estimate product-level first difference from detrended data. In a standard difference-in-difference (DID) analysis, the first difference is the difference between the results for a single product before and after the experiment begins.

Related content
Amazon Scholar David Card and Amazon academic research consultant Guido Imbens talk about the past and future of empirical economics.

Rather than simply subtracting the results before treatment from the results after treatment, however, we analyze historical trends to predict what would have happened if products were left untreated during the treatment period. We then subtract that prediction from the observed results.

Filter outliers. In pricing experiments, there are frequently unobserved factors that can cause extreme swings in our outcome measurements. We define a cutoff point for outliers as a percentage (quantile) of the results distribution that is inversely proportional to the number of products in the data. This approach has been used previously, but we validated it in simulations.

Estimate second difference from grouped products using causal forest. In DID analysis, the second difference is the difference between the treatment and control groups’ first differences. Because we’re considering groups of heterogeneous products, we calculate the second difference only for products that have strong enough affinities with each other to make the comparison informative. Then we average the second difference across products.

To compute affinity scores, we use a variation on decision trees called causal forests. A typical decision tree is a connected acyclic graph — a tree — each of whose nodes represents a question. In our case, those questions regard product characteristics — say, “Does it require replaceable batteries?”, or “Is its width greater than three inches?”. The answer to the question determines which branch of the tree to follow.

Related content
New method goes beyond Granger causality to identify only the true causes of a target time series, given some graph constraints.

A causal forest consists of many such trees. The questions are learned from the data, and they define the axes along which the data shows the greatest variance. Consequently, the data used to train the trees requires no labeling.

After training our causal forest, we use it to evaluate the products in our experiment. Products from the treatment and control groups that end up at the same terminal node, or leaf, of a tree are deemed similar enough that their second difference should be calculated.

Bootstrap data to estimate noise. To compute the standard error, we randomly sample products from our dataset and calculate their average treatment effect, then return them to the dataset and randomly sample again. Multiple resampling allows us to compute the variance in our outcome measures.

Spillover effect

At the Amazon Pricing Labs, we have also investigated ways to gauge the spillover effect, which occurs when treatment of one product causes a change in demand for another, similar product. This can throw off our measurements of treatment effect.

For instance, if a new pricing policy increases demand for, say, a particular kitchen chair, more customers will view that chair’s product page. Some fraction of those customers, however, may buy a different chair listed on the page’s “Discover similar items” section.

If the second chair is in the control group, its sales may be artificially inflated by the treatment of the first chair, leading to an underestimation of the treatment effect. If the second chair is in the treatment group, the inflation of its sales may lead to an overestimation of the treatment effect.

To correct for the spillover effect, we need to measure it. The first step in that process is to build a graph of products with correlated demand.

Related content
“Group testing” protocols tailored to particularities of the COVID-19 pandemic promise more-informative test results.

We begin with a list of products that are related to each other according to criteria such as their fine-grained classifications in the Amazon Store catalogue. For each pair of related items, we then look at a year’s worth of data to determine whether a change in the price of one affects demand for another. If those connections are strong enough, we join the products by an edge in our substitutable-items graph.

From the graph, we compute the probability that any given pair of substitutable products will find themselves included in the same experiment and which group, treatment or control, they’ll be assigned to. From those probabilities, we can use an inverse probability-weighting schema to estimate the effect of spillover on our observed outcomes.

Estimating spillover effect, however, is not as good as eliminating it. One way to do that is to treat substitutable products as a single product class and assign them to treatment or control groups en masse. This does reduce the power of our experiments, but it gives our business partners confidence that the results aren’t tainted by spillover.

To determine which products to include in each of our product classes, we use a clustering algorithm that searches the substitutable-product graph for regions of dense interconnection and severs those regions connections to the rest of the graph. In an iterative process, this partitions the graph into clusters of closely related products.

In simulations, we found that this clustering process can reduce spillover bias by 37%.

Research areas

Related content

US, WA, Seattle
Are you interested in working with top talent in Optimization, Operations Research and Supply Chain to help Amazon to efficiently match our Devices with worldwide customers? We have challenging problems and need your innovative solutions to make tremendous financial impacts! The Amazon Devices Science team is looking for a Research Scientist with background in Operations Research, Optimization, Supply Chain and/or Simulation to support science efforts to integrate across inventory management functionalities. Our team is responsible for science models (both deterministic and stochastic) that power world-wide inventory allocation for Amazon Devices business that includes Echo, Kindle, Fire Tablets, Amazon TVs, Amazon Fire TV sticks, Ring, and other smart home devices. We formulate and solve challenging large-scale financially-based optimization problems which ingest demand forecasts and produce optimal procurement, production, distribution, and inventory management plans. In addition, we also work closely with demand forecasting, material procurement, production planning, finance, and logistics teams to co-optimize the inventory management and supply chain for Amazon Devices given operational constraints. Key job responsibilities The successful candidate will be a self-starter, comfortable with ambiguity, with strong attention to detail, and ability to work in a fast-paced and ever-changing environment and a desire to help shape the overall business. Job responsibilities include: - Design and develop advanced mathematical, simulation, and optimization models and apply them to define strategic and tactical needs and drive appropriate business and technical solutions in the areas of inventory management and distribution, network flow, supply chain optimization, and demand planning - Apply mathematical optimization techniques (linear, quadratic, SOCP, robust, stochastic, dynamic, mixed-integer programming, network flows, nonlinear, nonconvex programming) and algorithms to design optimal or near optimal solution methodologies to be used by in-house decision support tools and software - Research, prototype and experiment with these models by using modeling languages such as Python; participate in the production level deployment - Create, enhance, and maintain technical documentation, and present to other Scientists, Product, and Engineering teams - Support project plans from a scientific perspective by managing product features, technical risks, milestones and launch plans - Influence organization's long-term roadmap and resourcing, and onboard new technologies onto the Science team's toolbox We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Seattle
Are you passionate about solving unique customer-facing problem in the Amazon scale? Are you excited by developing and productizing machine learning, deep learning algorithms and leverage tons of Amazon data to learn and infer customer shopping patterns? Do you enjoy working with a diversity of engineers, machine learning scientists, product managers and user-experience designers? If so, you have found the right match! Fashion is extremely fast-moving, visual, subjective, and it presents numerous unique problem domains such as product recommendations, product discovery and evaluation. The vision for Amazon Fashion is to make Amazon the number one online shopping destination for Fashion customers by providing large selections, inspiring and accurate recommendations and customer experience. The mission of Fit science team as part of Fashion Tech is to innovate and develop scalable ML solutions to provide personalized fit and size recommendation when Amazon Fashion customers evaluate apparels or shoes online. The team is hiring Applied Scientist who has a solid background in applied Machine Learning and a proven record of solving customer-facing problems via scalable ML solutions, and is motivated to grow professionally as an ML scientist. Key job responsibilities Tackle ambiguous problems in Machine Learning and drive full life-cycle Machine Learning projects. Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production. Run A/B experiments, gather data, and perform statistical tests. Establish scalable, efficient, automated processes for large-scale data mining, machine-learning model development, model validation and serving. Work closely with software engineers and product managers to assist in productizing your ML models. We are open to hiring candidates to work out of one of the following locations: San Diego, CA, USA | San Francisco, CA, USA | Santa Monica, CA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, MA, North Reading
Are you excited about developing generative AI and foundation models to revolutionize automation, robotics and computer vision? Are you looking for opportunities to build and deploy them on real problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience - at Amazon scale. We are looking for scientists, engineers and program managers for a variety of roles. The Research team at Amazon Robotics is seeking a passionate, hands-on Sr. Applied Scientist to help create the world’s first foundation model for a many-robot system. The focus of this position is how to predict the future state of our warehouses that feature a thousand or more mobile robots in constant motion making deliveries around the building. It includes designing, training, and deploying large-scale models using data from hundreds of warehouses under different operating conditions. This work spans from research such as alternative state representations of the many-robot system for training, to experimenting using simulation tools, to running large-scale A/B tests on robots in our facilities. Key job responsibilities * Research vision - Where should we be focusing our efforts * Research delivery - Proving/dis-proving strategies in offline data or in simulation * Production studies - Insights from production data or ad-hoc experimentation * Production implementation - Building key parts of deployed algorithms or models About the team You would join our multi-disciplinary science team that includes scientists with backgrounds in planning and scheduling, grasping and manipulation, machine learning, and operations research. We develop novel planning algorithms and machine learning methods and apply them to real-word robotic warehouses, including: - Planning and coordinating the paths of thousands of robots - Dynamic allocation and scheduling of tasks to thousands of robots - Learning how to adapt system behavior to varying operating conditions - Co-design of robotic logistics processes and the algorithms to optimize them Our team also serves as a hub to foster innovation and support scientists across Amazon Robotics. We also coordinate research engagements with academia, such as the Robotics section of the Amazon Research Awards. We are open to hiring candidates to work out of one of the following locations: North Reading, MA, USA | Westborough, MA, USA
US, WA, Bellevue
Inventory Planning and Control (IPC) is seeking an experienced senior data scientist to join its central science team. Our team owns the core decision models in the space of Buying, Placement, and Capacity Control. Our models decide when, where, and how much we should buy, flow, and hold inventories in our global fulfillment network to meet Amazon’s business goals and to make our customers happy. We do this for hundreds of millions of items and hundreds of product lines worth billions of dollars of world-wide for both our Retail and third-party seller business. Our systems are built entirely in-house, for which we constantly develop new technologies in automated inventory planning, prediction, optimization and simulation. Our systems operate at various scales, from real-time decision system that completes thousands of transactions per seconds, to large scale distributed system that optimizes the inventory decisions over millions of products simultaneously. IPC is also unique in that we are simultaneously developing the science and software of inventory optimization and solving some of the toughest computational/operational challenges in production. Our team members have an opportunity to be on the forefront of supply chain thought leadership by working on some of the most difficult problems in the industry with some of the best product managers, scientists, and software engineers in the industry. Key job responsibilities Candidates will be responsible for developing causal, machine learning and data driven models to enhance the various inventory optimization engines that the team owns. The successful candidate should have solid hands-on experience in applying machine learning or causal inference models. They will also be responsible for conducting data driven analysis to facilitate strategic decisions. They require superior logical thinkers who are able to quickly approach large ambiguous problems and develop a practical plan to tackle. Successful candidates must thrive in fast-paced environments, which encourage collaborative and creative problem solving. They are able to measure and estimate risks, and constructively critique peer research. As a senior scientist, you will also help coach/mentor junior scientists in the team. A day in the life The IPC science team contains a large group of scientists with different technical expertise, who will help and collaborate with you on your projects. In this role, you will also work with our internal customers from the Retail, third-party seller and operations departments worldwide. You will understand their challenges and pain points, and help develop data driven solutions that improve how Amazon manages inventory in our global supply chain. You will work closely with the product managers, engineers and other scientists to turn science proposals into production implementation. About the team We are a team of scientists, product managers and engineers focusing on innovation. We promote experimentation and learn by building. We often tackle the hardest problem in the organization and work cross-functionally. We are at the center of developing inventory solutions to support the rapid growth of Amazon's store business. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Seattle
Amazon Prime is looking for an ambitious Economist to help create econometric insights for world-wide Prime. Prime is Amazon's premiere membership program, with over 200M members world-wide. This role is at the center of many major company decisions that impact Amazon's customers. These decisions span a variety of industries, each reflecting the diversity of Prime benefits. These range from fast-free e-commerce shipping, digital content (e.g., exclusive streaming video, music, gaming, photos), and grocery offerings. Prime Science creates insights that power these decisions. As an economist in this role, you will create statistical tools that embed causal interpretations. You will utilize massive data, state-of-the-art scientific computing, econometrics (causal, counterfactual/structural, time-series forecasting, experimentation), and machine-learning, to do so. Some of the science you create will be publishable in internal or external scientific journals and conferences. You will work closely with a team of economists, applied scientists, data professionals (business analysts, business intelligence engineers), product managers, and software engineers. You will create insights from descriptive statistics, as well as from novel statistical and econometric models. You will create internal-to-Amazon-facing automated scientific data products to power company decisions. You will write strategic documents explaining how senior company leaders should utilize these insights to create sustainable value for customers. These leaders will often include the senior-most leaders at Amazon. The team is unique in its exposure to company-wide strategies as well as senior leadership. It operates at the cutting-edge of utilizing data, econometrics, artificial intelligence, and machine-learning to form business strategies. A successful candidate will have demonstrated a capacity for building, estimating, and defending statistical models (e.g., causal, counterfactual, time-series, machine-learning) using software such as R, Python, or STATA. They will have a willingness to learn and apply a broad set of statistical and computational techniques to supplement deep-training in one area of econometrics. For example, many applications on the team use structural econometrics, machine-learning, and time-series forecasting. They rely on building scalable production software, which involves a broad set of world-class software-building skills often learned on-the-job. As a consequence, already-obtained knowledge of SQL, machine learning, and large-scale scientific computing using distributed computing infrastructures such as Spark-Scala or PySpark would be a plus. Additionally, this candidate will show a track-record of delivering projects well and on-time, preferably in collaboration with other team members (e.g. co-authors). Candidates must have very strong writing and emotional intelligence skills (for collaborative teamwork, often with colleagues in different functional roles), a growth mindset, and a capacity for dealing with a high-level of ambiguity. Endowed with these traits and on-the-job-growth, the role will provide the opportunity to have a large strategic, world-wide impact on the customer experiences of Prime members. Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, visit https://www.amazon.jobs/en/disability/us We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Chicago, IL, USA | Seattle, WA, USA
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! The Books Advertising team owns the worldwide advertising business for books, including advertiser and shopper experiences. They develop long-term vision and drive improvements for category relevance, auction dynamics, and ad serving. Additionally, they drive advertiser engagement, represent advertisers' voice, and provide operational support for our programs. This means the team owns all book-specific experiences for Sponsored Products, Sponsored Brands, Sponsored Display, Lock Screen Advertising, the Ads Console, and the Public API. As an Senior Applied Scientist on this team, you will: - Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. - Perform hands-on analysis and modeling of enormous data sets to develop insights that increase traffic monetization and merchandise sales, without compromising the shopper experience. - Build machine learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. - Run A/B experiments, gather data, and perform statistical analysis. - Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. - Research new and innovative machine learning approaches. - Recruit Applied Scientists to the team and provide mentorship. Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. Team video https://youtu.be/zD_6Lzw8raE ** Candidates can be based within proximity of NYC, Seattle, Toronto, Arlington County/Virginia (HQ2), or Santa Monica ** We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | New York, NY, USA | Santa Monica, CA, USA
US, WA, Seattle
We are designing the future. If you are in quest of an iterative fast-paced environment, where you can drive innovation through scientific inquiry, and provide tangible benefit to hundreds of thousands of our associates worldwide, this is your opportunity. Come work on the Amazon Worldwide Fulfillment Design & Engineering Team! We are looking for an experienced and senior Research Scientist with background in Ergonomics and Industrial Human Factors, someone that is excited to work on complex real-world challenges for which a comprehensive scientific approach is necessary to drive solutions. Your investigations will define human factor / ergonomic thresholds resulting in design and implementation of safe and efficient workspaces and processes for our associates. Your role will entail assessment and design of manual material handling tasks throughout the entire Amazon network. You will identify fundamental questions pertaining to the human capabilities and tolerances in a myriad of work environments, and will initiate and lead studies that will drive decision making on an extreme scale. .You will provide definitive human factors/ ergonomics input and participate in design with every single design group in our network, including Amazon Robotics, Engineering R&D, and Operations Engineering. You will work closely with our Worldwide Health and Safety organization to gain feedback on designs and work tenaciously to continuously improve our associate’s experience. Key job responsibilities - Collaborating and designing work processes and workspaces that adhere to human factors / ergonomics standards worldwide. - Producing comprehensive and assessments of workstations and processes covering biomechanical, physiological, and psychophysical demands. - Effectively communicate your design rationale to multiple engineering and operations entities. - Identifying gaps in current human factors standards and guidelines, and lead comprehensive studies to redefine “industry best practices” based on solid scientific foundations. - Continuously strive to gain in-depth knowledge of your profession, as well as branch out to learn about intersecting fields, such as robotics and mechatronics. - Travelling to our various sites to perform thorough assessments and gain in-depth operational feedback, approximately 25%-50% of the time. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Bellevue
Are you excited about developing state-of-the-art deep learning foundation models, applied to the automation of labor for the future of Amazon’s Fulfillment network? Are you looking for opportunities to build and deploy them on real problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience - at Amazon scale. To this end, we are looking for an Applied Scientist who will build and deploy models that help automate labor utilizing a wide array of multi-modal signals. Together, we will be pushing beyond the state of the art in optimization of one of the most complex systems in the world: Amazon's Fulfillment Network. Key job responsibilities In this role, you will build models that can identify potential problems with Amazon’s vast inventory, including discrepancies between the physical and virtual manifest and efficient execution of inventory audit operations. You will work with a diverse set of real world structured, unstructured and potentially multimodal datasets to train deep learning models that identify current inventory management problems and anticipate future ones. Datasets include multiple separate inventory management event streams, item images and natural language. You will face a high level of research ambiguity and problems that require creative, ambitious, and inventive solutions. About the team Amazon Fulfillment Technologies (AFT) powers Amazon’s global fulfillment network. We invent and deliver software, hardware, and data science solutions that orchestrate processes, robots, machines, and people. We harmonize the physical and virtual world so Amazon customers can get what they want, when they want it. The AFT AI team has deep expertise developing cutting edge AI solutions at scale and successfully applying them to business problems in the Amazon Fulfillment Network. These solutions typically utilize machine learning and computer vision techniques, applied to text, sequences of events, images or video from existing or new hardware. We influence each stage of innovation from inception to deployment, developing a research plan, creating and testing prototype solutions, and shepherding the production versions to launch. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, VA, Herndon
The AWS Managed Operations Data Science (MODS) Team is looking for a Senior Data Scientist to lead the research and thought leadership to drive our data and insight strategy for AWS. You will develop sophisticated algorithms that involve learning from large amounts of data. You will be responsible for researching, experimenting and analyzing predictive and optimization models. You will provide to engineers, direction on technology strategy in data management, insights and Machine learning that will form the nucleus of providing data and insights as a service to our customers. You will work on ambiguous and complex business and research science problems with large opportunities. You are and comfortable working with cross-functional teams and systems. This position requires that the candidate selected be a US Citizen. Key job responsibilities - Interact with business and software teams to understand their business requirements and operational processes - Frame business problems into scalable solutions - Adapt existing and invent new techniques for solutions - Create and track accuracy and performance metrics - Gather data required for analysis and model building - Prototype models by using high-level modeling languages such as R or in software languages such as Python. A software team will be working with you to transform prototypes into production. - Interview, mentor, and develop scientists and engineers across the organization - Create, enhance, and maintain technical documentation A day in the life Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. We are dedicated to supporting our new team members. Our team has a broad mix of experience levels and Amazon tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our team also puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work. About the team The MODS team is driven by a shared vision of achieving operational excellence through data analytics and machine learning. We provide actionable insights to allow our stakeholders to manage operational posture and operator experience and drive sustainable, safe, and efficient operations. We define, monitor, and predict metrics to provide recommendations on AWS operations that are diagnostic (why something happened), predictive (what will happen) and prescriptive (best course of action) in nature. We are a customer obsessed team driving lean operations in all of AWS through actionable insights and data strategies that drive process improvement. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Herndon, VA, USA
US, CA, Santa Monica
Amazon Advertising is looking for a motivated and analytical self-starter to help pave the way for the next generation of insights and advertising products. You will use large-scale data, advertising effectiveness knowledge and business information needs of our advertising clients to envision new advertising measurement products and tools. You will facilitate innovation on behalf of our customers through end-to-end delivery of measurement solutions leveraging experiments, machine learning and causal inference. You will partner with our engineering teams to develop and scale successful solutions to production. This role requires strong hands-on skills in terms of effectively working with data, coding, and MLOps. However, the ideal candidate will also bring strong interpersonal and communication skills to engage with cross-functional partners, as well as to stay connected to insights needs of account teams and advertisers. This is a truly exciting and versatile position in that it allows you to apply and develop your hands-on data modeling and coding skills, to work with other scientists on research in new measurement solutions while at the same time partner with cross-functional stakeholders to deliver product impact. Key job responsibilities As an Applied Scientist on the Advertising Incrementality Measurement team you will: - Create new analytical products from conception to prototyping and scaling the product end-to-end through to production. - Scope and define new business problems in the realm of advertising effectiveness. Use machine learning and experiments to develop effective and scalable solutions. - Partner closely with the Engineering team. - Partner with Economists, Data Scientists, and other Applied Scientists to conduct research on advertising effectiveness using machine learning and causal inference. Make findings available via white papers. - Act as a liaison to product teams to help productize new measurement solutions. About the team Advertising Incrementality Measurement combines experiments with econometric analysis and machine learning to provide rigorous causal measurement of advertising effectiveness to internal and external customers. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Boulder, CO, USA | New York, NY, USA | Santa Monica, CA, USA