Evaluating the helpfulness of AI-enhanced catalogue data

Using causal random forests and Bayesian structural time series to extrapolate from sparse data ensures that customers get the most useful information as soon as possible.

Millions of customers around the world rely on Amazon's vast online catalogue, containing information on hundreds of millions of products, to make informed purchase decisions. To ensure that catalogue data is comprehensive, consistent, and accurate, the Amazon Catalog team uses a wide range of machine learning models — including generative-AI models that synthesize textual and visual information from seller listings, manufacturer websites, customer reviews, and other sources to enrich catalogue data.

Two versions of the same product page. Lengthy product descriptions in the "about this item" section of the version at left have been condensed in the version at right.
Generative-AI models can synthesize information from a range of sources with the aim of making product information clearer and more concise.

To ensure that the enriched data improves the customer experience, Amazon Catalog conducts A/B experiments that expose some of our customers to the enriched information, while others encounter the current alternative.

Two versions of the same product page, for a thermal-underwear shirt. At left, the shirt model is standing in front of a white background; at right, a background including shelves holding art and a potted plant have been added by a generative-AI model.
Generative-AI models can also enrich product-related images. At left is the control image in an A/B test, and at right is the treated image.

But A/B testing can incur opportunity costs, as we delay the rollout of catalogue improvements for some of our customers, and maintaining two backend systems is resource intensive. To tackle these challenges, we propose two different scientific approaches.

Machine-learning-based extrapolation model

First, we seek to run as few experiments as possible. We have developed a scalable machine-learning-based extrapolation model that effectively incorporates insights gained from prior experiments on enrichment initiatives and applies them to new contexts. We tailor the causal-random-forest approach, which is itself an extension of the classical random-forest algorithm, to our setting.

Related content
The requirement that at any given time, all customers see the same prices for the same products necessitates innovation in the design of A/B experiments.

During training on our existing A/B experiments, the algorithm randomly selects training and validation datasets and generates an ensemble of causal decision trees. Each tree splits the products involved in the experiments into smaller subgroups, sorted by similarity of features, balancing the within-sample fit of the observed outcomes by treatment status and the out-of-sample performance on the validation dataset. We then aggregate the different predictions from all causal trees to generate one prediction about the effect of enrichments given different product features. After training the model, we can validate it in additional experiments to compare predicted to actual treatment effects.

The validated model lets us test whether there are differences in the responses to an enrichment across products. If that is the case, we can focus our enrichment efforts on those product groups that respond particularly well. In addition, we can use the estimates to predict and document the impact of our planned enrichments on our customers as we tackle different product groups throughout the year. As an example, we can use the model to assess the impact of our efforts to correct and complete product information across the catalogue with only a limited set of experiments.

Bayesian structural time series

When conducting A/B experiments is not feasible, we can turn to observational modeling techniques such as Bayesian structural-time-series modeling. This approach synthesizes ideas from time-series analysis, the synthetic control method, and Bayesian statistics.

Related content
Amazon Scholar David Card and Amazon academic research consultant Guido Imbens talk about the past and future of empirical economics.

As we monitor sales of all our products over time, we can pair any product group with a synthetic twin that mirrors its sales performance, accounting for sales trends and seasonality. If we then enrich a group of products and observe a noticeable difference in sales performance, we can attribute the change to our enrichment efforts. Operating within the Bayesian framework enables us to integrate prior knowledge from our various analyses, including our A/B experiments, and effectively communicate uncertainty to our business stakeholders.

We have validated our observational model against our A/B experiments for selected use cases where both methods are feasible and have used this approach to evaluate the impact of our large-scale machine learning systems, which automatically classify products to improve their discoverability by customers through search or browsing.

With enriched catalogue data, Amazon customers are able to make better-informed and more-confident shopping decisions.

Acknowledgements: Philipp Eisenhauer

Research areas

Related content

US, WA, Bellevue
Does the idea of creating technology solutions for delivering 11 Billion+ packages across the globe excite you? If yes, come join a fun-loving, diverse, and creative team at Amazon Last Mile! The vision of the team is "To create Earth’s safest, most adaptive, and efficient plans for Last Mile logistics". The Last Mile Delivery Technology team is instrumental in impacting customer satisfaction directly, by devising innovative ways to deliver packages quickly and cost-effectively to the customers, and at scale using Artificial Intelligence (AI), Machine Learning and Operations Research solutions. Last Mile Delivery Technology organization supports the design, planning and execution of last mile transportation for Amazon’s various parcel and grocery delivery programs. All these programs require a large number of decision support systems to operate at scale and serve our customers, spanning demand planning, jurisdiction planning, delivery channel and network design, capacity planning for on the road and under the roof at delivery stations, routing inputs and route optimization. While these decision support systems have thus far been approached through the lens of traditional optimization and machine learning, we are looking to re-envision this space and pursue Foundational AI research, to innovate and advance the state of these decision support systems. Specifically, we are looking to develop foundational models (including Large Language Models, Multimodal Language Models, Multimodal Models), and adaptations to serve last mile use cases. Beyond Amazon the work developed will spur new fundamental knowledge and innovation in the logistics space. Job Location : Bellevue WA or Austin TX. Key job responsibilities You have deep expertise in ML/AI, staying current with the latest research and techniques. You also invent or adapt new scientific approaches based on customer needs, producing high-quality research reports and contributing to peer-reviewed publications when appropriate You are a highly skilled software engineer whose work is consistently of high quality, meets industry standards, and incorporates best practices. You work semi-autonomously, contribute to operational excellence. You have strong interpersonal and leadership skills, effectively collaborating with your team, championing scientific advancements, onboarding new teammates, setting a high standard for your scientific contributions, and actively participating in the wider scientific community
US, WA, Seattle
The Worldwide Defect Elimination (WWDE) Science team in Amazon Customer Service builds state-of-the-art Artificial Intelligence (AI) models to enable defect-free shopping experiences for Amazon customers. We develop technology and mechanisms to discover, root cause, measure, and escalate defects for resolution before they impact a broader range of customers. We are looking for a creative problem solver and technically-skilled Research Scientist able and interested in building AI solutions to address customer issues at scale. The ideal candidate will lead the development of innovative solutions that identify, root cause, attribute, and summarize problems embedded in large volumes of customer feedback in different modalities. They will also utilize the latest advances in GenAI technology to explore billions of customer contacts and automate defect resolution workflows. As a part of this role, this candidate will collaborate with a large team of experts in the field and move the state of defect elimination research forward. This candidate should have a knack for leveraging AI to translate complex data insights into actionable strategies and can communicate these effectively to both technical and non-technical audiences. Key job responsibilities * Apply science models to extract actionable information from large volumes and varying modalities of customer feedback * Leverage GenAI/Large Language Model (LLM) technology for scaling and automating defect elimination workflows * Design and implement metrics to evaluate the effectiveness of AI models * Present deep dives and analysis to both technical and non-technical stakeholders, ensuring clarity and understanding and influencing business partners * Perform statistical analysis and statistical tests including hypothesis testing and A/B testing * Recognize and adopt best practices in reporting and analysis: data integrity, test design, analysis, validation, and documentation A day in the life If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan About the team The Worldwide Defect Elimination (WWDE) team's mission is to understand and resolve all issues impacting customers at scale. The WWDE Science team is a force multiplier within this group, helping to to apply science solutions to eliminate defects and enhance customer experience.
US, TX, Austin
The Automated Reasoning Group in AWS Utility Computing is looking for a Senior Applied Scientist with experience in building scalable automated reasoning solutions that delight customers. You will be part of a world-class team building the next generation of automated reasoning tools and services. You will apply your knowledge to propose solutions, create software prototypes, and develop prototypes into production systems using software development tools. In addition, you will support and scale your solutions to meet the ever-growing demand of customer use. You have demonstrated leadership in automated reasoning positions in industry or academia, strong verbal and written communication skills, are self-driven and deliver high quality results in a fast-paced environment. Each day, hundreds of thousands of developers make billions of transactions worldwide on AWS. They harness the power of the cloud to enable innovative applications, websites, and businesses. Using automated reasoning technology and mathematical proofs, AWS allows customers to answer questions about security, availability, durability, and functional correctness. We call this provable security, absolute assurance in security of the cloud and in the cloud. https://aws.amazon.com/security/provable-security/ Key job responsibilities As a Senior Applied Scientist, you will help shape the definition and vision for applied science across teams within AWS. We have a diverse portfolio of projects that target protocol, code, and hardware verification, and leadership opportunities exist for: - Advance automated code-level reasoning and invariant synthesis and proof repair for cloud-scale web services. - Build new engines and extending foundational proof engines that apply to distributed systems. - Researching the application of automated reasoning to novel software applications. - Building automated reasoning solutions for critical AWS DSLs for architectural configuration, migration, code generation, and other areas. - Improving integration and user experience of tools to support large-scale adoption and use of automated reasoning techniques. You will work in an agile, startup-like development environment, where you are always working on the most important things, and you will design, implement, test, deploy and maintain innovative software solutions to transform service performance, durability, cost, and security. About the team About AWS Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. This team is part of AWS Utility Computing: Utility Computing (UC) AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps services in AWS, including support for customers who require specialized security solutions for their cloud services.
AE, Dubai
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The Generative AI Innovation Center at AWS is a new strategic team that helps AWS customers implement Generative AI solutions and realize transformational business opportunities. This is a team of strategists, data scientists, engineers, and solution architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and make plans for launching solutions at scale. The GenAI Innovation Center team provides guidance on best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for ML Data Scientists capable of using GenAI and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. Key job responsibilities As an ML Data Scientist, you will * Collaborate with ML scientist and architects to Research, design, develop, and evaluate cutting-edge generative AI algorithms to address real-world challenges * Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production * Create and deliver best practice recommendations, tutorials, blog posts, sample code, and presentations adapted to technical, business, and executive stakeholder * Provide customer and market feedback to Product and Engineering teams to help define product direction
US, WA, Bellevue
We are seeking a passionate, talented, and inventive individual to join the Applied AI team and help build industry-leading technologies that customers will love. This team offers a unique opportunity to make a significant impact on the customer experience and contribute to the design, architecture, and implementation of a cutting-edge product. The mission of the Applied AI team is to enable organizations within Worldwide Amazon.com Stores to accelerate the adoption of AI technologies across various parts of our business. We are looking for a Senior Applied Scientist to join our Applied AI team to work on LLM-based solutions. We are seeking an experienced Scientist who combines superb technical, research, analytical and leadership capabilities with a demonstrated ability to get the right things done quickly and effectively. This person must be comfortable working with a team of top-notch developers and collaborating with our research teams. We’re looking for someone who innovates, and loves solving hard problems. You will be expected to have an established background in building highly scalable systems and system design, excellent project management skills, great communication skills, and a motivation to achieve results in a fast-paced environment. You should be somebody who enjoys working on complex problems, is customer-centric, and feels strongly about building good software as well as making that software achieve its operational goals. Key job responsibilities You will be responsible for developing and maintaining the systems and tools that enable us to accelerate knowledge operations and work in the intersection of Science and Engineering. You will push the boundaries of ML and Generative AI techniques to scale the inputs for hundreds of billions of dollars of annual revenue for our eCommerce business. If you have a passion for AI technologies, a drive to innovate and a desire to make a meaningful impact, we invite you to become a valued member of our team. A day in the life On our team you will push the boundaries of ML and Generative AI techniques to scale the inputs for hundreds of billions of dollars of annual revenue for our eCommerce business. If you have a passion for AI technologies, a drive to innovate and a desire to make a meaningful impact, we invite you to become a valued member of our team.
US, WA, Seattle
We’re working to improve shopping on Amazon using the conversational capabilities of large language models, and are searching for pioneers who are passionate about technology, innovation, and customer experience, and are ready to make a lasting impact on the industry. You'll be working with talented scientists and engineers to innovate on behalf of our customers. If you're fired up about being part of a dynamic, driven team, then this is your moment to join us on this exciting journey! Key job responsibilities We seek strong Applied Scientists with domain expertise in machine learning and deep learning, transformers, generative models, large language models, computer vision and multimodal models. You will devise innovative solutions at scale, pushing the technological and science boundaries. You will guide the design, modeling, and architectural choices of state-of-the-art large language models and multimodal models. You will devise and implement new algorithms and new learning strategies and paradigms. You will be technically hands-on and drive the execution from ideation to productionization. You will work in collaborative environment with other technical and business leaders, to innovate on behalf of the customer.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics
IL, Tel Aviv
Are you a MS or PhD student interested in a 2024 Research Science Internship, where you would be using your experience to initiate the design, development, execution and implementation of scientific research projects? If so, we want to hear from you! Is your research in machine learning, deep learning, automated reasoning, speech, robotics, computer vision, optimization, or quantum computing? If so, we want to hear from you! We are looking for motivated students with research interests in a variety of science domains to build state-of-the-art solutions for never before solved problems You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science Key job responsibilities As a Research Science Intern, you will have following key job responsibilities; • Work closely with scientists and engineering teams (position-dependent) • Work on an interdisciplinary team on customer-obsessed research • Design new algorithms, models, or other technical solutions • Experience Amazon's customer-focused culture A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships and up to 12 months for part time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
US, WA, Seattle
Join us in building on AWS, for AWS! Amazon Web Services (AWS) provides companies of all sizes with an infrastructure web services platform in the cloud (“Cloud Computing”). With AWS you can requisition compute power, storage, and many other services – gaining access to a suite of elastic IT infrastructure services as your business demands them. AWS is the leading platform for designing and developing applications for the cloud and is growing rapidly with hundreds of thousands of companies in over 190 countries on the platform. Developers all over the world rely on our storage, compute, and virtualized services. Our success depends on our world-class selling and field teams, and these teams rely on the Worldwide Sales Strategy and Operations (SMGS Ops) team to power their activities. We’re handling massive scale, providing data that drives the AWS business internally, and delivering products and services to help our Amazon Web Service selling teams, marketing groups, and customers. We’re looking for a Data Scientist to design and deliver solutions that combine machine learning, human-in-the-loop input, and distributed big data technologies. We're building a cutting-edge data platform to enable us to arm our field teams with the actionable intelligence needed to engage and serve every possible AWS customer in the world, to the fullest. This position may be based in Seattle, WA; Dallas, TX Key job responsibilities - Design solutions to complex and ambiguous data challenges, starting from first principles - Apply Machine Learning to solve data problems, such as record matching, at scale - Leverage company data from third-party sources in combination with internal AWS data to develop quantitative models answering critical business questions - Build human-in-the-loop workflows, to complement and augment ML solutions - Work with AWS machine learning and big data technologies such as Amazon Sagemaker, EMR, S3, DynamoDB, Lambda, and more - Experiment and explore new technologies to create innovative solutions - Use Natural Language Processing and language models to derive insights from unstructured sources like public company regulatory filings and annual reports About the team Here at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and we host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team also puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here, which is why we aren’t focused on how many hours you spend at work or online. Instead, we’re happy to offer a flexible schedule so you can have a more productive and well-balanced life—both in and outside of work. About AWS Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.
US, WA, Bellevue
The AGI Data Service team is seeking a dedicated, skilled, and innovative Scientist with a robust background in deep learning, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI-DS team, a Research Scientist will collaborate closely with talented colleagues to lead the development of advanced approaches and modeling techniques, driving forward the frontier of LLM technology. This includes innovating model-in-the-loop and human-in-the-loop approaches to ensure the collection of high-quality data, safeguarding data privacy and security for LLM training, and more. A scientist will also have a direct impact on enhancing customer experiences through state-of-the-art products and services that harness the power of speech and language technology. A day in the life The Scientist with the AGI team will support the science solution design, run experiments, research new algorithms, and find new ways of optimizing the customer experience; while setting examples for the team on good science practice and standards. Besides theoretical analysis and innovation, the scientist will also work closely with talented engineers and scientists to put algorithms and models into practice. The ideal candidate should be passionate about delivering experiences that delight customers and creating robust solutions. They will also create reliable, scalable and high-performance products that require exceptional technical expertise, and a sound understanding of Machine Learning.