Interpretable ensemble models improve product retrieval

Gradient-boosted decision trees aggregate model outputs, and Shapley values help identify the most useful models for the ensemble.

The field of machine learning is evolving at a rapid pace, with the regular release of new models that promise improvements over their predecessors. Evaluating a new model for a particular use case, however, is a time-consuming and resource-intensive process. That poses a conundrum for online services like Amazon’s store, which are committed to offering their customers state-of-the-art technology but operate at high volume 24 hours a day.

In a paper we presented at this year’s Web Conference, we propose a solution to this conundrum. Rather than use a single model — or a pair of models, a language model and a graph neural network — to process customers’ queries, we propose using an ensemble of models, whose outputs are aggregated by gradient-boosted decision trees (GBDTs).

By using Shapley values to determine how much each model contributes to the GBDTs’ final decision, we can rank the models by utility. Then, depending on the computational resources available, we keep only as many of the most-useful models as are practical to run in parallel.

Related content
Using large language models to discern commonsense relationships can improve performance on downstream tasks by as much as 60%.

A new model, which hasn’t yet been thoroughly evaluated for a particular use case, can be trained on whatever data is available and added to the ensemble, where it takes its chances with the existing models. Shapley value analysis may remove it from the ensemble, or it may determine that the new model has rendered an existing model obsolete. Either way, the customer gets the benefit of the best current technology.

We tested our approach using our Shopping Queries Dataset, a public dataset that we released as part of a 2022 challenge at the Conference on Knowledge Discovery and Data Mining. The dataset consists of millions of query-product pairs in three languages, where the relationships between queries and products have been labeled according to the ESCI scheme (exact, substitute, complement, or irrelevant). We trained three large language models (LLMs) and three graph neural networks (GNNs) on the dataset and then used three different metrics (accuracy, macro F1, and weighted F1) to compare them to an ensemble of all six, which used our GBDT-based approach. Across the board, the ensemble outperformed the individual models, often dramatically.

esci-schema.jpg
In this graph, the edges represent the relationships between [brand 1] phone and other products. The information retrieval problem can be characterized as predicting the labels of the unlabeled edges (indicated by question marks).

ESCI classification

Historically, information retrieval models have been evaluated according to the relevance of the results they return; Amazon developed the ESCI scheme as a finer-grained alternative. Given a query, a product can be classified as an exact match (the brand and/or make specified in the query); as a substitute (a product in the same product class, but from a different manufacturer); as a complement (a complementary product, such as a phone case when the query is for a phone); or as irrelevant (an important classification, as it applies to the large majority of products for a given query).

Related content
Time series forecasting enables up-to-the-minute trend recognition, while novel two-step training process improves forecast accuracy.

There are two principal ways to do ESCI classification: one is to fine-tune a language model, which bases its output solely on the text of the product description and the query, and the other is to use a GNN, which can factor in observed relationships between products and between products and queries.

For instance, at Amazon’s store, we build graphs that capture information about which products in different categories tend to be purchased together, which products tend to be viewed together in the course of a single search session, which products are most frequently purchased in connection with particular query terms, and so on.

GNNs map graph information to a representation space in an iterative process, first embedding the data associated with each node; then creating new embeddings that combine the embeddings of nodes, their neighbors, and the relationships between them; and so on, usually to a distance of one to four hops. GNNs fine-tuned on the ESCI task thus factor in information beyond the semantic content of queries and product descriptions.

Model ensembles

At Amazon, we’ve found that combining the outputs of fine-tuned LLMs and GNNs usually yields the best performance on the ESCI task. In our WebConf paper, we describe a general method for expanding the number of models we include in our ensemble.

The outputs of the separate models are aggregated by GBDTs. A decision tree is a model that makes a series of binary decisions — usually, whether the value of a particular data feature exceeds some threshold. The leaves of the tree are correlated with particular data classifications.

Related content
Amazon ICML paper proposes information-theoretic measurement of quantitative causal contribution.

To calculate how much each model in our ensemble contributes to the final output, we use Shapley additive explanations, a method based on the game-theoretical concept of Shapley values. With Shapley values, we systematically vary the inputs to the GBDT model and track how each variation propagates through the decision trees; the Shapley value formalism provides a way to use that data to estimate aggregate effects across all possible inputs.

This, in turn, allows us to calculate how much each model in the ensemble contributes to the GBDT model’s output. On that basis, we can select only the most useful models for inclusion in our ensemble — up to whatever threshold we deem computationally practical.

Of course, running an ensemble of models is more computationally expensive than running a single model (or a pair of models, one language model and one GNN). But in our paper, we describe several techniques for making ensemble models more efficient, such as caching the labels of previously seen query-product pairs, for later reuse, and precomputing the GNN embeddings for the neighborhoods around frequently retrieved products. Our experiments show that ensemble models should be practical for real-time deployment.

Related content

US, CA, Sunnyvale
The Amazon Artificial General Intelligence (AGI) Personalization team is looking for a passionate, highly skilled and inventive Applied Scientist with strong machine learning background to build state-of-the-art ML systems for personalizing large-scale, high-quality conversational assistant systems. As a Applied Scientist, you will play a critical role in driving the development of personalization techniques enabling conversational systems, in particular those based on large language models, information retrieval, recommender systems and knowledge graph, to be tailored to customer needs. You will handle Amazon-scale use cases with significant impact on our customers' experiences. Key job responsibilities - Use deep learning, ML and NLP techniques to create scalable solutions for creation and development of language model centric solutions for building personalized assistant systems based on a rich set of structured and unstructured contextual signals - Innovate new methods for contextual knowledge extraction and information retrieval, using language models in combination with other learning techniques, that allows effective grounding in context providers when considering memory, compute, latency and quality - Research in advanced customer understanding and behavior modeling techniques - Collaborate with cross-functional teams of scientists, engineers, and product managers to identify and solve complex problems in personal knowledge aggregation, processing, modeling, and verification - Design and execute experiments to evaluate the performance of state-of-the-art algorithms and models, and iterate quickly to improve results - Think Big on conversational assistant system personalization over a multi-year horizon, and identify new opportunities to apply these technologies to solve real-world problems - Communicate results and insights to both technical and non-technical audiences, including through presentations and written reports About the team The AGI Personalization org uses various contextual signals to personalize Large Language Model output for our customers while maintaining privacy and security of customer data. We work across multiple Amazon products, including Alexa, to enhance the user experience by bringing more personal context and relevance to customer interactions.
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The SP team's primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! Why you love this opportunity Amazon is investing heavily in building a world-class advertising business. This team is responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven fundamentally from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. Key job responsibilities Key job responsibilities As an Applied Scientist III on this team you will: * Lead complex and ambiguous projects to deliver bidding recommendation products to advertisers. * Build machine learning models and utilize data analysis to deliver scalable solutions to business problems. * Perform hands-on analysis and modeling with very large data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. * Work closely with software engineers on detailed requirements, technical designs and implementation of end-to-end solutions in production. * Design and run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. * Work with scientists and economists to model the interaction between organic sales and sponsored content and to further evolve Amazon's marketplace. * Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. * Research new predictive learning approaches for the sponsored products business. * Write production code to bring models into production. * Mentor junior scientists and engineer in the team.
CA, ON, Toronto
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. As a core product offering within our advertising portfolio, Sponsored Products (SP) helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The SP team's primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! Why you love this opportunity Amazon is investing heavily in building a world-class advertising business. This team is responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven fundamentally from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. Key job responsibilities As an Applied Scientist on this team you will: * Build machine learning models and utilize data analysis to deliver scalable solutions to business problems. * Perform hands-on analysis and modeling with very large data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. * Work closely with software engineers on detailed requirements, technical designs and implementation of end-to-end solutions in production. * Design and run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. * Work with scientists and economists to model the interaction between organic sales and sponsored content and to further evolve Amazon's marketplace. * Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. * Research new predictive learning approaches for the sponsored products business. * Write production code to bring models into production.
US, WA, Seattle
We are seeking a Senior Applied Scientist to join our AI Security team, which builds security tooling and paved path solutions to ensure Generative AI (GenAI) based experiences developed by Amazon uphold our high security standards to protect customer data and trust. As a Senior Applied Scientist, you’ll be responsible for researching, modeling, designing, and implementing state-of-the-art solutions, to secure GenAI products and services at Amazon scale. You will collaborate with applied scientists, security engineers, software engineers, as well as internal partners and external researchers to develop innovative technologies to solve some of our hardest security problems, and build paved path solutions that support builder teams across Amazon throughout their software development journey, enabling Amazon businesses to accelerate the use of GenAI to enhance our user experiences and delight our customers. Key job responsibilities • Research and develop accurate and scalable methods to solve our hardest AI security problems. • Lead and partner with applied scientists and engineers to drive modeling and technical design for complex problems in GenAI security. • Build security tooling and paved path solutions that support builder teams throughout their software development journey. About the team About the team Diverse Experiences Amazon Security values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why Amazon Security At Amazon, security is central to maintaining customer trust and delivering delightful customer experiences. Our organization is responsible for creating and maintaining a high bar for security across all of Amazon’s products and services. We offer talented security professionals the chance to accelerate their careers with opportunities to build experience in a wide variety of areas including cloud, devices, retail, entertainment, healthcare, operations, and physical stores. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve. Inclusive Team Culture In Amazon Security, it’s in our nature to learn and be curious. Ongoing DEI events and learning experiences inspire us to continue learning and to embrace our uniqueness. Addressing the toughest security challenges requires that we seek out and celebrate a diversity of ideas, perspectives, and voices. Training & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, training, and other career-advancing resources here to help you develop into a better-rounded professional.
US, CA, Sunnyvale
The Amazon Artificial General Intelligence (AGI) Personalization team is looking for a passionate, highly skilled and inventive Senior Applied Scientist with strong machine learning background to lead the development and implementation of state-of-the-art ML systems for personalizing large-scale, high-quality conversational assistant systems. As a Senior Applied Scientist, you will play a critical role in driving the development of personalization techniques enabling conversational systems, in particular those based on large language models, information retrieval, recommender systems and knowledge graph, to be tailored to customer needs. You will handle Amazon-scale use cases with significant impact on our customers' experiences. Key job responsibilities - Use deep learning, ML and NLP techniques to create scalable solutions for creation and development of language model centric solutions for building personalized assistant systems based on a rich set of structured and unstructured contextual signals - Innovate new methods for contextual knowledge extraction and information retrieval, using language models in combination with other learning techniques, that allows effective grounding in context providers when considering memory, compute, latency and quality - Research in advanced customer understanding and behavior modeling techniques - Collaborate with cross-functional teams of engineers, product managers, and scientists to identify and solve complex problems in personal knowledge aggregation, processing, modeling, and verification - Design and execute experiments to evaluate the performance of state-of-the-art algorithms and models, and iterate quickly to improve results - Think Big about the arc of development of conversational assistant system personalization over a multi-year horizon, and identify new opportunities to apply these technologies to solve real-world problems - Communicate results and insights to both technical and non-technical audiences, including through presentations and written reports - Mentor and guide junior scientists and engineers, and contribute to the overall growth and development of the team About the team The AGI Personalization org uses various contextual signals to personalize Large Language Model output for our customers while maintaining privacy and security of customer data. We work across multiple Amazon products, including Alexa, to enhance the user experience by bringing more personal context and relevance to customer interactions.
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
US, NY, New York
Are you passionate about innovating and delivering shopping experiences that empower every Amazon customer based on their unique shopping intent? Are you looking for new challenges and solving hard problems while using top-notch LLM and GenAI techniques? Join us and you'll help millions of customers to make informed purchase decisions from a multi-billion product selection! Key job responsibilities You’ll be utilizing your Generative AI, time series and predictive modeling skills, and creative problem-solving skills to drive new projects from ideation to implementation. You will lead efforts in foundational models to develop new approaches to Personalization, and provide opportunities for scientists and engineers to invent and implement scalable ML recommendations supporting new Customer Experiences. Your science expertise will be leveraged to research and deliver often novel solutions to existing problems, explore emerging problems spaces, and create or organize knowledge around them. You’ll publish papers and file patents. And you’ll work closely with engineers to put your ideas into production. You will participate in the Amazon ML community by authoring scientific papers and submitting them to ML conferences. About the team The Discover Innovation team is dedicated to assisting customers in fulfilling their shopping objectives and empowering other Amazon teams by leveraging our deep understanding of customer intent. Our team’s mission is to delight customers by helping them discover the right content at the right time, tailoring the shopping experience to the customer’s intent and unique shopping objectives. We are a team of dedicated Scientists, Engineers, and Designers working together to deliver new, personalized shopping experiences. In tailoring the shopping experience to each customer’s intent, we envision operating seamlessly across Amazon as a talented personal shopping assistant — a partner that is trusted, knowledgeable, and understands your unique preferences.
US, CA, San Francisco
If you are interested in this position, please apply on Twitch's Career site https://www.twitch.tv/jobs/en/ About Us: Twitch is the world’s biggest live streaming service, with global communities built around gaming, entertainment, music, sports, cooking, and more. It is where thousands of communities come together for whatever, every day. We’re about community, inside and out. You’ll find coworkers who are eager to team up, collaborate, and smash (or elegantly solve) problems together. We’re on a quest to empower live communities, so if this sounds good to you, see what we’re up to on LinkedIn and Twitter, and discover the projects we’re solving on our Blog. Be sure to explore our Interviewing Guide to learn how to ace our interview process. About the Role: We are looking for an Applied Scientist to solve challenging and open-ended problems in the domain of recommendations, search, ranking and information retrieval. As an Applied Scientist on Twitch's Community team, you will use ML to help viewers find streamers and communities they’ll love. You will collaborate with a team of passionate scientists and engineers to develop these models and put them into production, where they can help Twitch's creators and viewers succeed and build communities. You will report to the Applied Science Manager on the Community Discovery Team. This position is located in San Francisco, CA. You Will: - Develop and Productionize ML algorithms for recommendations, ranking and search problems that can improve discovery on Twitch. - Collaborate with our Product and Engineering teams to work backwards from customer discovery problems, to determine the ML solution (algorithm and pipeline) to have the biggest impact on our user base in the real world. - Participate in the scientific community at Twitch, Amazon, and the broader ML and risk community. Perks - Medical, Dental, Vision & Disability Insurance - 401(k) - Maternity & Parental Leave - Flexible PTO - Amazon Employee Discount
US, CA, Sunnyvale
The Amazon Artificial General Intelligence (AGI) Personalization team is looking for a passionate, highly skilled and inventive Senior Applied Scientist with strong machine learning background to lead the development and implementation of state-of-the-art ML systems for personalizing large-scale, high-quality conversational assistant systems. As a Senior Applied Scientist, you will play a critical role in driving the development of personalization techniques enabling conversational systems, in particular those based on large language models, information retrieval, recommender systems and knowledge graph, to be tailored to customer needs. You will handle Amazon-scale use cases with significant impact on our customers' experiences. Key job responsibilities - Use deep learning, ML and NLP techniques to create scalable solutions for creation and development of language model centric solutions for building personalized assistant systems based on a rich set of structured and unstructured contextual signals - Innovate new methods for contextual knowledge extraction and information retrieval, using language models in combination with other learning techniques, that allows effective grounding in context providers when considering memory, compute, latency and quality - Research in advanced customer understanding and behavior modeling techniques - Collaborate with cross-functional teams of engineers, product managers, and scientists to identify and solve complex problems in personal knowledge aggregation, processing, modeling, and verification - Design and execute experiments to evaluate the performance of state-of-the-art algorithms and models, and iterate quickly to improve results - Think Big about the arc of development of conversational assistant system personalization over a multi-year horizon, and identify new opportunities to apply these technologies to solve real-world problems - Communicate results and insights to both technical and non-technical audiences, including through presentations and written reports - Mentor and guide junior scientists and engineers, and contribute to the overall growth and development of the team About the team The AGI Personalization org uses various contextual signals to personalize Large Language Model output for our customers while maintaining privacy and security of customer data. We work across multiple Amazon products, including Alexa, to enhance the user experience by bringing more personal context and relevance to customer interactions.
US, MA, North Reading
We are looking for experienced scientists and engineers to explore new ideas, invent new approaches, and develop new solutions in the areas of Controls, Dynamic modeling and System identification. Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Key job responsibilities Applied Scientists take on big unanswered questions and guide development team to state-of-the-art solutions. We want to hear from you if you have deep industry experience in the Mechatronics domain and : * the ability to think big and conceive of new ideas and novel solutions; * the insight to correctly identify those worth exploring; * the hands-on skills to quickly develop proofs-of-concept; * the rigor to conduct careful experimental evaluations; * the discipline to fast-fail when data refutes theory; * and the fortitude to continue exploring until your solution is found