Improving complementary-product recommendations

New modeling approach increases accuracy of recommendations by an average of 7%.

One way that e-commerce sites make life easier for customers is by recommending products that complement whatever the customer is looking for: someone buying a tennis racket, for instance, may also want to buy tennis balls; someone buying a camera may want an SD card for extra storage.

At this year’s Conference on Information and Knowledge Management, my colleagues at the University of California, Los Angeles, and Amazon and I will present a new deep-learning-based method for doing complementary-product recommendation (CPR) that, in our tests, was 7% more likely to find a product that the customer wanted to buy than existing methods. 

That improvement comes from three main strategies: better selection of training data for the CPR model; greater diversity in the types of products recommended; and respect for the asymmetry of the CPR problem (while an SD card may a be a good product to complement a camera, a camera is not a good product to complement an SD card).

Our approach also addresses the problem of cold start, or predicting complementary products for items that were added to the product catalogue after the machine learning model was trained. To do that, we use an embedding scheme developed at Amazon, called Product2vec, to represent the inputs to the CPR model — the products we seek to complement — according to their attributes and their relationships with other products, rather than simply using their names or ID numbers.

Implicit signals

For training data, our model, like most other CPR models, relies on implicit signals from customers. We consider three ways that product x might be related to product y: co-purchase, meaning customers who purchased 𝑥 also purchased y; co-view, meaning customers who viewed x also viewed y; and purchase after view, meaning customers who viewed x eventually bought y.

CPR models typically use co-views and purchase after view as an indication of similarity and co-purchase as an indication of complementarity. But there is considerable overlap between these three categories.

Our intuition was that training a CPR model on product pairs that show up in the co-purchase data but not in the co-view and purchase-after-view data would lead to better predictions. 

User studies in which participants rated pairs of products as substitutable, complementary, or irrelevant bore out this intuition: the complementarity ratings of co-purchase-only product pairs were 30% higher than those of co-purchase product pairs that also showed up in the co-view and purchase-after-view data. Accordingly, we used co-purchase-only product pairs to train our model.

The inputs to our model are Product2vec embedding vectors. Embeddings represent data items as points in a multidimensional space, such that proximity in the space indicates some relationship between the items. In our case, that relationship is similarity: points representing different brands of tennis rackets should cluster together in the space, as should points representing cameras, and so on.

Product2vec differs from other embedding schemes in that its inputs are graphs, data structures consisting of nodes (in our case, the nodes contain product information) and edges connecting the nodes (in our case, the edges represent relationships such as co-purchases and co-views).

Graphical representation of relationships between products.
In our graphical representation of relationships between products, each node includes information such as a product’s category, type, and image, and edges represent relationships such as the co-viewing and co-purchase of products.

In the same way that we train our CPR model on co-purchase-only data, we train Product2vec on pairs of products that show up in the co-view and purchase-after-view data but not in the co-purchase data. The idea is that customers might view variations of the same product before selecting one for purchase, but co-purchased products are likely to be complementary rather than similar.

Product2vec embedding helps solve the cold-start problem, as it will produce a meaningful embedding even for products it hasn’t seen before.

Diversification

CPR models are typically trained to output the most frequent co-purchases for each input product. But this can lead to homogeneity of outputs: the top three co-purchases for a tennis racket, for instance, might be three different brands of tennis balls. We believe that customers would prefer more-diverse complementary-product recommendations: for instance, the top three recommendations for a tennis racket should be something like a can of tennis balls, a pack of overgrips, and a headband.

We enforce diversity through our model architecture. For every input product, we pass its product-type embedding through a neural network (the type transition network) that outputs the embeddings of complementary product types. Each of those embeddings is then concatenated with the embedding of the input product before passing to the module that generates the recommendations (the type-item prediction module).

Diagram of the CPR model architecture.
The architecture of our model. For each input, the type transition module outputs a set of vectors representing complementary product types. These are combined with the representation of the input product before it passes to the type-item prediction module, to ensure diversity in the model’s outputs.

The whole model is trained end to end: that is, during training, the type transition network is evaluated solely according to the accuracy of the type-item prediction module’s outputs. But each output of the type transition network is associated with a single output of the type-item prediction module, which naturally leads to greater type diversity among recommendations.

The addition of the type transition network also breaks the symmetry between related products that can cause problems for the typical CPR system. The typical system bases its judgments of complementarity on proximity in the embedding space. But in that space, an SD card is as close to a camera as a camera is to an SD card.

The type transition network, however, learns to output different product-type embeddings for cameras and SD cards, which enables our model to better respond to other, asymmetric signals in the data.

In experiments, we used co-purchase data to compare our model’s performance to that of three leading CPR systems. We scored the models’ recommendations according to the frequency with which their recommended products were co-purchased with the input product.

On two different data sets — electronics and grocery — and three different accuracy measures — the accuracy of the top recommendation, the top three recommendations, and the top ten recommendations — our model outperformed the others across the board.

Related content

IL, Tel Aviv
Are you a MS or PhD student interested in a 2024 Research Science Internship, where you would be using your experience to initiate the design, development, execution and implementation of scientific research projects? If so, we want to hear from you! Is your research in machine learning, deep learning, automated reasoning, speech, robotics, computer vision, optimization, or quantum computing? If so, we want to hear from you! We are looking for motivated students with research interests in a variety of science domains to build state-of-the-art solutions for never before solved problems You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science Key job responsibilities As a Research Science Intern, you will have following key job responsibilities; • Work closely with scientists and engineering teams (position-dependent) • Work on an interdisciplinary team on customer-obsessed research • Design new algorithms, models, or other technical solutions • Experience Amazon's customer-focused culture A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships and up to 12 months for part time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
US, VA, Arlington
Amazon Web Services (AWS) is seeking a highly skilled Economist to help shape the future of our company and enhance the success of our customers. With AWS generating approximately $100B in annual revenue, we are expanding rapidly and need to identify the interventions that are most effective in helping both existing and potential customers throughout their cloud- adoption journey. As part of this role, you will apply advanced econometrics and machine learning techniques to determine which interventions yield the best outcomes across different stages of the customer journey, from early engagement to mature customer relationships. Your work will center on applying causal inference and machine learning to large, complex datasets, uncovering actionable insights that directly influence AWS's strategic decisions. You will be instrumental in developing scalable models that deepen our understanding of customer behavior and quantify the impact of marketing and sales initiatives. By working closely with key business stakeholders, you’ll ensure that AWS consistently delivers the most effective solutions tailored to the unique needs of our diverse and growing customer base. Key job responsibilities Key job responsibilities -Apply your expertise in econometrics and machine learning to evaluate the effectiveness of AWS interventions and customer engagement strategies. -Identify patterns and opportunities in customer data to suggest new interventions, such as credit offers, discounts, and service recommendations. -Formalize and document research processes, ensuring scientific rigor and knowledge sharing within Amazon’s science community. -Communicate insights and findings effectively to business leaders across various levels of the organization, influencing strategic decision-making.
US, MA, Westborough
Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart team of doers that work passionately to apply cutting edge advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even imagine yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Amazon Robotics is seeking Applied Science Interns and Co-ops with a passion for robotic research to work on cutting edge algorithms for robotics. Our team works on challenging and high-impact projects within robotics. Examples of projects include allocating resources to complete a million orders a day, coordinating the motion of thousands of robots, autonomous navigation in warehouses, identifying objects and damage, and learning how to grasp all the products Amazon sells. As an Applied Science Intern/Co-op at Amazon Robotics, you will be working on one or more of our robotic technologies such as autonomous mobile robots, robot manipulators, and computer vision identification technologies. The intern/co-op project(s) and the internship/co-op location are determined by the team the student will be working on. Please note that by applying to this role you would be considered for Applied Scientist summer intern, spring co-op, and fall co-op roles on various Amazon Robotics teams. These teams work on robotics research within areas such as computer vision, machine learning, robotic manipulation, navigation, path planning, perception, optimization and more. Learn more about Amazon Robotics: https://amazon.jobs/en/teams/amazon-robotics
LU, Luxembourg
Are you a MS or PhD student interested in a 2025 Internship in the field of machine learning, deep learning, speech, robotics, computer vision, optimization, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
LU, Luxembourg
At Global Mile Expansion team, our vision is to become the carrier of choice for all of our Selling Partners cross-border shipping needs, offering complete set of end to end cross border solutions from key manufacturing hubs to footprint countries supporting business who use Amazon to grow their business globally. As we expand, the need for comprehensive business insight and robust demand forecasting to aid decision making on asset utilization especially where we know demand will be variable becomes vital, as well as operational excellence. We are building business models involving large amounts of data and Macro economic inputs to produce the robust forecast to help the operational excellence and continue improving the customer experience. We are looking for an experienced economist who can apply innovative modelling techniques to real-world problems, and convert it to highly business-impacting solutions. Key job responsibilities - Experienced in using mathematical and statistical approach to create new, scalable solutions for business problems - Analyze and extract relevant information from business data to help automate and optimize key processes - Design, develop and evaluate highly innovative models for predictive learning - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Research and implement statistical approaches to understand the business long-term and short-term trend and support the strategies
ES, Madrid
Are you a MS or PhD student interested in a 2025 Internship in the field of machine learning, deep learning, speech, robotics, computer vision, optimization, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact, visionary person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Luxembourg, Netherlands, Poland, Romania, Spain, UAE, and UK). Please note these are not remote internships.
US, WA, Seattle
When customers search for products on the Amazon website, they often see brand advertisements displayed right below the search bar. These ads are part of the Sponsored Brands (SB) program. Our team, the SB Search and Relevance team, works on solving challenges to retrieve the most relevant ads for a customer's search query. A customer's search query is typically a short, free-form text consisting of just a few words. Our algorithm needs to understand the customer's underlying intention from this limited information. At the same time, each advertisement consists of various elements like text descriptions, images, videos, and more. Our algorithm also needs to comprehend the content of these ads and identify the most relevant one from the large pool of ad candidates. As Amazon's advertising business is growing rapidly, we are looking for experienced applied scientists. As an Applied Scientist on this team, you will: - Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale, complexity. - Apply deep learning and natural language processing to improve information retrieval and relevance. - Design and run A/B experiments. Evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Optimize deep learning inference latency by utilizing methods like knowledge distillation. - Work with software development engineers and write code to bring models into production. - Recruit Applied Scientists to the team and provide mentorship. Impact and Career Growth - You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! - Define a long-term science vision for our advertising business, driven fundamentally from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding.
US, MA, Boston
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? Machine learning (ML) has been strategic to Amazon from the early years. We are pioneers in areas such as recommendation engines, product search, eCommerce fraud detection, and large-scale optimization of fulfillment center operations. The AWS Industries Team at AWS helps AWS customers implement Generative AI solutions and realize transformational business opportunities for AWS customers in the most strategic industry verticals. This is a team of data scientists, engineers, and architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and build applications to launch these solutions at scale. The AWS Industries team provides guidance and implements best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for Applied Scientists capable of using GenAI and other techniques to design, evangelize, and implement state-of-the-art solutions for never-before-solved problems. Key job responsibilities As an Applied Scientist, you will- - Collaborate with AI/ML scientists, engineers, and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms and build ML systems to address real-world challenges - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production - Create and deliver best practice recommendations, tutorials, blog posts, publications, sample code, and presentations adapted to technical, business, and executive stakeholder. Publish novel developments in internal and external papers, forums, and conferences - Provide customer and market feedback to Product and Engineering teams to help define product direction About the team ABOUT AWS: Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, CA, San Diego
Are you looking to work at the forefront of Machine Learning and AI? Would you be excited to apply cutting edge Generative AI algorithms to solve real world problems with significant impact? The AWS Industries Team at AWS helps AWS customers implement Generative AI solutions and realize transformational business opportunities for AWS customers in the most strategic industry verticals. This is a team of data scientists, engineers, and architects working step-by-step with customers to build bespoke solutions that harness the power of generative AI. The team helps customers imagine and scope the use cases that will create the greatest value for their businesses, select and train and fine tune the right models, define paths to navigate technical or business challenges, develop proof-of-concepts, and build applications to launch these solutions at scale. The AWS Industries team provides guidance and implements best practices for applying generative AI responsibly and cost efficiently. You will work directly with customers and innovate in a fast-paced organization that contributes to game-changing projects and technologies. You will design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. In this Data Scientist role you will be capable of using GenAI and other techniques to design, evangelize, and implement and scale cutting-edge solutions for never-before-solved problems. Key job responsibilities As a Senior Data Scientist, you will- - Collaborate with AI/ML scientists, engineers, and architects to research, design, develop, and evaluate cutting-edge generative AI algorithms and build ML systems to address real-world challenges - Interact with customers directly to understand the business problem, help and aid them in implementation of generative AI solutions, deliver briefing and deep dive sessions to customers and guide customer on adoption patterns and paths to production - Create and deliver best practice recommendations, tutorials, blog posts, publications, sample code, and presentations adapted to technical, business, and executive stakeholder - Provide customer and market feedback to Product and Engineering teams to help define product direction About the team ABOUT AWS: Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship and Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, WA, Bellevue
Amazon Last Mile builds global solutions that enable Amazon to attract an elastic supply of drivers, companies, and assets needed to deliver Amazon's and other shippers' volumes at the lowest cost and with the best customer delivery experience. Last Mile Science team owns the core decision models in the space of jurisdiction planning, delivery channel and modes network design, capacity planning for on the road and at delivery stations, routing inputs estimation and optimization. We also own scalable solutions to reduce risks, improve safety, enhance personalized experiences of our delivery associates and partners. Our research has direct impact on customer experience, driver and station associate experience, Delivery Service Partner (DSP)’s success and the sustainable growth of Amazon. We are looking for a passionate individual with strong machine learning and analytical skills to join its Last Mile Science team in the endeavor of designing and improving the most complex planning of delivery network in the world. As a Senior Data Scientist, you will work with software engineers, product managers, and business teams to understand the business problems and requirements, distill that understanding to crisply define the problem, and design and develop innovative solutions to address them. Our team is highly cross-functional and employs a wide array of scientific tools and techniques to solve key challenges, including supervised and unsupervised machine learning, non-convex optimization, causal inference, natural language processing, linear programming, reinforcement learning, and other forecast algorithms. Key job responsibilities Key job responsibilities * Drive end-to-end Machine Learning projects that have a high degree of ambiguity, scale and complexity. * Build Machine Learning models, perform proof-of-concept, experiment, optimize, and deploy your models into production; work closely with software engineers to assist in productionizing your ML models. * Run A/B experiments, gather data, and perform statistical analysis. * Measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs. * Research new and innovative machine learning approaches. Help coach/mentor junior scientists in the team. * Willingness to publish research at internal and external top scientific venues. Write and pursue IP submissions.