Playlist search reinvented: LLMs behind the curtain

By Geetha Aluri, Siddharth Sharma, Tarun Sharma, Joaquin Delgado
2024
Download Copy BibTeX
Copy BibTeX
Improving search functionality poses challenges such as data scarcity for model training, metadata enrichment for comprehensive document indexing, and the labor-intensive manual annotation for evaluation. Traditionally, iterative methods relying on human annotators and customer feedback have been used. However, recent advancements in Large Language Models (LLMs) offer new solutions. This paper focuses on applying LLMs to playlist search. Leveraging LLMs’ contextual understanding and generative capabilities automates metadata enrichment, reducing manual efforts and expediting training. LLMs also address data scarcity by generating synthetic training data and serve as scalable judges for evaluation, enhancing search performance assessment. We demonstrate how these innovations enhance playlist search, overcoming traditional limitations to improve search result accuracy and relevance.

Latest news

US, CA, Pasadena
The Amazon Center for Quantum Computing in Pasadena, CA, is looking to hire a Fabrication R&D Scientist with experience in semiconductor process development who will aid in Amazon’s effort to bring cloud quantum computing services to its worldwide customer base. You will join a multi-disciplinary team of scientists, and hardware and software engineers working at the forefront of quantum computing. Through your work inside and outside of the cleanroom environment in the fabrication research and development group, you will solve problems related to developing next-generation quantum processors. Candidates must have a demonstrated background in sound scientific and engineering principles, and must have excellent data analysis, bias for action, problem solving, and communication skills, and be highly motivated and curious to research and learn new technical topics as needed. As a Fab R&D scientist you will be expected to work on new ideas and stay abreast of novel approaches in fabricating and packaging superconducting quantum processors. Working effectively within a team environment is critical. Diverse Experiences Amazon values diverse experiences. Even if you do not meet all the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Work/Life Balance Our team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Export Control Requirement Due to applicable export control laws and regulations, candidates must be either a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum, or be able to obtain a US export license. If you are unsure if you meet these requirements, please apply and Amazon will review your application for eligibility. Key job responsibilities Responsibilities include developing and optimizing processes to fabricate high-coherence superconducting qubits; developing advanced 3DI interconnect and routing technologies for integrating superconducting quantum technologies; analyzing inline metrology and electrical test data; developing and maintaining integration documentation, design rules, and standard operating procedures; interacting with project leads to provide feedback that continuously improves different processes; staying updated with the latest advancements and industry trends in process integration and apply knowledge to improve processes and drive innovation providing technical guidance and support to junior colleagues, fostering a collaborative and knowledge-sharing work environment. A day in the life The candidate will develop novel technologies using micro-/nano-fabrication techniques inside the cleanroom (independently or in collaboration with other scientists, engineers, and technicians) for next-generation quantum computing. Outside the cleanroom, the candidate will plan experiments, analyze data, and conceive future innovations.
US, CA, San Francisco
Amazon Industrial Robotics is on a mission to redefine the future of automation — and we're looking for exceptional talent to help lead the way. We are building the next generation of advanced robotic systems that seamlessly blend cutting-edge AI, sophisticated control systems, and novel mechanical design to create adaptable, intelligent automation solutions capable of operating safely alongside humans in dynamic, real-world environments. At Amazon Industrial Robotics, we leverage the power of machine learning, artificial intelligence, and advanced robotics to solve some of the most complex operational challenges at a scale unlike anywhere else in the world. Our fleet of robots spans hundreds of facilities globally, working in sophisticated coordination to deliver on our promise of customer excellence — and we're just getting started. As a Sr. Applied Scientist in Robot Perception, you will be at the forefront of this transformation. You will develop and deploy state-of-the-art perception algorithms that enable robots to truly understand and interact with the physical world — bridging the gap between theoretical research and realworld impact. Bringing deep expertise in Computer Vision and a nuanced understanding of the capabilities and limitations of modern Vision-Language Models (VLMs), you will innovate boldly and push the boundaries of what's possible. Our vision for the Perception layer is ambitious: to enable seamless, intelligent interaction between the user, the robot, and its environment. This is a rare opportunity to work at the intersection of deep learning, large language models, and robotics — contributing to research that doesn't just advance the field, but reshapes it. You will collaborate with world-class teams pioneering breakthroughs in dexterous manipulation, locomotion, and humanrobot interaction, all at an unprecedented scale. Key job responsibilities Design, develop, and deploy perception algorithms for robotics systems, including object detection, segmentation, tracking, depth estimation, and scene understanding • Lead research initiatives in computer vision, sensor fusion and 3D perception • Collaborate with cross-functional teams including robotics engineers, software engineers, and product managers to define and deliver perception capabilities • Drive end-to-end ownership of ML models — from data collection and labeling strategy to training, evaluation, and deployment • Mentor junior scientists and engineers; contribute to a culture of technical excellence • Define and track key metrics to measure perception system performance in real-world environments • Publish research findings in top-tier venues (CVPR, ICCV, ECCV, ICRA, NeurIPS, etc.) and contribute to patents A day in the life Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment • Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations • Actively contribute to brainstorming sessions on adjacent topics, bringing fresh perspectives that help peers grow and succeed — and in doing so, build lasting trust across the team • Mentor team members while maintaining significant hands-on contribution to technical solutions About the team Our Industrial Robotics Group is a diverse group of scientists and engineers passionate about building intelligent machines. We value curiosity, rigor, and a bias for action. We believe in learning from failure and iterating quickly toward solutions that matter.
IT, Turin
As a Senior Applied Scientist in the Alexa AI team, you will define and drive the science roadmap for state-of-the-art conversational AI systems powered by large language models, directly impacting how millions of customers interact with Alexa daily. You'll lead the design of LLM fine-tuning, alignment, and agentic architectures that operate reliably at scale, owning end-to-end delivery from research formulation through production deployment. Working at the intersection of research and production, you'll translate state of the art advances into customer-facing features. Your work will span the full ML lifecycle: developing novel evaluation frameworks, building automated training pipelines, and conducting rigorous experimentation across diverse devices and endpoints. Collaborating with engineering, product, and cross-functional science teams across Amazon, you'll tackle the team's most complex technical challenges while maintaining practical focus on customer value. This role offers the opportunity to publish at top-tier conferences, generate intellectual property, and see your innovations scale to one of the world's most popular voice assistants. Key job responsibilities As a Senior Applied Scientist in the Alexa AI team: - Define and drive the science roadmap for conversational AI capabilities powered by large language models - Design, implement, and evaluate novel approaches to LLM fine-tuning, alignment (RLHF, DPO), and distillation for production deployment - Architect agentic systems (multi-step reasoning, tool use, planning, and orchestration) that work reliably at scale - Develop evaluation frameworks and methodologies that go beyond standard benchmarks to capture real-world conversational quality - Translate research advances into customer-facing products, working closely with engineering, product, and cross-functional science teams - Own end-to-end delivery of complex, ambiguous research initiatives from problem formulation through experimentation to production deployment, with minimal guidance - Tackle the team's most complex technical problems while maintaining practical focus on customer value and solution generalizability - Advance the team's scientific reputation through high-impact publications and presentations at top-tier internal and external venues, and generate intellectual property through patents The applicable collective agreement for this role is CBA for employees of Telecommunication Sector. The position is classified at level 6 or above, depending on the candidate’s skills, competences and experience. The minimum gross annual base salary for this position is listed below. The base salary listed corresponds to working on a full-time basis. For part-time hours, the salary will be pro-rated. Amazon reserves the right to offer a higher salary and/or level, depending on the candidate's skills, competencies, and experience. Amazon's package may include a sign on payment. In addition, the candidate may be eligible to participate in a restricted stock unit scheme operated independently by Amazon.com Inc. in USA. Your recruiting team will share final salary and any restricted stock unit scheme if applicable, depending on skills and requirements. In addition to statutory benefits, and those applicable to the relevant CBA, company supplementary benefits may apply subject to further terms. Italy- EUR104,500 gross annually. A day in the life As a Senior Applied Scientist in the Alexa AI team, your day will involve leading cross-functional collaborations with engineering, product, and science teams to define the technical direction for our conversational assistant. You'll design experiments that shape the science roadmap, mentor junior scientists, and make high-judgment calls on architecture and deployment trade-offs. Working in a fast-paced, ambiguous environment, you'll own end-to-end delivery of complex initiatives: from formulating novel research problems to presenting strategic recommendations to senior leadership. Your ability to influence across organizational boundaries will drive measurable customer impact while raising the bar for millions of customers. About the team Alexa AI is building the science and technology behind Alexa+, Amazon's next-generation conversational assistant. Our team works at the intersection of large language models, reinforcement learning from human feedback and verifiable rewards, agentic architectures, and multilingual/multimodal understanding. We operate at massive scale: our models serve customers across dozens of languages and device types. If you want to push the frontier of conversational AI and see your work used by people every day, come join us.
US, WA, Bellevue
The Supply Chain Optimization Technologies (SCOT) team builds technology to automate and optimize Amazon’s supply chain of physical goods. We seek a Data Scientist with strong analytical and communication skills to join our team. SCOT manages Amazon's inventory under uncertainty of demand, pricing, promotions, supply, vendor lead times, and product life cycle. We optimize complex trade-offs between customer experience, inventory costs, fulfillment costs, fulfillment center capacity, etc. We develop sophisticated algorithms that involve learning from large amounts of data such as prices, promotions, similar products, and other data from our product catalog in order to automatically act on millions of dollars’ worth of inventory weekly and establish plans for tens of thousands of employees. As a Data Scientist, you will contribute to the research community, by working with other scientists across Amazon and our Supply Chain, as well as collaborating with academic researchers and publishing papers both internally and externally. Key job responsibilities Major responsibilities include: - Analysis of large amounts of data from different parts of the supply chain and their associated business functions - Improving upon existing machine learning methodologies by developing new data sources, developing and testing model enhancements, running computational experiments, and fine-tuning model parameters for new models - Formalizing assumptions about how models are expected to behave, creating definitions of outliers, developing methods to systematically identify these outliers, and explaining why they are reasonable or identifying fixes for them - Communicating verbally and in writing to business customers with various levels of technical knowledge, educating them about our research, as well as sharing insights and recommendations - Utilizing code (Python, R, Scala, etc.) for analyzing data and building statistical and machine learning models and algorithms A day in the life As a Data Scientist in SCOT, you will be tasked to understand and work with innovative research tools to enable the implementation of sophisticated models on big data. As a successful data scientist in the SCOT team, you are an analytical problem solver who enjoys diving into data from various businesses, is excited about investigations and algorithms, can multi-task, and can credibly interface between scientists, engineers and business stakeholders. Your expertise in synthesizing and communicating insights and recommendations to audiences of varying levels of technical sophistication will enable you to answer specific business questions and innovate for the future. Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: - Medical, Dental, and Vision Coverage - Maternity and Parental Leave Options - Paid Time Off (PTO) - 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply!
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the next-level. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Key job responsibilities * Develop, deploy, and operate scalable bioinformatics analysis workflows on AWS * Evaluate and incorporate novel bioinformatic approaches to solve critical business problems * Originate and lead the development of new data collection workflows with cross-functional partners * Partner with laboratory science teams on design and analysis of experiments About the team Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
US, CA, San Jose
Are you excited about using econometrics to make multi-million dollar decisions more Science and Data Driven? Are you interested in supporting Consumer Hardware device concepts from innovative idea inception to launch? Do you want to work on a Economics and Data Science team focused on tackling some of the hardest business questions within the Devices business at Amazon and then scaling those Statistics and Econometrics solutions via internal to Amazon tools? Then this could be the role for you! The Decision Science team owns demand estimates and pricing recommendations of concept devices before customers know they exist. We support analyses on hardware and services ranging from Echo Frames to Kindle Paperwhite to Blink Video Camera subscriptions to the Amazon Smart Plug - all prior to launch. In this role, you will develop science for high visible senior leadership decisions on new devices and services and work with a cross-functional team to apply and scale innovative science broadly. Key job responsibilities - Design, estimate, and scale Berry-Levinsohn-Pakes (BLP) random coefficients demand models to quantify consumer heterogeneity, own- and cross-price elasticities, and substitution patterns across large product markets. - Implement and optimize numerical routines—including GMM estimation, contraction mappings, and simulation-based inversion—to solve structural demand systems at scale in Python. - Develop and validate instrumental variables strategies to address price endogeneity in differentiated product markets, ensuring unbiased and robust demand parameter estimates. - Build production-grade pipelines that ingest large-scale observational datasets, estimate consumer preferences, and generate product-level demand forecasts on recurring schedules. - Collaborate with cross-functional teams including product management, marketing, and operations to translate structural model outputs—such as willingness-to-pay and competitive diversion ratios—into actionable pricing and portfolio strategies. - Advance the team's structural modeling capabilities by researching and deploying extensions to classical BLP frameworks (e.g., supply-side estimation, dynamic demand, micro-moments) and documenting approaches in clear technical reports.
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Our products are used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses a combination of econometrics, machine learning, and data science to build disruptive products for all our Advertising products. We also generate insights to guide Amazon Advertising strategy, providing direct support to senior leadership. We are looking for an experienced Economist with a deep passion for building econometric solutions and the ability to communicate data insights and scientific vision to execute on strategic projects. Key job responsibilities - Leverage econometrics and ML models to optimize advertising strategies on behalf of our customers. - Influence key business and product decisions based on insights from models you develop. - Perform hands-on analysis and modeling with enormous data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. - Work closely with software engineers on detailed requirements to productionize the models you build. - Run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Work with other scientists, software developers, and product partners to implement your solutions.
US, WA, Seattle
Amazon's Stores-Ads Science team operates at the intersection of Amazon's Stores and advertising businesses. We develop causal measurement systems, optimization algorithms, and machine learning models that inform how advertising affects shopper engagement, driving selling partner growth and marketplace economics. Our science shapes decisions both at the strategic level and in production systems. We are a team of interdisciplinary scientists who combine causal inference, economic modeling, and machine learning to drive measurable business impact. We are looking for an Applied Science Manager to lead our Ads Impact initiative. This team owns the science of understanding and optimizing how advertising creates value for shoppers and selling partners. What makes this role distinctive is its position at the frontier of AI and Economics: as Amazon's shopping experience evolves from traditional search toward LLM-powered, agentic commerce, the fundamental mechanisms through which advertising creates value are changing. This role will partner with leading scientists and academic researchers to measure these effects through large-scale causal experimentation, and develop novel methods to encode causal and economic reasoning into AI systems that optimize the shopping experience. Key job responsibilities In this role, you will lead a team of scientists, setting the technical vision and science roadmap for ads impact measurement and optimization. You will design experiments that identify the causal mechanisms through which advertising drives shopper engagement, advertiser value, and marketplace outcomes. You will develop optimization algorithms that integrate these causal signals into production and business decision-making, in close partnership with engineering and product teams across the organization. You will lead the research and communicate findings and recommendations to senior leadership through written narratives that connect technical science to business strategy. This role requires deep expertise in causal inference and experimental design, combined with strong applied ML skills and the engineering judgment to translate research into production systems. You will hire and develop future science leaders, think strategically, set ambitious roadmaps in highly ambiguous problem spaces, and foster a culture that values both intellectual depth and production impact. You will work cross-functionally, influencing across organizational boundaries to drive alignment on complex, multi-sided tradeoffs.
US, WA, Seattle
RISC's vision is to make Amazon Earth’s most trusted shopping destination for safe and compliant products. We do this by protecting customers from products that are unsafe, illegal, illegally marketed, controversial or otherwise in violation of Amazon's policies while enabling our Selling Partners (SPs) to offer their broadest selection of safe and compliant products. We are seeking an exceptional Applied Scientist to join a team of experts in the field of agentic AI, GenAI, Machine Learning, Software Engineers, and work together to tackle challenging problems across diverse compliance domains. We leverage and train state-of-the-art large-language-models (LLMs), multi-modal model, mixed with elegant harness engineering and SKILL building to 1) detect illegal and unsafe products across the Amazon catalog; 2) automation safety and compliance content authoring; 3) reasoning over enforcement action to provide actionable insights to Amazon sellers. We work on machine learning problems for content generation, multi-modal classification, global product taxonomy, intent detection, information retrieval, anomaly and fraud detection, agentic AI, generative AI and multi-agent system. This is an exciting and challenging position to deliver scientific innovations into production systems at Amazon-scale to make immediate, meaningful customer impacts while also pursuing ambitious, long-term research. You will work in a highly collaborative environment where you can analyze and process large amounts of image, text, unstructured and tabular data. You will work on challenging science problems that have not been solved before, conduct rapid prototyping to validate your hypothesis, and deploy your algorithmic ideas at scale. There will be something new to learn every day as we work in an environment with rapidly evolving regulations and adversarial actors looking to outwit your best ideas. Key job responsibilities • Design and evaluate state-of-the-art algorithms and approaches in content generation, multi-modal classification, global product taxonomy, intent detection, information retrieval, anomaly and fraud detection, agentic AI, generative AI and multi-agent system. • Translate product and CX requirements into measurable science problems and metrics. • Collaborate with product and tech partners and customers to validate hypothesis, drive adoption, and increase business impact • Key author in writing high quality scientific papers in internal and external peer-reviewed conferences. A day in the life • Understanding customer problems, project timelines, and team/project mechanisms • Proposing science formulations and brainstorming ideas with team to solve business problems • Writing code, and running experiments with re-usable science libraries • Reviewing labels and audit results with investigators and operations associates • Sharing science results with science, product and tech partners and customers • Writing science papers for submission to peer-review venues, and reviewing science papers from other scientists in the team. • Contributing to team retrospectives for continuous improvements • Driving science research collaborations and attending study groups with scientists across Amazon
US, NY, New York
About Sponsored Products and Brands: The Sponsored Products and Brands (SPB) organization at Amazon Ads is re-imagining the advertising landscape through industry leading generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. About Our Team: The Brand Beacon team is responsible for inventing impressions offerings for brands to increase share of voice via premium experiences, helping brands get discovered, acquire new customers and sustainably grow customer lifetime value. We build end-to-end solutions that enable brands to drive discovery, visibility and share of voice. This includes building advertiser controls, shopper experiences, monetization strategies and optimization features. We succeed when (1) shoppers discover, engage and build affinity with brands and (2) brands can grow their business at scale with our advertising products. About This Role: As a Senior Scientist for the team, you will have the opportunity to apply your deep subject matter expertise in the area of ML, LLM and GenAI models. You will invent new product experiences that enable novel advertiser and shopper experiences. This role will liaise with internal Amazon partners and work on bringing state-of-the-art GenAI models to production, and stay abreast of the latest developments in the space of GenAI and identify opportunities to improve the efficiency and productivity of the team. Additionally, you will define a long-term science vision for our advertising business, driven by our customer’s needs, and translate it into actionable plans for our team of applied scientists and engineers. This role will play a critical role in elevating the team’s scientific and technical rigor, identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. You will communicate learnings to leadership and mentor and grow Applied AI talent across org. * Develop AI solutions for advertiser and shopper experiences. Build monetization and optimization systems that leverage generative models to value and improve campaign performance. * Define a long-term science vision and roadmap for our advertising business, driven from our customers' needs, translating that direction into specific plans for applied scientists and engineering teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. * Design and conduct A/B experiments to evaluate proposed solutions based on in-depth data analyses. * Effectively communicate technical and non-technical ideas with teammates and stakeholders. * Stay up-to-date with advancements and the latest modeling techniques in the field. * Think big about the arc of development of Gen AI over a multi-year horizon and identify new opportunities to apply these technologies to solve real-world problems. #GenAI