Distributed “Re-Ranker” ensures that Alexa improvements reach customers ASAP

Suppose that you say to Alexa, “Alexa, play Mary Poppins.” Alexa must decide whether you mean the book, the video, or the soundtrack. How should she do it?

Each of Alexa’s core domains — such as the Books domain, the Video domain, or the Music domain — has its own natural-language-understanding (NLU) model, which estimates the likelihood that a given request is intended for it. But those models are trained on different data, so there’s no guarantee that their probability estimates are compatible. Should a 70% estimate from Music be given priority over a 68% estimate from Books, or is it possible that, when it comes to Mary Poppins, the Music model is slightly overconfident?

The standard way to address this problem is to train a machine learning system to take probability estimates from multiple domain models and weight them so that they’re more directly comparable. But in a paper we’re presenting at this year’s IEEE Spoken Language Technologies conference, my colleagues and I present an alternative: a method that lets each domain develop its own weighting procedure, independent of the others.

The advantage of this approach is that each domain can update its own weighting system whenever required, and multiple domains can perform updates in parallel, which is more efficient. We are already using this method in production, to ensure that Alexa customers benefit from system updates as quickly as possible. We call it re-ranking, because it takes a list of domain-specific hypotheses ranked according to an NLU model’s confidence scores and re-ranks them according to a learned set of weights.

Those weights don’t just apply to the domain classification probability. Alexa’s NLU models also classify utterances by intent, or the action the customer wants performed, and slot, or the data item the intent is supposed to act upon. And like the domain classifications, intent and slot classifications are probabilistic.

Within the Music domain, for instance, the utterance “play Thriller” would probably call up the PlayMusic intent (as opposed to, say, the CreateList intent), but the slot classifier might assign similar probabilities to the classifications of the word “Thriller” as AlbumName and SongName.

During training, we feed one of our domain-specific re-rankers not just the domain classification probability for each utterance but the most probable intent and slot hypotheses, too. It could be, for instance, that for purposes of re-ranking, intent confidence is more important than domain confidence, or vice versa.

The system learns these relative values during training, and it produces separate weights for each classification category — domain, intent, and slot. It can also factor contextual information into its re-rankings, learning, for instance, that on a Fire TV, it should give more precedence to Video domain hypotheses than it should on a voice-only device.

In experiments, we trained our re-ranker using several evaluation criteria, or “loss functions.” The first loss function relies on a metric we call semantic error, which counts the number of errors in a given classification hypothesis and divides it by the total number of intents and slots in the correct classification. If an utterance in the training set has one intent and two slots, for instance, then a hypothesis that gets the intent and one slot right but misses the other slot would have a semantic-error score of 1/3.

Semantic-error-rate_example.png._CB461160527_.png
Examples of semantic-error (SemER) scores for several Books domain hypotheses about the classification of the utterance “play Moana”.

Our first loss function is called expected semantic error. It imposes a penalty on rankings in which hypotheses with high semantic error rank ahead of hypotheses with low semantic error.

The second loss function is a so-called cross-entropy function. It eschews the comparatively fine-grained semantic-error measure in favor of a simple one-bit tag that indicates whether a given hypothesis is true or not. But it imposes a particularly stiff penalty on incorrect hypotheses that get high confidence scores, and it confers a large bonus on true hypotheses that get high confidence scores.

This is the loss function we introduce to calibrate scores across domains. It inclines toward weights that correct for overconfidence, but by rewarding the ability to distinguish true hypotheses so lavishly, it also prevents overcompensation.

Our third loss function simply combines the first two — the expected-semantic-error function and the cross-entropy function. We also compare systems trained on all three loss functions to one that uses no domain-specific weighting at all.

The graph below summarizes our results. The dotted line represents perfect calibration of confidence scores across all domains. The better our re-rankers approximate that line, the better calibration they afford.

Our first observation is that the unweighted baseline system is already fairly well calibrated, a tribute to the quality of the domain-specific NLU models. Our second is that, while the combination of expected semantic error and cross-entropy (R3) does indeed yield the best calibration, it’s only slightly better than cross-entropy alone (R2), which is significantly better than expected semantic error alone (R1). In production we use the combined loss function to ensure the best possible experience for our customers.

Calibration_plots.jpg._CB461173753_.jpg

Acknowledgments: Rahul Gupta, Shankar Ananthakrishnan, Spyros Matsoukas

Related content

US, WA, Seattle
Are you excited to play a strategic role in defining the customer experience on Tablets? Come join a dynamic, customer-obsessed, world-class science team to completely reshape the customer experience on our tablets. Key job responsibilities Using Amazon’s large-scale computing resources, you will design and deploy state-of-the-art recommendation models. You will ask research questions about customer behavior, design state-of-the-art models that help customers discover content, and deploy these models to production alongside other engineers. You will participate in the Amazon ML community and mentor other engineers with a strong interest in and knowledge of ML and generative AI. Your work will directly benefit customers! A day in the life We are looking for a passionate, hard-working, and talented Applied Scientist who has experience developing state-of-the-art models and deploying them to production. You will have an opportunity to make an enormous impact on the design, architecture, and implementation of products used everyday by real customers! About the team The Alexa Tablet team is a world-class team that delivers high-quality, innovative products that are loved by our customers. Our team focuses on spinning many of Amazon’s flywheels with bar-raising mobile experiences on our tablets. Over the last 12 years, we have built 4.0+ star rated products, and we continue to look for ways to scale our business. Join us as we continue our ambitious journey which will give you the opportunity to deliver disruptive products with global impact.
IN, KA, Bengaluru
Amazon.com’s Product Detail Page team is looking for talented, motivated and passionate applied scientist to be part of the design and development of a highly scalable multi-tiered shopping application to provide the best possible online shopping experience for Amazon customers world-wide. Our team is comprised of talented applied scientists, developers, testers, program managers, designers and product managers tasked with the singular goal to create THE world's best buying experience. Scientists on this team develop the next-generation technologies and experiences that change how millions interact and shop online. To provide the best possible online shopping at the scale of the web requires ideas from every area of computer science, including distributed computing, large-scale system design, machine learning, natural language processing, data compression and user interface design; the list goes on and is growing every day. We need our scientists to be versatile and always eager to tackle new problems as we continue to push technology forward. Our team leverages sophisticated econometric, machine learning, and big data technologies to help customers to discover the right products at the right prices from millions of trusted sellers billions of times a day. If you are looking for a career-defining opportunity on one of the most customer centric and business impacting teams within Amazon, we’d love to hear from you. We are looking for an Applied Scientist to help build the next generation of Detail Page optimization algorithms. These new set of algorithms will incorporate the continually changing preferences of our customers and continue to scale with numerous new programs that Amazon is introducing for our customers. You will work with multiple Amazon businesses and programs to identify big business opportunities and propose new business features and technical systems to improve customer experience on Amazon Detail Page, Search Page and many other widgets throughout the website. You will be responsible for the quality of algorithm design and will get the opportunity to present your ideas and share results of your deliverables with Amazon executives on a frequent basis. You will get an opportunity to work with senior scientists to define and enforce broad, company-wide technical standards in optimization techniques, statistical modeling and simulation techniques, and/or data analytics.
US, WA, Seattle
Applied Scientist II (AMZ1513176) (Multiple Positions), Amazon Data Services, Inc., Seattle, WA. Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists. (AMZ1513176) 40 hrs/week, Mon-Fri, 8:00 a.m. - 5:00 p.m. Salary Range: $163,617 - $193,200/yr. Amazon.com is an Equal Opportunity – Affirmative Action Employer – Minority/Female/Disability/Veteran/Gender Identity/Sexual Orientation MINIMUM REQUIREMENTS: Master's degree or foreign equivalent in Computer Science, Machine Learning, Statistics, or a related field and (1) one year of research or work experience in the job offered, or as a Research Scientist, Research Assistant, Software Engineer, or a related occupation. Employer will accept a Bachelor's degree or foreign equivalent in Computer Science, Machine Learning, Statistics, or a related field and (5) five years of research or work experience in a related occupation as equivalent to the Master's degree and one year of research or work experience. Must have (1) one year of research or work experience in the following skill(s): programming in Java, C++, Python, or equivalent programming language; and conducting the analysis and development of various supervised and unsupervised machine learning models for moderately complex projects in business, science, or engineering. Employer will accept any suitable combination of education, training or experience. Multiple Positions Available. Amazon is a total compensation company. Depending on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits.#0000
US, WA, Seattle
nnovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the extreme. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.
US, WA, Bellevue
Are you curious about turning complex data into actionable insights that shape how millions of customers experience Amazon? Do you enjoy building analytical models, testing hypotheses, and seeing your work drive real business decisions? We are looking for a Data Scientist to join our team at Amazon. In this role, you will apply statistical techniques, machine learning, and data analysis to solve well-defined business problems and deliver measurable improvements. You will work with large-scale datasets from multiple sources, build and validate models, and create metrics that quantify impact. This is an opportunity for a self-starter who is comfortable with ambiguity in data, thrives on collaboration with science and engineering partners, and is eager to grow their craft in a fast-paced environment. Key job responsibilities - Use statistical methods, machine learning algorithms, and data analysis techniques to translate well-defined business problems into data science solutions that improve processes and systems. - Gather, clean, and integrate large datasets from multiple sources to build and validate models, ensuring data integrity and adherence to best practices throughout the pipeline. - Design and deliver metrics, visualizations, and reports that compare results against benchmarks and original project goals, enabling clear and informed decision-making for stakeholders. - Collaborate with business, science, and engineering teams to understand customer needs, identify opportunities, and develop scalable analytical solutions using tools such as Python, R, or SQL. - Evaluate opportunities to integrate generative artificial intelligence tools into workflows, refining outputs and applying responsible practices to augment productivity and solution quality. About the team Our team is focused on using data science to help Amazon make smarter, faster decisions that benefit customers. We build analytical models and machine learning solutions that operate at scale, and we value rigorous methodology paired with clear communication. You will join a collaborative group of scientists and engineers who are passionate about learning from one another and pushing the boundaries of what data can do. If you are excited about growing your career in an inclusive, supportive environment where your contributions have tangible impact, we would love to hear from you.
CH, Zurich
Are you a MS or PhD student interested in a 2026 Applied Science Internship in the field of Robotics? Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way? If this describes you, come join our research teams at Amazon. Key job responsibilities As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept. We are looking for scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.
US, NY, New York
Twitch is looking for an Applied Scientist to lead computer vision and video manipulation work within Amazon Publisher Monetization (APM). You will apply large language models (LLMs), vision-language models (VLMs), and traditional computer vision techniques to build products that directly improve how millions of Twitch creators produce content and how viewers engage with it. This is a high-visibility project backed by senior leadership, and you will have the autonomy to evaluate both first-party and third-party solutions to deliver the highest quality results quickly. If you thrive in a fast-paced, startup-like environment where your scientific decisions shape the product roadmap, we want to hear from you. Key job responsibilities Design and deploy computer vision and video manipulation models using LLMs, VLMs, and traditional techniques, taking solutions from research prototype to production. Partner with senior Product and Twitch leadership to translate customer needs into well-defined scientific problems and deliver end-to-end solutions. Evaluate and integrate generative AI capabilities across Amazon Bedrock and third-party platforms, selecting the approach that maximizes product quality. Drive the team's scientific agenda by proposing new initiatives, authoring technical documents, and contributing to external publications. Mentor fellow scientists and engineers through code reviews, design guidance, and technical assessments that raise the overall quality bar. A day in the life You will split your time roughly evenly between independent research and collaborative sessions. Mornings often start with focused solo work — benchmarking a new VLM, writing model code, or analyzing experiment results. Midday you join working sessions with Product and Twitch leadership to align on requirements or review progress. Afternoons shift between pairing with engineers on production integration and mentoring teammates on evaluation methodology. Weekly science reviews and bi-weekly roadmap syncs keep the team aligned without over-scheduling your calendar. About the team The APM science team supports all of Amazon's Publisher Monetization efforts, building the models and systems behind Twitch's creator tools and ad experiences. The team includes applied scientists, software engineers, and product managers who collaborate daily to move ideas from whiteboard to production. We operate with a startup mentality — moving fast, wearing multiple hats, and iterating closely with Twitch and Product partners. We are accelerating investment in computer vision and generative AI, and this role will be central to that expansion.
TW, TPE, Hsinchu City
Are you passionate about robotics and research? Do you want to solve real customer problems through innovative technology? Do you enjoy working on scalable research and projects in a collaborative team environment? Do you want to see your science solutions directly impact millions of customers worldwide? At Amazon, we hire the best minds in technology to innovate and build on behalf of our customers. Customer obsession is part of our company DNA, which has made us one of the world's most beloved brands. We’re looking for current PhD or Master students with a passion for robotic research and applications to join us as Robotics Applied Scientist II Intern/Co-ops in 2027 to shape the future of robotics and automation at an unprecedented scale across. For these positions, our Robotics teams at Amazon are looking for students with a specialization in one or more of the research areas in robotics such as: robotics, robotics manipulation (e.g., robot arm, grasping, dexterous manipulation, end of arm tools/end effector), autonomous mobile robots, mobile manipulation, movement, autonomous navigation, locomotion, motion/path planning, controls, perception, sensing, robot learning, artificial intelligence, machine learning, computer vision, large language models, human-robot interaction, robotics simulation, optimization, and more! We're looking for curious minds who think big and want to define tomorrow's technology. At Amazon, you'll grow into the high-impact engineer you know you can be, supported by a culture of learning and mentorship. Every day brings exciting new challenges and opportunities for personal growth. By applying to this role, you will be considered for Robotics Applied Scientist II Intern/Co-op (2027) opportunities across various Robotics teams at Amazon with different robotics research focus, with internship positions available for multiple locations, durations (3 to 6+ months), and year-round start dates (winter, spring, summer, fall). Amazon intern and co-op roles follow the same internship structure. "Intern/Internship" wording refers to both interns and co-ops. Amazon internships across all seasons are full-time positions during vacation, and interns should expect to work in office, Monday-Friday, up to 40 hours per week typically between 9am-6pm. Specific team norms around working hours will be communicated by your manager. Interns should not have other employment during the Amazon work-day. Applicants should have a minimum of one quarter/semester/trimester remaining in their studies after their internship concludes. The robotics internship join dates, length, location, and prospective team will be finalized at the time of any applicable job offers. In your application, you will be able to provide your preference of research interests, start dates, internship duration, and location. While your preference will be taken into consideration, we cannot guarantee that we can meet your selection based on several factors including but not limited to the internship availability and business needs of this role.
US, WA, Seattle
Amazon Prime Air is seeking an Applied Scientist Manager to lead the team developing our next-generation drone navigation system. You will lead a team of 8–12 Applied Scientists, Research Scientists and Software Development Engineers who invent, model, develop, test, and deploy the state estimation algorithms. Your team will own the full algorithm lifecycle: early prototypes and modeling, simulation and hardware-in-the-loop testing, then flight test and deployment across a rapidly growing delivery network. You will also manage the feedback loop that brings real-world operational data back into the team to inform next generation of algorithm improvements. Because our navigation software is safety-critical, you will work closely with the Controls, Perception, Safety, Simulation and Flight Test teams to deliver robust, fully verified algorithms. This role combines deep technical leadership with people leadership. You will hire and develop science-focused engineers, run a fast-paced agile development team and help shape the technical roadmap for autonomous navigation at Prime Air. Key job responsibilities - Lead a team of 8–12 Applied Scientists, Research Scientists and Software Development Engineers developing novel navigation and state estimation algorithms, including Kalman filtering (EKF/UKF, error-state), tightly and loosely coupled GNSS/INS integration, multi-sensor fusion and observability analysis, integrity monitoring (RAIM, fault detection and exclusion), and visual navigation (VIO/VSLAM). - Drive day-to-day execution across the full development lifecycle: prototyping, modeling, simulation testing, hardware-in-the-loop testing, flight test and deployment to the production fleet. - Lead the team in a scrum-style agile process: set sprint priorities, manage dependencies and deliver against program milestones. - Build mechanisms that capture operational learnings from our delivery network, such as flight data, anomalies and performance trends, and feed them into the requirements and design of each new generation of algorithms. - Set technical direction and raise the bar on scientific rigor. - Partner with Flight controls, Perception, Hardware, Safety, Simulation and Flight Operations teams to define interfaces, requirements and verification evidence. - Hire, mentor and develop Applied Scientists, Research Scientists and SDEs, building a diverse, high-performing team with a strong safety culture. - Communicate technical strategy, trade-offs, risks and results clearly in writing and in reviews with senior leadership.
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the next-level. We focus on creating entirely new products and services with a goal of positively impacting the lives of our customers. No industries or subject areas are out of bounds. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. Here at Amazon, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We are constantly learning through programs that are local, regional, and global. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust. Our team highly values work-life balance, mentorship and career growth. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We care about your career growth and strive to assign projects and offer training that will challenge you to become your best.