HypRank: How Alexa determines what skill can best meet a customer’s need

Amazon Alexa currently has more than 40,000 third-party skills, which customers use to get information, perform tasks, play games, and more. To make it easier for customers to find and engage with skills, we are moving toward skill invocation that doesn’t require mentioning a skill by name (as highlighted in a recent post).

To enable name-free skill interaction, Alexa currently uses a two-step, scalable, and efficient neural shortlisting-reranking approach. (I described our approach to shortlisting in a post yesterday). The shortlisting step uses a scalable neural model to efficiently find the optimal (k-best) candidate skills for handling a particular utterance; the re-ranking step uses rich contextual signals to find the most relevant of those skills. We use the term “re-ranking” since we improve upon the initial confidence score provided by the shortlisting step.

This week, at the Human Language Technologies conference of the North American chapter of the Association for Computational Linguistics (NAACL 2018), my colleagues and I presented a paper, “A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding,” that describes our approach.

A high-level flow of the two-step shortlisting-reranking approach
A high-level flow of the two-step shortlisting-reranking approach

The Challenge

The problem here is essentially a domain classification problem over the k-best candidate skills returned by the shortlisting system, which we call Shortlister. The goal of Shortlister is to achieve high recall — to identify as many pertinent skills as possible — with maximum efficiency. On the other hand, the goal of the reranking network, HypRank, is to use rich contextual signals to achieve high precision — to select the most pertinent skills. Designing HypRank comes with its own challenges:

  • Hypothesis representation: It needs to use available contextual signals to produce an effective hypothesis representation for each skill in the k-best list;
  • Cross-hypothesis feature representation: It needs to efficiently and automatically compare features, such as a skill’s intent confidence, to those of other candidate skills in the k-best list;
  • Generalization: It needs to be language-agnostic; and
  • Robustness: It needs to be able to accommodate changes, such as independent modifications to Shortlister or to the natural-language-understanding models that provide skill-specific semantic interpretation of utterances.
Neural Hypotheses Reranker model architecture
Neural Hypotheses Reranker model architecture

The HypRank Neural Model

HypRank comprises two components:

1. Hypothesis representation for each skill; and 2. A bidirectional long short-term memory (bi-LSTM) model for re-ranking a list of hypotheses.

For each skill in the k-best list, we form a hypothesis based on additional semantic and contextual signals. For example, we perform intent-slot semantic analysis for a skill. If a user says “play Michael Jackson,” the Pop Music skill might infer the intent PlayMusic, while the Classic Music skill might infer the intent PlayTune. But the confidence scores that the skills assign their inferences could be useful for skill re-ranking. The hypothesis generator is constantly being updated, re-weighting signals and accommodating new functionality and changes in usage patterns.

HypRank is unique because of its list-wise ranking approach using a bi-LSTM layer. LSTM models are common in natural-language processing because they factor in the order in which data are received: if you’re trying to understand the sixth word in an utterance, it helps to know what the previous five were. Bidirectional LSTM models consider data sequences both forward and backward.

By leveraging the bi-LSTM layer, HypRank can evaluate an entire list of skill hypotheses before providing a re-ranking score for each hypothesis. This is distinct from point-wise approaches that look at each hypothesis in isolation or pair-wise approaches that look at pairs of hypotheses in a series of tournament-like competitions.

Whereas past re-ranking approaches relied on manually crafted cross-hypothesis features, our approach uses the bi-LSTM layer to automatically learn and encode appropriate cross-hypothesis features for improved re-ranking. The encoded cross-hypothesis vector then passes through a conventional feed-forward network, which determines the final score for each hypothesis.

HypRank is agnostic about both language and locale. The contextual information used to form a hypothesis is designed to be independent of whether the language spoken is English or French and whether the locale is the U.S. or France. Research is ongoing on making HypRank as generalizable as possible and also as robust as possible to changes in the upstream signals.

Paper: "A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding"

Alexa science

Acknowledgments: Sunghyun Park, Ameen Patel, Jihwan Lee, Joo-Kyung Kim, Dongchan Kim, Hammil Kerry, Ruhi Sarikaya, and all engineers in the Fan Sun, Bo Cao, and Yan Weng teams.

About the Author
Young-Bum Kim
Young-Bum Kim is an Amazon science leader in the Alexa AI organization.

Related content

US
Job summaryWant to help the largest global enterprises derive business value through the adoption of Artificial Intelligence (AI) and Machine Learning (ML)? Excited by using massive amounts of disparate data to develop ML models? Eager to learn to apply ML to a diverse array of enterprise use cases? Thrilled to be a part of Amazon who has been pioneering and shaping the world’s AI/ML technology for decades?At Amazon Web Services (AWS), we are helping large enterprises build ML models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. AWS Professional Services works together with AWS customers to address their business needs using AI solutions.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers. A successful candidate will be a person who enjoys diving deep into data, doing analysis, discovering root causes, and designing long-term solutions.This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed.Major responsibilities include: - Assist customers by being able to deliver a ML project from beginning to end, including understanding the business need, aggregating data, exploring data, building & validating predictive models, and deploying completed models with concept-drift monitoring and retraining to deliver business impact to the organization· Use AWS AI services (e.g., Personalize), ML platforms (SageMaker), and frameworks (e.g., MXNet, TensorFlow, PyTorch, SparkML, scikit-learn) to help our customers build ML models· Research and implement novel ML approaches, including hardware optimizations on platforms such as AWS Inferentia· Work with our other Professional Services consultants (Big Data, IoT, HPC) to analyze, extract, normalize, and label relevant data, and with our Professional Services engineers to operationalize customers’ models after they are prototypedInclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future.
US, NY, New York
Job summaryWant to help the largest global enterprises derive business value through the adoption of Artificial Intelligence (AI) and Machine Learning (ML)? Excited by using massive amounts of disparate data to develop ML models? Eager to learn to apply ML to a diverse array of enterprise use cases? Thrilled to be a part of Amazon who has been pioneering and shaping the world’s AI/ML technology for decades?At Amazon Web Services (AWS), we are helping large enterprises build ML models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. AWS Professional Services works together with AWS customers to address their business needs using AI solutions.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers. A successful candidate will be a person who enjoys diving deep into data, doing analysis, discovering root causes, and designing long-term solutions.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life harmony. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. As such, this is a customer facing role in a hybrid delivery model. Project engagements include remote delivery methods and onsite engagement that will include travel to customer locations as needed.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future.This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed.Major responsibilities include:· Assist customers by being able to deliver a ML project from beginning to end, including understanding the business need, aggregating data, exploring data, building & validating predictive models, and deploying completed models with concept-drift monitoring and retraining to deliver business impact to the organization· Use AWS AI services (e.g., Personalize), ML platforms (SageMaker), and frameworks (e.g., MXNet, TensorFlow, PyTorch, SparkML, scikit-learn) to help our customers build ML models· Research and implement novel ML approaches, including hardware optimizations on platforms such as AWS Inferentia· Work with our other Professional Services consultants (Big Data, IoT, HPC) to analyze, extract, normalize, and label relevant data, and with our Professional Services engineers to operationalize customers’ models after they are prototyped
ES, B, Barcelona
Job summaryAre you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon transportation network. Current research and areas of work within our team include machine learning forecast, online reinforcement learning, and anomaly detection models, among others.We are looking for a Senior Machine Learning Scientist with a strong academic background in the areas of machine learning, signal processing, statistics and/or optimisation. Expertise in either of the following areas will be appreciated:· Probabilistic Machine Learning· Graph Neural Networks (GNNs), Temporal Graph Networks (TGNs), and/or Graph Deep Learning (GDL)At Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
DE, BE, Berlin
Job summaryAre you a MS or PhD student interested in a 2022 Applied Science Internship in the field of Computer Vision, or Machine Learning/Deep Learning?Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?If this describes you, come join our research teams at Amazon. As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.We are looking for smart scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.We are hiring interns for the following teams in Berlin:Team Name: Alexa AI - Web InformationThe Domain/Research Focus: Question Answering and Machine Translation.Team Description: The team works on applied research for both Machine Translation for Amazon Translate and Question Answering for Alexa.
DE, BE, Berlin
Job summaryAre you a MS or PhD student interested in a 2022 Applied Science Internship in the field of Computer Vision, or Machine Learning/Deep Learning?Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?If this describes you, come join our research teams at Amazon. As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.We are looking for smart scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.We are hiring interns for the following teams in Berlin:Team Name: Alexa AI - Web InformationThe Domain/Research Focus: Question Answering and Machine Translation.Team Description: The team works on applied research for both Machine Translation for Amazon Translate and Question Answering for Alexa.
US, VA, Arlington
Job summaryAmazon Prime is looking for a talented Economist to join a team of scientists and engineers dedicated to understanding Prime customer behavior. As part of Amazon Prime, you will be working in one of the world's largest and most exciting big-data environments. The Economist role occupies a unique space at the intersection of technology, machine-learning, econometrics, large-scale scientific computing, social science, and product management.As an Economist within Amazon Prime, you will work closely with a team of economists, applied scientists, engineers, and product managers. You will propose and estimate novel statistical and econometric models to directly inform strategic decisions about characteristics of the Amazon Prime membership. These include what membership prices, benefits, and benefit content deliver the most value for our customers around the world. Since customer preferences are heterogeneous across marketplaces and customer segments, you will rely heavily on randomized experiments. You will also utilize the latest econometric tools at the intersection of structural econometrics, causal inference and machine learning. You will build massively parallelized science on modern distributed computing infrastructures such as Apache Spark. Additionally, the team is unique in its exposure to senior leaders of the Prime team and other Amazon business units.The successful candidate will demonstrate a capacity for building, estimating, and defending causal or structural statistical models using software such as R, Python, or STATA. They will also show a willingness to learn a broad set of tools from causal inference and structural econometrics, as well as how to create software. Knowledge of SQL, machine learning, and large-scale scientific computing using distributed computing infrastructures such as Spark-Scala or PySpark would be a plus. Additionally, this candidate will be good at communicating technical insights to peer scientists as well as business professionals. The role will provide the opportunity to have a large strategic, world-wide impact on the customer experiences of Prime members.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, visit https://www.amazon.jobs/en/disability/us
AE, Dubai
Job summaryAre you interested in building the next-generation services that will redefine shopping in Middle East? We are disrupting the local shopping experience and looking for talented data scientist to design and build a new product, ground-up.We are looking for data scientist with 3+ years of solid experience in solving complex problems using machine learning and data science. As a data scientist, you will provide machine learning expertise that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance customer experience. You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.Key job responsibilities* Manage your own process: identify and execute on high impact projects, triage external requests, and make sure you bring projects to conclusion in time for the results to be useful.* Apply state-of-the-art Machine Learning methods to large amounts of data from different sources to build and operationalize abuse prevention, detection and mitigation solutions.*Deep dive on the problems using SQL and scripting languages like Python/R to drive short term and long term solutions leveraging Statistical Analysis.*Collaborate closely with the development team to recommend and build innovations based on Data Science.*Collaborate in a fast paced environment with multiple teams
JO, Amman
Job summaryAre you interested in building the next-generation services that will redefine shopping in Middle East? We are disrupting the local shopping experience and looking for talented data scientist to design and build a new product, ground-up.We are looking for data scientist with 3+ years of solid experience in solving complex problems using machine learning and data science. As a data scientist, you will provide machine learning expertise that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance customer experience. You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.Key job responsibilities* Manage your own process: identify and execute on high impact projects, triage external requests, and make sure you bring projects to conclusion in time for the results to be useful.* Apply state-of-the-art Machine Learning methods to large amounts of data from different sources to build and operationalize abuse prevention, detection and mitigation solutions.*Deep dive on the problems using SQL and scripting languages like Python/R to drive short term and long term solutions leveraging Statistical Analysis.*Collaborate closely with the development team to recommend and build innovations based on Data Science.*Collaborate in a fast paced environment with multiple teams
US, WA, Bellevue
Job summaryAmazon is looking for an outstanding Senior Applied Scientist to help build next generation selection/assortment systems. On the Specialized Selection team within the Supply Chain Optimization Technologies (SCOT) organization, we own the selection to determine which products Amazon offers in our fastest delivery programs. We build tools and systems that enable our partners and business owners to scale themselves by leveraging our problem domain expertise, focusing instead on introspecting our outputs and iteratively helping us improve our ML models rather than hand-managing their assortment. We partner closely with our business stakeholders as we work to develop state-of-the-art, scalable, automated selection management systems.As a Senior Applied Scientist, you will work with software engineers, product managers, and business teams to understand the business problems and requirements, distill that understanding to crisply define the problem, and design and develop innovative solutions to address them. Our team is highly cross-functional and employs a wide array of scientific tools and techniques to solve key challenges, including supervised and unsupervised machine learning, non-convex optimization, causal inference, natural language processing, linear programming, reinforcement learning, and other forecast algorithms. Some critical research areas in our space include modeling substitutability between similar products, incorporating basket awareness and complementarity-aware logic, measuring speed sensitivity of products, modeling network capacity constraints, and supply and demand forecasting.You will be a science tech leader for the team. As a Senior Applied Scientist you will:• Lead a team of scientists to innovate on state-of-the-art assortment planning systems for limited-shelf businesses.• Set the scientific strategic vision for the team. You lead the decomposition of problems and development of roadmaps to execute on it.• Set an example for other scientists with exemplary scientific analyses; maintainable, extensible, and well-tested code; and simple, intuitive, and effective solutions.• Influence team business and engineering strategies.• Exercise sound judgment to prioritize between short-term vs. long-term and business vs. technology needs.• Communicate clearly and effectively with stakeholders to drive alignment and build consensus on key initiatives.• Foster collaborations between scientists across Amazon researching similar or related problems.• Actively engage in the development of others, both within and outside the team.• Engage with the broader scientific community through presentations, publications, and patents.To help describe some of our challenges, we created a short video about SCOT at Amazon: http://bit.ly/amazon-scotKey job responsibilities"Machine Learning", optimization, ML, python, NLP, "reinforcement learning", "causal inference", "supervised learning", "unsupervised learning", "assortment planning", "recommendation systems", "experimental design", "Natural Language Processing"
GB, Cambridge
Job summaryWe are looking for a passionate, talented, and inventive Machine Learning Scientist to help build industry-leading Speech and Language technology. Our mission is to push the envelope in Text-to-Speech (TTS) in order to provide the best-possible experience for our customers.Key job responsibilitiesPosition Responsibilities:· Participate in the design, development, evaluation, deployment and updating of data-driven models for text-to-speech applications.· Participate in research activities including the application and evaluation of text-to-speech techniques for novel applications.· Research and implement novel ML and statistical approaches to add value to the business.· Mentor junior engineers and scientists.A day in the lifeAs a ML Scientist at Amazon you will work with talented peers to develop novel algorithms and modelling techniques to advance the state of the art in speech synthesis.About the teamOur team undertakes research together with multiple organizations to advance the state-of-the-art in speech technologies and scale it to our whole portfolio of voices and languages.We are at the heart of making sure all our customers, in all languages or locations, get the same ground-breaking experience.