How Voice and Graphics Working Together Enhance the Alexa Experience

Last week, Amazon announced the release of both a redesigned Echo Show with a bigger screen and the Alexa Presentation Language, which enables third-party developers to build “multimodal” skills that coordinate Alexa’s natural-language-understanding systems with on-screen graphics.

Echo in use

One way that multimodal interaction can improve Alexa customers’ experiences is by helping resolve ambiguous requests. If a customer says, “Alexa, play Harry Potter”, the Echo Show screen could display separate graphics representing a Harry Potter audiobook, a movie, and a soundtrack. If the customer follows up by saying “the last one”, the system must determine whether that means the last item in the on-screen list, the last Harry Potter movie, or something else.

Alexa’s ability to handle these types of interactions derives in part from research that my colleagues and I presented earlier this year at the annual meeting of the Association for the Advancement of Artificial Intelligence. In our paper, we consider three different neural-network designs that treat query resolution as an integrated problem involving both on-screen data and natural-language understanding.

We find that they consistently outperform a natural-language-understanding network that uses hand-coded rules to factor in on-screen data. And on inputs that consist of voice only, their performance is comparable to that of a system trained exclusively on speech inputs. That means that extending the network to consider on-screen data does not degrade accuracy for voice-only inputs.

The other models we investigated are derivatives of the voice-only model, so I’ll describe it first.

All of our networks were trained to classify utterances according to two criteria, intent and slot. An intent is the action that the customer wants Alexa to perform, such as PlayAction<Movie>. Slot values designate the entities on which the intents act, such as ‘Harry Potter’->Movie.name. We have found, empirically, that training a single network to perform both classifications works better than training a separate network for each.

As inputs to the network, we use two different embeddings of each utterance. Embeddings represent words as points in a geometric space, such that strings with similar meanings (or functional roles) are clustered together. Our network learns one embedding from the data on which it is trained, so it is specifically tailored to typical Alexa commands. We also use a standard embedding, based on a much larger corpus of texts, which groups words together according to the words they co-occur with.

The embeddings pass to a bidirectional long short-term memory network. A long short-term memory (LSTM) network processes inputs in order, and its judgment about any given input reflects its judgments about the preceding inputs. LSTMs are widely used in both speech recognition and natural-language processing because they can use context to resolve ambiguities. A bidirectional LSTM (bi-LSTM) is a pair of LSTMs that process an input utterance both backward and forward.

Intent classification is based on the final outputs of the forward and backward LSTMs, since the networks’ confidence in their intent classifications should increase the more of the utterance they see. Slot classification is based on the total output of the LSTMs, since the relevant slot values can occur anywhere in the utterance.

A diagram describing the architectures of all four neural models we evaluated
A diagram describing the architectures of all four neural models we evaluated. The baseline system,which doesn’t use screen information, received only the (a) inputs. The three multimodal neuralsystems received, respectively, (a) and (b); (a), (b), and (c); and (a), (b), and (d).

The data on which we trained all our networks was annotated using the Alexa Meaning Representation Language, a formal language that captures more sophisticated relationships between the parts of an input sentence than earlier methods did. A team of Amazon researchers presented a paper describing the language earlier this year at the annual meeting of the North American chapter of the Association for Computational Linguistics.

The other four models we investigated factored in on-screen content in various ways. The first was a benchmark system that modifies the outputs of the voice-only network according to hand-coded rules.

If, for instance, a customer says, “Play Harry Potter,” the voice-only classifier, absent any other information, might estimate a 50% probability that the customer means the audiobook, a 40% probability that she means the movie, and a 10% probability that she means the soundtrack. If, however, the screen is displaying only movies, our rules would boost the probability that the customer wants the movie.

The factors by which our rules increase or decrease probabilities were determined by a “grid search” on a subset of the training data, in which an algorithm automatically swept through a range of possible modifications to find those that yielded the most accurate results.

The first of our experimental neural models takes as input both the embeddings of the customer’s utterances and a vector representing the types of data displayed on-screen, such as Onscreen_Movie or Onscreen_Book. We assume a fixed number of data types, so the input is a “one-hot” vector, with a bit for each type. If data of a particular type is currently displayed on-screen, its bit is set to 1; otherwise, its bit is set to 0.

The next neural model takes as additional input not only the type of data displayed on-screen but the specific name of each data item — so not just Onscreen_Movie but also ‘Harry Potter’ or ‘The Black Panther’. Those names, too, undergo an embedding, which the network learns to perform during training.

Our third and final neural model factors in the names of on-screen data items as well, but in a more complex way. During training, it uses convolutional filters to, essentially, identify the separate contribution that each name on the screen makes toward the accuracy of the final classification. During operation, it thus bases each of its classifications on the single most relevant name on-screen, rather than all the names at once.

So, in all, we built, trained, and evaluated five different networks: the voice-only network; the voice-only network with hand-coded rules; the voice-and-data-type network; the voice, data type, and data name network; and the voice, data type, and convolutional-filter network.

We tested each of the five networks on four different data sets: slots with and without screen information and intents with and without screen information.

We evaluated performance according to two different metrics, micro-F1 and macro-F1. Micro-F1 scores the networks’ performance separately on each intent and slot, then averages the results. Macro-F1, by contrast, pools the scores across intents and slots and then averages. Micro-F1 gives more weight to intents and slots that are underrepresented in the data, macro-F1 less.

According to micro-F1, all three multimodal neural nets outperformed both the voice-only and the rule-based system across the board. The difference was dramatic on the test sets that included screen information, as might be expected, but the neural nets even had a slight edge on voice-only test sets. On all four test sets, the voice, data type, and data name network achieved the best results.

According to macro-F1, the neural nets generally outperformed the baseline systems, although the voice, data type, and data name network lagged slightly behind the baselines on voice-only slot classification. There was more variation in the top-performing system, too, with each of the three neural nets achieving the highest score on at least one test. Again, however, the neural nets dramatically outperformed the baseline systems on test sets that included screen information.

Acknowledgments: Angeliki Metallinou, Rahul Goel

Related content

US, WA, Seattle
Are you a Ph.D. interested in the fields of machine learning, deep learning, automated reasoning, speech, robotics, computer vision, optimization, or quantum computing? Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way? If this describes you, come join our science teams at Amazon. As an Applied Scientist, you will have access to large datasets with billions of images and video to build large-scale systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept. We are looking for smart scientists capable of using a variety of domain expertise to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.
LU, Luxembourg
Have you ever wondered how Amazon delivers timely and reliably hundreds of millions of packages to customer’s doorsteps? Are you passionate about data and mathematics, and hope to impact the experience of millions of customers? Are you obsessed with designing simple algorithmic solutions to very challenging problems?If so, we look forward to hearing from you!Amazon Transportation Services is seeking a Postdoctoral Scientist with Operations Research or Applied Mathematics background, to join our team in the EU Headquarters in Luxembourg, for a one-plus-one year full-time research position. As a key member of the EU Research Science Team, this person will be responsible for designing and implementing beyond state of the art algorithmic frameworks that optimize the middle-mile Amazon Transportation Network. The successful applicant will ensure that our end-to-end strategies in terms of customer demand fulfillment, routing, consolidation locations, linehaul/airhaul/sea options and last-mile transportation are streamlined and optimizedKey job responsibilitiesIn this role you will:• Work closely with a senior science advisor, collaborate with other scientists and engineers, and be part of Amazon’s vibrant and diverse global science community.• Publish your innovation in top-tier academic venues and hone your presentation skills.• Be inspired by challenges and opportunities to invent cutting-edge techniques in your area(s) of expertise.
ES, B, Barcelona
Are you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon delivery network. Current research and areas of work within our team include machine learning forecast, anomaly detection models, model interpretability, graph neural nets, among others.We are looking for a Manager, Applied Science (Machine Learning) with a strong academic background and industry experience in the areas of probabilistic machine learning, time series forecasting, and/or anomaly detection.At Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
NL, Amsterdam
Are you a passionate scientist in the computer vision area who is aspired to apply your skills to bring value to millions of customers? Here at Ring, we have a unique possibility to innovate and see how the results of our work improve the lives of millions of people and make neighborhoods safer.You will be part of a team committed to pushing the frontier of computer vision and machine learning technology to deliver the best experience for our neighbors. This is a great opportunity for you to innovate in this space by developing highly optimized algorithms that will work on scale. This position requires experience with developing efficient computer vision algorithms on resource-constrained computing platforms on edge. You will collaborate with different Amazon teams to make informed decisions on the best practices in machine learning to build highly-optimized integrated hardware and software platforms.Key job responsibilities* Research and implement the state-of-the-art computer vision and sensor fusion algorithms for resource-constrained computing platforms on a large scale.* Collaborate with product managers and engineering teams to design and implement computer vision and machine learning based features for Ring devices* Influence system design and product vision by making informed decisions on the selection of technology, data sources, algorithms, and sensors.
US, WA, Seattle
Amazon internships are full-time (40 hours/week) for 12 consecutive weeks with start dates in May - July 2023. Our internship program provides hands-on learning and building experiences for students who are interested in a career in hardware engineering. This role will be based in Seattle, and candidates must be willing to work in-person.Corporate Projects (CPT) is a team that sits within the broader Corporate Development organization at Amazon. We seek to bring net-new, strategic projects to life by working together with customers and evolving projects from ZERO-to-ONE. To do so, we deploy our resources towards proofs-of-concept (POCs) and pilot programs and develop them from high-level ideas (the ZERO) to tangible short-term results that provide validating signal and a path to scale (the ONE). We work with our customers to develop and create net-new opportunities by relentlessly scouring all of Amazon and finding new and innovative ways to strengthen and/or accelerate the Amazon Flywheel.CPT seeks an Applied Science intern to work with a diverse, cross-functional team to build new, innovative customer experiences. Within CPT, you will apply both traditional and novel scientific approaches to solve and scale problems and solutions. We are a team where science meets application. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems.
US, IL, Chicago
MULTIPLE POSITIONS AVAILABLECompany: AMAZON.COM SERVICES LLCPosition Title: Data Scientist ILocation: Chicago, IllinoisPosition Responsibilities:Build the core intelligence, insights, and algorithms that support the real estate acquisition strategies for Amazon physical stores. Tackle cutting-edge, complex problems such as predicting the optimal location for new Amazon stores by bringing together numerous data assets, and using best-in-class modeling solutions to extract the most information out of them. Work with business stakeholders, software development engineers, and other data scientists across multiple teams to develop innovative solutions at massive scale.Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation #0000
US, WA, Seattle
Note that this posting is for a handful of teams within Amazon Robotics. Teams include: Robotics, Computer Vision, Machine Learning, Optimization, and more.Are you excited about building high-performance robotic systems that can perceive and learn to help deliver for customers? The Amazon Robotics team is creating new science products and technologies that make this possible, at Amazon scale. We work at the intersection of computer vision, machine learning, robotic manipulation, navigation, and human-robot interaction.Amazon Robotics is seeking broad, curious applied scientists and engineering interns to join our diverse, full-stack team. In addition to designing, building, and delivering end-to-end robotic systems, our team is responsible for core infrastructure and tools that serve as the backbone of our robotic applications, enabling roboticists, applied scientists, software and hardware engineers to collaborate and deploy systems in the lab and in the field. We will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun. Come join us!A day in the lifeAs an intern you will develop a new algorithm to solve one of the challenging computer vision and manipulation problems in Amazon's robotic warehouses. Your project will fit your academic research experience and interests. You will code and test out your solutions in increasingly realistic scenarios and iterate on the idea with your mentor to find the best solution to the problem.
US, WA, Seattle
Are you excited about building high-performance robotic systems that can perceive, learn, and act intelligently alongside humans? The Robotics AI team is creating new science products and technologies that make this possible, at Amazon scale. We work at the intersection of computer vision, machine learning, robotic manipulation, navigation, and human-robot interaction.The Amazon Robotics team is seeking broad, curious applied scientists and engineering interns to join our diverse, full-stack team. In addition to designing, building, and delivering end-to-end robotic systems, our team is responsible for core infrastructure and tools that serve as the backbone of our robotic applications, enabling roboticists, applied scientists, software and hardware engineers to collaborate and deploy systems in the lab and in the field. Come join us!
US, WA, Bellevue
Employer: Amazon.com Services LLCPosition: Research Scientist IILocation: Bellevue, WA Multiple Positions Available1. Research, build and implement highly effective and innovative methods in Statistical Modeling, Machine Learning, and other quantitative techniques such as operational research and optimization to deliver algorithms that solve real business problems.2. Take initiative to scope and plan research projects based on roadmap of business owners and enable data-driven solutions. Participate in shaping roadmap for the research team.3. Ensure data quality throughout all stages of acquisition and processing of the data, including such areas as data sourcing/collection, ground truth generation, data analysis, experiment, evaluation and visualization etc.4. Navigate a variety of data sources, understand the business reality behind large-scale data and develop meaningful science solutions.5. Partner closely with product or/and program owners, as well as scientists and engineers in cross-functional teams with a clear path to business impact and deliver on demanding projects.6. Present proposals and results in a clear manner backed by data and coupled with conclusions to business customers and leadership team with various levels of technical knowledge, educating them about underlying systems, as well as sharing insights.7. Perform experiments to validate the feature additions as requested by domain expert teams.8. Some telecommuting benefits available.The pay range for this position in Bellevue, WA is $136,000-$184,000 (yr); however, base pay offered may vary depending on job-related knowledge, skills, and experience. A sign-on bonus and restricted stock units may be provided as part of the compensation package, in addition to a full range of medical, financial, and/or other benefits, dependent on the position offered. This information is provided by the Washington Equal Pay Act. Base pay information is based on market location. Applicants should apply via Amazon's internal or external careers site.#0000
US, VA, Arlington
The Central Science Team within Amazon’s People Experience and Technology org (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal. As Director for PXT Central Science Technology, you will be responsible for leading multiple teams through rapidly evolving complex demands and define, develop, deliver and execute on our science roadmap and vision. You will provide thought leadership to scientists and engineers to invent and implement scalable machine learning recommendations and data driven algorithms supporting flexible UI frameworks. You will manage and be responsible for delivering some of our most strategic technical initiatives. You will design, develop and operate new, highly scalable software systems that support Amazon’s efforts to be Earth’s Best Employer and have a significant impact on Amazon’s commitment to our employees and communities where we both serve and employ 1.3 million Amazonians. As Director of Applied Science, you will be part of the larger technical leadership community at Amazon. This community forms the backbone of the company, plays a critical role in the broad business planning, works closely with senior executives to develop business targets and resource requirements, influences our long-term technical and business strategy, helps hire and develop engineering leaders and developers, and ultimately enables us to deliver engineering innovations.This role is posted for Arlington, VA, but we are flexible on location at many of our offices in the US and Canada.