Making Alexa more friction-free

Friction is any variable that impedes your progress toward a goal, whether it’s purchasing a product or navigating traffic to make your 9 a.m. meeting on time.

Amazon is obsessively focused on reducing or eliminating friction – think one-click ordering, Amazon Prime, or Amazon Go.

This morning, I am delivering a keynote talk at the World Wide Web Conference in Lyon, France, with the title, Conversational AI for Interacting with the Digital and Physical World. In my presentation, I’ll emphasize that while today’s computers are currently optimized to provide audiovisual output and receive tactile and motor skill input, we are on the cusp of voice becoming the primary input. This is significant as we evolve to a world of ambient computing, where we are surrounded at home, work and on the go by devices with internet connectivity and the ability to interact with cloud-based services via natural language understanding. Our goal is to enable more natural interaction with all of these IOT devices, and for these devices to more proactively engage with us.

The mobile computing era provides many benefits; we all wouldn’t be tethered to our phones if it didn’t. But when you think about it, what’s changed primarily with the phone is the form factor; the screen is smaller but we interact with our phones much the same way we do our PCs. It’s great to have a computing device where ever we go, yet we are still attached to a screen, touching, typing and swiping. With voice, you’re truly mobile. I’m often in the kitchen cooking, cleaning or putting groceries in the fridge, and without diverting my attention I can ask Alexa to play a song, or provide a weather update. Rarely am I looking directly at my Echo device when I ask a question, or make a request. In a sense, voice-enabled devices set me free. The profound difference in this emerging era is that with the benefit of AI and machine-learning technologies, Alexa and similar services can learn about you, and conform to your needs, instead of you having to conform to the system’s interaction model.

Alexa is similar to any other Amazon service. It is about removing friction in our customers’ interactions with the physical and digital world. The Alexa Brain initiative, which I lead, is one of many within the Alexa organization focused on making Alexa smarter and more natural to engage with. Our goals are to make it easier for users to discover and interact with the more than 40,000 third-party skills that developers have created for Alexa, and to improve Alexa’s ability to track context and memory within and across dialog sessions.

In my talk today, I’ll be updating conference goers on our progress against these goals, and outline the challenges that still exist in making interaction with Alexa more natural. I’ll also be highlighting three new capabilities we’ll soon make available to our customers.

Skills arbitration

We are always looking for ways to make it easier for customers to find and engage with skills. One of our approaches to this is the ability for Alexa to dynamically arbitrate among skills using machine learning. In the coming weeks, we’re rolling out this new capability that allows customers in the U.S. to automatically discover, enable and launch skills using natural phrases and requests. For example, using an Echo Show device, I recently asked: “Alexa, how do I remove an oil stain from my shirt?” She replied: “Here is Tide Stain Remover.” This beta experience was friction-free; the skill just walked me through the process of removing an oil stain from my shirt. Previously, I would have had to discover the skill on my own to use it. This is just one example, but it gives you a sense for how this capability will provide customers frictionless direct access to, and interaction with, third-party skills. We’re excited about what we’ve learned from our early beta users and will gradually make this capability available to more skills and customers in the U.S.

Context carryover

Soon, we will improve our understanding of multi-turn utterances, or what we refer to as context carryover. Initially, we will make this capability available to all of our customers in the U.S., U.K., and Germany. Previously, we’ve supported two-turn interactions with explicit pronoun references. For example, “Alexa, what was Adele’s first album?” “Alexa, play it.” We are expanding beyond this to include utterances without pronouns. For example: “Alexa, how is the weather in Seattle?” → “What about this weekend?” We are also supporting context across domains. For example: “Alexa, how’s the weather in Portland?” → “How long does it take to get there?” We are providing this more natural way of engaging with Alexa by adding deep learning models to our spoken language understanding (SLU) pipeline that allows us to carry customers’ intent and entities within and across domains (i.e., between weather and traffic).

Memory

In the U.S, we also soon will begin to roll out a new memory feature. With this capability, Alexa can remember any information for you so that you never forget. Alexa can store arbitrary information you want and retrieve it later. For example, a customer might ask: “Alexa, remember that Sean’s birthday is June 20th.” Alexa will reply: “Okay, I’ll remember that Sean’s birthday is June 20th.” This memory feature is the first of many launches this year that will make Alexa more personalized. It's early days, but with this initial release we will make it easier for customers to save information, as well as provide a natural way to recall that information later.

The challenges ahead

The work of our science and engineering teams to make Alexa smarter and more engaging has been extraordinary. It requires significant changes to Alexa’s existing architecture and incorporates contextual cues and customer preferences across all components of our system.

We have many challenges still to address, such as how to scale these new experiences across languages and different devices, how to scale skill arbitration across the tens of thousands of Alexa skills, and how to measure experience quality. Additionally, there are component-level technology challenges that span automatic speech recognition, spoken language understanding, dialog management, natural language generation, text-to-speech synthesis, and personalization.

As Rohit Prasad, vice president and head scientist of the Alexa Machine Learning team, said in a recent interview, we’ve only begun to scratch the surface of what’s possible. Skills arbitration, context carryover and the memory feature are early instances of a class of work Amazon scientists and engineers are doing to make engaging with Alexa more friction-free. We’re on a multi-year journey to fundamentally change human-computer interaction, and as we like to say at Amazon, it’s still Day 1.

About the Author
Ruhi Sarikaya is director of applied science, Alexa AI.

Related content

US
Job summaryWant to help the largest global enterprises derive business value through the adoption of Artificial Intelligence (AI) and Machine Learning (ML)? Excited by using massive amounts of disparate data to develop ML models? Eager to learn to apply ML to a diverse array of enterprise use cases? Thrilled to be a part of Amazon who has been pioneering and shaping the world’s AI/ML technology for decades?At Amazon Web Services (AWS), we are helping large enterprises build ML models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. AWS Professional Services works together with AWS customers to address their business needs using AI solutions.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers. A successful candidate will be a person who enjoys diving deep into data, doing analysis, discovering root causes, and designing long-term solutions.This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed.Major responsibilities include: - Assist customers by being able to deliver a ML project from beginning to end, including understanding the business need, aggregating data, exploring data, building & validating predictive models, and deploying completed models with concept-drift monitoring and retraining to deliver business impact to the organization· Use AWS AI services (e.g., Personalize), ML platforms (SageMaker), and frameworks (e.g., MXNet, TensorFlow, PyTorch, SparkML, scikit-learn) to help our customers build ML models· Research and implement novel ML approaches, including hardware optimizations on platforms such as AWS Inferentia· Work with our other Professional Services consultants (Big Data, IoT, HPC) to analyze, extract, normalize, and label relevant data, and with our Professional Services engineers to operationalize customers’ models after they are prototypedInclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 14 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life balance. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future.
US, NY, New York
Job summaryWant to help the largest global enterprises derive business value through the adoption of Artificial Intelligence (AI) and Machine Learning (ML)? Excited by using massive amounts of disparate data to develop ML models? Eager to learn to apply ML to a diverse array of enterprise use cases? Thrilled to be a part of Amazon who has been pioneering and shaping the world’s AI/ML technology for decades?At Amazon Web Services (AWS), we are helping large enterprises build ML models on the AWS Cloud. We are applying predictive technology to large volumes of data and against a wide spectrum of problems. AWS Professional Services works together with AWS customers to address their business needs using AI solutions.AWS Professional Services is a unique consulting team. We pride ourselves on being customer obsessed and highly focused on the AI enablement of our customers. If you have experience with AI, including building ML models, we’d like to have you join our team. You will get to work with an innovative company, with great teammates, and have a lot of fun helping our customers. A successful candidate will be a person who enjoys diving deep into data, doing analysis, discovering root causes, and designing long-term solutions.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have thirteen employee-led affinity groups, reaching 85,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences. Amazon’s culture of inclusion is reinforced within our 16 Leadership Principles, which remind team members to seek diverse perspectives, learn and be curious, and earn trust.Work/Life BalanceOur team puts a high value on work-life harmony. Striking a healthy balance between your personal and professional life is crucial to your happiness and success here. We are a customer-obsessed organization—leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. As such, this is a customer facing role in a hybrid delivery model. Project engagements include remote delivery methods and onsite engagement that will include travel to customer locations as needed.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded professional and enable them to take on more complex tasks in the future.This is a customer-facing role and you will be required to travel to client locations and deliver professional services as needed.Major responsibilities include:· Assist customers by being able to deliver a ML project from beginning to end, including understanding the business need, aggregating data, exploring data, building & validating predictive models, and deploying completed models with concept-drift monitoring and retraining to deliver business impact to the organization· Use AWS AI services (e.g., Personalize), ML platforms (SageMaker), and frameworks (e.g., MXNet, TensorFlow, PyTorch, SparkML, scikit-learn) to help our customers build ML models· Research and implement novel ML approaches, including hardware optimizations on platforms such as AWS Inferentia· Work with our other Professional Services consultants (Big Data, IoT, HPC) to analyze, extract, normalize, and label relevant data, and with our Professional Services engineers to operationalize customers’ models after they are prototyped
ES, B, Barcelona
Job summaryAre you interested in building state-of-the-art machine learning systems for the most complex, and fastest growing, transportation network in the world? If so, Amazon has the most exciting, and never-before-seen, challenges at this scale (including those in sustainability, e.g. how to reach net zero carbon by 2040).Amazon’s transportation systems get millions of packages to customers worldwide faster and cheaper while providing world class customer experience – from online checkout, to shipment planning, fulfillment, and delivery. Our software systems include services that use tens of thousands of signals every second to make business decisions impacting billions of dollars a year, that integrate with a network of small and large carriers worldwide, that manage business rules for millions of unique products, and that improve experience of over hundreds of millions of online shoppers.As part of this team you will focus on the development and research of machine learning solutions and algorithms for core planning systems, as well as for other applications within Amazon Transportation Services, and impact the future of the Amazon transportation network. Current research and areas of work within our team include machine learning forecast, online reinforcement learning, and anomaly detection models, among others.We are looking for a Senior Machine Learning Scientist with a strong academic background in the areas of machine learning, signal processing, statistics and/or optimisation. Expertise in either of the following areas will be appreciated:· Probabilistic Machine Learning· Graph Neural Networks (GNNs), Temporal Graph Networks (TGNs), and/or Graph Deep Learning (GDL)At Amazon, we strive to continue being the most customer-centric company on earth. To stay there and continue improving, we need exceptionally talented, bright, and driven people. If you'd like to help us build the place to find and buy anything online, and deliver in the most efficient and greenest way possible, this is your chance to make history.
DE, BE, Berlin
Job summaryAre you a MS or PhD student interested in a 2022 Applied Science Internship in the field of Computer Vision, or Machine Learning/Deep Learning?Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?If this describes you, come join our research teams at Amazon. As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.We are looking for smart scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.We are hiring interns for the following teams in Berlin:Team Name: Alexa AI - Web InformationThe Domain/Research Focus: Question Answering and Machine Translation.Team Description: The team works on applied research for both Machine Translation for Amazon Translate and Question Answering for Alexa.
DE, BE, Berlin
Job summaryAre you a MS or PhD student interested in a 2022 Applied Science Internship in the field of Computer Vision, or Machine Learning/Deep Learning?Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?If this describes you, come join our research teams at Amazon. As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.We are looking for smart scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.We are hiring interns for the following teams in Berlin:Team Name: Alexa AI - Web InformationThe Domain/Research Focus: Question Answering and Machine Translation.Team Description: The team works on applied research for both Machine Translation for Amazon Translate and Question Answering for Alexa.
US, VA, Arlington
Job summaryAmazon Prime is looking for a talented Economist to join a team of scientists and engineers dedicated to understanding Prime customer behavior. As part of Amazon Prime, you will be working in one of the world's largest and most exciting big-data environments. The Economist role occupies a unique space at the intersection of technology, machine-learning, econometrics, large-scale scientific computing, social science, and product management.As an Economist within Amazon Prime, you will work closely with a team of economists, applied scientists, engineers, and product managers. You will propose and estimate novel statistical and econometric models to directly inform strategic decisions about characteristics of the Amazon Prime membership. These include what membership prices, benefits, and benefit content deliver the most value for our customers around the world. Since customer preferences are heterogeneous across marketplaces and customer segments, you will rely heavily on randomized experiments. You will also utilize the latest econometric tools at the intersection of structural econometrics, causal inference and machine learning. You will build massively parallelized science on modern distributed computing infrastructures such as Apache Spark. Additionally, the team is unique in its exposure to senior leaders of the Prime team and other Amazon business units.The successful candidate will demonstrate a capacity for building, estimating, and defending causal or structural statistical models using software such as R, Python, or STATA. They will also show a willingness to learn a broad set of tools from causal inference and structural econometrics, as well as how to create software. Knowledge of SQL, machine learning, and large-scale scientific computing using distributed computing infrastructures such as Spark-Scala or PySpark would be a plus. Additionally, this candidate will be good at communicating technical insights to peer scientists as well as business professionals. The role will provide the opportunity to have a large strategic, world-wide impact on the customer experiences of Prime members.Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, visit https://www.amazon.jobs/en/disability/us
AE, Dubai
Job summaryAre you interested in building the next-generation services that will redefine shopping in Middle East? We are disrupting the local shopping experience and looking for talented data scientist to design and build a new product, ground-up.We are looking for data scientist with 3+ years of solid experience in solving complex problems using machine learning and data science. As a data scientist, you will provide machine learning expertise that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance customer experience. You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.Key job responsibilities* Manage your own process: identify and execute on high impact projects, triage external requests, and make sure you bring projects to conclusion in time for the results to be useful.* Apply state-of-the-art Machine Learning methods to large amounts of data from different sources to build and operationalize abuse prevention, detection and mitigation solutions.*Deep dive on the problems using SQL and scripting languages like Python/R to drive short term and long term solutions leveraging Statistical Analysis.*Collaborate closely with the development team to recommend and build innovations based on Data Science.*Collaborate in a fast paced environment with multiple teams
JO, Amman
Job summaryAre you interested in building the next-generation services that will redefine shopping in Middle East? We are disrupting the local shopping experience and looking for talented data scientist to design and build a new product, ground-up.We are looking for data scientist with 3+ years of solid experience in solving complex problems using machine learning and data science. As a data scientist, you will provide machine learning expertise that helps accelerate the business. You will build various data and machine learning models that help us innovate different ways to enhance customer experience. You will need to be entrepreneurial, wear many hats, and work in a highly collaborative environment. We like to move fast, experiment, iterate and then scale quickly, thoughtfully balancing speed and quality.Key job responsibilities* Manage your own process: identify and execute on high impact projects, triage external requests, and make sure you bring projects to conclusion in time for the results to be useful.* Apply state-of-the-art Machine Learning methods to large amounts of data from different sources to build and operationalize abuse prevention, detection and mitigation solutions.*Deep dive on the problems using SQL and scripting languages like Python/R to drive short term and long term solutions leveraging Statistical Analysis.*Collaborate closely with the development team to recommend and build innovations based on Data Science.*Collaborate in a fast paced environment with multiple teams
US, WA, Bellevue
Job summaryAmazon is looking for an outstanding Senior Applied Scientist to help build next generation selection/assortment systems. On the Specialized Selection team within the Supply Chain Optimization Technologies (SCOT) organization, we own the selection to determine which products Amazon offers in our fastest delivery programs. We build tools and systems that enable our partners and business owners to scale themselves by leveraging our problem domain expertise, focusing instead on introspecting our outputs and iteratively helping us improve our ML models rather than hand-managing their assortment. We partner closely with our business stakeholders as we work to develop state-of-the-art, scalable, automated selection management systems.As a Senior Applied Scientist, you will work with software engineers, product managers, and business teams to understand the business problems and requirements, distill that understanding to crisply define the problem, and design and develop innovative solutions to address them. Our team is highly cross-functional and employs a wide array of scientific tools and techniques to solve key challenges, including supervised and unsupervised machine learning, non-convex optimization, causal inference, natural language processing, linear programming, reinforcement learning, and other forecast algorithms. Some critical research areas in our space include modeling substitutability between similar products, incorporating basket awareness and complementarity-aware logic, measuring speed sensitivity of products, modeling network capacity constraints, and supply and demand forecasting.You will be a science tech leader for the team. As a Senior Applied Scientist you will:• Lead a team of scientists to innovate on state-of-the-art assortment planning systems for limited-shelf businesses.• Set the scientific strategic vision for the team. You lead the decomposition of problems and development of roadmaps to execute on it.• Set an example for other scientists with exemplary scientific analyses; maintainable, extensible, and well-tested code; and simple, intuitive, and effective solutions.• Influence team business and engineering strategies.• Exercise sound judgment to prioritize between short-term vs. long-term and business vs. technology needs.• Communicate clearly and effectively with stakeholders to drive alignment and build consensus on key initiatives.• Foster collaborations between scientists across Amazon researching similar or related problems.• Actively engage in the development of others, both within and outside the team.• Engage with the broader scientific community through presentations, publications, and patents.To help describe some of our challenges, we created a short video about SCOT at Amazon: http://bit.ly/amazon-scotKey job responsibilities"Machine Learning", optimization, ML, python, NLP, "reinforcement learning", "causal inference", "supervised learning", "unsupervised learning", "assortment planning", "recommendation systems", "experimental design", "Natural Language Processing"
GB, Cambridge
Job summaryWe are looking for a passionate, talented, and inventive Machine Learning Scientist to help build industry-leading Speech and Language technology. Our mission is to push the envelope in Text-to-Speech (TTS) in order to provide the best-possible experience for our customers.Key job responsibilitiesPosition Responsibilities:· Participate in the design, development, evaluation, deployment and updating of data-driven models for text-to-speech applications.· Participate in research activities including the application and evaluation of text-to-speech techniques for novel applications.· Research and implement novel ML and statistical approaches to add value to the business.· Mentor junior engineers and scientists.A day in the lifeAs a ML Scientist at Amazon you will work with talented peers to develop novel algorithms and modelling techniques to advance the state of the art in speech synthesis.About the teamOur team undertakes research together with multiple organizations to advance the state-of-the-art in speech technologies and scale it to our whole portfolio of voices and languages.We are at the heart of making sure all our customers, in all languages or locations, get the same ground-breaking experience.