Screenshot shows a portion of the what should I watch experience
The new What Should I Watch (WSIW) experience, released in mid-September, combines Alexa AI and Fire TV recommendations to turn Alexa into an entertainment expert who provides relevant suggestions with a conversational customer experience.

The science behind the new “Alexa, what should I watch?” Fire TV experience

The phrase launches a feature built to help customers navigate an increasingly complex and diverse world of content.

"What should I watch?"

In an entertainment universe filled with a rapidly expanding catalog of shows across myriad channels and apps, this might be one of the most common questions to pop up in many households. And if you are among those who have trouble keeping up with all the latest shows and pinpointing which ones are worth your time, you are not alone.

In fact, more than half of respondents in a recent survey from the consulting firm Deloitte found it difficult to access content across multiple services, and 49% were frustrated if a service failed to provide them with good recommendations. Viewers find themselves surfing … and surfing. It takes the average smart TV owner 12 minutes to land on a show, according to a 2020 survey by Tivo — and for some viewers that can take up to half an hour.

"It's kind of shocking how much time customers have to spend on finding content instead of just sitting down on the couch and jumping into a TV show or a movie that they really enjoy," said Cosmin Laslau, a technical program manager who works on spoken language understanding as part of the Amazon Alexa Entertainment team. "We wanted to leverage new technology to help solve that problem for customers."

Image shows the new Fire TV Cube, left, the Fire TV Omni QLED Series, middle, and the Alexa Voice Remote Pro, right
The What Should I Watch experience works with many Fire TV devices, including the new Fire TV Cube, left, the Fire TV Omni QLED Series, middle, and the Alexa Voice Remote Pro announced at the 2022 Devices and Services event.

The team did that by launching What Should I Watch (WSIW). The new experience, released in mid-September, combines Alexa AI and Fire TV recommendations to turn Alexa into an entertainment expert who provides relevant suggestions with a conversational customer experience. The experience also works with the new Fire TV Cube, the Fire TV Omni QLED Series, and the Alexa Voice Remote Pro announced at the 2022 Devices and Services event.

“We built WSIW to rapidly experiment with new Alexa technologies and push the envelope on discovery experiences to address the core customer need to find something interesting to watch,” explained Parthasarathi Dutta Sharma, a product manager who helped bring WSIW to customers.

WSIW displays personalized recommendations when customers ask, “Alexa, what should I watch?” or a variant of that phrase. Customers can then customize the recommendations using voice prompts (for example, “just the ones that are free to me”) or by using their remote to select filters on the screen, watch trailers, view additional information (eg genre, ratings), and initiate playback.

Related content
Rohit Prasad on the pathway to generalizable intelligence and what excites him most about his re:MARS keynote.

The experience combines innovation for both Fire TV, with its extensive catalog, search and recommendation features, and the conversational AI that drives Alexa.

"We wanted to layer on these new innovations that have been developed around Alexa Conversations specifically," Laslau said. "We've given customers a broad range of natural ways to interact with Alexa, without being limited to a single utterance."

Since previewing WSIW last fall and beginning beta testing with customers, teams have worked to refine the customer experience.

“We used beta testing to closely observe how customers interacted with WSIW and to validate our core hypotheses on what works for customers,” explained Dutta Sharma. “A prime hypothesis we validated was viewers naturally gravitate to using natural language, with variability in inputs, while interacting with Alexa.”

Related content
Dialogue simulator and conversations-first modeling architecture provide ability for customers to interact with Alexa in a natural and conversational manner.

For example, to customize recommendations, the team found that initially customers might say, “I am in the mood for something funny”. They would then follow that by asking, “Which of these are on Prime Video?” or simply stating, “free to me”. So, the team worked to ensure WSIW could support those types of interactions with Alexa. It proved to be a feature customers responded to enthusiastically.

The team also responded to early feedback by introducing more gradual introductions to autoplay trailers and swapped an intro video on how to use the WSIW feature with on-screen contextual hints.

“Another insight was that customers wanted to be able to view only the titles they were already entitled to — versus those for rent or purchase — so we added a permanent free-to-me filter. Customers routinely call that out as a highlight,” Dutta Sharma said.

Building AI for the entertainment space

The What Should I Watch experience builds upon existing Alexa natural language understanding and automatic speech recognition capabilities.

"But bringing natural conversation to the entertainment domain has its own set of unique challenges," Laslau explained. Maybe a show, like The Boys or The Expanse, is ambiguously named, or a movie starts to trend that wasn't in the catalog a week or two ago. Optimizing the feature required combining core advances in AI around natural, multi-turn conversations with a fast-changing catalog.

"We are making sure those natural conversations are intelligent enough to reflect the very latest of what's happening in entertainment," he said.

The team also worked to ensure a mix of personalization based on your preferences— those British detective series you always gravitate toward — and something new that you might not have seen otherwise.

They did this by customizing Fire TV's existing recommender technology, mixing personalization with popular titles and randomizing subsets of these lists so that viewers encounter fresh ideas each time they turn on the TV.

A flywheel effect on innovation

The deep-learning-based Alexa Conversations makes it far simpler to develop the thousands of potential dialogue turns that a “What Should I Watch?” utterance might generate.

Alexa Conversations comprises three models: entity recognition (identifying Tom Cruise as an actor, for example), action prediction (utilizing the “movie searching” API to find movies), and argument filling (indicating the movies to be those with Tom Cruise).

“Alexa Conversations is designed to reduce the burden on developers, generating variations of dialogue automatically. The team has added several new features recently,” said Jiun-Yu Kao, an applied scientist within the Alexa AI Natural Understanding organization.

The WSIW experience is the first to launch with enhanced understanding of screen context.
Jiun-Yu Kao

Those include conversational Q&A which allow customers to ask broad questions about the recommended titles, such as which movies won an Oscar; a context reset function that allows a user to "start over" with a blank slate; and visual context, which enhances Alexa’s ability to respond correctly when a viewer says something like, "play the one on the left,” referencing what’s on the screen instead of naming the movie title.

“The WSIW experience is the first to launch with enhanced understanding of screen context,” Kao said. “It is also the first to combine all above-listed features for improved customer experience.”

Alexa and Fire TV science, engineering, and product teams collaborated to build the different components of the new feature.

Related content
A behind-the-scenes look at the unique challenges the engineering teams faced, and how they used scientific research to drive fundamental innovation to overcome those challenges.

“What’s super cool is that we are tapping into so many different services in parts of Alexa and Fire TV,” said Carlos Mattoso, a Fire TV software development engineer. “We are using a lot of the domain knowledge and capabilities that Fire TV has built around the recommendation space, for instance. But where we do that, we’re also trying to raise the bar: How can we use the information we’re gleaning from usage of What Should I Watch back into the system so that we have this flywheel that continuously improves?”

Mattoso noted that work with the Alexa team enabled not just suggestions but new in-context commands for Fire TV playback and volume changes, for example, that weren’t previously available.

“For instance, when we were building the first beta, we did not really have a way of initiating playback of a title from within an Alexa skill for Fire TV,” he explained. “So, we worked together with the Alexa Video team to extend the existing capability and then add support for that feature so that we could use it on WSIW.”

Looking ahead

Teams continue to work on making What Should I Watch faster and smarter.

One possibility is for users to explicitly guide Alexa by saying something like, "I'm a big sci-fi fan," or "I don't like horror movies." This type of interaction represents an opportunity for Alexa to adapt to customer engagement preferences, with some preferring to guide the service directly, and others wanting to lean back and take in recommendations.

As collaboration on the experience continues, both Alexa and Fire TV are becoming more capable. That could have a broader effect, particularly for the Alexa skill development community.

“We’re really trying to raise the bar,” Mattoso said, “and the capabilities we develop may eventually benefit third-party skill developers. Those might include improved long-term memory, better context resetting, and better visual context understanding.”

Related content

US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Do you have a strong machine learning background and want to help build new speech and language technology? Amazon is looking for PhD students who are ready to tackle some of the most interesting research problems on the leading edge of natural language processing. We are hiring in all areas of spoken language understanding: NLP, NLU, ASR, text-to-speech (TTS), and more! A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will develop and implement novel scalable algorithms and modeling techniques to advance the state-of-the-art in technology areas at the intersection of ML, NLP, search, and deep learning. You will work side-by-side with global experts in speech and language to solve challenging groundbreaking research problems on production scale data. The ideal candidate must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon has positions available for Natural Language Processing & Speech Intern positions in multiple locations across the United States. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. Please visit our website to stay updated with the research our teams are working on: https://www.amazon.science/research-areas/conversational-ai-natural-language-processing
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. The Research team at Amazon works passionately to apply cutting-edge advances in technology to solve real-world problems. Do you have a strong machine learning background and want to help build new speech and language technology? Do you welcome the challenge to apply optimization theory into practice through experimentation and invention? Would you love to help us develop the algorithms and models that power computer vision services at Amazon, such as Amazon Rekognition, Amazon Go, Visual Search, etc? At Amazon we hire research science interns to work in a number of domains including Operations Research, Optimization, Speech Technologies, Computer Vision, Robotics, and more! As an intern, you will be challenged to apply theory into practice through experimentation and invention, develop new algorithms using mathematical programming techniques for complex problems, implement prototypes and work with massive datasets. Amazon has a culture of data-driven decision-making, and the expectation is that analytics are timely, accurate, innovative and actionable. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Amazon Scientist use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science.
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. The Research team at Amazon works passionately to apply cutting-edge advances in technology to solve real-world problems. Do you have a strong machine learning background and want to help build new speech and language technology? Do you welcome the challenge to apply optimization theory into practice through experimentation and invention? Would you love to help us develop the algorithms and models that power computer vision services at Amazon, such as Amazon Rekognition, Amazon Go, Visual Search, etc? At Amazon we hire research science interns to work in a number of domains including Operations Research, Optimization, Speech Technologies, Computer Vision, Robotics, and more! As an intern, you will be challenged to apply theory into practice through experimentation and invention, develop new algorithms using mathematical programming techniques for complex problems, implement prototypes and work with massive datasets. Amazon has a culture of data-driven decision-making, and the expectation is that analytics are timely, accurate, innovative and actionable. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Amazon Scientist use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science.
CA, ON, Toronto
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Are you a Masters student interested in machine learning, natural language processing, computer vision, automated reasoning, or robotics? We are looking for skilled scientists capable of putting theory into practice through experimentation and invention, leveraging science techniques and implementing systems to work on massive datasets in an effort to tackle never-before-solved problems. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Our scientists use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science.
CA, ON, Toronto
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Are you a PhD student interested in machine learning, natural language processing, computer vision, automated reasoning, or robotics? We are looking for skilled scientists capable of putting theory into practice through experimentation and invention, leveraging science techniques and implementing systems to work on massive datasets in an effort to tackle never-before-solved problems. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Our scientists use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science.
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. We are looking for Masters or PhD students excited about working on Automated Reasoning or Storage System problems at the intersection of theory and practice to drive innovation and provide value for our customers. AWS Automated Reasoning teams deliver tools that are called billions of times daily. Amazon development teams are integrating automated-reasoning tools such as Dafny, P, and SAW into their development processes, raising the bar on the security, durability, availability, and quality of our products. AWS Automated Reasoning teams are changing how computer systems built on top of the cloud are developed and operated. AWS Automated Reasoning teams work in areas including: Distributed proof search, SAT and SMT solvers, Reasoning about distributed systems, Automating regulatory compliance, Program analysis and synthesis, Security and privacy, Cryptography, Static analysis, Property-based testing, Model-checking, Deductive verification, compilation into mainstream programming languages, Automatic test generation, and Static and dynamic methods for concurrent systems. AWS Storage Systems teams manage trillions of objects in storage, retrieving them with predictable low latency, building software that deploys to thousands of hosts, achieving 99.999999999% (you didn’t read that wrong, that’s 11 nines!) durability. AWS storage services grapple with exciting problems at enormous scale. Amazon S3 powers businesses across the globe that make the lives of customers better every day, and forms the backbone for applications at all scales and in all industries ranging from multimedia to genomics. This scale and data diversity requires constant innovation in algorithms, systems and modeling. AWS Storage Systems teams work in areas including: Error-correcting coding and durability modeling, system and distributed system performance optimization and modeling, designing and implementing distributed, multi-tenant systems, formal verification and strong, practical assurances of correctness, bits-IOPS-Watts: the interplay between computation, performance, and energy, data compression - both general-purpose and domain specific, research challenges with storage media, both existing and emerging, and exploring the intersection between storage and quantum technologies. As an Applied Science Intern, you will work closely with Amazon scientists and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A successful candidate will be a self-starter with strong attention to detail and the ability to thrive in a fast-paced, ever-changing environment who is comfortable with ambiguity. Amazon believes that scientific innovation is essential to being the world’s most customer-centric company. Our ability to have impact at scale allows us to attract some of the brightest minds in Automated Reasoning and related fields. Our scientists work backwards to produce innovative solutions that delight our customers. Please visit https://www.amazon.science (https://www.amazon.science/) for more information.
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. We are looking for PhD students excited about working on Automated Reasoning or Storage System problems at the intersection of theory and practice to drive innovation and provide value for our customers. AWS Automated Reasoning teams deliver tools that are called billions of times daily. Amazon development teams are integrating automated-reasoning tools such as Dafny, P, and SAW into their development processes, raising the bar on the security, durability, availability, and quality of our products. AWS Automated Reasoning teams are changing how computer systems built on top of the cloud are developed and operated. AWS Automated Reasoning teams work in areas including: Distributed proof search, SAT and SMT solvers, Reasoning about distributed systems, Automating regulatory compliance, Program analysis and synthesis, Security and privacy, Cryptography, Static analysis, Property-based testing, Model-checking, Deductive verification, compilation into mainstream programming languages, Automatic test generation, and Static and dynamic methods for concurrent systems. AWS Storage Systems teams manage trillions of objects in storage, retrieving them with predictable low latency, building software that deploys to thousands of hosts, achieving 99.999999999% (you didn’t read that wrong, that’s 11 nines!) durability. AWS storage services grapple with exciting problems at enormous scale. Amazon S3 powers businesses across the globe that make the lives of customers better every day, and forms the backbone for applications at all scales and in all industries ranging from multimedia to genomics. This scale and data diversity requires constant innovation in algorithms, systems and modeling. AWS Storage Systems teams work in areas including: Error-correcting coding and durability modeling, system and distributed system performance optimization and modeling, designing and implementing distributed, multi-tenant systems, formal verification and strong, practical assurances of correctness, bits-IOPS-Watts: the interplay between computation, performance, and energy, data compression - both general-purpose and domain specific, research challenges with storage media, both existing and emerging, and exploring the intersection between storage and quantum technologies. As an Applied Science Intern, you will work closely with Amazon scientists and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A successful candidate will be a self-starter with strong attention to detail and the ability to thrive in a fast-paced, ever-changing environment who is comfortable with ambiguity. Amazon believes that scientific innovation is essential to being the world’s most customer-centric company. Our ability to have impact at scale allows us to attract some of the brightest minds in Automated Reasoning and related fields. Our scientists work backwards to produce innovative solutions that delight our customers. Please visit https://www.amazon.science (https://www.amazon.science/) for more information.
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Help us develop the algorithms and models that power computer vision services at Amazon, such as Amazon Rekognition, Amazon Go, Visual Search, and more! We are combining computer vision, mobile robots, advanced end-of-arm tooling and high-degree of freedom movement to solve real-world problems at huge scale. As an intern, you will help build solutions where visual input helps the customers shop, anticipate technological advances, work with leading edge technology, focus on highly targeted customer use-cases, and launch products that solve problems for Amazon customers. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. You will own the design and development of end-to-end systems and have the opportunity to write technical white papers, create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Amazon Scientist use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Help us develop the algorithms and models that power computer vision services at Amazon, such as Amazon Rekognition, Amazon Go, Visual Search, and more! We are combining computer vision, mobile robots, advanced end-of-arm tooling and high-degree of freedom movement to solve real-world problems at huge scale. As an intern, you will help build solutions where visual input helps the customers shop, anticipate technological advances, work with leading edge technology, focus on highly targeted customer use-cases, and launch products that solve problems for Amazon customers. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. You will own the design and development of end-to-end systems and have the opportunity to write technical white papers, create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Amazon Scientist use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science
US, WA, Seattle
To ensure a great internship experience, please keep these things in mind. This is a full time internship and requires an individual to work 40 hours a week for the duration of the internship. Amazon requires an intern to be located where their assigned team is. Amazon is happy to provide relocation and housing assistance if you are located 50 miles or further from the office location. Are you a Masters or PhD student interested in machine learning? We are looking for skilled scientists capable of putting Machine Learning theory into practice through experimentation and invention, leveraging machine learning techniques (such as random forest, Bayesian networks, ensemble learning, clustering, etc.), and implementing learning systems to work on massive datasets in an effort to tackle never-before-solved problems. A successful candidate will be a self-starter comfortable with ambiguity, strong attention to detail, and the ability to work in a fast-paced, ever-changing environment. As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to create technical roadmaps, and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists, and other science interns to develop solutions and deploy them into production. The ideal scientist must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. Amazon Science gives insight into the company’s approach to customer-obsessed scientific innovation. Amazon fundamentally believes that scientific innovation is essential to being the most customer-centric company in the world. It’s the company’s ability to have an impact at scale that allows us to attract some of the brightest minds in artificial intelligence and related fields. Our scientists use our working backwards method to enrich the way we live and work. For more information on the Amazon Science community please visit https://www.amazon.science.