Image shows Amazon science intern Michael Saxon standing in front of two office buildings
Michael Saxon, an Amazon science intern, is completing his PhD in computer science at the University of California, Santa Barbara, with a core focus on natural language processing.

“Alexa, how do you know everything?”

How Amazon intern Michael Saxon uses his experience with automatic speech recognition models to help Alexa answer complex queries.

“Alexa, play ‘Rhapsody in Blue’.”

“Playing ‘Rhapsody in Blue’.”

Customers often describe this kind of interaction with Alexa as magical; less than a decade ago it would have seemed fanciful.

A black and white profile shot of the Nobel Prize-winning biologist Peter Medawar
By Digitised for CODEBREAKERS, MAKERS OF MODERN GENETICS
The Nobel Prize-winning biologist Peter Medawar published "Advice to a Young Scientist" in 1979. Here are some of Medawar’s key insights from the book.

One component of the science behind Alexa is automatic speech recognition — the process that Alexa utilizes to interpret semantic meaning from a speech signal. And scientists like Michael Saxon, PhD student and three-time Amazon applied science intern, encounter interesting challenges when a customer’s request is more complex than asking for a song to play.

Saxon is one of more than 10,000 interns Amazon hosted virtually this summer. More than 10 percent of those internships were for applied science and data science roles with teams across the company. The majority of science-related internships run between 12 and 16 weeks.

A growing interest in NLP

Saxon completed his undergraduate degree in electrical engineering and received a master’s in computer engineering at Arizona State University. He’s now completing his PhD in computer science at the University of California, Santa Barbara, with a core focus on natural language processing (NLP).

He became interested in speech and NLP as an undergrad; in his final year, a professor recruited him for a project. Saxon studied the progression of neurological disorders by using automatic speech recognition models to detect and track hypernasality in dysarthric speech

Saxon later met some Amazon recruiters who were looking for applied science interns at the AAAI Conference on Artificial Intelligence. “Based on my interests in speech and NLP, they offered for me to join the Alexa Hybrid Science team in Pittsburgh,” Saxon says. “And my experience with automatic speech recognition models was a plus.”

Solving end-to-end SLU

A core research direction of the Alexa Hybrid Science team has been the development of neural end-to-end spoken language understanding (SLU) models. For his 2019 internship project, Saxon was given a task that seemed relatively easy to him at the outset: develop an end-to-end intent SLU system that can make a decision after hearing as few words as possible.

However, he found the project proved to be deceptively difficult. Using training data, Saxon and the team were unable to replicate high-performance results from prior SLU publications.

Toward the end of the summer 2019 internship, the team identified the reason why. There was a mismatch between levels of semantic complexity in the training data and the publicly available datasets from the existing literature.

Semantic complexity refers to the number of possible expressions and their various meanings that a collection of language data contains. The more semantically complex the collection, the more ways a program can interpret a single utterance from it.

Due to their relatively low semantic complexity, the publicly available datasets required less training data and ultimately restricted the research systems to choose from a fixed list of predetermined exact command permutations.

Saxon’s team applied the model architecture from the existing literature to Amazon’s training data, which has much higher semantic complexity.

“We found for similarly sized datasets, and similar architectures, that we couldn’t reproduce these strong results from prior work, and we suspected that it was due to this semantic-complexity mismatch,” says Saxon. “The models were fundamentally designed for domains with lower semantic complexity.”

However, this setback in his first internship project inspired the direction for the next one.

Getting results

When Saxon returned to the Alexa Hybrid Science team for his second internship in January 2020, the team hit the ground running. While he was finishing his master’s coursework at ASU, the team began a research effort toward demonstrating usable measures of semantic complexity to facilitate objective comparisons of SLU tasks.

To produce useful measures, the team needed to compare the relationship between an SLU task’s complexity measures and the accuracy they could achieve with a model if they applied it to different datasets, each less semantically complex than the last.

The team artificially generated datasets of different levels of semantic complexity by repeatedly removing batches of rare words. This led to a continuum of virtual SLU problems ranging from Alexa-level tasks in large artificial datasets to effectively spotting keywords from a short list.

Michael Saxon and team published their findings on the importance of contextualizing results to demonstrate an SLU system’s scope of applicability in “Semantic Complexity in End-to-End Spoken Language Understanding”.

“There is a strong, nearly linear relationship between these semantic complexity measures and the maximum accuracy we were able to get across several different models,” Saxon says. “So that suggests that there is a fundamental relationship between a given model’s performance ceiling and the semantic complexity of the task it solves.”

Saxon and team published their findings on the importance of contextualizing results to demonstrate an SLU system’s scope of applicability in “Semantic Complexity in End-to-End Spoken Language Understanding” and presented them at Interspeech 2020.  

Considering the challenges of semantic complexity, the team then set out to develop an end-to-end model for generalized SLU that could enable voice assistants like Alexa to process any utterance with improved accuracy over other models.

"End-to-end spoken language understanding for generalized voice assistants" presents an approach to developing an E2E model for generalized SLU in commercial voice assistants.

The result: a second publication, “End-to-End Spoken Language Understanding for Generalized Voice Assistants.” The team produced an end-to-end SLU system that could both be pretrained on speech and accept the drop-in insertion of a large language model. This allowed the team to separately adjust the system’s transcription and interpretation capabilities.

Consequently, the system could process many more combinations of intent and argument interpretations. Of note, the SLU system’s speech-to-interpretation accuracy achieved a 43 percent improvement over similarly capable end-to-end baselines.

Answering any question using the web

This summer, Saxon is completing his third applied science internship at Amazon, working remotely for the Alexa AI team in Manhattan Beach, Calif. The team’s work focuses on getting Alexa to provide highly accurate responses to customers’ questions. 

“I’ve been on this journey where I've started on the speech side of things and transitioned further down the technology stack to where I am now in the web information domain, where there are still echoes of this previous work,” explains Saxon.

Michael’s internship helped us build substantial expertise and reach the level of maturity that we have in the team today in end-to-end SLU.
Athanasios Mouchtaris

The challenge this time involves an even more semantically complex use case: the Alexa AI team needs to train web information–based models that can correctly answer any possible question — even the most confounding ones — so that Alexa can provide useful responses to customers’ questions.

Often, the most important words in a question sentence that an ASR system needs to transcribe correctly are very rare. They increase the sentence’s semantic complexity and are also the hardest words for the system to transcribe.

Without correctly hearing one of those words, the system won’t be able to answer the question. Saxon’s current work brings his previous experiences in end-to-end SLU to bear on this task.

“Michael’s internship helped us build substantial expertise and reach the level of maturity that we have in the team today in end-to-end SLU,” says his former manager, Athanasios Mouchtaris. “Everything we learned from Michael’s work during his internship was crucial to our success.”

Looking ahead

Having only completed the first year of his PhD, Saxon is still in an exploratory phase of finding a research direction. He has four years left of his PhD and intends to complete additional internships — and he said he can see himself returning to Amazon again.

“I’ve really bought into the leadership principles and culture here. And I particularly like the emphasis on taking ownership and ‘disagree and commit,’ which have served me well during these research projects,” he says. “I would definitely consider coming back for full-time work after I graduate.”

Amazon hosted more than 10,000 interns virtually this summer. If you’re a student with interest in an Amazon internship, you can learn more about internship opportunities at Amazon Student Programs.

Related content

US, WA, Seattle
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Knowledge of econometrics, as well as basic familiarity with Python (or R, Matlab, or equivalent) is necessary, and experience with SQL would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Bellevue
As an applied scientist, you will use your experience to initiate the design, development, execution and implementation of scientific research projects. Working closely with fellow research scientists and product managers, you will use your experience in modeling, statistics, and simulation to design models of new policies, simulate their performance, and evaluate their benefits and impacts to cost, reliability, and speed of our fulfillment network. Our teams are looking for experience in network and combinatorial optimization, algorithms, data structures, statistics, and/or machine learning. This position requires superior analytical thinking, and ability to apply their technical and statistical knowledge to identify opportunities for real world applications. You should be able to mine and analyze large data, and be able to use necessary programming and statistical analysis software/tools to do so. Amazon has positions available for Research Scientists in multiple locations across the US and Canada.
US, WA, Virtual Contact Center-WA
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. About the team The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. Within the Science team, our goal is to understand the impact of changing fees on Seller (supply) and Customers (demand) behavior (e.g. price changes, advertising strategy changes, introducing new selection etc.) as well as using this information to optimize our fee structure and maximizing our long term profitability.
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
This is a unique opportunity to build technology and science that millions of people will use every day. Are you excited about working on large scale Natural Language Processing (NLP), Machine Learning (ML), and Deep Learning (DL)? We are embarking on a multi-year journey to improve the shopping experience for customers globally. Amazon Search team creates customer-focused search solutions and technologies that makes shopping delightful and effortless for our customers. Our goal is to understand what customers are looking for in whatever language happens to be their choice at the moment and help them find what they need in Amazon's vast catalog of billions of products. As Amazon expands to new geographies, we are faced with the unique challenge of maintaining the bar on Search Quality due to the diversity in user preferences, multilingual search and data scarcity in new locales. We are looking for an applied researcher to work on improving search on Amazon using NLP, ML, and DL technology. As an Applied Scientist, you will lead our efforts in query understanding, semantic matching (e.g. is a drone the same as quadcopter?), relevance ranking (what is a "funny halloween costume"?), language identification (did the customer just switch to their mother tongue?), machine translation (猫の餌を注文する). This is a highly visible role with a huge impact on Amazon customers and business. As part of this role, you will develop high precision, high recall, and low latency solutions for search. Your solutions should work for all languages that Amazon supports and will be used in all Amazon locales world-wide. You will develop scalable science and engineering solutions that work successfully in production. You will work with leaders to develop a strategic vision and long term plans to improve search globally. We are growing our collaborative group of engineers and applied scientists by expanding into new areas. This is a position on Global Search Quality team in Seattle Washington. We are moving fast to change the way Amazon search works. Together with a multi-disciplinary team you will work on building solutions with NLP/ML/DL at its core. Along the way, you’ll learn a ton, have fun and make a positive impact on millions of people. Come and join us as we invent new ways to delight Amazon customers.
US, WA, Seattle
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies to deliver game changing value to our customers. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon’s Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon’s on-line retail business. As an economist on our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design new methods and pricing strategies with the potential to deliver game changing value to our customers. This is an opportunity for a high-energy individual to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon’s on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon’s goods and services are aligned with Amazon’s corporate goals.
US, CA, San Francisco
The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals. We are seeking an experienced high-energy Economist to help envision, design and build the next generation of retail pricing capabilities. You will work at the intersection of statistical inference, experimentation design, economic theory and machine learning to design new methods and pricing strategies for assessing pricing innovations. Roughly 85% of previous intern cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. Key job responsibilities Amazon's Pricing Science and Research team is seeking an Economist to help envision, design and build the next generation of pricing capabilities behind Amazon's on-line retail business. As an economist on our team, you will will have the opportunity to work with our unprecedented retail data to bring cutting edge research into real world applications, and communicate the insights we produce to our leadership. This position is perfect for someone who has a deep and broad analytic background and is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis. We are particularly interested in candidates with research background in experimentation design, applied microeconomics, econometrics, statistical inference and/or finance. A day in the life Discussions with business partners, as well as product managers and tech leaders to understand the business problem. Brainstorming with other scientists and economists to design the right model for the problem in hand. Present the results and new ideas for existing or forward looking problems to leadership. Deep dive into the data. Modeling and creating working prototypes. Analyze the results and review with partners. Partnering with other scientists for research problems. About the team The retail pricing science and research group is a team of scientists and economists who design and implement the analytics powering pricing for Amazon's on-line retail business. The team uses world-class analytics to make sure that the prices for all of Amazon's goods and services are aligned with Amazon's corporate goals.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of interns from previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US
The Amazon Supply Chain Optimization Technology (SCOT) organization is looking for an Intern in Economics to work on exciting and challenging problems related to Amazon's worldwide inventory planning. SCOT provides unique opportunities to both create and see the direct impact of your work on billions of dollars’ worth of inventory, in one of the world’s most advanced supply chains, and at massive scale. We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. We are looking for a PhD candidate with exposure to Program Evaluation/Causal Inference. Knowledge of econometrics and Stata/R/or Python is necessary, and experience with SQL, Hadoop, and Spark would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com.
US, WA, Seattle
The Selling Partner Fees team owns the end-to-end fees experience for two million active third party sellers. We own the fee strategy, fee seller experience, fee accuracy and integrity, fee science and analytics, and we provide scalable technology to monetize all services available to third-party sellers. We are looking for an Intern Economist with excellent coding skills to design and develop rigorous models to assess the causal impact of fees on third party sellers’ behavior and business performance. As a Science Intern, you will have access to large datasets with billions of transactions and will translate ambiguous fee related business problems into rigorous scientific models. You will work on real world problems which will help to inform strategic direction and have the opportunity to make an impact for both Amazon and our Selling Partners.