Image shows Amazon science intern Michael Saxon standing in front of two office buildings
Michael Saxon, an Amazon science intern, is completing his PhD in computer science at the University of California, Santa Barbara, with a core focus on natural language processing.

“Alexa, how do you know everything?”

How Amazon intern Michael Saxon uses his experience with automatic speech recognition models to help Alexa answer complex queries.

“Alexa, play ‘Rhapsody in Blue’.”

“Playing ‘Rhapsody in Blue’.”

Customers often describe this kind of interaction with Alexa as magical; less than a decade ago it would have seemed fanciful.

A black and white profile shot of the Nobel Prize-winning biologist Peter Medawar
By Digitised for CODEBREAKERS, MAKERS OF MODERN GENETICS
The Nobel Prize-winning biologist Peter Medawar published "Advice to a Young Scientist" in 1979. Here are some of Medawar’s key insights from the book.

One component of the science behind Alexa is automatic speech recognition — the process that Alexa utilizes to interpret semantic meaning from a speech signal. And scientists like Michael Saxon, PhD student and three-time Amazon applied science intern, encounter interesting challenges when a customer’s request is more complex than asking for a song to play.

Saxon is one of more than 10,000 interns Amazon hosted virtually this summer. More than 10 percent of those internships were for applied science and data science roles with teams across the company. The majority of science-related internships run between 12 and 16 weeks.

A growing interest in NLP

Saxon completed his undergraduate degree in electrical engineering and received a master’s in computer engineering at Arizona State University. He’s now completing his PhD in computer science at the University of California, Santa Barbara, with a core focus on natural language processing (NLP).

He became interested in speech and NLP as an undergrad; in his final year, a professor recruited him for a project. Saxon studied the progression of neurological disorders by using automatic speech recognition models to detect and track hypernasality in dysarthric speech

Saxon later met some Amazon recruiters who were looking for applied science interns at the AAAI Conference on Artificial Intelligence. “Based on my interests in speech and NLP, they offered for me to join the Alexa Hybrid Science team in Pittsburgh,” Saxon says. “And my experience with automatic speech recognition models was a plus.”

Solving end-to-end SLU

A core research direction of the Alexa Hybrid Science team has been the development of neural end-to-end spoken language understanding (SLU) models. For his 2019 internship project, Saxon was given a task that seemed relatively easy to him at the outset: develop an end-to-end intent SLU system that can make a decision after hearing as few words as possible.

However, he found the project proved to be deceptively difficult. Using training data, Saxon and the team were unable to replicate high-performance results from prior SLU publications.

Toward the end of the summer 2019 internship, the team identified the reason why. There was a mismatch between levels of semantic complexity in the training data and the publicly available datasets from the existing literature.

Semantic complexity refers to the number of possible expressions and their various meanings that a collection of language data contains. The more semantically complex the collection, the more ways a program can interpret a single utterance from it.

Due to their relatively low semantic complexity, the publicly available datasets required less training data and ultimately restricted the research systems to choose from a fixed list of predetermined exact command permutations.

Saxon’s team applied the model architecture from the existing literature to Amazon’s training data, which has much higher semantic complexity.

“We found for similarly sized datasets, and similar architectures, that we couldn’t reproduce these strong results from prior work, and we suspected that it was due to this semantic-complexity mismatch,” says Saxon. “The models were fundamentally designed for domains with lower semantic complexity.”

However, this setback in his first internship project inspired the direction for the next one.

Getting results

When Saxon returned to the Alexa Hybrid Science team for his second internship in January 2020, the team hit the ground running. While he was finishing his master’s coursework at ASU, the team began a research effort toward demonstrating usable measures of semantic complexity to facilitate objective comparisons of SLU tasks.

To produce useful measures, the team needed to compare the relationship between an SLU task’s complexity measures and the accuracy they could achieve with a model if they applied it to different datasets, each less semantically complex than the last.

The team artificially generated datasets of different levels of semantic complexity by repeatedly removing batches of rare words. This led to a continuum of virtual SLU problems ranging from Alexa-level tasks in large artificial datasets to effectively spotting keywords from a short list.

Michael Saxon and team published their findings on the importance of contextualizing results to demonstrate an SLU system’s scope of applicability in “Semantic Complexity in End-to-End Spoken Language Understanding”.

“There is a strong, nearly linear relationship between these semantic complexity measures and the maximum accuracy we were able to get across several different models,” Saxon says. “So that suggests that there is a fundamental relationship between a given model’s performance ceiling and the semantic complexity of the task it solves.”

Saxon and team published their findings on the importance of contextualizing results to demonstrate an SLU system’s scope of applicability in “Semantic Complexity in End-to-End Spoken Language Understanding” and presented them at Interspeech 2020.  

Considering the challenges of semantic complexity, the team then set out to develop an end-to-end model for generalized SLU that could enable voice assistants like Alexa to process any utterance with improved accuracy over other models.

"End-to-end spoken language understanding for generalized voice assistants" presents an approach to developing an E2E model for generalized SLU in commercial voice assistants.

The result: a second publication, “End-to-End Spoken Language Understanding for Generalized Voice Assistants.” The team produced an end-to-end SLU system that could both be pretrained on speech and accept the drop-in insertion of a large language model. This allowed the team to separately adjust the system’s transcription and interpretation capabilities.

Consequently, the system could process many more combinations of intent and argument interpretations. Of note, the SLU system’s speech-to-interpretation accuracy achieved a 43 percent improvement over similarly capable end-to-end baselines.

Answering any question using the web

This summer, Saxon is completing his third applied science internship at Amazon, working remotely for the Alexa AI team in Manhattan Beach, Calif. The team’s work focuses on getting Alexa to provide highly accurate responses to customers’ questions. 

“I’ve been on this journey where I've started on the speech side of things and transitioned further down the technology stack to where I am now in the web information domain, where there are still echoes of this previous work,” explains Saxon.

Michael’s internship helped us build substantial expertise and reach the level of maturity that we have in the team today in end-to-end SLU.
Athanasios Mouchtaris

The challenge this time involves an even more semantically complex use case: the Alexa AI team needs to train web information–based models that can correctly answer any possible question — even the most confounding ones — so that Alexa can provide useful responses to customers’ questions.

Often, the most important words in a question sentence that an ASR system needs to transcribe correctly are very rare. They increase the sentence’s semantic complexity and are also the hardest words for the system to transcribe.

Without correctly hearing one of those words, the system won’t be able to answer the question. Saxon’s current work brings his previous experiences in end-to-end SLU to bear on this task.

“Michael’s internship helped us build substantial expertise and reach the level of maturity that we have in the team today in end-to-end SLU,” says his former manager, Athanasios Mouchtaris. “Everything we learned from Michael’s work during his internship was crucial to our success.”

Looking ahead

Having only completed the first year of his PhD, Saxon is still in an exploratory phase of finding a research direction. He has four years left of his PhD and intends to complete additional internships — and he said he can see himself returning to Amazon again.

“I’ve really bought into the leadership principles and culture here. And I particularly like the emphasis on taking ownership and ‘disagree and commit,’ which have served me well during these research projects,” he says. “I would definitely consider coming back for full-time work after I graduate.”

Amazon hosted more than 10,000 interns virtually this summer. If you’re a student with interest in an Amazon internship, you can learn more about internship opportunities at Amazon Student Programs.

Related content

US, WA, Seattle
Job summaryEmployer: Amazon.com Services LLCPosition: Data Scientist IILocation: Seattle WAMultiple Positions Available:1. Perform analytical tasks on vast amounts of structured and unstructured data to extract actionable business insights2. Develop algorithms using advanced mathematical and statistical techniques including machine learning to predict business outcomes and recommend optimal actions to management3. Use advanced algorithms to solve business problems, and transform one-off models into automated systems4. Analyze and validate data to ensure high data quality and reliable insights5. Run analytical experiments in a methodical manner to find opportunities for product and process optimization6. Communicate analytical output and business insights to management using visualization techniques and data storytelling7. Write concise documents communicating results to stakeholders and visualize data to drive decision making#0000
US, VA, Arlington
Job summaryAmazon is looking for a passionate, talented, and inventive Research Scientists with a strong machine learning background to help build industry-leading Speech and Language technology. This includes delivering innovative HR experiences for our employees. To get there, we need exceptionally talented, bright, and driven people. The Employee Services Technology (ES Tech) is growing rapidly and is looking for a talented senior research scientist to create massive scale and powerful new solutions that our customers need to be successful. In this role, you will work to establish world class data science, analytics and reporting for Amazonians as part of building the Personalization Engine for myHR. This key role will work closely with internal partners to assist in developing and managing solutions for ES Tech. We are looking for candidates who want to invent the future for employee engagement.This role will play critical part in forecasting initiatives, responsible for building models and prototypes for ES Tech division, and will require close collaboration with other scientists on the team that are developing state-of-the-art optimization algorithms to scale. This team plays a significant role in various stages of the innovation pipeline from identifying employee needs, developing new algorithms, prototyping/simulation, to implementation by working closely with colleagues in engineering, product management and employee benefits.We are seeking a Senior Research Scientist with expertise in mixed-methods research, preferably in social science and behavioral research. In this role, you will lead and support research efforts within the Recruiting, Talent Management, and Leadership & Development space.You will help set the direction for science and research in the organization. You will be a thought leader on the team, partnering with a diverse set of stakeholders to identify and develop impactful areas for novel research about talent and recruiting outcomes, mechanisms, and programs. You will mentor and provide scientific expertise/peer review to other scientists and analysts on the team.The ideal candidate should have strong problem solving skills, excellent business acumen, as well as an expertise in both qualitative and quantitative methods. This role will need to navigate complex and ambiguous business challenges by asking the right questions, understanding what methodologies to employ, and communicating results to multiple audiences (e.g., technical peers, functional teams, business leaders). Translating business and stakeholder needs into realistic and actionable scientific research will be a regular challenge in this role.Responsibilities include:· Leading scientific direction, providing consult, mentorship, and peer review.· Partnering closely and driving effective collaborations across multi-disciplinary science, analytics, HR, and business teams. Reviewing and scoping research requests, and recommending appropriate scientific methodologies.· Designing, developing, and executing qualitative and quantitative data collection, research, and experimentation. Strong experience with statistical analysis required. Experience with survey development and experimental design (including quasi-experimental) preferred.· Communicating findings and business impact effectively (written and verbally) with both technical and non-technical stakeholders
US, VA, Arlington
Job summaryAmazon is looking for a passionate, talented, and inventive Research Scientists with a strong machine learning background to help build industry-leading Speech and Language technology. This includes delivering innovative HR experiences for our employees. To get there, we need exceptionally talented, bright, and driven people. The Employee Services Technology (ES Tech) is growing rapidly and is looking for a talented senior research scientist to create massive scale and powerful new solutions that our customers need to be successful. In this role, you will work to establish world class data science, analytics and reporting for Amazonians as part of building the Personalization Engine for myHR. This key role will work closely with internal partners to assist in developing and managing solutions for ES Tech. We are looking for candidates who want to invent the future for employee engagement.This role will play critical part in forecasting initiatives, responsible for building models and prototypes for ES Tech division, and will require close collaboration with other scientists on the team that are developing state-of-the-art optimization algorithms to scale. This team plays a significant role in various stages of the innovation pipeline from identifying employee needs, developing new algorithms, prototyping/simulation, to implementation by working closely with colleagues in engineering, product management and employee benefits.We are seeking a Senior Research Scientist with expertise in mixed-methods research, preferably in social science and behavioral research. In this role, you will lead and support research efforts within the Recruiting, Talent Management, and Leadership & Development space.You will help set the direction for science and research in the organization. You will be a thought leader on the team, partnering with a diverse set of stakeholders to identify and develop impactful areas for novel research about talent and recruiting outcomes, mechanisms, and programs. You will mentor and provide scientific expertise/peer review to other scientists and analysts on the team.The ideal candidate should have strong problem solving skills, excellent business acumen, as well as an expertise in both qualitative and quantitative methods. This role will need to navigate complex and ambiguous business challenges by asking the right questions, understanding what methodologies to employ, and communicating results to multiple audiences (e.g., technical peers, functional teams, business leaders). Translating business and stakeholder needs into realistic and actionable scientific research will be a regular challenge in this role.Responsibilities include:· Leading scientific direction, providing consult, mentorship, and peer review.· Partnering closely and driving effective collaborations across multi-disciplinary science, analytics, HR, and business teams. Reviewing and scoping research requests, and recommending appropriate scientific methodologies.· Designing, developing, and executing qualitative and quantitative data collection, research, and experimentation. Strong experience with statistical analysis required. Experience with survey development and experimental design (including quasi-experimental) preferred.· Communicating findings and business impact effectively (written and verbally) with both technical and non-technical stakeholders
US, CA, San Francisco
Job summaryAbout Us:Launched in 2011, Twitch is a global community that comes together each day to create multiplayer entertainment: unique, live, unpredictable experiences created by the interactions of millions. We bring the joy of co-op to everything, from casual gaming to world-class esports to anime marathons, music, and art streams. Twitch also hosts TwitchCon, where we bring everyone together to celebrate, learn, and grow their personal interests and passions. We're always live at Twitch. Stay up to date on all things Twitch on Linkedin, Twitter and on our Blog.About the Role:Amazon Interactive Video Service (IVS) is a managed live video service, within Amazon Web Services (AWS), built on top of Twitch Video. It has one of the world's largest live video networks. IVS powers the many companies that are using live video to uniquely engage and build communities, just as it does for Twitch, where more than a billion hours are streamed live each month.The IVS Video Intelligence team, a central data team for all of IVS, is looking for a principal applied scientist to create a strategic impact for the entire IVS business. As part of the team, you will shape the data and science strategy for the whole of IVS. You will both invent new solutions and bring your knowledge of ML ops best practices to establish cross-company standards. You will oversee our novel end-to-end predictive simulation of our video streaming infrastructure to test product hypotheses, forecast demand and capacity needs, and cover failure scenarios. You will report to the Video Intelligence team lead.If you are experienced in predictive analytics and have a passion for deep technical problems at consumer web scale, come help us grow an enterprise cloud service. You will work in our San Francisco office, but we will also consider other locations including Seattle (WA), Salt Lake City (UT), or US-Virtual.You Will:• Create the organizational science strategy.• Invent new technologies to give live video broadcasters and viewers a better overall experience.• Become a domain expert real-time video streaming tracking and measurement, building trust with your product and infrastructure partners.• Mentor junior team members and promote analytics and experimentation best practices throughout the company.
US, WA, Bellevue
Job summaryAt Amazon, we're working to be the most customer-centric company on earth. To get there, we need exceptionally talented, bright, result oriented, and driven people. Amazon is seeking a Simulation Engineer to assist in designing and optimizing the middle mile concepts and process improvements using discrete event simulations. Successful candidates will be natural self-starters who have the drive to design, model, and simulate new concepts and processes. The Simulation Engineer will be expected to deep dive problems and drive relentlessly towards creative solutions. This individual needs to be comfortable interfacing and driving various functional teams and individuals at all levels of the organization in order to be successful. Perform process modelling and simulation using discrete event simulation software’s, process optimization, statistical data analysis, and Design of Experiments (DOE) etc. to drive decisions on process and designs. Need based remote work option is available.Responsibilities:· Lead system level complex Discrete Event Simulation (DES) projects to build , simulate, and optimize the fulfillment center operational process flow models using FlexSim, Demo 3D, AnyLogic or any other Discrete Event Simulation (DES) software packages· Understand process flows , analyze data, perform Design of Experiments and effectively represent in simulation model to achieve better correlation and process improvements· Manage multiple DES simulation projects and tasks simultaneously and effectively influence, negotiate, and communicate with internal and external business partners, contractors and vendors.· Facilitate process improvement initiatives among site operations, engineering, and corporate systems groups.· Utilize code (python or another object oriented language) for data analysis and modeling algorithms· Analyze historical data to identify trends and support decision making using Statistical Techniques· Lead and coordinate simulation efforts between internal teams and outside vendors to develop optimal solutions for the network, including equipment specification, material flow, process design, and site layout.· Deliver results according to project schedules and quality· Provide written and verbal presentations to share insights and recommendations to audiences of varying levels of technical sophistication.· Make technical trade-offs for long term/short-term needs considering challenges in business area by applying relevant data science disciplines, and interactions among systems.
US, PA, Pittsburgh
Job summaryAmazon is looking for passionate Applied Scientists with expertise in Machine Translation and Information retrieval to help us expand our MT and NLP technology for search.Amazon’s International Search team owns defining and delivering Amazon’s cutting-edge Search and Machine Translation initiatives to customers and cultures across multiple regions. We do this by developing our own customer-facing features, services and platforms based on customer needs, and through partnerships with Amazon technical teams around the world including Central organizations.You will work with the largest online retail search application in the world, both in terms of users, catalogue size, and computing resources, and your work will directly impact millions of our customers. You will collaborate with recognized experts in business, science and engineering.Our mission is to enable a superior experience for all of Amazon’s customers in their native language by developing and deploying state-of-the art technology and applications in Machine Translation (MT), Natural Language Processing (NLP) and Machine Learning (ML). The global scale of Amazon means true big data problems, big data opportunities, and building for global customers. Machine Translation allows Amazon to reach people in their preferred language. We are a team tackling the tough language problems that general-purpose MT cannot solve.As part of our MT research & development team, you will work on developing, deploying, maintaining and supporting our fleet of Machine Translation systems, particularly focused on Search. You will collaborate with recognized experts in science and engineering to develop novel algorithms and modeling techniques to advance the state-of-the-art in MT, search and information retrieval. You will partner closely with our engineering teams, publish scientific papers and apply for patents for your inventions. Your work will directly impact millions of our customers in the form of products and services that make use of our MT and NLP technology. You will gain hands-on experience with Amazon’s heterogeneous language data sources and large-scale computing resources to accelerate advances in MT and NLP.Key job responsibilities* Drive scientific exploration and innovation for MT improvements and evaluation techniques* Train and evaluate neural MT models using Amazon's proprietary MT technology* Analyze the accuracy and performance of MT models for search* Assist in deploying, maintaining, and supporting Amazon's fleet of dedicated MT systems in production* Design and implement software to collect, analyze and prepare large natural language data sets
DE, BY, Munich
Job summaryGlobal Workforce Staffing (WFS), a division of Amazon’s Worldwide Operations People Experience and Technology (PxT, aka Human Resources in other companies), manages Amazon’s Tier 1 talent supply chain. We attract, hire, and onboard the associates who, by fulfilling orders at the frontlines of the company, make Amazon a global leader in delivery and logistics. The Market Intelligence ensures that Amazon can deliver an industry leading customer delivery experience while raising the bar for its largest candidate and employee population--Tier 1 Associates. We ensure that Amazon can take scale globally in a consistent manner, accounting for the specific regional needs and characteristics of European countries.We are seeking a Manager, Data Science, with a heavy focus on quantitative data analysis and evaluation, and a deep focus on understanding European labor markets. You will be responsible for leading a new global expansion team from the ground up, develop research roadmaps, run experiments, and drive business impact through your research at global scale.The ideal candidate should be well versed in quantitative methods, including classical statistics and machine learning approaches. Competitive candidates will be very comfortable with at least one computational language (e.g., Python, R). Candidates should be comfortable selecting and leading their team through deployment of the best fit computational models and machine learning algorithms for analyzing their output.Candidates should have demonstrated experience leading data science and analytics projects related to labor market research and analysis, including research on wage sensitivity and elasticity, addressable workforce sizing, competition, and other factors.A day in the lifeYou will lead a team of Data Scientists and BI Engineers, and partner with Data and Software Engineers in scaling intelligence by country pertaining to market presence (site selection), launch risk, and hiring risk. You will build strong trusting relationships with key business partners to influence and drive success in their operations. You will continually coach and mentor a team to be successful in agile research iterations, working hand-in-hand with subject matter experts as humans-in-the-loop refining and ensuring the adoption of your models and products.
GB, MLN, Edinburgh
Job summaryAre you a MS or PhD student interested in a 2022 Applied Science Internship in the fields of Speech, Robotics, Computer Vision, or Machine Learning/Deep Learning?Do you enjoy diving deep into hard technical problems and coming up with solutions that enable successful products that improve the lives of people in a meaningful way?If this describes you, come join our research teams at Amazon. As an Applied Science Intern, you will have access to large datasets with billions of images and video to build large-scale machine learning systems. Additionally, you will analyze and model terabytes of text, images, and other types of data to solve real-world problems and translate business and functional requirements into quick prototypes or proofs of concept.We are looking for smart scientists capable of using a variety of domain expertise combined with machine learning and statistical techniques to invent, design, evangelize, and implement state-of-the-art solutions for never-before-solved problems.Machine Learning Science:Amazon has multiple positions available for Applied Scientists in Berlin, Munich, Tuebingen, Cambridge, Edinburgh, London, Iasi and Barcelona..A few of the teams that are hiring include:· Core AI· · Amazon Search· · AWS AI· · Advertising Technologies· · Community Shopping· · Prime VideoSpeech and Language Technology:We are hiring in all areas of spoken language understanding: ASR, NLP, NLU, text-to-speech (TTS), and Dialog Management. Amazon has multiple positions available for Speech Scientists in Aachen, Barcelona, Berlin, Cambridge, Edinburgh, Gdansk, Haifa, Tel Aviv and Turin.A few of the teams that are hiring currently include:· Alexa ML· · Alexa Brain· · Alexa Shopping· · Amazon Search· · CS TechnologyComputer Vision:Amazon has multiple positions available for Computer Vision Scientists in locations such as Berlin, Barcelona, Tuebingen, Haifa and Tel Aviv.We are currently hiring for multiple teams including:· Visual Search· Amazon AI (AWS Rekognition)· Amazon Go· Lab126
TW, TPE, Taipei
Job summaryThere is nothing COOLER than thinking and developing a brand-new IoT protocol like Sidewalk.Amazon Sidewalk is looking for a Sr. Data Scientist to help build and maintain analytics platforms in Sidewalk Network Health and Analytics team.About the RoleAs a Sr. Data Scientist, you will work closely with internal teams to improve and solve large scale analytics. You will leverage your analytical skills to provide insights on the end-to-end architecture and drive optimization. You must be willing to own, you will be responsive, flexible and able to succeed within an open collaborative peer environment. You must also be able and willing to multi-task and open to learn new technologies quickly. You are comfortable with proactive outward communication and technical leadership and you never shy away from a challenge.Ideally, you have hands-on experience with python and jupyter notebooks, you have familiarity with AWS infrastructure and running cloud operations.A successful candidate will need excellent communication skills and the ability to interact with software developers and managers that will be contributing and leveraging the analytics platform. If you are interested on building and managing large scale platforms, this position will provide you with a unique opportunity to deploy analytics at scale in a platform for real-time operations.About the team : Amazon SidewalkAmazon Sidewalk is a new long-term effort to greatly extend the working range of low-bandwidth, low-power, smart lights, sensors, and other low-cost devices customers install at the edge of their home network. We believe customers shouldn’t have to settle for connected devices that lose functionality past the front door. With Amazon Sidewalk, customers will be able to place smart devices anywhere on their property and know the devices will work, even in dead spots where Wi-Fi and Bluetooth don’t reach, helping bridge the connectivity gaps around our homes!Key job responsibilities· Broad cross-functional engineering interaction to deliver analytics real-time for Amazon Sidewalk· Strong analytical and quantitative data-analysis; use hard data and metrics to back up assumptions, recommendations, and drive actions· Drive system level integration of telemetry and analytics.
IT, Turin
Job summaryAmazon is looking for a Data Scientist with a passion for languages to join our Alexa AI - Natural Understanding Turin team. We are seeking a candidate with strong analytical skills and Natural Language Processing (NLP) experience to help us develop language components for a variety of Alexa products. Come join the Alexa team, building the speech and language solutions behind Alexa, Amazon Echo and other Amazon products and services! You will help us invent the future. As a NLP Data Scientist of the Alexa AI - NU Team, you will work close with Language Engineers to build and releases NLU models in production and improve them. You will gain hands-on experience with Amazon’s heterogeneous structured data sources; as well as large-scale computing resources to accelerate advances in training deep neural networks for natural language understanding. You will take lead on solving highly visible and impactful business problems in areas of automation, self-service solution and quality improvement to continue delight Alexa customers and help driving Amazon business performance. The ideal candidate is clearly passionate about delivering experiences that delight customers and creating solutions that are robust. Creating reliable, scalable and high performance products requires exceptional technical expertise, and a sound understanding of the fundamentals of Machine Learning, NLP, Linguistic and Problem solving. This role requires working closely with business, engineering and other scientists within the team and across Amazon to raise the bar in operational excellence, improving tools and automating workflows. You will lead high visibility and high impact programs collaborating with various teams across Amazon. You will focus on deliver results with the right quality and in a timely fashion. Your bias for action will be critical to move quickly on projects, with calculated risk taking.