Amazon Research Award recipient Shrikanth Narayanan is on a mission to make inclusive human-AI conversational experiences.
Amazon Research Award recipient Shrikanth Narayanan, university professor and Niki & C. L. Max Nikias Chair in Engineering at the University of Southern California, is on a mission to make inclusive human-AI conversational experiences.
USC

“Who we are shapes what we say and how we say it”

Amazon Research Award recipient Shrikanth Narayanan is on a mission to make inclusive human-AI conversational experiences.

To hear Shrikanth Narayanan describe it, every single human conversation is a feat of engineering — a complex system for creating and interpreting a dizzying array of signals.

“When I'm speaking, I'm producing this audio signal, which you're able to make sense out of by processing it in your auditory system and neural systems,” Narayanan says. “Meanwhile, you’re decoding my intent and emotions. I've always been fascinated by that.”

Narayanan uses signal processing and machine learning to better understand this sort of real-world information transfer as university professor and Niki & C. L. Max Nikias Chair in Engineering at the University of Southern California (USC).

In 2020, his lab earned an Amazon Research Award for work on creating “inclusive human-AI conversational experiences for children." Today, he continues to collaborate with Amazon researchers through The Center for Secure and Trusted Machine Learning at the USC Viterbi School of Engineering. He’s also gained a reputation for training future Amazon scientists, with dozens of his former students now working full time for the company.

They’re finding new approaches to machine learning privacy, security, and trustworthiness that are helping to shape a future that Narayanan hopes will be more equitable, more secure, and more empathetic.

A signal with ‘complex underpinnings’

Narayanan recalls being fascinated by the scientific side of the human experience as early as high school. At the time, he says, he was mainly interested in our physiology. But in retrospect, he says, his curiosity had the tenor of a tinkering engineer.

Related content
With little training data and no mapping of speech to phonemes, Amazon researchers used voice conversion to generate Irish-accented training data in Alexa’s own voice.

“I was always interested in how it all worked,” he says. “I wanted to know how the heart worked, what happened in the brain, how it worked together. I was looking at humans through this lens of systems — the information flow that happens within individuals and between individuals.”

It was in the early ‘90s, while he was pursuing a PhD in electrical engineering at the University of California, Los Angeles, that he managed to combine his diverse interests.
“I was training in electrical engineering, but I really wanted the chance to look at something more directly connected to those human systems,” he says. He got the chance to intern at AT&T Bell Laboratories and realized human language held all the sorts of mysteries he’d been hoping to help solve.

Related content
Alexa Fund company unlocks voice-based computing for people who have trouble using their voices.

“Human speech is a signal that has these complex underpinnings,” he says. “There’s a cognitive aspect, the mind, and motoric aspects. We use the vocal instrument to create the signal, which in turn gets processed by people.”

Narayanan was fascinated by all the data involved in helping a conversation go right — and how easily conversations can go wrong.

He also became interested in the ways developmental disorders and health conditions could change the process of creating and interpreting speech, as well as how the rich diversity of human cultural contexts could impact the efficacy of voice recognition and synthesis.

In 2000, Narayanan founded USC’s Signal Analysis and Interpretation Laboratory (SAIL) to focus “on human-centered signal and information processing that address key societal needs.”

Over the last two decades, SAIL has enabled advances in audio, speech, language, image, video and bio signal processing, human and environment sensing and imaging, and human-centered machine learning. The lab also applies their findings to create “technologies that are inclusive, and technologies that support inclusion,” Narayanan says.

Related content
In a top-3% paper at ICASSP, Amazon researchers adapt graph-based label propagation to improve speech recognition on underrepresented pronunciations.

By that, he means that in addition to making sure technologies like voice recognition actually work for everyone — some of his earliest work involved helping AI pick up on a speaker’s emotional state regardless of their spoken language — he uses signal analysis and interpretation to help uncover and spotlight inequality.

In 2017, SAIL created algorithms for analyzing movie script dialogue in order to measure representation of BIPOC characters. Another SAIL tool analyzed footage directly to track and tally female screen time and speaking time.

In 2019, the lab reported that an algorithm trained on human speech patterns could predict whether or not couples facing hard times would actually stay together. It did so even better than a trained therapist presented with video recordings of the couples in question. Instead of interpreting the content of the discussions —or any visual cues— the algorithm focused on factors like cadence and pitch. A similar tool predicted changes in mental well-being in psychiatric patients as well as human physicians could.

Building trust in AI

“Even if we speak the same language,” Narayanan says, “who we are shapes what we say and how we say it. And this is particularly fascinating for children, because their speech represents a moving target with ongoing developmental changes.”

Even if we speak the same language, who we are shapes what we say and how we say it. And this is particularly fascinating for children, because their speech represents a moving target with ongoing developmental changes.
Shrikanth Narayanan

It’s not just that a child’s vocal instrument is constantly changing as they grow. They’re also developing cognitively and socially. That can mean rapid shifts in the words they use and how they use them. When you add in other factors that might make those speech shifts different from the already diverse average —cultural contexts, speaking or hearing impairments, cognitive differences, or developmental delays — training a voice assistant to effectively communicate with kids poses a real challenge.

The analysis gets even more complicated when interacting with two humans at once, especially if one is an adult and one is a child. Using Amazon Elastic Compute Cloud (Amazon EC2) to process their data, SAIL made advances in core competences like automatic speech recognition to improve speaker diarization — the process of partitioning audio of human speech to determine which person is speaking when.

Related content
Alexa Fund company’s assisted reality tech could unlock speech for hundreds of millions of people who struggle to communicate.

In 2021, SAIL also published a detailed empirical study of children’s speech recognition. They found that the state-of-the-art end-to-end systems setting high benchmarks on adult speech had serious shortcomings when it came to understanding children. The following year, the lab proposed a novel technique for estimating a child’s age based on temporal variability in their speech.

By measuring the same aspects of speech that make children difficult for AI to interact with — like variations in pause length and the time it takes to pronounce certain sounds — his team was able to reliably measure a child’s developmental stage. That could help AI adapt to the needs of users with less sophisticated language skills. Because the analysis relies on signals that can be stripped of other identifying information, the method also has the potential to help protect a child’s privacy.

Narayanan refers to this and similar projects as “trustworthy speech processing,” and says he and collaborators he’s found through Amazon are working to spread interest in the idea across their booming field. In March, the International Speech Communication Association (ISCA) awarded him their ISCA Medal for Scientific Achievement — the group’s most prestigious award — for his sustained and diverse contributions to speech communication science and technology and its application to human-centered engineering systems. He will receive the medal and deliver the opening keynote lecture in August at Interspeech 2023, held in Dublin, Ireland.

Narayanan notes that the last five years have seen radical changes in our ability to gather and analyze information about human behavior.

Related content
Generative AI raises new challenges in defining, measuring, and mitigating concerns about fairness, toxicity, and intellectual property, among other things. But work has started on the solutions.

“The technology systems have made this sort of engineering leap and allowed applications we hadn’t even imagined yet,” he says. “All these people are interacting with these devices in open, real-world environments, and we have the machine learning and deep learning advances to actually use that audio data.”

The next big challenge, he says, is figuring out how to process that data in a way that not only serves the user, but ensures their trust. In addition to continuing to study how various developmental differences might impact voice recognition—and how AI can learn to adapt to them—Narayanan hopes to find new ways to mask as much user data as possible for privacy while pulling out the signals that voice assistants need.

Ushering in the next generation of researchers

Working with Amazon enables Narayanan’s lab to explore key research themes through a practical lens. He notes that collaborations of this nature provide academics like himself with the time and support to tackle complex, delicate research questions — such as those involving children and other vulnerable populations.

In addition, Naraynan’s graduate students get to work directly with Amazon scientists to understand the potential practical applications of their research.

“This kind of partnership really takes research to the next level,” he says.

The AI revolution that's happening has a very nice connection to what's happening at Amazon, so naturally it was a place where my students found the most exciting challenges and opportunities.
Shrikanth Narayanan

Narayanan has also encouraged dozens of his students to pursue internships at Amazon to explore what industry has to offer. Just as his time at Bell Laboratories helped to crystalize his own interests, he says, he’s watched countless young engineers find exciting new applications for their skills at Amazon.

What started as a gentle nudge to consider Amazon internships and job postings has grown into a steady pipeline of Amazon hires — one that Narayanan says owes entirely to the merits of his lab’s alums.

Angeliki Metallinou, a senior applied science manager for Alexa AI, joined Amazon fulltime in 2014 with Narayanan’s encouragement. Alexa was a top-secret project at the time, so she didn’t know exactly what she’d be working on until she got there. She credits Narayanan with encouraging her to dive in.

Related content
How he parlayed an internship to land an expanded role at Amazon while pursuing his master’s degree.

“As a student, I hadn’t realized the extent that Amazon scientists collaborate with academia and are able to publish their work at top tier venues and conferences,” she recalls. “I wasn’t even aware that there was such a strong science community here. But Shri already had a few former PhD students working at Amazon, and he recommended it as a great place for an industry career.”

Rahul Gupta, a senior applied scientist for Amazon Alexa, first connected with Amazon for an internship near the end of his SAIL PhD in 2015. These days, he says, he has one or two SAIL students doing summer internships in his group alone.

“There's really good cultural alignment between SAIL and Amazon,” Gupta says.

Narayanan, who proudly displays photos of all of his lab graduates on the wall of his office, admits he’s lost count of how many have worked at Amazon over the years.

“It's exciting,” he says. “The AI revolution that's happening has a very nice connection to what's happening at Amazon, so naturally it was a place where my students found the most exciting challenges and opportunities. But I’ve also seen many of them progress into leadership positions, which I did my best to set them up for — I always encourage creativity and collaboration, and I don’t micromanage them in my lab.”

Now that his graduates are thriving at Amazon, he says, the internship opportunities for his current students are all the more robust.

“It sustains itself,” he says. “They shine in what they do at Amazon and in the community, and that connects back to the lab. It’s incredibly exciting.”

Related content

GB, London
Our team's mission is to improve Shopping experience for customers interacting with Amazon devices via voice. We work with Alexa and multiple other teams to research and develop advanced state-of-the-art speech technologies. Do you want to be part of the team developing the latest technology that impacts the customer experience of ground-breaking products? Then come join us and make history. Key job responsibilities We are looking for a passionate, talented, and inventive Senior Applied Scientist with a background in Machine Learning to help build industry-leading Speech and Language technology. As a Senior Applied Scientist at Amazon you will work with talented peers to develop novel algorithms and modelling techniques to drive the state of the art in speech synthesis. Position Responsibilities: * Participate in the design, development, evaluation, deployment and updating of data-driven models for Speech and Language applications. * Participate in research activities including the application and evaluation of Speech and Language techniques for novel applications. * Research and implement novel ML and statistical approaches to add value to the business. * Mentor junior engineers and scientists. We are open to hiring candidates to work out of one of the following locations: London, GBR
ES, M, Madrid
Amazon's International Technology org in EU (EU INTech) is creating new ways for Amazon customers discovering Amazon catalog through new and innovative Customer experiences. Our vision is to provide the most relevant content and CX for their shopping mission. We are responsible for building the software and machine learning models to surface high quality and relevant content to the Amazon customers worldwide across the site. The team, mainly located in Madrid Technical Hub, London and Luxembourg, comprises Software Developer and ML Engineers, Applied Scientists, Product Managers, Technical Product Managers and UX Designers who are experts on several areas of ranking, computer vision, recommendations systems, Search as well as CX. Are you interested on how the experiences that fuel Catalog and Search are built to scale to customers WW? Are interesting on how we use state of the art AI to generate and provide the most relevant content? Key job responsibilities We are looking for Applied Scientists who are passionate to solve highly ambiguous and challenging problems at global scale. You will be responsible for major science challenges for our team, including working with text to image and image to text state of the art models to scale to enable new Customer Experiences WW. You will design, develop, deliver and support a variety of models in collaboration with a variety of roles and partner teams around the world. You will influence scientific direction and best practices and maintain quality on team deliverables. We are open to hiring candidates to work out of one of the following locations: Madrid, M, ESP
US, WA, Seattle
Here at Amazon, we embrace our differences. We are committed to furthering our culture of diversity and inclusion of our teams within the organization. How do you get items to customers quickly, cost-effectively, and—most importantly—safely, in less than an hour? And how do you do it in a way that can scale? Our teams of hundreds of scientists, engineers, aerospace professionals, and futurists have been working hard to do just that! We are delivering to customers, and are excited for what’s to come. Check out more information about Prime Air on the About Amazon blog (https://www.aboutamazon.com/news/transportation/amazon-prime-air-delivery-drone-reveal-photos). If you are seeking an iterative environment where you can drive innovation, apply state-of-the-art technologies to solve real world delivery challenges, and provide benefits to customers, Prime Air is the place for you. Come work on the Amazon Prime Air Team! Our Prime Air Drone Vehicle Design and Test team within Flight Sciences is looking for an outstanding engineer to help us rapidly configure, design, analyze, prototype, and test innovative drone vehicles. You’ll be responsible for developing, improving, and maintaining a suite of multi-disciplinary optimization (MDO) tools across all aircraft design disciplines. You’ll use these to explore new and novel drone vehicle conceptual designs in both focused and wide open design spaces, with the ultimate goal of meeting our customer requirements. You’ll have the opportunity to prototype vehicle designs and support wind tunnel and other testing of vehicle designs. You will directly support the Office of the Chief Program Engineer, and work closely across all vehicle subsystem teams to ensure integrated designs that meet performance, reliability, operability, manufacturing, and cost requirements. In addition, you’ll own the Flight Sciences assessments and analysis methods for the drone vehicle design as it progresses through later stages of development. About the team Our Flight Sciences Vehicle Design & Test organization includes teams that span the following disciplines: Aerodynamics, Performance, Stability & Control, Configuration & Spatial Integration, Loads, Structures, Mass Properties, Multi-disciplinary Optimization (MDO), Wind Tunnel Testing, Noise Testing, Flight Test Instrumentation, and Rapid Prototyping. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, MA, Boston
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Knowledge of applied econometrics is necessary, and experience with SQL and Python would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will build data sets and perform applied econometric analysis, collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with future job market placement. Roughly 85% of previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: Boston, MA, USA | Seattle, WA, USA
ES, M, Madrid
Amazon's International Technology org in EU (EU INTech) is creating new ways for Amazon customers discovering Amazon catalog through new and innovative Customer experiences. Our vision is to provide the most relevant content and CX for their shopping mission. We are responsible for building the software and machine learning models to surface high quality and relevant content to the Amazon customers worldwide across the site. The team, mainly located in Madrid Technical Hub, London and Luxembourg, comprises Software Developer and ML Engineers, Applied Scientists, Product Managers, Technical Product Managers and UX Designers who are experts on several areas of ranking, computer vision, recommendations systems, Search as well as CX. Are you interested on how the experiences that fuel Catalog and Search are built to scale to customers WW? Are interesting on how we use state of the art AI to generate and provide the most relevant content? Key job responsibilities We are looking for Applied Scientists who are passionate to solve highly ambiguous and challenging problems at global scale. You will be responsible for major science challenges for our team, including working with text to image and image to text state of the art models to scale to enable new Customer Experiences WW. You will design, develop, deliver and support a variety of models in collaboration with a variety of roles and partner teams around the world. You will influence scientific direction and best practices and maintain quality on team deliverables. We are open to hiring candidates to work out of one of the following locations: Madrid, M, ESP
US, WA, Bellevue
Amazon’s Modeling and Optimization Team (MOP) is looking for a passionate individual with strong optimization and analytical skills to join us in the endeavor of designing and planning the most complex supply chain in the world. The team is responsible for optimizing the global supply chain for Amazon.com and ensuring that the company is able to inbound goods from seller and vendors, transport them to their target fulfillment center, and deliver to our customers as quickly, accurately, and cost effectively as possible. We work on problems ranging from network design to inventory management, in order to support strategic decisions. It is a terrific opportunity to have a direct impact in the business while pushing the boundaries of science. Key job responsibilities We are seeking an experienced scientist who has solid background in Operations Research, Operations Management, Applied Mathematics or other similar domain. In this role, you will develop models and solution algorithms that are innovative and scalable to solve new challenges in the inventory management space. You will collaborate with other scientists across teams to create integrated solutions that improves fulfillment speed, cost, and carbon emission. You have deep understanding of business challenges and provide scientific analysis to support business decision using a range of methodologies. You will also work with engineering teams to identify new data requirements, deploy new models or simplifying existing processes. About the team https://www.aboutamazon.com/news/innovation-at-amazon/how-artificial-intelligence-helps-amazon-deliver We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, CA, Santa Clara
Do you wish to create the greatest possible worldwide impact in healthcare? We, at Amazon Health Store Tech, are working towards the best-in-class healthcare storefront to make high-quality healthcare reliable, accessible, and intuitive. Our mission is to make it dramatically easier for customers to access the healthcare products and services they need to get and stay healthy. Towards this mission, we are building the technology, products and services, that help customers find, buy, and engage with the healthcare solutions they need. We are looking to hire and develop subject-matter experts in AI who focus on data analytics, machine learning (ML), natural language understanding (NLP), and deep learning for healthcare. We target high-impact algorithmic unlocks in areas such as natural language understanding (NLU), Foundation Models, Large Language Models (LLMs), document understanding, and knowledge representation systems—all of which are of high-value to our healthcare products and services. If you are a seasoned, hands-on Principal Applied Scientist with a track record of delivering to timelines with high quality, deeply technical and innovative, we want to talk to you. You will bring AI and machine learning advancements to real-time analytics for customer-facing solutions in healthcare. You will explore, innovate, and deliver advanced ML-based technologies that involve clinical and medical data. You are a domain expert in one or more of the following areas: natural language processing and understanding (language models, transformers like BERT, GPT-3, T-5, etc.), Foundation Models and LLMs, deep learning, active learning, reinforcement learning, and bioinformatics. Key job responsibilities As an Principal Applied Scientist, you will take on challenging and ambiguous customer problems, distill customer requirements, and then deliver solutions that either leverage existing academic and medical research or utilize your own out-of-the-box but pragmatic thinking. In addition to coming up with novel solutions and prototypes, you will directly contribute to its implementation. A successful candidate has excellent technical depth, scientific vision, great implementation skills, and a drive to achieve results in a collaborative team environment. You should enjoy the process of solving real-world, open-ended problems that, quite frankly, haven’t been solved at scale anywhere before. Along the way, we guarantee you’ll get opportunities to be a fearless disruptor, prolific innovator, and a reputed problem solver—someone who truly enables machine learning and statistics to truly impact the lives and health of millions of customers. You mentor and help develop a team of Applied Scientists and SDEs and work with key leaders to guide this top talent to push the boundary of science and next generation of product. They will lead the technical implementation of our evidence-based retrieval sub-system that ingests, indexes and retrieves relevant data in different forms and from multiple sources given the customer question and context. We are open to hiring candidates to work out of one of the following locations: Santa Clara, CA, USA | Seattle, WA, USA
US, WA, Bellevue
Imagine being part of an agile team where your ideas have the potential to reach millions of customers. Picture working on cutting-edge, customer-facing solutions, where every team member is a critical voice in the decision making process. Envision being able to leverage the resources of a Fortune 500 company within the atmosphere of a start-up. Welcome to Amazon’s NCRC team. We solve complex problems in an ambiguous space, focusing on reducing return costs and improving the customer experience. We build solutions that are distributed on a large scale, positively impacting experiences for our customers and sellers. Come innovate with the NCRC team! The Net Cost of Refunds and Concessions (NCRC) team is looking for a Senior Manager Data Science to lead a team of economists, business intelligence engineers and business analysts who investigate business problems, develop insights and build models & algorithms that predict and quantify new opportunity. The team instigates and productionalizes nascent solutions around four pillars: outbound defects, inbound defects, yield optimization and returns reduction. These four pillars interact, resulting in impacts to our overall return rate, associated costs, and customer satisfaction. You may have seen some downstream impacts of our work including Amazon.com customer satisfaction badges on the website and app, new returns drop off optionality, and faster refunds for low cost items. In this role, you will set the science vision and direction for the team, collaborating with internal stakeholders across our returns and re-commerce teams to scale and advance science solutions. This role is based in Bellevue, WA Key job responsibilities * Single threaded leader responsible for setting and driving science strategy for the organization. * Lead and provide coaching to a team of Scientists, Economists, Business Intelligence Engineers and Business Analysts. * Partner with Engineering, Product and Machine Learning leaders to deliver insights and recommendations across NCRC initiatives. * Lead research and development of models and science products powering return cost reduction. * Educate and evangelize across internal teams on analytics, insights and measurement by writing whitepapers, knowledge documentation and delivering learning sessions. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Bellevue
We are designing the future. If you are in quest of an iterative fast-paced environment, where you can drive innovation through scientific inquiry, and provide tangible benefit to hundreds of thousands of our associates worldwide, this is your opportunity. Come work on the Amazon Worldwide Fulfillment Design & Engineering Team! We are looking for an experienced and Research Scientist with background in Ergonomics and Industrial Human Factors, someone that is excited to work on complex real-world challenges for which a comprehensive scientific approach is necessary to drive solutions. Your investigations will define human factor / ergonomic thresholds resulting in design and implementation of safe and efficient workspaces and processes for our associates. Your role will entail assessment and design of manual material handling tasks throughout the entire Amazon network. You will identify fundamental questions pertaining to the human capabilities and tolerances in a myriad of work environments, and will initiate and lead studies that will drive decision making on an extreme scale. .You will provide definitive human factors/ ergonomics input and participate in design with every single design group in our network, including Amazon Robotics, Engineering R&D, and Operations Engineering. You will work closely with our Worldwide Health and Safety organization to gain feedback on designs and work tenaciously to continuously improve our associate’s experience. Key job responsibilities - Collaborating and designing work processes and workspaces that adhere to human factors / ergonomics standards worldwide. - Producing comprehensive and assessments of workstations and processes covering biomechanical, physiological, and psychophysical demands. - Effectively communicate your design rationale to multiple engineering and operations entities. - Identifying gaps in current human factors standards and guidelines, and lead comprehensive studies to redefine “industry best practices” based on solid scientific foundations. - Continuously strive to gain in-depth knowledge of your profession, as well as branch out to learn about intersecting fields, such as robotics and mechatronics. - Travelling to our various sites to perform thorough assessments and gain in-depth operational feedback, approximately 25%-50% of the time. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA