International Women's Day 2020.png
Credit: Glynis Condon

Seven Amazon scientists shaping the future of AI

To commemorate International Women’s Day, we spoke to women scientists across a variety of research areas at Amazon.

To commemorate International Women’s Day (IWD), during Women's History Month, we asked scientists across a variety of Amazon research areas about their backgrounds, and the most exciting innovations in their fields. Here’s what they had to say.

Xin Luna Dong, principal scientist

Xin Luna Dong
Xin Luna Dong, principal scientist

Dong is a principal scientist, leading the efforts to develop the Amazon Product Knowledge Graph. Dong received her PhD in computer science, with a focus on data integration, from the University of Washington. The personal information management system in Dong’s dissertation (which won the Best Demo award in Sigmod’ 2005), is a personal knowledge graph developed at least five years before the phrase “knowledge graph” was coined. After graduation, Dong led the development of the Knowledge-based Trust project at Google. Dong has co-authored the book Big Data Integration. She is an ACM Distinguished Member, and has received the VLDB Early Career Research Contribution Award for "advancing the state of the art of knowledge fusion”. Dong was program committee co-chair for Sigmoid 2018, and is program committee co-chair for VLDB 2020. She also serves on the VLDB endowment and PVLDB advisory committees.

Innovations I find exciting

A recent innovation that I’m most excited about is graph neural network (GNN). Unlike recurrent neural networks (RNNs) and convolutional neural networks (CNNs), which focus much more on regular data such as word sequences, 2-D images, and 3-D videos, GNNs allow us to leverage graphs to capture much more complex relationships. These include elements in the graph, represented by nodes in the graph, and their relationships, represented by edges between the nodes. Examples of graphs influenced by GNNs include social networks, world wide web (WWW) topology, knowledge graphs, and molecular graphs. As we build knowledge graphs for products, it is amazing how many different ways we can benefit from GNNs.

Naturally, we can apply GNNs on the knowledge graphs we have built, to discern interesting patterns to find popular artists in the music domain. We also model webpage layouts as graphs, and model customer behaviors as graphs, so GNNs help us extract relevant knowledge and enrich our knowledge graphs. This new technique enables us to be so much more creative in the practice of constructing knowledge graphs, and applying the findings to real-world applications.

Claire Law, senior technical program manager

ClaireLaw.jfif
Claire Law, senior technical program manager

Law is senior technical program manager on Amazon’s physical retail team, the team behind the Just Walk Out technology used in Amazon Go and Amazon Go Grocery stores. She studied nanotechnology engineering at the University of Waterloo. Early on, she realized that she didn’t enjoy the type of lab work expected from a researcher in material science. She leveraged a university program, and interned as a software developer, marketer, hardware test engineer, project control officer in consulting, and software program manager. These experiences, coupled with work experiences at Microsoft and Research in Motion, led Law to pursue a career in software.

After a stint in Amazon’s international organization, Law joined the physical retail team to work on machine vision initiatives. On this team, Law is able to leverage her experience in cloud computing and knowledge of optics and photography to build new experiences for physical retail.

Innovations I find exciting

We are only now reaching a level where computer vision can solve real-world problems in a meaningful way. While we still need to be creative in where we look for simplifiers, algorithms are able to solve more and more problems every day. Challenges that looked insurmountable just a couple years ago are now part of production systems across the industry. Checkout-free stores seemed like science fiction before Amazon Go was launched, and now customers are loving this effortless shopping experience in the 25 Amazon Go stores, and the new Amazon Go Grocery store we have open today.

Yoelle Maarek, vice president

Yoelle Maarek, vice president of research and science for Alexa Shopping
Yoelle Maarek, vice president of research and science for Alexa Shopping

Maarek is vice president of research and science, Alexa Shopping. Prior to Amazon, Maarek served in engineering and research leadership roles at Yahoo, Google and IBM. Maarek has been regularly serving as program committee (PC) chair and senior PC committee member at leading academic research conferences related to Web search and data mining, such as SIGIR, The Web Conference, and Web Search and Data Mining (WSDM). She is currently serving on the steering committees of WSDM and the Web Conference series.

She is a member of the Technion Board of Governors and was inducted as an ACM Fellow in 2013. Maarek obtained a PhD in computer science from the Technion, Israel in 1989. She holds an engineering degree from the Ecole des Ponts et Chaussées, and a DEA (graduate degree) in computer science from Paris VI university. Maarek completed her PhD at the Technion in Israel and was a visiting student at Columbia University. She played a pioneering role within industry in researching the field of information retrieval, the computer science discipline behind search, in the pre-Web era, and led the launch of Google Suggest, the query auto-completion capability. As such, she jokingly refers to herself as a “search dinosaur”.

Innovations I find exciting

We are on the verge of making ambient computing happen, and Alexa is pioneering this long-awaited revolution. It forces us to revisit all our assumptions across multiple domains. I see this prevalent especially in search and question answering. These are topics close to my heart. I have been following progress in these areas since I got my PhD thirty years ago. The focus on ambient computing is also a unique opportunity for us at Amazon to demonstrate what we mean by customer-obsessed science. As humans are learning to interact with machines, their behavior is evolving and we need to follow suit. It not only challenges scientists to keep inventing on behalf of customers but also forces all of us to remain humble. We are not here to teach customers how to speak to a machine, but rather to do everything in our power to understand, satisfy and predict their needs so as to constantly wow and delight them.

Angeliki Metallinou, applied science manager

angeliki.jpeg
Angeliki Metallinou, applied science manager

Metallinou is an applied science manager within the Amazon Alexa AI Natural Understanding group. She received both her PhD, and master’s degree in electrical engineering from the University of Southern California. Her interests and experience lie in the areas of spoken and natural language understanding, dialogue systems, machine learning, deep learning, affective computing and applications for education and healthcare.

She has published papers in the areas of speech, language, dialogue, artificial intelligence and multimodal human computer interaction at leading science conferences such as Interspeech, the AAAI Conference on Artificial Intelligence, and the Association for Computational Linguistics (ACL), has served as an area chair for Interspeech 2016, and as a reviewer of papers for several science conferences.

Innovations I find exciting

It is exciting to see how new techniques in deep learning continuously push the boundaries of the state of the art in the fields of dialogue and spoken language processing. I’m very interested in advances around unsupervised, semi-supervised and transfer learning, which allow deep learning models to leverage the power of large corpora without relying on costly and time-consuming manual annotations. Pre-trained language models like BERT and GPT-2 and their use in downstream applications are just a few examples. These innovations are particularly relevant for industry applications where scalability is key.

I am also excited about recent literature in deep learning that is allowing us to develop models to perform complex tasks like higher-level reasoning, for example, over the contents of a document or an image or both, as opposed to simpler classification tasks. I’m also excited to see how these methods can have a positive impact on people through their deployment in products, especially in applications of healthcare, accessibility and education.

Priya Ponnapalli, principal deep learning scientist

Priya Ponnapalli
Priya Ponnapalli, principal deep learning scientist

Ponnapalli is a senior manager and principal deep learning scientist within the Amazon ML Solutions Lab, where she leads a global team of data scientists that help AWS customers accelerate their adoption of ML and cloud technologies across industries, from healthcare and finance to sports. As the leader of Amazon ML Solutions Lab’s sports business, Ponnapalli works with customers including National Football League (NFL), Six Nations Rugby, and Formula 1 (F1), just to name a few, to enhance the fan experience and transform sports using ML.

Ponnapalli is also a senior research affiliate at the Genomic Signal Processing Lab at the University of Utah, and a faculty member at Rutgers Business School, where she teaches ML to business leaders, and works to inspire the next generation of leaders. Prior to joining AWS, she co-founded Eigengene, a data-driven personalized medicine startup and has helped companies like Genentech and Roche establish and build data science teams. For her PhD in electrical and computer engineering at the University of Texas at Austin, Ponnapalli defined and demonstrated the higher-order generalized singular value decomposition (HO GSVD), the only framework that can create a single coherent model from multiple two-dimensional datasets by extending the GSVD from two to more than two matrices.

Innovations I find exciting

As an Amazon ML Solutions Lab scientist, I’m most excited about real-world applications of ML across industries. I’m interested in innovations to overcome challenges with small, limited datasets that companies often have to contend with. I’m also intrigued by model interpretability and explainability which are key to earning trust and spurring broad adoption. I’m passionate about making ML accessible to all, so it can be used to solve some of the most important problems we are facing, from fighting climate change to treating cancer.

Ana Pinheiro Privette, senior program manager

ana.jpg
Ana Pinheiro Privette, senior program manager

Ana Pinheiro Privette is a senior program manager with Amazon's Sustainability group. She joined the Sustainability Science and Innovation team in September 2017 as the program lead for the Amazon Sustainability Data Initiative (ASDI), a program that seeks to leverage Amazon’s scale, technology, and infrastructure to help create more global innovation for sustainability. ASDI is a Tech-for-Good project and is a joint effort between Amazon Sustainability and the AWS Open Data team focusing on democratizing access to key data and analytical capabilities to anyone working in the sustainability space.

Privette was trained as an environmental engineer and as an earth sciences researcher at the New University of Lisbon (Portugal) and at MIT. She did her doctoral research work at NASA in the Washington D.C. area and as part of her project, she spent a couple of years running scientific field work sites in Africa to support a NASA international field campaign. After spending most of her career at NASA and NOAA as a scientist, Privette led projects for the White House climate portfolio, including the Obama Climate Data Initiative and the Partnership for Resilience and Preparedness (PREP), both focused on delivering better access and use of US Federal climate data to support decision makers.

Innovations I find exciting

As part of ASDI, I work very closely with AWS customers developing applications in the space of sustainability to understand what challenges they may be experiencing and how we may accelerate sustainability research and innovation by minimizing the cost and time required to acquire and analyze large sustainability datasets. The ASDI currently works with scientific organizations like NOAA, NASA, the UK Met Office and Government of Queensland to identify, host, and deploy key datasets on the AWS Cloud, including weather observations, weather forecasts, climate projection data, satellite imagery, hydrological data, air quality data, and ocean forecast data. These datasets are publicly available to anyone.

In addition, ASDI provides cloud grants to those interested in exploring the use of AWS’ technology and scalable infrastructure to solve big, long-term sustainability challenges with this data. The dual-pronged approach allows sustainability researchers to analyze massive amounts of data in mere minutes, regardless of where they are in the world or how much local storage space or computing capacity they can access.

Nashlie Sephus, manager, applied science

sephus.jpg
Nashlie Sephus, applied scientist, Amazon Web Services machine learning team.
Credit: Terrence Wells@PoetWilliamsPhotography

Sephus is an applied scientist on AWS’s artificial intelligence team, focusing on computer vision. In this role, Sephus focuses on the fairness and accuracy of the team’s algorithms. Sephus formerly led the Amazon Visual Search team in Atlanta, which launched visual search for replacement parts on the Amazon Shopping app in June 2018. This technology was a result of former startup Partpic (Atlanta) being acquired by Amazon, for which she was the chief technology officer (CTO). Prior to working at Partpic, she received her PhD in 2014 from the School of Electrical and Computer Engineering at the Georgia Institute of Technology. She received her bachelor’s degree in computer engineering in 2007 from Mississippi State University.

Innovations I find exciting

Since the onset of machine learning and artificial intelligence, neural networks (such as convolutional neural networks (CNNs), and generative adversarial networks (GANs), etc.) and learning algorithms have always excited me. It’s being able to quickly and automatically draw patterns from data, whether it be images, video, or audio at scale, that fascinates me. Since music was my first love (along with karaoke!), music information retrieval has always been a passion of mine. These innovations, when used responsibly and fairly, are able to benefit people in their everyday activities.

Research areas

Related content

ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.