A decade of NFL Next Gen Stats innovation

Every NFL game generates millions of tracking data points from 22 RFID-equipped players. Seventy-five machine learning models running on AWS process that data in under a second, transforming football into a sport where every movement is measured, modeled, and instantly analyzed.

Every snap in the NFL triggers a deluge of physical data. Twenty-two players accelerate, collide, and change direction in fractions of a second, while the ball traces a path through the controlled chaos.

Yet for most of the sport’s history, much of that complexity went unmeasured. “Football, for 100-plus years, has been a box score game: you've got yards, you've got touchdowns, you've got tackles … ,” says Mike Band, senior manager of research and analytics with NFL’s Next Gen Stats.

Those numbers could capture only a sliver of what actually unfolded on the field. Coaches pored over game recordings and made educated guesses. Fans argued from the stands and the sofa. Officials occasionally made judgment calls based on partial, often obstructed views. “Looking at box score stats, you didn’t even know which 22 players were on the field for a given play,” says Mike Lopez, senior director of NFL Football Data and Analytics.

In 2015, the NFL decided to expand beyond box scores by launching Next Gen Stats (NGS). RFID chips were placed in every set of shoulder pads and inside the football, and more than 20 ultrawideband receivers were mounted around each stadium. The system began streaming the coordinates of all 22 players (10 times a second) and the ball (25 times per second). For the first time, the league was capturing comprehensive player location data, accurate to a few inches, for every moment of every play.

RFID chips were placed in every set of shoulder pads and inside the football, and more than 20 ultrawideband receivers were mounted around each stadium.

At first, each club could access only its own tracking data. That shifted in 2018, when teams gained league-wide access, putting coaches, scouts, and analysts on common analytic footing. Also that year, the league formalized and deepened its partnership with AWS, marking the start of the gradual transformation of NGS from a tracking experiment into critical NFL infrastructure, with live broadcasts only its most visible expression.

Today, NGS underpins decision making across the league, from how clubs evaluate players and design game plans to how the NFL studies officiating, player safety, and rule changes. Every team, and much of the league itself, now works from the same continuously expanding data backbone.

But it started simply, says Band. “Our early metrics were low-hanging fruit — player separation, speed, and time to throw — easily derivable from the data we had. Modeling more-complex game metrics takes much more effort, and that’s where AWS came in.”

The first complex stat the partnership delivered, in 2018, was completion probability. It was built to answer a simple question: can the difficulty of a pass be quantified?

The answer came, in part, courtesy of an XGBoost machine learning (ML) model hosted on Amazon’s SageMaker platform. It blended the factors that shape a throw’s outcome, from quarterback pressure to throw depth, receiver separation, and sideline proximity. The model returned a single percentage that captured both likelihood and difficulty. “That became our entry point into machine learning,” Band says.

Beyond SageMaker, the NFL’s analytics work has expanded into a broad suite of AWS tools, including Amazon Quick, which the League uses to deliver real-time, interactive visualizations and answers to fans, analysts, and broadcast partners. Lopez says the members of the league’s football data analytics group “call ourselves an AWS shop.” By 2018, with league-wide access in place and AWS’s ML pipelines running, NGS began to illuminate deeper questions across the sport.

Every NFL game generates millions of raw-tracking data points, yet the raw feed is only the substrate. The real data growth comes from the models that convert coordinates into usable football insight. Pressure probability, for example, estimates how likely a defender is to affect the quarterback at each moment of a pass rush and produces more than a dozen secondary metrics.

Band estimates that NGS now produces between 500 and 1,000 stats — per play. Keeping the system responsive depends on AWS infrastructure to ingest the feed, run the models, return results within seconds for teams and broadcasters, and store the wider data trove for deeper analysis.

Overview of Amazon Quick Sight dashboard.
The NFL uses the Amazon Quick’s agentic AI and BI capabilities to deliver real-time, interactive visualizations and answers to fans, analysts, and broadcast partners. Using the capabilities of Amazon Quick, the NFL provided unprecedented access to insights on player performance and draft predictions to over a million fans with their Combine IQ, Draft IQ, and Draft IQ Assistant experiences. 

Big Data Bowl

The roots of that deeper analysis extend back to 2018, with the inaugural Big Data Bowl. Led by Lopez, it became the league’s first large-scale effort to open player-tracking data to external researchers, inviting them to tackle questions such as which defenders close space most effectively or how to predict post-throw player movement.

Structured as a months-long hackathon, the annual competition challenges participants to train ML models on historical tracking data and test their ability to generalize to unseen plays. The emphasis is increasingly on prediction — models that can anticipate what would happen next.

An early success was the 2020 development of rush yards over expectation (RYOE). The metric measures the difference between actual yards gained and expected rushing yards, or what a league-average player would be predicted to gain on the same carry, considering the location, speed, and direction of blockers and defenders. It helps contextualize how strong a given run was and, when aggregated, how well a back performed over a game or season.

NFL Big Data Bowl Explained | AWS Events

RYOE moved from the Big Data Bowl to national broadcasts quickly. Lopez recalls the moment he first saw it appear, during the 2021 NFC Championship Game between the Buccaneers and Packers: “Leonard Fournette had a good run, and immediately a graphic popped up with his rush yards over expectation. That was less than 10 months after we got the winning solution.” He adds: “I took a photo of my TV screen, and colleagues were sending me theirs. It was a proud moment.”

That pipeline has turned the Big Data Bowl into a proving ground for both ideas and data science talent. In its first decade, the Big Data Bowl has become a central part of the league’s analytics ecosystem. As then New Orleans Saints coach Sean Payton quipped in 2015 about the rise of real-time data on the sidelines, “ I think it means there are going to be more MIT grads coaching.”

Key metrics

Over the past decade, NGS has grown into a portfolio of more than 75 ML models, spanning offense, defense, special teams, and game strategy. Among those, tackle probability and defensive alerts perhaps best illustrate how raw tracking data can be converted into clearer insights for teams, broadcasters, and fans.

Tackle probability estimates the likelihood of a defender completing a tackle at the moment of contact, factoring in speed, angle, distance, leverage, and pursuit. That data allows NGS to identify true tackle opportunities, quantify missed tackles, and calculate the yards a defender saves or concedes.

Defensive alerts assess defensive alignment and movement before the snap to predict which players are likely to rush. The model uses acceleration patterns and presnap shifts, combines them with situational context such as down, distance, and game state, and then applies generative AI to predict likely rushers, who are highlighted with red circles for viewers.

“Defensive alerts had a big impact, from a broadcast perspective,” says Dashiell Flynn, AWS’s principal sports consultant. He highlights how the model exposes deliberate misdirection: “Sometimes the prediction is wrong because the defense itself is using misdirection, trying to trick the offense into thinking a blitz is coming.” Those moments give game commentators a natural way to discuss disguised defensive pressure and the intent behind it.

Together, these metrics show how NGS models can turn fast, ambiguous moments into clear visual and tactical explanations.

Overview of how Next Gen Stats uses data to make accurate predictions.

Player safety and rule changes

The same tracking foundation that fuels performance analysis also gives the league clearer visibility into player safety. By capturing every player’s speed, spacing, and movement, it gives the league a concrete understanding of the dynamics behind plays long considered risky.

The new dynamic kickoff, introduced for the 2024 season, is a clear example. Kickoffs were producing too many dangerous, high-speed collisions. NGS helped quantify and ultimately change that.

“The season before, we were showing Next Gen Stats animations of the space and relative speeds of the players, and that analysis became a critical part of the rules change,” says Lopez.

The NFL Competition Committee tested alternative formations and identified a design that reduced high-speed contact without removing the competitive element. Two seasons of data show the dynamic kickoff is working: the 2025 return rate jumped to 75% (from 32% in 2024), and even with 1,157 more plays, lower-extremity injuries dropped 35% while concussion rates remain below the old kickoff format. The change is delivering both more action and fewer injuries.

Pose tracking

The infrastructure for the next major advance — optical tracking — is already embedded in every NFL venue. Rather than recording only a player’s two-dimensional location, the system uses 4K cameras to capture the full three-dimensional position of key joints such as shoulders, elbows, knees, hips, and hands.

The result is pose estimation, a digital skeletal model for every player on every play. This season marks the first year the league has had what Band calls “full installation, full capture” across every game, although the data remains internal while it is validated, structured, and stored for future use.

For the NGS team, pose estimation arrives at the right moment. A decade of two-dimensional tracking has deepened understanding of the game, Band says, “but this new skeletal data is going to unlock the next level. It’s an inflection point.”

The scale of the data capture is worth pausing over. Standard location tracking collects a single x,y coordinate for each player 10 times per second. Optical tracking captures high-resolution video from 16 angles to derive x,y,z coordinates for 29 body parts per player, 60 times a second. “The explosion in the volume of data can be daunting,” says Flynn. “But once folks wrap their heads around it, the ideas start flowing very quickly.”

The pipeline behind optical tracking runs in three stages: local capture, on-site processing, and cloud analysis. High-bandwidth video from 4K cameras cannot be sent to the cloud fast enough, so each stadium hosts AWS servers that process the data within about 700 milliseconds. The processed, simplified data is then sent to the cloud, where ML models run in under 100 milliseconds and return analysis to the production team. This keeps the full capture-to-analysis pipeline under a second. And because broadcasts such as Thursday Night Football operate with a roughly two-second delay, Next Gen Stats derived from this new data can be delivered effectively in real time as plays develop on screen.

The promise of pose data lies in the detail it adds to football’s geometry. It also resolves ambiguities that two-dimensional data cannot, says Lopez. “On a pass play now, we can see the ball pass a player using RFID data, but we don’t know if it rolled between their legs or flew 20 yards over their head.”

The ultimate goal is a hybrid system that uses RFID to identify each player’s center of mass and combines it with full skeletal data, with algorithms filling in gaps when players obscure one another from camera view.

Pose tracking will also unlock a new kind of training environment. Quarterbacks could use VR headsets to face a virtual pass rush that unfolds exactly as it did on the field. “You’re seeing those linemen coming at you and learning to keep your eye level down the field for that extra half second,” says Flynn.

This realism makes it possible to both train safely and correct habits that get young quarterbacks into trouble, while also helping them make quicker decisions in the pocket. “Josh Allen took a couple of seasons to become Josh Allen. Perhaps that could happen in half a year instead of three,” Flynn says.

Each stage in the evolution of NGS has pushed the league closer to modeling the game’s underlying mechanics rather than just its outcomes. As these capabilities come together, the wider transformation becomes clearer. Ten years after expanding box scores, the NFL’s partnership with AWS has evolved from a tracking experiment into something closer to the sport’s nervous system. By combining football expertise with scalable cloud infrastructure, Next Gen Stats continues to shape how the game is played, coached, and understood.

But in the end, it’s the subtle depth of football that hooks people. “It’s like quantum physics,” says Band. “You can zoom in as much as you want, and every shift in scale reveals something new. There are games within the game, happening all over the field.” It turns out that illuminating the intricate mechanics of the sport doesn't spoil the magic but only deepens the awe.

Research areas

Related content

US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to apply their causal inference and/or structural econometrics skillsets to solve real world problems. The intern will work in the area of Economics Intelligence in Amazon Returns and Recommerce Technology and Innovation and develop new, data-driven solutions to support the most critical components of this rapidly scaling team. Our PhD Economist Internship Program offers hands-on experience in applied economics, supported by mentorship, structured feedback, and professional development. Interns work on real business and research problems, building skills that prepare them for full-time economist roles at Amazon and beyond. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. About the team The WWRR Economics Intelligence (RREI) team brings together Economists, Data Scientists, and Business Intelligence Engineers experts to delivers economic solutions focused on forecasting, causality, attribution, customer behavior for returns, recommerce, and sustainability domains.
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to apply their causal inference and/or structural econometrics skillsets to solve real world problems. The intern will work in the area of Economics Intelligence in Amazon Returns and Recommerce Technology and Innovation and develop new, data-driven solutions to support the most critical components of this rapidly scaling team. Our PhD Economist Internship Program offers hands-on experience in applied economics, supported by mentorship, structured feedback, and professional development. Interns work on real business and research problems, building skills that prepare them for full-time economist roles at Amazon and beyond. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. About the team The WWRR Economics Intelligence (RREI) team brings together Economists, Data Scientists, and Business Intelligence Engineers experts to delivers economic solutions focused on forecasting, causality, attribution, customer behavior for returns, recommerce, and sustainability domains.
US, WA, Seattle
Amazon has co-founded and signed The Climate Pledge, a commitment to reach net zero carbon by 2040. As a team, we leverage GenAI, sensors, smart home devices, cloud services, material science, and Alexa to build products that have a meaningful impact for customers and the climate. In alignment with this bold corporate goal, the Amazon Devices & Services organization is looking for a passionate, talented, and inventive Senior Applied Scientist to help build revolutionary products with potential for major societal impact. Great candidates for this position will have expertise in the areas of agentic AI applications, deep learning, time series analysis, LLMs, and multimodal systems. This includes experience designing autonomous AI agents that can reason, plan, and execute multi-step tasks, building tool-augmented LLM systems with access to external APIs and data sources, implementing multi-agent orchestration, and developing RAG architectures that combine LLMs with domain-specific knowledge bases. You will strive for simplicity and creativity, demonstrating high judgment backed by statistical proof. Key job responsibilities As a Senior Applied Scientist on the Energy Science team, you'll design and deploy agentic AI systems that autonomously analyze data, plan solutions, and execute recommendations. You'll build multi-agent architectures where specialized AI agents coordinate to solve complex optimization problems, and develop tool-augmented LLM applications that integrate with external data sources and APIs to deliver context-aware insights. Your work involves creating multimodal AI systems that synthesize diverse data streams, while implementing RAG pipelines that ground large language models in domain-specific knowledge bases. You'll apply advanced machine learning and deep learning techniques to time series analysis, forecasting, and pattern recognition. Beyond technical innovation, you'll drive end-to-end product development from research through production deployment, collaborating with cross-functional teams to translate AI capabilities into customer experiences. You'll establish rigorous experimentation frameworks to validate model performance and measure business impact, building AI-driven products with potential for major societal impact.
US, CA, San Francisco
Amazon launched the AGI Lab to develop foundational capabilities for useful AI agents. We built Nova Act - a new AI model trained to perform actions within a web browser. The team builds AI/ML infrastructure that powers our production systems to run performantly at high scale. We’re also enabling practical AI to make our customers more productive, empowered, and fulfilled. In particular, our work combines large language models (LLMs) with reinforcement learning (RL) to solve reasoning, planning, and world modeling in both virtual and physical environments. Our lab is a small, talent-dense team with the resources and scale of Amazon. Each team in the lab has the autonomy to move fast and the long-term commitment to pursue high-risk, high-payoff research. We’re entering an exciting new era where agents can redefine what AI makes possible. We’d love for you to join our lab and build it from the ground up! Key job responsibilities This role will lead a team of SDEs building AI agents infrastructure from launch to scale. The role requires the ability to span across ML/AI system architecture and infrastructure. You will work closely with application developers and scientists to have a impact on the Agentic AI industry. We're looking for a Software Development Manager who is energized by building high performance systems, making an impact and thrives in fast-paced, collaborative environments. About the team Check out the Nova Act tools our team built on on nova.amazon.com/act
US, WA, Seattle
MULTIPLE POSITIONS AVAILABLE Employer: AMAZON WEB SERVICES, INC. Offered Position: Applied Scientist III Job Location: Seattle, Washington Job Number: AMZ9674037 Position Responsibilities: Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data, and run and analyze experiments in a production environment. Identify new opportunities for research in order to meet business goals. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists. Position Requirements: Master’s degree or foreign equivalent degree in Computer Science, Machine Learning, Engineering, or a related field and two years of research or work experience in the job offered, or as a Research Scientist, Research Assistant, Software Engineer, or a related occupation. Employer will accept a Bachelor’s degree or foreign equivalent degree in Computer Science, Machine Learning, Engineering, or a related field and five years of progressive post-baccalaureate research or work experience in the job offered or a related occupation as equivalent to the Master’s degree and two years of research or work experience. Must have one year of research or work experience in the following skill(s): (1) programming in Java, C++, Python, or equivalent programming language; and (2) conducting the analysis and development of various supervised and unsupervised machine learning models for moderately complex projects in business, science, or engineering. Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation. 40 hours / week, 8:00am-5:00pm, Salary Range $167,100/year to $226,100/year. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits.#0000
IN, KA, Bengaluru
Interested to build the next generation Financial systems that can handle billions of dollars in transactions? Interested to build highly scalable next generation systems that could utilize Amazon Cloud? Massive data volume + complex business rules in a highly distributed and service oriented architecture, a world class information collection and delivery challenge. Our challenge is to deliver the software systems which accurately capture, process, and report on the huge volume of financial transactions that are generated each day as millions of customers make purchases, as thousands of Vendors and Partners are paid, as inventory moves in and out of warehouses, as commissions are calculated, and as taxes are collected in hundreds of jurisdictions worldwide. Key job responsibilities • Understand the business and discover actionable insights from large volumes of data through application of machine learning, statistics or causal inference. • Analyse and extract relevant information from large amounts of Amazon’s historical transactions data to help automate and optimize key processes • Research, develop and implement novel machine learning and statistical approaches for anomaly, theft, fraud, abusive and wasteful transactions detection. • Use machine learning and analytical techniques to create scalable solutions for business problems. • Identify new areas where machine learning can be applied for solving business problems. • Partner with developers and business teams to put your models in production. • Mentor other scientists and engineers in the use of ML techniques. A day in the life • Understand the business and discover actionable insights from large volumes of data through application of machine learning, statistics or causal inference. • Analyse and extract relevant information from large amounts of Amazon’s historical transactions data to help automate and optimize key processes • Research, develop and implement novel machine learning and statistical approaches for anomaly, theft, fraud, abusive and wasteful transactions detection. • Use machine learning and analytical techniques to create scalable solutions for business problems. • Identify new areas where machine learning can be applied for solving business problems. • Partner with developers and business teams to put your models in production. • Mentor other scientists and engineers in the use of ML techniques. About the team The FinAuto TFAW(theft, fraud, abuse, waste) team is part of FGBS Org and focuses on building applications utilizing machine learning models to identify and prevent theft, fraud, abusive and wasteful(TFAW) financial transactions across Amazon. Our mission is to prevent every single TFAW transaction. As a Machine Learning Scientist in the team, you will be driving the TFAW Sciences roadmap, conduct research to develop state-of-the-art solutions through a combination of data mining, statistical and machine learning techniques, and coordinate with Engineering team to put these models into production. You will need to collaborate effectively with internal stakeholders, cross-functional teams to solve problems, create operational efficiencies, and deliver successfully against high organizational standards.
IN, KA, Bengaluru
Amazon Health Services (One Medical) About Us: At Health AI, we're revolutionizing healthcare delivery through innovative AI-enabled solutions. As part of Amazon Health Services and One Medical, we're on a mission to make quality healthcare more accessible while improving patient outcomes. Our work directly impacts millions of lives by empowering patients and enabling healthcare providers to deliver more meaningful care. Role Overview: We're seeking an Applied Scientist to join our dynamic team in building state of the art AI/ML solutions for healthcare. This role offers a unique opportunity to work at the intersection of artificial intelligence and healthcare, developing solutions that will shape the future of medical services delivery. Key job responsibilities • Lead end-to-end development of AI/ML solutions for Amazon Health organization, including Amazon Pharmacy and One Medical • Research, design, and implement state-of-the-art machine learning models, with a focus on Large Language Models (LLMs) and Visual Language Models (VLMs) • Optimize and fine-tune models for production deployment, including model distillation for improved latency • Drive scientific innovation while maintaining a strong focus on practical business outcomes • Collaborate with cross-functional teams to translate complex technical solutions into tangible customer benefits • Contribute to the broader Amazon Health scientific community and help shape our technical roadmap
US, CA, Santa Clara
Amazon Quick Suite is an enterprise AI platform that transforms how organizations work with their data and knowledge. Combining generative AI-powered search, deep research capabilities, intelligent agents and automations, and comprehensive business intelligence, Quick Suite serves tens of thousands of users. Our platform processes thousands of queries monthly, helping teams make faster, data-driven decisions while maintaining enterprise-grade security and governance. From natural language interactions with complex datasets to automated workflows and custom AI agents, Quick Suite is redefining workplace productivity at unprecedented scale. We are seeking a Data Scientist II to join our Quick Data team, focusing on evaluation and benchmarking data development for Quick Suite features, with particular emphasis on Research and other generative AI capabilities. Our mission is to engineer high-quality datasets that are essential to the success of Amazon Quick Suite. From human evaluations and Responsible AI safeguards to Retrieval-Augmented Generation and beyond, our work ensures that Generative AI is enterprise-ready, safe, and effective for users at scale. As part of our diverse team—including data scientists, engineers, language engineers, linguists, and program managers—you will collaborate closely with science, engineering, and product teams. We are driven by customer obsession and a commitment to excellence. Key job responsibilities In this role, you will leverage data-centric AI principles to assess the impact of data on model performance and the broader machine learning pipeline. You will apply Generative AI techniques to evaluate how well our data represents human language and conduct experiments to measure downstream interactions. Specific responsibilities include: * Design and develop comprehensive evaluation and benchmarking datasets for Quick Suite AI-powered features * Leverage LLMs for synthetic data corpora generation; data evaluation and quality assessment using LLM-as-a-judge settings * Create ground truth datasets with high-quality question-answer pairs across diverse domains and use cases * Lead human annotation initiatives and model evaluation audits to ensure data quality and relevance * Develop and refine annotation guidelines and quality frameworks for evaluation tasks * Conduct statistical analysis to measure model performance, identify failure patterns, and guide improvement strategies * Collaborate with ML scientists and engineers to translate evaluation insights into actionable product improvements * Build scalable data pipelines and tools to support continuous evaluation and benchmarking efforts * Contribute to Responsible AI initiatives by developing safety and fairness evaluation datasets About the team Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Hybrid Work We value innovation and recognize this sometimes requires uninterrupted time to focus on a build. We also value in-person collaboration and time spent face-to-face. Our team affords employees options to work in the office every day or in a flexible, hybrid work model near one of our U.S. Amazon offices.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Mission Build AI safety systems that protect millions of Alexa customers every day. As conversational AI evolves, you'll solve challenging problems in Responsible AI by ensuring LLMs provide safe, trustworthy responses, building AI systems that understand nuanced human values across cultures, and maintaining customer trust at scale. What You'll Build You'll pioneer breakthrough solutions in Responsible AI at Amazon's scale. Imagine training models that set new safety standards, designing automated testing systems that hunt for vulnerabilities before they surface, and certifying the systems that power millions of daily conversations. You'll create intelligent evaluation systems that judge responses with human-level insight, build models that truly understand what makes interactions safe and delightful, and craft feedback mechanisms that help Alexa+ grasp the nuances of complex customer conversations. Here's where it gets even more exciting: you'll build AI agents that act as your team's safety net—automatically detecting and fixing production issues in real-time, often before anyone notices there was a problem. Your innovations won't just improve Alexa+; they'll fundamentally shape how it learns, evolves, and earns customer trust. As Alexa+ continues to delight customers, your work ensures it becomes more trustworthy, safer, and deeply aligned with customer needs and expectations. Your work directly protects customer trust at Amazon's scale. Every innovation you create—from novel safety mechanisms to sophisticated evaluation techniques—shapes how millions of people interact with AI confidently. You're not just building products; you're defining industry standards for responsible AI. This is frontier research with immediate real-world impact. You'll tackle problems that require innovative solutions: training models that remain truthful and grounded across diverse contexts, building reward models that capture the nuanced spectrum of human values across cultures and languages, and creating automated systems that continuously discover and address potential issues before customers encounter them. You'll collaborate with world-class scientists, product managers, and engineers to transform state-of-the-art ideas into production systems serving millions. What We're Looking For * Deep expertise in state-of-the-art NLP and Large Language Models * Track record of building scalable ML systems * Passion for impactful research—where frontier science meets real-world responsibility at scale * Excitement about solving problems that will shape the future of AI Ready to work on AI safety challenges that define the industry? Join us. Key job responsibilities This is where you'll make your mark. You'll architect breakthrough Responsible AI solutions that become industry benchmarks, pioneering algorithms that eliminate false information, designing frameworks that hunt down vulnerabilities before bad actors find them, and developing models that understand human values across every culture we serve. Working with world-class engineers and scientists, you'll push the boundaries of model training—transforming bold research into production systems that protect millions of customers daily while withstanding attacks and delivering exceptional experiences. But here's what makes this role truly special: you'll shape the future. You'll lead certification processes, advance optimization techniques, build evaluation systems that reason like humans, and mentor the next generation of AI safety experts. Every innovation you drive will set new standards for trustworthy AI at the world's largest scale. A day in the life As a Responsible AI Scientist, you're at the frontier of AI safety—experimenting with breakthrough techniques that push the boundaries of what's possible. You partner with engineering to transform research into production-ready solutions, tackling complex optimization challenges. You brainstorm with Product teams, translating ambitious visions into concrete objectives that drive real impact. Your expertise shapes critical deployment decisions as you review impactful work and guide go/no-go calls. You mentor the next generation of AI safety leaders, watching ideas spark and capabilities grow. This is where science meets impact—building AI that's not just intelligent, but trustworthy and aligned with human values. About the team Our team pioneers Responsible AI for conversational assistants. We ensure Alexa delivers safe, trustworthy experiences across all devices, modalities, and languages worldwide. We work on frontier AI safety challenges—and we're looking for scientists who want to help shape the future of trustworthy AI.