The science behind the new FBA capacity management system

The new Fulfillment by Amazon system empowers sellers to have more transparency and control over their capacity within Amazon’s fullfilment network by applying market-based principles.

It’s September. You’re a toy seller using Fulfillment by Amazon (FBA) and most of your sales come right before the holidays. However, this summer you overstocked, which degraded the health of your inventory turns, resulting in lower remaining capacity as you near the holiday season. Meanwhile, you just launched a new line of robot assembly kits that are becoming popular with customers.

How can you ensure access to additional space in Amazon fulfillment centers to set yourself up for a great Black Friday and Cyber Monday using FBA? And how does Amazon manage capacity without overbuilding space, potentially incurring costs that do not benefit customers, as well as without underbuilding space and limiting product selection, availability, and fast delivery?

The journey of a package
Brands and resellers seeking to leverage Amazon's science- and technology-driven fulfillment and transportation networks are turning to Fulfillment by Amazon (FBA) to outsource order fulfillment and customer service to Amazon.

The answer lies in an innovative new FBA Capacity management system, driven by Amazon’s Supply Chain Optimization Technologies (SCOT) and Selling Partners Services (SPS) teams. The new system provides most sellers with greater capacity limits, and also greater control to obtain additional capacity when they need it.

FBA is an optional service for sellers to outsource order fulfillment and customer service to Amazon. Since its launch in 2006, FBA has expanded rapidly, with selling partners ranging from family-owned snack makers to multinational high-tech companies.

FBA is an increasingly popular choice for both brands and resellers seeking to leverage Amazon's science- and technology-driven fulfillment and transportation networks. FBA provides sellers with significant cost savings and faster delivery speeds that together provide a great value for customers and sellers. The accelerated delivery speed has led to significant increases in customer demand for products, while the surge in demand, particularly during peak events such as Prime Day, Black Friday, Cyber Monday, and holidays, has created unprecedented demand for FBA capacity.

Amazon needed to find a way to match demand with available capacity while building additional capacity to meet sellers’ needs.

The capacity management challenge

Amazon’s forecasting capabilities — powered by a combination of machine learning, simulation, and optimization modeling — are well documented. Scientists have tackled research questions such as how to manage inventory for peak events and how to best distribute products across the Amazon fulfillment network for the fastest delivery.

Related content
The pandemic turbo-charged retail growth — teams of scientists at Amazon forged a path forward to handle the scale.

But when it comes to managing capacity in the FBA program, even Amazon’s predictive mechanisms have limitations. FBA serves a wide breadth of independent sellers who own their inventory and make independent decisions about which products to carry, how much inventory to stock, and what prices to offer.

“There's only so much we can really understand about what sellers’ business plans are,” said Garrett van Ryzin, distinguished scientist at SCOT. “It’s fundamentally limiting to rely solely on our own predictions.”

Additionally, Amazon does not have full visibility into those sellers’ business needs or decision-making processes. This became more relevant as seller usage of FBA network capacity grew significantly within the span of a few years.

“Most of the time our predictive models allocate sufficient capacity to each seller. Yet we observed, in some situations, sellers had insufficient capacity, which impacted their ability to pursue certain growth opportunities. So we set out to design a release valve so sellers can tell us when they need more capacity for products they are confident customers will love, even if we were not predicting that they needed more space based on the data that Amazon had,” said Seamus Browne, a principal product manager on the FBA team.

To manage capacity for FBA, Amazon originally created a system that scores sellers according to how well they manage their inventory. A few weeks in advance of each allocation period, sellers are told how much capacity they will have for the next three months — a number that is based on sellers’ projected sales, and previous inventory performance (everything else equal, this number increases for sellers who use allocated space efficiently).

Our initial system worked well for managing capacity. However, we needed a new approach that could handle sellers’ current and future plans, such as a new product introduction, marketing and advertising campaigns, or other information that only sellers knew about.
Özalp Özer

“Our initial system worked well for managing capacity. However, we needed a new approach that could handle sellers’ current and future plans, such as a new product introduction, marketing and advertising campaigns, or other information that only sellers knew about,” said Özalp Özer, who is leading the FBA Science team in his capacity as director of research science.

Allocating space based on past performance and Amazon’s forecasts meant sellers who had recently underperformed with respect to sales and inventory efficiency might end up having lower leftover capacity to launch potentially popular new products or campaigns. Sellers could submit exception requests for additional capacity; Amazon was getting many such requests each quarter, and each one had to be reviewed manually.

To address that customer pain point, the Amazon FBA team created a new way for sellers to provide a simple, objective measure of how much value they could create if given additional capacity. The new tool applies economic principles to empower sellers to communicate credible needs for additional capacity.

Related content
Why multimodal identification is a crucial step in automating item identification at Amazon scale.

Here is how it works: during each capacity allocation period, sellers can place requests to increase their capacity limit. Those sellers submit a “reservation fee” — the maximum fee they would be willing to pay per cubic foot to reserve additional capacity, along with their desired maximum amount of additional capacity.

Requests are granted objectively, starting with the highest reservation fee per cubic foot until all capacity available under this program has been allocated. When additional capacity is granted, sellers pay no more than their chosen maximum reservation fee. In addition, sellers receive performance credits from the sales they generate using the extra capacity — up to 100% of the reservation fee.

A different approach to capacity management

The FBA team designed the tool to optimize the value for customers who use additional capacity effectively — and not to maximize proceeds for Amazon. In fact, sellers who win more space can earn credits from the extra sales they generate, and these credits can offset the entire amount of the reservation fee.

“The mechanism is designed to reward sellers who use their additional capacity productively to generate sales,” van Ryzin explained. “We are applying economic market design principles to manage supply chain capacity, and it's the first time we've really done it at this scale within FBA.”

This new tool is inspired by earlier economic research on selling securities, where payments are based on the value generated from the asset after it is allocated, but the details posed unique challenges.

“You also need something that is simple for sellers and holds a level of fairness in its decisions,” explained Tolga Seyhan, a principal research scientist on the FBA team.

“We make sure that if you come early and you paid a high fee, you will ultimately get the same fee as someone who paid less in a later round,” said Alexandre Belloni, an Amazon Scholar from Duke University’s Fuqua School of Business who has been leading market-design-related projects. “We want to reward early requests, which helps with our capacity planning and allows sellers to better utilize the additional capacity.”

Related content
How Customer Order and Network Density OptimizeR (CONDOR) has led to improved delivery routes.

Take a theoretical example: Say a seller has a Halloween shop on Amazon and needs space for a new costume. They want 5,000 additional cubic feet (“cubes”) on top of their usual allocation of 10,000 ft3. They place a reservation fee of $6 per cube in an early round of the quarter for a total potential cost of $30,000. Their request is granted, but then the lowest reservation fee granted in subsequent rounds drops to $4 per cube. The new tool reduces the Halloween shop’s reservation fee to $4 per cube, matching the lowest reservation fee granted in the later round. Hence, the seller’s total reservation fee would be a maximum of $20,000.

As is often the case with these capacity reservations, in this theoretical example, the seller correctly forecasts that their new costume would justify the space. That extra space generates $300,000 in incremental sales. Amazon awards a 15% performance credit on the incremental sales, up to a maximum of the $20,000 total commitment. In this case, the seller has earned performance credits of $45,000, which fully offset the entire reservation fee. The seller ends up paying nothing for the additional space, and because they used the space efficiently, their capacity allocations are also likely to increase in the future.

The system was successfully launched for a small number of sellers in early 2022.

Related content
The story of a decade-plus long journey toward a unified forecasting model.

“When we launched the first experiment, we were excited to see that sellers were using this rationally and that it was helping address information asymmetry. Sellers had enough information to confidently forecast both the need and price they’d be willing to pay for additional space. As a result, the vast majority of sellers got their reservation fee back and ended up paying nothing for the additional space,” Belloni said, adding that demand for the program has increased.

“The productivity of the space allocated through this mechanism is higher than what we would get by doing the allocation ourselves, and that benefits everyone: Amazon, our sellers, and our customers,” Özer said. “Building this new system required our scientists to use state-of-the-art tools from optimization, simulation, and economics and work closely with product managers and engineers. Together we innovate on behalf of our customers and sellers around the world.”

Related content

ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.