A head and shoulders profile photo of Haluk Demirkan, senior manager of Devices Demand Science
Haluk Demirkan, senior manager of Devices Demand Science, says his goal at Amazon "is to build an ecosystem in which technology is doing the labor-intensive tasks, freeing my team to do more smart work and value-added tasks."

How Haluk Demirkan is using ML-powered forecasts to get the right devices to the right place at the right time

Part-time sabbatical plan turns into full-time role for author of five books and more than 170 research articles.

For years, Amazon has been at the forefront of machine learning and data science. At the same time, the company has pioneered the large-scale automation of processes at all levels of its supply chains. But in its fast-moving commercial world, the constant challenge is to integrate these complementary fields to create processes that optimize the delivery of customer value.

Related content
A combination of audio and visual signals guide the device’s movement, so the screen is always in view.

In 2021, Haluk Demirkan joined Amazon to boost the company’s efforts for that kind of integration. As the senior manager of Devices Demand Science, Demirkan is building and leading a team dedicated to using cutting-edge data analytics, machine learning, and process optimization — among other techniques — to transform sales predictions for Amazon devices, from the Kindle, the Echo family of devices, and Fire Tablets, to Fire TVs, and Ring Video Doorbell — an enormously important task.

To understand why the company’s Devices organization is excited about Demirkan’s arrival, a little background is required. Demirkan first came to the US from Turkey in 1991 on a three-month language course to improve his English. Three decades later, he hasn’t left. Instead, he has built a pioneering transdisciplinary career at the intersection of data science, service science, smart machines, and industry.

Bridging industry and academia

He earned a master’s in industrial and systems engineering and in 2002 completed a dual-degree PhD in information systems and operations management at the University of Florida. He gained these qualifications while simultaneously working full-time for AT&T Bell Labs (as it was then known) and Citibank in data analytics, process engineering, and price and supply chain optimization.

By the time he earned his PhD, Demirkan had already spent 11 years in industry, so in 2002 he decided to give full-time academia a try. He joined Arizona State University as an assistant professor, primarily teaching information systems, analytics and supply chain management. While at ASU, Demirkan co-edited two seminal research books in the emerging fields of service science and systems, and its industry-based implementation.

Related content
The collaboration will focus on advancing innovation in core robotics and AI technologies and their applications.

In 2013, he moved to the University of Washington-Tacoma as an associate professor of service innovation and business analytics, tasked with expanding research and education programs in business analytics. He eventually became founding director of the Milgard School’s Center for Business Analytics and Master of Science Business Analytics, and the founding assistant dean of the Analytics Innovations Hub.

So far, he has published five books and more than 170 research articles. In 2021, Demirkan’s work and community building were recognized by the university with an award for both Distinguished Research and Community Engagement.

Throughout this academic stretch of his career, Demirkan maintained strong links with industry, developing data science, engineering and smart analytics solutions for dozens of leading companies, including IBM, GE, Cisco, HP, Intel, Bank of America, and Mayo Clinic. With AWS Academy Educator Accreditation, he still teaches a data analytics course at the University of Washington-Tacoma on weekends, guiding his students in developing AI and data-analytics-based solutions to novel business problems.

Researching “big problems”

So how did he end up full-time at Amazon?

“I’d come close to one of the highest positions in the academic career path,” says Demirkan. “The next step would have been to become a dean somewhere. But I didn’t want to be dean for near term: I prefer working on research for big problems.”

So, after almost two decades in academia, he decided to take a sabbatical. However, his restless nature meant he couldn’t be idle, so Demirkan applied to become an Amazon Scholar, a flexible program designed for academics who want to tackle large-scale technical challenges.

His plan: work one day a week during his sabbatical. During the interview with Amazon, however, it became clear that some of Amazon’s big business challenges dovetailed with Demirkan’s skillset so strongly — and offered him the opportunity to make such a big impact — that he decided to join the company full time by taking a leave from UW.

“My wife was like, ‘This is not a sabbatical!’”, recalls Demirkan.

The power of demand prediction

In his new Seattle-based role, Demirkan has two primary areas of business focus. The first is in making sales demand predictions for most Amazon devices. His team produces sales predictions for the majority of device types, globally, in which Amazon has a presence, and for every day from now until a year in the future.

To do this, Demirkan’s team ingests device sales data to train machine learning algorithms to generate increasingly accurate sales forecasts. Specifically, the team is employing advanced time-series forecasting methods, such as Random Forest, XGBoost, and Ridge Regression.

Related content
The story of a decade-plus long journey toward a unified forecasting model.

“Based on that predicted demand, another Amazon team ships just the right amount of product to the right locations, so it’s where it needs to be just as a customer chooses to make that purchase,” says Demirkan. “Basically, our goal is to get the right amount of devices to the right place at the right time to better meet our customers’ needs. That's our goal.”

The forecasts Demirkan’s team creates do more than mere prediction. The team can also, for example, make projections for sales based on varying promotional prices.

“How many customers in London are going to buy a Kindle on April 21, if the price is x? The forecast assists our executive management teams to make decisions around how many units to manufacture, how many to ship, and when to ship,” says Demirkan.

By providing the company’s supply chain with increasingly accurate demand forecasts, Amazon simultaneously reduces delivery times and supply chain costs, helping the company keep prices low, while increasing customer responsiveness.

Demirkan is also developing a comprehensive, science-driven forecasting model called “Intelligent Demand Plan”. It will combine a wider range of inputs, including product cannibalization, macro-economic factors, traffic, social media and lots more, to sense demand and customer preferences with greater nuance, and to gain early insight into emerging market trends.

Automation and optimization

Demirkan’s second area of business focus: process and task automation and optimization, which utilizes his expertise in AI, process engineering and supply chain management. He and his team are analyzing the forecasting processes in Amazon’s devices group, identifying opportunities for improvement.

Related content
How Amazon’s scientists developed a first-of-its-kind multi-echelon system for inventory buying and placement.

“Amazon is one of the fastest-moving companies I have ever seen,” he says. “Everything is about providing the best possible service to customers, and innovation is happening so quickly here that processes designed as recently as six months ago may no longer be optimal.”

This is one of his research passions: machine-assisted cognition, also known as intelligence augmentation with artificial intelligence.

“Computers are already our assistants today, of course. I want to take these computing technologies to a more advanced level, using machine learning to, for example, train computers to teach themselves to provide me with what I need to know to make better business decisions,” Demirkan explains. “By making processes more automated, efficient, and error proof, we humans have time to do more value-added tasks.”

Breaking research silos

Demirkan said he expects his team to grow in the next six months. “I have applied scientists, research scientists, and data scientists. It’s one of many fast-growing teams at Amazon,” he says.

Demirkan’s transdisciplinary expertise — that combination of deep research knowledge and broad applications experience — is something he will be infusing into his team’s culture. Many education systems, he argues, with their tendencies to silo students in particular domains of expertise, are producing a generation of people who can find it hard to adapt to the wider commercial world.

Related content
Belinda Zeng, head of applied science and engineering at Amazon Search Science and AI, shares her perspective.

“With the digital transformation of companies in every industry, the coming generations of employees need to be more ‘T-shaped’ — innovators with not only a depth of expertise but also a broad, integrated understanding of other disciplines, such as management, engineering, and social sciences,” he says.

Last year, Demirkan’s insights into the changing nature of the high-tech workplace were sought by the US Department of Labor, Employment and Training Administration, when he was invited as an occupation expert to provide guidance on the definition of the occupation Operations Research Analysts.

“This role is about the combination of operations management, IT, data science, and machine learning — a very multidisciplinary, new occupation,” says Demirkan. “I'm hoping that in the future, academia will have more programs geared to preparing people for these crucial kinds of roles.”

With new roles in mind, how has Demirkan enjoyed his work with Amazon?

“A friend of mine said to me: ‘Haluk, you are going 35 miles an hour in academia, and now you are switching to 200 miles an hour?!’,” he says. “I love being a professor and making a difference in students’ lives, but I am relishing being back in industry because in 19 years, things have changed. I’m absorbing so much, and I can take this updated knowledge back to my classes when I teach on the weekend.”

From his professional perspective, Demirkan sees more clearly than most the gaps between academic education and the expectations of professional workplaces. “Globally, we have an ongoing mismatch problem. With my experience with Amazon, I can do my bit to close this gap,” he says.

Giving back

Doing his bit is central to Demirkan’s ethos. When Covid-19 struck in 2020, and hospitals all over the planet were suddenly critically overloaded, Demirkan was contacted by Virginia Mason Franciscan Health, one of the largest healthcare service and hospital providers in Washington state. They wanted his help to optimize their hospital bed allocations, among other things.

“We met online every week, trying to predict demand and capacity, which patients to move to other hospital facilities, looking at doctors’ and nurses’ scheduling — everything.” The urgency of the situation meant Demirkan was more than a volunteer advisor. “I was writing machine learning scripts, literally writing the code myself, to exploit the data quickly being gathered by the hospitals. I was proud to be involved in that work, because it was the first time I was able to make that sort of critical difference in people's lives.”

Related content
While these systems look like other robot arms, they embed advanced technologies that will shape Amazon's robot fleet for years to come.

In terms of making a difference at Amazon, Demirkan says: “My goal is to build an ecosystem in which technology is doing the labor-intensive tasks, freeing my team to do more smart work and value-added tasks. That's my idea of success.” That, and taking Amazon’s device-sales forecasting to a whole new level. “I want my team forecasting at a comprehensive, granular level,” says Demirkan. “I want to say with unprecedented accuracy that in this location, this device — in this color, size, and detail — will sell x units on this day.”

But Demirkan also sees additional potential in developing approaches that go beyond traditional forecasting. “I want to develop machine learning and data analytics that can discern what it is that customers really want and expect from Amazon devices; to generate insights powerful enough to actually impact the design decisions for brand new products and services.”

Research areas

Related content

  • Staff writer
    December 29, 2025
    From foundation model safety frameworks and formal verification at cloud scale to advanced robotics and multimodal AI reasoning, these are the most viewed publications from Amazon scientists and collaborators in 2025.
  • Staff writer
    December 29, 2025
    From quantum computing breakthroughs and foundation models for robotics to the evolution of Amazon Aurora and advances in agentic AI, these are the posts that captured readers' attention in 2025.
  • Staff writer
    October 21, 2025
    Initiative will fund over 100 doctoral students researching machine learning, computer vision, and natural-language processing at nine universities.
ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.