New AWS tool recommends removal of unused permissions

IAM Access Analyzer feature uses automated reasoning to recommend policies that remove unused accesses, helping customers achieve “least privilege”.

AWS Identity and Access Management (IAM) policies provide customers with fine-grained control over who has access to what resources in the Amazon Web Services (AWS) Cloud. This control helps customers enforce the principle of least privilege by granting only the permissions required to perform particular tasks. In practice, however, writing IAM policies that enforce least privilege requires customers to understand what permissions are necessary for their applications to function, which can become challenging when the scale of the applications grows.

To help customers understand what permissions are not necessary, we launched IAM Access Analyzer unused access findings at the 2023 re:Invent conference. IAM Access Analyzer analyzes your AWS accounts to identify unused access and creates a centralized dashboard to report its findings. The findings highlight unused roles and unused access keys and passwords for IAM users. For active IAM roles and users, the findings provide visibility into unused services and actions.

Related content
New IAM Access Analyzer feature uses automated reasoning to ensure that access policies written in the IAM policy language don’t grant unintended access.

To take this service a step further, in June 2024 we launched recommendations to refine unused permissions in Access Analyzer. This feature recommends a refinement of the customer’s original IAM policies that retains the policy structure while removing the unused permissions. The recommendations not only simplify removal of unused permissions but also help customers enact the principle of least privilege for fine-grained permissions.

In this post, we discuss how Access Analyzer policy recommendations suggest policy refinements based on unused permissions, which completes the circle from monitoring overly permissive policies to refining them.

Policy recommendation in practice

Let's dive into an example to see how policy recommendation works. Suppose you have the following IAM policy attached to an IAM role named MyRole:

{
  "Version": "2012-10-17",
  "Statement": [
   {
      "Effect": "Allow",
      "Action": [
        "lambda:AddPermission",
        "lambda:GetFunctionConfiguration",
        "lambda:UpdateFunctionConfiguration",
        "lambda:UpdateFunctionCode",
        "lambda:CreateFunction",
        "lambda:DeleteFunction",
        "lambda:ListVersionsByFunction",
        "lambda:GetFunction",
        "lambda:Invoke*"
      ],
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:my-lambda"
   },
  {
    "Effect" : "Allow",
    "Action" : [
      "s3:Get*",
      "s3:List*"
    ],
    "Resource" : "*"
  }
 ]
}

The above policy has two policy statements:

  • The first statement allows actions on a function in AWS Lambda, an AWS offering that provides function execution as a service. The allowed actions are specified by listing individual actions as well as via the wildcard string lambda:Invoke*, which permits all actions starting with Invoke in AWS Lambda, such as lambda:InvokeFunction.
  • The second statement allows actions on any Amazon Simple Storage Service (S3) bucket. Actions are specified by two wildcard strings, which indicate that the statement allows actions starting with Get or List in Amazon S3.

Enabling Access Analyzer for unused finding will provide you with a list of findings, each of which details the action-level unused permissions for specific roles. For example, for the role with the above policy attached, if Access Analyzer finds any AWS Lambda or Amazon S3 actions that are allowed but not used, it will display them as unused permissions.

Related content
Amazon Web Services (AWS) is a cloud computing services provider that has made significant investments in applying formal methods to proving correctness of its internal systems and providing assurance of correctness to their end-users. In this paper, we focus on how we built abstractions and eliminated specifications to scale a verification engine for AWS access policies, Zelkova, to be usable by all AWS

The unused permissions define a list of actions that are allowed by the IAM policy but not used by the role. These actions are specific to a namespace, a set of resources that are clustered together and walled off from other namespaces, to improve security. Here is an example in Json format that shows unused permissions found for MyRole with the policy we attached earlier:

[
 {
    "serviceNamespace": "lambda",
    "actions": [
      "UpdateFunctionCode",
      "GetFunction",
      "ListVersionsByFunction",
      "UpdateFunctionConfiguration",
      "CreateFunction",
      "DeleteFunction",
      "GetFunctionConfiguration",
      "AddPermission"
    ]
  },
  {
    "serviceNamespace": "s3",
    "actions": [
        "GetBucketLocation",
        "GetBucketWebsite",
        "GetBucketPolicyStatus",
        "GetAccelerateConfiguration",
        "GetBucketPolicy",
        "GetBucketRequestPayment",
        "GetReplicationConfiguration",
        "GetBucketLogging",
        "GetBucketObjectLockConfiguration",
        "GetBucketNotification",
        "GetLifecycleConfiguration",
        "GetAnalyticsConfiguration",
        "GetBucketCORS",
        "GetInventoryConfiguration",
        "GetBucketPublicAccessBlock",
        "GetEncryptionConfiguration",
        "GetBucketAcl",
        "GetBucketVersioning",
        "GetBucketOwnershipControls",
        "GetBucketTagging",
        "GetIntelligentTieringConfiguration",
        "GetMetricsConfiguration"
    ]
  }
]

This example shows actions that are not used in AWS Lambda and Amazon S3 but are allowed by the policy we specified earlier.

Related content
Rungta had a promising career with NASA, but decided the stars aligned for her at Amazon.

How could you refine the original policy to remove the unused permissions and achieve least privilege? One option is manual analysis. You might imagine the following process:

  • Find the statements that allow unused permissions;
  • Remove individual actions from those statements by referencing unused permissions.

This process, however, can be error prone when dealing with large policies and long lists of unused permissions. Moreover, when there are wildcard strings in a policy, removing unused permissions from them requires careful investigation of which actions should replace the wildcard strings.

Policy recommendation does this refinement automatically for customers!

The policy below is one that Access Analyzer recommends after removing the unused actions from the policy above (the figure also shows the differences between the original and revised policies):

{
  "Version": "2012-10-17",
  "Statement" : [
   {
      "Effect" : "Allow",
      "Action" : [
-       "lambda:AddPermission",
-       "lambda:GetFunctionConfiguration",
-       "lambda:UpdateFunctionConfiguration",
-       "lambda:UpdateFunctionCode",
-       "lambda:CreateFunction",
-       "lambda:DeleteFunction",
-       "lambda:ListVersionsByFunction",
-       "lambda:GetFunction",
        "lambda:Invoke*"
      ],
      "Resource" : "arn:aws:lambda:us-east-1:123456789012:function:my-lambda"
    },
    {
     "Effect" : "Allow",
     "Action" : [
-      "s3:Get*",
+      "s3:GetAccess*",
+      "s3:GetAccountPublicAccessBlock",
+      "s3:GetDataAccess",
+      "s3:GetJobTagging",
+      "s3:GetMulti*",
+      "s3:GetObject*",
+      "s3:GetStorage*",
       "s3:List*"
     ],
     "Resource" : "*"
   }
  ]
}

Let’s take a look at what’s changed for each policy statement.

For the first statement, policy recommendation removes all individually listed actions (e.g., lambda:AddPermission), since they appear in unused permissions. Because none of the unused permissions starts with lambda:Invoke, the recommendation leaves lambda:Invoke* untouched.

For the second statement, let’s focus on what happens to the wildcard s3:Get*, which appears in the original policy. There are many actions that can start with s3:Get, but only some of them are shown in the unused permissions. Therefore, s3:Get* cannot just be removed from the policy. Instead, the recommended policy replaces s3:Get* with seven actions that can start with s3:Get but are not reported as unused.

Related content
Amazon scientists are on the cutting edge of using math-based logic to provide better network security, access management, and greater reliability.

Some of these actions (e.g., s3:GetJobTagging) are individual ones, whereas others contain wildcards (e.g., s3:GetAccess* and s3:GetObject*). One way to manually replace s3:Get* in the revised policy would be to list all the actions that start with s3:Get except for the unused ones. However, this would result in an unwieldy policy, given that there are more than 50 actions starting with s3:Get.

Instead, policy recommendation identifies ways to use wildcards to collapse multiple actions, outputting actions such as s3:GetAccess* or s3:GetMulti*. Thanks to these wildcards, the recommended policy is succinct but still permits all the actions starting with s3:Get that are not reported as unused.

How do we decide where to place a wildcard in the newly generated wildcard actions? In the next section, we will dive deep on how policy recommendation generalizes actions with wildcards to allow only those actions that do not appear in unused permissions.

A deep dive into how actions are generalized

Policy recommendation is guided by the mathematical principle of “least general generalization” — i.e., finding the least permissive modification of the recommended policy that still allows all the actions allowed by the original policy. This theorem-backed approach guarantees that the modified policy still allows all and only the permissions granted by the original policy that are not reported as unused.

To implement the least-general generalization for unused permissions, we construct a data structure known as a trie, which is a tree each of whose nodes extends a sequence of tokens corresponding to a path through the tree. In our case, the nodes represent prefixes shared among actions, with a special marker for actions reported in unused permissions. By traversing the trie, we find the shortest string of prefixes that does not contain unused actions.

The diagram below shows a simplified trie delineating actions that replace the S3 Get* wildcard from the original policy (we have omitted some actions for clarity):

Access Analyzer trie.png
A trie delineating actions that can replace the Get* wildcard in an IAM policy. Nodes containing unused actions are depicted in orange; the remaining nodes are in green.

At a high level, the trie represents prefixes that are shared by some of the possible actions starting with s3:Get. Its root node represents the prefix Get; child nodes of the root append their prefixes to Get. For example, the node named Multi represents all actions that start with GetMulti.

Related content
Automated reasoning and optimizations specific to CPU microarchitectures improve both performance and assurance of correct implementation.

We say that a node is safe (denoted in green in the diagram) if none of the unused actions start with the prefix corresponding to that node; otherwise, it is unsafe (denoted in orange). For example, the node s3:GetBucket is unsafe because the action s3:GetBucketPolicy is unused. Similarly, the node ss is safe since there are no unused permissions that start with GetAccess.

We want our final policies to contain wildcard actions that correspond only to safe nodes, and we want to include enough safe nodes to permit all used actions. We achieve this by selecting the nodes that correspond to the shortest safe prefixes—i.e., nodes that are themselves safe but whose parents are not. As a result, the recommended policy replaces s3:Get* with the shortest prefixes that do not contain unused permissions, such as s3:GetAccess*, s3:GetMulti* and s3:GetJobTagging.

Together, the shortest safe prefixes form a new policy that, while syntactically similar to the original policy, is the least-general generalization to result from removing the unused actions. In other words, we have not removed more actions than necessary.

You can find how to start using policy recommendation with unused access in Access Analyzer. To learn more about the theoretical foundations powering policy recommendation, be sure to check out our science paper.

Related content

ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.