Custom policy checks help democratize automated reasoning

New IAM Access Analyzer feature uses automated reasoning to ensure that access policies written in the IAM policy language don’t grant unintended access.

To control access to resources in the Amazon Web Services (AWS) Cloud, customers can author AWS Identity and Access Management (IAM) policies. The IAM policy language is expressive, allowing you to create fine-grained policies that control who can perform what actions on which resources. This control can be used to enforce the principle of least privilege, granting only the permissions required to perform a task.

But how can you verify that your IAM policies meet your security requirements? At AWS’s 2023 re:Invent conference, we announced the launch of IAM Access Analyzer custom policy checks, which help you benchmark policies against your security standards. Custom policy checks abstract away the task of converting policy statements into mathematical formulas, so customers can enjoy the benefits of automated reasoning without expertise in formal logic.

Policy checks in context.png
The role of IAM Access Analyzer custom policy checks in the development pipeline.

The IAM Access Analyzer API CheckNoNewAccess ensures that you do not inadvertently add permissions to a policy when you update it. With the CheckAccessNotGranted API, you can specify critical permissions that developers should not grant in their IAM policies.

We built custom policy checks on an internal AWS service called Zelkova, which uses automated reasoning to analyze IAM policies. Previously, we used Zelkova to build preventative and detective managed controls, such as Amazon S3 Block Public Access and IAM Access Analyzer public and cross-account findings. Now, with the release of custom policy checks, you can set a security standard and prevent policies that do not meet this standard from being deployed.

How does Zelkova work?

Zelkova models the semantics of the IAM policy language by translating policies into precise mathematical expressions. It then uses automated engines called satisfiability modulo theories (SMT) solvers to check properties of the policies. Satisfiability (SAT) solvers check if it is possible to assign true or false values to Boolean variables to satisfy a set of constraints; SMT is a generalization of SAT to include strings, integers, real numbers, or functions. The benefit of using SMT to analyze policies is that it is comprehensive. Unlike tools that simulate or evaluate a policy for a given request or a small set of requests, Zelkova can check properties of a policy for all possible requests.

Consider the following Amazon S3 bucket policy:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": ["s3:PutObject"],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
      }
   ]
}

Zelkova translates this policy into the following formula:

(Action = “s3:PutObject”) 
∧ (Resource = “arn:aws:s3:::DOC-EXAMPLE-BUCKET”)

In this formula, "∧" is the mathematical symbol for “and”. Action and Resource are variables that represent values from any possible request. The formula is true only when a request is allowed by the policy. This precise mathematical representation of a policy is useful because it allows us to answer questions about the policy exhaustively. For example, we can ask if the policy allows public access, and we receive the answer that it does.

For simple policies such as the preceding policy, we could perform manual reviews to determine whether they allow public access: the "Principal": "*" in the policy’s statement means that anyone (the public) is allowed access. But manual review can be error prone and is not scalable.

Alternatively, we could write simple syntactic checks for patterns such as "Principal": "*". However, these syntactic checks can miss the subtleties of policies and the interactions between different parts of a policy. Consider the following modification of the preceding policy, which adds a Deny statement with "NotPrincipal": "123456789012"; the policy still has the pattern "Principal": "*", but it no longer allows public access:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": ["s3:PutObject"],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
      },
      {
         "Effect": "Deny",
         "NotPrincipal": "123456789012",
         "Action": "*",
         "Resource": "*"
      }
   ]
}

With the mathematical representation of policy semantics in Zelkova, we can answer questions about access privileges precisely.

Answering questions with Zelkova

As an example, let’s consider a relatively simple question. With IAM policies, you can grant cross-account access to resources you want to share. For sensitive resources, you’d like to check that cross-account access is not possible.

Suppose we wanted to check whether the preceding policies allow anyone outside my account, 123456789012, to access my S3 bucket. Just as we translated the policy into a mathematical formula, we can translate the question we want to ask (or property we want to check) into a mathematical formula. To check whether all allowed accesses are from my account, we can translate the property to the following formula:

(Principal = 123456789012)

To show that the property holds true for the policy, we can now try to prove that only requests with (Principal = 123456789012) are allowed by the policy. A common trick used in mathematics is to flip the question around. Instead of trying to prove that the property holds, we can prove that it does not hold by finding requests that do not satisfy it — in other words, requests that satisfy (Principal 123456789012). To find such a counterexample, we look for assignments to the variables Principal, Action, and Resource such that the following is true:

(Action = “s3:PutObject”)
∧ (Resource = “arn:aws:s3:::DOC-EXAMPLE-BUCKET”)
∧ (Principal ≠ 123456789012)

Zelkova translates the policy and property into the preceding mathematical formula, and it efficiently searches for counterexamples using SMT solvers. For the preceding formula, the SMT solver can produce a counterexample showing that such access is indeed allowed by the policy (for example, with Principal = 111122223333).

For the previously modified policy with the Deny statement, the SMT solver can also prove that no solution is possible for the resulting formula and that no access is allowed for the policy from outside my account, 123456789012:

(Action = “s3:PutObject”) 
∧ (Resource = “arn:aws:s3:::DOC-EXAMPLE-BUCKET”) 
∧ (Principal = 123456789012) ∧ (Principal ≠ 123456789012)

The Deny statement in the policy with "NotPrincipal": "123456789012" is translated to the constraint (Principal = 123456789012). By inspecting the preceding formula, we can see that it can’t be satisfied: the constraints on Principal from the policy and from the property are contradictory. An SMT solver can prove this and more complicated formulas by exhaustively ruling out solutions.

Custom policy checks

To democratize access to Zelkova, we needed to abstract the construction of mathematical formulas behind a more accessible interface. To that end, we launched IAM Access Analyzer custom policy checks with two predefined checks: CheckNoNewAccess and CheckAccessNotGranted.

With CheckNoNewAccess, you can confirm that you do not accidentally add permissions to a policy when updating it. Developers often start with more-permissive policies and refine them over time toward least privilege. With CheckNoNewAccess, you can now compare two versions of a policy to confirm that the new version is not more permissive than the old version.

Suppose a developer updates the first example policy in this post to disallow cross-account access but at the same time also adds a new action:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": "123456789012",
         "Action": [ 
            "s3:PutObject",
            "s3:DeleteBucket" 
         ],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
      }
   ]
}

CheckNoNewAccess translates the two versions of the policy into formulas Pold and Pnew, respectively. It then searches for solutions to the formula (Pnew ¬Pold) that represent requests that are allowed by the new policy but not allowed by the old policy (“¬” is the mathematical symbol for “not”). Because the new policy allows principals in 123456789012 to perform an action that the old policy did not, the check fails, and a security engineer can review whether this policy change is acceptable.

With CheckAccessNotGranted, security engineers can be more prescriptive by specifying critical permissions to be checked against policy updates. Let’s say we want to ensure that developers are not granting permissions to delete an important bucket. In our previous example, CheckNoNewAccess detected this only because the permission was added with an update. With CheckAccessNotGranted, the security engineer can specify s3:DeleteBucket as a critical permission. We then translate the critical permissions into a formula such as (Action = “s3:DeleteBucket”) and search for requests with that action that are allowed by the policy. Because the preceding policy allows this action, the check fails and that prevents the permission from being deployed.

With the ability to specify critical permissions as parameters to the CheckAccessNotGranted API, you can now check policies against your standards — and not just for canned, broadly applicable checks.

Debugging failures

By democratizing policy checks, without the need for costly and time-consuming manual reviews, custom policy checks help developers move faster. When policies pass the checks, developers can make updates with confidence. If policies fail the checks, IAM Access Analyzer provides additional information so that developers can debug and fix them.

Suppose a developer writes the following identity-based policy:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "ec2:DescribeInstance*",
            "ec2:StartInstances", 
            "ec2:StopInstances" 
         ],
         "Resource": "arn:aws:ec2:*:*:instance/*"
      },
      {
         "Effect": "Allow",
         "Action": [ 
            "s3:GetObject*", 
            "s3:PutObject",
            "s3:DeleteBucket" 
         ],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
      }
   ]
}

Let’s also suppose that a security engineer has specified critical permissions that include s3:DeleteBucket. As described above, CheckAccessNotGranted fails on this policy.

For any given policy, it can sometimes be hard to understand why a check failed. To give developers more clarity, IAM Access Analyzer uses Zelkova to solve additional problems that pinpoint the failure to a specific statement in the policy. For the preceding policy, the check failed with the description "New access in the statement with index: 1". This description indicates that the second statement contains a critical permission.

The key to democratizing automated reasoning is to make it simple to use and easy to specify properties. With additional custom checks, we will continue to enable our customers on their journey to least privilege.

Research areas

Related content

US, WA, Bellevue
We are seeking a passionate, talented, and inventive individual to join the Applied AI team and help build industry-leading technologies that customers will love. This team offers a unique opportunity to make a significant impact on the customer experience and contribute to the design, architecture, and implementation of a cutting-edge product. The mission of the Applied AI team is to enable organizations within Worldwide Amazon.com Stores to accelerate the adoption of AI technologies across various parts of our business. We are looking for a Senior Applied Scientist to join our Applied AI team to work on LLM-based solutions. On our team you will push the boundaries of ML and Generative AI techniques to scale the inputs for hundreds of billions of dollars of annual revenue for our eCommerce business. If you have a passion for AI technologies, a drive to innovate and a desire to make a meaningful impact, we invite you to become a valued member of our team. You will be responsible for developing and maintaining the systems and tools that enable us to accelerate knowledge operations and work in the intersection of Science and Engineering. You will push the boundaries of ML and Generative AI techniques to scale the inputs for hundreds of billions of dollars of annual revenue for our eCommerce business. If you have a passion for AI technologies, a drive to innovate and a desire to make a meaningful impact, we invite you to become a valued member of our team. We are seeking an experienced Scientist who combines superb technical, research, analytical and leadership capabilities with a demonstrated ability to get the right things done quickly and effectively. This person must be comfortable working with a team of top-notch developers and collaborating with our research teams. We’re looking for someone who innovates, and loves solving hard problems. You will be expected to have an established background in building highly scalable systems and system design, excellent project management skills, great communication skills, and a motivation to achieve results in a fast-paced environment. You should be somebody who enjoys working on complex problems, is customer-centric, and feels strongly about building good software as well as making that software achieve its operational goals.
IN, KA, Bengaluru
Do you want to lead the development of advanced machine learning systems that protect millions of customers and power a trusted global eCommerce experience? Are you passionate about modeling terabytes of data, solving highly ambiguous fraud and risk challenges, and driving step-change improvements through scientific innovation? If so, the Amazon Buyer Risk Prevention (BRP) Machine Learning team may be the right place for you. We are seeking a Senior Applied Scientist to define and drive the scientific direction of large-scale risk management systems that safeguard millions of transactions every day. In this role, you will lead the design and deployment of advanced machine learning solutions, influence cross-team technical strategy, and leverage emerging technologies—including Generative AI and LLMs—to build next-generation risk prevention platforms. Key job responsibilities Lead the end-to-end scientific strategy for large-scale fraud and risk modeling initiatives Define problem statements, success metrics, and long-term modeling roadmaps in partnership with business and engineering leaders Design, develop, and deploy highly scalable machine learning systems in real-time production environments Drive innovation using advanced ML, deep learning, and GenAI/LLM technologies to automate and transform risk evaluation Influence system architecture and partner with engineering teams to ensure robust, scalable implementations Establish best practices for experimentation, model validation, monitoring, and lifecycle management Mentor and raise the technical bar for junior scientists through reviews, technical guidance, and thought leadership Communicate complex scientific insights clearly to senior leadership and cross-functional stakeholders Identify emerging scientific trends and translate them into impactful production solutions
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
GB, London
We are looking for a Senior Economist to work on exciting and challenging business problems related to Amazon Retail’s worldwide product assortment. You will build innovative solutions based on econometrics, machine learning, and experimentation. You will be part of a interdisciplinary team of economists, product managers, engineers, and scientists, and your work will influence finance and business decisions affecting Amazon’s vast product assortment globally. If you have an entrepreneurial spirit, you know how to deliver results fast, and you have a deeply quantitative, highly innovative approach to solving problems, and long for the opportunity to build pioneering solutions to challenging problems, we want to talk to you. Key job responsibilities * Work on a challenging problem that has the potential to significantly impact Amazon’s business position * Develop econometric models and experiments to measure the customer and financial impact of Amazon’s product assortment * Collaborate with other scientists at Amazon to deliver measurable progress and change * Influence business leaders based on empirical findings
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions that attack India first (and other Emerging Markets across MENA and LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
EG, Cairo
Are you a MS or PhD student interested in a 2026 internship in the field of machine learning, deep learning, generative AI, large language models and speech technology, robotics, computer vision, optimization, operations research, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Jordan, Luxembourg, Netherlands, Poland, Romania, Spain, South Africa, UAE, and UK). Please note these are not remote internships.
US, CA, San Diego
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to apply their macroeconomics and forecasting skillsets to solve real world problems. The intern will work in the area of forecasting, developing models to improve the success of new product launches in Private Brands. Our PhD Economist Internship Program offers hands-on experience in applied economics, supported by mentorship, structured feedback, and professional development. Interns work on real business and research problems, building skills that prepare them for full-time economist roles at Amazon and beyond. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis About the team The Amazon Private Brands Intelligence team applies Machine Learning, Statistics and Econometrics/economics to solve high-impact business problems, develop prototypes for Amazon-scale science solutions, and optimize key business functions of Amazon Private Brands and other Amazon orgs. We are an interdisciplinary team, using science and technology and leveraging the strengths of engineers and scientists to build solutions for some of the toughest business problems at Amazon, covering areas such as pricing, discovery, negotiation, forecasting, supply chain and product selection/development.