Social
Economics

Measuring the effectiveness of software development tools and practices

New cost-to-serve-software metric that accounts for the full software development lifecycle helps determine which software development innovations provide quantifiable value.

By Willie deButts
July 29, 2025
9 min read
Share

At Amazon, we constantly seek ways to optimize software development tools, processes, and practices in order to improve outcomes and experiences for our customers. Internally, Amazon has the variety of businesses, team sizes, and technologies to enable research on engineering practices that span a wide variety of circumstances. Recently, we've been exploring how generative artificial intelligence (genAI) affects our cost-to-serve-software (CTS-SW) metric. This post delves into the research that led to CTS-SW’s development, how various new AI-powered tools can lower CTS-SW, and our future plans in this exciting area.

Understanding CTS-SW

We developed cost to serve software as a metric to quantify how investments in improving the efficiency of building and supporting software enable teams to easily, safely, and continually deploy software to customers. It bridges the gap between our existing framework, which tracks many metrics (similar to DORA and SPACE), and the quantifiable bottom-line impact on the business. It allows developer experience teams to express their business benefits in either effective capacity (engineering years saved) or the monetary value of those savings. In a recent blog post on the AWS Cloud Enterprise Strategy Blog, we described how CTS-SW can evaluate how initiatives throughout the software development lifecycle affect the ability to deliver for customers.

Guido Imbens AMLC keynote figure
Related content
Economics Nobelist on causal inference
In a keynote address at the latest Amazon Machine Learning Conference, Amazon academic research consultant, Stanford professor, and recent Nobel laureate Guido Imbens offered insights on the estimation of causal effects in “panel data” settings.

At a high level, CTS-SW tracks the dollars spent per unit of software reaching customers (i.e., released for use by customers). The best unit of software to use varies based on the software architecture. Deployment works well for microservices. Code reviews or pull requests that are shipped to a customer work well for monolith-based teams or software whose release is dictated by a predetermined schedule. Finally, commits that reach customers make sense for teams that contribute updates to a central code “trunk”. We currently use deployments, as it fits our widespread use of service-oriented architecture patterns and our local team ownership.

CTS-SW is based on the same theory that underlies the cost-to-serve metric in Amazon’s fulfillment network, i.e., that the delivery of a product to a customer is the result of an immeasurably complex and highly varied process and would be affected by the entirety of any changes to it. That process is so complex, and it changes so much over time, that the attempt to quantify each of its steps and assign costs to them, known as activity-based costing, is likely to fail. This is especially true of software engineering today, as new AI tools are changing the ways software engineers do their jobs.

Cost to serve simplifies this complex process by modeling only the input costs and the output units. We can then work backwards to understand drivers and opportunities for improvement.

CTS-16x9.gif
This equation represents the high-level CTS-SW setup.

In the context of software development, working backwards means that we investigate changes that could affect the metric, beyond the core coding experience of working in an IDE and writing logic. We also include continuous integration/continuous delivery (CI/CD) practices, work planning, incident management practices, maintenance of existing systems, searching for information, and many other factors that characterize software development at Amazon. By working backwards, we look across the collective software builder experience and investigate how changes in different areas, such as reducing the number of alarms engineers receive, affects developers’ ability to build new experiences for customers. We have used a variety of research methods to explore these relationships, but we have primarily relied on mathematical models.

From a science perspective, Amazon is an interesting place in which to build these models because of our established culture of small software teams that manage their own services. A longstanding Amazon principle is that these teams should be small enough to be fed by two pizzas, so we refer to them as “two-pizza teams”. This local-ownership model has led to the creation of thousands of distinct services solving customer problems across the company.

Amazon’s practice of working backwards from the best possible customer experience means software teams choose the optimal combination of tooling and technology to enable that experience. These choices have led to the implementation of many different software architectures at Amazon. That variety offers an opportunity to explore how different architectures affect CTS-SW.

Continuous Treatment architecture.png
Related content
Causal inference when treatments are continuous variables
Combining a cutting-edge causal-inference technique and end-to-end machine learning reduces root-mean-square error by 27% to 38%.

The Amazon Software Builder Experience (ASBX) team, our internal developer experience team, has access to rich telemetry data about these architectures and different ways of working with them. Using this data, we created a panel dataset representing the work of thousands of two-pizza teams over the past five years and including features we thought could affect CTS-SW. We model CTS-SW using the amount of developer time — the largest component of CTS-SW — per deployment. This data offers an opportunity for modeling the complete process from inception to delivery at a scale rarely seen in developer experience research.

Last year, as a first exploration of this dataset, we fit a set of linear mixed models to CTS-SW, to identify other metrics and behaviors that are highly correlated with it. Within ASBX, we were looking for input metrics that teams could optimize to lower CTS-SW. Correlations with linear mixed models can also help establish causal links between factors in the linear mixed models and CTS-SW. Linear mixed models are a good fit for this sort of problem because they have two components, one that captures the underlying relation between the outcome variable and the predictors, irrespective of team, and one that captures differences across teams.

Once we’d fit our models, we found that the following input metrics stood out as being the largest potential drivers of CTS-SW after a sensitivity analysis:

  • Team velocity: This measures how many code reviews (CRs) a software team merges each week per developer on the team. Teams that check in more code have a lower CTS-SW. Our science validates that software is a team sport, and framing this as a team-level outcome instead of an individual one prevents using CR flow as a performance metric for individual engineers. Having strong engineering onboarding and deployment safety helps teams reach and sustain high velocity. This was our largest single predictor of CTS-SW.
  • Delivery health (interventions per deploy, rollback rates): We find that teams that have implemented CI/CD with automation and change safety best practices have better CTS-SW outcomes. Our data demonstrates that when you spend less time wrestling with deployment friction and more time creating value, both productivity and job satisfaction improve.
  • Pages per on-call builder: This measures how many pages a team gets per week. We find that an increase in paging leads to lower CTS-SW, as paging can result in a deployment to production. However, we believe that work done in this reactive way may not be the most useful to customers in the long term. Understanding how this urgent, unplanned work interacts with new-feature delivery is an area for future research.

Our research has shown strong relationships between development factors and CTS-SW, making it an effective tool for measuring software development efficiency. We are working to expand the data we use in these models to better capture the ways in which teams build and operate their services. With this data, we will investigate the effects of software architecture decisions, informing architecture recommendations for teams across Amazon.

Validating linear mixed models with causal inference

Once we found that model fitting implied a correlation between team velocity and CTS-SW, we started looking for natural experiments that would help us validate the correlation with causal evidence. The rapidly emerging set of generative AI-powered tools provided that set of natural experiments.

Root-cause analysis.png
Related content
AWS contributes novel causal machine learning algorithms to DoWhy
New features go beyond conventional effect estimation by attributing events to individual components of complex systems.

The first of these tools adopted at scale across Amazon was Amazon Q Developer. This tool automatically generates code completions based on existing code and comments. We investigated the tool’s effect on CR velocity by building a panel regression model with dynamic two-way fixed effects.

This model uses time-varying covariates based on observations of software builder teams over multiple time periods during a nine-month observation window, and it predicts either CR velocity or deployment velocity. We specify the percentage of the team using Q Developer in each week and pass that information to the model as well.

We also evaluate other variables passed to the model to make sure they are exogenous, i.e., not influenced by Q Developer usage, to ensure that we can make claims of a causal relationship between Q Developer usage and deployment or CR velocity. These variables include data on rollbacks and manual interventions in order to capture the impact of production and deployment incidents, which may affect the way builders are writing code.

Here’s our model specification:

yit = ai + λt + βyi,t-1 + γXit + εit

In this equation, 𝑦𝑖𝑡 is the normalized deployments per builder week or team weekly velocity for team 𝑖 at time 𝑡, 𝑎𝑖 is the team-specific fixed effect, 𝜆𝑡 is the time-specific fixed effect, 𝑦𝑖,𝑡―1 is the lagged normalized deployments or team velocity, 𝑋𝑖𝑡 is the vector of time-varying covariates (Q Developer usage rate, rollback rate, manual interventions), 𝛽𝑖𝑡 is the persistence of our dependent variable over time (i.e., it shows how much of the past value of 𝑦 carries over into the current period), and 𝜀𝑖𝑡 is the error term.

Conditioning set.gif
Related content
Determining causality in correlated time series
New method goes beyond Granger causality to identify only the true causes of a target time series, given some graph constraints.

Early evidence shows that Q Developer has accelerated CR velocity and deployment velocity. More important, we found causal evidence that the launch of a new developer tool can lower CTS-SW for adopting teams and that we can measure that impact. As agentic AI grows, there will be agents for a range of tasks that engineers perform, beyond just writing code. That will require a unit of measurement that can capture their contributions holistically, without overly focusing on one area. CTS-SW enables us to measure the effects of AI across the software development lifecycle, from agents giving feedback on design docs to agents suggesting fixes to failed builds and deployments.

The road ahead

We recognize that combining experimental results can sometimes overstate an intervention’s true impact. To address this, we're developing a baseline model that we can use to normalize our tool-based approach to ensure that our estimates of AI impact are as accurate as possible.

Looking ahead, we plan to expand our analysis to include AI's impact on more aspects of the developer experience. By leveraging CTS-SW and developing robust methodologies for measuring AI's impact, we're ensuring that our AI adoption is truly customer obsessed, in that it makes Amazon’s software development more efficient. As we continue to explore and implement AI solutions, we remain committed to using data-driven approaches to improve outcomes and experiences for our customers. We look forward to sharing them with you at a later date.

Research areas
Tags
About the Author
Willie deButts
Willie deButts is a principal data scientist in the Amazon Software Builder Experience (ASBX) organization.

Related content

Work with us

See more jobs See more jobs
Data Scientist , Leo Customer Terminal
US, WA, Redmond
At Amazon, we’re inventing on behalf of customers, and with Amazon Leo, we’re redefining what global connectivity looks like. Our mission is to deliver fast, affordable broadband to unserved and underserved communities around the world through a constellation of low Earth orbit (LEO) satellites. Every system we build helps connect people to education, healthcare, opportunity, and each other. As a Data Scientist, you will be responsible for developing advanced analytics and machine learning solutions for user terminals. You will develop predictive models to proactively identify possible user terminal failures in the field. You will work in a collaborative environment with a multi-disciplinary team, including constellation, RF, antenna, silicon, algorithm, and software engineers. Key job responsibilities As a Data Scientist, you will develop analytic tools for a team developing current and future user terminals. Your responsibilities include: • Develop statistical and analytical tool to enable the regression decision from on-orbit and lab measurement of user terminals • Publish documents and create compelling visualizations and presentations to communicate insights to stakeholders • Create and manage datasets for continued pre-training and supervised fine-tuning of LLMs • Develop scalable visualizations for analysis of user terminal performance • Work closely with constellation, RF, antenna, silicon, algorithm, and software engineers to root-cause the failures using data as the primary tool • Drive consensus on metrics and analysis approaches to support product development strategy Export Control Requirement: Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. A day in the life As a Data Scientist in the LEO Customer Terminal Team, you will work daily with satellite constellation, algorithm, RF, antenna, silicon, hardware, and software teams in a collaborative environment. Your focus will be using data as an intelligence source to enable design decisions for the team. About the team The LEO Customer Terminal team is responsible for developing both outdoor and indoor devices that enable customers to access internet service via the LEO satellite network. We own the entire process from early prototypes through mass production, including requirements documentation, architecture definition, hardware development, algorithm development, and all integration and verification testing.
Senior Applied Scientist, Fauna
US, NY, New York
We are seeking an Applied Scientist to lead the development of evaluation frameworks and data collection protocols for robotic capabilities. In this role, you will focus on designing how we measure, stress-test, and improve robot behavior across a wide range of real-world tasks. Your work will play a critical role in shaping how policies are validated and how high-quality datasets are generated to accelerate system performance. You will operate at the intersection of robotics, machine learning, and human-in-the-loop systems, building the infrastructure and methodologies that connect teleoperation, evaluation, and learning. This includes developing evaluation policies, defining task structures, and contributing to operator-facing interfaces that enable scalable and reliable data collection. The ideal candidate is highly experimental, systems-oriented, and comfortable working across software, robotics, and data pipelines, with a strong focus on turning ambiguous capability goals into measurable and actionable evaluation systems. Key job responsibilities - Design and implement evaluation frameworks to measure robot capabilities across structured tasks, edge cases, and real-world scenarios - Develop task definitions, success criteria, and benchmarking methodologies that enable consistent and reproducible evaluation of policies - Create and refine data collection protocols that generate high-quality, task-relevant datasets aligned with model development needs - Build and iterate on teleoperation workflows and operator interfaces to support efficient, reliable, and scalable data collection - Analyze evaluation results and collected data to identify performance gaps, failure modes, and opportunities for targeted data collection - Collaborate with engineering teams to integrate evaluation tooling, logging systems, and data pipelines into the broader robotics stack - Stay current with advances in robotics, evaluation methodologies, and human-in-the-loop learning to continuously improve internal approaches - Lead technical projects from conception through production deployment - Mentor junior scientists and engineers
Satellite GNC Engineer, Amazon Leo
US, WA, Redmond
Amazon Leo is building a constellation of thousands of low Earth orbit satellites to deliver fast, reliable internet beyond the reach of existing networks. As a Guidance, Navigation & Control engineer, you will design the algorithms, simulations, and onboard autonomy that keep every satellite precisely pointed and stable so the payload can serve customers below, and keep the fleet flying safely. This is satellite GNC with demanding pointing and performance requirements, and you will see your work go from concept to orbit on a constellation that is launching and growing today. Amazon Leo will delight customers with reliable connectivity regardless of where they are. In this role, you will: - Take your GNC algorithms from concept all the way to orbit, and stay close to them through integration, test, and on-orbit operations - Solve hard precise-pointing, payload-cueing, and fleet-autonomy problems against demanding performance requirements - Work across the full lifecycle — design, simulation, test, and flight operations — rather than a narrow slice - Join a small team of some of the top GNC engineers in the industry, building a product people will love to use This position is part of the Satellite Attitude Determination and Control team, where we own the full lifecycle of our work — from algorithm design and flight software, through lab integration, to flying our own satellites on orbit. You will design and analyze the control system and algorithms, support development of our flight hardware and software, help integrate the satellite in our labs, and operate the GNC system in flight as a constellation of satellites flows through the production line into orbit. **Export Control Requirement:** Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. Key job responsibilities - Design and analyze algorithms for state estimation, attitude determination and control, precise pointing, and payload cueing against demanding performance requirements. - Write the production C++ flight code that runs these algorithms on the satellite. Our GNC engineers own their algorithms all the way into flight software, working alongside software development engineers rather than handing prototypes off to be reimplemented by another team. - Design concepts of operation that balance pointing and payload cueing, momentum management, power, thermal, and communications across all phases of the mission, including off-nominal and contingency scenarios. - Develop momentum management strategies using reaction wheels and magnetic torquers, and help size and lay out the GNC sensor and actuator suite — star trackers, magnetometers, sun sensors, IMUs, and GPS receivers — and the algorithms that consume them. - Develop onboard autonomy, including Fault Detection, Isolation, and Recovery, so the fleet operates safely at scale with minimal intervention. - Develop and validate models and simulations of the satellite and constellation across a range of fidelity levels, including 6-DOF time-domain simulation and Monte Carlo analysis, and run hardware-in-the-loop testing to verify closed-loop performance. - Support component-level environmental testing, functional and performance checkout, and subsystem and satellite integration in our labs. - Analyze fleet telemetry in bulk and build the metrics, dashboards, and tooling that let us monitor and continuously improve pointing performance and payload operations time as the constellation grows. - Operate the GNC system on orbit — reviewing fleet data, supporting maneuvers, and resolving anomalies when on-orbit behavior diverges from the ground model. - Contribute to architecture, design, and code reviews, and document your algorithms, analyses, and results clearly. A day in the life This role spans a diverse set of problems across the spacecraft and network system. You will derive and prototype estimation and control algorithms, turn them into production flight code, and prove them out through 6-DOF and Monte Carlo simulation and hardware-in-the-loop testing in the lab. You will support satellite integration, and when something on orbit does not behave the way the ground model predicted, you will dig into fleet telemetry to find out why and drive the fix. You will own problems end to end and have the focus to go deep on each one. Because GNC sits at the center of safe flight, you will work across the whole program. The payload, power, thermal, and operations teams all depend on GNC, so you will balance their needs against your pointing and momentum budgets. To get your algorithms running reliably on the spacecraft, you will partner closely with avionics, flight software, and our embedded software development engineers. You will help the mechanical teams shape the next generation of satellite hardware, from reaction wheels to star trackers. And as satellites move from design through manufacturing into orbit, you will work alongside systems engineering and Assembly, Integration & Test. The problems are genuinely hard — demanding pointing and performance requirements, a large constellation to keep flying safely, and the chance to invent advanced capabilities that have not been built before. The work is staffed with some of the strongest engineers in the industry, and the constant collaboration across teams makes it a place to keep learning and growing. About the team Our team brings deep experience across many satellite systems and other flight vehicles, with real bench strength in both our mission and the core GNC disciplines. We design, prototype, test, iterate, and learn together, and because GNC is central to safe flight, we tend to drive Concepts of Operation and many of the system-level analyses on the program. We also have a lot of freedom. We trust our engineers to pursue the problems they face in whatever way they find best, to choose their own tools and approaches, and to own the results. If you want the autonomy to solve hard problems your way, and a team of strong engineers to do it with, you will feel at home here.
Applied Scientist, Fauna
US, NY, New York
We are seeking an Applied Scientist to lead the development of evaluation frameworks and data collection protocols for robotic capabilities. In this role, you will focus on designing how we measure, stress-test, and improve robot behavior across a wide range of real-world tasks. Your work will play a critical role in shaping how policies are validated and how high-quality datasets are generated to accelerate system performance. You will operate at the intersection of robotics, machine learning, and human-in-the-loop systems, building the infrastructure and methodologies that connect teleoperation, evaluation, and learning. This includes developing evaluation policies, defining task structures, and contributing to operator-facing interfaces that enable scalable and reliable data collection. The ideal candidate is highly experimental, systems-oriented, and comfortable working across software, robotics, and data pipelines, with a strong focus on turning ambiguous capability goals into measurable and actionable evaluation systems. Key job responsibilities - Design and implement evaluation frameworks to measure robot capabilities across structured tasks, edge cases, and real-world scenarios - Develop task definitions, success criteria, and benchmarking methodologies that enable consistent and reproducible evaluation of policies - Create and refine data collection protocols that generate high-quality, task-relevant datasets aligned with model development needs - Build and iterate on teleoperation workflows and operator interfaces to support efficient, reliable, and scalable data collection - Analyze evaluation results and collected data to identify performance gaps, failure modes, and opportunities for targeted data collection - Collaborate with engineering teams to integrate evaluation tooling, logging systems, and data pipelines into the broader robotics stack - Stay current with advances in robotics, evaluation methodologies, and human-in-the-loop learning to continuously improve internal approaches - Lead technical projects from conception through production deployment - Mentor junior scientists and engineers
Applied Scientist, Advertiser Growth Engine
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering a collection of advertising products that drive discovery and sales. Our products are strategically important to our businesses driving long term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! Advertiser Growth Engine (AGE) team owns and builds services and applications across Amazon World-Wide Advertising that make advertising across multi-marketplaces as easy as flipping on a switch. We are focused on: (1) expanding Amazon Ads advertiser base, and (2) eliminating localization, operational, and marketplace knowledge gap barrier for Advertisers who advertise across multiple marketplaces. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth globally. We're looking for an experienced Applied Scientist with exceptional technical, analytical, and innovative capabilities to research, design, and create elegant machine learning solutions. The solutions will help our advertisers with multi-media and multi-lingual advertising offerings. You will use ideas from various domains of machine learning, including supervised and unsupervised methods, Deep Neural Networks, Natural Language Processing (NLP), and Computer Vision (CV) to build ML models that localizes multi-media advertising contents, including text, images and videos. You will also identify opportunities to leverage ML beyond localization, including, international expansion and global campaigns. Your work will directly impact our customers in the form of products and services used directly by our advertisers as well as our third-party integrators. As an Applied Scientist on this team, you will: - Build and deliver end-to-end machine learning solutions; build ML models and perform data analysis to deliver scalable solutions to business problems. - Perform hands-on analysis and modeling with enormous data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. - Work closely with software engineers on detailed requirements to productionize the ML models you build. - Run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Establish scalable, efficient, automated processes for large-scale data analysis, machine-learning model development, model validation and serving. - Research new innovate machine learning approaches. Why you will love this opportunity: Amazon is investing heavily in building a world-class advertising business. This team defines and delivers a collection of advertising products that drive discovery and sales. Our solutions generate billions in revenue and drive long-term growth for Amazon’s Retail and Marketplace businesses. We deliver billions of ad impressions, millions of clicks daily, and break fresh ground to create world-class products. We are a highly motivated, collaborative, and fun-loving team with an entrepreneurial spirit - with a broad mandate to experiment and innovate. Impact and Career Growth: You will invent new experiences and influence customer-facing shopping experiences to help suppliers grow their retail business and the auction dynamics that leverage native advertising; this is your opportunity to work within the fastest-growing businesses across all of Amazon! Define a long-term science vision for our advertising business, driven from our customers' needs, translating that direction into specific plans for research and applied scientists, as well as engineering and product teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. Team video https://youtu.be/zD_6Lzw8raE
Applied Science Manager, Personalization
IL, Tel Aviv
Are you a scientist interested in pushing the state of the art in Information Retrieval, Large Language Models and Recommendation Systems? Are you interested in innovating on behalf of millions of customers, helping them accomplish their every day goals? Do you wish you had access to large datasets and tremendous computational resources? Do you want to join a team of capable scientist and engineers, building the future of e-commerce? Answer yes to any of these questions, and you will be a great fit for our team at Amazon. Our team is part of Amazon’s Personalization organization, a high-performing group that leverages Amazon’s expertise in machine learning, generative AI, large-scale data systems, and user experience design to deliver the best shopping experiences for our customers. Our team is building next-generation personalization systems powered by Large Language Models. We are tackling novel research challenges to help customers discover products they'll love - at Amazon scale and latency requirements. We are a team uniquely placed within Amazon, to have a direct window of opportunity to influence how customers will think about their shopping journey in the future. As an Applied Science Manager, you will lead a team of scientists working at the frontier of LLM-based personalization. You will set the technical vision, drive the research agenda, and ensure your team delivers production-ready solutions. You will hire, mentor, and develop world-class scientists while fostering a culture of innovation and scientific rigor. You will partner closely with engineering and product teams to translate ambitious research into customer-facing impact, and represent your team's work to senior leadership. Please visit https://www.amazon.science for more information.
Sr. Applied Scientist, AWS Applied AI Solutions - Life Sciences
US, WA, Seattle
AWS Applied AI Solutions (AAIS) is building toward a future where every business innovates with Amazon AI teammates. To get there, we build AI solutions that improve human capabilities and transform entire business functions. We create end-to-end products that surprise and delight out-of-the-box, making complex things easy and hard things possible, with no cloud experience required. We start with customers who embrace the future and build bridges to meet the rest where they are. We pursue ambitious opportunities with conviction, and we are looking for builders who share that mindset. The Team Join the next science revolution at AWS Life Sciences Applied AI Solutions where you'll work alongside world-class scientists to build AI that transforms how therapeutics are discovered, developed, and brought to patients. We're out to revolutionize how medicines are discovered, developed, and brought to patients powered by a new generation of AI. Our team tackles some of the hardest open problems at the intersection of frontier AI and life sciences. We apply biological foundation models large language models and agentic reasoning systems to life sciences problems then put them into the hands of customers as applications and managed services they can fine-tune tailor and deploy on their own data. The science challenges are deep: how do you design agentic systems that reason correctly over complex biological regulatory and clinical logic? How do you enable customers to tailor foundation models to their proprietary data and get better outputs with less effort? How do you adapt models to reason faithfully in high-stakes scientific and regulatory domains? Today we're focused on two areas. In clinical trials we're building AI that automates and optimizes regulatory and clinical development workflows. In drug design our products (including Amazon Bio Discovery) accelerate discovery by giving bench scientists AI-guided protein engineering and antibody design capabilities. We combine frontier research with production-scale delivery to put breakthrough science into the hands of customers solving humanity's hardest problems. We value scientific rigor encourage publication and support conference participation. If you want to do research that ships this is the team. The Role We are seeking an Applied Scientist to build the models and methods behind our life sciences AI products with a primary focus on clinical trial operations and agentic reasoning. You will design train and evaluate systems that reason over complex clinical and operational logic and ship them into products customers use directly. You will work closely with senior and principal scientists on well-scoped research problems own your results end to end and see your work reach production. This role combines expertise in LLM reasoning and agentic AI with applied impact in life sciences. You will work on how large language models reason plan and act in complex scientific domains while applying domain knowledge to ensure models produce scientifically valid outputs. The problems span multiple fronts: • How do you build LLM-based agentic systems that correctly reason over clinical protocols regulatory standards and complex multi-step operational workflows? • How do you evaluate agent reliability and faithfulness rigorously enough to trust in high-stakes clinical settings? • How do you develop model customization methods (fine-tuning retrieval augmentation domain adaptation) that let customers get strong results from foundation models on their own data? You will focus on clinical trial operations (agentic automation structured reasoning evaluation domain adaptation) with opportunities to contribute across drug discovery (protein engineering antibody design) as the portfolio grows. You will own end-to-end scientific solutions from research through production and your work will directly shape the tools that scientists use daily. Key job responsibilities • Design train fine-tune and evaluate LLM-based agentic systems that reason over clinical protocols regulatory standards and operational workflows • Build rigorous evaluation harnesses and benchmarks to measure agent reliability faithfulness and failure modes in high-stakes domains • Develop model customization methods (fine-tuning RLHF retrieval augmentation domain adaptation) that help customers get better outputs on their own data with less effort • Contribute to graph-based and causal modeling approaches for clinical trial operations • Partner with Life Sciences domain experts product and engineering to translate scientific challenges into shipped capabilities • Own experiments end to end: problem framing implementation evaluation iteration and hand-off to production • Publish at top-tier venues where the work supports it • Contribute to drug discovery efforts (protein engineering antibody design) as opportunities arise A day in the life • Design and run an experiment to validate a new agentic reasoning or fine-tuning method then ship it as a capability customers can use • Diagnose why a model is failing on a new class of inputs and implement a fix to unblock a delivery milestone • Build or extend an evaluation benchmark to measure how faithfully an agent reasons over clinical logic • Meet with domain experts to scope what the next model release needs to do • Review results with a senior scientist sharpen the approach and get it over the finish line • Prototype a new idea that could become the next capability in the product About the team Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying. Amazon Web Services (AWS) is the world's most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that's why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there's nothing we can't achieve in the cloud. Here at AWS, it's in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity and AmazeCon conferences, inspire us to never stop embracing our uniqueness. We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
Data Scientist III, Amazon 1P Credito, Payments
BR, SP, Sao Paulo
Do you feel the challenge and the adrenaline kick when a huge data-set stares you in the face and you know that somewhere inside are hidden very important business insights that can fundamentally alter the way top business leaders think and act? Do you enjoy presenting strong data backed insights to business leaders; insights that can topple their long held beliefs and compel them to change their direction completely? If yes, then you are the one we are looking for. We are looking to invite passionate leaders, with expertise in generate power business insights from very large datasets, on a journey where the primary aim would be to enable needle moving business impacts through statistical analysis. We are looking for leaders who can envision the design and development of analytical infrastructure which can support strategic and tactical decision-making. Those who join this high visibility team would have to navigate through significant ambiguity in defining business problems and converting them to analytical problems. This role requires additional exposure and experience to Machine Learning. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems • Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes • Design, development, evaluate and deploy innovative and highly scalable ml models such as risk scorecards, income models, fraud models for predictive learning in credit risk applications • Research and implement novel machine learning and statistical approaches • Work closely with software engineering teams to drive real-time model implementations and new feature creations • Work closely with business owners and operations staff to optimize various business operations • Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation • Mentor other scientists and engineers in the use of ML techniques • Innovate with the latest GenAI technology to build highly automated solutions for efficient customer promotions • Design, develop and deploy end-to-end machine learning solutions in the Amazon production environment to delight Amazon customers • Collaborate with cross-functional teams to develop comprehensive ML/statistical models that can scale to millions of customers to multiple countries Understand the credit risk data and evaluate the best ml model/ solution for dynamic business problems. About the team Brazil Payments is part of the International Emerging Stores Payments team and focuses on supporting the launch of new payment and financial products to our customers in Brazil.
Data Scientist II, Amazon 1P Credito, Payments
BR, SP, Sao Paulo
Do you feel the challenge and the adrenaline kick when a huge data-set stares you in the face and you know that somewhere inside are hidden very important business insights that can fundamentally alter the way top business leaders think and act? Do you enjoy presenting strong data backed insights to business leaders; insights that can topple their long held beliefs and compel them to change their direction completely? If yes, then you are the one we are looking for. We are looking to invite passionate leaders, with expertise in generate power business insights from very large datasets, on a journey where the primary aim would be to enable needle moving business impacts through statistical analysis. We are looking for leaders who can envision the design and development of analytical infrastructure which can support strategic and tactical decision-making. Those who join this high visibility team would have to navigate through significant ambiguity in defining business problems and converting them to analytical problems. This role requires additional exposure and experience to Machine Learning. Key job responsibilities • Use machine learning and analytical techniques to create scalable solutions for business problems • Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes • Design, development, evaluate and deploy innovative and highly scalable models for predictive learning • Research and implement novel machine learning and statistical approaches • Work closely with software engineering teams to drive real-time model implementations and new feature creations • Work closely with business owners and operations staff to optimize various business operations • Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation • Mentor other scientists and engineers in the use of ML techniques • Innovate with the latest GenAI technology to build highly automated solutions for efficient customer promotions • Design, develop and deploy end-to-end machine learning solutions in the Amazon production environment to delight Amazon customers • Collaborate with cross-functional teams to develop comprehensive ML/statistical models that can scale to millions of customers to multiple countries About the team Brazil Payments is part of the International Emerging Stores Payments team and focuses on supporting the launch of new payment and financial products to our customers in Brazil.
Data Scientist III, Amazon 1P Credito, Payments
BR, SP, Sao Paulo
Do you feel the challenge and the adrenaline kick when a huge data-set stares you in the face and you know that somewhere inside are hidden very important business insights that can fundamentally alter the way top business leaders think and act? Do you enjoy presenting strong data backed insights to business leaders; insights that can topple their long held beliefs and compel them to change their direction completely? If yes, then you are the one we are looking for. We are looking to invite passionate leaders, with expertise in generate power business insights from very large datasets, on a journey where the primary aim would be to enable needle moving business impacts through statistical analysis. We are looking for leaders who can envision the design and development of analytical infrastructure which can support strategic and tactical decision-making. Those who join this high visibility team would have to navigate through significant ambiguity in defining business problems and converting them to analytical problems. This role requires additional exposure and experience to Machine Learning. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems • Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes • Design, development, evaluate and deploy innovative and highly scalable ml models such as risk scorecards, income models, fraud models for predictive learning in credit risk applications • Research and implement novel machine learning and statistical approaches • Work closely with software engineering teams to drive real-time model implementations and new feature creations • Work closely with business owners and operations staff to optimize various business operations • Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation • Mentor other scientists and engineers in the use of ML techniques • Innovate with the latest GenAI technology to build highly automated solutions for efficient customer promotions • Design, develop and deploy end-to-end machine learning solutions in the Amazon production environment to delight Amazon customers • Collaborate with cross-functional teams to develop comprehensive ML/statistical models that can scale to millions of customers to multiple countries Understand the credit risk data and evaluate the best ml model/ solution for dynamic business problems. About the team Brazil Payments is part of the International Emerging Stores Payments team and focuses on supporting the launch of new payment and financial products to our customers in Brazil.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social