Social
Economics

Measuring the effectiveness of software development tools and practices

New cost-to-serve-software metric that accounts for the full software development lifecycle helps determine which software development innovations provide quantifiable value.

By Willie deButts
July 29, 2025
9 min read
Share

At Amazon, we constantly seek ways to optimize software development tools, processes, and practices in order to improve outcomes and experiences for our customers. Internally, Amazon has the variety of businesses, team sizes, and technologies to enable research on engineering practices that span a wide variety of circumstances. Recently, we've been exploring how generative artificial intelligence (genAI) affects our cost-to-serve-software (CTS-SW) metric. This post delves into the research that led to CTS-SW’s development, how various new AI-powered tools can lower CTS-SW, and our future plans in this exciting area.

Understanding CTS-SW

We developed cost to serve software as a metric to quantify how investments in improving the efficiency of building and supporting software enable teams to easily, safely, and continually deploy software to customers. It bridges the gap between our existing framework, which tracks many metrics (similar to DORA and SPACE), and the quantifiable bottom-line impact on the business. It allows developer experience teams to express their business benefits in either effective capacity (engineering years saved) or the monetary value of those savings. In a recent blog post on the AWS Cloud Enterprise Strategy Blog, we described how CTS-SW can evaluate how initiatives throughout the software development lifecycle affect the ability to deliver for customers.

Guido Imbens AMLC keynote figure
Related content
Economics Nobelist on causal inference
In a keynote address at the latest Amazon Machine Learning Conference, Amazon academic research consultant, Stanford professor, and recent Nobel laureate Guido Imbens offered insights on the estimation of causal effects in “panel data” settings.

At a high level, CTS-SW tracks the dollars spent per unit of software reaching customers (i.e., released for use by customers). The best unit of software to use varies based on the software architecture. Deployment works well for microservices. Code reviews or pull requests that are shipped to a customer work well for monolith-based teams or software whose release is dictated by a predetermined schedule. Finally, commits that reach customers make sense for teams that contribute updates to a central code “trunk”. We currently use deployments, as it fits our widespread use of service-oriented architecture patterns and our local team ownership.

CTS-SW is based on the same theory that underlies the cost-to-serve metric in Amazon’s fulfillment network, i.e., that the delivery of a product to a customer is the result of an immeasurably complex and highly varied process and would be affected by the entirety of any changes to it. That process is so complex, and it changes so much over time, that the attempt to quantify each of its steps and assign costs to them, known as activity-based costing, is likely to fail. This is especially true of software engineering today, as new AI tools are changing the ways software engineers do their jobs.

Cost to serve simplifies this complex process by modeling only the input costs and the output units. We can then work backwards to understand drivers and opportunities for improvement.

CTS-16x9.gif
This equation represents the high-level CTS-SW setup.

In the context of software development, working backwards means that we investigate changes that could affect the metric, beyond the core coding experience of working in an IDE and writing logic. We also include continuous integration/continuous delivery (CI/CD) practices, work planning, incident management practices, maintenance of existing systems, searching for information, and many other factors that characterize software development at Amazon. By working backwards, we look across the collective software builder experience and investigate how changes in different areas, such as reducing the number of alarms engineers receive, affects developers’ ability to build new experiences for customers. We have used a variety of research methods to explore these relationships, but we have primarily relied on mathematical models.

From a science perspective, Amazon is an interesting place in which to build these models because of our established culture of small software teams that manage their own services. A longstanding Amazon principle is that these teams should be small enough to be fed by two pizzas, so we refer to them as “two-pizza teams”. This local-ownership model has led to the creation of thousands of distinct services solving customer problems across the company.

Amazon’s practice of working backwards from the best possible customer experience means software teams choose the optimal combination of tooling and technology to enable that experience. These choices have led to the implementation of many different software architectures at Amazon. That variety offers an opportunity to explore how different architectures affect CTS-SW.

Continuous Treatment architecture.png
Related content
Causal inference when treatments are continuous variables
Combining a cutting-edge causal-inference technique and end-to-end machine learning reduces root-mean-square error by 27% to 38%.

The Amazon Software Builder Experience (ASBX) team, our internal developer experience team, has access to rich telemetry data about these architectures and different ways of working with them. Using this data, we created a panel dataset representing the work of thousands of two-pizza teams over the past five years and including features we thought could affect CTS-SW. We model CTS-SW using the amount of developer time — the largest component of CTS-SW — per deployment. This data offers an opportunity for modeling the complete process from inception to delivery at a scale rarely seen in developer experience research.

Last year, as a first exploration of this dataset, we fit a set of linear mixed models to CTS-SW, to identify other metrics and behaviors that are highly correlated with it. Within ASBX, we were looking for input metrics that teams could optimize to lower CTS-SW. Correlations with linear mixed models can also help establish causal links between factors in the linear mixed models and CTS-SW. Linear mixed models are a good fit for this sort of problem because they have two components, one that captures the underlying relation between the outcome variable and the predictors, irrespective of team, and one that captures differences across teams.

Once we’d fit our models, we found that the following input metrics stood out as being the largest potential drivers of CTS-SW after a sensitivity analysis:

  • Team velocity: This measures how many code reviews (CRs) a software team merges each week per developer on the team. Teams that check in more code have a lower CTS-SW. Our science validates that software is a team sport, and framing this as a team-level outcome instead of an individual one prevents using CR flow as a performance metric for individual engineers. Having strong engineering onboarding and deployment safety helps teams reach and sustain high velocity. This was our largest single predictor of CTS-SW.
  • Delivery health (interventions per deploy, rollback rates): We find that teams that have implemented CI/CD with automation and change safety best practices have better CTS-SW outcomes. Our data demonstrates that when you spend less time wrestling with deployment friction and more time creating value, both productivity and job satisfaction improve.
  • Pages per on-call builder: This measures how many pages a team gets per week. We find that an increase in paging leads to lower CTS-SW, as paging can result in a deployment to production. However, we believe that work done in this reactive way may not be the most useful to customers in the long term. Understanding how this urgent, unplanned work interacts with new-feature delivery is an area for future research.

Our research has shown strong relationships between development factors and CTS-SW, making it an effective tool for measuring software development efficiency. We are working to expand the data we use in these models to better capture the ways in which teams build and operate their services. With this data, we will investigate the effects of software architecture decisions, informing architecture recommendations for teams across Amazon.

Validating linear mixed models with causal inference

Once we found that model fitting implied a correlation between team velocity and CTS-SW, we started looking for natural experiments that would help us validate the correlation with causal evidence. The rapidly emerging set of generative AI-powered tools provided that set of natural experiments.

Root-cause analysis.png
Related content
AWS contributes novel causal machine learning algorithms to DoWhy
New features go beyond conventional effect estimation by attributing events to individual components of complex systems.

The first of these tools adopted at scale across Amazon was Amazon Q Developer. This tool automatically generates code completions based on existing code and comments. We investigated the tool’s effect on CR velocity by building a panel regression model with dynamic two-way fixed effects.

This model uses time-varying covariates based on observations of software builder teams over multiple time periods during a nine-month observation window, and it predicts either CR velocity or deployment velocity. We specify the percentage of the team using Q Developer in each week and pass that information to the model as well.

We also evaluate other variables passed to the model to make sure they are exogenous, i.e., not influenced by Q Developer usage, to ensure that we can make claims of a causal relationship between Q Developer usage and deployment or CR velocity. These variables include data on rollbacks and manual interventions in order to capture the impact of production and deployment incidents, which may affect the way builders are writing code.

Here’s our model specification:

yit = ai + λt + βyi,t-1 + γXit + εit

In this equation, 𝑦𝑖𝑡 is the normalized deployments per builder week or team weekly velocity for team 𝑖 at time 𝑡, 𝑎𝑖 is the team-specific fixed effect, 𝜆𝑡 is the time-specific fixed effect, 𝑦𝑖,𝑡―1 is the lagged normalized deployments or team velocity, 𝑋𝑖𝑡 is the vector of time-varying covariates (Q Developer usage rate, rollback rate, manual interventions), 𝛽𝑖𝑡 is the persistence of our dependent variable over time (i.e., it shows how much of the past value of 𝑦 carries over into the current period), and 𝜀𝑖𝑡 is the error term.

Conditioning set.gif
Related content
Determining causality in correlated time series
New method goes beyond Granger causality to identify only the true causes of a target time series, given some graph constraints.

Early evidence shows that Q Developer has accelerated CR velocity and deployment velocity. More important, we found causal evidence that the launch of a new developer tool can lower CTS-SW for adopting teams and that we can measure that impact. As agentic AI grows, there will be agents for a range of tasks that engineers perform, beyond just writing code. That will require a unit of measurement that can capture their contributions holistically, without overly focusing on one area. CTS-SW enables us to measure the effects of AI across the software development lifecycle, from agents giving feedback on design docs to agents suggesting fixes to failed builds and deployments.

The road ahead

We recognize that combining experimental results can sometimes overstate an intervention’s true impact. To address this, we're developing a baseline model that we can use to normalize our tool-based approach to ensure that our estimates of AI impact are as accurate as possible.

Looking ahead, we plan to expand our analysis to include AI's impact on more aspects of the developer experience. By leveraging CTS-SW and developing robust methodologies for measuring AI's impact, we're ensuring that our AI adoption is truly customer obsessed, in that it makes Amazon’s software development more efficient. As we continue to explore and implement AI solutions, we remain committed to using data-driven approaches to improve outcomes and experiences for our customers. We look forward to sharing them with you at a later date.

Research areas
Tags
About the Author
Willie deButts
Willie deButts is a principal data scientist in the Amazon Software Builder Experience (ASBX) organization.

Related content

Work with us

See more jobs See more jobs
Principal Applied Scientist, Frontier AI Assets , Frontier AI Assets
US, CA, Sunnyvale
The Frontier AI Assets (FAI) team is looking for a passionate, talented, and inventive Principal Applied Scientist with a strong deep learning and reinforcement learning background to build the training and evaluation assets that advance frontier Generative AI (GenAI) with Large Language Models (LLMs). AI models become more capable largely because of the learning environments, the RL gyms, in which they are trained. FAI builds those gyms and the platform that certifies them, so that model training teams across Amazon can rely on high-quality, graded assets. As a Principal on this team, you will set the technical direction for how these gyms and the tooling around them are built, evaluated, and scaled. Key job responsibilities As a Principal Applied Scientist with FAI, you will: •Set technical direction across data collection, sampling, and evaluation for RL gyms focused on agentic tool use across a range of real-world domains. •Design and lead experiments that produce reliable, efficient signal on what drives model improvement. •Build gym-building tooling and automated quality control that scale with demand. •Advance the science of how gyms are evaluated and validated before they're used in model training. •Act as a hands-on technical leader: produce exemplary, high-quality assets yourself, and audit and support more junior scientists and engineers. •Partner closely with domain experts and cross-functional teams, turning real-world problems into challenging, high-value tasks. •Lead scientific reviews, scrutinize experimental design and verification, probe assumptions, and align teams toward coherent strategies. Contribute to hiring, mentorship, and the development of other scientists. About the team The FAI team has the mission to push the envelope in GenAI with LLMs, in order to provide the best-possible experience for our customers. We build frontier training and evaluation assets, RL gyms and the certification platform behind them, that directly shape how Amazon's models are trained and improved. We value scientific depth, hands-on building, and turning research into measurable impact at scale.
Senior Research Scientist, RBS Tech
IN, KA, Bengaluru
RBS (Retail Business Services) Tech team works towards enhancing the customer experience (CX) and their trust in product data by providing technologies to find and fix Amazon CX defects at scale. Our platforms help in improving the CX in all phases of customer journey, including selection, discoverability & fulfilment, buying experience and post-buying experience (product quality and customer returns). The team also develops GenAI platforms for automation of Amazon Stores Operations. As a Sciences team in RBS Tech, we focus on foundational ML research and develop scalable state-of-the-art ML solutions to solve the problems covering customer experience (CX) and Selling partner experience (SPX). We work to solve problems related to multi-modal understanding (text and images), task automation through multi-modal LLM Agents, supervised and unsupervised techniques, multi-task learning, multi-label classification, aspect and topic extraction for Customer Anecdote Mining, image and text similarity and retrieval using NLP and Computer Vision for product groupings and identifying duplicate listings in product search results Key job responsibilities As a Senior Research Scientist, you will be responsible to design and deploy scalable GenAI, NLP and Computer Vision solutions that will impact the content visible to millions of customer and solve key customer experience issues. You will develop novel LLM, deep learning and statistical techniques for task automation, text processing, image processing, pattern recognition, and anomaly detection problems. You will define the research and experiments strategy with an iterative execution approach to develop AI/ML models and progressively improve the results over time. You will partner with business and engineering teams to identify and solve large and significantly complex problems that require scientific innovation. You will help the team leverage your expertise, by coaching and mentoring. You will contribute to the professional development of colleagues, improving their technical knowledge and the engineering practices. You will independently as well as guide team to file for patents and/or publish research work where opportunities arise. The RBS org deals with problems that are directly related to the selling partners and end customers and the ML team drives resolution to organization level problems. Therefore, the Senior Research Scientist role will impact the large product strategy, identifies new business opportunities and provides strategic direction which is very exciting.
Applied Scientist, Amazon Robotics
US, CA, Sunnyvale
Amazon Robotics is seeking an exceptional Applied Scientist to join our Foundation Models and Agentic AI team. This role presents an opportunity to shape the future of robotics through innovative applications of large vision-language-action (VLA) models, reasoning and task planning models, and reinforcement learning. At Amazon Robotics, we leverage advanced robotics, machine learning, and artificial intelligence to solve complex operational challenges at unprecedented scale. Our fleet of robots operates across hundreds of facilities worldwide, working in sophisticated coordination to fulfill our mission of customer excellence. We are pioneering the development of robotics foundation models that: - Enable unprecedented generalization across diverse tasks - Integrate multi-modal learning capabilities (visual, depth, force, tactile, linguistic) - Accelerate policy skill acquisition through imitation learning and reinforcement learning - Enhance robotic perception and environmental understanding - Streamline development processes through reusable capabilities The ideal candidate will contribute to research that bridges the gap between theoretical advancement and practical implementation in robotics. You will be part of a team that's revolutionizing how robots learn, adapt, and interact with their environment. Join us in building the next generation of intelligent robotics systems that will transform the future of automation and human-robot collaboration. Key job responsibilities As an Applied Scientist in the Applied AI and Robotics Foundations Model team, you will: - Model Development and Training: Designing and implementing the model architectures, training and fine tuning the foundation models using various datasets, and optimize the model performance through iterative experiments - Data Management: Process and prepare training data, including data governance, provenance tracking, data quality checks and creating reusable data pipelines. - Experimentation and Validation: Design and execute experiments to test model capabilities on the simulator and on the embodiment, validate performance across different scenarios, create a baseline and iteratively improve model performance. - Code Development: Write clean, maintainable, well commented and documented code, contribute to training infrastructure, create tools for model evaluation and testing, and implement necessary APIs - Research: Stay current with latest developments in foundation models and robotics, assist in literature reviews and research documentation, prepare technical reports and presentations, and contribute to research discussions and brainstorming sessions. - Collaboration: Work closely with senior scientists, engineers, and leaders across multiple teams, participate in knowledge sharing, support integration efforts with robotics hardware teams, and help document best practices and methodologies. A day in the life Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: - Medical, Dental, and Vision Coverage - Maternity and Parental Leave Options - Paid Time Off (PTO) - 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! About the team Are you inspired by invention? Is problem solving through teamwork in your DNA? Do you like the idea of seeing how your work impacts the bigger picture? Answer yes to any of these and you’ll fit right in here at Amazon Robotics. We are a smart, collaborative team of enthusiastic doers that work passionately to apply innovative advances in robotics and software to solve real-world challenges that will transform our customers’ experiences in ways we can’t even image yet. We invent new improvements every day. We are Amazon Robotics and we will give you the tools and support you need to invent with us in ways that are rewarding, fulfilling and fun.
Sr Applied Scientist, Amazon Shipping
IN, HR, Gurugram
Lead ML teams building large-scale forecasting and optimization systems that power Amazon’s global transportation network and directly impact customer experience and cost. As an Sr Applied Scientist, you will set scientific direction, mentor applied scientists, and partner with engineering and product leaders to deliver production-grade ML solutions at massive scale. Key job responsibilities 1. Lead and grow a high-performing team of Applied Scientists, providing technical guidance, mentorship, and career development. 2. Define and own the scientific vision and roadmap for ML solutions powering large-scale transportation planning and execution. 3. Guide model and system design across a range of techniques, including tree-based models, deep learning (LSTMs, transformers), LLMs, and reinforcement learning. 4. Ensure models are production-ready, scalable, and robust through close partnership with stakeholders. Partner with Product, Operations, and Engineering leaders to enable proactive decision-making and corrective actions. 5. Own end-to-end business metrics, directly influencing customer experience, cost optimization, and network reliability. 6. Help contribute to the broader ML community through publications, conference submissions, and internal knowledge sharing. A day in the life Your day includes reviewing model performance and business metrics, guiding technical design and experimentation, mentoring scientists, and driving roadmap execution. You’ll balance near-term delivery with long-term innovation while ensuring solutions are robust, interpretable, and scalable. Ultimately, your work helps improve delivery reliability, reduce costs, and enhance the customer experience at massive scale.
Applied Scientist, North America Sort Centers, Amazon Transportation Services, North America Sort Centers, Amazon Transportation Services
US, WA, Bellevue
Have you ever ordered a product on Amazon and when that box with the smile arrived you wondered how it got to you so fast? Have you wondered where it came from and how much it cost Amazon to deliver it to you? Have you also wondered what are different ways that the transportation assets can be used to delight the customer even more. If so, the Amazon transportation Services, North America Sort Center Science team is for you . We manage the delivery of tens of millions of products every week to Amazon’s customers, achieving on-time delivery in a cost-effective manner. We are looking for an enthusiastic, customer obsessed Applied Scientist with strong scientific thinking, good software and statistics experience, skills to help manage projects and operations, improve metrics, and develop scalable processes and tools. The primary role of an Applied Scientist within Amazon is to address business challenges through building a compelling case, and using data to influence change across the organization. This individual will be given responsibility on their first day to own those business challenges and the autonomy to think strategically and make data driven decisions. Decisions and tools made in this role will have significant impact to the customer experience, as it will have a major impact on how we operate the middle mile network. Ideal candidates will be a high potential, strategic and analytic graduate with a PhD in (Operations Research, Statistics, Engineering, and Supply Chain) ready for challenging opportunities in the core of our world class operations space. Great candidates have a history of operations research, machine learning , and the ability to use data and research to make changes. This role requires robust skills in research and implementation of scalable products and models . This individual will need to be able to work with a team, but also be comfortable making decisions independently, in what is often times an ambiguous environment. Key job responsibilities Key job responsibilities - Design and develop advanced mathematical optimization and machine learning solutions in the domains of network design and optimization. - Research, prototype, simulate, and experiment with these models using programming languages such as Java and Python; participate in the production level deployment. - Create, enhance, and maintain technical documentation and science designs. - Present to other Scientists, Product, and Software Engineering teams, as well as Stakeholders. - Lead project plans from a scientific perspective by managing product features, technical risks, milestones and launch plans.
Applied Scientist II, Prime Video - Ads Science
US, WA, Seattle
Prime Video is a first-stop entertainment destination offering customers a vast collection of premium programming in one app available across thousands of devices. Prime members can customize their viewing experience and find their favorite movies, series, documentaries, and live sports – including Amazon MGM Studios-produced series and movies; licensed fan favorites; and programming from Prime Video subscriptions such as Apple TV+, HBO Max, Peacock, Crunchyroll and MGM+. All customers, regardless of whether they have a Prime membership or not, can rent or buy titles via the Prime Video Store, and can enjoy even more content for free with ads. Are you interested in shaping the future of entertainment? Prime Video's technology teams are creating best-in-class digital video experience. As a Prime Video team member, you’ll have end-to-end ownership of the product, user experience, design, and technology required to deliver state-of-the-art experiences for our customers. You’ll get to work on projects that are fast-paced, challenging, and varied. You’ll also be able to experiment with new possibilities, take risks, and collaborate with remarkable people. We’ll look for you to bring your diverse perspectives, ideas, and skill-sets to make Prime Video even better for our customers. With global opportunities for talented technologists, you can decide where a career Prime Video Tech takes you! Key job responsibilities The science challenges include: - Heterogeneous customer responses. The same customer responds differently to ads depending on what they’re watching, how engaged they are, and their broader streaming context. We need to understand and predict this variation to make better advertising decisions in real time. - Signal sparsity in streaming. Unlike search or retail advertising, streaming offers no clicks, no conversions, and limited direct feedback. We must develop creative approaches to infer customer preferences, intent, and tolerance from indirect behavioral signals. - Personalization at scale. A one-size-fits-all ad experience leaves value on the table for both customers and the business. We build systems that adapt ad load, placement, and content to individual viewers across 100M+ customers and 100k+ titles. - Small effects, large variance. Ad interventions typically produce 0.1-2% shifts in engagement, effects easily overwhelmed by natural behavioral variance. Measuring, attributing, and optimizing these small signals requires rigorous experimental design and causal methodologies. - Competing objectives. Revenue, customer engagement, long-term retention, and advertiser value are in tension. We develop principled frameworks to navigate these tradeoffs and optimize for sustainable outcomes rather than any single metric. A day in the life - Contribute to the research and development of ML models that personalize advertising decisions for 100M+ customers across 100k+ titles, with production deployment in mind - Develop deep learning architectures (multi-task learning, embedding-based representations) for customer behavior prediction at scale - Design and analyze large-scale A/B experiments, applying causal inference techniques to measure and optimize the impact of ad strategies on customer engagement and monetization - Partner with engineering to ensure models meet production latency and scalability requirements - Collaborate with product managers to frame business problems as tractable ML problems and translate findings into product decisions - Shape the team’s scientific roadmap, identifying high-impact research directions About the team Prime Video Ads is pioneering the use of Generative AI, Deep Learning and Causal ML to empower the next generation of Ads experience in Prime Video. Our vision is to build the world’s most valuable and differentiated ad-supported streaming platform: one where every ad is relevant to the viewer, and every dollar spent by an advertiser delivers measurable results. We are seeking lead applied scientists who have demonstrated experience in advancing state of the art ML algorithms, particularly in Personalization, Relevance, Casual Inference and Gen-AI areas. Your role will be to deliver these innovations as production-ready systems at Amazon scale.
Research Scientist, Pricing Science
US, WA, Seattle
Pricing is one of the most consequential decisions Amazon makes, and the science behind it needs to be causally rigorous, not just predictive. The P2 Optimization Science team builds the machine learning (ML) systems that power Amazon's pricing decisions at scale: demand lift models, customer lifetime value (LTV) frameworks, and the experimentation infrastructure that validates whether pricing changes actually work. We're looking for a Research Scientist to own causal inference at the intersection of ML and pricing experimentation. Our team has identified a real gap: the methodological bridge between econometric analysis and production-scale ML pipelines needs a practitioner who lives in both worlds. You'll build conditional average treatment effect (CATE) estimation models, design analysis workflows for pricing experiments, and develop reusable causal ML infrastructure that the broader team — including non-ML scientists — can rely on. If you're a scientist who wants to tackle hard causal identification problems in a high-stakes production environment, and who finds satisfaction in making rigorous methods accessible to a broader team, this role is for you. Key job responsibilities - Design, train, evaluate, and deploy end-to-end causal ML pipelines for pricing use cases, including CATE estimation models and heterogeneous treatment effect frameworks. - Serve as the team's causal ML methodology expert — defining identification strategies, model selection criteria, evaluation standards, and the tradeoffs between econometric and ML approaches to causal estimation. - Contribute causal analysis methodology to pricing A/B test post-analysis and build reusable tooling that economists and other non-ML scientists can adopt without requiring deep ML expertise. - Assess emerging causal inference methods (e.g., synthetic difference-in-differences, generalized random forests, causal representation learning) for applicability to Amazon's pricing context, and author internal methodology proposals for adoption. - Partner with senior economists on identification strategy and causal assumptions, collaborate with software and data engineering partners on production deployment, and align with product managers on experiment design requirements. A day in the life Your work directly shapes the prices customers see on hundreds of millions of Amazon products. In a given week you might investigate an optimization anomaly in simulation and trace it to an unmodeled market dynamic, then design an offline evaluation framework to benchmark competing approaches before committing to online testing. You'll collaborate with senior economists on identification strategy for a pricing experiment, present a science proposal for incorporating a new competitiveness signal into an optimization system, or work with the experimentation platform team on randomization design. You'll also have opportunities to write up novel scientific findings for submission to venues such as KDD, NeurIPS, or the ACM Conference on Economics and Computation. About the team The P2 Optimization Science team sits at the center of Amazon's pricing decisions. We build the ML systems, causal inference pipelines, and experimentation infrastructure that determine how prices are set across Amazon's global catalog. Our team brings together research scientists, economists, and engineers who collaborate closely to turn rigorous methodology into production-quality systems with measurable business impact. We value scientific rigor, clear communication across disciplines, and a culture where every team member's work is designed to be extensible and useful beyond their own project. If you want your science to directly influence how hundreds of millions of customers experience Amazon, this is the team to join.
Data Scientist , Music DISCO
US, CA, San Francisco
Amazon Music is an immersive audio entertainment service that deepens connections between fans, artists, and creators. From personalized music playlists to exclusive podcasts, concert livestreams to artist merch, Amazon Music is innovating at some of the most exciting intersections of music and culture. We offer experiences that serve all listeners with our different tiers of service: Prime members get access to all the music in shuffle mode, and top ad-free podcasts, included with their membership; customers can upgrade to Amazon Music Unlimited for unlimited, on-demand access to 100 million songs, including millions in HD, Ultra HD, and spatial audio; and anyone can listen for free by downloading the Amazon Music app or via Alexa-enabled devices. Join us for the opportunity to influence how Amazon Music engages fans, artists, and creators on a global scale. The Data, Insights, Science and Optimization, Consumer Product and Tech (DISCO CPT) team is looking for a Data Scientist to join a team of Data Scientists, Analytics Leads, Business Intelligence Engineers and Data Engineers who analyze big data, provide analytics and insights as well as build models and algorithms that power product for personalized experiences. The team focuses on accelerating Amazon Music customer growth by empowering Product teams to make sound, customer-centric decisions through data and insights. We build data pipelines, science models, self-service analytics, insights and dashboards, deliver engagement and retention at scale. In this role, you will pioneer and execute advanced analytics and experimentation efforts for the Amazon Music product team. You will work with stakeholders and partners to deliver experimentation insights, author scientific artifacts, build scalable models to advance our analytics and science products. The ideal candidate must be willing to effectively lead science solutions, prioritize across multiple stakeholders and projects and be ready to jump into a fast-paced, dynamic and fun environment. Key job responsibilities 1. Data Scientist I on this role will be a thought partner to Product and Engineering organizations through their expertise in experimentation, statistical modeling and advanced analytics 2. Conceptualize and drive strategic growth areas: app performance, stablity and latency experimentation, Gen AI initiatives, and product excellence. 3. Bridge science, analytics, and engineering to deliver solutions informing strategic decisions on AM technology investments and long-term business performance 4. Drive experimentation, measurement efficiency tooling, and ROI tracking for AM leadership 5. Build and maintain causal models to understand customer sentiment and its impact on key engagement/retention metrics 6. Develop structural and predictive models using data science workflows to deliver measurable customer results A day in the life 1. Lead strategic initiatives across visual and voice experiences, experimentation and customer retention stakeholders 2. Conduct experimentation, build causal models to better understand key drivers for customer retention 3. Evaluate existing metrics and lead deep dives to ensure the right measurement framework is leveraged to measure and enhance user adoption through visual and voice interfaces 4. Partner with product managers to translate business questions into actionable science projects 5. Influence product roadmaps through data-driven insights on customer behavior patterns 6. Identify and resolve customer experience friction points with cross-functional teams Drive adoption of best practices and innovative solutions across the organization
Principal Applied Scientist, Ads Measurement Science
US, NY, New York
The Measurement Science team in the Measurement, Ad Tech, and Data Science (MADS) organization of Amazon Ads serves a centralized role developing solutions for a multitude of performance measurement products. We create solutions which measure the comprehensive impact of their ad spend, including sales impacts both online and offline and across timescales, and provide actionable insights that enable our advertisers to optimize their media portfolios. We leverage a host of scientific technologies to accomplish this mission, including Generative AI, classical ML, Causal Inference, Natural Language Processing, and Computer Vision. This Principal Applied Scientist leads the technical vision and execution for Amazon Advertising's performance measurement. Our measurement products and tools help customers understand the holistic impact of their spend on Amazon Ads and optimize their portfolios. This is a complex, multi-disciplinary scientific space spanning machine learning, causal inference, statistics and experimentation, natural language processing, and generative AI. As a GenAI-first organization, we build foundational and agentic models that power advertiser use cases across Ads, while empowering our Applied Scientists to directly build and ship products. You will guide a team of scientists with expertise across different domains to come together and innovate on some of the hardest problems in advertising, made more challenging by evolving privacy regulations and technological changes. In measuring the comprehensive impact of advertising, key challenges you will tackle include capturing impact: 1) both for identified and increasingly unidentified web traffic in light of an evolving global privacy landscape; 2) for outcomes across retailers, including both within and beyond Amazon; and 3) for both short- and long-term impacts of advertising. Key job responsibilities * Driving the scientific vision of the teams in your organization and you advise and influence its technical leadership on performance measurement models and products. * Identifying, tackling, and proposing innovative solutions to intrinsically hard, previously unsolved problems. You solve problems pragmatically, applying judgment and experience to balance trade-offs between competing interests. * Bringing clarity to complex problems, probing assumptions, illuminating pitfalls, fostering shared understanding, and guiding towards effective solutions. * Serving and being recognized by internal and external peers as a thought leader in modeled measurement in Ads. You will engage routinely with the broader scientific community, e.g. as a reviewer, program committee member, workshop organizer, or chair at peer-reviewed internal/external conferences. * Leading by example, with your papers, analyses, and code setting the standard in your organization for scientific engineering excellence. * Leading science and design reviews, aligning teams across your organization around coherent algorithmic strategies. * Influencing your team’s science and business strategy by driving one or more team roadmaps contributing to the organization’s roadmap and taking responsibility for some organizational goals. You drive multiple new product features from inception to production launch. * Guiding the career development of others, actively mentoring and educating the larger applied science community on trends, technologies, and best practices. You drive the documentation of scientific innovation. You attract bar-raising talent from academia and industry. * Staying abreast of academic and industry trends, knowing the state of the art, and building on top of existing solutions instead of reinventing when possible. About the team We are a team of scientists across Applied, Research, Data Science and Economist disciplines. You will work with colleagues with deep expertise in ML, NLP, CV, Gen AI, and Causal Inference with a diverse range of backgrounds. We partner closely with top-notch engineers, product managers, sales leaders, and other scientists with expertise in the ads industry and on building scalable modeling and software solutions.
Robotics Applied Scientist
US, CA, Sunnyvale
Amazon is seeking exceptional talent to help develop the next generation of advanced robotics systems that will transform automation at Amazon's scale. We're building robotic systems that combine frontier AI, sophisticated control, and advanced mechanical design to create adaptable automation capable of working safely alongside humans in dynamic environments. This is an opportunity to shape the future of robotics and automation at an unprecedented scale, working with world-class teams pushing the boundaries of what's possible in robotic manipulation, locomotion, and human-robot interaction. We are seeking an Applied Scientist to develop tactile- and force-driven manipulation policies for highly dexterous, multi-fingered robotic hands and grippers. In this role you will research and build policy learning algorithms that enable robotic hands to perform grasping and in-hand manipulation in unstructured, real-world environments, using touch and force feedback as first-class signals rather than afterthoughts. Much of this work is grounded in reinforcement learning. To be successful you need to be highly motivated, dive deep, and deliver to the highest standards. You will demonstrate strong working knowledge of modern policy learning methods, with expertise in taking algorithms from simulation onto real hardware. That means being comfortable working against real physical constraints and treating sim-to-real as a core research problem rather than optimizing for benchmark numbers alone. You will work across tactile signal processing, contact-rich manipulation, sim-to-real transfer, and multi-modal sensor fusion to build robust solutions for autonomous grasping, dexterous re-orientation, and fine motor control. You will bring the desire to learn from new challenges, and the problem-solving and communication skills to work within a highly interactive and experienced team. Key job responsibilities - Design and train manipulation policies that leverage tactile and force feedback for dexterous grasping, in-hand manipulation, and contact-rich tasks, using reinforcement learning, imitation learning, or hybrid approaches. - Drive sim-to-real transfer, closing the gap between simulated and physical performance on real robotic hands. - Shape objectives and control strategies with the hardware in mind, accounting for how motors and actuators actually behave: transmission ratios, torque versus power, backdrivability, and thermal limits. - Build simulation-based and on-robot evaluation frameworks, with benchmarks and metrics that make tactile perception and policy performance systematically comparable across iterations. - Own scientific and technical projects within the tactile sensing and manipulation workstreams, driving from research concept through deployment on physical robotic systems — and, at the senior level, set direction across a workstream. - Collaborate with hardware, mechanical design, firmware, and controls teams — and with partner applied science organizations — to translate research advances into deployable robotic capabilities, and to inform sensor and actuator design decisions with policy-level requirements. - Publish at top-tier venues and build collaborations with the external research community. - Contribute to a strong scientific bar on the team through code and design review, and mentor engineers and interns working on real-world manipulation and sensing problems. A day in the life Your morning might start with a standup alongside hardware and controls engineers reviewing overnight sim-to-real training runs on a multi-finger gripper, debugging why a grasp policy that worked in simulation is slipping on a real sensor array. Mid-morning you join the weekly tactile sensing workstream to align on sensor integration milestones and share early results from a contact-state estimator you have been prototyping. After lunch you spend a focused block iterating on a reinforcement learning reward formulation, testing variations in simulation before queuing runs on the cluster. Later you pair with a mechanical engineer to review sensor placement trade-offs on an upcoming gripper revision, then wrap the day by drafting a short experiment write-up and sketching next steps for a conference submission. No two days look exactly the same: one week you may be collecting teleoperated demonstrations on the physical robot to seed an imitation learning pipeline; the next you could be deep in a codebase refactor to support a new tactile modality. Throughout, you balance hands-on research — writing algorithms, running experiments, analyzing data — with cross-team collaboration and mentoring. The common thread is moving from research insight to working capability on real hardware, with a tight feedback loop between simulation and the physical world. About the team Our is developing next-generation manipulation capabilities for highly dexterous multi-fingered grippers and hands. Our work spans tactile sensor development, manipulation policy learning, and the tight integration of sensing hardware with intelligent control software. We are building systems that can feel, adapt, and manipulate with human-level dexterity. You will join a team working at the frontier of tactile sensing and contact-rich manipulation, helping define how robots perceive and interact with objects through touch. Because we build the hands as well as the policies that drive them, you will have a direct line to the sensor, actuator, and mechanism designs your algorithms depend on — and real influence over how they evolve. This is an opportunity to solve hard problems across perception, learning, and physical interaction alongside a team of scientists and engineers building toward real-world deployment.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social