Responsible AI in the wild: Lessons learned at AWS

Real-world deployment requires notions of fairness that are task relevant and responsive to the available data, recognition of unforeseen variation in the “last mile” of AI delivery, and collaboration with AI activists.

When we first joined AWS AI/ML as Amazon Scholars over three years ago, we had already been doing scientific research in the area now known as responsible AI for a while. We had authored a number of papers proposing mathematical definitions of fairness and machine learning (ML) training algorithms enforcing them, as well as methods for ensuring strong notions of privacy in trained models. We were well versed in adjacent subjects like explainability and robustness and were generally denizens of the emerging responsible-AI research community. We even wrote a general-audience book on these topics to try to explain their importance to a broader audience.

Related content
Generative AI raises new challenges in defining, measuring, and mitigating concerns about fairness, toxicity, and intellectual property, among other things. But work has started on the solutions.

So we were excited to come to AWS in 2020 to apply our expertise and methodologies to the ongoing responsible-AI efforts here — or at least, that was our mindset on arrival. But our journey has taken us somewhere quite different, somewhere more consequential and interesting than we expected. It’s not that the definitions and algorithms we knew from the research world aren’t relevant — they are — but rather that they are only one component of a complex AI workstream comprising data, models, services, enterprise customers, and end-users. It’s also a workstream in which AWS is uniquely situated due to its pioneering role in cloud computing generally and cloud AI services specifically.

Our time here has revealed to us some practical challenges of which we were previously unaware. These include diverse data modalities, “last mile” effects with customers and end-users, and the recent emergence of AI activism. Like many good interactions between industry and academia, what we’ve learned at AWS has altered our research agenda in healthy ways. In case it’s useful to anyone else trying to parse the burgeoning responsible-AI landscape (especially in the generative-AI era), we thought we’d detail some of our experiences here.

Modality matters

One of our first important practical lessons might be paraphrased as “modality matters”. By this we mean that the particular medium in which an AI service operates (such as visual images or spoken or written language) matters greatly in how we analyze and understand it from both performance and responsible-AI perspectives.

Consider specifically the desire for trained models be “fair”, or free of significant demographic bias. Much of the scientific literature on ML fairness assumes that the features used to compare performance across groups (which might include gender, race, age, and other attributes) are readily available, or can be accurately estimated, in both training and test datasets.

Related content
Two of the world’s leading experts on algorithmic bias look back at the events of the past year and reflect on what we’ve learned, what we’re still grappling with, and how far we have to go.

If this is indeed the case (as it might be for some spreadsheet-like “tabular” datasets recording things like medical or financial records, in which a person’s age and gender might be explicit columns), we can more easily test a trained model for bias. For instance, in a medical diagnosis application we might evaluate the model to make sure the error rates are approximately the same across genders. If these rates aren’t close enough, we can augment our data or retrain the model in various ways until the evaluation is passed to satisfaction.

But many cloud AI/ML services operate on data that simply does not contain explicit demographic information. Rather, these services live in entirely different modalities such as speech, natural language, and vision. Applications such as our speech recognition and transcription services take as input time series of frequencies that capture spoken utterances. Consequently, there are not direct annotations in the data of things like gender, race, or age.

But what can be more readily detected from speech data, and are also more directly related to performance, are regional dialects and accents — of which there are dozens in North American English alone. English-language speech can also feature non-native accents, influenced more by the first languages of the speakers than by the regions in which they currently live. This presents an even more diverse landscape, given the large number of first languages and the international mobility of speakers. And while spoken accents may be weakly correlated or associated with one or more ancestry groups, they are usually uninformative on things like age and gender (speakers with a Philadelphia accent may be young or old; male, female or nonbinary; etc.). Finally, the speech of even a particular person may exhibit many other sources of variation, such as situational stress and fatigue.

Regional dialects.jpeg
Data — such as regional variations in word choice and accents — may lead toward alternative notions of fairness that are more task-relevant, as with word error rates across dialects and accents.

What is the responsible-AI practitioner to do when confronted with so many different accents and other moving parts, in a task as complex as speech transcription? At AWS, our answer is to meet the task and data on their own terms, which in this case involves some heavy lifting: meticulously gathering samples from large populations of representative speakers with different accents and carefully transcribing each word. The “representative” is important here: while it might be more expedient to (for instance) gather this data from professional actors trained in diction, such data would not be typical of spoken language in the wild.

Related content
Both secure multiparty computation and differential privacy protect the privacy of data used in computation, but each has advantages in different contexts.

We also gather speech data that exhibits variability along other important dimensions, including the acoustic conditions during recording (varying amounts and types of background noise, recordings made via different mobile-phone handsets, whose microphones may vary in quality, etc.). The sheer number of combinations makes obtaining sufficient coverage challenging. (In some domains such as computer vision, coverage issues that are similar — variability across visual properties such as skin tone, lighting conditions, indoor vs. outdoor settings, and so on — have led to increased interest in synthetic data to augment human-generated data, including for fairness testing here at AWS.)

Once curated, such datasets can be used for training a transcription model that is not only good overall but also roughly equally performant across accents. And “performant” here means something more complex than in a simple prediction task; speech recognition typically uses a measure like the word error rate. On top of all the curation and annotations above, we also annotate some data by self-reported speaker demographics to make sure we’re fair not just by accent but by race and gender as well, as detailed in the service’s accompanying service card.

Our overarching point here is twofold. First, while as a society we tend to focus on dimensions such as race and gender when speaking about and assessing fairness, sometimes the data simply doesn’t permit such assessments, and it may not be a good idea to impute such dimensions to the data (for instance, by trying to infer race from speech signals). And second, in such cases the data may lead us toward alternative notions of fairness that might be more task-relevant, as with word error rates across dialects and accents.

The last mile of responsible AI

The specific properties of individuals that can or cannot (or should not) be gleaned from a particular dataset or modality are not the only things that may be out of the direct control of AI developers — especially in the era of cloud computing. As we have seen above, it’s challenging work to get coverage of everything you can anticipate. It’s even harder to anticipate everything.

The supply chain phrase “the last mile” refers to the fact that “upstream” providers of goods and products may have limited control over the “downstream” suppliers that directly connect to end-users or consumers. The emergence of cloud providers like AWS has created an AI service supply chain with its own last-mile challenges.

Related content
The team’s latest research on privacy-preserving machine learning, federated learning, and bias mitigation.

AWS AI/ML provides enterprise customers with API access to services like speech transcription because many want to integrate such services into their own workflows but don’t have the resources, expertise, or interest to build them from scratch. These enterprise customers sit between the general-purpose services of a cloud provider like AWS and the final end-users of the technology. For example, a health care system might want to provide cloud speech transcription services optimized for medical vocabulary to allow doctors to take verbal notes during their patient rounds.

As diligent as we are at AWS at battle-testing our services and underlying models for state-of-the-art performance, fairness, and other responsible-AI dimensions, it is obviously impossible to anticipate all possible downstream use cases and conditions. Continuing our health care example, perhaps there is a floor of a particular hospital that has new and specialized imaging equipment that emits background noise at a specific regularity and acoustic frequency. In the likely event that these exact conditions were not represented in either the training or test data, it’s possible that overall word error rates will not only be higher but may be so differentially across accents and dialects.

Such last-mile effects can be as diverse as the enterprise customers themselves. With time and awareness of such conditions, we can use targeted training data and customer-side testing to improve downstream performance. But due to the proliferation of new use cases, it is an ever-evolving process, not one that is ever “finished”.

AI activism: from bugs to bias

It’s not only cloud customers whose last miles may present conditions that differ from those during training and testing. We live in a (healthy) era of what might be called AI activism, in which not only enterprises but individual citizens — including scientists, journalists, and members of nonprofit organizations — can obtain API or open-source access to ML services and models and perform their own evaluations on their own curated datasets. Such tests are often done to highlight weaknesses of the technology, including shortfalls in overall performance and fairness but also potential security and privacy vulnerabilities. As such, they are typically performed without the AI developer’s knowledge and may be first publicized in both research and mainstream media outlets. Indeed, we have been on the receiving end of such critical publicity in the past.

Related content
Technique that mixes public and private training data can meet differential-privacy criteria while cutting error increase by 60%-70%.

To date, the dynamic between AI developers and activists has been somewhat adversarial: activists design and conduct a private experimental evaluation of a deployed AI model and report their findings in open forums, and developers are left to evaluate the claims and make any needed improvements to their technology. It is a dynamic that is somewhat reminiscent of the historical tensions between more traditional software and security developers and the ethical and unethical hacker communities, in which external parties probe software, operating systems, and other platforms for vulnerabilities and either expose them for the public good or exploit them privately for profit.

Over time the software community has developed mechanisms to alter these dynamics to be more productive than adversarial, in particular in the form of bug bounty programs. These are formal events or competitions in which software developers invite the hacker community to deliberately find vulnerabilities in their technology and offer financial or other rewards for reporting and describing them to the developers.

Bias bounties.png
In a fair-ML (“bias bounty”) competition, different teams (x-axis) focus on different demographic features (y-axis) in the dataset, indicating that crowdsourced bias mitigation can help contend with the breadth of possible sources of bias. (The darker the blue, the greater the use of the feature.)

In the last couple of years, the ideas and motivations behind bug bounties have been adopted and adapted by the AI development community, in the form of “bias bounties”. Rather than finding bugs in traditional software, participants are invited to help identify demographic or other biases in trained ML models and systems. Early versions of this idea were informal hackathons of short duration focused on finding subsets of a dataset on which a model underperformed. But more recent proposals incubated at AWS and elsewhere include variants that are more formal and algorithmic in nature. The explosion of models, interest in, and concerns about generative AI have also led to more codified and institutionalized responsible-AI methodologies such as the HELM framework for evaluating large language models.

We view these recent developments — AI developers opening up their technology and its evaluation to a wider community of stakeholders than just enterprise customers, and those stakeholders playing an active role in identifying necessary improvements in both technical and nontechnical ways — as healthy and organic, a natural outcome of the complex and evolving AI industry. Indeed, such collaborations are in keeping with our recent White House commitments to external testing and model red-teaming.

Responsible AI is neither a problem to be “solved” once and for all, nor a problem that can be isolated to a single location in the pipeline stretching from developers to their customers to end-users and society at large. Developers are certainly the first line where best practices must be established and implemented and responsible-AI principles defended. But the keys to the long-term success of the AI industry lie in community, communication, and cooperation among all those affected by it.

Related content

US, CA, San Diego
Do you want to join an innovative team of scientists who use deep learning, natural language processing, large language models to help Amazon provide the best seller experience across the entire Seller life cycle, including recruitment, growth, support and provide the best customer and seller experience by automatically mitigating risk? Do you want to build advanced algorithmic systems that help manage the trust and safety of millions of customer interactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data and creating state-of-the-art algorithms to solve real world problems? Are you excited by the opportunity to leverage GenAI and innovate on top of the state-of-the-art large language models to improve customer and seller experience? Do you like to build end-to-end business solutions and directly impact the profitability of the company? Do you like to innovate and simplify processes? If yes, then you may be a great fit to join the Machine Learning Accelerator team in the Amazon Selling Partner Services (SPS) group. Key job responsibilities The scope of an Applied Scientist III in the Selling Partner Services (SPS) Machine Learning Accelerator (MLA) team is to research and prototype Machine Learning applications that solve strategic business problems across SPS domains. Additionally, the scientist collaborates with engineers and business partners to design and implement solutions at scale when they are determined to be of broad benefit to SPS organizations. They develop large-scale solutions for high impact projects, introduce tools and other techniques that can be used to solve problems from various perspectives, and show depth and competence in more than one area. They influence the team’s technical strategy by making insightful contributions to the team’s priorities, approach and planning. They develop and introduce tools and practices that streamline the work of the team, and they mentor junior team members and participate in hiring. We are open to hiring candidates to work out of one of the following locations: San Diego, CA, USA
US, WA, Seattle
Amazon is looking for a strategic, innovative science leader within the Global Talent and Compensation (GTMC) organization to lead an interdisciplinary team charged with developing data-driven solutions to model, automate, and inform high judgement decision making by bringing together science and technology in consumer grade internal talent products. GTMC delivers employee-focused experiences by providing scalable and responsive mechanisms for employees, as well as listening and signaling mechanisms for managers and leaders. They do this through intelligent, flexible, and extensible products and scalable data and science services. They set out to deliver a singular experience supporting multiple employee talent journeys (e.g., onboarding, evaluation, compensation, movement, promotion, exit), to generate and capture signals from product data, surface outliers, increase personalization, and improve the efficacy of “next best action” recommendations, for 1.6 million Amazonians around the world. In this role you will lead multiple research teams across the disciplines of Talent Management, Diversity Equity and Inclusion, and Compensation. You will interface with the most senior leaders at Amazon to develop and deliver on a strategic research roadmap that crosses all lines of Amazon businesses (e.g., Consumer, AWS, Devices, Advertising). This role will then partner with engineering and product management leader to deliver the outcomes of this research in production environments. Successful candidates will have an established background expertise in machine learning with some experience in applying this expertise to the fields of talent management, product management and/or software development. We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
IN, KA, Bangalore
Are you interested in changing the Digital Reading Experience? We are from Kindle Books Team looking for a set of Scientists to take the reading experience in Kindle to next level with a set of innovations! We envision Kindle as the place where readers find the best manifestation of all written content optimized with features that enable them to get the most out of reading, and creators are able to realize their vision to customers quickly and at scale. Every time customers open their content, regardless of surface, they start or restart their reading in a familiar, useful and engaging place. We achieve this by building a strong foundation of core experiences and act as a force multiplier and partner for content creators (directly or indirectly) to easily innovate on top of Kindle's purpose built content experience stack in a simple and extensible way. We will achieve this by providing a best-in-class reading experience, unique content experiences, and remaining agile in meeting the evolving needs and preferences of our users. Our goal is to foster long-lasting reading habits and make us the preferred destination for enriching literary experiences. We are building a In The Book Science team and looking for Scientists, who are passionate about Reading and are willing to take Reading to the next level. Every Book is a complex structure with different entities, layout, format and semantics, with more than 17MM eBooks in our catalog. We are looking for experts in all domains like core NLP, Generative AI, CV and Deep Learning Techniques for unlocking capabilities like analysis, enhancement, curation, moderation, translation, transformation and generation in Books based on Content structure, features, Intent & Synthesis. Scientists will focus on Inside the book content and semantically learn the different entities to enhance the Reading experience overall (Kindle & beyond). They have an opportunity to influence in 2 major phases of life-cycle - Publishing (Creation of Books process) and Reading experience (building engaging features & representation in the book thereby driving reading engagement). Key job responsibilities - 5+ years of building machine learning models for business application experience - PhD, or Master's degree and 6+ years of applied research experience - Knowledge of programming languages such as C/C++, Python, Java or Perl - Experience programming in Java, C++, Python or related language - You have expertise in one of the applied science disciplines, such as machine learning, natural language processing, computer vision, Deep learning - You are able to use reasonable assumptions, data, and customer requirements to solve problems. - You initiate the design, development, execution, and implementation of smaller components with input and guidance from team members. - You work with SDEs to deliver solutions into production to benefit customers or an area of the business. - You assume responsibility for the code in your components. You write secure, stable, testable, maintainable code with minimal defects. - You understand basic data structures, algorithms, model evaluation techniques, performance, and optimality tradeoffs. - You follow engineering and scientific method best practices. You get your designs, models, and code reviewed. You test your code and models thoroughly - You participate in team design, scoping and prioritization discussions. You are able to map a business goal to a scientific problem and map business metrics to technical metrics. - You invent, refine and develop your solutions to ensure they are meeting customer needs and team goals. You keep current with research trends in your area of expertise and scrutinize your results. - Experience in mentoring junior scientists A day in the life You will be working with a group of talented scientists on researching algorithm and running experiments to test solutions to improve our experience. This will involve collaboration with partner teams including engineering, PMs, data annotators, and other scientists to discuss data quality, model development and productionizing the same. You will mentor other scientists, review and guide their work, help develop roadmaps for the team. We are open to hiring candidates to work out of one of the following locations: Banagalore, KA, IND | Bangalore, IND | Bangalore, KA, IND
IN, KA, Bangalore
Are you interested in changing the Digital Reading Experience? We are from Kindle Books Team looking for a set of Scientists to take the reading experience in Kindle to next level with a set of innovations! We envision Kindle as the place where readers find the best manifestation of all written content optimized with features that enable them to get the most out of reading, and creators are able to realize their vision to customers quickly and at scale. Every time customers open their content, regardless of surface, they start or restart their reading in a familiar, useful and engaging place. We achieve this by building a strong foundation of core experiences and act as a force multiplier and partner for content creators (directly or indirectly) to easily innovate on top of Kindle's purpose built content experience stack in a simple and extensible way. We will achieve this by providing a best-in-class reading experience, unique content experiences, and remaining agile in meeting the evolving needs and preferences of our users. Our goal is to foster long-lasting reading habits and make us the preferred destination for enriching literary experiences. We are building a In The Book Science team and looking for Scientists, who are passionate about Reading and are willing to take Reading to the next level. Every Book is a complex structure with different entities, layout, format and semantics, with more than 17MM eBooks in our catalog. We are looking for experts in all domains like core NLP, Generative AI, CV and Deep Learning Techniques for unlocking capabilities like analysis, enhancement, curation, moderation, translation, transformation and generation in Books based on Content structure, features, Intent & Synthesis. Scientists will focus on Inside the book content and semantically learn the different entities to enhance the Reading experience overall (Kindle & beyond). They have an opportunity to influence in 2 major phases of life-cycle - Publishing (Creation of Books process) and Reading experience (building engaging features & representation in the book thereby driving reading engagement). Key job responsibilities - 3+ years of building machine learning models for business application experience - PhD, or Master's degree and 2+ years of applied research experience - Knowledge of programming languages such as C/C++, Python, Java or Perl - Experience programming in Java, C++, Python or related language - You have expertise in one of the applied science disciplines, such as machine learning, natural language processing, computer vision, Deep learning - You are able to use reasonable assumptions, data, and customer requirements to solve problems. - You initiate the design, development, execution, and implementation of smaller components with input and guidance from team members. - You work with SDEs to deliver solutions into production to benefit customers or an area of the business. - You assume responsibility for the code in your components. You write secure, stable, testable, maintainable code with minimal defects. - You understand basic data structures, algorithms, model evaluation techniques, performance, and optimality tradeoffs. - You follow engineering and scientific method best practices. You get your designs, models, and code reviewed. You test your code and models thoroughly - You participate in team design, scoping and prioritization discussions. You are able to map a business goal to a scientific problem and map business metrics to technical metrics. - You invent, refine and develop your solutions to ensure they are meeting customer needs and team goals. You keep current with research trends in your area of expertise and scrutinize your results. A day in the life You will be working with a group of talented scientists on researching algorithm and running experiments to test solutions to improve our experience. This will involve collaboration with partner teams including engineering, PMs, data annotators, and other scientists to discuss data quality, model development and productionizing the same. You will mentor other scientists, review and guide their work, help develop roadmaps for the team. We are open to hiring candidates to work out of one of the following locations: Bangalore, IND | Bangalore, KA, IND
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques We are open to hiring candidates to work out of one of the following locations: Bengaluru, KA, IND
IN, KA, Bengaluru
How to use the world’s richest collection of e-commerce data to improve payments experience for our customers? Amazon Payments Global Data Science team seeks a Senior Data Scientist for building analytical and scientific solutions that will address increasingly complex business questions in the Gift-Cards space. Amazon.com has a culture of data-driven decision-making and demands intelligence that is timely, accurate, and actionable. This team operates at WW level and provides a fast-paced environment where every day brings new challenges and opportunities. As a Senior Data Scientist in this team, you will be driving the Data Science/ML roadmap for business continuity & growth. You will develop statistical and machine learning models to solve for complex business problems in Gift-Cards space, design and run global experiments, and find new ways to optimize the customer experience. You will need to collaborate effectively with internal stakeholders, cross-functional teams to solve problems, create operational efficiencies, and deliver successfully against high organizational standards. You will explore GenAI use-cases within Gift-Cards space and also work on cross-disciplinary efforts with other scientists within Amazon. Key job responsibilities - You should be detail-oriented and must have an aptitude for solving unstructured and ambiguous problems. You should work in a self-directed environment, own tasks and drive them to completion - You should be passionate about working with huge data sets and be someone who loves to bring datasets together to answer business questions - You should demonstrate thorough technical expertise on feature engineering of massive datasets, exploratory data analysis, and model building using state-of-art ML algorithms - Random Forest, Gradient Boosting, SVM, Neural Nets, DL, Reinforcement Learning etc. You should be aware of automating feedback loops for algorithms in production - You should work closely with internal stakeholders like the business teams, engineering teams and partner teams and align them with respect to your focus areas - You should have excellent business and communication skills to be able to work with business owners to develop and define key business questions and build mechanisms that answer those questions We are open to hiring candidates to work out of one of the following locations: Bengaluru, KA, IND
US, NY, New York
The Automated Reasoning Group in AWS Platform is looking for an Applied Scientist with experience in building scalable solver solutions that delight customers. You will be part of a world-class team building the next generation of automated reasoning tools and services. AWS has the most services and more features within those services, than any other cloud provider–from infrastructure technologies like compute, storage, and databases–to emerging technologies, such as machine learning and artificial intelligence, data lakes and analytics, and Internet of Things. You will apply your knowledge to propose solutions, create software prototypes, and move prototypes into production systems using modern software development tools and methodologies. In addition, you will support and scale your solutions to meet the ever-growing demand of customer use. You will use your strong verbal and written communication skills, are self-driven and own the delivery of high quality results in a fast-paced environment. Each day, hundreds of thousands of developers make billions of transactions worldwide on AWS. They harness the power of the cloud to enable innovative applications, websites, and businesses. Using automated reasoning technology and mathematical proofs, AWS allows customers to answer questions about security, availability, durability, and functional correctness. We call this provable security, absolute assurance in security of the cloud and in the cloud. See https://aws.amazon.com/security/provable-security/ As an Applied Scientist in AWS Platform, you will play a pivotal role in shaping the definition, vision, design, roadmap and development of product features from beginning to end. You will: - Define and implement new solver applications that are scalable and efficient approaches to difficult problems - Apply software engineering best practices to ensure a high standard of quality for all team deliverables - Work in an agile, startup-like development environment, where you are always working on the most important stuff - Deliver high-quality scientific artifacts - Work with the team to define new interfaces that lower the barrier of adoption for automated reasoning solvers - Work with the team to help drive business decisions The AWS Platform is the glue that holds the AWS ecosystem together. From identity features such as access management and sign on, cryptography, console, builder & developer tools, to projects like automating all of our contractual billing systems, AWS Platform is always innovating with the customer in mind. The AWS Platform team sustains over 750 million transactions per second. Learn and Be Curious. We have a formal mentor search application that lets you find a mentor that works best for you based on location, job family, job level etc. Your manager can also help you find a mentor or two, because two is better than one. In addition to formal mentors, we work and train together so that we are always learning from one another, and we celebrate and support the career progression of our team members. Inclusion and Diversity. Our team is diverse! We drive towards an inclusive culture and work environment. We are intentional about attracting, developing, and retaining amazing talent from diverse backgrounds. Team members are active in Amazon’s 10+ affinity groups, sometimes known as employee resource groups, which bring employees together across businesses and locations around the world. These range from groups such as the Black Employee Network, Latinos at Amazon, Indigenous at Amazon, Families at Amazon, Amazon Women and Engineering, LGBTQ+, Warriors at Amazon (Military), Amazon People With Disabilities, and more. Key job responsibilities Work closely with internal and external users on defining and extending application domains. Tune solver performance for application-specific demands. Identify new opportunities for solver deployment. About the team Solver science is a talented team of scientists from around the world. Expertise areas include solver theory, performance, implementation, and applications. Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Hybrid Work We value innovation and recognize this sometimes requires uninterrupted time to focus on a build. We also value in-person collaboration and time spent face-to-face. Our team affords employees options to work in the office every day or in a flexible, hybrid work model near one of our U.S. Amazon offices. We are open to hiring candidates to work out of one of the following locations: New York, NY, USA
US, WA, Bellevue
Amazon’s Automated Inventory Management (AIM) Planning Organization is looking for a Data Scientist to help invent the next generation of Amazon's Capacity and Constraint Management system - Automated Planning System (APS). APS will herald a a new era in Sales and Operations Planning (S&OP). APS emerges as a next-generation decision-making framework for Amazon's Worldwide (WW) fulfillment networks. In an industry first, APS seamlessly aligns Amazon's business controls by uniting cutting-edge supply and demand forecasts with a state-of-the-art coordination framework – respecting the distributed ownership of business logic and outcomes. As the centralized planning system, APS takes charge of coordinating all fulfillment, inventory, and operational decisions, maximizing WW Long Term Free Cash Flow (LTFCF) over a 1-year horizon The AIM team is part of the Supply Chain Optimization Technology (SCOT) Team within the Operations Organization. The charter of the SCOT team is to maximize Amazon’s return on our inventory investment in terms of Free Cash Flow and customer satisfaction. The planning organization within Amazon leads the S&OP, IPE and Capacity Planning functions. As a Data Scientist on the this team, you will build a deep understanding of Amazon's supply chain systems, lead innovation in our forecasting capabilities and build principled solutions to identify improvement opportunities in our supply chain using the latest machine learning techniques. You will also work with a team of Product Managers, Business Intelligence Engineers and Software Engineers to research and build accurate predictive models and deploy automated software solutions to provide insights to business leaders at the most senior levels throughout the company. You will build models that make our data more actionable and help us make complex business decisions at scale. To help describe some of our challenges, we created a short video about Supply Chain Optimization at Amazon - http://bit.ly/amazon-scot Key job responsibilities - Implement statistical and machine learning methods to solve complex business problems - Research new ways to improve predictive and explanatory models - Directly contribute to the design and development of automated prediction systems and ML infrastructure - Build models that can detect supply chain defects and explain variance to the optimal state - Collaborate with other researchers, software developers, and business leaders to define the scientific roadmap for this team We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Seattle
Do you want to join an innovative team of scientists who use machine learning to help Amazon provide the best experience to our Selling Partners by automatically understanding and addressing their challenges, needs and opportunities? Do you want to build advanced algorithmic systems that are powered by state-of-art ML, such as Natural Language Processing, Large Language Models, Deep Learning, Computer Vision and Causal Modeling, to seamlessly engage with Sellers? Are you excited by the prospect of analyzing and modeling terabytes of data and creating cutting edge algorithms to solve real world problems? Do you like to build end-to-end business solutions and directly impact the profitability of the company and experience of our customers? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Selling Partner Experience Science team. Key job responsibilities - Use statistical and machine learning techniques to create the next generation of the tools that empower Amazon's Selling Partners to succeed. - Design, develop and deploy highly innovative models to interact with Sellers and delight them with solutions. - Work closely with teams of scientists and software engineers to drive real-time model implementations and deliver novel and highly impactful features. - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation. - Research and implement novel machine learning and statistical approaches. - Participate in strategic initiatives to employ the most recent advances in ML in a fast-paced, experimental environment. About the team Selling Partner Experience Science is a growing team of scientists, engineers and product leaders engaged in the research and development of the next generation of ML-driven technology to empower Amazon's Selling Partners to succeed. We draw from many science domains, from Natural Language Processing to Computer Vision to Optimization to Economics, to create solutions that seamlessly and automatically engage with Sellers, solve their problems, and help them grow. Focused on collaboration, innovation and strategic impact, we work closely with other science and technology teams, product and operations organizations, and with senior leadership, to transform the Selling Partner experience. We are open to hiring candidates to work out of one of the following locations: Denver, CO, USA | Seattle, WA, USA
US, WA, Seattle
Amazon is investing heavily in building a world class advertising business and developing a collection of self-service performance advertising products that drive discovery and sales. Our products are strategically important to our Retail and Marketplace businesses for driving long-term growth. We deliver billions of ad impressions and millions of clicks daily and are breaking fresh ground to create world-class products. We are highly motivated, collaborative and fun-loving with an entrepreneurial spirit and bias for action. With a broad mandate to experiment and innovate, we are growing at an unprecedented rate with a seemingly endless range of new opportunities. Key job responsibilities Search Supply and Experiences, within Sponsored Products, is seeking a Senior Data Scientist to join a fast growing team with the mandate of creating new ads experience that elevates the shopping experience for our hundreds of millions customers worldwide. We are looking for a top analytical mind capable of understanding our complex ecosystem of advertisers participating in a pay-per-click model– and leveraging this knowledge to help turn the flywheel of the business. As a Senior Data Scientist on this team you will: - Lead Data Science solutions from beginning to end. - Deliver with independence on challenging large-scale problems with ambiguity. - Manage and drive the technical and analytical aspects of Advertiser segmentation; continually advance approach and methods. - Write code (Python, R, Scala, etc.) to analyze data and build statistical models to solve specific business problems - Retrieve, synthesize, and present critical data in a format that is immediately useful to answering specific questions or improving system performance. - Analyze historical data to identify trends and support decision making. - Improve upon existing methodologies by developing new data sources, testing model enhancements, and fine-tuning model parameters. - Provide requirements to develop analytic capabilities, platforms, and pipelines. - Apply statistical and machine learning knowledge to specific business problems and data. - Formalize assumptions about how our systems should work, create statistical definitions of outliers, and develop methods to systematically identify outliers. Work out why such examples are outliers and define if any actions needed. - Given anecdotes about anomalies or generate automatic scripts to define anomalies, deep dive to explain why they happen, and identify fixes. - Build decision-making models and propose solution for the business problem you defined - Conduct written and verbal presentation to share insights and recommendations to audiences of varying levels of technical sophistication. - Write code (python or another object-oriented language) for data analyzing and modeling algorithms. A day in the life The Senior Data Scientist will have the opportunity to use one of the world's largest eCommerce and advertising data sets to influence the evolution of our products. This role requires an individual with excellent business, communication, and technical skills, enabling collaboration with various functions, including product managers, software engineers, economists and data scientists, as well as senior leadership. This role will create and enhance performance monitoring reports to find insights that product and business team should focus on. The successful candidate will be a self-starter comfortable with ambiguity, with strong attention to detail, and with an ability to work in a fast-paced, high-energy and ever-changing environment. The drive and capability to shape the direction is a must. This role will influence the direction of the business by leveraging our data to deliver insights that drive decisions and actions. The role will involve translating broad business problems into specific analytics projects, conducting deep quantitative analyses, and communicating results effectively. The role will help the organization identify, evaluate, and evangelize new techniques and tools to continue to improve our ability to deliver value to Amazon’s customers. About the team We are a customer-obsessed team of engineers, technologists, product leaders, and scientists. We are focused on continuous exploration of contexts and creatives where advertising delivers value to customers and advertisers. We specifically work on new ads experiences globally with the goal of helping shoppers make the most informed purchase decision. We obsess about our customers and we are continuously innovating on their behalf to enrich their shopping experience on Amazon We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA