NeurIPS: Why causal-representation learning may be the future of AI

Francesco Locatello on the four NeurIPS papers he coauthored this year, which largely concern generalization to out-of-distribution test data.

In a conversation right before the 2021 Conference on Neural Information Processing Systems (NeurIPS), Amazon vice president and distinguished scientist Bernhard Schölkopf — according to Google Scholar, the most highly cited researcher in the field of causal inference — said that the next frontier in artificial-intelligence research was causal-representation learning.

Where existing approaches to causal inference use machine learning to discover causal relationships between variables — say, the latencies of various interrelated services on a website — causal-representation learning learns the variables themselves. “These kinds of causal representations will also go toward reasoning, which we will ultimately need if we want to move away from this pure pattern recognition view of intelligence,” Schölkopf said.

Francesco.jpg
Senior applied scientist Francesco Locatello.

Francesco Locatello, a senior applied scientist with Amazon Web Services, leads Amazon’s research on causal-representation learning, and he’s a coauthor on four papers at this year’s NeurIPS.

Assaying out-of-distribution generalization in transfer learning” concerns one of the most compelling applications of causal inference in machine learning: generalizing models trained on data with a particular probability distribution to real-world data with a different distribution.

“When you do standard machine learning, you are drawing independent samples from some probability distribution, and then you train a model that's going to generalize to the same distribution,” Locatello explains. “You're describing a physical system using a single probability distribution. Causal models are different because they model every possible state that this physical system can take as a result of an intervention. So instead of having a single probability distribution, you have a set of distributions.

Related content
Amazon Science hosts a conversation with Amazon Scholars Michael I. Jordan and Michael Kearns and Amazon distinguished scientist Bernhard Schölkopf.

“What does it mean that your test data comes from a different distribution? You have the same underlying physical system; the causal structure is the same. It's just a new intervention you have not seen. Your test distribution is different than the training, but now it's not an arbitrary distribution. It’s well posed because it's entailed by the causal structure, and it's a meaningful distribution that may happen in the real world.”

In “Assaying out-of-distribution generalization in transfer learning”, Locatello explains, “what we do is to collect a huge variety of datasets that are constructed for or adapted to this scenario where you have a very narrow data set that you can use for transfer learning, and then you have a wide variety of test data that is all out of distribution. We look at the different approaches that have been studied in the literature and compare them on fair ground.”

Although none of the approaches canvassed in the paper explicitly considers causality, Locatello says, “causal approaches should eventually be able to do better on this benchmark, and this will allow us to evaluate our progress. That's why we built it.”

Neural circuits

Today’s neural networks do representation learning as a matter of course: their inputs are usually raw data, and they learn during training which aspects of the data are most useful for the task at hand. As Schölkopf pointed out in conversation last year, causal-representation learning would simply bring causal machine learning models up to speed with conventional models.

Related content
New method goes beyond Granger causality to identify only the true causes of a target time series, given some graph constraints.

“The important thing to realize is that most machine learning applications don't come structured as a set of well-defined random variables that fully align with the underlying functioning of a physical system,” Locatello explains. “We still want to model these systems in terms of abstract variables, but nobody gives these variables to us. So you may want to learn them in order to be able to perform causal inference.”

Among his and his colleagues’ NeurIPS papers, Locatello says, the one that comes closest to the topic of causal-representation learning is “Neural attentive circuits”. Causal models typically represent causal relationships using graphs, and a neural network, too, can be thought of as an enormous graph. Locatello and his collaborators are trying to make that analogy explicit, by training a neural network to mimic the structure of a causal network.

Neural attentive circuits.png
Visualizations of graph structures learned by neural attentive circuits, from "Neural attentive circuits".

“This is a follow-up on a paper we had last year in NeurIPS,” Locatello says. “The inspiration was to design architectures that behave more similarly to causal models, where you have the noise variables — that's the data — and then you have variables that are being manipulated by functions, and they simply communicate with each other in a graph. And this graph can change dynamically when a distribution changes, for example, because of an intervention.

“In the first paper, we developed an architecture that behaves exactly like that: you have a set of neural functions that can be composed on the fly, depending on the data and the problem. The functions, the routing, and the stitching of the functions are learned. Everything is learned. But it turns out that dynamic stitching is not very scalable.

“In this new work, we essentially compiled the stitching of the functions so that for each sample it's decided beforehand — where it's going to go through the network, how the functions are going to be composed. Instead of doing it on the fly one layer at a time, you decide for the overall forward pass. And we demonstrated that these sparse learned connectivity patterns improve out-of-distribution generalization.”

Success stories

Locatello’s other NeurIPS papers are on more-conventional machine learning topics. “Self supervised amodal video object segmentation” considers the problem of reconstructing the silhouette of an occluded object, which is crucial to robotics applications, including autonomous cars.

Locatello 16_9.png
Segmentations of partially occluded objects, from "Self supervised amodal video object segmentation".

“We exploit the principle that you can build information about an object over time in a video,” Locatello explains. “Perhaps in past frames you've seen parts of the objects that are now occluded. If you can remember that you've seen this object before, and this was its segmentation mask, you can build up your segmentation over time.”

The final paper, “Are two heads the same as one? Identifying disparate treatment in fair neural networks”, considers models whose training objectives are explicitly designed to minimize bias across different types of inputs. Locatello and his colleagues find that frequently, such models — purely through training, without any human intervention — develop two “heads”: that is, they learn two different pathways through the neural network, one for inputs in the sensitive class, and one for all other inputs.

Related content
Amazon ICML paper proposes information-theoretic measurement of quantitative causal contribution.

The researchers argue that, since the network is learning two heads, anyway, it might as well be designed with a two-headed architecture: that would improve performance while meeting the same fairness standard. But this approach hasn’t been adopted, as it runs afoul of rules prohibiting disparate treatment of different groups. In this case, however, disparate treatment could be the best way to ensure fair treatment.

These last two papers are only obliquely related to causality. But, Locatello says, “causal-representation learning is a very young field. So we are trying to identify success stories, and I think these papers are going in that direction.”

“It's clear that causality will have a role in future machine learning,” he adds, “because there are a lot of open problems in machine learning that can at least be partially addressed when you start looking at causal models. And my goal really is to realize the benefits of causal models in mainstream machine learning applications. That's why some of these works are not necessarily about causality, but closer to machine learning. Because ultimately, that's our goal.”

Learn more about Amazon at NeurIPS 2022

For more on the Amazon research being presented at this year's NeurIPS, see our quick guide to Amazon's NeurIPS 2022 papers.

Related content

US, WA, Seattle
AWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we’re the people who keep the cloud running. We support all AWS data centers and all of the servers, storage, networking, power, and cooling equipment that ensure our customers have continual access to the innovation they rely on. We work on the most challenging problems, with thousands of variables impacting the supply chain — and we’re looking for talented people who want to help. You’ll join a diverse team of software, hardware, and network engineers, supply chain specialists, security experts, operations managers, and other vital roles. You’ll collaborate with people across AWS to help us deliver the highest standards for safety and security while providing seemingly infinite capacity at the lowest possible cost for our customers. And you’ll experience an inclusive culture that welcomes bold ideas and empowers you to own them to completion. Come work for M13 - an AWS team specializing in the deception and disruption of cyber threats. We are looking for an Applied Scientist who is passionate about the security domain. You will build services and tools for security engineers and developers that leverage artificial intelligence and machine learning to pull unique insights about the cyber threat landscape. You will be part of a team building Large Language Model (LLM)-based services with the focus on enabling AWS teams to interact with our threat data. The team works in close collaboration with other AWS security services to power mitigations that protect the global AWS network and features in external security services such as Amazon GuardDuty, AWS WAF, and AWS Network Firewall. If you are excited about combating the ever evolving threat landscape then we'd love to talk to you. As an Applied Scientist, you are recognized for your expertise, advise team members on a range of machine learning topics, and work closely with software engineers to drive the delivery of end-to-end modeling solutions. Your work focuses on ambiguous problem areas where the business problem or opportunity may not yet be defined. The problems that you take on require scientific breakthroughs. You take a long-term view of the business objectives, product roadmaps, technologies, and how they should evolve. You drive mindful discussions with customers, engineers, and scientist peers. You bring perspective and provide context for current technology choices, and make recommendations on the right modeling and component design approach to achieve the desired customer experience and business outcome. Key job responsibilities • Understand the challenges that security engineers and developers face when building software today, and develop generalizable solutions. • Collaborate with the team to pave the way towards bringing your solution into production systems. Lead cross team projects and ensure technical blockers are resolved • Communicate and document your research via publishing papers in external scientific venues. About the team *Why AWS* Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. *Diverse Experiences* Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. *Work/Life Balance* We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. *Inclusive Team Culture* Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. *Mentorship and Career Growth* We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.
US, WA, Seattle
We are seeking a senior scientist with demonstrated experience in A/B testing along with related experience with observational causal modeling (e.g. synthetic controls, causal matrix completion). Our team owns "causal inference as a service" for the Pricing and Promotions organization; we run A/B tests on new pricing, promotions, and pricing/promotions CX algorithms and, where experimentation is impractical, conduct observational causal studies. Key job responsibilities We are seeking a senior scientist to help envision, design, and build the next generation of pricing, promotions, and pricing/promotions CX for Amazon. On our team, you will work at the intersection of economic theory, statistical inference, and machine learning to design and implement in production new statistical methods for measuring causal effects of an extensive array of business policies. This position is perfect for someone who has a deep and broad analytic background, is passionate about using mathematical modeling and statistical analysis to make a real difference. You should be familiar with modern tools for data science and business analysis and have experience coding with engineers to put projects into production. We are particularly interested in candidates with research background in experimental statistics. A day in the life - Discuss with business problems with business partners, product managers, and tech leaders - Brainstorm with other scientists to design the right model for the problem at hand - Present the results and new ideas for existing or forward looking problems to leadership - Dive deep into the data - Build working prototypes of models - Work with engineers to implement prototypes in production - Analyze the results and review with partners About the team We are a team of scientists who design and implement the econometrics powering pricing, promotions, and pricing/promotions CX.
US, WA, Seattle
Do you want to join a team of innovative scientists to research and develop generative AI technology that would disrupt the industry? Do you enjoy dealing with ambiguity and working on hard problems in a fast-paced environment? Amazon Connect is a highly disruptive cloud-based contact center from AWS that enables businesses to deliver intelligent, engaging, dynamic, and personalized customer service experiences. As an Applied Scientist on our team, you will work closely with senior technical and business leaders from within the team and across AWS. You distill insight from huge data sets, conduct cutting edge research, foster ML models from conception to deployment. You have deep expertise in machine learning and deep learning broadly, and extensive domain knowledge in natural language processing, generative AI and LLMs, etc. The ideal candidate has the ability to understand, implement, innovate and on the state-of-the-art generative AI based systems. You are comfortable with quickly prototyping and iterating your ideas to build robust ML models using technology such as PyTorch, Tensorflow, AWS Sagemaker, and SparkML. Our team is at an early stage, so you will have significant impact on our ML deliverables with little operational load from existing models/systems. We have a rapidly growing customer base and an exciting charter in front of us that includes solving highly complex engineering and scientific problems. We are looking for passionate, talented, and experienced people to join us to innovate on modern contact centers in the cloud. The position represents a rare opportunity to be a part of a fast-growing business soon after launch, and help shape the technology and product as we grow. You will be playing a crucial role in developing the next generation contact center, and get the opportunity to design and deliver scalable, resilient systems while maintaining a constant customer focus. Our team is leading ML and optimization features in Amazon Connect. We are a team of scientists and engineers working on multiple science projects for Amazon Connect. We use state-of-the-art science and engineering practices to address the hard problems in contact center operation and management for our customers, and we move fast to implement solutions and refine them based on customer feedback. Learn more about Amazon Connect here: https://aws.amazon.com/connect/ About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Hybrid Work We value innovation and recognize this sometimes requires uninterrupted time to focus on a build. We also value in-person collaboration and time spent face-to-face. Our team affords employees options to work in the office every day or in a flexible, hybrid work model near one of our U.S. Amazon offices.
US, CA, San Diego
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to help Amazon provide the best customer experience by preventing eCommerce fraud? Are you excited by the prospect of analyzing and modeling terabytes of data and creating state-of-the-art algorithms to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you enjoy collaborating in a diverse team environment? If yes, then you may be a great fit to join the Amazon Buyer Risk Prevention (BRP) Machine Learning group. We are looking for a talented scientist who is passionate to build advanced algorithmic systems that help manage safety of millions of transactions every day. Key job responsibilities Use machine learning and statistical techniques to create scalable risk management systems Learning and understanding large amounts of Amazon’s historical business data for specific instances of risk or broader risk trends Design, development and evaluation of highly innovative models for risk management Working closely with software engineering teams to drive real-time model implementations and new feature creations Working closely with operations staff to optimize risk management operations, Establishing scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation Tracking general business activity and providing clear, compelling management reporting on a regular basis Research and implement novel machine learning and statistical approaches
US, WA, Seattle
Amazon’s Global Media and Entertainment (GME) organization is creating a future of entertainment where creative content, innovation, and commerce come together. We leverage Amazon’s unique expertise across video, music, gaming, and more to create a truly immersive entertainment experience. Our team, GME Science, is focused on building science tools to optimize Amazon’s entertainment offerings, so that we can provide a great customer experience while operating as a sustainable and profitable business. We push ourselves to Think Big, building ambitious models that create value in multiple GME businesses. This role will expand our team’s measurement work. Business leaders need to quickly understand the long-term impact of various investments, such as new website features, content creation, or marketing campaigns. Our team figures out how to take short-term signals – such as clicks or signups – and turn them into estimates of long-term financial impacts. We work with measurement teams in each business as well as central teams to build foundational measurement science and adapt it for unique use cases. One particular application for this role is to build a principled approach to valuing content/talent deals that include multiple GME businesses. Each deal is unique, featuring talent from film, sports, music, and other top industries, with contract terms that could include video content, podcasts, live appearances, and more. Our valuations need to be structured so that they are comparable across deals, yet flexible enough to account for diverse contracts. To be successful in this role, you will need effective communication, an ability to work closely with stakeholders across our many GME partner teams, and the skill to translate data-driven findings into actionable insights. This includes developing a deep understanding of our business context, which is ambiguous and can change quickly. Your work will be used by decision-makers across GME to deliver the best entertainment experience for our customers, which means we have a high bar. Our healthy team culture is supportive and fast-paced, and we prioritize learning, growth, and helping each other to continuously raise the bar. Impact and Career Growth In today’s entertainment landscape, critical decisions are made with data and economic models. You’ll help GME leaders ask the right questions, and then deliver data-driven answers, creating the future of GME at Amazon. You’ll help define a long-term science vision in this space and translate it into an actionable roadmap. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding – a perfect recipe for career growth as an economist in tech. Key job responsibilities • Design and build econometric models, especially causal models, to measure the value of the business and its many features • Develop science products from concept to prototype to production, incorporating feedback from scientists and business partners • Independently identify and pursue new opportunities to leverage economic insights across GME businesses • Write business and technical documents communicating business context, methods, and results to business leadership and other scientists • Serve as a technical reviewer for our team and related teams, including document and code reviews
GB, Cambridge
The Artificial General Intelligence team (AGI) has an exciting position for an Applied Scientist with a strong background NLP and Large Language Models to help us develop state-of-the-art conversational systems. As part of this team, you will collaborate with talented scientists and software engineers to enable conversational assistants capabilities to support the use of external tools and sources of information, and develop novel reasoning capabilities to revolutionise the user experience for millions of Alexa customers. Key job responsibilities As an Applied Scientist, you will develop innovative solutions to complex problems to extend the functionalities of conversational assistants . You will use your technical expertise to research and implement novel algorithms and modelling solutions in collaboration with other scientists and engineers. You will analyse customer behaviours and define metrics to enable the identification of actionable insights and measure improvements in customer experience. You will communicate results and insights to both technical and non-technical audiences through written reports, presentations and external publications.
US, WA, Seattle
Amazons Price Optimization science team is seeking a Senior Scientist to harness planet scale multi-modal datasets, navigate a continuously evolving competitor landscape, in order to regularly generate fresh customer-relevant prices on billions of Amazon and Third Party Seller products worldwide. This is a high visibility, high impact role for a seasoned, intellectually curious scientist able to partition customer problems into discrete solvable components, build or leverage existing approaches to deliver those components, and innovate to deploy the science into measurable customer-improving outputs. This role requires an individual with exceptional machine learning and reinforcement learning modeling expertise, a strong statistical background, excellent cross-functional collaboration skills, outstanding business acumen, and an entrepreneurial spirit. We are looking for an experienced innovator, who is a self-starter, comfortable with ambiguity, demonstrates strong attention to detail, and has the ability to work in a fast-paced and ever-changing environment. Price is a highly relevant input into many partner-team architectures, and is highly relevant to the customer, therefore this role creates the opportunity to drive extremely large impact (measured in Bs not Ms), but demands careful thought and clear communication. Key job responsibilities We are hiring a senior applied scientist to drive our pricing optimization initiatives. The Price Optimization science team drives cross-domain and cross-system improvements through: * Using cross-ASIN signals to optimally price bundles, ensure price rationality across products, and discovering and launch optimal promotional bundles * invent and deliver price optimization, simulation, and competitiveness tools for 3p Sellers. * shape and extend our bandit optimization platform - a pricing centric multi-armed bandit platform that automates the optimization of various system parameters and price inputs. * Promotion optimization initiatives exploring CX, discount amount, and cross-product optimization opportunities. * Identifying opportunities to optimally price across systems and contexts (marketplaces, request types, event periods) About the team The Pricing Optimization science team owns price quality, discovery and discount optimization initiatives across Amazon’s internal pricing architecture as well as upwards into the customer discovery funnel. We leverage planet scale data on billions of Amazon and external competitor products to build advanced optimization models for pricing, elasticity estimation, product substitutability, and optimization. We preserve long term customer trust by ensuring Amazon's prices are always competitive and error free.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a highly skilled and experienced Senior Applied Scientist, to lead the development and implementation of cutting-edge algorithms and models for supervised fine-tuning and reinforcement learning through human feedback; with a focus across text, image, and video modalities. As a Senior Applied Scientist, you will play a critical role in driving the development of Generative AI (GenAI) technologies that can handle Amazon-scale use cases and have a significant impact on our customers' experiences. Key job responsibilities - Collaborate with cross-functional teams of engineers, product managers, and scientists to identify and solve complex problems in GenAI - Design and execute experiments to evaluate the performance of different algorithms and models, and iterate quickly to improve results - Think big about the arc of development of GenAI over a multi-year horizon, and identify new opportunities to apply these technologies to solve real-world problems - Communicate results and insights to both technical and non-technical audiences, including through presentations and written reports - Mentor and guide junior scientists and engineers, and contribute to the overall growth and development of the team
US, WA, Bellevue
Amazon’s Last Mile Team is looking for a passionate individual with strong optimization and analytical skills to join its Last Mile Science team in the endeavor of designing and improving the most complex planning of delivery network in the world. Last Mile builds global solutions that enable Amazon to attract an elastic supply of drivers, companies, and assets needed to deliver Amazon's and other shippers' volumes at the lowest cost and with the best customer delivery experience. Last Mile Science team owns the core decision models in the space of jurisdiction planning, delivery channel and modes network design, capacity planning for on the road and at delivery stations, routing inputs estimation and optimization. Our research has direct impact on customer experience, driver and station associate experience, Delivery Service Partner (DSP)’s success and the sustainable growth of Amazon. Optimizing the last mile delivery requires deep understanding of transportation, supply chain management, pricing strategies and forecasting. Only through innovative and strategic thinking, we will make the right capital investments in technology, assets and infrastructures that allows for long-term success. Our team members have an opportunity to be on the forefront of supply chain thought leadership by working on some of the most difficult problems in the industry with some of the best product managers, scientists, and software engineers in the industry. Key job responsibilities Candidates will be responsible for developing solutions to better manage and optimize delivery capacity in the last mile network. The successful candidate should have solid research experience in one or more technical areas of Operations Research or Machine Learning. These positions will focus on identifying and analyzing opportunities to improve existing algorithms and also on optimizing the system policies across the management of external delivery service providers and internal planning strategies. They require superior logical thinkers who are able to quickly approach large ambiguous problems, turn high-level business requirements into mathematical models, identify the right solution approach, and contribute to the software development for production systems. To support their proposals, candidates should be able to independently mine and analyze data, and be able to use any necessary programming and statistical analysis software to do so. Successful candidates must thrive in fast-paced environments, which encourage collaborative and creative problem solving, be able to measure and estimate risks, constructively critique peer research, and align research focuses with the Amazon's strategic needs.
IN, KA, Bangalore
Alexa is the voice activated digital assistant powering devices like Amazon Echo, Echo Dot, Echo Show, and Fire TV, which are at the forefront of this latest technology wave. To preserve our customers’ experience and trust, the Alexa Sensitive Content Intelligence (ASCI) team creates policies and builds services and tools through Machine Learning techniques to detect and mitigate sensitive content across Alexa. We are looking for an experienced Applied Science Manager to lead a team to build industry-leading technologies in attribute extraction and sensitive content detection across all languages and countries. A Manager, Applied Science will be a tech leader for a team of exceptional scientists to develop novel algorithms and modeling techniques to advance the state of the art in NLP or CV related tasks. You will work in a hybrid, fast-paced organization where scientists, engineers, and product managers work together to build customer facing experiences. You will collaborate with and mentor other scientists to raise the bar of scientific research in Amazon. Your work will directly impact our customers in the form of products and services that make use of speech, language, and computer vision technologies. We are looking for a leader with strong technical experiences a passion for building scientific driven solutions in a fast-paced environment. You should have good understanding of NLP models (e.g. LSTM, transformer based models) or CV models (e.g. CNN, AlexNet, ResNet) and where to apply them in different business cases. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience of building large-scale distributed systems to creating reliable, scalable, and high-performance products. In addition to technical depth, you must possess exceptional communication skills and understand how to influence key stakeholders. You will be joining a select group of people making history producing one of the most highly rated products in Amazon's history, so if you are looking for a challenging and innovative role where you can solve important problems while growing as a leader, this may be the place for you. Key job responsibilities You'll lead and manage the science driven solution development including design, run experiments, research new algorithms, and find new ways of optimizing customer experience. You set examples for the team on good science practice and standards. Besides theoretical analysis and innovation, you will work closely with talented engineers and ML scientists to put your algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally. You contribute directly to our growth by hiring smart and motivated Scientists to establish teams that can deliver swiftly and predictably, adjusting in an agile fashion to deliver what our customers need. A day in the life You will be working with a group of talented scientists as well as stakeholder from different functional areas (e.g. product, engineering) on researching algorithm and running experiments to test scientific proposal/solutions to improve our sensitive contents detection and mitigation. This will involve collaboration with partner teams including engineering, PMs, data annotators, and other scientists to discuss data quality, policy, and model development. You will mentor other scientists, review and guide their work, help develop roadmaps for the team. You work closely with partner teams across Alexa to deliver platform features that require cross-team leadership. About the team The mission of the Alexa Sensitive Content Intelligence (ASCI) team is to (1) minimize negative surprises to customers caused by sensitive content, (2) detect and prevent potential brand-damaging interactions, and (3) build customer trust through appropriate interactions on sensitive topics. The term “sensitive content” includes within its scope a wide range of categories of content such as offensive content (e.g., hate speech, racist speech), profanity, content that is suitable only for certain age groups, politically polarizing content, and religiously polarizing content. The term “content” refers to any material that is exposed to customers by Alexa (including both 1P and 3P experiences) and includes text, speech, audio, and video.