Improving explainable AI’s explanations

Causal analysis improves both the classification accuracy and the relevance of the concepts identified by popular concept-based explanatory models.

Explainability is an important research topic in AI today. If we’re going to trust deep-learning systems to make decisions for us, we often want to know why they make the decisions they do.

One popular approach to explainable AI is concept-based explanation. Instead of simply learning to predict labels from input features, the model learns to assign values to a large array of concepts. For instance, if the inputs are images of birds, the concepts might be things like bill shape, breast color, and wing pattern. Then, on the basis of the concept values, the model classifies the input: say, a yellow grosbeak.

More ICLR-related content

Today, as part of our ICLR coverage, Amazon Science also features a profile of Michael Bronstein, a professor of computer science at Imperial College London who received an Amazon Research Award for work that pushes the boundaries of drug design, reveals the cancer-fighting properties of food — and even decodes whale-speak.

But this approach can run into trouble if there are confounders in the training data. For instance, if birds with spatulate bills are consistently photographed on the water, the model could learn to associate water imagery with the concept “bill shape: spatulate”. And that could produce nonsensical results in the case of, say, a starling that happened to be photographed near a lake.

In a paper that Amazon distinguished scientist David Heckerman and I are presenting this week at the International Conference on Learning Representations (ICLR), we adapt a technique for removing confounders from causal models, called instrumental-variable analysis, to the problem of concept-based explanation.

In tests on a benchmark dataset of images annotated with concept labels, we show that our method increases the classification accuracy of a concept-based explanatory model by an average of 25%. Using the remove-and-retrain (ROAR) methodology, we also show that our method improves the model’s ability to identify concepts relevant to the correct image label.

Oversimplified causal graph.png
A simple (too simple) causal graph of a concept-based explanatory model.

Our analysis begins with a causal graph, which encodes our prior belief about the causal relationships among the variables. In our case, the belief is that a prediction target (y) causes a concept representation (c), which in turn causes an input (x). Note that prediction happens in the opposite direction, but this doesn’t matter as the statistical relationships between data and concept and concept and label remain the same.

Confounders complicate this simple model. In the figure below, u is a confounder, which influences both the input and the concept (c) learned by the model; d is the debiased concept we wish to learn.  

Causal graph with confounder.png
A more realistic causal graph of a concept-based explanatory model, with a confounding variable (u) and a debiased concept variable (d).

In the terms of our example, u represents the watery backgrounds common to images of birds with spatulate bills, c is the confounded concept of bill shape, and d is a debiased concept of bill shape, which correlates with actual visual features of birds’ bills.

Note, too, that there is a second causal path between input and label, which bypasses concept representation. The experts who label images of birds, for instance, may rely on image features not captured by the list of concepts.

Our approach uses a trick from classic instrumental-variable analysis, which considers the case in which a variable p has a causal effect on the variable q, but that effect is obscured by a confounding variable, u, which influences both p and q. The analysis posits an instrumental variable, z, which is correlated with p but not with q. Instrumental-variable analysis uses regression to estimate p from z; since z is independent of the confounder u, so is the estimate of p, known as p̂. A regression of q on p̂ is thus an estimate of the causal impact of p on q.

Causal graph with instrumental variable.png
Our updated causal model, in which we use regression to estimate concepts (ĉ) from labels (y).

In our causal graph above, we can use regression to estimate d from y and c from d, breaking the causal link between u and the estimate of cĉ. (In practice, we just set the estimate of c equal to the estimate of d.)

Using a benchmark dataset that contains 11,788 images of 200 types of birds, annotated according to 312 concepts, we trained two concept-based explanatory models, which were identical except that one used regression to estimate concepts and one didn’t. The model that used regression was 25% more accurate than the one that didn’t.

Debiased performance.png
Our debiased model (red) exhibits greater relative accuracy improvements than baseline (blue) as we successively remove more and more irrelevant concepts from the training data, indicating that it does a better job of identifying relevant concepts.

The accuracy of the classifier, however, doesn’t tell us anything about the accuracy of the concept identification, which is the other purpose of the model. To evaluate that, we used the ROAR method. First, we train both models using all 312 concepts for each training example. Then we discard the least relevant 31 concepts (10%) for each training example and re-train the models. Then we discard the next least relevant 31 concepts per example and re-train, and so on.

We find that, as irrelevant concepts are discarded, our debiased model exhibits greater relative improvement in accuracy than the baseline model. This indicates that our model is doing a better job than baseline of identifying relevant concepts.

Research areas
About the Author
Taha Bahadori is a senior machine learning scientist in the Amazon Devices organization.

Related content

US, WA, Seattle
Job summaryAre you passionate about conducting measurement research and experiments to assess and evaluate talent? Would you like to see your research in products that will drive key talent management behaviors globally to ensure we are raising the bar on our talent? If so, you should consider joining the CXNS team!Amazon CXNS team is an innovative organization that exists to propel Amazon HR toward being the most scientific HR organization on earth. CNXS mission is to use Science to assist and measurably improve every talent decision made at Amazon. CXNS does this by discovering signals in workforce data, infusing intelligence into Amazon’s talent products, and guiding the broader CXNS team to pursue high-impact opportunities with tangible returns. This multi-disciplinary approach spans capabilities, including: data engineering, reporting and analytics, research and behavioral sciences, and applied sciences such as economics and machine learning.In this role, you will support measurement efforts for Amazon Connections (an innovative program that gives Amazonians a confidential and effective way to give feedback on the workplace to help shape the future of the company and improve the employee experience). You will own the research development strategy to evaluate, diagnose, understand, and surface drivers and moderators for key research streams. These include (but are not limited to) attrition, engagement, productivity, diversity, and Amazon culture. You will deep dive and analyze what research should be conducted and to what end, develop hypotheses that can be tested, and support a larger research program to deliver deeper insights that we can surface to leaders on our platform (short term and long).You will use both quantitative and qualitative data as well as conduct research studies to test your hypotheses. You will use a variety of statistical approaches to model and understand behavior. You will develop algorithms and thresholds to surface personalized results to managers/leaders, and partner with machine learning scientists to build these statistical models into production that scales. You will work with an interdisciplinary team of psychologists, economists, ML scientists, UX researchers, engineers, and product managers to inform and build product features to surface deeper people and business insights for our leaders.What you'll do:· Lead a global research strategy to drive more effective decisions and improve the employee experience across all of Amazon· Execute a scalable global content development and research strategy Amazon-wide· Conduct psychometrics analyses to evaluate integrity and practical application of content· Identify research streams to evaluate how to mitigate or remove sources of measurement error· Partner closely and drive effective collaborations across multi-disciplinary research and product teams· Manage full life cycle of large scale research programs (Develop strategy, gather requirements, execute, and evaluate)This person will possess knowledge of different assessment approaches to evaluate performance, a strong psychometrics background, scientific survey methodology, and computing various content validity analyses.
US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for an Economist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.Economists at Amazon are solving some of the most challenging applied economics questions in the tech sector. Amazon economists apply the frontier of economic thinking to market design, pricing, forecasting, program evaluation, online advertising and other areas. Our economists build econometric models using our world class data systems, and apply economic theory to solve business problems in a fast-moving environment. A career at Amazon affords economists the opportunity to work with data of unparalleled quality, apply rigorous applied econometric approaches, and work with some of the most talented applied econometricians in the trade.As the Economist within WW Installments, you will be responsible for building long-term causal inference models and experiments. These analysis represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how objective functions are designed and which inputs are consumed for modeling. You will work across functions including machine learning, business intelligence, data engineering, software development, and finance to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing a causal inference and experimentation roadmap for the WW Installments Competitive Pricing team.• Apply expertise in causal and econometric modeling to develop large-scale systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the project plan from a scientific perspective on product launches including identifying potential risks, key milestones, and paths to mitigate risks• Own key inputs to reports consumed by VPs and Directors across Amazon.• Identifying new opportunities to influence business strategy and product vision using causal inference.• Continually improve the WW Installments experimentation roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver analytical projects and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for an Applied Scientist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.As an Applied Scientist within WW Installments, you will be responsible for building machine learning models and pipelines with direct customer impact. These models represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how they interact with financing options to make purchases. You will work across functions including data engineering, software development, and business to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing production machine learning models and pipelines for the WW Installments Competitive Pricing team that directly impact customers.• Apply expertise in machine learning to develop large-scale production systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the implementation of production ML from a scientific perspective including identifying potential risks, key milestones, and paths to mitigate risks.• Identifying new opportunities to influence business strategy and product vision using data science and machine learning.• Continually improve the WW Installments ML roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver ML pipelines, analytics projects, and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, WA, Seattle
Job summaryWW Installments is one of the fastest growing businesses within Amazon and we are looking for a Data Scientist to join the team. This group has been entrusted with a massive charter that will impact every customer that visits Amazon.com. We are building the next generation of features and payment products that maximize customer enablement in a simple, transparent, and customer obsessed way. Through these products, we will deliver value directly to Amazon customers improving the shopping experience for hundreds of millions of customers worldwide. Our mission is to delight our customers by building payment experiences and financial services that are trusted, valued, and easy to use from anywhere in any way.As a Data Scientist within WW Installments, you will be responsible for building machine learning models and pipelines with direct customer impact. These models represent a core capability for WW Installments and businesses across Amazon. Your work will directly impact customers by influencing how they interact with financing options to make purchases. You will work across functions including data engineering, software development, and business to induce data driven decisions at every level of the organization.Key job responsibilitiesThis role will be responsible for:• Developing machine learning models and pipelines for the WW Installments Competitive Pricing team.• Apply expertise in machine learning to develop large-scale systems that are deployed across Amazon businesses.• Identify business opportunities, define and execute modeling approach, then deliver outcomes to various Amazon businesses with an Amazon-wide perspective for solutions.• Lead the project plan from a scientific perspective on product launches including identifying potential risks, key milestones, and paths to mitigate risks.• Own key inputs to reports consumed by VPs and Directors across Amazon.• Identifying new opportunities to influence business strategy and product vision using data science and machine learning.• Continually improve the WW Installments ML roadmap automating and simplifying whenever possible.• Coordinate support across engineers, scientists, and stakeholders to deliver ML pipelines, analytics projects, and build proof of concept applications.• Work through significant business and technical ambiguity delivering on analytics roadmap across the team with autonomy.
US, CA, Palo Alto
Job summaryAmazon is the 4th most popular site in the US (http://www.alexa.com/topsites/countries/US). Our product search engine is one of the most heavily used services in the world, indexes billions of products, and serves hundreds of millions of customers world-wide. We are working on a new AI-first initiative to re-architect and reinvent the way we do search through the use of extremely large scale next-generation deep learning techniques. Our goal is to make step function improvements in the use of advanced Machine Learning (ML) on very large scale datasets, specifically through the use of aggressive systems engineering and hardware accelerators. This is a rare opportunity to develop cutting edge ML solutions and apply them to a problem of this magnitude. Some exciting questions that we expect to answer over the next few years include:· Can a focus on compilers and custom hardware help us accelerate model training and reduce hardware costs?· Can combining supervised multi-task training with unsupervised training help us to improve model accuracy?· Can we transfer our knowledge of the customer to every language and every locale ? The Search Science team is looking for a Senior Applied Science Manager to drive roadmap on making large business impact through application of Deep Learning models via close collaboration with partner teams. The team also has a focus on technology solution for deep-learning based embedding generation, sensitive data ingestion and applications, data quality measurement, improvement, data bias identification and reduction to achieve model fairness.Success in this role will require the courage to chart a new course. You will manage your own team to understand all aspects of the customer journey. You and your team will inform other scientists and engineers by providing insights and building models to help improving training data quality and reducing bias. The research focus includes but not limited to Natural Language Processing, recommendation, applications relevant to Amazon buyers, sellers and more. You will be working with cutting edge technologies that enable big data and parallelizable algorithms. You will play an active role in translating business and functional requirements into concrete deliverables and working closely with software development teams to put solutions into production.
US, WA, Seattle
Job summaryAmazon EC2 provides cloud computing which forms the foundation for the majority of AWS services, as well as a large portion of compute use cases for businesses and individuals around the world. A critical factor in the continued success of EC2 is the ability to provide reliable and cost effective computing. The EC2 Fleet Health and Lifecycle (EC2 FHL) organization is responsible for ensuring that the global EC2 server fleet continues to raise the bar for reliability, security, and efficiency. We are looking for seasoned engineering leaders with passion for technology and an entrepreneurial mindset. At Amazon, it is all about working hard, having fun and making history. If you are ready to make history, we want to hear from you!Come join a brand new team, EC2 Health Analytics, under EC2 Foundational Technology, to solve complex cutting-edge problems to power a faster, more robust and performant EC2 of tomorrow. The charter of our team is to improve customer experience on the EC2 fleet by analyzing hundreds of signals and driving next-generation detection and remediation tools. We apply Machine Learning to predict outcomes and optimize decisions that improve customer experience and operational efficiency. As an Applied Scientist in the EC2 Health Analytics team, you will join an industry-leading engineering team solving challenging problems at massive scale.· Build a world-class forecasting platform that scales to handling billions of time series data in real time.· Drive fleet utilization improvement where each 1% means tens of millions of additional free cash flow.· Automate tactical and strategic capacity planning tools to optimize for service availability and infrastructure cost.· Build recommendation algorithms for improving the AWS customer experience.· · Reduce dependence on manual troubleshooting for deep-dives.What you will learn:· State-of-the-art analytics and forecasting methodologies.· Application of machine learning to large-scale data sets.· · Product recommendation algorithms.· Resource management and admission control for the Cloud.· The internals of all AWS services.Inclusive Team CultureHere at AWS, we embrace our differences. We are committed to furthering our culture of inclusion. We have ten employee-led affinity groups, reaching 40,000 employees in over 190 chapters globally. We have innovative benefit offerings, and host annual and ongoing learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences.Work/Life BalanceOur team puts a high value on work-life balance. It isn’t about how many hours you spend at home or at work; it’s about the flow you establish that brings energy to both parts of your life. We believe striking the right balance between your personal and professional life is critical to life-long happiness and fulfillment. We offer flexibility in working hours and encourage you to find your own balance between your work and personal lives.Mentorship & Career GrowthOur team is dedicated to supporting new members. We have a broad mix of experience levels and tenures, and we’re building an environment that celebrates knowledge sharing and mentorship. Our senior members enjoy one-on-one mentoring and thorough, but kind, code reviews. We care about your career growth and strive to assign projects based on what will help each team member develop into a better-rounded engineer and enable them to take on more complex tasks in the future.
US, CA, Palo Alto
Job summaryThe Amazon Search team creates powerful, customer-focused search and advertising solutions and technologies. Whenever a customer visits an Amazon site worldwide and types in a query or browses through product categories, the Amazon Search services go to work. We design, develop, and deploy high performance, fault-tolerant distributed search systems used by millions of Amazon customers every day. Our team works to maximize the quality and effectiveness of the search experience for visitors to Amazon websites worldwide.
US, WA, Bellevue
Job summaryThe primary mission of ADECT Monitoring team is to protect customer trust and improve customer experience with Alexa skills and devices. As part of this role, you will build models to improve customer’s experience on Alexa. The team uses various signals to ensure that customers get delightful experiences. This could be through experience improvements or ensuring that only high quality experiences reach customers. We use a lot of data along with multiple approaches such as machine learning and other algorithmic approaches to solve challenges that customers face interacting with Alexa. The ideal candidate will be an expert in the areas of data science, machine learning and statistics, having hands-on experience with multiple improvement initiatives as well as balancing technical and business judgment to make the right decisions about technology, models and methodologies. This involves building conversation arbitration models, which validate conversation quality and metrics to measure and continuously improve on it. These are some of the challenges that have not been solved in the industry before. The candidate needs experience with data science / business intelligence, analytics, and reporting systems while striving for simplicity, and demonstrating significant creativity and high judgment backed by statistical proof. The candidate is also expected to take these models into production, so they need to have some experience with software systems as well. There will be guidance provided on the software front though.
US, WA, Bellevue
Job summaryThe primary mission of ADECT Monitoring team is to protect customer trust and improve customer experience with Alexa skills and devices. The team uses various signals to ensure that customers get delightful experiences. This could be through experience improvements or ensuring that only high quality experiences reach customers. We use a lot of data along with multiple approaches such as machine learning and other algorithmic approaches to solve challenges that customers face interacting with Alexa. The ideal candidate will be an expert in the areas of data science, machine learning and statistics, having hands-on experience with multiple improvement initiatives as well as balancing technical and business judgment to make the right decisions about technology, models and methodologies. As part of this role, you will build models to improve customer’s experience on Alexa. This involves building conversation arbitration models, which validate conversation quality and metrics to measure and continuously improve on it. These are some of the challenges that have not been solved in the industry before. The candidate needs experience with data science / business intelligence, analytics, and reporting systems while striving for simplicity, and demonstrating significant creativity and high judgment backed by statistical proof. The candidate is also expected to work on ML models to improve customer trust. This role will have an opportunity to convert to an Applied Scientist.
US, CA, Sunnyvale
Job summaryAmazon Lab126 is an inventive research and development company that designs and engineers high-profile consumer electronics. Lab126 began in 2004 as a subsidiary of Amazon.com, Inc., originally creating the best-selling Kindle family of products. Since then, we have produced groundbreaking devices like Fire tablets, Fire TV and Amazon Echo. What will you help us create?The Role:As a Design Analysis Engineer, you will be responsible for bringing new product designs through to manufacturing. Thermal and structural engineering contributes unique, in-depth technical knowledge to solve complex engineering problems in concert with multi-disciplinary teams including Industrial Design, Hardware Engineering, and Operations.You will work closely with multi-disciplinary groups including Product Design, Industrial Design, Hardware Engineering, and Operations, to drive key aspects of engineering of consumer electronics products. In this role, you will:· Perform analysis and testing of complex electronic assemblies using advanced simulation and experimentation tools and techniques· Strong fundamentals in dynamics with emphasis on system dynamics, mechanism analysis (Multi Body Dynamics analysis) and co-simulation· Develop, analyze and test thermal, acoustic and structural solutions; from concept design, feature development, product architecture, through system validation· Support creative developments through application of analysis and testing of complex electronic assemblies using advanced simulation and experimentation tools and techniques· Use simulation tools like Abaqus, LS-Dyna, Simpack for analysis and design of products· Validate design modifications using simulation and actual prototypes· Use of programming languages like Python and Matlab for analytical/statistical analyses and automation· Establish noise thresholds for usability and compliance requirements· Determine and validate structural performance under use and test conditions· Have strong knowledge of various materials such as heat spreaders solutions to resolve thermal issues, damping materials for noise and vibration suppression· Use various data acquisition systems with thermocouples, accelerometers, strain gauges and IR cameras· Collaborate as part of the device team to iterate and optimize design parameters of enclosures and structural parts to establish and deliver project performance objectives· Design and execute tests using statistical tools to validate analytical models, identify risks and assess design margins· Create and present analytical and experimental results· Develop and apply design guidelines based on project results