Determining causality in correlated time series

New method goes beyond Granger causality to identify only the true causes of a target time series, given some graph constraints.

Given observed time series and a target time series of interest, can we identify the causes of the target, without excluding the presence of hidden time series? This question arises in many fields — such as finance, biology, and supply chain management — where sequences of data constitute partial observations of a system.

Imagine, for instance, that we have time series for the prices of dairy products. From the data alone, can we identify the causes of fluctuations in the price of butter?

Dairy prices.png
The prices of dairy products in Germany are correlated, but do any of those correlations imply causation?

The standard way to represent causal relationships between variables that are associated with each other is with a graph whose nodes represent variables and whose edges represent causal relationships.

In a paper that we presented at the International Conference on Machine Learning (ICML) 2021, coauthored by Bernhard Schölkopf, we described a new technique for detecting all the direct causal features of a target time series — and only the direct or indirect causal features — given some graph constraints. The proposed method yielded false-positive rates of detected causes close to zero.

The constraints we observe refer to the target and the “memory” of some hidden time series (the lack of dependency on their own pasts, in some cases). We wanted to limit our assumptions to those that can be naturally derived from the setting and that could not be avoided otherwise. Therefore, we wanted to avoid strong assumptions made by other methods, such as excluding hidden common causes (unobserved time series that caused multiple observed ones).

We also wanted to avoid other drawbacks of prior methods, such as requiring interventions on the system (to test for particular causal sequences) and requiring large conditioning sets (sets of variables that must be controlled for to detect dependences) or exhaustive conditional-independence tests, which hinder the statistical strength of the outcome.

Our method, by contrast, accounts for hidden common causes, uses only observational data, and constructs conditioning sets that are small and efficient in terms of signal-to-noise ratio, given some graph constraints that seemed hard to avoid.

Conditioning set.gif
The researchers' new method constructs a conditioning set — a set of variables that must be controlled for — that enables tests for conditional dependence and independence in a causal graph.

Conditional independence

As is well known, statistical dependence (i.e., correlation in linear cases) does not imply causation. The graphs we use to represent causal relationships between associated variables are so-called directed acyclic graphs (DAGs), meaning the edges have direction and there are no loops. The direction of the edges (represented by arrows in the graphs below) indicates the direction of causal influence. In the time series case, we use “full time DAGs”, where each node represents a different time step from a time series. 

To analyze whether a third variable, S, explains a statistical dependency (i.e., correlation) between two other variables, one checks whether the dependency disappears after restricting the statistics to data points with fixed values of S. In larger graphs, S can be a whole set of variables, which we call a conditioning set. Controlling for all the variables in a conditioning set is known as conditional independence testing and is the main tool we use in our method. 

Another important notion is that of confounding. If two variables, X and Y, are dependent, not because one causes the other, but because they’re both caused by a third variable, U, we say that they are confounded by U.

Before we get into the complex graphs of time series, let's present the intuition behind our method with simple graphs. 

In the graphs below, we manage to distinguish between causal influence and confounding relationships by searching for different patterns of conditional independence. In both graphs, X and Y are dependent (i.e., they vary together). But in the left-hand graph, Z and Y are independent when we condition on the cause X; i.e., when we control for X, variations in Y become independent from variations in Z

When, however, there is a hidden confounder between X and Y, as in the graph at right, Z and Y become dependent when conditioning on X.

This can seem counterintuitive. When we condition on a variable, we treat it as if we know its outcome. In the graph below, because we know how Z contributes to X, the difference between this contribution and the actual value of X comes from U (with some variation from noise). Since Y varies with U, it reflects that variation as well, and Z and Y become dependent.

simple_iid_case.png
An example of how the presence of a confounder can create causal dependence.

Causality in time series

This idea of finding similar characteristic patterns of conditional independences to distinguish causes from confounders is very relevant to our method. In the time series case, the graph is much more complicated than in the examples above. Here we show such a time series graph:

Baseline causal graph.png
A full time graph with hidden time series (U).

Here, we have a univariate (one-dimensional) target time series, Y, whose causes we want to find. Then we have several observed candidate time series, Xi, which might be causing the target or have different dependencies with it. Finally, we allow for the existence of several hidden time series, U.

We know the directions of some edges from the time order, which is helpful. On the other hand, time series’ dependence on their own pasts complicates the picture, because it creates common-cause schemes between nodes. 

For each candidate time series, we want to isolate the current and previous node and the corresponding target node. We thus extract triplets like the one indicated by green and yellow in the graph below.

Causal graph conditional tests.png
Tests for conditional dependence and independence in the full time graph.

If we manage to do that, then it is enough to check whether the green nodes become independent when we simultaneously condition on the yellow node and all the purple ones. 

If there is a hidden confounder between the yellow node and the target’s green node, then, conditioning on the yellow node will force a dependence between the two green nodes, as in the first example above. But to perform that test, we need to isolate our triplet from the causal influences of other time series. 

To do that, we construct a conditioning set, S, that includes at most one node from each time series that is dependent on the target. This node corresponds to the one that enters the previous time stamp of the target (Yt in the graph above). And of course, we also need to include the previous time stamp of the target node itself (Yt, above) to remove the target's past dependency, as well as the yellow node.

Here we see that indeed the relationship between Xj and Y is confounded (Xj does not cause Y, although they appear to be related). We see that the second condition of our method is violated, and consequently, Xj is correctly rejected (as it is not a cause of Y).

Given some restrictions on the graph, which we do not consider extreme given the hardness of hidden confounding, we propose and prove two theorems for the identification of direct and indirect causes in single-lag graphs — that is, graphs in which a node in a candidate time series shares only one edge with nodes in the target time series. These theorems result in an algorithm with only two conditional-independence tests and well-defined conditioning sets, which scales linearly with the number of candidate time series. 

dairy_experiments_graphs.PNG
Graphs of the causal relationships between dairy-product prices in Germany, Ireland, and the UK, with the true-positive rates (TPR) and true-negative rates (TNR) achieved by the researchers' new method.

We now return to our original motivational example, predicting the price of butter. The real-world data we used to test our approach included the price of raw milk, the price of butter, and, depending on the country, the prices of other dairy products, such as cheese and whey powder. Our method correctly deduced that the price of butter was caused by the price of raw milk but not by the prices of other dairy products, although they were strongly dependent on it. In one dataset, where the data did not include the price of raw milk, our method correctly deduced that the dependencies between the price of butter and the prices of other dairy products did not imply causation. 

Research areas

Related content

US, WA, Bellevue
Imagine being part of an agile team where your ideas have the potential to reach millions of customers. Picture working on cutting-edge, customer-facing solutions, where every team member is a critical voice in the decision making process. Envision being able to leverage the resources of a Fortune 500 company within the atmosphere of a start-up. Welcome to Amazon’s NCRC team. We solve complex problems in an ambiguous space, focusing on reducing return costs and improving the customer experience. We build solutions that are distributed on a large scale, positively impacting experiences for our customers and sellers. Come innovate with the NCRC team! The Net Cost of Refunds and Concessions (NCRC) team is looking for a Senior Manager Data Science to lead a team of economists, business intelligence engineers and business analysts who investigate business problems, develop insights and build models & algorithms that predict and quantify new opportunity. The team instigates and productionalizes nascent solutions around four pillars: outbound defects, inbound defects, yield optimization and returns reduction. These four pillars interact, resulting in impacts to our overall return rate, associated costs, and customer satisfaction. You may have seen some downstream impacts of our work including Amazon.com customer satisfaction badges on the website and app, new returns drop off optionality, and faster refunds for low cost items. In this role, you will set the science vision and direction for the team, collaborating with internal stakeholders across our returns and re-commerce teams to scale and advance science solutions. This role is based in Bellevue, WA Key job responsibilities * Single threaded leader responsible for setting and driving science strategy for the organization. * Lead and provide coaching to a team of Scientists, Economists, Business Intelligence Engineers and Business Analysts. * Partner with Engineering, Product and Machine Learning leaders to deliver insights and recommendations across NCRC initiatives. * Lead research and development of models and science products powering return cost reduction. * Educate and evangelize across internal teams on analytics, insights and measurement by writing whitepapers, knowledge documentation and delivering learning sessions. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
US, WA, Seattle
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and Scala would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time economics employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: Chicago, IL, USA | Seattle, WA, USA | Washington, DC, USA
US, CA, Santa Clara
Machine learning (ML) has been strategic to Amazon from the early years. We are pioneers in areas such as recommendation engines, product search, eCommerce fraud detection, and large-scale optimization of fulfillment center operations. The Generative AI team helps AWS customers accelerate the use of Generative AI to solve business and operational challenges and promote innovation in their organization. As an applied scientist, you are proficient in designing and developing advanced ML models to solve diverse challenges and opportunities. You will be working with terabytes of text, images, and other types of data to solve real-world problems. You'll design and run experiments, research new algorithms, and find new ways of optimizing risk, profitability, and customer experience. We’re looking for talented scientists capable of applying ML algorithms and cutting-edge deep learning (DL) and reinforcement learning approaches to areas such as drug discovery, customer segmentation, fraud prevention, capacity planning, predictive maintenance, pricing optimization, call center analytics, player pose estimation, event detection, and virtual assistant among others. AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest growing small- and mid-market accounts to enterprise-level customers including public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to customer success. AWS Support also partners with a global list of customers that are building mission-critical applications on top of AWS services. Key job responsibilities The primary responsibilities of this role are to: Design, develop, and evaluate innovative ML models to solve diverse challenges and opportunities across industries Interact with customer directly to understand their business problems, and help them with defining and implementing scalable Generative AI solutions to solve them Work closely with account teams, research scientist teams, and product engineering teams to drive model implementations and new solutions About the team Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why flexible work hours and arrangements are part of our culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. We are open to hiring candidates to work out of one of the following locations: San Francisco, CA, USA | Santa Clara, CA, USA
US, WA, Bellevue
We are seeking a passionate, talented, and inventive individual to join the Amazon AGI team and help build industry-leading technologies that customers will love. This team offers a unique opportunity to make a significant impact on the customer experience and contribute to the design, architecture, and implementation of a cutting-edge product. Our goal is to provide conversational, visual and proactive mechanisms to delight customers, increase customer engagement, reduce defects, and enable natural interactions across Amazon touch points (e.g., Alexa, Stores, Mobile). To achieve our mission, we build automated, scalable, self-serve AI systems that use customer, device and ambient signals to offer personalized suggestions, comprehend customer inputs, learn from customer interactions, reduce defects, and propose appropriate actions to serve millions of Amazon customers across the globe. Key job responsibilities As a Principal Applied Scientist, you will be a trusted part of the technical leadership. You bring business and industry context to science and technology decisions. You tackle intrinsically hard problems, acquiring expertise as needed and decompose complex problems into straightforward solutions. You align teams toward coherent strategies. You will develop long-term strategies, persuade teams to adopt those strategies, propose goals and deliver on them. You will also participate in organizational planning, hiring, mentorship and leadership development. You will serve as a key scientific resource in full-cycle development (conception, design, implementation, testing to documentation, delivery, and maintenance). About the team The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Principal Scientist to help build industry-leading conversational technologies that customers love. Our mission is to push the envelope in Artificial Intelligence (AI), Natural Language Understanding (NLU), Machine Learning (ML), Dialog Management, Automatic Speech Recognition (ASR), and Audio Signal Processing, in order to provide the best-possible experience for our customers We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA | Boston, MA, USA | Seattle, WA, USA | Sunnyvale, CA, USA
US, WA, Seattle
Are you excited about developing models to revolutionize automation, robotics and computer vision? Are you looking for opportunities to build and deploy them on real problems at truly vast scale? At Amazon Fulfillment Technologies and Robotics we are on a mission to build high-performance autonomous systems that perceive and act to further improve our world-class customer experience - at Amazon scale. We are looking for scientists, engineers and program managers for a variety of roles. The Amazon Robotics software team is seeking a collaborative Applied Scientist to focus on computer vision machine learning models. This includes building multi-viewpoint and time-series computer vision systems. It includes building large-scale models using data from many different tasks and scenes. This work spans from basic research such as cross domain training, to experimenting on prototype in the lab, to running wide-scale A/B tests on robots in our facilities. Key job responsibilities * Research vision - Where should we be focusing our efforts * Research delivery – Proving/dis-proving strategies in offline data or in the lab * Production studies - Insights from production data or ad-hoc experimentation. A day in the life Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! We are open to hiring candidates to work out of one of the following locations: Seattle, WA, USA
US, NY, New York
We are looking for a motivated and experienced Senior Data Scientist with experience in Machine Learning (ML), Artificial Intelligence (AI), Big Data, and Service Oriented Architecture with deep understanding in advertising businesses, to be part of a team of talented scientists and engineers to innovate, iterate, and solve real world problem with cutting-edge AWS technologies. In this role, you will take a leading role in defining the problem, innovating the ML/AI solutions, and information the tech roadmap. You will join a cross-functional, fun-loving team, working closely with scientists and engineers in a daily basis. You will innovate on behalf of our customers by prototyping, delivering functional proofs of concept (POCs), and partnering with our engineers to productize and scale successful POCs. If you are passionate about creating the future, come join us as we have fun, and make history. Key job responsibilities - Define and execute a research & development roadmap that drives data-informed decision making for marketers and advertisers - Establish and drive data hygiene best practices to ensure coherence and integrity of data feeding into production ML/AI solutions - Collaborate with colleagues across science and engineering disciplines for fast turnaround proof-of-concept prototyping at scale - Partner with product managers and stakeholders to define forward-looking product visions and prospective business use cases - Drive and lead of culture of data-driven innovations within and outside across Amazon Ads Marketing orgs About the team Marketing Decision Science provides science products to enable Amazon Ads Marketing to deliver relevant and compelling guidance across marketing channels to prospective and active advertisers for success on Amazon. We own the product, technology and deployment roadmap for AI- and analytics-powered products across Amazon Ads Marketing. We analyze the needs, experiences, and behaviors of Amazon advertisers at petabytes scale, to deliver the right marketing communications to the right advertiser at the right team to help them make the data-informed advertising decisions. Our science-based products enable applications and synergies across Ads organization, spanning marketing, product, and sales use cases. We are open to hiring candidates to work out of one of the following locations: New York, NY, USA
US, WA, Bellevue
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to work with large and complicated data sets. Some knowledge of econometrics, as well as basic familiarity with Python is necessary, and experience with SQL and UNIX would be a plus. These are full-time positions at 40 hours per week, with compensation being awarded on an hourly basis. You will learn how to build data sets and perform applied econometric analysis at Internet speed collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. Roughly 85% of previous cohorts have converted to full time scientist employment at Amazon. If you are interested, please send your CV to our mailing list at econ-internship@amazon.com. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA
IN, KA, Bangalore
Alexa is the voice activated digital assistant powering devices like Amazon Echo, Echo Dot, Echo Show, and Fire TV, which are at the forefront of this latest technology wave. To preserve our customers’ experience and trust, the Alexa Sensitive Content Intelligence (ASCI) team creates policies and builds services and tools through Machine Learning techniques to detect and mitigate sensitive content across Alexa. We are looking for an experienced Senior Applied Scientist to build industry-leading technologies in attribute extraction and sensitive content detection across all languages and countries. An Applied Scientist will be a tech lead for a team of exceptional scientists to develop novel algorithms and modeling techniques to advance the state of the art in NLP or CV related tasks. You will work in a hybrid, fast-paced organization where scientists, engineers, and product managers work together to build customer facing experiences. You will collaborate with and mentor other scientists to raise the bar of scientific research in Amazon. Your work will directly impact our customers in the form of products and services that make use of speech, language, and computer vision technologies. We are looking for a leader with strong technical experiences a passion for building scientific driven solutions in a fast-paced environment. You should have good understanding of NLP models (e.g. LSTM, transformer based models) or CV models (e.g. CNN, AlexNet, ResNet) and where to apply them in different business cases. You leverage your exceptional technical expertise, a sound understanding of the fundamentals of Computer Science, and practical experience of building large-scale distributed systems to creating reliable, scalable, and high-performance products. In addition to technical depth, you must possess exceptional communication skills and understand how to influence key stakeholders. You will be joining a select group of people making history producing one of the most highly rated products in Amazon's history, so if you are looking for a challenging and innovative role where you can solve important problems while growing as a leader, this may be the place for you. Key job responsibilities You'll lead the science solution design, run experiments, research new algorithms, and find new ways of optimizing customer experience. You set examples for the team on good science practice and standards. Besides theoretical analysis and innovation, you will work closely with talented engineers and ML scientists to put your algorithms and models into practice. Your work will directly impact the trust customers place in Alexa, globally. You contribute directly to our growth by hiring smart and motivated Scientists to establish teams that can deliver swiftly and predictably, adjusting in an agile fashion to deliver what our customers need. A day in the life You will be working with a group of talented scientists on researching algorithm and running experiments to test scientific proposal/solutions to improve our sensitive contents detection and mitigation. This will involve collaboration with partner teams including engineering, PMs, data annotators, and other scientists to discuss data quality, policy, and model development. You will mentor other scientists, review and guide their work, help develop roadmaps for the team. You work closely with partner teams across Alexa to deliver platform features that require cross-team leadership. About the hiring group About the team The mission of the Alexa Sensitive Content Intelligence (ASCI) team is to (1) minimize negative surprises to customers caused by sensitive content, (2) detect and prevent potential brand-damaging interactions, and (3) build customer trust through appropriate interactions on sensitive topics. The term “sensitive content” includes within its scope a wide range of categories of content such as offensive content (e.g., hate speech, racist speech), profanity, content that is suitable only for certain age groups, politically polarizing content, and religiously polarizing content. The term “content” refers to any material that is exposed to customers by Alexa (including both 1P and 3P experiences) and includes text, speech, audio, and video. We are open to hiring candidates to work out of one of the following locations: Bangalore, KA, IND
US, WA, Seattle
Amazon Web Services (AWS) is building a world-class marketing organization, and we are looking for an experienced Applied Scientist to join the central data and science organization for AWS Marketing. You will lead AWS Measurement, targeting, recommendation, forecasting related AI/ML products and initiatives, and own mechanisms to raise the science and measurement standard. You will work with economists, scientists and engineers within the team, and partner with product and business teams across AWS Marketing to build the next generation marketing measurement, valuation and machine learning capabilities directly leading to improvements in our key performance metrics. A successful candidate has an entrepreneurial spirit and wants to make a big impact on AWS growth. You will develop strong working relationships and thrive in a collaborative team environment. You will work closely with business leaders, scientists, and engineers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed services. The ideal candidate will have experience with machine learning models and causal inference. Additionally, we are seeking candidates with strong rigor in applied sciences and engineering, creativity, curiosity, and great judgment. You will work on high-impact, high-visibility products, with your work improving the experience of AWS leads and customers. Key job responsibilities - Lead the design, development, deployment, and innovation of advanced science models in the strategic area of marketing measurement and optimization. - Partner with scientists, economists, engineers, and product leaders to break down complex business problems into science approaches. - Understand and mine the large amount of data, prototype and implement new learning algorithms and prediction techniques to improve long-term causal estimation approaches. - Design, build, and deploy effective and innovative ML solutions to improve components of our ML and causal inference pipelines. - Publish and present your work at internal and external scientific venues in the fields of ML and causal inference. - Influence long-term science initiatives and mentor other scientists across AWS. We are open to hiring candidates to work out of one of the following locations: Arlington, VA, USA | Austin, TX, USA | New York, NY, USA | Santa Clara, CA, USA | Seattle, WA, USA
US, WA, Bellevue
Where will Amazon's growth come from in the next year? What about over the next five? Which product lines are poised to quintuple in size? Are we investing enough in our infrastructure, or too much? How do our customers react to changes in prices, product selection, or delivery times? These are among the most important questions at Amazon today. The Topline Forecasting team in the Supply Chain Optimization Technologies (SCOT) group is looking for innovative, passionate and results-oriented Principal Economist to provide thought-leadership to help answer these questions. You will have an opportunity to own the long-run outlook for Amazon’s global consumer business and shape strategic decisions at the highest level. The successful candidate will be able to formalize problem definitions from ambiguous requirements, build econometric models using Amazon’s world-class data systems, and develop cutting-edge solutions for non-standard problems. Key job responsibilities • You understand the state-of-the-art in time series and econometric modeling. • You apply econometric tools and theory to solve business problems in a fast moving environment. • You excel at extracting insights and correct interpretations from data using advanced modeling techniques. • You communicate insights in a digestible manner to senior leaders in Finance and Operations within the company. • You are able to anticipate future business challenges and key questions, and have the ability to design modeling solutions to tackle them. • You have broad influence over the Topline team’s scientific research agenda and deliverables. • You contribute to the broader Econ research community in Amazon. • You advise other economists on scientific best-practices and raise the bar of research. • You will actively mentor other scientists and contribute to their career development. We are open to hiring candidates to work out of one of the following locations: Bellevue, WA, USA | New York, NY, USA