Publications

Randomized Control Trials (RCTs) are widely used across Amazon to causally estimate impacts of proposed feature changes, in order to make data-driven launch decisions. A key element of experimental design is the level of randomization, and the choice often relies on the cross-unit interaction structure. For instance, in the context of advertiser experiments, a treatment may affect the outcome of control

There are many experimental settings that may suffer from cross-unit (customers, seller, advertiser, etc.) spillovers, for instance through network effects. Such effects introduce bias and prevent the experimenter from drawing trustworthy insights on the data. One approach to dealing with such spillovers is to group units into clusters and randomize treatment status at the cluster level. Examples of clusters

Enabling a circular economy aims to reduce the amount of global waste generated from electrical and electronic equipment, mitigate the associated risk to the ecosystem and human health, and address concerns over limited material resources. Durability is a critical concern because keeping products in use for a longer time should reduce resource consumption and waste. Assessing the durability of products

Due to numerous applications in retail and (online) advertising the problem of assortment selection has been widely studied under many combinations of discrete choice models and feasibility constraints. In many situations, however, an assortment of products has to be constructed gradually and without accurate knowledge of all possible alternatives; in such cases, existing offline approaches become inapplicable

We introduce OpportunityFinder, a code-less framework for performing a variety of causal inference studies with panel data for non-expert users. In its current state, OpportunityFinder only requires users to provide raw observational data and a configuration file. A pipeline is then triggered that inspects/processes data, chooses the suitable algorithm(s) to execute the causal study. It returns the causal

Brands usually invest in a portfolio of digital ad products for brand consideration and conversion, and their performance is commonly evaluated with ad - attributed metrics. However, these metrics limit the measurement of advertising effectiveness within a short time window, typically of two weeks. Therefore, they could underestimate the total effect if some ad products' efficacy lasts beyond the measurement

Motivated by bid recommendation in online ad auctions, this paper considers a general class of multi-level and multi-agent games, with two major characteristics: one is a large number of anonymous agents, and the other is the intricate interplay between competition and cooperation. To model such complex systems, we propose a novel and tractable bi-objective optimization formulation with mean-field approximation

This paper proposes a learning model of online ad auctions that allows for the following four key realistic characteristics of contemporary online auctions: (1) ad slots can have different values and click-through rates depending on users’ search queries, (2) the number and identity of competing advertisers are unobserved and change with each auction, (3) advertisers only receive partial, aggregated feedback

Companies offering web services routinely run randomized online experiments to estimate the “causal impact” associated with the adoption of new features and policies on key performance metrics of interest. These experiments are used to estimate a variety of effects: the increase in click rate due to the repositioning of a banner, the impact on subscription rate as a consequence of a discount or special

Causal Impact (CI) measurement is broadly used across the industry to inform both short- and long-term investment decisions of various types. In this paper, we apply the double machine learning (DML) methodology to estimate average and conditional average treatment effects across 100s of customer action types for e-commerce and digital businesses and 100s of millions of customers that can be used in decisions

The majority of students who choose to major in engineering do so to become a part of the community of practice of professional engineers (Johri & Olds, 2011), meaning that they want to have adequate exposure to what a career as a professional engineer could potentially be as part of their college experience. However, according to Jonassen (2014), engineering graduates are not well trained to contribute

Network interference, where observed outcomes are influenced by interaction with nearby units, is a fundamental issue in A/B testing and experimentation in social and economic networks. Clustered randomization is a frequently-used strategy that aims to prevent confounding by limiting interaction between treated and untreated units. We study a model of least-squares estimation under network interference,

Computer science (CS) is special among STEM subjects: it aims at an industry sector that has the most job growth but has a constant shortage in the workforce; it is a relatively young and burgeoning subject in K-12 education that has a shortage of classroom teachers; and it is one of a very few STEM subjects that large number of students can master by learning it completely out-of-school. To inspire the

Classical Randomized Controlled Trials (RCTs), or A/B tests, are designed to draw causal inferences about a population of units, for example, individuals, plots of land or visits to a website. A key assumption underlying a standard RCT is the absence of interactions between units, or the stable unit treatment value assumption (Ann. Statist. 6 (1978) 34-58). Modem experimentation, however, is often conducted

In order to identify expertise, forecasters should not be tested by their calibration score, which can always be made arbitrarily small, but rather by their Brier score. The Brier score is the sum of the calibration score and the refinement score; the latter measures how good the sorting into bins with the same forecast is, and thus attests to “expertise.” This raises the question of whether one can gain