Learning the Covariance of Treatment Effects Across Many Weak Experiments
arxiv(2024)
摘要
When primary objectives are insensitive or delayed, experimenters may instead
focus on proxy metrics derived from secondary outcomes. For example, technology
companies often infer long-term impacts of product interventions from their
effects on weighted indices of short-term user engagement signals. We consider
meta-analysis of many historical experiments to learn the covariance of
treatment effects on different outcomes, which can support the construction of
such proxies. Even when experiments are plentiful and large, if treatment
effects are weak, the sample covariance of estimated treatment effects across
experiments can be highly biased and remains inconsistent even as more
experiments are considered. We overcome this by using techniques inspired by
weak instrumental variable analysis, which we show can reliably estimate
parameters of interest, even without a structural model. We show the Limited
Information Maximum Likelihood (LIML) estimator learns a parameter that is
equivalent to fitting total least squares to a transformation of the
scatterplot of estimated treatment effects, and that Jackknife Instrumental
Variables Estimation (JIVE) learns another parameter that can be computed from
the average of Jackknifed covariance matrices across experiments. We also
present a total-covariance-based estimator for the latter estimand under
homoskedasticity, which we show is equivalent to a k-class estimator. We show
how these parameters relate to causal quantities and can be used to construct
unbiased proxy metrics under a structural model with both direct and indirect
effects subject to the INstrument Strength Independent of Direct Effect
(INSIDE) assumption of Mendelian randomization. Lastly, we discuss the
application of our methods at Netflix.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要