Chapter 20 Overlap in Observational Studies: Difficulties and Opportunities

100 Implications of overlap

In Part III of this book, causal inference with observational studies relies on two critical assumptions: ignorability

Z\ind \{Y(1),Y(0)\}\mid X

and overlap

0<e(X)<1.

D’Amour et al., (2021) pointed out the tension between these two assumptions: typically, more covariates make the ignorability assumption more plausible (ignoring M-bias discussed in Chapter 84.1), but more covariates make the overlap assumption less plausible because the treatment becomes more predictable given more covariates.

If some units have e(X)=0 or e(X)=1, then we have philosophic difficulty in thinking about the counterfactual potential outcomes (King and Zeng,, 2006). In particular, if a unit deterministically receives the treatment, then it may not be meaningful to conceive its potential outcome under the control; if a unit deterministically receives the control, then it may not be meaningful to conceive its potential outcome under the treatment. Even if the true propensity score is not exactly 0 or 1, the estimated propensity score can be very close to 0 or 1 in finite samples, which makes the estimators based on IPW numerically unstable. I have discussed this issue in Chapter 11.

Many statistical analyses in fact require a strict version of overlap:

Assumption 20.1 (strict overlap)

\eta\leq e(X)\leq 1-\eta for some \eta\in(0,1/2).

However, D’Amour et al., (2021, Corollary 1) showed that Assumption 20.1 has very strong implications when the number of covariates is large. For simplicity, I present only one of their results. Let X_{k}\ (k=1,\ldots,p) be the kth component of the covariate X=(X_{1},\ldots,X_{p}), and

e=\textup{pr}(Z=1)

be the proportion of the treated units.

Theorem 20.1

Assumption 20.1 implies that \eta\leq e\leq 1-\eta and

\displaystyle p^{-1}\sum_{k=1}^{p}\big|E(X_{k}\mid Z=1)-E(X_{k}\mid Z=0)\Big| (92)
\displaystyle\leq \displaystyle p^{-1/2}C^{1/2}\left\{e\lambda_{1}^{1/2}+(1-e)\lambda_{0}^{1/2}\right\},

where

C=\frac{(e-\eta)(1-e-\eta)}{e^{2}(1-e)^{2}\eta(1-\eta)}

is a positive constant depending only on (e,\eta), and \lambda_{1} and \lambda_{0} are the maximum eigenvalues of the covariance matrices \textup{cov}(X\mid Z=1) and \textup{cov}(X\mid Z=0), respectively.

What is the order of the maximum eigenvalues in Theorem 20.1? D’Amour et al., (2021) showed that it is usually smaller than O(p) unless the components of X are highly correlated. If the components of X are highly correlated, then some components are redundant after including other components. If the components of X are not highly correlated, then the right-hand side converges to zero. So the average difference in means of the covariates is close to zero, that is, the treatment and control groups are nearly balanced in means averaged over all dimensions of the covariates. Mathematically, the left-hand side of (92) converging to zero rules out the possibility that all dimensions of X have non-vanishing differences in means across the treatment and control groups. It is a strong requirement in observational studies with many covariates.

100.1 Trimming in the presence of limited overlap

When Assumption 20.1 does not hold, it is common to trim the units based on the estimated propensity scores (Crump et al.,, 2009; Yang and Ding, 2018b, ). Trimming drops units within regions of little overlap, which changes the population and estimand. The restrictive implications of overlap in Assumption 20.1 suggest that trimming must be employed more often and one may need to trim a large proportion of units to achieve desirable overlap in high dimensions.

100.2 Outcome modeling in the presence of limited overlap

The somewhat negative results in D’Amour et al., (2021) also highlight the limitation of focusing only on the propensity score in the presence of limited overlap. This in some sense challenges Rubin, (2008)’s view that “for objective causal inference, design trumps analysis.” Rubin, (2008) argued strongly for the role of the “design” stage in observational studies. The “design” stage focuses on the propensity score which may not satisfy the overlap condition with many covariates.

With high dimensional covariates, outcome modeling becomes more important. In particular, if the outcome means only depend on a function of the original covariates in that

E\{Y(z)\mid X\}=f_{z}(r(X)),\quad(z=0,1)

then it suffices to control for r(X), a lower-dimensional summary of the original covariates. See Problem 20.1 for more details. Due to the dimension reduction, the strict overlap condition on r(X) can be much weaker than the strict overlap condition on X. This is conceptually straightforward, but the corresponding theory and methods are missing.

101 Causal inference with no overlap: regression discontinuity

Let us start from the simple case with a univariate X. An extreme treatment assignment mechanism is a deterministic one:

Z=I(X\geq x_{0}),

where x_{0} is a predetermined threshold. An interesting consequence of this assignment is that the ignorability assumption holds automatically:

Z\ind \{Y(1),Y(0)\}\mid X

because Z is a deterministic function of X and a constant is independent of any random variables. However, the overlap assumption is violated by definition:

e(X)=\textup{pr}(Z=1\mid X)=1(X\geq x_{0})=\begin{cases}1&\text{ if }X\geq x_{0},\\
0&\text{ if }X<x_{0}.\end{cases}

So our analytic strategies discussed in Part IV are no longer applicable here. We must change our perspective.

The discussion above seems contrived, with a deterministic treatment assignment. Interestingly, it has many applications in practice and is called regression discontinuity. Below, I first review some canonical examples and then give a mathematical formulation of this type of study.

101.1 Examples and graphical diagnostics
Example 20.1

Thistlethwaite and Campbell, (1960) first proposed the idea of regression-discontinuity analysis. Their motivating example was to study the effect of students’ winning the Certificate of Merit on later career plans, where the Certificate of Merit was determined by whether the Scholarship Qualifying Test score was above a certain threshold. Their initial analysis was mainly graphical. Figure 26 shows one of their graphs.

Refer to caption
Figure 26: A graph from Thistlethwaite and Campbell, (1960) with minor modifications of the unclear text in the original paper
Example 20.2

Bor et al., (2014) used regression discontinuity to study the effect of when to start HIV patients with antiretroviral on their mortality, where the treatment is determined by whether the patients’ CD4 counts were below 200 cells/\muL (note that CD4 cells are white blood cells that fight infection.)

Example 20.3

Carpenter and Dobkin, (2009) studied the effect of alcohol consumption on mortality, which leverages the minimum legal drinking age as a discontinuity for alcohol consumption. They derived mortality data from the National Center for Health Statistics, including the decedent’s date of birth and date of death. They computed the age profile of deaths per 100,000 person-years with outcomes measured by the following nine variables: all all deaths, the sum of internal and external internal deaths due to internal causes external deaths due to external causes, the sum of the rest homicide homicides suicide suicides mva motor vehicle accidents alcohol deaths with a mention of alcohol drugs deaths with a mention of drug use externalother deaths due to other external causes

Figure 27 plots the number of deaths per 100,000 person-years for nine measures based on the data used by Angrist and Pischke, (2014). From the jumps at age 21, it seems obvious that there is an increase in mortality at age 21, primarily due to motor vehicle accidents. I leave the formal analysis as Problem 20.4.

插图
Figure 27: Minimum legal drinking age example from Carpenter and Dobkin, (2009)
101.2 A mathematical formulation of regression discontinuity

The technical term for the variable X that determines the treatment is the running variable. Intuitively, regression discontinuity can identify a local average causal effect at the cutoff point x_{0}:

\tau(x_{0})=E\{Y(1)-Y(0)\mid X=x_{0}\}.

In particular, for the potential outcome under treatment, we have

\displaystyle E\{Y(1)\mid X=x_{0}\} \displaystyle= \displaystyle\lim_{\varepsilon\rightarrow 0+}E\{Y(1)\mid X=x_{0}+\varepsilon\} (93)
\displaystyle= \displaystyle\lim_{\varepsilon\rightarrow 0+}E\{Y(1)\mid Z=1,X=x_{0}+\varepsilon\} (94)
\displaystyle= \displaystyle\lim_{\varepsilon\rightarrow 0+}E(Y\mid Z=1,X=x_{0}+\varepsilon), (95)

where (93) holds if E\{Y(1)\mid X=x\} is continuous from the right at x_{0} and (94) follows by the definition of Z. Similarly, for the potential outcome under control, we have

E\{Y(0)\mid X=x_{0}\}=\lim_{\varepsilon\rightarrow 0+}E(Y\mid Z=0,X=x_{0}-\varepsilon)

if E\{Y(0)\mid X=x\} is continuous from the left at x_{0}. So the local average causal effect at x_{0} can be identified by the difference of the two limits. I summarize the key identification result below.

Theorem 20.2

Assume that the treatment is determined by Z=I(X\geq x_{0}) where x_{0} is a predetermined threshold. Assume that E\{Y(1)\mid X=x\} is continuous from the right at x_{0} and E\{Y(0)\mid X=x\} is continuous from the left at x_{0}. Then the local average treatment effect at X=x_{0} is identified by

\tau(x_{0})=\lim_{\varepsilon\rightarrow 0+}E(Y\mid Z=1,X=x_{0}+\varepsilon)-\lim_{\varepsilon\rightarrow 0+}E(Y\mid Z=0,X=x_{0}-\varepsilon).

Since the right-hand side of the above equation only involves observables, the parameter \tau(x_{0}) is nonparametrically identified. However, the form of the identification formula is totally different from what we derived before. In particular, the identification formula involves limits of two conditional expectation functions.

101.3 Regressions near the boundary

If we are lucky, graphical diagnostics sometimes can clearly show the causal effect at the cutoff point. However, many outcomes are noisy so graphical diagnostics are not enough in finite samples. Figure 28 shows two simulated examples with obvious jumps at the cutoff point and two examples without obvious jumps, although the underlying data-generating processes all have discontinuities.

插图
Figure 28: Examples of regression discontinuity. The black points are the observed outcomes whereas the grey points are the unobserved counterfactual outcomes. In the first column, the data-generating processes result in visible jumps at the cutoff points; in the second column, the jumps are not so visible. In the first row, the data generating processes have constant \tau(x); in the second row, \tau(x) varies with x.

Assume that E(Y\mid Z=1,X=x)=\gamma_{1}+\beta_{1}x and E(Y\mid Z=0,X=x)=\gamma_{0}+\beta_{0}x are linear in x. We can run OLS based on the treated and control data to obtain the fitted lines \hat{\gamma}_{1}+\hat{\beta}_{1}x and \hat{\gamma}_{0}+\hat{\beta}_{0}x, respectively. We can then estimate the average causal effect at the point X=x_{0} as

\hat{\tau}(x_{0})=(\hat{\gamma}_{1}-\hat{\gamma}_{0})+(\hat{\beta}_{1}-\hat{\beta}_{0})x_{0}.

Numerically, \hat{\tau}(x_{0}) is identical to the coefficient of Z_{i} in the OLS

\displaystyle Y_{i}\sim\{1,Z_{i},X_{i}-x_{0},Z_{i}(X_{i}-x_{0})\}, (96)

and it is also identical to the coefficient of Z_{i} in the OLS

\displaystyle Y_{i}\sim\{1,Z_{i},R_{i},L_{i}\}, (97)

where

R_{i}=\max(X_{i}-x_{0},0),\qquad L_{i}=\min(X_{i}-x_{0},0)

indicate the right and left parts of X_{i}-x_{0}, respectively. I leave the algebraic details to Problem 20.2.

However, this approach may be sensitive to the violation of the linear model assumption. Theory suggests running regression using only the local observations near the cutoff point43. However, the rules for choosing the “local points” are quite involved. Fortunately, the rdrobust function in the rdrobust package in R implements various choices of “local points.” Since choosing the “local points” is the key to regression discontinuity, it seems more sensible to report estimates and confidence intervals based on various choices of the “local points.” This can be viewed as a sensitivity analysis for regression discontinuity.

101.4 An example

Lee, (2008) gave a famous example of using regression discontinuity to study the incumbency advantage in the U.S. House. He wrote that “incumbents are, by definition, those politicians who were successful in the previous election. If what makes them successful is somewhat persistent over time, they should be expected to be somewhat more successful when running for re-election.” Therefore, this is a fundamentally challenging causal inference problem. The regression discontinuity is a clever study design to study this problem.

The running variable is the lagged vote in the previous election centered at 0, and the outcome is the vote in the current election, with units being the congressional districts. The treatment is the binary indicator for being the current incumbent party in a district, determined by the lagged vote. The following R code generates Figure 29 that shows the raw data.

house = read.csv("house.csv")[, -1]
plot(y ~ x, data = house, pch = 19, cex = 0.1)
abline(v = 0, col = "grey")
插图
Figure 29: Raw data of Lee, (2008)

The rdrobust function gives three sets of the point estimate and confidence intervals. They all suggest a positive incumbency advantage.

> library(rdrobust)
> RDDest = rdrobust(house$y, house$x)
Warning message:
In rdrobust(house$y, house$x) :
Mass points detected in the running variable.
> cbind(RDDest$coef, RDDest$ci)
Coeff CI Lower CI Upper
Conventional 0.06372533 0.04224798 0.08520269
Bias-Corrected 0.05937028 0.03789292 0.08084763
Robust 0.05937028 0.03481238 0.08392818

We can also obtain the point estimates and the confidence intervals based on OLS with different choices of the local points defined by |X|<h based on the following R code.

house$z = (house$x >= 0)
hh = seq(0.05, 1, 0.01)
local.lm = sapply(hh, function(h){
Greg = lm(y ~ z + x + z*x, data = house,
subset = (abs(x)<=h))
cbind(coef(Greg)[2], confint(Greg, ’zTRUE’))
})
plot(local.lm[1, ] ~ hh, type = "p",
pch = 19, cex = 0.3,
ylim = range(local.lm),
xlab = "h",
ylab = "point and interval estimates",
main = "subset linear regression: |X|<h")
lines(local.lm[2, ] ~ hh, type = "p",
pch = 19, cex = 0.1)
lines(local.lm[3, ] ~ hh, type = "p",
pch = 19, cex = 0.1)

Figure 30 shows the point estimates and the confidence intervals as a function of h. While the point estimates and the confidence intervals are sensitive to the choice of h, the qualitative result remains the same as above.

插图
Figure 30: Estimates and confidence intervals based on local linear regressions
101.5 Problems of regression discontinuity

What can go wrong with the regression discontinuity analysis? The technical challenge is to specify the neighborhood near the cutoff point. We have discussed this issue above.

In addition, Theorem 20.2 holds under a continuity condition. It may be violated in practice. For instance, if the mortality rate jumps at the age of 21, then we may not be able to attribute the jumps in Figure 27 to the change in drinking behavior due to the legal drinking age. However, it is hard to check the violation of the continuity condition empirically.

McCrary, (2008) proposed an indirect test for the validity of the regression discontinuity. He suggested checking the density of the running variable at the cutoff point. The discontinuity in the density of the running variable at the cutoff point may suggest that some units were able to manipulate their treatment status perfectly. The DCdensity function in the R package rdd implements this test. I omit the details.

102 Homework Problems

20.1 A theorem for the role of outcome modeling in the presence of limited overlap

This problem extends the discussion in Section 100.2. I state a formal theorem below.

Theorem 20.3

Assume

Z\ind Y(z)\mid X,\quad(z=0,1)

and

E\{Y(z)\mid X\}=f_{z}(r(X)),\quad(z=0,1)

for some function r(X). The average causal effect \tau=E\{Y(1)-Y(0)\} can be identified by

\tau=E\{E(Y\mid Z=1,r(X))-E(Y\mid Z=0,r(X))\}

or

\tau=E\left\{\frac{ZY}{e(r(X))}\right\}-E\left\{\frac{(1-Z)Y}{1-e(r(X))}\right\}

where e(r(X))=\textup{pr}\{Z=1\mid r(X)\}.

Prove Theorem 20.3.

Remark: When r(X)=X, Theorem 20.3 reduces to the standard IPW formula for the average causal effect. When r(X) has a lower dimension than X, Theorem 20.3 has broader applicability if the overlap condition on e(X) fails whereas the overlap condition on e(r(X)) may still hold.

20.2 Linear potential outcome models

This problem gives more details for the numerical equivalence in Section 101.3.

Show that \hat{\tau}(x_{0}) equals the coefficients of Z_{i} in OLS fits (96) and (96).

Remark: It is helpful to start with the figures of Z_{i}(X_{i}-x_{0}), L_{i}, and R_{i} with X_{i} on the x-axis. The conclusion holds by a reparametrization of the OLS regressions.

20.3 Simulation for regression discontinuity

In Figure 28, the potential outcomes are simulated from linear models. Change them to nonlinear models, and compare different point estimators and confidence intervals, including the biases and variances of the point estimators, and the coverage properties of confidence intervals.

20.4 Re-analysis of the data on the minimum legal drinking age

Figure 27 shows the jumps at the cutoff point. Analyze the data mlda.csv of Carpenter and Dobkin, (2009).

20.5 Re-analysis of Lee, (2008)’s data

Figure 30 plots the confidence intervals based on the standard errors assuming homoskedasticity. Generate a figure with confidence intervals based on the EHW standard errors in OLS.

20.6 Recommended reading

D’Amour et al., (2021) discussed the implications of overlap with high dimensional covariates.

Thistlethwaite and Campbell, (1960)’s original paper on regression discontinuity was re-printed as Thistlewaite and Campbell, (2016) with many insightful comments. Coincidentally, Thistlethwaite and Campbell, (1960) and Rubin, (1974) were both published in the Journal of Educational Psychology.

脚注

43. This is called local linear regression in nonparametric statistics, which belongs to a broader class of local polynomial regression (Fan and Gijbels,, 1996).

本书:A First Course in Causal Inference,作者 Peng Ding。原文:arXiv:2305.18793v2,按 CC BY 4.0 许可发布。本站(yuanens.com)把 arXiv 网页版改成分章的纯 HTML,把公式渲染成图片,方便在 Kindle 等电子书阅读器上看,内容没有改动。转换难免有错,以原文为准。本站与作者和出版社无关。