Chapter 6 Rerandomization and Regression Adjustment

Stratification and post-stratification in Chapter 5 are duals for discrete covariates in the design and analysis of randomized experiments. How should we deal with multidimensional, possibly continuous, covariates? We can discretize continuous covariates, but this is not an ideal strategy with many covariates. Rerandomization and regression adjustment are duals for general covariates, which are the topics for this chapter.

The following table summarizes the topics of Chapters 5 and 6:

design analysis
discrete covariate stratification post-stratification
general covariate rerandomization regression adjustment

29 Rerandomization

29.1 Experimental design

Again we consider a finite population of n experimental units, where n_{1} of them receive the treatment and n_{0} of them receive the control. Let \bm{Z}=(Z_{1},\ldots,Z_{n}) be the treatment vector for these units. Unit i has covariate X_{i}\in\mathbb{R}^{K} which can have continuous or binary components. Concatenate them as an n\times K covariate matrix \bm{X}=(X_{1},\ldots,X_{n})^{\mkern-1.5mu\mathsf{T}} and center them at mean zero \bar{X}=n^{-1}\sum_{i=1}^{n}X_{i}=0 to simplify the presentation.

The CRE balances the covariates in the treatment and control groups on average, for instance, the difference in means of the covariates

\hat{\tau}_{X}=n_{1}^{-1}\sum_{i=1}^{n}Z_{i}X_{i}-n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})X_{i}

has mean zero under the CRE. However, it can result in undesired covariate balance across the treatment and control groups in the realized treatment allocation, that is, the realized value of \hat{\tau}_{X} is often not zero. Using the vector form of Neyman, (1923) in Problem 4.6 before, we can show that

\textup{cov}(\hat{\tau}_{X})=\frac{1}{n_{1}}S_{X}^{2}+\frac{1}{n_{0}}S_{X}^{2}=\frac{n}{n_{1}n_{0}}S_{X}^{2},

where S_{X}^{2}=(n-1)^{-1}\sum_{i=1}^{n}X_{i}X_{i}^{\mkern-1.5mu\mathsf{T}} is the finite-population covariance matrix of the covariates. The following Mahalanobis distance measures the difference between the treatment and control groups:

M=\hat{\tau}_{X}^{\mkern-1.5mu\mathsf{T}}\textup{cov}(\hat{\tau}_{X})^{-1}\hat{\tau}_{X}=\hat{\tau}_{X}^{\mkern-1.5mu\mathsf{T}}\left(\frac{n}{n_{1}n_{0}}S_{X}^{2}\right)^{-1}\hat{\tau}_{X}. (26)

Technically, the above formula for M in (26) is meaningful only if S_{X}^{2} is invertible, which means that the columns of the covariate matrix \bm{X} are linearly independent. If a column can be represented by a linear combination of other columns, it is redundant and should be dropped before the experiment. A nice feature of M is that it is invariant under non-degenerate linear transformations of X. Lemma 6.1 below summarizes the result with the proof relegated to Problem 6.2.

Lemma 6.1

M in (26) remains the same if we transform X_{i} to b_{0}+BX_{i} for all units i=1,\ldots,n where b_{0}\in\mathbb{R}^{K} and B\in\mathbb{R}^{K\times K} is invertible.

The finite population CLT (Li and Ding,, 2017) ensures that with large n, the Mahalanobis distance M is approximately \chi^{2}_{K} under the CRE. Therefore, it is likely that M has a large realized value under the CRE with asymptotic mean K and variance 2K. Rerandomization avoids covariate imbalance by discarding the treatment allocations with large values of M. Below I give a formal definition of the rerandomization using the Mahalanobis distance (ReM), which was proposed by Cox, (1982) and further studied by Morgan and Rubin, (2012).

Definition 6.1 (ReM)

Draw \bm{Z} from CRE and accept it if and only if

M\leq a,

for some predetermined constant a>0.

The problem of choosing a is similar to the problem of choosing the number of strata in the SRE, which is non-trivial in practice. At one extreme, a=\infty, we just conduct the CRE. At the other extreme, a=0, there are very few feasible treatment allocations, and consequently, the experiment has little randomness, rendering randomization-based inference useless. As a compromise, we choose a small but not extremely small a, for example, a=0.001 or some upper quantile of a \chi^{2}_{K} distribution.

ReM has many desirable properties. As mentioned above, it is invariant to linear transformations of the covariates. Moreover, it has nice geometric properties and elegant mathematical theory. This chapter will focus on ReM.

29.2 Statistical inference

An important question is how to analyze the data under ReM. Bruhn and McKenzie, (2009) and Morgan and Rubin, (2012) argued that we can always use the FRT as long as we simulate \bm{Z} under the constraint M\leq a. This always yields finite-sample exact p-values under the sharp null hypothesis. See Problem 6.1.

It is a challenging problem to derive the finite sample properties of ReM without assuming the sharp null hypothesis. Instead, Li et al., 2018b () derived the asymptotic distribution of the difference in means of the outcome \hat{\tau} under ReM and the regularity conditions below.

Condition 6.1

As n\rightarrow\infty,

  1. 1.

    n_{1}/n and n_{0}/n have positive limits;

  2. 2.

    the finite population covariance of \{X_{i},Y_{i}(1),Y_{i}(0),\tau_{i}\} has a finite limit;

  3. 3.

    \max_{1\leq i\leq n}\{Y_{i}(1)-\bar{Y}(1)\}^{2}/n\rightarrow 0, \max_{1\leq i\leq n}\{Y_{i}(0)-\bar{Y}(0)\}^{2}/n\rightarrow 0, and \max_{1\leq i\leq n}X_{i}^{\mkern-1.5mu\mathsf{T}}X_{i}/n\rightarrow 0,

Below is the main theorem for ReM, which relies on additional notation. Let

L_{K,a}\sim D_{1}\mid\bm{D}^{\mkern-1.5mu\mathsf{T}}\bm{D}\leq a

where \bm{D}=(D_{1},\ldots,D_{K}) follows a K-dimensional standard Normal distribution; let \varepsilon follow a univariate standard Normal distribution; L_{K,a}\ind \varepsilon.

Theorem 6.1

Under ReM with M\leq a and Condition 6.1, we have18

\hat{\tau}-\tau\stackrel{{\scriptstyle\cdot}}{{\sim}}\sqrt{\textup{var}(\hat{\tau})}\left\{\sqrt{R^{2}}L_{K,a}+\sqrt{1-R^{2}}\varepsilon\right\},

where

\textup{var}(\hat{\tau})=\frac{S^{2}(1)}{n_{1}}+\frac{S^{2}(0)}{n_{0}}-\frac{S^{2}(\tau)}{n}

is Neyman, (1923)’s variance formula proved in Chapter 4, and

R^{2}=\textup{corr}^{2}(\hat{\tau},\hat{\tau}_{X})

is the squared multiple correlation coefficient (see Section 30.A.1 for the definition) between \hat{\tau} and \hat{\tau}_{X} under the CRE.

Although the proof of Li et al., 2018b () is technical, the asymptotic distribution in Theorem 6.1 has a clear geometric interpretation, as shown in Figure 12. It shows that \hat{\tau} decomposes into a component that is a linear combination of \hat{\tau}_{X} and a component that is orthogonal to \hat{\tau}_{X}. Geometrically, \cos^{2}\theta=R^{2}, where \theta is the angle between \hat{\tau} and \hat{\tau}_{X}. ReM affects the first component but does not change the second component. The truncated Normal distribution L_{K,a} is due to the restriction of ReM on the first component.

Refer to caption
Figure 12: Geometry of the asymptotic distribution of \hat{\tau} under ReM

When a=\infty, the asymptotic distribution simplifies to the one under the CRE:

\hat{\tau}-\tau\stackrel{{\scriptstyle\cdot}}{{\sim}}\sqrt{\textup{var}(\hat{\tau})}\varepsilon.

When the threshold a is close to zero, the asymptotic distribution simplifies to

\hat{\tau}-\tau\stackrel{{\scriptstyle\cdot}}{{\sim}}\sqrt{\textup{var}(\hat{\tau})(1-R^{2})}\varepsilon;

see Wang and Li, (2022) for a rigorous proof. So with a small threshold a, the efficiency gain due to ReM depends on R^{2}, which has the following equivalent form.

Proposition 6.1

Under the CRE, we have

R^{2}=\textup{corr}^{2}(\hat{\tau},\hat{\tau}_{X})=\frac{n_{1}^{-1}S^{2}(1\mid X)+n_{0}^{-1}S^{2}(0\mid X)-n^{-1}S^{2}(\tau\mid X)}{n_{1}^{-1}S^{2}(1)+n_{0}^{-1}S^{2}(0)-n^{-1}S^{2}(\tau)},

where \{S^{2}(1),S^{2}(0),S^{2}(\tau)\} are the finite population variances of \{Y_{i}(1),Y_{i}(0),\tau_{i}\}_{i=1}^{n}, and \{S^{2}(1\mid x),S^{2}(0\mid x),S^{2}(\tau\mid x)\} are the corresponding finite population variances of their linear projections on (1,X_{i}); see Section 31.B for the definition of linear projections. Under the constant causal effect assumption with \tau_{i}=\tau, the R^{2} reduces to S^{2}(0\mid X)/S^{2}(0), the finite population squared multiple correlation between Y_{i}(0) and X_{i}.

I leave the proof of Proposition 6.1 to Problem 6.4.

When 0<a<\infty, the asymptotic distribution of \hat{\tau} has a more complicated form and is more concentrated at \tau and thus the difference in means \hat{\tau} is more precise under ReM than under the CRE.

If we ignore the design of ReM and still use the confidence interval based on Neyman, (1923)’s variance formula and the Normal approximation, it is overly conservative and overcovers \tau even if the individual causal effects are constant. Li et al., 2018b () proposed to construct confidence intervals based on Theorem 6.1. We omit the discussion here but will come back to the inference issue in Section 31.

30 Regression adjustment

What if we do not conduct rerandomization in the design stage but want to adjust for covariate imbalance in the analysis stage of the CRE? We will discuss several regression adjustment strategies.

30.1 Covariate-adjusted FRT

The covariates \bm{X} are all fixed, and furthermore, under H_{0\textsc{f}}, the observed outcomes are all fixed. Therefore, we can simulate the distribution of any test statistic T=T(\bm{Z},\bm{Y},\bm{X}) and calculate the p-value. The basic idea of the FRT remains the same in the presence of additional covariates.

There are two general strategies to construct the test statistic. Problem 3.6 before hints at both of them. I summarize them below, using the terminology from Zhao and Ding, 2021a ().

Definition 6.2 (pseudo-outcome strategy for covariate-adjusted FRT)

We can construct the test statistic based on residuals from fitted statistical models. We can regress Y_{i} on X_{i} to obtain residual \hat{\varepsilon}_{i}, and then treat \hat{\varepsilon}_{i} as the pseudo outcome to construct test statistics.

Definition 6.3 (model-output strategy for covariate-adjusted FRT)

We can use a regression coefficient as a test statistic. We can regress Y_{i} on (Z_{i},X_{i}) to obtain the coefficient of Z_{i} as the test statistic.

In the pseudo-outcome strategy in Definition 6.2, the regression of Y_{i} on X_{i} should not include the treatment Z_{i} because we want to ensure that the pseudo outcome satisfies H_{0\textsc{f}} if the original outcome satisfies H_{0\textsc{f}}. In the model-output strategy in Definition 6.3, the regression of Y_{i} on (Z_{i},X_{i}) includes the treatment Z_{i} because we want to use the coefficient of Z_{i} to measure the deviation from H_{0\textsc{f}}. Computationally, in strategy one, we only need to run regression once, whereas in strategy two, we need to run regression many times.

In the above, “regression” is a generic term, which can be linear regression, logistic regression, or even machine learning algorithms. The FRT with any test statistics from these two strategies will be finite-sample exact under H_{0\textsc{f}} although they differ under alternative hypotheses. The rest of this section will review some test statistics based on OLS.

30.2 Analysis of covariance and extensions

Now we turn to direct estimation of the average causal effect \tau that adjusts for the observed covariates.

Historically, Fisher, (1925) proposed to use the analysis of covariance (ANCOVA) to improve estimation efficiency. This remains a standard strategy in many fields. He suggested running the OLS of Y_{i} on (1,Z_{i},X_{i}) and obtaining the coefficient of Z_{i} as an estimator for \tau. Let \hat{\tau}_{\textsc{F}} denote Fisher’s ANCOVA estimator.

A former Berkeley Statistics Professor, David A. Freedman, reanalyzed Fisher’s ANCOVA under Neyman, (1923)’s potential outcomes framework. Freedman, 2008a (); Freedman, 2008b () found the following negative results:

  1. [(1)]

  2. 1.

    \hat{\tau}_{\textsc{F}} is biased, but the simple difference in means \hat{\tau} is unbiased.

  3. 2.

    The asymptotic variance of \hat{\tau}_{\textsc{F}} may be even larger than that of \hat{\tau} when n_{1}\neq n_{0}.

  4. 3.

    The standard error from the OLS is inconsistent for the true standard error of \hat{\tau}_{\textsc{F}} under the CRE.

A former Berkeley Ph.D. student, Winston Lin, wrote a thesis in response to Freedman’s critiques. Lin, (2013) found the following positive results:

  1. [(1)]

  2. 1.

    The bias of \hat{\tau}_{\textsc{F}} is small in large samples, and it goes to zero as the sample size approaches infinity.

  3. 2.

    We can improve the asymptotic efficiency of both \hat{\tau} and \hat{\tau}_{\textsc{F}} by using the coefficient of Z_{i} in the OLS of Y_{i} on (1,Z_{i},X_{i},Z_{i}\times X_{i}). Let \hat{\tau}_{\textsc{L}} denote Lin, (2013)’s estimator. Moreover, the EHW standard error is a conservative estimator for the true standard error of \hat{\tau}_{\textsc{L}} under the CRE.

  4. 3.

    The EHW standard error19 for \hat{\tau}_{\textsc{F}} in the OLS fit of Y_{i} on (1,Z_{i},X_{i}) is a conservative estimator for the true standard error of \hat{\tau}_{\textsc{F}} under the CRE.

30.2.1 Some heuristics for Lin, (2013)’s results

Neyman, (1923)’s result demonstrates that the variance of the difference-in-means estimator \hat{\tau} depends on the variances of the potential outcomes. Intuitively, we can reduce the variance of the estimator by reducing the variances of the potential outcomes. A family of linearly adjusted estimators is

\displaystyle\hat{\tau}(\beta_{1},\beta_{0}) \displaystyle= \displaystyle n_{1}^{-1}\sum_{i=1}^{n}Z_{i}(Y_{i}-\beta_{1}^{\mkern-1.5mu\mathsf{T}}X_{i})-n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-\beta_{0}^{\mkern-1.5mu\mathsf{T}}X_{i}) (27)
\displaystyle= \displaystyle\left\{\hat{\bar{Y}}(1)-\beta_{1}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(1)\right\}-\left\{\hat{\bar{Y}}(0)-\beta_{0}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(0)\right\}, (28)

where \{\hat{\bar{Y}}(1),\hat{\bar{Y}}(0)\} are the sample means of the outcomes, and \{\hat{\bar{X}}(1),\hat{\bar{X}}(0)\} are the sample means of the covariates. This covariate-adjusted estimator \hat{\tau}(\beta_{1},\beta_{0}) tries to reduce the variance of \hat{\tau} by residualizing the potential outcomes. It reduces to \hat{\tau} with \beta_{1}=\beta_{0}=0. It has mean \tau for any fixed values of \beta_{1} and \beta_{0} because \bar{X}=0. We are interested in finding the (\beta_{1},\beta_{0}) that minimize the variance of \hat{\tau}(\beta_{1},\beta_{0}). This estimator is essentially the difference in means of the adjusted potential outcomes \{Y_{i}(1)-\beta_{1}^{\mkern-1.5mu\mathsf{T}}X_{i},Y_{i}(0)-\beta_{0}^{\mkern-1.5mu\mathsf{T}}X_{i}\}_{i=1}^{n}. Applying Neyman, (1923)’s result, this estimator has variance

\textup{var}\{\hat{\tau}(\beta_{1},\beta_{0})\}=\frac{{S}^{2}(1;\beta_{1})}{n_{1}}+\frac{{S}^{2}(0;\beta_{0})}{n_{0}}-\frac{S^{2}(\tau;\beta_{1},\beta_{0})}{n},

where {S}^{2}(z;\beta_{z}) (z=1,0) and S^{2}(\tau;\beta_{1},\beta_{0}) are the finite population variances of the adjusted potential outcomes and individual causal effects, respectively; moreover, a conservative variance estimate is

\hat{V}(\beta_{1},\beta_{0})=\frac{\hat{S}^{2}(1;\beta_{1})}{n_{1}}+\frac{\hat{S}^{2}(0;\beta_{0})}{n_{0}},

where

\displaystyle\hat{S}^{2}(1;\beta_{1}) \displaystyle= \displaystyle(n_{1}-1)^{-1}\sum_{i=1}^{n}Z_{i}(Y_{i}-\gamma_{1}-\beta_{1}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2},
\displaystyle\hat{S}^{2}(0;\beta_{0}) \displaystyle= \displaystyle(n_{0}-1)^{-1}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-\gamma_{0}-\beta_{0}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2}

are the sample variances of the adjusted potential outcomes with \gamma_{1} and \gamma_{0} being the sample means of Y_{i}-\beta_{1}^{\mkern-1.5mu\mathsf{T}}X_{i} under treatment and Y_{i}-\beta_{0}^{\mkern-1.5mu\mathsf{T}}X_{i} under control. To minimize \hat{V}(\beta_{1},\beta_{0}), we need to solve two OLS problems20:

\min_{\gamma_{1},\beta_{1}}\sum_{i=1}^{n}Z_{i}(Y_{i}-\gamma_{1}-\beta_{1}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2},\quad\min_{\gamma_{0},\beta_{0}}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-\gamma_{0}-\beta_{0}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2}.

We run OLS of Y_{i} on X_{i} for the treatment and control groups separately and obtain (\hat{\gamma}_{1},\hat{\beta}_{1}) and (\hat{\gamma}_{0},\hat{\beta}_{0}). The final estimator is

\displaystyle\hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) \displaystyle= \displaystyle n_{1}^{-1}\sum_{i=1}^{n}Z_{i}(Y_{i}-\hat{\beta}_{1}^{\mkern-1.5mu\mathsf{T}}X_{i})-n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-\hat{\beta}_{0}^{\mkern-1.5mu\mathsf{T}}X_{i})
\displaystyle= \displaystyle\left\{\hat{\bar{Y}}(1)-\hat{\beta}_{1}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(1)\right\}-\left\{\hat{\bar{Y}}(0)-\hat{\beta}_{0}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(0)\right\}.

From the properties of the OLS fits (see (162)), we know

\hat{\bar{Y}}(1)=\hat{\gamma}_{1}+\hat{\beta}_{1}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(1),\quad\hat{\bar{Y}}(0)=\hat{\gamma}_{0}+\hat{\beta}_{0}^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(0).

Therefore, we can rewrite the estimator as

\hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0})=\hat{\gamma}_{1}-\hat{\gamma}_{0} (29)

The equivalent form in (29) suggests that we can obtain \hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) from a single OLS fit below.

Proposition 6.2

The estimator \hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) in (29) equals the coefficient of Z_{i} in the OLS fit of Y_{i} on (1,Z_{i},X_{i},Z_{i}\times X_{i}), which is Lin, (2013)’s estimator \hat{\tau}_{\textsc{L}} introduced before.

I leave the proof of Proposition 6.2 to Problem 6.7, which is a pure linear algebra fact.

Based on the discussion above, a conservative variance estimator for \hat{\tau}_{\textsc{L}} is

\displaystyle\hat{V}(\hat{\beta}_{1},\hat{\beta}_{0}) \displaystyle= \displaystyle\frac{1}{n_{1}(n_{1}-1)}\sum_{i=1}^{n}Z_{i}(Y_{i}-\hat{\gamma}_{1}-\hat{\beta}_{1}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2}
\displaystyle\qquad+\frac{1}{n_{0}(n_{0}-1)}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-\hat{\gamma}_{0}-\hat{\beta}_{0}^{\mkern-1.5mu\mathsf{T}}X_{i})^{2}.

Based on quite technical calculations, Lin, (2013) further showed that the EHW standard error from the OLS in Proposition 6.2 is almost identical to \hat{V}(\hat{\beta}_{1},\hat{\beta}_{0}) which is a conservative estimator of the true standard error of \hat{\tau}_{\textsc{L}} under the CRE. Intuitively, this is because we do not assume that the linear model is correctly specified, and the EHW standard error is robust to model misspecification.

There is a subtle issue with the discussion above. The variance formula \textup{var}\{\hat{\tau}(\beta_{1},\beta_{0})\} works for fixed (\beta_{1},\beta_{0}), but the estimator \hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) uses two estimated coefficients (\hat{\beta}_{1},\hat{\beta}_{0}). The additional uncertainty in the estimated coefficients may cause finite-sample bias in the final estimator. Lin, (2013) showed that this issue goes away asymptotically because \hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) behaves similarly to \hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0}), where \tilde{\beta}_{1} and \tilde{\beta}_{0} are the limit of \hat{\beta}_{1} and \hat{\beta}_{0}, respectively. Heuristically, the difference between \hat{\tau}(\hat{\beta}_{1},\hat{\beta}_{0}) and \hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0}) depends on

(\hat{\beta}_{z}-\tilde{\beta}_{z})^{\mkern-1.5mu\mathsf{T}}\hat{\bar{X}}(z),\quad(z=0,1)

which is a product of two terms of small order. As a warning, the asymptotic theory requires a large sample size and some regularity conditions on the potential outcomes and covariates. In finite samples, regression adjustment can be even harmful due to the additional uncertainty in the estimated coefficients \hat{\beta}_{1} and \hat{\beta}_{0}.

30.2.2 Understanding Lin, (2013)’s estimator via predicting the potential outcomes
Table 1: Predicting the potential outcomes
X Z Y(1) Y(0) \hat{Y}(1) \hat{Y}(0)
X_{1} 1 Y_{1}(1) ? \hat{\mu}_{1}(X_{1}) \hat{\mu}_{0}(X_{1})
⋮
X_{n_{1}} 1 Y_{n_{1}}(1) ? \hat{\mu}_{1}(X_{n_{1}}) \hat{\mu}_{0}(X_{n_{1}})
X_{n_{1}+1} 0 ? Y_{n_{1}+1}(0) \hat{\mu}_{1}(X_{n_{1}+1}) \hat{\mu}_{0}(X_{n_{1}+1})
⋮
X_{n} 0 ? Y_{n}(0) \hat{\mu}_{1}(X_{n}) \hat{\mu}_{0}(X_{n})

We can view Lin, (2013)’s estimator as a predictive estimator based on OLS fits of the potential outcomes on the covariates. We build a prediction model for Y(1) based on X using the data from the treatment group:

\displaystyle\hat{\mu}_{1}(X)=\hat{\gamma}_{1}+\hat{\beta}_{1}^{\mkern-1.5mu\mathsf{T}}X. (30)

Similarly, we build a prediction model for Y(0) based on X using the data from the control group:

\displaystyle\hat{\mu}_{0}(X)=\hat{\gamma}_{0}+\hat{\beta}_{0}^{\mkern-1.5mu\mathsf{T}}X. (31)

Table 1 illustrates the prediction of the potential outcomes. If we predict the missing potential outcomes, then we have the following predictive estimator:

\displaystyle\hat{\tau}_{\text{pred}}=n^{-1}\left\{\sum_{Z_{i}=1}Y_{i}+\sum_{Z_{i}=0}\hat{\mu}_{1}(X_{i})-\sum_{Z_{i}=1}\hat{\mu}_{0}(X_{i})-\sum_{Z_{i}=0}Y_{i}\right\}. (32)

We can verify that with (30) and (31), the predictive estimator equals Lin, (2013)’s estimator:

\displaystyle\hat{\tau}_{\text{pred}}=\hat{\tau}_{\textsc{L}}. (33)

If we predict all potential outcomes even if they are observed, we have the following projective estimator:

\displaystyle\hat{\tau}_{\text{proj}}=n^{-1}\sum_{i=1}^{n}\{\hat{\mu}_{1}(X_{i})-\hat{\mu}_{0}(X_{i})\}. (34)

We can verify that with (30) and (31), the projective estimator equals Lin, (2013)’s estimator:

\displaystyle\hat{\tau}_{\text{proj}}=\hat{\tau}_{\textsc{L}}. (35)

I leave the proofs of (33) and (35) to Problem 6.8.

The terminology “predictive” and “projective” is from the survey sampling literature (Firth and Bennett,, 1998; Ding and Li,, 2018). The more general formulas (32) and (34) are well-defined with other predictors of the potential outcomes. To make connections with Lin, (2013)’s estimator, I focus on the linear predictors here. The predictors \hat{\mu}_{1}(X) and \hat{\mu}_{0}(X) can be quite general, including much more complicated machine learning algorithms. However, constructing a point estimator is just the first step in analyzing the CRE. A more important second step is to quantify the uncertainty associated with the estimator, which depends on the properties of the predictors of the potential outcomes. Nevertheless, without doing additional theoretical analysis, we can always use (32) and (34) as the test statistics in the FRT.

30.2.3 Understanding Lin, (2013)’s estimator via adjusting for covariate imbalance

The linearly adjusted estimator has an equivalent form

\displaystyle\hat{\tau}(\beta_{1},\beta_{0})=\hat{\tau}-\gamma^{\mkern-1.5mu\mathsf{T}}\hat{\tau}_{X} (36)

where \gamma=\frac{n_{0}}{n}\beta_{1}+\frac{n_{1}}{n}\beta_{0}, so we can also write it as \hat{\tau}(\gamma)=\hat{\tau}(\beta_{1},\beta_{0}). Similarly, Lin, (2013)’s estimator has an equivalent form

\displaystyle\hat{\tau}_{\textsc{L}}=\hat{\tau}-\hat{\gamma}^{\mkern-1.5mu\mathsf{T}}\hat{\tau}_{X}, (37)

where \hat{\gamma}=\frac{n_{0}}{n}\hat{\beta}_{1}+\frac{n_{1}}{n}\hat{\beta}_{0}. I leave the proofs of (36) and (37) to Problem 6.9. The forms (36) and (37) are the mathematical statements of “adjusting for the covariate imbalance.” They essentially subtract some linear combinations of the difference in means of the covariates. Since \hat{\tau} and \hat{\tau}_{X} are correlated under the CRE, the covariate adjustment with an appropriately chosen \gamma reduces the variance of \hat{\tau}. Another interesting feature of (36) and (37) is that the final estimators depend only on \gamma or \hat{\gamma}, so the choice of the \beta-coefficients is not unique. Li and Ding, (2020) pointed out this simple fact. Therefore, Lin, (2013)’s estimator is just one of the optimal estimators. However, it can be easily implemented via the standard OLS with the EHW standard error. That’s why this book focuses on it.

30.3 Some additional remarks on regression adjustment
30.3.1 Duality between ReM and regression adjustment

Li et al., 2018b () pointed out that ReM and Lin, (2013)’s regression adjustment are duals in using covariates in the design and analysis stages of the experiment. To be more specific, when a is small, the asymptotic distribution of \hat{\tau} under ReM is almost identical to the asymptotic distribution of \hat{\tau}_{\textsc{L}} under the CRE. So ReM uses covariates in the design stage and Lin, (2013)’s regression adjustment uses covariates in the analysis stage, achieving nearly the same asymptotic efficiency gain when a is small.

30.3.2 Equivalence of regression adjustment and post-stratification

If we have discrete covariate C_{i} with K categories, we can create K-1 centered dummy variables

X_{i}=(I(C_{i}=1)-\pi_{[1]},\ldots,I(C_{i}=K-1)-\pi_{[K-1]})

where \pi_{[k]} equals the proportion of units with C_{i}=k. In this case, Lin, (2013)’s regression adjustment is equivalent to post-stratification, as summarized by the following proposition.

Proposition 6.3

\hat{\tau}_{\textsc{L}} based in X_{i} is numerically identical to the post-stratification estimator \hat{\tau}_{\textsc{PS}} based on C_{i} (see Chapter 26).

I leave the proof of Proposition 6.3 as Problem 6.11.

30.3.3 Difference-in-difference as a special case of covariate adjustment \hat{\tau}(\beta_{1},\beta_{0})

An important covariate X in many studies is the lagged outcome before the treatment. For instance, the covariate X is the pre-test score if the outcome Y is the post-test score in educational research; the covariate X is the log wage before the job training program if the outcome Y is the log wage after the job training program. With the lagged outcome X as a covariate, a popular estimator is the gain score or difference-in-difference estimator with \beta_{1}=\beta_{0}=1 in (27) and (28):

\displaystyle\hat{\tau}(1,1) \displaystyle= \displaystyle n_{1}^{-1}\sum_{i=1}^{n}Z_{i}(Y_{i}-X_{i})-n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})(Y_{i}-X_{i})
\displaystyle= \displaystyle\left\{\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)\right\}-\left\{\hat{\bar{X}}(1)-\hat{\bar{X}}(0)\right\}.

The first form of \hat{\tau}(1,1) justifies the name gain score because it is essentially the difference in means of the gain score g_{i}=Y_{i}-X_{i}. The second form of \hat{\tau}(1,1) justifies the name difference-in-difference because it is the difference between two differences in means. This estimator is different from Lin, (2013)’s estimator: it fixes \beta_{1}=\beta_{0}=1 in advance while Lin, (2013)’s estimator involves two estimated \beta’s. The \hat{\tau}(1,1) is unbiased with a conservative variance estimator

\displaystyle\hat{V}(1,1) \displaystyle= \displaystyle\frac{1}{n_{1}(n_{1}-1)}\sum_{i=1}^{n}Z_{i}\{g_{i}-\hat{\bar{g}}(1)\}^{2}
\displaystyle\qquad+\frac{1}{n_{0}(n_{0}-1)}\sum_{i=1}^{n}(1-Z_{i})\{g_{i}-\hat{\bar{g}}(0)\}^{2},

where \hat{\bar{g}}(1) and \hat{\bar{g}}(0) are the sample means of the gain score g_{i}=Y_{i}-X_{i} under the treatment and control, respectively. When the lagged outcome is a strong predictor of the outcome, the gain score g_{i}=Y_{i}-X_{i} often has a much smaller variance than the outcome itself. In this case, \hat{\tau}(1,1) often greatly reduces the variance of the simple difference in means of the outcome. See Problem 6.12 for more details.

In theory, Lin, (2013)’s estimator is always more efficient than \hat{\tau}(1,1) with large samples. However, Lin, (2013)’s estimator is biased in finite samples, whereas \hat{\tau}(1,1) is always unbiased.

30.4 Extension to the SRE

It is possible that we have an experiment stratified on a discrete variable C and we also observe additional covariates X. If all strata are large, then we can obtain Lin, (2013)’s estimators within strata \hat{\tau}_{\textsc{L},[k]} and obtain the final estimator as

\hat{\tau}_{\textsc{L},\textsc{S}}=\sum_{k=1}^{K}\pi_{[k]}\hat{\tau}_{\textsc{L},[k]}.

A conservative variance estimator is

\hat{V}_{\textsc{L},\textsc{S}}=\sum_{k=1}^{K}\pi_{[k]}^{2}\hat{V}_{\textsc{ehw},[k]},

where \hat{V}_{\textsc{ehw},[k]} is the EHW variance estimator from the OLS fit of the outcome on the intercept, the treatment indicator, the covariates, and their interactions within stratum k. Importantly, we need to center covariates by their stratum-specific means.

31 Unification, combination, and comparison

Li and Ding, (2020) unified the literature and showed that we can combine rerandomization and regression adjustment. That is, if we rerandomize in the design stage, we can use Lin, (2013)’s estimator with the EHW standard error in the analysis stage. The combination of rerandomization and regression adjustment improves covariate balance in the design stage and estimation efficiency in the analysis stage.

Table 2 summarizes the literature from Neyman, (1923) to Li and Ding, (2020). Arrow 1 illustrates the efficiency gain of covariate adjustment in the CRE: asymptotically, \hat{\tau}_{\textsc{L}} has a smaller variance than \hat{\tau}. Arrow 2 illustrates the efficiency gain of ReM: asymptotically, \hat{\tau} has narrower quantile ranges under the ReM than under the CRE. Arrows 3 and 4 illustrate the benefits of the combination of ReM and the CRE.

Table 2: Design and analysis of experiments

32 Simulation

Angrist et al., (2009) conducted an experiment to evaluate different strategies to improve academic performance among college freshmen in a Canadian university. Here I use a subset of the original data, focusing on the control group and the treatment group offered academic support services and financial incentives for good grades. The outcome is the GPA at the end of the first year. I impute the missing outcomes with the observed average which is somewhat arbitrary (see Problem 6.14). Two covariates are the gender and baseline GPA.

angrist = read.dta("star.dta")
angrist2 = subset(angrist, control == 1||sfsp == 1)
## imputing missing outcomes
y = angrist2$GPA_year1
meany = mean(y, na.rm = TRUE)
y = ifelse(is.na(y), meany, y)
z = angrist2$sfsp
x = angrist2[, c("female", "gpa0")]

The following code gives the results based on the unadjusted and adjusted estimators.

> ## unadjusted estimator
> fit_unadj = lm(y ~ z)
> ace_unadj = coef(fit_unadj)[2]
> se_unadj = sqrt(hccm(fit_unadj, type = "hc2")[2, 2])
>
> ## regression adjustment
> x = scale(x)
> fit_adj = lm(y ~ z*x)
> ace_adj = coef(fit_adj)[2]
> se_adj = sqrt(hccm(fit_adj, type = "hc2")[2, 2])
>
> res = c(ace_unadj, ace_adj, se_unadj, se_adj)
> dim(res) = c(2, 2)
> t.stat = res[, 1]/res[, 2]
> p.value = 2*pnorm(abs(t.stat), lower.tail = FALSE)
> res = cbind(res, t.stat, p.value)
> rownames(res) = c("Neyman", "Lin")
> colnames(res) = c("estimate", "s.e.", "t-stat", "p-value")
> round(res, 3)
estimate s.e. t-stat p-value
Neyman 0.054 0.076 0.719 0.472
Lin 0.075 0.072 1.036 0.300

The adjusted estimator has a smaller standard error although it gives the same insignificant result as the unadjusted estimator.

I also use this dataset to conduct simulation studies to evaluate the four design and analysis strategies summarized in Table 2. I fit quadratic functions of the outcome on the covariates and use them to impute all the missing potential outcomes, separately for the treated and control groups. To show the improvement of ReM and regression adjustment, I also rescale the error terms by 0.1 and 0.25 to increase the signal-to-noise ratio. With the imputed Science Table, I generate 2000 treatments, obtain the observed data, and calculate the estimators. In the simulation, the “true” outcome model is nonlinear, but we still use linear adjustment for estimation. By doing this, we can evaluate the properties of the estimators even when the linear model is misspecified.

Figure 13 shows the violin plots21 of the four combinations, subtracting the true \tau from the estimates. As predicted by the theory, all estimators are nearly unbiased, and both ReM and regression adjustment improve efficiency. They are more effective when the noise level is smaller.

插图
Figure 13: Simulation with 2000 Monte Carlo replicates and a=0.05 for ReM

33 Final remarks

ReM uses the Mahalanobis distance as the balance criterion. We can consider general rerandomization with the balance criterion defined as a function of \bm{Z} and \bm{X}. For example, we can use the following criterion based on marginal tests for all coordinates of X_{i}=(x_{i1},\ldots,x_{iK})^{\mkern-1.5mu\mathsf{T}}. We accept \bm{Z} if and only if

\left|\frac{\hat{\tau}_{xk}}{\sqrt{\frac{n}{n_{1}n_{0}}S_{xk}^{2}}}\right|\leq a\quad(k=1,\ldots,K) (38)

for some predetermined constant a>0, where S_{xk}^{2} is the finite-population variance of covariate x_{ik}. For example, a is some upper quantile of a standard Normal distribution. See Zhao and Ding, 2021b () for the theory for the rerandomization based on criterion (38) as well as other criteria.

With a continuous outcome, Fisher’s ANCOVA has been the standard approach for many years. Lin, (2013)’s improvement has better theoretical properties even when the linear model is misspecified. With a binary outcome, it is common to use the coefficient of the treatment in the logistic regression of the observed outcome on the intercept, the treatment indicator, and covariates to estimate the causal effects. However, Freedman, 2008c () showed that this logistic regression does not have nice properties under the potential outcomes framework. Even if the logistic model is correct, the coefficient estimates the conditional odds ratio (see Chapter 31.F) which may not be the parameter of interest; when the logistic model is incorrect, it is even harder to interpret the coefficient. From the discussion above, if the parameter of interest is the average causal effect, we can still use Lin, (2013)’s estimator to analyze the binary outcome data in the CRE. Guo and Basse, (2023) extend Lin, (2013)’s theory to allow for using generalized linear models to construct estimators for the average causal effect under the potential outcomes framework.

Other extensions of Lin, (2013)’s theory focus on high dimensional covariates. Bloniarz et al., (2016) focus on the regime with many covariates than the sample size, and suggest replacing the OLS fits with the least absolute shrinkage and selection operator (LASSO) fits (Tibshirani,, 1996) of the outcome on the intercept, the treatment, covariates and their interactions. Lei and Ding, (2021) focus on the regime with a diverging number of covariates without assuming sparsity, and under certain regularity conditions, they show that Lin, (2013)’s estimator is still consistent and asymptotically Normal. Wager et al., (2016) propose to use machine learning methods to analyze high dimensional experimental data.

34 Homework Problems

6.1 FRT under ReM

Describe the FRT under ReM.

6.2 Invariance of the Mahalanobis Distance

Prove Lemma 6.1.

6.3 Bias of the difference-in-means estimator under rerandomization

Assume that we draw \bm{Z}=(Z_{1},\ldots,Z_{n}) from a CRE and accept it if and only if \phi(\bm{Z},\bm{X})=1, where \phi is a predetermined balance criterion. Show that if n_{1}=n_{0} and

\displaystyle\phi(\bm{Z},\bm{X})=\phi(\bm{1}_{n}-\bm{Z},\bm{X}), (39)

then \hat{\tau} is unbiased for \tau. Verify that rerandomization using the Mahalanobis distance satisfies (39) if n_{1}=n_{0}. Give a counterexample that \hat{\tau} is biased for \tau when these two conditions do not hold.

Remark: \phi(\bm{Z},\bm{X}) can be a general balance criterion in this problem. ReM is a special case with \phi(\bm{Z},\bm{X})=I(M\leq a).

6.4 Equivalent form of R^{2} in the CRE

Prove Proposition 6.1.

6.5 More on linear projections of the potential outcomes onto covariates

Show that

S^{2}(1\mid X)=S_{Y(1)X}(S_{X}^{2})^{-1}S_{XY(1)}

where S_{XY(1)} is the finite population covariance between Y_{i}(1)’s and X_{i}’s, S_{Y(1)X}=S_{XY(1)}^{\mkern-1.5mu\mathsf{T}}, and S_{X}^{2} is the finite population covariance matrix of X_{i}’s. Find the analogous formulas for S^{2}(0\mid X) and S^{2}(\tau\mid X).

6.6 Comparing the true variances within the family of linearly adjusted estimator

Show that the variance \hat{\tau}(\beta_{1},\beta_{0}) decomposes into

\textup{var}\{\hat{\tau}(\beta_{1},\beta_{0})\}=\textup{var}\{\hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0})\}+\textup{var}\{\hat{\tau}(\beta_{1},\beta_{0})-\hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0})\}

where \tilde{\beta}_{1} and \tilde{\beta}_{0} are the coefficients of X_{i} in the OLS projection of Y_{i}(1)’s and Y_{i}(0)’s onto (1,X_{i})’s, respectively.

Remark: Li and Ding, (2017, Example 9) derived the decomposition. It shows that based on the true variance, \hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0}) is an optimal choice within the family of linearly adjusted estimator because \textup{var}\{\hat{\tau}(\beta_{1},\beta_{0})-\hat{\tau}(\tilde{\beta}_{1},\tilde{\beta}_{0})\} is always non-negative.

6.7 Lin’s estimator for covariate adjustment

Prove Proposition 6.2.

6.8 Predictive and projective estimators

Prove (33) and (35).

6.9 Equivalent form of the covariate-adjusted estimator

Prove (36) and (37).

6.10 ANCOVA also adjusts for covariate imbalance

This problem gives a result for ANCOVA that is similar to (37).

Show that

\hat{\tau}_{\textsc{F}}=\hat{\tau}-\hat{\gamma}_{\textsc{F}}^{\mkern-1.5mu\mathsf{T}}\hat{\tau}_{X},

where \hat{\gamma}_{\textsc{F}} is the coefficient of X_{i} in the OLS fit of Y_{i} on (1,Z_{i},X_{i}).

6.11 Regression adjustment / post-stratification of CRE

Prove Proposition 6.3.

Remark: Sometimes \hat{\tau}_{\textsc{ps}} or \hat{\tau}_{\textsc{L}} may not be well-defined. In those cases, we treat \hat{\tau}_{\textsc{ps}} and \hat{\tau}_{\textsc{L}} as equal. You can ignore this complexity in the proof.

6.12 More on the difference-in-difference estimator in the CRE

This problem gives more details for the difference-in-difference estimator in the CRE in Section 30.3.3.

Show that \hat{\tau}(1,1) is unbiased for \tau, calculate its variance, and show that \hat{V}(1,1) is a conservative estimator for the true variance of \hat{\tau}(1,1). When does E\{\hat{V}(1,1)\}=\textup{var}\{\hat{\tau}(1,1)\} hold?

Compare the variances of \hat{\tau}(0,0) and \hat{\tau}(1,1) to show that

\textup{var}\{\hat{\tau}(0,0)\}\geq\textup{var}\{\hat{\tau}(1,1)\}

if and only if

2\frac{n_{0}}{n}\beta_{1}+2\frac{n_{1}}{n}\beta_{0}\geq 1,

where

\beta_{1}=\frac{\sum_{i=1}^{n}(X_{i}-\bar{X})\{Y_{i}(1)-\bar{Y}(1)\}}{\sum_{i=1}^{n}(X_{i}-\bar{X})^{2}},\quad\beta_{0}=\frac{\sum_{i=1}^{n}(X_{i}-\bar{X})\{Y_{i}(0)-\bar{Y}(0)\}}{\sum_{i=1}^{n}(X_{i}-\bar{X})^{2}}

are the coefficients of X_{i} in the OLS fits of Y_{i}(1) and Y_{i}(0) on (1,X_{i}), respectively.

Remark: Gerber and Green, (2012, page 28) discussed a special case of this problem with n_{1}=n_{0}.

6.13 Data re-analyses of the Penn Bonus Experiment

Re-analyze the Penn Bonus Experiment data. The analysis in Chapter 5 uses the treatment indicator, the outcome, and the block indicator. Now we want to use all other covariates.

Conduct regression adjustments within the strata of the experiment, and then combine these adjusted estimators to estimate the average causal effect. Report the point estimator, estimated standard error, and 95% confidence interval. Compare them with those without regression adjustment.

6.14 Missing outcomes in randomized experiments

The data analysis in Section 32 uses a naive imputation method to deal with missing outcomes. It is somewhat arbitrary.

Impute the missing outcomes under the treatment and control, respectively, based on the observed means. Do the results change?

Impute the missing outcomes under the treatment and control, respectively, based on linear regressions of the observed outcomes on the covariates. Do the results change?

Do you have other ways to deal with missing outcomes? Justify them and implement them to analyze the dataset.

6.15 Recommended reading

The title of this chapter is the same as that of Li and Ding, (2020), which studied the roles of rerandomization and regression adjustment in the design and analysis stages of randomized experiments, respectively.

脚注

18. The notation “A\stackrel{{\scriptstyle\cdot}}{{\sim}}B” means that A and B have the same asymptotic distributions.

19. Without covariates, the HC2 correction yields identical variance estimator as Neyman, (1923)’s classic one; see Problem 4.3. For coherence, we can also use the HC2 correction for Lin, (2013)’s estimator with covariate adjustment. When the number of covariates is small compared with the sample size and the covariates do not contain outliers, the variants of the EHW standard error perform similarly to the original one. When the number of covariates is large compared with the sample size or the covariates contain outliers, the variants can outperform the original one. In those cases, Lei and Ding, (2021) recommend using the HC3 variant of the EHW standard error.

20. We can also minimize the true variance of \hat{\tau}(\beta_{1},\beta_{0}). See Problem 6.6 for more details.

21. A violin plot is similar to a box plot, with the addition of a rotated kernel density plot on each side.

本书:A First Course in Causal Inference,作者 Peng Ding。原文:arXiv:2305.18793v2,按 CC BY 4.0 许可发布。本站(yuanens.com)把 arXiv 网页版改成分章的纯 HTML,把公式渲染成图片,方便在 Kindle 等电子书阅读器上看,内容没有改动。转换难免有错,以原文为准。本站与作者和出版社无关。