Chapter 8 Unification of the Fisherian and Neymanian Inferences in Randomized Experiments

Chapters 3–7 cover both the Fisherian and Neymanian inferences for different types of experiments. The Fisherian perspective focuses on the finite-sample exact p-values for testing the strong null hypothesis of no causal effects for any units whatsoever, whereas the Neymanian perspective focuses on unbiased estimation with a conservative large-sample confidence interval for the average causal effect. Both of them are justified by the physical randomization which is ensured by the design of the experiments. Because of this, they are both called randomization-based inference or design-based inference. Because they concern a finite population of units in the experiments, they are also called finite-population inference. They are related but also have distinct features.

In 1935, Neyman presented his seminal paper on randomization-based inference to the Royal Statistical Society. His paper (Neyman,, 1935) was attacked by Fisher in the discussion session. Sabbaghi and Rubin, (2014) reviewed this famous Neyman–Fisher controversy and presented some new results for this old problem. Instead of going to the philosophical issues, this chapter tries to provide a unified discussion.

43 Testing strong and weak null hypotheses in the CRE

Let us revisit the treatment-control CRE. The Fisherian perspective focuses on testing the strong null hypothesis

H_{0\textsc{f}}:Y_{i}(1)=Y_{i}(0)\text{ for all units }i=1,\ldots,n.

The FRT delivers a finite-sample exact p_{\textsc{frt}} defined in (8).

By duality of the confidence interval and hypothesis testing, the Neymanian perspective gives a test for the weak null hypothesis

H_{0\textsc{n}}:\tau=0\Longleftrightarrow H_{0\textsc{n}}:\bar{Y}(1)=\bar{Y}(0)

based on

t=\frac{\hat{\tau}}{\sqrt{\hat{V}}}.

By the CLT of \hat{\tau} and the conservativeness of the variance estimator, we have

t=\sqrt{\frac{\textup{var}(\hat{\tau})}{\hat{V}}}\times\frac{\hat{\tau}}{\sqrt{\textup{var}(\hat{\tau})}}\rightarrow C\times\textsc{N}(0,1)

in distribution, where C is smaller than or equal to 1 but depends on the unknown potential outcomes. Using \textsc{N}(0,1) quantiles for the studentized statistic t, we have a conservative large-sample test for H_{0\textsc{n}}.

Furthermore, Ding and Dasgupta, (2017) show that the FRT with the studentized statistic t has the dual guarantees:

  1. 1.

    p_{\textsc{frt}} is finite-sample exact under H_{0\textsc{f}};

  2. 2.

    p_{\textsc{frt}} is asymptotically conservative under H_{0\textsc{n}}.

Importantly, this is a feature of the studentized statistic t. Ding and Dasgupta, (2017) showed that the FRT with other test statistics may not have the dual guarantees. In particular, the FRT with \hat{\tau} may be asymptotically anti-conservative under H_{0\textsc{n}}. I give some heuristics below to illustrate the importance of studentization in the FRT.

Under H_{0\textsc{n}}, we have

\hat{\tau}\stackrel{{\scriptstyle\cdot}}{{\sim}}\textsc{N}\left(0,\frac{S^{2}(1)}{n_{1}}+\frac{S^{2}(0)}{n_{0}}-\frac{S^{2}(\tau)}{n}\right).

The FRT pretends that the Science Table is (Y_{i},Y_{i})_{i=1}^{n} induced by H_{0\textsc{f}}, so the randomization distribution of \hat{\tau} is

(\hat{\tau})^{\pi}\stackrel{{\scriptstyle\cdot}}{{\sim}}\textsc{N}\left(0,\frac{s^{2}}{n_{1}}+\frac{s^{2}}{n_{0}}\right),

where (\cdot)^{\pi} denotes the randomization distribution24 and s^{2} is the sample variance of the observed outcomes. Based on (13) in Chapter 3, we can approximate the asymptotic variance of (\hat{\tau})^{\pi} under H_{0\textsc{f}} as

\displaystyle\frac{s^{2}}{n_{1}}+\frac{s^{2}}{n_{0}} \displaystyle= \displaystyle\frac{n}{n_{1}n_{0}}\left\{\frac{n_{1}-1}{n-1}\hat{S}^{2}(1)+\frac{n_{0}-1}{n-1}\hat{S}^{2}(0)+\frac{n_{1}n_{0}}{n(n-1)}\hat{\tau}^{2}\right\}
\displaystyle\approx \displaystyle\frac{\hat{S}^{2}(1)}{n_{0}}+\frac{\hat{S}^{2}(0)}{n_{1}}
\displaystyle\approx \displaystyle\frac{S^{2}(1)}{n_{0}}+\frac{S^{2}(0)}{n_{1}},

which does not match the asymptotic variance of \hat{\tau}. Ideally, we should compute the p-value under H_{0\textsc{n}} based on the true distribution of \hat{\tau}, which, however, depends on the unknown potential outcomes. In contrast, we use the FRT to compute the p_{\textsc{frt}} based on the permutation distribution (\hat{\tau})^{\pi}, which does not match the true distribution of \hat{\tau} under H_{0\textsc{n}} even with large samples. Therefore, the FRT with \hat{\tau} may not control the type one error rate under H_{0\textsc{n}} even with large samples.

Fortunately, the undesired property of the FRT with \hat{\tau} goes away if we replace the test statistic \hat{\tau} with the studentized version t. Under H_{0\textsc{n}}, we have

t\stackrel{{\scriptstyle\cdot}}{{\sim}}\textsc{N}(0,C^{2})

where C^{2}\leq 1 with equality holding if Y_{i}(1)-Y_{i}(0)=\tau for all units i=1,\ldots,n. The FRT pretends that Y_{i}(1)=Y_{i}(0)=Y_{i} for all i’s and generates the permutation distribution

t^{\pi}\stackrel{{\scriptstyle\cdot}}{{\sim}}\textsc{N}(0,1)

where the variance equals 1 because the Science Table used by the FRT has zero individual causal effects. Under H_{0\textsc{n}}, because the true distribution of t is more dispersed than the corresponding permutation distribution, the p_{\textsc{frt}} based on t is asymptotically conservative.

44 Covariate-adjusted FRTs in the CRE

Extending the discussion in Section 43 to the case with covariates, Zhao and Ding, 2021a () recommend using the FRT with the studentized Lin, (2013)’s estimator:

t_{\textsc{L}}=\frac{\hat{\tau}_{\textsc{L}}}{\sqrt{\hat{V}_{\textsc{L}}}},

which is the robust t-statistic for the coefficient of Z_{i} in the OLS fit of Y_{i} on (1,Z_{i},X_{i},Z_{i}X_{i}). They show that the FRT with t_{\textsc{L}} has multiple guarantees:

  1. 1.

    p_{\textsc{frt}} is finite-sample exact under H_{0\textsc{f}};

  2. 2.

    p_{\textsc{frt}} is asymptotically conservative under H_{0\textsc{n}};

  3. 3.

    p_{\textsc{frt}} is asymptotically more powerful than the FRT with t when H_{0\textsc{n}} does not hold and the covariates are predictive to the outcomes;

  4. 4.

    the above properties hold even when the linear outcome model is misspecified.

Similarly, this is a feature of the studentized statistic t_{\textsc{L}}. Zhao and Ding, 2021a () show that other covariate-adjusted FRTs reviewed in Section 30.1 may be either anti-conservative under H_{0\textsc{n}} or less powerful than the FRT with t_{\textsc{L}} when H_{0\textsc{n}} does not hold.

45 A simulation study

Now I use simulation to evaluate the finite-sample properties of the p_{\textsc{frt}}’s under the weak null hypothesis. I will use the following twelve test statistics.

  1. 1.

    The first three test statistics are from the OLS implementation of Neyman, (1923), including the coefficient of the treatment, the t-statistic based on the classic standard error, and the t-statistic based on the EHW standard error.

  2. 2.

    The next three test statistics are based on the pseudo-outcome strategy in Definition 6.2, advocated by Rosenbaum, 2002a (). We first residualize the outcomes by regressing Y_{i} on (1,X_{i}), and then obtain the three test statistics similar to the first three.

  3. 3.

    The next three test statistics are based on the OLS implementation of Fisher, (1925), as discussed in Chapter 30.2.

  4. 4.

    The final three test statistics are based on the OLS implementation of Lin, (2013), as discussed in Chapter 30.2.

Consider a finite population of n=100 units subjected to a CRE of size (n_{1},n_{0})=(20,80). For each i, we draw a univariate covariate X_{i} from Unif(-1,1) and generate potential outcomes as Y_{i}(1)\sim\textsc{N}(X_{i}^{3},1) and Y_{i}(0)\sim\textsc{N}(-X_{i}^{3},0.5^{2}). Center the Y_{i}(1)’s and Y_{i}(0)’s to ensure \tau=0. Fix \{Y_{i}(1),Y_{i}(0),X_{i}\}_{i=1}^{n} in simulation. We draw a random permutation of n_{1} 1’s and n_{0} 0’s to obtain the observed outcomes and conduct FRTs. The procedure is repeated 500 times, with the p-values approximated by 500 independent permutations of the treatment vector in each replication.

Figure 16 shows the p-values under the weak null hypothesis. The four robust t-statistics, as shown in the last row, are the only ones that preserve the correct type one error rates. In fact, they are conservative, which is coherent with the theory. All the other eight statistics yield type one error rates greater than the nominal levels and are thus not proper for testing the weak null hypothesis.

插图
Figure 16: Histograms of p_{\textsc{frt}} under the weak null hypothesis

46 General recommendations

The recommendations for the SRE parallel those for the CRE if both the strong and weak null hypotheses are of interest. Recall the estimators in Chapters 5 and 30.4. Without additional covariates, Zhao and Ding, 2021a () recommend using the FRT with

t_{\textsc{S}}=\frac{\hat{\tau}_{\textsc{S}}}{\sqrt{\hat{V}_{\textsc{S}}}};

with additional covariates, they recommend using the FRT with

t_{\textsc{L,S}}=\frac{\hat{\tau}_{\textsc{L,S}}}{\sqrt{\hat{V}_{\textsc{L,S}}}}.

The analysis of ReM is trickier. Zhao and Ding, 2021a () show that the FRT with t does not have the dual guarantees in Section 43, but the FRT with t_{\textsc{L}} still has the guarantees in Section 44. This highlights the importance of both covariate adjustment and studentization in ReM.

Similar results hold for the MPE. Without covariates, we recommend using the FRT with the t-statistic for the intercept in the OLS fit of \hat{\tau}_{i} on 1; with covariates, we recommend using the FRT with the t-statistic for the intercept in the OLS fit of \hat{\tau}_{i} on 1 and \hat{\tau}_{X,i}. Figure 15 in Chapter 7 are based on these recommended FRTs.

Overall, the FRTs with studentized statistics are safer choices. When the large-sample Normal approximations to the studentized statistics are accurate, the p_{\textsc{frt}}’s are almost identical to the p-values based on Normal approximations. When the large-sample approximations are inaccurate, the FRTs at least guarantee valid p-values under the strong null hypotheses. This is the recommendation of this book.

47 A case study

Recall the SRE in Chong et al., (2016) analyzed in Chapter 26.2. I compare the “soccer” arm versus the “control” arm and the “physician” arm versus the “control” arm. We also compare the FRTs with and without using the covariate indicating the baseline anemia status. We use their dataset to illustrate the FRTs in the CRE and SRE. The ten subgroup analyses within the same class levels use the FRTs with t and t_{\textsc{L}} for the CRE and the two overall analyses averaging over all class levels use the FRTs with t_{\textsc{S}} and t_{\textsc{L,S}} for the SRE.

Table 4 shows the point estimators, standard errors, the p-value based on the Normal approximation of the robust t-statistics, and the p-value based on the FRTs. In most strata, covariate adjustment decreases the standard error since the baseline anemia status is predictive of the outcome. Table 4 also exhibits two exceptions: within class 2, covariate adjustment increases the standard error when comparing “soccer” and “control”; in class 4, covariate adjustment increases the standard error when comparing “physician” and “control”. This is due to the small group sizes within these strata, causing the asymptotic approximation inaccurate. Nevertheless, in these two scenarios, the differences in the standard error are in the third digit. The p-values from the Normal approximation and the FRT are close with the latter being slightly larger in most cases. Based on the theory, the p-values based on the FRT should be trusted since it has an additional guarantee of being finite-sample exact under the sharp null hypothesis. This becomes important in this example since the group sizes are quite small within strata.

Table 4: Re-analyzing Chong et al., (2016)’s data. “N” corresponds to the unadjusted estimators and tests due to Neyman, (1923), and “L” corresponds to the covariate-adjusted estimators and tests due to Lin, (2013).
(a) soccer versus control
est s.e. p_{\text{Normal}} p_{\textsc{frt}}
class 1
N 0.051 0.502 0.919 0.924
L 0.050 0.489 0.919 0.929
class 2
N -0.158 0.451 0.726 0.722
L -0.176 0.452 0.698 0.700
class 3
N 0.005 0.403 0.990 0.989
L -0.096 0.385 0.803 0.806
class 4
N -0.492 0.447 0.271 0.288
L -0.511 0.447 0.253 0.283
class 5
N 0.390 0.369 0.291 0.314
L 0.443 0.318 0.164 0.186
all
N -0.051 0.204 0.802 0.800
L -0.074 0.200 0.712 0.712
(b) physician versus control
est s.e. p_{\text{Normal}} p_{\textsc{frt}}
class 1
N 0.567 0.426 0.183 0.192
L 0.588 0.418 0.160 0.174
class 2
N 0.193 0.438 0.659 0.666
L 0.265 0.409 0.517 0.523
class 3
N 1.305 0.494 0.008 0.012
L 1.501 0.462 0.001 0.003
class 4
N -0.273 0.413 0.508 0.515
L -0.313 0.417 0.454 0.462
class 5
N -0.050 0.379 0.895 0.912
L -0.067 0.279 0.811 0.816
all
N 0.406 0.202 0.045 0.047
L 0.463 0.190 0.015 0.017

Figure 17 compares the histograms of the randomization distributions of the robust t-statistics with the asymptotic approximations. In the subgroup analysis, we can observe discrepancies between the randomization distributions and \textsc{N}(0,1); averaged over all class levels, the discrepancy becomes unnoticeable. Overall, in this application, the p-values based on the Normal approximation do not differ substantially from those based on the FRTs. Two approaches yield coherent conclusions: the video with a physician telling the benefits of iron supplements improved academic performance and the effect was most significant among students in class 3; in contrast, the video with a famous soccer player telling the benefits of the iron supplements did not have any significant effect.

插图
Figure 17: Re-analyzing Chong et al., (2016)’s data: randomization distributions with 5\times 10^{4} Monte Carlo draws and the \textsc{N}(0,1) approximations

48 Homework Problems

8.1 Re-analyzing Angrist and Lavy, (2009)’s data

This is the Fisherian counterpart of Problem 7.8. Report the p_{\textsc{frt}}’s from the FRTs with studentized statistics.

8.2 Replication of Zhao and Ding, 2021a ()’s Figure 1

Zhao and Ding, 2021a () use simulation to evaluate the finite-sample properties of the p_{\textsc{frt}}’s from the FRTs with various test statistics. Based on their Figure 1, they recommend using the FRT with t_{\textsc{L,S}} to analyze the SRE. Replicate their Figure 1.

8.3 Recommended readings

Using the studentized statistics in permutation tests is not a new idea; see Janssen, (1997) and Chung and Romano, (2013). Under the design-based framework, Ding and Dasgupta, (2017) and Wu and Ding, (2021) made the recommendation for multiarmed experiments, and Zhao and Ding, 2021a () extended the recommendation to covariate adjustment.

脚注

24. \pi is a standard notation for a random permutation.

本书:A First Course in Causal Inference,作者 Peng Ding。原文:arXiv:2305.18793v2,按 CC BY 4.0 许可发布。本站(yuanens.com)把 arXiv 网页版改成分章的纯 HTML,把公式渲染成图片,方便在 Kindle 等电子书阅读器上看,内容没有改动。转换难免有错,以原文为准。本站与作者和出版社无关。