Chapter 3 The Completely Randomized Experiment and the Fisher Randomization Test

The potential outcomes framework has intrinsic connections with randomized experiments. Understanding causal inference in various randomized experiments is fundamental and helpful for understanding causal inference in more complicated non-experimental studies.

Part II of this book focuses on randomized experiments. This chapter focuses on the simplest experiment, the completely randomized experiment (CRE).

10 CRE

Consider an experiment with n units, with n_{1} receiving the treatment and n_{0} receiving the control. We can define the CRE based on its treatment assignment mechanism6.

Definition 3.1 (CRE)

Fix n_{1} and n_{0} with n=n_{1}+n_{0}. A CRE has the treatment assignment mechanism:

\textup{pr}(\bm{Z}=\bm{z})=1\Big/\binom{n}{n_{1}},

where \bm{z}=(z_{1},\ldots,z_{n}) satisfies \sum_{i=1}^{n}z_{i}=n_{1} and \sum_{i=1}^{n}(1-z_{i})=n_{0}.

In Definition 3.1, we assume that the potential outcome vector under treatment \bm{Y}(1)=(Y_{1}(1),\ldots,Y_{n}(1)) and the potential outcome vector under control \bm{Y}(0)=(Y_{1}(0),\ldots,Y_{n}(0)) are both fixed. Even if we view them as random, we can condition on them and the treatment assignment mechanism becomes

\textup{pr}\{\bm{Z}=\bm{z}\mid\bm{Y}(1),\bm{Y}(0)\}=1\Big/\binom{n}{n_{1}}

because \bm{Z}\ind \{\bm{Y}(1),\bm{Y}(0)\} in a CRE. In a CRE, the treatment vector \bm{Z} is from a random permutation of n_{1} 1’s and n_{0} 0’s.

In his seminal book Design of Experiments, Fisher, (1935) pointed out the following advantages of randomization:

  1. [(1)]

  2. 1.

    It creates comparable treatment and control groups on average.

  3. 2.

    It serves as a “reasoned basis” for statistical inference.

Point 1 is intuitive because the random treatment assignment does not bias toward the treatment or the control. Most people understand point 1 well. Point 2 is more subtle. What Fisher meant is that randomization justifies a statistical test, which is now called the Fisher Randomization Test (FRT). This chapter illustrates the basic idea of the FRT under a CRE.

11 FRT

Fisher, (1935) was interested in testing the following null hypothesis:7

H_{0\textsc{f}}:Y_{i}(1)=Y_{i}(0)\text{ for all units }i=1,\ldots,n.

Rubin, (1980) called it the sharp null hypothesis in the sense that it can determine all the potential outcomes based on the observed data: \bm{Y}(1)=\bm{Y}(0)=\bm{Y}=(Y_{1},\ldots,Y_{n}), the vector of the observed outcomes. It is also called the strong null hypothesis (Wu and Ding,, 2021, e.g.,).

Conceptually, under H_{0\textsc{f}}, the FRT works for any test statistic

\displaystyle T=T(\bm{Z},\bm{Y}), (7)

which is a function of the observed data. The observed outcome vector \bm{Y} is fixed under H_{0\textsc{f}}, so the only random component in the test statistic T is the treatment vector \bm{Z}. The experimenter determines the distribution of \bm{Z}, which in turn determines the distribution of T under H_{0\textsc{f}}. This is the basis for calculating the p-value. I will give more details below.

In a CRE, \bm{Z} is uniform over the set

\{\bm{z}^{1},\ldots,\bm{z}^{M}\}

where M=\binom{n}{n_{1}}, and the \bm{z}^{m}’s are all possible vectors with n_{1} 1’s and n_{0} 0’s. For instance, with n=5 and n_{1}=3, we can enumerate M=\binom{5}{3}=10 vectors as follows:

> permutation10 = function(n, n1){
+ M = choose(n, n1)
+ treat.index = combn(n, n1)
+ Z = matrix(0, n, M)
+ for(m in 1:M){
+ treat = treat.index[, m]
+ Z[treat, m] = 1
+ }
+ Z
+ }
>
> permutation10(5, 3)
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 1 1 1 1 1 1 0 0 0 0
[2,] 1 1 1 0 0 0 1 1 1 0
[3,] 1 0 0 1 1 0 1 1 0 1
[4,] 0 1 0 1 0 1 1 0 1 1
[5,] 0 0 1 0 1 1 0 1 1 1

As a consequence, T is uniform over the set (with possible duplications)

\{T(\bm{z}^{1},\bm{Y}),\ldots,T(\bm{z}^{M},\bm{Y})\}.

That is, the distribution of T is known due to the design of the CRE. We will call this distribution of T the randomization distribution.

If larger values are more extreme for T, we can use the following tail probability to measure the extremeness of the test statistic with respect to its randomization distribution:8

p_{\textsc{frt}}=M^{-1}\sum_{m=1}^{M}I\{T(\bm{z}^{m},\bm{Y})\geq T(\bm{Z},\bm{Y})\}, (8)

which is called the p-value by Fisher. Figure 3 illustrates the computational process of p_{\textsc{frt}}.

插图
Figure 3: Illustration of the FRT

The p-value, p_{\textsc{frt}}, in (8) works for any choice of test statistic and any outcome-generating process. It also extends naturally to any experiment, which will be a topic repeatedly discussed in the following chapters. Importantly, it is finite-sample exact in the sense9 that under H_{0\textsc{f}},

\displaystyle\textup{pr}(p_{\textsc{frt}}\leq u)\leq u\quad\text{ for all }\quad 0\leq u\leq 1. (9)

In practice, M is often too large (e.g., with n=100,n_{1}=50, we have M>10^{29}), and it is computationally infeasible to enumerate all possible values of the treatment vector. We often approximate p_{\textsc{frt}} by Monte Carlo (see Section 30.E for a review of the basic Monte Carlo method). To be more specific, we take independent random draws from all possible values of the treatment vector, or, equivalently, we randomly permute \bm{Z}, and approximate p_{\textsc{frt}} by

\hat{p}_{\textsc{frt}}=R^{-1}\sum_{r=1}^{R}I\{T(\bm{z}^{r},\bm{Y})\geq T(\bm{Z},\bm{Y})\}, (10)

where the \bm{z}^{r}’s are the R random permutations of \bm{Z}. The p-value in (10) has Monte Carlo error decreasing fast with an increasing R; see Problem 3.2. Because the calculation of the p-value in (10) involves permutations of \bm{Z}, the FRT is sometimes called the permutation test in the context of the CRE. However, the idea of FRT is more general than the permutation test in more complex experiments.

12 Canonical choices of the test statistic

From the above discussion, the FRT generates finite-sample exact p-value for any choice of the test statistic. This is a feature of the FRT. However, this feature should not encourage arbitrary choice of the test statistic. Intuitively, we must choose test statistics that give information for possible violations of H_{0\textsc{f}}. Below I will review some canonical choices.

Example 3.1 (difference-in-means)

The difference-in-means statistic is

\hat{\tau}=\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)

where

\hat{\bar{Y}}(1)=n_{1}^{-1}\sum_{Z_{i}=1}Y_{i}=n_{1}^{-1}\sum_{i=1}^{n}Z_{i}Y_{i}

is the sample mean of the outcomes under the treatment and

\hat{\bar{Y}}(0)=n_{0}^{-1}\sum_{Z_{i}=0}Y_{i}=n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})Y_{i}

is the sample mean of the outcomes under the control, respectively. Under H_{0\textsc{f}}, it has mean

E(\hat{\tau})=n_{1}^{-1}\sum_{i=1}^{n}E(Z_{i})Y_{i}-n_{0}^{-1}\sum_{i=1}^{n}E(1-Z_{i})Y_{i}=0

and variance

\displaystyle\textup{var}(\hat{\tau}) \displaystyle= \displaystyle\textup{var}\left\{n_{1}^{-1}\sum_{i=1}^{n}Z_{i}Y_{i}-n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})Y_{i}\right\}
\displaystyle= \displaystyle\textup{var}\left(\frac{n}{n_{0}}\frac{1}{n_{1}}\sum_{i=1}^{n}Z_{i}Y_{i}\right)
\displaystyle=_{*} \displaystyle\frac{n^{2}}{n_{0}^{2}}\left(1-\frac{n_{1}}{n}\right)\frac{s^{2}}{n_{1}}
\displaystyle= \displaystyle\frac{n}{n_{1}n_{0}}s^{2},

where =_{*} follows from Lemma 32.2 for simple random sampling with

\bar{Y}=n^{-1}\sum_{i=1}^{n}Y_{i},\quad s^{2}=(n-1)^{-1}\sum_{i=1}^{n}(Y_{i}-\bar{Y})^{2}.

Furthermore, the randomization distribution of \hat{\tau} is approximately Normal due to the finite population central limit theorem (CLT) in Lemma 32.4:

\frac{\hat{\tau}}{\sqrt{\frac{n}{n_{1}n_{0}}s^{2}}}\rightarrow\textsc{N}(0,1) (11)

in distribution. Since s^{2} is fixed under H_{0\textsc{f}}, it is equivalent to use

\frac{\hat{\tau}}{\sqrt{\frac{n}{n_{1}n_{0}}s^{2}}}

as the test statistic in the FRT, which is asymptotically \textsc{N}(0,1) as shown in (11). Then we can calculate an approximate p-value.

The observed data are \{Y_{i}:Z_{i}=1\} and \{Y_{i}:Z_{i}=0\}, so the problem is essentially a two-sample problem. Under the assumption of independent and identically distributed (IID) Normal outcomes (see Section 30.D.1), the classic two-sample t-test assuming equal variance is based on

\frac{\hat{\tau}}{\sqrt{\frac{n}{n_{1}n_{0}(n-2)}\left[\sum_{Z_{i}=1}\{Y_{i}-\hat{\bar{Y}}(1)\}^{2}+\sum_{Z_{i}=0}\{Y_{i}-\hat{\bar{Y}}(0)\}^{2}\right]}}\sim t_{n-2}. (12)

Based on some algebra (see Problem 3.8), we have the expansion

(n-1)s^{2}=\sum_{Z_{i}=1}\{Y_{i}-\hat{\bar{Y}}(1)\}^{2}+\sum_{Z_{i}=0}\{Y_{i}-\hat{\bar{Y}}(0)\}^{2}+\frac{n_{1}n_{0}}{n}\hat{\tau}^{2}. (13)

With a large sample size n, we can ignore the difference between \textsc{N}(0,1) and t_{n-2} and the difference between n-1 and n-2. Moreover, under H_{0\textsc{f}}, \hat{\tau} converges to zero in probability, so n_{1}n_{0}/n\hat{\tau}^{2} can be ignored asymptotically. Therefore, under H_{0\textsc{f}}, the approximate p-value in Example 3.1 is close to the p-value from the classic two-sample t-test assuming equal variance, which can be calculated by t.test with var.equal = TRUE. Under alternative hypotheses with nonzero \tau, the additional term \frac{n_{1}n_{0}}{n}\hat{\tau}^{2} in the expansion (13) can make the FRT less powerful than the usual t-test; Ding, (2016) made this point.

Based on the above discussion, the FRT with \hat{\tau} effectively uses a pooled variance ignoring the heteroskedasticity between these two groups. In classical statistics, the two-sample problem with heteroskedastic Normal outcomes is called the Behrens–Fisher problem (see Section 30.D.1). In the Behrens–Fisher problem, a standard choice of the test statistic is the studentized statistic below.

Example 3.2 (studentized statistic)

The studentized statistic10 is

t=\frac{\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)}{\sqrt{\frac{\hat{S}^{2}(1)}{n_{1}}+\frac{\hat{S}^{2}(0)}{n_{0}}}},

where

\hat{S}^{2}(1)=(n_{1}-1)^{-1}\sum_{Z_{i}=1}\{Y_{i}-\hat{\bar{Y}}(1)\}^{2},\quad\hat{S}^{2}(0)=(n_{0}-1)^{-1}\sum_{Z_{i}=0}\{Y_{i}-\hat{\bar{Y}}(0)\}^{2}

are the sample variances of the observed outcomes under the treatment and control, respectively. Under H_{0\textsc{f}}, the finite population CLT again implies that t is asymptotically Normal:

t\rightarrow\textsc{N}(0,1)

in distribution. Then we can calculate an approximate p-value which is close to the p-value from t.test with var.equal = FALSE.

An extremely important point is that the FRT justifies the traditional t-tests using t.test with either var.equal = TRUE or var.equal = FALSE, even if the underlying distributions are not Normal. Standard statistics textbooks motivate the t-tests based on the Normality assumption, but the assumption can be too strong in practice. Fortunately, the t-test procedures can still be used as long as the finite population CLTs hold. Even if we do not believe the CLTs, we can still use \hat{\tau} and t as test statistics in the FRT to obtain finite-sample exact p-values.

We will motivate this studentized statistic from another perspective in Chapter 8. The theory there shows that using t in FRT is more robust to heteroskedasticity across the two groups.

The following test statistic is robust to outliers resulting from heavy-tailed outcome data.

Example 3.3 (Wilcoxon rank sum)

The difference-in-means statistic \hat{\tau} uses the sample means of the original outcomes, and its sampling distribution depends on the variances of the outcomes. The studentized statistic t uses the sample means and variances of the original outcomes, and its sampling distribution depends on higher moments of the outcomes. This makes \hat{\tau} and t sensitive to outliers.

Another popular test statistic is based on the ranks of the pooled observed outcomes. Let R_{i} denote the rank of Y_{i} in the pooled samples \bm{Y}:

R_{i}=\#\{j:Y_{j}\leq Y_{i}\}.

The Wilcoxon rank sum statistic is the sum of the ranks under treatment:

W=\sum_{i=1}^{n}Z_{i}R_{i}.

For algebraic simplicity, we assume that there are no ties in the outcomes.11 Because the sum of the ranks of the pooled samples is fixed at 1+2+\cdots+n=n(n+1)/2, the Wilcoxon statistic is equivalent to the difference in the means of the ranks under the treatment and control. Under H_{0\textsc{f}}, the R_{i}’s are fixed, so W has mean

E(W)=\sum_{i=1}^{n}E(Z_{i})R_{i}=\frac{n_{1}}{n}\sum_{i=1}^{n}i=\frac{n_{1}}{n}\times\frac{n(n+1)}{2}=\frac{n_{1}(n+1)}{2}

and variance

\displaystyle\textup{var}(W) \displaystyle= \displaystyle\textup{var}\left(n_{1}\frac{1}{n_{1}}\sum_{i=1}^{n}Z_{i}R_{i}\right)
\displaystyle=_{*} \displaystyle n_{1}^{2}\left(1-\frac{n_{1}}{n}\right)\frac{1}{n_{1}}\frac{1}{n-1}\sum_{i=1}^{n}\left(R_{i}-\frac{n+1}{2}\right)^{2}
\displaystyle= \displaystyle\frac{n_{1}n_{0}}{n(n-1)}\sum_{i=1}^{n}\left(i-\frac{n+1}{2}\right)^{2}
\displaystyle= \displaystyle\frac{n_{1}n_{0}}{n(n-1)}\left\{\sum_{i=1}^{n}i^{2}-n\left(\frac{n+1}{2}\right)^{2}\right\}
\displaystyle= \displaystyle\frac{n_{1}n_{0}}{n(n-1)}\left\{\frac{n(n+1)(2n+1)}{6}-n\left(\frac{n+1}{2}\right)^{2}\right\}
\displaystyle= \displaystyle\frac{n_{1}n_{0}(n+1)}{12},

where =_{*} follows from Lemma 32.2. Furthermore, under H_{0\textsc{f}}, the finite population CLT ensures that the randomization distribution of \widehat{\tau} is approximately Normal:

\displaystyle\frac{\sum_{i=1}^{n}Z_{i}R_{i}-\frac{n_{1}(n+1)}{2}}{\sqrt{\frac{n_{1}n_{0}(n+1)}{12}}}\rightarrow\textsc{N}(0,1) (14)

in distribution. Based on (14), we can conduct an asymptotic test. In R, the function wilcox.test can compute both exact and asymptotic p-values based on the statistic W-n_{1}(n_{1}+1)/2. Based on some asymptotic analyses, Lehmann, (1975) showed that the FRT using W has reasonable power over a wide range of data-generating processes.

Example 3.4 (Kolmogorov–Smirnov statistic)

The treatment may affect the outcome in different ways. It seems natural to summarize the treatment outcomes and control outcomes based on the empirical distributions:

\hat{F}_{1}(y)=n_{1}^{-1}\sum_{i=1}^{n}Z_{i}I(Y_{i}\leq y),\quad\hat{F}_{0}(y)=n_{0}^{-1}\sum_{i=1}^{n}(1-Z_{i})I(Y_{i}\leq y).

Comparing these two empirical distributions yields the famous Kolmogorov–Smirnov statistic

D=\max_{y}\Big|\hat{F}_{1}(y)-\hat{F}_{0}(y)\Big|.

It is a challenging mathematics problem to derive the distribution of D. With large sample sizes n_{1}\rightarrow\infty and n_{0}\rightarrow\infty, its distribution function converges to

\textup{pr}\left(\frac{n_{1}n_{0}}{n}D\leq x\right)\rightarrow\frac{\sqrt{2\pi}}{x}\sum_{j=1}^{\infty}e^{-(2j-1)^{2}\pi^{2}/(8x^{2})},

based on which we calculate an asymptotic p-value (Van der Vaart,, 2000). In R, the function ks.test can compute both exact and asymptotic p-values.

13 A case study of the LaLonde experimental data

I use LaLonde, (1986)’s experimental data to illustrate the FRT. The data are available in the Matching package (Sekhon,, 2011):

> library(Matching)
> data(lalonde)
> z = lalonde$treat
> y = lalonde$re78

In the above, z denotes the binary treatment vector indicating whether a unit was randomly assigned to the job training program or not, and y is the outcome vector representing a unit’s real earnings in 1978. Figure 4 shows the histograms of the outcomes under the treatment and control.

插图
Figure 4: Histograms of the outcomes in the LaLonde experimental data: the treatment in white and the control in grey. The vertical lines are the sample means of the outcomes: the solid line for the treatment and the dashed line for the control.

The following code computes the observed values of the test statistics using existing functions:

> tauhat = t.test(y[z == 1], y[z == 0],
+ var.equal = TRUE)$statistic
> tauhat
t
2.835321
> student = t.test(y[z == 1], y[z == 0],
+ var.equal = FALSE)$statistic
> student
t
2.674146
> W = wilcox.test(y[z == 1], y[z == 0])$statistic
> W
W
27402.5
> D = ks.test(y[z == 1], y[z == 0])$statistic
> D
D
0.1321206

By randomly permuting the treatment vector, we can obtain the Monte Carlo approximation of the randomization distributions of the test statistics, stored in four vectors Tauhat, Student, Wilcox, and Ks.

> MC = 10^4
> Tauhat = rep(0, MC)
> Student = rep(0, MC)
> Wilcox = rep(0, MC)
> Ks = rep(0, MC)
> for(mc in 1:MC)
+ {
+ zperm = sample(z)
+ Tauhat[mc] = t.test(y[zperm == 1], y[zperm == 0],
+ var.equal = TRUE)$statistic
+ Student[mc] = t.test(y[zperm == 1], y[zperm == 0],
+ var.equal = FALSE)$statistic
+ Wilcox[mc] = wilcox.test(y[zperm == 1],
+ y[zperm == 0])$statistic
+ Ks[mc] = ks.test(y[zperm == 1],
+ y[zperm == 0])$statistic
+ }

The one-sided p-values based on the FRT are all smaller than 0.05:

> exact.pv = c(mean(Tauhat >= tauhat),
+ mean(Student >= student),
+ mean(Wilcox >= W),
+ mean(Ks >= D))
> round(exact.pv, 3)
[1] 0.002 0.002 0.006 0.040

Without using Monte Carlo, we can also compute the asymptotic p-values which are all smaller than 0.05:

> asym.pv = c(t.test(y[z == 1], y[z == 0],
+ var.equal = TRUE)$p.value,
+ t.test(y[z == 1], y[z == 0],
+ var.equal = FALSE)$p.value,
+ wilcox.test(y[z == 1], y[z == 0])$p.value,
+ ks.test(y[z == 1], y[z == 0])$p.value)
> round(asym.pv, 3)
[1] 0.005 0.008 0.011 0.046

The differences between the p-values are due to the asymptotic approximations as well as the fact that the default choices for t.test and wilcox.test are two-sided tests. To make fair comparisons, we need to multiply the first three p_{\textsc{frt}}’s by a factor of 2.

Figure 5 shows the histograms of the randomization distributions of four test statistics, as well as their corresponding observed values. For the first three test statistics, the Normal approximations work quite well even though the underlying outcome data distribution is far from Normal as shown in Figure 4. In general, a figure like Figure 5 can give clearer information for testing the sharp null hypothesis. Recently, Bind and Rubin, (2020) proposes, in the title of their paper, that “when possible, report a Fisher-exact p-value and display its underlying null randomization distribution.” I agree.

插图
Figure 5: The randomization distributions of four test statistics based on the LaLonde experimental data

14 Some history of randomized experiments and FRT

14.1 James Lind’s experiment

James Lind (1716—1794) was a Scottish doctor and a pioneer of naval hygiene in the Royal Navy. At his time, scurvy was a major cause of death among sailors. He conducted one of the earliest randomized experiments with clear documentation of the details and concluded that citrus fruits cured scurvy before the discovery of Vitamin C.

In Lind, (1753), he described the following randomized experiment with 12 patients of scurvy assigned to 6 groups. With some simplifications, the 6 groups are:

  1. [(1)]

  2. 1.

    two received a quart of cider every day;

  3. 2.

    two received twenty-five drops of sulfuric acid three times every day;

  4. 3.

    two received two spoonfuls of vinegar three times every day;

  5. 4.

    two received half a pint of seawater every day;

  6. 5.

    two received two oranges and one lemon every day;

  7. 6.

    two received a spicy paste plus a drink of barley water every day.

After six days, patients in the fifth group recovered, but patients in other groups did not. If we simplify the treatment as

Z_{i}=1(\text{unit }i\text{ received citrus fruits})

and the outcome as

Y_{i}=1(\text{unit }i\text{ recovered after six days}),

then we have a two-by-two table

Y=1 Y=0
Z=1 2 0
Z=0 0 10

This is the extremest possible two-by-two table we can observe under this experiment, and the data contain strong evidence for the positive effect of citrus fruits for curing scurvy. Statistically, how do we measure the strength of the evidence?

Following the logic of the FRT, if the treatment has no effect at all (under H_{0\textsc{f}}), this extreme two-by-two table will occur with probability

\frac{1}{\binom{12}{2}}=\frac{1}{66}=0.015

which is the p_{\textsc{frt}}. This seems a surprise under H_{0\textsc{f}}: we can easily reject H_{0\textsc{f}} at the level of 0.05.

14.2 Lady tasting tea

Fisher, (1935) described the following famous experiment of Lady Tasting Tea.12 A lady claimed that she could tell the difference between the two ways of making milk tea: one with milk added first, and the other with tea added first. This might sound odd to most people. As a statistician, Fisher designed an experiment to test whether the lady could actually tell the difference between the two ways of making milk tea.

He made 8 cups of tea, 4 with milk added first and the other 4 with tea added first. Then he presented these 8 cups of tea in a random order to the lady and asked the lady to pick up the 4 with milk added first. The final experiment result can be summarized in the following two-by-two table:

milk first (lady) tea first (lady) column sum
milk first (Fisher) X 4-X 4
tea first (Fisher) 4-X X 4
row sum 4 4 8

The X can be 0,1,2,3,4. In the real experiment, X=4, which is the most extreme data, strongly suggesting that the lady could tell the difference between the two ways of making milk tea. Again, how do we measure the strength of the evidence?

Under the null hypothesis that the lady could not tell the difference, only one of the \binom{8}{4}=70 possible orders yields the two-by-two table with X=4. So the p-value is

p_{\textsc{frt}}=\frac{1}{70}=0.014.

Given the significance level of 0.05, we reject the null hypothesis.

14.3 Two Fisherian principles for experiments

In the above two examples in Sections 14.1 and 14.2, the p_{\textsc{frt}}’s are justified by the randomization of the experiments. This highlights the first Fisherian principle of experiments: randomization.

Moreover, the above two experiments are in some sense the smallest possible experiments that can yield statistically meaningful results. For instance, if Lind only assigned one patient to each of the six groups, then the smallest p-value is

\frac{1}{\binom{6}{1}}=\frac{1}{6}=0.167;

if Fisher only made 6 cups of tea, 3 with milk added first and the other 3 with tea added first, then the smallest p-value is

\frac{1}{\binom{6}{3}}=\frac{1}{20}=0.05.

We can never reject the null hypotheses at the level of 0.05. This highlights the second Fisherian principle of experiments: replications. This is, to ensure the FRT to have power, we must have enough units in experiments.

Chapter 5 will discuss the third Fisherian principle of experiments: blocking.

15 Discussion

15.1 Other sharp null hypotheses and confidence intervals

I focus on the sharp null hypothesis H_{0\textsc{f}} above. In fact, the logic of the FRT also works for other sharp null hypotheses. For instance, we can test

H_{0}(\bm{\tau}):Y_{i}(1)-Y_{i}(0)=\tau_{i}\text{ for all }i=1,\ldots,n

for a known vector \bm{\tau}=(\tau_{1},\ldots,\tau_{n}). Because the individual causal effects are all known under H_{0}(\bm{\tau}), we can impute all missing potential outcomes based on the observed data. With known potential outcomes, the distribution of any test statistic T=T(\bm{Z},\bm{Y}(1),\bm{Y}(0)) is completely determined by the treatment assignment mechanism, and therefore, we can compute the corresponding p_{\textsc{frt}} as a function of \bm{\tau}, denoted by p_{\textsc{frt}}(\bm{\tau}). If we can specify all possible \bm{\tau}’s, then we can compute a series of p_{\textsc{frt}}(\bm{\tau})’s. By duality of hypothesis testing and confidence set (see Section 30.B.5), we can obtain a (1-\alpha)-level confidence set for the average causal effect:

\left\{\tau=n^{-1}\sum_{i=1}^{n}\tau_{i}:p_{\textsc{frt}}(\bm{\tau})\geq\alpha\right\}.

Although this strategy is conceptually straightforward, it has practical complexities due to the large number of all possible \bm{\tau}’s. In the special case of a binary outcome, Rigdon and Hudgens, (2015) and Li and Ding, (2016) proposed some computationally feasible approaches to constructing confidence intervals for \tau based on the FRT. For general unbounded outcomes, this strategy is often computationally infeasible.

A canonical simplification (Rosenbaum, 2002b, ; Rosenbaum,, 2010) is to consider a subclass of the sharp null hypotheses with constant individual causal effects:

H_{0}(c):Y_{i}(1)-Y_{i}(0)=c\text{ for all }i=1,\ldots,n

for a known constant c. Given c, we can compute p_{\textsc{frt}}(c). By duality, we can obtain a (1-\alpha)-level confidence set for the average causal effect:

\{c:p_{\textsc{frt}}(c)\geq\alpha\}.

Because this procedure only involves a one-dimensional search, it is computationally feasible. However, the constant individual causal effect assumption is too strong. In particular, it does not hold for binary outcomes unless all units have effects 0, -1, or 1. In general, the constant individual causal effect assumption has testable implications that can be rejected by the observed data; Ding et al., (2016) proposed a formal statistical test.

15.2 Other test statistics

The FRT is a general strategy because it is applicable in any randomized experiment with any test statistic. I have given several examples of test statistics in Section 12. In fact, the definition of a test statistic can be much more general. For instance, with pre-treatment covariate matrix \bm{X} with the ith row being X_{i} for unit i (i=1,\ldots,n) 13, we can allow the test statistic T(\bm{Z},\bm{Y},\bm{X}) to be a function of the treatment vector, outcome vector, and the covariate matrix. Problem 3.6 gives an example.

15.3 Final remarks

For a general experiment, the probability distribution of \bm{Z} is not uniform over all possible permutations of n_{1} 1’s and n_{0} 0’s. However, its distribution is completely known by the experimenter. Therefore, we can always simulate its distribution which in turn implies the distribution of any test statistic under the sharp null hypothesis. A finite-sample exact p-value equals

p_{\textsc{frt}}=\textup{pr}^{\prime}\{T(\bm{Z}^{\prime},\bm{Y})\geq T(\bm{Z},\bm{Y})\}

where \textup{pr}^{\prime} is average over the distribution of \bm{Z}^{\prime} conditional on data. I will discuss other experiments in the subsequent chapters and I want to emphasize that the FRT works beyond the specific experiments discussed in this book.

The FRT works with any test statistic. However, this does not answer the practical question of how to choose a test statistic in the data analysis. If the goal is to find surprises with respect to the sharp null hypothesis, it is desirable to choose a test statistic that yields high power under alternative hypotheses. In general, no test statistic can dominate others in terms of power because power depends on the alternative hypothesis. The four test statistics in Section 12 are motivated by different alternative hypotheses. For instance, \hat{\tau} and t are motivated by an alternative hypothesis with a nonzero average treatment effect; W is motivated by an alternative hypothesis with a constant causal effect with heavy-tailed outcomes. Specifying a working alternative hypothesis is often helpful for constructing a test statistic although it does not have to be precise to guarantee the validity of the FRT. Problems 3.6 and 3.7 illustrate the idea of using a working alternative hypothesis or statistical model to construct test statistics.

16 Homework Problems

3.1 Exactness of p_{\textsc{frt}}

Prove (9).

3.2 Monte Carlo error of \hat{p}_{\textsc{frt}}

Given data, p_{\textsc{frt}} is a fixed number while its Monte Carlo estimator \hat{p}_{\textsc{frt}} in (10) is random. Show that

E_{\text{mc}}(\hat{p}_{\textsc{frt}})=p_{\textsc{frt}}

and

\textup{var}_{\text{mc}}(\hat{p}_{\textsc{frt}})\leq\frac{1}{4R},

where the subscript “mc” signifies the randomness due to Monte Carlo, that is, \hat{p}_{\textsc{frt}} is random because \bm{z}^{r}’s are R independent random draws from all possible values of \bm{Z}.

Remark: p_{\textsc{frt}} is random because \bm{Z} is random. But in this problem, we condition on data, so p_{\textsc{frt}} becomes a fixed number. \hat{p}_{\textsc{frt}} is random because the \bm{z}^{r}’s are random permutations of \bm{Z}. Problem 3.2 shows that \hat{p}_{\textsc{frt}} is unbiased for p_{\textsc{frt}} over the Monte Carlo randomness and gives an upper bound on the variance of \hat{p}_{\textsc{frt}}. Luo et al., (2021, Theorem 2) gives a more delicate bound on the Monte Carlo error.

3.3 A finite-sample valid Monte Carlo approximation of p_{\textsc{frt}}

Although \hat{p}_{\textsc{frt}} is unbiased for p_{\textsc{frt}} by the result in Problem 3.2, it may not be a valid p-value in the sense that \textup{pr}(\hat{p}_{\textsc{frt}}\leq u)\leq u for all u\in(0,1) due to Monte Carlo error with a finite R. The following modified Monte Carlo approximation is always a finite-sample valid p-value. Phipson and Smyth, (2010) pointed out this trick in the permutation test.

Define

\tilde{p}_{\textsc{frt}}=\frac{1+\sum_{r=1}^{R}I\{T(\bm{z}^{r},\bm{Y})\geq T(\bm{Z},\bm{Y})\}}{1+R}

where the \bm{z}^{r}’s the R random permutations of \bm{Z}. Show that with an arbitrary R, the Monte Carlo approximation \tilde{p}_{\textsc{frt}} is always a finite-sample valid p-value in the sense that \textup{pr}(\tilde{p}_{\textsc{frt}}\leq u)\leq u for all u\in(0,1).

Remark: You can use the following two basic probability results to prove the claim in Problem 3.3.

Lemma 3.1

For two Binomial random variables X_{1}\sim\text{Binomial}(R,p_{1}) and X_{2}\sim\text{Binomial}(R,p_{2}) with p_{1}\geq p_{2}, we have \textup{pr}(X_{1}\leq x)\leq\textup{pr}(X_{2}\leq x) for all x.

Lemma 3.2

If p\sim\text{Uniform}(0,1) and X\mid p\sim\text{Binomial}(R,p), then, marginally, X is a uniform random variable over \{0,1,\ldots,R\}.

3.4 Fisher’s exact test

Consider a CRE with a binary outcome, with data summarized in the following two-by-two table:

Y=1 Y=0 total
Z=1 n_{11} n_{10} n_{1}
Z=0 n_{01} n_{00} n_{0}

Under H_{0\textsc{f}}, show that any test statistic T(n_{11},n_{10},n_{01},n_{00}) is a function of n_{11} and other non-random fixed constants, and the exact distribution of n_{11} is Hypergeometric. Specify the parameters for the Hypergeometric distribution.

Remark: Barnard, (1947) and Ding and Dasgupta, (2016) pointed out the equivalence of Fisher’s exact test (reviewed in Section 30.C.1) and the FRT under a CRE with a binary outcome.

3.5 More details for lady tasting tea

Recall the example in Section 14.2. Calculate \textup{pr}(X=k) for k=0,1,2,3,4.

3.6 Covariate-adjusted FRT

This problem gives more details for Section 15.2.

Section 13 re-analyzed the LaLonde experimental data using the FRT with four test statistics. With additional covariates, the FRT can be more general with at least the following two additional strategies. Assume all potential outcomes and covariates are fixed numbers.

First, we can use test statistics based on residuals from the linear regression. Run a linear regression of the outcomes on the covariates, and obtain the residuals (i.e., view the residuals as the “pseudo outcomes”). Then define the four test statistics based on the residuals. Conduct the FRT using these four new test statistics. Report the corresponding p-values.

Second, we can define the test statistic as the coefficient in the linear regression of the outcomes on the treatment and covariates. Conduct the FRT using this test statistic. Report the corresponding p-value.

Why are the five p-values from the above two strategies finite-sample exact? Justify them.

3.7 FRT with a generalized linear model

Use the same dataset as Problem 3.6 but change the outcome to a binary indicator for whether re78 is positive or not. Run logistic regression of the outcome on the treatment and covariates. Is the coefficient of the treatment significant and what is the p-value? Calculate the p-value from the FRT using the coefficient of the treatment as the test statistic.

3.8 An algebraic detail

Verify (13).

3.9 Recommended reading

Bind and Rubin, (2020) is a recent paper advocating the use of p-values as well as the display of the corresponding randomization distributions in analyzing complex experiments.

脚注

6. Readers may think that a CRE has Z_{i}’s as independent and identically distributed (IID) Bernoulli random variables with probability \pi, in which n_{1} is a Binomial(n,\pi) random variable. This is called the Bernoulli randomized experiment (BRE), which reduces to the CRE if we condition on (n_{1},n_{0}). I will give more details for the BRE in Problem 4.7 in Chapter 4.

7. Actually, Fisher, (1935) did not use this form of H_{0\textsc{f}} since he did not use the notation of potential outcomes. This form was due to Rubin, (1980).

8. Because of this, the p_{\textsc{frt}} in (8) is one-sided. Sometimes, we may want to define two-sided p-values. One possibility is to use the absolute value of T if it is distributed around 0.

9. This is the standard definition of the p-value in mathematical statistics. The inequality is often due to the discreteness of the test statistic, and when the equality holds, the p-value is Uniform(0,1) under the null hypothesis. Let F(\cdot) be the distribution function of T(\bm{Z},\bm{Y}). Even though it is a step function, we assume that it is continuous and strictly increasing as if it is the distribution function of a continuous random variable taking values on the whole real line. So p_{\textsc{frt}}=1-F(T), and \textup{pr}(p_{\textsc{frt}}\leq u)=\textup{pr}\{1-F(T)\leq u\}=\textup{pr}\{T\geq F^{-1}(1-u)\}=1-F(F^{-1}(1-u))=u. The discreteness of T does cause some technical issues in the proof, yielding an inequality instead of an equality. I leave the technical details to Problem 3.1.

10. The t notation is intentional because it is related to the t distribution. But it should not be confused with the notation t_{\nu}, which denotes the t distribution with degrees of freedom \nu.

11. The FRT can be applied regardless of the existence of ties. With ties, we can either add small random noises to the original outcomes or use the average ranks for the tied outcomes. For the case with ties, see Lehmann, (1975, Chapter 1 Section 4).

12. This later became the title of a book on the modern history of statistics by Salsburg, (2001).

13. In causal inference, we call X_{i} a covariate if it is not affected by the treatment. That is, if the covariate has two potential outcomes X_{i}(1) and X_{i}(0), then they must satisfy X_{i}(1)=X_{i}(0). Standard statistics books often do not distinguish the treatment and covariates because they often appear on the right-hand side of a regression model for the outcome. They are both called covariates in those statistical models. This book distinguishes the treatment and covariates because they play different roles in causal inference.

本书:A First Course in Causal Inference,作者 Peng Ding。原文:arXiv:2305.18793v2,按 CC BY 4.0 许可发布。本站(yuanens.com)把 arXiv 网页版改成分章的纯 HTML,把公式渲染成图片,方便在 Kindle 等电子书阅读器上看,内容没有改动。转换难免有错,以原文为准。本站与作者和出版社无关。