Chapter 30 Probability and Statistics

This book assumes that the readers have basic knowledge of probability theory and statistical inference. Therefore, this chapter is not a comprehensive review of probability and statistics. For easy reference, I review the key concepts that are crucial for the main text.

Appendix 30.A Probability

30.A.1 Pearson correlation coefficient and squared multiple correlation coefficient

For two random variables Y and X, define the Pearson correlation coefficient as

\rho_{YX}=\frac{\textup{cov}(Y,X)}{\sqrt{\textup{var}(Y)\textup{var}(X)}}

which measures the linear dependence of Y on X. The definition is symmetric in Y and X in that

\rho_{YX}=\rho_{XY}.

With a random variable Y and a random vector X, define the squared multiple correlation coefficient as

R^{2}_{YX}=\textup{corr}^{2}(Y,X)=\frac{\textup{cov}(Y,X)\textup{cov}(X)^{-1}\textup{cov}(X,Y)}{\textup{var}(Y)}

where \textup{cov}(Y,X) is a row vector and \textup{cov}(X,Y) is a column vector. It also measures the linear dependence of Y on X. But this definition is not symmetric in Y and X.

30.A.2 Multivariate Normal random vector

A multivariate Normal random vector \textsc{N}(\mu,\Sigma) is determined by its mean vector \mu and covariance matrix \Sigma. Partition it into two parts:

\left(\begin{array}[]{c}Y_{1}\\
Y_{2}\end{array}\right)\sim\textsc{N}\left(\left(\begin{array}[]{c}\mu_{1}\\
\mu_{2}\end{array}\right),\left(\begin{array}[]{cc}\Sigma_{11}&\Sigma_{12}\\
\Sigma_{21}&\Sigma_{22}\end{array}\right)\right).

First, the marginal distributions are Normal:

Y_{1}\sim\textsc{N}\left(\mu_{1},\Sigma_{11}\right),\quad Y_{2}\sim\textsc{N}\left(\mu_{2},\Sigma_{22}\right).

Second, if \Sigma_{22} is positive definite, then the conditional distribution is also Normal:

Y_{1}\mid Y_{2}=y_{2}\sim\textsc{N}\left(\mu_{1}+\Sigma_{12}\Sigma_{22}^{-1}(y_{2}-\mu_{2}),\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\right).
30.A.3 \chi^{2} and t distributions

Assume X_{1},\ldots,X_{n} are IID \textsc{N}(0,1). Then

\sum_{i=1}^{n}X_{i}^{2}

follows a \chi^{2}_{n} distribution with degrees of freedom n. The \chi^{2}_{n} distribution has mean n and variance 2n.

Assume X\sim\textsc{N}(0,1),Q_{n}\sim\chi^{2}_{n} and X\ind Q_{n}. Then

\frac{X}{\sqrt{Q_{n}/n}}

follows a t_{n} distribution with degrees of freedom n. The t_{n} distribution has mean 0 if n>1. When n=1, the t_{1} distribution is also called the Cauchy distribution, which does not have a finite mean.

30.A.4 Cauchy–Schwarz inequality

The Cauchy–Schwarz inequality has many forms. With two random variables A and B, we have

|E(AB)|\leq\sqrt{E(A^{2})E(B^{2})}

with equality holding when B=\beta A for some \beta. Centering A and B to have mean 0, we have

|\textup{cov}(A,B)|\leq\sqrt{\textup{var}(A)\textup{var}(B)}

with equality holding when B=\alpha+\beta A for some \alpha and \beta.

When A and B are uniform random variables over finite sets \{a_{1},\ldots,a_{n}\} and \{b_{1},\ldots,b_{n}\} respectively, we have

\left|\sum_{i=1}^{n}a_{i}b_{i}\right|\leq\sqrt{\sum_{i=1}^{n}a_{i}^{2}\sum_{i=1}^{n}b_{i}^{2}}

with equality holding if there exists \beta such that b_{i}=\beta a_{i} for all i’s.

30.A.5 Tower property and variance decomposition

Given random variables or vectors A,B,C, we have

E(A)=E\{E(A\mid B)\}

and

E(A\mid C)=E\{E(A\mid B,C)\mid C\}.

Given a random variable A and random variables or vectors B,C, we have

\textup{var}(A)=E\{\textup{var}(A\mid B)\}+\textup{var}\{E(A\mid B)\}

and

\textup{var}(A\mid C)=E\{\textup{var}(A\mid B,C)\mid C\}+\textup{var}\{E(A\mid B,C)\mid C\}.

When I was in graduate school, my professors Carl Morris and Joe Blitzstein called this formula the Eve’s Law due to the “EVVE” form of the formula. They then went back to call the first tower property the Adam’s Law.

Similarly, we can decompose the covariance as

\textup{cov}(A_{1},A_{2})=E\left\{\textup{cov}(A_{1},A_{2}\mid B)\right\}+\textup{cov}\{E(A_{1}\mid B),E(A_{2}\mid B)\}

and

\textup{cov}(A_{1},A_{2}\mid C)=E\left\{\textup{cov}(A_{1},A_{2}\mid B,C)\mid C\right\}+\textup{cov}\{E(A_{1}\mid B,C),E(A_{2}\mid B,C)\mid C\}.
30.A.6 Limiting theorems
Definition 30.1 (convergence in probability)

A sequence of random variables (X_{n})_{n\geq 1} converges to X in probability, if for every \varepsilon>0, we have

\textup{pr}(|X_{n}-X|>\varepsilon)\rightarrow 0

as n\rightarrow\infty.

Definition 30.2 (convergence in distribution)

A sequence of random variables (X_{n})_{n\geq 1} converges to X in distribution, if

\textup{pr}(X_{n}\leq x)\rightarrow\textup{pr}(X\leq x)

for all continuity point x of \textup{pr}(X\leq x), as n\rightarrow\infty.

Convergence in probability is stronger than convergence in distribution. Definitions 30.1 and 30.2 are useful for stating the following two fundamental theorems on the sample average of independent and identically distributed (IID) random variables.

Theorem 30.1 (law of large numbers)

If X_{1},\ldots,X_{n}\stackrel{{\scriptstyle\textup{IID}}}{{\sim}}X with E|X|<\infty, then \bar{X}=n^{-1}\sum_{i=1}^{n}X_{i}\rightarrow E(X) in probability.

The law of large numbers in Theorem 30.1 states that the sample average is close to the population mean in the limit.

Theorem 30.2 (central limit theorem (CLT))

If X_{1},\ldots,X_{n}\stackrel{{\scriptstyle\textup{IID}}}{{\sim}}X with \textup{var}(X)<\infty, then

\frac{\bar{X}-E(X)}{\sqrt{\textup{var}(X)/n}}\rightarrow\textsc{N}(0,1)

in distribution.

The CLT in Theorem 30.2 states that the standardized sample average is close to a standard Normal random variable in the limit.

Theorems 30.1 and 30.2 assume IID random variables for convenience. There are also many laws of large numbers and CLTs for the sample mean of independent or weakly dependent random variable (Durrett,, 2019, e.g.,).

30.A.7 Delta method

The delta method is a power tool to derive the asymptotic Normality of nonlinear functions of an asymptotically Normal random vector. I review a special case of the delta method below.

Theorem 30.3 (delta method)

Assume \sqrt{n}(X_{n}-\mu)\rightarrow\textsc{N}(0,\Sigma) in distribution and the function g(x) has non-zero derivative \nabla g(\mu) at \mu. Then

\sqrt{n}\{g(X_{n})-g(\mu)\}\rightarrow\textsc{N}(0,(\nabla g(\mu))^{\mkern-1.5mu\mathsf{T}}\Sigma\nabla g(\mu))

in distribution.

I will omit the proof of Theorem 30.3. It is intuitive based on the first-order Taylor expansion:

g(X_{n})-g(\mu)\approx(\nabla g(\mu))^{\mkern-1.5mu\mathsf{T}}(X_{n}-\mu).

So \sqrt{n}\{g(X_{n})-g(\mu)\} is close to the linear transformation of \textsc{N}(0,\Sigma), which is \textsc{N}(0,(\nabla g(\mu))^{\mkern-1.5mu\mathsf{T}}\Sigma\nabla g(\mu)).

As illustrations, we can use the delta method to obtain the asymptotic Normality of the ratio and product.

Example 30.1 (asymptotic Normality for the ratio)

Assume

\sqrt{n}\begin{pmatrix}Y_{n}-\mu_{Y}\\
X_{n}-\mu_{X}\end{pmatrix}\rightarrow\textsc{N}\left(\begin{pmatrix}0\\
0\end{pmatrix},\begin{pmatrix}\sigma_{Y}^{2}&\sigma_{YX}\\
\sigma_{YX}&\sigma_{X}^{2}\end{pmatrix}\right) (154)

in distribution with \mu_{X}\neq 0. Apply Theorem 30.3 to obtain that

\sqrt{n}\left(\frac{Y_{n}}{X_{n}}-\frac{\mu_{Y}}{\mu_{X}}\right)\rightarrow\textsc{N}\left(0,\frac{\sigma_{Y}^{2}}{\mu_{X}^{2}}+\frac{\mu_{Y}^{2}\sigma_{X}^{2}}{\mu_{X}^{4}}-\frac{2\mu_{Y}\sigma_{YX}}{\mu_{X}^{3}}\right) (155)

in distribution. In the special case that X_{n} and Y_{n} are asymptotically independent with \sigma_{YX}=0, the asymptotic variance of Y_{n}/X_{n} simplifies to \sigma_{Y}^{2}/\mu_{X}^{2}+\mu_{Y}^{2}\sigma_{X}^{2}/\mu_{X}^{4}. I leave the details to Problem AD.2.

The asymptotic variance in Example 30.1 is a little cumbersome. An easier way to memorize it is based on the following approximation:

\frac{Y_{n}}{X_{n}}-\frac{\mu_{Y}}{\mu_{X}}=\frac{Y_{n}-\mu_{Y}/\mu_{X}\cdot X_{n}}{X_{n}}\approx\frac{Y_{n}-\mu_{Y}/\mu_{X}\cdot X_{n}}{\mu_{X}}, (156)

so the asymptotic variance of the ratio equals the asymptotic variance of

\frac{Y_{n}-\mu_{Y}/\mu_{X}\cdot X_{n}}{\mu_{X}},

which is a linear combination of Y_{n} and X_{n}. Slutsky’s theorem can make the approximation in (156) rigorous but it is beyond this book.

Example 30.2 (asymptotic Normality for the product)

Assume (154). Apply Theorem 30.3 to obtain that

\sqrt{n}\left(X_{n}Y_{n}-\mu_{X}\mu_{Y}\right)\rightarrow\textsc{N}\left(0,\mu_{Y}^{2}\sigma_{X}^{2}+\mu_{X}^{2}\sigma_{Y}^{2}+2\mu_{X}\mu_{Y}\sigma_{XY}\right) (157)

in distribution. In the special case that X_{n} and Y_{n} are asymptotically independent with \sigma_{YX}=0, the asymptotic variance of X_{n}Y_{n} simplifies to \mu_{Y}^{2}\sigma_{X}^{2}+\mu_{X}^{2}\sigma_{Y}^{2}. I leave the details to Problem AD.3.

Appendix 30.B Statistical inference

30.B.1 Point estimation

Assume that \theta is the parameter of interest. Oftentimes, the problem also contains other parameters not of interest, denoted by \eta. Statisticians call \eta the nuisance parameter. Based on the data, we can compute an estimator \hat{\theta}. Throughout this book, we take the frequentist’s perspective by assuming that \theta is a fixed number and \hat{\theta} is random due to the randomness of data. Two basic requirements for an estimator are below.

Definition 30.3 (unbiasedness)

The estimator \hat{\theta} is unbiased for \theta if

E(\hat{\theta})=\theta

for all possible values of \theta and \eta.

Definition 30.4 (consistency)

The estimator \hat{\theta} is consistent for \theta if

\hat{\theta}\rightarrow\theta

in probability as the sample size approaches to infinity, for all possible values of \theta and \eta.

Unbiasedness requires that the mean of the estimator is identical to the parameter of interest. Consistency requires that the estimator is close to the true parameter in the limit. Unbiasedness does not imply consistency, and consistency does not imply unbiasedness either. Unbiasedness can be restrictive because it is impossible even in some simple statistics problems. Consistency is often the basic requirement in most statistics problems.

30.B.2 Confidence interval

A point estimator \hat{\theta} is a random variable that differs from the true parameter \theta. Statisticians are often interested in finding an interval that covers the true parameter with a certain given probability. This interval is computed based on the data, and it is random.

Definition 30.5 (confidence interval)

A data-dependent interval [\hat{\theta}_{\textsc{l}},\hat{\theta}_{\textsc{u}}] is a confidence interval for \theta with coverage probability at least 1-\alpha if

\textup{pr}(\hat{\theta}_{\textsc{l}}\leq\theta\leq\hat{\theta}_{\textsc{u}})\geq 1-\alpha

for all possible values of \theta and \eta.

Definition 30.6 (asymptotic confidence interval)

A data-dependent interval [\hat{\theta}_{\textsc{l}},\hat{\theta}_{\textsc{u}}] is an asymptotic confidence interval for \theta with coverage probability at least 1-\alpha if

\textup{pr}(\hat{\theta}_{\textsc{l}}\leq\theta\leq\hat{\theta}_{\textsc{u}})\rightarrow 1-\alpha^{\prime},\qquad\text{ as }n\rightarrow\infty

with \alpha^{\prime}\leq\alpha, for all possible values of \theta and \eta.

A standard choice is \alpha=0.05. In Definitions 30.5 and 30.6, the coverage probabilities can be larger than the nominal level 1-\alpha. That is, the definitions allow for over-coverage but do not allow for under-coverage. With over-coverage, we say that the confidence interval is conservative. Of course, we hope the confidence interval to be as narrow as possible. Otherwise, the definition of the confidence interval can be arbitrary.

30.B.3 Hypothesis testing

Many applied problems can be formulated as testing a hypothesis:

H_{0}:\theta=0.

The decision rule \phi is a binary function of the data: \phi=1 if we reject H_{0}; \phi=0 if we fail to reject H_{0}. The type one error rate of the test is the probability of rejection if the null hypothesis holds. I review the definition below.

Definition 30.7 (type one error rate)

When H_{0} holds, define the type one error rate of the test \phi as the maximum value of the probability

\textup{pr}(\phi=1)

over all possible values of \theta and \eta.

A standard choice is to make sure that the type one error rate is below \alpha=0.05. The type two error rate of the test is the probability of no rejection if the null hypothesis does not hold. I review the definition below.

Definition 30.8 (type two error rate)

When H_{0} does not hold, define the type two error rate of the test \phi as the maximum value of the probability

\textup{pr}(\phi=0)

over all possible values of \theta and \eta.

Given the control of the type one error rate under H_{0}, we hope the type two error rate is as low as possible when H_{0} does not hold.

30.B.4 Wald-type confidence interval and test

Many statistics problems have the following structure. The parameter of interest is \theta. We first find a consistent estimator \hat{\theta} that converges in probability to \theta, and show that it is asymptotically Normal with mean \theta and variance v which may depend on \theta as well as the nuisance parameter \eta. We then find a consistent estimator \hat{v} for v, based on analytic formulas or the bootstrap reviewed in Chapter 30.F later. The square root of \hat{v} is called the standard error. We finally construct the Wald-type confidence interval for \theta as

\hat{\theta}\pm z_{1-\alpha/2}\sqrt{\hat{v}}

where z_{1-\alpha/2} is the 1-\alpha/2 upper quantile of the standard Normal random variable. It covers \theta with probability approximately 1-\alpha. When this interval excludes a particular c, for example, c=0, we reject the null hypothesis H_{0}(c):\theta=c, which is called the Wald test.

30.B.5 Duality between constructing confidence sets and testing null hypotheses

Consider the statistical inference problem for a scalar parameter \theta. A fundamental result in statistics is that constructing confidence sets for \theta is equivalent to testing null hypotheses about \theta. This is often called the duality between constructing confidence sets and testing null hypotheses.

Section 30.B.4 has reviewed the duality based on the Wald-type confidence interval and test. The duality also holds in general. Assume that \hat{\Theta} is a (1-\alpha)-level confidence set for \theta:

\textup{pr}(\theta\in\hat{\Theta})\geq 1-\alpha.

Then we can reject the null hypothesis H_{0}(c):\theta=c if c is not in the set \hat{\Theta}. This is a valid test because when \theta indeed equals c, we have the correct type one error rate \textup{pr}(\theta\not\in\hat{\Theta})\leq\alpha. Conversely, if we test a sequence of null hypotheses H_{0}(c):\theta=c, we can obtain the corresponding p-values, p(c), as a function of c. Then the values of c that we fail to reject at level \alpha form a confidence set for \theta:

\hat{\Theta}=\{c:p(c)\geq\alpha\}=\{c:\text{ fail to reject }H_{0}(c)\text{ at level }\alpha\}.

It is a valid confidence set because

\textup{pr}(\theta\in\hat{\Theta})=\textup{pr}\{\text{fail to reject }H_{0}(\theta)\text{ at level }\alpha\}\geq 1-\alpha.

Here I use “confidence set” instead of “confidence interval” because \hat{\Theta} based on inverting tests may not be an interval. See the use of the duality in Chapters 30.D.2 and 15.1.

Appendix 30.C Inference with two-by-two tables

30.C.1 Fisher’s exact test

Fisher proposed an exact test for H_{0}:p_{1}=p_{0} under the statistical model:

n_{11}\sim\text{Binomial}(n_{1},p_{1}),\quad n_{01}\sim\text{Binomial}(n_{0},p_{0}),\quad n_{11}\ind n_{01}.

The table below summarizes the data.

1 0 row sum
sample 1 n_{11} n_{10} n_{1}
sample 0 n_{01} n_{00} n_{0}
column sum n_{\cdot 1} n_{\cdot 0} n

He argued that the sums n_{11}+n_{01}=n_{\cdot 1} and n_{10}+n_{00}=n_{\cdot 0} contain little information about the difference between p_{1} and p_{0}, and conditional on them, n_{11} follows a Hypergeometric distribution that does not depend on the unknown parameter p_{1}=p_{0} under H_{0}:

\textup{pr}(n_{11}=k)=\frac{\binom{n_{\cdot 1}}{k}\binom{n-n_{\cdot 1}}{n_{1}-k}}{\binom{n}{n_{1}}}.

In R, the function fisher.test implements this test.

30.C.2 Estimation with two-by-two tables

Based on the model in Section 30.C.1, we can estimate the parameters p_{1} and p_{0} by sample frequencies:

\hat{p}_{1}=\frac{n_{11}}{n_{1}},\quad\hat{p}_{0}=\frac{n_{01}}{n_{0}}.

Therefore, we can estimate the risk difference, log risk ratio, and log odds ratio

rd \displaystyle= \displaystyle p_{1}-p_{0},
\displaystyle\log\textsc{rr} \displaystyle= \displaystyle\log\frac{p_{1}}{p_{0}},
\displaystyle\log\textsc{or} \displaystyle= \displaystyle\log\frac{p_{1}/(1-p_{1})}{p_{0}/(1-p_{0})}

by the sample analogs

\displaystyle\hat{\textsc{rd}} \displaystyle= \displaystyle\hat{p}_{1}-\hat{p}_{0},
\displaystyle\log\hat{\textsc{rr}} \displaystyle= \displaystyle\log\frac{\hat{p}_{1}}{\hat{p}_{0}},
\displaystyle\log\hat{\textsc{or}} \displaystyle= \displaystyle\log\frac{\hat{p}_{1}/(1-\hat{p}_{1})}{\hat{p}_{0}/(1-\hat{p}_{0})}=\log\frac{n_{11}n_{00}}{n_{10}n_{01}}.

Based on the asymptotic approximation (see Problem AD.5), the estimated variance for the above parameters are

\displaystyle\frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_{1}}+\frac{\hat{p}_{0}(1-\hat{p}_{0})}{n_{0}},
\displaystyle\frac{1-\hat{p}_{1}}{n_{1}\hat{p}_{1}}+\frac{1-\hat{p}_{0}}{n_{0}\hat{p}_{0}},
\displaystyle\frac{1}{n_{1}\hat{p}_{1}(1-\hat{p}_{1})}+\frac{1}{n_{0}\hat{p}_{0}(1-\hat{p}_{0})},

respectively. The log transformation above yields better Normal approximations because the risk ratio and odds ratio are always positive.

Appendix 30.D Two famous problems in statistics

30.D.1 Behrens–Fisher problem

Consider the two-sample problem with n_{1} units under the treatment and n_{0} units under the control, respectively. Assume the outcomes under the treatment \{Y_{i}:Z_{i}=1\} are IID from \textsc{N}(\mu_{1},\sigma_{1}^{2}) and the outcomes under the control \{Y_{i}:Z_{i}=0\} are IID from \textsc{N}(\mu_{0},\sigma_{0}^{2}), respectively. The goal is to test H_{0}:\mu_{1}=\mu_{0}.

Start with the easier case with \sigma_{1}^{2}=\sigma_{0}^{2}. Coherent with Chapter 3, let

\hat{\bar{Y}}(1)=n_{1}^{-1}\sum_{Z_{i}=1}Y_{i},\quad\hat{\bar{Y}}(0)=n_{0}^{-1}\sum_{Z_{i}=0}Y_{i}

denote the sample means of the outcomes under the treatment and control, respectively. A standard result is that

t_{\textup{equal}}=\frac{\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)}{\sqrt{\frac{n}{n_{1}n_{0}(n-2)}\left[\sum_{Z_{i}=1}\{Y_{i}-\hat{\bar{Y}}(1)\}^{2}+\sum_{Z_{i}=0}\{Y_{i}-\hat{\bar{Y}}(0)\}^{2}\right]}}\sim t_{n-2}.

Based on t_{\textup{equal}}, we can construct a test for H_{0}.

Now consider the more difficult case with possibly different \sigma_{1}^{2} and \sigma_{0}^{2}. The distribution of t_{\textup{equal}} is no longer t_{n-2}. Estimating the variances separately, we can also define

t_{\textup{unequal}}=\frac{\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)}{\sqrt{\frac{\hat{S}^{2}(1)}{n_{1}}+\frac{\hat{S}^{2}(0)}{n_{0}}}},

where

\hat{S}^{2}(1)=(n_{1}-1)^{-1}\sum_{Z_{i}=1}\{Y_{i}-\hat{\bar{Y}}(1)\}^{2},\quad\hat{S}^{2}(0)=(n_{0}-1)^{-1}\sum_{Z_{i}=0}\{Y_{i}-\hat{\bar{Y}}(0)\}^{2}

are the sample variances of the outcomes under the treatment and control, respectively. Unfortunately, the exact distribution of t_{\textup{unequal}} depends on the known variances. Testing H_{0} without assuming equal variances is the famous Behrens–Fisher problem. With large sample sizes n_{1} and n_{0}, the CLT ensures that t_{\textup{unequal}} is approximately \textsc{N}(0,1). So we can construct an approximate test for H_{0}. By duality, a large-sample Wald-type confidence interval for \mu_{1}-\mu_{0} is

\hat{\bar{Y}}(1)-\hat{\bar{Y}}(0)\pm z_{1-\alpha/2}\sqrt{\frac{\hat{S}^{2}(1)}{n_{1}}+\frac{\hat{S}^{2}(0)}{n_{0}}}

where z_{1-\alpha/2} is the 1-\alpha/2 upper quantile of the standard Normal random variable.

30.D.2 Fieller–Creasy problem

Consider the two-sample problem with n_{1} units under the treatment and n_{0} units under the control, respectively. Assume the outcomes under the treatment \{Y_{i}:Z_{i}=1\} are IID from \textsc{N}(\mu_{1},1) and the outcomes under the control \{Y_{i}:Z_{i}=0\} are IID from \textsc{N}(\mu_{0},1), respectively. The goal is to estimate \gamma=\mu_{1}/\mu_{0}. We can use \hat{\gamma}=\hat{\bar{Y}}(1)/\hat{\bar{Y}}(0) to estimate \gamma. However, the point estimator has a complicated distribution, which does not yield a simple procedure to construct the confidence interval for \gamma.

Fieller’s confidence interval can be formulated as inverting tests for a sequence of null hypotheses: H_{0}(c):\gamma=c. Under H_{0}(c), we have

\frac{\hat{\bar{Y}}(1)-c\hat{\bar{Y}}(0)}{\sqrt{1/n_{1}+c^{2}/n_{0}}}\sim\textsc{N}(0,1)

which motivates the confidence interval

\left\{c:\Big|\frac{\hat{\bar{Y}}(1)-c\hat{\bar{Y}}(0)}{\sqrt{1/n_{1}+c^{2}/n_{0}}}\Big|\leq z_{1-\alpha/2}\right\}

where z_{1-\alpha/2} is the 1-\alpha/2 upper quantile of the standard Normal random variable.

Appendix 30.E Monte Carlo method in statistics

The Monte Carlo method is a powerful tool in statistics. I will review its basic use in approximating expectations or averages, which is fundamental in understanding the idea of FRT introduced in Chapter 3.

If our goal is to calculate \theta=E\{g(Y)\} with Y being a random variable, we can simply draw IID samples Y_{1},\ldots,Y_{n} from the distribution of Y and obtain the moment estimator for \theta:

\hat{\theta}=n^{-1}\sum_{i=1}^{n}g(Y_{i}).

As a special case, Y is a uniform distribution over \{y_{1},\ldots,y_{N}\} and \theta=N^{-1}\sum_{i=1}^{N}g(y_{i}). We can draw n IID samples \{Y_{1},\ldots,Y_{n}\} from \{y_{1},\ldots,y_{N}\} to obtain the moment estimator \hat{\theta} defined above. This is called sampling with replacement, which is different from sampling without replacement reviewed in Chapter 32.

Appendix 30.F Bootstrap

It is often very tedious to derive the variance formulas for complex estimators. Efron, (1979) proposed the bootstrap as a general tool for variance estimation. There are many versions of the bootstrap (Davison and Hinkley,, 1997). In this book, we only need the most basic one: the nonparametric bootstrap, which will be simply called the bootstrap.

Consider the generic setting with

Y_{1},\ldots,Y_{n}\stackrel{{\scriptstyle\textup{IID}}}{{\sim}}Y,

where Y_{i} can be a general random element denoting the observed data for unit i. An estimator \hat{\theta} is a function of the observed data: \hat{\theta}=T(Y_{1},\ldots,Y_{n}). When T is a complex function, it may not be easy to obtain the variance or asymptotic variance of \hat{\theta}.

The uncertainty of \hat{\theta} is driven by the IID sampling of Y_{1},\ldots,Y_{n} from the true distribution. Although the true distribution is unknown, it can be well approximated by its empirical version

\hat{F}_{n}(y)=n^{-1}\sum_{i=1}^{n}I(Y_{i}\leq y),

when the sample size n is large. If we believe this approximation, we can simulate \hat{\theta} by sampling

(Y_{1}^{*},\ldots,Y_{n}^{*})\stackrel{{\scriptstyle\textup{IID}}}{{\sim}}\hat{F}_{n}(y).

Because \hat{F}_{n}(y) is a discrete distribution with mass 1/n on each observed data point, the simulation of \hat{\theta} reduces to the following procedure:

  1. 1.

    sample (Y_{1}^{*},\ldots,Y_{n}^{*}) from \{Y_{1},\ldots,Y_{n}\} with replacement;

  2. 2.

    compute \hat{\theta}^{*}=T(Y_{1}^{*},\ldots,Y_{n}^{*});

  3. 3.

    repeat the above two steps B times to obtain the bootstrap replicates \{\hat{\theta}^{*}_{1},\ldots,\hat{\theta}^{*}_{B}\}.

We can then approximate the (asymptotic) variance of \hat{\theta} by the sample variance of the bootstrap replicates:

\hat{V}_{\text{boot}}=(B-1)^{-1}\sum_{b=1}^{B}(\hat{\theta}^{*}_{b}-\bar{\theta}^{*})^{2},

where \bar{\theta}^{*}=B^{-1}\sum_{b=1}^{B}\hat{\theta}^{*}_{b}. The bootstrap confidence interval based on the Normal approximation is then

\hat{\theta}\pm z_{1-\alpha/2}\sqrt{\hat{V}_{\text{boot}}},

where z_{1-\alpha/2} is the 1-\alpha/2 upper quantile of the standard Normal random variable.

I use the following simple example to illustrate the idea of the bootstrap. With n=100 IID observations Y_{i}’s from \textsc{N}(1,1), the sample mean should be close to 1 with variance 1/100=0.01. Over 500 simulations, the classic variance estimator and the bootstrap variance estimator with B=200 both have average values close to 0.01.

> ## sample size
> n = 100
> ## number of Monte Carlo simulations
> MC = 500
> ## number of bootstrap replicates B
> n.boot = 200
> simulation = replicate(MC, {
+ Y = rnorm(n, 1, 1)
+ boot.mu.hat = replicate(n.boot, {
+ index = sample(1:n, n, replace = TRUE)
+ mean(Y[index])
+ })
+ c(mean(Y), var(Y)/n, var(boot.mu.hat))
+ })
> ## summarize the results
> apply(simulation, 1, mean)
[1] 0.997602961 0.010006303 0.009921895

Appendix 30.G Homework problems

AD.1 Independent but not IID data

Assume that the Y_{i}’s are independent with mean \mu_{i} and variances \sigma_{i}^{2} for i=1,\ldots,n. The parameter of interest is \mu=n^{-1}\sum_{i=1}^{n}\mu_{i}. Show that \hat{\mu}=n^{-1}\sum_{i=1}^{n}Y_{i} is unbiased for \mu and find its variance. Show that the usual variance estimator for IID data

\hat{v}=\{n(n-1)\}^{-1}\sum_{i=1}^{n}(Y_{i}-\hat{\mu})^{2}

is a conservative estimator for the variance of \hat{\mu} in the sense that

E(\hat{v})-\textup{var}(\hat{\mu})=\{n(n-1)\}^{-1}\sum_{i=1}^{n}(\mu_{i}-\mu)^{2}\geq 0.

Remark: Consider a simpler case with \mu_{i}=\mu and \sigma_{i}^{2}=\sigma^{2} for all i=1,\ldots,n. The sample mean is unbiased for \mu with variance \sigma^{2}/n. Moreover, an unbiased estimator for the variance \sigma^{2}/n is \hat{\sigma}^{2}/n=\hat{v}, where \hat{\sigma}^{2}=(n-1)^{-1}\sum_{i=1}^{n}(Y_{i}-\hat{\mu})^{2}. This problem states a more general result for the case with independent but not identically distributed observations. The result has an important implication for Theorem 7.1 for the matched-pairs experiment discussed in Chapter 7.

AD.2 Asymptotic Normality of ratio

Prove (155).

AD.3 Asymptotic Normality of product

Prove (157).

AD.4 Product of two independent Normals

Assume X\sim\textsc{N}(\mu_{X},\sigma_{X}^{2}),Y\sim\textsc{N}(\mu_{Y},\sigma_{Y}^{2}) and X\ind Y. Show that

\textup{var}(XY)=\sigma_{X}^{2}\sigma_{Y}^{2}+\mu_{X}^{2}\sigma_{Y}^{2}+\mu_{Y}^{2}\sigma_{X}^{2}.

Remark: This problem gives a non-asymptotic form of Example 30.2.

AD.5 Variance estimators in two-by-two tables

Use the delta method to show the variance estimators in Section 30.C.2.

AD.6 Fisher weighting

Assume that we have p independent unbiased estimators \hat{\theta}_{1},\ldots,\hat{\theta}_{p} for a common parameter \theta:

E(\hat{\theta}_{j})=\theta,\quad(j=1,\ldots,p).

The estimator \hat{\theta}_{j} has variance v_{j} (j=1,\ldots,p).

Construct a new estimator

\hat{\theta}=\sum_{j=1}^{p}w_{j}\hat{\theta}_{j}

with nonrandom constants w_{j}’s. Show that \hat{\theta} is unbiased for estimating \theta if \sum_{j=1}^{p}w_{j}=1. Show that the optimal w_{j}’s such that \hat{\theta} has the smallest variance under the constraint \sum_{j=1}^{p}w_{j}=1 are

w_{j}^{*}=\frac{1/v_{j}}{\sum_{j^{\prime}=1}^{p}1/v_{j^{\prime}}},\quad(j=1,\ldots,p).

Show that the resulting estimator \hat{\theta}^{*}=\sum_{j=1}^{p}w_{j}^{*}\hat{\theta}_{j} has variance

\textup{var}(\hat{\theta}^{*})=\frac{1}{\sum_{j=1}^{p}1/v_{j}}.

Remark: This is called Fisher weighting. The optimal weights are proportional to the inverses of the variances.

本书:A First Course in Causal Inference,作者 Peng Ding。原文:arXiv:2305.18793v2,按 CC BY 4.0 许可发布。本站(yuanens.com)把 arXiv 网页版改成分章的纯 HTML,把公式渲染成图片,方便在 Kindle 等电子书阅读器上看,内容没有改动。转换难免有错,以原文为准。本站与作者和出版社无关。