Consequences of Kolmogorov’s axioms

Proof. Finite additivity follows from countable additivity and \(\mathbb P(\varnothing)=0\). Apply it to the disjoint decompositions

\[ \Omega=A\mathbin{\dot\cup}A^c,\qquad A\cup B=A\mathbin{\dot\cup}(B\setminus A),\qquad B=(A\cap B)\mathbin{\dot\cup}(B\setminus A). \]

If \(A\subseteq B\), the last decomposition becomes \(B=A\mathbin{\dot\cup}(B\setminus A)\). Nonnegativity gives \(\mathbb P(B)\ge \mathbb P(A)\).

Measurable maps

Let \((S,\mathcal S)\) and \((T,\mathcal T)\) be measurable spaces.

Measurability of random vectors

Proof \((\Rightarrow)\). Fix \(j\in\{1,\ldots,d\}\). The projection \(\pi_j:\mathbb R^d\to\mathbb R\), \(\pi_j(x)=x_j\), is continuous.

With the Borel \(\sigma\)-algebras, \(\pi_j\) is measurable by Example 1.8 (ii).

If \(\mathbf X\) is measurable, then for every \(B\in\mathcal B(\mathbb R)\),

\[ X_j^{-1}(B)=\mathbf X^{-1}\!\left(\pi_j^{-1}(B)\right)\in\mathcal F. \]

Thus \(X_j\) is measurable for every \(j\).

Measurability of random vectors II

Proof \((\Leftarrow)\). Define

\[ \mathcal C=\{A\subseteq\mathbb R^d:\mathbf X^{-1}(A)\in\mathcal F\}. \]

Inverse images preserve complements and countable unions, and \(\mathbf X^{-1}(\mathbb R^d)=\Omega\). Thus \(\mathcal C\) is a \(\sigma\)-algebra.

For open intervals \(I_1,\ldots,I_d\), put \(R=\prod_{j=1}^d I_j\). Then

\[ \mathbf X^{-1}(R)=\bigcap_{j=1}^dX_j^{-1}(I_j)\in\mathcal F. \]

Open boxes \(I_1 \times \ldots \times I_d\) with rational endpoints form a countable basis of the topology on \(\mathbb R^d\), so they generate \(\mathcal B(\mathbb R^d)\). Hence \(\mathcal B(\mathbb R^d)\subseteq\mathcal C\), proving measurability.

The distribution of a random element

Let \(X:(\Omega,\mathcal F,\mathbb P)\to(S,\mathcal S)\) be a random element.

Proof that \(\mathbb P_X\) is a probability measure. Measurability of \(X\) gives \(X^{-1}(B)\in\mathcal F\) for every \(B\in\mathcal S\). Hence \(\mathbb P_X\) is well-defined, \(\mathbb P_X(B)\ge0\), and \(\mathbb P_X(S)=\mathbb P(\Omega)=1\).

For pairwise disjoint \(B_1,B_2,\ldots\in\mathcal S\), put \(A_n=X^{-1}(B_n)\). These events are pairwise disjoint, so countable additivity of \(\mathbb P\) gives

\[ \mathbb P_X\!\left(\bigcup_{n\ge1}B_n\right) =\mathbb P\!\left(\bigcup_{n\ge1}A_n\right) =\sum_{n\ge1}\mathbb P(A_n) =\sum_{n\ge1}\mathbb P_X(B_n). \]

Measurable transformations

Proof. Let \(C\in\mathcal T\). Since \(g\) is measurable, \(g^{-1}(C)\in\mathcal S\). Measurability of \(X\) therefore gives

\[ Y^{-1}(C)=(g\circ X)^{-1}(C)=X^{-1}\!\left(g^{-1}(C)\right)\in\mathcal F. \]

Thus \(Y\) is a random element. By the definitions of \(\mathbb P_Y\) and \(\mathbb P_X\),

\[ \mathbb P_Y(C)=\mathbb P\!\left(Y^{-1}(C)\right) =\mathbb P\!\left(X^{-1}\!\left(g^{-1}(C)\right)\right) =\mathbb P_X\!\left(g^{-1}(C)\right). \]

Example: Tossing a coin

Cumulative distribution functions

Right-continuity and the endpoint limits

Proof. Fix \(x\in\mathbb R\) and let \(h_n\searrow0\) with \(h_n>0\). By Proposition 1.5,

\[ X^{-1}((-\infty,x+h_n])\searrow X^{-1}((-\infty,x]) \quad\Longrightarrow\quad F_X(x+h_n)\searrow F_X(x). \]

For the endpoint limits, the same result gives

\[ \begin{aligned} X^{-1}((-\infty,-n])\searrow\varnothing &\quad\Longrightarrow\quad F_X(-n)\searrow0,\\ X^{-1}((-\infty,n])\nearrow\Omega &\quad\Longrightarrow\quad F_X(n)\nearrow1. \end{aligned} \]

The inclusion \(X^{-1}((-\infty,x])\subseteq X^{-1}((-\infty,y])\) for \(x\le y\) makes \(F_X\) nondecreasing. The integer limits therefore extend to \(x\to\pm\infty\).

Left limits and point probabilities

Proof. The bounded, nondecreasing function \(F_X\) has a left limit at every \(x\). Since \(X^{-1}((-\infty,x-1/n])\nearrow X^{-1}((-\infty,x))\), continuity of probability gives

\[ F_X(x^-)=\lim_{n\to\infty}\mathbb P(X\le x-1/n)=\mathbb P(X<x). \]

The second identity follows by additivity from the disjoint decomposition

\[ X^{-1}((-\infty,x])=X^{-1}((-\infty,x))\mathbin{\dot\cup}X^{-1}(\{x\}). \]

Interval probabilities from the CDF

Proof. The following are disjoint decompositions:

\[ \begin{aligned} X^{-1}((-\infty,b])&=X^{-1}((-\infty,a])\mathbin{\dot\cup}X^{-1}((a,b]),\\ X^{-1}((-\infty,b])&=X^{-1}((-\infty,a))\mathbin{\dot\cup}X^{-1}([a,b]). \end{aligned} \]

Apply additivity, use \(\mathbb P(X\le a)=F_X(a)\) and \(\mathbb P(X<a)=F_X(a^-)\), and rearrange.

Countability of CDF discontinuities

Proof. For each integer \(m\ge1\), define

\[ D_m=\{x\in\mathbb R:\mathbb P(X=x)\ge1/m\}. \]

There can be at most \(m\) points in \(D_m\): any \(m+1\) such disjoint singleton events would have total probability greater than \(1\).

Every atom belongs to some \(D_m\), so

\[ \{x\in\mathbb R:\mathbb P(X=x)>0\}=\bigcup_{m=1}^{\infty}D_m \]

is countable. Since \(F_X\) is nondecreasing and right-continuous, every discontinuity is a jump.

The generalized inverse

Proof. The endpoint limits make \(\{x\in\mathbb R:F(x)\ge u\}\) nonempty and bounded below. Put \(q=F^{-1}(u)\) and choose \(x_n\searrow q\) with \(F(x_n)\ge u\). Right-continuity gives

\[ F(q)=\lim_{n\to\infty}F(x_n)\ge u. \]

A prescribed probability distribution function

Proof. For \(u\in(0,1)\) and \(x\in\mathbb R\),

\[ Q(u)\le x\quad\Longleftrightarrow\quad u\le F(x). \]

The forward implication follows from \(F(x)\ge F(Q(u))\ge u\). The reverse implication follows from the definition of the infimum.

Thus \(Q^{-1}(({-\infty},x])=(0,1)\cap(0,F(x)]\) is Borel. These half-lines generate \(\mathcal B(\mathbb R)\), so \(Q\) is measurable. Moreover,

\[ \mathbb P(Q(U)\le x)=\mathbb P(U\le F(x))=F(x). \]

Taking complements also gives \(F(x)<u\Longleftrightarrow x<Q(u)\).

Continuity of quantiles

Proof. For \(0<u<v<1\),

\[ \{x\in\mathbb R:F(x)\ge v\}\subseteq\{x\in\mathbb R:F(x)\ge u\} \quad\Longrightarrow\quad Q(u)\le Q(v). \]

To prove left-continuity, let \(u_n\nearrow u\in(0,1)\). Then \(Q(u_n)\nearrow\ell\le Q(u)\), and

\[ F(\ell)\ge F(Q(u_n))\ge u_n \quad\Longrightarrow\quad F(\ell)\ge u. \]

The inverse relation gives \(Q(u)\le\ell\). Therefore

\[ \lim_{v\nearrow u}Q(v)=Q(u). \]

Discrete distributions

For finite support, the CDF is a step function with a jump of \(p_X(x)\) at each \(x\in\operatorname{supp}(X)\).

Radon–Nikodym derivatives

Proof at continuity points of \(f_X\). For \(h\ne0\),

\[ \frac{F_X(x+h)-F_X(x)}{h} =\frac1h\int_x^{x+h}f_X(t)\,dt \ \longrightarrow\ f_X(x). \]

The fundamental theorem for Lebesgue integrals gives continuity of \(F_X\) and the general almost-everywhere assertion. In particular,

\[ \mathbb P(X=x)=\int_{\{x\}}f_X(t)\,dt=0 \qquad\text{for every }x. \]

A density value is not a point probability: for \(U\sim U(0,1/2)\), \(f_U(x)=2\) on \((0,1/2)\).

Joint CDFs: one-sided limits

Proof. The lower rectangles satisfy

\[ S_{x+h_n\mathbf1}\searrow S_x, \qquad S_{x-h_n\mathbf1}\nearrow S_x^\circ. \]

Apply continuity of probability to their inverse images under \(\mathbf X\).

Continuity of a joint CDF

Proof. If the boundary has probability zero, the upper and lower limits on the previous slide both equal \(F_{\mathbf X}(x)\). For \(\|y-x\|_\infty<h\),

\[ F_{\mathbf X}(x-h\mathbf1) \le F_{\mathbf X}(y) \le F_{\mathbf X}(x+h\mathbf1). \]

For any \(\varepsilon>0\), choose \(h>0\) so that both bounds differ from \(F_{\mathbf X}(x)\) by less than \(\varepsilon\). Then \(|F_{\mathbf X}(y)-F_{\mathbf X}(x)|<\varepsilon\) whenever \(\|y-x\|_\infty<h\), proving continuity at \(x\).

Conversely, continuity gives \(\mathbb P_{\mathbf X}(S_x^\circ)=\mathbb P_{\mathbf X}(S_x)\), so the boundary has probability zero.

Independent events

For \(\mathbb P(B)>0\), this is equivalent to \(\mathbb P(A\mid B)=\mathbb P(A)\). However, the definition also applies when \(\mathbb P(B)=0\) or \(1\).

Disjoint events of positive probability are not independent:

\[ A\cap B=\varnothing,\quad \mathbb P(A)\mathbb P(B)>0 \quad\Longrightarrow\quad \mathbb P(A\cap B)=0\ne \mathbb P(A)\mathbb P(B). \]

An independence criterion

Proof. Put \(\alpha=\mathbb P(A\mid B)\) and \(\beta=\mathbb P(A\mid B^c)\).

Since \(A=(A\cap B)\mathbin{\dot\cup}(A\cap B^c)\),

\[ \mathbb P(A)=\alpha \mathbb P(B)+\beta \mathbb P(B^c). \]

If \(\alpha=\beta\), this gives \(\mathbb P(A)=\alpha\) and hence \(\mathbb P(A\cap B)=\mathbb P(A)\mathbb P(B)\).

Conversely, if \(A\) and \(B\) are independent, then

\[ \mathbb P(A\cap B^c)=\mathbb P(A)-\mathbb P(A\cap B)=\mathbb P(A)\mathbb P(B^c), \]

so \(\alpha=\beta=\mathbb P(A)\). The same calculation proves independence is preserved under complementation.

The \(\sigma\)-algebra generated by a finite family

Proof. The nonempty sets \(C_\varepsilon\) partition \(\Omega\). Their unions form a \(\sigma\)-algebra \(\mathcal G\): complements select the remaining sets, and countable unions are again unions of these sets.

Since \(A_i=\bigcup_{\varepsilon:\varepsilon_i=1}C_\varepsilon\), we have \(\sigma(\mathcal C)\subseteq\mathcal G\). Conversely, every \(C_\varepsilon\in\sigma(\mathcal C)\), so every union of these finitely many sets belongs to \(\sigma(\mathcal C)\).

Independence and joint CDFs

Proof. (i) ⇒ (ii). Take the events \(X_i^{-1}(({-\infty},x_i])\).

(ii) ⇒ (iii). Equality holds for \(B_i=(-\infty,x_i]\). With the other sets fixed, both sides are finite measures in \(B_i\). Uniqueness extends equality to all Borel sets, one coordinate at a time.

From rectangles to independence

Proof of Theorem 2.14, (iii) ⇒ (i).

Let \(I=\{i_1,\ldots,i_k\}\subseteq\{1,\ldots,n\}\) be nonempty, and choose \(A_i\in\sigma(X_i)\) for \(i\in I\).

By Definition 2.10, write \(A_i=X_i^{-1}(B_i)\) with \(B_i\in\mathcal B(\mathbb R)\). Set \(B_i=\mathbb R\) for \(i\notin I\).

Condition (iii) and \(\mathbb P_{X_i}(\mathbb R)=1\) give

\[ \begin{aligned} \mathbb P\!\left(\bigcap_{i\in I}A_i\right) &=\mathbb P\!\left(\mathbf X^{-1}(B_1\times\cdots\times B_n)\right)\\ &=\mathbb P_{\mathbf X}(B_1\times\cdots\times B_n)\\ &=\mathbb P_{X_1}(B_1)\cdots \mathbb P_{X_n}(B_n)\\ &=\mathbb P(A_{i_1})\cdots \mathbb P(A_{i_k}). \end{aligned} \]

This holds for every \(I\) and every choice of the events \(A_i\). Thus the \(\sigma\)-algebras \(\sigma(X_1),\ldots,\sigma(X_n)\) are independent, as required.

Joint probability mass functions

Proof. The union of the disjoint events \((X,Y)^{-1}(\{(x,y)\})\), \((x,y)\in D\) has probability one. Both identities follow thus from countable additivity.

Marginal probability mass functions

Proof. For fixed \(x\), the events \(X^{-1}(\{x\})\cap Y^{-1}(\{v\})\), \(v\in D_Y\), are disjoint. Their union is \(X^{-1}(\{x\})\cap Y^{-1}(D_Y)\). Since \(\mathbb P(Y^{-1}(D_Y))=1\), countable additivity gives

\[ p_X(x)=\mathbb P\bigl(X^{-1}(\{x\})\bigr) =\sum_{v\in D_Y}\mathbb P\bigl(X^{-1}(\{x\})\cap Y^{-1}(\{v\})\bigr). \]

Interchanging \(X\) and \(Y\) gives the second identity. The marginals are the row and column sums of a joint PMF table.

Conditional probability mass functions

The marginal identity proves normalization:

\[ \sum_{y\in\operatorname{supp}(Y)} p_{Y\mid X=x}(y) =\frac{\sum_{y\in\operatorname{supp}(Y)} p_{X,Y}(x,y)}{p_X(x)}=1. \]

For \(p_Y(y)>0\), the reverse conditional is

\[ p_{X\mid Y=y}(x):=\frac{p_{X,Y}(x,y)}{p_Y(y)}. \]

Independence and PMF factorization

Proof. Independence gives the identity for the events \(X^{-1}(\{x\})\) and \(Y^{-1}(\{y\})\).

Conversely, summing a factorized PMF over any Borel rectangle \(A\times B\) gives

\[ \begin{aligned} \mathbb P(X\in A,Y\in B) &=\sum_{x\in A\cap D_X}\sum_{y\in B\cap D_Y}p_X(x)p_Y(y)\\ &=\mathbb P(X\in A)\mathbb P(Y\in B). \end{aligned} \]

Joint probability density functions

Taking \(A=\mathbb R^2\) gives \(\iint_{\mathbb R^2}f_{X,Y}(x,y)\,dx\,dy=1\).

Marginal probability density functions

Proof. For every Borel subset \(A\subseteq\mathbb R\), Tonelli’s theorem gives

\[ \mathbb P(X\in A) =\int_A\left[\int_{\mathbb R}f_{X,Y}(x,y)\,dy\right]dx. \]

The inner integral is measurable and nonnegative, with total integral one. It is therefore finite almost everywhere and defines a density of \(X\). Interchanging the coordinates proves the claim for \(Y\).

Conditional probability density functions

Verification. For such \(x\), \(f_{Y\mid X=x}\) is measurable and nonnegative, and

\[ \int_{\mathbb R}f_{Y\mid X=x}(y)\,dy =\frac{\int_{\mathbb R}f_{X,Y}(x,y)\,dy}{f_X(x)}=1. \]

Conditional distributions from densities

Proof. Put \(D=\{x\in\mathbb R:0<f_X(x)<\infty\}\). Since \(\int_{\mathbb R}f_X(x)\,dx=1\), \(\mathbb P_X(D)=1\).

On \(D\), \(f_{Y\mid X=x}(y)f_X(x)=f_{X,Y}(x,y)\). Thus, by Tonelli,

\[ \begin{aligned} \int_A\left[\int_Bf_{Y\mid X=x}(y)\,dy\right]d\mathbb P_X(x) &=\int_{A\cap D}\int_Bf_{X,Y}(x,y)\,dy\,dx\\ &=\mathbb P(X\in A,Y\in B). \end{aligned} \]

Independence and density factorization

Proof. Put \(g(x,y)=f_X(x)f_Y(y)\). For Borel subsets \(A,B\subseteq\mathbb R\), Tonelli gives

\[ \iint_{A\times B}g(x,y)\,dx\,dy=\mathbb P_X(A)\mathbb P_Y(B). \]

Taking \(A=B=\mathbb R\) shows that \(g\) is a probability density.

(⇐) If \(f_{X,Y}=g\) almost everywhere, then \(\mathbb P(X\in A,Y\in B)=\mathbb P_X(A)\mathbb P_Y(B)\).

(⇒) If \(X\) and \(Y\) are independent, the measures with densities \(f_{X,Y}\) and \(g\) agree on Borel rectangles. This class is closed under finite intersections and generates \(\mathcal B(\mathbb R^2)\). By uniqueness of probability measures and of densities, \(f_{X,Y}=g\) almost everywhere.

Bayes’ formula

Proof. By the definition of conditional probability,

\[ \mathbb P(A\cap B)=\mathbb P(A\mid B)\mathbb P(B)=\mathbb P(B\mid A)\mathbb P(A). \]

Now, divide by \(\mathbb P(A)\).

Properties of expectation

We say that a property holds almost surely (a.s.), if it holds with probability one. Precisely, we say that \(X\le Y\) holds almost surely, if \(\mathbb P (\{\omega\in \Omega \mid X(\omega) \le Y(\omega) \}) = 1\). Likewise, we say that \(X = Y\) holds almost surely, if \(\mathbb P (\{\omega\in \Omega \mid X(\omega) = Y(\omega) \}) = 1\). By \([X]\), we denote the equivalence class of all random variables \(Y\) which are equal to \(X\) almost surely.

Comparison of \(L^p\) norms

Proof. The function \(t\mapsto t^{q/p}\) is concave on \([0,\infty)\). Jensen’s inequality applied to \(|X|^p\) gives \(\mathbb E(|X|^q)\le(\mathbb E(|X|^p))^{q/p}<\infty\). Taking \(q\)th roots proves the claim.

Variance

Proof. Expand the square and apply linearity for the first identity. The second follows from \(aX+b-\mathbb E(aX+b)=a(X-\mathbb E(X))\).

Finally, the nonnegative variable \((X-\mathbb E(X))^2\) has expectation zero if and only if it is zero almost surely.

Covariance

Proof. Expand the product, use linearity, and apply Cauchy–Schwarz to the centered variables.

Covariance and inner products

Proof. Cauchy–Schwarz gives \(\mathbb E(|XY|)<\infty\). Symmetry and bilinearity follow from the integral. Also, \(\langle X,X\rangle_2=\mathbb E(X^2)\ge0\), with equality exactly when \(X=0\) almost surely, that is, \(X=0\) in \(L^2(\mathbb P)\).

For centered variables the covariance formula is \(\mathbb E(XY)\). On all of \(L^2(\mathbb P)\), covariance is not an inner product: \(\operatorname{Cov}(1,1)=0\) although \(1\ne0\).

Variance of a sum

Proof. We showed in Proposition 3.12 that the covariance is bilinear. This can be used to expand \(\mathbb E\!\left(\left(\sum_j(X_j-\mathbb E(X_j))\right)^2\right)\). The cross terms are twice the pairwise covariances.

Expectation of a product

Proof. By Theorem 2.14, the joint distribution is the product of the marginals. Tonelli’s theorem gives

\[ \begin{aligned} \mathbb E(|XY|) &=\int_{\mathbb R}\int_{\mathbb R}|xy|\,d\mathbb P_Y(y)\,d\mathbb P_X(x)\\ &=\mathbb E(|X|)\mathbb E(|Y|)<\infty. \end{aligned} \]

Fubini’s theorem therefore applies to the signed product:

\[ \mathbb E(XY) =\int_{\mathbb R}x\left(\int_{\mathbb R}y\,d\mathbb P_Y(y)\right)d\mathbb P_X(x) =\mathbb E(X)\mathbb E(Y). \]

Uncorrelatedness and independence

In each condition, the integral required to vanish is

\[ \int_\Omega \bigl(f(X)-\mathbb E(f(X))\bigr) \bigl(g(Y)-\mathbb E(g(Y))\bigr)\,d\mathbb P. \]

The integrability restriction in (ii) can instead be replaced by requiring \(f,g\) to be bounded.

Proof of the comparison

Proof of Proposition 3.17.

(i) If \(f(x)=ax+b\) and \(g(y)=cy+d\), Proposition 3.12 gives

\[ \operatorname{Cov}(f(X),g(Y))=ac\,\operatorname{Cov}(X,Y). \]

This vanishes for all \(a,b,c,d\) exactly when \(\operatorname{Cov}(X,Y)=0\), by taking \(a=c=1\) and \(b=d=0\).

(ii), forward implication. Measurability gives \(\sigma(f(X))\subseteq\sigma(X)\) and \(\sigma(g(Y))\subseteq\sigma(Y)\), so \(f(X)\) and \(g(Y)\) are independent. By Proposition 3.16, their covariance is zero.

Reverse implication. For any Borel sets \(A,B\subseteq\mathbb R\), choose the bounded and measurable functions \(f=\mathbf1_A\) and \(g=\mathbf1_B\). Then

\[ 0=\operatorname{Cov}(\mathbf1_A(X),\mathbf1_B(Y)) =\mathbb P(X\in A,Y\in B)-\mathbb P(X\in A)\mathbb P(Y\in B). \]

By Theorem 2.14, \(X\) and \(Y\) are independent.

Tail integrals for moments

Proof. Tonelli permits the interchange below because the integrand is nonnegative.

\[ \begin{aligned} \mathbb E(X^r) &=\int_\Omega X(\omega)^r\,d\mathbb P(\omega) =\int_\Omega\int_0^{X(\omega)}r t^{r-1}\,dt\,d\mathbb P(\omega)\\ &=\int_0^\infty\int_{\{\omega\in\Omega\mid X(\omega)>t\}}r t^{r-1}\,d\mathbb P(\omega)\,dt\\ &=r\int_0^\infty\mathbb P(\{\omega\in\Omega\mid X(\omega)>t\})t^{r-1}\,dt =r\int_0^\infty t^{r-1}\mathbb P(X>t)\,dt. \end{aligned} \]

Tail formulas for the mean

Both equalities allow the value \(+\infty\). The integral follows from \(r=1\) in Proposition 3.19. The sum follows by taking expectations in \(X=\sum_{k\ge1}\mathbf1_{[k,\infty)}(X)\).

Tail sums for higher moments

Proof. For \(t\in[k,k+1)\), \(\mathbb P(X>t)=\mathbb P(X>k)\). Splitting the nonnegative integral in Proposition 3.19 gives

\[ \begin{aligned} \mathbb E(X^r) &=\sum_{k=0}^\infty\mathbb P(X>k)\int_k^{k+1}r t^{r-1}\,dt\\ &=\sum_{k=0}^\infty\bigl((k+1)^r-k^r\bigr)\mathbb P(X>k). \end{aligned} \]

Expectation as a quantile integral

Proof. Put \(Q=F_X^{-1}\) and let \(U\) be uniform on \((0,1)\). By Proposition 1.25, \(Q\) is measurable and \(Q(u)\le x\) if and only if \(u\le F_X(x)\). Hence, for every \(x\in\mathbb R\),

\[ \mathbb P(Q(U)\le x)=\mathbb P(U\le F_X(x))=F_X(x). \]

Thus \(Q(U)\) and \(X\) have the same distribution. In particular, \(\mathbb E(|Q(U)|)=\mathbb E(|X|)<\infty\). Applying Theorem 3.2 to \(U\) and \(Q\) gives

\[ \mathbb E(X)=\mathbb E(Q(U))=\int_0^1Q(u)\,du. \]

The binomial distribution

The multinomial distribution

Counting interpretation. Roll an \(n\)-sided die \(d\) times independently. Write \(C_\ell\) for the outcome of roll \(\ell\), with \(\mathbb P(C_\ell=j)=p_j\) for each face \(j\). Let \(X_j\) count the occurrences of face \(j\).

Exactly \(d!/(x_1!\cdots x_n!)\) sequences have counts \(\mathbf x\). Each has probability \(p_1^{x_1}\cdots p_n^{x_n}\), giving the displayed formula.

Multinomial moments

Proof. Each of the \(d\) independent rolls shows face \(j\) with probability \(p_j\), so \(X_j\sim B(d,p_j)\) by Proposition 3.25.

For \(i\ne j\), \(X_iX_j\) counts ordered pairs of distinct rolls showing faces \(i\) and \(j\). Linearity and independence give

\[ \begin{aligned} \mathbb E(X_iX_j) &=\sum_{\substack{1\le\ell,m\le d\\\ell\ne m}}\mathbb P(C_\ell=i,C_m=j) =d(d-1)p_ip_j,\\ \operatorname{Cov}(X_i,X_j) &=d(d-1)p_ip_j-(dp_i)(dp_j)=-dp_ip_j. \end{aligned} \]

The Poisson distribution

Its mean and variance are both \(\lambda\):

\[ \mathbb E(X)=\mathbb V(X)=\lambda. \]

If \(X_j\sim\operatorname{Poisson}(\lambda_j)\) are independent, then

\[ \sum_{j=1}^nX_j\sim\operatorname{Poisson}\!\left(\sum_{j=1}^n\lambda_j\right). \]

The exponential distribution

Integration gives

\[ \mathbb P(X>x)=e^{-\lambda x}\quad(x\ge0), \qquad \mathbb E(X)=\frac1\lambda, \qquad \mathbb V(X)=\frac1{\lambda^2}. \]

In the call-arrival example below, \(1/\lambda\) is the mean waiting time.

Memorylessness

Proof. Since \(\mathbb P(X>s)=e^{-\lambda s}>0\),

\[ \mathbb P(X>s+t\mid X>s) =\frac{\mathbb P(X>s+t)}{\mathbb P(X>s)} =\frac{e^{-\lambda(s+t)}}{e^{-\lambda s}} =e^{-\lambda t}. \]

This is the survival probability \(\mathbb P(X>t)\) for every \(t\ge0\), so it determines the same distribution.

Moment generating functions

Some distributions, including the Cauchy distribution, have no finite MGF for any \(t\ne0\).

Affine linear maps and independent sums

Proof. Substitute \(aX+b\) in the definition and factor out \(e^{ibt}\). For the sum, independence gives

\[ \mathbb E\!\left(e^{it\sum_jX_j}\right) =\mathbb E\!\left(\prod_je^{itX_j}\right) =\prod_j\mathbb E(e^{itX_j}). \]

The corresponding MGF identities hold wherever all MGFs involved are finite.

Derivatives of characteristic functions

Proof. Since \(|(ix)^je^{itx}|=|x|^j\le1+|x|^k\), dominated differentiation under the integral is valid through order \(k\). Lebesgue’s dominated convergence theorem also shows that the derivatives are continuous. Set \(t=0\) for the final identity.

Almost sure convergence

Almost sure convergence: criterion

Proof of “\(\implies\)”. Suppose \(X_n\xrightarrow{\mathrm{a.s.}}X\). Outside a set \(A\) of probability zero, \(X_n(\omega)\) converges (pointwise) to \(X(\omega)\). For each \(\varepsilon>0\) and \(\omega \in \Omega\setminus A\), there can thus only be finitely many \(n\ge1\) such that \(|X_n(\omega)-X(\omega)| \ge \varepsilon\). Hence, any \(\omega \in \Omega\setminus A\) cannot lie in the limit superior of the sets \(\bigl(|X_n-X|\bigr)^{-1}((\varepsilon,\infty))\). Thus the limit superior is contained in \(A\) and must therefore have probability zero.

Almost sure convergence: converse

Proof. By the assumption and by countable subadditivity,

\[ A:=\bigcup_{m=1}^{\infty}\limsup_{n\to\infty}\bigl(|X_n-X|\bigr)^{-1}((1/m,\infty)) \qquad\text{satisfies }\mathbb P(A)=0. \]

For \(\omega\notin A\) and every \(m\), \(|X_n(\omega)-X(\omega)|\le1/m\) holds for all large enough \(n\). Hence \(X_n(\omega)\) converges pointwise to \(X(\omega)\) for \(\omega\notin A\), proving \(X_n\xrightarrow{\mathrm{a.s.}}X\).

The space \(L^\infty(\mathbb P)\)

\(L^\infty\) convergence

Proof. For each \(n\), choose a set \(A_n\) of probability zero outside which \(|X_n-X|\le\|X_n-X\|_\infty\). The union \(A=\bigcup_{n\ge1}A_n\) has probability zero.

For \(\omega\notin A\), we have \(|X_n(\omega)-X(\omega)|\le\|X_n-X\|_\infty\to0\), proving \(X_n\xrightarrow{\mathrm{a.s.}}X\).

Convergence in probability

Proof. For fixed \(\varepsilon>0\), the indicators \(\mathbf1_{(\varepsilon,\infty)}(|X_n-X|)\) converge to zero almost surely and are bounded by \(1\). Applying Lebesgue’s dominated convergence theorem, we see

\[ \mathbb P(|X_n-X|>\varepsilon) =\mathbb E\!\left(\mathbf1_{(\varepsilon,\infty)}(|X_n-X|)\right)\to0. \]

Markov’s inequality

Proof. The pointwise inequality \(f(\varepsilon)\mathbf1_{[\varepsilon,\infty)}(Z)\le f(Z)\) and monotonicity of expectation give

\[ f(\varepsilon)\,\mathbb P(Z\ge\varepsilon) =\mathbb E(f(\varepsilon)\mathbf1_{[\varepsilon,\infty)}(Z)) \le\mathbb E(f(Z)). \]

Now, divide by \(f(\varepsilon)>0\).

For \(f(t)=t\), this gives \(\mathbb P(Z\ge\varepsilon)\le\mathbb E(Z)/\varepsilon\).

Chebyshev’s inequality

Proof. Apply Proposition 4.10 to \(Z=|X-\mathbb E(X)|\), with \(f(t)=t^2\):

\[ \mathbb P\!\left(|X-\mathbb E(X)|\ge\varepsilon\right) \le\frac{\mathbb E(|X-\mathbb E(X)|^2)}{\varepsilon^2} =\frac{\mathbb V(X)}{\varepsilon^2}. \]

\(L^p\) and probability convergence

Proof. For every \(\varepsilon>0\), apply Proposition 4.10 with \(Z=|X_n-X|\) and \(f(t)=t^p\):

\[ \begin{aligned} \mathbb P(|X_n-X|>\varepsilon) &\le\frac{\mathbb E(f(|X_n-X|))}{f(\varepsilon)}\\ &=\frac{\mathbb E(|X_n-X|^p)}{\varepsilon^p}\longrightarrow0. \end{aligned} \]

Under \(L^p\) convergence the numerator tends to zero, proving convergence in probability.

Typewriter: \(L_p\) convergence without almost sure convergence

Proof. If \(Y_n=X_{m,k}\), then \(\|Y_n\|_p^p=\int_\Omega\mathbf1_{I_{m,k}}\,d\mathbb P=2^{-m}\).

Each row is finite, so \(m\to\infty\) as \(n\to\infty\). Hence \(\|Y_n\|_p\to0\). Convergence in probability follows from Proposition 4.12.

Fix \(\omega\in\Omega\). In every row \(m\ge1\), exactly one indicator equals \(1\) at \(\omega\), and the others equal \(0\). Let \(n_m(\omega)\) and \(r_m(\omega)\) be the indices in \((Y_n)\) of this \(1\) and the first \(0\) in that row. Both index sequences are strictly increasing, and

\[ Y_{n_m(\omega)}(\omega)=1,\qquad Y_{r_m(\omega)}(\omega)=0 \qquad(m\ge1). \]

Thus \((Y_n(\omega))\) does not converge for any \(\omega\in\Omega\).

Dominated convergence in \(L^p\)

Proof. Not examinable Step 1) For every \(\varepsilon>0\), the triangle inequality and \(|X_n|\le Y\) almost surely give

\[ \mathbb P(|X|>Y+\varepsilon) \le\mathbb P(|X-X_n|>\varepsilon)\longrightarrow0. \]

The probability on the left does not depend on \(n\), so it is zero. Taking \(\varepsilon=1/m\) for \(m=1,2,\ldots\) and using countable subadditivity gives \(|X|\le Y\) almost surely.

Thus \(\mathbb E(|X|^p)\le\mathbb E(Y^p)<\infty\), so \(X\in L^p(\mathbb P)\).

Dominated convergence: the \(L^p\) limit

Proof of Theorem 4.16. Not examinable Step 2) Put \(Z_n:=|X_n-X|^p\) and \(Z:=(2Y)^p\). Step 1 gives \(0\le Z_n\le Z\) almost surely, with \(\mathbb E(Z)<\infty\).

Fix \(\delta,M>0\). On \(Z\le M\), use \(Z_n\le\delta+M\mathbf1_{(\delta,\infty)}(Z_n)\). On \(Z>M\), use \(Z_n\le Z\). Taking expectations gives

\[ \mathbb E(Z_n)\le\delta+M\,\mathbb P(Z_n>\delta) +\mathbb E\!\left(Z\mathbf1_{(M,\infty)}(Z)\right). \]

By convergence in probability,

\[ \mathbb P(Z_n>\delta) =\mathbb P(|X_n-X|>\delta^{1/p})\longrightarrow0. \]

First take the limit superior as \(n\to\infty\). Then let \(M\to\infty\): the last expectation tends to zero by Lebesgue’s dominated convergence theorem. This yields \(\limsup_{n\to\infty}\mathbb E(Z_n)\le\delta\).

Since \(\delta>0\) is arbitrary, \(\mathbb E(|X_n-X|^p)\to0\), as required.

Weak convergence

Let \(S\) be a metric space and let \(C_b(S)\) denote the space of bounded continuous functions \(f\colon S\to \mathbb R\).

The random variables may be defined on different probability spaces.

Uniform continuity and continuity sets

The Portmanteau theorem

Portmanteau: closed sets

Proof of Theorem 4.20, (1) implies (2) implies (3). Not examinable The first implication follows because uniform continuity is more special than just continuity.

For a nonempty closed set \(F\subseteq S\) and integers \(m\ge1\), put

\[ d(x,F):=\inf_{y\in F}d(x,y),\qquad g_m(x):=\max\{1-m\,d(x,F),0\}. \]

Then \(|g_m(x)-g_m(y)|\le m\,d(x,y)\), so \(g_m\) is uniformly continuous. Also \(0\le g_m\le1\) and \(g_m\searrow\mathbf1_F\), since \(F\) is closed. By (2),

\[ \limsup_{n\to\infty}\mathbb P_n(F) \le\lim_{n\to\infty}\int_Sg_m\,d\mathbb P_n =\int_Sg_m\,d\mathbb P. \]

Let \(m\to\infty\). Lebesgue’s dominated convergence theorem makes the last integral tend to \(\mathbb P(F)\). For \(F=\varnothing\), the inequality is immediate.

Portmanteau: open and continuity sets

Proof of Theorem 4.20, (3) and (4) are equivalent. Not examinable If \(G\) is open, then \(G^c\) is closed, so (3) gives

\[ \liminf_{n\to\infty}\mathbb P_n(G) =1-\limsup_{n\to\infty}\mathbb P_n(G^c) \ge1-\mathbb P(G^c)=\mathbb P(G). \]

The same calculation with complements proves the reverse implication.

These conditions imply (5). For a Borel set \(A\), use \(A^\circ\subseteq A\subseteq\overline A\) to obtain

\[ \mathbb P(A^\circ) \le\liminf_{n\to\infty}\mathbb P_n(A) \le\limsup_{n\to\infty}\mathbb P_n(A) \le\mathbb P(\overline A). \]

If \(\mathbb P(\partial A)=0\), both outer terms equal \(\mathbb P(A)\). Hence \(\mathbb P_n(A)\to\mathbb P(A)\).

Portmanteau: level sets

Proof of Theorem 4.20, (5) implies (1). Not examinable Assume (5). Fix a continuous function \(h:S\to[0,1]\) and, for \(0<t<1\), define

\[ A_t:=h^{-1}((t,\infty)). \]

By continuity, a point with \(h(x)>t\) has a neighbourhood inside \(A_t\), while a point with \(h(x)<t\) has a neighbourhood outside \(A_t\). Thus

\[ \partial A_t\subseteq h^{-1}(\{t\}). \]

By Proposition 1.22, the distribution of \(h\) under \(\mathbb P\) has at most countably many atoms. For all other \(t\), \(\mathbb P(\partial A_t)=0\), so (5) gives

\[ \mathbb P_n(A_t)\longrightarrow\mathbb P(A_t). \]

Portmanteau: integrals of continuous functions

Proof of Theorem 4.20, conclusion. Not examinable With \(h\) and \(A_t\) as above, the tail integral formula Proposition 3.20 gives

\[ \int_Sh\,d\mathbb P_n=\int_0^1\mathbb P_n(A_t)\,dt \longrightarrow\int_0^1\mathbb P(A_t)\,dt=\int_Sh\,d\mathbb P. \]

The limit follows from Lebesgue’s dominated convergence theorem: the integrands lie in \([0,1]\) and converge outside a countable set, which has Lebesgue measure zero.

For general \(g\in C_b(S)\), choose \(M>0\) with \(|g|\le M\) and put \(h=(g+M)/(2M)\). Then \(h:S\to[0,1]\) is continuous and

\[ \int_Sg\,d\mathbb P_n=2M\int_Sh\,d\mathbb P_n-M \longrightarrow2M\int_Sh\,d\mathbb P-M=\int_Sg\,d\mathbb P. \]

This is weak convergence, completing all the equivalences.

Convergence in distribution: the CDF criterion

At a jump of \(F\), convergence of \(F_n(x)\) is not required.

Proof, forward implication. Suppose \(X_n\xrightarrow{\mathmakebox[1.5em]{d}}X\) and \(F\) is continuous at \(x\). By Proposition 1.18,

\[ \mathbb P_X(\partial(-\infty,x]) =\mathbb P_X(\{x\})=F(x)-F(x^-)=0. \]

Condition (5) in Theorem 4.20 gives \(F_n(x)=\mathbb P_{X_n}((-\infty,x])\to\mathbb P_X((-\infty,x])=F(x)\).

The CDF criterion: intervals

Proof of Theorem 4.21, converse. Assume \(F_n(x)\to F(x)\) at every continuity point of \(F\).

By Proposition 1.22, the discontinuities of \(F\) are countable. Choose a continuity point in each nonempty open interval with rational endpoints. Their set \(D\) is countable and dense in \(\mathbb R\).

For \(a,b\in D\) with \(a<b\),

\[ \begin{aligned} \mathbb P_{X_n}((a,b]) &=F_n(b)-F_n(a)\\ &\longrightarrow F(b)-F(a)=\mathbb P_X((a,b]). \end{aligned} \]

The same convergence holds for every finite union of such intervals. To see this, rewrite it as a disjoint union of finitely many half-open intervals with endpoints in \(D\), and use finite additivity.

Remark. Every open \(G\subseteq\mathbb R\) is a countable union of such intervals. Approximating by increasing finite unions gives \(\liminf_{n\to\infty}\mathbb P_{X_n}(G)\ge\mathbb P_X(G)\), so Theorem 4.20 yields \(X_n\xrightarrow{\mathmakebox[1.5em]{d}}X\).