Statistical Inference

Week 6 · 2026 · Statistical models and statistics

Week 6

Statistical models

Sample spaces, parameter spaces and assumptions about distributions

Statistics

Functions of a sample and their sampling distributions

Sufficiency

Preserving information about a parameter and finding a minimal sufficient statistic

Exponential families

Natural parameters, completeness and preparation for estimation

Lecture 11

Week 6 · Statistical models and statistics

Models, sampling distributions and sufficient statistics.

Statistical models

A model specifies which probability distributions could describe the data.

Definition 11.1

A measurable sample space \((\mathcal X,\mathcal A)\) together with a family \(\mathcal P\) of probability measures on it is called a statistical model.

A model is called parametric if its distributions are described by a fixed finite number \(d\) of real parameters. It has the form \[ \mathcal P=\{P_\theta:\theta\in\Theta\},\qquad \Theta\subseteq\mathbb R^d,\quad d<\infty, \] where \(\Theta\) is called the parameter space.

A model is called nonparametric if its family of distributions is not described by a fixed finite number of free parameters.

The parameter \(\theta\) is fixed but unknown. The sample \(X\) is random with distribution \(P_\theta\).

Examples of parametric models

We now assume a sample of fixed size \(n\) with i.i.d. coordinates.

Bernoulli model

\(X_i\sim\operatorname{Bernoulli}(p)\), \(p\in(0,1)\). \[ \mathcal X=\{0,1\}^n,\qquad \Theta=(0,1). \] \[ f_p(x)=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i}. \]

Normal model

\(X_i\sim N(\mu,\sigma^2)\), \(\mu\in\mathbb R\), \(\sigma^2>0\). \[ \mathcal X=\mathbb R^n,\qquad \Theta=\mathbb R\times(0,\infty). \] \[ f_{\mu,\sigma^2}(x)=\prod_{i=1}^n f(x_i\mid\mu,\sigma^2). \]

Example: Measuring the speed of light

The newspaper style clipping in Eliana's slides, titled The Newcomb Michelson Velocity of Light Experiments.

Original optical apparatus diagram from Eliana's slides, showing the light path through lenses and mirrors.

A light pulse travels a known total path length \(d\). Its travel time is measured repeatedly.

The example contains 66 travel time measurements.

Let \(\mu>0\) be the true travel time. The corresponding speed is \(c=d/\mu\).

Model the measurements as

\[ X_i=\mu+\varepsilon_i, \]

where \(\varepsilon_i\) is the measurement error.

Two models for the measurement errors

For the travel time measurements \(X_i=\mu+\varepsilon_i\), assume independent errors with the same distribution.

A parametric model

\[ \varepsilon_i\sim N(0,\sigma^2). \]

The unknown parameters are \((\mu,\sigma^2)\), with \(\mu>0\) and \(\sigma^2>0\).

A nonparametric model

The error CDF \(F\) is continuous and symmetric about zero:

\[ F(u)=1-F(-u)\quad(u\in\mathbb R). \]

Both \(\mu\) and \(F\) are unknown.

Original plot of the 66 travel time measurements from Eliana's slides. The horizontal coordinate is measurement number. The vertical coordinate is deviation from 24800 nanoseconds.

Horizontal axis: measurement number. Vertical axis: deviation from 24800 nanoseconds.

Statistics

Definition 11.2

A measurable function \(T\) of the sample that does not depend on unknown parameters is called a statistic. It may be vector valued.

For \(n\ge2\), examples are \[ \overline X=\frac1n\sum_{i=1}^n X_i,\qquad S^2=\frac1{n-1}\sum_{i=1}^n(X_i-\overline X)^2. \]

But \(n^{-1}\sum_i(X_i-\mu)^2\) is not a statistic when \(\mu\) is unknown.

One distinguishes the map \(T\), the random value \(T(X)\), and the observed value \(T(x)\).

Sampling distributions

Definition 11.3

Under a specified sampling model, the probability distribution of \(T(X)\) is called its sampling distribution.

A Bernoulli sample

If \(X_i\overset{\mathrm{iid}}\sim\operatorname{Bernoulli}(p)\), then

\[ T=\sum_iX_i\sim\operatorname{Bin}(n,p). \]

The sample mean is \(\overline X=T/n\).

A normal sample

If \(X_i\overset{\mathrm{iid}}\sim N(\mu,\sigma^2)\), then

\[ \overline X\sim N\!\left(\mu,\frac{\sigma^2}{n}\right). \]

This holds exactly for every \(n\).

Example: Modelling the orbit of Mars

Side by side schematic comparison. Galilei is associated with a circular model, with the Sun at the centre. Kepler's elliptical model places the Sun at one focus. The diagrams show idealized orbit shapes, not observations.

Example: A statistical model for Mars

At a known time \(t_i\), let \(Y_i\) denote the measured longitude of Mars.

An orbital model predicts the longitude \(m_\theta(t_i)\). The unknown parameters \(\theta\) describe quantities such as the orbital period and orientation.

Allow the measured position to differ from the prediction:

\[ Y_i=m_\theta(t_i)+\varepsilon_i. \]

For example, assume \(\varepsilon_i\overset{\mathrm{iid}}\sim N(0,\sigma^2)\).

Then the observations are independent normal variables with common variance, but their means \(m_\theta(t_i)\) generally differ. Thus the observations need not be identically distributed.

The times are treated as fixed. The parameters are fixed but unknown and the responses are random. The Gaussian error distribution is an assumption, not a property established by the historical table. We use unwrapped angular coordinates. A simple circular prediction is m(t)=alpha+bt. For an ellipse, set M(t)=alpha+bt−omega, solve E−e sin E=M, convert eccentric anomaly E to true anomaly nu and form longitude omega+nu. Here e is eccentricity, the distance from the ellipse centre to a focus divided by the semimajor axis. The period is 2 pi/b. The code preserves complete revolutions. At zero eccentricity the circular model is recovered and omega is not identifiable. Coordinate reduction and timing uncertainty can create dependence or systematic errors not captured by the proposed error distribution.

Observed positions and fitted residuals

Let \(m(t_i)\) be a fitted prediction and \(y_i\) a recorded position.

The root mean square (RMS) residual over the ten fitting observations is

\[ \operatorname{RMS}=\sqrt{\frac1{10}\sum_{i=1}^{10}\bigl(y_i-m(t_i)\bigr)^2}. \]

Observed minus fitted residuals for circular and elliptical predictions. Filled dots are the ten observations from 1580 to 1600 used for fitting. Hollow diamonds show the later observations from 1602 and 1604, not used for fitting. The vertical axes have different scales.

Historical reduced data. Different vertical scales. Smaller residuals do not validate the error assumptions. Recorded values and fitted predictions

Sufficient statistics

Definition 11.4

Let \(X\) have distribution \(P_\theta\) on \((\mathcal X,\mathcal A)\).

A statistic \(T(X)\) is called sufficient if, for every \(A\in\mathcal A\), there is a measurable function \(q_A\), not depending on \(\theta\), such that for every \(\theta\),

\[ \mathbb P_\theta(X\in A\mid T(X))=q_A(T(X)) \quad P_\theta\text{ almost surely}. \]

Bernoulli model: Sufficiency

Let \(X_i\overset{\mathrm{iid}}\sim\operatorname{Bernoulli}(p)\), \(p\in(0,1)\), and \(T=\sum_iX_i\).

For \(t=0,\ldots,n\) and \(x\in\{0,1\}^n\) with \(\sum_ix_i=t\), \[ \mathbb P_p(X=x\mid T=t) =\frac{p^t(1-p)^{n-t}}{\binom nt p^t(1-p)^{n-t}} =\binom nt^{-1}. \] It is zero for vectors with a different count.

Proposition 11.5

The total number of successes is sufficient for \(p\). Conditional on the count, their arrangement is uniform and does not depend on \(p\).

The Neyman and Fisher factorization theorem

Theorem 11.6

Let \(f_\theta\) be the joint probability mass function or joint density of the sample. The counting measure or Lebesgue measure used for it is the same for every parameter.

A statistic \(T\) is sufficient if and only if there are nonnegative measurable functions \(g_\theta\) and \(h\) such that \[ f_\theta(x)=g_\theta(T(x))\,h(x) \quad\text{for almost every }x,\text{ for every }\theta. \] The factor \(h\) does not depend on \(\theta\). the other factor uses the sample only through \(T(x)\).

Factorization proof for a countable sample space

From factorization to sufficiency. Suppose \(f_\theta(x)=g_\theta(T(x))h(x)\) and put \[ H(t)=\sum_{z:T(z)=t}h(z). \] For \(T(x)=t\) and \(\mathbb P_\theta(T=t)>0\), \[ \mathbb P_\theta(X=x\mid T=t) =\frac{g_\theta(t)h(x)}{g_\theta(t)H(t)} =\frac{h(x)}{H(t)}. \]

Conversely. If \(T\) is sufficient with common conditional pmf \(k(x\mid t)\), take \[ g_\theta(t)=\mathbb P_\theta(T=t),\qquad h(x)=k(x\mid T(x)). \] Then \(f_\theta(x)=g_\theta(T(x))h(x)\).

A density ratio characterisation

Proposition 11.7

Assume the sample densities \(f_\theta\) have a common positive support. Let \(q_\theta\) be sampling densities of \(T\), with respect to one common measure.

Then \(T\) is sufficient if and only if \[ \frac{f_\theta(x)}{q_\theta(T(x))} \] does not depend on \(\theta\), wherever the denominator is positive.

Proof. If the ratio is \(r(x)\), then \(f_\theta(x)=q_\theta(T(x))r(x)\) is a factorization.

Conversely, factorization and a fixed reference parameter \(\theta_0\) give \(f_\theta(x)=a_\theta(T(x))f_{\theta_0}(x)\). Integrating over \(\{T\in B\}\) yields \(q_\theta(t)=a_\theta(t)q_{\theta_0}(t)\), so the ratio is independent of \(\theta\).

The sample densities are with respect to a common dominating measure and the statistic densities are with respect to another common measure, which need not have the same dimension. The equalities hold almost everywhere with coherent density versions. On a common positive support, divide a factorization g_theta(T)h by its value at theta0 to obtain a_theta=g_theta/g_theta0. For each measurable B, integrating the sample density over the inverse image of B gives the displayed relation between sampling densities. This avoids interpreting a density ratio as a Euclidean conditional density on a fibre. For discrete data, the ratio at T(x)=t is directly the conditional pmf of X given T=t. The support condition in this proposition is important. The factorization theorem remains applicable to the uniform model, whose support changes with theta.

Normal observations with known variance

Assume \(X_i\overset{\mathrm{iid}}\sim N(\mu,\sigma_0^2)\) with known \(\sigma_0^2>0\).

The identity \[ \sum_i(x_i-\mu)^2=\sum_i(x_i-\overline x)^2+n(\overline x-\mu)^2 \] follows by expansion and \(\sum_i(x_i-\overline x)=0\).

It gives the factorization

\[ f_\mu(x)= \underbrace{(2\pi\sigma_0^2)^{-n/2} \exp\!\left(-\frac{\sum_i(x_i-\overline x)^2}{2\sigma_0^2}\right)}_{h(x)} \underbrace{\exp\!\left(-\frac{n(\overline x-\mu)^2}{2\sigma_0^2}\right)}_{g_\mu(\overline x)}. \]

Thus \(\overline X\) is sufficient for \(\mu\).

Normal observations with both parameters unknown

Now \(X_i\overset{\mathrm{iid}}\sim N(\mu,\sigma^2)\), with both parameters unknown. Put \[ s(x)=\sum_i x_i,\qquad q(x)=\sum_i x_i^2. \] Then \[ f_{\mu,\sigma^2}(x)=(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{q(x)-2\mu s(x)+n\mu^2}{2\sigma^2}\right). \]

Proposition 11.8

\(T(X)=(\sum_iX_i,\sum_iX_i^2)\) is sufficient for \((\mu,\sigma^2)\).

The coefficient of the spread is unknown: it can no longer be absorbed into a factor that does not depend on the parameter.

A uniform sample with unknown endpoint

Let \(X_i\overset{\mathrm{iid}}\sim U(0,\theta)\), \(\theta>0\). Let \(x_{(1)}=\min_i x_i\) and \(x_{(n)}=\max_i x_i\).

The whole joint density is \[ f_\theta(x)=\theta^{-n}\, \mathbf1_{\{x_{(1)}>0\}}\, \mathbf1_{\{x_{(n)}<\theta\}}. \] Factor it as \[ g_\theta(m)=\theta^{-n}\mathbf1_{\{m<\theta\}},\qquad h(x)=\mathbf1_{\{x_{(1)}>0\}}. \] Thus the sample maximum \(M=X_{(n)}\) is sufficient for \(\theta\).

Order statistics: Sufficiency

Proposition 11.9

If \(X_i\) are i.i.d. from an unknown density \(f\), then the sorted vector \[ T(X)=(X_{(1)},\ldots,X_{(n)}) \] is sufficient for the density family.

Proof. For every candidate \(f\), \[ f_X(x)=\prod_{i=1}^n f(x_i)=\prod_{i=1}^n f(x_{(i)}). \] Apply factorization. The original order is discarded, but all \(n\) values remain.

This argument does not justify sorting responses while forgetting their different observation times.

Lecture 12

Week 6 · Minimality and completeness

The coarsest sufficient information. exponential structure. uniqueness from expectations.

Minimal sufficient statistics

Definition 12.1

A sufficient statistic \(T(X)\) is called minimal sufficient if, for every sufficient statistic \(U(X)\), there is a measurable function \(\varphi\) such that \[ T(X)=\varphi(U(X)) \quad P_\theta\text{ almost surely for every }\theta. \] The function \(\varphi\) may depend on \(U\), but not on \(\theta\).

Minimal sufficient statistics need not be unique. Measurable invertible transformations preserve the information retained.

Criterion for minimal sufficiency

Let \(f_\theta\) be the joint density or probability mass function of the sample under \(P_\theta\).

Theorem 12.2

Let \(T\) be sufficient. Suppose that, for all sample points \(x,y\), \[ T(x)=T(y) \quad\Longleftrightarrow\quad f_\theta(x)=c(x,y)f_\theta(y)\text{ for every }\theta, \] for some \(c(x,y)>0\) independent of \(\theta\).

Then \(T\) is minimal sufficient.

If the support depends on \(\theta\), the equality must also hold for parameter values at which one of the densities is zero.

Bernoulli model: Minimal sufficiency

For \(x,y\in\{0,1\}^n\), let \(t_x=\sum_i x_i\) and \(t_y=\sum_i y_i\). \[ \frac{f_p(x)}{f_p(y)} =\left(\frac{p}{1-p}\right)^{t_x-t_y}. \]

If \(t_x=t_y\), the ratio is \(1\).
If \(t_x\ne t_y\), the ratio varies as \(p\) ranges over \((0,1)\).

Together with sufficiency, the criterion proves:

Proposition 12.3

For an i.i.d. Bernoulli\((p)\) sample, \(\sum_iX_i\) is minimal sufficient. So is \(\overline X\), because \(n\) is known.

Normal model: Minimal sufficiency

Write \(s(x)=\sum_i x_i\) and \(q(x)=\sum_i x_i^2\), and similarly for \(y\). Then \[ \log\frac{f_{\mu,\sigma^2}(x)}{f_{\mu,\sigma^2}(y)} =\frac{\mu(s(x)-s(y))}{\sigma^2}-\frac{q(x)-q(y)}{2\sigma^2}. \]

Known variance. The ratio is constant in \(\mu\in\mathbb R\) exactly when \(s(x)=s(y)\). Thus \(\overline X\) is minimal sufficient.

Both parameters unknown. Varying \(\mu\) forces \(s(x)=s(y)\). Varying \(\sigma^2>0\) then forces \(q(x)=q(y)\). Conversely, if both equalities hold, the density ratio equals \(1\) for every parameter value.

Hence \((\sum_iX_i,\sum_iX_i^2)\) is minimal sufficient for \((\mu,\sigma^2)\).

Normal model: Equivalent statistics

For a normal sample of size \(n\ge2\), define the sample variance by \[ S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\overline X)^2. \]

The statistics \((\sum_iX_i,\sum_iX_i^2)\) and \((\overline X,S^2)\) determine one another: \[ \overline X=\frac{\sum_iX_i}{n},\qquad S^2=\frac{\sum_iX_i^2-(\sum_iX_i)^2/n}{n-1}, \] \[ \sum_iX_i=n\overline X,\qquad \sum_iX_i^2=(n-1)S^2+n\overline X^2. \]

Proposition 12.4

A measurable one to one transformation with measurable inverse preserves sufficiency and minimal sufficiency.

Uniform model: Minimal sufficiency

For a positive sample vector \(x\), the joint density is \[ f_\theta(x)=\theta^{-n}\mathbf1_{\{\theta>x_{(n)}\}}. \]

If \(x_{(n)}=y_{(n)}\), then \(f_\theta(x)=f_\theta(y)\) for every \(\theta\).
If \(x_{(n)}<y_{(n)}\), choose \(x_{(n)}<\theta<y_{(n)}\). Then \(f_\theta(x)>0\) but \(f_\theta(y)=0\).

Thus the proportionality condition in Theorem 12.2 holds exactly when \(x_{(n)}=y_{(n)}\).

Proposition 12.5

For an i.i.d. \(U(0,\theta)\) sample, \(X_{(n)}\) is minimal sufficient for \(\theta\).

Exponential families

Definition 12.6

A family whose support does not depend on \(\theta\) is called an exponential family if, for some integer \(k\ge1\), its densities can be written as \[ f(x\mid\theta)=h(x)c(\theta) \exp\!\left(\sum_{j=1}^k w_j(\theta)t_j(x)\right), \] where \(c(\theta)>0\), \(h(x)\ge0\), each \(w_j,t_j\) takes values in \(\mathbb R\), and each \(t_j\) is measurable.

The functions \(h\) and \(t=(t_1,\ldots,t_k)\) do not involve \(\theta\). The common support is \(\{x:h(x)>0\}\).

Here \(x\) denotes one observation. The coefficients \(w_j(\theta)\) are called natural parameters. Write \(w=(w_1,\ldots,w_k)\).

For i.i.d. \(X_1,\ldots,X_n\), the statistic \(T_n=\sum_{i=1}^n t(X_i)\) is called the canonical statistic.

Examples of exponential families: Binomial

Let \(Y\sim\operatorname{Bin}(m,p)\), where \(m\) is known and \(p\in(0,1)\). For \(y=0,\ldots,m\), \[ f(y\mid p) =\binom my p^y(1-p)^{m-y} =\underbrace{\binom my}_{h(y)}\, \underbrace{(1-p)^m}_{c(p)}\, \exp\!\left(\underbrace{\log\frac{p}{1-p}}_{w(p)}\, \underbrace{y}_{t(y)}\right). \]

The natural parameter is \(\eta=\log[p/(1-p)]\in\mathbb R\).

\[ p=\frac{e^\eta}{1+e^\eta},\qquad c(p)=(1+e^\eta)^{-m}. \]

For an i.i.d. sample \(X_i\sim\operatorname{Bin}(m,p)\), the canonical statistic is \(T_n=\sum_iX_i\).

Examples of exponential families: Normal

For a single \(N(\mu,\sigma^2)\) observation, \[ f(x\mid\mu,\sigma^2) =\underbrace{(2\pi\sigma^2)^{-1/2}e^{-\mu^2/(2\sigma^2)}}_{c(\mu,\sigma^2)} \exp\!\left(\frac{\mu}{\sigma^2}x-\frac{1}{2\sigma^2}x^2\right). \]

One choice is \(h(x)=1\) and \[ t_1(x)=x,\quad t_2(x)=x^2,\qquad w_1=\frac{\mu}{\sigma^2},\quad w_2=-\frac1{2\sigma^2}. \]

The two natural coordinates range over \(\mathbb R\times(-\infty,0)\) when both normal parameters are free. For an i.i.d. sample, the canonical statistic is \(T_n=(\sum_iX_i,\sum_iX_i^2)\).

Natural parameter spaces

Definition 12.7

For fixed \(h\) and \(t=(t_1,\ldots,t_k)\), define \[ Z(\eta)=\int_{\mathcal X}h(x)e^{\eta\cdot t(x)}\,d\nu(x),\qquad \mathcal N=\{\eta\in\mathbb R^k:0<Z(\eta)<\infty\}. \] The set \(\mathcal N\) is called the natural parameter space. Here \(\nu\) is counting measure or Lebesgue measure. For \(\eta\in\mathcal N\), put \(A(\eta)=\log Z(\eta)\).

Distribution of each \(X_i\) (i.i.d.)	Canonical statistic \(T_n\)	Natural parameter \(\eta\)	Natural parameter space \(\mathcal N\)
\(\operatorname{Bin}(m,p)\), fixed \(m\ge1\), \(0<p<1\)	\(\sum_iX_i\)	\(\log[p/(1-p)]\)	\(\mathbb R\)
\(\operatorname{Pois}(\lambda)\), \(\lambda>0\)	\(\sum_iX_i\)	\(\log\lambda\)	\(\mathbb R\)
Exponential with rate \(\lambda>0\)	\(\sum_iX_i\)	\(-\lambda\)	\((-\infty,0)\)
\(N(\mu,\sigma^2)\), both unknown	\((\sum_iX_i,\sum_iX_i^2)\)	\((\mu/\sigma^2,-1/(2\sigma^2))\)	\(\mathbb R\times(-\infty,0)\)

The allowed parameter set is \(\Omega=w(\Theta)\subseteq\mathcal N\). A family is called full if \(\Omega=\mathcal N\). It is called curved if \(\Omega\) is a smooth set of dimension less than \(k\).

The definition concerns the fixed representation h and t for one observation. Counting measure gives a sum for discrete models, while Lebesgue measure gives the displayed integral for continuous models. The normalized density is f_eta(x)=h(x) exp(eta dot t(x) minus A(eta)). For an i.i.d. sample of size n, the joint density has the factor exp(eta dot T_n minus n A(eta)), with carrier equal to the product of the h(X_i). Its normalizing integral is Z(eta) to the power n. Since n is a positive integer, this is positive and finite exactly when Z(eta) is positive and finite. Thus the natural parameter space is unchanged, while the canonical statistic is the sum over observations. The original family may use only the subset Omega=w(Theta) of all values for which the density exists.

For the binomial representation, h(x)=binomial(m,x) on {0,…,m} and Z(eta)=(1+exp(eta)) to the power m, which is positive and finite for every real eta. The parameter p is restricted to (0,1). For the Poisson representation, h(x)=1/x! on the nonnegative integers and Z(eta)=exp(exp(eta)), finite for every real eta. Here lambda is positive. For the exponential representation, h(x) is the indicator of x>0 and Z(eta)=minus 1/eta for eta<0. The integral diverges for eta greater than or equal to zero. The rate lambda is positive. For the normal representation, h=1 and t=(x,x squared). Completing the square gives Z(eta)=sqrt(pi divided by minus eta2) times exp(minus eta1 squared divided by four eta2), finite exactly when eta2<0. If eta2 is nonnegative, the integral on the whole real line diverges for every eta1. The inverse map is sigma squared=minus 1/(2 eta2), mu=minus eta1/(2 eta2).

The full family uses all of the natural parameter space. A curved family constrains the natural parameters to a smooth set of smaller dimension. No condition of nonzero geometric curvature is intended by this statistical terminology. Full and curved refer to the chosen canonical representation. A subfamily of full dimension need not exhaust the natural parameter space.

Sufficiency in exponential families

Theorem 12.8

Let \(X_1,\ldots,X_n\) be i.i.d. with density or probability mass function \[ f(x\mid\theta)=h(x)c(\theta) \exp\!\left(\sum_{j=1}^k w_j(\theta)t_j(x)\right), \] where \(h,t_1,\ldots,t_k\) do not depend on \(\theta\). Write \(t=(t_1,\ldots,t_k)\). Then the canonical statistic \(T_n=\sum_{i=1}^n t(X_i)\) is sufficient for \(\theta\).

Proof. The joint density is \[ f_\theta(x)=\left(\prod_{i=1}^n h(x_i)\right) c(\theta)^n\exp\!\left(\sum_{j=1}^k w_j(\theta)\sum_{i=1}^n t_j(x_i)\right). \] The first factor is independent of \(\theta\), so Theorem 11.6 applies.

Completeness

Definition 12.9

A statistic \(T\) is called complete if every measurable \(a\) satisfying \(\mathbb E_\theta(|a(T)|)<\infty\) for every \(\theta\) obeys \[ \mathbb E_\theta(a(T))=0\ \text{for every }\theta \quad\Longrightarrow\quad \mathbb P_\theta(a(T)=0)=1\ \text{for every }\theta. \]

Bernoulli model: Completeness

Let \(T\sim\operatorname{Bin}(n,p)\) with \(p\in(0,1)\). Suppose \[ 0=\mathbb E_p(a(T)) =\sum_{t=0}^n a(t)\binom nt p^t(1-p)^{n-t} \quad\text{for every }p. \] Divide by the positive factor \((1-p)^n\) and set \(z=p/(1-p)\): \[ \sum_{t=0}^n a(t)\binom nt z^t=0\qquad\text{for every }z>0. \] A polynomial vanishing on \((0,\infty)\) has every coefficient zero. Since \(\binom nt>0\), all \(a(t)=0\).

Therefore the Bernoulli count is complete and sufficient.

Completeness in exponential families

Theorem 12.10

For an exponential family, let \(\Omega=w(\Theta)\subseteq\mathcal N\) be the set of natural parameter values used by the model.

If \(\Omega\) contains a nonempty open rectangle in \(\mathbb R^k\), then the canonical statistic \[ T_n=\sum_{i=1}^n t(X_i) \] of an i.i.d. sample is complete and sufficient.

For the normal model with both parameters unknown, \(\Omega=\mathcal N=\mathbb R\times(-\infty,0)\). Thus \((\sum_iX_i,\sum_iX_i^2)\) is complete and sufficient.

A curved family has no such rectangle in \(\Omega\). This theorem does not decide completeness in that case.

The open rectangle is a sufficient condition, not a necessary one. The hypothesis concerns the actual allowed set, not the natural space of a larger family. A proof uses uniqueness of Laplace transforms. Choose an interior eta0 and let Q_eta be the distribution of the canonical sum. If a(T) is integrable and has expectation zero throughout the family, define finite positive measures with densities a positive part and a negative part relative to Q_eta0. Exponential tilting gives equal finite transforms in a neighbourhood of zero. Uniqueness makes the two measures equal, so a is zero Q_eta0 almost surely. The distributions Q_eta are mutually absolutely continuous and the conclusion holds for every parameter. The one parameter normal family with known variance also satisfies the open interval condition, so the mean is complete there.

Complete sufficient statistics

Theorem 12.11

Suppose the model admits a minimal sufficient statistic. Then every statistic that is complete and sufficient is minimal sufficient.

Completeness alone is not enough. In the Bernoulli model, \(T\equiv0\) is complete but not sufficient, since \[ \mathbb P_p(X_1=1\mid T=0)=p. \]

Completeness will give uniqueness in estimation. If two integrable functions of \(T\) have the same expectation for every parameter, their difference has expectation zero, so they agree almost surely.

A constant statistic is complete because expectation zero forces its only function value to vanish. It is not sufficient in a nontrivial Bernoulli model because conditioning on it leaves the parameter dependence unchanged. For the theorem, let M be minimal sufficient. Since T is sufficient, M=phi(T). For a bounded measurable b, sufficiency of M gives a common r(M)=E(b(T) given M). Completeness makes b(T)−r(phi(T)) zero almost surely throughout the model. Applying this to indicators of a countable determining family shows that T is measurable with respect to M, establishing minimality. Minimal sufficiency does not in general imply completeness. In the curved model N(mu,mu squared), mu>0, the sum and sum of squares remain minimal by the criterion for minimal sufficiency, but (1+1/n)S squared−mean squared has expectation zero for every mu without vanishing almost surely. At a vector of ones it equals minus one and stays nonzero on an open neighbourhood of positive normal probability. This converse is not needed as an additional classroom example. Uniqueness among functions of a complete statistic does not alone prove a variance optimum among all estimators.

Mars example: Sufficiency

For the Mars longitudes, assume \(Y_i=\alpha+bt_i+\varepsilon_i\), with fixed known times \(t_i\) and independent \(N(0,\sigma^2)\) errors. Here \(\alpha\) is the initial longitude and \(b\) the constant angular speed. Longitudes include completed revolutions.

\[ \begin{aligned} f_{\alpha,b,\sigma^2}(y) ={}&\underbrace{(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{n\alpha^2+2\alpha b\sum_i t_i+b^2\sum_i t_i^2}{2\sigma^2}\right)}_{\text{independent of }y}\\ &\quad\times\exp\!\left(-\frac{\sum_i y_i^2}{2\sigma^2} +\frac{\alpha}{\sigma^2}\sum_i y_i +\frac{b}{\sigma^2}\sum_i t_i y_i\right). \end{aligned} \]

The sums \(\sum_i t_i\) and \(\sum_i t_i^2\) are known constants, so they need not be included in the statistic.

In the factorization \(f_\theta(y)=h(y)g_\theta(T(y))\) from Theorem 11.6, we can choose:

Variance	\(h(y)\)	Sufficient statistic \(T(Y)\)
\(\sigma^2\) known	\(\exp[-\sum_i y_i^2/(2\sigma^2)]\)	\((\sum_iY_i,\sum_i t_iY_i)\)
\(\sigma^2\) unknown	\(1\)	\((\sum_iY_i,\sum_i t_iY_i,\sum_iY_i^2)\)

If \(\sigma^2\) is unknown, we additionally retain \(\sum_iY_i^2\) because its coefficient is unknown.

This is the Gaussian model associated with the simple constant angular speed prediction for Mars. Longitude is recorded on the real line, including full revolutions rather than resetting after a full turn. The initial longitude and angular speed are unknown parameters. The observation times are fixed and known, not random observations in this model. All their functions, including the sum of the times and the sum of squared times, are therefore already known. They still occur in the joint density, but do not need to be stored as components of a statistic of Y.

The displayed expression follows by multiplying the independent normal densities and expanding sum (y_i minus alpha minus b t_i) squared. The first factor contains only the parameters and known times, so it belongs in g_theta. For known variance, take h(y)=exp(minus sum y_i squared divided by twice the known variance). Dividing the displayed joint density by this h leaves a function of the sample only through sum y_i and sum t_i y_i. Thus g_theta is a function of those two statistic values, as required by the Neyman and Fisher factorization theorem. For unknown variance, the coefficient of sum y_i squared is an unknown parameter, so that factor cannot be put into h. Instead use h=1 and retain all three sums in T. The displayed factorization then gives sufficiency for alpha, b and sigma squared.

This is an exact mathematical statement for the assumed Gaussian statistical model, not an established property of the historical reduction process. The observations need not be identically distributed because their means depend on time. No sufficiency or completeness claim is made for the nonlinear ellipse fit. A fitted parameter vector does not automatically determine all the data dependence in the joint density.