Estimators and Unbiased Estimates

Scope Label

Core 9758. This branch develops the estimation side of Sampling and Estimation: what an estimator is, what unbiasedness means, and how to compute unbiased estimates of the population mean and variance.

Use it with the hub Sampling and Estimation and the sampling-design branch Sampling Methods.

Parameters, Statistics, Estimators, and Estimates

A population parameter is a numerical feature of the whole population. In H2 estimation, the main parameters are:

• population mean $\mu$;
• population variance ${\sigma }^2$.

These are fixed but usually unknown.

A sample statistic is computed from sample data. Examples are:

• sample mean $\bar{X}$ or observed value $\bar{x}$;
• sample variance $S^2$ or observed value $s^2$.

An estimator is a statistic used as a rule for estimating an unknown population parameter. A point estimate is the numerical value obtained from one observed sample.

For example:

• $\bar{X}$ is an estimator of $\mu$;
• after observing one sample, $\bar{x}=12.4$ may be a point estimate of $\mu$.

The distinction matters:

• an estimator is random before sampling;
• a point estimate is fixed after the sample has been observed.

How to read this figure. A parameter is fixed but unknown. An uppercase estimator is random before sampling; its lowercase realised value is the numerical point estimate obtained from one sample.

What Unbiasedness Means

An estimator is unbiased if its expected value equals the parameter it is estimating.

If $T$ estimates a population parameter $\theta$, then $T$ is unbiased if

$$E(T)=\theta .$$

Its bias is defined by

$$\mathrm{Bias}(T)=E(T)-\theta .$$

This is a long-run statement. It does not mean every estimate from every sample is correct.

It means:

• one sample may overestimate the parameter;
• another sample may underestimate it;
• across all possible random samples of the same size, the estimator is centred at the true parameter.

How to read this figure. Each dot is an estimate from one repeated random sample. Unbiasedness concerns the centre of the full estimator distribution, not whether any one dot equals $\theta$.

Bias Versus Variability

Unbiasedness is not the only property of an estimator.

Two estimators may both be unbiased, but one may fluctuate much more from sample to sample. The estimator with smaller variability is usually more useful in practice.

So estimator quality has two separate dimensions:

• bias: is the estimator centred correctly?
• variability: how widely does it fluctuate across repeated samples?

How to read this figure. Horizontal displacement of the estimate cloud from $\theta$ shows bias; its width shows sampling variability. Two estimators can share a centre but differ greatly in precision.

Unbiased Estimate of the Population Mean

For iid observations with finite mean, the estimator $\bar{X}$ satisfies $E(\bar{X})=\mu$. After observing data, the corresponding point estimate is $\bar{x}$.

For raw data $x_1,x_2,\dots ,x_n$,

$$\bar{x}=\frac{1}{n}\sum x.$$

For grouped or tabulated data with frequency $f$,

$$\bar{x}=\frac{\sum fx}{\sum f}.$$

This is conceptually natural:

• the population mean is the average of the population;
• the sample mean is the average of the observed sample;
• under proper random sampling, the sample mean is centred at the population mean.

Unbiased Estimate of the Population Variance

The population variance ${\sigma }^2$ measures spread in the whole population. For iid observations and $n\ge 2$, define the random estimator

$$S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2.$$

It satisfies $E(S^2)={\sigma }^2$. After data are observed, its realised value is the point estimate $s^2$.

For a sample, the unbiased estimate of ${\sigma }^2$ uses denominator $n-1$:

$$s^2=\frac{1}{n-1}\sum (x-\bar{x}{)}^2.$$

The computational form is often more convenient:

$$s^2=\frac{\sum x^2-\frac{(\sum x{)}^2}{n}}{n-1}.$$

For grouped data,

$$s^2=\frac{\sum f{x}^2-\frac{(\sum fx{)}^2}{n}}{n-1},n=\sum f.$$

Here $x$ denotes an observed value and $f$ its frequency. If $x$ is a class midpoint for interval-grouped data, the resulting mean and variance are approximations because the original within-class values are unknown.

The $n-1$ correction is needed because the deviations are measured from $\bar{x}$, a centre fitted from the same sample. This makes the deviations slightly too small on average if we divide by $n$.

How to read this figure. Fitting $\bar{x}$ to the same data forces $\sum (x_i-\bar{x})=0$, leaving $n-1$ free deviations. The $n-1$ denominator corrects the downward bias caused by division by $n$.

Why $n-1$ Appears

The sample mean $\bar{x}$ is chosen from the data. Once $\bar{x}$ is fixed, the deviations

$$x_i-\bar{x}$$

must satisfy

$$\sum (x_i-\bar{x})=0.$$

So the deviations cannot all vary freely. Informally, one degree of freedom has been used to estimate the centre. Only $n-1$ deviations are free.

That is why

$$\frac{1}{n}\sum (x-\bar{x}{)}^2$$

tends to underestimate ${\sigma }^2$, while

$$\frac{1}{n-1}\sum (x-\bar{x}{)}^2$$

is the unbiased estimator used in H2.

More precisely, under the iid model,

$$E\left[\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2\right]=\frac{n-1}{n}{\sigma }^2,$$

so multiplying by $n/(n-1)$—equivalently dividing the squared-deviation total by $n-1$—gives $E(S^2)={\sigma }^2$.

Worked Example 1: Raw Data

A random sample gives

$$6,8,9,10,12.$$

Find unbiased estimates of the population mean and variance.

First compute:

$$n=5,\sum x=45,\sum x^2=425.$$

Then

$$\bar{x}=\frac{45}{5}=9.$$

Also

$$s^2=\frac{425-\frac{45^2}{5}}{5-1}=\frac{425-405}{4}=5.$$

Therefore,

$$\bar{x}=9,s^2=5.$$

Worked Example 2: Summarised Data

A random sample of size $20$ gives

$$\sum x=314,\sum x^2=5128.$$

Find unbiased estimates of the population mean and variance.

The estimated population mean is

$$\bar{x}=\frac{314}{20}=\mathrm{15.7.}$$

The estimated population variance is

$$s^2=\frac{5128-\frac{314^2}{20}}{20-1}.$$

Since

$$\frac{314^2}{20}=4929.8,$$

we get

$$s^2=\frac{5128-4929.8}{19}=\frac{198.2}{19}\approx \mathrm{10.43.}$$

Thus

$$\bar{x}=15.7,s^2\approx \mathrm{10.43.}$$

Shifted-data check

The same calculation may be supplied in coded form. If $y=x-10$, then for this sample

$$\sum y=114,\sum y^2=848.$$

Hence

$$\bar{y}=\frac{114}{20}=5.7,\bar{x}=\bar{y}+10=15.7,$$

while a constant shift does not change variance:

$$s_x^2=s_y^2=\frac{848-114^2/20}{19}\approx \mathrm{10.43.}$$

Do not expand every original value unnecessarily: shifting changes the mean predictably and leaves variance unchanged.

Worked Example 3: Grouped Data

The following frequency table is obtained from a sample.

$x$	1	2	3	4	5
$f$	2	5	7	4	2

First,

$$n=\sum f=20.$$

Next,

$$\sum fx=2(1)+5(2)+7(3)+4(4)+2(5)=59,$$

and

$$\sum f{x}^2=2(1^2)+5(2^2)+7(3^2)+4(4^2)+2(5^2)=199.$$

Hence

$$\bar{x}=\frac{59}{20}=2.95,$$

and

$$s^2=\frac{199-\frac{59^2}{20}}{19}=\frac{199-174.05}{19}=\frac{24.95}{19}\approx \mathrm{1.31.}$$

Therefore,

$$\bar{x}=2.95,s^2\approx \mathrm{1.31.}$$

Calculator Discipline

Many graphing calculators can compute sample standard deviation directly. In H2 work, still check what the calculator output means:

• population standard deviation usually divides by $n$;
• sample standard deviation usually divides by $n-1$.

When estimating ${\sigma }^2$ from a sample, use the sample-variance convention with $n-1$.

Although $S^2$ is unbiased for ${\sigma }^2$, its square root $S$ is not generally an unbiased estimator of $\sigma$.

Common Pitfalls

• Treating “unbiased” as meaning “always exactly correct”.
• Forgetting that an estimator is a rule while a point estimate is one observed value.
• Using the $n$ denominator when the question asks for an unbiased estimate of population variance.
• Confusing sample standard deviation with population standard deviation on a calculator.
• Treating raw-data, summarised-data, and grouped-data formulas as unrelated methods.

Revision Checklist

• Can you distinguish a parameter, statistic, estimator, and point estimate?
• Can you explain unbiasedness as a repeated-sampling idea?
• Can you compute $\bar{x}$ from raw and grouped data?
• Can you compute $s^2$ from raw, summarised, and grouped data?
• Can you explain why $n-1$ appears in the variance estimate?
• Can you identify whether a calculator output divides by $n$ or by $n-1$?