What Is an Efficient Estimator?

The efficiency of an estimator relates to its variance.

When comparing estimators of the same unknown quantity, an estimator with a lower variance is said to be more efficient. In particular, given a choice among unbiased or equally biased estimators, we generally prefer the one with the smallest variance.

Efficiency is another desirable property of estimators, in addition to unbiasedness.

Accuracy and Precision

An unbiased estimator is accurate, or “correct on average”: its expected value is the true quantity we are trying to estimate. But this does not tell us how widely individual estimates are likely to vary around that value.

An estimator with a small variance tends to produce estimates that are more tightly grouped around its own mean. In this sense, variance measures precision rather than accuracy.

In particular, two estimators might both be unbiased, while one produces much more precise estimates than the other.

The archery analogy on the slide illustrates the distinction. An archer whose shots are centred on the bullseye is accurate on average, while an archer whose shots are tightly grouped is precise. Ideally, we would like both.

Example: Estimating a Population Mean

Let

$$Y_1,Y_2,\dots,Y_n$$

be a random sample from a population, with common mean

$$\mu=\mathbb{E}(Y_i)$$

and common variance

$$\sigma^2=\operatorname{Var}(Y_i).$$

The sample mean

$$\bar Y=\frac{Y_1+Y_2+\dots+Y_n}{n}$$

is an unbiased estimator of $\mu$:

$$\mathbb{E}(\bar Y)=\mu.$$

Now, consider a much less useful estimator, where we simply use the first observation $Y_1$ and ignore the rest.

Intuitively, this should be much less effective. However, it is also unbiased:

$$\mathbb{E}(Y_1)=\mu.$$

So unbiasedness alone cannot tell us which estimator is preferable.

To see the difference, compare their variances:

$$\operatorname{Var}(\bar Y)=\frac{\sigma^2}{n},\qquad \operatorname{Var}(Y_1)=\sigma^2.$$

For $n>1$, the sample mean has the smaller variance, and is therefore more efficient. This solidifies the intuition that using more observations gives us a better estimate of the population mean.

Efficiency Is Usually Comparative

We usually use efficiency comparatively: one estimator is more efficient than another because it has a smaller variance.

There is also a theoretical limit on how small the variance of an unbiased estimator can be. This can help us judge whether an estimator is making good use of the information contained in our sample. We’ll return to this later!

Bias and Variance

A lower variance is desirable, but it is not the only thing we care about.

Sometimes there is a trade-off between bias and variance. An estimator with a very small variance might nevertheless be systematically off target, while an unbiased estimator might vary considerably from sample to sample.

In some cases, we might prefer to tolerate a little bias in exchange for improved efficiency.

Background

• Variance and Standard Deviation

• Estimators as Random Variables