What Is Statistics?

Statistics is about collecting and analysing data to draw conclusions and estimate unknown quantities, often in the real world.

This is rather different from probability theory. For instance, in probability theory, we would not estimate a quantity of interest, but rather begin with a probability model and use it to calculate it.

Probability and statistics therefore involve rather different kinds of activity, even if they are tightly related and are often taught together.

We use a simple example to see the distinction more clearly.

Let $X$ represent the outcome of rolling a fair die, and suppose we are interested in learning its expected value, $\mathbb{E}(X)$.

We will approach exactly the same question in two different ways.

The Probability Approach

Since we know that the die is fair, we know the probability of each possible outcome.

We can therefore calculate the expected value directly:

$$\mathbb{E}(X)=1\times\frac{1}{6}+2\times\frac{1}{6}+3\times\frac{1}{6}+4\times\frac{1}{6}+5\times\frac{1}{6}+6\times\frac{1}{6}=3.5.$$

This is the probability approach. We begin with a known probability distribution and use it to calculate the quantity we are interested in.

In this case,

$$\mathbb{E}(X)=3.5.$$

The Statistics Approach

Now suppose that, instead of doing this calculation, we try to learn about $\mathbb{E}(X)$ by collecting some data.

We roll the same fair die $100$ times. Let $x_i$ denote the outcome of the $i^{th}$ roll. Our data come out as follows:

$$x_1=2,\quad x_2=4,\quad x_3=2,\quad x_4=2,\quad\dots,\quad x_{98}=4,\quad x_{99}=4,\quad x_{100}=6.$$

A natural way to try to figure out $\mathbb{E}(X)$ is to take the average of these die rolls:

$$\frac{x_1+x_2+\dots+x_{100}}{100}=\frac{2+4+2+\dots+6}{100}=3.42.$$

In general, we call this kind of process estimation, and the resulting number an estimate. This sounds fancier than “guessing”!

So, in this case, our estimate of the expected value is

$$3.42.$$

Taking a “sample average” like this is one of our main techniques for estimating unknowns.

Probability and Statistics Together

From the probability approach, we happen to know that the true value is $3.5$. So, we see that our estimate is fairly close — but not exactly right.

If we rolled the die another $100$ times, we would probably get a different answer. In the real world, data are fickle like that! But it would also likely be close to $3.5$.

Of course, most of the time in statistics, we will not know the true answer to compare our estimate with. That is precisely when we resort to guessing with statistics – when we are unable to simply calculate the answer with probability!

Probability theory then returns in another role: we use it to investigate whether our statistical methods are likely to work well.