The Variation in $X$ Assumption

Our third assumption says, roughly, that there must be some variation in the independent variable $X$.

This sounds straightforward enough. However, later we will need to consider two points of view; both after and before the $X_i$ have been observed. For this reason, the assumption comes in two slightly different flavours.

From the conditional point of view, we have already observed

$$X_1=x_1,\ldots,X_n=x_n,$$

and simply require that the observed numbers $x_i$ in our sample are not all the same.

From the unconditional point of view, the $X_i$ are still random variables. Now we instead require that the probability of obtaining a sample in which they are all the same is zero.

Below we consider these two versions in turn.

SLR3a: Variation in the Observed $x_i$

Once the $x_i$ have been observed, define the sample sum of squares for $X$:

$$SS_x=\sum_{i=1}^n(x_i-\bar{x})^2.$$

SLR3a: Sample Variation in the $x_i$ requires

$$SS_x>0.$$

This is equivalent to saying that the observed $x_i$ are not all the same.

Why do we need this? From a mathematical point of view, for precisely one reason!

Recall that the OLS slope estimate is

$$\hat\beta_1=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n}(x_i-\bar{x})^2},$$

If every $x_i$ is identical, then

$$SS_x=0,$$

so, since we can’t divide by $0$, the formula for $\hat\beta_1$ is not defined.

What Goes Wrong Without Variation?

The graph on the slide gives a more intuitive visual explanation of the same issue.

Suppose that, by coincidence, we survey 30 workers and every one of them has exactly 13.95 years of experience.

Our observations then form a vertical stack of points. There is plenty of variation in wages, but no variation at all in experience.

Since every worker has the same value $x=13.95$, any particular line gives the same fitted value to every worker: namely, the height at which that line crosses $x=13.95$.

The SSR is minimised when this common fitted value is $\bar{y}$. In other words, an OLS line must pass through

$$(\bar{x},\bar{y})=(13.95,\bar{y}).$$

But there are, of course, infinitely many different lines passing through this one point.

So, OLS has no way to choose a unique slope.

Intuitively, if everybody in our sample has the same amount of experience, the data cannot tell us anything about how wages change as experience changes.

SLR3b: Variation in the Random $X_i$

There is a second version of the assumption.

Before observing our sample, the quantities

$$X_1,\ldots,X_n$$

are still random variables.

We can therefore define the random variable

$$SS_X=\sum_{i=1}^n(X_i-\bar{X})^2.$$

SLR3b: Sample Variation in the $X_i$ requires

$$\mathbb{P}(SS_X=0)=0.$$

In words, before observing the data, the probability of obtaining a sample in which all the $X_i$ are identical must be zero.

If SLR3b holds, then with probability 1 we will obtain a sample satisfying SLR3a.

The importance of this is that, with probability 1, our OLS slope estimator will be well-defined:

$$\hat\beta_1=\frac{\sum_{i=1}^{n}(X_i-\bar{X})(Y_i-\bar{Y})}{\sum_{i=1}^{n}(X_i-\bar{X})^2},$$

This will be important when considering its expectation and variance!

For continuous distributions, SLR3b follows naturally from SLR2: Random Sampling. That is, the probability that several independent continuously distributed $X_i$ all take exactly the same value is always zero.

It need not always hold, though. For example, suppose the $X_i$ are independent Bernoulli random variables with probability $p\in(0,1)$. There is a positive probability that every observation is 1, and also a positive probability that every observation is 0.

The probability that all $n$ observations are 1 is

$$p^n,$$

while the probability that they are all 0 is

$$(1-p)^n.$$

So the total probability that our sample contains no variation in $X$ is

$$\mathbb{P}(SS_X=0)=p^n+(1-p)^n>0.$$

In any such sample, our OLS slope estimator $\hat\beta_1$ is undefined. This causes problems when considering its expectation.

Not a lot of people know that!

Background

• Random Variables
• Random Samples
• Evaluating OLS
• Conditioning on X
• SLR2: Random Sampling