Is OLS a Good Method?

We have now developed our basic method for estimating a simple linear regression model:

$$Y=\beta_0+\beta_1X+U.$$

As a recap, the unknown coefficients $\beta_0$ and $\beta_1$ describe the population model, while the error term $U$ captures the other factors affecting $Y$.

Given some sample data, OLS draws a line of best fit that matches the data we’ve collected as well as possible, and uses this to guess where the population line lies. That is: we use the intercept and slope of this “line of best fit” as our estimates of $\beta_0$ and $\beta_1.$

Now, the obvious question is: is this actually a good method?

The short answer is: sometimes. It depends!

OLS can work very well in some settings, but badly in others. Our goal in this section is to understand what makes the difference, as well as to formalise what “works well” even means.

Using Probability and Statistics

As we saw in the previous topic, before collecting the data, our OLS estimators are random variables:

$$\hat\beta_1=\frac{\sum_{i=1}^{n}(X_i-\bar X)(Y_i-\bar Y)}{\sum_{i=1}^{n}(X_i-\bar X)^2},$$

and

$$\hat\beta_0=\bar Y-\hat\beta_1\bar X.$$

This allows us to use probability and statistics to investigate how our estimates are likely to behave.

For example, we can ask whether they tend to be close to the true values, whether they are correct on average, and how much they vary.

The answers turn out to depend on how the data are generated, and on the relationships between the different parts of our simple linear model:

$$Y=\beta_0+\beta_1X+U.$$

The Gauss-Markov Assumptions

This brings us to the Gauss-Markov assumptions.

These are a set of conditions under which our OLS estimators have desirable properties, as understood in statistics. Over the next few pages, we will introduce them one at a time, explain precisely what they mean, and use graphs and simulations to visualise what can happen when they fail.

Several of them require paying close attention to the error term $U$, and especially to its relationship with the independent variable $X.$ This will turn out to be one of the main factors determining whether OLS estimates can be trusted.

Some of the assumptions are fairly innocuous. Others are much more demanding, and indeed often fail in real applications. If they do fail, another method may do better. However, often these more advanced methods are still recognisably based on a least squares approach – that is, “OLS with extra steps”.

Background

• Estimators as Random Variables
• Unbiasedness
• Efficiency
• Consistency
• Linear Models
• OLS Estimation
• Evaluating OLS