Linear Regression Basics

We begin our study of linear regression by carefully developing its central idea from scratch.

Suppose two variables are related in some unknown way, and we collect some data on them. We would like to use these data to learn about the underlying relationship. In particular, our goal is to estimate the unknown coefficients in a linear model expressing one in terms of the other – plus an error term.

We first look carefully at what such a model means. We then represent our data on a graph and introduce the familiar idea of a line of best fit, and make the important distinction between errors and residuals.

Of course, simply drawing a line that just looks about right is not a very satisfactory statistical method! We therefore introduce ordinary least squares (OLS), which gives us a precise rule for choosing the line.

Finally, we put the method into action and consider how its results should be interpreted and evaluated.

These basic ideas provide the foundation for almost everything that follows in linear regression.