18 Least Squares Estimation

We wold like to describe the child’s height using the parent’s height, to do this we’ll use linear regression to do this.

Really, we want to find out two parameters that we do not know. Namely; The intercept \(\beta_0\) and the slope \(\beta_1\).

Let \(Y_i\) be he \(i^{th}\) child’s height and \(X_i\) be the \(i^{th}\) (average over the pair of) parent’s heights.
Consider finding the best line
- Child’s height = \(\beta_o\) + parent’s height \(\beta_1\)
Use least squares \[\sum_{i=1}^n [Y_i-(\beta_0 + \beta_1 X_i)]^2\]

18.1 Results

Using the least squares model fit to the line \(Y = \beta_0 + \beta_1 X\) though the data pairs \((X_i, Y_i)\) with \(Y_i\) as the outcome, obtains the line \(Y = \hat{\beta_0} + \hat{\beta_1}X\) where; \[\hat{\beta_1} = Cor(Y,X){Sd(Y) \over Sd(X)}\] \[\hat{\beta_0} = \bar{Y} -\hat{\beta_1} \bar{X}\]

\(\hat{\beta_1}\) has the units of \(Y/X\), \(\hat{\beta_0}\) has the units of \(Y\)
The line passes through the point \((\bar{X}, \bar{Y})\)
The slope of the regression line with \(X\) as the outcome

18.2 Linear Least Squares Coding Example

y <- galton$child
x <- galton$parent

beta1 <- cor(x,y)*sd(y)/sd(x)
beta0 <- mean(y) - beta1*mean(x)

rbind(c(beta0,beta1), coef(lm(y~x)))

##      (Intercept)         x
## [1,]    23.94153 0.6462906
## [2,]    23.94153 0.6462906

18.3 Normalising variables results in the slope being the correlation

# First, normalise both y and x:
yn <- (y - mean(y))/sd(y)
xn <- (x - mean(x))/sd(x)

# Now we can compare the cor of y,x, normalised y and x, and the output of the linear model coefficients for normalised y and x:
c(cor(y,x), cor(yn,xn), coef(lm(yn~xn))[2])

##                            xn 
## 0.4587624 0.4587624 0.4587624