Recall what the least-squares line minimises
RSS=i=1∑n(yi−a−bxi)2 The vertical distances are squared, so points above and below the line both count against it.
Differentiate the residual sum of squares with respect to a
∂a∂RSS=−2i=1∑n(yi−a−bxi) Treat b as a constant and use the chain rule on each square.
Differentiate with respect to b
∂b∂RSS=−2i=1∑nxi(yi−a−bxi) Each square contributes a factor xi from the inner derivative.
Set both derivatives to zero
∑(yi−a−bxi)=0,∑xi(yi−a−bxi)=0 At a minimum both partial derivatives vanish; the RSS is a positive quadratic, so this stationary point IS the minimum.
Recall the normal equations
∑yi=na+b∑xi,∑xiyi=a∑xi+b∑xi2 These are what the two partial derivatives give when they are set to zero.
Divide the first normal equation by n
yˉ=a+bxˉ ⇒ a=yˉ−bxˉ This is why the least-squares line must pass through the mean point.
Eliminate a from the second normal equation
∑xiyi−yˉ∑xi+bxˉ∑xi−b∑xi2=0 Substituting a=yˉ−bxˉ leaves an equation in b alone.
Recognise the two sums of squares
Sxy−bSxx=0 ⇒ b=SxxSxy ∑xiyi−nxˉyˉ=Sxy and ∑xi2−nxˉ2=Sxx.
Find the means for these data
xˉ=20,yˉ=35 Now the general result can be evaluated.
Find Sxx
Sxx=2900−71402=100 Sxx is the total variation in x about xˉ.
Find Sxy
Sxy=4986−7(140)(245)=86 Sxy measures how x and y vary together; its sign is the sign of the gradient.
Find the gradient of the line of y on x
b=SxxSxy=10086=0.86 This is the least-squares gradient, obtained from the normal equations.
Find the intercept
a=yˉ−bxˉ=35−(0.86)(20)=17.8 Forcing the line through (xˉ,yˉ) fixes the intercept.
Recall the least-squares summary statistics
Sxx=∑x2−n(∑x)2,Sxy=∑xy−n(∑x)(∑y) Everything in linear regression is built from these two sums of squares.
Recall the least-squares gradient
b=SxxSxy Setting the partial derivatives of the residual sum of squares to zero gives this value of b.
Recall the least-squares intercept
a=yˉ−bxˉ The fitted line always passes through the mean point (xˉ,yˉ).
State the derived estimates
b=SxxSxy=0.86,a=yˉ−bxˉ=17.8 Both come from the normal equations, not from any rearrangement of them.