Hard Further Maths Linear regression Questions

Challenging, exam-style Further Maths Linear regression questions with worked solutions. Stretch yourself on the hardest least-squares, regression-line, raw-data, summary-statistics problems.

least-squaresregression-lineraw-datasummary-statisticsresidual-sum-of-squaresresidual
Further Maths34 questionsStep-by-step solutions
Question 1
9 markschallenging
A chemist measures the initial rate of a reaction at several catalyst concentrations. For each observation, xx denotes the catalyst concentration (mmol per litre) and yy denotes the initial rate (micromol per second). Both variables are subject to random measurement error, but in this later specimen it is yy that has been measured and xx that must be estimated. For the original 55 observations the regression line of yy on xx is y=19.25+2.15xy = 19.25 + 2.15x and the regression line of xx on yy is x=7.9+0.43yx = -7.9 + 0.43y. A later specimen is found to have y=28y=28. Which line should be used to estimate the corresponding value of xx, and what estimate does it give?
Show worked solution

Worked solution

  1. Recall what the least-squares line minimises

    RSS=i=1n(yiabxi)2\text{RSS}=\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2

    The vertical distances are squared, so points above and below the line both count against it.

  2. Recall the regression line of xx on yy

    x=a+by,b=SxySyy,a=xˉbyˉx=a'+b'y,\qquad b'=\frac{S_{xy}}{S_{yy}},\qquad a'=\bar{x}-b'\bar{y}

    This line minimises the squared HORIZONTAL distances, so it is the one to use when xx is the response.

  3. Decide which variable is the response here

    y is known,x is to be estimatedy\ \text{is known},\qquad x\ \text{is to be estimated}

    The quantity being estimated is the one whose deviations should be squared.

  4. Choose the line that minimises the deviations in xx

    minimise (xiabyi)2  x on y\text{minimise}\ \sum\left(x_i-a'-b'y_i\right)^{2}\ \Rightarrow\ x\ \text{on}\ y

    That is the regression line of xx on yy.

  5. Write down the line of xx on yy

    x=7.9+0.43yx = -7.9 + 0.43y

    Its gradient is b=SxySyyb'=\frac{S_{xy}}{S_{yy}}, not 1b\frac{1}{b}.

  6. Substitute y=28y=28

    x^=7.9+0.43×28=4.14\hat{x}=-7.9+0.43\times28=4.14

    This is the required estimate.

  7. See what rearranging the wrong line would have given

    x=2819.252.154.07x=\frac{28-19.25}{2.15}\approx4.07

    A genuinely different answer, because the two lines are different lines.

  8. Explain why the two disagree

    bb=r2=0.9245<1b\,b'=r^{2}=0.9245<1

    The two lines coincide only when r2=1r^{2}=1; the weaker the correlation, the wider they open out.

  9. Confirm both lines meet at the mean point

    (xˉ,yˉ)=(5, 30)\left(\bar{x},\bar{y}\right)=\left(5,\ 30\right)

    They cross there and nowhere else.

  10. Recall the least-squares summary statistics

    Sxx=x2(x)2n,Sxy=xy(x)(y)nS_{xx}=\sum x^2-\frac{\left(\sum x\right)^2}{n},\qquad S_{xy}=\sum xy-\frac{\left(\sum x\right)\left(\sum y\right)}{n}

    Everything in linear regression is built from these two sums of squares.

  11. Recall the least-squares gradient

    b=SxySxxb=\frac{S_{xy}}{S_{xx}}

    Setting the partial derivatives of the residual sum of squares to zero gives this value of bb.

  12. Recall the least-squares intercept

    a=yˉbxˉa=\bar{y}-b\bar{x}

    The fitted line always passes through the mean point (xˉ,yˉ)\left(\bar{x},\bar{y}\right).

  13. Recall what the least-squares line minimises

    RSS=i=1n(yiabxi)2\text{RSS}=\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2

    The vertical distances are squared, so points above and below the line both count against it.

  14. Recall the definition of a residual

    ei=yiy^i=yi(a+bxi)e_i=y_i-\hat{y}_i=y_i-\left(a+bx_i\right)

    A residual is the signed vertical distance from a data point to the fitted line.

  15. Recall the computational form of the residual sum of squares

    RSS=SyySxy2Sxx\text{RSS}=S_{yy}-\frac{S_{xy}^{2}}{S_{xx}}

    Substituting the least-squares aa and bb into (yiabxi)2\sum\left(y_i-a-bx_i\right)^2 collapses it to this.

  16. State the line and the estimate

    x on y:x^=4.14x\ \text{on}\ y:\qquad \hat{x}=4.14

    Rearranging the line of yy on xx is the classic error and gives 4.074.07 instead.

Answer
x^=4.14\hat{x}=4.14
Question 2
9 markschallenging
A telecommunications engineer measures the strength of a signal received at several distances from a transmitter mast. For each observation, xx denotes the distance from the mast (km) and yy denotes the received signal strength (dB). The 77 paired observations are shown in the table. x20212225282930y76737368696665\begin{array}{c|ccccccc}x & 20 & 21 & 22 & 25 & 28 & 29 & 30\\\hline y & 76 & 73 & 73 & 68 & 69 & 66 & 65\end{array} The least-squares regression line of yy on xx is the line y=a+bxy=a+bx that minimises the residual sum of squares i=1n(yiabxi)2\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2. Every summary statistic is to be found exactly from the data; for these data every regression quantity is an exact terminating decimal, so give the answer exactly. The least-squares regression line of yy on xx for these data is y=93.750.95xy = 93.75 - 0.95x. Which statement about the residuals of this least-squares line is correct?
Show worked solution

Worked solution

  1. Recall the definition of a residual

    ei=yiy^i=yi(a+bxi)e_i=y_i-\hat{y}_i=y_i-\left(a+bx_i\right)

    A residual is the signed vertical distance from a data point to the fitted line.

  2. Recall the normal equations

    yi=na+bxi,xiyi=axi+bxi2\sum y_i=na+b\sum x_i,\qquad \sum x_iy_i=a\sum x_i+b\sum x_i^{2}

    These are what the two partial derivatives give when they are set to zero.

  3. Read off the first normal equation

    i=1n(yiabxi)=0\sum_{i=1}^{n}\left(y_i-a-bx_i\right)=0

    That sum IS ei\sum e_i, so the residuals must sum to zero for ANY least-squares line.

  4. Find the residual at x=20x=20

    e=76(93.750.95x)x=20=7674.75=1.25e=76-\left(93.75-0.95x\right)_{x=20}=76-74.75=1.25

    The residual is observed minus predicted.

  5. Find the residual at x=21x=21

    e=73(93.750.95x)x=21=7373.8=0.8e=73-\left(93.75-0.95x\right)_{x=21}=73-73.8=-0.8

    The residual is observed minus predicted.

  6. Find the residual at x=22x=22

    e=73(93.750.95x)x=22=7372.85=0.15e=73-\left(93.75-0.95x\right)_{x=22}=73-72.85=0.15

    The residual is observed minus predicted.

  7. Find the residual at x=25x=25

    e=68(93.750.95x)x=25=6870=2e=68-\left(93.75-0.95x\right)_{x=25}=68-70=-2

    The residual is observed minus predicted.

  8. Find the residual at x=28x=28

    e=69(93.750.95x)x=28=6967.15=1.85e=69-\left(93.75-0.95x\right)_{x=28}=69-67.15=1.85

    The residual is observed minus predicted.

  9. Find the residual at x=29x=29

    e=66(93.750.95x)x=29=6666.2=0.2e=66-\left(93.75-0.95x\right)_{x=29}=66-66.2=-0.2

    The residual is observed minus predicted.

  10. Find the residual at x=30x=30

    e=65(93.750.95x)x=30=6565.25=0.25e=65-\left(93.75-0.95x\right)_{x=30}=65-65.25=-0.25

    The residual is observed minus predicted.

  11. Check that the residuals sum to zero

    ei=(1.25)+(0.8)+(0.15)+(2)+(1.85)+(0.2)+(0.25)=0\sum e_i=\left(1.25\right)+\left(-0.8\right)+\left(0.15\right)+\left(-2\right)+\left(1.85\right)+\left(-0.2\right)+\left(-0.25\right)=0

    This is guaranteed by the first normal equation and catches arithmetic slips.

  12. Divide the first normal equation by nn

    yˉabxˉ=0\bar{y}-a-b\bar{x}=0

    The mean residual is zero, which is the same statement.

  13. Deduce that the line passes through the mean point

    yˉ=a+bxˉ\bar{y}=a+b\bar{x}

    So (xˉ,yˉ)\left(\bar{x},\bar{y}\right) lies on the line.

  14. Verify with the numbers

    93.75+(0.95)(25)=70=yˉ93.75+\left(-0.95\right)\left(25\right)=70=\bar{y}

    The mean point checks out exactly.

  15. Note that the residual sum of SQUARES is not zero

    ei2=9.750\sum e_i^{2}=9.75\neq0

    Minimising a sum of squares does not make it zero unless the points are collinear.

  16. State the correct conclusion

    ei=0and(xˉ,yˉ)=(25, 70) lies on the line\sum e_i=0\quad\text{and}\quad\left(\bar{x},\bar{y}\right)=\left(25,\ 70\right)\ \text{lies on the line}

    Both facts are consequences of the first normal equation.

Answer
ei=0, line through (25, 70)\sum e_i=0,\ \text{line through}\ \left(25,\ 70\right)
Question 3
9 markschallenging
A hydrologist compares the daily rainfall over a catchment with the flow measured in the river that drains it. For each observation, xx denotes the rainfall (mm) and yy denotes the river flow (cubic metres per second). For the 77 paired observations the summary statistics are n=7n=7, x=140\sum x=140, y=245\sum y=245, x2=2900\sum x^{2}=2900, y2=8655\sum y^{2}=8655 and xy=4986\sum xy=4986. The residual sum of squares is RSS(a,b)=i=1n(yiabxi)2\text{RSS}\left(a,b\right)=\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^{2}. Every summary statistic is to be found exactly from the data; for these data every regression quantity is an exact terminating decimal, so give the answer exactly. Which of the following correctly derives the least-squares estimates and gives their values for these data?
Show worked solution

Worked solution

  1. Recall what the least-squares line minimises

    RSS=i=1n(yiabxi)2\text{RSS}=\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2

    The vertical distances are squared, so points above and below the line both count against it.

  2. Differentiate the residual sum of squares with respect to aa

    RSSa=2i=1n(yiabxi)\frac{\partial\text{RSS}}{\partial a}=-2\sum_{i=1}^{n}\left(y_i-a-bx_i\right)

    Treat bb as a constant and use the chain rule on each square.

  3. Differentiate with respect to bb

    RSSb=2i=1nxi(yiabxi)\frac{\partial\text{RSS}}{\partial b}=-2\sum_{i=1}^{n}x_i\left(y_i-a-bx_i\right)

    Each square contributes a factor xix_i from the inner derivative.

  4. Set both derivatives to zero

    (yiabxi)=0,xi(yiabxi)=0\sum\left(y_i-a-bx_i\right)=0,\qquad \sum x_i\left(y_i-a-bx_i\right)=0

    At a minimum both partial derivatives vanish; the RSS is a positive quadratic, so this stationary point IS the minimum.

  5. Recall the normal equations

    yi=na+bxi,xiyi=axi+bxi2\sum y_i=na+b\sum x_i,\qquad \sum x_iy_i=a\sum x_i+b\sum x_i^{2}

    These are what the two partial derivatives give when they are set to zero.

  6. Divide the first normal equation by nn

    yˉ=a+bxˉ  a=yˉbxˉ\bar{y}=a+b\bar{x}\ \Rightarrow\ a=\bar{y}-b\bar{x}

    This is why the least-squares line must pass through the mean point.

  7. Eliminate aa from the second normal equation

    xiyiyˉxi+bxˉxibxi2=0\sum x_iy_i-\bar{y}\sum x_i+b\bar{x}\sum x_i-b\sum x_i^{2}=0

    Substituting a=yˉbxˉa=\bar{y}-b\bar{x} leaves an equation in bb alone.

  8. Recognise the two sums of squares

    SxybSxx=0  b=SxySxxS_{xy}-b\,S_{xx}=0\ \Rightarrow\ b=\frac{S_{xy}}{S_{xx}}

    xiyinxˉyˉ=Sxy\sum x_iy_i-n\bar{x}\bar{y}=S_{xy} and xi2nxˉ2=Sxx\sum x_i^{2}-n\bar{x}^{2}=S_{xx}.

  9. Find the means for these data

    xˉ=20,yˉ=35\bar{x}=20,\qquad \bar{y}=35

    Now the general result can be evaluated.

  10. Find SxxS_{xx}

    Sxx=290014027=100S_{xx}=2900-\frac{140^{2}}{7}=100

    SxxS_{xx} is the total variation in xx about xˉ\bar{x}.

  11. Find SxyS_{xy}

    Sxy=4986(140)(245)7=86S_{xy}=4986-\frac{\left(140\right)\left(245\right)}{7}=86

    SxyS_{xy} measures how xx and yy vary together; its sign is the sign of the gradient.

  12. Find the gradient of the line of yy on xx

    b=SxySxx=86100=0.86b=\frac{S_{xy}}{S_{xx}}=\frac{86}{100}=0.86

    This is the least-squares gradient, obtained from the normal equations.

  13. Find the intercept

    a=yˉbxˉ=35(0.86)(20)=17.8a=\bar{y}-b\bar{x}=35-\left(0.86\right)\left(20\right)=17.8

    Forcing the line through (xˉ,yˉ)\left(\bar{x},\bar{y}\right) fixes the intercept.

  14. Recall the least-squares summary statistics

    Sxx=x2(x)2n,Sxy=xy(x)(y)nS_{xx}=\sum x^2-\frac{\left(\sum x\right)^2}{n},\qquad S_{xy}=\sum xy-\frac{\left(\sum x\right)\left(\sum y\right)}{n}

    Everything in linear regression is built from these two sums of squares.

  15. Recall the least-squares gradient

    b=SxySxxb=\frac{S_{xy}}{S_{xx}}

    Setting the partial derivatives of the residual sum of squares to zero gives this value of bb.

  16. Recall the least-squares intercept

    a=yˉbxˉa=\bar{y}-b\bar{x}

    The fitted line always passes through the mean point (xˉ,yˉ)\left(\bar{x},\bar{y}\right).

  17. State the derived estimates

    b=SxySxx=0.86,a=yˉbxˉ=17.8b=\frac{S_{xy}}{S_{xx}}=0.86,\qquad a=\bar{y}-b\bar{x}=17.8

    Both come from the normal equations, not from any rearrangement of them.

Answer
b=0.86, a=17.8b=0.86,\ a=17.8
Question 4
9 markschallenging
A manufacturer records the annual maintenance cost of a machine against the number of hours it has been run. For each observation, xx denotes the running time (hundreds of hours) and yy denotes the maintenance cost (hundreds of pounds). The 88 paired observations are shown in the table. x1314171821242627y4446485051535355\begin{array}{c|cccccccc}x & 13 & 14 & 17 & 18 & 21 & 24 & 26 & 27\\\hline y & 44 & 46 & 48 & 50 & 51 & 53 & 53 & 55\end{array} The least-squares regression line of yy on xx is the line y=a+bxy=a+bx that minimises the residual sum of squares i=1n(yiabxi)2\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2. Every summary statistic is to be found exactly from the data; for these data every regression quantity is an exact terminating decimal, so give the answer exactly. Find the regression sum of squares SyyRSSS_{yy}-\text{RSS}, giving your answer exactly.
Show worked solution

Worked solution

  1. Add the xx values and the yy values

    x=160,y=400\sum x=160,\qquad \sum y=400

    These two totals give the means and appear in both SxxS_{xx} and SxyS_{xy}.

  2. Find the means

    xˉ=1608=20,yˉ=4008=50\bar{x}=\frac{160}{8}=20,\qquad \bar{y}=\frac{400}{8}=50

    The regression line must pass through this mean point.

  3. Find x2\sum x^{2}

    x2=3400\sum x^{2}=3400

    Squaring each xx and adding gives the sum needed for SxxS_{xx}.

  4. Find xy\sum xy

    xy=8138\sum xy=8138

    Multiplying each pair and adding gives the sum needed for SxyS_{xy}.

  5. Find y2\sum y^{2}

    y2=20100\sum y^{2}=20100

    This is the sum needed for SyyS_{yy}, the total variation in yy.

  6. Find SxxS_{xx}

    Sxx=340016028=200S_{xx}=3400-\frac{160^{2}}{8}=200

    SxxS_{xx} is the total variation in xx about xˉ\bar{x}.

  7. Find SyyS_{yy}

    Syy=2010040028=100S_{yy}=20100-\frac{400^{2}}{8}=100

    SyyS_{yy} is the total variation in yy about yˉ\bar{y}.

  8. Find SxyS_{xy}

    Sxy=8138(160)(400)8=138S_{xy}=8138-\frac{\left(160\right)\left(400\right)}{8}=138

    SxyS_{xy} measures how xx and yy vary together; its sign is the sign of the gradient.

  9. Find the gradient of the line of yy on xx

    b=SxySxx=138200=0.69b=\frac{S_{xy}}{S_{xx}}=\frac{138}{200}=0.69

    This is the least-squares gradient, obtained from the normal equations.

  10. Find the intercept

    a=yˉbxˉ=50(0.69)(20)=36.2a=\bar{y}-b\bar{x}=50-\left(0.69\right)\left(20\right)=36.2

    Forcing the line through (xˉ,yˉ)\left(\bar{x},\bar{y}\right) fixes the intercept.

  11. State the equation of the regression line of yy on xx

    y=36.2+0.69xy=36.2+0.69x

    Both coefficients are exact for these data.

  12. Find the residual sum of squares

    RSS=SyySxy2Sxx=100(138)2200=10095.22=4.78\text{RSS}=S_{yy}-\frac{S_{xy}^{2}}{S_{xx}}=100-\frac{\left(138\right)^{2}}{200}=100-95.22=4.78

    This closed form is what (yiabxi)2\sum\left(y_i-a-bx_i\right)^2 reduces to at the least-squares aa and bb.

  13. Subtract the residual sum of squares from the total

    SyyRSS=1004.78=95.22S_{yy}-\text{RSS}=100-4.78=95.22

    This is the part of the variation in yy that the regression accounts for.

  14. Check against the closed form

    Sxy2Sxx=(138)2200=95.22\frac{S_{xy}^{2}}{S_{xx}}=\frac{\left(138\right)^{2}}{200}=95.22

    The regression sum of squares is exactly the term subtracted in RSS=SyySxy2Sxx\text{RSS}=S_{yy}-\frac{S_{xy}^{2}}{S_{xx}}.

  15. Express it as a proportion

    SyyRSSSyy=95.22100=0.9522\frac{S_{yy}-\text{RSS}}{S_{yy}}=\frac{95.22}{100}=0.9522

    That proportion is r2r^{2}, which ties the two ideas together.

  16. Recall the least-squares summary statistics

    Sxx=x2(x)2n,Sxy=xy(x)(y)nS_{xx}=\sum x^2-\frac{\left(\sum x\right)^2}{n},\qquad S_{xy}=\sum xy-\frac{\left(\sum x\right)\left(\sum y\right)}{n}

    Everything in linear regression is built from these two sums of squares.

  17. State the regression sum of squares

    SyyRSS=95.22S_{yy}-\text{RSS}=95.22

    Total variation splits exactly into explained plus unexplained: Syy=(SyyRSS)+RSSS_{yy}=\left(S_{yy}-\text{RSS}\right)+\text{RSS}.

Answer
SyyRSS=95.22S_{yy}-\text{RSS}=95.22
Question 5
9 markschallenging
A food technologist measures the moisture content of pasta after several drying times. For each observation, xx denotes the drying time (minutes) and yy denotes the moisture content (%). The 55 paired observations are shown in the table. x810121416y3731292825\begin{array}{c|ccccc}x & 8 & 10 & 12 & 14 & 16\\\hline y & 37 & 31 & 29 & 28 & 25\end{array} The least-squares regression line of yy on xx is the line y=a+bxy=a+bx that minimises the residual sum of squares i=1n(yiabxi)2\sum_{i=1}^{n}\left(y_i-a-bx_i\right)^2. Every summary statistic is to be found exactly from the data; for these data every regression quantity is an exact terminating decimal, so give the answer exactly. Find the residual of largest magnitude, giving your answer exactly.
Show worked solution

Worked solution

  1. Add the xx values and the yy values

    x=60,y=150\sum x=60,\qquad \sum y=150

    These two totals give the means and appear in both SxxS_{xx} and SxyS_{xy}.

  2. Find the means

    xˉ=605=12,yˉ=1505=30\bar{x}=\frac{60}{5}=12,\qquad \bar{y}=\frac{150}{5}=30

    The regression line must pass through this mean point.

  3. Find x2\sum x^{2}

    x2=760\sum x^{2}=760

    Squaring each xx and adding gives the sum needed for SxxS_{xx}.

  4. Find xy\sum xy

    xy=1746\sum xy=1746

    Multiplying each pair and adding gives the sum needed for SxyS_{xy}.

  5. Find SxxS_{xx}

    Sxx=7606025=40S_{xx}=760-\frac{60^{2}}{5}=40

    SxxS_{xx} is the total variation in xx about xˉ\bar{x}.

  6. Find SxyS_{xy}

    Sxy=1746(60)(150)5=54S_{xy}=1746-\frac{\left(60\right)\left(150\right)}{5}=-54

    SxyS_{xy} measures how xx and yy vary together; its sign is the sign of the gradient.

  7. Find the gradient of the line of yy on xx

    b=SxySxx=5440=1.35b=\frac{S_{xy}}{S_{xx}}=\frac{-54}{40}=-1.35

    This is the least-squares gradient, obtained from the normal equations.

  8. Find the intercept

    a=yˉbxˉ=30(1.35)(12)=46.2a=\bar{y}-b\bar{x}=30-\left(-1.35\right)\left(12\right)=46.2

    Forcing the line through (xˉ,yˉ)\left(\bar{x},\bar{y}\right) fixes the intercept.

  9. State the equation of the regression line of yy on xx

    y=46.21.35xy=46.2-1.35x

    Both coefficients are exact for these data.

  10. Find the residual at x=8x=8

    e=37(46.21.35x)x=8=3735.4=1.6e=37-\left(46.2-1.35x\right)_{x=8}=37-35.4=1.6

    The residual is observed minus predicted.

  11. Find the residual at x=10x=10

    e=31(46.21.35x)x=10=3132.7=1.7e=31-\left(46.2-1.35x\right)_{x=10}=31-32.7=-1.7

    The residual is observed minus predicted.

  12. Find the residual at x=12x=12

    e=29(46.21.35x)x=12=2930=1e=29-\left(46.2-1.35x\right)_{x=12}=29-30=-1

    The residual is observed minus predicted.

  13. Find the residual at x=14x=14

    e=28(46.21.35x)x=14=2827.3=0.7e=28-\left(46.2-1.35x\right)_{x=14}=28-27.3=0.7

    The residual is observed minus predicted.

  14. Find the residual at x=16x=16

    e=25(46.21.35x)x=16=2524.6=0.4e=25-\left(46.2-1.35x\right)_{x=16}=25-24.6=0.4

    The residual is observed minus predicted.

  15. Check that the residuals sum to zero

    ei=(1.6)+(1.7)+(1)+(0.7)+(0.4)=0\sum e_i=\left(1.6\right)+\left(-1.7\right)+\left(-1\right)+\left(0.7\right)+\left(0.4\right)=0

    This is guaranteed by the first normal equation and catches arithmetic slips.

  16. State the residual of largest magnitude

    e=1.7 at x=10e=-1.7\ \text{at}\ x=10

    This observation lies furthest from the line vertically and dominates the residual sum of squares.

Answer
e=1.7e=-1.7

Unlock 29 more Linear regression questions

Create a free account to work through every Further Maths Linear regression question with instant step-by-step worked solutions, progress tracking and interactive lessons.

  • Full worked solutions for every question
  • Interactive lessons and instant feedback
  • Track your mastery across every topic
Create a Free Account

No card required · Free forever

More Linear regression practice

Related Statistics topics