Regression Lines and Estimation

Scope Label

Core 9758. Regression lines, line selection, interpolation, extrapolation, and reliability of estimates are core parts of correlation and linear regression.

Role in the Topic

This branch starts after the scatter diagram suggests that a linear model is sensible.

Use it with:

• Correlation and Linear Regression
• Scatter, Correlation, and PMCC

Regression as Best-Fit Modelling

A regression line is a best-fit line used to estimate one variable from another.

The phrase “best fit” is made precise by least squares.

For regression of $y$ on $x$, the line is chosen to minimise the sum of squared vertical errors.

If ${\hat{y}}_i$ is the fitted value, the vertical residual is

$$e_i=y_i-{\hat{y}}_i,$$

and least squares minimises $\sum e_i^2$. Squaring prevents positive and negative residuals cancelling and penalises large errors. Regression of $x$ on $y$ uses the analogous horizontal residuals.

For regression of $x$ on $y$, the line is chosen to minimise the sum of squared horizontal errors.

How to read this figure. The residual direction defines the fitted line. Use a known dependence direction first; only when no direction is known should the variable being estimated determine which residuals are minimised.

This is the key reason the two regression lines usually differ.

Regression of $y$ on $x$

The regression line of $y$ on $x$ is used when $y$ is modelled or estimated from $x$.

It has the form

$$y=a+bx$$

or equivalent calculator notation.

This line treats $x$ as the explanatory or input variable and $y$ as the response or output variable.

If the fitted line is

$$y=a+bx,$$

then the gradient $b$ means:

for each one-unit increase in $x$, the predicted value of $y$ changes by approximately $b$ units.

The intercept $a$ is the predicted value of $y$ when $x=0$, but it should only be interpreted if $x=0$ is meaningful in context and not far outside the observed data range.

Regression of $x$ on $y$

The regression line of $x$ on $y$ is used when $x$ is modelled or estimated from $y$.

It has the form

$$x=a+by.$$

If this line is later rearranged to draw it on usual $x$-$y$ axes, it is still the regression line of $x$ on $y$. The direction of least-squares error has not changed.

Choosing the Correct Regression Line

The safest question is:

Which variable is being modelled as depending on the other?

If the context gives a dependence direction, follow it.

Context	Suitable model
$y$ depends on $x$	regression of $y$ on $x$
$x$ depends on $y$	regression of $x$ on $y$
no clear dependence, estimate $y$ from $x$	regression of $y$ on $x$
no clear dependence, estimate $x$ from $y$	regression of $x$ on $y$

How to read this figure. Start with known dependence direction. Only when neither variable naturally depends on the other should the variable being estimated determine the regression direction.

The subtle case is calibration. If an instrument reading $y$ depends on the true concentration $x$, the modelling direction may still be $y$ on $x$ even when the question asks for $x$ from a given reading.

Mean Point

Both regression lines pass through

$$(\bar{x},\bar{y}).$$

This point is the centre of the bivariate data.

Formula insight (derivation not required). With $S_{xx}=\sum (x-\bar{x}{)}^2$, $S_{yy}=\sum (y-\bar{y}{)}^2$, and $S_{xy}=\sum (x-\bar{x})(y-\bar{y})$, the $y$-on-$x$ slope is $S_{xy}/S_{xx}=r(s_y/s_x)$ and its intercept is $\bar{y}-b\bar{x}$. The $x$-on-$y$ slope is $S_{xy}/S_{yy}=r(s_x/s_y)$. These forms explain why both lines pass through $(\bar{x},\bar{y})$; calculator use remains the core procedure.

These formulae require $S_{xx}>0$ for regression of $y$ on $x$ and $S_{yy}>0$ for regression of $x$ on $y$. If all $x$-values are equal, the $y$-on-$x$ slope is undefined; if all $y$-values are equal, the $x$-on-$y$ slope is undefined.

How to read this figure. Both fitted lines must pass through $(\bar{x},\bar{y})$, but that property alone does not identify which line is $y$ on $x$ or make the two lines interchangeable.

The stronger the linear correlation, the closer the two regression lines are. If $r=1$ or $r=-1$, all points are collinear and the two regression lines coincide.

For $0<|r|<1$, the regression line of $x$ on $y$, rearranged into finite-gradient $y=mx+c$ form, is steeper in absolute value than the $y$-on-$x$ line. If $|r|=1$, the lines coincide. If $r=0$, the lines are $y=\bar{y}$ and $x=\bar{x}$; the latter is vertical and cannot be rearranged into finite-gradient form.

How to read this figure. For $0<|r|<1$, stronger linear correlation brings the two lines closer. At $|r|=1$ they coincide; at $r=0$ the two regression lines are horizontal and vertical through the mean point.

Interpolation and Extrapolation

An estimate is interpolation if the input value lies within the observed data range.

An estimate is extrapolation if the input value lies outside the observed data range.

How to read this figure. Interpolation avoids assuming that a trend continues beyond the observed predictor range, but it still requires a suitable linear model. Extrapolation needs high caution even when $|r|$ is large.

Interpolation is generally more reliable because it stays within the evidence supplied by the data.

Extrapolation assumes the same relationship continues outside the observed range. That assumption may be false even when $|r|$ is close to $1$ within the sample.

Reliability of Estimates

To judge reliability, ask:

1. Is the input value within the observed data range?
2. Is the scatter diagram roughly linear?
3. Is $|r|$ close to $1$?
4. Are there outliers or clusters that weaken the model?
5. Does the estimate make sense in context?

How to read this figure. These checks make an estimate more defensible, not guaranteed correct. Name the predictor range being checked, inspect curvature/outliers/clusters, and apply contextual judgement.

A good reliability statement names both range and linear strength:

The estimate is likely to be reliable because the input is within the observed range and the scatter diagram shows a strong linear pattern.

or:

The estimate is not reliable because it is extrapolation beyond the observed data range.

Core Example: Choosing a Line

Suppose $y$ is a machine reading and $x$ is the true concentration of a chemical. The machine reading depends on the true concentration.

If the question gives a machine reading and asks for the concentration, it may seem natural to use the regression line of $x$ on $y$.

But if the modelling relationship is

$$x⟶y,$$

then the regression line of $y$ on $x$ may be the correct calibration model. After fitting it, solve the equation for $x$.

The reason is not algebraic convenience. It is that the reading error is in $y$.

Core Example: Reliability

Suppose the observed $x$-values range from $10$ to $80$, and the regression line is used to estimate $y$ when $x=55$.

This is interpolation, so it may be reasonable if the scatter diagram is roughly linear and $|r|$ is close to $1$.

If the same line is used at $x=140$, the estimate is extrapolation. It should be treated as unreliable unless there is strong external justification that the same linear pattern continues.

Core Example: Complete Regression Workflow

Consider the original paired data

$x$	1	2	3	4	5
$y$	2.2	3.9	5.8	8.2	9.9

A calculator gives (coefficients are displayed here to suitable accuracy; retain the stored calculator values for computation),

$$\text{y on x:}y=0.09+1.97x,$$ $$\text{x on y:}x=-0.03543914+0.50590652y,$$

with $r\approx 0.9983$. Since $(\bar{x},\bar{y})=(3,6)$, both equations pass through the mean point.

If there is no known dependence direction, estimating $y$ at $x=3.5$ uses the $y$-on-$x$ line and gives $\hat{y}=6.985$. Estimating $x$ at $y=7$ uses the $x$-on-$y$ line and gives $\hat{x}\approx 3.506$. Both are interpolation.

If context instead says $y$ is an instrument reading that depends on true input $x$, the calibration model remains $y$ on $x$. For a reading $y=7$, solve $7=0.09+1.97x$ to obtain $x\approx 3.508$.

Keep calculator coefficients at full precision during substitution and round only the final estimate. Premature coefficient rounding can shift the reported answer.

Common Pitfalls

• Choosing the regression line only by looking at which variable is unknown.
• Forgetting that context may determine the dependence direction.
• Treating the two regression lines as interchangeable.
• Rearranging $x=a+by$ and forgetting it is still regression of $x$ on $y$.
• Forgetting both regression lines pass through $(\bar{x},\bar{y})$.
• For $0<|r|<1$, forgetting that the rearranged $x$ on $y$ line has a larger absolute gradient than the $y$ on $x$ line on usual axes. At $r=0$, that rearranged line is vertical and has no finite gradient.
• Interpreting an intercept even when $x=0$ is outside the meaningful context.
• Calling an extrapolated estimate reliable just because $|r|$ is close to $1$.
• Ignoring outliers or curvature when using a regression line.

Revision Checklist

• Can you explain why regression of $y$ on $x$ minimises vertical errors?
• Can you explain why regression of $x$ on $y$ minimises horizontal errors?
• Can you choose the correct regression line from the context?
• Can you explain why both regression lines pass through $(\bar{x},\bar{y})$?
• Can you interpret the gradient and intercept of a regression line in context?
• Can you distinguish interpolation from extrapolation?
• Can you write a reliability sentence that mentions both range and linear strength?