Class 3 & 4 · Multiple Linear Regression
STAT 517: Advanced Statistical Models · Fall 2026
This interactive companion follows the Class 3 and 4 notes. We extend simple linear regression to several predictors, work with categorical predictors, distinguish joint from conditional inference, compare marginal and conditional associations, examine model-selection criteria, and evaluate held-out predictions.
Each WebR code box runs R directly in your browser. Press Run Code, inspect the result, change something, and run it again. Run the cells in order because later cells use objects created earlier. Checkpoint cells intentionally omit answers so they can be completed during class.
3.1 From one predictor to several
A multiple linear regression model for a continuous response is
\[ Y_i = \beta_0 + \beta_1X_{i1} + \cdots + \beta_pX_{ip} + \varepsilon_i, \qquad \varepsilon_i \overset{\mathrm{iid}}{\sim} N(0,\sigma^2). \]
Here, \(p\) counts the non-intercept columns in the design matrix and \(k=p+1\) counts all regression coefficients, including the intercept. Simple linear regression is the special case \(p=1\).
The assumptions concern the errors after conditioning on the included predictors:
- observations have independent errors;
- the linear conditional mean is correctly specified;
- errors are normally distributed for the usual small-sample inference; and
- errors have constant variance across observations.
3.2 Least squares and model fit
Least squares minimizes the residual sum of squares. This small example makes the parameter counts and residual degrees of freedom explicit.
For an ordinary least-squares model with an intercept,
\[ \operatorname{MSE}=\frac{\operatorname{RSS}}{n-k} =\frac{\operatorname{RSS}}{n-p-1}, \qquad R^2=1-\frac{\operatorname{RSS}}{\operatorname{TSS}}. \]
In simple linear regression, MSE uses denominator \(n-2\). Explain why multiple regression uses \(n-k=n-p-1\). Then explain why raw \(R^2\) is not enough to choose among models with different numbers of predictors.
3.3 Georgia voting data
The gavote data contain one row for each of Georgia’s 159 counties in the 2000 election. We create Gore vote share and relative undercount, then retain the variables used in this lab.
Use exploratory plots to see the response across equipment groups and the relationship between two demographic variables. These plots motivate questions; they do not determine a final model or establish causation.
3.4 Categorical predictors and reference levels
R stores categorical predictors as factors. With the default treatment coding, the first level is the reference level and an \(m\)-level factor contributes \(m-1\) indicator columns.
Inspect the columns R creates for a model containing four factors.
Changing a reference level changes the displayed coefficients, but it does not change fitted values, residuals, or overall fit for a model containing the same factor terms.
3.5 Fitting a full model
The dot in the formula uses every other column in new_gavote as a predictor. This is convenient for a demonstration, but a scientific model should justify each candidate variable.
Extract the main fit summaries and coefficient table directly.
Each coefficient is conditional on the other included model terms. For example, the equipOS-PC estimate compares OS-PC with the LEVER reference group among counties with the same modeled economic category, demographics, location, and Gore vote share. This is an adjusted observational association, not a causal effect of replacing equipment.
A large \(p\)-value can reflect little conditional association, limited precision, collinearity, or an unsuitable functional form. It is not proof of “no relationship.”
3.6 Diagnostics still matter
The familiar four-panel display asks four different questions:
| Diagnostic evidence | Question addressed | Good if |
|---|---|---|
| Residuals versus fitted values | Does the fitted linear conditional mean leave a visible pattern? | No clear curve or funnel pattern. |
| Normal Q-Q plot | Are residuals reasonably consistent with normal errors for small-sample inference? | Points are close to a straight line, especially away from extreme tails. |
| Scale-location plot | Is residual spread approximately constant across fitted values? | Vertical spread is roughly constant. |
| Residuals versus leverage and Cook’s distance | Are observations unusually influential for the fitted coefficients? | No observation combines a large residual with high leverage or large Cook’s distance. |
Diagnostics concern the specified model, not the data in isolation. A labeled case deserves investigation, but it should not be deleted automatically.
3.7 Inference in multiple regression
3.7.1 The overall \(F\) test
The overall test compares the fitted model with an intercept-only model:
\[ H_0:\beta_1=\cdots=\beta_p=0 \qquad\text{versus}\qquad H_A:\text{at least one }\beta_j\ne0. \]
Its numerator degrees of freedom are \(p=k-1\) and its denominator degrees of freedom are \(n-k\).
Failing to reject the joint null warns that the chosen linear predictor set may not be useful as a group. It does not prove that every possible relationship is absent.
3.7.2 Conditional coefficient \(t\) tests
For one coefficient, the statistic is
\[ T_j=\frac{\widehat\beta_j}{\operatorname{SE}(\widehat\beta_j)}, \qquad T_j\sim t_{n-k}\text{ under }H_0:\beta_j=0. \]
Inspect one coefficient test and its confidence interval.
Failing to reject \(H_0:\beta_j=0\) means the data do not provide sufficient evidence that \(X_j\) explains additional linear variation after accounting for the other model terms. It does not show that \(X_j\) has no marginal association with the response.
3.7.3 Confidence intervals for coefficients
A two-sided \(100(1-\alpha)\%\) interval is
\[ \widehat\beta_j \mathbin{\pm}t_{n-k,1-\alpha/2} \operatorname{SE}(\widehat\beta_j). \]
For a two-sided test at \(\alpha=0.05\), the coefficient test and 95% confidence interval give the same reject/fail-to-reject decision.
For the full undercount model, distinguish these questions: Does the model contain any predictive signal? Does OS-PC differ from LEVER after adjustment? Which output line or test would you use for each?
3.8 Marginal and conditional relationships
A marginal association does not adjust for other predictors. A conditional association describes how the response varies with one predictor at fixed values of the other included model terms.
The variables perAA and pergore are strongly related in these data. The second command demonstrates why ordinary correlation is not defined between a numeric variable and a factor.
Compare the marginal coefficient summaries with the corresponding conditional rows from the full model.
Strongly related predictors can inflate standard errors and make individual coefficients sensitive to the model specification. Prediction may still be useful even when the separate coefficients are difficult to estimate precisely.
How does “Do counties with larger perAA values tend to have different undercount?” differ from “Among counties with the same modeled equipment, economic category, location, and Gore vote share, how does undercount vary with perAA?”
3.9 Model fit and complexity
Raw \(R^2\) cannot choose among candidate models of different sizes because it cannot decrease when predictors are added. Adjusted \(R^2\) and AIC penalize additional complexity:
\[ R^2_{\mathrm{adj}} =1-\frac{\operatorname{RSS}/(n-k)}{\operatorname{TSS}/(n-1)}, \qquad \operatorname{AIC}=-2\ell(\widehat\theta)+2q. \]
Here \(k=p+1\) counts regression coefficients. For a normal linear model with unknown variance, AIC also counts \(\sigma^2\), so \(q=k+1\).
Compare a prespecified reduced model, the full model, and an AIC-selected model. Higher adjusted \(R^2\) and lower AIC are preferred when the models use the same response and observations.
Inspect the selected formula and identify coefficient rows omitted from the full model.
Why can the equipment-and-economic-category model have higher adjusted \(R^2\) than the full model even though the full model has at least as high a raw \(R^2\)?
The same data are used to search across models and report the selected model. Ordinary coefficient \(p\)-values and confidence intervals do not account for that search. Use subject-matter reasoning, prespecified goals, diagnostics, and out-of-sample evaluation rather than treating step() as an automatic scientific decision maker.
3.10 Prediction on held-out observations
A confidence interval targets a conditional mean. A prediction interval targets one new observed response and is wider because it includes individual variation around that mean.
Create a reproducible train/test split and fit a model for Gore vote share using only the training counties.
Compare observed and predicted responses in the held-out counties.
Construct both interval types and use the observed test responses only for evaluation.
Use AIC to check whether every training-set predictor remains in the selected model.
Why is it unsurprising that only about one-third of the held-out pergore values fall inside the confidence intervals for the mean? Which interval should be used when checking whether individual held-out counties are covered?
Results can change with the random split, especially for a modest dataset. Repeated resampling or cross-validation gives a more stable estimate of predictive performance. An ordinary linear model can also predict a proportion below 0 or above 1, so the response scale must be checked.
3.11 Takeaways
- Multiple-regression coefficients are interpreted conditionally on the other included model terms.
- An \(m\)-level factor contributes \(m-1\) indicator columns relative to a reference level.
- The overall \(F\) test addresses a joint null; individual \(t\) tests and confidence intervals address conditional coefficient questions.
- Marginal and conditional associations answer different questions, and collinearity can make separate coefficients imprecise.
- Adjusted \(R^2\) and AIC balance fit and complexity, but automatic selection is exploratory.
- Held-out data should evaluate predictions, and the interval must match its intended target.
Next class (September 1): likelihood, log-likelihood, and maximum likelihood estimation. Generalized linear models begin on September 3.
3.12 Answer key
Checkpoint answers are intentionally omitted from this class-demo companion. The lecture notes provide the answer key for review after class discussion.