Class 11 · Multinomial Logistic Regression

STAT 517: Advanced Statistical Models · Fall 2026

Published

October 6, 2026

This interactive companion follows Chapter 10 and the reviewed class11_multinomial_regression.R. The high-school program example develops a model for a response with three nominal categories; the breast-tissue example evaluates predictions for four categories on held-out observations.

Press Run Code on each box in order. Later boxes reuse earlier objects, and you can edit a box and run it again. Both data files are included and the packages load in your browser; the first run takes longer while VGAM and caret load. Questions 10.1–10.4 match the handout. Record your answers where the instructor posts this lab.

Goals

  • Specify a baseline-category logit model and count its coefficients.
  • Distinguish the baseline outcome category from the reference level of a predictor.
  • Convert fitted logits into a full probability vector for a new case.
  • Interpret an adjusted odds ratio for one pair of outcome categories.
  • Test a multi-coefficient predictor jointly with a likelihood-ratio test.
  • Read a multiclass confusion matrix and compute precision, recall, and \(F_1\).

Start here

Run this box once. VGAM fits the multinomial model; caret provides the stratified split and the labeled classification summary.

Part 1 · Model and probabilities

1. Academic-program data and reference categories

There are 200 students and three program categories: general, vocation, and academic. The labels have no natural order, so this is a nominal response. Two reference choices matter and they are different decisions: academic is the baseline category of the outcome, and middle is the reference level of the predictor ses.

TipQuestion 10.1: Specify the academic-program model

Use academic as the baseline category and writing score \(x\) as the only predictor.

  1. Write both baseline-category logit equations. How many coefficients, including intercepts, must be estimated?
  2. Write the log odds of general versus vocation using those coefficients.
  3. If we change the baseline category to general, do the fitted probabilities change?

2. Fit the two baseline-category logits

With \(K=3\) categories the model has \(K-1=2\) logits, each comparing one category with the baseline:

\[ \log\frac{p_{i\ell}}{p_{iK}} = \beta_{0\ell}+\beta_{1\ell}x_{i1}+\cdots+\beta_{p\ell}x_{ip}, \qquad \ell = 1,\ldots,K-1 . \]

The ratio \(p_{i\ell}/p_{iK}\) is the baseline-category odds of category \(\ell\) versus \(K\): the odds of \(\ell\) among responses that fall in either \(\ell\) or \(K\). It differs from \(p_{i\ell}/(1-p_{i\ell})\), which compares \(\ell\) with all other categories combined. vglm() estimates a separate coefficient vector for each non-baseline category.

Read the output carefully. The suffixes :1 and :2 index the two baseline-category logits named under Names of linear predictors: log(mu[,1]/mu[,3]) is general versus academic and log(mu[,2]/mu[,3]) is vocation versus academic. The last line confirms that level 3, academic, is the baseline.

Each \(e^{\widehat\beta_{j\ell}}\) is an adjusted odds ratio (Chapter 7): the multiplicative change in the baseline-category odds of \(\ell\) versus academic for a one-unit increase in \(x_j\), holding the other predictor fixed. For example, write:1 gives \(e^{-0.0579}=0.944\): comparing two students with the same SES, one additional writing point is associated with about 5.6% lower odds of general rather than academic. Every interpretation must name both outcome categories and hold the other predictor fixed.

3. Predict for individual students

type = "response" returns all three category probabilities at once, including the baseline category academic, which has no coefficients of its own. Each row is a full probability vector and sums to one.

Practice (not collected): student 3 has low SES and writing score 49. Use the coefficients above to calculate both logits by hand, then the three probabilities with their common denominator \(1+e^{\eta_{G}}+e^{\eta_{V}}\). Only afterwards add a predict() line to the box below to check your numbers. Which program has the largest probability, and how confident does the full vector make you about that single label?

4. Follow probabilities across writing scores

To see how the fitted probabilities change with writing score, build a grid of new students at each SES level and ask predict() for all three probabilities. The grid uses writing scores 30 to 70 at each SES level.

Reshape to one row per SES, writing score, and program, then plot. Compare the same SES line across the three panels: for any fixed writing score, those three probabilities sum to one.

The sign of a coefficient describes a change in the odds of one category versus the baseline. Its probability also depends on every other logit, so the general-program probability can rise and then fall even though its logit against academic is a straight line in writing score. Changing the baseline category changes coefficient labels and values but leaves the fitted probabilities unchanged.

Part 2 · Inference and classification

5. Wald tests and confidence intervals

A category-specific Wald test asks whether one coefficient is zero, after adjusting for the other predictors in the model. Exponentiating a coefficient interval gives an interval for its adjusted odds ratio; the coefficient interval contains 0 exactly when the odds-ratio interval contains 1. Identify the outcome contrast before interpreting any row.

For write:1, one additional writing point is associated with 1.6% to 9.5% lower odds of general rather than academic (95% confidence), holding SES fixed. An interval containing zero means the data do not establish an association at this level; it does not prove that none exists.

6. Test SES as a whole

SES enters the model through two indicators in each of two logits. Asking whether SES matters at all is a joint question about four coefficients, which four separate Wald tests do not answer. Fit the model without SES on the same students and compare maximized log-likelihoods:

\[ G^2 = 2\{\ell(\widehat{\boldsymbol\beta}_{\rm full})-\ell(\widehat{\boldsymbol\beta}_{\rm reduced})\} \;\mathrel{\dot\sim}\;\chi^2_q , \]

where \(q\) is the number of restrictions.

TipQuestion 10.2: Test socioeconomic status jointly

Compare prog ~ ses + write with prog ~ write. SES has levels low, middle, and high; the outcome has three categories.

  1. State the joint null hypothesis in terms of coefficients.
  2. Find the degrees of freedom and explain your count.
  3. Find the \(\alpha=0.05\) cutoff with qchisq(0.95, df = ...) and compare \(G^2\) with it. Confirm with pchisq(joint.ses.LR, df = ..., lower.tail = FALSE) and state the conclusion. Why is one category-specific Wald test insufficient?

7. Read a multiclass confusion matrix

For \(K\) classes the confusion matrix is \(K\times K\). In our tables rows are predictions and columns are actual classes. For class \(\ell\):

  • precision divides its diagonal entry by its row total (predicted-class total);
  • recall divides its diagonal entry by its column total (actual-class total);
  • \(F_1 = 2\,\dfrac{\text{Precision}_\ell\times\text{Recall}_\ell}{\text{Precision}_\ell+\text{Recall}_\ell}\), and macro \(F_1\) averages the \(K\) class values.

Recall is the sensitivity of Chapter 8 applied to class \(\ell\) versus all other classes. Specificity is less useful here, because the pooled “other” classes make it high for every class, so precision takes its place as recall’s partner. Read the labels before calculating so the two denominators do not get interchanged.

The original fruit example supplies these counts:

The Orange and Mango rows below were computed from this matrix.

Class Precision Recall \(F_1\)
Apple ______ ______ ______
Orange 2/6 = 0.333 2/12 = 0.167 4/18 = 0.222
Mango 1/6 = 0.167 1/13 = 0.077 2/19 = 0.105
TipQuestion 10.3: Evaluate the fruit classifier
  1. Fill in the Apple row: its precision, recall, and \(F_1\).
  2. Compute the macro \(F_1\) score and overall accuracy. Keep unrounded intermediate calculations when finding macro \(F_1\).
  3. Give a short assessment of the classifier, using both overall and class-specific results.

8. Classify breast tissues on held-out data

The data contain 106 tissue samples in four classes: adi (adipose), car (carcinoma), con (connective), and fgm (fibro-adenoma, glandular, and mastopathy combined). Following the original analysis, use Max.IP and DA as predictors, seed 1234, and a stratified 60/40 split. Rounding within each class gives 66 training and 40 test observations.

Fit only on the training observations. The baseline category is fgm, so the suffixes :1, :2, :3 mean adi, car, and con versus fgm.

Now predict for the observations that were not used to estimate the model. The classification rule assigns each test case to its most probable class. confusionMatrix() takes the predictions as data and the observed classes as reference; name both arguments so the orientation is unambiguous.

Rows are predictions and columns are observed classes. Note that caret reports both vocabularies: the Recall row repeats Sensitivity, and Precision repeats Pos Pred Value.

TipQuestion 10.4: Identify carcinoma

Treat carcinoma (car) as the class of interest.

  1. Identify TP, FP, and FN for carcinoma in the displayed test matrix.
  2. Calculate its precision, recall, and \(F_1\) by hand from the matrix, then check them against the Class: car column. Explain the different denominators.
  3. Calculate overall accuracy by hand and check it against the Accuracy line. Why does this value alone fail to describe how well the classifier detects carcinoma?
  4. Could we use this test set to select predictors and still describe the reported performance as an independent evaluation of that selected model?

The largest-probability label is a decision rule for equal error costs. Class counts and class-specific metrics reveal mistakes that overall accuracy can conceal. With only 40 test observations, these rates also carry substantial sampling uncertainty.

Optional · Score the probabilities with log loss

Every metric above uses only the predicted label, \(\arg\max_\ell \widehat p_{i\ell}\): a case labeled fgm at probability 0.51 and one at 0.99 count the same. The log loss is the cross-entropy between the one-hot observed vector \(y_i\) and the predicted vector \(\widehat p_i\), averaged over the test set:

\[ \text{log loss} =-\frac{1}{n_{\rm test}}\sum_{i\in\text{test}}\sum_{\ell=1}^{K} y_{i\ell}\log\widehat p_{i\ell} =-\frac{1}{n_{\rm test}}\sum_{i\in\text{test}}\log\widehat p_{i,y_i}, \]

since only the observed class has \(y_{i\ell}=1\). Lower is better. It is the same quantity that maximum likelihood minimized on the training data, now evaluated on new data, so the model is judged out of sample by the criterion it was fitted with. caret::mnLogLoss() expects a data frame with the observed class in obs and one probability column per class.

Explore: which test cases contribute most to the log loss? The box below lists each case’s penalty \(-\log\widehat p_{i,y_i}\) with the largest first. Compare the misclassified cases with the correctly classified ones.

Log loss is a proper scoring rule: you cannot improve it by bluffing confidence, only by making the probabilities more accurate. Compare probabilistic classifiers by log loss as well as by confusion matrix.

Optional · Stepwise selection from the original analysis

The original lecture also fit the academic-program model with all available predictors and reduced it by backward AIC using VGAM::step4(). This is kept for discussion; it is not needed for Questions 10.1–10.4, and it may take a little while to run in the browser. The post-selection Wald tests and intervals below do not account for the selection step, so treat them as exploratory summaries rather than prespecified inference.

Check your understanding

  • Identify both the baseline outcome category and any predictor reference level.
  • Use the common denominator to recover every category probability.
  • Match a category-specific inference to the coefficient it tests, and name both outcome categories.
  • Test a multi-coefficient predictor jointly when that is the question.
  • Keep model selection separate from the final test evaluation.
  • Read the matrix labels before computing precision and recall.