Class 9 · Classification with Logistic Regression

STAT 517: Advanced Statistical Models · Fall 2026

Work through two examples: credit-card default and ten-year coronary heart disease. Each example explains the commands and interprets their output, from fitting probabilities to choosing a classification threshold.

Press Run Code on each box in order. Later boxes reuse earlier objects; you can edit a box and run it again to explore a different choice. The data are included, and the required packages load automatically in your browser. The first run may take longer while the packages load. Profile-likelihood intervals and stepwise selection may also take a little time.

Goals

• Fit on training observations and predict outcomes for test observations.
• Read a confusion matrix with an explicitly defined positive class.
• Calculate sensitivity, specificity, and accuracy and interpret their tradeoffs.
• Interpret ROC curves, AUC, and Youden’s J.
• Choose a threshold for a stated objective and distinguish selection from testing.

Start here

Run this box once before the examples. It loads the two packages used below.

Example 1: Predicting default on credit-card debt

The ISLR::Default dataset contains 10,000 customers and whether they defaulted on their credit cards. We fit a model on training customers and evaluate its predictions on separate test customers. We first estimate a probability, then choose a classification rule.

The response is default: No (\(y=0\)) or Yes (\(y=1\)). The predictors are:

• student: whether the customer is a student;
• balance: the customer’s credit-card balance; and
• income: the customer’s annual income.

Split into 80% training and 20% test data

Fit the model using the training data. Reserve the test data to examine how well its predictions work for observations not used to estimate coefficients. set.seed() lets us reproduce this particular random split.

Fit the model and interpret inference

The dot in default ~ . includes all other columns as predictors. binomial() uses logistic regression (the default link is logit).

Predict probabilities for test customers

type = "response" returns P(default = Yes), between 0 and 1. Without it, predict.glm() returns the linear predictor (log-odds).

Classify customers at four prespecified cutoffs

Our rule: predict Yes if the probability is AT LEAST t; otherwise predict No. Apply each threshold to the SAME probabilities and the SAME test customers. Only the decision rule changes; we do not refit the logistic regression.

Worked cutoff t = 0.5: read the four cells

First identify the four cells, then use the actual positive and negative group totals as the denominators for sensitivity and specificity.

Worked cutoff t = 0.9: why accuracy is not enough

Raise the cutoff and compare the results. Pay attention to how many actual defaults are detected, even when the overall accuracy looks high.

ROC curve: examine all possible cutoffs

Each cutoff produces a sensitivity and a false-positive rate (1 - specificity). An ROC curve plots these pairs: false-positive rate horizontally, sensitivity vertically. Moving along the curve changes the cutoff, not the fitted model. levels and direction make explicit that larger scores predict Yes.

Youden’s J: choose a cutoff that maximizes J

\(J = \mathrm{sensitivity} + \mathrm{specificity} - 1\). A large J favors high values of both rates. The cutoff that maximizes J is a probability threshold; it is not J itself. This criterion weights sensitivity and specificity equally. It need not represent the actual costs of missed defaults and false alarms.

We use the test ROC here to demonstrate how cutoff selection works. This is a retrospective illustration: once test outcomes help choose a rule, they cannot independently evaluate that rule. For a prediction study, select cutoffs on validation data (or resampling within training data), then evaluate the chosen rule once on a separate, untouched test set.

Suppose we require sensitivity of at least 98%

Suppose missing defaults is especially costly. For this illustration, require sensitivity >= 0.98 and, among eligible cutoffs, maximize specificity. First filter the rows; then optimize the other rate within the eligible set.

Example 2: Ten-year coronary heart disease (Framingham)

We now predict the binary 10-year CHD outcome using the Framingham data. TenYearCHD is coded NotAtRisk (\(y=0\)) and AtRisk (\(y=1\)) in the supplied file. Here AtRisk is the positive outcome label; it is not a probability cutoff. Predictors include age, smoking measures, cholesterol, blood pressure, BMI, and glucose. Binary predictors retain their numerical 0/1 coding.

Split, fit, and select on training data

Use a stratified split again. Fit the full model and choose predictors using training data only, leaving test outcomes out of the model-selection process.

Predict test outcomes and examine AUC

Use the selected model and return the probability of the positive class.

Suppose we require specificity of at least 98%

A false positive here means predicting AtRisk for an actual NotAtRisk outcome. Suppose avoiding such false alarms is the priority because follow-up has costs. For this classroom example, require specificity >= 0.98, then maximize sensitivity among the eligible rules. This is not a clinical recommendation. A low false-positive rate also does not tell us the fraction of positive predictions that are false; that fraction depends on prevalence as well.

Practice: Question 8.3

Return to the raw confusion tables for thresholds 0.01 and 0.10. Use those counts to answer the following in your notes:

1. Calculate accuracy, sensitivity, and specificity at each threshold.
2. Which threshold would you choose if detecting nearly all defaults were the priority? State the cost in false positives.
3. If you used these test results to choose a threshold, could those same results independently evaluate the selected rule?

The worked 0.5 and 0.9 examples show the calculation method. The 0.01 and 0.10 metric answers are left for you to calculate.

Takeaways

• A probability becomes a class prediction only after a threshold is chosen.
• High accuracy can coexist with poor detection of a rare outcome.
• Sensitivity concerns actual positives; specificity concerns actual negatives.
• AUC describes ranking across thresholds, not accuracy at one cutoff.
• Threshold choice depends on the consequences of false positives and false negatives.
• Choose the model and threshold before evaluating them on the final test set.