Class 10 · Train, Validate, Test Three Binary Links
STAT 517: Advanced Statistical Models · Fall 2026
STAT 517 · Chapter 9 · 30 minutes · One submission per group
Use the familiar credit-card example to compare logit, probit, and complementary log-log regression. Every group fits all three models with the same predictors: student status, balance, and income. The event is default = Yes.
Our workflow is 60% training → 20% validation → 20% test:
| Subset | What you do with it |
|---|---|
| Training | Estimate the coefficients of each model. |
| Validation | Choose a separate classification threshold for each link. |
| Test | Compare the three fixed classification rules on new observations. |
All three links must use the same split within your group. Groups use different fixed random seeds so we can see how results vary across splits. Keep your assigned seed and model formula throughout the lab.
Group assignments
Group1: @Kent-Robinson, @swalser01, @blurissa
Group2: @BrooklynTyner, @cjdow, @jmames17
Group3: @Peyton-Math, @jcarp333, @zhuang-jy
Group4: @jess-l-b, @zoemelton, @maggieleporin
Group5: @nmayers517, @jth27, @joshyoung803
Group6: @Isabellhaines, @asearly60
Choose someone to run the code, someone to check the data flow, and someone to record the interpretation. The two-person group can combine roles. The six student groups are separate from the three data subsets.
1. Make the 60/20/20 split — 5 minutes
Run the code boxes in order using Run Code. R and pROC run in your browser, and Default.csv is included; no installation or upload is needed. The first run may take longer while R and the package load. Set your group number in the first box. If you change it or restart the page, rerun every box from the top.
Change group to your group number. The supplied loop shuffles customers separately within the two outcome categories, keeping the default rate similar across subsets. For these data, rounding gives exactly 6,000 training, 2,000 validation, and 2,000 test customers.
Discuss: Why should each link use the same training customers? Which subset will choose the threshold, and which will assess its performance?
2. Fit the three links on training data — 5 minutes
Only train enters these fits. A binomial model with the factor levels No, Yes models the probability of Yes. Keep the same predictors under every link.
Check: Identify where the training data and link function enter each call. Why would comparing the raw coefficient magnitudes across links be misleading?
3. Choose thresholds on validation data — 9 minutes
Predict default when the fitted probability is at least the threshold. For this lab, choose each threshold to maximize validation balanced accuracy:
\[ \text{Balanced accuracy} = \frac{\text{Sensitivity} + \text{Specificity}}{2}. \]
This weights the two class-specific success rates equally, which is useful here because defaults are rare. It is our common teaching criterion; a real application may use different error costs. Keep this criterion fixed for all three models.
pROC::coords(..., x = "best", best.method = "youden") finds thresholds that maximize sensitivity plus specificity, so it also maximizes balanced accuracy. The probabilities and outcomes supplied to the ROC objects below are validation values. If several cutoffs tie, we use the smallest tied cutoff, favoring sensitivity.
Use these validation ROC objects to select and record the thresholds.
Before continuing: Record all three thresholds and identify the link with the highest validation balanced accuracy, noting any tie. Why might the links choose different thresholds? Why is the largest validation score not an independent assessment of the chosen rule?
Freeze the three fitted models and thresholds now. Keep their full numerical precision; round only the displayed results. For this lab, do not refit the models on training plus validation data, because that would change the fitted rules whose thresholds you just selected.
4. Compare the fixed rules on test data — 6 minutes
Use the test set for final evaluation. The following code uses the saved thresholds; it does not search for new ones. Report all three models even if their ranking differs from validation.
Now calculate test metrics at the saved cutoffs and test AUC for each link.
Check one row by hand: Use its confusion matrix to calculate sensitivity and specificity. Confirm that their average equals the reported balanced accuracy. AUC summarizes ranking across thresholds; it is not the accuracy at your chosen cutoff.
5. Submit and compare across groups — 5 minutes
Submit one Markdown response per group where the instructor posts this lab. Include your group number, seed (2400 + group), validation-selected link or tie, and this table. Report four decimal places.
| Link | Validation threshold | Validation balanced accuracy | Test sensitivity | Test specificity | Test accuracy | Test balanced accuracy | Test AUC |
|---|---|---|---|---|---|---|---|
| Logit | |||||||
| Probit | |||||||
| Cloglog |
Add brief answers:
- Which link has the highest test balanced accuracy? Does the test AUC ranking agree? Describe the size of the differences.
- Explain why using
x = "best"on a test ROC object and then reporting the resulting test sensitivity and specificity as independent performance would double dip the data. - Compare your link rankings with one other group. What changed, and what does this suggest about declaring one link universally best? The groups’ data subsets overlap, so these are examples of split variation, not six independent replications.
Do not change the seed, formula, threshold criterion, or thresholds after seeing test results. Our goal is to compare the three prespecified procedures honestly, not to improve their reported test scores.
Reference for the threshold calculation: pROC coords() documentation.