What Predicts Negative ROI? — Right Fidelity AI

Model Performance

How Well Can We Predict Bad Programs?

A gradient-boosted classifier trained on non-tautological features—things a decision-maker would know before seeing outcomes—achieves strong predictive performance across 5-fold cross-validation on 65,935 programs.

0.927

AUC

85.1%

Recall

72.6%

Precision

0.784

F1 Score

Gradient boosting classifier, 5-fold cross-validation, non-tautological features only (no outcome leakage). N = 65,935 programs; 22,692 negative ROI.

Feature Importance

What Drives the Prediction?

The model relies heavily on just three features: the institution's overall track record, the field's national success rate, and credential level. Together these account for 93% of the model's predictive power.

Feature Importance (Gradient Boosting)

Key Findings

What the Model Tells Us

Finding

A school's overall track record is the strongest predictor (53%). If the institution has lots of bad programs, any new program there is likely bad too.

Finding

The field's national success rate is second (27%). Cosmetology programs are negative almost everywhere—knowing the field tells you a lot.

Finding

Credential level matters (13%). Certificates and associate's degrees carry more risk than bachelor's and graduate degrees.

Implication

What This Means for a University CFO

Check your institution's overall track record first. If more than 40% of your programs have negative ROI, every program should be under review—not just the obvious ones. The institution-level signal is so strong that individual program characteristics are secondary.

← Prev Q2: Field vs. Institution Next → Q4: Policy Simulation