All articles
Credit scoring
Model risk
Underwriting

Credit scoring you can defend

AI underwriting is easy to demo and hard to defend. This is the scorecard methodology we hold ourselves to: weight-of-evidence features, validation with confidence intervals, drift monitoring, and a decision that survives a model-risk review.

Levmetrics Team

Document intelligence & fraud forensics

7 min read
Credit scoring
On this page⌄

A credit model is easy to demo. You feed it an applicant, it returns a number, and the number looks confident. The hard part comes later, in the room where a lender's model-risk team asks the questions that decide whether the model is allowed anywhere near a real lending decision: How predictive is it, with what confidence interval? Which features carry the weight? How will you know when the population drifts? Can you explain a decline to the applicant, and to a regulator?

This is the methodology our credit-scoring engine is built around. None of it is novel; that is the point. It follows the practitioner literature, mainly Siddiqi's Intelligent Credit Scoring and the model-risk guidance in the US Federal Reserve's SR 11-7, because "defensible" beats "clever" every time a real loan is on the line.

A score is only as good as its paper trail

The first design decision is that a score never travels alone. Every decision the engine emits carries, in one payload:

  • a score on the familiar 300 to 850 scale, plus the probability of default it maps to,
  • the IFRS 9 Expected Credit Loss for the exposure,
  • the top reason codes that drove the score,
  • an adverse-action notice when the outcome is a decline,
  • and a model card describing the model that produced it.

The reason is simple: a decision you cannot reconstruct is a decision you cannot defend. Tying the output back to the exact model and policy that produced it is what turns "the AI said no" into an auditable, contestable decision.

Picking features that actually predict

Before any model is fit, the question is which signals deserve to be in it. The classical tool here is Weight of Evidence (WoE) and its summary statistic, Information Value (IV). WoE re-expresses each bin of a feature as the log-odds of good versus bad outcomes; IV rolls those up into a single number for how much the feature separates the two populations.

IV gives you a blunt, honest filter. Using the strength bands from Siddiqi (2017, p. 184):

Information Value Band Decision
< 0.02 Not predictive Drop
0.02 to 0.10 Weak Investigate, may drop
0.10 to 0.30 Medium Keep
0.30 to 0.50 Strong Keep
> 0.50 Suspiciously strong Audit for target leakage

That last row is the one people forget. A feature that looks too good is usually leaking the answer, for example a field that is only populated after the loan has already gone bad. Treating "suspiciously strong" as a red flag rather than a win is part of being honest with yourself before a regulator does it for you. (Zero-count bins are handled with Laplace smoothing, the half-count convention from Siddiqi p. 183, so a single empty bucket does not blow up the math.)

Validating with confidence intervals, not point estimates

A single accuracy number is not a validation. The engine reports the standard discrimination metrics, Kolmogorov-Smirnov (KS), Gini, and AUC, and it reports each one with a 95% confidence interval from a 1000-resample bootstrap.

This matters more than it sounds. A point estimate of "Gini 0.70" tells you nothing about whether the next sample would land at 0.68 or 0.55. The confidence interval does. A model card without confidence intervals is a red flag in any model-validation memo, so we put them on by default rather than as an afterthought.

The verdict bands, again from Siddiqi (2017, p. 260), are deliberately conservative:

KS Gini AUC Verdict
< 20 < 0.40 < 0.70 Red, rebuild
20 to 30 0.40 to 0.55 0.70 to 0.775 Amber, marginal
30 to 50 0.55 to 0.80 0.775 to 0.90 Green, production-acceptable
> 50 > 0.80 > 0.90 Audit for target leakage

Note the top band again: extremely high KS is not a trophy, it is a prompt to go looking for leakage. The honest range to live in is "green," not "perfect."

Watching for drift after you ship

A model is validated against the population it was built on. Production populations move. The Population Stability Index (PSI) measures how far the live score distribution has drifted from the training distribution, with the standard action bands (Siddiqi 2017, p. 365):

PSI Band Action
< 0.10 Green No action
0.10 to 0.25 Amber Investigate, monitor over one to two periods
≥ 0.25 Red Recalibrate or rebuild

PSI alone only tells you that something moved. Pairing it with a Scorecard Characteristic Index, the per-feature average point shift attributable to the change, tells you which feature moved, which is the difference between a useful alert and a vague one.

Turning a probability into a decision

Discrimination metrics rank applicants; they do not set a cutoff. A gains table does that: bin the population best-first and read off the cumulative bad rate and approval rate at each band, so you can pick the cutoff that hits your target approval rate or your target bad rate.

From there, probability of default feeds the IFRS 9 impairment calculation directly:

Expected Credit Loss = PD x LGD x EAD

where PD is the probability of default, LGD the loss given default, and EAD the exposure at default. The same probability that drives the approve/decline decision also drives the loss provisioning, from one consistent number rather than two models that disagree.

Explaining every decision

A decline is not finished until it can be explained. Two artifacts handle this:

  • Reason codes. A SHAP-style decomposition attributes the score to its drivers, so "declined" comes with "because debt-to-income was high and recurring income was thin," not a shrug.
  • Adverse-action notices. When the outcome is a decline, the engine generates a notice consistent with the content requirements of the US FCRA and ECOA Regulation B. The applicant gets the principal reasons, which is both the law and the decent thing to do.

Fairness by construction

The cleanest way to avoid a discriminatory model is to never feed it the inputs that enable discrimination. Features are limited to financial-behavior signals, and each is registered with its regulatory class. No protected-class attributes, and no proxies for them. Fairness is a property of the feature set first, not a filter bolted on at the end.

Why this is the boring version on purpose

There is a flashier way to write about AI underwriting. But a lending decision is a regulated act with a real person on the other side, and the version that survives a model-risk review is the one built on feature selection you can justify, metrics with confidence intervals, drift monitoring that names the culprit, and decisions you can explain. Defensible first. Clever second.

A note of thanks

Much of how we think about scorecards, the WoE/IV discipline, the validation bands, the drift monitoring, traces back to one book: Naeem Siddiqi's Intelligent Credit Scoring. It is the rare text that is both rigorous and genuinely practical, and it has shaped this work more than any other single source. If you build credit models for a living and have not read it, start there. Thank you, Naeem.

References

  • Siddiqi, N. (2017). Intelligent Credit Scoring: Building and Implementing Better Credit Risk Scorecards, 2nd ed. Wiley/SAS.
  • Federal Reserve / OCC (2011). SR 11-7, Supervisory Guidance on Model Risk Management.
  • IASB (2014). IFRS 9 Financial Instruments, section 5.5 (Impairment).
  • Lundberg, S. & Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS.
  • World Bank ICCR (2019). Credit Scoring Approaches Guidelines.
  • FCRA (15 U.S.C. section 1681m) and ECOA Regulation B (12 CFR section 1002.9), adverse-action notice content.

More from the blog

Lifecycle
Lifecycle
Observability

A SpaceX rocket crashed into the Moon. Deployment is not the end of the lifecycle.

A SpaceX Falcon 9 upper stage crashed into the Moon on August 5 after 18 months derelict. What the lunar impact says about systems you stopped maintaining.

10 min read
Fraud
Fraud
Lending

How to spot a fake or tampered bank statement

Altered and fabricated bank statements are one of the most common fraud vectors in lending. Here is what to look for, why manual review misses so much of it, and how to catch tampering at scale.

4 min read