insights from a decade of model validations

Insights from a decade of model validations

Date: June 26, 2020

VantageScore’s arrival in 2006 brought the first meaningful competition to the credit scoring industry and introduced several new best practices, including publication of annual model validation results. Validations test credit scoring models to determine whether they continue to meet performance expectations. Publishing the results of those validations promotes greater transparency in the credit scoring industry and assists financial institutions with model governance.1

After 10 years, VantageScore still remains the only credit scoring model developer that publishes annual validation results. (Competing developers typically validate models only at their inception.) We’ve used our 2016 Validation white paper to report not just our latest findings but also to reflect on trends that emerged over the course of a decade of analysis.

As in years past, the 2016 validation results found VantageScore credit scoring models consistently outperforming in-house models from the three national credit reporting companies (CRCs: Equifax, Experian, and TransUnion) in the performance dimensions that matter most to lenders—predictiveness, universe expansion, and score consistency. Once again, this performance superiority bore out across all three major lending categories: mortgage, auto, and bankcard.

The study further examined how the three VantageScore models (VantageScore 1.0, 2.0, and 3.0) performed during the time period of 2013-2015, as well as over the past years.

“We put the validations process in place to help pinpoint opportunity as risk profiles shift and to introduce transparency into what was an otherwise opaque industry,” said Sarah Davies, VantageScore senior vice president for analytics, product management, and research. “All the VantageScore models surpass competitor models in both normalized and volatile risk periods, demonstrating unparalleled long-term stability.”

The results

VantageScore models are Recession-resistant: Over the last 10 years, the economy has seen both a recession and recovery. For this study, VantageScore validated all its models on consumer originations during the decade’s most volatile risk period, 2007-2009, and compared the results for VantageScore 1.0, which was developed pre-recession (2003-05); VantageScore 2.0, which was developed mid-recession (2007-10); and VantageScore 3.0, which was developed post-recession (2009-12).

It is not surprising that VantageScore 2.0, developed using data from the recession time frame, performed optimally on credit files taken from that period. VantageScore 3.0 performance, however, was also strong, coming within a half Gini-point2 of VantageScore 2.0 for the bankcard, installment, and auto industries. For mortgages, where score performance was most challenged during the recession, the Gini for VantageScore 3.0 (59.3) was within three points of that for VantageScore 2.0 (61.9), while VantageScore 1.0 lagged version 2.0 by nearly nine Gini points (53.2).

With probability of default (PD), shift happens: Although the validation process has shown that all VantageScore credit scoring models were highly effective risk assessment tools during the last 10 years, the risk levels associated with specific score cutoffs have varied over the years. Analysis of shifts in the probability of default (PD) associated with specific score cutoff values revealed considerable variation over the past decade—and showed that risk levels have finally returned to near pre-recession levels. (This variation in risk levels makes it critical for lenders to regularly and systematically review changes in the risk associated with their score cut-offs and to update policies accordingly to manage risk exposure effectively.)

VantageScore remains best in class: Despite economic instability, VantageScore 3.0, in particular, was shown to deliver superior and stable performance:

  • Predictive power: During the performance period of 2013-2015, VantageScore 3.0 outperformed all CRC models by an average of 1.7%-3.4% in key industries.
  • Consistency across the board: The accuracy of VantageScore 3.0 is also highly consistent across all three CRCs (Experian, Equifax, and TransUnion) with a minimal variance of just .34 Gini points between the CRCs.
  • Universe expansion: VantageScore 3.0 continues to score 30-35 million more consumers than competing credit scoring models, 9.5 million of whom are Hispanic- or African-American consumers.

For more information on the validation process and study results, please visit VantageScore.com/Validation2016

1Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency, Supervisory Guidance on Model Risk Management, April 4, 2011: http://www.occ.treas.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf.
2The Gini coefficient of a credit score compares the distribution of defaulting consumers with the distribution of nondefaulting consumers across the credit score range. The coefficient has a value of 0 to 100. A value of 0 indicates that defaulting consumers are equally distributed across the entire credit score range. A coefficient value of 100 indicates that the credit score has successfully assigned all defaulting consumers to the lowest score possible. A Gini coefficient of 45 or greater is considered a good result by industry standards.