Milliman’s latest 10T “analysis” (published May 18) purporting to compare 10T and VantageScore 4.0 is statistically false and misleading. It needs to be withdrawn immediately. Paid for by VantageScore’s competitor, the paper’s only aim appears to be to mislead market participants with statistical swindles.
Swindle #1: Using Only One Credit Score When Mortgage Lenders Are Required to Use Three
Anyone who has ever gone through the process of applying for a mortgage knows that lenders pull three credit scores and use the middle score to qualify the borrower and underwrite the loan. Milliman’s analysis incorrectly only uses one score from one credit bureau. For many mortgage loans, that one specific bureau credit score would not be the middle score and, therefore, would not be the score used by Fannie Mae, Freddie Mac or the lender making the actual mortgage loan decision. In the real world, most mortgage lenders use the VantageScore 4.0 middle score, which reflects a more robust assessment of the borrower’s risk. This error renders Milliman’s entire analysis unreliable and misleading. The method of using the middle VantageScore 4.0 is already being successfully used by many of the nation’s 30 largest mortgage lenders in real-world underwriting environments with great success.
Swindle #2: Excluding the COVID Stress Period
The COVID stress period was one of the most meaningful reality checks for credit risk performance. Milliman’s analysis excludes that period, instead cherry-picking a dataset that omits the very macro-environment where models are supposed to prove their value. Losses rarely show up when the sun is shining on the economy. Instead, mortgage defaults tend to appear when the economy is tough, like during COVID. VantageScore 4.0 performed best when macro-economic conditions were the most difficult, which explains why we don’t hesitate to include the COVID period in any of our analytical results and comparisons. However, Milliman excludes this period to falsely make 10T results look better than they actually are.
Swindle #3: Misleading the LLPA Analysis
The very premise of Milliman’s pricing analysis rests on the deeply flawed assumption that the same LLPA grids used for legacy Classic FICO credit scores will automatically apply to VantageScore 4.0. That is not how mortgage pricing works. LLPAs (Loan-Level Price Adjustments) are the risk-based pricing grids used by the GSEs to determine how much a mortgage loan should cost based on a borrower’s risk profile. In simple terms, the credit score is one input used to estimate risk, and the LLPA grid translates that risk assessment into specific mortgage loan pricing. But the grid is not tied to the raw score numbers alone; it is calibrated to how well a specific credit model predicts actual loan performance. Because different credit scoring models are intentionally designed differently, each model requires its own calibrated LLPA framework. FHFA policy and GSE implementation guidelines explicitly recognize this and VantageScore 4.0 LLPA grids align pricing with the actual risk predicted by VantageScore 4.0.
Swindle #4: Presenting Results in Aggregate
Using vintage aggregation instead of vintage-level performance is a convenient way to blur out the details that matter. By rolling ten years of vintages into a single aggregate view, the Milliman study avoids showing how the models performed across dramatically different economic conditions, policy environments, and borrower behavior regimes. When everything is averaged together, inconvenient differences for 10T start to disappear. Established best practices in credit model evaluation, which VantageScore follows, require both aggregate and vintage-level disclosures, especially across periods involving economic expansion, contraction, and unprecedented policy intervention. If Milliman disaggregates mortgage loan vintages, the analysis would reveal the truth: VantageScore 4.0 is better.
Swindle #5: Inclusion of Loan Modifications in the Default Definition
Milliman’s 10T analysis also conflates two different types of mortgage loans: a) mortgages with loan modifications and b) mortgage defaults. These are not the same and they are not interchangeable concepts. Treating them the same is a faulty departure from standard credit practice. A mortgage loan modification is not the same thing as a 90-day delinquency. Loan modifications are driven by servicing rules, investor policies, and loss mitigation programs, and they do not inherently represent a borrower’s credit failure. But, the Milliman study treats every modification as if it were evidence of default, conveniently inflating delinquency rates and contaminating the analysis along the way.
Swindle #6: A Vague, False Definition of ‘First-Time Homebuyer”
It is unclear what data source Milliman’s 10T analysis uses to define “first-time homebuyers,” since that information cannot be reliably determined from a credit report alone. As mortgage tradelines disappear from credit reports after seven years and cash purchases leave no credit file footprint, the Milliman approach provides a failed measure of first-time homebuyer status. As a result, the paper’s conclusions related to first-time home buyers cannot be relied upon.
— By Dr. Andrada Pacheco, EVP and Chief Data Scientist, VantageScore
