1、Measuring the Performance of Market-Based Credit Risk Models William Morokoff, Managing Director Quantitative Analytics Research Group Standard & Poors PRMIA-CIRANO Lunch Conference February 2, 2011,Agenda,Introduction A Brief History of Credit Risk Models Credit Model Performance Measures Independe
2、nt Obligors Correlated Obligors Relationship Between Price and Credit Risk CDS Spread-based Models for Credit Transitions,Disclaimer,The models and analyses presented here are exclusively part of a research effort intended to better understand the strengths and weaknesses of various approaches to ev
3、aluating model performance and interpreting credit market pricing data. No comment or representation is intended or should be inferred regarding Standard & Poors ratings criteria or models that are used in the ratings process for any type of security.,Introduction,Themes,Questions: What credit risk
4、information is embedded in the market price of a credit-risky instrument? How does the market price-derived credit risk measure complement fundamental credit analysis? Related Questions: How do you measure the effectiveness of a model for determining credit risk? What credit market price information
5、 is available now to drive models?,Definition and Focus of Credit Risk,Credit Risk The risk of not receiving timely principal and interest payments set forth according to financial contractsPrimary focus of credit models: Single Obligor Default Risk (Probability of Default PD) Single Obligor Credit
6、Quality Transition/Deterioration Risk (Transition Matrices) Portfolio Credit Risk (Correlation, Default Dependencies) Recovery Estimation (Loss Given Default LGD) Exposure at Default (Bank Loan Portfolios),Evolution of Credit Models - A Brief Incomplete History,Evolution of Credit Models 1980s and e
7、arlier,The credit business was mostly buy and hold. Investment grade corporate bond portfolios for institutional investors and retail bond funds, investment grade portfolios of corporate and prime consumer loans. Trading was often driven by interest rate risk. Late 80s saw rise of high yield bonds a
8、nd early CDOs first credit derivatives. Shorting or hedging credit risk was difficult limited ability to sell loans (some syndications of large loans to highly rated companies). Analysis and Modeling: Fundamental, qualitative assessment of individual obligors loan to those you know. Capital charges
9、typically fixed. Large concentration risk and inefficient use of capital in bank portfolios. Illiquid debt market not much to calibrate models to. Merton Structural Models for Probability of Default and bond pricing based on equity markets had not found practical application.,1990s - The Expansion o
10、f Credit Markets,Credit Derivatives begin to grow total return swaps on bonds, credit default swaps, CBOs, synthetic CDOs referencing bank loan portfolios first opportunities to efficiently short, hedge and securitize credit risk to create customized credit risk profiles. Corporations tend to more l
11、everage, lower credit quality fewer extremely high grade bond issuers. Investors attracted to highly rated securitized debt. Securitization of consumer loans (mortgages, auto, credit cards, student loans, ) grows. Concept of Active Credit Portfolio Management forms Credit Value-at-Risk, risk-adjuste
12、d capital allocation, marginal capital for investment decisions, measures of concentration risk and diversification benefits. Banks measure economic capital.,1990s The Rise of Credit Default Models,Merton-style Structural Models for Probability of Default prove effective and commercially viable with
13、 KMV as an industry leader. This provides a more dynamic, equity market-based view of credit quality to compliment fundamental analysis. Reduced form default intensity models to price bonds and options are introduced (Jarrow and Turnbull). Regression-based PD models incorporating firm-specific finan
14、cial ratios and macro-economic variables prove effective, particularly for private firms when sufficiently large default databases are collected. Mortgage foreclosure frequency and loss severity models appear based on consumer characteristics and loan properties.,1990s The Rise of Credit Portfolio M
15、odels,KMV and RiskMetrics develop credit portfolio models in structural model framework with joint default dependencies derived from equity market correlations. KMV model captures changes in portfolio value due to both credit quality transitions and default and becomes a benchmark economic capital m
16、odel for large banks. Default Time models with a Gaussian copula used to create joint default dependencies are introduced (Li) and widely adopted for pricing portfolio credit derivatives. Basel I is adopted to bring uniformity to capital measures in the banking industry and Basel II development begi
17、ns.,2000 - 2007 Active Credit Markets Grow Rapidly,Structured Finance markets experience a huge growth in securitization of mortgages, including new mortgage products (subprime, Alt-A, ARMs, etc.). CLOs market grows fueled by increasing leveraged loan lending and private equity. CDS market explodes
18、and overtakes cash bond market in notional traded. CDS indices introduced creating a liquid index market, as well as a liquid index securitization market for tranches. Numerous credit derivative products are introduced or grow in popularity including ABS CDOs, SIVs, CDPCs, CPDOs, etc. By mid-decade,
19、 credit spreads are extremely tight and investors turn to new products for higher yield.,2007 2010: Credit Crisis Stresses Financial System,In 2007 the housing bubble bursts, property values collapse sharply, and mortgage default rates begin to rise. RMBS bonds and CDOs back by RMBS bonds deteriorat
20、e sharply in credit quality, leading to many defaults and great loss in value. In 2008, financial institutions with large mortgage exposure either fail (Lehman, Bear Stearns, Countrywide), are subject to distressed take-overs (Merrill Lynch, WAMU, Wachovia), or received extraordinary government supp
21、ort (AIG, Fannie, Freddie). For part of this period, credit markets freeze with little lending and extremely high credit spreads. Private mortgage securitization market mostly disappears, along with new-issuance in CLOs.,2000s Wide Spread Adoption of Quantitative Credit Models,KMVs EDFs become widel
22、y accepted as predictors of default (KMV acquired by Moodys in 2002). Other PD models are developed commercially (S&P, Kamakura, etc.) Credit Portfolio models are increasingly used for active portfolio management Default Time/Gaussian copula model becomes industry standard for pricing and hedging sy
23、nthetic CDOs and index tranches with the introduction of base correlation idea Semi-Analytic numerical methods speed index tranche calculations Top-Down portfolio models are introduced for pricing index tranches to address Gaussian copula calibration issues Credit valuation models are introduced tha
24、t price illiquid loans and bonds based on PDs and estimates of market price of risk Consumer asset credit models further developed,Credit Modeling Today,Studying Probability of Default and Credit Transition Models e.g. applying information decay theory for PDs at longer horizons Developing new or up
25、dated models for a range of assets: Residential mortgages, commercial mortgages, municipal bonds, SMEs/private firms, consumer assets Incorporating credit marketing pricing data in models: CDS spreads, Bond OAS, House Price Appreciation indices, etc. Understanding fair credit value vs market prices
26、that incoporate liquidity risk, counterparty risk and supply and demand Expanding credit portfolio models to cover more asset classes, better dependence modeling, credit cycle effects, correlated recoveries, changes in value due to credit migration, etc. Improving measures of counterparty risk, syst
27、emic risk and contagion.,Credit Model Performance Measures,Credit Model Performance Measures,Credit model performance is often determined by a models ability to: Rank obligors by default/downgrade risk to discriminate between potential defaulters and non-defaulters Anticipate realized default/downgr
28、ade rates: compare probabilities to observed rates Performance evaluated with statistical measures on a validation data set Method 1: Cumulative Accuracy Profile (CAP) Sort obligors from riskiest to safest as predicted by the credit model (x-axis) and plot against fraction of all defaulted obligors.
29、 Accuracy Ratio = B / (B + A),Credit Model Performance Measures,Method 2: Receiver Operating Characteristic (ROC) Plot the distribution of model scores, R, for defaulters and non-defaulters Note: For a perfect model, no overlap in the distributions whereas for a random / uninformative model, 100% ov
30、erlap (i.e., identical distributions) Specify a cutoff value C such that scores less than C are potential defaulters and rank scores higher than C are potential survivorsGiven C, 4 outcomes are possible: Incorrect decisions: R C and default (Type I error) Correct decisions: R C and survive,Credit Mo
31、del Performance Measures,Method 2: ROC (cont.) Define Hit Rate (given C), HC = # of predicted defaulters/total defaulters Define False Alarm Rate (given C), FC = # of non-defaulters predicted to default/ total non-defaulters Plot HC vs. FC for all values of C to generate the ROC curveThe larger the
32、ROC (i.e. the area under the ROC curve - shaded region), implying higher hit rate to false alarm rate, the better the ranking model Note: Hit rate corresponds to 1 Type I Error,Credit Model Performance Measures,Relationship between ROC and CAP If the same weight is attributed to Type I vs. Type II e
33、rrors, it can be shown that ROC and CAP communicate the same information AR = 2 x ROC 1(Engelmann, Hayden and Tasche, “Testing Rating Accuracy,” Risk, January 2003) ROC is however a more general measure as different weights may be given to Type I and II errors. Typically, more weight may be given to
34、 Type I vs. Type II error Example: Giving a loan to a defaulting firm (I) vs. losing potential interest income by not extending credit to a non-defaulting firm (II) We assume equivalence of ROC and CAP for this study (i.e. equal weights to I vs II errors),Performance Measures: Independent Obligors,R
35、ankings based on PD vs. Default Prediction,Typical measures: focus on correct identification of defaulters vs. non-defaulters (binary) Credit models: rankings may be based on relative risk (probability or likelihood) Implication: A perfect (PD based) ranking model cannot have a perfect Accuracy Rati
36、o. Even the highly ranked “buckets” can have defaults (albeit low) The lowest ranked buckets can have survivors (1-PD) Example Very large pool categorized into 10 equally weighted Risk Category (RC) buckets on the basis of PD Safest bucket (RC1) has PD 2bp, RC2 has PD 4bp, RC3 has PD 8bp, and so on
37、i.e., each subsequent bucket twice as risky in PD terms Large pool (theoretically infinite) ensures realized defaults in each RC bucket equal expected defaults,Rankings based on PD vs. Default Prediction (Large Pools),Even with perfect risk categorization on a PD basis, accuracy ratio is only 71.66%
38、.Default rate matters: If the PDs for all buckets are 5 x larger, CAP curve remains the same, but the AR = 78.19%.,Additional Observations for Large Pools,Even if obligors are categorized on the basis of “noisy“ PD estimates, all results hold provided expected PD for each bucket is monotonic Large p
39、ools of uncorrelated obligors diversify away “idiosyncratic“ defaults perfectly making realized defaults always equal to expected defaults Buckets defined in terms of PD ranges both overlapping and non overlapping will yield similar results PDs range can be viewed equivalent to a noisy PD Small “noi
40、se“ component can be made to correspond to distinct PD ranges Large “noise“ component can be made to correspond to overlapping PD ranges,Finite Sample Size: Impact of Smaller Pools,Size of the pool has most impact on AR values. Similar results hold for smaller pools as large pools with regard to PD
41、distribution within buckets, both non-overlapping and overlapping.,Finite Sample Size: Impact of PD levels,Study impact on AR of reducing absolute PD levels (and PD differences) across buckets Base case: Safest bucket 2 Bps PD; each subsequent bucket PD 2x preceding one Other cases: Safest bucket 2
42、Bps; subsequent buckets 2.5x, 1.5x, 1.25x, 1.1xAbsolute levels of PD determine the level of heterogeneity in the pool Small PDs cause larger variability in number of realized defaults, including frequent instances of low or no defaults Small PD differences across buckets imply more homogenous pools,
43、 making default rates similar across buckets Both factors impact AR negatively, even though the categorization is “perfect“ in PD terms,Performance Measures: Correlated Obligors,Correlation,Fact: - Correlation does not affect a pools expected defaults, - but correlation introduces more variability i
44、n realized defaults Recall: AR being a non-linear function of defaults, i.e., ED(AR) AR(ED), any change in shape of the default distribution introduced by correlation can be expected to impact AR Correlation modeled in the context of a Systemic Risk factor model For a pool of N obligors, most generi
45、c model would be to specify a joint distribution of u1, u2, , uN, ui U0,1 through a copula function and define obligor i as a default if ui PDi Factor model representation allows for systemic/idiosyncratic separationNote: Z may be dependent on several independent factors. We assume a single factor m
46、odel for the following analysis.,Correlation (continued),Key feature of model: Conditional independence i.e., conditioned on systemic latent factors: Unconditional PDs for correlated obligors become conditional PDs for independent obligors A given periods observation corresponds to a specific system
47、ic latent factor realization and can be viewed as a description of a given state of the economy Unconditional results can be viewed as an average (over systemic factors) of conditional, independent results,Correlation (continued),Base Case Example: Z N(0,1), fi N(0,1) Unconditional PDsConditional PD
48、s: r = 20% r = 80%,Correlation: Large Pool,AR can be calculated using semi-analytic methods Given Z, determine conditional PDs. PDs determine exact default rates for each bucket AR|Z can be calculated analytically Integrate AR|Z over Z to obtain unconditional ARs for different values of r.Main resul
49、t: Increasing correlation improves AR for large pools, except for maximum correlation of 100% Large pool eliminates variance around conditional default rates Increasing correlation introduces more heterogeneity in conditional PDs for negative shocks,Correlation: Finite pools 1000 observations per Bu
50、cket,Simulation method: Estimate mean AR over 100,000 trials Conditioned on Z, finite pool buckets have a default rate distribution instead of a point mass for large pools, bringing into play the effect of the non-linear relationship between AR and default rates, ED(AR) AR(ED) Main observation: Correlation improves mean AR for heterogeneous (in PD) portfolios but is detrimental to homogenous portfolios,
