Software - On the Limitations of Models

 Introduction

 Famously in 1987 George Box the well-known statistician wrote in a paper on Empirical Model Building and Response Surfaces that: “essentially, all models are wrong, but some are useful"

  

 On reflection this is – of course – obvious. However (mathematical) models can be very seductive to quants … in much the same way as (fashion) models are to premier league footballers.

  

 But why do quants fall for mathematical models and sometimes lose sight of the (apparently obvious) fact that they are wrong?

  

 The answer to that comes over a long chat and a beer or a coffee - but in this article we will focus on the challenge of the risk manager who has now been given ownership of the internal or ORSA model.
 We have chosen three topics that cause models to have limitations, and that risk managersneed to make allowance for in managing their models.

  

 Model Scope and the Historical Data

  

 Most models tend to limit their scope to the world that is – in some sense – measureable. In other words they look to include what Solvency II would term quantifiable risks.

 What this is typically thought to mean is that a risk is defined, some data is found for that risk factor and a distribution is fitted to that data.

  

 In practice what tends to happen is that some data is found, the risk factor is defined to be whatever the data represents and a distribution is fitted to the historical data.

  

 There are a number of limitations that immediately jump out.

  

 Firstly and perhaps most importantly,this makes the very strong assumption that the shape of the distribution of the risk factor has been the same over the entire history of the data and that this same distribution is applicable over the next 12 months or so. But is this true? Given some of the very different economic paradigms that we have lived through – even in the last 5 years - this seems quite a bold assumption.

  

 Secondly the data assumed is usually some kind of aggregate which has been historically collected and is therefore available for fitting models to. The actual risk factors of the insurer are usually mapped to the aggregated data – and it the company specificrisk factors that the insurer is exposed to financially. But what is the difference between the aggregate and the company specific risk factors? This is known as basis risk and while recognised (at least for financial risks) its inclusion in models is relatively rare.

  

 Lastly, the distribution assumed is invariably some mono-modal distribution (i.e. one hump) whereas there may be a strong argument for a bi-modal (i.e. two or more humps), especially if there is a significant but uncertain event that may or may not occur and the outcome of which will influence the distribution – for example a political election or a piece of proposed legislation.

  

 Associative Dependency

  

 Modelling the dependency between risks is a critical area of models as it substantially reduces the capital that is held compared to just summing the capital required for each risk factor.

  

 In practice, models use associative dependence to model how the risks move together – and therefore how they reduce their capital for the diversification benefit. When we talk about associative dependence modelling we are talking about observing how often two risks move together concurrently and recreating that dependence using statistical distribution.

  

 The first issue we highlight is that all the issues we highlighted for risk distributions in the previous section all apply to associative dependency modelling too. In fact the problems are often more acute as associative dependency modelling needs much more data for reliable statistical estimate. There is rarely, if ever, sufficient data for a reliable statistical estimate. The dependency between two risks is also only reliably captured if the two risks have moved together in the same way on a reasonable number occasions. Which highlights that the extreme event which capital is held against may not only be rare in the historical data series, it may not be in there at all.

  

 The second issue we highlight is the concurrent nature of associative modelling. If one risk tends to occur after another risk – i.e. perhaps the year after – then the dependence will be missed from the model. It doesn’t take much imagination to conceive of events that lead onto other events that lead onto yet further events. The events post 2007 attest to that.

  

 Expert Judging Model Excellence

  

 Another limitation of modelling that we have seen is the problem of elegance. On its own, elegance sounds like a virtuous quality – who would not want an elegant model? However an increasing number of people would appear to be in this camp as they dig into their (black box) models and find some quite critical pieces of expert judgement being applied in the mathematics.

  

 In some models is not uncommon to find that mathematical forms have been chosen because they permit a wave of mathematical cancellations and substitutions which reduce the mess of complexity that represents reality to an apparently simple form.

  

 Yet such mathematical tricks can hide significant expert judgement calls that were they clearly identified to the risk owners in the business – i.e. the people that actually have to manage the money riding on the risk – would be rejected as quite unrealistic.

  

 At best these pieces of expert judgement (let’s call them assumptions) are all separated out, clearly identified and subject to challenge and scrutiny. At worst there is no acknowledgement of embedded assumptions as the mathematical derivation jumps from one line to the next.

  

 Expert judgement and simplification are inevitable in any model. But expert judgement and simplifications need to be transparent and clearly understood. Insurers have got into most trouble with models where mathematics has been used as a substitute for transparency and understanding.

  

 This idea of elegance touches on the question we raising in the introduction about why Quants can be blinded to the fact that their models are wrong. It not uncommon for a Quant to get quite attached to a model they have developed and nurtured. Anyone who has tried to question the validity of aelegantly constructed model - that someone has spent the last 15 years developing - will know that the reaction is similar to that of telling a new mother that her baby looks a bit on the ugly side. You are pretty much off the Christmas Card list for good and physical violence may result.

  

 Conclusion

  

 In conclusion model building is at the core of the insurance sector and we will never stop building models. But awareness of their limitations and the techniques to manage those limitations is crucial to successful risk management.

  

 Simplifications and expert judgement are a fundamental part of building models too and should be recognised as such. However they should be applied with full transparency and in full knowledge of their limitations rather than couched in mathematical derivations where they cannot be readily challenged.

  

 Acquiring a model will often be a good idea, but without the proper care for its limitations you can soon find that it doesn’t do what you thought it was going to do and in few years it could all end in acrimony and the loss of several millions of pounds.

  

 We are still talking from the perspective of risk managers of course… just in case you thought we were back onto premiership footballers.