Model Validation in Banking Series, Part 1: Conflicts Between Modeling and Validation
If you are managing or setting up an internal validation team in a bank, this article is for you! In a series of blog articles in the coming weeks, we will elaborate on the prevailing trends in model validation in the banking industry. Internal validation teams are preparing themselves more and more for taking the lead (or playing an active role) in effective management of models (i.e., the portfolio of models) that the institutions possess as well as model risk. Below, we kick the discussion off with a core function that any validation team should play on the way to effective management of models and model risk: independent challenge.
Why is there conflict between modeling and validation? And, why is it a good thing?
Everybody agrees that no model in the world is perfect! Indeed, no model will ever be! No matter how good a modeler you are or how much effort you put in your model, you will always face some model risk. A modeler will always have some work to do to improve a model. (S)he will always be looking for and carrying out some actions to make a model even more good-looking than it is today.
While researching and executing these actions, a modeler is (almost) always constrained by its environment. (S)he has to manage many conflicting constraints like managing data, conflicting or badly defined model requirements, diverging expectations, scarce resources, people, money and so on. Modeling is the art of building something that works just fine no matter how nasty the environment or the organization is.
Validation, on the other hand, lives in a different planet. Often criticized to be talking from the ivory tower, a validator does not really understand or even get a faint feeling of the nasty obstacles that a modeler faces. The validator always talks about this ideal model or vision that only people from Pleasantville could imagine. (S)he formulates recommendations to the modeler often in the form of ‘should do’, ‘should document’, ‘must improve’ and so on.
But, unlike a Pleasantville dweller, the validator’s life is not that easy either. A validator should (always) keep a sharp eye on finding out the fractures and the cracks in a model. If possible, (s)he should also be able to tell (just tell, not more) how to repair or patch up these. Moreover, the validator is usually the main responsible to figure out and explain what a model can and cannot do (obviously, modelers could also tell that but they usually do not, for understandable reasons).
Such divergence of lifestyles (well, a little bit exaggerated here for the sake of illustration) explains why there is (almost) always a conflict between Modeling and Validation. But, is this conflict devastating for the world? The answer is short and simple: No! On the contrary, it is needed. Everybody in the world (except the modeler and the validator) benefits from this conflict.
At the end of the day, the validator’s focus on the transparency, model limitations and ‘what should be done’ is complementary to the modeler’s focus on the possible actions and ‘what could be done’. Experience shows that adopting only the modeler’s perspective and watering-down the validation activity gives a false sense of comfort (“We did our best, so it is good!”), and makes the institutions blind against the inherent limitations and boundaries of models’ capabilities, which generally leads to issues linked to under-used, over-used or misused model outputs. On the other hand, adopting the validator’s focus alone by ignoring the modeling realities would lead to no action or, to no model at all.
Some modeling experts say that the best way to develop a good model is to start with a bad one and improve it. It is only possible through this on-going struggle between the modeler and the validator that a bad model gets better and better over time. And, it is only possible through this battle that an institution knows best when a model can be used and for what purpose, when a model gets dangerous and what to do in such case.
About the Author: Dr Aykut Ozsoy is Manager with Risk Dynamics, specialized in banking risk models validation and optimization. He also works actively on Risk Dynamics model validation frameworks and methodologies. Click here to email Aykut.