The Strengths and Weaknesses of Operational Risk Frameworks
Summary
Operational Risk Frameworks put some structure around a vast expanse of risk management can should be best viewed as an imperfect answer but far better than any of the alternatives. They are littered with acronyms (which I've listed below for information). Their value is best appreciated at the 30,000 feet level where they can provide a subjective summary of the operational processes of the bank indicating areas which are unable to supply control data as well as areas where the controls are being reported as stressed. There's no other tool that can indicate the condition of the control processes across the bank, but their construction and operation is riddled with subjective decisions that they are unlikely to be a useful predictor operational losses or where the next big blow up might occur. They work best where the potential losses can measured precisely and are at best indicative where it cannot, especially if the loss is dependent on variable external factors. A good framework should be transparent about the level of objective and subjective data within it. Any gaps in the ORF or any of the KRI coverage should be escalated. A department or process that hasn't got the time to deliver MIS about its operation can hardly claim to be under control!
The Structure of
the ORF
The ORF is one of the main tools currently used by banks in their Operational Risk Management Framework. At its simplest, a bank's operations can be viewed as a collection of business processes (selling, trading, advising, settling, reporting) and a collection of control processes (reconciliation, validation, confirmation) to confirm that the first collection are correct. The ORF should cover all the key processes above, irrespective of whether it is a business activity or a control, performed manually or automated. Having documented all the processes the BCBS recommend that firms first estimate the inherent risk in a process before any controls are applied but this is usually about as useful as estimating the survivability of an aeroplane with no engines or control surfaces and also likely to lead to a similar incredulous response. Banks then look at the controls over them to estimate how much risk of loss they contain after the control operation deemed the residual risk with some guesstimates of the probable loss and its frequency. The most significant controls are identified and performance thresholds for these RCSAs need to be set. This is where the weaknesses of the approach becomes apparent. The thresholds are usually set by the operators with validation by an independent team such as Operational Risk or Internal Audit. If the thresholds are too strict the metric will trigger RED too easily. It's very common for managers to set a "perfection" control metric which they then regret as they have to explain the constant breaches. I've even seen managers attempt to define metrics for controls that they want that do not exist in this process! At the other extreme if the metric is too lax it will never trigger, even if the function is in meltdown. Similarly, the independent teams may not be 100% impartial, Operational Risk may feel it's in their interests to generate some RED metrics to reinforce the importance of their function. If we compare this to a VaR model for market risk you can see far less subjective inputs but also a process that's leading to a monetary loss.
The other problem with any self-assessment process is the obvious lack of segregation. Complete falsification of records would be rare but being somewhat "economical with the truth" is a risk if neither Audit or Operational Risk review how the reporting should work. Too often audits only go as far as confirming that the reporting complies with the documented procedures. Managers may also de-prioritise (or allow to lapse) controls that are not part of the self assessment reporting, which may not reflect their actual importance, and the ORF will be completely unaware.
The other weakness here is that some monetary consequences are pretty explicit while other are very hard to set even within a range. For example the risk of a non-receipt on a nostro is clearly the amount itself, but failing to follow market conduct rules could lead to client lawsuits and regulatory sanctions which are far less easy to estimate. The ORF usually lumps all this together with assumptions which ought to be documented (but often are not). ORFs like most other risk tools are quite historically focussed. Just as a risk committee in 2006 would never have imagined their market stress scenarios would be exceed by actual market moves, it would not have predicted the severity of the fines imposed by regulatory bodies in a climate of "Bank Bashing" after government bailouts and the creation of new regulatory bodies with wide ranging powers. What we really need is a framework that clearly distinguishes between hard and soft estimates and also clarity on whether the estimate is of future losses or immediate spot losses. In my nostro example above we have a spot loss based on immediate data, which is not a hard projection for nostro losses in the future. This brings up another topic, how do operational losses change over time, if at all. I think individual losses broadly follow these characteristics;
|
Characteristic |
Example |
Expected Loss |
Unexpected Loss |
|
Static |
· Nostro break · Unreconciled debit balance in the GL |
The loss is restricted to the size of the amount, and it is not likely to change over time. |
The loss could only grow if it was found that actually it
was an net amount and the credits were valid whilst
the debit they had been netted against was unrecoverable. |
|
Linear, i.e. the loss grows steadily over time. |
Minor fraud · Padding expenses · Overcharging · Theft of client dividends ·
Late trade reporting fines |
The loss will grow over time as the fraud costs a static amount each time. |
Unlikely, for these examples, with the exception of penal fines from the authorities if for example they deemed the late trade reporting was wilful non compliance rather than just operational issues. |
|
Stochastic |
· Dealer fraud · Position control error · Collateral collection failure |
The loss might potentially be a profit but the magnitude of
the impact is dependent on market moves over the length of time the item is
left exposed to market moves. A VaR/
initial margin calculation would be an appropriate model to use in such
cases. |
If the position gets carried through a period of market stress then the magnitude of its impact could be as severe as that of any proprietary position. |
|
Random Noise |
· Regulatory breaches with penal fines |
A truly random process where personal, political similar external social factors affect the outcome. |
Regulatory and legal processes that are not bound by rules of precedent and may also view non-voting, non-domestic rule breakers as an easy target for extremely large penalties. |
Predictive models for expected losses can be built for distributions that model the first three provided there is sufficient data to calibrate them. The last is truly random, more like the noise hiss coming out of an un-tuned radio, (compared with a stochastic process which is sometimes described as the path a very drunk person takes).
I think a better ORF model would delineate between hard data and soft data, so that the inherent uncertainty in say regulatory fines is not confused with the more factual problems such as unreconciled GL accounts.
In Conclusion;
1. ORFs are not perfect, their biggest weaknesses are the level of subjectivity and the lack of independent data. The framework should explicitly recognise this and build compensating processes.
2. Gaps in coverage are serious deficiencies and must be escalated.
3. Operational loss types display different characteristics and this should be part of the framework.
4. The framework should be able to explain how hard or soft any loss estimate actually is.
Common Acronyms;
ORF Operational Risk Framework - A process to document and report the performance of the key processes and controls across the organisation. They do not cover every process and control usually, but work on the basis that the performance of the key controls is a good guide to the performance of the more minor controls.
RCSA Risk Control Self Assessment - The operator of a control self assesses it's performance and this enables the ORF to standardise its MIS. So for example a position control might set the following standards; < 5 breaks more than $10k in size = GREEN, 6-20 breaks more that $10k in size = AMBER, >20 breaks more than $10k in size = RED.
CST Control Sample Tests - Are tests that reperform a process reporting a KCSA to provide confidence that the control is really operating as documented.
KPI Key Performance Indicator - metrics separate from KCSAs which give additional information about the performance of a function and any stress it might be undergoing. They could be such things as exception queue length, system batch performance times. They can be very useful in trend analysis or observing seasonal factors.
KRI Key Risk Indicators - Are similar to KPIs except that they focus on metrics which amount to risk, such as the number of external unauthorised system access attempts in a period of time.
BPM Business Process Mapping - A Structured (usually visual) description of the bank's processes and controls to clearly map out inputs, outputs, actors, timing, dependencies, controls.