The new approach seeks to emulate the constant evaluation of mistakes make by certain sectors – most notably the aviation industry – where data from “black box” flight recorders in aircraft are used to identify and understand the cause of both major accidents and near misses, and, in turn, to drive constant improvement in the safety of the sector as a whole. In 2017, through the consistent application of this “open loop” approach, there was not a single passenger jet crash for the major airlines anywhere in the world.
The Black Box Thinking framework was developed by Matthew Syed, author and broadcaster.
Applying this approach to pension schemes, the report has sought to address three key questions:
1. What mistakes are being made by DB trustee boards today? What are the errors that emerge in strategy setting that Black Box Thinking can be applied to?
2. How do boards evaluate errors? Do boards have a culture of recognising and measuring errors and are they able to learn from their mistakes to improve future decision making?
3. Ways to improve. Based on the analysis, what can schemes do?
In assessing errors, with the exception of quantitative information on fund deficits, there are few, if any, yardsticks that can be used to measure mistakes in DB pension schemes in the same way that mortality is used in aviation. The pensions industry is currently characterised by a lack of measurement and hence an absence of the data to make an informed judgement.
Professor David Blake, Director of the Pensions Institute, and one of the authors of the report, said: “It is clear from our research that too many pension schemes are making the same mistakes again and again. As an industry, trustees are not good at evaluating their failures, learning from them and sharing this knowledge. If we can emulate the open-loop ‘Black Box Thinking’ approach that the airline industry uses to such great effect, we might actually be able to address many of the issues facing DB pension schemes in the UK at the moment. Examples include using post-mortems with lessons learned where things go wrong, and using pre-mortems as mechanisms for avoiding future mistakes, such as considering a new investment idea, a move in liability-driven investing, or a forthcoming valuation or enhanced transfer value exercise. There is also an important role for the regulator to play as a clearing house for post-mortems of failed schemes and the lessons that can be learned.”
Kerrin Rosenberg, CEO of Cardano and sponsor of the report commented: “Many trustees have been early adopters of tried and tested risk management practices that have generated good outcomes for both their members and sponsors. However, there are also a large number who haven’t. Making decisions that don’t deliver the hoped for outcomes is not wrong in itself, if lessons can be learnt and shared. However, making the same poor decisions many times over is the problem. If we can create a culture of shared learning, we may be able to give the pensions industry the knowledge to avoid the mistakes of the past and of their peers.
“This exciting report presents a radical new approach that needs to be adopted quickly and effectively, so that we can move to a far more efficient governance system that is in a state of continuous improvement.”
The key findings developed from a series of interviews with senior trustees and industry experts are summarised below:
Key mistakes
In decision making, Black Box Thinking suggests mistakes occur when processes that mitigate cognitive biases are absent. Common mistakes include:
(1) Focusing on what trustees know and failing to focus on areas where trustees had little expertise or understanding.
(2) The separation of investment and funding decisions, and the failure to challenge the sponsor’s recovery plan or dividend policy.
(3) A short-termist attitude, especially by sponsor-appointed trustees, e.g., in respect of investment performance and hedging strategies.
(4) Failing to recognise biases in others, such as the career concerns of finance directors who need to show that the company is doing well on their watch, or of advisers who temper their advice to clients to avoid losing the contract.
(5) Being distracted from their main focus, e.g., by being drawn into making decisions that are really outside their remit, such as resolving disputes over benefit payment or implementing adviser recommendations, when this should be an executive task handled by a project manager.
Evaluating errors
The report suggests that schemes do not systematically measure mistakes, although there are individual examples of best practice. Issues raised include:
(1) An absence of clear goals and instead a vaguely defined long-term strategy which cascades down into poor measurement of errors in decision making.
(2) Even in schemes with clearer goals, the timescale is often so flexible that the problem can be repeatedly kicked down the road.
(3) While all boards pay attention when the size of the deficit changes, there are few other commonly agreed metrics that boards can monitor that would help to detect mistakes being made.
(4) In the absence of a narrow and clearly defined set of indicators, there is the danger that trustees have so much information potentially available they have trouble identifying signals from noise.
(5) There are two issues in particular that emerge as result of a lack of benchmarks and indicators: risk not being measured and poor spending decisions, particularly on consultants fees.
The report goes on to highlight that there is no industry-wide approach for trustees and boards to learn from their mistakes, with many boards not having a culture of seeking out and revealing mistakes – consistent with a closed loop mindset. Examples raised include:
(1) No ownership of mistakes. A typical example is where an increase in the scheme deficit is blamed on unforeseeably low interest rates rather than seeing this, with the benefit of introspection and hindsight, as an error in their decision making about hedging interest rates.
(2) Executives not pointing out mistakes. The lack of ownership of mistakes is reinforced in schemes that have a culture where the executive tries to massage the ego of the trustees, so they do not feel that they made a bad decision.
(3) Inertia and herding. Inertia is the most powerful force of all: there is a strong behavioural bias against taking action, particularly if the action is new, such as liability-driven investing. Herding, as in the rush to adopt LDI once it has become “conventional wisdom”, is another powerful behavioural bias that needs to be recognised and overcome.
(4) Blaming others is classic closed loop thinking consistent with cognitive dissonance – where our minds are in a state of denial about a mistake and so we reframe the evidence and look elsewhere for scapegoats.
Examples of poor practice include:
(1) Principal-agent gamesmanship between the trustee and sponsor – with both sides holding on to information to protect their power, e.g., trustees who don’t want the sponsor to see the investment strategy as they fear the sponsor will unduly try to influence it.
(2) Failing even to attempt to identify mistakes for a variety of reasons, including the skills of the boards, sponsor domination of boards, a particular board chair, poor advice, poor resourcing and low exposure to the workings of peer schemes.
(3) Boards spending too little time on strategic thinking, and instead lurching from one short term problem to another.
Ways to improve
Black Box Thinking suggests that information sharing is an effective means of addressing many of the problems facing schemes: opening up routes that trustees can acquire and share best practices themselves.
First and foremost, from the top down, there is a role for the regulator to play as a clearing house for post-mortems of failed schemes.
Then, from the bottom up, there are a number of different ways of achieving the much needed improvements:
(1) “New blood” board members and advisers appointed on a regular basis.
(2) Improving diversity – one benefit of member trustees is that they are not experts and they ask “naïve” questions in board meetings that are often actually challenging for advisers and board chairs to answer.
(3) Sympathetic advisers sharing best practice experience.
(4) Conferences and forums – these are normally organised by advisers and trustees can feel it difficult to speak openly at them: mistakes are rarely discussed. They could also be used to discuss key performance indicators that would help boards monitor their performance and make detailed comparisons of fund investment allocations.
(5) Away days with presentations from other larger schemes, but not from existing advisers.
(6) Administrative pooling arrangements and consolidation. Having pooled support can help the sharing of experience.
Examples of good practice include:
(1) Routinely asking each board member one thing we could do better.
(2) Using post-mortems with lessons learned where things go wrong.
(3) Using pre-mortems as mechanisms for avoiding future mistakes, such as considering a new investment idea, a move in LDI, or a forthcoming valuation or enhanced transfer value exercise.
|
