The Shortcuts Trap – Key Indicators Under Fire

How common key indicator errors can increase risk and reduce resiliency.

This is the second in a series of four blogs about the ways in which common shortcuts can undermine overall operational risk management success within organizations. You can access the first blog here Risk and Control Self Assessments.

Everyone loves metrics, and particularly key indicators such as key performance indicators (KPIs), key risk indicators (KRIs), and key control indicators (KCIs). In a perfect world, you’ll be using risk management software and each data point should provide the organization with importance intelligence. However, most operational risk teams wind up burdened with hundreds of KPIs, KRIs and KCIs, and then find they cannot see the forest for the trees. At some point, most operational risk teams will find themselves needing to sit down and shake up their stock of key indicators. And that’s where the temptation to use shortcuts can arise. Common shortcut mistakes include:

• Deeming the
  KPIs to be KRIs – Often the business will have a list of KPIs that they
  track for operational and financial purposes. There can be a temptation to just
  adopt all of these KPIs as KRIs, and call it a day. However, KRIs are just what
  they say they are – and so they need to be identified with a specific risk. For
  example, a KPI that everyone looks at is profitability, and this is an
  excellent KPI. However, it tells the operational risk team almost nothing specific about
  the way the firm is managing any of its operational risks or controls. Operational risk
  teams should challenge the use of KPIs as KRIs by asking which specific risk or
  control a KPI relates to.
• Regarding
  KCIs as KRIs – Once again, it’s important to distinguish between what is a
  KRI and what is a KCI, and to treat them differently. KCIs are vitally
  important for an operational risk program – they measure the strength of the
  control environment. Treating them as KRIs leads to failing to consider the
  control environment on its own. KCIs should be paired with a relevant KRI, for
  maximum operational risk framework robustness. As well, they should be tested regularly
  through the control testing that the operational risk team performs.
• Using the
  data to set the thresholds – Although using the data might seem to be a
  straight-forward way to set the thresholds of key indicators, it is deeply
  problematic. For example, some operational risk teams will look at the data for a key
  indicator and say to themselves, “we’re generally where we want to be, but
  sometimes we’re not so that must be amber, and remember that really big one two
  years ago, that must be red.” However, this thinking doesn’t really capture
  what is happening on the ground within the business. In fact, the process the
  key indicator is capturing may be experiencing problems, so what looks like
  “green” from the data might actually be amber or red for the team. It’s
  important to have a conversation with the business for each KRI and KCI being
  used, to set the thresholds according to what is right for the business.










• Believing
  all KRIs are early warning signals – Certainly some KRIs are early warning
  signals. In particular, those KRIs which relate to the likelihood of a risk
  happening can fall into this category. However, those KRIs which are indicators
  of the size of the impact of a risk event are not early warning indicators –
  they simply indicate the possible magnitude of the impact. As a result, impact
  KRIs are more of a lagging indicator. An example of a good operationa risk early
  warning signal is a staff turnover metric. Staff leaving a settlements
  processing team leads to a loss of operational history as well as the need to
  train new staff. Existing staff can become demoralized and not train new team
  members properly – and so risk can rise. Therefore, indicators of likelihood
  are good early warning signals because the right ones can provide an insight
  that a risk is more likely to happen. However, in this example, the number of
  settlement fails would be an impact indicator – it is an indicator of the
  impact of losing key staff – and not providing an early warning of anything.


• Tossing
  out indicators that aren’t predictive – Most financial services
  organizations have far too many key indicators – they tend to accumulate over
  time. However, because there is a commonly-held belief that all key indicators
  are early warning signals, some operationa risk teams have simply put other types of
  key indicators in the bin. This is a big mistake. For example, KRIs that
  suggest the potential impact of a risk event, as well as KCIs are important for
  understanding operational resilience. Instead, operationa risk teams seeking to reduce
  the number of key indicators should look towards industry best practice and indicator
  effectiveness as benchmarks.

In short, it’s important to take a more thoughtful approach
to restructuring an ecosystem of key indicators. Taking shortcuts can reduce
the ability of certain KRIs to be predictive, and damage the firms’
understanding of its operational resiliency, through other KRIs and KCIs, as
well.