Date: Tue 27 July 2021
Title: Why is Risk ID All Over the Map?
We are discussing High Quality Risk Assessment, and set out last time the many benefits of writing a Context Paper as good preparation. But what now? Looking carefully at conventional advice, we discover why risk identification is typically so ineffectual and leaves people cold!
Let’s discuss the confusion entrained by the supposed risk ID methods set out in conventional literature:
· interviews and surveys, questionnaires
· audits, physical inspection
· networking with peers, industry groups
· judgmental – speculative, conjectural, intuitive
· history, failure analysis
· examination of personal experience or past agency experience
· incident, accident and injury investigation
· scenario analysis
· decision trees
· SWOT analysis
· flow charting, system design review
· work breakdown structure
We find that the items in this list are a mix of (often problematic) procedures, mere ways of thinking with no associated process, and examinations of risks already matured (compliance breaches, accidents and incidents that occurred in the past).
I surmise that the reason for such dismal survey results extended over years (see Episode 1) is that managers who had no experience in risk ID tried it without clear methods or definitions, and so quickly became disillusioned with the quality of the results.
Make sure you understand the pitfalls and deficiencies of random “methods”.
Following upon the very definition of risk given in the standards, a complete methodology is required. We present High Quality Risk Assessment. In our discussion so far you will find:
– High Quality Risk Assessment definition (Ep. 004)
– procedural grounding in proper planning (Ep. 005, 006)
– preparation of the Context Paper for the risk ID session (Ep. 007,008).
“Such a multiplicity of [risk ID] methods might entrain confusion about the object of the exercise.” (Robertson, p.42)
[edited for clarity]
Welcome to Episode #9: Why is Risk ID All Over the Map? So far on our podcast series, we’ve discussed core concepts of enterprise risk management, the way I conceptualize and recommend it, and part of that is to focus on what I call High Quality Risk Assessment — that is, as opposed to some informal or ad hoc method to identify risk.
So the first step in high quality risk assessment is to establish the context, which we discussed in detail in the last two episodes, and discovered why it was so important to set out a context paper to explain the scope and assumptions that will feed into the discussion of risk. Doing that prevents so many misconceptions and procedural hiccups, so that your meeting time is really used to best advantage. That now leads us to the point where we have to actually start to identify risk.
[01:45] You know, in the first few episodes we discussed some survey results; that is, how are folks are making out with the idea of trying to implement ERM in their organizations — how is their risk ID process going?; do they feel at all confident in their ability to identify risk? and so on. A lot of the survey results have been quite dismal, and that’s an extended trend over many years. Now you would think that developing risk information would be really beneficial for the organization, and intuitively, that is correct. The trouble enters in when there are so many different conceptualizations, definitions and methods that are sort of thrown in — and all characterized as legitimate risk ID methods.
What we’re going to do in today’s episode is deconstruct all of that. Not simply to be negative or to be grumpy, but to lead us to some clarity as to what a real method really is, and how it follows closely on a carefully considered definition of risk.
Now I’m going to be reading from a piece of conventional advice. (If you want to see the reference, you can go to my book and find out what it is, but for the purposes of today’s discussion, I could’ve used virtually any piece of conventional advice from the standards, government literature, and some of the consultants’ literature…) The premise is that risk as identified [defined] in at least three of the standards, they sort of overlap on this point, has to do with the uncertainty that is associated with intended goals and objectives of the organization. Okay, so keeping that definition in mind, let’s go through this list of supposed risk ID methods.
[03:36] #1. interviews and surveys. All right, well, I grant you that interviews and surveys are indeed legitimate methods to actually gather in ideas about what risks are; in other words, to collect risk information. The great advantage of interviews and surveys is that you don’t have to convene any meetings. You can go and visit the people, or electronically visit them, so to speak, by administering a survey.
Now the difficulty here is that interviews and surveys have pitfalls for those who are not trained in research methods. Why do I say that? Am I just being too picky? No, the fact is that respondents, in individual interviews and surveys, will have widely varying ideas as to the definition of terms. For example, the definition of what a risk is; then again what the scope of the discussion is; what the context is (as we covered in the last two episodes).
People who are not trained in research methods will often sort of go out and willy nilly gather in all kinds of risk information — and struggle to aggregate that information, I might add — without having a clear idea about the validity of data, about bias that can enter into the collection of data. I’ve seen this happen, where people have collected risk information from senior executive using the interview method (because it was easier to how to get to those people using that method) — except that they didn’t take into account the fact that respondents, as I said, were going to have a varying ideas as to what the definition of a risk was. They responded to the notion of risk in different ways, sometimes in an alarmist fashion; sometimes not having the same concept as to what the goals and objectives of the organization actually were.
Without checking all of that at then you simply have no reliability. There’s no validity of data, because there are too many elements of bias that are entering in and ruining the quality of your data.
[05:51] Now here I can almost hear people objecting: “No, this is a valid thing. What you’re objecting to, Edward, is just on the margin. Generally speaking, people understand what their goals and objectives are, and when we talk to senior executive about risk, we get answers that makes sense.”
Well, here’s my answer to all that. I’m betting that the results of your risk assessment could be much better. It’s much more likely, especially in the first go-round when you try to do this, that the results of your risk ID efforts will be to simply have a vague rehash of all of the issues that are already known and understood, and bandied about, but without much concrete conclusion being drawn from them. I’ve seen this time and again.
So you can continue on this method, if you simply want a general, vague rehash to be the result for your risk ID efforts. But what I’m suggesting is a method that is much more rigorous, that really is incisive and insightful — and I will get to that. For the purposes of today’s discussion, let’s just be clear that interviews and surveys have problems with regard to validity of data, especially when you don’t bother to set out the scope and assumptions with the participants beforehand.
[07:19] #2. audits or physical inspections. Okay, so that’s supposed to be a way to identify risk. Well, no, if you conduct an audit on what already exists, it’s a way to identify compliance issues. Therefore, the risk has already matured; it’s already happened. It does not consist of uncertainty, it consists of a breach in compliance. That’s what an audit will tend to unveil.
Same thing with physical inspection, whether it’s in an engineering environment, or looking for fire hazard… Are you simply going to conduct your risk assessment by virtue of compliance? If so, I’m not saying that’s entirely invalid, but I recommend you take a look at “Busting Myths” — that was a previous episode. Myth #11 was the idea that compliance constituted a truly effective enterprise risk management regime.
[08:20] #3. brainstorming. No, brainstorming is a method to generate all kinds of ideas without limit, without constraint, without definitions, without any assumptions. That’s the whole point of brainstorming — to be creative in a wild and woolly fashion. Therefore, any risk that you identify with such a method is going to be mixed in with so many Illegitimate, improbable and unworkable ideas, that it’s simply not a practical method.
Well going down my list here… The next one is
[08:56] #4. questionnaire. I characterize questionnaire just the same as I do survey or interview: in other words, there’s going to be a huge risk for data validity problems.
Notice, too, that in a questionnaire or in a survey or individual interview, there’s the problem of isolation. You’ve got people responding to these questions in isolation without the benefit of a group process. So I will get into that in more detail later on…
#5. networking with peers industry groups or professional associations. Okay, well “networking” is too vague. First of all it’s not really a defined process, and there’s no differentiation between general discussion and risk identification that is implied in the term networking.
#6. judgmental, speculative, conjectural, intuitive. All right, well, is that a method or is that simply mental process to identify risk? It’s a mental process — but it will have to have some procedural context, and it doesn’t tell me what that is. It just tells me that the mental process of intuition is is valid.
I actually agree with that — but within certain constraints, a certain procedural context, which is not explained here in this list [that I’m quoting from].
[10:31] #6. history and failure analysis. Well, there again, we’re looking at what happened in the past. So if we examine the history of risks that matured, of failures that occurred, let’s say, in engineering or any other domain, then we’ve seen where risks came to pass and actually resulted in real problems, in real crises or accidents. That is not following the definition. The definition has to do with uncertainty. Of course, you can take that information and project it, and use it as inspiration. But that’s not what it explains in this list. So, following on that point:
#7. examination of personal experience or past agency experience — Of course, when people identify risk, invariably, they’re going to be drawing upon their professional memories. I think that almost goes without saying.
#8. incident accident and injury investigation. Well there again we’ve got the risk that has already matured, and that’s going to inform our risk identification for future actions.
#9. scenario analysis. That is useful for developing (as the term implies) an entire scenario, a whole picture of a certain situation where all kinds of risk will be incurred in one situation. What comes to mind there is business continuity and emergency planning, so of course that’s a valid method. But it does not really apply to identify risk in a given administrative context [with] a set of goals and objectives. A scenario applies to a specific incident of crisis, not to a general plan.
[12:23] #10. decision trees. I’ve seen examples in textbooks of influence diagrams, where all kinds of various influences feed into a certain decision, with probabilities attached to each of the elements — the various branches of the tree.
Well, I find that very impractical. A typical risk ID session is going to generate, in my experience, between thirty and fifty risks. If you have to trace through the decision tree, or the influences that lead to each of those risks, you’ll never get done; you won’t even get started. And the other thing that enters into this part of the discussion is that we don’t have the actuarial data. We don’t have the statistical analysis to assign probabilities — statistical probabilities — to each element of a given decision. The only context that I can think of where this would make sense is where you already have a body of highly developed and highly specific statistical information to guide a certain decisions… such as an actuarial setup, where you’re got certain insurance coverages, and all of the historical statistical data that you need to support your policy.
But in the dozens or even hundreds of risk ID sessions that I’ve facilitated, I have not seen people bring data like that to bear upon all of the administrative decisions that they need to make, in all their diverse contexts.
[13:57] #11. strengths weaknesses opportunities threats. This is the old familiar SWOT analysis. There’s seemingly a relationship between risk ID and this technique of SWOT, because “threats” are sort of synonymous with “risks”, are they not?
Now here, SWOT is presented as a risk identification technique, and yet to my mind, it belongs under the general rubric of planning techniques. In other words, you’ve got environmental scan: you’re going to be analyzing the general industry trends and conditions and so on, as preparation for making plans. And that’s where strengths, weaknesses, opportunities and threats enter in.
Risk identification requires something much more specific, and an analysis that will apply to each risk that is identified and formulated. And that is not facilitated by the SWOT technique, which really just bears some similarity to the idea of risk, because of the word “threat”. But it’s not, strictly speaking, risk ID.
[15:12] #12. flow charting. Yes, I described in my last podcast episode how creating a flow chart for an administrative or technical process is not a bad way to set up a context for risk identification.
There again, that doesn’t identify the risk. It simply is an aid or facilitation tool. Similarly — and here’s the last item in the list —
#13. work breakdown structure analysis. In other words, this is the project management work breakdown structure. Of course, that gives you a lovely structure to trace through, and try to identify risk. So again, that’s a facilitation tool: it’s a means to access the content, but it doesn’t tell you exactly how to conceptualize and formulate a risk. It’s not telling you how to identify risk.
So here, in summary, what we’ve got in this long list a mix of ways of thinking; ways of accessing people; ways of looking at risks that have already matured and caused some sort of accident or failure; various procedures, facilitation tools, and so on. These various so-called risk ID methods are not even commensurable, they’re not qualitatively the same thing. You can’t compare one to the other because one is talking about process, one is talking about conceptualization, the third one is talking about examining physical history, etc.
[16:41] One thing that’s remarkable about this list is that there’s nothing — except for the idea of perhaps speculative or conjectural mind process — there’s nothing that really allows people in a procedural way to go into the future, to try to detect uncertainty, new uncertainty, fresh ideas of things that people had not encountered before, by virtue of what is contained in their plan.
So if you try to follow this kind of advice, especially people who are non-experts, people who are busy doing other things and they want to get on with some kind of risk management, but they’re not quite sure how to proceed (and they don’t have time to do all of this analysis that I’m doing right here and now) — they’re going to be misled. They’re going to be led down the garden path of trying to cobble together some sort of a risk ID methodology, but very quickly discover that the whole thing just falls apart.
The risk information that they have generated is just all over the map, and so it’s really no surprise that the information that they developed did not really inspire them to take action or to gain any new insight into their program.
[17:49] So the question that naturally follows on all this is: “What is high quality risk assessment?” and the answer is that it has to comprise all of the things that we were talking about: It has to have a certain process; conceptual grounding; and procedures that precede (in terms of planning and context preparation)…
Then we can explain with much more profit, with much more clarity, what the risk ID session actually looks like — how to conduct it, and the kind of results that we can expect to get.
So in our next episode, we’ll start talking about how to conduct the risk identification session, following the method of high quality risk assessment.