A 6-Step Process To Assess Risk And Benefit In ICH Q9 Decision Making

By Lori Richter, ValSource, LLC

ICH Q9 Quality Risk Management states, “The process of Risk Control includes decision making to reduce and/or accept risks. The purpose of risk control is to reduce the risk to an acceptable level. The amount of effort used for risk control should be proportional to the significance of the risk. Decision makers might use different processes, including benefit-cost analysis, for understanding the optimal level of risk control.” One of the questions risk control seeks to answer is: What is the appropriate balance among benefits, risks, and resources? What is the best way to determine the answer to this question? The following risk-benefit tool has been developed to answer such a question. A case study will explain how this tool can be leveraged to provide the valuable information described in ICH Q9 to decision makers.

Risk-benefit analysis provides valuable guidance to decision makers for determining the best options to control risk.

• If several options are being evaluated to control a risk, a risk-benefit analysis brings a standardized approach to evaluate all options equally. A decision can then be made by the team to determine the best option.
• The risk-benefit analysis provides the rationale and justification needed to present options to decision makers.

Risk Control Decisions

Prior to the risk control process, teams have completed a risk assessment. The team has identified the risks based on the risk question, scored the risks with current prevention and detection controls in consideration, and prioritized based on an overall risk level. At this point, the team must now determine which risks they will recommend reducing and which they may recommend accepting. This process can be very tedious and fraught with potential bias. There are likely projects in a queue waiting for approval in a project portfolio process or some that are already approved, that can sometimes show up during the risk control discussions. This is where bias can become crucial. If in-process projects are chosen as risk control options, are these the correct risk control options or are they convenient in that these projects have already been scoped and approved? Do we understand and have we documented the risks or benefits associated with accepting some risks? Using a risk-benefit analysis tool should drive to remove this bias and provide data to support decision making.

To develop a risk-benefit analysis tool, there are two aspects that must be considered. The risk criteria must consider the negative consequence if the risk control recommendation is selected and the “benefit,” or the positive impact to the business, if the risk control recommendation is chosen. Cost, resources, and time must also not be neglected within this process. How we gather this information and what we do to drive to a conclusion may vary based on the nature of the risk control options in question. We must also determine a final comprehensive analysis of the data to provide the information needed to draw a conclusion to the best fit risk control.

Figure 1 describes the process that should be followed when performing a risk-benefit analysis. Once the risk control options have been discussed, the team must move forward with a risk-benefit analysis to determine the best option. It should be noted that if there is a clear risk control decision and the rationale is best captured in a technical report, then the risk-benefit tool may not be a necessary approach. The tool is best utilized with a cross-functional team of subject matter experts (SMEs) who can assess both the risks and benefits of an option.

Figure 1: Risk-benefit analysis process for risk control decisions

Step 1 – Identify Risk Control Options

After a risk assessment exercise, the team discusses risk control options for risks that need additional control. During this time, the team should be thinking of the possible controls that would best manage the high-risk items on their risk assessment. Likewise, this may be an opportunity to accept risk. If so, the same tool can apply. The question can then be addressed, “If this risk is accepted, what are the potential issues that may have to be dealt with, and in addition, what are the benefits of accepting this risk?”

To better illustrate what is expected in each step of the process, a case study example will be used (the case study is fictitious and does not represent an actual pharmaceutical company.) In the example, let’s review what the team has determined:

• Risk description (example): The current lab at the Providence, RI site for Lifesaving Pharma is not sufficiently designed to perform release testing for the newest biologic drug being transferred to the site. There are inadequate storage space, personnel, and equipment to perform the required testing needed for product release. There is a potential for delayed release and drug shortages if this risk is not appropriately reduced.
• Risk control options (examples):
  • Option 1 – Construct a new on-site lab that is connected to the existing manufacturing facility, hire new staff, and purchase new equipment.
  • Option 2 – Contract with an off-site lab that is in the same city as the manufacturing facility.
  • Option 3 – Use the lab space in another manufacturing facility in Lifesaving Pharma’s network to perform the lot release testing for the Providence site. This site is in Raleigh, NC.

Step 2 – Develop “Risk” Criteria

The team should now look at the risk description that is under discussion as well as the options being considered and start a structured brainstorming. The main question being addressed at this point is “What am I concerned about?” Or, in other words, what is it about each of these options that makes me feel uncomfortable? Draw a fishbone diagram on a flip chart or whiteboard and provide your team members with sticky notes. On these sticky notes, they are expected to document what they’re concerned about regarding each of these options.

Figure 2: Example fishbone diagram to brainstorm “risk” criteria (not all-inclusive)

As expected, there will be several concerns, so be prepared to group them into common themes. This process provides the framework for developing risk categories. Risk categories are important to ensure all options are being evaluated equally against the same criteria.

Figure 3: Example of a risk category and risk ranking definitions (not all-inclusive)

Figure 3 provides an example of one category that may have resulted from the fishbone exercise. This activity will generate other categories as well. As the risk categories are defined, definitions for High, Medium, and Low need to be developed as well. Remember, the point of this exercise is comparing the risk control options against the same standardized criteria to gather information to best feed the risk-informed decision-making process.

Step 3 – Develop “Benefit” Criteria

The benefit criteria should evaluate what benefit the risk control option might bring to the process or system. In identifying the benefits, the team should perform another structured brainstorm. Instead of evaluating the question “What am I concerned about?” the team will be looking at “What am I excited about?” In other words, what is unique about this change that brings a benefit to the process or system it will be changing? Again, a fishbone diagram is a great option for this exercise to ensure ideas are looked at together as a team and in an open, less structured environment.

Figure 4: Example fishbone diagram to brainstorm “benefit” criteria (not all-inclusive)

Just as brainstorming risks generated many concerns, the benefits brainstorming can end up with many benefits. Take these benefits and develop common themes with a definition for each scoring criteria.

Figure 5: Example of a benefit category and ranking definition (not all-inclusive)

Figure 5 provides an example of one category that may have resulted from using an affinity approach for some of the ideas that were generated during the benefits brainstorming activity. Several other benefit categories would also develop. As the categories for benefits are defined, definitions for High, Medium, and Low need to be developed as well. Define the benefit in terms of positive impact with respect to the category.

A good rule of thumb to keep the risk-benefit analysis manageable is to not have more than 10 categories each.

Step 4 – Risk Rank the Risk Control Options

Once the categories and scoring are defined, the risk-benefit analysis can be performed. Gather a cross-functional team of SMEs and evaluate each individual risk control option. Capture the rationale for each of the scores provided to document the decision-making process. Once each category has been scored, take the average to get an overall score of risk and benefit for each option. Of course, it is best to understand how the overall score came to be. Make sure to investigate each category once the overall score has been obtained. Special emphasis should be spent on those categories that were high risk or low benefit. It is important to understand the rationale of these scores and if there is additional mitigation that may need to be considered if a risk control option is chosen with, for example, high-risk categories.

Figure 6: Example of risk ranking risk control options

Summary of the data:

• Option 1 is a high risk and may appear to not be an ideal choice. It’s important to do a deeper dive to understand why these risks are high within each category. Upon performing the benefit analysis, it appears this option has a high amount of benefit associated with it as well. Four out of five of the benefit categories were ranked high.

• Option 2 has some risk associated with it, including two high-risk categories. However, the risk level is lower than Option 1. The benefit of this option is low.
• Option 3 has very little risk associated with it and zero high-risk categories. There is some benefit, including three categories that were ranked as very beneficial with this option.

While there is some great information using this basic tool, it is still not a comprehensive picture of the options being evaluated. This tool does provide the risks and benefits with respect to the categories identified but is lacking the cost, resource impact, and duration of this project. While there is risk associated with these topics, when comparing options for risk control, it is preferred to look at these three items in relation to their cost and time impacts.

Step 5 ­– Identify the Cost, Resource, and Time for Each Risk Control Option

Now that the basic risk ranking is complete for each of the options, data must be collected to have a basic understanding of the cost, resource impact, and time impact. This will allow for a better understanding of the options to guide the risk-informed decision.

Figure 7: Cost, resource, time example

Summary of the data:

• Option 1 is a high-expense project with the highest amount of resource hours and the longest implementation time.
• Option 2 is 10 percent of the cost of Option 1, with a third of the resource hours and half the time.
• Option 3 is 1 percent of the cost of Option 1, with less than 10 percent of the resources and half the time.

With just this view, it is clear to see the expense, resource impact, and time required. However, without understanding the risk and benefit information, a decision could be made at this point without understanding which option will be most effective at managing risk and benefiting the business.

Step 6 ­– Perform Final Comprehensive Assessment

This last step of the process combines both the risk and benefit assessment along with the expense, resource impact, and time. With the comprehensive assessment, decision makers now have a holistic picture to inform the decision-making process.

Figure 8: Comprehensive final assessment example

In evaluating the final comprehensive assessment, decision makers can see the considerations to choose the best option.

Questions for decision makers:

• Is it more important to choose the option that has the least risk?
• Is it more important to choose the option that brings the most benefit?
• Is cost a major factor and is there a limit to the budget?
• Will resource hours be a showstopper if they are too high?
• How much time is too much time to complete this risk mitigation?
• What balance of the factors evaluated best suits the company’s current strategy?

Decision makers now have a comprehensive assessment to determine the best option. While this tool does not tell the decision maker which option to choose, it does provide a picture that brings all the information together. In addition to evaluating risk control options, this tool can also be used to evaluate risk control acceptance. It can provide information to answer the questions “Should this risk be accepted?” “If so, what are the risks and benefits to accepting such a risk?”

Identifying the best risk control options can be a challenging process without the ability to view all the information needed for the investment. By using a risk-benefit analysis tool, the unique opportunity arises to see the potential negative impact and the positive impact an option may pose. It also enables the comparison of these options side by side, with the same criteria being evaluated for each. While it may take time for a team to build this tool, it should be noted that once this is put together, it provides not just the information for decision makers, but also the knowledge management to document the rationale for the option chosen. Lastly, this tool addresses the question in ICH Q9, “What is the appropriate balance among benefits, risks, and resources?” All that is left is for the decision maker to leverage this data to make the decision and document the rationale in a location where it can be retrieved to use for future risk-benefit analysis.

References:

1. ICH.  ICH Q9: Quality Risk Management. Jun 2005

About The Author:

Lori Richter is a senior consultant at ValSource. She holds a microbiology degree and over 19 years of experience in the pharmaceutical industry. Her areas of expertise include quality risk management (QRM), quality systems, business continuity management, biotechnology manufacturing processes, and business process development. Prior to becoming a consultant, she held roles such as a site risk manager at a biologics facility, leading the development of an integrated risk management program. She was also a member of the global team responsible for the development and deployment of the QRM program across the Roche/Genentech Pharmaceutical Division. She has developed QRM training modules and delivered training to both health authorities and industry teams. She is an instructor for the PDA Quality Risk Management Certificate Program and was an author for PDA TR 54-5, “Quality Risk Management for the Design, Qualification, and Operation of Manufacturing Systems.”