Risk Management Planning: Be Prepared When Disaster Strikes

By David Shenberger, Belay Station Advisors

Few companies are ready when the black swan appears. Business continuity becomes important when the unexpected happens and the plan on the shelf can’t be executed. Continued operations through unforeseen events aren’t an accident; true continuity requires an integrated approach to cyber preparedness, the manufacturing infrastructure, supply chain robustness, operational readiness, process modeling, and a dedicated team. We can’t wait for the mythical superhero to arrive when the unexpected occurs, but instead, we must plan, train, and instill a resilient mindset and culture of excellence in the whole team. This involves assessing risks, engaging stakeholders, and developing multidimensional strategies that can adapt to various unforeseen events.

Risk Assessment And Management: The Foundation

Natural disasters — floods, tornados, hurricanes, earthquakes, and pandemics — are often the first risks considered in business continuity discussions. In today’s environment, cyber preparedness is where most organizations need to start. Many companies think they are too small or inconsequential to be attacked; they are wrong, and hundreds of companies come to regret their lack of attention to cybersecurity. Security starts with the least trained employee, and the person or system where penetration occurs has often escaped investment and training. Third-party systems, building management systems, phishing schemes, and even card readers and scanners have provided the access bad actors need to steal data and ransom systems. However, without formal risk assessments, key risks may be overrated, underrated, or not assessed at all.

For a biomanufacturing facility, data is everything. Data should drive decision-making around process performance, quality, and maintenance. Intellectual property (IP) is also key data. Therefore, protecting data is one of the highest priorities. Today’s threats are legion: ransomware, malware, industrial espionage, inadvertent and malicious employee actions, and data manipulation or loss.

In just the past 10 years, major pharmaceutical manufacturers have 1) been shut down by ransomware due to a third-party provider, 2) had a plant closed for months by a major hurricane that destroyed the electricity infrastructure, and 3) had a major distribution warehouse destroyed by a tornado. In each instance, considerable continuity discussions and planning occurred after the event, including major capital projects to prepare for “next time.” These responses are important but also show the lack of detailed risk management necessary to keep manufacturing plants operational through unforeseen events instead of recovering from them and then hardening for the future.

The first step in building for the unexpected is assessing risks and developing mitigation strategies. Many companies have learned the hard way that assessing these risks can’t just be left to chance.

Risk assessments and mitigation strategies need to be multidimensional and based on relative risk of occurrence. Including the right stakeholders at this stage is critical to determining the risks and gaining staff buy-in. If all stakeholders aren’t consulted and their input included, risks are not identified or are minimized, and mitigation strategies are incomplete or not suitable to maintain operations.

Commonly used risk assessment methodologies include fishbone, five whys, and failure mode and effects analysis (FMEA). There are many commercially available software tools that integrate the principles of these approaches to provide visualization, prioritization, and progress tracking. An integrated approach often works best, using the fishbone diagram to visualize the risks and drive stakeholder collaboration, then using the five whys to delve deeper into the root cause of the identified critical risks to determine the core risk. Finally, the FMEA brings a quantitative analysis, identifying existing controls and setting a risk score as a starting point to identify the most important risks to mitigate and a potential order.

By using an integrated approach, organizations can move beyond procrastination or the inclination to address everything at once, establishing a prioritized list based on risk numbers, fat tail risks, and budget constraints. This allows for a strategic and phased approach to risk mitigation over a three to five-year period.

The Planning Process

As an organization focuses on building continuity, it is important to think about the plan, the people, and the infrastructure as an integrated whole. If you picture a three-legged stool, as soon as one leg is missing or a different length than the others, the stool becomes unstable. The same is true with building continuity.

So, when budgeting for infrastructure investments, leadership must link plans to what the site staff can accomplish and the maturity level of both the people and the systems. Clear and timely communications are fundamental to stakeholder acceptance and engagement.

As the team develops the plan, there must be a balance between short-term and long-term thinking, investment costs, and the critical risks. This is where integrating stakeholders from the following groups will make all the difference:

• Operations
• Engineering
• Maintenance
• Quality assurance
• Quality control
• Tech transfer and supply chain
• Science & technology or R&D

The IT and infrastructure portion of the business continuity plan must address how process data and IP will be stored, protected, and recovered. Three high-level items to be considered are:

• sufficient backup power supply
• segregated networks to isolate the manufacturing network from the corporate network
• local data storage and buffering for communication gateways.

Integrating business continuity and digital transformation can be a win-win for the organization. Often, companies segment these into isolated budget items, but integrating them can result in significant savings by achieving both objectives at a lower cost and with better overall outcomes.

A focus on building continuity aligns well with achieving goals of digital transformation and even the disruption of broken and archaic operational processes. One important investment to prioritize is the development of a functional digital twin for both the manufacturing process and the IT infrastructure. Properly protecting, collecting, and storing data drives better decision-making and enhances the usefulness of your digital twin. Implemented properly, the digital twin will “learn” the process as operations adds relevant data to the model. This in turn will allow better simulation of interruptions, improving the continuity plan and response times — a virtuous circle.

Utilizing a digital twin enhances the output of the risk assessment by rapidly and inexpensively testing prospective controls, and the value of the digital twin grows over time as it uses data from operations to fine-tune simulations and improve the accuracy and precision of the projections. Using simulations of natural disasters, cyberattacks, and human error validates the projected risks, tests multiple controls, and produces risk mitigation strategies identifying robust approaches without expensive physical process experimentation and potential harm to physical components or to personnel.

Increasing digital maturity and digital disruption requires a data-first mindset: What data do we need and what do we do with it? How is the data used to make decisions? How do we maintain data integrity and validate the data appropriately for its intended use? This is too complex of a topic for this discussion, but thoughtful data management and increasing digital maturity are critical to the sustainability of operations.

The following are a few data-focused items that make the facility more resilient and sustainable:

• Unified namespace data structure to simply data usage and access
• Flexible systems that allow for implementation of machine learning and predictive process improvements
• Minimizing technical debt and application switching costs

People And Training: Empowering The Human Element

Investment and planning only get you so far; even with significant automation, your team are the ones who will deliver in an emergency. The planning process is the first step to successfully achieving this goal. Properly done, planning drives engagement, brings the viewpoint of the teams who operate day in and day out and know the idiosyncrasies of the equipment, and gives confidence to the team executing that there is a procedure for what they are about to do.

The process of planning increases knowledge of system operational characteristics and develops team problem-solving skills. The next step is training teams on processes and procedures as one team, leveraging the skills developed during the planning process. When a team trains together, they learn from each other, understand the unique skills each brings to a specific challenge, and develop a sixth sense of how each member reacts in high-pressure situations.

For training to properly support continuity, it must be multi-modal and persistent. Here is how that works. Multi-modal training leverages on-demand and classroom training with direct experiential learning in the manufacturing plant. If the business continuity plan calls for manual activities, then there must be an SOP that defines the steps necessary to successfully complete the manual operation. Leadership trains and certifies the operators on that SOP and drills these atypical operations at some regular interval, such as yearly or more frequently.

Building true business continuity starts with a holistic focus on future operations and having a clear picture of desired outcomes. The process of developing a business continuity plan and associated training shakes out the problems in your plan, drives buy-in with the team in the field, and builds confidence that a facility can manage whatever comes its way.

The path to sustainability depends on a clear, well thought out plan, focused training, and leadership buy-in. When all the parts come together, your facility will weather unexpected events and deliver critical products to patients no matter what is thrown at it.

About The Author:

David Shenberger is the principal at Belay Station Advisors and a partner at Shenberger Technology. He has served in numerous leadership, project management, sales, and technical roles over 30 years in the pharmaceutical industry. In leadership roles, he has been responsible for operations, marketing and sales, and strategic development. He has served on a major pharmaceutical client’s merger integration team and is a member of ISPE. He has extensive experience in pharmaceutical, advanced therapies, and biotech manufacturing, focusing on automation and facility start-up project management. He speaks frequently on computer system validation, project management, and quality systems. He has a B.S. in technical management, has an executive certificate in the business of life sciences from Indiana University, and is a certified PMP.