How To Generate The Right Input Data For An Effective Quality Management Review System

A successful quality management system (QMS) must objectively and accurately measure, review, and continuously improve quality and current good manufacturing practice (CGMP) compliance. This process often takes place within the context of a quality management review (QMR) system, with related meetings and formalized content and outputs. However, the review and improve elements of this equation are compromised if the input data are inaccurate, incomplete, or misleading.

This article is the second in a two-part series about demonstrating CGMP compliance during inspections by establishing a successful quality culture and related systems. Part 1 discussed the importance of “top-down” quality management and “bottom-up” communication of compliance risks and concerns, culminating in an effective QMR system. Here in Part 2, we turn our attention to the input data for an effective QMR. In addition to the risk register(s) and top risks and mitigation actions described in Part 1, these include the review of:

1. Quality and CGMP compliance metrics
2. Data for recurring deviations, laboratory investigations, and complaints
3. Data for overdue compliance activities — CAPAs, especially from inspections and audits, product quality review reporting, completion of stability testing, critical preventive maintenance and calibration, internal audits and self-inspections, closure of change control, and scheduled review of QMS documentation.
4. Data for supplier performance
5. Heads-up on emerging compliance trends, standards, and regulations

Note:  Some QMRs become distracted by focusing on efficiency data, such as the number of days required to release a batch.  This is not a measure of compliance and should not take over the valuable time required during a QMR for the review and discussion of compliance. Also, it is not recommended to describe change control as “planned deviations.” Deviations are unexpected events and therefore not planned.

Quality And CGMP Compliance Metrics

Objective, accurate metrics can be valuable inputs to QMR meetings to facilitate the required discussions, to determine the actions required to prevent non-compliance and the recurrence of non-compliance, and to continuously improve the QMS.  However, if the output data are derived from a distorted or misleading formula, the value of the metric is greatly reduced or can be damaging.  A common approach is to describe and use a “comparison metric,” i.e., a number representing the non-compliance over a period of time is compared to another number representing the source of that number over the same period of time. For example:

Product Quality Complaint Rate = (number of product quality complaints received for the product ÷ total number of dosage units distributed in the current reporting time frame)

In this example the metric is distorted and might be misleading, as the two components of the calculation are derived from different periods of production, which are unlikely to be directly comparable.  The number of complaints represents dosage units on the market and will therefore relate to a much earlier period of production, compared to the number of dosage units distributed in the current reporting time frame. To correct and align this metric, the definition would need to be adjusted, e.g.:

Product Quality Complaint Rate = (number of product quality complaints received for the product in the current reporting time frame ÷ the number of dosage units distributed during the time frame represented by the distribution date range of the complaint batches/lots).

An alternative here would be to simply report the number of complaints received or, preferably, the number of recurring complaints received for each time period.  In this way, a trend indicating general compliance or non-compliance is visible, although the source of this is in the past.  Consequently, CAPAs completed to correct an adverse trend will not actually influence the trend until the corrected units have reached the source of the complaints.  Complaint metrics are therefore not a leading indicator of current compliance.

Metrics are also often calculated to describe manufacturing compliance by comparing the amount of non-compliance to the amount of manufacturing over the same period of time. Again, care is needed to avoid distortion and misinformation, e.g.:

Deviation Rate = number of deviations ÷ number of batches manufactured

However, what does “manufactured” relate to — batches started, batches completed, batches tested, or batches released? Each of the “manufactured” numbers will be different; therefore, to ensure this metric is not distorted, e.g.:

Deviation Rate = number of deviations ÷ number of batches represented by the date range of the deviations initiated during the reporting period.

In this way, a real-time deviation metric is reported each time, which might include the same batch over multiple reporting periods if this batch experiences more than one deviation during manufacturing. Nevertheless, this provides a more accurate indication of current compliance.

An adjustment of another common metric aimed at indicating manufacturing compliance is, e.g.:

Right the First Time Manufacturing Compliance Rate = number of batches with zero manufacturing documentation errors and zero deviations ÷ number of batches started (and finished) during the reporting period.

Additionally:

Right the First Time Manufacturing Rate = number of first time (not reprocessed or reworked) batches passing all release testing ÷ number of first time batches which completed all release testing during the reporting period.

The above examples show the importance of aligning both parts of the calculation to avoid the generation and use of misleading data.

Recurrence Is The Best Metric

If the investigation of a non-compliance is effective in determining the true root cause and the action(s) completed (the CAPAs) are effective in eliminating the root cause, the non-compliance and related non-compliances should not recur. To confirm that the true root cause has been identified this should be challenged using, for example, the 5 why? approach, where the question why? is asked five times to challenge each answer.

Determining the effectiveness of CAPAs is typically a requirement for product quality review reporting; therefore, monitoring for recurrence should be in place. Metrics related to recurring deviations, complaints, and laboratory investigations should be collected and used to determine if further action is required.

What Is The Target For Metrics?

Targets are usually assigned as a means to drive improvements. The ultimate target is of course 100 percent compliance; however, initially, the metrics data should be collected to determine the current/baseline performance in each case. Thereafter, it’s reasonable to assign a target of reducing the gap (between the baseline and 100 percent) by 50 percent of the gap over a realistic period of time. Eventually, the ceiling performance will be reached, where closing the gap by a further 50 percent requires a significant change. At this point the risks associated with the remaining gap might be acceptable as described by “risk acceptance” in ICH Q9. Senior management should review and decide whether to accept residual risk or not.

Overdue Must Be Avoided

If planned compliance activities with assigned completion dates become overdue, the initial conclusion is that management oversight is not adequate. However, foreseen and justified delay can be acceptable where the risk has been determined and recorded to be small. Examples would be a delay to scheduled non-critical maintenance or calibration activities, where evidence exists that the delay will not compromise the performance or accuracy. An exception is the completion of actions promised in a response to a regulatory authority. These actions must be tracked very closely, in line with the written response accepted by the authority. Where any overdue activity is accepted and approved, it is advisable that the number of allowed delays (extensions to the due date) is restricted and that the approval of delays requires escalation to increasing levels of management.

Note: To improve efficiency, some firms prioritize release testing over stability testing.  This presents a risk of non-compliance, as stability testing represents the manufacturing of products that are already on the market, whereas release testing represents the manufacturing of products that are not yet on the market.  Timelines must be in place for the completion of each stage of stability testing, from the removal of samples from storage to the reporting and typicality trending of the sample data.  Compliance with the procedural timelines in place should be tracked in real time and reviewed at the QMR.

Don’t Tolerate Poor Supplier Performance

If supplied materials do not meet the required and agreed standards, they should not be used. Such materials should be returned to the supplier as a complaint for investigation and remediation as required before further supply. If the lead time for resupply is long, alternative sources of supply from stock should be approved for use when required. If an alternative source of supply is not available, the failing material should be formally evaluated for possible use under controlled conditions; however, this is not a favorable situation, as the supplier is being rewarded for failure. If the material supplied is categorized as critical, a “for cause” audit of the supplier should be considered before the material is used or resupplied.

Maintaining The “Current” In CGMP

A GMP quality system is an evolving structure that requires constant adjustment to remain current with regulator and industry expectations, best practices, and topical trends. It is important, therefore, that the quality system and organization supports the gathering of external data and information to support continuous assessment and adjustment internally aimed at keeping the system current and aligned with CGMP.

Bringing It All Together

Holistically, the existence of a strong top-down quality culture, driving the open, transparent communication of risks, together with the existence of effective quality risk management and quality management review (where accurate and objective metrics and inputs are used to determine and facilitate action completion), will ensure that a firm can achieve and demonstrate a high level of overall CGMP compliance and compliance management capability.

When I perform audits, I state at the opening that my greatest interest is not the level of compliance I experience during the audit, but the level of compliance I’m not able to experience before and after the audit.  In other words, the “performance” experienced during a typical inspection or audit may not accurately reflect the actual level of compliance during the routine manufacturing, testing, and supply activities.  A top-down approach provides an insight into what this actual level of compliance might be.

About The Author:

Grant Mordue is the director of Pro-Active GMP Consulting Ltd., a UK-based consultancy founded in April 2020 to help companies to successfully implement a proactive level of quality management and CGMP compliance. Mordue has more than 30 years of management experience across the CGMP compliance of manufacturing and supply operations at local (national) and global levels, including the management of regulatory inspections. He has a BSc (Hons) degree in applied chemistry and is a Chartered Chemist and Member of the Royal Society of Chemistry in the UK. You can connect with him on LinkedIn.