Is Your AI Model Trustworthy And Credible In GMP Processes?
A conversation between Gillian Buckley, Ph.D., of PhRMA and Life Science Connect's Jon O'Connell
Artificial intelligence's spread into daily life has been shaped by the "move fast and break stuff" philosophy of Silicon Valley. The mentality is anathema in GMP environments, where each component must be validated and trusted.
For manufacturers, regulators, and technology developers, the urgent question is whether an AI model can or should be trusted when product quality, process control, and patient safety hang in the balance. Drugmakers that want to connect artificial intelligence to bioprocessing production lines must prove that it fits its intended use, remains under control, and continues to perform as expected over time. Two concepts, trustworthiness and credibility, neatly contain these critical challenges.
Gillian Buckley, Ph.D., a senior director at PhRMA, will speak about establishing both in AI models during the inaugural 2026 ISPE AI in Life Sciences Summit — Powered By GAMP at the end of June.
Ahead of the summit, Buckley gave us a preview and helped explain the difference between credibility and trustworthiness, the importance of context of use, and how to recognize and manage model drift, among other key considerations.
Can you describe the role of credibility and trustworthiness in the context of GMP AI manufacturing tools?
Buckley: Credibility and trustworthiness are related but distinct. Credibility concerns the quality of the data used to train a model, whether it is fit for purpose and supports transparency into how the model works and evolves. Trustworthiness is more of a feature when a model operates with greater autonomy, making or influencing decisions without direct human sign-off. In pharmaceutical manufacturing today, meaningful human oversight remains the norm. It’s also important to note that machine learning models generally start from a strong baseline of trustworthiness because their outputs are reproducible and generalizable. Together, these attributes influence the overall assessment of the model risk.
Can you describe what a monitoring program looks like for AI tools in GMP environments? How do manufacturers know when a model that was trustworthy at deployment is starting to drift?
Buckley: Machine learning models generally become stronger over time because experience with the tool encourages greater confidence in its use. This relates to the answer above — if a company has good insight into how the model is working, when the results are reproducible and explainable, the understanding of the model should improve over time.
It is also important to note that manufacturers typically have standard operating procedures (SOPs) that describe the verification, validation, applicability, and procedures that they monitor for model qualification. These SOPs are typically conveyed in the pharmaceutical quality system (PQS). The context of use and the explainability of the model inform the need for the level of human oversight. Manufacturers also typically implement risk assessment procedures that assess the planned use of the model, the training data sets and their robustness, the limits for drift in the model, and the procedure for retraining the model. Taken together these checks provide a signal to a human when there is a potential need to address any identified model drift.
Can you describe an example that helped build a case for credibility?
Buckley: An example to spotlight involves a model that combines traditional scientific equations with machine learning to monitor what's happening inside a bioreactor. The model can spot early signs that a batch may be heading in the wrong direction and suggest corrective actions, such as adjusting the feed or physical conditions, to get it back on track.
What makes this a good credibility story is twofold. First, you can evaluate how well the model performs against documented data that confirms its predictions. Second, over many batches, that track record compounds, building confidence. It's also worth noting that even small changes to how the model is being used — its context of use — can affect model credibility. Credibility is assessed dynamically, in relation to the specific job the model is doing.
How close do these kinds of examples get us to fully trusting the models, or will we always deploy them with skepticism baked in?
Buckley: We are working toward a risk-based framework where the level of oversight matches the stakes of the decision and where the explainability of a model informs how much confidence you can place in its outputs. The FDA has already laid out a framework for using AI in drug manufacturing that takes this approach, designed to build a shared understanding between regulators and industry of how sponsors may use AI to support regulatory decision-making. As the science matures and both sides gain more experience with these tools, we'd expect that confidence to grow and AI to become more seamlessly integrated into manufacturing. We're not there yet, but the trajectory is positive — and conversations like this one are an important part of building that shared foundation.
AI models can improve with more data over time, but change control frameworks were built around static, deterministic software. How are manufacturers navigating that tension?
Buckley: It's not as new of a problem as it might seem. AI and machine learning models share similarities with other computational tools manufacturers have been managing for years, and the same best practices apply, such as version control, structured development and validation, and rigorous data governance, including thorough testing of training data and documentation for future reference.
Models are built and tested in secure, qualified environments with the computational infrastructure needed to support them. That foundation is critical to using AI in a GMP setting. Such controls enable a risk-based approach to change control. Not every update requires the same level of documentation and review. That risk-based thinking is what allows manufacturers to keep pace with evolving models while maintaining the controls a GMP environment demands.
What characteristics of real-time process data have you seen manufacturers use to establish the credibility of an AI tool?
Buckley: Real-time data is one of the most powerful tools manufacturers have for building and maintaining confidence in an AI model. It can catch failures, flag deviations early, and confirm the model is still performing as expected. Real-time data is generated in ongoing model maintenance, with formal procedures— SOPs — governing how such maintenance is performed.
In the digital twin bioreactor example we're presenting, the model can detect deviations in titers early in the manufacturing process and suggest steps to correct any identified potential deviations. If it detects a batch falling behind, it suggests specific interventions, like adjusting nutrient concentrations, for example. And the manufacturer can compare the projected trajectory without action against what actually happens when the model's recommendation is followed. This process is repeated over many batches and with each repetition the manufacturer’s confidence in the real-time data increases.
Finally, what do PhRMA members most frequently report as being the greatest barrier to trusting AI in GMP manufacturing?
Buckley: The biggest barrier our members point to is the challenge of using innovative tools in a highly regulated environment. Regulators are understandably cautious about issuing guidance on AI before they've seen enough real-world examples of how it's used, and industry is reluctant to move forward without clearer regulatory expectations. Both sides are waiting on each other.
The way to break that cycle is through exactly the kind of open dialogue that forums like the ISPE AI in Life Sciences Summit make possible. Open discussion encourages a shared view of the landscape, which in turn builds a common understanding. We strongly encourage continued engagement with FDA and other regulators because advancing the use of AI in manufacturing ultimately benefits public health.
About The Expert:
Gillian Buckley leads the global quality and manufacturing portfolio at PhRMA. Before joining PhRMA, she was a study director at the National Academies of Sciences, Engineering, and Medicine where she led research on a range of topics including health financing, antimicrobial resistance, infectious disease, and global health. She has a Master of Public Health degree and a Ph.D. in human nutrition, both from Johns Hopkins University.