Controls For Retrieval-Augmented Generation In GxP

A conversation between Ulrich Köllisch of UMK Consulting and Life Science Connect's Jon O'Connell

Generative artificial intelligence, by its probabilistic nature, is at odds with pharma's highly-regulated and anti-variability stance. Retrieval-augmented generation, or RAG, presents a meaningful bridge and, if harnessed, offers drug developers access to powerful generative AI tools that draw from limited and validated data sets.

By pairing large language models (LLMs) with controlled knowledge bases and source retrieval, RAG systems can help users find and synthesize information across validated data repositories. However, In GxP settings, that added capability still creates new validation, cybersecurity, and data governance questions that traditional computerized system frameworks were not designed to answer directly.

Ulrich Köllisch, an independent consultant and data integrity expert, says that existing frameworks such as GAMP 5, ICH Q9(R1), data integrity expectations, and emerging regulatory positions can be extended to cover the full RAG life cycle — from data ingestion, preprocessing, embedding, retrieval, and model versioning through to user interaction, audit trails, and operational monitoring.

The key difference, Köllisch says, is that RAG validation must account for both probabilistic model behavior and the quality, integrity, and permissions governing the underlying data pipeline. To that end, manufacturers need controlled model registries, risk-based performance verification, strong role-based access controls, cybersecurity oversight across every connected data source, and audit records that link responses back to original source material.

Köllisch will present on validating RAG systems in GxP environments during the ISPE 2026 Annual Meeting & Expo. He agreed to preview his talk and answer our questions on where human approval remains essential and where FDA and EU expectations appear to be converging. His conference co-presenter, Pradeep Ravindra from Körber Pharma, helped respond to our questions.

RAG architectures pull in retrieved data alongside a language model's own outputs, which adds a new layer of complexity to standard AI validation frameworks. What specifically about RAG makes it harder to validate compared to a stand-alone LLM or a traditional deterministic system?

Köllisch: Models that operate probabilistically do not necessarily produce the same output for the same input. This makes traditional testing against fully predefined expected results impossible. However, this challenge is not completely new to validation teams. GMP processes with inherent variability, such as visual inspection, are already validated using appropriate statistical methods. Similar principles can be applied to the verification of probabilistic AI systems.

For RAG systems, the entire end-to-end data process must be considered, including data selection and ingestion, preprocessing, embedding, retrieval, and generation. Appropriate data governance controls should therefore be applied across the complete data pipeline to ensure the required quality and integrity of the data throughout the process.

How did your team apply existing GAMP 5 principles and framework elements across the life cycle of your RAG system?

Köllisch: All principles outlined in GAMP 5 are fully applicable to AI/ML systems, such as, for example, the concept of a computerized system consisting of hardware, software (now with an AI/ML model), and processes and procedures. The life cycle model is fully applicable; however, several phases are even more important for RAGs, including the operational phase, monitoring, and change control, as we need to have our data pipelines under control. The data flow of all components – the LLM during its operation, the model data, and the data and metadata of the knowledge base of the RAG – implies rigorous governance that can very well be supported by the GAMP guidelines on data integrity.

Can you explain what a versioned and approved model registry looks like in practice? What triggers a new version, and what does the approval process for a model version involve compared to, for instance, approving a traditional software update?

Köllisch: In practice, a versioned and approved model registry can be considered a controlled repository and release mechanism for ML models, similar to controlled software releases but including additional model-specific information.

A new model version would typically be triggered by any change that could affect model behavior or performance. Such changes may be required due to declining performance, identified issues (e.g., within the data pipeline), or the availability of updated models with improved expected performance. Examples include changes to the underlying model, the data ingestion or embedding pipeline, or other relevant data preprocessing steps.

Does an agentic AI system in a GxP environment have permission to perform any tasks on its own? If so, is there a point where you draw the line requiring human approval?

Köllisch: In my experience, not yet. The EU GMP Annex 22 draft, published in July 2025, is quite explicit in prohibiting probabilistic models from independently making decisions for critical GMP activities. Notably, the term “critical GMP activities” is not further defined.

I expect this to become possible in the future. However, a key prerequisite will be demonstrating that the overall performance of the new process, for example, an agentic system without a human in the loop, is at least equivalent to, and preferably better than, the current best practice.

How does cybersecurity risk change once you're dealing with a RAG system pulling from multiple connected data sources, versus a traditional walled-off validated system?

Köllisch: In principle, all cybersecurity controls remain applicable, and they are evaluated based on data criticality (the impact of downstream decisions on product quality and patient safety) and data risk. All three layers must be secured: the data layer (verifying ingestion inputs), the infrastructure layer (firewalls, network segmentation), and the application layer (access controls, MFA).

Challenges introduced by the RAG architecture include an expanded attack surface and hosted risks. Connecting multiple databases increases the risk of unauthorized lateral movement. Strict identity management (least privilege, unique accounts) is required. Furthermore, if these services are hosted in the cloud, the manufacturer remains fully accountable and must validate the provider's security and data management policies

Can you walk us through the user experience in your information-as-a-service case study? How does the system maintain ALCOA++ traceability back to source data through that interaction?

Köllisch: From the user’s perspective, interaction is simple: a question is submitted through a controlled application interface, and the response includes citations to the source records used. The system serves as an information assistance tool, not an autonomous decision maker. Users remain responsible for reviewing source information before GMP-relevant decisions.

Behind the interface, a controlled RAG process retrieves relevant information. The AI agent operates within the authenticated user’s RBAC (role-based access control) permissions and may access only authorized tools and data sources, effectively acting as an extension of the user.

For traceability, an audit record captures user identity, timestamp, query, RBAC context, system configuration, retrieved source identifiers and versions, generated response, and relevant subsequent actions. Authoritative source content remains within the controlled knowledge base.

This supports ALCOA++ by ensuring interactions are attributable and contemporaneous, linking responses to original authoritative sources, enabling verification for accuracy, and maintaining complete and consistent records of inputs, retrieval context, configuration, outputs, and relevant user actions.

Where do you see the clearest points of alignment between evolving FDA guidance and EU GMP Annex 22? More specifically, where might companies operating in both regions run into conflicting expectations?

Köllisch: There is substantial alignment between the European agencies and the FDA. Both have repeatedly expressed openness to the use of new technologies when they are applied in a scientifically sound and risk-based manner to improve patient safety and product quality. Drug availability is also becoming an increasingly important consideration on both sides of the Atlantic.

At the latest EMA Interested Parties Workshop, the meaningful application of ICH Q9(R1) was repeatedly emphasized as a core principle. The respective regulatory expectations also show significant overlap, although some regional differences remain, such as the stronger emphasis on access to test data in the European context. Overall, considering the guidelines together, it is relatively straightforward to develop a strategy that aligns with the expectations of both regions.

For teams new to validating probabilistic systems, can you describe the important first steps to governing generative AI the way this framework demands?

Köllisch: The first step is to establish a common level of understanding among the relevant stakeholders, as risks can only be meaningfully assessed when the underlying technology is sufficiently understood. It is equally important to establish a shared risk appetite for the use of new technologies.

New technologies inherently involve a degree of uncertainty, which should be considered when determining the appropriate level of formality, in line with ICH Q9(R1). For RAG architectures specifically, teams must also ensure that the data feeding the knowledge base are available at the required level of quality. The technology cannot compensate for poor-quality source data, following the well-established principle of “garbage in, garbage out.”

About The Expert:

Ulrich Köllisch is founder and principal of UMK Consulting with expertise in life sciences compliance, data integrity, and audits. He has led data integrity initiatives and supported many organizations in the pharmaceutical and biotech sector executing data integrity campaigns. Ulrich has experience in consulting for audit preparation and conduction in the GMP and GCP areas, is co-heading a PDA special interest group, and is an active participant in other industry knowledge groups. He has a Ph.D. in physics in the area of medical imaging from the Technical University Munich.