Building Trust In AI For Computerized System Validation

A conversation between Frank Henrichmann at Q-Finity and Life Science Connect's Jon O'Connell

Misguided confidence is one of artificial intelligence's greatest traps. Anybody who has watched ChatGPT return an incorrect response with utter confidence can imagine the danger it poses when allowed to play within validated systems, let alone assist in the validation to begin with.

With a measured level of trust and the correct guiderails, however, AI can slice out some of the repetitive work of computer system validation (CSV) and create time savings for pharma process validation professionals.

Existing validation frameworks like GAMP 5 (Second Edition) support its use, and, at the upcoming 2026 ISPE Annual Meeting & Expo, Frank Henrichmann, a senior executive consultant at Q-Finity, will explore the unique and evolving considerations of using AI as part of the validation repertoire. He agreed to preview his talk for us and answer questions.

Using AI tools for CSV might sound like the professor's letting the students grade each other's papers. What are some of the areas where we can safely rely on AI to help while safeguarding integrity?

Henrichmann: The picture is almost right, with an important caveat.

We have not specified what kind of paper we are looking at. Grading an essay is a judgment call with no answer key. Much of what we ask AI to do in CSV is not that. It is closer to checking a calculation against the answer key. Does this test trace back to this approved requirement? Is this reference missing? Is the terminology consistent with the specification? Those are derivations from a source, and they can be checked against that same source.

So, I tend to sort AI-supported work by what I call certainty, meaning the degree to which an output can be objectively derived from source evidence and independently reproduced, not the degree to which the system sounds confident. At one end, you have outputs that are essentially deterministic:

• completeness checks,
• reconciliation,
• identification of missing references,
• traceability statistics,
• consistency checks against defined rules and terminology, and
• reformatting of approved content into a structured record.

These can be reproduced and independently verified by a person, or even by the system itself. Here, I am quite relaxed.

At the other end, you have outputs that are conclusions:

• Is this test evidence adequate for this requirement?
• Is this deviation significant?
• Has this requirement really been implemented?

That is interpretation and, here, the human effort must increase, not decrease.

And yes, there is a human in the loop, but I want to be careful with that phrase, because it has become a comfort word in our industry. A human signature is not automatically a detection control. If the reviewer is looking at fluent, well-structured, mostly correct output in high volume, the psychology is against them.

Three effects are well documented here:

• Automation bias is our tendency to prefer what a machine proposes over our own judgment, even when contradicting information is available.
• Cognitive ease is the effect whereby information that is easy to read and looks well-ordered simply feels truer.
• Review fatigue is what happens when you assess large volumes of similar, mostly correct output.

Even the most qualified reviewer is vulnerable to vigilance erosion under these conditions.

That is why human oversight should never be the only layer. Independent technical controls or guardrails must remain active here, meaning technical and procedural limits built into the tool itself that define what it may take as input, what it may output, and when it must escalate to a human.

Automated test case generation is one of the more concrete promises here. How does an AI system generate a test case for a GxP computerized system, and what data feeds it?

Henrichmann: Mechanically, it is less mysterious than people expect. The system takes an approved requirement, ideally a precise, testable user requirement that describes intended use inside a business process, together with the functional and design specification, the risk assessment for that requirement or function, the configuration of the system, and the company's own templates and terminology. From that, it derives what needs to be verified, at which depth, and in which format, and it drafts the test case, including preconditions, steps, expected results, acceptance criteria, and evidence to be captured.

The other important inputs are historical: previous validation packages, previous test cases for comparable systems, defect and deviation history, and relevant SOPs. This is company-specific knowledge, and without it even a very strong model produces something generic. With it, the output starts to reflect how your organization works.

But here is the part I always emphasize: the quality ceiling is set by the input, not by the model. If a requirement says, "the system shall have an audit trail," an AI will happily generate a plausible test that confirms an audit trail exists. It will not tell you that you actually needed to verify completeness, immutability, and attribution because you never asked for it.

Traceability matrices are tedious to build and maintain by hand and also easy to get subtly wrong. Are you seeing better accuracy, or just faster documentation of the same accuracy?

Henrichmann: Both, but not equally.

Notably, speed and coverage improve dramatically. A traceability agent is an AI tool that reads your life cycle documents and builds the relationships between requirements and the artifacts downstream of them. Rather than relying on links a person maintains by hand, the agent can analyze the whole document set and find orphan requirements, missing links, inconsistent identifiers, and requirements that were changed but whose downstream artifacts were not. Structural and mechanical accuracy improves because these are the same errors humans make when they maintain a matrix in a spreadsheet over the course of a project spanning many months. Maintenance under change is, in my experience, the biggest single win.

Substantive accuracy is a different story. We often think of traceability in CSV as asserting whether a link exists. It must go deeper than that to confirm whether the link is justified and whether the verification behind it is adequate. An agent can connect an audit trail requirement to a test case because the test mentions the words "audit trail" and reports traceability as complete. If that test only verifies that the function is present, the matrix is formally perfect but substantively empty.

And now the trap closes. The output arrives clean and highly convincing, but the human brain does not distinguish between the two well.

So, my answer is this: faster and more consistent everywhere, more accurate on the structural layer, and unchanged on the semantic layer unless you deliberately design a review that targets the depth and adequacy of testing. That review is where I would now spend the time that the tool has given back to me. If an organization simply takes the time savings and moves on, it has bought speed and accepted the same risk in a nicer format.

Can you explain the shifting validation roles evolving from documentation management toward quality architect responsibilities? How are day-to-day tasks different, and what new skills are required?

Henrichmann: I find this change the most encouraging of all.

Today, a large part of a validation professional's day is managing documents. Very few people entered our profession because they love this. It absorbs the effort that should go into thinking about risk.

In an AI-supported setup, much of that documentation management work is automated, and it allows the expert to focus on two things. The first is input quality: defining the business process clearly, eliciting and writing precise, testable requirements, identifying intended use, and structuring the risk assessment. The second is a critical review of AI-produced outputs with attention on coverage, anomalies, and adequacy rather than on formatting.

That is why I describe it as moving from document manager to quality architect. You are designing the arrangement that produces compliance, instead of manually assembling its artifacts by applying critical thinking in the ISPE GAMP 5 (Second Edition) sense, including awareness of your own subjectivity.

Reviewing AI-generated validation output requires knowing what a subtle error looks like, which might be more difficult than writing the documentation from scratch. How do you train a team to critically review AI output rather than rubber-stamp output that looks convincingly complete?

Henrichmann: Here, we have identified the hardest problem in this whole field. Training alone will not solve it because the effect comes from the process design and working conditions.

So, I would start with process design and use training to support it.

In the process design, we should limit review volume per person. Furthermore, we can use risk-based triggers instead of a uniform full review, so that attention goes where it changes something, and build-in cognitive forcing functions like small design elements that make it impossible to approve something without actually engaging with it.

Instead of an "approve" button, require a short documented justification or rotate reviewers and use peer review. Individual attitude toward automation is one of the strongest predictors of whether errors get corrected.

Then comes training. To solidify the risks in operators' minds, specify each of the failure modes by name: automation bias, cognitive ease, review fatigue, and sycophancy. People should recognize the feeling of "this looks right" as a signal to slow down. Show examples of plausible output that lacks substance.

And then you need to measure. This is the part many organizations skip. You need to ask: How many errors have slipped through, and how many are caught? The detection rate tells you whether your human control works. Without it, you are assuming detectability, which is precisely the illusion we should avoid.

As with any emerging technology, hype often runs ahead of reality. What are some of the capabilities that everyone loves touting but lack real-world pressure-testing?

Henrichmann: Let me name a few without being cynical, as I am genuinely optimistic about this field.

First, to the best of my knowledge, there is no fully autonomous "zero-touch" or self-validating system. Maybe in the near future, we will have agentic AI systems that plan and execute a whole sequence of validation steps on their own, using tools and other systems, thereby automatically and independently validating the system in question in accordance with your SOPs. That's not happening today.

The second is "AI checks AI," presented as independent verification where one model generates the test case, and a second one reviews it. It can be a valuable control, but its effectiveness depends entirely on genuine independence: different models and an adversarial setup. Two models from similar families, prompted similarly, share assumptions and blind spots, leaving you with the appearance of a second opinion, which is worse than having none, because it creates a feeling of safety.

Third, coverage claims as AI-supported testing can exercise a far greater number of scenarios, branches, and states than manual testing and that is a real and significant benefit. But 100% coverage is not possible, even with AI support.

What is genuinely working today, in my experience, is less glamorous and includes drafting, consistency and completeness checking, gap identification, structuring and reformatting, comparison of large document sets, and first draft test scripts from good requirements. That is already a lot of value.

GAMP 5 (Second Edition) prioritizes critical thinking first. Is AI compatible with that philosophy?

Henrichmann: It is compatible, but not without risk.

Critical thinking in GAMP 5 (Second Edition) means scaling effort to risk and justifying what you do, rather than following a template because the template exists. Nothing about AI conflicts with that. In fact, AI can support it through risk assessments informed by prior assessments. Controls across comparable systems give a broader, more consistent basis for judgment than one person's memory and the last project's document. It does not replace expert judgment, but rather widens the evidence that judgment is applied to.

AI, however, can accelerate the dangers that GAMP 5 was written against. Our industry has a long tradition of producing documentation as a substitute for thinking. If you take a template-driven, tick-box validation culture and add AI, you get template-driven tick-box validation at 10 times the throughput with better formatting.

On the other side, there is a good amount of critical thinking required before the AI does anything. If intended use, written requirements, and risk assessment are all done well, you can expect more robust AI outputs. For a validation team that's never used AI tools in this way, can you recommend some practical starting points? Are there any warnings you'd give before taking the leap?

Henrichmann: I would start in two places, in this order.

First, fix the inputs before buying anything. Take one system and genuinely improve the requirements: process-oriented, precise, testable, traceable, and stored as structured digital records rather than as documents. This has value even if you never introduce AI, and without it no tool will help you. Honestly, most disappointing projects failed here, not on the technology.

Second, run it in parallel. Take validation packages you already completed and trust, let the tool work on them, and compare. You learn where it is strong and, more importantly, what its characteristic mistakes look like.

Do not begin with your most inspection-critical artifact. Do not let a pilot drift into production use without change control, as this happens surprisingly often. Qualify the tool for your intended use; a supplier's compliance claim is not evidence.

There is no need to buy an expensive tool to get your toes wet. Use a generic LLM for starters, but consider confidentiality before pasting your specifications into a public service. You can use this outcome as the baseline to assess the performance of more sophisticated tools. Establish AI literacy training and a clear policy before enthusiasm outruns governance. Finally, engage your QA colleagues from day one, as bringing them in at the end is the most reliable way to lose a lot of time.

About The Author:

Frank Henrichmann is a senior executive consultant at Q-Finitiy Quality Management in Germany. He is an expert in quality management, computer system validation, and compliance. Previously, he worked at Parexel and Bayer with a background that includes extensive involvement in conducting internal and vendor audits. He is a co-author of both editions of the ISPE GAMP Good Practice Guide: Validation and Compliance of Computerized GCP Systems and Data and contributed to the GAMP 5 (Second Edition) guide as a chapter lead. He is an ISPE-certified GAMP trainer and the current chair of the GAMP Global Steering Committee.