Brian Drapeau is founder, GxPFrame and a cell and gene therapy manufacturing and quality systems consultant.
He can be contacted at bdrapeau@gxpframe.com for any consulting or advisory needs. 

Brian Drapeau

In December 2025, FDA announced that it was deploying agentic AI capabilities across the agency. FDA defined these systems as AI designed to achieve goals through planning, reasoning, and multi-step actions, and said its deployment would allow employees to create more complex workflows using multiple models. The agency specifically identified applications ranging from premarket review and postmarket surveillance to inspections and compliance.1

Four months later, FDA issued a warning letter to a pharmaceutical manufacturer with an unusually direct subsection: “Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing.” The manufacturer had used AI agents to create drug-product specifications, procedures, and master production or control records. FDA's objection was not simply that AI had been used. The firm had failed to adequately review what the agents produced, and FDA stated that AI outputs used for current good manufacturing practice (cGMP) activities must be reviewed and cleared by an authorized human representative of the quality unit.2

There is something useful in the proximity of those two events. The same regulator deploying agents internally had just demonstrated where accountability remained when agents entered a regulated workflow. The agent did not receive the warning letter. The manufacturer did.

For the first wave of generative AI, pharma spent considerable time asking whether models could hallucinate, whether their outputs could be trusted, and whether humans needed to review them. Those questions do not disappear with agentic AI, but they are no longer enough. A chatbot produces an answer. An agent can participate in the process that determines what happens next.

Once that happens, the more important question becomes: what did we give it permission to decide?

Most generative-AI governance was built around a relatively simple interaction. A person asks a model a question, the model generates something, and the person reviews it. Prompt, output, review. Whatever uncertainty existed inside the model, there was an obvious control point downstream where a human could evaluate the artifact before it entered the regulated process.

Agentic AI changes that architecture. FDA's own definition captures the distinction: these systems can plan, reason, and execute multi-step actions toward a defined goal. Recent work on agentic systems in regulated environments similarly distinguishes systems that reason over supplied information from systems capable of committed writes to authoritative records.1,3

Put that inside a pharmaceutical quality system and the distinction stops being theoretical. Imagine an AI assistant drafting a deviation investigation from information a human provides. Now imagine an agent that retrieves the deviation itself, searches previous investigations, identifies relevant procedures, selects potentially related events, develops possible causal relationships, drafts the investigation, recommends a classification, and routes the record to the next workflow step.

Both might appear on an AI inventory as “AI supporting deviation investigations,” but they are not remotely the same use case. The first generates content. The second participates in work, and participation creates something output accuracy alone cannot describe: authority.

There is an upstream problem here that is easy to miss. The industry talks about whether “AI works in pharma” as though AI were a capability rather than a category of technologies. It is not particularly useful to ask whether an LLM is good at pharmaceutical research, deviation management, quality review, or drug development in the abstract. The useful question is narrower: which part of the task matches what this particular system has demonstrated it can do?

An LLM may perform well at retrieving and summarizing scientific literature. That does not by itself establish that it can independently reason about the underlying biology. An agent may reliably retrieve previous deviation investigations and summarize potentially related events. That does not establish that it can determine root cause. Same system, same records, completely different claim.

Draft EU GMP Annex 22 already moves in this direction by requiring the intended use of an AI model and the specific tasks it assists or automates to be described in detail, based on an in-depth understanding of the process in which the model operates. The draft also requires personnel involved with AI systems to have adequate qualifications, defined responsibilities, and appropriate access.4

This is why successful AI use cases often look less ambitious than failed ones. The task is narrow. The expected output is defined. Failure can be detected. Human intervention occurs before the consequential decision. There is some way to determine whether the system actually performed acceptably. That is not a lack of ambition; it is specificity.

Specificity has to come before autonomy, because an organization cannot responsibly delegate authority for a function it has never demonstrated the system can perform.

“Human in the Loop” Tells Us Less than We Think

Draft Annex 22 explicitly contemplates human-in-the-loop controls for non-critical generative-AI applications, assigning qualified personnel responsibility for determining whether outputs are suitable for their intended use.4 The problem is that saying a human is “in the loop” tells us that a person exists somewhere in the workflow. It does not necessarily tell us what that person still controls.

Observe → Human decides → AI executes.

AI observes → AI interprets → AI decides → AI executes → Human approves.

Both contain a human. They do not contain the same human control.

A reviewer who enters after an agent has already selected the evidence, excluded alternatives, interpreted the information, classified the event, and initiated the workflow may technically remain in the loop. But many consequential choices have already occurred. Human review can only operate on what reaches the human.

If an agent searches 10 years of deviation records and returns five “relevant” events, a reviewer can competently assess those five records and still have no idea that the sixth, most consequential record was excluded upstream. The final output can be correct relative to the evidence presented and still be wrong relative to the evidence that existed. The model did not hallucinate anything. It made a selection decision.

Human-in-the-Loop Is Not a Control Strategy by Itself. It Is a Description of Topology

The control question is where the human sits relative to the consequential decision, what information reaches them, and whether intervention still occurs early enough to change the outcome.

This is where agentic AI collides with something pharmaceutical quality systems have been defining for decades: decision rights.

Regulation 21 CFR 211.22 does not merely assign the quality unit responsibility. It assigns responsibility and authority. The quality control unit has authority to approve or reject components and drug products, review production records, and approve or reject procedures and specifications affecting product identity, strength, quality, and purity.5

That language matters. GMP deliberately distinguishes between performing work and possessing authority over the consequential decision attached to that work. A manufacturing operator may understand why a batch should be rejected and still lack authority to disposition it. An investigator may recommend a corrective action and preventive action (CAPA) without having authority to approve closure.

Agentic AI introduces a new actor into that architecture, yet many AI inventories still describe systems primarily by function: documentation assistant, investigation copilot, quality agent, regulatory assistant. Those labels tell us what the system is supposed to help with. They tell us almost nothing about what the system is permitted to do.

FDA's Purolea letter makes the distinction unusually concrete. FDA did not merely require that AI-generated cGMP documents eventually encounter a person. It said outputs or recommendations from an AI agent must be reviewed and cleared by an authorized human representative of the quality unit.2

Authority was already part of the regulatory architecture before the agent arrived.

The Table maps the distinct decision rights an AI agent may support, from observing information through approving an accountable GMP determination.

Table I. AI Agent Decision Rights

Retrieving five related deviations is not equivalent to deciding that those deviations constitute a recurring quality event. Recommending that a CAPA be opened is not equivalent to opening one, and opening a CAPA is not equivalent to approving its closure.

There is emerging technical precedent for thinking about agents this way. Recent work separates what an agent is technically capable of doing from the autonomy it should actually be allowed to exercise,6 while other work on regulated agentic architectures distinguishes low-agency reasoning from actions that commit writes to authoritative records.3

Yet an implementation containing any of those activities could still be summarized internally as “AI used in CAPA management.” That description is nearly useless for governance because the meaningful unit is not simply the use case. It is the decision right embedded inside the use case.

Capability and Permission Are Different Questions

Capability asks what the agent can do. Permission asks what the organization has decided it may do.

A sufficiently capable agent might technically be able to retrieve an investigation, classify it, modify the record, open a CAPA, assign actions, and route the record. Nothing requires the organization to permit all of those actions simply because the technology can perform them.

Recent agent-governance research makes this distinction by separating an agent's technical capability from the autonomy it should actually be allowed to exercise, based on risk, reversibility, oversight, and accountability.6

That distinction maps well onto GMP. Capability should establish the outer technical boundary. Risk and accountability should determine how much of that capability the organization actually authorizes.

Agent governance is beginning to develop language for this bounded permission structure. The useful idea is an authority envelope: the set of actions and permissions within which an agent may operate without additional authorization.

For GMP, I would define that envelope at the level of decision rights. It should specify what the agent can observe, interpret, recommend, decide, and execute; which systems and data it can access; which actions require human authorization; which actions are prohibited; when the system must stop and escalate; and who remains accountable for the outcome.

The narrower the envelope, the easier the assurance problem. As the envelope expands, the evidence supporting it should expand with it.

This is also where cybersecurity provides a useful precedent. The National Institute of Standards and Technology's (NIST) 2025 preliminary Cybersecurity Framework Profile for AI recommends treating AI systems as separate entities with their own permissions and authorization policies. For AI agents specifically, it recommends least privilege: agents should receive only the permissions necessary to perform their assigned role.7

GMP does not currently use the term “authority envelope” as a regulatory requirement. The point is not to pretend it does. The point is that existing GMP decision rights and emerging agent permission models are converging on the same control problem.

The Dangerous Decision May Happen Before the Final Answer

This becomes particularly important when organizations rely on final-output review as the primary control. Suppose an investigation agent produces an excellent final draft. The facts presented are accurate, the language is appropriate, and the causal argument is reasonable. A qualified QA reviewer reads it and approves it.

But the reviewer sees the artifact, not necessarily everything that produced it. Did the agent search every relevant prior event? Did terminology differences cause it to exclude an older investigation? Did it select the correct SOP revision? Did it discard an apparently unrelated result that would have changed the causal hypothesis? Did it decide that an intermediate finding did not warrant escalation?

The dangerous agentic failure can therefore occur several steps before the output anybody reviews. An upstream retrieval, selection, or interpretation can constrain everything downstream while leaving the final artifact looking perfectly reasonable. By the time the human enters the loop, the decision space may already have been narrowed.

This is also why 21 CFR 211.68 deserves more attention in the agent discussion. The regulation allows computerized and automated systems to perform pharmaceutical manufacturing functions, but requires appropriate controls so that changes to master production and control records or other records are made only by authorized personnel. It also requires computer-system inputs and outputs to be checked for accuracy, with the degree and frequency of verification based on the system's complexity and reliability.8

The regulation predates agentic AI by decades, so applying it to agent trajectories requires interpretation. But the control logic is strikingly relevant: authorization, accuracy, complexity, and reliability determine how much assurance the computerized process requires.

Audit Trails Record Events. Agents Raise a Provenance Problem

Pharma already understands audit trails. If User X changed field Y from A to B at 2:32 pm, the audit trail tells you what happened.

An agent introduces a different evidentiary problem because the consequential information may include what objective the agent received, which information it retrieved, which information it excluded, what tools it called, which intermediate conclusions affected subsequent actions, what model and configuration were operating, where human intervention occurred, and what happened after that intervention.

There is already a name for the broader concept: decision provenance. Singh, Cobbe, and Norval proposed it in 2018 as a way of exposing decision pipelines: the chains of inputs, decisions, actions, and downstream effects that operate across complex automated systems. Their argument was explicitly tied to accountability, audit, compliance, and oversight.9

The GMP application is straightforward. Logging that an agent changed a field establishes the event. Decision provenance would attempt to establish the information and intermediate actions that produced it.

That distinction becomes particularly important when the final output does not reveal the path that generated it.

Suppose a QA director has broad permissions across the quality management system because the role requires them, and the director launches an agent. Should the agent automatically inherit those permissions?

There is little reason it should. The director's authorization reflects role, training, qualification, organizational responsibility, and accountability. The agent possesses none of those merely because an authorized person initiated it.

NIST's emerging guidance points in the same direction: AI agents should have their own permissions and authorization policies, constrained by least privilege.7 And the technical evidence suggests this is not merely administrative housekeeping.

ToolPrivBench, published in June 2026, tested whether LLM agents select higher-privilege tools when a lower-privilege option would suffice. Across multiple domains and models, unnecessary high-privilege tool selection was common and became worse after transient tool failures. General safety alignment did not reliably prevent the behavior.10

Earlier work on Progent approached the same problem architecturally, proposing deterministic privilege policies around agent tool calls so that agents can perform necessary actions while unnecessary actions remain unavailable.11

That suggests a useful GMP principle: do not ask the agent to exercise restraint over authority it never needed in the first place. A deviation-triage agent may require read access across deviation records without requiring modification rights. An investigation-support agent may need to draft content without being able to approve the record containing that content.

The control is stronger when unnecessary authority is unavailable rather than merely discouraged.

Segregation of Duties Gets Strange Very Quickly

Agentic systems also complicate one of the oldest controls in quality systems: segregation of duties. A human investigator drafts an investigation. QA reviews it. Another authorized role may approve a consequential action. Those separations exist because concentrating incompatible responsibilities in one actor creates risk.

Now consider an agent that retrieves the evidence, determines which evidence is relevant, interprets it, drafts the investigation, recommends the classification, and routes the record. How many roles is that agent performing? More importantly, can the same system generate an analysis and then evaluate whether its own analysis is sufficient?

Using a second agent does not automatically solve the problem. If both agents depend on the same foundation model, retrieval architecture, or underlying data, the second system may reproduce the same blind spot rather than provide genuinely independent review.

Current GMP regulations do not prescribe an agent-specific segregation-of-duties architecture. This is an inference from existing quality and access-control principles, not a regulatory requirement. But it is precisely the kind of question manufacturers should resolve before automation quietly collapses boundaries that were deliberately separated when humans performed the work.

This extends the qualification model I have argued for previously. For generative AI, the qualified state is not “AI trained.” It is a relationship:

The use case itself has to be bounded by what the system has demonstrated it can actually do. Agentic AI adds another variable because the same system can perform the same nominal use case while possessing very different permissions.

Person × Agent × Version × Use Case × Authority Envelope

Suppose a QA reviewer demonstrates competent use of an investigation agent that retrieves historical records and drafts recommendations. Six months later, the organization enables the same agent to initiate workflows automatically. Same reviewer, same product, same model, same apparent use case, but different authority.

That is not the same qualified state. The system crossed a decision boundary, and the assurance case should cross with it.

Draft Annex 22 already provides part of the regulatory foundation for this argument. It requires the specific tasks an AI system assists or automates to be defined, assigns responsibilities and access levels to qualified personnel, and makes human responsibility explicit for generative AI used in non-critical GMP applications.4 What it does not yet provide is an agent-specific model for connecting those requirements to progressively greater decision authority.

There is an irony in all of this. FDA is not standing outside the technology telling industry to avoid it. The agency deployed agentic capabilities in December 2025 and has continued integrating AI more deeply into its own workflows.1

Meanwhile, FDA's first explicit manufacturing enforcement language around AI agents made human quality-unit authorization unmistakable.2

Those developments should not be collapsed into a claim that FDA has established a comprehensive regulatory framework for agents. It has not. But together they make something else clear: agentic technology is entering regulated work while the underlying accountability architecture remains human and organizational.

That matters because the value proposition of agents comes precisely from delegation. If a person still performs every retrieval, interpretation, decision, and action, there is not much agency left. The economic value comes from delegated work, but the regulatory problem changes when delegated work becomes delegated judgment.

The answer is not to slow everything down until another governance committee produces another framework. Pharma already knows how to deliberate. Nor is the answer to deploy broadly and trust that human review will catch whatever goes wrong.

The better approach is smaller and more concrete: narrow tasks, demonstrated capability boundaries, separate agent permissions, observable actions, reversible consequences where possible, named owners, predefined acceptance criteria, and explicit conditions that stop the system or force escalation.

This is where capability and authority need to remain separate. A system may be capable of acting autonomously without being authorized to do so. Emerging governance research, NIST guidance, and agent-security research independently support limiting permissions according to the actual task.6,7,10,11

Move quickly where failure is bounded. Demand more evidence as authority expands.

The first major agentic-AI quality failure probably will not look like science fiction. There will be no rogue machine independently deciding to release a batch. It will look like a normal workflow: a deviation gets classified, a record gets routed, a trend gets missed, a CAPA does not get opened, or a specification moves forward. Each individual action may look reasonable, and the final reviewer may even approve the result.

Somewhere three steps upstream, an agent will have made a decision nobody realized had become a decision. That is the frontier pharma needs to govern now, not because agents are uniquely dangerous, but because they are unusually easy to mistake for tools after they have started exercising authority.

Articles

Agentic AI changes the question from what a model can produce to what a system is allowed to decide.

Pharma's Next AI Problem Is Authority

Europe Map with virtual gavel and sound block and AI word. Concept of the EU recently adopted the AI Act, ushering in new restrictions on Artificial Intelligence use cases and mandating transparency | Image Credit: ©tanaonte -stock.adobe.com

In July 2025, European regulators published a draft annex to the European good manufacturing practice (GMP) guidelines that would keep ChatGPT-class systems out of any manufacturing decision touching patient safety, product quality, or data integrity. The annex does not restrict its use. It excludes it from scope, with the plain instruction that such models should not be used in critical GMP applications.¹

Industry pushed back hard enough that EMA is now reconsidering the boundaries of that position.²

 Volume 4 is 6 pages long. It was written by EMA’s GMP/GDP Inspectors Working Group (GMDP IWG) in cooperation with Pharmaceutical Inspection
Co-operation Scheme (PIC/S), whose members include regulators well beyond the European Union, and the FDA and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) sat in as observers. It supplements Annex 11 rather than replacing it. Treating this as a European problem is a mistake, and manufacturers exporting into the EU are expected to meet EU GMP wherever they operate.¹

The scope section does the work. Annex 22 covers machine learning models that got their functionality from training data rather than explicit programming, and only those that are static, meaning the parameters are frozen and do not adapt during use, and deterministic, meaning identical inputs produce identical outputs. Dynamic models that continuously learn are not covered and, in the draft’s words, should not be used in critical GMP applications.

The annex states that the document “does not apply to Generative AI and large language models (LLM), and such models should not be used in critical GMP applications.” They remain available for non-critical work, where qualified personnel are responsible for confirming the output is suitable for its intended use.¹

Everything after that is machinery: intended use characterization including rare inputs and bias risks; test metrics with acceptance criteria approved before testing begins; test data that is representative, stratified, correctly labeled, and rigorously walled off from training; explainability through feature attribution; confidence scores with thresholds that force an undecided flag rather than an unreliable answer; change control, configuration control, and drift monitoring on the input sample space.¹

All of it is also built around a category of model that excludes what most people mean when they say AI.

The FDA went a different way. Its January 2025 draft guidance does not propose validating AI models at all. It proposes a risk-based credibility assessment for a defined context of use, executed in 7 steps;³ credibility, for a context, not validation, for a system.

Neither agency extended the conventional validated-state concept cleanly across a technology whose behavior, provenance, and lifecycle may not remain sufficiently stable or observable for that claim to mean what it traditionally has.

Deterministic, and Why the Distinction Matters

For anyone who has not had to care about this distinction before, it is worth being plain about it, because much of the regulatory debate rests on what evidence collected today allows you to make claims about performance tomorrow.

A deterministic system produces the same output from the same input; press the button, get the result.

Traditional computerized-system validation does not require every regulated process to be literally deterministic. Pharma validates analytical methods, biological assays, manufacturing processes, and other systems that contain variability every day. What matters is that the expected performance and relevant sources of variability can be characterized, bounded, monitored, and maintained sufficiently for the intended use.

The validated state is therefore a claim about continuity. The evidence generated during qualification and validation remains meaningful because the system operating tomorrow is sufficiently connected to the system that was tested yesterday, and change control exists to identify when that claim may no longer hold.

Generative AI makes that connection harder.

Ask an LLM the same question twice, and you can get 2 different answers because the model samples from a probability distribution over possible next tokens rather than selecting a fixed one. But stochastic output alone is not the entire problem. Pharma already controls variable processes.

The harder problem is the combination: stochastic outputs, training data you may not be able to inspect, confidence signals that do not necessarily correspond to factual correctness, limited ability to reconstruct why an output occurred, and vendor-controlled changes to an underlying model you may not own.

And yet pharma already manages another highly variable component in virtually every GMP process it operates: People.

Ask me how I'm doing on Monday and I say, “living the dream.” Ask the same question Tuesday, and I might say, “my car's in the shop. I hit 49 minutes of traffic, and I have to drive to PA after work to see my in-laws.”

Nobody responds to human variability by demanding that people become deterministic. GMP controls the risk another way. It defines responsibilities. It establishes procedures. It qualifies people for assigned functions. It monitors performance. It reassesses competency when conditions change. And when the decision matters, it assigns accountability to a named person with authority to make it.

Every drug approved in the past century passed through systems full of people who were having different days, carrying different stress loads, and occasionally reaching different conclusions from the same information.

Pharma did not solve that problem by eliminating variability.

That distinction matters because the bind created by generative AI is not simply technical. It is epistemological. The industry keeps asking how to make a probabilistic component behave enough like conventional software to fit the existing assurance model.

The better question may be what exactly needs to be demonstrated for the human-AI system to be fit for a defined use.

The industry conversation tends to stop at stochastic output, as though the difficulty were a single technical inconvenience.

Read Annex 22 closely with a generative system in mind and 4 of its own controls become difficult to apply as written. Not because the annex is poorly drafted, but because those controls assume properties foundation models may not give you.

Section 6 is the most rigorous part of the draft, and rightly so. Test data must never have been used in development, training, or validation. Staff who touched the test set should not work on training the same model. There should be no copies of test data outside the controlled repository, with access logged.¹

Now apply that to a model trained on a web-scale corpus you did not assemble and cannot inspect.

You want to test it on deviation investigations, batch record review, or specification drafting. How do you demonstrate that your test cases, or documents functionally identical to them, were not in the training data?

You cannot establish that from information you do not possess.

This is contamination, and in machine learning, it is the difference between measuring generalization and measuring recall. Annex 22 asks for evidence of independence that users of externally developed foundation models may be unable to produce.

Section 8 asks that systems capture the features contributing to a classification, using techniques like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-Agnostic Explanations).¹ SHAP estimates each feature’s contribution to a prediction using Shapley values, whereas LIME approximates the model locally by fitting a simpler interpretable model around a specific prediction.

Those approaches can provide useful feature-attribution information for model classes where the relationship between inputs and outputs is sufficiently tractable.

Apply the same expectation to a large transformer and the problem changes.

A model can generate a plausible account of why it produced an answer. Whether that account faithfully describes the computation responsible for the answer is a separate question.

A model asked to explain its reasoning produces more model output.

That is not automatically a faithful trace.

For a regulator asking why a specific GMP decision was made, a plausible narrative can be worse than no explanation if the narrative creates confidence that the underlying evidence does not support.

Section 9 is elegant on paper. Log the confidence score for each prediction, establish a threshold, and have the model flag an outcome as undecided rather than produce an unreliable answer.¹

For a classifier calibrated against a defined distribution, that can be a meaningful control.

For generative models, token probability is not an estimate of factual correctness. A system can produce fluent, high-probability content that is entirely wrong without generating an internal signal equivalent to “I am hallucinating.”

The undecided flag depends on the system reliably knowing when it does not know.

That is precisely the capability in question.

4. The frozen state may not be yours to freeze

Section 10 puts the model under change and configuration control, with measures to detect unauthorized changes.¹

With a vendor-hosted model behind an API, you may not. The weights can change on the provider's schedule. System prompts can change. Safety layers can change. Supporting infrastructure can change. The practical question is whether those changes are observable, versioned, and governed sufficiently for the manufacturer to determine when its existing evidence no longer applies.

A change-control trigger you cannot observe is not much of a control.

And this one becomes contractual before it becomes procedural.

There is a fifth trap that applies to every model, deterministic or not. Section 4.3 says acceptance criteria should be at least as high as the performance of the process being replaced.¹

This seems straightforward, until you try to state the error rate of the human process you are automating.

Most organizations have never measured it.

That clause quietly requires a baseline almost nobody has.

Vendor after vendor is now selling agents into pharma with systems that chain steps, call tools, and take actions rather than return a single answer.

Draft Annex 22 does not use the word ‘agent’ anywhere in its 6 pages.

That does not mean agents are unaddressed. It means they get sorted by the properties the draft does define. An agentic system built on a LLM is generative. One that adapts during use is dynamic. One that can take a different path through the same task on 2 runs is probabilistic.

Any of those properties affects where it fits under the draft and what assurance problem the manufacturer inherits.

So, the category the industry is deploying fastest is the category the draft handles largely by implication.

If you are running an agent anywhere near a batch record, the exercise is not waiting for somebody to define the word for you. It is identifying exactly what the system does, what decisions it influences, which components can change, where human review occurs, and who owns the outcome when it is wrong.

The consultation closed in October 2025. EMA’s own account is that stakeholders indicated support for enabling technologies such as generative AI and LLMs.²

Then EMA did something more interesting than issue a response.

On June 30 and July 1, 2026, EMA convened a 2-day workshop, open session followed by a closed drafting-group review, to gather expert input on control and mitigation measures such as guardrails, described as part of a proposed risk-based approach for manufacturers wanting to use these technologies.²

The conversation has moved toward the conditions under which these technologies might be controlled. I do not think regulators build a 2-day evidence-gathering exercise around guardrails for a technology they have no intention of reconsidering, though that is my read on institutional behavior rather than anything EMA has stated.

The workshop is now complete. EMA's public record confirms that the second day was a closed review by the Annex 22 drafting group and that the Agency expects a report capturing the expert input. As of August 11, 2026, no final Annex 22 appears in 

 Volume 4, and PIC/S still lists Annex 22 under draft guidelines. The July 2025 exclusion, therefore, remains the published draft position. EMA's current GMDP Inspectors Working Group plan targets Q4 2026 to provide the European Commission with a final text.² ⁴ ⁵ ⁶

So here is the useful question, and it is not the one the industry is asking.

If EMA does open a pathway, what could the guardrail possibly be?

Not determinism alone, not a frozen state alone, not confidence scores, explainability, or test independence alone.

Each can contribute evidence where technically appropriate. None absorbs accountability for the final GMP decision.

What remains at that decision point is a person.

Section 3.3 addresses the case where a model provides input to a decision made by a human operator. Where the effort to test the model has been diminished, the description of intended use should include the responsibility of the operator.

And then: the training and consistent performance of the operator should be monitored like any other manual process.

Section 10.5 completes it, noting that depending on process criticality and the level of model testing, human review may require consistent review or test of every output, according to a procedure.¹

The less assurance you can establish around the model, the more assurance you need around the human control layer.

The scope section points in the same direction: generative models remain available for non-critical applications, where qualified personnel carry responsibility for confirming output suitability.¹

EMA convened a workshop to find guardrails for a problem its own draft had already exposed.

The final control layer was never going to be purely technical.

It was always going to end with a qualified human at the decision point.

The vocabulary problem is doing more damage here than the technology.

The industry keeps asking whether AI can be validated.

Pharma has never limited validation to processes that produce mathematically identical results every time. A cell-based potency assay varies by nature, and nobody pretends otherwise. It is controlled through defined performance characteristics, acceptance criteria, system suitability, monitoring, and qualified execution.

Draft Annex 22 to EU GMP would exclude generative AI/LLMs from critical manufacturing decisions, while permitting non‑critical use under qualified human responsibility. It applies only to static, deterministic ML models and specifies controls for intended use, independent test sets, explainability, confidence thresholds, change control, and drift monitoring. Extending these controls to foundation models is often infeasible because training data are opaque, “confidence” miscalibrates truth, explanations become narration, and vendors can update models. A workable assurance model qualifies Person × System × Version × Use Case, akin to media-fill qualification.

EMA’s draft GMP Annex 22 sidelines LLMs in critical manufacturing, fueling a push for risk-based guardrails and human accountability.

Europe Tried to Ban Generative AI From Critical GMP. The Ban May Not Survive. It Does Not Matter.

A recent McKinsey analysis of AI adoption in pharma, synthesizing perspectives from R&D and data-science leaders at Johnson & Johnson, GSK, Boehringer Ingelheim, AbbVie, and Genentech who appeared on the firm's Eureka! podcast, opens against a backdrop that should stop anyone in the industry cold.

Global corporate investment in AI reached $252.3 billion in 2024, and the pharma-specific AI market is projected to grow from roughly $4.35 billion in 2025 to nearly $35 billion by 2031, a compound annual growth rate above 41% across the forecast window.1,2 Despite that scale of investment, McKinsey reports that the industry has yet to see meaningfully shorter development timelines or improved preclinical and clinical success rates.3 In other words, the money is flowing and the pilots are launching. But the needle on outcomes that actually matter, like getting therapies to patients faster and more reliably, has not moved.

McKinsey's proposed explanation is a set of six enablers for successful AI deployment: treating AI as part of a broader business strategy rather than a bolt-on tool, using analytics to convert data into decisions, building an integrated technology stack, generating and curating purpose-built data, developing cross-functional talent that blends machine learning expertise with domain knowledge, and adopting flexible, iterative change management.3 The throughline in all six is a call to “redesign” rather than “tinker,” to rebuild workflows around AI instead of layering it onto legacy processes. It is a reasonable framework. None of the six points are wrong. But taken together, they diagnose the wrong disease. They describe symptoms of a much more basic failure: pharma companies adopted AI the way they would adopt a marketing trend, not the way they would adopt anything that touches a regulated process or a patient outcome.

Here is what actually happened across the industry over the past three or four years, and it is not really a mystery. AI became the defining topic in every boardroom, every investor call, and every competitor's press release. No pharma executive wanted to be the one explaining to a board why their company was behind on “AI transformation” while a competitor announced a flashy partnership with a cloud provider or a splashy new digital-twin initiative. That competitive anxiety, not a well-reasoned strategic thesis, is what actually drove the pace of adoption. Companies approved budgets fast. They greenlit proof-of-concept pilots fast. They selected vendors fast. And in that rush, pharma quietly skipped an entire category of due diligence it applies to virtually everything else in its operations: workforce qualification.

Consider how a pharma company treats a new piece of laboratory equipment, a new assay, or a new manufacturing process. Nothing goes live without a validation protocol. Nobody touches the equipment without documented training. There is a paper trail proving that the people operating the system understand its failure modes, its limitations, and the conditions under which its outputs can and cannot be trusted. This is not bureaucratic overhead for its own sake. It exists because the cost of an unqualified person misreading an assay or mishandling a manufacturing step can mean bad data, a failed batch, or in the worst case, patient harm. The entire culture of the industry is built around the idea that capability without qualification is a liability, not an asset.

I have run environmental monitoring performance qualifications on cleanrooms before those rooms were released for human product. The test is not whether the room holds its specification on its own. You fill it to maximum occupancy, more bodies than the process would ever normally put in there, and you have operators mimic the manipulations the work requires. Nobody is running the batch record end to end. That is not the point. The point is load. EM samples throughout: non-viable counts, viable air, surfaces, personnel. The room passes or fails on what those plates read after incubation. The whole protocol rests on a single assumption, which is that the people are the contamination source. So you put the worst realistic version of them in the room and find out whether sterility survives it. And it does not end at release. EMPQ gets repeated on a defined interval, because personnel are the largest variable in that environment and that variable never stops moving.

Now look at how the industry validates an AI model. Benchmark data set, held-out test set, performance metrics, sign-off. The model is tested by itself, against clean inputs, one time. There is no maximum occupancy condition. Nobody puts the actual users in the actual workflow under actual pressure and measures what comes out the other side. Pharma worked out decades ago that the humans are the challenge condition. It stopped applying that the moment the capability showed up as software.

The pushback here is predictable. An AI model is not a bioreactor. It is software, and pharma does not qualify people on software. So look at what actually triggers a qualification requirement. The output has to be read and acted on by a person. The failure modes do not announce themselves in the output. There are operating conditions outside of which the result cannot be trusted. An assay meets all three. A bioreactor meets all three. A model sitting inside a regulated workflow meets all three. A textbook does not, which is why nobody qualifies anyone on one. Nobody signs a batch record on the strength of what a textbook said. The moment a model's output feeds a decision that ends up in a regulated record, the software argument is finished.

AI escaped that scrutiny entirely. Pharma deployed models into discovery workflows, trial design, and patient identification without applying the same rigor to who was allowed to use them, how their outputs should be interpreted, and what the escalation path looked like when a model was wrong. Scientists and clinicians received powerful tools with a webinar and a slide deck, not a qualification program. That is the actual root cause of the return on investment (ROI) gap McKinsey is describing, and it is conspicuously absent from their list of six enablers.

None of this is hypothetical, and FDA has already acted on it. On April 2, 2026, the agency issued Warning Letter 320-26-58 to a Michigan drug manufacturer, one of the first CGMP letters with a dedicated section on inappropriate AI use.5 The firm had used AI agents to create drug product specifications, procedures, and master production records. When investigators cited it for distributing product without process validation, the firm's answer was that the AI agent it used had never told it validation was required. FDA cited 21 CFR 211.22(c) and stated that any output from an AI agent must be reviewed and cleared by an authorized human representative of the quality unit. Read the fact pattern closely. The failure was not a bad model. It was an operator who did not know what the tool could and could not be trusted to do, and who had nothing in place telling them to check. That is an unqualified user. The firm has since ceased drug production.

Why “Redesign the Workflow” Doesn't Fix What's Actually Broken

McKinsey's framing implicitly assumes that if a company gets the strategy, the tech stack, the data, and the talent mix right, the training problem takes care of itself. It does not. A company can have a beautifully redesigned workflow, a state-of-the-art tech stack, and a strong cross-functional team. It will still fail if the people using the AI day to day do not know how to evaluate whether a model's output is trustworthy in a given context. A pathologist reviewing an AI-flagged tissue sample, a trial designer relying on a digital-twin simulation, or a discovery scientist trusting a molecule-design model all need the same thing: a structured understanding of what the model was validated to do, where its blind spots are, and what a compliant record of that decision-making looks like. It is not something that gets solved by hiring more “trilingual” data scientists, as J&J's approach is described in the piece, or by building a better data pipeline, as GSK apparently has. Those are necessary conditions. They are not sufficient ones.

This is precisely why so many AI pilots in pharma stall out exactly where McKinsey observes them stalling, unable to scale past an isolated proof of concept. Scaling an AI tool across an organization means trusting hundreds or thousands of employees to use it appropriately, under real regulatory scrutiny, without a data scientist standing over their shoulder. If those employees were never actually trained and qualified on the tool in a documented, auditable way, scaling is not a workflow problem but a trust and compliance problem. No amount of iterative change management resolves that.

What Genuine Qualification Actually Requires

If pharma wants to treat AI the way it treats every other capability that affects patient outcomes, a few things need to happen. None of them are new. Role-based training needs to exist for every function that touches an AI system, from the bench scientist using a molecule-design tool to the clinical operations lead relying on a patient-matching algorithm to the regulatory affairs team responsible for explaining that algorithm to the FDA or the European Medicines Agency (EMA). That training needs to be specific to how each role actually uses the tool, not a generic “intro to machine learning” course that treats a clinician and a data engineer identically.

Validation needs to extend beyond the model itself to the humans operating it. It is not enough to validate that a model performs well on a benchmark data set; someone needs to validate that the people using that model in production understand its intended use, its limitations, and the conditions that would trigger escalation to a human expert or a fallback process. That is the qualification layer that mirrors how pharma validates equipment and processes today, and it is the layer that has been almost entirely missing from AI rollouts across the industry.

Pharma must document all of this in a way that would satisfy a regulator asking, after the fact, how a given AI-assisted decision was made and by whom. Compliance cannot be retrofitted onto a tool that has already been in production for two years. Pharma must build it into the qualification process from day one, the same way it builds compliance into every other validated system it operates.

The consequences of skipping this step are exactly what McKinsey's podcast guests are describing, even if they are not naming it directly. Pilots that produce promising results in a controlled setting fail to scale because the broader workforce was never brought along. Clinicians and scientists who were never given the tools to evaluate outputs treat them as suspect or ignore them, so the model's insights sit unused. Governance becomes an afterthought, bolted on only after a near-miss or an audit finding forces the issue, rather than being built in from the start. And perhaps most damaging long-term, trust in AI as a category erodes internally, making the next rollout even harder, because employees remember that vendors handed them a powerful, poorly explained tool and told them to figure it out.

None of this is a strategy failure or a talent-mix failure in the way McKinsey frames it. It is the predictable result of skipping the qualification step that every other regulated capability in this industry goes through as a matter of course. Qualification is not the only constraint on AI returns in pharma. It is the binding one, because it is the only one the industry skipped entirely while it went to work on the others. Close it and a company finds out whether the technology it bought was worth buying. Skip it and it never will.

The evidence outside pharma points the same way. It does not settle anything yet. YouGov surveyed more than 500 US and UK business leaders for DataCamp in early 2026. Just over one in five reported significant positive ROI on their AI investment. Among organizations running mature data and AI literacy programs, that number was 42%.4 Treat that as a signal, not a finding. It is self-reported. It is correlational. The company that paid for it sells upskilling. General AI literacy is also not qualification in the way 21 CFR 211.25 uses the word. But the direction holds, and pharma should clear that bar faster than anyone, because it already runs the qualification infrastructure every other industry is now trying to build from scratch.

A Solution that Treats this Like the Compliance Problem it Is

The fix is not complicated. It requires discipline that the “move fast” instinct around AI has made unfashionable. Pharma needs to build AI workforce qualification programs with the same seriousness it applies to equipment validation and process training: role-specific curricula, documented competency assessments, and governance frameworks that are explicitly mapped to FDA and EMA expectations rather than invented internally on an ad hoc basis. The goal is to build a qualified workforce that actually lets AI-driven redesign, the kind McKinsey is calling for, succeed instead of stalling at the pilot stage.

Pharma does not have a problem with AI's potential. The technology clearly works, and the use cases McKinsey describes, from prescreening trial candidates to simulating patient outcomes with digital twins, are genuinely valuable. What the industry has is a self-inflicted training and validation gap, born from the pressure to move fast on the hottest technology of the decade instead of treating it like any other internal capability that requires rigorous, compliant qualification before it is trusted with real decisions. Close that gap and the industry finds out what its AI investment was actually worth. Keep skipping it, and no amount of strategic redesign will save the next round of pilots from the same fate as the last one.

1. Economy. In: The 2025 AI Index Report. Chapter 4. Stanford Institute for Human-Centered Artificial Intelligence; 2025. Accessed August 7, 2026. 

2. Artificial intelligence (AI) in pharmaceutical market size and share analysis: growth trends and forecast (2026-2031). Mordor Intelligence. Updated August 7, 2026. Accessed August 7, 2026. 

3. Devereson A, Nagra N. How pharma is rewriting the AI playbook: perspectives from industry leaders. McKinsey & Company. November 6, 2025. Accessed August 7, 2026. 

4. AI training efforts give ROI a boost. CIO Dive. March 2, 2026. Accessed August 7, 2026. 

5. Warning letter 320-26-58: Purolea Cosmetics Lab. US Food and Drug Administration. April 2, 2026. Accessed August 7, 2026. 

AI investment in pharma is accelerating, yet development timelines and success rates have not meaningfully improved. The critical gap is not strategy, data, or technology stacks, but workforce qualification for AI embedded in regulated decisions. Unlike equipment and processes, AI tools have been deployed without role-based, documented training on intended use, limitations, and escalation pathways. FDA’s April 2026 warning letter illustrates how unqualified reliance on AI-generated quality documents can precipitate CGMP failures. Closing the qualification gap is framed as the binding constraint on scalable ROI.

Pharma's AI ROI gap isn't strategy or tech; it's the missing workforce qualification step every other regulated capability requires.