Insights and Analysis
AI-washing – when AI hype becomes a litigation risk
We spend a lot of time with manufacturing leaders, quality teams, and in-house counsel who are all talking about the same basic issue, just from different angles: automation, AI, data, and how far to push “digital transformation” in a regulated environment. From our perspective as FDA lawyers focused on medical devices, and as ISO 13485 auditors, the question is no longer whether these tools are coming. They are already here. The real question is how to use them without creating a validation problem, a data-integrity problem, or an inspection story the company will later regret.
Join us for our fifth annual AI Health Law & Policy Summit in Washington, D.C., on May 13-14, where panelists will explore these and other rapidly evolving health care AI regulatory concerns.
In regulated manufacturing, the usual case for automation is easy to understand: fewer manual touches, fewer chances for error. But in this space, automation is not just an efficiency play. It is increasingly part of the company's process-control story. If you can show that a process is performed consistently, monitored appropriately, and controlled when it drifts, you are not just making the line run more efficiently to meet targeted yields. You are putting yourself in a better position to defend the system.
That compliance framing matters because QMSR doesn't reward “we trained our people well” nearly as much as it rewards demonstrable control—validation, monitoring, documented outcomes, and defined responses when things drift. We see companies investing in automation not only because it makes the line faster, but because it helps answer uncomfortable inspection questions: How do you know the process is executed the same way every time? How do you detect drift? Where is it documented?
At the same time, there is a real tension between modern flexible manufacturing and the way regulated operations actually work. Companies want rapid changeovers, software-driven adjustments, and room to optimize. But once a process is part of the validated state, even a small software update, sensor replacement, or robotics change can trigger change control, revalidation, and a round of documentation updates. We often tell clients to think about that early. In a regulated environment, flexibility is possible, but only if the quality system is set up to handle it.
We are seeing more AI tools show up in quality systems, sometimes clearly labeled as decision support and sometimes embedded in platforms that trend data, flag issues, or prioritize work in the background. The appeal is obvious: faster signal detection, better complaint trending, and the possibility of catching a problem before it turns into a nonconformance, corrective action, or even an escalation for a possible field action.
In practice, the AI discussion gets practical very quickly. The issue is not just whether the tool can find signals. It is whether the company can explain how it is being used and stand behind the result. We generally advise clients to treat AI output like any other input into the quality system: be clear about what the tool does, what data it uses, what counts as an acceptable output, and who is responsible for reviewing, accepting, or rejecting it.
When we say “defend it,” we mean something very specific: if an inspector, notified body, or internal auditor asks why a certain complaint cluster was escalated—or why it wasn't—the answer comes down to whether the basis for that decision can be explained in a way that is consistent with company procedures and records. “The model said so” is not a quality-system rationale. The more opaque the algorithm, the more pressure will be felt to show controls around training data, model changes, performance monitoring, and the boundaries of intended use.
That's where validation and transparency collide with “black box” decision-making. If an AI tool is influencing quality decisions—especially decisions that impact product release, complaint investigations, CAPA prioritization, or supplier controls—then the company should expect hard questions about validation, traceability, and governance. We're not saying FDA is anti-AI—indeed, FDA is using it as well. We are saying that regulators are pro‑control, and AI has to fit into a controlled system.
One practical point we emphasize: even if AI is “only” used for prioritization or trending, it can still shape the record. If an algorithm pushes the organization toward certain investigations and away from others, that pattern may become visible during an inspection. Over time, we expect AI-supported workflows to be evaluated not just on technical performance, but on whether they reliably drive timely detection, investigation, and escalation—because that is how quality decisions become enforcement narratives.
Automation and AI generate more data. In a QMSR environment, that is not just a technical issue; it raises the stakes on governance, traceability, and record integrity. As companies connect equipment, QMS platforms, complaint systems, supplier portals, and post-market information, they can move very quickly from not having enough information to having more than they can realistically manage with success.
In our audits and legal work, we keep coming back to a familiar theme: data integrity is not an IT afterthought. It is a quality system issue. If records are incomplete, overwritten, unauditable, or effectively untraceable, the organization may struggle to show objective evidence that processes were executed as intended. That's true whether the records are calibration logs, in-process QC activities, batch release data, complaint intake fields, or automated inspection outputs.
Cross-border operations add another layer. We increasingly see global quality systems that need centralized visibility, while local jurisdictions impose data-residency, privacy, cybersecurity, or government-access constraints. Those constraints can limit where data is stored, who can access it, and how quickly it can be shared during an investigation or field action. Ignoring these realities can create delays in investigation timelines and inconsistencies in records—both of which can frustrate investigators and draw regulatory attention.
For companies operating globally, alignment is the goal: you want a data governance approach that supports FDA expectations while also fitting within EU and other international requirements. Practically, that often means defining clear ownership of data sets, access controls, audit trails, retention rules, and escalation paths that work across systems and time zones. When those fundamentals are missing, “digital transformation” can unintentionally become “digital uncertainty.”
A “digital thread” sounds like a technology initiative, but the driver is often traceability pressure. QMSR-aligned systems, risk management expectations, and lifecycle management all push organizations toward connected records—from design inputs and verification, through supplier controls and manufacturing history, to post-market surveillance and complaint handling, with a heavy overlay of risk management.
From our standpoint, QMSR accelerates this trend because it rewards organizations that can demonstrate a clear chain of objective evidence. When information is siloed—different systems, different owners, inconsistent definitions—traceability breaks down right when the organization needs it most (e.g., during a complaint spike, a supplier failure, a recall decision, or an inspection).
Supplier data is a common area where traceability can break down. A manufacturer may have decent internal systems, but supplier records are frequently in different formats, use different identifiers, and operate with very different levels of control. That is how companies end up relying on spreadsheets, manual reconciliations, and workarounds right when the pressure is highest. In our view, supplier data quality should be treated as part of supplier qualification and ongoing oversight, not as a cleanup exercise after something goes wrong.
And when the organization can't produce the data, the consequences are rarely limited to “we'll improve next quarter.” “We don't have the data” can become a regulatory problem (because it undermines objective evidence) and a contractual problem (because it affects supplier obligations, customer commitments, and transaction diligence). In our experience, the strongest programs are the ones that can explain traceability clearly, support it with reliable records, and withstand scrutiny when regulators, counterparties, or investigators start asking hard questions.
As systems become more automated, we expect root-cause investigations to change shape. Instead of “operator error,” we anticipate more situations where the hard questions are about configuration control, sensor calibration, software updates, model drift, or data pipelines. Automation can reduce variability—but it can also centralize risk. When a single automated decision point fails, it may fail at scale.
That shift will also create new fault lines in responsibility allocation. If an algorithm fails, if data is incomplete, or if robotics behave unexpectedly, the inevitable question is: who owns the outcome? The device manufacturer still owns compliance, but the technical reality may involve automation vendors, software providers, system integrators, and suppliers. We expect quality agreements, vendor oversight, and change-control terms to become more detailed—and more litigated—because automated systems can blur the boundary between “tool” and “decision-maker.”
We also expect QMSR-related findings to show up more often in supplier disputes, diligence reviews, and enforcement matters. Automated systems can create better records, but they can also create a clearer record of what the company saw and what it failed to do. If trending was available but ignored, or if exceptions were repeatedly overridden without explanation, that can become a central issue later. The goal is not automation for its own sake. The goal is to make it easier to do the right thing consistently, while keeping meaningful human judgment where it belongs.
Even though the recent FDA warning letter addressing AI use in a drug manufacturing facility is not directly applicable to medical device manufacturers operating under QMSR, it is still worth reading closely. The basic lesson is not really about drug-specific current good manufacturing practice requirements; it is about regulatory expectations when companies begin to rely on AI in quality-affecting activities. In that letter, FDA objected not simply to the use of AI, but to the firm's apparent reliance on AI-generated specifications, procedures, and production records without adequate human review, and to the firm's suggestion that it missed a foundational validation requirement because the AI system did not identify it. For device manufacturers, the specific citations may differ, but the underlying enforcement logic will feel familiar: the company remains responsible for the quality system, the record, and the decision, even when software or AI helps generate the draft, recommendation, or workflow.
This warning letter provides a useful preview of how regulators may frame AI issues across regulated manufacturing more broadly. It reinforces several themes already running through this discussion: human oversight cannot be outsourced to a model; validation still matters even when the tool is marketed as “assistive”; AI-generated content that enters the quality system must be reviewed, approved, and controlled; and overreliance on an opaque tool can quickly become an inspection narrative. Put differently, if AI is helping draft procedures, prioritize investigations, trend complaints, shape production decisions, or influence release-related activities, then the organization should expect to justify intended use, define review responsibilities, monitor performance, and document when humans accept, reject, or override the output. The pharma letter does not rewrite device law, but it does offer a practical warning: FDA is unlikely to accept “the AI told us” or “the AI did not tell us” as a substitute for a governed, validated, inspection-ready quality system.
We do not see automation, robotics, and AI as future concepts anymore. They are already part of how many device manufacturers operate, and they are becoming harder to separate from the quality system itself. The companies that handle this well will be the ones that treat these tools like regulated inputs to a regulated system, with clear intended use, risk management, validation, change control, and reliable records.
If there is one practical takeaway, it is this: start with governance, not with the sales pitch. Identify the decisions the system will affect, decide what the record needs to show, and be clear about how changes will be reviewed and documented. From a legal and audit standpoint, that groundwork is often what separates a useful tool from an imminent problem.
Mike Heyl and Jodi Scott are FDA lawyers who advise medical device manufacturers, suppliers, and investors on QMSR and product-related issues. They are also ISO 13485 qualified auditors. Together, they have more than 50 years of experience helping medical device companies prepare for, manage, and respond to FDA inspections, warning letters, enforcement actions, and other high-stakes regulatory matters.
This article is the 28th in our thought leadership series, “DigiCure: Legal insights at the intersection of Technology and Life Sciences and Health Care,” which aims to help you stay informed about the broad array of legal and regulatory issues affecting companies operating at the intersection of the technology and life sciences & health care sectors. From using AI in clinical studies, to evolving patient data concerns, to the entire digital health product lifecycle, our team will discuss novel issues arising in all parts of the world, including unique deal-making, litigation, and compliance concerns. Ensure you are subscribed to Our Thinking to receive these new insights!