AI Governance in Healthcare: Beyond HIPAA into FDA Territory

AI Governance in Healthcare: Beyond HIPAA into FDA Territory

When Ridgeway Health - a mid-sized regional health system - launched an LLM-based clinical documentation assistant, leaders expected improvements in physician time-to-note and coding accuracy. Six months later the deployment taught them an expensive lesson: the assistant's "smart summarization" had begun surfacing inferred treatment suggestions and order prompts in a subset of cases. Those prompts crossed the line from administrative documentation into clinical recommendation. Faced with potential FDA device classification and state scope-of-practice questions, Ridgeway paused the feature, brought in regulatory counsel, separated documentation only workflows from any clinical-recommendation features, and stood up a formal AI Clinical Governance Committee. The move cost time and money up front - but it turned an operational risk into a controlled, auditable asset.

This is the new reality for health systems: HIPAA compliance and data security are necessary but not sufficient. AI governance in healthcare must extend into FDA territory, state practice law, and lifecycle change control. Below is a pragmatic guide - grounded in Ridgeway's experience - to help clinical, compliance, and IT leaders get AI-economy-ready.

Why HIPAA is just the floor

HIPAA protects patient privacy and sets rules for data handling. Most health systems have that covered - or at least they think they do. But HIPAA tells you nothing about whether a software function is a medical device that requires FDA oversight, how to manage intended-use drift, or how to control iterative AI model changes that alter clinical behavior.

AI tools in care delivery can be:

- Administrative: documentation autocompletion, coding suggestions, workflow triage that do not recommend clinical actions.
- Clinical support: tools that interpret data, prioritize diagnoses, recommend tests/treatments, or otherwise change clinical decision-making.

The distinction matters. Administrative tools are usually low regulatory risk. Clinical support functionality may be regulated as (software as) a medical device, subject to FDA expectations around validation, labeling, post-market monitoring, and - increasingly - predetermined change control for AI/ML modifications.

FDA's evolving playbook: what to watch

The FDA has signaled how it will regulate AI/ML-based software through guidance, action plans, and a risk-based lens:

- 21st Century Cures Act carveouts (2016) and subsequent FDA guidance describe when Clinical Decision Support (CDS) software is not a device - for example, if it allows independent review of the basis for recommendations and does not directly interpret images or signals.
- For AI/ML-based Software as a Medical Device (SaMD), the FDA has proposed a Total Product Life-Cycle approach and the concept of a Predetermined Change Control Plan (PCCP). A PCCP pre-defines the types of modifications a manufacturer may make without submitting a new premarket application and describes methods to control, validate, and monitor those changes.
- FDA's AI/ML Action Plan emphasizes Good Machine Learning Practices (GMLP), real-world performance monitoring, and transparency.

Practical implications:
- If your assistant begins to generate therapy suggestions, dose recommendations, or trigger orders, treat it as high-risk until counsel and regulators say otherwise.
- If you or your vendor plan frequent model updates that change clinical outputs, expect to document and validate those changes - and in future, regulators may expect a PCCP-like plan.

The line between administrative AI and CDS

The boundary between documentation and clinical decision support is often a matter of how information is presented and whether a clinician can readily reconstruct the reasoning:

- Documentation assistant (lower risk): auto-populates a note, summarizes conversations, suggests coding based on documented facts, but does not recommend specific clinical actions. The clinician can independently verify the facts the assistant used.
- CDS/device (higher risk): analyzes data to recommend diagnostic or therapeutic actions, prioritizes patients for escalation, or automates orders. If the software hides its logic or prevents independent clinician review, it trends toward device classification.

Ridgeway's misstep was subtle: a convenience feature that turned a synthesized problem list into a prioritized set of suggested orders when certain keywords appeared. That made the output actionable in a way that undermined the independent review box under CDS exemptions.

State scope-of-practice matters - and often gets overlooked

Even when a feature is crafted as "documentation only," outputs that enable non-physician staff to take clinical actions can trigger state scope-of-practice rules. Examples:

- A nurse relying on an AI-generated triage recommendation to change a standing order - some states limit such delegation without explicit protocols.
- Pharmacists using predictive dosing suggestions to adjust medication without pharmacist-prescriber agreements in place.
- Telehealth workflows where non-physician moderators use AI summaries to make escalation decisions.

Governance must include state regulatory review, especially for multi-state systems. A feature that's permissible in one state might be impermissible in another.

Building the governance structure that holds it together

Ridgeway's recovery hinged on standing up a formal AI Clinical Governance Committee with clearly defined roles and decision rights. Your governance should ensure economic readiness (costs, vendor risk), workflow readiness (safe integration, clinician training), and governance readiness (regulatory, legal, auditability).

Suggested structure and responsibilities:

- Executive sponsor (e.g., CMO or COO)
- Sets risk tolerance and prioritizes clinical oversight.
- AI Clinical Governance Committee (cross-functional)
- Members: CMO (chair), Chief Nursing Officer, Chief Compliance Officer, General Counsel, CIO/Chief Digital Officer, Chief Information Security Officer, Chief Data & Analytics Officer, Head of Patient Safety/Risk, Procurement lead, and patient representative.
- Responsibilities: classify AI by risk, authorize deployments, approve labeling and clinician disclaimers, oversee pilot criteria and rollouts, evaluate state practice implications, and maintain the AI inventory.
- Technical subcommittee (IT, data science, vendors)
- Responsibilities: model validation, performance monitoring, cybersecurity, logging, integration architecture, and rollback procedures.
- Clinical review panels (specialty-specific)
- Responsibilities: domain validation, clinician training curricula, escalation workflows, and acceptance testing.
- Legal & compliance
- Responsibilities: regulatory analysis (FDA/CDS), contracts and indemnity, reporting obligations, and privacy/HIPAA oversight.
- Risk & patient safety
- Responsibilities: incident review, post-market surveillance, adverse event reporting, and mitigation plans.

This committee should meet regularly and be empowered to pause deployments - Ridgeway's quick pause prevented greater downstream exposure.

Operational controls and the PCCP mindset

Whether your AI is vendor-provided or in-house, design governance to reflect PCCP principles even if the FDA has not yet required them for your product:

- Define the intended use, performance baseline, and validated endpoints before deployment.
- Pre-specify allowable modifications and the validation strategy for each (data drift monitoring, shadow testing, A/B rollout windows).
- Implement robust logging and audit trails that tie outputs to model version, training data snapshot, and inference time context.
- Maintain a model change-control register with impact assessments signed off by clinical leadership.
- Create a post-market monitoring plan: performance metrics, feedback loops from clinicians, adverse event thresholds that trigger escalation.
- Contractually ensure vendors deliver transparency on model changes, provide reproducible testing artifacts, and accept responsibilities for regulatory compliance where applicable.

These controls create predictability - and regulators will view predictable, auditable processes more favorably than ad hoc updates.

Workflow readiness: labeling, UI, and clinician authority

Small UI choices have big regulatory consequences. Make them intentionally:

- Separate documentation UI from clinical recommendation UI. Use distinct screens, badges, and disclaimers.
- Require explicit clinician acknowledgment for any suggested order or recommendation (no auto-accept).
- Surface the information basis for a recommendation - patient data elements that led to the suggestion - to enable independent review.
- Train clinicians on when to treat outputs as a convenience versus a clinical input and how to document reliance.
- Provide easy mechanisms for clinicians to flag erroneous outputs and for those flags to feed into model retraining and safety monitoring.

Ridgeway's decision to separate documentation flows from recommendation flows reduced the chance of accidental clinical action and clarified clinician responsibility.

Economic readiness: measuring the cost of not doing governance

Governance isn't just compliance theater - it protects revenue and avoids deep operational costs:

- Pause costs and rework are real: Ridgeway estimated a 6-week pause, engineer hours, vendor rework, and retraining costs. That was cheaper than potential regulatory enforcement, product recalls, or malpractice claims.
- Poorly governed AI can erode clinician trust, increasing documentation burden and turnover.
- Well-governed AI improves throughput and reduces downstream billing errors - but only if its outputs are safe and auditable.

Budget for governance early: legal reviews, validation testing, clinician time for acceptance testing, and technical monitoring infrastructure.

The final layer: vendor management and contracts

Most health systems do not build foundational LLMs themselves. Contracts must allocate regulatory responsibilities, change notifications, and transparency obligations:

- Require vendors to disclose model architecture class, update cadence, and validation evidence for clinical uses.
- Include clauses that require prior notice and a cooling-off period for model updates that may affect clinical outputs.
- Demand access to logs and model versioning metadata for incident investigations.
- Define liability and indemnity for regulatory actions or adverse clinical outcomes attributable to vendor software.

Conclusion - a single readiness move to start today

Ridgeway's story has a simple moral: HIPAA compliance is necessary but not sufficient. If any AI product touches clinical decisions, treat it as potentially within FDA and state practice scopes and govern it accordingly.

Concrete first move (AI economy-ready): create an AI inventory and classify each tool by risk within 30 days. For every tool that touches the clinical care pathway, require a short regulatory impact assessment authored by legal and clinical leadership that answers:
- Does this tool produce a clinical recommendation or enable independent review of its basis?
- Could any output change treatment, dosing, ordering, or escalation?
- Will model updates change clinical outputs without predictable validation?

That one document will tell you which tools can continue with HIPAA-plus controls, which need tighter FDA-style change control and validation, and which must be paused and redesigned. From there, stand up the cross-functional AI Clinical Governance Committee to enforce the next moves.

AI can improve care and clinician experience - but only if governance turns risk into a managed, auditable capability. Executive leaders who prioritize regulatory, workflow, and economic readiness now will avoid expensive retrofits and convert AI into a durable advantage rather than a compliance crisis.