HIPAA and AI Tools: What Healthcare Leaders Are Getting Wrong

A health system in the Midwest got flagged by OCR last year because physicians were pasting patient notes into ChatGPT to generate summaries. No one had approved the tool. No one had signed a business associate agreement. No one had even asked whether one was required. The doctors thought they were being efficient. Leadership thought AI adoption was happening "organically." What actually happened was a cascade of HIPAA violations that could have been avoided with ten minutes of clarity on how the rules work.

I'm seeing this pattern repeat across healthcare organizations. Executives want to deploy AI tools—ambient scribes, diagnostic support, workflow automation—but they're making decisions based on incomplete mental models of HIPAA AI compliance. The confusion isn't about obscure regulatory edge cases. It's about foundational questions: when does a vendor become a business associate, what constitutes PHI in an AI context, and whether the BAA you signed actually covers what your teams are doing with the tool.

This isn't a theoretical problem. The Office for Civil Rights has already issued guidance. Enforcement actions are coming. The organizations that wait until after an incident to figure this out will pay more than those who get it right now.

The Business Associate Agreement Gap Everyone Ignores

Most healthcare leaders know they need a BAA when a vendor handles PHI. Where they go wrong is assuming their existing BAA covers new AI use cases, or that a BAA on file somewhere means the vendor is actually compliant. Neither assumption holds up under scrutiny.

Here's what I see in practice: a hospital signs a BAA with an AI scribe vendor. The contract covers transcription services in the exam room. Six months later, clinical staff start using the same vendor's new "ambient intelligence" feature that continuously listens to patient interactions and suggests documentation improvements in real time. No one checks whether the BAA covers this expanded scope. No one confirms whether the vendor's subprocessors—often third-party speech recognition engines or cloud infrastructure providers—are also covered under downstream BAAs.

The assumption is that one BAA equals universal coverage. It doesn't. BAAs are use-case specific. When the use case changes, the agreement needs to be reviewed and often amended. When subprocessors change, you need to verify the chain of agreements still holds. This isn't legal hair-splitting. It's the difference between a defensible compliance posture and an OCR investigation that uncovers you've been transmitting PHI to unsecured endpoints for months.

The second gap is more subtle: assuming a signed BAA means the vendor is actually doing what they promised. A BAA is a contract, not a compliance certification. It obligates the vendor to implement safeguards. It doesn't prove they've done so. I've reviewed vendor security programs where the BAA promised encryption at rest and in transit, logging of all PHI access, and annual penetration testing. When we asked for evidence, we got marketing materials and a SOC 2 report that didn't actually scope the AI processing environment. The BAA existed. Compliance did not.

What to Verify Before You Trust the BAA

When evaluating an AI vendor's BAA, you need to confirm three things that most contracts leave ambiguous:

• Scope of PHI processing: Does the agreement explicitly cover the AI model training, inference, and any intermediate processing steps? Many BAAs are written for traditional SaaS tools and don't address whether patient data is used to fine-tune models or improve algorithms.
• Subprocessor transparency: Does the vendor disclose all downstream parties that will handle PHI, and do they maintain BAAs with each one? AI vendors often rely on hyperscale cloud providers, third-party model APIs, and data labeling services. Each one is a potential breach point.
• Data residency and persistence: Where does the PHI go, and how long does it stay there? Some AI tools cache patient data to improve response times. Others store conversation logs for quality assurance. If the BAA doesn't address data retention and deletion, you don't have a complete agreement.

If you can't get clear answers to these questions, you don't have HIPAA AI compliance. You have a document that says "business associate agreement" at the top and a lot of risk underneath.

For more on how to evaluate whether an AI vendor actually needs a BAA in the first place, see Do AI Vendors Need to Sign a BAA? The Answer Is More Complex Than You Think.

ChatGPT and Consumer AI Tools in Clinical Workflows

The most common violation I see isn't from a failed enterprise deployment. It's from individual clinicians using consumer AI tools because no one told them not to. A physician uses ChatGPT to draft a patient letter. A nurse pastes lab results into an AI assistant to generate a summary for handoff. A scheduler uses a free transcription tool to document a phone call with a patient about appointment details.

Each of these actions is a HIPAA violation. Consumer AI tools—ChatGPT, Claude, Gemini, and others in their standard public-facing forms—do not sign BAAs. They do not commit to HIPAA compliance. They explicitly disclaim any responsibility for PHI in their terms of service. When a clinician pastes a patient's information into one of these tools, that data leaves your control, enters the vendor's training pipeline, and potentially gets stored in ways you can't audit or delete.

OpenAI offers an enterprise version of ChatGPT that includes BAA coverage. Google offers a HIPAA-compliant tier of its AI services. These versions exist because the default consumer offerings are not compliant. The problem is that most healthcare workers don't know the difference. They see "ChatGPT" and assume it's one product. They don't realize that the free version they access from a browser is categorically different from the enterprise version with data processing agreements in place.

Leadership often discovers this after the fact. An IT audit finds API calls to openai.com from clinical workstations. A staff survey reveals that 40% of providers have used a generative AI tool in the past month. No one logged the use. No one tracked what data was entered. No one can confirm what happened to it afterward.

The Policy Gap That Enables Shadow AI

Why do clinicians use non-compliant tools? Because they work, they're free, and no one has given them a compliant alternative. Telling staff "don't use AI" without providing an approved option is a recipe for shadow IT. People will solve their problems with whatever tools are available. If you haven't deployed sanctioned AI tools with BAAs in place, you're just ensuring that non-compliant tools will spread undetected.

The fix isn't just policy. It's providing a viable path. Deploy an enterprise AI tool with a proper BAA. Train staff on what's approved and what isn't. Make the compliant option easier than the non-compliant one. Monitor for API calls to consumer AI services and block them at the network level if you have to. You can't enforce HIPAA AI compliance by memo alone.

I wrote a detailed breakdown of the specific risks and mitigations for this scenario in ChatGPT in Healthcare: HIPAA Risks and How to Manage Them.

Ambient Listening Tools and the Conduit Exception Myth

Ambient AI scribes are one of the fastest-growing categories of healthcare AI. A device in the exam room listens to the patient-provider conversation, transcribes it, extracts clinical information, and auto-populates the EHR. The promise is significant time savings and better documentation. The compliance risk is that most organizations don't understand when these tools qualify as business associates and when they don't.

Some vendors claim they're acting as a "conduit" and therefore don't need a BAA. The conduit exception in HIPAA applies to entities that merely transport data without accessing it—like the postal service or a courier. The argument from some AI vendors is that their tool just listens and passes structured data to the EHR without "accessing" PHI in a meaningful way.

This argument is almost always wrong. The conduit exception requires that the entity have no more than transient access to the data and that it not store, process, or analyze the information. An AI scribe that converts speech to text, applies natural language processing to extract clinical entities, and generates structured output is doing far more than transient transport. It's processing PHI. It's applying logic to it. It's creating derivative data products from it. That makes the vendor a business associate, full stop.

OCR has been clear about this. If a vendor is performing a function or activity on behalf of a covered entity that involves PHI, and if that involvement goes beyond mere transportation, a BAA is required. The fact that the processing happens in milliseconds or that the vendor doesn't "store" the data long-term doesn't change the obligation. Transient access during processing still counts as access.

What Ambient AI Vendors Should Be Doing

Legitimate ambient AI vendors know they're business associates. They offer BAAs as a standard part of their sales process. They architect their systems to minimize data retention. They provide audit logs that show when PHI was accessed and by whom. They undergo third-party assessments and share the results with prospective customers.

If a vendor resists signing a BAA or insists they don't need one, that's a red flag. Either they don't understand HIPAA, or they're trying to avoid the compliance obligations that come with business associate status. Neither scenario is acceptable for a healthcare organization that's serious about regulatory compliance.

The due diligence checklist for ambient AI tools should include:

• Confirmation that the vendor will sign a BAA that explicitly covers the AI processing use case
• Documentation of how PHI is encrypted during capture, processing, and transmission
• Evidence of subprocessor agreements if the vendor relies on third-party cloud or AI services
• Details on data retention policies, including how long audio, transcripts, and derived data are kept
• Access to security audit reports that cover the AI environment, not just the vendor's corporate IT infrastructure

If the vendor can't provide this, you're not deploying an enterprise tool. You're introducing an unmanaged risk into your clinical environment.

AI Model Training and the De-Identification Fiction

One of the more dangerous misconceptions about HIPAA AI compliance is the belief that de-identified data is "safe" for AI training and that de-identification is straightforward. Both assumptions are wrong, and organizations that act on them are creating compliance exposure they don't understand.

HIPAA permits the use of de-identified data without restriction, provided the data meets one of two standards: the Safe Harbor method, which requires removal of 18 specific identifiers, or the Expert Determination method, which requires a qualified statistician to certify that re-identification risk is very small. Most organizations that claim they're using "de-identified data" for AI haven't done either.

What they've actually done is remove names and medical record numbers and called it good enough. That's not de-identification under HIPAA. It's incomplete anonymization that still qualifies as PHI if enough indirect identifiers remain. And with AI models, the risk of re-identification is higher than most people realize. Modern machine learning techniques can infer identities from patterns in clinical data, especially when combined with external datasets. A model trained on "de-identified" patient encounters can still leak information about individuals if it memorizes specific cases or if adversarial queries can extract details about the training set.

The Expert Determination Problem

Even if you follow the Safe Harbor method precisely, you haven't necessarily eliminated re-identification risk in an AI context. Safe Harbor was designed for static datasets and traditional disclosure scenarios. It wasn't designed for machine learning models that can encode complex relationships between variables and potentially reconstruct aspects of the training data through inference.

Expert Determination is the theoretically better path, but it requires a qualified expert to analyze your specific dataset and use case. Most healthcare organizations don't have that expertise in-house, and hiring a statistician to perform this analysis is expensive and time-consuming. The result is that many organizations skip it, assume their data is de-identified based on a gut-level assessment, and proceed with AI training under the assumption that HIPAA doesn't apply.

If OCR investigates and finds that your "de-identified" data wasn't properly de-identified, every use of that data becomes a potential violation. Every vendor you shared it with becomes a business associate you should have had a BAA with. Every model you trained becomes evidence of unauthorized disclosure. This is not a small compliance issue. It's a foundation-level failure that can unravel an entire AI program.

The safe approach is to treat any patient-level data used for AI training as PHI unless you have formal Expert Determination or have meticulously followed Safe Harbor and documented the process. If you're not sure, treat it as PHI. Get the BAAs. Implement the safeguards. The cost of over-compliance is far lower than the cost of getting this wrong.

The Incident Response Gap for AI-Related Breaches

Most healthcare organizations have incident response plans for traditional breaches: ransomware, stolen laptops, unauthorized EHR access. Very few have updated their plans to cover AI-specific breach scenarios. When an AI-related breach happens, the response often flounders because no one knows who owns the problem or what the notification obligations are.

Consider a scenario: your ambient scribe vendor discloses that a misconfigured cloud storage bucket exposed patient transcripts for a two-week period. The vendor didn't discover the exposure themselves; a security researcher found it and reported it. The vendor can't confirm whether anyone accessed the data, but the logs show the bucket was publicly readable. What's your notification obligation?

Under HIPAA, if there's an unauthorized disclosure of PHI and you can't demonstrate a low probability that the data was compromised, you're required to notify affected individuals, OCR, and potentially the media if the breach affects more than 500 people. The burden of proof is on you to show the data wasn't accessed. In a cloud misconfiguration scenario, that's nearly impossible to prove. The default assumption is that a breach occurred.

But here's where the AI dimension complicates things: the vendor is the business associate, so they have the initial obligation to investigate and report the breach to you. Many AI vendors are startups or companies that have never dealt with a HIPAA breach before. They don't know the timelines. They don't know what counts as a breach. They notify you three weeks after discovery because they spent that time trying to figure out if it was "really a breach" or just a "configuration issue." By the time you're notified, you're already past the 60-day window for individual notifications.

What Your Incident Response Plan Should Cover

Your plan needs specific provisions for AI vendor breaches, including:

• Contractual requirements for breach notification timelines from business associates, with penalties for late reporting
• Defined ownership for investigating AI-related incidents, since they often span IT, compliance, legal, and clinical operations
• Pre-established criteria for what constitutes a "breach" versus an "incident" in AI contexts, especially for transient data processing
• Communication templates for notifying patients about AI tool breaches in plain language that doesn't require a technical background to understand

You also need to test the plan. Tabletop exercises should include AI scenarios. What happens if your diagnostic AI tool leaks patient imaging data? What if a chatbot logs conversations that include PHI and the database is compromised? These aren't hypothetical. They're scenarios that have already happened at other organizations.

Vendor Risk Management When the Vendor Is an AI Startup

Healthcare organizations are accustomed to vetting large, established software vendors. They know what to look for in a SOC 2 report. They know how to assess financial stability and business continuity plans. AI vendors—especially startups—don't fit that mold, and the standard vendor risk framework often misses the things that matter most.

A typical AI startup is two years old, venture-backed, burning cash, and pivoting its product every six months based on customer feedback. It has ten engineers, no dedicated compliance staff, and a security posture that's best described as "we use AWS and everything's encrypted." The company genuinely believes it's secure because it's using cloud infrastructure and has enabled the default security settings. What it doesn't have is a formal risk assessment, a tested incident response plan, or any experience handling PHI at scale.

When you ask for a SOC 2 report, they tell you it's "in progress" and offer a completed security questionnaire instead. When you ask about subprocessors, they list AWS and OpenAI but can't provide copies of the underlying data processing agreements. When you ask about business continuity, they say they have backups but haven't tested a restore in a production-scale failure scenario.

None of this means the vendor is acting in bad faith. It means they're a startup operating at a different level of maturity than your organization is accustomed to. The question is whether you can accept that risk, and if so, what controls you put in place to manage it.

Due Diligence That Actually Addresses AI-Specific Risks

Standard vendor questionnaires don't cut it for AI tools. You need to ask different questions:

• Model provenance: Where did the AI model come from? Was it trained on public data, proprietary data, customer data? If it was trained on customer data, was consent obtained and were BAAs in place?
• Model updates: How often is the model retrained or updated? Do those updates go through a validation process, or are they pushed to production automatically? Can you opt out of model updates if you need stability for a regulated environment?
• Data lineage: Can the vendor trace where patient data flows within their system? If you request deletion of a patient's data, can they confirm it's been removed from training sets, caches, logs, and backups?
• Third-party dependencies: What happens if OpenAI changes its API terms, or if a hyperscale cloud provider has an outage? Does the vendor have fallback options, or is your clinical workflow entirely dependent on a single external service?

These questions often reveal gaps the vendor hasn't thought about. That's useful information. If the vendor responds with curiosity and a plan to address the gaps, that's a sign of maturity. If they respond defensively or with vague assurances, that's a sign you're taking on more risk than you should.

The Compliance Program Implications of AI Deployment

Deploying AI tools in healthcare isn't just a procurement decision. It's a compliance program change. Your policies, training, auditing, and risk assessments all need to account for the new technology. Most organizations treat AI as "just another vendor" and don't update their compliance infrastructure accordingly. That's a mistake.

Your HIPAA policies need to explicitly address AI tools. What's the approval process for new AI deployments? Who evaluates whether a BAA is required? What's the acceptable use policy for generative AI in clinical settings? If your policy manual doesn't answer these questions, your staff will make their own decisions, and those decisions won't align with your compliance obligations.

Training programs need to evolve as well. Annual HIPAA training that covers "don't share passwords" and "lock your workstation" isn't sufficient when your staff are using AI scribes and diagnostic assistants. They need to understand what counts as PHI in an AI context, why consumer AI tools aren't HIPAA-compliant, and how to recognize when a new tool requires compliance review before adoption.

Audit procedures need to include AI-specific checks. Are you logging which staff are using which AI tools? Are you reviewing API calls to external AI services? Are you periodically testing whether your BAAs are still in force and cover the current use cases? If your audit plan doesn't include these items, you're not auditing your actual risk landscape.

Building AI Governance Into Your Compliance Program

The organizations that are getting this right are treating AI governance as a formal extension of their compliance program. They're establishing cross-functional teams that include IT, compliance, legal, and clinical leadership. They're creating approval workflows for new AI tools that require sign-off from all stakeholders before deployment. They're maintaining an inventory of AI tools in use, the BAAs in place, and the risk assessments that justified the deployment.

This isn't bureaucracy for its own sake. It's the recognition that AI introduces risks your existing compliance program wasn't designed to handle. If you don't adapt the program, you're trying to manage 2024 risks with 2014 controls. That gap is where violations happen.

For a broader look at how to structure AI governance across the enterprise, not just in healthcare, see What Is AI Governance? A Framework for Organizations Deploying AI.

What CISOs and Compliance Leaders Should Do This Quarter

If you're responsible for HIPAA AI compliance in a healthcare organization, you have work to do that can't wait for the next budget cycle or strategic planning session. Start with an inventory. Find out what AI tools are deployed, what's being piloted, and what your staff are using informally. You can't manage risk you don't know about.

Once you have the inventory, map each tool to a business associate agreement or document why one isn't required. If you can't produce a BAA for a tool that's processing PHI, that's a priority remediation. Either get the agreement in place or shut the tool down until you do. There's no middle ground on this.

Review your policies and training materials. If they don't mention AI, update them. Make sure your workforce knows what's approved and what isn't. Make sure they know how to request approval for new tools instead of deploying them in the shadow. Make the compliant path visible and accessible.

Finally, engage your leadership team. If your executives think HIPAA AI compliance is "handled" because you have BAAs on file, they're not seeing the risks this article has outlined. Your job is to show them the gaps in plain language and get their support for the resources needed to close them. This isn't a technical problem. It's a leadership and resource allocation problem, and it requires executive engagement to solve.

The healthcare organizations that will navigate AI deployment successfully aren't the ones with the most advanced technology. They're the ones that understood the compliance requirements before they deployed the technology and built their programs accordingly. That clarity is available to you right now. The question is whether you'll act on it before OCR forces the issue.