How to Build an AI Medical Scribe: Ambient Capture, Architecture & FHIR

The pipeline has to record an encounter responsibly and keep the clinician in charge. It also has to deliver structured data the Electronic Health Record (EHR) can use. Your users need an accurate clinical note, and a fast, wrong note costs more than no note. When the pipeline cannot produce clean output, it leaves you with integration debt.

According to a study in the American Journal of Managed Care, by June 2025, 62.6% of Epic-using US hospitals had adopted ambient Artificial Intelligence (AI). As ambient AI technology spreads across health systems, your product can differentiate itself through its architecture. 

This guide covers the four parts of an AI medical scribe: capture, structuring, human review, and FHIR output. We’ll walk you through the main intricacies of building a medical scribe that meets HealthTech requirements, including healthcare data interoperability.

Key Takeaways

• An AI scribe is a pipeline of four layers, and each layer fails in its own way.

• The AI drafts the note, and a clinician approves it. The whole design rests on that approval.

• Health systems buy on clinician time, patient attention, and note quality, so design each layer against one of them.

• Capture is the most sensitive layer. Settle consent, retention, and access before you write code.

• Plan the FHIR mapping and EHR write-back from the first sprint.

What Do Health Systems Expect From Ambient AI Technology?

Your buyers judge an ambient AI scribe by what it changes in the clinic, so design against their list. Each expectation below maps to a layer in the architecture that follows.

Less charting and less stress

Physicians and other providers want less time on charting and fewer evenings staying late to finish notes.

They also want less stress from a documentation burden that grows as workforce shortages continue. Easing that administrative burden is what a scribe is sold to do.

Ambient documentation also aims to cut after-hours charting and lower the mental load of manual note-taking. It should make documentation more consistent, too.

More attention on the patient

Many clinicians spend hours on charting after the clinic closes, and they would rather spend that time on patient care. They also want to look at the patient instead of the computer.

Eye contact and a present, focused conversation are what a screen tends to take away from an appointment, and patients notice. Patient experience and patient engagement improve when doctors are talking with the patient instead of typing.

What health systems add

Health systems add two requirements of their own. Documentation quality has to hold up across specialties, from psychiatry to internal medicine, with the same consistency from one visit to the next.

The note also has to support coding and revenue integrity, which means it reflects what happened in the encounter and nothing else.

What the evidence shows

The evidence supports modest, real gains. A JAMA study across five academic medical centers found that ambient scribes cut documentation time by 16.0 minutes and total EHR time by 13.4 minutes. According to the American Hospital Association, ambient AI was also linked to 0.49 more clinician visits per week.

Cleveland Clinic’s AI scribe documented 1 million patient encounters in its preliminary results. It cut time spent writing and reviewing EHR notes by two minutes per appointment and 14 minutes per day. The average time saved is modest, so note quality and review time matter as much as speed.

Looking for a reliable AI development partner?

We are happy to share our expertise

Let's talk

What Does an Ambient Clinical Documentation Architecture Look Like?

An ambient clinical documentation architecture is a pipeline, not a single model. While the AI listens, audio capture feeds speech-to-text.

A Large Language Model (LLM) then structures the transcript, and a clinician reviews the draft. The approved note leaves as Fast Healthcare Interoperability Resources (FHIR). The rest of this guide follows those layers in order.

Automatic Speech Recognition (ASR) turns audio into text, and every later layer works from that text, so its errors travel with the note. Look at where review sits: between the model and the record. Nothing the model writes reaches the chart without a person approving it.

How Does Ambient Capture Work in a Clinical Setting?

Ambient capture clinical documentation systems record an encounter passively instead of waiting for the clinician to dictate. That raises the bar on consent, transparency, and what audio you keep after transcription. In an AI medical scribe, ambient documentation starts with raw audio, which holds Protected Health Information (PHI) in its most direct form.

Getting awareness before recording

The patient and the clinician should both know recording is about to start. Patients tend to want this.

According to a UC Davis Health survey of nearly 1,900 patients, most wanted to hear about AI use at booking, arrival, or check-in. Fifty-seven percent wanted to be told face to face.

Build the notice into the workflow, such as a prompt on the clinician's screen or a line at check-in. That way it does not depend on memory.

The same survey shows what worries patients. Note accuracy came first at 39%, ahead of privacy and security (13%) and being recorded (13%). Your review step answers the top concern.

What to keep and what to discard

Decide early whether you keep raw audio, keep only the transcript, or keep neither past a short window. Each choice changes your storage, access controls, and exposure.

Keeping less is easier to defend. If you do keep audio, give it a retention period and a deletion job that runs on its own.

Multiple speakers and clinical terminology

Exam rooms are noisy. You have a clinician, a patient, sometimes a relative, and background sound. Test your ASR on audio that resembles your real settings, with the drug names, abbreviations, accents, and languages your users produce.

Speaker labels matter too. The structuring step has to know whether the patient described a symptom or the clinician gave an instruction.

Natural language processing then filters casual conversation from the critical medical details. The draft keeps the comprehensive details of the encounter and drops the small talk.

Consent and privacy checklist

What a compliant architecture needs to account for at the capture layer:

• Patient and clinician are both informed before recording starts

• A clear policy exists for how long raw audio is retained, if at all

• Access to raw audio and transcripts is limited and logged

• The patient can decline ambient capture without losing access to care

How Do You Structure a Clinical Note with an LLM?

Structuring means prompting or fine-tuning an LLM to turn a raw transcript into a note format clinicians already use, such as SOAP (Subjective, Objective, Assessment, Plan). Every statement must stay traceable to something said in the encounter. The model organizes what was said and does not interpret it.

Matching the format clinicians expect

Clinicians know where things belong in a note, so a draft in the format they already use is quicker to check and easier to trust. Documentation needs differ by specialty, so make the format configurable. Pass the clinical context of the visit, such as the specialty and the reason for it, so abbreviations resolve correctly.

Keeping the draft traceable

Store the transcript timestamps next to each generated statement. A reviewer can then verify a claim in one click and check a note in seconds. You can also measure how often the model adds something nobody said.

Give the model a rule for gaps too. If allergies never came up, the field stays empty and reads "not discussed."

A model asked for a complete note will fill blanks with plausible text. Design the prompt and output schema to accept "not stated."

Where structuring goes wrong

The usual failures are an invented detail, a dropped negative ("no chest pain" turning into "chest pain"), a misattributed speaker, and a wrong dose or unit. Each reads fluently, which is how they slip past a tired reviewer, and they get caught when the screen shows the source text beside the draft. 

Our LLM engineering work follows the same rule: human-in-the-loop from the first sprint, with prompt design, retrieval, and injection hardening treated as engineering.

Why Does the Human Review Step Matter So Much?

Human review keeps an ambient AI scribe assistive instead of autonomous. The clinician reads, edits, and approves every note before it becomes part of the patient's medical record, and that approval is not optional. The scribe transcribes and structures what was said, and the clinician decides what goes in the chart.

Build this into the data model as well as the interface. A note has a draft state and an approved state, and only an approved note can leave your system. No background job, retry, or integration should be able to push a draft to the EHR.

What a good review screen needs

• Easy to scan. Short sections in the clinician's usual format, with medications, doses, and plan items easy to find.

• Easy to verify. One click from any statement to the transcript passage behind it, and a marker on anything the model could not tie to the transcript.

• Editable in place. Corrections should take seconds, and your system should keep them. The gap between draft and approved note is the best measure of documentation quality.

• A deliberate approval action. One explicit step, tied to the clinician's identity and timestamped.

What happens when review is rushed

Reviewing notes takes real attention, so review fails quietly. If drafts run long or approval is a single "accept all" button, clinicians start approving without reading, and the tool becomes autonomous in practice. Watch review time and edit rates in production, because a review that takes two seconds on every note is a warning sign.

Ambient Clinical Documentation and FHIR: How Does the Note Reach the EHR?

Once a note is approved, it has to be mapped into standard FHIR resources so the electronic health record and downstream systems can use it. Most scribe software integrates with the EHR for exactly this reason.

The quality of that integration is a significant part of the difference between a demo and a product. A well-written note that stays in free text recreates the problem the scribe was meant to solve.

Mapping an approved note to FHIR resources

A workable starting pattern uses a small set of resources:

• DocumentReference holds the note and its metadata: patient, encounter, author, and status. Use a preliminary status for drafts and final for the approved version.

• Composition carries the note's sections, such as the SOAP parts, as discrete data instead of one block of text.

• Provenance records what produced the note: the software that drafted it and the clinician who approved it.

• Observation entries can hold values the clinician stated and confirmed during the visit, such as measurements. Create them only from confirmed content, never from model inference.

Keep the mapping deterministic. The LLM writes the draft, but turning an approved note into FHIR should be ordinary, testable code that gives the same output for the same input.

Where this connects to the EHR

What an EHR accepts, and through which route, varies by vendor. Confirm early what your target systems support for write-back, because the answer can reshape the design.

A FHIR-native backend simplifies the middle of the pipeline. Our Medplum development work uses the open-source platform's resource store, access policies, and audit events, and its Bots can react when an approved note lands. Your team is then free to build what is specific to your product.

What Mistakes Do Teams Make Building an AI Medical Scribe?

Most problems trace back to a few early choices that are cheap to avoid at design time and costly to fix after launch.

• Weakening the review step. Auto-approval, "accept all" buttons, and background syncs that push drafts early turn an assistive tool into an autonomous one.

• Treating ASR accuracy as good enough. A demo on clean audio proves little. Test on real clinical speech, because errors here feed every later layer.

• Planning the note but not the delivery. Teams polish the draft, then learn the target EHR takes a different resource or none at all. Prototype write-back while you tune prompts.

• Keeping audio by default. Recordings pile up as sensitive data with no clear purpose. Keep what you can justify, for as long as you can justify it.

• Relying on the prompt as the safety control. A line saying "do not invent details" is a request. The controls are structural: traceability, empty fields for unstated information, and a review step nobody can bypass.

How TechMagic Approaches This

We have not shipped a complete ambient scribe, so we will not claim one. Three projects cover the parts of this work. Documentation is also a common first step in AI automation in healthcare, and the same human-in-the-loop rules apply to the AI scribes and workflows that follow it.

An LLM assistant on proprietary data

For Elements.cloud, we built ElementsGPT, an AI guide that turns a plain-language process description into diagrams, user stories, and implementation advice based on the client's own data. The build dealt with limited model context, grounding answers through a vector database, and hardening prompts against injection, all of which a scribe faces too. It is not a healthcare product.

One integration layer for many EHRs

A US HealthTech company captures patient intake and referral data and delivers it into each customer's EMR. We built the integration layer behind that delivery, with FHIR R4, HL7v2, and proprietary REST clients behind one interface and per-client settings stored as data.

The product now connects to 10+ EMRs from one codebase, and every delivery is logged and traceable. That is the write-back problem a scribe meets at its last step. The project had no AI layer.

A FHIR-native, HIPAA-ready platform

For a digital health startup focused on early cancer detection, we used Medplum as the FHIR foundation. The result was a HIPAA-ready platform with role-based access for patients, physicians, and administrators.

It had no AI component either, but it shows the foundation any scribe sits on. The compliance obligation stays with you, and our part is building software that supports it.

Preventive care platform built with Medplum: Read our case study

HIPAA-compliant app built with Medplum to support early cancer detection

Read more

Wrapping Up

Building an AI medical scribe well means treating capture, structuring, human review, and FHIR output as four separate engineering problems, and never letting the review step become optional. The clinician approves the note, and every other part of the design supports that while keeping attention on patient care.

Where the market is heading

In a recent Deloitte survey, more than 80% of health system executives said they are prioritizing agentic Artificial Intelligence for clinical operations and care delivery. The same analysis says safety has to be built in from the start. It adds that AI works best when it fits existing clinical workflows without adding steps.

As the future of this market takes shape, expect buyers at health systems to ask for a review trail they can inspect. They will also want notes that land in the chart without extra clicks.

Where to start

Pick one specialty, one capture setting, and one EHR target. Build the whole pipeline for that slice, including review and write-back, before you widen it.

Planning an ambient scribe and want a second opinion on the architecture?

We're here to help

Healthcare AI agent development

FAQ

What is an AI medical scribe?

An AI medical scribe is software that listens to a clinical encounter, turns the audio into text, and drafts a clinical note for the clinician to review. It assists with documentation and does not diagnose or finalize anything on its own.

What is ambient clinical documentation?

Ambient clinical documentation is the practice of capturing a patient encounter passively, without the clinician dictating, and producing a draft note from it. The clinician reviews and approves that draft before it joins the record.

How does an AI scribe turn a conversation into a clinical note?

A speech-to-text layer converts the audio to a transcript. An LLM then organizes the transcript into a standard format such as SOAP, keeping each statement tied to what was said. A clinician reviews the draft and approves it.

Does an AI medical scribe replace the clinician's review?

No. The clinician reads, edits, and approves every note before it becomes part of the record.

How does an AI-generated note get into the EHR in FHIR format?

After approval, the note is mapped to FHIR resources such as DocumentReference and Composition, with Provenance recording who drafted and approved it. The resources reach the EHR through its FHIR API or a write-back integration.

How does TechMagic approach building an AI medical scribe differently?

We design for human-in-the-loop from the start, so the clinician stays in control of every note. We pair LLM engineering experience with FHIR-native delivery, and we build the pipeline to support your compliance work without claiming to certify it.