Before anyone sold one for care: designing an AI scribe that's safe to trust.

When someone starts receiving home care, a supervisor visits them for an initial assessment, which typically takes around an hour and a half in their own home and covers their health, abilities, routines, preferences and the risks in their home. Everything gathered in that conversation then has to be typed into a highly structured digital care plan, with dozens of sections, conditional questions and risk assessments. Writing one took at least five hours, often spread over several days, which created a risk that care could begin before a plan was in place. Supervisors saw it as the most tedious part of their job and the biggest drain on their time.

I was the senior product manager responsible for care-management products, including digital care plans. In late 2022, within weeks of ChatGPT's release, I set out to test whether a supervisor could simply hold the conversation, record it with consent, and have AI draft the care plan as they went, ready for them to review. The aim was to more than halve the time it took to write a care plan, which would save money, remove one of the most frustrating parts of the role and reduce the risk of care starting without a plan, without making plans any less complete or safe.

The difficulty came from three directions at once, and only one of them was technical.

The models weren't ready, or at least not publicly. The options in late 2022 were narrow, and the only one good enough was GPT-4, which wasn't released to the public until March 2023.

Real assessments are messy. There are often several people in the room, clients can struggle to answer, meaning is frequently implied rather than said, and much of what matters is visible to everyone present and therefore never said aloud.

The governance stakes were high. We would be recording vulnerable people in their own homes, often with relatives present and sometimes in homes with no phone signal; moreover, the more fluent a draft care plan became, the greater the risk that a busy supervisor would trust it more than it deserved.

• I secured a capable model early, without cutting corners on data. Having found nothing else that could do the job, I negotiated early access to GPT-4 ahead of its public launch in March 2023, and set up a private, gated environment to run it in, so recordings of real assessments never went anywhere they shouldn't. Consent was captured when each assessment was booked and confirmed again at the start of the visit.
• I proved it by hand before asking anyone to build it. I turned the care plan into a structured schema with a short instruction for each field describing what good looked like, wrote a mock assessment for a fictional client, and compared models against it. I then ran every stage myself on recordings of real assessments, and only once the approach worked end to end did I bring in an engineer to automate it. Engineering time went on something already shown to work, and I understood every failure first-hand.
• I went into people's homes and listened, and what I heard changed the design more than any model did. In one assessment a client struggled to say what she liked for her tea, and the supervisor, kindly helping her along, offered "Me, I like to eat…"; a model could easily have recorded the supervisor's answer as the client's. Relatives answered on people's behalf too, so alongside the technical work on who was speaking, we planned to train supervisors to open each recording by stating who was present and in what role. Supervisors also referred to things everyone could see but nobody needed to say. "Does she use pads?" wasn't recognised as a question about continence needs. Early drafts also filled in answers the conversation never gave; in a care plan, "no", "not mentioned" and "the model guessed" are three different answers, and the design had to keep them apart.
• I designed for the right amount of trust, not the most. A fluent draft invites people to stop reading, so supervisors would review the AI's suggestions field by field inside the care-planning system, deliberately accepting, rejecting or editing each one, with its source shown alongside: the transcript, the quality team's guidance or the client's existing records. Because the review happened inside the system of record, its own rules still applied, and no risk assessment that an answer should trigger could be skipped. Human oversight became a designed check rather than a person somewhere in the process.
• I sequenced the work by risk. Risk assessments carried the most risk, so the first phase would test the review mechanism using suggestions written by the quality team rather than generated by AI: someone using an emollient cream, for example, should prompt a fire-risk assessment with a standard set of mitigations, each editable to fit the person and their home. We would prove that supervisors reviewed suggestions properly before letting a model make them.
• I brought in the people who'd live with the risks before any code was written. In May 2023 I ran a design sprint with a head of UX, a tech lead and a care expert, followed by two pre-mortems: one with engineers and data scientists, and one with clinical, quality, branch and health-and-safety colleagues. What they raised went into the refined proposal I put forward in July 2023.
• I did the zero-to-one work myself, so my team could stay focused. I was responsible for several products with one team, and that team was busy on medication management. Rather than pull them off it, I ran the discovery, wrote the schema and prompts and ran the manual pipeline myself, drawing on conversations with the head of UX, and brought in a contract engineer with some AI experience for one day a week in June and July 2023 to automate what I'd proved by hand. The roadmap kept moving while the scribe went from idea to tested design.
• I made the case for people when priorities collided. The plan was for my team to move onto the scribe once the proof of concept was done, but the medication-management product needed them for longer than planned. I put together a business case for three extra hires so both could go ahead; it was financially sound, but the company was in a hiring freeze and it was rejected, so in autumn 2023 the scribe was parked.

I'd have fought much harder for that business case. Frankly, letting the scribe be parked was a huge mistake, and one I sincerely regret. I don't regret keeping my team on the medication work: it was a clinical product, and safety required it. However, the case was financially sound, and three salaries were very little against what was at stake. I think that if we'd built it then, we'd have been ahead of a market that has since become huge.

This was a prototype and a designed workflow, not a shipped product: by mid-2023, a complete, schema-driven design for AI care-plan drafting with deliberate human review at its centre, proven by hand on real assessments about two and a half years before NHS England published guidance on AI-enabled ambient scribing in April 2025.

When ambient scribes later became products you could buy, recording, transcribing and drafting were exactly what they did well. The parts I'd spent most of my time designing still largely weren't available out of the box: filling the care-planning system's own fields with its rules enforced, deliberate review of each suggestion, a visible source for every statement, and a risk-first start.

That gap is the lesson I took from it and still apply: the model is the easiest part of an AI product and the first to be commoditised, while the value, and most of the risk, sits in the workflow around it. It shaped how I later rolled out AI report drafting inside clinical governance. How I built the prototype, and what has changed technically since, is written up in my lab.