My View - Building AI you can trust

VIEW - Kate Rhind is a Founding Fellow of HiNZ and founder and chief executive of Healthpoint 

Ask any New Zealander who has tried to find urgent care for a sick child on a Sunday, and they will tell you the hard part is rarely the treatment. It is knowing where to go for help. 

Our health system is full of the right services. They are simply hard to find. Consider something as ordinary as “finding a GP”. Healthpoint profiles at least ten distinct types of general practice and urgent care: standard general practice, after-hours practice, accident and urgent medical care, nurse-led and paramedic-led practices, youth and tertiary student health, virtual care, aged residential care, and armed forces clinics. 

From the outside they look alike, but each serves a different population, under different rules.

A perfect problem for AI

Knowing what kind of service can help is only the first step. The questions that actually decide whether someone gets care are practical and personal: who is open right now? Who will see me, given the access criteria? What will it cost, and do I qualify for a lower fee? 

As the Minister for Mental Health has noted, not knowing where to go or who to talk to is the reported reason for unmet mental health needs in one third of children and more than a quarter of adults. 

When people cannot find the door, they wait, they give up, or they arrive
in the wrong part of the system. Cost, distance, language, literacy and disability stack the barriers higher.

This is the unmet need - not a shortage of services, but a shortage of navigation.

Navigation across a complex, forever-changing system is exactly the kind of problem AI appears built for. 

Type what you need in plain words; receive a clear answer, but in health, a useful answer is only trustworthy when it is grounded in verified data, shaped by domain expertise, and constrained by careful design. 

That is my central argument to colleagues in health informatics: what looks easy is not.

The pillars of good AI

Good AI over a health system rests on three pillars: take away any one of them, and the whole structure collapses.

The first is the data. Healthpoint operates New Zealand's national health services directory: more than 12,000 verified public, private and NGO services, authored by 8,500 service editors who own their own records, and changing constantly, between 100 and 400 updates a day. 

This trusted foundation is only possible because thousands of health and social
service organisations across Aotearoa maintain accurate, up-to-date information and
contribute to the shared digital health ecosystem. 

Ask Healthpoint is New Zealand's first public-facing health AI to draw on international data standards, FHIR and SNOMED CT, so that clinical meaning is exchanged precisely rather than guessed at.

The second is domain expertise. Twenty-two years have taught us that the real world is messy: eligibility, referral pathways, cost to access. Data alone does not capture that.

Understanding how the sector fits together is what lets us make the directory usable, keep it current, and connect people with confidence. Crucially, it is what lets us know what good navigation looks like.

The third is carefully controlled and implemented AI. We use AI narrowly, to understand what a person means, and then return real services through conventional queries against
verified data. 

That design choice deliberately sidesteps the hallucinations generative models
produce. 

Ask Healthpoint does not diagnose or triage. It holds no personal information.
Neither directory data nor conversations are ever used to train the underlying model.

Knowing what good looks like

I want to dwell on why that third pillar demands the first two, because this is where the sector is most exposed. 

AI is a confident performer, producing answers that sound fluent, plausible and authoritative, and be wrong. 

Ask a general model where to find a needle exchange and it offers generic advice and a helpline number. Ask Healthpoint and it names the pharmacy, the opening hours, that it is a free service, and that no identification is required. 

The difference is not cleverness. It is verified data and disciplined constraint.
You cannot govern what you cannot recognise.

If you do not know what good looks like, you cannot tell when an AI is quietly leading people astray. Domain expertise is therefore not a nice-to-have wrapped around the technology; it is the thing that lets you judge the technology at all.

So we built to constrain, not to impress. Over nearly twelve months we ran 25 rounds of testing across 600 real-world scenarios, and had the tool independently security-tested the way a real attacker would probe it. 

Humans remain in the loop, reviewing and escalating.

Enabling the consumer experience 

None of this is visible to the person who types “I have a sore ear” and gets a useful answer and that is the point. The ease is earned; behind it sits governance, expertise and restraint.

My caution to the sector is simple. The barrier to deploying AI in health has never been lower, and the temptation to move fast never greater, but plausible is not the same as correct, and in health the consequences of that gap fall on real people. 

Before we trust AI to guide people through the system, we should be able to say, precisely, what good looks like, and prove our tools meet it.

If you want to contact eHealthNews.nz regarding this View, please email the editor Rebecca McBeth.

Read more VIEWS

Return to eHealthNews.nz home page