Drawing up test-and-learn criteria for Ask the NHS

Drawing up test-and-learn criteria for Ask the NHS In recent weeks the team has sharpened its focus on how we can launch a test-and-learn pilot for Ask the NHS within this financial year.

We will need to work quickly to hit that target so our first release will need to balance meeting essential user needs with rapid delivery. Those trade-offs have got me thinking about the fundamental questions we need to answer about this new capability. What are we testing? What can we learn?

The answers we need will shape what features we include in the first release. And the data we get will shape what we prioritise for future iterations and how we want to position this capability within the larger digital estate.

By offering a slimmed-down first release of Ask the NHS to a limited set of users we can address some big unknowns.

Will users engage with this service?

This is a bedrock question, answered by users voting with their clicks. So out of all the times users see Ask the NHS as an option, how often do users choose to enter a query? We expect a significant number of users will click on the service, have a look, but not go on to make an enquiry so we will need to consider what a “good” conversion rate might be.

We understand willingness to use Ask the NHS will be impacted by how much the public trusts the NHS to use AI technology safely. But it will also be a result of how they perceive this service in relation to the rest of their digital routes to NHS support and advice.

What is the range of questions people ask?

It is important for us to understand what users think Ask the NHS is for.

Is it useful for every digital transaction or a lifeline when they feel stuck? That is, will users treat Ask the NHS as a simple search bar, one they might even prefer over the information architecture laid out across, say, the NHS App or NHS.UK? Or will users feed Ask the NHS tricky questions about getting help in complex circumstances or from under-the-radar services? To answer this we will need to do some qualitative reviews and classification of user input.

If Ask the NHS is viewed as a tool for routine transactions, I would expect to see a handful of topics overwhelm all other types of queries.

For example, 70% of questions relate to new symptoms, GP appointments, and prescriptions with a long tail of other topics in the remaining 30%.

If users rely on Ask the NHS when their needs feel less straightforward, I would expect users to ask about a wide variety of topics with no one category claiming perhaps more than 20% of transactions

Will users understand the service offering and use it intuitively?

We anticipate users will enter requests that are too brief or ambiguous to classify.

From the beginning we plan to offer interactive steps that allow them to confirm, clarify or restate their query. We can count how often this happens to get a sense of how well we have framed the service proposition. But we should also do some qualitative investigation to identify common reasons for this.
We also anticipate some people will want to play about with Ask the NHS and test its limits, for reasons both good and ill.

There will be guardrails in place to stop queries that are malicious, erroneous or abusive, and we should monitor how often those safety measures are needed.
The more we know about these challenges the easier it will be to plan future iterations that can accommodate unexpected user behaviours.

Will users trust our recommendations?

This capability needs to be helpful and accurate, and users need to feel they receive a considered response to their question. Already in prototype testing we have seen positive feedback from interviewees when the service offers a bit of reasoning as to why, for example, they should they contact their GP surgery. When Ask the NHS recommends a course of action, how often will users click through to try it?

Are we meeting user expectations for AI-powered assistance?

First release won’t be able to offer a very joined-up user experience. It will signpost to helpful service, but Ask the NHS will not have access to users’ demographic profiles and medical histories. We might not even know the user’s NHS number.

This means Ask the NHS won’t be able to check instantly their eligibility, for example, to receive a flu vaccination on the NHS or to visit a pharmacy for antibiotics to treat a UTI. And it won’t offer agentic services like cancelling an appointment or renewing a prescription on a user’s behalf.

We will need to collect qualitative feedback so users can say if they were hoping for a more personalised or agentic experience. This can be done by offering on-page feedback forms.

Will Ask the NHS handle queries safely?

Ask the NHS will be built with safety features to support people in concerning circumstances. This includes users who may have life-threatening symptoms, are suffering a mental health crisis, or need safeguarding help. We intend to offer immediate exit routes for users in urgent need. We need to collect data on how often this happens, and we need to review regularly a selection of enquiries, especially in the test-and-learn phase, to ensure Ask the NHS is not missing chance to support users at higher risk of harm.

What is the cost per transaction?

Even if Ask the NHS meets every user expectations and performs safely, the service needs to make prudent use of public money. A test-and-learn offering will help us judge whether we can feasibly run and maintain of an open-access AI-enabled service. Transaction costs from the pilot period will provide a baseline for measuring future savings that might be achievable after a wider roll-out, for example, by scaling up the service’s infrastructure or reserving larger amounts of compute in advance.

The answers to these questions will shape our decisions about whether to offer this capability more widely, which users to target, and how we ought to iterate our roadmap to meet their most pressing or beneficial needs.