Opening hours: Are the NHS asking patients the right questions?

Last month saw headlines in the media about the 'vast majority of patients' being satisfied with their GP's opening hours. Sceptical? MHF website editor and self-confessed stats cynic Jim Pollard looks at the facts behind the headlines.

The headlines on the BBC (right) and elsewhere reporting widespread satisfaction with GP opening hours and other aspects of the service followed publication of the GP Patient Survey (GPPS) conducted by the Department of Health. Some interpreted the results as proof that GP practices are open at times to suit the vast majority of patients.

Spin or our old friend 'damned lies and statistics'?

Either way, if you find this figure hard to credit, our experience at the MHF suggests you're right.

Time and again surveys on our health information website malehealth.co.uk have seen the vast majority of male respondants express dissatisfaction with GP opening hours. The anecdotal evidence is equally strong.

Even the DH's own sources suggest that this supposed satisfaction is a statistical anomaly rather than a true picture. After all, have those trumpeting the findings already forgotten the DH-commissioned Healthcare Commission (HC) survey from January? We certainly would have done had it not been for the elephantine memory and eternal vigilance of our policy officer David Wilkins. In short, the HC survey gives a very different impression.

In the HC survey, based on an admittedly smaller but still large sample, 25% of people said specifically that inconvenient opening hours had put them off going to the GP within the preceding 12 months. That's one in four. Furthermore, only 28% of people said that additional opening hours were not necessary.

So why the disparity?

These two surveys make an interesting case-study in the sort of scrutiny to which we should be subjecting survey findings. The resulting headline may be less interesting but the picture will be more complete.

The GPPS carried out by Ipsos MORI survey appears to have actually been two simultaneous surveys. One of these surveys was not postal or online but completed by the GP with their patients. Moreover, both surveys within the GPPS chose, or were weighted to cover, only patients who had made an appointment in the previous six months.

People sitting with their GP are less likely to be critical while those who have had a recent appointment are demonstrably less inconvenienced by opening hours than the many thousands who didn't make an appointment.

Don Redding, head of communications at the Picker Institute which carried out the HC survey, says: 'In general the results differ because the surveys themselves differ. The questions are worded differently, there is a big difference in the number of response options, and the base for analysis is different. The access questions in the GPPS survey are based on those who made an appointment with a GP within the 6 months prior to the survey, whereas our sample was based on anyone registered with the practice. Not surprisingly this gives a massive disparity.'

Specifically, the two questions on opening hours could hardly have been more different and still be about the same subject.

The GPPS asked 'Over the last 6 months or so, were you satisfied with the hours your GP surgery was open?' There were just two options: 'Yes I was satisfied with the opening hours' or 'No, I was dissatisfied with the opening hours'. In this context, it is less surprising that 84% of people said they were satisfied.

By contrast, the HC survey asked: 'In the last 12 months, have you ever been put off going to your GP practice/ health centre because the opening times are inconvenient for you?' There were three possible responses: Yes, often; Yes, sometimes; and No. As we have seen, in response, 25% stated that in the last 12 months they had been put off because they found opening hours inconvenient.

Don Redding explains the differences: 'GPPS asks respondents to generalise their satisfaction with the opening hours over the last 6 months. The HC question asks more specifically about experience — have they EVER been put off from going. The HC response options discriminate between people more. Those 18% who said "yes, sometimes" could have said they were satisfied with the opening hours in the GPPS survey.'

There was a similar gulf with other questions. Take, for example, the GPPS finding: 86% of people reported that they were satisfied with their ability to get through to their doctor's surgery on the phone. This contrasts starkly with the HC survey in which 58% reported difficulty contacting the local practice/health centre by phone.

The GPPS question was 'In general, are you satisfied with how easy it is to get through to someone on the phone at your doctor's surgery?' The HC question was 'Have you had a problem getting through to your GP practice/ health centre on the phone?'

Do Redding says: 'The questions are asking two conceptually different things — the GPPS question is asking for patients to generalise and to rate their satisfaction with getting through on the phone, where as our question asks more specifically about the patient's experience of having a problem getting through. Our methodology at Picker is to always use reports of actual experience. We consider satisfaction questions as less precise and less useful.'

As far as opening hours are concerned, both surveys share the shortcoming of only asking people who were registered with a GP and still living at their registered address. What about those who are not registered? Surely one of the reasons for this might be the inconvenience of the opening hours?

A parliamentary answer in 2003 revealed how little data is held about how many people are not registered with a GP.

The then secretary of state John Hutton said: 'The vast majority of the population is registered with a general practitioner' but admitted that registration figures are actually 'greater than population figures. Therefore, it is not possible to express accurately the percentages of the population that are covered. This is due to "list inflation".'

Talk to A&E doctors, or those working with the homeless or refugees and we can deduce that many thousands if not millions are unregistered.

As MHF CEO Peter Baker said in his letter to the BMJ on this subject: 'Relying just on the last set of statistics is always an unwise path to tread'. Arguably, relying on any set of statistics is unwise if their methodology has not been scrutinised.