Background

Many studies in health sciences research rely on collecting participant-reported outcomes. Although some of these are participant reports of factual information, such as adherence to drug regimes, that could be objectively validated, there is an increasing recognition of the importance of subjective measures such as attitude to, and perceptions of, health and services provision. Alongside the exponential increase in health-related literature devoted to participant-reported outcomes, attention is being paid to the method or mode of data collection. Much of this has been driven by the rapid development of new technologies, which can lead to increased ease, speed and efficiency of data capture alongside an increasing drive for maximising response rates. Survey methodologies (e.g. in the business, marketing, social and political sciences) have a literature base of their own, covering theory to practice, much of which has been only slowly recognised in the health arena. Few health-related outcome development papers indicate a theoretical approach to eliciting survey response and the focus in choosing a mode for a study is often based predominantly on improving response rates and minimising cost. The impact on the validity of response is not generally a consideration. In addition to this, in order to gain as complete a data set as possible, many studies are using multiple modes either to enhance participants’ choice (e.g. opting for web- or paper-based surveys) or to improve follow-up rates (e.g. non-responders getting telephone data collection). Although for practical reasons these choices are entirely justifiable, consideration needs to be given to the validity of response via different modes and the impact that the choice of mode or modes might have on the conclusions from a study.

Objectives

• To provide an overview of the theoretical models of survey response and how they relate to health research.
• To review all studies comparing two modes of administration for subjective outcomes and assess the impact of mode of administration on response quality.
• To explore the impact of findings for key identified health-related measures.
• To create an accessible resource for health science researchers, which will advise on the impact of the selection of different modes of data collection on response.
• To inform the analysis of multimode studies.

Methods

In order to inform the systematic review of mode comparison studies, a review of the theoretical models and how they relate to the health domain was undertaken. This clarified the need to focus on features of mode rather than crude modes per se in order to understand the way in which responses to subjective outcomes could be affected. From this, a theoretical model based on Tourangeau was proposed with four main features: administration (interviewer or self), use of the telephone, use of the computer and sensory stimuli (audio, visual or both). Additional features were proposed that may belong in a model of response as well as potential mediating factors, such as cognitive challenge of questions. This approach was used to define the data extraction and coding classifications for studies.

Owing to the large body of literature relating to survey methodology which is published outside the health research arena, all studies that incorporate a mode comparison were included, regardless of setting. This led to a broad search strategy covering a wide range of disciplines. In order to target methodological studies, some innovations in search strategy that separate out the process from traditional reviews of the effectiveness of interventions were undertaken.

Results

The search strategy identified 39,253 unique references, of which 2156 were considered as full papers. Of these, 597 progressed to data extraction, with 381 finally included in the review. The most common reason (44%) for exclusion once the full paper was considered was that there was no actual mode comparison in the study. The majority of included studies were from North America (62%), with only 10% being from the UK.

Study designs were relatively evenly divided into between- and within-person studies (52% and 47%, respectively), with only 39% using some form of randomisation (random allocation for between-person studies and random ordering for within-person studies). In terms of quality assessment, most studies described their hypotheses and study design well, and drew appropriate conclusions (89%, 83% and 81% – good, respectively), but the description of participants, group allocation, potential impact of timing of data collection and presenting of variances was less good (22%, 50%, 27% and 35% – poor, respectively).

The 381 studies provided descriptions on 1282 outcome measures, of which 57% were health related. The most frequently reported outcomes were the SF-36 (17 studies) and the MMPI (9 studies). Thirty per cent of studies considered only a single outcome in their mode comparison, but most considered more (ranging from 1 to 21 outcomes). These studies also described a number of mode comparisons, giving in total 1522 comparisons between modes on multiple outcomes for analysis. Of these, 977 reported enough data to be included in the analysis of absolute mean differences, 910 in the analysis of the ratio of variances and 912 in the analysis of the ES.

Two features of mode were clearly associated with bias in response; however, none of the features of mode was associated with changes in precision. How the measure was administered, by an interviewer or by the person themselves, was highly significantly associated with bias (p < 0.001). A difference in sensory stimuli was also significant (p = 0.03). When both of these were present the average overall bias was < 1 point on a percentage scale. In terms of mediating factors, there was some suggestion that there was an interaction between both telephone and computer for data collection and date of publication, supporting the theory that differences disappear as new technologies become commonplace. Single-item measures were also related to greater degrees of bias than multi-item scales (p = 0.01).

Individual analysis of the SF-36 and MMPI showed a varied pattern across the different subscales, with conflicting results between the two types of study. None of the MMPI measures used to detect deviant responding showed a relationship with the mode features tested. The limits of agreement analysis showed how variable measures were between modes at an individual rather than at a group mean level.

Conclusions

Funding

Funding for this study was provided by the Health Technology Assessment programme of the National Institute for Health Research.

Publication

• Hood K, Robling M, Ingledew D, Gillespie D, Greene G, Ivins R, et al. Mode of data elicitation, acquisition and response to surveys: a systematic review. Health Technol Assess 2012;16(27). [PubMed: 22640750]

NIHR Health Technology Assessment programme

The Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’.

The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, from the public and consumer groups and from professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA programme then commissions the research by competitive tender.

Second, the HTA programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA programme in April 2007 under the newly established NIHR Methodology Panel. The HTA programme project number is 06/91/07. The contractual start date was in June 2005. The draft report began editorial review in September 2010 and was accepted for publication in March 2011. The commissioning brief was devised by the NCCRM who specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley CBE

Series Editors: Dr Martin Ashton-Key, Professor Aileen Clarke, Dr Peter Davidson, Dr Tom Marshall, Professor John Powell, Dr Rob Riemsma and Professor Ken Stein