Collecting patient-reported outcome (PRO) data is an integral function of clinical research, providing the patient perspective that enriches the evaluation of treatment effectiveness. Historically, patient-reported outcomes measurement (PROM) was performed predominantly with paper-based assessments. However, over the past two decades, there has been a notable paradigm shift towards the electronic collection of data.
Electronic data capture typically relies on devices equipped with pre-programmed software applications. They are distributed to study participants for use during a trial and subsequently returned to the study site upon trial completion.
In recent years, with the proliferation of personal smartphones, there has been a growing interest in having participants utilize their own smartphones or tablets to complete assessments in clinical studies. This practice is commonly referred to as “bring your own device” or BYOD. It is worth noting that, until recently, industry groups expressed concerns regarding proving the equivalence of various devices for data collection, which has posed challenges to the widespread adoption of BYOD.
This article delves into the journey of “mixed-modes” data collection from the prior emphasis on rigorous comparability testing to the adoption of an evidence-based approach, which recognizes the comparability of devices and paves the way to BYOD.
The Early Years: Establishing a Foundation
As the collection of PRO data became commonplace in clinical trials, the Food and Drug Administration (FDA) in 2009 developed guidance on how it would review and evaluate existing, modified, or newly-created PRO instruments (questionnaires) used to support claims in approved medical product labeling.
Part of this guidance called for sponsors to provide evidence to confirm the equivalence of an instrument when changing from a paper to electronic format, but the guidance was vague and didn’t specify the evidence needed to prove the change was acceptable.
As a result, a task force created by The Professional Society for Health Economics and Outcomes Research—better known as ISPOR—published recommendations addressing the implications of the FDA’s draft guidance for the collection of PRO data using electronic data capture modes of administration (ePRO). The ISPOR recommendations noted that migrating from paper to electronic PRO administration was considered a “minor” change and testing to show equivalence should be performed with a cognitive debrief (also referred to as a cognitive interview) and usability testing. Cognitive debriefing aims to assess the patient’s understanding of the instrument in the paper and electronic modes to ensure measurement comparability. The usability testing was to ensure the system was usable by patients.
To some extent, the “minor” classification was adopted by ISPOR because there already was evidence accumulating that paper and electronic forms of assessments were equivalent, including a meta-analysis of 46 unique studies evaluating 278 measurements all showing equivalence.
Notably absent from the ISPOR recommendations was a trend taking hold in this and other industries: BYOD.
The Era of Bring Your Own Device (BYOD)
As time went on, the industry witnessed the rise of BYOD strategies. There are many positives to allowing BYOD in a clinical study. BYOD allows study participants to use devices they are comfortable with and often carry with them. This could lead to higher compliance and engagement in data collection activities. Implementing BYOD can also potentially reduce the costs associated with provisioning dedicated devices to participants.
At the same time, BYOD implementations can add a layer of complexity to PROMs strategies. Sponsors must be able to verify the accuracy and completeness of collected data and must adhere to FDA guidelines governing PROs. Researchers also need to ensure that the data collection platform or app is compatible with a wide range of devices and operating systems to accommodate participants’ diverse technology preferences and to ensure a consistent user experience.
In 2014, an ISPOR task force introduced guidelines on mixing modes within a trial. The crux of this guidance discouraged the mixing of paper and electronic data collection modes in a study, and went on to stipulate the need for equivalence studies if mixing electronic modes.
This essentially halted BYOD efforts as sponsors grappled with how to comply with ISPOR’s recommendation for cognitive debriefs to show equivalency for every potential device.
Evidence Leads to Relaxed Testing Requirements
In the ensuing years, multiple studies were conducted to address the issue of mixed-modes, including a large meta-analysis demonstrating the equivalence of paper and electronic measures. In 2017, a study of paper and BYOD measures showed strong evidence supporting the use of BYOD for PROM collection across scale types, as well as a high patient acceptance of BYOD.
Then in 2018, a large meta-analysis of cognitive debriefs and usability studies involving 53 studies with 68 unique instruments and 101 instrument evaluations found that accumulating evidence pointed to the possibility of relaxing the need to routinely conduct cognitive interviews and usability studies when implementing minor changes during instrument migration. In other words, BYOD strategies using best practices and conducted with a vendor with a good user interface should no longer need formal validation studies for every potential device.
Finally, an Evidence-Based Approach
In 2023, the ISPOR task force embraced a paradigm shift by recommending an evidence-based approach to PROM. This shift in approach recognized that the rigorous comparability testing previously required was becoming excessive in terms of resources and was no longer justified.
The evidence-based approach emphasizes that testing is appropriate only when there is a lack of evidence of measurement comparability, significant changes are introduced, or sponsors lack best practices for faithful migration. However, when sponsors have credible evidence of measurement comparability, even with minor or moderate changes, no further testing is needed. The implication is then that testing across all potential devices for BYOD is not needed and therefore BYOD strategies may move forward, thereby making BYOD an acceptable data collection method.
While some in the industry question why it took so long to recognize what the cognitive debriefs indicated five years ago, the evolution of PROM guidance from stringent comparability testing to an evidence-based approach is welcome news for sponsors. ISPOR’s updated recommendations simplify the path for study sponsors, reducing testing burdens and encouraging the adoption of modern electronic data capture methods.
Editor’s Note: Dr. Jill Platko is vice president of scientific services at Suvoda. In her 16 years in the industry, Jill led the Scientific Advisory practice at Signant Health and held eCOA-related positions at Covance and PHT Corporation. She earned her doctorate in Genetics and Development from Cornell University. She can be reached at jplatko@suvoda.com.