Despite the growing demand for research that engages stakeholders, there is limited evidence in the literature to demonstrate its value -or return on investment. This gap indicates a general lack of evaluation of engagement activities. To adequately inform engagement activities, we need to further investigate the dividends of engaged research, and how to evaluate these effects. This paper synthesizes the literature on hypothesized impacts of engagement, shares what has been evaluated and identifies steps needed to reduce the gap between engagement's promises and the underlying evidence supporting its practice. This assessment provides explicit guidance for better alignment of engagement's promised benefits with evaluation efforts and identifies specific areas for development of evaluative measures and better reporting processes. Keywords: comparative effectiveness research • evaluation • patient engagement • patient-centered outcomes research • PCOR • review • stakeholder engagement BackgroundEngagement of patients and other healthcare stakeholders (herein, stakeholder partners) is increasingly recognized as essential to patient-centered comparative effectiveness research (CER), which is intended to answer questions of importance to patients and their caregivers [1]. Often described as researchers doing research with patients, rather than for, at or to them [2][3][4], patient-engaged research implies a level of involvement that extends beyond the role of research subject [5]. This shift has been fueled in part by the 2010 creation of the Patient-Centered Outcomes Research Institute (PCORI) by Congress with the passage of the Patient Protection and Affordable Care Act. PCORI is committed to producing and promoting high-integrity CER that is 'guided by patients, caregivers, and the broader healthcare community.' [6].The PCORI Methodology Report states that patient engagement can include: defining topics and formulating study questions, identifying a study population and choosing interventions, comparators and outcomes, developing optimal strategies for recruitment and retention of study participants, conducting a study and analyzing results and disseminating research findings into clinical practice [7]. Generally, the value of engaging stakeholder partners in research is that it can help reorient and improve the research enterprise, reduce clinical uncertainty and speed adoption of meaningful findings that hold great promise with the ultimate goal of improving patients' care experience, decision-making and health outcomes [6].Despite the growing interest in and demand for research that engages stakeholder partners -or, 'research done differently,' [8] there is limited evidence in the published literature to demonstrate the value -or return on investment -of this engagement. This evidence gap is reflective of a general lack of evaluation of research engagement activities. In fact, the majority of published research with a stakeholder partner engagement dimension does not include an evaluation component [9][10][11]...

Aims Stakeholder engagement is fundamental to comparative effectiveness research (CER), but lacks consistent terminology. This paper aims to define stakeholder engagement and present a conceptual model for involving stakeholders in CER. Materials & methods The definitions and model were developed from a literature search, expert input and experience with the Center for Comparative Effectiveness Research in Cancer Genomics, a proof-of-concept platform for stakeholder involvement in priority setting and CER study design. Results Definitions for stakeholder and stakeholder engagement reflect the target constituencies and their role in CER. The ‘analytic-deliberative’ conceptual model for stakeholder engagement illustrates the inputs, methods and outputs relevant to CER. The model differentiates methods at each stage of the project; depicts the relationship between components; and identifies outcome measures for evaluation of the process. Conclusion While the definitions and model require testing before being broadly adopted, they are an important foundational step and will be useful for investigators, funders and stakeholder groups interested in contributing to CER.

Background There is growing interest in patient and stakeholder engagement in research, yet limited evidence about effective methods. Since 2012, the Patient-Centered Outcomes Research Institute (PCORI) has funded patient-centered comparative effectiveness research with a requirement for engaging patients and other stakeholders as research partners in study planning, conduct, and dissemination. This requirement, unique among large healthcare research funders in the US, provides an opportunity to learn about challenges encountered and specific strategies used by PCORI-funded study teams. The primary objective of this study is to describe -- from the perspective of PCORI investigators and research partners—the most common engagement challenges encountered in the first two years of the projects and promising strategies to prevent and overcome these challenges. Methods Descriptive information about investigators, partners, and their engagement was collected from investigators via annual (N = 235) and mid-year (N = 40) project progress reporting to PCORI, and from their partners (N = 260) via voluntary survey. Qualitative data were analyzed using content and thematic analyses. Results Investigators and partners most often described engagement challenges in three domains: (1) infrastructure to support engagement, (2) building relationships, and (3) maintaining relationships. Infrastructure challenges related to financial and human resources, including funding support and dedicated staff, identifying diverse groups of partners, and partners’ logistical needs. Challenges for both building and maintaining relationships encompass a variety of aspects of authentic, positive interactions that facilitate mutual understanding, full participation, and genuine influence on the projects. Strategies to prevent or mitigate engagement challenges also corresponded overall to the same three domains. Both groups typically described strategies more generally, with applicability to a range of challenges rather than specific actions to address only particular challenges. Conclusion Meaningful engagement of patients and other stakeholders comes with challenges, as does any innovation in the research process. The challenges and promising practices identified by these investigators and partners, related to engagement infrastructure and the building and maintenance of relationships, reveal actionable areas to improve engagement, including organizational policies and resources, training, new engagement models, and supporting engagement by viewing it as an investment in research uptake and impact.

Background US research organizations increasingly are supporting patient and stakeholder engagement in health research with a goal of producing more useful, relevant and patient‐centered evidence better aligned with real‐world clinical needs. The Patient‐Centered Outcomes Research Institute (PCORI) engages patients, family caregivers and other health‐care stakeholders, including clinicians, payers and policymakers, as active partners in prioritizing, designing, conducting and disseminating research as a key strategy to produce useful evidence for health‐care decision making. Objective To inform effective engagement practices and policies, we sought to understand what motivates patients and caregivers to engage as partners on PCORI‐funded research projects and how such engagement changed their lives. Methods We conducted thematic analysis of open‐ended survey responses from 255 patients, family caregivers and individuals from advocacy and community‐based organizations who engaged as partners on 139 PCORI‐funded research projects focusing on a range of health conditions. Results Partners' motivations for engaging in research were oriented primarily towards benefiting others, including a desire to improve patients' lives and to support effective health‐care interventions. In addition to feeling they made a positive difference, many partners reported direct benefits from engagement, such as new relationships and improved health habits. Discussion and Conclusions By identifying patient and caregiver motivations for engaging in research partnerships and what they get out of the experience, our study may help research teams and organizations attract partners and foster more satisfying and sustainable partnerships. Our findings also add to evidence that engagement benefits the people involved as partners, strengthening the case for more widespread engagement.

BackgroundOne of the most common barriers to improving immunization coverage rates is human resources and its management. In the Republic of Georgia, a country where widespread health care reforms have taken place over the last decade, an intervention was recently implemented to strengthen performance of immunization programs. A range of measures were taken to ensure that immunization managers carry out their activities effectively through direct, personal contact on a regular basis to guide, support and assist designated health care facility staff to become more competent in their immunization work. The aim of this study was to document the effects of "supportive" supervision on the performance of the immunization program at the district(s) level in Georgia.MethodsA pre-post experimental research design is used for the quantitative evaluation. Data come from baseline and follow-up surveys of health care providers and immunization managers in 15 intervention and 15 control districts. These data were supplemented by focus group discussions amongst Centre of Public Health and health facility staff.ResultsThe results of the study suggest that the intervention package resulted in a number of expected improvements. Among immunization managers, the intervention independently contributed to improved knowledge of supportive supervision, and helped remove self-perceived barriers to supportive supervision such as availability of resources to supervisors, lack of a clear format for providing supportive supervision, and lack of recognition among providers of the importance of supportive supervision. The intervention independently contributed to relative improvements in district-level service delivery outcomes such as vaccine wastage factors and the DPT-3 immunization coverage rate. The clear positive improvement in all service delivery outcomes across both the intervention and control districts can be attributed to an overall improvement in the Georgian population's access to health care.ConclusionProvider-based interventions such as supportive supervision can have independent positive effects on immunization program indicators. Thus, it is recommended to implement supportive supervision within the framework of national immunization programs in Georgia and other countries in transition with similar institutional arrangements for health services organization.Abstract in RussianSee the full article online for a translation of this abstract in Russian.

Objective The objective of this study was to evaluate the feasibility and outcomes of incorporating value of information (VOI) analysis into a stakeholder-driven research prioritization process in a US-based setting. Methods Within a program to prioritize comparative effectiveness research areas in cancer genomics, over a period of 7 months, we developed decision-analytic models and calculated upper-bound VOI estimates for three previously selected genomic tests. Thirteen stakeholders representing patient advocates, payers, test developers, regulators, policy-makers, and community-based oncologists ranked the tests before and after receiving VOI results. The stakeholders were surveyed about the usefulness and impact of the VOI findings. Results The estimated upper-bound VOI ranged from $33M to $2.8 billion for the three research areas. Seven stakeholders indicated the results modified their rankings, nine stated VOI data was useful, and all indicated they would support its use in future prioritization processes. Some stakeholders indicated expected value of sampled information might be the preferred choice when evaluating specific study designs. Limitations Our study was limited by the size and the potential for selection bias in the composition of the external stakeholder group, lack of a randomized design to assess effect of VOI data on rankings, and the use of expected value of perfect information versus expected value of sample information methods. Conclusions Value of information analyses may have a meaningful role in research topic prioritization for comparative effectiveness research in the US, particularly when large differences in VOI across topic areas are identified. Additional research is needed to facilitate the use of more complex value of information analyses in this setting.

BackgroundIn September 2003, the Canadian government committed to developing legislation that would facilitate greater access to affordable medicines for developing countries. Over the course of eight months, the legislation, now known as Canada's Access to Medicines Regime (CAMR), went through a controversial policy development process and the newspaper media was one of the major venues in which the policy debates took place. The purpose of this study was to examine how the media framed CAMR to determine how policy goals were conceptualized, which stakeholder interests controlled the public debate and how these variables related to the public policy process.MethodsWe conducted a qualitative content analysis of newspaper coverage of the CAMR policy and implementation process from 2003-2008. The primary theoretical framework for this study was framing theory. A total of 90 articles from 11 Canadian newspapers were selected for inclusion in our analysis. A team of four researchers coded the articles for themes relating to access to medicines and which stakeholders' voice figured more prominently on each issue. Stakeholders examined included: the research-based industry, the generic industry, civil society, the Canadian government, and developing country representatives.ResultsThe most frequently mentioned themes across all documents were the issues of drug affordability, intellectual property, trade agreements and obligations, and development. Issues such as human rights, pharmaceutical innovation, and economic competitiveness got little media representation. Civil society dominated the media contents, followed far behind by the Canadian government, the research-based and generic pharmaceutical industries. Developing country representatives were hardly represented in the media.ConclusionsMedia framing obscured the discussion of some of the underlying policy goals in this case and failed to highlight issues which are now significant barriers to the use of the legislation. Using the media to engage the public in more in-depth exploration of the policy issues at stake may contribute to a more informed policy development process. The media can be an effective channel for those stakeholders with a weaker voice in policy deliberations to raise public attention to particular issues; however, the political and institutional context must be taken into account as it may outweigh media framing effects.

Introduction Complex health interventions (CHIs) are increasingly studied in comparative effectiveness research (CER), and there is a need for improvements in CHI research practices. The Patient-Centered Outcomes Research Institute (PCORI) Methodology Committee (MC) launched an effort in 2016 to develop formal guidance on this topic. Objective To develop a set of minimal standards for scientifically valid, transparent, and reproducible CER studies of CHIs. The standards are intended to apply to research examining a broad range of healthcare interventions including delivery system, behavior change, and other non-pharmacological interventions. Methods We conducted a literature review, reviewed existing methods guidance, and developed standards through an iterative process involving the MC, two panels of external research methods experts, and a 60-day public comment period. The final standards were approved by the PCORI MC and adopted by the PCORI Board of Governors on April 30, 2018. Results The final standards include the following: (1) fully describe the intervention and comparator and define their core functions, (2) specify the hypothesized causal pathways and their theoretical basis, (3) specify how adaptations to the form of the intervention and comparator will be allowed and recorded, (4) plan and describe a process evaluation, and (5) select patient outcomes informed by the causal pathway. Discussion The new standards offer three major contributions to research: (1) they provide a simple framework to help investigators address the major methodological features of a CHI study, (2) they emphasize the importance of the causal model and the need to understand how a CHI achieves its effects rather than simply measuring these effects, and (3) they require description of a CHI using the concepts of core functions and forms. While these standards apply formally to PCORI-funded CER studies, they have broad applicability.