Analysis of Stakeholder Engagement in the Public Comments of ICER Draft Evidence Reports

Health technology assessment is a dynamic tool used to inform real-world decisions about the value of technologies and resource allocation in healthcare.¹ Although cost-effectiveness analysis has long formed the backbone of health technology assessment decision criteria, there has been increasing recognition that value frameworks should capture the multidimensional aspects of value to better inform stakeholder decisions.² Indeed, value frameworks are becoming increasingly vital components of health economics in the United States.³

The Institute for Clinical and Economic Review (ICER) is an independent, nonprofit research organization that creates value-based price benchmarks for prescription drugs and other healthcare technologies. The organization represents the most comprehensive attempt by a US-based group to complete detailed, publicly available health technology assessment. Like its analogous international counterparts such as the UK’s National Institute for Health and Care Excellence or Australia’s Pharmaceutical Benefits Advisory Committee,⁴ ICER conducts clinical and economic reviews centered on cost-effectiveness analyses that generate a cost per quality-adjusted life-year (QALY) ratio.⁵ The rise of ICER’s legitimacy in the United States has, in part, reflected demand among healthcare payers for health technology assessment in the absence of a governmental health technology assessment body.⁵

Although value frameworks, including those promulgated by ICER,⁶ can facilitate clinical and resource allocation decisions, their development and interpretation have not been without criticism. Stakeholders with diverse perspectives and goals often value costs and benefits differently.^4,7 For example, policymakers must balance the social values with the need to create evidence-based strategies to adequately manage healthcare costs.⁸ US health plans use comparative clinical and cost-effectiveness analyses in their decision-making, but they may not always prioritize the cost per QALY calculations that are common to traditional health technology assessment bodies.⁹

Patients and physicians view value broadly,^10,11 and patient advocates have expressed concern that ICER’s prioritization of cost and QALY measures has overshadowed the contextual considerations that are important to patients^10,12 and has not captured the individuality of needs for each patient or patient population.^10,13 Industry representatives have echoed patients’ concerns that an overemphasis on cost-effectiveness limits patient centricity and adaptability.¹⁴

ICER must therefore work to conduct transparent cost-effectiveness and budget impact analyses while refining its value framework and reviews according to stakeholder input and evolving recommendations.^15,16 To that end, ICER aims to engage with diverse stakeholders, including patients, clinical experts, drug manufacturers, health plans, and industry representatives, to mitigate concerns, improve transparency, and incorporate feedback into its reviews.^5,6 Specifically, ICER allows stakeholders and interest groups to submit input on publicly posted draft evidence reports.

The present study aimed to understand the frequency and characteristics of written stakeholder engagement with the draft evidence reports. Specifically, the goals of this study were to describe the number, type, and source of comments submitted in response to ICER draft reports, and to examine the associations between healthcare stakeholder type, comment type and frequency, and the therapeutic area.

Methods

We obtained letters submitted from stakeholders in response to ICER’s publicly posted evidence reports for 7 ICER reviews published between 2014 and 2016. Table 1 describes the 7 therapeutic areas for the ICER reports and the pharmaceutical therapies discussed in each report. The reports were selected based on availability at the time of the data collection and were refined by the ICER value framework version, preceding the 2017-2019 update, that was applied at the time of our review.

Each letter was first classified by stakeholder type of pharmaceutical manufacturer, patient or patient group, provider or provider group, government (state or federal), health plan or pharmacy benefit manager (PBM), employer or employer coalition, or other (including consultant and other organizations not fitting a particular stakeholder type).

Next, individual comments from stakeholder letters were identified and classified. Specifically, a comment was defined as a simple, self-contained thought with a clear and interpretable meaning. Each comment was assigned a unique identifier and was independently evaluated by 2 trained doctoral students (authors JAG and RV) to categorize the comment type as methodological, missing data, patient experience, provider experience, general support, or general criticism. The comments regarding methodology included, for example, those that referred to the population characteristics and subgroups, cost and efficacy assumptions, or the time horizon of ICER’s proposed model structure. In cases where the 2 evaluators disagreed, a third evaluator (author KMP), who provided the initial training to the doctoral students, reviewed the comment for classification.

This study involves the analysis of ICER draft evidence reports and does not constitute research on human subjects; therefore, it was exempt from Institutional Review Board review.

Statistical Analysis

Descriptive statistics were used to describe the number and percentage of comments by report, stakeholder, and comment type. To test for differences across the categories, 3 pairwise Fisher’s exact tests were performed as follows, with simulated P values based on 10,000 iterations for each test, and a Bonferroni-corrected significance threshold of α = 0.0167:

• Test 1: comment type × stakeholder

• Test 2: comment type × report

• Test 3: stakeholder × report

For tests 1 and 2, the comment types considered were methodological, missing data (published or unpublished), general support, and general criticism; comments in which patients or patient groups or providers or provider groups shared their experiences were omitted from these analyses, because of their relative infrequency. For tests 1 and 3, the stakeholder types considered were pharmaceutical manufacturer, patient or patient group, and provider or provider group. Employers or employer coalitions were omitted from these analyses because neither contributed any comments to the data set, and government (state or federal) and health plan or PBM were similarly omitted because of their infrequency. All analyses were performed in R software version 3.5.1 (R Foundation for Statistical Computing; Vienna, Austria).

Results

A total of 463 comments, submitted in 55 total communications, were identified in response to the 7 ICER draft evidence reports (Table 2). The stakeholder comments we reviewed most often pertained to ICER’s reports on multiple sclerosis (N = 169; 36.5%), multiple myeloma (N = 108; 23.3%), and insulin degludec (N = 51; 11%). The number of comments tended to increase with the number of therapies included in a draft report, except for the insulin degludec report, which garnered 51 comments despite evaluating only a single drug.

Drug manufacturers were the most engaged stakeholder type, as assessed by the number of comments (N = 292; 63.1%), followed by patient or patient advocacy groups (N = 84; 18.1%), provider or provider groups (N = 45; 9.7%), and other entities (N = 37; 8%). Government entities (N = 3; 0.65%), health plans (N = 2; 0.43%), and employer stakeholders (N = 0; 0%) rarely or never submitted comments.

Comments made on the ICER draft reports most often pertained to the methodology of the value assessment (N = 249; 53.8%). Comments about missing data (N = 65; 14%), general criticism (N = 38; 8.2%), and general support (N = 10; 2.2%) were less common, and few comments (<1%) reflected patient or provider experiences. A total of 95 (20.5%) comments were categorized as other.

The stakeholder comment type varied significantly (P<.001) across the reports (Table 3). The proportion of comments that highlighted methodological considerations varied by report from 40% to 78%, with methodological comments comprising the majority of comments for all reports, except for the congestive heart failure report. Furthermore, the number of generally critical comments equaled or exceeded the number of generally supportive comments made for each report (Table 3). Although the distribution of comment types varied by report, it was consistent across stakeholder types (P = .2842). No patients or patient groups submitted comments of general support.

The distribution of stakeholder types was significantly different (P<.001) across the reports (Table 4). Drug manufacturers submitted the majority of the total comments on 5 of the 7 (71.4%) reports and were the only stakeholders who engaged with the congestive heart failure report. Many comments were submitted by provider groups for the hepatitis C (N = 20 comments) and multiple myeloma (N = 19) reports; provider engagement was much more limited for the remaining reports (≤5 comments for each). Patients engaged most often with the multiple sclerosis (N = 34 comments) and multiple myeloma (N = 27) reports.

Discussion

With rising healthcare spending on prescription drugs, effective measures to assess and communicate the value of healthcare services are increasingly important to payers’ decision-making.³ However, our analysis shows that payers were underrepresented among ICER report commentators during the study period. The comments on the ICER reports that were included in this study were most often (63.1%) submitted by drug manufacturers. Although pharmaceutical manufacturers have expressed strong support for the use of evidence-driven value frameworks, industry representatives of pharmaceutical research and manufacturing have also criticized the cost per QALY metric’s limited ability to capture all aspects of complex, real-world healthcare decision-making or to reflect true patient preferences.¹⁴

In our study, these concerns were likely reflected in the frequency and the nature of pharmaceutical manufacturer comments, which were primarily (53.8%) about methodological considerations. Drug manufacturers’ high level of engagement may also be interpreted in the context of recent studies showing that a minority (29%-38%) of therapies that were evaluated were deemed cost-effective in ICER reports^17,18 and that ICER has recommended, on average, price discounts for 53% to 83% of the evaluated therapies.¹⁸ The use of budget impact models in ICER’s value assessment is controversial among drug manufacturers, who may believe that affordability should not influence the long-term value of a drug.¹⁶

In addition, before their 2017-2019 value framework update,6 ICER included a drug or intervention uptake estimate that assumed no provider-imposed constraints on uptake. This estimate is no longer calculated because of a concern that stakeholders may view it as a prediction of true marketplace behavior.¹⁶ ICER now demonstrates budget impact to payers using a range of price points and uptake assumptions. Drug manufacturers may therefore have strong incentives to advocate for methodological changes that improve the favorability of ICER’s assessments. Future research is warranted to examine changes in the frequency and type of manufacturer engagement with ICER reviews, because ICER’s influence and track record have grown.

In contrast to drug manufacturers, payers, including governmental entities, health plans, and employers, rarely engaged early in the ICER review process. This finding was unexpected, given that health technology assessment reports are used primarily for coverage decisions internationally; furthermore, nearly half (46%) of US payers expressed an intention to use ICER reports when surveyed after the first 3 reports were published.¹⁹ Several reasons, however, may underlie payers’ limited engagement during the ICER review process.

First, although US payers often use comparative-­effectiveness analyses for their coverage decisions, their use of explicit cost per QALY metrics has been limited,⁹ and payers have stated that they use the ICER reports primarily as an evidence source during their drug evaluation processes.^19,20 Second, ICER’s models remain proprietary, precluding payer customization to reflect plan-specific patient populations and costs.²⁰

The timing of ICER reports’ availability may also limit their usefulness to payers¹⁹; ICER has more recently announced a desire to publish reports as close to the US Food and Drug Administration drug approval as possible.²¹ Finally, several aspects of ICER’s methodology, including its health system perspective, emphasis on randomized controlled clinical trial data over real-world evidence, and time horizons, may limit its relevance to US payers.²⁰ Increased payer involvement during ICER reviews may improve ICER’s perceived usefulness to payers.

Our findings also suggest that there is an opportunity for greater government involvement in the ICER process. In the United States, government-sponsored Medicare and Medicaid health plans have a combined expenditure that exceeds that of private health plans, making them an essential stakeholder to be involved in the health technology assessment process.²²

Particularly, cost-benefit and cost-effectiveness models that use short and long time horizons may be useful in determining costs and cost offsets to the Centers for Medicare & Medicaid Services (CMS) when evaluating coverage for new drugs. Increasing CMS engagement in the ICER review process could help to generate more robust models that would be of use to CMS, as well as to a wider set of payers. Although the Agency for Healthcare Research and Quality provides health technology assessment evidence to CMS, its review is focused on clinical utility, not on economic implications.²³

With the final update of ICER’s 2017-2019 Value Assessment Framework came the publication of patient participation and manufacturer engagement guides that detail the most effective ways to affect the health technology assessment process.²⁴ ICER allows public stakeholder engagement to influence the draft scoping document, draft evidence report and voting questions, and the public meeting discussion.

Before 2017, opportunities for public involvement were limited to the draft evidence report and physical presence at the public meeting. ICER now also releases a stakeholder list with the draft scoping document, which indicates the stakeholders with whom ICER intends to consult directly throughout the health technology assessment process.²⁵ The selection of these stakeholders is based on stakeholder engagement in various channels and assumed appropriateness, with no implications for ICER endorsements.

Our study showed that patient engagement with the ICER reports varied, with patient comments comprising 0% to 34% of the total stakeholder comments submitted for each report. Of note, no comments submitted by patients or patient groups represented general support for the ICER analysis, which may reflect the previously expressed concerns that QALY-based analyses lack patient-centeredness.¹⁰ A lack of well-defined patient roles in health technology assessment, and uncertainty surrounding the way in which their evidence will be handled and valued, have also previously garnered criticism from patient groups.^26,27

In the future, ICER may promote patient engagement, and potentially increase the relative balance of criticism and support received from patients, by improving transparency about the role of patient input and patient-centeredness in its reports. More consistent patient participation may, in turn, improve the model’s credibility, salience, and responsiveness.^26,28,29

Furthermore, the variability in patient and provider participation in the ICER process shows that the visibility of ICER reviews in these communities may vary. Future research is needed to examine whether ICER’s increasing prominence and/or the involvement of advocacy groups affect the degree to which patients and providers engage with each review.

Limitations

This study analyzed ICER’s Draft Evidence Reports between 2014 and 2016, which does indeed date the study’s findings. However, reports outside of these dates were intentionally omitted, because of changes in the ICER Value Assessment Framework of 2017-2019, and so on. Today, ICER continues to complete its Draft Evidence Report review process as explained in this study, with additional opportunities for public review. The ICER Value Framework that was applied during the creation of the studied Draft Evidence Reports is consistent and creates value in this research, as well as a potential benchmark for future comparative research studies.

This study has inherent limitations regarding potential bias in the identification and categorization of comments within the ICER public comment reports. To minimize this potential bias, we used 2 independent reviewers to identify and classify the comments, and a third independent reviewer was used to resolve discrepancies.

The number of comment categories was also limited for feasibility purposes; future studies may be needed to explore additional categories and include a larger sample of comments.

Furthermore, this study’s generalizability to more recent patterns of stakeholder engagement may be limited by changes in the ICER review process beginning in 2017. ICER has since instituted a second opportunity for stakeholder feedback during reviews, and it now proactively requests input from a published stakeholder list, which most often includes payers, drug manufacturers, and patient advocacy groups. Future research is needed to explore changes in the type and frequency of stakeholder engagement because ICER’s review process has evolved.

Conclusion

ICER—the first major nonprofit US-based group to complete detailed, publicly available health technology assessment—has aimed to engage healthcare stakeholders during the development of its value framework and its reviews. This study of stakeholder involvement in ICER reviews shows that drug manufacturers had strong incentives to provide input regarding ICER’s methods, whereas healthcare payers were not as engaged in the process. Government bodies that serve as essential payers in our healthcare system were noticeably not engaged in this review. Patients and patient groups had varying degrees of involvement in the reviews and most notably gave no comments of “general support” to the ICER reports, which may be because of concerns about a lack of patient centeredness in the ICER reports. These findings suggest that stakeholders are viewing and engaging with ICER reports in different ways and identify opportunities for further discussion and research.

Acknowledgments

The authors wish to thank Dr Nina Lyons for her assistance in constructing and populating the study database.

Author Disclosure Statement

Ms Prioli has received honorarium from Cerus Corp, and is a health economics and outcomes research modeling consultant to the Institute for Integrated Healthcare. Dr Patterson was an employee of Indivior before her work on this project. Dr Pizzi has received grant support from Merck and Co, GlaxoSmithKline, Cerus Corp, NovoNordisk, Janssen Pharmaceuticals, and Allergan. Ms Gerlach, Dr Snow, and Dr Vertsman have no conflicts of interest to report.

References

1. Drummond M, Jönsson B, Rutten F, Stargardt T. Reimbursement of pharmaceuticals: reference pricing versus health technology assessment. Eur J Health Econ. 2011;12:263-271.
2. Willke RJ, Neumann PJ, Garrison LP Jr, Ramsey SD. Review of recent US value frameworks—a health economics approach: an ISPOR Special Task Force report [6]. Value Health. 2018;21:155-160.
3. Neumann PJ, Willke RJ, Garrison LP Jr. A health economics approach to US value assessment frameworks—introduction: an ISPOR Special Task Force report [1]. Value Health. 2018;21:119-123.
4. Freund D, Choi J. Is ICER NICEr? Pharmacoeconomics. 2018;36:385-386.
5. Neumann PJ, Cohen JT. ICER’s revised Value Assessment Framework for 2017–2019: a critique. Pharmacoeconomics. 2017;35:977-980.
6. Institute for Clinical and Economic Review. Overview of the ICER Value Assessment Framework and update for 2017-2019. https://icer-review.org/wp-­content/uploads/2018/03/ICER-value-assessment-framework-update-FINAL-062217.pdf. Accessed April 12, 2019.
7. Philipson TJ, Kamal-Bahl S, Jena AB. Defining value: the need for a longer, broader view. Pharmacoeconomics. 2017;35:669-672.
8. Norman R, Chalkidou K, Culyer AJ. A health economics approach to US value frameworks: serving the needs of decision making. Value Health. 2018;21:117-118.
9. Solow B, Pezalla EJ. ISPOR’s Initiative on US Value Assessment Frameworks: the use of cost-effectiveness research in decision making among US insurers. Value Health. 2018;21:166-168.
10. Perfetto EM. ISPOR’s Initiative on US Value Assessment Frameworks: a missed opportunity for ISPOR and patients. Value Health. 2018;21:169-170.
11. Frois C, Howe A, Jarvis J, et al. Drug treatment value in a changing oncology landscape: a literature and provider perspective. J Manag Care Spec Pharm. 2019;25:246-259.
12. Weber JS, Drakeman DL. Comment on the American Society of Clinical Oncology value statement. J Clin Oncol. 2016;34:2937-2938.
13. FasterCures; Avalere. Integrating the patient perspective into the development of value frameworks. March 2016. https://synapse.pfmd.org/resources/13a48a0d-dd52-4654-80d1-b69a0d24d8df/integrating-the-patient-perspective-into-the-development-of-value-frameworks/download. Accessed May 10, 2019.
14. Burkholder R, Dougherty JS, Neves LA. ISPOR’s Initiative on US Value Assessment Frameworks: an industry perspective. Value Health. 2018;21:173-175.
15. Garrison LP Jr, Neumann PJ, Willke RJ, et al. A health economics approach to US value assessment frameworks—summary and recommendations of the ISPOR Special Task Force report [7]. Value Health. 2018;21:161-165.
16. Pearson SD. The ICER value framework: integrating cost effectiveness and affordability in the assessment of health care value. Value Health. 2018;21:258-265.
17. Macaulay R, Majeed B, Mcnaughton E, Anastasaki E. ICER reports – 4 years of cost/QALY drug evaluations in the USA…what does the future hold? Value Health. 2018;21(suppl 1):Abstract PHP24.
18. Ramirez Guevara G, Shields GE, McCrea C, Treharne C. A review of recent appraisals from the Institute for Clinical and Economic Review: how important is the ICER? Value Health. 2018;21(suppl 3):Abstract PHP294.
19. Lising A, Drummond M, Barry M, Augustovski F. Payers’ use of independent reports in decision making – will there be an ICER effect? Value Outcomes Spotlight. 2017;3:7-10.
20. Pyenson B, Sawhney TG, Buzby E. The Utility of ICER Reports for Private Payer Drug Coverage Decision-Making: Cost-Effectiveness Assessments. New York, NY: Milliman; September 19, 2018. https://milliman-cdn.azureedge.net/-/media/milliman/importedfiles/uploadedfiles/insight/2018/utility-icer-reports-private-payer.ashx. Accessed June 24, 2020.
21. Institute for Clinical and Economic Review. Institute for Clinical and Economic Review announces potential topics for review in 2018. July 26, 2017. https://icer-review.org/announcements/icer-topic-selection-2018/. Accessed June 24, 2020.
22. Centers for Medicare & Medicaid Services. NHE fact sheet. Updated March 24, 2020. www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NHE-Fact-Sheet. Accessed June 24, 2020.
23. Agency for Healthcare Research and Quality. Technology Assessment Program. Updated April 2020. www.ahrq.gov/research/findings/ta/index.html. Accessed June 24, 2020.
24. Institute for Clinical and Economic Review. Institute for Clinical and Economic Review releases final Value Assessment Framework for 2017-2019. June 22, 2017. https://icer-review.org/announcements/vaf-update-2017-2019/. Accessed June 18, 2019.
25. Institute for Clinical and Economic Review. Patient, manufacturer, and other stakeholder engagement. https://icer-review.org/methodology/stakeholder-­engagement/. Accessed June 18, 2019.
26. Rosenberg-Yunger ZRS, Thorsteinsdóttir H, Daar AS, Martin DK. Stakeholder involvement in expensive drug recommendation decisions: an international perspective. Health Policy. 2012;105:226-235.
27. Barham L. Public and patient involvement at the UK National Institute for Health and Clinical Excellence. Patient. 2011;4:1-10.
28. van Voorn GAK, Vemer P, Hamerlijnck D, et al. The missing stakeholder group: why patients should be involved in health economic modelling. Appl Health Econ Health Policy. 2016;14:129-133.
29. Boothe K. “Getting to the table”: changing ideas about public and patient involvement in Canadian drug assessment. J Health Polit Policy Law. 2019;44:631-663.