This week, our In Focus section highlights HMA Medicaid Market Solutions’ (MMS) efforts to support state flexibility in designing and implementing Section 1115 Demonstration Waivers promoting member engagement and personal responsibility. Over the coming weeks, HMA MMS will present a series of articles providing in-depth analyses of the many facets of these new Medicaid models. This week, we examine the implications for designing consumerism and personal responsibility waivers.
Section 1115 demonstration waivers allow states to request the Secretary of Health and Human Services to “waive” certain federal Medicaid requirements to implement state-specific policy approaches to serve Medicaid populations. Recently, states have used these waivers to advance consumerism and personal responsibility policies, such as mandatory premium payments and community engagement requirements. The Centers for Medicare and Medicaid Services (CMS) requires that all states evaluate their waivers. The evaluation must be conducted by an independent third-party evaluator. Evaluation components of 1115 waivers are set and reported at five key points in the demonstration process:
- Initial Application: Section 1115 demonstration applications must include the goals and objectives of the waiver, including a general description of how the state intends to evaluate whether the goals and objectives are met.
- Application Approval: During the CMS review, negotiation, and approval of section 1115 demonstrations, detailed evaluation components are outlined and included within the special terms and conditions (STCs), which accompany each approved waiver.
- Evaluation Design: After a waiver is approved, states are required to submit a comprehensive evaluation plan, which describes how each requirement with the STCs will be assessed.
- Demonstration Renewal: Generally, states must submit an evaluation describing findings to date, as part applications to renew or extend their demonstration.
- Demonstration End: States are required to submit a final evaluation plan upon the conclusion of each section 1115 demonstration.
NEW DEVELOPMENTS IN 1115 EVALUATION
In February of this year, the U.S. Government Accountability Office (GAO) published an analysis of state evaluations of Section 1115 Medicaid Demonstrations. The publication, titled “Evaluations Yielded Limited Results, Underscoring Need for Changes to Federal Policies and Procedures”, found that state 1115 evaluations were in need of considerable improvement. The report stated lack of standardization prevented the evaluation of critical program elements and dissemination of results. Even when findings were released, methodological limitations—such as small sample sizes and the lack of comparison groups—prevented the evaluations from being useful.
The GAO report concludes with three formal recommendations for CMS:
- The Administrator of CMS should establish written procedures for implementing the agency’s policy that requires all states to submit a final evaluation report after the end of each demonstration cycle, regardless of renewal status.
- The Administrator of CMS should issue written criteria for when CMS will allow limited evaluation of a demonstration or a portion of a demonstration, including defining conditions, such as what it means for a demonstration to be longstanding or noncomplex, as applicable.
- The Administrator of CMS should establish and implement a policy for publicly releasing findings from federal evaluations of demonstrations, including findings from rapid cycle, interim, and final reports; and this policy should include standards for timely release.
As a result of the GAO report, states should expect more stringent standards for Section 1115 Medicaid Demonstration evaluations. In its written response to the GAO report, the Department of Health and Human Services (HHS) referenced recent actions to enhance the quality of 1115 evaluations, such as including specific evaluation requirements and timelines within demonstration STCs and requiring states to adhere to the prevailing standards of academic rigor in their evaluation reports (e.g., controlling for confounding variables). The HHS response also cited a recent CMS informational bulletin which highlights the agency’s work to develop standardized evaluation templates to support the alignment and comparison of 1115 evaluations.
INCREASED EXPECTATIONS FROM 1115 EVALUATION
CMS is increasing expectations of state 1115 evaluations. Comparison of STCs for Indiana’s previous and most recent Healthy Indiana Plan 1115 demonstrations reveals more stringent evaluation requirements, including two additional addendums outlining specific components of the evaluation.
The first addendum describes the required elements of the evaluation plan including the following:
- A description of how demonstration goals translate into measurable targets for improvement
- The use of driver diagrams to visually aid readers in understanding the rationale for program policies and outcomes
- The identification of the specific statistical testing (e.g., t-tests, chi-square, odds ratio, ANOVA, regression, etc.) for each measure
- A discussion of how evaluation methodologies including “difference-in-differences” and propensity score analyses may be used to adjust for variances in comparison populations over time
The addendum includes an additional mandate that the state post its evaluation plan to its website.
Indiana’s second STC addendum outlines the structure for completing the formal evaluation reports for the demonstration. It requires the state to:
- Discuss how the demonstration interacts with other aspects of the state’s Medicaid program
- Assess the implications of the evaluation’s findings at both the state and national levels
- Describe lessons learned from the demonstration
- Make strategy recommendations for other states interested in implementing a similar approach, as well as other related requirements
CMS is also enhancing evaluation efforts at the federal level. The agency has contracted for separate federal evaluations of 1115 waivers and recently released several reports that describe preliminary findings from its national assessment of section 1115 evaluations.
One report focused on member engagement strategies detailed that although engagement in one state (Indiana) was relatively high (i.e., 90 percent of members made monthly payments to partake in enhanced “HIP Plus” benefits), less than half of the members in the remaining demonstrations participated in incentivized behaviors. The report suggested a need for ongoing member education if demonstrations are to be successful with member engagement.
Another report compared monthly payment requirements across demonstrations finding that the proportion of members who are required or encouraged to make monthly payments vary considerably (e.g., 25% in Michigan versus 100% in Indiana). The report also found that the consequences of failing to comply with required payments varied considerably – being subject to copayments (Iowa), tax return garnishment (Michigan) and disenrollment from programs (Iowa, Indiana, and Montana).
A third report described health plan enrollment continuity among members enrolled in demonstrations that provide premium assistance for members to purchase coverage from the Health Insurance Marketplace. The report highlighted the potential for a high rate of enrollment continuity among demonstrations with complete public and private health plan overlap (Arkansas and New Hampshire), while outlining the possibility of considerable enrollment discontinuity where health plan overlap was not absolute (Iowa). The report identified strategies to encourage or mandate health plan participation in both the public and private sectors as a means of maximizing health plan enrollment continuity for members who transition between eligibility for Medicaid and the Marketplace.
The federal analysis and publication of the results of states’ 1115 evaluations demonstrates the increasing accountability and scrutiny of state evaluations. States should be prepared to incorporate CMS’ recommendations and best practices within their demonstration designs. States should expect that Section 1115 Medicaid demonstration applications, which include strategies identified as ineffective or problematic within the CMS evaluation reports, may face a more difficult road toward federal approval.
CONSIDERATIONS FOR CURRENT & UPCOMING EVALUATION EFFORTS
In addition to meeting CMS requirements, the evaluation should provide value to the program itself. It is important to consider specific data needed to improve identification of demonstration outcomes and to inform state policy decision making. The following elements are essential for designing effective program evaluations:
- Plan in Advance: Evaluation should be built into the waiver application, long before it is approved. In addition, states should incorporate recommended strategies from CMS’ evaluation of similar demonstrations into their evaluation design.
- Budget: Comprehensive evaluations can be expensive. CMS requires states to appoint independent evaluators but does not provide additional evaluation funding outside of administrative funding. Research and plan for associated costs prior to waiver approval.
- Data Availability & Quality: Where will the data to perform the evaluation come from? Does the data structure allow for the analyses proposed in the evaluation? Most evaluations will comprise a mix of both quantitative (e.g., claims) and qualitative (e.g., surveys) data. Depending on the availability and quality of quantitative data, additional investment in the acquisition of qualitative data may be needed.
- Ongoing Monitoring & Evaluation: CMS reserves the right to request ad hoc reports to assess the success of Section 1115 Medicaid Demonstrations, outside of the reports listed within each waiver’s STCs. Planning to have regular monitoring and ongoing reporting capabilities will enable the ability to support additional requests from federal contractors, and allows states to course correct, as needed.
 Centers for Medicare and Medicaid Services (CMS). About Section 1115 Demonstrations. Available at https://www.medicaid.gov/medicaid/section-1115-demo/about-1115/index.html. Accessed August 2, 2018.
 United States Government Accountability Office. Medicaid Demonstrations. Evaluations Yielded Limited Results, Underscoring Need for Changes to Federal Policies and Procedures. Available at https://www.gao.gov/assets/690/689506.pdf. Accessed August 2, 2018.
 U.S. Government Accountability Office. Medicaid Demonstrations:
Evaluations Yielded Limited Results, Underscoring Need for Changes to Federal Policies and Procedures. Available at https://www.gao.gov/products/GAO-18-220. Accessed September 13, 2018.
 U.S. Department of Health and Human Services (HHS). GAO Report on Medicaid Demonstration Evaluations—Appendix II: Comments from the Department of Health and Human Services. Available at https://www.gao.gov/assets/690/689506.pdf. Accessed September 25, 2018.