The Importance of Evaluating Adult Protective Services
to Enhance Program Effectiveness
The Administration for Community Living (ACL) has created a framework for improving state and local adult protective services (APS) programs throughout the country. The framework consists of a complementary set of initiatives, centered on giving APS jurisdictions tools and information to enhance the effectiveness of their programs. With an estimated 1 in 10 of older adults being at risk of abuse, neglect or exploitation, APS programs play a most critical role in protecting and sustaining the health and well-being of all vulnerable adults, including adults with disabilities and older Americans.
The proposed evaluation plan addresses several major gaps in knowledge and understanding of APS:
There are many similarities and differences across state and local APS programs. These similarities and differences are not well-documented and even less well understood. This lack of knowledge and understanding hampers efforts to enhance the effectiveness of APS programs. This means that increasing knowledge and understanding should be a priority for investing in APS program improvement. Without this knowledge and understanding, it is very difficult for the government to make informed decisions about priorities, policies, and programs.
One of the ACL initiatives is implementation of the National Adult Maltreatment Reporting System (NAMRS). NAMRS will establish, for the first time, a national database of information and data on APS programs. Data will provide a broader understanding of what APS does, provide information for advocacy for resources and improvement, and be used in research to establish best practice and service innovations.
A second initiative, The Voluntary Consensus Guidelines for State APS Systems (Guidelines), provides a framework for defining good APS practice. Based on current best practice literature and program input, the Guidelines are a core set of principles and common expectations to encourage consistency in policy and practices across the country. They will help ensure that adults are afforded similar protections and service delivery, regardless of which state or jurisdiction they are in, support interdisciplinary and interagency coordination, and enhance effective delivery both of services to victims and responses to perpetrators.
Through NAMRS and the Guidelines, ACL is providing state and county APS programs a valuable set of resources for program improvement. These resources, as valuable as they are, would be strengthened with the addition of complementary program evaluation activities.
NAMRS data will be useful in answering questions focused on program improvement. An APS program evaluation will provide the right questions to ask of the data captured by NAMRS, greatly increasing the utility of NAMRS.
Similarly, an APS program evaluation will enhance the efficacy of the Guidelines. The Guidelines are evidence-based only to a limited extent since the field of APS lacks a solid foundation of evidence-based research. The Guidelines and a dedication to APS program evaluation would have a synergistic effect: the Guidelines will help inform a theoretical framework for an APS program evaluation and APS program evaluation will provide an evidence base to improve future versions of the Guidelines.
Program evaluation is a valuable tool for improving social service programs. ACL has used it to assess the effectiveness of programs like the Older American’s Act caregiving and long-term care ombudsman programs. There has not been a major federal program evaluation of APS programs. A theoretical framework for thinking about how to enhance the effectiveness of APS programs needs to be developed. The ad hoc evaluations of APS programs that have been conducted, while useful for small picture research questions, did not apply rigorous research criteria or analytical methods and have not focused on developing generalizable knowledge and understanding that can be applied to the APS system as a whole.
Program evaluation provides a rigorous and consistent method for transforming information based on data into evidence based knowledge. ACL can help all APS agencies to improve the efficiency and effectiveness of their programs through a program evaluation that maximizes existing resources and is designed in a building block manner to develop a firm foundation of knowledge and understanding.
The evaluation approach being used applies a rigorous APS theoretical framework to comprehensively analyzing the major functions of APS: intake, investigation, post investigation, and quality assurance. These functions will be examined through the domains of policy, practice, personnel, partners, and performance.
- What policies are currently in place in each U.S. state and territory?
- What are the similarities and differences among policies across the states and territories?
- What practices are being used? Which practices are innovative?
- What are the specific outcomes of APS practices, and do they vary by type of practice?
- Are certain policies and practices associated with different outcomes for aging adults and for adults with disabilities?
Increasing knowledge and understanding through evaluation is necessary to help improve the effectiveness of APS programs in all major program administration areas.
Program Planning: An APS program evaluation will provide federal, state, and local governments with information they need to wisely invest resources in improving APS programs. Government officials need to know what aspects of APS programs need improvement.
One leading program evaluation expert says that evaluation answers three questions:
Effectiveness: An APS program evaluation will help APS program administrators understand what approaches and interventions add the most value in achieving positive client outcomes for APS clients. The evaluation will establish the guide for future research efforts into best practices and service delivery innovation.
Quality Improvement: By establishing a theoretical framework, an APS program evaluation will provide program administrators with information they need to establish quality improvement processes. It will inform future technical assistance by the APS TARC.
Outcome assessment: Development of a theoretical framework, as part of an evaluation of APS, will lay the groundwork for future definition, measurement, and assessment of outcomes of APS services. An APS program evaluation will help identify measurable, achievable outcomes. Completion of the proposed evaluation will, for the first time, provide a data set for rigorous analysis of these outcomes.
- So What?
- Now What?
He adds that the purpose of evaluation is to inform thought and action. It is a means to move from data to interpretation to action; and to move from acting on belief and hope to acting on data and feedback.
The APS program evaluation aligns closely with these concepts. There is a fundamental need to better understand What APS does and how it does, which is addressed in the first component. Beyond understanding, there is a need to answer the So What question in APS—how do various practices and innovations make a difference in the lives of APS clients, which aligns with the second component. Finally, the third component will provide a rich resource for answering the all-important question of Now What—that is, what are the recommendations to enhance the effectiveness of APS programs.
In summary, the answer to the question of “why evaluate APS services?” is:
- Program Improvement: Knowledge is a pre-requisite for program improvement. There is a lack of fundamental knowledge and understanding about the nature and scope of APS programs. Knowledge and understanding should precede investment.
- Initiative Improvement: An evaluation of APS complements other federal efforts, particularly NAMRS and the Guidelines, to improve APS services. Data is useful to the extent you know the right questions to ask of it and guidelines are useful when supported by evidence.
- System Improvement: Program evaluation will create the framework and knowledge base to move the APS system forward. In particular, it will inform the nature and types of technical assistance that the APS TARC will provide and the types of systems improvement ACL should foster in the future.
- Impact Improvement: Within current federal resources and authority, program evaluation is currently the best way for ACL to have a long-term impact in improving APS programs.
Michael Quinn Patton, Essentials of Utilization-Focused Evaluation,
Sage Publishing, 2012.