APS Technical Assistance
Resource Center
Home / Evaluation of APS
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
HHS Administration for Community Living websites devoted to supporting Elder Justice
Evaluation of Adult Protective Services
The National Adult Protective Services (APS) Process Evaluation is based on research conducted by the APS Technical Assistance Resource Center. This evaluation was designed and implemented to describe the current landscape of APS program structure and operations across the United States. It is available in two reports:
- Full Report (PDF | 3.09 MB)
- Summary Report (PDF | 1.88 MB)
Why evaluate APS?
The Administration for Community Living (ACL) has created a framework for improving state and local adult protective services (APS) programs throughout the country. The framework consists of a complementary set of initiatives, centered on giving APS jurisdictions tools and information to enhance the effectiveness of their programs. With an estimated one in 10 of older adults being at risk of abuse, neglect or exploitation, APS programs play a critical role in protecting and sustaining the health and well-being of all vulnerable adults, including adults with disabilities and older Americans.
There are many similarities and differences across state and local APS programs. These similarities and differences are not well-documented and even less understood. This lack of knowledge and understanding hampers efforts to enhance the effectiveness of APS programs. This means that increasing knowledge and understanding should be a priority for investing in APS program improvement. Without this knowledge and understanding, it is very difficult for the government to make informed decisions about priorities, policies, and programs.
Program evaluation provides a rigorous and consistent method for transforming information based on data into evidence based knowledge. ACL can help all APS agencies to improve the efficiency and effectiveness of their programs through a program evaluation that maximizes existing resources and is designed in a building block manner to develop a firm foundation of knowledge and understanding.
The evaluation approach being used - a process evaluation - applies a rigorous APS theoretical framework to comprehensively analyzing the major functions of APS: intake, investigation, post investigation, and quality assurance. These functions will be examined through the domains of policy, practice, personnel, partners, and performance.
The proposed evaluation plan addresses several major gaps in knowledge and understanding of APS:
- What policies are currently in place in each U.S. state and territory?
- What are the similarities and differences among policies across the states and territories?
- What practices are being used? Which practices are innovative?
- What are the specific outcomes of APS practices, and do they vary by type of practice?
- Are certain policies and practices associated with different outcomes for aging adults and for adults with disabilities?
One leading program evaluation expert says that evaluation answers three questions:
1. What?
2. So What?
3. Now What?
He adds that the purpose of evaluation is to inform thought and action. It is a means to move from data to interpretation to action; and to move from acting on belief and hope to acting on data and feedback. ¹
The APS program evaluation aligns closely with these concepts. There is a fundamental need to better understand What APS does and how it does, which is addressed in the first component. Beyond understanding, there is a need to answer the So What question in APS—how do various practices and innovations make a difference in the lives of APS clients, which aligns with the second component. Finally, the third component will provide a rich resource for answering the all-important question of Now What—that is, what are the recommendations to enhance the effectiveness of APS programs.
¹ Michael Quinn Patton, Essentials of Utilization-Focused Evaluation, p. 3. Sage Publishing, 2012.
How do you evaluate APS?
The Administration for Community Living (ACL) established the Adult Protective Service Technical Assistance Resource Center (APS TARC) to enhance the effectiveness of APS programs. As a first step in that process, the APS TARC is conducting the first-ever national evaluation of APS. The following provides an overview and approach of the plan.
The plan:
- Includes an APS Logic Model to use as a theoretical framework for conducting the program evaluation.
- Includes a compilation of potential research questions, based on the APS Logic Model.
- Presents a study comprised of three components to evaluate APS. A phased or stepwise approach was used to address the fact that there is currently only a basic understanding of the services provided by APS in each state, with little knowledge about the effectiveness of these services at reducing abuse and preventing re-referral to the APS system.
The APS Logic Model is a one-page depiction of the following elements of APS programs: context, inputs/resources, activities, activity metrics, and results. Activities, activity metrics, and results are divided into the typical case flow of intake (also often called “prescreening”), investigation, and post-investigation services. Quality assurance is also included and is comprised of a number of activities (e.g., documentation and supervisory review) that are critical aspects of APS programs.
The identification of research questions is a critical step in the development of a program evaluation plan. Research questions help formulate what will be studied and evaluated, and provide a tool for prioritizing topics to be studied. The APS TARC went one step further and elaborated upon five areas or constructs of program operations, which were used to organize the research questions. The five constructs are policy, practice, personnel, partners, and performance. The constructs are defined in Exhibit A.
Exhibit A - Program Operation Constructs
| Construct | Definition |
| Policy | Policy consists of the mission, goals, and objectives of a program. Often, policy also provides guidance or requirements in terms of procedures. Included in policy may also be requirements for staff and funding, and cooperation with other agencies. Formal policy is always written and found in state statutes, administrative code, and agency manuals. Policy may be created by the legislative or judicial branches of government, or by the executive branch. |
| Practice | Practice entails those activities and actions of APS agency staff that implement policy or other management expectations. |
| Personnel | Personnel includes the qualifications of staff, as well as the hiring, training, and performance review of staff, which are implemented based upon policy. |
| Partners | Partners are other agencies (governmental, non-governmental, and private) that support the program or program’s clients. |
| Performance | Performance refers to the measurement of activities and outcomes. |
To develop the research questions, these constructs were cross-referenced with the critical elements of the APS Logic Model (context, intake, etc.) and research questions developed for each intersection. The result is a comprehensive set of potential broad research questions to guide the evaluation. Ultimately, based on feedback from ACL and development of research hypotheses, the research questions were refined and focused on specific areas of inquiry consistent with the selected research methodology. Exhibit B show a summary of the cross-referencing and how it is addressed in the evaluation plan.
Exhibit B - Cross Referencing
| Construct | Definition | Key Questions | Methods | Component |
| Policy | Policy is found in state statutes, administrative code, and agency policy manuals. All policy is written. | What policies direct key program activities? | Document review of extant state codes, regulations, and policy manuals and analysis of initial year NAMRS data (Agency Component). | 1 |
| Practice | Practice entails those activities and actions of APS agency staff to implement policy or other management expectations. | How are critical activities conducted? | Surveys of APS staff, on-site focus groups, and analysis of NAMRS data. | 2 |
| Personnel | Personnel includes the qualifications of staff, as well as the hiring, training, performance review of staff. | How do APS programs employ and train staff and what are key activities related to staff effectiveness? | Document review of extant state codes, regulations and policy manuals and analysis of initial year NAMRS data (Agency Component). Survey of APS staff, on-site focus groups, and analysis of NAMRS data. | 1, 2 |
| Partners | Partners are other agencies (government, non-governmental, and private) that support the program or program's clients. | Who are the critical partners of the APS agency and what type of assistance do they provide? | Surveys of APS staff, on-site focus groups, and analysis of NAMRS data. | 2 |
| Performance | Performance refers to the measurement of activities and outcomes. | How do APS programs measure activity metrics and system-level outcomes? What are the system-level outcomes for APS activities? | Document review of extant state codes, regulations, and policy manuals and analysis of intial year NAMRS data (Agency Component). Surveys of APS staff, on-site focus groups, and analysis of NAMRS data. Analysis of state-level NAMRS data sets. | 1, 2, 3 |
The APS program evaluation is comprised of a three-component study. The first two components are process evaluations, which build towards a more system outcome-oriented evaluations (defined in the third component below). The process-oriented components start with a recognition, discussed with ACL, that knowledge-building is the first type of evaluation to undertake when evaluating APS. Out of this knowledge base, more sophisticated evaluations can be undertaken. Given this, the three evaluation components were developed. A summary of these components are as follows:
Exhibit C - Summary of Study Components
| Component 1 | Component 2 | Component 3 | |
| Objectives | To document the: Different approaches to state administration of APS. Various approaches to conducting the basic functions of APS as outlined in the logic model. Variations in definitions of key concepts such as eligible clients; abuse, neglect, and exploitation; and timeliness of APS responses. | Establish a baseline of understanding about status of APS programs: Identify practice variations in serving older adults and adults with disabilities. Identify barriers to meeting policy mandates. Identify practice innovations or model programs that are designed to address such barriers or community-minded needs. | Assess the impact of policies, practices, and system-level outcomes. |
| Methods | Document reviews and NAMRS data analysis. | Surveys and NAMRS data analysis. | Create and analyze previously collected data and NAMRS data. |
What does an APS Logic Model look like?
Over the past several decades, state and local initiatives developed APS programs without a national framework or a national consensus about what adult maltreatment is, and what role government should have to assist victims. Lacking a unifying national framework, APS programs developed with variation in most aspects of programming and service delivery. A recent initiative of ACL, the Guidelines, is a step toward greater consistency among programs, but its impact has not yet been fully achieved.
The professional literature also reflects this lack of uniformity. An existing theoretical framework for conducting an evaluation was not identified by the APS TARC in preparing the evaluation plan. Consequently, one of the first tasks of the APS TARC was to develop a logic model to provide a theoretical framework for the evaluation. The APS Logic Model was drafted by the APS TARC and was reviewed informally by several APS administrators and researchers. Their comments were incorporated into the current model.
In developing the APS Logic Model, the APS TARC consulted a case flow diagram developed by the National Adult Protective Services Association (NAPSA). This case flow diagram portrays the major activities undertaken by APS agencies when investigating an allegation of maltreatment. It shows the characteristic steps in an APS investigation, beginning with the intake report and concluding with case closure.
The APS Logic Model elaborates upon this case flow and identifies results of standard APS activities, as well as the context under which these activities occur. The APS Logic Model is a depiction of the following elements of APS programs: context, inputs/resources, activities, activity metrics, and results. Activities, activity metrics, and results are divided into the typical case flow of intake (also often called prescreening), investigation, and post-investigation services. Quality assurance is also included and is comprised of a number of activities (e.g., documentation and supervisory review) that are critical aspects of APS programs.
The APS Logic Model provides a framework for identifying potential evaluation research questions, which are discussed in the following section. For a downloadable PDF version of this model, click here.
Last Modified: 11/30/2023