Evaluation of Adult Protective Services

The APS TARC is conducting the first-ever process national evaluation of adult protective services programs. This evaluation will provide the foundation for ongoing and future technical assistance efforts. To learn more about this important effort, review the information below.

• Why Evaluate APS?

  The Administration for Community Living (ACL) has created a framework for improving state and local adult protective services (APS) programs throughout the country. The framework consists of a complementary set of initiatives, centered on giving APS jurisdictions tools and information to enhance the effectiveness of their programs. With an estimated one in 10 of older adults being at risk of abuse, neglect or exploitation, APS programs play a critical role in protecting and sustaining the health and well-being of all vulnerable adults, including adults with disabilities and older Americans.
  
  There are many similarities and differences across state and local APS programs. These similarities and differences are not well-documented and even less understood. This lack of knowledge and understanding hampers efforts to enhance the effectiveness of APS programs. This means that increasing knowledge and understanding should be a priority for investing in APS program improvement. Without this knowledge and understanding, it is very difficult for the government to make informed decisions about priorities, policies, and programs.
  
  Program evaluation provides a rigorous and consistent method for transforming information based on data into evidence based knowledge. ACL can help all APS agencies to improve the efficiency and effectiveness of their programs through a program evaluation that maximizes existing resources and is designed in a building block manner to develop a firm foundation of knowledge and understanding.
  
  The evaluation approach being used - a process evaluation - applies a rigorous APS theoretical framework to comprehensively analyzing the major functions of APS: intake, investigation, post investigation, and quality assurance. These functions will be examined through the domains of policy, practice, personnel, partners, and performance.

  The proposed evaluation plan addresses several major gaps in knowledge and understanding of APS:

  1. What policies are currently in place in each U.S. state and territory?
  2. What are the similarities and differences among policies across the states and territories?
  3. What practices are being used? Which practices are innovative?
  4. What are the specific outcomes of APS practices, and do they vary by type of practice?
  5. Are certain policies and practices associated with different outcomes for aging adults and for adults with disabilities?

  Increasing knowledge and understanding through evaluation is necessary to help improve the effectiveness of APS programs in all major program administration areas.

  Program Planning: An APS program evaluation will provide federal, state, and local governments with information they need to wisely invest resources in improving APS programs. Government officials need to know what aspects of APS programs need improvement.
  
  Effectiveness: An APS program evaluation will help APS program administrators understand what approaches and interventions add the most value in achieving positive client outcomes for APS clients. The evaluation will establish the guide for future research efforts into best practices and service delivery innovation.
  
  Quality Improvement: By establishing a theoretical framework, an APS program evaluation will provide program administrators with information they need to establish quality improvement processes. It will inform future technical assistance by the APS TARC.
  
  Outcome assessment: Development of a theoretical framework, as part of an evaluation of APS, laid the groundwork for future definition, measurement, and assessment of outcomes of APS services through a separate client outcome evaluation. 

  One leading program evaluation expert says that evaluation answers three questions:

  1. What? 
  2. So What?
  3. Now What? 

  He adds that the purpose of evaluation is to inform thought and action. It is a means to move from data to interpretation to action; and to move from acting on belief and hope to acting on data and feedback. ¹
  
  The APS program evaluation aligns closely with these concepts. There is a fundamental need to better understand What APS does and how it does, which is addressed in the first component. Beyond understanding, there is a need to answer the So What question in APS—how do various practices and innovations make a difference in the lives of APS clients, which aligns with the second component. Finally, the third component will provide a rich resource for answering the all-important question of Now What—that is, what are the recommendations to enhance the effectiveness of APS programs.
  
  In summary, the answer to the question of “why evaluate APS services?” is:

  • Program Improvement: Knowledge is a pre-requisite for program improvement. There is a lack of fundamental knowledge and understanding about the nature and scope of APS programs. Knowledge and understanding should precede investment.
  • Initiative Improvement: An evaluation of APS complements other federal efforts, particularly NAMRS and the Guidelines to improve APS services. Data is useful to the extent you know the right questions to ask of it and guidelines are useful when supported by evidence.
  • System Improvement: Program evaluation will create the framework and knowledge base to move the APS system forward. In particular, it will inform the nature and types of technical assistance that the APS TARC will provide and the types of systems improvement ACL should foster in the future.
  • Impact Improvement: Within current federal resources and authority, program evaluation is currently the best way for ACL to have a long-term impact in improving APS programs.

  ---

  
  ¹ Michael Quinn Patton, Essentials of Utilization-Focused Evaluation, p. 3. Sage Publishing, 2012.
• How do you evaluate APS?

  The Administration for Community Living (ACL) established the Adult Protective Service Technical Assistance Resource Center (APS TARC) to enhance the effectiveness of APS programs. As a first step in that process, the APS TARC is conducting the first-ever national evaluation of APS. The following provides an overview and approach of the plan. 

  The plan:

  • Includes an APS Logic Model to use as a theoretical framework for conducting the program evaluation.

  • Includes a compilation of potential research questions, based on the APS Logic Model.

  • Presents a study comprised of three components to evaluate APS. A phased or stepwise approach was used to address the fact that there is currently only a basic understanding of the services provided by APS in each state, with little knowledge about the effectiveness of these services at reducing abuse and preventing re-referral to the APS system.

  The APS Logic Model is a one-page depiction of the following elements of APS programs: context, inputs/resources, activities, activity metrics, and results. Activities, activity metrics, and results are divided into the typical case flow of intake (also often called “prescreening”), investigation, and post-investigation services. Quality assurance is also included and is comprised of a number of activities (e.g., documentation and supervisory review) that are critical aspects of APS programs.
  
  The identification of research questions is a critical step in the development of a program evaluation plan. Research questions help formulate what will be studied and evaluated, and provide a tool for prioritizing topics to be studied. The APS TARC went one step further and elaborated upon five areas or constructs of program operations, which were used to organize the research questions. The five constructs are policy, practice, personnel, partners, and performance. The constructs are defined in Exhibit A.

  Exhibit A - Program Operation Constructs

  Construct	Definition
  Policy	Policy consists of the mission, goals, and objectives of a program. Often, policy also provides guidance or requirements in terms of procedures. Included in policy may also be requirements for staff and funding, and cooperation with other agencies. Formal policy is always written and found in state statutes, administrative code, and agency manuals. Policy may be created by the legislative or judicial branches of government, or by the executive branch.
  Practice	Practice entails those activities and actions of APS agency staff that implement policy or other management expectations.
  Personnel	Personnel includes the qualifications of staff, as well as the hiring, training, and performance review of staff, which are implemented based upon policy.
  Partners	Partners are other agencies (governmental, non-governmental, and private) that support the program or program’s clients.
  Performance	Performance refers to the measurement of activities and outcomes.

  To develop the research questions, these constructs were cross-referenced with the critical elements of the APS Logic Model (context, intake, etc.) and research questions developed for each intersection. The result is a comprehensive set of potential broad research questions to guide the evaluation. Ultimately, based on feedback from ACL and development of research hypotheses, the research questions were refined and focused on specific areas of inquiry consistent with the selected research methodology. Exhibit B show a summary of the cross-referencing and how it is addressed in the evaluation plan.

  Exhibit B - Cross Referencing

  Construct	Definition	Key Questions	Methods	Component
  Policy	Policy is found in state statutes, administrative code, and agency policy manuals. All policy is written.	What policies direct key program activities?	Document review of extant state codes, regulations, and policy manuals and analysis of initial year NAMRS data (Agency Component).	1
  Practice	Practice entails those activities and actions of APS agency staff to implement policy or other management expectations.	How are critical activities conducted?	Surveys of APS staff, on-site focus groups, and analysis of NAMRS data.	2
  Personnel	Personnel includes the qualifications of staff, as well as the hiring, training, performance review of staff.	How do APS programs employ and train staff and what are key activities related to staff effectiveness?	Document review of extant state codes, regulations and policy manuals and analysis of initial year NAMRS data (Agency Component). Survey of APS staff, on-site focus groups, and analysis of NAMRS data.	1, 2
  Partners	Partners are other agencies (government, non-governmental, and private) that support the program or program's clients.	Who are the critical partners of the APS agency and what type of assistance do they provide?	Surveys of APS staff, on-site focus groups, and analysis of NAMRS data.	2
  Performance	Performance refers to the measurement of activities and outcomes.	How do APS programs measure activity metrics and outcomes? What are the outcomes for APS activities?	Document review of extant state codes, regulations, and policy manuals and analysis of intial year NAMRS data (Agency Component). Surveys of APS staff, on-site focus groups, and analysis of NAMRS data. Analysis of state-level NAMRS data sets.	1, 2, 3

  The APS program evaluation is comprised of a three-component study.  The first two components are process evaluations, which build towards a more system outcome-oriented evaluation. The process-oriented components start with a recognition, discussed with ACL, that knowledge-building is the first type of evaluation to undertake when evaluating APS. Out of this knowledge base, more sophisticated, system outcome-oriented evaluations can be undertaken. Given this, the three evaluation components were developed. A summary of these components are as follows:

  • The first component is a detailed examination of extant policies of state APS agencies, which is needed for foundational knowledge-building. This consists of documenting state policies pertaining to APS programs, and developing and comparing state profiles. The emphasis of this component is: what are APS policies? This report is available in two volumes: Volume I and Volume II. 

  • The second component, a comprehensive inventory of state agency APS practices of delivering client services, collected more specific information about how services are actually delivered and the parameters and restrictions on the scope of client services. This resulted in an environmental scan of state practices under each of the major stages in the APS Logic Model: intake, investigation, post-investigation services, and quality assurance. The emphasis of this component is: what activities are conducted by APS and how are they conducted?

    • The third component examines state patterns and relationships of APS program characteristics, key policies and practices, and APS system-level outcomes and to gain an understanding of what impacts program effectiveness. The study team computed descriptive statistics for the administrative structures, policies, and practices that characterize each state’s APS program (predictor variables) and system-level outcomes. The study team also conducted an exploratory latent class analysis to examine whether groups of states can be classified into distinct subtypes with similar policies and practices.
      
      Exhibit C provides a summary of the components.

      Exhibit C - Summary of Study Components

      	Component 1	Component 2	Component 3
      Objectives	To document the: Different approaches to state administration of APS. Various approaches to conducting the basic functions of APS as outlined in the logic model. Variations in definitions of key concepts such as eligible clients; abuse, neglect, and exploitation; and timeliness of APS responses.	Establish a baseline of understanding about status of APS programs: Identify practice variations in serving older adults and adults with disabilities. Identify barriers to meeting policy mandates. Identify practice innovations or model programs that are designed to address such barriers or community-minded needs.	Assess the impact of policies, practices, and system outcomes.
      Methods	Document reviews and NAMRS data analysis.	Surveys and NAMRS data analysis.	Create and analyze previously collected data and NAMRS data.

• What does an APS Logic Model look like?

  Over the past several decades, state and local initiatives developed APS programs without a national framework or a national consensus about what adult maltreatment is, and what role government should have to assist victims. Lacking a unifying national framework, APS programs developed with variation in most aspects of programming and service delivery. A recent initiative of ACL, the Guidelines, is a step toward greater consistency among programs, but its impact has not yet been fully achieved.
  
  The professional literature also reflects this lack of uniformity. An existing theoretical framework for conducting an evaluation was not identified by the APS TARC in preparing the evaluation plan. Consequently, one of the first tasks of the APS TARC was to develop a logic model to provide a theoretical framework for the evaluation. The APS Logic Model was drafted by the APS TARC and was reviewed informally by several APS administrators and researchers. Their comments were incorporated into the current model.
   
  In developing the APS Logic Model, the APS TARC consulted a case flow diagram developed by the National Adult Protective Services Association (NAPSA). This case flow diagram portrays the major activities undertaken by APS agencies when investigating an allegation of maltreatment. It shows the characteristic steps in an APS investigation, beginning with the intake report and concluding with case closure.
   
  The APS Logic Model elaborates upon this case flow and identifies results of standard APS activities, as well as the context under which these activities occur. The APS Logic Model is a depiction of the following elements of APS programs: context, inputs/resources, activities, activity metrics, and results. Activities, activity metrics, and results are divided into the typical case flow of intake (also often called prescreening), investigation, and post-investigation services. Quality assurance is also included and is comprised of a number of activities (e.g., documentation and supervisory review) that are critical aspects of APS programs.
   
  The following description and assumptions explain the APS Logic Model:

  • The model is focused primarily on APS client services. It does not include other program activities such as public awareness campaigns or budget planning. The chart includes elements related to APS investigations of providers or facilities, but is not an exhaustive list of potential provider investigation activities conducted by some APS programs or licensing and regulatory agencies.

  • The chart shows the typical stages of an APS case in the activities, activity metrics, and results columns. The overall case stages are from the top of the column to the bottom. It is recognized that actual activities, depending on the program and case, may occur in different boxes than shown. For example, case initiation activities in some programs may be performed as part of intake and not as part of the investigation.

  • The model represents an overall depiction of elements of APS programs, but no program is expected to include all elements. Specific state processes will differ. For example, some APS programs only investigate allegations and do not provide services. Policies may differ across jurisdictions regarding an alleged victim’s right to refuse an investigation or services. Some APS programs have funding to purchase services for victims as part of their program budgets, while many do not or the funding may be insufficient. Long term post-investigation management of guardianship cases is not included in this model. Consultative experts can be internal or external to a program.

  • The listed activity metrics are associated with the activities column and are not a comprehensive list of potential metrics for APS programs. The results column does not list “outcomes” or “impact,” which are often included in logic models; instead, it more definitively and concretely lists results of the items in the activities column.

  • Finally, the chart is generally consistent with the Guidelines and with terminology used in NAMRS.

  The APS Logic Model provides a framework for identifying potential evaluation research questions, which are discussed in the following section. For a downloadable version of this model, click here.
  
  

   

  ---

  Several acronyms are used throughout the logic model: ANE=abuse, neglect, exploitation; AV=alleged victims; CV=confirmed victims; AP=alleged perpetrator; MDT=multi-disciplinary team