Research and Evaluation Strategy of the R&R Project
The project’s evaluation plan employs a mixed methods approach designed to gather formative, process and outcome evaluation data. These methods include the following:
- Embedded evaluations strengthen learning in the training, as well as provide feedback on curriculum effectiveness.
- Measures of satisfaction and learning provide an understanding of the experiences of supervisors, managers and directors who attend retention training.
- Transfer of learning (TOL) survey measures how and how often training is utilized in the workplace. Based upon the results of the TOL, individuals who have experienced high and low levels of transfer will be selected for follow-up interviews which will provide in-depth information about organizational factors that facilitate or impede application of the training.
- Toolkit evaluations measure the use and impact of recruitment and selection resources.
- Outcome evaluation assesses changes in key performance indicators over a period of time, using both administrative and survey data.
It is this outcome evaluation process that is the focus of this web page.
Understanding and Predicting Who Actually Leaves
The persistent shortage of a competent child welfare workforce has hampered the ability of agencies to provide effective services to families and children. Recruitment, selection and retention issues faced by child welfare agencies are numerous and readily acknowledged throughout the professional literature. Yet, there is a lack of rigorous research about the nature of the child welfare workforce crisis and initiatives to address the issues. The R&R Project is not only about training curriculum, recruitment tools and selection methods, it is also an experimental study about child welfare worker retention in North Carolina.
Project Participation by Random Assignment
Selection of counties for project participation was a two stage process. First, four of the state‘s seven geographic regions were randomly selected. These four regions contained 34 of North Carolina‘s 100 counties which were then randomly assigned to either an intervention group or a control group. The 17 counties in the intervention group (blue) participate in and implement the project‘s procedures—recruitment and selection activities, retention training and technical assistance. The control counties (yellow) are continuing with their original methods of recruitment, selection and retention.
Human Resources Database
Each project county maintains a web-based human resources database used to track actual child welfare worker turnover. The table below shows the combined turnover and vacancy rates for project counties in 2005 and 2006. County-specific tables show results on child welfare worker turnover and agency vacancy rates across child welfare positions in 33 county child welfare agencies.
| Year | Quarter | Number of Exiting Child Welfare Workers* | Turnover Rate for Participating Counties** | Average Number of Days Position is Vacant | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2005 | ||||||||||
| 1 Jan-Mar | 40 | 3.98% | 105 | |||||||
| 2 Apr-Jun | 80 | 7.95% | 83 | |||||||
| 3 Jul-Sept | 82 | 8.15% | 84 | |||||||
| 4 Oct-Dec | 50 | 4.97% | 93 | |||||||
| Overall | 252 | 25.00% | ||||||||
| 2006 | ||||||||||
| 1 Jan-Mar | 65 | 6.46% | 84 | |||||||
| 2 Apr-Jun | 76 | 7.55% | 55*** | |||||||
| 3 Jul-Sept | 46 | 4.57% | 28*** | |||||||
| Overall | 187 | 19.00% | ||||||||
Child Welfare Worker Survey
A web-based survey measures perceptions, characteristics and attitudes of child welfare workers shown in the literature to be related to retention. Workers are also asked how likely they are to leave their positions within the next six months (”intent to leave”). There have been 3 administrations of the child welfare worker survey to date. In June 2005, 716 child welfare workers from 33 project county departments of social services were invited to complete the R&R web-based survey. With a response rate of 49.5‰, 356 surveys were suitable for analysis. The second survey administration occurred in November 2005, with a similar response rate. In November 2006, 829 child welfare workers from 33 project counties were invited to complete the third administration of the child welfare survey. With a response rate of 46‰, 386 surveys were suitable for analysis. The results presented below are from this third administration.
Worker Survey Results
The survey measured workers’ perceptions about their work, supervisor, and agency, all shown in the literature to be related to intent to leave. These domains were in turn defined by 16 scales which measure specific aspects of a particular domain. The work domain is represented on the survey by the scales depersonalization, desire to help, self-efficacy, work load, match of skills to work, and realistic job portrayal. The supervisor domain is represented by scales describing the supervisor’s ability to provide practice support, team support, and emotional support. The agency domain is represented by scales relating to the agency’s public image, its negative image, the agency’s ability to provide worker recognition, salary and benefits, the clarity and visibility of the agency mission and the degree to which workers feel part of a joint decision making process in the agency. You can see graphs and descriptions for the aggregate mean scores (on a 6-point Likert scale) of 386 child welfare workers on the domains of work, supervisor and agency by clicking the links.
Findings from this research highlight the complex nature of child welfare worker turnover and have implications for such interventions as supervisor training, improved recruitment and selection methods and agency climate changes.
County Specific Results
On our password-protected Counties page these project-wide results are compared with individual county results from the worker survey and the HR database.
