Social behavioral scientists are increasingly employing data collected by multiple persons (raters) in longitudinal research. However, rater effects (bias) in such studies are often overlooked. Many studies using multiple-rater data assume no or negligible measurement errors in the ratings, and apply statistical methods that lead to misleading and biased results.
The consequence of ignoring rater effects is significant. For example, the evaluation of the Social and Character Development (SACD) program, a project of more than $34 million funded by the U.S. Department of Education and the Centers for Disease Control and Prevention, did not take into consideration the fact that students’ behavioral ratings were made by more than one teacher. As a consequence, the SACD evaluation (IES, 2010) did not confirm that the intervention was effective; however, this finding is flawed due to the failure to control for rater effects in data analysis.
Shenyang Guo, Wallace H. Kuralt Distinguished Professor at the UNC School of Social Work, and Kenneth Bollen, H.R. Immerwahr Distinguished Professor of Sociology at UNC, recently examined this important methodological issue and developed new methods to correct for rater effects to enhance analytic rigor and accuracy of longitudinal research.
Their study has resulted in two peer-reviewed publications:
- Guo, S., & Bollen, K. A. (2013). Research using longitudinal ratings collected by multiple raters: one methodological problem and approaches to its solution. Social Work Research, doi: 10.1093/swr/svs029.
- Guo, S. (2013, Forthcoming). Correction of rater effects in longitudinal research with a cross-classified random effects model. Applied Psychological Measurement.
Guo and Bollen’s study reiterates the importance of using effective and robust methods to correct for measurement error in longitudinal research. It shows that longitudinal ratings without Rater IDs have little, if any, usefulness.
