Interrater Agreement of the Group Care Quality Standards and the Group Care Quality Standards Assessment

The Family First Prevention Services Act (Family First) emphasizes the importance of children growing up in families and when foster care is needed, being placed in the least restrictive, most family-like setting appropriate to their special needs. As a result, Family First limits the use of group care to Qualified Residential Treatment Programs, which are required to demonstrate family engagement, use a trauma-informed treatment model, and to be licensed and accredited. However, existing measures of quality of group care are limited and mostly not quantifiable.

This study is part of a research project to develop the Quality Standards for residential care and the Quality Standards Assessment (QSA), a quantifiable measure of residential care quality. Developed by researchers and practitioners in Florida in 2014, the QSA contains 8 domains, each of which has 4-20 items. The domains include: 1) Assessment, Admission, and Service Planning, 2) Positive, Safe Living Environment, 3) Monitor and Report Problems, 4) Family, Culture, and Spirituality, 5) Professional and Competent Staff, 6) Program Elements, 7) Education, Skills, and Positive Outcomes, and 8) Pre-Discharge/Post Discharge Processes. Different versions of the QSA were to be completed by youth, group care providers (i.e., directors and director care workers), lead contract agency staff (e.g., case managers, contract managers, placement coordinators), and state licensing specialists/external auditors.

In this study, we evaluated interrater agreement (IRA) of QSA. IRA is the absolute consensus in rating scores from multiple raters on the same targets. High IRA justifies aggregation of scores from multiple raters.