On this page, jump to: Overview | Strategies | CQI Strategies | Resources
MODULE 1: Organizational Foundations for Successful Family Engagement and Partnership
Building an Outcome Culture of Care
Special Topic Overview:
- By developing a culture of evaluation and outcome measurement, residential organizations can ask critical questions about the process and efficacy of services. By using data to inform practice, organizations can answer questions such as:
- Do organizational practices put a priority focus on ensuring permanency for every child?
- Are organizational practices family-driven?
- Are organizational practices culturally responsive to the youth and families served?
- Are residential interventions leading to lasting, positive impacts for youth and families?
- Ongoing assessment and evaluation processes help an organization to assess the extent to which programs are adhering to and implementing best practices for permanency and family-driven care, which correlate to improved outcomes for youth and families. A culture of evaluation includes continuous quality improvement (CQI) and tracking the long-term outcomes of youth and families to measure the organization’s impact.
- Outcome measurement for residential intervention goes beyond studying youth gains within the residential program; it assesses the long-term impact of residential interventions in the lives of youth and families.
Strategies:
Commit to using data to inform practice. For residential organizations, a culture of evaluation includes continuous quality improvement (CQI), as well as evaluation focused on long-term outcomes for youth and families. CQI entails planning and implementing ongoing proactive improvements in processes of care by identifying, describing, and analyzing strengths and problems, and then testing, implementing, learning from, and revising solutions. Long-term outcomes reflect how youth and families are doing at home, in the community, and at school and/or work in the months and years following the residential intervention. The BBI Outcomes Monitoring Readiness can help organizations identify the strengths and needs in their organizational evaluation capacity.
Ensure that evaluation is a collaborative and inclusive process. Consistent with the principle of family engagement, all aspects of evaluation (CQI and long-term outcome tracking) are meant to be inclusive. Successful residential organizations engage staff at all levels, as well as family members, youth, Family/Parent Partners, Youth Partners/Peer Mentors, and other stakeholders in the evaluation process.
Module 1
KEY CONCEPTS & DEFINITIONS
Continuous quality improvement (CQI) is a structured organizational process that is dependent upon the participation of staff at all levels of the organization, as well as families, youth, and other stakeholders, especially in the areas of family-driven care and permanency. It entails planning and implementing ongoing proactive improvements in processes of care by identifying, describing, and analyzing strengths and problems, and then testing, implementing, learning from, and revising solutions. The strategies in this module address just two components of this comprehensive process: CQI as it pertains to permanency and family-driven care. There are many CQI areas of focus not addressed in this module. These may include specific focus areas chosen by individual organizations, such as addressing cultural and linguistic competence, as well as state licensing and regulatory requirements, such as tracking and reviewing incidents and safety measures.
A data dashboard is an information management tool that visually tracks, analyzes, and displays important performance indicators, metrics, and relevant insights from key datasets to monitor the health of an organization or program. Key residential data points specific to the focus of this family engagement toolkit include areas relevant to achieving permanency and successfully engaging, partnering with, supporting, and offering culturally responsive skill building for families. Data dashboards can also display information related to recidivism; seclusion and restraint; issues of diversity, equity and inclusion; results of follow-up studies; scores on instruments (such as the CANS/CAFAS/ROLES); intake/discharge data; and the financial health of the program.
Long-term outcomes refer to how youth and families are doing at home, in the community, and at school and/or work in the months and years following the residential intervention. Outcome measures most often capture how the youth is functioning, but it is also important to collect data regarding how parents are doing.
CQI Strategies:
Select metrics that can quantify the organization’s progress toward prioritizing permanency and becoming family-driven. BBI offers residential stakeholders several sample metrics that organizations can draw from to begin their journeys in assessing permanency and family-driven care practices. Organizations can set initial benchmarks when they begin to make improvements regarding successful engagement of families. For example:
- 95% of youth with no identified permanency resource at admission will have an identified permanency resource approved by the youth within six months post residential admission.
- 100% of youth will spend every holiday during their residential intervention at home or at the home of a family member.
- 95% of youth will spend time at home at least one time
- 95% of staff will complete at least three training modules, and will participate in supervision/mentoring regarding family-driven care practices and successful family engagement at least two times monthly, in a given year.
- 75% of families will have clinical staff provide support to them in their homes or communities at least once weekly
Use data dashboards to focus your organization’s priorities on permanency and family engagement. Use a data dashboard for important metrics for executive leaders to review weekly, such as:
- number of children and number of days for each child without a permanent family
- number and percentage of youth spending time weekly with families at home
- number and percentage of staff who represent the ethnicities and speak the primary languages of families served
- number and/or percentage of families receiving clinical support in their homes; and
- number and/or percentage of families for whom a comprehensive community support network is in place by three months post residential admission.
Build a feedback loop. In a data-driven organization, leaders can build an expectation of continuous quality improvement. Establish recurring processes by which key stakeholders (family, youth, Family/Parent Partners and Youth Partners/Peer Mentors, staff at all levels, leadership, board) review CQI data and have input into improvement strategies and follow-up actions.
For instance, within an organization’s strategic plan, each department and program can develop measurable goals, such as the examples above. Additionally, each staff member, in consultation with their supervisor, can also develop measurable personal goals, which can be formally reviewed a minimum of twice annually. Examples of personal goals for programs in the early stages of transitioning to a strong focus on family-driven care include the following:
- Clinical staff will spend a minimum of 50% of their time working with families in their homes and communities.
- Direct-care staff will call each family member of the youth for whom they are assigned as a “primary” at least five times weekly.
Managers are held accountable for their team results, and program directors are accountable for their department/program results. The CEO is accountable to the board and all staff for the performance of the whole organization. The CEO also reports the results of the agency’s evaluation findings through a quarterly review, with an end-of-the-year report summarizing the results of the annual operating plan.
Use parts of the BBI Self-Assessment Tool (SAT) specific to family-driven care to provide a mechanism for CQI of residential organizations. The BBI SAT provides a set of practice indicators in key domains relevant to residential interventions, such as Child and Family Teams, family-driven practices, cultural and linguistic competence, and other areas. The BBI SAT is designed to examine the supports and services youth and families are receiving through residential interventions and to inform key stakeholders (e.g., family members, youth, staff, leadership, community partners, etc.) about programmatic strengths and opportunities to improve.
Tracking Long-Term Outcomes:
The strategies below for long-term outcome tracking are focused specifically on measuring outcomes related to youth and families living together successfully in the community. However, there are a host of other outcomes that programs can track, based on their individual organizational goals for different years and contractual obligations, including educational achievement, job placement, and other areas.
Develop a data collection approach that is realistic and impactful. Post-residential outcome tools are best when they’re short and user-friendly, such that they can be administered to a family member or youth within a ten-to-fifteen-minute phone interview (as is the BBI Post- Residential-Discharge Survey) and administered at set intervals (e.g., three months, six months, and twelve months post residential discharge, etc.).
Include a focus on family functioning measures, as identified by each family. Consider using the Family Assessment of Needs and Strengths (FANS), especially the Family Peer Support Version. The FANS is utilized by a Family/Parent Partner with the parent(s); it serves as both an assessment of the parents’ needs and strengths as identified by the parent(s) and an outcome tool administered quarterly. The FANS allows families to identify and voice their own needs, versus solely tracking program-identified clinical measures. This creates a bridge for the Family/Parent Partner to work with other staff to coordinate support for the entire family on “real” identified needs and desired outcomes, as voiced by the family members themselves.
Consider correlating post-discharge survey data with family-driven care and permanency outcomes to determine practice level impact. The results of a post-discharge survey of how youth and families are doing can be correlated with data from a practice indicator survey (such as specific metrics chosen by the organization, results of the FANS, and/or parts of the BBI SAT) to assess the impact of changes at the practice level. For example, an increase in permanency practices will likely correlate with increased successful permanency outcomes for youth.
Develop a process to share findings on long-term outcomes. When developing a culture of evaluation, successful organizations develop a process for sharing data with all staff, families, youth, and other stakeholders to reflect on what is working well or less well for families in the long term. This demonstrates an organization’s accountability and commitment to family- driven care, as well as the organization’s urgency toward the outcome of permanency. Consider holding data meetings that are engaging, with a meal, to foster a comfortable atmosphere and to convey a sense of gratitude for everyone’s efforts.
Additional Recommendations for Evaluation:
Develop a culturally and linguistically competent process. It is critical for organizations to ensure that all evaluation instruments (e.g., surveys, interviews, etc.) are culturally appropriate for the family. Interviewers should be available to conduct the post-residential survey or interview in the family’s spoken language. It is important for culturally diverse families to be involved in selecting and reviewing measurement tools to ensure cultural relevance and responsiveness.
When planning for evaluation (CQI and long-term outcomes tracking), consider how staff will be allocated to data collection efforts. Ideally, to allow for sufficient capacity, several staff, including Family/Parent Partners, would be trained and supervised on how to administer the phone interviews. Interviewer training includes cultural sensitivity, how to build rapport, how to ask the questions accurately, and how to recognize needs that families express by phone that may necessitate referrals. The BBI Post-Residential-Discharge survey contains a sample script to assist staff with the phone interview process. By providing flexibility in the timing of calls, providers can reach more families. An organization may also consider hiring an external evaluator to lead evaluation activities; external evaluators also bring expertise with analysis and improvement recommendations.
Obtain informed consent upon admission to residential intervention. Residential organizations can make a practice of obtaining, at admission, informed consent from family members regarding post-residential-evaluation phone surveys. Staff can describe when families can expect to receive follow-up calls and can answer any questions from the family about their voluntary participation.
Consider providing incentives to families for completing surveys/interviews. Providing incentives to youth and families for completing surveys or interviews is a way to convey respect for their time and effort. A small monetary incentive or gift card often helps increase the rate of response.
Resources:
The BBI Outcomes Workgroup has developed a framework and set of resources for assessing residential intervention practices and outcomes. The framework consists of practice indicators, a tool for measurement, with accompanying instructions; recommendations for measuring outcomes; a post-discharge survey and study of feasibility of collecting post-discharge outcomes; and a readiness assessment to help guide organizations in developing the infrastructure for outcomes measurement and instrumentation. Specific resource materials for the framework are listed below and are available (in many cases, in Spanish as well as English) on the our website.
Practice Indicators and Measurement
Performance Guidelines and Indicators Matrix. (2008). Building Bridges Initiative.
Building Bridges Initiative Self-Assessment Tool (BBI SAT). [Spanish version forthcoming.] (2020, September). Building Bridges Initiative.
Outcomes Measurement
Dougherty, R., Strod, D., Fisher, S., Broderick, S., & Lieberman, R.E. (2014). Tracking long-term, strength-based outcomes. In G.M. Blau, B. Caldwell, & R.E. Lieberman (Eds.), Residential Interventions for Children, Adolescents, and Families: A Best Practice Guide (pp. 182–194). Routledge.
Weiner, D., Thompson, R., & Alexander, M. (2020). Measuring the impact of residential interventions. In B. Caldwell, R. Lieberman, J. LeBel, & G. M. Blau (Eds.), Transforming Residential Interventions: Practical Strategies and Future Directions (pp. 235–253). Routledge.
Post-Residential-Discharge Measurement Feasibility Study and Survey Tool
Post-Residential-Discharge Measurement Survey. (2018). Building Bridges Initiative.
- Caregiver survey; English: Click here.
- Caregiver survey; Spanish: Click here.
- Young Adult survey; English: Click here.
- Young Adult survey; Spanish: Click here.
Additional Resources
Praed Foundation (2018). Family Assessment of Needs and Strengths (FANS)¾Family Peer Support Version.
(For more information on the FANS, contact John Lyons or Suzanne Button at info@praedfoundation.org; https://praedfoundation.org; or Nancy Craig, Families Together in NYS, ncraig@ftnys.org, www.FTNYS.org.)