FAMILY & COMMUNITY SUPPORT

Proposed measures:

https://a257.g.akamaitech.net/7/257/2422/01jan20051800/edocket.access.gpo.gov/2005/
pdf/05-22095.pdf (PDF file; Requires Adobe Reader.)

The Child Welfare League of America (CWLA) and the National Working Group to Improve Child Welfare Data, representing all 51 states, are pleased to respond to the proposed CFSR measures outlined in the Federal Register of November 30, 2005.

Part 1: General Comments

CWLA and the National Working Group support the use of composite measures to assess the general bureaucratic function of child welfare systems, accounting for the interaction of multiple indicators. We further support the Children’s Bureau’s plan to perform analyses regarding the viability of inclusion of specific performance areas.

However, CWLA and the National Working Group have several concerns with the composites as presented in this notice. First, CWLA strongly recommends that the Children’s Bureau provide a more detailed notice and further opportunity to respond prior to implementing the composites, given the many unanswered questions and short time frame provided for this notice. A more detailed notice would include results of the aforementioned analyses, more details about the indicators, external validity criteria, methodology for calculating and applying the composites, and the threshold for the standards (e.g. percent that must be met). CWLA also recommends that the indicators and composites be pilot tested prior to being implemented as part of the CFSR review.

Second, CWLA notes that the addition of composite measures should not shift the focus from actual child outcomes. The Child and Family Service Reviews were designed to promote continuous improvements in programs, policies and services. The addition of new indicators is consistent with this goal, allowing for a more holistic view of child outcomes. In order to improve child welfare services, states should be supported as they track and work to improve their performance on a range of indicators. Putting too much emphasis on composite scores could detract from the important focus on improving child outcomes.

Third, CWLA and the National Working Group are concerned that the definitional discrepancies among states still need to be addressed in order to make the National Standards more meaningful. National data are an important source of information as states monitor their performance and strive to improve. The National Standards would become a more effective tool to gauge the performance of child welfare agencies in achieving positive outcomes for children if we could improve definitional consistency and develop evidence-based benchmarks. Part 2 of this response describes our concerns and some suggestions to improve quantitative evaluation of states.

Finally, the performance areas described in the notice represent important factors in a child’s safety and permanency, yet some of the indicators do not represent reliable, valid measures of the topic described. First, now that it is possible to link cases across reporting periods, CWLA advocates for linking AFCARS files and using entry cohorts for some of the indicators. Second, as noted in the advisory meetings last year, there needs to be a differentiation between older and younger children in some cases (such as placement stability). There also needs to be a way to differentiate child populations (e.g. child welfare, juvenile justice, mental health) or to evaluate states separately when they serve different populations. This is important, for instance, when assessing re-entry rates in a state that serves the juvenile justice population, in which re-entries are beyond control of the agency. In addition, some measures need to have the denominator more clearly defined (e.g., timeliness of adoptions should consider cases with the goal of adoption). Further, some areas are better suited to analysis over time within a state to identify and address the true problem areas. Finally, not all domains may be suitable for composite measures at this time. CWLA’s thoughts on the specific performance areas for each composite are offered below in Part 3.

Finally, CWLA offers thoughts on the definitional issues, methodological approach and need for further training and technical assistance in Part 4.

Part 2: National Standards

The federal register notes “National standards will be developed for each of the domains represented by the six data composites.” National data are an important source of information as states monitor their performance and strive to improve. However, CWLA is concerned that the issues plaguing the current standards not be repeated in a new set of standards.

First, CWLA is concerned that differences in state definitions diminish the reliability of cross-state comparisons using child welfare data. CWLA and the National Working Group to Improve Child Welfare Data have implemented three surveys that provide evidence of definitional inconsistencies among states in the areas of placement stability, child maltreatment in foster care, and time to reunification. These inconsistencies are outlined in Part 4. Although CWLA, the state child welfare agencies, the Children’s Bureau and others are working to achieve more consistency, this effort takes considerable time. CWLA did note that some of the proposed measures attempt to address the inconsistencies, but others will take time and effort over the course of several years.

Second, CWLA is concerned that some of the standards were not developed in the context of federal definitions. For instance, the current standard for reunification, 76.2% or more reunification in < 12 months, is set too high considering that the AFCARS definitions consider a child discharged after a “trial home visit” (after end of legal custody and supervision), not once the child is physically returned home, and children may be on trial home visit for six months or more in some states.

Third, the current federal standards do not reflect an evidence base for good practice. States wish to improve their services and outcomes for children by setting goals that are based in good practice. Developing a 75th percentile score based on data from some (or all) states does not necessarily reach that goal, particularly given the definitional issues that persist.

The National Standards would become a more effective tool to gauge the performance of child welfare agencies in achieving positive outcomes for children if we could improve definitional consistency and develop evidence-based benchmarks. In this light, CWLA makes the following recommendations:

• Emphasize state performance over time, comparing a state to itself, as done in the federal Program Improvement Plans. Given the definitional inconsistencies (as well as differences in populations served), it is more meaningful to work with states in this manner to achieve improved results for children in care in this manner.
• For now, use national data as an important reference, but not to judge or penalize a state given the definitional problems that persist. The danger is that sanctions may be applied when the selected data are not indicative of good or bad practice, but of variations in definitions.
• Increase federal support of efforts to improve data quality and definitional consistency among states. This includes funding for states to alter their data systems to become more consistent, as well as support of state-driven efforts to reach more consistency.
• Develop performance benchmarks that are consistent with the evidence base for good practice. Where evidence is lacking, support further research to build an appropriate evidence base. Pilot test the indicators and benchmarks before implementing them across all states.
• Compute future standards in the context of federal definitions, as described earlier.

Part 3: Performance Areas

The performance areas described in the federal register represent important factors in a child’s safety and permanency, yet some of the indicators do not represent reliable, valid measures of the topic described. First, now that it is possible to link cases across reporting periods, CWLA advocates for linking AFCARS files and using entry cohorts for some of the indicators. Second, as noted in the advisory meetings last year, there needs to be a differentiation between older and younger children in some cases (e.g. placement stability). There also needs to be a way to differentiate child populations (e.g. child welfare, juvenile justice, mental health) or to evaluate states separately when they serve populations other than child welfare, such as juvenile justice. This is important, for instance, when assessing re-entry rates in a state that serves the juvenile justice population. Further, some measures need to have the denominator more clearly defined (e.g., timeliness of adoptions should consider cases with the goal of adoption). Finally, some areas are better suited to analysis over time within a state to identify and address the true problem areas.

Safety Composite 1, Recurrence of Maltreatment.

Performance Area 1: Recurrence of Substantiated or Indicate Maltreatment Reports.
CWLA supports the continued use of this indicator as a rough estimate of recurrence. However, we note that there continue to be concerns regarding data quality (e.g. multiple reports of same incident, reports occurring a long time after the incident, and the use of report date in lieu of incident date in this measure) and the absence of consideration of recurrence after longer periods of time. Further, an increasing number of states are implementing alternative response systems. States vary in their use of disposition categories and identification of “victims” or “non-victims” in these cases, which raises the question of comparability of the maltreatment data.

Performance Area 2: Multiple Unsubstantiated Maltreatment Reports
Multiple unsubstantiated maltreatment reports should be eliminated from consideration as an indicator for a composite measure. CWLA shares some of the concerns ACF listed in the register. In addition, we are concerned that differences in state statutes, definitions, criteria for substantiating child abuse and neglect, and variations in the use of disposition categories make the unsubstantiated category invalid for cross state comparison. Further, it seems that the measure makes states accountable for following their own laws and rules for substantiation.

This performance area represents a significant issue that should be reviewed. Given the variations among states, CWLA recommends that multiple unsubstantiated reports be reviewed as part of the on-site review, in addition to internal state examination of the issue.

Performance Area 3, Timeliness of Initiating Investigations of Child Maltreatment Reports
Timeliness of investigations is an extremely important practice issue that should be monitored closely and improved upon within states. However, CWLA and the National Working Group agree with the concerns ACF expressed about this indicator in the federal notice. Further, while 24 hours is appropriate for cases in which the child’s safety is at risk, it would be difficult to measure this in the context of all cases. Variations in screening procedures, which may cause one state to include a case and another to exclude it, would affect the proportion of cases that should be seen within 24 hours. In a state that casts a wide net it may be appropriate to initiate a larger percentage of cases after 24 hours. Timeliness should be measured within states in the context of their laws, policies, and priority systems.

Performance Area 4, Timeliness of Dispositions of Child Maltreatment Reports
CWLA recommends that this indicator not be used in the composite. This measure does not seem to reflect a state’s protection of a child, since states often take action to protect the child’s safety prior to reaching a disposition. In addition, some states may provide services while an investigation remains open and before reaching disposition. Finally, this measure has the potential to reward poor practice if workers reach a disposition without adequate investigation.

Safety Composite 2: Child Maltreatment in Foster Care

Performance Area 1, Maltreatment In Foster Care By A Foster Care Provider Or Facility Staff Member
CWLA is pleased by the inclusion of the AFCARS identifier in the NCANDS file, which will help distinguish children maltreated in foster care from those maltreated in other systems. However, we are concerned about two issues that affect this measure. First, as mentioned in the notice, the perpetrator category used for relative foster care providers (relative vs. foster care provider) varies among and within states. Second, the use of report date as a proxy for incident date makes the number unreliable (see below).

Performance Area 2, Maltreatment In Foster Care By Their Parents 
This indicator represents an important area for analysis, but is not appropriate for inclusion in a national standard. The numbers would be quite low, which lends itself to fluctuation and makes it very difficult to demonstrate improvement. Also, the use of report date as a proxy for incident date is of concern because a child in foster care may reveal abuse that occurred prior to coming into care (see below). Further, some states are concerned that the court may order visitation with parents against the child welfare agency’s recommendation, and the child welfare agency should not be held accountable for this.

If this indicator is used, CWLA and the National Working Group suggest several changes. First, consider broadening the category to include relatives in addition to parents. This change reflects a fuller picture of family perpetrators and helps capture relative foster care providers who are either not licensed or not recorded as foster parents in the investigation process. Second, the indicator may fit better in Safety Composite 1, since the parents are perpetrators and since the children may actually be living at home with parents but included in the AFCARS foster care population based on AFCARS’ definition of “trial home visit”.

Incident Date
As mentioned in the notice in the federal register, the report date is used as a proxy for incident date. This is a concern for measurement of maltreatment in foster care since some abuse may be discovered and reported well after the time of the incident. Using the report date, it may appear that abuse occurred while the child was in foster care rather than while the child was still in the home. Likewise, it may appear that abuse occurred once the child was returned home, when in fact it occurred while the child was in foster care. In either case, the inaccuracy of the information may provide misleading data.

Permanency Composite 1 – Timeliness and Permanency of Reunifications

Performance Area 1, Timeliness to Reunifications of children Exiting Foster Care
CWLA recommends that this indicator be adjusted to measure entry cohorts, now that it is possible to link AFCARS files to create a longitudinal file. This would allow for a better measure of current policy and practice, which exit cohorts cannot capture. Whether or not this is done, CWLA further offers the following comments:

CWLA endorses the attempts by the Children’s Bureau to propose indicators that address the definitional issues caused by state differences in use of protective custody and trial home visits, as shown in bullets 2 and 4, and that views reunification from the perspective of the child (bullet 4, trial home visit). CWLA supports the use the 7 day time frame (bullet 2) rather than the 30 days timeframe (bullet 3) in order to account for differences in state use of temporary protective custody without negating state efforts to return children quickly to their families.

CWLA advocates for two separate indicators to assess timeliness to reunification in order to accommodate differences in state use of trial home visits. The first would determine the percent of children who were either placed on “trial home visit” (physically reunified) or discharged to reunification (or “living with other relatives”) in less than 12 months from the latest removal from home, as seen in bullet 4. The second would determine the percent of children discharged to reunification (or “living with other relatives”) in less than 12 months from the time of the latest removal from home. Both would focus on children in care for more than 7 days, as seen in bullet 2. With these two indicators (instead of one) the composite evaluates states on a combination of time to the child’s physical return home (reunification from the child’s perspective) and time to the end of the state’s legal custody of the child (discharge from AFCARS).

CWLA notes that the proposed indicator that uses the AFCARS placement setting “trial home visit” is not meaningful for a small number of states that include children who are home on brief visits (e.g. weekend) and do not use this term to mean children who have physically returned to live at home. CWLA also notes that there continues to be some confusion among states as to the use of the AFCARS term “trial home visits” and how it relates to reunification. The proposed indicators would be improved by modifying AFCARS to capture a separate date for physical return and the end of legal custody and supervision, rather than relying on the placement setting “trial home visit.” States would need some support to adjust their systems and reporting for this purpose.

Finally, CWLA advocates the use of percent rather than median in order to be consistent with the indicator in area 2.

Performance Area 2, Timeliness to Reunifications for children Entering Foster Care 
CWLA commends the Children’s Bureau on proposing an indicator that factors in recent implemented state efforts to reunify children in a timely manner. However, the proposed indicator does not account for trial home visits. CWLA advocates for a similar indicator that counts children who have been discharged or placed on trial home visit, as discussed above. The seven day timeframe should be used here also. The indicator would be further improved by focusing on children with a case plan goal of reunification.

Performance Area 3 – Permanency of Reunifications
In general, CWLA supports the Children’s Bureau’s inclusion of reunification and re-entry in the same composite. We further support the effort to link children across years to capture children re-entering foster care.

CWLA does have several concerns regarding this measure. First, if the timeliness indicators use the date placed on trial home visit to equate to a discharge, then the re-entry calculation will also need to take this into consideration.

Second, this indicator does not take into consideration the age or population, and how that effects re-entry rates. Children on probation and children that come into care because of behavioral issues may impact the re-entry data. This is of greatest concern in states that include juvenile justice children in their AFCARS population. This indicator could be improved by focusing on child welfare children (excluding juvenile justice youth) and/or focusing on younger children (excluding teenagers).

Permanency Composite 2, Timeliness of Adoptions

Performance Area 1, Timeliness of Adoptions of Children Discharged From Foster Care to a Finalized Adoption
CWLA recommends that this indicator be adjusted to measure entry cohorts, now that it is possible to link AFCARS files to create a longitudinal file. This would allow for a better measure of current policy and practice, which exit cohorts cannot capture. We also note that more emphasis should be placed on working with individual states to identify what contributes to the state’s score on the measure. It would be particularly useful to view the data by age group, race/ethnicity, and relative vs. non-relative adoptions.

Also, CWLA supports the use of an indicator that measures median time to adoption (in months). If the median is not used, we support the use of a broader timeframe for the adoption indicator, as demonstrated in the second bullet.

Performance Area 2, Timeliness of Adoptions of Children who are in Foster Care for 17 months or longer
CWLA supports the addition of an indicator that focuses on children who are adopted after having been in care for longer periods of time. This is particularly important to capture positive results from adoption initiatives focusing on children who have been in care for longer periods of time. However, this indicator may be improved by focusing on children with a goal of adoption. Also, as noted above, CWLA advocates the use of entry cohort data to measure time to adoption. Finally, a different timeframe should be considered to avoid the overlap with Performance Area 1.

Performance Area 3, Timeliness of Adoptions of Children for Whom Parental Rights Terminated
CWLA supports the idea that permanency should be reached especially quickly for children once parental rights have been terminated. However, this indicator should not be used unless several concerns are addressed: 1) add voluntary relinquishment to the indicator, as this occurs fairly often instead of TPR and is just as important (or clarify the language here and in the AFCARS element that voluntary termination dates should be included); 2) indicate which of two possible TPR dates would be used; 3) focus on children with a case plan goal of adoption, and 4) take into account state appeals processes that extend the timeframe beyond the control of the agency. Also, the data quality requires considerable improvement to make this indicator more meaningful.

Performance Area 4, Timeliness of TPR for children who have been in foster care for 17 months or longer at the start of a fiscal year
This indicator should not be used as written. This indicator may reward the inappropriate pursuit of TPR and penalize good practice. If the indicator is used, it would be improved by focusing on children with a case plan goal of adoption whose parental rights are terminated. Also, the data quality requires improvement.

Permanency Composite 3, Placement Stability

CWLA is concerned with the placement stability indicators for several reasons. First, the measures are based on AFCARS FC Element #24, which is calculated by the states. A National Working Group survey revealed significant differences in how states report certain changes, such as hospitalization and detention or incarceration (see Part 4). These variations make the measures unreliable. More time and attention are needed to address the variations among states. Second, age and population (e.g. child welfare, juvenile justice, and children entering care for behavioral reasons) are critical factors in assessing placement stability. The proposed indicators do not consider age or population, as they should. Third, the indicators do not take into account appropriate movement to less restrictive settings.

Performance Area 1 – Placement Stability during the first year in care
The measure does not address definitional variations, so the measure continues to be unreliable. Also, the measure would be more meaningful stratified by age or population. Finally, the 12 month timeframe is misleading as some children in this group really are in care for just a few days.

Performance Area 2 – Placement Stability for children in care for longer than 12 months
CWLA supports the need to measure placement stability for children who have been in care for longer periods of time. However, the concerns about definitional inconsistencies and need to stratify by age and population apply equally to this indicator. Also, in order to better measure recent efforts to improve placement stability we recommend that, for children in care longer than 24 months, the measure focus on an increase in placements between the beginning and end of the reporting period (or year).

Third Performance Area
If the placement stability measures are used despite the above listed concerns, CWLA recommends that a third indicator be added to capture the percent of children with 3 placements within 12 – 24 months.

Permanency Composite 4, Achieving Permanency for Children

In general, Permanency Composite 4 addresses important performance areas. However, it is difficult to comment on these indicators without knowing the standard that will be applied for each. Performance area 1, children growing up in foster care, adds an important dimension to the performance assessment. Performance area 3, exits to families of children with TPR, addresses the existence of children who are “legal orphans.” For each, we would be interested in hearing more details.

Performance area 2, however, raises questions. It is unclear as to its purpose and effectiveness in measuring permanency for children. The rationale mentions that a number of states have a high percentage of children for whom the data element regarding case plan goal is “not yet determined.” It seems this issue may be more effectively addressed with those states to determine whether this is a data entry issue or a practice issue, and how it may be improved upon.

Part 4: Additional Comments

CWLA and the National Working Group continue to be concerned with the definitional issues and methodological issues outlined below. In addition, we advocate for increased training and technical assistance to help states improve the quality of the data and better use data to assess and improve child welfare outcomes.

Definitional Issues

CWLA and the National Working Group to Improve Child Welfare Data have implemented three surveys that provide evidence of definitional inconsistencies among states in the data reported to federal sources (NCANDS and AFCARS). Other inconsistencies are evident in clear differences in child welfare policy and programs in different states. These inconsistencies make cross-state comparison unreliable, even with the use of composite measures. Examples pertinent to the proposed measure include the following:

Child maltreatment in foster care
When a child is abused or neglected by a relative foster care provider, the incident is captured differently among states. Since the caregiver is both a relative and a foster care provider, the relative relationship may be captured in the perpetrator data in some states or in some cases (not the foster care relationship), which is not part of the federal measure on child maltreatment in foster care.

Residential facility staff are counted as perpetrators of maltreatment in most, but not all, states. The victims in these cases are not always in foster care, and thus the measure of maltreatment in foster care in these states has the potential to be inflated.

Reunification
Inclusion of children discharged to relatives: States vary in the AFCARS discharge category used for relative guardianship and, in a few states, relative adoption; some include these children as discharged to “living with other relatives”, while others do not. Since “living with other relatives” is included as “reunified” in the federal measure, this effects the federal reunification rate.

Date of discharge and trial home visits. The date of discharge is reported as the date the child physically goes home in some states when the state maintains custody and supervision, while more states use the date legal custody is returned to parents. Also, reporting of trial home visits varies, as does the length of time children spent on trial home visits (if these are used at all in a state). The date of discharge affects the length of time to reunification. In some states children may be returned to parents within 6 to 12 months but will never meet the federal standard of 12 months due to long trial home visits that may exceed six months.

Placement Stability
States are required to report the number of placement changes a child has experienced in care for FC Element #24. For this element, states vary as to whether, and when, they count certain changes such as hospital placements and detention or incarceration placements. Some federal guidance is provided, but states are interpreting the guidance differently.

States vary as to their inclusion of juvenile justice youth in the national data. Since placement issues are different in juvenile justice, the placement stability data may be affected. Therefore, when analyzing placement stability data it is important to group states that serve similar populations or limit the data to just the child welfare population.

Methodological Issues
CWLA supports the use of the current national data sources (NCANDS and AFCARS) at this time since they are available for all states. However, we advocate the use of longitudinal data to evaluate performance over time. Some researchers have demonstrated the potential to convert AFCARS into longitudinal files, which could then be used to help states follow their performance over time. Further, some states have the capacity to do longitudinal analysis, either internally or through external arrangements, and these data should be taken into consideration when state performance is evaluated in the Child and Family Services Reviews.

Training and Technical Assistance
Additional training and technical assistance is needed to help states understand federal guidance and how it relates to their state terms/policies. Also, concrete guides are needed to integrate the federal definitions and guidance with real scenarios to help states report their data more consistently. CWLA and the state child welfare agencies (the National Working Group to Improve Child Welfare Data) are developing such a guide relevant to the federal placement stability measure, and we are requesting feedback from the Children’s Bureau. Finally, the resource centers (such as NRC-CWDT) should be available to provide consistent information and assistance to all states as the outcome measures are developed.

CWLA and the National Working Group support efforts to improve practice and achieve positive outcomes for children and families in child welfare systems across the country. We are actively working to improve the data quality so that as a field we may work toward improved outcomes. It is critical that the CFSR measures support real improvements in practice, and not be marred by unreliable comparisons among states or methodology that provides a misleading view of the outcomes. The CFSR process has proven to be a positive experience, helping states identify problem areas and improve services that will lead to better outcomes. With improvements in the data indicators, the process promises to be even more effective. We thank the Children’s Bureau for the opportunity to comment and look forward to further collaboration among the Children’s Bureau, state child welfare agencies and national organizations as we work to improve outcomes for children.