Below we provide responses to frequently asked questions (FAQs). We first present responses to general FAQs. The remaining FAQs are organized according to the review process.
A downloadable copy of the Prevention Services Clearinghouse Handbook of Standards and Procedures can be found here.
The Prevention Services Clearinghouse website includes the working list of programs and services that are currently under review. This can be found on the FAQ page of the website under the question: “What are the next programs and services planned for systematic review?”
It is difficult to predict the timeline for review of an individual program and service. Review timelines are influenced by a number of factors including, but not limited to:
- The number of eligible studies examining a particular program or service;
- Whether eligible studies have all the information necessary to complete the review or whether author queries will be needed to obtain additional information;
- If author queries are needed, the timeline for authors to respond to the queries and the information they provide; and,
- Whether any questions come up during the course of the review of studies that require internal or external expert consultation.
Our goal is to review and rate as many programs and services as quickly as possible to support states’ efforts to improve outcomes for children and families through implementation of the Family First Prevention Services Act. We encourage stakeholders to sign up for email updates on the Clearinghouse website to receive notifications about changes to the working list of programs and services planned for systematic review and the release of new program or service ratings.
Consistent with the practice of other federal evidence reviews, standards and procedures may be revised over time as research methods evolve, the needs of the field change, and lessons are learned during the review process. Potential revisions will be informed by a transparent process and build from the existing knowledge of States, federal agencies, researchers, evaluators, program and service developers, stakeholders and experts, and the general public. Opportunities for public feedback through Federal Register Notices will be announced via our email list. To receive updates, please join our email list.
The Clearinghouse has developed a working list of the next programs and services planned for systematic review. To learn more about this list, please visit the About page.
The Prevention Services Clearinghouse continues to select programs and services for review using an inclusive process. The Prevention Services Clearinghouse staff have logged all recommendations received since 2018. Programs and services may be identified from the following sources:
- A public call for recommendations on at least an annual basis. The latest public call occurred in the fall of 2022 and is now closed.
- Recommendations received via email to PreventionServices@abtassoc.com; and
- An environmental scan and an inventory of the literature.
Given the high volume of recommendations, the Clearinghouse must prioritize programs and services for review. Particular consideration is given to programs and services recommended by State or local government administrators; rated by other clearinghouses (such as CEBC or HomVEE); recommended by federal partners; and/or evaluated as part of any grants supported by the Children’s Bureau (such as the Title IV-E Child Welfare Demonstrations or Regional Partnership Grants).
To learn more about how we select and prioritize programs and services for review, please see Chapter 2 of the Clearinghouse’s Handbook of Standards and Procedures.
In identifying the initial group of programs and services for review, the Children’s Bureau and the Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families considered comments and recommendations from more than 360 unique responses to the Federal Register Notice (83 FR 29122) issued June 22, 2018.
The initial programs and services selected for review met at least two of the following conditions: (1) recommendation from State or local government administrators in response to the Notice; (2) rated by the California Evidence-Based Clearinghouse for Child Welfare; (3) evaluated by Title IV-E Child Welfare Waiver Demonstrations; (4) recipient of a Family Connection Discretionary Grant; and (5) recommendation solicited from federal partners in the Administration for Children and Families, Health Resources and Services Administration, the National Institutes of Health, the Centers for Disease Control and Prevention, the Office of the Assistant Secretary for Planning and Evaluation, and the Substance Abuse and Mental Health Services Administration.
Our goal is to review and rate as many programs and services as quickly as possible to support states’ efforts to improve outcomes for children and families through implementation of the Family First Prevention Services Act (FFPSA).
The Prevention Services Clearinghouse will continue to select additional programs and services for review on a rolling basis starting in summer 2019. Join our email list to be notified of updates!
The Prevention Services Clearinghouse has received a high volume of program and service recommendations to date. This includes more than 360 comments received in response 2018 FRN as well as nearly 400 emails received in response to the 2019 public call for program and service recommendations. Many of these recommendations identify multiple programs and services. The Prevention Services Clearinghouse staff log and review all program and service recommendations and respond to stakeholders noting that their recommendations have been received. Given the high volume of recommendations, it is not currently feasible within resource and time constraints, for staff to send individualized responses to stakeholders indicating why a program or service recommended has not yet been selected for review. The Clearinghouse’s systematic prioritization process is outlined in the Handbook of Standards and Procedures (Section 2). The Prevention Services Clearinghouse is working to review and rate as many services and programs as quickly as possible.
As described in the Clearinghouse’s Handbook of Standards and Procedures, programs and services are identified for review using an inclusive process (see Chapter 1). Once identified, the Clearinghouse then selects and prioritizes programs and services for review (see Chapter 2). The Clearinghouse maintains a working list of the next programs and services planned for systematic review. This can be found on the Programs & Services Planned for Review page on the website. Candidate programs and services on this list are evaluated against program or service eligibility criteria, as described in Chapter 2 of the Handbook. Programs and services added to the working list may later be deemed ineligible for review if they do not meet these eligibility criteria. Reasons a program or service might not meet these eligibility criteria are listed in the Handbook under Program Eligibility Criterion 2.1.1 and 2.1.2. These include: 1) a written protocol, manual, or other documentation that describes how to implement or administer the practice is not available to the public at this time or 2) the program or service does not meet the criteria for any of the four program or service areas eligible for inclusion. When this happens, the program appears on the Find a Program or Service page with a description of “This program or service is not eligible for review at this time.”
As noted in Chapter 3 of the Handbook, the Prevention Services Clearinghouse begins the search by identifying citations from other evidence clearinghouses or. A number of evidence clearinghouses overlap in content with the Prevention Services Clearinghouse (see Exhibit 3.1). The list of clearinghouses may change and additional clearinghouses may be used, depending on the program or service selected. We currently review the following clearinghouses:
Blueprints for Healthy Youth Development
The California Evidence-Based Clearinghouse for Child Welfare
The Campbell Collaboration
Cochrane Reviews
CrimeSolutions
Home Visiting Evidence of Effectiveness
National Registry of Evidence-based Programs and Practices (via Results First Clearinghouse)
Social Programs That Work
Teen Pregnancy Prevention Evidence Review
Washington State Institute of Public Policy
The following procedures apply when the Prevention Services Clearinghouse identifies more than 15 eligible studies for a program or service under review. As noted in Section 4.2. Study Review Prioritization Criteria of the Handbook, the Prevention Services Clearinghouse reviews all eligible studies.
If a program or service has 15 or fewer eligible studies, all studies are reviewed using the design and execution standards described and assessed for risk of harm.
If a program or service has more than 15 eligible studies, all eligible studies are assessed for risk of harm. Study review prioritization criteria are used to determine the order of eligible studies reviewed using the design and execution standards. Once ordered, the first 15 eligible studies will be reviewed using the design and execution standards. If, after review of 15 eligible studies, a program or service has not achieved a rating of well-supported, additional studies will continue to be reviewed in order until the program or service has achieved a rating of well-supported or all eligible studies have been reviewed.
Many programs and services have been adapted (e.g., modified to address particular issues or populations). The Prevention Services Clearinghouse reviews each adaptation to determine if the program or service has been substantially modified or adapted from the version that has been selected for review. The Prevention Services Clearinghouse reviews eligible studies only of the version selected for review.
Exhibit 4.1. within the Handbook of Standards and Procedures (shown below) provides examples of ‘eligible’ adaptations (i.e., adaptations that have not substantially modified or adapted the program or service from the one selected for review). For example, if a program modestly changes the session frequency or duration, this may be reviewed as part of the version that was selected for review. Exhibit 4.1 below also provides examples of adaptations that would be considered substantial and result in a different version of the program or service than the one selected for review. For example, substantial changes to a program or service’s enrollment or eligibility criteria.
When there are multiple versions of a program or service, the Prevention Services Clearinghouse may select just one version as indicated by a specific book/manual/other documentation to review at a particular time. Other versions of the program or service may be reviewed by the Prevention Services Clearinghouse in the same round of review or in later rounds of review.
For additional information see section 4.1.6 of the Handbook of Standards and Procedures.
As described in Section 4.1.4 of the Handbook of Standards and Procedures, eligible studies must use a randomized or quasi-experimental group design with at least one intervention condition and at least one comparison condition. Intervention and comparison conditions may be formed through either randomized or non-randomized procedures and the unit of assignment to conditions may be either individuals or groups of individuals (e.g., families, providers, centers). Comparison groups may be formed using a variety of methodologies, including, for example, both propensity score matching and weighting strategies. Designs in which the same group of individuals is evaluated before and after receiving an intervention are not eligible (i.e., single group pretest posttest designs). This includes interrupted time series designs without comparison groups. The Prevention Services Clearinghouse does not currently review studies that use regression discontinuity designs.
Studies must measure and report program or service impacts on at least one eligible target outcome, as described in Sections 2.2.1 and 4.1.5 of the Handbook. Programs and services in the areas of mental health, substance abuse, and in-home parent skills must target outcomes in the domains of (a) Child Safety, (b) Child Permanency, (c), Child Well-Being, and/or (d) Adult Well-Being. Programs and services in the area of kinship navigator must target outcomes in the domains of (a) Child Safety, (b) Child Permanency, (c) Child Well-Being, (d) Adult Well-Being, (e) Access to Services, (f) Referral to Services, and/or (g) Satisfaction with Programs and Services. Eligible target outcomes are defined as follows:
The Prevention Services Clearinghouse reviews the following domains of Child Safety:
- Child Welfare Administrative Reports. Substantiated or unsubstantiated child maltreatment from administrative records. Eligible indicators include, but are not limited to, substantiated and unsubstantiated reports of abuse or neglect, investigations of abuse and neglect from administrative records, and recurrence of abuse and neglect from administrative records.
- Self-Reports of Maltreatment. Eligible indicators include victim and perpetrator reports of abuse or neglect and questionnaire or interview instruments that directly assess abusive behavior or neglect.
- Maltreatment Risk Assessment. Eligible indicators include child maltreatment risk assessments.
- Medical Indicators of Maltreatment Risk. Eligible indicators include administrative, questionnaire, or interview instruments assessing childhood injuries, ingestions, emergency room visits, hospitalizations and any other indicators of childhood injuries.
The Prevention Services Clearinghouse reviews the following domains of Child Permanency:
- Out-of-Home Placement. Any situation where a child is removed from the family home. Eligible indicators include, but are not limited to, any out-of-home placement, placement to foster care, reports of the caregiver relinquishing her or his role, and time to placement in out-of-home care.
- Least Restrictive Placement. Included in this subdomain are measures that assess the restrictiveness or disruptiveness of out-of-home placement. This subdomain focuses on improving the environments/settings into which children are placed, including favoring kinship placements over non-kin or institutional placements or placements that maintain connections to the child’s community versus those that do not. Outcomes must be operationalized with more than two placement settings, as binary measures for which the reference category is another out-of-home placement setting, or as movement from more restrictive/disruptive to less restrictive/disruptive settings. Eligible indicators include, but are not limited to, hierarchies of least restrictive preference (e.g., kin placement, family foster care, therapeutic care, group home, residential, hospitalization, and incarceration).
- Placement Stability. Placement stability refers to the stability of out-of-home placement (e.g., that children are in placements that are disrupted infrequently). This subdomain focuses on the number of placement disruptions (planned and unplanned) or number of out-of-home placements. Eligible indicators include, but are not limited to, number of placement changes or disruptions of placements, and re-entries or failed exits/reunifications or adoptions.
- Planned Permanent Exits. Planned permanent exits from out-of-home care refer to placements or time to placement to a more permanent status, including reunification, guardianship, and adoption. Eligible indicators include, but are not limited to, measures of the amount of time to reunification, guardianship, or adoption and reunification rates.
The Prevention Services Clearinghouse reviews the following domains of child well-being, the specific nature of which may vary with age:
- Behavioral and Emotional Functioning. Characteristics and behaviors relating to the ability to realize one’s potential, cope with daily activities, and work and play productively and fruitfully. Both strengths-based and deficit-based indicators are eligible. Examples include measures of externalizing behaviors (e.g., aggressive behavior, disruptiveness, impulsive behavior), internalizing behaviors (e.g., depression, anxiety, mood or thought problems), mental/behavioral health diagnoses, positive behavior, resilience, self-regulation or self-control, and emotional adjustment.
- Social Functioning. Skills and capabilities relating to the ability to develop, maintain, and manage interpersonal relationships (e.g., social skills, assertiveness, cooperation, empathy, social adjustment, peer relations, rebelliousness, defiance, and other similar characteristics related to interpersonal interactions and relationships).
- Cognitive Functions and Abilities. Abilities related to reasoning, knowledge, problem-solving, mental processing, executive functioning, and the like. Eligible measures include intelligence tests, developmental assessments, measures of visual or spatial processing, and other indicators of cognitive functions and abilities.
- Educational Achievement and Attainment. Educational achievement refers to the extent to which students master academic content. Eligible measures include composite or subject-specific (e.g., reading, mathematics) standardized achievement test scores or overall grade point averages. Educational attainment refers to student progress through school or the completion of a degree, certificate, or program. Eligible measures of attainment include grade promotion, high school graduation or dropout rates, certificate or degree completion rates, and other indicators for educational attainment.
- Physical Development and Health. Characteristics related to the healthy functioning of the body may include indicators of physical health (e.g., Body Mass Index), physical capabilities (e.g., motor skills), normative indicators of healthy development (e.g., height), and any other measure relating to healthy (or unhealthy) physical development.
- Substance Use or Misuse. Measures of substance use or misuse may involve any substances and may be self- or other-reported, clinical tests such as urinalysis, or any other measure that provides an assessment of the participants’ substance use behavior. Measures must describe actual use or misuse, such as frequency or quantity of use, type of use, use/no use, time since last use, etc. Substance use diagnoses (e.g., from a clinical interview or DSM criteria) are considered eligible outcomes in this domain. Measures that do not directly index substance use or misuse (e.g., drug-related criminal or delinquency activity such as selling drugs, drug knowledge, behavioral intentions to use or not, etc.) are not eligible in this domain, but may meet the requirements for other outcome domains.
- Delinquent Behavior. Delinquent behavior refers to behavior chargeable under applicable laws, whether or not apprehension occurs or charges are brought. Chargeable offenses also include “status” offenses (e.g., runaway, truancy, curfew violations).
The Prevention Services Clearinghouse reviews the following domains of adult well-being:
- Parenting Practices. Parenting practices include a range of practices and behaviors focused on developing strong, functional relations between parents or caregivers and children and the parents or caregivers’ abilities to successfully manage child socialization and support child development, health, and well-being in an effective and constructive manner. Measures may include items about basic elements of caregiving, such as feeding and physical care; communication and listening; nurturing, loving, or supportive behavior; rules and consequences; setting boundaries; warmth; scaffolding children’s behavior to develop self-discipline; parent-child relationships, and the like. Measures may index either positive parenting practices or negative parenting practices.
- Parent/Caregiver Mental or Emotional Health. Mental or emotional health refers to a parent’s/caregiver’s ability to cope with daily activities, realize his or her potential, and interact productively in the world. Both strengths-based and deficit-based indicators are eligible. Examples include measures of externalizing behaviors (e.g., aggressive behavior), internalizing behaviors (e.g., depression, anxiety, mood or thought problems), mental/behavioral health diagnoses, parent/caregiver stress, relationship stress, positive behavior, resilience, and emotional adjustment.
- Parent/Caregiver Substance Use or Misuse. Measures of substance use or misuse may involve any substances and may be self- or other-reported, clinical tests such as urinalysis, or any other measure that provides an assessment of the participants’ substance use or misuse. Measures must describe actual use or misuse, such as frequency or quantity of use, type of use or misuse, use/no use, time since last use, etc. Substance use diagnoses (e.g., from a clinical interview or DSM criteria) are considered eligible in this domain. Measures that do not directly index substance use or misuse (e.g., drug-related criminal or delinquency activity such as selling drugs, drug knowledge, behavioral intentions, etc.) are not eligible in this domain, but may meet the requirements for other outcome domains.
- Parent/Caregiver Criminal Behavior. Criminal behavior refers to behavior chargeable under applicable laws, whether or not apprehension occurs or charges are brought.
- Family Functioning. Family functioning refers to the capacity or lack of capacity of a family to meet the needs of its members and includes physical care and maintenance of family members; socialization and education of children; and, economic and financial support of the family.
- Physical Health. Refers to the physical health of parents or caregivers and can include a variety of indicators including blood pressure; weight, obesity, or body mass index (BMI); chronic conditions such as asthma or diabetes; and, healthy lifestyle behaviors such as diet and exercise.
- Economic and Housing Stability. Economic and housing stability includes indicators of financial or economic stability (e.g., level of income, employment/unemployment, financial assistance) and/or housing stability (e.g., number of moves, quality of housing, homelessness).
Additional outcomes that are eligible for kinship navigator programs include:
- Access to Services. Access to services refers to a parent, caregiver, or family’s knowledge of and ability to access, or utilization of services to support the family’s financial, legal, social, educational, and/or health needs such as medical care, financial assistance, and social services. Parent/caregiver self-reports, informed collateral reports (e.g., from therapists or case managers), or administrative records are eligible indicators for Prevention Services Clearinghouse reviews.
- Referral to Services. Referral to services may include referrals to any needed financial, legal, social, educational, or health services. Measures may be obtained from parent/caregiver self-reports, therapist or provider reports or records, or administrative records. Examples include the presence or absence of referrals or counts/frequencies of referrals.
- Satisfaction with Programs and Services. Satisfaction with programs and services refers to parent or caregiver satisfaction with the programs and services to which they are referred or which they receive as part of a kinship navigator program.
Section 4.1.4. of the Handbook of Standards and Procedures indicates that eligible comparison groups must be “no or minimal intervention” or “treatment as usual” groups.
“No or minimal intervention” comparison group members may receive handouts, referrals to available services, or similar nominal interventions. Such additional services must be brief (i.e., one session or less) and/or predominantly information-based (e.g., pamphlets about child development, psychoeducation about a specific disorder).
“Treatment as usual” comparison group members may be receiving services in their communities already or they may be offered services as part of the research study. Services must be clearly described as the usual or typical services available for that population in the study. The Prevention Services Clearinghouse considers a comparison groups to be “treatment as usual” under either of the following two conditions.
- Condition 1: Usual or typical services. Comparison groups that are already receiving services in their community or are offered services that they would have received in the absence of the study meet this condition. These comparison groups do not receive services because of the study (i.e., they do not receive anything they wouldn’t already receive in the absence of the study). However, it is acceptable for studies with comparison groups in this category to provide or offer minimal intervention (as defined above) along with the treatment as usual. To be eligible under Condition 1, the study or supporting documentation must provide specific information to demonstrate that only usual or typical services were available and that any services offered by the study in addition to usual or typical services were minimal.
- Condition 2: Services consistent with usual or typical services. Comparison groups that are offered services as part of the study that are clearly described in the study as consistent with the usual or typical services that would be received by individuals or families similar to those in the study may also be eligible. In such cases, the study or supporting documentation must provide specific information to justify that the services offered to the comparison group participants are consistent with what individuals or families like those in the study could be expected to receive in the absence of a study.
In studies with multiple eligible comparison groups, reviewers select one eligible comparison per eligible outcome at each time point instead of comparing the same intervention group to multiple comparison groups for all outcomes and time points. Selection of an eligible comparison group for each outcome at each time point is based on the group that receives the least intensive services in order to maximize the treatment contrast.
Studies in which the program or service under review is delivered in combination (or bundled) with another program or service may exhibit a design confound. In such cases, it is impossible to separate how much of the observed effect was related to the intervention and how much to the confounding factor. Thus, the study cannot meet evidence standards and will receive a low rating (see Section 5.9.3. of the Handbook of Standards and Procedures for additional information on design confounds). We illustrate three scenarios of relevance to bundled interventions below:
- Intervention + Treatment as Usual vs. Treatment as Usual Only. Studies of bundled intervention conditions do not exhibit a design confound when the program or service under review is bundled with treatment as usual and the comparison group also receives treatment as usual.
- Intervention + Treatment as Usual vs. No/Minimal Treatment. Studies of bundled intervention conditions may exhibit a design confound when the program or service under review is bundled with treatment as usual but the comparison group does not receive treatment as usual. Studies with this configuration of conditions would only be determined to be free of a design confound if the treatment as usual services bundled with the program or service under review are weak, minimal, or unlikely to affect study outcomes.
- Intervention + Second Intervention vs. No/Minimal Treatment or Treatment as Usual. Studies of bundled intervention conditions in which the second intervention in the bundle is not treatment as usual (i.e., the intervention condition receives the program or service under review and another program or service) are ineligible for review unless both programs or services are selected for review as a bundle. If a bundle of services is selected for review, the eligibility criteria for comparison groups apply; that is, the comparison groups must be no treatment or minimal treatment, or treatment as usual groups.
The Prevention Services Clearinghouse uses unique standards and procedures developed in accordance with the statutory requirements detailed in the Family First Prevention Services Act (FFPSA) of 2018. For example, though the California Evidence-Based Clearinghouse for Child Welfare (CEBC) uses a peer-review process to review and rate studies of programs and services, the Prevention Services Clearinghouse is required to conduct an independent, systematic evidence review. In addition, different from CEBC, the Title IV-E Prevention Services Clearinghouse allows programs or services with rigorous quasi-experimental evidence to achieve a well-supported or supported rating. The Prevention Services Clearinghouse also includes review of publicly available research studies including those that do not appear in peer-reviewed journals.
Similarly, different from Home Visiting Evidence of Effectiveness (HomVEE) review, the rating criteria and outcome domains identified in FFPSA do not directly align with the criteria established by HHS for evidence-based home visiting models. For example, in accordance with the statutory requirement of FFPSA, the Prevention Services Clearinghouse criteria for sustained effects are based on length of time after the end of treatment, whereas HomVEE criteria for sustained effects are based on length of time after enrollment.
Yes. As noted within section 7.4.1 of the Handbook of Standards and Procedures, programs and services reviewed for the Prevention Services Clearinghouse may be considered for re-review due to missing information or errors in the original review, or due to the emergence of substantial new evidence that has the potential to change program or service ratings. Prevention Services Clearinghouse staff keep track of the dates that programs and services are reviewed and periodically assess the extent of new evidence available. Periodically, content experts may be consulted to determine if new research is available and if the available research has the potential to change the rating of the program or service. Stakeholders may request a re-review of the program or service rating by emailing PreventionServices@abtassoc.com. To help facilitate the re-review, stakeholders may specify the rationale for the request and provide any supporting documentation.
As noted within section 7.4.2 of the Handbook of Standards and Procedures, individual studies reviewed by the Prevention Services Clearinghouse may be considered for re-review due to missing information or errors in the original review. If errors or missing information are identified, the Prevention Services Clearinghouse follows standard procedures for re-review. This includes assigning different, blinded reviewers to conduct any re-reviews. If the re-review determines the original review to be in error, the error is corrected on the website. All correspondence regarding re-reviews is logged and maintained by Prevention Services Clearinghouse staff. Stakeholders may request a re-review of an individual study by emailing PreventionServices@abtassoc.com. To help facilitate the re-review, stakeholders may specify the rationale for the request and provide any supporting documentation.
