FAQs

Below we provide responses to frequently asked questions (FAQs). We first present responses to general FAQs. The remaining FAQs are organized according to the steps in the Prevention Services Clearinghouse systematic review process described in the Handbook of Standards and Procedures, Version 2.0. All Prevention Services Clearinghouse program and service reviews and re-reviews are now conducted using the Handbook of Standards and Procedures, Version 2.0.

The FAQs are informed by questions submitted to the Prevention Services Clearinghouse via email and by ongoing active engagement activities. FAQs will continue to be updated based on questions received via email and future active engagement activities.

Responses to FAQs relevant to the Handbook of Standards and Procedures, Version 1.0 are available to download.

General FAQs

Where can I download a copy of the Handbook of Standards and Procedures?

A downloadable copy of Version 2.0 can be found on the Handbook of Standards and Procedures, Version 2.0 page.

A downloadable copy of Version 1.0 can be found on the Handbook of Standards and Procedures, Version 1.0 page.

What updates were made to the Handbook of Standards and Procedures in Version 2.0?

A detailed summary of revisions made to the Handbook of Standards and Procedures can be found in the “Summary of Revisions to the Handbook of Standards and Procedures” appendix in the Handbook of Standards and Procedures, Version 2.0. For more information about the handbook revision process, please see the Introduction of Handbook Version 2.0.

Do you have an estimated timeline to complete the review of a particular program or service?

The Prevention Services Clearinghouse website includes a working list of programs and services that are planned for systematic review. This can be found on the Programs & Services Planned For Review page.

It is difficult to predict the timeline for review of an individual program or service. Review timelines are influenced by a number of factors including, but not limited to:

• Whether the Prevention Services Clearinghouse has questions for program or service developers about manual availability or adaptations, and if so, the time it takes for developers to respond.
• The number of eligible studies evaluating the impact of a particular program or service on at least one target outcome.
• Questions that arise during the review of studies that require internal or external expert consultation.
• Whether eligible studies have all the information necessary to complete the review or whether author queries are needed to obtain additional information – and, if required, the time it takes for study authors to respond.

For more information, see the fact sheet The Review Process: Navigating Timeline Challenges.

Our goal is to review and rate as many programs and services as quickly as possible to support states’ efforts to improve outcomes for children and families through implementation of the Family First Prevention Services Act. We encourage individuals to sign up for email updates on the Clearinghouse website to receive notifications about changes to the working list of programs and services planned for systematic review and the release of new program or service ratings.

Will the Prevention Services Clearinghouse periodically update the Handbook of Standards and Procedures?

The Handbook of Standards and Procedures, Version 2.0 was released in July 2024. To download a copy, visit our Handbook of Standards and Procedures, Version 2.0 page. Consistent with the practice of other federal evidence reviews, standards and procedures may be revised over time as research methods evolve, the needs of the field change, and lessons are learned during the review process. Potential revisions will be informed by a transparent process and build on the existing knowledge of States, federal agencies, researchers, evaluators, program and service developers, experts, and the general public. Opportunities for public feedback through Federal Register Notices will be announced via our email list. To receive updates, please join our email list.

Does the Prevention Services Clearinghouse provide individualized evaluation technical assistance?

Due to the need to maintain independence and objectivity, the Prevention Services Clearinghouse cannot provide individualized technical assistance to those seeking to align design, implementation, or analysis of their evaluations with the Handbook of Standards and Procedures. However, the Prevention Services Clearinghouse encourages interested parties, including study authors, program or service developers, and other members of the public to submit technical clarification questions pertaining to the results of a study or program or service review via email to PreventionServices@abtglobal.com. The Prevention Services Clearinghouse responds to all such requests, though the timeline for response may vary depending on the number and complexity of questions received and the volume of requests received. Because of the volume of correspondence the Prevention Services Clearinghouse receives and the need to maintain a written record of key decisions, individual briefings with members of the public are not possible.

Where can I find information about study settings and participant characteristics on the Prevention Services Clearinghouse website?

The Prevention Services Clearinghouse records and displays information about study settings and participant characteristics for all studies that receive a moderate or high rating on design and execution, when reported in study documents. That information is available on the individual program and service pages on the website under the Study Participant Characteristics heading. Individual program and service pages can be found by clicking the “Find a Program or Service” button on the home page or by navigating to the “Programs and Services Reviewed” page under the “Find a Program or Service” menu. Study settings and participant characteristics are not available for programs and services without studies that received a moderate or high rating on design and execution.

The Prevention Services Clearinghouse is planning to conduct a pilot to understand the considerations, including methodological, timeline, and resource related considerations, for the systematic review of subgroup analyses, including how such analyses may contribute to future program or service ratings. The results of the pilot will be communicated to the public via the Prevention Services Clearinghouse email list. To receive updates, please join our email list.

For programs and services reviewed under the Handbook of Standards and Procedures, Version 2.0, the Prevention Services Clearinghouse will report whether certain subgroup analyses are present in studies where the full-sample analysis receives a high or moderate design and execution rating.

How does the Prevention Services Clearinghouse engage program or service developers during the review process?

In order to support transparency and leverage the expertise developers have about the programs and services they have developed, the Prevention Services Clearinghouse actively engages program and service developers at multiple steps of the review process, related to:

• Program or service eligibility, prioritization, or selection criteria. Program or service developers receive a query if information is needed to apply the program or service eligibility, prioritization, or selection criteria, as described in Chapter 2 of the Handbook of Standards and Procedures, Version 2.0
• Review of draft program or service description, including manual citation(s). Developers are provided with a copy of the proposed public description of the program or service, inclusive of the manual citations, for comment in advance of the publication of the results of the review of the program or service.
• Adaptations identified. Program or service developers may be consulted regarding identified adaptations if it is not clear from the program or service manual whether identified adaptations are substantial.
• Notification of program or service review or re-review. Program and service developers are also notified if their program or service has been identified for a program or service re-review or a study re-review. Program or service developers are notified of the results of a program or service review (or re-review) through the publication of the review results on the Prevention Services Clearinghouse website.

Program and service developers also may receive queries from the Prevention Services Clearinghouse for information deemed necessary to understand the program or service as it is intended to be delivered; to understand alternative manual editions, manual variants, or other changes in the program or service over time; or to inform a public description of the program or service on the Prevention Services Clearinghouse website. All queries provide clearly defined timelines for developers to respond in order to facilitate timely completion of reviews.

Developers can support the review process by: clearly listing contact information on their program or service websites; documenting any adaptations made between manual editions or variants; promptly responding to developer queries; and reviewing the Prevention Services Clearinghouse’s description of the program in a timely manner.

The Reporting Guide for Study Authors highlights how developers and study authors can describe any adaptations completely and consistently to help facilitate the Prevention Services Clearinghouse’s review process.

How does the Prevention Services Clearinghouse engage with study authors?

The Prevention Services Clearinghouse actively engages with study authors throughout the systematic review process to support transparency and leverage the expertise study authors have about their own research.

The Prevention Services Clearinghouse will query study authors for information deemed necessary (1) to determine the eligibility of a study for review; (2) to assign a design and execution rating of high, moderate, or low; (3) to assess statistical significance; (4) to determine the length of sustained effects; or (5) to assess risk of harm. Author queries may request information about the intervention or comparison conditions, sample sizes, baseline statistics, group formation (e.g., whether randomization was used), characteristics of the outcome measures required to determine whether outcome requirements are met, and may ask clarifying questions about analytic models (e.g., whether covariates are included in the models estimating impacts). Author queries may also request information needed to assign program or service ratings.

All queries provide clearly defined timelines for authors to respond in order to facilitate timely completion of reviews. If authors do not respond to queries, the Prevention Services Clearinghouse proceeds using available information. Information provided after the defined timeline is retained by the Prevention Services Clearinghouse but is not guaranteed to be included in the review.

Study authors can facilitate reviews by doing the following:

• Provide information needed in reporting. As authors write up study results, they can proactively include information required for the review process. Comprehensive reporting up front can prevent the need for author queries. For more information, see the Reporting Guide for Study Authors.
• Responding promptly to author queries, when applicable. Prompt responses to author queries can help ensure a timely review. For more information, see the author query policies in Section 8.4.2 of the Handbook of Standards and Procedures, Version 2.0.

How does the Prevention Services Clearinghouse identify substantial program or service adaptations for the purposes of review?

If any adaptations between manual editions or variants are identified, the Prevention Services Clearinghouse assesses whether the adaptations represent a substantial adaptation from the program or service as described in the focal manual and therefore will be considered a separate program or service for the purposes of review. This assessment is conducted using a systematic stepwise process shown in Exhibit 2.4 in the Handbook of Standards and Procedures, Version 2.0 and displayed below.

What are examples of program or service adaptations that are substantial and not substantial for the purposes of review?

If any adaptations between manual editions or variants are identified, the Prevention Services Clearinghouse assesses whether the adaptations represent a substantial adaptation from the program or service as described in the focal manual. Examples of adaptations that are substantial and not substantial are shown below.

Note. Adaptations may affect multiple program or service components simultaneously and in this case the Prevention Services Clearinghouse will review changes to each component independently. For example, adding two sessions may not constitute a substantial adaptation based on dosage alone; however, it may still constitute a substantial adaptation based on content if new subject matter is added.

How are program or service adaptations in studies reviewed?

To be eligible for review, studies of a program or service must not indicate substantial differences from the program or service selected for review, as specified in the books, manuals, or other documentation that describe how to implement or administer the program or service.

If a study exhibits adaptations relative to the focal manual for the program or service under review, a systematic stepwise process (illustrated in Exhibit 4.1 in Handbook of Standards and Procedures, Version 2.0 and described in detail below) is then used to determine whether the study is an eligible study of the program or service or is ineligible (i.e., not a study of the program or service under review). Studies with substantial adaptations are ineligible for review for the program or service under review.

• Step 1: Is the study adaptation (a) explicitly prohibited in the focal manual for the program or service under review or (b) the result of adding a separate program or service to the program or service under review?
• Step 2: Is the study adaptation explicitly allowed by the focal manual for the program or service under review?
• Step 3: Does the study adaptation substantially change a key program or service component in the focal manual for the program or service under review?
• Step 4: After gathering any additional information needed, have experts determined that the study adaptation is substantial?

For additional information see Section 4.1.9 of the Handbook of Standards and Procedures, Version 2.0.

Identify Programs and Services

How do I recommend a program or service for review?

The Prevention Services Clearinghouse continues to identify programs and services for review using an inclusive process. The Prevention Services Clearinghouse staff have logged all program and service recommendations received since 2018. Programs and services may be identified from the following sources:

• A public call for recommendations on at least an annual basis. The latest public call occurred in the fall of 2024 and is now closed.
• Recommendations received via email to PreventionServices@abtglobal.com; and
• An environmental scan and an inventory of the literature.

All recommendations sent to the Clearinghouse are logged and considered, including those submitted in response to annual public calls and ad hoc requests submitted at any time.

A comprehensive list of programs and services that have been recommended for review is available on the website.

For more information on what to include in a program or service recommendation, see the next question “What information is helpful to include when I recommend a program or service?”

What information is helpful to include when I recommend a program or service for review?

Recommendations should be sent to PreventionServices@abtglobal.com. All submitters will receive acknowledgment of their submission’s receipt. Due to the large volume of submissions, the Prevention Services Clearinghouse is not able to inform individual submitters when the program or service may be prioritized for review.

Only Information Needed:

• Name of the program or service you are recommending: Some programs or services have changed names over time or are known by multiple names. If you are aware of multiple names, please list them.

Optional Information:

Additional information is optional but is helpful to include to support the review process.

• Version or adaptation of program or service (optional): If multiple versions or adaptations are available, please specify which you are recommending for review.
• Program or service contact information (optional): If available, also include contact information for the program or service developer, which may include the official program or service website, the organizational email address, and/or a contact phone number.
• Program or service area(s) addressed (optional): Specify: [1] mental health prevention and treatment programs or services, [2] substance use prevention and treatment programs or services, [3] in-home parent skill-based programs or services, and/or [4] kinship navigator programs.
• Citation(s) for available books, manuals, or other documentation (optional): Programs and services reviewed by the Clearinghouse must be clearly defined and replicable. Programs and services must have available written or recorded books, manuals, or other documentation that specify the components of the practice and describe how to implement or administer the practice. Please provide a citation with the title, author, date, and publisher for such materials.
• Indicate how the program or service manual(s) can be accessed (optional). If the manual is available to the public (for free or for purchase), please provide a link to the website where individuals may download, request, or purchase the manual. The purpose of this request is to ensure that Clearinghouse staff understand the program as it is designed to be delivered. Any materials included with a nomination will not be made public or re-distributed.
• Information about the child welfare relevance of the recommended program or service (optional). Evidence that the program or service is designed for, or is commonly used to serve, children, youth, young adults, and/or families receiving child welfare services (or populations similar to those receiving child welfare services or at-risk for receiving child welfare services) is considered.
• Information about the population(s) served for the recommended program or service (optional). The Prevention Services Clearinghouse records the populations the program or service is intended to serve and considers the particular needs of populations served across programs and services reviewed. Examples of population characteristics that may indicate particular need include status as an underserved community, tribal groups and nations, and needs associated with parent/caregiver or child age. Other considerations related to populations served may include the presence of culturally grounded or adapted programs and services, culturally adapted program or service content, program adaptations for different populations or age groups, studies conducted with diverse or underserved populations, and other information relevant to this criterion submitted to the Prevention Services Clearinghouse (e.g., via program or service recommendations, periodic engagement sessions, or other means).
• Information about available implementation supports for the recommended program or service (optional). The Prevention Services Clearinghouse prioritizes programs and services for which there are implementation supports, including (but not limited to) implementation manuals or frameworks, fidelity checklists or other fidelity-monitoring tools (such as regular site supervision meetings or regular submission of video-taped therapist sessions for quality monitoring), videos, training programs, coaching programs, or any similar resources available for potential program or service adopters. To meet this criterion, there must be affirmative, documented evidence that such supports are available to the public, either at no cost or for purchase.
• Information about eligible studies of the program or service recommended (optional): The Prevention Services Clearinghouse conducts a comprehensive literature search for each program or service prioritized for review but welcomes the submission of citations for studies and/or copies of studies or other supporting documentation (e.g., lists of published research). Any materials included with a recommendation will not be made public or re-distributed. For more information about the Prevention Services Clearinghouse study eligibility criteria, see Section 4.1 of the Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 2.0.

For more information about how the Prevention Services Clearinghouse prioritizes and selects programs and services for review, see Section 2.2 of the Handbook of Standards and Procedures, Version 2.0.

Select and Prioritize Programs and Services

What programs and services are eligible for review?

To be eligible for review, programs and services must:

• Meet eligibility criteria for at least one of the four program or service areas: (1) mental health prevention and treatment programs or services, (2) substance use prevention and treatment programs or services, (3) in-home parent skill-based programs or services, and/or (4) kinship navigator programs.
• Have books, manuals, or other documentation available: Programs and services must be clearly defined and replicable. To meet this criterion, they must have available written or recorded books, manuals, or other documentation that specifies the components of the practice and describes how to implement or administer the practice. There must be affirmative, documented evidence that the materials that satisfy this requirement exist and are available to the public to download, request, or purchase; materials may be presented in a web-based format. As part of ensuring that programs and services are clearly defined and replicable, sufficient information must be available about the program or service content, dosage, modality, providers, and/or other key components of the program or service.

To learn about the Clearinghouse’s standards and procedures, including further detail about program or service eligibility, please download the Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 2.0.

What are the next programs and services planned for systematic review?

The Clearinghouse has developed a working list of the next programs and services planned for systematic review. To learn more about this list, please visit the Programs & Services Planned for Review page.

When will new programs and services be added to the list?

Our goal is to review and rate as many programs and services as quickly as possible to support states’ efforts to improve outcomes for children and families through implementation of the Family First Prevention Services Act (FFPSA). The current list of programs and services planned for systematic review is available on the Programs & Services Planned for Review page.

The Prevention Services Clearinghouse will continue to select additional programs and services for review on a rolling basis. Join our email list to be notified of updates.

How does the Prevention Services Clearinghouse prioritize programs and services for review?

Given the high volume of recommendations, the Clearinghouse must prioritize programs and services for review. As described in the Handbook of Standards and Procedures, Version 2.0, the Prevention Services Clearinghouse prioritizes programs and services for review based on the following criteria:

• Available evidence of eligibility. The Prevention Services Clearinghouse prioritizes programs or services that have evidence they would meet the eligibility criteria in Section 2.1 of Handbook Version 2.0 if selected for review.
• In use/active. The Prevention Services Clearinghouse prioritizes programs or services that are in active use.
• Program or service recommendations received. The Prevention Services Clearinghouse considers both the total number of recommendations and the recommendation source. Particular consideration is given to programs and services recommended by state, tribal, or local government child welfare administrators and federal partners.
• Child welfare relevance. Evidence that the program or service is designed for, or is commonly used to serve, children, youth, young adults, and/or families receiving child welfare services (or populations similar to those receiving child welfare services or at-risk for receiving child welfare services) is considered.
• Population(s) served. The Prevention Services Clearinghouse records the populations the program or service is intended to serve and considers the particular needs of populations served across programs and services reviewed. Examples of population characteristics that may indicate particular need include status as an underserved community, tribal groups and nations, and needs associated with parent/caregiver or child age.
• Previous evaluations and studies. Information suggesting the program or service has been evaluated using an eligible study design is considered.
• Implementation supports. The Prevention Services Clearinghouse prioritizes programs and services for which there are implementation supports, including (but not limited to) implementation manuals or frameworks, fidelity checklists or other fidelity-monitoring tools (such as regular site supervision meetings or regular submission of video-taped therapist sessions for quality monitoring), videos, training programs, coaching programs, or any similar resources available for potential program or service adopters.

The Prevention Services Clearinghouse also prioritizes programs and services in a way that ensures representation of programs and services across the four program or service areas: mental health prevention and treatment programs and services, substance use prevention and treatment programs and services, in-home parent skill-based programs and services, and kinship navigator programs.

To learn more about how we prioritize and select programs and services for review, please see Section 2.2 of the Clearinghouse’s Handbook of Standards and Procedures, Version 2.0.

Can the Prevention Services Clearinghouse tell me why the program or service I have recommended has not yet been selected for review?

The Prevention Services Clearinghouse has received a high volume of program and service recommendations to date. This includes hundreds of emails received in response to each of the public calls that solicited recommendations for programs and services. Many of these recommendations identify multiple programs and services. The Prevention Services Clearinghouse staff log and consider all program and service recommendations and respond to submitters noting that their recommendations have been received. Given the high volume of recommendations, it is not currently feasible within resource and time constraints for staff to send individualized responses to submitters indicating why a program or service recommended has not yet been selected for review.

The Clearinghouse’s systematic prioritization process is outlined in the Handbook of Standards and Procedures, Version 2.0 (Section 2.2). The Prevention Services Clearinghouse is working to review and rate as many services and programs as quickly as possible.

A comprehensive, searchable list of programs and services that have been recommended for review is available on the website.

What does it mean when a program or service is listed as “not eligible for review at this time” on the “Find a Program or Service” page?

Programs and services are identified for review using an inclusive process. Once identified, the Clearinghouse then selects and prioritizes programs and services for review. The Clearinghouse maintains a working list of the next programs and services planned for systematic review. This can be found on the Programs & Services Planned for Review page on the website. Program or service eligibility criteria are then applied to the candidate programs and services on this list. Programs and services added to the working list may later be deemed ineligible for review if they do not meet these eligibility criteria. Reasons a program or service might not meet these eligibility criteria include: (1) the program or service does not meet the criteria for any of the four program or service areas eligible for inclusion, (2) a written or recorded book, manual, or other documentation that specifies the components of the practice and describes how to implement or administer the practice is not available to the public at this time, or (3) the Prevention Services Clearinghouse does not have sufficient information to define the program or service based on content, dosage, modality, providers, and/or other key components of the program or service and apply its adaptation procedures. When this happens, the program appears on the Programs and Services Reviewed page with a description of “This program or service is not eligible for review at this time.”

To learn about the Clearinghouse’s standards and procedures, including further detail about program or service eligibility, please download the Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 2.0. Program or service eligibility criteria are described in Chapter 2.

How does the Prevention Services Clearinghouse handle programs and services with more than one manual?

Many programs and services have more than one manual edition or version. Some of these are different editions created as a program or service evolves over time or expands beyond its original developer(s) (referred to here as manual editions). Other cases with more than one manual represent variants of a program or service, which may be designed to address new issues or different populations or may present alternative approaches to delivering the program or service (referred to here as manual variants).

When there is more than one manual edition available for a program or service, the Prevention Services Clearinghouse typically selects the most current publicly available manual edition as the focal manual for the program or service, based on the prioritization criterion for programs or services that are in active use (Section 2.2.2 of the Handbook of Standards and Procedures, Version 2.0).

If more than one manual variant of a program or service exists, the Prevention Services Clearinghouse will generally attempt to identify a focal manual that represents the standard or most comprehensive or complete version of the program or service under review. The Prevention Services Clearinghouse will typically treat each manual variant as describing a distinct program or service unless there is indication that the manual variant describes a program or service that is not substantially different from the program or service described in the focal manual identified (as outlined in the processes for identifying and assessing substantial adaptations in Section 2.3.2 of the Handbook of Standards and Procedures, Version 2.0).

Literature Search

Which other clearinghouses are used to identify relevant research?

As noted in Chapter 3 of the Handbook of Standards and Procedures, Version 2.0, the Prevention Services Clearinghouse begins the literature search by identifying citations from other evidence clearinghouses. A number of evidence clearinghouses overlap in content with the Prevention Services Clearinghouse. The list of clearinghouses may change and additional clearinghouses may be used, depending on the program or service selected. We currently identify relevant research from the following clearinghouses:

Study Eligibility Screening and Prioritization

Which studies are eligible for review?

To be eligible for review, a study must:

• Evaluate the version of the program or service currently under review.
• Use a randomized controlled trial (RCT) or quasi-experimental (QED) design with at least one intervention condition involving the program or service under review and one or more appropriate comparison conditions that utilize some form of control.
  • The Prevention Services Clearinghouse is planning a pilot to develop and implement standards for reviewing studies that use single case designs and determine how such designs may contribute to promising ratings.
• Measure and report program or service impacts on at least one eligible outcome.
• Be published or prepared in or after 1990.
• Be publicly available and published in journals or in reports prepared or commissioned by federal, state, tribal, or local government agencies or departments, private agencies or organizations, universities, research institutes, research firms, foundations or other funding entities, or other similar organizations.
• Be available in English.
  • The Prevention Services Clearinghouse is planning a pilot to understand the feasibility of identifying and reviewing studies only published in Spanish.
  • The results of the pilot will be communicated to the public via the Prevention Services Clearinghouse email list. To receive updates, please join our email list.

For more information about study eligibility criteria, see Section 4.1 of the Handbook of Standards and Procedures, Version 2.0.

Does the Prevention Services Clearinghouse have a minimum sample size requirement for studies to be eligible for review?

There is no minimum sample size for studies to be eligible for review by the Prevention Services Clearinghouse. The Handbook of Standards and Procedures, Version 2.0 describes the study eligibility criteria in Chapter 4, none of which include sample size restrictions. There were no sample size restrictions for studies to be eligible for review under the Handbook of Standards and Procedures, Version 1.0.

In the “Individual Study Findings” sections of the respective program and service pages, the Clearinghouse reports sample sizes for each study by contrast in the “N of Participants” column for all contrasts that meet moderate or high evidence standards. Examples of studies with favorable contrasts that meet the moderate or high evidence standards include:

• Study 11178
• Study 12908
• Study 10814

What are common reasons studies are not eligible for review?

Studies may be ineligible for a variety of reasons, and some studies are ineligible for multiple reasons. The most common reasons that studies are ineligible are listed below:

• They do not examine the version of the program or service currently under review. To be eligible for review, studies of a program or service must all represent similar implementations of the program or service under review; that is, programs or services may not be substantially modified or adapted from the book, manual, or other documentation (manual) or version of the program or service selected for review. The procedures for assessing study adaptations are addressed in Section 4.1.9 of the Handbook of Standards and Procedures, Version 2.0.
  • If a study indicates that content was substantially adapted, the Clearinghouse may determine that the study is not evaluating the program or service that was selected for review and is not eligible for review.
  • If a study implements an older version of a program or service selected for review and the older version of the manual is substantially different from the manual selected for review, the Clearinghouse may determine that the study is not eligible for review.
• The study does not use a design that establishes that the program or service, and not other factors, is responsible for the outcomes observed.
  • Eligible research designs include randomized controlled trial (RCT) or quasi-experimental (QED) designs with at least one intervention condition and at least one comparison condition. Not all studies are designed to support causal inferences about a program or service. For example, studies that use a pre-post design in which all study participants received the intervention cannot isolate the effect of the program from other factors that may explain changes in outcomes, such as children maturing or symptoms naturally improving over time. In such cases, the Clearinghouse cannot be confident that the outcomes reflect the causal impact of the program.

For more information, Section 4.1 of the Handbook of Standards and Procedures, Version 2.0 provides a complete list of study eligibility criteria. The Reporting Guide for Study Authors offers advice to help study authors describe their studies completely and consistently to help facilitate the Prevention Services Clearinghouse’s eligibility review process.

How does the Clearinghouse review process differ based on the number of eligible studies available for a particular program or service?

If a program or service has 15 or fewer eligible studies, all studies are reviewed using the design and execution standards and assessed for risk of harm.

If a program or service has more than 15 eligible studies, all eligible studies are assessed for risk of harm. Study review prioritization criteria are used to determine the order of eligible studies reviewed using the design and execution standards. Once ordered, the first 15 eligible studies will be reviewed using the design and execution standards. If, after review of 15 eligible studies, a program or service has not achieved a rating of well-supported, additional studies will continue to be reviewed using the design and execution standards in prioritization order only if there is potential for the program or service rating to improve.

For more information, see Section 4.2 of the Handbook of Standards and Procedures, Version 2.0.

What kinds of quasi-experimental designs are eligible for review by the Prevention Services Clearinghouse?

As described in Section 4.1.5 of the Handbook of Standards and Procedures, Version 2.0, currently eligible studies must use a randomized controlled trial (RCT) or quasi-experimental (QED) design with at least one intervention condition involving the program or service under review and one or more appropriate comparison conditions that utilize some form of control. Intervention and comparison conditions may be formed through either randomized or non-randomized procedures and the unit of assignment to conditions may be either individuals or groups of individuals (e.g., families, providers, centers).

The Prevention Services Clearinghouse has no specific requirements for how intervention and comparison conditions are formed for a quasi-experimental design. A wide range of designs can be eligible including, but not limited to, matched and non-matched designs, natural experiments, before-after-control-impact designs, controlled before-and-after designs, and difference-in-difference designs. Conditions may be formed through natural processes or formed by researchers. One example of natural variation would be if a state rolls out a new intervention with natural variation in the start dates across counties, such that there is a potential comparison condition of individuals who did not yet have the intervention available in their county. Researchers may also use statistical techniques to match individuals (or groups of individuals) who participated in an intervention to individuals who did not participate based on their measurable characteristics (e.g., propensity score matching). Intervention and comparison conditions may be formed before or after outcome data are collected.

What are the eligible outcome domains?

Studies must measure and report program or service impacts on at least one eligible outcome, as defined in Section 4.1.8 of the Handbook of Standards and Procedures, Version 2.0. The Prevention Services Clearinghouse classifies outcomes into outcome domains. The Prevention Services Clearinghouse outcome domains were selected to align with the important child and parent outcomes described in the authorizing legislation for the Prevention Services Clearinghouse, the Family First Prevention Services Act of 2018. There are four outcome domains used by the Prevention Services Clearinghouse for all programs and services: Child Safety, Child Permanency, Child Well-being, and Adult Well-being. Outcomes in three additional outcome domains (Access to Services, Referral to Services, Satisfaction with Programs and Services) are eligible for kinship navigator programs.

The eligible outcome measures listed in Handbook Version 2.0 under each outcome domain are examples and not an exhaustive list. The Prevention Services Clearinghouse may engage with individuals with lived expertise, individuals from relevant populations and communities, and other experts to ensure the inclusivity of eligible outcome measures used in varying cultural contexts.

What kinds of comparison groups are eligible for review by the Prevention Services Clearinghouse?

In alignment with the Family First Prevention Services Act, eligible RCT or QED studies must use an appropriate comparison condition. Comparison conditions for RCTs and QEDs must be exclusive (i.e., participants in the comparison condition may not overlap with those in the intervention condition).

As described in Section 4.1.7 of the Handbook of Standards and Procedures, Version 2.0, the Prevention Services Clearinghouse reviews the following types of appropriate comparison conditions:

• No intervention, untreated group, or wait list. Participants are offered no services or participants are assigned to receive the intervention under study at a later date.
• Minimal intervention. Participants may receive informational materials or psychoeducation, referrals to available services, or similar nominal services. Such additional services must be brief and/or predominantly information-based (e.g., pamphlets about child development, psychoeducation about a specific disorder).
• Placebo or attention control. Includes psychological or pharmacological placebos, attention placebos, and nonspecific therapy in which participants receive the same or similar amount of attention or contact as the participants in the intervention condition. Typically, these conditions are designed to account for nonactive effects of treatment, such as participants’ expectations, contact time with an interventionist, or the relationship between interventionist and participants.
• Treatment as usual. Participants in treatment as usual comparison conditions may already be receiving services in their communities or they may be offered services as part of the research study. The Prevention Services Clearinghouse considers a comparison condition to be “treatment as usual” under either of the following two conditions:
  • Condition 1: Usual or typical services. This condition refers to comparison conditions in which individuals are already receiving services in their communities or are offered services that they would have received in the absence of the study (i.e., they do not receive anything they would not have been able to receive anyway). In such cases, the study must clearly describe these services as the usual or typical services available for the population included in the study. The amount of contact and type or content of services can differ across participants. It is acceptable for studies with comparison conditions in this category to provide or offer a minimal intervention (as defined above) along with usual services. Studies may also constrain or standardize available usual services that are provided in the context of the study.
  • Condition 2: Services consistent with usual or typical services. This condition refers to comparison conditions that are offered services as part of the study that are not offered in the community but are clearly described as consistent with the usual or typical services that would be received by individuals or families similar to those in the study. In such cases, the study must provide specific information to justify that the services offered to the comparison condition participants are consistent with what individuals or families like those in the study could be expected to receive in the absence of a study.

Treatment as usual services may include therapeutic or pharmacological interventions that meet the criteria for either of these two conditions. If it is unclear from study documentation whether the services described in the study constitute usual or typical services offered in the community (Condition 1) or services consistent with usual or typical services (Condition 2), the Prevention Services Clearinghouse may query study authors to gather information needed to make this determination.

• Head-to-head comparisons. Head-to-head comparisons can also be referred to as alternative interventions, active interventions, active control interventions, or comparator interventions. Participants are assigned to another intervention that is not a variant of the program or service under review and does not meet the criteria for treatment as usual. This may include studies in which a new or emerging intervention is compared to an established one, studies that compare interventions that ascribe to different or contrasting theoretical orientations, or studies that compare two or more commonly used manualized interventions to each other. Excluded are comparisons to pharmacological interventions or psychotropic medications that do not meet the definition of treatment as usual above.

How does the Prevention Services Clearinghouse review studies with multiple comparison groups?

In studies with more than one eligible comparison condition, reviewers will review contrasts from all eligible comparison conditions against the eligible intervention condition(s) in the study.

Evidence Review

How can I tell why a study received a low design and execution rating?

When an individual study is listed in the Studies Reviewed section of a program page as having a rating of Low, the study citation is followed by the reason for this rating in bold-italic. These reasons include:

• This study received a low rating because it did not meet design confound standards. These standards are described in Section 5.9.3 of the Handbook of Standards and Procedures, Version 2.0.
• This study received a low rating because none of the target outcomes met measurement standards. These standards are described in Section 5.9.2 of Handbook Version 2.0.
• This study received a low rating because baseline equivalence of the intervention and comparison groups was necessary and not demonstrated. These standards are described in Sections 5.7 and 5.8 of Handbook Version 2.0.
• This study received a low rating because the standards for addressing missing data were not met. These standards are described in Section 5.9.4 of Handbook Version 2.0.
• This study received a low rating because it did not meet the statistical model standards. These standards are described in Section 5.9.1 of Handbook Version 2.0.

How does the Prevention Services Clearinghouse review bundled interventions?

Studies in which the program or service under review is delivered in combination (or bundled) with another program or service may exhibit a design confound. In such cases, it is impossible to separate how much of the observed effect was related to the intervention and how much to the confounding factor. Thus, the study cannot meet evidence standards and will receive a low rating (see Section 5.9.3. of the Handbook of Standards and Procedures, Version 2.0 for additional information on design confounds). We illustrate three scenarios of relevance to bundled interventions below:

• Intervention + Treatment as Usual vs. Treatment as Usual Only. Studies of bundled intervention conditions do not exhibit a design confound when the program or service under review is bundled with treatment as usual and the comparison group also receives treatment as usual.
• Intervention + Treatment as Usual vs. No/Minimal Treatment. Studies of bundled intervention conditions may exhibit a design confound when the program or service under review is bundled with treatment as usual but the comparison group does not receive treatment as usual. Studies with this configuration of conditions would only be determined to be free of a design confound if the treatment as usual services bundled with the program or service under review are weak, minimal, or unlikely to affect study outcomes.
• Intervention + Second Intervention vs. No/Minimal Treatment or Treatment as Usual. Studies of bundled intervention conditions in which the second intervention in the bundle is another program or service are ineligible for review unless both programs or services are selected for review as a bundle. If a bundle of services is selected for review, the eligibility criteria for comparison groups apply (Section 4.1.7).

Program and Service Ratings

Why are the Prevention Services Clearinghouse program and service ratings different from those of other clearinghouses?

The Prevention Services Clearinghouse uses unique standards and procedures developed in accordance with the statutory requirements detailed in the Family First Prevention Services Act (FFPSA) of 2018. For example, though the California Evidence-Based Clearinghouse for Child Welfare (CEBC) uses a peer-review process to review and rate studies of programs and services, the Prevention Services Clearinghouse is required to conduct an independent, systematic evidence review. In addition, different from CEBC, the Title IV-E Prevention Services Clearinghouse allows programs or services with rigorous quasi-experimental evidence to achieve a well-supported or supported rating. The Prevention Services Clearinghouse also includes review of publicly available research studies including those that do not appear in peer-reviewed journals.

Similarly, different from Home Visiting Evidence of Effectiveness (HomVEE) review, the rating criteria and outcome domains identified in FFPSA do not directly align with the criteria established by HHS for evidence-based home visiting models. For example, in accordance with the statutory requirement of FFPSA, the Prevention Services Clearinghouse criteria for sustained effects are based on length of time after the end of treatment, whereas HomVEE criteria for sustained effects are based on length of time after enrollment.

The Prevention Services Clearinghouse leverages other clearinghouses during the review process to gather information relevant to the program and service prioritization criteria, during the literature search, and as sources of information for the program and service descriptions, the target populations, and program and service delivery and implementation information.

Procedures for Reviewing Programs and Services for the Clearinghouse

Will the Prevention Services Clearinghouse re-review all previously reviewed programs using the new Handbook of Standards and Procedures, Version 2.0?

No. As noted within Section 8.5.1 of the Handbook of Standards and Procedures, Version 2.0, programs and services reviewed by the Prevention Services Clearinghouse under a prior version of the Handbook of Standards and Procedures may be considered for re-review under the current version of the Handbook of Standards and Procedures if a re-review has the potential to change the program or service rating. Requests for the re-review of a program or service rating should be submitted by emailing PreventionServices@abtglobal.com. To help facilitate consideration of the re-review, requests should specify the rationale for the request and provide any supporting documentation. Examples of supporting documentation could include citations for new research studies not previously reviewed or documentation of potential errors or missing information identified.

Can a program or service be re-reviewed?

Yes. As noted in Section 8.5.1 of the Handbook of Standards and Procedures, Version 2.0, programs and services reviewed for the Prevention Services Clearinghouse may be considered for re-review due to missing information or errors in the original review or due to the emergence of substantial new evidence that has the potential to change program or service ratings. Prevention Services Clearinghouse staff keep track of the dates that programs and services are reviewed and periodically assess the extent of new evidence available – including keeping track of programs or services where no eligible studies were identified during the review and considering ad hoc submissions of research to the Prevention Services Clearinghouse. Study authors and program or service developers can submit new evidence or studies as part of requests to re-review a program or service.

Programs and services reviewed by the Prevention Services Clearinghouse under a prior version of the Handbook of Standards and Procedures also may be considered for re-review under the current version of the Handbook of Standards and Procedures if a re-review has the potential to change the program or service rating. Program or service ratings assigned under any prior versions of the Handbook of Standards and Procedures remain in effect until such time that a re-review occurs.

Requests for the re-review of a program or service rating should be submitted by emailing PreventionServices@abtglobal.com. To help facilitate consideration of the re-review, requests should specify the rationale for the request and provide any supporting documentation. Examples of supporting documentation could include citations for new research studies not previously reviewed or documentation of potential errors or missing information identified.

How many times can a program or service be re-reviewed?

Programs and services can be re-reviewed multiple times provided that they follow the procedures noted in Section 8.5.1 of the Handbook of Standards and Procedures, Version 2.0, which indicate that programs and services reviewed for the Prevention Services Clearinghouse may be considered for re-review due to missing information or errors in the original review or due to the emergence of substantial new evidence that has the potential to change program or service ratings. There is no limit to the number of times a program or service may be re-reviewed.

Can an individual study of a program or service be re-reviewed?

Yes. As noted within Section 8.5.2 of the Handbook of Standards and Procedures, Version 2.0, individual studies reviewed by the Prevention Services Clearinghouse may be considered for re-review due to missing information or errors in the original review.

The Prevention Services Clearinghouse will conduct study re-reviews under the version of the handbook used to review the program or service. That is, for a program or service reviewed under Handbook Version 1.0 where the program or service has not been re-reviewed under Handbook Version 2.0, a study re-review would be conducted under Handbook Version 1.0. For a program or service where a program or service rating has been assigned using Handbook Version 2.0, study re-reviews would be conducted using Handbook Version 2.0.

If errors or missing information are identified, the Prevention Services Clearinghouse follows standard procedures for re-review. This includes assigning different, blinded reviewers to conduct any re-reviews. If the re-review determines the original review to be in error, the error is corrected on the website. All correspondence regarding re-reviews is logged and maintained by Prevention Services Clearinghouse staff. Requests for the re-review of an individual study should be submitted by emailing PreventionServices@abtglobal.com. To help facilitate the re-review, specify the rationale for the request and provide any supporting documentation.

When an individual study is listed in the Studies Reviewed section of a program page as having a rating of Low or Not Eligible for Review, the study citation is followed by the reason for this rating in bold-italic. For more information on these reasons, see the questions “How can I tell the reason why a study received a low rating?” and “What are common reasons studies are not eligible for review?” above.

For more information on why studies do not meet the Clearinghouse’s design and execution standards, or receive low ratings for design and execution, see Chapter 5 of the Handbook of Standards and Procedures, Version 2.0 and the fact sheet, Does the Prevention Services Clearinghouse Rate the Design and Execution of Studies?

How does the Prevention Services Clearinghouse handle requests to update a program or service manual citation if a new manual is released for a program or service that has already been reviewed?

The public may request an update of manual citation for the program or service on the basis of missing information, errors in citation, or the emergence of new information, such as a new manual edition being published.

As current priorities and resources allow, the Clearinghouse may elect to review the new manual to determine whether it is consistent with the manual reviewed. In considering whether an update to a manual citation is warranted, the Prevention Services Clearinghouse must have sufficient information available to be able to apply the procedures specified in Section 2.3 of the Handbook of Standards and Procedures, Version 2.0 for determining whether any substantial adaptations are present in the newer manual edition compared to the original edition reviewed. If there is insufficient information available to apply these procedures, the Prevention Services Clearinghouse will default to retaining the existing manual citation.

If the new manual is determined to be consistent with the manual that was reviewed, the Clearinghouse updates the program or service page to reflect the new manual edition. The original program page is then archived, but available via a link at the bottom of the current program page. To signal that changes have been made, the Clearinghouse updates the dates on the program or service page (see “Date Program or Service Description Updated”) and displays both manuals in the “Program or Service Documentation” section.

If the new manual is determined to be a substantial adaptation of the manual that was reviewed, the Clearinghouse considers the new manual to be a separate program or service. This separate program or service is then subject to standard procedures for selecting and prioritizing new programs to review and is dependent on the Clearinghouse’s current priorities and resources.

The Clearinghouse may also identify new manuals in the process of conducting a re-review of an existing program or service, even though the Clearinghouse may not have been notified of a new manual by the public. If a new manual is discovered in the process of conducting a re-review, the same procedures described above are applied to determine whether the manual citation is updated.