1. Overview
2. Background
3. Development
1. Goals
2. Development Team
3. Domains
4. Components
   1. Target Population
   2. Parent Baseline Assessment
   1. Diagnosis at Baseline
   2. Outcomes of Care
   3. Prognostic Characteristics
   3. Patient Baseline Assessment
   1. Diagnosis at Baseline
   2. Outcomes of Care
   4. Clinician Baseline Assessment
   1. Eligibility
   2. Diagnosis at Baseline
   3. Treatment
   5. Parent Follow-up Assessment
   1. Outcome of Care
   2. Treatment
   6. Patient Follow-up Assessment
   1. Outcome of Care
   2. Treatment
   7. Medical Record Review
5. Reliability & Validity
1. Gold Standards
2. Reliability
3. Validity
4. Sensitivity to Change
5. Feasibility
6. Potential Uses
1. Strengths
2. Limitations
7. Data Analysis and Presentation
1. Individual Patient Reports
2. Aggregate Patient Reports
3. Scoring
4. Usage
8. Summary
9. Bibliography

Overview   Top

Adolescents experience a variety of mental disorders, from severe emotional disturbance that requires prolonged and intensive treatment to more prevalent, but milder, disorders. The Adolescent Treatment Outcomes Module (ATOM) measures the outcomes of care by assessing the clinical status of adolescent patients with emotional and behavioral disorders, including depression, anxiety, attention deficit, oppositional defiant, and conduct disorders; the functioning of patients; family burden; and satisfaction with care. Data generated by the ATOM will enable users to make better treatment decisions, improve the quality of mental health care, and provide evidence of the utility of mental health services for adolescents. As the module is further refined, it may be used to determine the most effective and highest quality services by comparing outcomes of different treatments in similar settings or of similar treatments in different settings

Background   Top

It is estimated that as many as one in five adolescents in the United States suffer from emotional or behavioral disorders that are serious enough to require mental health services.(1-3) In spite of the more than $5 billion that is spent each year on services for these adolescents,(4,5) little evidence is available to show whether or not routine mental heath care is effective. Until the ATOM was developed, no comprehensive, standardized set of instruments was available that was brief, easy to administer, and inexpensive. (See Table 8.1 for a comparison of the ATOM with other available instruments.) The ATOM was designed to provide common, risk-adjusted outcomes data for youth receiving mental health treatment across settings and time. The ATOM can also determine whether efficacious interventions in controlled trials deteriorate when implemented as routine care or whether outcomes assessments adequately capture the range of effects on youth.

Table 8.1 Domains Assessed by ATOM and other Widely Used Instruments

ATOM=Adolescent Treatment Outcomes Module, CBCL=Child Behavior Checklist, DISC=Diagnostic Interview Schedule for Children, CAPA=Children and Adolescent Psychiatric Assessment, CAS=Child Assessment Schedule, DICA=Diagnostic Interview for Children and Adolescents, KSADS=Schedule for Affective Disorders and Schizophrenia, CAFAS=Children and Adolescent Functional Assessment Scale, CIS=Columbia Impairment Scale, CGAS=Children's Global Assessment Scale, VFI=Vanderbilt Functioning Index, BIS=Brief Impairment Scale, BCQ=Burden of Care Questionnaire, BAS=Burden Assessment Scale, CABA=Child and Adolescent Burden Assessment, CSQ=Client Satisfaction Questionnaire, VSS=Vanderbilt Satisfaction Scales, CASA=Child and Adolescent Services Assessment, CHIP=Child Health and Illness Profile, CHQ= Child Health Questionnaire.

Development  Top

Goal

The ATOM is designed to be used in routine clinical care settings as part of an outcomes management system for continuous quality improvement by assessing patients' symptom severity and functioning over time. The module provides information on a comprehensive set of variables that are relevant to understanding how treatment affects outcomes in adolescents with various emotional and behavioral problems. The module is used to measure the process and type of care patients receive, the outcomes of that care, and the prognostic factors that influence either the types or outcomes of care.

Development Team

A 13-member multi-institutional, multidisciplinary panel advised UAMS on the development of relevant clinical and methodological assessments for issues that surround the treatment and outcomes of adolescents with emotional and behavioral problems. The module's authors are James M. Robbins, Ph.D.; Kathryn M. Rost, Ph.D.; Audrey Burnam, Ph.D.; Barbara J. Burns, Ph.D.; J. Lynn Taylor, MD; G. Richard Smith, Jr., MD; and Teresa L. Kramer, Ph.D.

Domains

The ATOM covers three categories of outcomes. Clinical status is measured by assessing the presenting or focal problems and symptom severity. Functioning is measured by determining impairments in interactions in the home, school, community, and with peers; a general health status assessment is also made. A consumer perspective includes family burden and satisfaction with care assessments. Prognostic factors that may influence the outcome of care are also measured.

Components  Top

The ATOM consists of six components: Patient and Parent Baseline Assessments, Clinician Baseline Assessment, Patient and Parent Follow-up Assessments, and the optional Medical Record Review. The patient, parent, and clinician components are designed as self-administered questionnaires or personal interviews.

Target Population

Patients aged 11 to 19 years of age who experience moderate to severe functional impairments caused by emotional or behavioral problems, including depression, anxiety, attention deficit, oppositional defiant, and conduct disorders, are eligible for the ATOM protocol. Children younger than 11 years may participate if a reader is available to administer the assessments. A parent, guardian, or other knowledgeable person who has had contact with the adolescent during the previous six months must be available to serve as an informant. Exclusion criteria include mental retardation(IQ<70) and current psychotic episodes. Patients and informants must complete the Wide Range Achievement Test (WRAT)(6), a 1-2-minute assessment of reading ability. If a participant's literacy is below the 5th grade level, interviewers should assist participants completing the self-administered instruments.

Parent Baseline Assessment

Within three days of the initial visit for an emotional or behavioral problem, one of the adolescent's parents or caregivers should complete the assessment. The assessment provides information to establish a primary diagnostic grouping or caseness, symptom severity, functional impairment levels, and patient characteristics that may influence the prognosis. The 132-item assessment requires approximately 25 minutes to complete.

Diagnosis at Baseline

Several items establish a likely, though not definitive, diagnosis of conduct disorder, oppositional defiant disorder, major depression or dysthymia, anxiety disorder, or attention deficit-hyperactivity. ATOM uses items that match the Diagnostic and Statistical Manual of Mental Disorder (DSM-IV) criteria for these disorders.(7) These DSM-IV criteria are accepted by the Health Care Financing Administration and most third-party payers in their reimbursement classification systems.

Outcomes of Care

By examining how symptom levels and functioning change over time, the outcomes of treatment can be measured and addressed. The Parent Baseline Assessment measures symptom severity and six scales of functioning, including functioning in the home, school functioning, trouble with rules and laws, leisure functioning, friendship, and dysfunctional friends. Baseline responses are compared with follow-up responses to determine the outcomes of specific treatments. The ATOM also assesses sentinel indicators such as expulsion, school suspension, or frequent arrests.

Prognostic Characteristics

Prognostic characteristics that influence treatment outcomes are measured to allow comparisons of outcomes across sites, adjusting for important differences in patient populations. Variables that the ATOM measures include:

Patient Baseline Assessment

The Patient Baseline Assessment provides information from the adolescent on symptom severity and functional impairment levels to establish a primary diagnosis. The 132-item assessment takes approximately 20-25 minutes to administer.

Diagnosis at Baseline

Adolescents often disagree with parents when providing information needed to establish a diagnosis of emotional or behavioral disorder. By obtaining information from both adolescents and caregivers, the clinician is able to more fully understand the adolescent's primary problems. A brief set of items parallel to those in the Parent Baseline Assessment establish a likely, though not definitive, diagnosis of conduct disorder, oppositional defiant disorder, major depression or dysthymia, anxiety disorder, or attention deficit-hyperactivity. Detailed questions about substance use are also included in the assessment to determine the potential influence of drug and alcohol use on emotional and behavioral problems.

Outcomes of Care

The Patient Baseline Assessment measures symptom severity and functioning in the areas of family, school, and community. Baseline responses are compared with follow-up responses to determine the outcomes of specific treatments.

Clinician Baseline Assessment

This assessment, comprised of 24 items, is used to confirm the patient's eligibility for the ATOM, diagnosis at baseline, and treatments begun at baseline. It requires 1.5 minutes to complete.

Eligibility

The clinician should confirm that the patient is able to read the module and is not actively psychotic or mentally retarded.

Diagnosis at Baseline

The diagnosis of emotional or behavioral disorders and the symptoms that the clinician considers in making the diagnosis are recorded.

Treatment

Information on forms of therapy and medication prescribed during the index treatment are recorded.

Parent Follow-up Assessment

The follow-up assessment covers the six-month period after baseline. The information gathered establishes the current primary diagnostic grouping, symptom severity, and functional impairment levels. About 25 minutes are needed to complete this assessment.

Outcomes of Care

Measures of symptom severity and functioning in the family, school, and community are repeated at follow-up.

Treatment

The assessment collects reports of the mental health services used by the adolescent in the six months following the index treatment.

Patient Follow-up Assessment

The Patient Follow-up Assessment also covers the six-month period after baseline and is used to establish the current primary diagnostic grouping, symptom severity, and functional impairment levels. The questionnaire requires approximately 20 minutes to complete.

Outcomes of Care

Measures of symptom severity and functioning in the family, school, and community are repeated at follow-up.

Treatment

The assessment collects the adolescent's reports of the mental health services he/she used in the six months following the index treatment.

Medical Record Review

Characterization of the frequency and types of treatment provided by the provider or payer is completed through the optional 13-item Medical Record Review at a six-month follow-up. Treatment information includes data on the variety and amount of services received, involvement of the family and other agencies in treatment, and psychotropic medications prescribed. The review takes an average of 10 to 15 minutes to complete.

Reliability & Validity  Top

The reliability and validity of the ATOM were tested against gold standard instruments to determine its sensitivity to clinical changes and its feasibility for use in routine clinical settings. Self-reported data were collected from 72 persons, aged 11 to 18 years, who received treatment at two inpatient and two outpatient units. Both adolescents and parents were administered the ATOM and validating instruments within three days of the start of treatment and at six months after baseline.(8)

Gold Standards

An expert panel evaluated the validity of the content. Instruments used to validate the ATOM for criterion concurrent validity are shown in Table 8.3.

Reliability

Oppositional defiant, conduct, anxiety, depression, and attention deficit hyperactivity disorders were predicted by 10 self-reported symptoms. Test-retest correlations for 10 outcomes variables were excellent for parent report (r ³0.70). Adolescents were less consistent in their reporting (range=0.38-0.71). Correlations of the ATOM with the expected direction of global measures of functioning were moderate for measures of functioning at home, in school, and in the community. The test-retest reliability coefficients, internal consistency alpha coefficients for key measures, and correlations for caseness indicators and prognostic factors are shown in Table 8.2.(8)

Table 8.2 Test-Retest and Internal Consistency of Parent and Adolescent Responses on ATOM

(a) Intraclass correlation coefficient.
(b) Administered to parent only.
BAS=Burden Assessment Scale, CSQ= Client Satisfaction Questionnaire, FAD= Family Assessment Device

Validity

Responses on the ATOM were compared with those received on the relevant validation instruments which included the Child Behavior Checklist (CBCL),(9) the Child Health Questionnaire (CHQ),(10) and the Columbia Impairment Scale (CIS).(11) Almost all measures achieved significant correlations with established validation instruments (Table 8.3).(8) Focal problem severity was moderately correlated with parental responses but not with adolescents. Symptom severity was strongly associated with the CBCL and CHQ. Functioning scales were moderately correlated with the CIS and certain scales of the CHQ.

Table 8.3 ATOM with Established Concurrent Validation of Outcomes Measures and Sensitivity of Clinically Relevant Change (a)

(a) Spearman correlation
(b) CBCL=Child Behavior Checklist(9); CHQ=Child Health Questionnaire(12); CIS=Columbia Impairment Scale(11)

Sensitivity to Change

Changes in outcomes scores were correlated with changes in validation measures. The ATOM generally detected change in the expected direction. The predictive validity of prognostic factors for changes in outcomes was weak, with correlations in the range of 0.11 to 0.25.(8)

Feasibility

The ATOM is readily completed, via mailings, by adolescents and their parents. The most common reason for incomplete interviews in the field test was repeatedly missed appointments by parents.(8)

Potential Uses  Top

The ATOM generates a comprehensive set of measurements that are relevant for understanding how treatment affects outcomes in adolescents with conduct disorder, oppositional defiant disorder, depression, anxiety, or attention deficit-hyperactivity disorder.

Strengths

The ATOM has been field-tested on inpatients, outpatients, and court-referred adolescents. The ATOM was shown to be useful for sampling constructs from the full range of outcome domains that are relevant to adolescent treatment for certain emotional and behavioral problems. Field-testing has shown the module to be relatively brief to administer and acceptable to respondents. It also has good reliability and concurrent validity and is able to detect change and the absence of change in a patient's functioning and health status. As such, it can be used to assess the effectiveness of care and identify and address problems in quality of care.

Limitations

The ATOM module is not a static product. Additional research with large samples is needed to improve the sensitivity and specificity of some caseness indicators. In addition, age- and gender-specific norms have not yet been established for the key mental health indicators.

Data Analysis and Presentation  Top

The ATOM provides extensive data on a patient's demographic status, functioning, and symptoms that can be extremely valuable to the provider as treatment proceeds. In addition, aggregate reporting in these patient characteristics can be integrated into the system's quality improvement review and planning to provide "best practice" care for minors.

Individual Patient Reports

Individual patient reports can be generated via the Internet at www.NetOutcomes.net to provide data on diagnostic criteria, mental and physical health functioning, and other issues pertinent to adolescent development.

Aggregate Patient Reports

Aggregate reports can depict the age, sex, race, and other important background characteristics of the treatment population for programmatic and staffing needs. Other more clinically relevant indicators can demonstrate the overall effectiveness of a treatment process or assess what improvements should be undertaken. For example, a system may find from the ATOM data that a large majority of the adolescents have a comorbid disorder, such as attention deficit/hyperactivity, that has not been adequately addressed in the treatment plan.

The NetOutcomes^TM outcomes management systems have three types of aggregate reports-provider, program, and facility reports. Provider reports present individual providers with quarterly summaries of baseline and follow-up information as well as derived data such as the percentage of patients who receive guideline-concordant care. The provider's data is also compared to the provider's program data and to national benchmarks.

Program reports present data on each provider within a program as well as the aggregate data for the entire programmed national benchmark. Each provider is distinguished by a code known only to the provider and the clinical director.

Facility reports are used in situations where similar programs are to be compared within a facility or system of care. National benchmarks are also presented.

Scoring

Providers can gain access to the module and scoring through NetOutcomes^TM website at www.NetOutcomes.net.

Usage

The ATOM is protected under copyright and is available for unlimited free use, provided that there is no charge associated with its administration. To that end, UAMS permits unlimited reproduction and distribution of the module by the public for nonprofit, educational, or research purposes. However, any commercial use of our work, including the creation of electronic versions or other derivative works of the module for sale, constitutes a violation of the copyright of the university unless prior authorization has been granted in writing.

Summary  Top

Background.

The Adolescent Treatment Outcomes Module (ATOM) seeks to improve care for a variety of emotional and behavioral disorders experienced by adolescents, ranging from severe emotional disturbances that require prolonged and intensive treatment to more prevalent, but milder disorders. As many as one in five US adolescents may suffer from emotional or behavioral disorders serious enough to require mental health services. Although more than $5 billion is spent each year on these services, little evidence is available to show whether routine mental health care is effective. Until the ATOM was developed, no comprehensive, standardized set of instruments was available that was brief, easy to administer, and inexpensive to use.

Development.

The ATOM is specifically designed to be used in routine clinical care settings as part of an outcomes management system for continuous quality improvement by assessing patient symptom severity and functioning over time. The ATOM measures the outcomes of care by assessing clinical status of adolescent patients with emotional and behavioral disorders. It also measures the process and type of care patients receive and the prognostic factors that influence either the types or outcomes of care. A multi-institutional, multidisciplinary panel served as advisors to the University of Arkansas for Medical Sciences professionals who developed the ATOM's clinical and methodological assessments relating to treatment and outcomes of adolescents with emotional and behavioral problems. The assessments cover clinical status, functioning, the consumer perspective, and prognostic factors that may influence the outcome of care.

Components.

Patients aged 11 to 19 years who experience moderate to severe functional impairments caused by emotional or behavioral problems are eligible for the ATOM protocol. Younger children and patients reading below the fifth grade level can participate if a reader/interviewer is available to administer assessments. Patients with mental retardation and a current psychotic episode are excluded. A parent, guardian, or other knowledgeable person who has had contact with the adolescent during the past six months must be available to serve as an informant. The module consists of Patient, Parent, and Clinician Baseline Assessments; Patient and Parent Follow-up Assessments; a Medical Record Review, and a User's Manual. Individual reports and aggregate provider reports are also available. Obtaining information from both adolescents and caregivers allows the clinician to more fully understand the patient's primary problems because adolescents often disagree with their parents when providing information needed to establish a diagnosis of emotional or behavioral disorder. The parent assessment should be completed within three days of the initial visit by a parent or guardian who has had contact with the adolescent during the past six months. Follow-up assessments and an optional medical record review are completed at six-month follow-up. In keeping with the ATOM's emphasis on use in routine clinical settings, its basic components are relatively quick to complete. Parent and patient assessments take approximately 20-25 minutes to administer, and the clinician's assessment requires 1.5 minutes to complete.

The ATOM uses items that match the Diagnostic and Statistical Manual of Mental Disorder (DSM-IV) criteria to establish a likely, though not definitive, diagnosis of conduct disorder, oppositional defiant disorder, major depression or dysthymia, anxiety disorder, and/or attention deficit-hyperactivity. Detailed questions about substance use are also included in the assessment to determine the potential influence of drug and alcohol use on emotional and behavioral problems.

ATOM individual reports provide information on diagnostic criteria, mental and physical health functioning, and other issues pertinent to adolescent development. Aggregate patient reports can depict the age, sex, race, and other important background characteristics of the treatment population for programmatic and staffing needs. Other more clinically relevant indicators can demonstrate the overall effectiveness of treatment processes or assess what improvements should be undertaken.

Reliability and Validity.

ATOM's reliability and validity were tested against gold standard instruments using self-reported data from 72 persons, aged 11 to 18 years. Both adolescents and parents were administered the ATOM and validating instruments within three days of the start of treatment and at six months after baseline. Oppositional defiant, conduct, anxiety, depression, and attention deficit/hyperactivity disorders were predicted by 10 self-reported symptoms. Test-retest correlations for 10 outcomes variables were excellent for parent report, but adolescents were less consistent in their reporting. Correlations of the ATOM with the expected direction of global measures of functioning were moderate for measures of functioning at home, in school, and in the community.

When ATOM responses were compared with those on three relevant validation instruments, almost all measures achieved significant correlations. Focal problem severity was moderately correlated with parental responses but not with adolescent responses. Symptom severity was strongly associated with the Child Behavior Checklist and Child Health Questionnaire (CHQ). Functioning scales were moderately correlated with the Columbia Impairment Scale and certain scales of the CHQ. Changes in outcomes scores were correlated with changes in validation measures. The ATOM generally detected change in the expected direction. The predictive validity of prognostic factors for changes in outcomes was weak.

Potential Uses.

The ATOM generates a comprehensive set of measurements that are relevant for understanding how treatment affects outcomes in adolescents with several emotional and behavioral disorders. Field-testing has shown the ATOM to be relatively brief to administer, acceptable to respondents, and useful for sampling constructs for the full range of outcome domains that are relevant to adolescent treatment for certain emotional and behavioral problems. It also has good reliability and concurrent validity and is able to detect change and absence of change in a patient's functioning and health status. Additional research with large samples is needed to improve the sensitivity and specificity of some caseness indicators, and age- and gender-specific norms have not yet been established for the key mental health indicators.

Bibliography   Top