1. Overview
2. Background
3. Development
1. Goals
2. Development Team
3. Domains
4. Components
   1. Target Population
   2. Patient Baseline Assessment
      1. Diagnosis at Baseline
      2. Outcomes of Care
      3. Prognostic Characteristics
   3. Clinician Baseline Assessment
      1. Eligibility
      2. Diagnosis
      3. Prognostic Characteristics
      4. Treatment
   4. Patient Follow-up Assessment
      1. Outcomes
   5. Medical Record Review
5. Reliability & Validity
   1. Diagnostic Accuracy
   2. Reliability
   3. Concurrent Validity
   4. Remission
   5. Sensitivity to Clinical Change
6. Potential Uses
   1. Strengths
   2. Limitations
7. Data Analysis and Presentation
   1. Individual Patient Reports
   2. Aggregate Patient Reports
   3. Scoring
   4. Usage
8. Summary
9. Bibliography

Overview  Top

Substance abuse is a significant problem in our society. Treatment costs for substance abuse were approximately $14 billion in 1996. The Substance Abuse and Mental Health Services Administration (SAMHSA) estimates that approximately one million patients were in treatment on any given day in 1995. The Substance Abuse Outcomes Module (SAOM) measures the types of care received by adults under treatment for substance abuse or dependence, the outcomes of that care, and the prognostic factors that influence either the types or the outcomes of care. Data generated by this module may be used in understanding how treatment affects outcomes of patients with substance abuse or dependence, and, thus, how the outcomes of such treatment can be improved.

Background  Top

Almost one in every seven Americans over the age of 12 has used illicit drugs one or more times in the past year, according to estimates.(1) In addition, approximately 18% of men and 5% of women are classified as heavy drinkers, while 6% of men and 2% of women are reportedly intoxicated weekly. These figures for alcohol and drug abuse translate into billions of dollars spent annually on hospitalizations, medical care, accidents, injuries, and crime, with costs projected to increase in the coming years.

Research has demonstrated the effectiveness of treatment for substance abuse, with overall improvement in functioning and subsequent decreases in consumption and substance-related problem behaviors such as absence from work, arrests, and family conflicts. Observational studies such as the Drug Abuse Treatment Outcomes Study (DATOS) and Project Match indicate that treatment is effective, and Project Match indicates that treatment is efficacious.(2-4) Studies have also shown a relationship between treatment and decreased utilization of medical services over time.(5)

Although considerable work has been done in this area, little has been done to monitor care in clinical settings for the purposes of quality improvement. Specifically, little is known systematically about which treatments work for whom or which treatments are relatively more effective than others. One approach to developing these data is for patients and providers to participate in an outcomes management system with valid and reliable measures. The SAOM, which permits collection of data on the outcomes of care, prognostic variables, and treatment components, was developed for this purpose. Specifically designed for use in routine clinical care, the SAOM seeks to reduce the burden of assessment for individuals with alcohol and other drug problems. It also provides diagnosis and determines remission through patient self-report.

Development  Top

Goal

The SAOM identifies patients with substance abuse or dependence disorders using the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) criteria and assesses their symptom severity and functioning over time. It is a tool specifically designed for use in routine clinical care to systematically assess patient characteristics, treatment elements, and patient outcomes of care in order to monitor or improve the outcomes of care. This outcomes management system for continuous quality improvement efforts was developed by combining the psychometrically sound and clinically relevant measures and scales from a previously developed alcohol abuse outcomes module and the prototype for a drug abuse outcomes module.

Development Team

A multi-institutional, multidisciplinary panel advised UAMS on the development of relevant clinical and methodological assessments on substance abuse and dependence. The module was developed by G. Richard Smith, Jr., MD; Thomas Babor, Ph.D.; Audrey Burnam, Ph.D.; Kathryn M. Rost, Ph.D.; Robert Drake, MD; Barbara J. Burns, Ph.D.; and Kim Heithoff, Sc.D

Domains

The SAOM assesses diagnostic criteria to identify a homogeneous group of patients and determines whether patients meet a diagnosis of substance abuse or dependence based upon their own reports. The patient's general functioning is measured using the SF-36 or Health Status Questionnaire referenced previously. The patient's change in consumption of controlled substances, physical and emotional symptoms of dependence, and general functioning over time make up the outcomes domain. Prognostic variables or case-mix variables, including those that predict treatment seeking, choice of treatment modality, and sociodemographic characteristics, are included in the assessment to allow analysis of the relationships between treatment and outcomes. Treatment components included in the domain are treatment type, extent, and setting.(6)

Components  Top

The SAOM is made up of the following components: Patient Baseline Assessment, Clinician Baseline Assessment, Patient Follow-up Assessment, and the optional Medical Record Review.

Target Population

Patients aged 18 years or older may be entered onto the SAOM protocol after a clinician diagnoses substance abuse or dependence of alcohol or other drugs, except nicotine. Polysubstance disorders are also covered by the protocol.

Patient Baseline Assessment

The Patient Baseline Assessment, comprised of 110 items, is administered to all patients after diagnosis of a new episode of substance abuse. It provides the information necessary to establish a diagnosis, determine the severity of illness, and measure prognostic characteristics that may affect the outcomes of treatment received. Patients complete follow-up assessments at three and six months after baseline in order to assess patient outcomes and the processes of care. This self-administered questionnaire takes approximately 20 minutes to complete.

Diagnosis at Baseline

Using DSM-IV criteria, a diagnosis is established of alcohol and/or drug abuse or dependence. These diagnoses are accepted by the Health Care Financing Administration and most third-party payers in their reimbursement classification systems.

Outcomes of Care

By examining how substance use problems change over time, the severity of substance-related disorders and the outcomes of treatment can be measured and addressed. The Patient Baseline Assessment measures current consumption, general functioning using the SF-36, and substance-related problems. These problems include common physical, psychological, and social consequences of using controlled drugs and alcohol and related symptoms of abuse and dependence. Baseline responses are compared with follow-up responses to determine the outcome of specific treatments.

Prognostic Characteristics

Prognostic characteristics that influence treatment outcomes are measured to allow for comparisons of outcomes across sites, adjusted for inpatient differences in patient populations. Variables that the SAOM measures include:

Clinician Baseline Assessment

Eligibility

The clinician should determine that patients are able to understand and speak English, have no active psychoses, demonstrate adequate levels of cognitive functioning, and are sober at the time they complete the Patient Baseline Assessment. Patients who do not meet all of these eligibility criteria are excluded from the outcomes assessment protocol.

Diagnosis

Like the Patient Baseline Assessment, the Clinician Baseline Assessment is obtained at the beginning of a new episode of substance abuse treatment. Specific information on eligibility, diagnosis, prognostic characteristics, outcomes, and treatment is detailed by the clinician making the diagnosis of substance abuse or dependence immediately after seeing the patient. The clinician records a specific diagnosis of either substance abuse or dependence and the symptoms considered in making this diagnosis. The Clinician Baseline takes 1½ minutes to complete.

Prognostic Characteristics

The Clinician Baseline Assessment includes assessment of substance-related physical symptoms, such as hepatitis, cirrhosis, delirium tremens, cognitive deficits, and HIV.

Treatment

Information is collected to describe the treatment setting where the diagnosis was made and recommendations and referrals that were made during the visit when the substance disorder was diagnosed.

Patient Follow-up Assessment

Patients complete this follow-up questionnaire every three months after baseline for a minimum of six months. If substance abuse continues or recurs, follow-up is continued. The Patient Follow-up Assessment is designed to be completed in 20 minutes by patients who have not used any substances on the day of administration.

Outcomes

Disease-specific outcomes information gathered by the SAOM includes data on the patient's current substance abuse use, general functioning, symptom severity, bed days, and days lost at work. The module's scales assess measures of the quantity and frequency of substance abuse, as well as the consequences of substance abuse, including physical, interpersonal, intrapersonal, impulse-control, and social consequences.

Medical Record Review

The optional Medical Record Review, which characterizes treatment the patient received from the provider, is completed at the same time as the Patient Follow-up (every three months after baseline), if it is used. This six-item Medical Record Review may take 15-25 minutes to complete when records are reviewed manually, but requires substantially less time if patient records are computerized. The only outcome measured in the Medical Record Review is mortality, including cause of death.

Table 7.1 Domains and Content of the SAOM

Instruments-AUDIT=Alcohol Use Disorders Identification Test(7), DIS=Diagnostic Interview Schedule (8), DrinC2R(9), MOS=Medical Outcomes Study (10), National Health Survey (11), SF-3626, Typology Questionnaire (12), Your Workplace (13)

Reliability & Validity  Top

Strengths

Like its predecessor, the Alcohol Outcomes Module, the SAOM has a clear advantage over other tools because it is able to make substance abuse diagnoses and determine remission by patient self-report. In addition, SAOM users can use key constructs with good reliability and validity to discern clinical improvement in groups of patients based on differences in treatment. The cost of data collection is significantly reduced because there is no need to train interviewers or clinical staff to administer the module's self-reporting sections and because these sections can be mailed or administered by phone.

Limitations

Future studies are needed to determine the usefulness of these data for monitoring and improving patient outcomes, and the SAOM will need to prove itself in the clinical settings for which it was designed. Further research is also needed to determine if the variables that predict outcomes should be differentially weighted.

Potential Uses  Top

Individual Patient Reports

Depending on the needs of the organization, individual reports can be created from the SAOM to provide information about the patient's specific substance abuse symptoms, symptom severity, and aspects of general functioning obtained from the SF-36. These individual reports can be used to track a patient's progress and alert the clinician to changes in symptomatology, including the severity of critical items. Individual patient reports may be obtained from www.NetOutcomes.net.

Aggregate Patient Reports

As mentioned previously, statistics can either be descriptive or inferential. Probably, in the initial stages of using the outcomes management systems, organizations will be interested in baseline descriptive statistics. These statistics characterize substance patients by demographics, prior mental health status, comorbid physical and psychiatric conditions, and level of disease-specific and generic symptoms. This information, which allows organizations to profile the types of patients who present for treatment of substance abuse symptoms, can be used for treatment planning and potential system remediation.

Inferential statistics should be conducted whenever there is a large enough group to conduct more complex data analyses. These analyses may indicate differences between groups or highlight prognostic variables that predict treatment failure. If an organization is interested in pursuing this level of data analysis, it is recommended that a statistical consultant be included in the project's developmental stages to advise the organization on power calculations, software purchases and utilization, and appropriate statistical tests.

Scoring

Providers can also gain access to the user's manual and modules through NetOutcomes^TM website at www.NetOutcomes.net and have individual patient reports scored automatically without a charge. Scoring for programming and facility reports is available from the website for a modest fee.

Usage

The SAOM is protected under copyright and is available for unlimited free use, provided that there is no charge associated with its administration. To that end, UAMS permits unlimited reproduction and distribution of the module by the public for nonprofit, educational, or research purposes. However, any commercial use of our work, including the creation of electronic versions or other derivative works of the module for sale, constitutes a violation of the copyright of the university unless prior authorization has been granted in writing.

Data Analysis and Presentation  Top

The reliability and validity of the SAOM was tested in a longitudinal study(14) of 100 patients who were beginning treatment for an episode of DSM-IV alcohol and/or drug abuse/dependence at an outpatient methadone maintenance treatment center and two private psychiatric hospitals. Many of these patients had multiple substance abuse/dependence diagnoses. Instruments used in this validation study included the Addiction Severity Index (ASI) 5th Edition,(15) the Alcohol Use Disorders Identification Test (AUDIT),(16) the Diagnostic Interview Schedule (DIS),(8) the Inventory of Drug Use Consequences (INDUC-2R) (Personal communication with Scott Tonigan, Ph.D.), the Medical Outcomes Study Social Support Survey (MOS),(17) the Substance Abuse Module of the Composite International Diagnostic Interview (CIDI-SAM),(18) and the Timeline Follow-Back Assessment.(19)

Diagnostic Accuracy

The accuracy of the SAOM's diagnostic component is key to its use as a clinical assessment tool. At baseline, the 17-item diagnostic component of the SAOM had excellent internal reliability, with an alpha of 0.89. There was a 93% agreement between this component and the research standard, (the CIDI-SAM) regarding the presence of a substance abuse or dependence diagnosis. Instances of non-agreement were evenly balanced between over- and under-diagnosis by the SAOM's diagnostic component. A similar analysis of the diagnostic accuracy of the module at follow-up showed agreements in the high range (89-90%) and with a kappa of 0.76 when compared to the CIDI-SAM.

Reliability

Case-mix variables for patient characteristics generally had good to excellent internal reliability, with all but two alphas at 0.89 or above. The Interclass Correlation Coefficient (ICC) and kappa used to measure the test/retest reliability of key constructs for patient characteristics were also strong, with most of the case-mix variables above 0.90. The consequences scales demonstrated high internal consistency. Both the substance use variables, which include multiple measures of quantity and frequency of substance use, and the consequences of substance use categories showed high test-retest reliability, with most ICCs or kappas above 0.80. Table 7.2 presents these data in detail.

Table 7.2 Reliability of Key Constructs of the SAOM

(a) ICC= Interclass Correlation Coefficient

Concurrent Validity

SAOM patient characteristics variables, including case-mix variables and both medical and psychiatric comorbidities, produced correlations ranging from 0.36 to 0.94 when compared with the corresponding research standards. These correlations were all 0.48 or higher, with the exceptions of severity of drug abuse at baseline and medical comorbidity. All chi square values were strong and all kappa values were in the moderate to strong range. Correlations between the patient outcomes variables and the research standards were very high, all exceeding 0.60 with the exception of alcohol quantity/frequency at baseline. The chi square and kappa comparisons were also very high. These data are presented in Table 7.3.

Table 7.3 Concurrent Validity of Patient Characteristics and Patient Outcomes Variables at Baseline and Follow-up

(a)ASI=Addiction Severity Index; AUDIT=Alcohol Use Disorders Identification Test;

CIDI-SAM=Substance Abuse Module of the Composite International Diagnostic Interview; DIS-Section=Diagnostic Interview Schedule;

INDUC-2R=Inventory of Drug Use Consequences; MOS=Medical Outcomes Study; TFB=Timeline Follow-Back

Remission

Validating the SAOM's ability to assess remission for substance abuse according to DSM-IV criteria is important because follow-up remission rates are often used as an indication of quality of care. The diagnostic component's sensitivity to determine remission compared to the CIDI-SAM was 82.5%, and the specificity was 94.4%. It had a positive predictive value of 98.1%; its negative predictive value was 39.3%; and the kappa was 0.64.

Sensitivity to Clinical Change

Analyses of the SAOM's sensitivity to clinically important change indicated that its variables change in the clinically expected direction, with all effects-size scores positive. With the exception of two subscale scores from the SF-36, all scores were in the moderate or large range, indicating that the SAOM measures are sensitive to important clinical changes.

Summary  Top

Background.

The SAOM seeks to improve care for individuals suffering from substance abuse problems, which cost our society billions of dollars annually. Although research has demonstrated the effectiveness of treatment for substance abuse, with subsequent decreases in consumption and substance-related problem behaviors, little has been done to monitor care in clinical settings for the purposes of quality improvement.

Development.

The SAOM is a tool specifically designed for use in routine clinical care to systematically assess patient characteristics, treatment elements, and patient outcomes of care in order to monitor or improve the outcomes of care. A multi-institutional, multidisciplinary panel served as advisors to the University of Arkansas for Medical Sciences professionals who developed the SAOM's clinical and methodological assessments on substance abuse and dependence. The assessments include domains for diagnostic criteria, general patient functioning, outcomes, prognostic variables, and treatment components.

Components.

Patients aged 18 years or older may be entered into the SAOM protocol after a diagnosis is established of alcohol and/or drug abuse or dependence using the DSM-IV criteria. The module consists of Patient and Clinician Baseline Assessments, a Patient Follow-up Assessment, a Medical Record Review, and a User's Manual. Individual and aggregate patient reports are also available. Baseline assessments are administered to all patients after diagnosis of a new episode of substance abuse.

The follow-up assessment and optional medical record review are completed three months after baseline for a minimum of 12 months. In keeping with the SAOM's emphasis on use in routine clinical settings, its basic components are relatively quick to complete. Both self-report patient assessments can be completed in about 20 minutes, and the clinician's assessment usually takes less than 5 minutes to complete.

SAOM individual reports can provide information about a patient's abuse severity, dependence severity, or general functioning and can be used to track a patient's progress and alert the clinician to changes in symptomatology. Aggregate patient reports can provide both descriptive and inferential statistics, although the more complex data analyses for inferential statistics require data from large groups of patients. Descriptive statistics characterize substance abuse patients by demographics, prior mental health status, comorbid physical and psychiatric conditions, and level of disease-specific and general symptoms. They also allow an organization to profile the kind of patients who present for substance abuse treatment. Analyses using inferential statistics can indicate differences between groups and highlight prognostic variables that predict treatment failure.

Reliability and Validity.

An 11-month SAOM validation study indicated a strong agreement between the SAOM's diagnostic component and the Substance Abuse Module of the Composite International Diagnostic Interview (CIDI-SAM) regarding the presence of a substance abuse or dependence diagnosis at baseline and follow-up. This statistic is important because the accuracy of the SAOM's diagnostic component is key to its use as a clinical assessment tool. The study also showed high internal reliability and strong test-retest reliability. Concurrent validity of patient characteristics and patient outcomes variables with research standards was strong overall. The sensitivity of the SAOM's diagnostic component to determining remission compared to the CIDI-SAM was 82.5% with a 94.4% specificity and a 98.1% positive predictive value. This validation of the SAOM's ability to predict remission for substance abuse according to DSM-IV criteria is very important because follow-up remission rates are often used as an indication of quality of care. In addition, each of the module's case-mix measures significantly predicted at least one patient outcome, and SAOM measures were shown to be sensitive to important clinical changes.

Uses.

The SAOM's abilities to make substance abuse diagnoses and determine remission by patient self-report give it clear advantages over other tools. Another strength is that good reliability and validity for key constructs allow users to discern clinical improvement in groups of patients based on differences in treatment. The module also offers significantly reduced cost of data collection because of the self-reporting sections. One of the module's limitations is that additional studies are needed to determine the usefulness of these data for monitoring and improving patient outcomes. Another is that the SAOM will need to prove itself in the clinical settings for which it was designed.

Bibliography  Top