If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Emergency Department Safety Assessment and Follow-up Evaluation Screening Outcome Evaluation examined whether universal suicide risk screening is feasible and effective at improving suicide risk detection in the emergency department (ED).
A three-phase interrupted time series design was used: Treatment as Usual (Phase 1), Universal Screening (Phase 2), and Universal Screening + Intervention (Phase 3). Eight EDs from seven states participated from 2009 through 2014. Data collection spanned peak hours and 7 days of the week. Chart reviews established if screening for intentional self-harm ideation/behavior (screening) was documented in the medical record and whether the individual endorsed intentional self-harm ideation/behavior (detection). Patient interviews determined if the documented intentional self-harm was suicidal. In Phase 2, universal suicide risk screening was implemented during routine care. In Phase 3, improvements were made to increase screening rates and fidelity. Chi-square tests and generalized estimating equations were calculated. Data were analyzed in 2014.
Across the three phases (N=236,791 ED visit records), documented screenings rose from 26% (Phase 1) to 84% (Phase 3) (χ2 [2, n=236,789]=71,000, p<0.001). Detection rose from 2.9% to 5.7% (χ2 [2, n=236,789]=902, p<0.001). The majority of detected intentional self-harm was confirmed as recent suicidal ideation or behavior by patient interview.
Universal suicide risk screening in the ED was feasible and led to a nearly twofold increase in risk detection. If these findings remain true when scaled, the public health impact could be tremendous, because identification of risk is the first and necessary step for preventing suicide.
ED-SAFE used an interrupted time series design with three phases:
Treatment as Usual;
Universal Screening; and
Universal Screening + Intervention.
Eight participating sites were randomly assigned to one of four cohorts, and each cohort was randomly assigned to a start date and progressed through the three phases sequentially (random stepped wedge design).
The Screening Outcome Evaluation used data from all three phases. The study was approved by the IRBs of all participating institutions. Because primary data were collected by chart review, the requirement for informed consent was waived. Patients who were approached for additional questioning provided verbal informed consent to be interviewed. All adult ED patients who were treated in one of the eight participating sites during any of the three study phases were eligible for chart review.
Eight hospitals located in seven states participated. Annual ED census ranged from 31,000 to 54,000. Like most U.S. EDs, participating sites did not have dedicated psychiatric EDs
to increase the representativeness of the findings.
At baseline, sites generally screened for suicide risk only among those who presented with frank psychiatric symptoms. One site had already implemented universal screening during Phase 1 using a single-item screener created by the site. This implementation occurred after the site had been selected for participation (at the time of selection, it did not have universal screening) but before Phase 1 had started and was attributed to the hospital’s risk management concerns about complying with Joint Commission requirements.
Implementing Universal Screening
The investigative team and representatives from all eight sites used the best evidence available to create a screener feasible to implement in emergency settings. The Patient Safety Screener-3 (PSS-322) assesses depressed mood, active suicidal ideation in the past 2 weeks, and lifetime suicide attempt. If lifetime attempt is endorsed, the timing of the most recent attempt is identified. A positive screen was defined as active suicidal ideation in the past 2 weeks or a suicide attempt within the past 6 months. The depression item is not factored into screening results; it was included as segue to help “ease into” the suicide items, which can be jarring when asked without context. A positive screen on the PSS-3 has strong agreement (κ=0.95) with the Beck Scale for Suicide Ideation,
Prior to Phase 2, each site convened a performance improvement team to develop strategies the site could use to improve implementation of universal screening. The team composition and performance improvement methods followed those commonly used by the Institute for Healthcare Improvement,
an industry leader. The team was multidisciplinary, including practicing clinicians to administrators, and focused on using continuous quality improvement cycles. The Plan–Do–Check–Act (PDCA) strategy was used, which included planning the desired changes (Plan); implementing them (Do); measuring performance (Check); and making adjustments (Act). The PDCA process was repeated as often as needed. Each site’s team received centralized training on these methods and their relationship to suicide risk screening to ensure a common knowledge base. Although the precise clinical protocols were not dictated, there were two requirements:
The site needed to use the PSS-3.
The PSS-3 was to be administered universally for all patients, with notation if the individual was inappropriate to screen because of his or her medical condition (e.g., unconscious, cognitively disabled).
Screening during triage for patients who presented with a psychiatric complaint was maintained. However, screening during triage for individuals with non-psychiatric complaints was strongly discouraged because of concerns about slowing ED flow. For most sites, the PSS-3 was administered during the primary nursing assessment once the patient was placed in a treatment area. Some sites chose to use the PSS-3 during triage for psychiatric patients as well. Each site’s performance improvement team created written clinical protocols, which stipulated how positive screens would be managed.
Nurses at each site were either trained by trainers recruited from the site’s clinical staff or through a webinar provided by the ED-SAFE trainer. Each site’s performance improvement team was responsible for collecting its own data on performance independent of the research data (e.g., screening rates); making adjustments to protocols as needed; and providing feedback, additional training, and other small incentives (e.g., $5 gift cards and lunches) to promote performance. An a priori “Excellence” screening rate of 75% was set by consensus.
During Phase 3, sites continued to improve screening using PDCA cycles. See Methods paper18(pp15–16) for additional information on the additional interventions introduced during Phase 3.
Across all phases, sites staffed the ED with research assistants (RAs) at least 40 hours/week during peak volume hours (12noon to 10:00pm), with at least 1 weekend day/month. As both volume and enrollment rates decline after 10:00pm, this efficiently maximized the representativeness of the sample. All adult patients who entered the ED during data collection shifts were documented on a Screening Log, and the RA reviewed their medical charts in real time throughout the visit until the patient was discharged. This was done both to obtain screening data (goal of the present study) and to identify subjects for the longitudinal clinical trial. Although the RAs were not blinded to phase, additional data sources were used to expand and validate outcomes abstracted by chart review, including patient interview and random chart review. Information on the fidelity interviews used to validate medical record documentation is reported in Appendix A (available online). Data were collected from 2009 through 2014 and analyzed in 2014.
The study was directed at improving suicide risk screening and detection. There were two primary outcomes. The first outcome, documentation of intentional self-harm ideation or behavior screening, was defined broadly as any documentation of past or current intentional self-harm ideation or behavior appearing in the record as either present or absent. The second outcome, intentional self-harm risk detection, was defined as past or current intentional self-harm ideation or behavior documented as positive in the ED medical record. A broad initial threshold comprising any intentional self-harm ideation or behavior was used because ED documentation may be vague or incomplete as to whether intentional self-harm is suicidal or not, particularly during treatment as usual, and could introduce error and unreliability across raters (Limitations section). Consequently, the protocol tracked documentation of any intentional self-harm ideation or behavior, not just suicidal self-harm ideation or behavior, and clarified the nature of the self-harm using a subsequent direct research interview with the patient. Passive ideation only, such as a desire to go to sleep and not wake up, was not considered a positive screen.
The Screening Log extracted data from the medical record. The RA recorded demographic information, whether any screening for intentional self-harm ideation or behavior was documented anywhere on the patient’s ED medical record, and whether self-harm ideation or behavior was documented as present (positive screen) or absent (negative screen). A positive screen was further classified by the RA using chart documentation only as current (i.e., self-harm ideation or behavior occurring during or immediately preceding the current ED visit); past (i.e., noted in the past but not currently); or unknown time (i.e., documented without a clear time reference).
In addition, through all three phases, an RA approached all individuals with documentation reflecting intentional self-harm ideation or behavior and directly interviewed the patient. Patients who were incarcerated or too medically, cognitively, or emotionally ill to be interviewed were excluded. Intentional self-harm ideation or behavior was further clarified as suicidal ideation or behavior within the past week, including the day of the visit (yes/no).
Because RAs did not staff the ED 24 hours/day, 7 days/week, a separate sample of 2,400 randomly selected charts was collected (100/site X 8 sites X 3 phases) as a validation sample. The RA recorded whether there was documentation in the chart of
any intentional self-harm ideation or behavior screening;
any intentional self-harm ideation or behavior detection;
suicidal ideation (past or current); and
suicide attempts (past or current).
More information on these procedures is published.
Analyses were performed using Stata, version 13.1. Data are presented as proportions with 95% CIs and medians with interquartile ranges. Changes in documentation of screening and detection were evaluated by analyzing data from the Screening Log using chi-square tests. Analyses were repeated using random chart review data as a confirmation of trends. To examine trends across the three study phases and control for potential clustering by site, generalized estimating equations with a logit link function were used. All p-values are two-tailed, with p<0.05 considered statistically significant. Time series figures displaying screening and detection rates were generated from Screening Log data.
Across the three phases, RAs reviewed 236,791 ED visit records (Figure 1, Table 1), identified 10,625 patients with any intentional self-harm ideation or behavior, and completed 3,101 patient interviews. The demographics were compared to the National Hospital Ambulatory Medical Care Survey,
Documentation of screening for any intentional self-harm ideation or behavior improved from 26% in Phase 1 to 73% and 84% in Phases 2 and 3, respectively (χ2[2, n=236,789]=71,000, p<0.001, p for trend adjusted for site clustering <0.001). Similar increases were noted using random chart review data (24%, 72%, and 84% in Phases 1, 2, and 3, respectively [χ2(2, n=2,398)=679, p<0.001, p for trend adjusted for site clustering <0.001]). The rate of screening across phases for each site can be seen in Figure 2. The Phase 3 screening rates by site ranged from 63% to 98%, with seven sites reaching >75%.
Detection of any intentional self-harm ideation or behavior rose from 2.9% in Phase 1 to 5.2% and 5.7% in Phases 2 and 3, respectively (χ2[2, n=236,789]=902, p<0.001), with the largest increase occurring between Phases 1 and 2 (p with adjustment for site clustering=0.008) (Table 2). Documentation of intentional self-harm ideation or behavior in the week prior to the visit rose from 2.7% to 3.9% (χ2[2, n=236,789]=205, p<0.001), whereas past intentional self-harm ideation or behavior at any time other than the past week rose from 0.3% to 1.9% (χ2[2, n=236,789]=1,200, p<0.001). Analyses with random chart review data revealed similar increases (2.8%, 5.8%, and 7.3% in Phases 1, 2, and 3, respectively, χ2[2, n=2,398]=17, p<0.001). Appendix Figure B1 (available online) depicts the risk detection rates across all three phases over time. Multivariable logistic regression models adjusting for patient clustering by site and demographics did not materially affect the results (Appendix Table B1, available online).
Table 2Screening Log and Random Chart Review Risk Detection Rates by Phase
Direct patient interview data (n=3,101) found that, across all three phases, approximately 74% (95% CI=73%, 76%) of patients with any intentional self-harm reported suicidal ideation or an attempt within the past week, supporting that the majority of the self-harm detection was relatively recent suicidal ideation or behavior.
Although research has repeatedly suggested that ED patients have significant undetected suicide risk, this is the first study to address the key question of whether detection feasibly can be increased by implementing universal suicide risk screening protocols in the ED. Two main conclusions can be reached. First, increasing documented suicide risk screening rates in the general adult ED population during routine care was achieved using a simple screener and Institute for Healthcare Improvement performance improvement protocols. Second, increased screening led to nearly twice as many patients being identified as having suicide risk, either by virtue of endorsing active suicidal ideation or a past suicide attempt. If applied to the approximately 350,000 ED visits that occur annually at the eight participating EDs alone, nearly 10,000 additional patients with previously undetected suicide risk would be identified every year.
Documented screenings rose from an average of 26% in Phase 1 to an average of 84% in Phase 3, representing an increase of more than 300% (Figure 1). Moreover, this increase in screening led to a nearly twofold increase in identification of patients positive for suicide risk. Documentation of any past or current intentional self-harm ideation or behavior nearly doubled over the course of the study from 2.9% to 5.7%. Patient interviews confirmed that the majority of this intentional self-harm was suicidal ideation or behavior that occurred in the weeks before the ED visit. Findings using data obtained during active research shifts, which disproportionately captured hours from 9:00am to 10:00pm, were reinforced by findings from random chart reviews. Furthermore, the positive trends in screening and detection persisted even after adjusting for factors such as site, demographics, and fidelity (Appendix A, available online).
Importantly, steps were taken to foster successful screening adoption. A simple screening instrument and clinical protocols designed to be easily integrated into the ED routine were developed. Training was brief and available through multiple channels, including in-person by a site trainer or online. Sites used widely available performance improvement methodologies, including PDCA cycles, for integrating the protocols into routine care and monitoring performance.
There are several potential limitations with this study. The participating EDs, though quite diverse, may not represent the nation’s EDs. Further study will be needed to examine whether these protocols, and their subsequent increase in risk detection, can be successfully translated to other EDs.
The RAs were not blinded to study phase. The most important bias this could have caused was to artificially inflate the improvements in suicide risk screening and detection. However, the authors reduced the potential for this bias in several ways. First, RAs were trained on research protocols, which were highly structured, designed to minimize the role of RA interpretations, and closely monitored using both site-specific and centralized quality assurance procedures. Second, RAs were not involved in any of the intervention training or activities. Finally, RAs interviewed all patients who screened positive on the chart to validate the documentation with the patient. If the RAs had been influenced by their knowledge of phase and artificially inflated the documentation of positive screenings, there would have been more patients identified as a positive screen on the chart but who denied any intentional self-harm. This rarely happened (less than 1% of the time across the study).
The study did not randomize individuals or sites due to study design, ethical, and legal considerations (described in Methods paper
). The quasi-experimental design could have introduced a change in clinician behavior simply as a result of being observed (Hawthorne Effect). There was no way to eliminate or measure this, but research protocols were designed to be unobtrusive. The random chart reviews captured data from all days and times to provide an indicator of performance less likely to be influenced by RA presence. Finally, the time series plots show a trend of performance improving over time, which is the opposite of tapering trends usually observed with a Hawthorne Effect as clinicians acclimate to being observed.
The RAs were trained to first identify any documentation of intentional self-harm, whether it was specifically noted as suicidal or not, and to interview these patients to ascertain whether the self-harm was suicidal. This was done because clinical documentation may be inadequate to determine intent. Relying on RAs to ascertain if the self-harm noted was suicidal based solely on documentation might have excluded valid cases. This would have been especially true during treatment as usual prior to the systematic changes in protocols that trained clinicians to document suicidal ideation and behavior more precisely using the PSS-3. The most likely impact of including only charts with clear documentation of suicidal self-harm would be to artificially inflate the screening rates for Phases 2 and 3 relative to Phase 1. Clinicians simply could have become more specific in their self-harm documentation as a result of training and improved templates, rather than increasing screening. Consequently, the inclusion criterion was broad, followed by direct interview to ascertain whether the intentional self-harm documented was indeed suicidal in nature. Patient interviews and careful examination of PSS-3 response patterns in Phase 3 confirmed that the vast majority of intentional self-harm was composed of recent suicidal ideation or behavior.
Such a remarkable and robust increase in screening by clinicians during routine care in busy EDs, and the subsequent increase in risk detection that would not have been identified if universal screening had not been implemented, has never before been published. This landmark finding, if scaled beyond the ED-SAFE sites, has the potential to dramatically improve identification of nascent suicide risk in a highly vulnerable group, thereby paving the way for applying interventions to reduce subsequent suicidal behavior.
We would like to acknowledge the time and effort of the research coordinators and research assistants from the eight participating sites: Edward Boyer, MD, PhD, who provided evaluations of research and clinical protocols (compensated for effort); Robin Clark, PhD, who advised on health economics–related issues (compensated for effort); Barry Feldman, PhD, who led the development and deployment of training materials (compensated for effort); and Brianna Haskins, MS, who assisted with copyediting and formatting the paper (compensated for effort).
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health (NIMH) or NIH.
The project described was supported by Award Number U01MH088278 from NIMH. This grant was funded as a cooperative award by NIMH. NIMH was represented on the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) Steering Committee by Amy Goldstein, PhD. She collaborated with the other committee members and investigators to oversee the conduct of the study, data collection, analysis, and interpretation. NIMH provided Data and Safety Monitoring Board (DSMB) oversight of the study. The NIMH DSMB liaison was Adam Haim, PhD.
This activity is available for CME credit. See page A4 for information.
☆On behalf of the ED-SAFE Investigators (Collaborators), including: Marian E. Betz, MD, MPH, University of Colorado Hospital, Aurora, Colorado; Jeffrey M. Caterino, MD, The Ohio State University Medical Center, Columbus, Ohio; Talmage Holmes, PhD, MPH, University of Arkansas Medical Center, Little Rock, Arkansas; Maura Kennedy, MD, MPH, Beth Israel Deaconess Medical Center, Boston, Massachusetts; Frank LoVecchio, DO, MPH, Maricopa Medical Center, Phoenix, Arizona; Lisa A. Uebelacker, PhD, Memorial Hospital of Rhode Island, Providence, Rhode Island; and Wesley Zeger, DO, University of Nebraska Medical Center, Omaha, Nebraska