If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Center on Social Dynamics and Policy, The Brookings Institution, Washington, District of ColumbiaBrown School, Washington University in St. Louis, St. Louis, MissouriSanta Fe Institute, Santa Fe, New Mexico
Prevention Research Center, Brown School at Washington University in St. Louis, St. Louis, MissouriPublic Health Sciences Division, Department of Surgery, Washington University School of Medicine in St. Louis, St. Louis, MissouriAlvin J. Siteman Cancer Center, Washington University School of Medicine in St. Louis, St. Louis, Missouri
The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes.
Methods
The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation.
Results
Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation.
Conclusions
The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.
INTRODUCTION
The term misimplementation refers to decision makers ending effective activities prematurely (discontinuation misimplementation) or continuing ineffective ones (continuation misimplementation).
In a U.S. study, 36.5% of state health department (SHD) employees reported that programs often or always end that should have continued; 24.7% of respondents reported that programs often or always continue when they should have ended.
Early termination of effective activities results in negative outcomes, including continued early onset or inadequate management of diabetes and other chronic conditions.
The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.
The purpose of this innovative study is to build on previous work using agent-based modeling (ABM) to gain insight into why misimplementation occurs and what feasible approaches might reduce it.
ABM is a computational simulation methodology in which individual entities (e.g., employees), their behaviors, and the environments in which they operate are explicitly (and typically, stochastically) modeled over time.
Until now, it has not been used to understand the complex and contextual drivers of SHD decision making. Thus, this research serves as a first foray into the application of ABM to an important topic, specifically to (1) develop an ABM with sufficient explanatory power to reproduce observed misimplementation patterns, (2) use this ABM to explore counterfactual conditions to determine what feasible approaches might reduce misimplementation frequency, and (3) consider how ABM could be further applied to explore the drivers of and potential approaches to reducing misimplementation.
Existing literature, supplemented by input from an Expert Advisory Group with domain and practical expertise, highlighted the potential key determinants of misimplementation. Following the practices for participatory research, the team collaboratively identified factors within and external to public health departments that may drive occurrences of misimplementation.
Four broad hypotheses emerged: (1) lack of evidence-based decision making (EBDM), defined as “an approach to decision-making that combines the appropriate research evidence, practitioner expertise, and the characteristics, needs, and preferences of the community”
; (2) organizational culture that prevents leadership from having sufficient information about intervention effectiveness; (3) organizational structure that prevents leadership from having sufficient information about intervention effectiveness; and (4) internal and external pressures that induce leadership to make suboptimal decisions by considering factors other than intervention effectiveness. These hypotheses are neither exhaustive nor mutually exclusive. The causal pathways potentially connecting all the 4 to misimplementation are likely to be intraorganizational in nature, may be bidirectional, may change over time, and might operate synergistically. To navigate the obstacles introduced by the complex nature of these phenomena (i.e., heterogeneity, interdependence, and dynamic adaptation), an ABM research approach was used.
Figure 1 depicts an ABM design aligned with characterizing and testing the hypothesized determinants of misimplementation described earlier. It dynamically represents how information, filtered through organizational structure, capacity, culture, and leadership priorities, shapes decisions about whether to continue active interventions. The model design is summarized in this paper and described in detail in the Appendix (available online).
Figure 1Visual summary of model design.
Note: Circles represent employees (agents) within a hierarchically structured organization, rectangles represent the organization-level set of interventions active at any given point in time, and gray arrows represent upward and downward interactions between agents that collectively comprise key organizational dynamics over time that drive the outcome of interest (misimplementation frequency).
In the model, agents represent individual health department employees situated in a formal organizational structure, with overall organizational size, number of hierarchical levels, and number of employees per supervisor stochastically initialized. The organization has a set of active interventions, each with attributes representing age, evidence support for effectiveness given current implementation and context, and levels of support from external stakeholders and from funders. Agents have 2 attributes: EBDM ability and information-sharing propensity. EBDM ability reflects the accuracy with which an agent assesses the evidence support for intervention effectiveness for each active intervention; individual-level EBDM abilities collectively comprise organizational capacity for EBDM.
Information-sharing propensity reflects comfort with reporting these assessments to supervisors or adjusting their own assessment on the basis of reports from supervisees; individual-level information sharing collectively comprises organizational communication culture.
Each simulation run represents 36 months to reflect a combination of typical funding cycles, state health officer terms of office, and time periods for governmental public health organizations to make capacity-building modifications.
At the start of each run, agents in the organization are initialized along with a set of current, active interventions. During each simulated month, agents’ EBDM abilities can change, with employees’ values gravitating toward those of their supervisors to represent personnel activities such as training, hiring, and retention. In any given month, employees might report their assessments of active interventions to their supervisors, with the probability that they do so on the basis of their current information-sharing values. Information-sharing values either increase or decrease on the basis of whether agents’ reports to supervisors result in an adjustment of supervisors’ assessments. Thus, information about interventions continuously flows from the lowest level of the organization to leadership, filtered through individual-level EBDM ability and information-sharing propensity values.
Interventions are evaluated by leadership annually, with some probability that any given intervention will be reviewed off-cycle as well. Leadership makes continuation decisions on the basis of their current assessments as well as interventions’ other attributes. If an intervention is discontinued, it may be replaced by a new one. Except for age, for the sake of model parsimony, intervention attributes are fixed during simulations.
Data Inputs Into Model
Parameter values for the baseline model condition were derived from 5 broad sources of data:
1.
three surveys of SHD employees conducted in 2014 (n=1,237),2 2016 (n=571),
The outcome measures of perceived frequency of misimplementation are from 2 samples of U.S. public health practitioners who completed cognitive response testing (n=12, n=11) followed by survey test‒retest, 2–3 weeks apart (n=54, n=39).
Percent agreement of frequency responses of continuation and discontinuation misimplementation in the 2 samples were 80.0% and 83.8% and 79.2% and 97.3%, respectively.
Promoting state health department evidence-based cancer and chronic disease prevention: a multi-phase dissemination study with a cluster randomized trial component.
Assessing organizational supports for evidence-based decision making in Local Public Health Departments in The United States: development and psychometric properties of a new measure.
supplementary stakeholder interviews conducted in 2020 with a set of participants with current or previous experience as directors of chronic disease units in SHDs (n=13). Questions were structured to solicit model input data (e.g., On a scale of 1 to 10… how much are [intervention age] and [external support] related?);
4.
iterative feedback from an Expert Advisory Group; and
5.
ABM calibration to survey responses from 2014 and 2018 (described earlier) corresponding to the outcome of interest (misimplementation frequency).
Table 1 summarizes how these data sources informed specific model elements. Surveys and interviews were conducted, and response data were analyzed following the protocols approved by the Washington University IRB.
Researchers assessed the ability of the model to reproduce observed fact patterns such as frequency of misimplementation, given the model inputs grounded in available real-world data (i.e., the baseline condition). The team then compared the baseline condition with misimplementation frequencies produced by counterfactual scenarios representing approaches to reducing misimplementation, varying organizational attributes, or decision-making processes alone or in combination. Counterfactual conditions were selected with input from the Expert Advisory Group and on the basis of findings from previous studies. They included the following:
1.
Increased EBDM: representing an organization-wide shift in EBDM capacity, the parameter used to initialize agents’ EBDM was increased by 10%, 30%, or 50%.
Increased information sharing: reflecting a shift in organizational culture and practices that makes transmission of and responsiveness to reports about assessed intervention effectiveness from employees to their supervisors, the parameter used to initialize agents’ information-sharing propensity attributes is increased by 30% or 50%, applied either organization wide or targeted at managers (i.e., the top 3 hierarchical levels).
It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
Organizational restructuring: keeping organizational size (i.e., the number of employees) consistent, organizations were made taller by increasing the parameter that initializes the number of hierarchical levels and reducing the one initializing the number of employees per supervisor or were made wider by doing the inverse. On the basis of the relatively tall nature of real-world health departments at baseline, the model team considered 1 formulation of the former and 2 of the latter.
Intervention continuation decision making: representing a shift in training, incentives, protocols, and practices, the model considered scenarios in which leadership utilizes different criteria when making continuation decisions.
It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
This set of scenarios was characterized by incremental removal of intervention age, stakeholder support, and funder support from continuation decisions. Thus, in the last case, decisions were made solely on the basis of the department leader's assessment of intervention effectiveness.
Experimentation involved a full combinatorial sweep of the variations described earlier and stochastic repetition of runs under each condition to capture variation in organizations and interventions.
First, the study team compared model output under baseline conditions with real-world reports of misimplementation frequency. To compare categorical survey responses with continuous frequency outputs from the model, there were several simplifying assumptions. In Figure 2, the left and right panels (respectively) show the frequency with which ineffective interventions were continued (continuation misimplementation) and effective programs were discontinued (discontinuation misimplementation) when reviewed by leadership. The x-axes show the frequency with which each type of misimplementation occurs. The y-axis shows the probability density, normalized for equivalent comparison between survey and model data. Categorical survey responses are shown with histogram bars, evenly distributed on the x-axes between 0 and 1 (e.g., with never placed between 0 and.2). Continuous model output values taken from 50 simulation runs, smoothed using a Gaussian kernel for ease of visual interpretation, are shown with solid lines. These comparisons are not intended as a formal test but rather to qualitatively gauge the model's ability to broadly reproduce output patterns observed in the real world.
Next, the team conducted counterfactual condition experimentation. Figure 3 depicts the misimplementation frequencies for each single change condition (i.e., those that differ from the baseline in only 1 respect), with the baseline misimplementation frequencies shown for comparison. Across these scenarios, interventions were more likely to be discontinued than they were in the baseline condition. This tended to manifest itself as a reduction in continuation misimplementation relative to baseline but also, in many of the scenarios, a concomitant increase in discontinuation misimplementation. From Figure 3, experiment effects fall into the following 4 broad categories:
1.
Entirely negative: both types of misimplementation increased relative to baseline. The very wide (an average of approximately 3 hierarchical levels and 14 employees per supervisor) scenario displayed this behavior, with average frequencies of each type of misimplementation approximately 2 percentage points higher than the baseline.
2.
Net negative: continuation misimplementation decreased less than discontinuation misimplementation increased. The small (10%) EBDM increase scenario displayed this behavior, although the impact on both types of misimplementation (and thus the difference between them) was very small.
3.
Net positive: continuation misimplementation decreased more than discontinuation misimplementation increased. The moderate (30%) and large (50%) organization-wide information sharing increase, tall (an average of 6 hierarchical levels with approximately 4 employees per supervisor), and somewhat wide (an average of 4 hierarchical levels with 8 employees per supervisor) scenarios all displayed this behavior.
4.
Entirely positive: both types of misimplementation decreased. The other 7 scenarios displayed this desirable behavior. The reduction in continuation misimplementation in scenarios where leadership did not include intervention age in their decisions was notable (an average reduction of over 20 percentage points), as were scenarios in which leadership also excluded other factors (i.e., external leadership or funder support) from their decision-making process; excluding both results in an average reduction of approximately 35 percentage points.
Figure 3Box-plot distributions of misimplementation frequency.
Note: Continuation of ineffective interventions or discontinuation of effective ones, respectively, shown in the left and right panels, under the baseline as well as all single intervention policy value conditions are shown. This includes EBDM boost, conditions in which agents are initialized with larger EBDM values, and sharing boost, conditions in which agents are initialized with larger information-sharing values, alternative organizational structures in which the organization is initialized such that it is either wider or taller than in the baseline condition, and conditions in which leadership utilizes different strategies for making continuation decisions. Median values are shown as vertical lines; the 25th and 75th percentile values are shown as left and right box edges, respectively; 95% CIs are shown as horizontal lines; and outlier values are shown as dots. Frequency values are shown on the x-axis, and the sole deviation from the baseline condition is noted on the y-axis (other than the baseline itself). For ease of comparison, the baseline median is presented as a dashed line.
The Appendix (available online) contains specific values for outcome distributions depicted in Figure 3 and results from conditions where 2 or more of the experiment categories varied from baseline.
DISCUSSION
This research introduces a novel ABM of public health department organizational information flow dynamics and intervention continuation decision making; within the constraints of available testing data, it shows satisfactory explanatory power. The main results presented in Figure 3 suggest actionable strategies that align with existing literature and the experts’ experiences. By identifying and operationalizing for the first time the specific dynamic pathways driving misimplementation, this model also serves as a starting point for further efforts to inform and improve public health practice as well as to guide future data collection.
Analysis of model results indicated that increasing organizational EBDM capacity tends to decrease misimplementation frequency. This is not unexpected a priori, but the results quantify the strength of this relationship. EBDM helps public health departments to identify the best available evidence about an intervention's potential impact given the context in which it will be deployed.
Emerging qualitative research on ending ineffective efforts highlights the importance of this capacity in reducing misimplementation because participants indicate that when successful, they leveraged evaluation data.
Findings also suggest that changes in organizational culture that facilitate information sharing can reduce misimplementation, with that reduction more pronounced when changes are applied to the whole organization than to only management. To fully activate EBDM, employees must have two-way street relationships with their supervisors where they speak and are then heard.
When an employee is aware of a problem or has an idea, they must be comfortable sharing it with a supervisor, which is more likely to occur when that supervisor is open to the views of others, is willing to reflect on and shift their own perspectives, and can help shepherd information that they receive into observable change.
Contrary to a priori expectations, changes in organizational structure (e.g., flattening the organizational hierarchies) did not consistently reduce misimplementation.
Following best practices in systems science, the study team incorporated sustained expert guidance, feedback, and engagement with ABM into the research plan.
Participants concurred that the results had face validity on the basis of their experiences and intuition. One finding that not only has support from literature
but particularly resonated with this group was that shifting decision-making processes to place additional emphasis on intervention effectiveness has the potential to dramatically reduce misimplementation. An approach that effectively removed intervention age from leadership's continuation decisions was described as viewing them with fresh eyes and approvingly seen as a way to remove organizational inertia and sunk cost mentality in favor of prioritizing effective interventions.
It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
ABMs are highly extensible, and the research reported in this paper suggests ways to add sophistication in future iterations of the model. First, is an exploration of additional formal and informal information-sharing dynamics between employees within or between workgroups, allowing for consideration of arrangements such as matrix management and horizontal communication? Second, relevant decisions may be influenced by the degree of centralization of public health activities in different states. Third, there is a need to explore alternative EBDM dynamics, such as peer-based or employee-led learning. Fourth, research is needed on the role of relative implementability of specific evidence-based interventions. Fifth, more information is needed on whether and how leadership might employ an option beyond continuation or discontinuation, for example, adjusting intervention design or implementation targeting to improve effectiveness. Finally, in addition to iteratively improving this model, additional applications to an exploration of how misimplementation occurs—and might be addressed—at the local public health department level (with significant input from local-level partners) are envisioned.
The application of ABM to this important problem is highly innovative. This research presents an opportunity to extend beyond existing (often cross-sectional) efforts to improve organizational effectiveness, combining data from multiple sources to engage in thought experiments aided by computational simulation. Thus, without incurring costs associated with organizational initiatives or risking negative health outcomes from ineffective or counterproductive efforts, one can obtain valuable insights.
Limitations
The biggest challenge faced stemmed from limited previous research into misimplementation, meaning that there was a dearth of existing, relevant data to populate models. Previous work has shown that ABM can be a useful tool to advance the field in such circumstances.
This research effort identified the types of data that should be collected (along with when and how data should be gathered) to shed additional light on the causes of and solutions to misimplementation. Specifically, future misimplementation research will benefit from a validated measure of misimplementation that does not rely on programmatic employees’ self-reported perceptions and longitudinal data describing intervention continuation patterns over time as well as more detailed data on decision-making processes that result in continuation.
CONCLUSIONS
Misimplementation has previously been defined and shown to be widespread, with an important impact on public health, but neither the dynamic pathways that drive it nor the most effective ways to address have been well understood.
The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.
ABMs and similar computational modeling techniques have proven useful in public health because they examine the complex interplay among systems, organizations, community contexts, and individuals that influence population health and extend beyond existing data to address counterfactual conditions.
Kasman M, Hammond RA, Heuberger B, et al. Activating a community: an agent-based model of romp & chomp, a whole-of-community childhood obesity intervention. Obesity (Silver Spring). 2019;27(9):1494–1502. https://doi.org/10.1002/oby.22553.
The first-generation research presented in this paper, along with related studies, suggests that 2 priorities for training in the public health workforce should be EBDM and effective communication, skills that are applicable to employees regardless of supervisory status.
Operationalizing insights gained from this research into leadership decision making will require an intentional rethinking of how leaders are selected and trained and how they engage in decision-making processes: identifying and weighing priorities that might be in conflict as well as navigating relationships with stakeholders and funders to advocate for evidence-based continuation decisions.
In public health, one size often does not fit all. Computational modeling tools make it easier for decision makers to select policies and practices that are likely to effect sustainable, positive change.
Tools to show context-relevant simulation output can help convey potential impacts and be useful springboards for informing specific recommendations. For example, ABMs that have been iteratively developed and applied have provided actionable guidance on the selection of tobacco control policies such as menthol sales restrictions and retailer density reduction across communities.
This model might similarly shape recommendations to reduce misimplementation in specific public health contexts.
ACKNOWLEDGMENTS
Special thanks go to the Expert Advisory Group, especially the late Julia Thorsness, who provided important input and feedback throughout the research. Carol Brownson, Randy Schwartz, Frank Bright, and Paula Clayton provided invaluable advice and feedback on the guide for supplementary stakeholder interviews; Carol Brownson additionally assisted by conducting many of these interviews. David O'Gara helped to review and finalize manuscript figures.
The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the NIH or the Centers for Disease Control and Prevention.
This project is funded by the National Cancer Institute of the NIH (R01CA214530, P50CA244431), the Centers for Disease Control and Prevention (Number U48DP006395), and the Foundation for Barnes-Jewish Hospital.
No financial disclosures were reported by the authors of this paper.
CRediT AUTHOR STATEMENT
Matt Kasman: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Validation, Writing - Original Draft Preparation, Writing - Review and Editing. Ross A. Hammond: Conceptualization, Methodology, Project administration, Writing - Review and Editing. Rob Purcell: Conceptualization, Software, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review and Editing. Louise Farah Saliba: Conceptualization, Formal analysis, Investigation, Writing - Original Draft Preparation, Writing - Review and Editing. Stephanie Mazzucca: Conceptualization, Formal analysis, Investigation, Writing - Review and Editing. Margaret Padek: Writing - Review and Editing, Conceptualization, Formal analysis, Investigation, Writing - Review and Editing. Peg Allen: Conceptualization, Formal analysis, Investigation, Writing - Original Draft Preparation, Writing - Review and Editing. Douglas A. Luke: Conceptualization, Writing - Review and Editing. Sarah Moreland-Russell: Conceptualization, Writing - Review and Editing. Paul C. Erwin: Conceptualization, Writing - Review and Editing. Ross C. Brownson: Conceptualization, Funding acquisition, Project administration, Supervision, Writing - Review and Editing.
The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.
Promoting state health department evidence-based cancer and chronic disease prevention: a multi-phase dissemination study with a cluster randomized trial component.
Assessing organizational supports for evidence-based decision making in Local Public Health Departments in The United States: development and psychometric properties of a new measure.
It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
Kasman M, Hammond RA, Heuberger B, et al. Activating a community: an agent-based model of romp & chomp, a whole-of-community childhood obesity intervention. Obesity (Silver Spring). 2019;27(9):1494–1502. https://doi.org/10.1002/oby.22553.
00531-1&id=gr1.jpg){kind=link}
00531-1&id=gr2.jpg){kind=link}
00531-1&id=gr3.jpg){kind=link}