Advertisement

Understanding Misimplementation in U.S. State Health Departments: An Agent-Based Model

Open AccessPublished:December 10, 2022DOI:https://doi.org/10.1016/j.amepre.2022.10.011

      Introduction

      The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes.

      Methods

      The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation.

      Results

      Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation.

      Conclusions

      The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.

      INTRODUCTION

      The term misimplementation refers to decision makers ending effective activities prematurely (discontinuation misimplementation) or continuing ineffective ones (continuation misimplementation).
      • Padek M
      • Allen P
      • Erwin PC
      • et al.
      Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.
      In a U.S. study, 36.5% of state health department (SHD) employees reported that programs often or always end that should have continued; 24.7% of respondents reported that programs often or always continue when they should have ended.
      • Brownson RC
      • Allen P
      • Jacob RR
      • et al.
      Understanding mis-implementation in public health practice.
      Early termination of effective activities results in negative outcomes, including continued early onset or inadequate management of diabetes and other chronic conditions.

      The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.

      Continuation of interventions that are not effective in positively impacting intended priority population groups can exacerbate health disparities.

      2021 Health disparities report. America's Health Rankings, United Health Foundation. https://www.americashealthrankings.org/learn/reports/2021-disparities-report. Updated June 28, 2021. Accessed May 16, 2022.

      ,
      Centers for Disease Control and Prevention
      CDC Health Disparities and Inequalities Report – United States, 2013.
      Recent research provides nascent, suggestive evidence about factors related to misimplementation.
      • Padek M
      • Allen P
      • Erwin PC
      • et al.
      Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.
      ,
      • Brownson RC
      • Allen P
      • Jacob RR
      • et al.
      Understanding mis-implementation in public health practice.
      ,
      • Padek MM
      • Mazzucca S
      • Allen P
      • et al.
      Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
      ,
      • Allen P
      • Jacob RR
      • Parks RG
      • et al.
      Perspectives on program mis-implementation among U.S. local public health departments.
      The purpose of this innovative study is to build on previous work using agent-based modeling (ABM) to gain insight into why misimplementation occurs and what feasible approaches might reduce it.
      ABM is a computational simulation methodology in which individual entities (e.g., employees), their behaviors, and the environments in which they operate are explicitly (and typically, stochastically) modeled over time.
      • Hammond RA
      Considerations and best practices in agent-based modeling to inform policy.
      ABM has been increasingly utilized in guiding policy and practice in the social sciences in general and public health specifically.
      • Tracy M
      • Cerdá M
      • Keyes KM.
      Agent-based modeling in public health: current applications and future directions.
      • Silverman E
      • Gostoli U
      • Picascia S
      • et al.
      Situating agent-based modelling in population health research.
      • Huang W
      • Chang CH
      • Stuart EA
      • et al.
      Agent-based modeling for implementation research: an application to tobacco smoking cessation for persons with serious mental illness.
      • McKay VR
      • Hoffer LD
      • Combs TB
      • Margaret Dolcini M
      The dynamic influence of human resources on evidence-based intervention sustainability and population outcomes: an agent-based modeling approach.
      • Li Y
      • Padrón NA
      • Mangla AT
      • Russo PG
      • Schlenker T
      • Pagán JA.
      Using systems science to inform population health strategies in local health departments: a case study in San Antonio, Texas.
      • Li Y
      • Kong N
      • Lawley M
      • Weiss L
      • Pagán JA.
      Advancing the use of evidence-based decision-making in local health departments with systems science methodologies.
      • Combs TB
      • McKay VR
      • Ornstein J
      • et al.
      Modelling the impact of menthol sales restrictions and retailer density reduction policies: insights from tobacco town Minnesota.
      • Luke DA
      • Hammond RA
      • Combs T
      • et al.
      Tobacco town: computational modeling of policy options to reduce tobacco retailer density.
      • Hammond R
      • Ornstein JT
      • Purcell R
      • Haslam MD
      • Kasman M.
      Modeling robustness of COVID-19 containment policies [preprint].
      • Zhang D
      • Giabbanelli PJ
      • Arah OA
      • Zimmerman FJ.
      Impact of different policies on unhealthy dietary behaviors in an urban adult population: an agent-based simulation model.
      There is also a growing body of evidence that ABM is particularly well suited to studying organizations.
      • Chang MH
      • Harrington Jr, JE
      Agent-based models of organizations.
      • Carley KM.
      Computational organization science: A new frontier.
      • Fioretti G.
      Agent-based simulation models in organization science.
      • Secchi D.
      A case for agent-based models in organizational behavior and team research.
      Until now, it has not been used to understand the complex and contextual drivers of SHD decision making. Thus, this research serves as a first foray into the application of ABM to an important topic, specifically to (1) develop an ABM with sufficient explanatory power to reproduce observed misimplementation patterns, (2) use this ABM to explore counterfactual conditions to determine what feasible approaches might reduce misimplementation frequency, and (3) consider how ABM could be further applied to explore the drivers of and potential approaches to reducing misimplementation.
      Existing literature, supplemented by input from an Expert Advisory Group with domain and practical expertise, highlighted the potential key determinants of misimplementation. Following the practices for participatory research, the team collaboratively identified factors within and external to public health departments that may drive occurrences of misimplementation.
      • Aarons GA
      • Hurlburt M
      • Horwitz SM.
      Advancing a conceptual model of evidence-based practice implementation in public service sectors.
      • Moullin JC
      • Dickson KS
      • Stadnick NA
      • Rabin B
      • Aarons GA.
      Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.
      • Ramanadhan S
      • Davis MM
      • Armstrong R
      • et al.
      Participatory implementation science to increase the impact of evidence-based cancer prevention and control.
      • Moreland-Russell S
      • Weno ER
      • Padek M
      • Saliba LF
      • Brownson RC.
      Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
      Four broad hypotheses emerged: (1) lack of evidence-based decision making (EBDM), defined as “an approach to decision-making that combines the appropriate research evidence, practitioner expertise, and the characteristics, needs, and preferences of the community”
      • Padek MM
      • Mazzucca S
      • Allen P
      • et al.
      Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
      ; (2) organizational culture that prevents leadership from having sufficient information about intervention effectiveness; (3) organizational structure that prevents leadership from having sufficient information about intervention effectiveness; and (4) internal and external pressures that induce leadership to make suboptimal decisions by considering factors other than intervention effectiveness. These hypotheses are neither exhaustive nor mutually exclusive. The causal pathways potentially connecting all the 4 to misimplementation are likely to be intraorganizational in nature, may be bidirectional, may change over time, and might operate synergistically. To navigate the obstacles introduced by the complex nature of these phenomena (i.e., heterogeneity, interdependence, and dynamic adaptation), an ABM research approach was used.
      • Kasman M
      • Breen N
      • Hammond RA.
      Complex systems science.
      • Luke DA
      • Stamatakis KA.
      Systems science methods in public health: dynamics, networks, and agents.
      • Miller JH
      • Page SE
      • Page S
      Complex Adaptive Systems.
      • Brownson RC
      • Shelton RC
      • Geng EH
      • Glasgow RE.
      Revisiting concepts of evidence in implementation science.

      METHODS

      Model Design

      Figure 1 depicts an ABM design aligned with characterizing and testing the hypothesized determinants of misimplementation described earlier. It dynamically represents how information, filtered through organizational structure, capacity, culture, and leadership priorities, shapes decisions about whether to continue active interventions. The model design is summarized in this paper and described in detail in the Appendix (available online).
      Figure 1
      Figure 1Visual summary of model design.
      Note: Circles represent employees (agents) within a hierarchically structured organization, rectangles represent the organization-level set of interventions active at any given point in time, and gray arrows represent upward and downward interactions between agents that collectively comprise key organizational dynamics over time that drive the outcome of interest (misimplementation frequency).
      In the model, agents represent individual health department employees situated in a formal organizational structure, with overall organizational size, number of hierarchical levels, and number of employees per supervisor stochastically initialized. The organization has a set of active interventions, each with attributes representing age, evidence support for effectiveness given current implementation and context, and levels of support from external stakeholders and from funders. Agents have 2 attributes: EBDM ability and information-sharing propensity. EBDM ability reflects the accuracy with which an agent assesses the evidence support for intervention effectiveness for each active intervention; individual-level EBDM abilities collectively comprise organizational capacity for EBDM.
      • Brownson RC
      • Fielding JE
      • Green LW.
      Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research.
      • Jacob RR
      • Brownson CA
      • Deshpande AD
      • et al.
      Long-term evaluation of a course on evidence-based public health in the U.S. and Europe.
      • Brownson RC
      • Fielding JE
      • Maylahn CM.
      Evidence-based public health: a fundamental concept for public health practice.
      Information-sharing propensity reflects comfort with reporting these assessments to supervisors or adjusting their own assessment on the basis of reports from supervisees; individual-level information sharing collectively comprises organizational communication culture.
      Each simulation run represents 36 months to reflect a combination of typical funding cycles, state health officer terms of office, and time periods for governmental public health organizations to make capacity-building modifications.
      • Brownson RC
      • Fielding JE
      • Green LW.
      Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research.
      ,
      • Brownson RC
      • Allen P
      • Duggan K
      • Stamatakis KA
      • Erwin PC.
      Fostering more-effective public health by identifying administrative evidence-based practices: a review of the literature.
      At the start of each run, agents in the organization are initialized along with a set of current, active interventions. During each simulated month, agents’ EBDM abilities can change, with employees’ values gravitating toward those of their supervisors to represent personnel activities such as training, hiring, and retention. In any given month, employees might report their assessments of active interventions to their supervisors, with the probability that they do so on the basis of their current information-sharing values. Information-sharing values either increase or decrease on the basis of whether agents’ reports to supervisors result in an adjustment of supervisors’ assessments. Thus, information about interventions continuously flows from the lowest level of the organization to leadership, filtered through individual-level EBDM ability and information-sharing propensity values.
      Interventions are evaluated by leadership annually, with some probability that any given intervention will be reviewed off-cycle as well. Leadership makes continuation decisions on the basis of their current assessments as well as interventions’ other attributes. If an intervention is discontinued, it may be replaced by a new one. Except for age, for the sake of model parsimony, intervention attributes are fixed during simulations.

      Data Inputs Into Model

      Parameter values for the baseline model condition were derived from 5 broad sources of data:
      Table 1 summarizes how these data sources informed specific model elements. Surveys and interviews were conducted, and response data were analyzed following the protocols approved by the Washington University IRB.
      • Padek M
      • Allen P
      • Erwin PC
      • et al.
      Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.
      Model parameterization details (including which measures from each source were used and how) are provided in the Appendix (available online).
      Table 1Summary of Model Parameterization
      Model element/description of key parametersData source
      Organizational structure
       Distribution used for a number of organizational levelsSupplemental stakeholder interviews
       Distribution used for the number of supervisees assigned to supervisorsSurvey data
      Active interventions
       Number of active interventions at the start of runSupplemental stakeholder interviews
       Distribution used for initialization of intervention agesInitial stakeholder interviews
       Distributions used for initialization of intervention evidence support, external stakeholder support, and funder supportSupplemental stakeholder interviews
       Correlations between age, evidence support, external stakeholder support, and funder supportSupplemental stakeholder interviews
       Probability that the discontinued intervention is replaced with a new interventionExpert Advisory Group
      Leadership review
       Probability of off-cycle intervention evaluationExpert Advisory Group
      Continuation decisions
       Continuation decision function termsModel calibration
      EBDM ability
       Distributions used for agents’ initial EBDM ability valuesSurvey data
       EBDM update magnitudes (upward or downward based on supervisor value; upward value is higher because it incorporates employee training)Expert Advisory Group
      Information sharing propensity
       Distributions used for agents’ initial information-sharing propensity valuesSurvey data
       Information-sharing propensity update magnitudeModel calibration
      Intervention assessment reporting
       Report to supervisor probability function termsModel calibration
       Assessment update probability function termsModel calibration
       Assessment update magnitudeModel calibration
      EBDM, evidence-based decision making.

      Statistical Analysis

      Researchers assessed the ability of the model to reproduce observed fact patterns such as frequency of misimplementation, given the model inputs grounded in available real-world data (i.e., the baseline condition). The team then compared the baseline condition with misimplementation frequencies produced by counterfactual scenarios representing approaches to reducing misimplementation, varying organizational attributes, or decision-making processes alone or in combination. Counterfactual conditions were selected with input from the Expert Advisory Group and on the basis of findings from previous studies. They included the following:
      • 1.
        Increased EBDM: representing an organization-wide shift in EBDM capacity, the parameter used to initialize agents’ EBDM was increased by 10%, 30%, or 50%.
        • Brownson RC
        • Fielding JE
        • Green LW.
        Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research.
        • Jacob RR
        • Brownson CA
        • Deshpande AD
        • et al.
        Long-term evaluation of a course on evidence-based public health in the U.S. and Europe.
      • 2.
        Increased information sharing: reflecting a shift in organizational culture and practices that makes transmission of and responsiveness to reports about assessed intervention effectiveness from employees to their supervisors, the parameter used to initialize agents’ information-sharing propensity attributes is increased by 30% or 50%, applied either organization wide or targeted at managers (i.e., the top 3 hierarchical levels).
        • Mazzucca S
        • Saliba LF
        • Smith R
        • et al.
        It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
      • 3.
        Organizational restructuring: keeping organizational size (i.e., the number of employees) consistent, organizations were made taller by increasing the parameter that initializes the number of hierarchical levels and reducing the one initializing the number of employees per supervisor or were made wider by doing the inverse. On the basis of the relatively tall nature of real-world health departments at baseline, the model team considered 1 formulation of the former and 2 of the latter.
        • Padek MM
        • Mazzucca S
        • Allen P
        • et al.
        Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
      • 4.
        Intervention continuation decision making: representing a shift in training, incentives, protocols, and practices, the model considered scenarios in which leadership utilizes different criteria when making continuation decisions.
        • Brownson RC
        • Allen P
        • Duggan K
        • Stamatakis KA
        • Erwin PC.
        Fostering more-effective public health by identifying administrative evidence-based practices: a review of the literature.
        • Mazzucca S
        • Saliba LF
        • Smith R
        • et al.
        It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
        • Jacob RR
        • Baker EA
        • Allen P
        • et al.
        Training needs and supports for evidence-based decision making among the public health workforce in the United States.
        This set of scenarios was characterized by incremental removal of intervention age, stakeholder support, and funder support from continuation decisions. Thus, in the last case, decisions were made solely on the basis of the department leader's assessment of intervention effectiveness.
      Experimentation involved a full combinatorial sweep of the variations described earlier and stochastic repetition of runs under each condition to capture variation in organizations and interventions.
      • Kasman M
      • Kreuger LK.
      Best practices for systems science research.

      RESULTS

      First, the study team compared model output under baseline conditions with real-world reports of misimplementation frequency. To compare categorical survey responses with continuous frequency outputs from the model, there were several simplifying assumptions. In Figure 2, the left and right panels (respectively) show the frequency with which ineffective interventions were continued (continuation misimplementation) and effective programs were discontinued (discontinuation misimplementation) when reviewed by leadership. The x-axes show the frequency with which each type of misimplementation occurs. The y-axis shows the probability density, normalized for equivalent comparison between survey and model data. Categorical survey responses are shown with histogram bars, evenly distributed on the x-axes between 0 and 1 (e.g., with never placed between 0 and.2). Continuous model output values taken from 50 simulation runs, smoothed using a Gaussian kernel for ease of visual interpretation, are shown with solid lines. These comparisons are not intended as a formal test but rather to qualitatively gauge the model's ability to broadly reproduce output patterns observed in the real world.
      • Epstein JM
      Generative Social Science.
      Overall, the model appeared capable of reproducing expected misimplementation frequencies under baseline conditions.
      Figure 2
      Figure 2Comparison of frequencies of misimplementation from survey response data and model output.
      Note: The lines represent the model output, and the histogram bars depict the frequencies of survey responses.
      Next, the team conducted counterfactual condition experimentation. Figure 3 depicts the misimplementation frequencies for each single change condition (i.e., those that differ from the baseline in only 1 respect), with the baseline misimplementation frequencies shown for comparison. Across these scenarios, interventions were more likely to be discontinued than they were in the baseline condition. This tended to manifest itself as a reduction in continuation misimplementation relative to baseline but also, in many of the scenarios, a concomitant increase in discontinuation misimplementation. From Figure 3, experiment effects fall into the following 4 broad categories:
      • 1.
        Entirely negative: both types of misimplementation increased relative to baseline. The very wide (an average of approximately 3 hierarchical levels and 14 employees per supervisor) scenario displayed this behavior, with average frequencies of each type of misimplementation approximately 2 percentage points higher than the baseline.
      • 2.
        Net negative: continuation misimplementation decreased less than discontinuation misimplementation increased. The small (10%) EBDM increase scenario displayed this behavior, although the impact on both types of misimplementation (and thus the difference between them) was very small.
      • 3.
        Net positive: continuation misimplementation decreased more than discontinuation misimplementation increased. The moderate (30%) and large (50%) organization-wide information sharing increase, tall (an average of 6 hierarchical levels with approximately 4 employees per supervisor), and somewhat wide (an average of 4 hierarchical levels with 8 employees per supervisor) scenarios all displayed this behavior.
      • 4.
        Entirely positive: both types of misimplementation decreased. The other 7 scenarios displayed this desirable behavior. The reduction in continuation misimplementation in scenarios where leadership did not include intervention age in their decisions was notable (an average reduction of over 20 percentage points), as were scenarios in which leadership also excluded other factors (i.e., external leadership or funder support) from their decision-making process; excluding both results in an average reduction of approximately 35 percentage points.
      Figure 3
      Figure 3Box-plot distributions of misimplementation frequency.
      Note: Continuation of ineffective interventions or discontinuation of effective ones, respectively, shown in the left and right panels, under the baseline as well as all single intervention policy value conditions are shown. This includes EBDM boost, conditions in which agents are initialized with larger EBDM values, and sharing boost, conditions in which agents are initialized with larger information-sharing values, alternative organizational structures in which the organization is initialized such that it is either wider or taller than in the baseline condition, and conditions in which leadership utilizes different strategies for making continuation decisions. Median values are shown as vertical lines; the 25th and 75th percentile values are shown as left and right box edges, respectively; 95% CIs are shown as horizontal lines; and outlier values are shown as dots. Frequency values are shown on the x-axis, and the sole deviation from the baseline condition is noted on the y-axis (other than the baseline itself). For ease of comparison, the baseline median is presented as a dashed line.
      EBDM, evidence-based decision making.
      The Appendix (available online) contains specific values for outcome distributions depicted in Figure 3 and results from conditions where 2 or more of the experiment categories varied from baseline.

      DISCUSSION

      This research introduces a novel ABM of public health department organizational information flow dynamics and intervention continuation decision making; within the constraints of available testing data, it shows satisfactory explanatory power. The main results presented in Figure 3 suggest actionable strategies that align with existing literature and the experts’ experiences. By identifying and operationalizing for the first time the specific dynamic pathways driving misimplementation, this model also serves as a starting point for further efforts to inform and improve public health practice as well as to guide future data collection.
      Analysis of model results indicated that increasing organizational EBDM capacity tends to decrease misimplementation frequency. This is not unexpected a priori, but the results quantify the strength of this relationship. EBDM helps public health departments to identify the best available evidence about an intervention's potential impact given the context in which it will be deployed.
      • Brownson RC
      • Fielding JE
      • Green LW.
      Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research.
      • Jacob RR
      • Brownson CA
      • Deshpande AD
      • et al.
      Long-term evaluation of a course on evidence-based public health in the U.S. and Europe.
      • Brownson RC
      • Fielding JE
      • Maylahn CM.
      Evidence-based public health: a fundamental concept for public health practice.
      Emerging qualitative research on ending ineffective efforts highlights the importance of this capacity in reducing misimplementation because participants indicate that when successful, they leveraged evaluation data.
      • Moreland-Russell S
      • Weno ER
      • Padek M
      • Saliba LF
      • Brownson RC.
      Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
      ,
      • Rodriguez Weno E
      • Allen P
      • Mazzucca S
      • et al.
      Approaches for ending ineffective programs: strategies from state public health practitioners.
      Findings also suggest that changes in organizational culture that facilitate information sharing can reduce misimplementation, with that reduction more pronounced when changes are applied to the whole organization than to only management. To fully activate EBDM, employees must have two-way street relationships with their supervisors where they speak and are then heard.
      • Moreland-Russell S
      • Weno ER
      • Padek M
      • Saliba LF
      • Brownson RC.
      Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
      ,
      • Baker EA
      • Brownson RC
      • Dreisinger M
      • McIntosh LD
      • Karamehic-Muratovic A.
      Examining the role of training in evidence-based public health: a qualitative study.
      When an employee is aware of a problem or has an idea, they must be comfortable sharing it with a supervisor, which is more likely to occur when that supervisor is open to the views of others, is willing to reflect on and shift their own perspectives, and can help shepherd information that they receive into observable change.
      • Moreland-Russell S
      • Weno ER
      • Padek M
      • Saliba LF
      • Brownson RC.
      Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
      ,
      • Morrison EW
      • Wheeler-Smith SL
      • Kamdar D.
      Speaking up in groups: a cross-level study of group voice climate and voice.
      • Detert JR
      • Treviño LK.
      Speaking up to higher-ups: how supervisors and skip-level leaders influence employee voice.
      • Morrison EW.
      Employee voice and silence.
      • Hartley J
      • Alford J
      • Hughes O
      • Yates S.
      Public value and political astuteness in the work of public managers: the art of the possible.
      Contrary to a priori expectations, changes in organizational structure (e.g., flattening the organizational hierarchies) did not consistently reduce misimplementation.
      • Padek MM
      • Mazzucca S
      • Allen P
      • et al.
      Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
      ,
      • Allen P
      • Jacob RR
      • Parks RG
      • et al.
      Perspectives on program mis-implementation among U.S. local public health departments.
      ,
      • Joseph J
      • Gaba V.
      Organizational structure, information processing, and decision-making: a retrospective and road map for research.
      Following best practices in systems science, the study team incorporated sustained expert guidance, feedback, and engagement with ABM into the research plan.
      • Hammond RA
      Considerations and best practices in agent-based modeling to inform policy.
      ,
      • Kasman M
      • Breen N
      • Hammond RA.
      Complex systems science.
      Participants concurred that the results had face validity on the basis of their experiences and intuition. One finding that not only has support from literature
      • Allen P
      • Jacob RR
      • Parks RG
      • et al.
      Perspectives on program mis-implementation among U.S. local public health departments.
      ,
      • Hu H
      • Allen P
      • Yan Y
      • Reis RS
      • Jacob RR
      • Brownson RC.
      Organizational supports for research evidence use in state public health agencies: a latent class analysis.
      but particularly resonated with this group was that shifting decision-making processes to place additional emphasis on intervention effectiveness has the potential to dramatically reduce misimplementation. An approach that effectively removed intervention age from leadership's continuation decisions was described as viewing them with fresh eyes and approvingly seen as a way to remove organizational inertia and sunk cost mentality in favor of prioritizing effective interventions.
      • Mazzucca S
      • Saliba LF
      • Smith R
      • et al.
      It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
      ABMs are highly extensible, and the research reported in this paper suggests ways to add sophistication in future iterations of the model. First, is an exploration of additional formal and informal information-sharing dynamics between employees within or between workgroups, allowing for consideration of arrangements such as matrix management and horizontal communication? Second, relevant decisions may be influenced by the degree of centralization of public health activities in different states. Third, there is a need to explore alternative EBDM dynamics, such as peer-based or employee-led learning. Fourth, research is needed on the role of relative implementability of specific evidence-based interventions. Fifth, more information is needed on whether and how leadership might employ an option beyond continuation or discontinuation, for example, adjusting intervention design or implementation targeting to improve effectiveness. Finally, in addition to iteratively improving this model, additional applications to an exploration of how misimplementation occurs—and might be addressed—at the local public health department level (with significant input from local-level partners) are envisioned.
      The application of ABM to this important problem is highly innovative. This research presents an opportunity to extend beyond existing (often cross-sectional) efforts to improve organizational effectiveness, combining data from multiple sources to engage in thought experiments aided by computational simulation. Thus, without incurring costs associated with organizational initiatives or risking negative health outcomes from ineffective or counterproductive efforts, one can obtain valuable insights.

      Limitations

      The biggest challenge faced stemmed from limited previous research into misimplementation, meaning that there was a dearth of existing, relevant data to populate models. Previous work has shown that ABM can be a useful tool to advance the field in such circumstances.
      • Hammond RA.
      Complex systems modeling for obesity research.
      • Hammond RA
      • Ornstein JT.
      A model of social influence on body mass index.
      • Hall KD
      • Hammond RA
      • Rahmandad H.
      Dynamic interplay among homeostatic, hedonic, and cognitive feedback circuits regulating body weight.
      • Hawkes C
      • Smith TG
      • Jewell J
      • et al.
      Smart food policies for obesity prevention.
      This research effort identified the types of data that should be collected (along with when and how data should be gathered) to shed additional light on the causes of and solutions to misimplementation. Specifically, future misimplementation research will benefit from a validated measure of misimplementation that does not rely on programmatic employees’ self-reported perceptions and longitudinal data describing intervention continuation patterns over time as well as more detailed data on decision-making processes that result in continuation.

      CONCLUSIONS

      Misimplementation has previously been defined and shown to be widespread, with an important impact on public health, but neither the dynamic pathways that drive it nor the most effective ways to address have been well understood.
      • Padek M
      • Allen P
      • Erwin PC
      • et al.
      Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.
      • Brownson RC
      • Allen P
      • Jacob RR
      • et al.
      Understanding mis-implementation in public health practice.

      The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.

      2021 Health disparities report. America's Health Rankings, United Health Foundation. https://www.americashealthrankings.org/learn/reports/2021-disparities-report. Updated June 28, 2021. Accessed May 16, 2022.

      Centers for Disease Control and Prevention
      CDC Health Disparities and Inequalities Report – United States, 2013.
      • Padek MM
      • Mazzucca S
      • Allen P
      • et al.
      Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
      • Allen P
      • Jacob RR
      • Parks RG
      • et al.
      Perspectives on program mis-implementation among U.S. local public health departments.
      ABMs and similar computational modeling techniques have proven useful in public health because they examine the complex interplay among systems, organizations, community contexts, and individuals that influence population health and extend beyond existing data to address counterfactual conditions.
      • Li Y
      • Kong N
      • Lawley M
      • Weiss L
      • Pagán JA.
      Advancing the use of evidence-based decision-making in local health departments with systems science methodologies.
      ,
      • Kasman M
      • Breen N
      • Hammond RA.
      Complex systems science.
      ,
      • Li Y
      • Lawley MA
      • Siscovick DS
      • Zhang D
      • Pagán JA.
      Agent-based modeling of chronic diseases: a narrative review and future research directions.

      Kasman M, Hammond RA, Heuberger B, et al. Activating a community: an agent-based model of romp & chomp, a whole-of-community childhood obesity intervention. Obesity (Silver Spring). 2019;27(9):1494–1502. https://doi.org/10.1002/oby.22553.

      • Morshed AB
      • Kasman M
      • Heuberger B
      • Hammond RA
      • Hovmand PS.
      A systematic review of system dynamics and agent-based obesity models: evaluating obesity as part of the global syndemic.
      • Burke JG
      • Lich KH
      • Neal JW
      • Meissner HI
      • Yonas M
      • Mabry PL.
      Enhancing dissemination and implementation research using systems science methods.
      • Combs T
      • Nelson KL
      • Luke D
      • et al.
      Simulating the role of knowledge brokers in policy making in state agencies: an agent-based model.
      The first-generation research presented in this paper, along with related studies, suggests that 2 priorities for training in the public health workforce should be EBDM and effective communication, skills that are applicable to employees regardless of supervisory status.
      • Moreland-Russell S
      • Weno ER
      • Padek M
      • Saliba LF
      • Brownson RC.
      Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
      ,
      • Jacob RR
      • Brownson CA
      • Deshpande AD
      • et al.
      Long-term evaluation of a course on evidence-based public health in the U.S. and Europe.
      ,
      • Bogaert K
      • Castrucci BC
      • Gould E
      • Rider N
      • Whang C
      • Corcoran E.
      Top training needs of the governmental public health workforce.
      ,
      • Koh S
      • Lee M
      • Brotzman LE
      • Shelton RC.
      An orientation for new researchers to key domains, processes, and resources in implementation science.
      Operationalizing insights gained from this research into leadership decision making will require an intentional rethinking of how leaders are selected and trained and how they engage in decision-making processes: identifying and weighing priorities that might be in conflict as well as navigating relationships with stakeholders and funders to advocate for evidence-based continuation decisions.
      In public health, one size often does not fit all. Computational modeling tools make it easier for decision makers to select policies and practices that are likely to effect sustainable, positive change.
      • Hammond RA
      Considerations and best practices in agent-based modeling to inform policy.
      ,
      • Combs TB
      • McKay VR
      • Ornstein J
      • et al.
      Modelling the impact of menthol sales restrictions and retailer density reduction policies: insights from tobacco town Minnesota.
      ,
      • Li Y
      • Lawley MA
      • Siscovick DS
      • Zhang D
      • Pagán JA.
      Agent-based modeling of chronic diseases: a narrative review and future research directions.
      Tools to show context-relevant simulation output can help convey potential impacts and be useful springboards for informing specific recommendations. For example, ABMs that have been iteratively developed and applied have provided actionable guidance on the selection of tobacco control policies such as menthol sales restrictions and retailer density reduction across communities.
      • Combs TB
      • McKay VR
      • Ornstein J
      • et al.
      Modelling the impact of menthol sales restrictions and retailer density reduction policies: insights from tobacco town Minnesota.
      ,
      • Luke DA
      • Hammond RA
      • Combs T
      • et al.
      Tobacco town: computational modeling of policy options to reduce tobacco retailer density.
      This model might similarly shape recommendations to reduce misimplementation in specific public health contexts.

      ACKNOWLEDGMENTS

      Special thanks go to the Expert Advisory Group, especially the late Julia Thorsness, who provided important input and feedback throughout the research. Carol Brownson, Randy Schwartz, Frank Bright, and Paula Clayton provided invaluable advice and feedback on the guide for supplementary stakeholder interviews; Carol Brownson additionally assisted by conducting many of these interviews. David O'Gara helped to review and finalize manuscript figures.
      The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the NIH or the Centers for Disease Control and Prevention.
      This project is funded by the National Cancer Institute of the NIH (R01CA214530, P50CA244431), the Centers for Disease Control and Prevention (Number U48DP006395), and the Foundation for Barnes-Jewish Hospital.
      No financial disclosures were reported by the authors of this paper.

      CRediT AUTHOR STATEMENT

      Matt Kasman: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Validation, Writing - Original Draft Preparation, Writing - Review and Editing. Ross A. Hammond: Conceptualization, Methodology, Project administration, Writing - Review and Editing. Rob Purcell: Conceptualization, Software, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review and Editing. Louise Farah Saliba: Conceptualization, Formal analysis, Investigation, Writing - Original Draft Preparation, Writing - Review and Editing. Stephanie Mazzucca: Conceptualization, Formal analysis, Investigation, Writing - Review and Editing. Margaret Padek: Writing - Review and Editing, Conceptualization, Formal analysis, Investigation, Writing - Review and Editing. Peg Allen: Conceptualization, Formal analysis, Investigation, Writing - Original Draft Preparation, Writing - Review and Editing. Douglas A. Luke: Conceptualization, Writing - Review and Editing. Sarah Moreland-Russell: Conceptualization, Writing - Review and Editing. Paul C. Erwin: Conceptualization, Writing - Review and Editing. Ross C. Brownson: Conceptualization, Funding acquisition, Project administration, Supervision, Writing - Review and Editing.

      Appendix. SUPPLEMENTAL MATERIAL

      REFERENCES

        • Padek M
        • Allen P
        • Erwin PC
        • et al.
        Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.
        Implement Sci. 2018; 13: 49https://doi.org/10.1186/s13012-018-0742-9
        • Brownson RC
        • Allen P
        • Jacob RR
        • et al.
        Understanding mis-implementation in public health practice.
        Am J Prev Med. 2015; 48: 543-551https://doi.org/10.1016/j.amepre.2014.11.015
      1. The Guide to Community Preventive Services (the Community Guide). Community Preventive Services Task Force. https://www.thecommunityguide.org/. Updated January 24, 2020. Accessed May 18, 2022.

      2. 2021 Health disparities report. America's Health Rankings, United Health Foundation. https://www.americashealthrankings.org/learn/reports/2021-disparities-report. Updated June 28, 2021. Accessed May 16, 2022.

        • Centers for Disease Control and Prevention
        CDC Health Disparities and Inequalities Report – United States, 2013.
        Centers for Disease Control and Prevention, Atlanta, GA2013 (Accessed May 18, 2022)
        • Padek MM
        • Mazzucca S
        • Allen P
        • et al.
        Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.
        BMC Public Health. 2021; 21: 101https://doi.org/10.1186/s12889-020-10101-z
        • Allen P
        • Jacob RR
        • Parks RG
        • et al.
        Perspectives on program mis-implementation among U.S. local public health departments.
        BMC Health Serv Res. 2020; 20: 258https://doi.org/10.1186/s12913-020-05141-5
        • Hammond RA
        Considerations and best practices in agent-based modeling to inform policy.
        Assessing the Use of Agent-Based Models for Tobacco Regulation. National Academies Press (U.S.), Washington, DC2015
        https://www.ncbi.nlm.nih.gov/books/NBK305917/
        Date accessed: December 5, 2022
        • Tracy M
        • Cerdá M
        • Keyes KM.
        Agent-based modeling in public health: current applications and future directions.
        Annu Rev Public Health. 2018; 39: 77-94https://doi.org/10.1146/annurev-publhealth-040617-014317
        • Silverman E
        • Gostoli U
        • Picascia S
        • et al.
        Situating agent-based modelling in population health research.
        Emerg Themes Epidemiol. 2021; 18: 10https://doi.org/10.1186/s12982-021-00102-7
        • Huang W
        • Chang CH
        • Stuart EA
        • et al.
        Agent-based modeling for implementation research: an application to tobacco smoking cessation for persons with serious mental illness.
        Implement Res Pract. 2021; 2https://doi.org/10.1177/26334895211010664
        • McKay VR
        • Hoffer LD
        • Combs TB
        • Margaret Dolcini M
        The dynamic influence of human resources on evidence-based intervention sustainability and population outcomes: an agent-based modeling approach.
        Implement Sci. 2018; 13: 77https://doi.org/10.1186/s13012-018-0767-0
        • Li Y
        • Padrón NA
        • Mangla AT
        • Russo PG
        • Schlenker T
        • Pagán JA.
        Using systems science to inform population health strategies in local health departments: a case study in San Antonio, Texas.
        Public Health Rep. 2017; 132: 549-555https://doi.org/10.1177/0033354917722149
        • Li Y
        • Kong N
        • Lawley M
        • Weiss L
        • Pagán JA.
        Advancing the use of evidence-based decision-making in local health departments with systems science methodologies.
        Am J Public Health. 2015; 105: S217-S222https://doi.org/10.2105/AJPH.2014.302077
        • Combs TB
        • McKay VR
        • Ornstein J
        • et al.
        Modelling the impact of menthol sales restrictions and retailer density reduction policies: insights from tobacco town Minnesota.
        Tob Control. 2020; 29: 502-509https://doi.org/10.1136/tobaccocontrol-2019-054986
        • Luke DA
        • Hammond RA
        • Combs T
        • et al.
        Tobacco town: computational modeling of policy options to reduce tobacco retailer density.
        Am J Public Health. 2017; 107: 740-746https://doi.org/10.2105/AJPH.2017.303685
        • Hammond R
        • Ornstein JT
        • Purcell R
        • Haslam MD
        • Kasman M.
        Modeling robustness of COVID-19 containment policies [preprint].
        OSF Preprint. 2021; (Online May 13,)https://doi.org/10.31219/osf.io/h5ua7
        • Zhang D
        • Giabbanelli PJ
        • Arah OA
        • Zimmerman FJ.
        Impact of different policies on unhealthy dietary behaviors in an urban adult population: an agent-based simulation model.
        Am J Public Health. 2014; 104: 1217-1222https://doi.org/10.2105/AJPH.2014.301934
        • Chang MH
        • Harrington Jr, JE
        Agent-based models of organizations.
        Handb Comput Econ. 2006; 2: 1273-1337https://doi.org/10.1016/S1574-0021(05)02026-5
        • Carley KM.
        Computational organization science: A new frontier.
        Proc Natl Acad Sci U S A. 2002; 99: 7257-7262https://doi.org/10.1073/pnas.082080599
        • Fioretti G.
        Agent-based simulation models in organization science.
        Organ Res Methods. 2013; 16: 227-242https://doi.org/10.1177/1094428112470006
        • Secchi D.
        A case for agent-based models in organizational behavior and team research.
        Team Perform Manag. 2015; 21: 37-50https://doi.org/10.1108/TPM-12-2014-0063
        • Aarons GA
        • Hurlburt M
        • Horwitz SM.
        Advancing a conceptual model of evidence-based practice implementation in public service sectors.
        Adm Policy Ment Health. 2011; 38: 4-23https://doi.org/10.1007/s10488-010-0327-7
        • Moullin JC
        • Dickson KS
        • Stadnick NA
        • Rabin B
        • Aarons GA.
        Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.
        Implement Sci. 2019; 14: 1https://doi.org/10.1186/s13012-018-0842-6
        • Ramanadhan S
        • Davis MM
        • Armstrong R
        • et al.
        Participatory implementation science to increase the impact of evidence-based cancer prevention and control.
        Cancer Causes Control. 2018; 29: 363-369https://doi.org/10.1007/s10552-018-1008-1
        • Moreland-Russell S
        • Weno ER
        • Padek M
        • Saliba LF
        • Brownson RC.
        Leading the way: qualities of leaders in preventing mis-implementation of public health programs [preprint].
        Research Square. 2020; (Online December 15,)https://doi.org/10.21203/rs.3.rs-127907/v1
        • Kasman M
        • Breen N
        • Hammond RA.
        Complex systems science.
        The Science of Health Disparities Research. John Wiley & Sons, Ltd, Hoboken, NJ2021: 243-256https://doi.org/10.1002/9781119374855.ch15
        • Luke DA
        • Stamatakis KA.
        Systems science methods in public health: dynamics, networks, and agents.
        Annu Rev Public Health. 2012; 33: 357-376https://doi.org/10.1146/annurev-publhealth-031210-101222
        • Miller JH
        • Page SE
        • Page S
        Complex Adaptive Systems.
        Princeton University Press, Princeton, NJ2009
        • Brownson RC
        • Shelton RC
        • Geng EH
        • Glasgow RE.
        Revisiting concepts of evidence in implementation science.
        Implement Sci. 2022; 17: 26https://doi.org/10.1186/s13012-022-01201-y
        • Brownson RC
        • Fielding JE
        • Green LW.
        Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research.
        Annu Rev Public Health. 2018; 39: 27-53https://doi.org/10.1146/annurev-publhealth-040617-014746
        • Jacob RR
        • Brownson CA
        • Deshpande AD
        • et al.
        Long-term evaluation of a course on evidence-based public health in the U.S. and Europe.
        Am J Prev Med. 2021; 61: 299-307https://doi.org/10.1016/j.amepre.2021.03.003
        • Brownson RC
        • Fielding JE
        • Maylahn CM.
        Evidence-based public health: a fundamental concept for public health practice.
        Annu Rev Public Health. 2009; 30: 175-201https://doi.org/10.1146/annurev.publhealth.031308.100134
        • Brownson RC
        • Allen P
        • Duggan K
        • Stamatakis KA
        • Erwin PC.
        Fostering more-effective public health by identifying administrative evidence-based practices: a review of the literature.
        Am J Prev Med. 2012; 43: 309-319https://doi.org/10.1016/j.amepre.2012.06.006
        • Eyler AA
        • Valko C
        • Ramadas R
        • Macchi M
        • Fershteyn Z
        • Brownson RC.
        Administrative evidence-based practices in state chronic disease practitioners.
        Am J Prev Med. 2018; 54: 275-283https://doi.org/10.1016/j.amepre.2017.09.006
        • Budd EL
        • Ying X
        • Stamatakis KA
        • et al.
        Developing a survey tool to assess implementation of evidence-based chronic disease prevention in public health settings across four countries.
        Front Public Health. 2019; 7: 152https://doi.org/10.3389/fpubh.2019.00152
        • Reis RS
        • Duggan K
        • Allen P
        • Stamatakis KA
        • Erwin PC
        • Brownson RC.
        Developing a tool to assess administrative evidence-based practices in local health departments.
        Front Public Health Serv Syst Res. 2014; 3: 1-5https://doi.org/10.13023/FPHSSR.0303.02
        • Allen P
        • Sequeira S
        • Jacob RR
        • et al.
        Promoting state health department evidence-based cancer and chronic disease prevention: a multi-phase dissemination study with a cluster randomized trial component.
        Implement Sci. 2013; 8: 141https://doi.org/10.1186/1748-5908-8-141
        • Mazzucca S
        • Parks RG
        • Tabak RG
        • et al.
        Assessing organizational supports for evidence-based decision making in Local Public Health Departments in The United States: development and psychometric properties of a new measure.
        J Public Health Manag Pract. 2019; 25: 454-463https://doi.org/10.1097/PHH.0000000000000952
        • Mazzucca S
        • Saliba LF
        • Smith R
        • et al.
        It's good to feel like you're doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA.
        Implement Sci Commun. 2022; 3: 1-12https://doi.org/10.1186/s43058-021-00252-4
        • Jacob RR
        • Baker EA
        • Allen P
        • et al.
        Training needs and supports for evidence-based decision making among the public health workforce in the United States.
        BMC Health Serv Res. 2014; 14: 564https://doi.org/10.1186/s12913-014-0564-7
        • Kasman M
        • Kreuger LK.
        Best practices for systems science research.
        National Institutes of Health Office of Behavioral and Social Sciences Research, Bethesda, MD2022
        • Epstein JM
        Generative Social Science.
        Princeton University Press, Princeton, NJ2012
        • Rodriguez Weno E
        • Allen P
        • Mazzucca S
        • et al.
        Approaches for ending ineffective programs: strategies from state public health practitioners.
        Front Public Health. 2021; 9: 1-10https://doi.org/10.3389/fpubh.2021.727005
        • Baker EA
        • Brownson RC
        • Dreisinger M
        • McIntosh LD
        • Karamehic-Muratovic A.
        Examining the role of training in evidence-based public health: a qualitative study.
        Health Promot Pract. 2009; 10: 342-348https://doi.org/10.1177/1524839909336649
        • Morrison EW
        • Wheeler-Smith SL
        • Kamdar D.
        Speaking up in groups: a cross-level study of group voice climate and voice.
        J Appl Psychol. 2011; 96: 183-191https://doi.org/10.1037/a0020744
        • Detert JR
        • Treviño LK.
        Speaking up to higher-ups: how supervisors and skip-level leaders influence employee voice.
        Organ Sci. 2010; 21: 249-270https://doi.org/10.1287/orsc.1080.0405
        • Morrison EW.
        Employee voice and silence.
        Annu Rev Organ Psychol Organ Behav. 2014; 1: 173-197https://doi.org/10.1146/annurev-orgpsych-031413-091328
        • Hartley J
        • Alford J
        • Hughes O
        • Yates S.
        Public value and political astuteness in the work of public managers: the art of the possible.
        Public Admin. 2015; 93: 195-211https://doi.org/10.1111/padm.12125
        • Joseph J
        • Gaba V.
        Organizational structure, information processing, and decision-making: a retrospective and road map for research.
        Acad Manag Ann. 2020; 14: 267-302https://doi.org/10.5465/annals.2017.0103
        • Hu H
        • Allen P
        • Yan Y
        • Reis RS
        • Jacob RR
        • Brownson RC.
        Organizational supports for research evidence use in state public health agencies: a latent class analysis.
        J Public Health Manag Pract. 2019; 25: 373-381https://doi.org/10.1097/PHH.0000000000000821
        • Hammond RA.
        Complex systems modeling for obesity research.
        Prev Chronic Dis. 2009; 6: A97
        • Hammond RA
        • Ornstein JT.
        A model of social influence on body mass index.
        Ann N Y Acad Sci. 2014; 1331: 34-42https://doi.org/10.1111/nyas.12344
        • Hall KD
        • Hammond RA
        • Rahmandad H.
        Dynamic interplay among homeostatic, hedonic, and cognitive feedback circuits regulating body weight.
        Am J Public Health. 2014; 104: 1169-1175https://doi.org/10.2105/AJPH.2014.301931
        • Hawkes C
        • Smith TG
        • Jewell J
        • et al.
        Smart food policies for obesity prevention.
        Lancet. 2015; 385: 2410-2421https://doi.org/10.1016/S0140-6736(14)61745-1
        • Li Y
        • Lawley MA
        • Siscovick DS
        • Zhang D
        • Pagán JA.
        Agent-based modeling of chronic diseases: a narrative review and future research directions.
        Prev Chronic Dis. 2016; 13: E69https://doi.org/10.5888/pcd13.150561
      3. Kasman M, Hammond RA, Heuberger B, et al. Activating a community: an agent-based model of romp & chomp, a whole-of-community childhood obesity intervention. Obesity (Silver Spring). 2019;27(9):1494–1502. https://doi.org/10.1002/oby.22553.

        • Morshed AB
        • Kasman M
        • Heuberger B
        • Hammond RA
        • Hovmand PS.
        A systematic review of system dynamics and agent-based obesity models: evaluating obesity as part of the global syndemic.
        Obes Rev. 2019; 20: 161-178https://doi.org/10.1111/obr.12877
        • Burke JG
        • Lich KH
        • Neal JW
        • Meissner HI
        • Yonas M
        • Mabry PL.
        Enhancing dissemination and implementation research using systems science methods.
        Int J Behav Med. 2015; 22: 283-291https://doi.org/10.1007/s12529-014-9417-3
        • Combs T
        • Nelson KL
        • Luke D
        • et al.
        Simulating the role of knowledge brokers in policy making in state agencies: an agent-based model.
        Health Serv Res. 2022; 57: 122-136https://doi.org/10.1111/1475-6773.13916
        • Bogaert K
        • Castrucci BC
        • Gould E
        • Rider N
        • Whang C
        • Corcoran E.
        Top training needs of the governmental public health workforce.
        J Public Health Manag Pract. 2019; 25 (suppl): S134-S144https://doi.org/10.1097/PHH.0000000000000936
        • Koh S
        • Lee M
        • Brotzman LE
        • Shelton RC.
        An orientation for new researchers to key domains, processes, and resources in implementation science.
        Transl Behav Med. 2020; 10: 179-185https://doi.org/10.1093/tbm/iby095