Barriers and Facilitators to the Implementation of the Early-Onset Sepsis Calculator: A Multicenter Survey Study

1. Introduction

Use of the Early-Onset Sepsis (EOS) Calculator, a risk-assessment tool helping physicians narrow antibiotic use in neonates at risk for EOS, is quickly spreading internationally. The calculator was developed by the Kaiser Permanente Research division and is based on a multivariate risk prediction model, combining both maternal intrapartum factors and objective neonatal clinical findings to estimate individual EOS risk and subsequently give policy recommendations [1,2,3]. Studies comparing the EOS calculator to conventional management strategies showed a significant decrease in neonatal empiric antibiotic use [4,5,6,7]. A number of Dutch hospitals already use the calculator in the study context of a multicenter cluster randomized controlled trial [8]. Similar to its inclusion in the latest National Institute for Health and Care Excellence (NICE) guidelines [9], the tool is likely to be included in the upcoming revision of the Dutch national guidelines, as well as in international guidelines, and will be implemented in clinical practice.

However, successful implementation of a revised guideline is not self-evident, as the translation of novel evidence into daily clinical practice is found to often be suboptimal [10]. Three large systematic reviews on guideline adherence in different healthcare settings showed adherence rates ranging from 0 to 98%, with mean rates around 50–70% [11,12,13]. A study investigating adherence to the current Dutch guidelines on EOS management reported only 42.5% adherence to recommendations regarding the start of antibiotic treatment [14]. Though no data are available yet on the EOS calculator adherence in daily clinical practice, few effectiveness and safety studies did mention some concerns regarding EOS calculator use, including safety concerns, incorrect use and practical concerns, which may contribute to low adherence rates [15,16,17,18]. Since low guideline adherence leads to variations in daily clinical practice, a waste of resources and suboptimal patient outcomes, research should not end with evidence for a guideline or tool but continue with studies on how to achieve successful implementation [10,19,20]. Prior to implementing an innovation, gaining insight into the factors that may hinder or promote its adoption is an essential step in the preparation process.

The primary objective of this study is to identify the key barriers and facilitators perceived by stakeholders of EOS calculator implementation. By understanding these factors, this study seeks to inform a comprehensive implementation strategy that will optimize the successful integration and utilization of the EOS calculator in practice.

2. Materials and Methods

2.1. Study Design and Participants

A multicenter cross-sectional online survey was conducted among stakeholders of EOS calculator implementation in thirteen Dutch hospitals. For selecting hospitals, purposive sampling was used in order to reflect different types of neonatal care in the Netherlands and ensure representation of all groups of stakeholders. Among included hospitals, the highest level of neonatal care was the Neonatal Intensive Care Unit (NICU) in 2 hospitals, High Care in 7 hospitals and Medium Care in 4 hospitals. The NICUs are categorized as tertiary care facilities, offering the most advanced level of care for critically ill neonates and severely preterm infants (<30 weeks of gestation). High-Care departments are situated at regular pediatric departments and provide care for ill neonates and preterm infants >30 weeks of gestation, facilitating continuous monitoring and respiratory support. Medium-Care departments are also situated at regular pediatric departments and provide simple medical care for neonates >32 weeks of gestation. In some cases, the care provided in the Medium-Care department may be extended to the maternity ward, contingent on local agreements.

To identify relevant stakeholder groups, Dutch workflows of managing neonates at risk for EOS were explicated. In case of EOS calculator use, physicians of a neonatology ward examine neonates at risk for EOS and fill in the EOS calculator as they are responsible for neonatal policy. Physicians of the obstetrics ward play an important role in sampling and sharing information about maternal factors needed for the EOS calculator. Nurses of both the neonatology and obstetrics wards have to execute some of the calculators’ practical recommendations, such as measuring neonatal vital signs. Based on this workflow, four main groups of stakeholders were identified: (1) physicians of the neonatology ward (PN) (pediatricians, neonatologists, pediatric residents and physician assistants); (2) physicians of the obstetrics ward (PO) (gynecologists, perinatologists, clinical obstetricians and gynecologic residents); (3) nurses of the obstetrics ward (NO) (maternity nurses and obstetric nurses); and (4) nurses of the neonatology ward (NN) (NICU nurses, neonatal nurses and pediatric nurses).

2.2. Survey Development

The Consolidated Framework for Implementation Research (CFIR) was used to structure survey development and analysis. The CFIR is a frequently used determinant framework and combines different theories and existing models into a list of 5 domains and 39 constructs of implementation success in healthcare settings [21]. The five domains are (1) intervention characteristics, (2) outer setting, (3) inner setting, (4) characteristics of individuals and (5) implementation process (Figure 1). Steps of survey development were informed by the guide of Burns et al. on designing and conducting self-administered surveys of clinicians [22]. The CROSS checklist (Consensus-Based Checklist for Reporting of Survey Studies) was used as a reporting checklist [23].

Figure 1. Graphic overview of the five domains of the CFIR, focused on the setting of this study.

2.3. Data Analysis

Survey data were exported and analyzed using IBM SPSS Statistics 28.0. Only fully completed surveys were included for analysis. Likert-data were coded from 1 to 5, with 1 = totally disagree/very unimportant to 5 = totally agree/very important. Reversely formulated statements were recoded so that a higher score indicated participant’s stronger association with the topic. For data analysis and presentation, Likert-data were divided into 2 categories: irrelevant/neutral (I) = 1 + 2 + 3 and relevant (R) = 4 + 5. Relevant facilitators were defined as facilitators that were rated ‘R’ by >50% of respondents. Relevant barriers were defined as barriers that were rated ‘R’ by >10% of respondents, as barriers applicable to only small numbers of stakeholders may also have a large impact on implementation outcomes. For all closed questions, the frequency distribution of reported barriers and facilitators in the total group and, if applicable, sub-groups were calculated in percentages. If deemed relevant for practice, differences between subgroups were tested using the chi-square test of independence, followed by post hoc Bonferroni testing. p-values of <0.05 were considered statistically significant. Open-ended questions were analyzed through open coding. Abbreviations (PN, PO, NO, NN) were used to describe subgroups.

3. Results

3.1. Participants and Guidelines

The survey was sent to 2054 eligible participants of 13 Dutch hospitals. The survey link was opened 1154 times, resulting in 522 survey responses, of which 465 (23%) were fully completed (Figure 2). The response rate was 39% in the PN group, 20% in the PO group, 17% in the NO group and 21% in the NN group. No information on unique visitors was available. Of the 465 respondents, 40.2% reported to use the current Dutch guidelines, 16.6% the EOS calculator, 9.9% a local protocol, 12.7% a combination of protocols and 20.6% reported unknown.

Figure 2. Study flow chart.

3.2. Survey Part 1: Reported Barriers and Facilitators by All Respondents

Relevant facilitators and barriers for the total group of respondents, allocated to the CFIR domain, are displayed in . Relevant barriers were found only in the inner setting domain and concerned capacity shortage and problems with maternal information transfer between the departments. Capacity shortage was significantly more reported by physicians and nurses of the obstetrics ward compared to those of the neonatology ward and was mainly expected at the maternity ward, both due to staff and room shortage . Problems with the handover of maternal information from the obstetric to neonatology department were significantly more reported by physicians of the neonatology ward, compared to the other three subgroups.

Relevant facilitators were found in all five domains of the CFIR, though the majority of facilitators concerned intervention characteristics, including the reduction in neonatal antibiotic prescriptions, reduction in neonatal side-effects, less mother–child separation, reduction in blood tests and net shorter hospital stays. Endorsement of the EOS calculator by the NVK (outer setting), integration of the EOS calculator in EHR systems (inner setting), the belief that the EOS calculator is safe and effective (individual characteristics), the provision of training on the EOS calculator and the provision of feedback on implementation results at the own department (process) were found to be other relevant facilitators. An additional question regarding education showed clear differences in preferences between subgroups . An instructional video about the EOS calculator was chosen by the majority of all stakeholders. However, education about scientific evidence was specifically rated important by physicians, whereas a clinical lesson about EOS was clearly more preferred among nurses.

3.3. Survey Part 2: Reported Barriers and Facilitators per Group

Relevant facilitators and barriers reported by subgroups, allocated to CFIR domain, are displayed in . A PN-specific barrier was the belief that maternity nurses are not adequately trained to measure neonatal vital signs. PN-specific facilitators concerned tension for change of the current NVK guidelines and the general feeling that too many antibiotics are currently prescribed. PO-specific barriers concerned the expectation that the EOS calculator will increase workload, more neonates from primary care will be admitted to the hospital and the belief that maternity nurses are not adequately trained to measure neonatal vital signs. No PO-specific facilitators were found. NO-specific barriers concerned feeling incompetent to measure neonatal heart and respiratory rate and not being informed in a timely manner about changes in physicians’ protocols. NO-specific facilitators concerned the availability of an EOS calculator smartphone app, clear communication by physicians about reasons for policy choices and availability of a local implementation team. An NN-specific barrier concerned nurses not being informed in a timely manner about changes in physicians’ protocols. NN-specific facilitators concerned clear communication by physicians about reasons for policy choices and the availability of a local implementation team.

3.4. Survey Part 3: Reported Barriers and Facilitators by EOS Calculator Users

Relevant facilitators and barriers reported by EOS calculator users, allocated to the CFIR domain, are displayed in . Facilitators for the total group of EOS calculator users concerned the feeling that the care for neonates is more uniform since the implementation of the EOS calculator and the thought that parents agree with EOS calculator recommendations. A PN-specific barrier concerned encountering some textual or substantive uncertainties when using the EOS calculator. A PN-specific facilitator concerned the experience that the EOS calculator is supportive in making clinical decisions.

4. Discussion

To our knowledge, this study described the first systematical pre-implementation evaluation of factors influencing implementation of the EOS calculator. Our primary focus centered on identifying the barriers and facilitators of EOS calculator implementation as perceived by stakeholders within Dutch hospitals. Analysis revealed the presence of facilitators and barriers across all five domains of the CFIR. The facilitators were generally applicable across diverse stakeholder groups, while the barriers exhibited a tendency to be more discipline-specific.

Two main organizational barriers to EOS calculator implementation emerged: insufficient capacity in neonatal care departments and challenges in transferring maternal information between departments. Notably, capacity concerns were more prominent among obstetric physicians and nurses, likely tied to issues primarily expected at the maternity ward. However, this expectation contrasts with research indicating shorter hospital stays for neonates when using the EOS calculator [41]. The discrepancy might be attributed to the anticipated shift of care from the neonatology ward to the maternity ward, accompanied by increased clinical examinations.

Since comprehensive maternal information is vital for utilizing the EOS calculator effectively, streamlining the process of obtaining and transferring these data is critical. Respondents in this study suggested implementing standardized consultation forms with smart text features and enabling autofill for the EOS calculator through integration into the EHR. Similar strategies, like checklists or fill-in-the-blank handovers, have proven successful in improving information transfer in other healthcare settings [44,45].

Moreover, this study identified a barrier among nurses in the obstetrics departments, indicating that a small but relevant group of them does not feel confident in measuring neonatal respiratory rate and heart rate. This lack of self-efficacy is crucial to take into account, as it plays a significant role in determining one’s motivation to perform tasks [46]. Additionally, both groups of physicians expressed concerns about the adequacy of training for maternity nurses in measuring neonatal vitals. These combined findings emphasize the importance of implementing an intervention targeting this specific issue.

Facilitators identified in this study were several relative advantages of the EOS calculator, such as less mother–child separation and net shorter hospital stays. However, it is important to underscore that the extent to which relative advantages foster promotion is significantly contingent upon their visibility in daily practice [47,48]. Visibility may be improved through stakeholder education, wherein emphasis is placed on elucidating advantages. As stakeholders in this study have reported diverse educational preferences, the adaptation of educational methods for the target group should be considered. However, the standalone impact of education possesses certain limitations and should therefore be combined with other strategies [49,50]. The significance of evaluation and feedback has been widely recognized in the literature [47,51,52]. Facilitating this process involves acquiring and disseminating objective data pertaining to implementation success [53]. Notably, a recent meta-analysis revealed that the efficacy of feedback is strongly contingent on various factors such as the methodology employed, the recipients of the feedback and the context in which it is provided, thereby offering implications for practical implementation. For instance, feedback aimed at directly aiding clinical behavior demonstrated the highest level of effectiveness [54].

Integration of the EOS calculator in the EHR was deemed facilitative by a significant majority of stakeholders. This observation aligns with prior research, which demonstrated that the inclusion of clinical decision tools, exemplified by the EOS calculator, within the EHR, enhances their integration into clinical workflows [15,42,55,56]. Conversely, the importance of a smartphone application was only reported by a few stakeholders, which is an interesting finding, given that the concurrent RCT relies entirely on an EOS calculator smartphone application as its methodological approach [8]. The forthcoming findings from this RCT hold the potential to offer further insights and guidance on this subject matter.

The vast majority of all respondents expected the EOS calculator to be safe and effective. The reported general expectation of safety and effectivity is considered a beneficial starting point for implementing the EOS calculator. Tension for change of the current Dutch guidelines was reported by a large majority of physicians of the neonatology ward, as was expected based on these group’s reported low adherence to the current NVK guidelines [14]. It should be noted, however, that tension for change was not clearly present in the other groups of stakeholders, possibly because of their more indirect roles with regard to antibiotic use. It is crucial to pay attention to these stakeholder groups as a sense of urgency and the need for change are essential factors for the successful implementation of any new guidelines [57,58].

4.1. Strengths and Limitations

A major asset of this study was the inclusion of different groups of stakeholders, resulting in a complete overview of factors influencing implementation. By selecting hospitals with different levels of neonatal care, optimal reflection of the target population was pursued. As all stakeholders employed in the participating centers were invited to take part in the survey, we aimed to avoid selection bias. The survey being conducted before starting large-scale implementation ensures that the implementation can be tackled properly from the beginning. Still, additional factors may come forward during actual implementation, emphasizing the importance of evaluation. The CFIR framework, recommended for implementation projects, was used as available during the study, providing a robust theoretical base [59,60,61]. Currently, a revised CFIR has been published, which should be considered for future implementation research [62].

We recognize that our study has certain limitations. Firstly, the response rate was relatively low, which introduces a significant risk of non-response bias, a common challenge in healthcare professionals’ surveys [63,64,65]. As our aim was to obtain as many perspectives as possible and avoid selection bias, we chose to invite every single stakeholder in the included hospitals. Though many responses were gathered, this approach did not necessarily result in optimal response rates. While acknowledging non-response bias risk, we assert that our findings remain practically valuable. The absolute number of nearly 500 responses across 13 hospitals of varying care levels is robust and allowed for a diverse sample that effectively captured stakeholder perspectives. Although there is a chance that this study did not capture certain barriers or facilitators that non-responders could have brought to light, the factors identified in our research hold undeniable significance for their incorporation into clinical practice. Secondly, a clear disparity in response rate was seen among physicians from the neonatology ward (39%) compared to other stakeholders (PO:20%, NO: 17%, NN: 21%). It is described that people with more interest in the topic are more likely to answer a survey. At the time of our study, the EOS calculator was best known by physicians of the neonatal ward, mainly through pediatric journal publications and congresses. For physicians of the obstetric ward and both groups of nurses, the topic was further away, which likely contributed to lower response rates in these groups. Besides the lack of familiarity, the lack of time and the large number of surveys nurses might have diminished the response rate [66]. In our pilot phase, length and readability of the survey were not found to be an issue. Furthermore, hospitals typically have employed a greater number of physicians of the obstetrics ward and nurses, leading to a higher volume of surveys distributed to them compared to physicians in the neonatology ward. As the core importance of this topic rests with physicians of the neonatology ward, there might have been a relatively uneven distribution. Thirdly, potential bias was created by dichotomizing data for analysis into ‘relevant’ and ‘irrelevant’, ignoring the nuance of the ‘neutral’ option. However, it was well considered to do this, as this study’s aim was to identify genuinely relevant barriers and facilitators, which were not reflected by the ‘neutral’ option.

4.2. Clinical Implications

Our study provides a blueprint and benchmark for countries and networks looking to successfully implement the EOS calculator. Moreover, it may serve as an example for implementing other diagnostic score strategies aiming to narrow antibiotic use, such as serial clinical examination. Findings of this study may inform and facilitate an implementation strategy for the EOS calculator specifically (see , taking into account that additional strategies are needed when other factors come to light. Sufficient information supply is essential to both stimulate facilitators and tackle barriers. To show relative advantages of the EOS calculator and foster tension for change among stakeholders, local discussion and educational meetings should be conducted. Early adopters and local champions should be identified, so that they can be stimulated to motivate and educate colleagues. Education should consist of both ready-to-use materials, such as an instructional video, as well as educational meetings tailored to the target group. For nurses of the obstetrics ward, access to training on measuring neonatal vital signs should be provided. Focus must be placed on tackling reported organizational problems, including capacity shortage and suboptimal information transfer. A culture of collective responsibility of both the obstetrics and neonatology wards should be created by collective meetings and a collective point of contact. Local capacity analysis should be conducted to asses local needs and inform the team. Information transfer from the obstetrics to neonatology ward should be made as easy as possible, by using checklists, smart phrases, order sets and autofill. Integration of the EOS calculator in all Dutch EHR programs should be pursued. A smartphone application tailored to the national setting might be helpful, yet should not be obligatory. Evaluation should both measure the success of implementation and success of the innovation. It should be evaluated to what extent the expected facilitators and barriers found is this study are comparable to those after implementation, and data in the EHR should be used to analyze if implementation indeed resulted in the intended effects of the EOS calculator, such as the reduction in antibiotic prescriptions. Evaluation-based points of action should be clearly communicated with the team in a timely manner to create an environment of shared learning.

5. Conclusions

Our study showed a variety of barriers and facilitators of EOS calculator implementation among all relevant groups of stakeholders. Based on these findings, the EOS calculator can potentially be implemented, improving the experience of healthcare personnel as well as ultimately improving patient outcomes.

References

1. Kuzniewicz, M.W.; Walsh, E.M.; Li, S.; Fischer, A.; Escobar, G.J. Development and Implementation of an Early-Onset Sepsis Calculator to Guide Antibiotic Management in Late Preterm and Term Neonates. Jt. Comm. J. Qual. Patient Saf. 2016, 42, 232–239. [Google Scholar] [CrossRef] [PubMed]
2. Puopolo, K.M.; Draper, D.; Wi, S.; Newman, T.B.; Zupancic, J.; Lieberman, E.; Smith, M.; Escobar, G.J. Estimating the Probability of Neonatal Early-Onset Infection on the Basis of Maternal Risk Factors. Pediatrics 2011, 128, e1155–e1163. [Google Scholar] [CrossRef] [PubMed]
3. Escobar, G.J.; Puopolo, K.M.; Wi, S.; Turk, B.J.; Kuzniewicz, M.W.; Walsh, E.M.; Newman, T.B.; Zupancic, J.; Lieberman, E.; Draper, D. Stratification of Risk of Early-Onset Sepsis in Newborns ≥34 Weeks’ Gestation. Pediatrics 2014, 133, 30–36. [Google Scholar] [CrossRef] [PubMed]
4. Achten, N.B.; Klingenberg, C.; Benitz, W.E.; Stocker, M.; Schlapbach, L.J.; Giannoni, E.; Bokelaar, R.; Driessen, G.J.A.; Brodin, P.; Uthaya, S.; et al. Association of Use of the Neonatal Early-Onset Sepsis Calculator with Reduction in Antibiotic Therapy and Safety. JAMA Pediatr. 2019, 173, 1032–1040. [Google Scholar] [CrossRef] [PubMed]
5. Deshmukh, M.; Mehta, S.; Patole, S. Sepsis calculator for neonatal early onset sepsis—a systematic review and meta-analysis. J. Matern. Neonatal Med. 2021, 34, 1832–1840. [Google Scholar] [CrossRef] [PubMed]
6. Goel, N.; Shrestha, S.; Smith, R.; Mehta, A.; Ketty, M.; Muxworthy, H.; Abelian, A.; Kirupaalar, V.; Saeed, S.; Jain, S.; et al. Screening for early onset neonatal sepsis: NICE guidance-based practice versus projected application of the Kaiser Permanente sepsis risk calculator in the UK population. Arch. Dis. Child. Fetal Neonatal Ed. 2020, 105, 118–122. [Google Scholar] [CrossRef] [PubMed]
7. Kuzniewicz, M.W.; Puopolo, K.M.; Fischer, A.; Walsh, E.M.; Li, S.; Newman, T.B.; Kipnis, P.; Escobar, G.J. A Quantitative, Risk-Based Approach to the Management of Neonatal Early-Onset Sepsis. JAMA Pediatr. 2017, 171, 365–371. [Google Scholar] [CrossRef] [PubMed]
8. van der Weijden, B.M.; van der Weide, M.C.; Plötz, F.B.; Achten, N.B. Evaluating safety and effectiveness of the early-onset sepsis calculator to reduce antibiotic exposure in Dutch at-risk newborns: A protocol for a cluster randomised controlled trial. BMJ Open 2023, 13, e069253. [Google Scholar] [CrossRef]
9. Neonatal Infection: Antibiotics for Prevention and Treatment NICE Guideline 2021. Available online: https://www.nice.org.uk/guidance/ng195 (accessed on 30 January 2022).
10. Grol, R.; Grimshaw, J. From best evidence to best practice: Effective implementation of change in patients’ care. Lancet 2003, 362, 1225–1230. [Google Scholar] [CrossRef]
11. McGlynn, E.A.; Aschm, S.M.; Adams, J.; Keesey, J.; Hicks, J.; DeCristofaro, A.; Kerr, E.A. The Quality of Health Care Delivered to Adults in the United States. N. Engl. J. Med. 2003, 26, 2635–2680. [Google Scholar] [CrossRef]
12. Ebben, R.H.; Vloet, L.C.; Verhofstad, M.H.; Meijer, S.; Groot, J.A.M.-D.; van Achterberg, T. Adherence to guidelines and protocols in the prehospital and emergency care setting: A systematic review. Scand. J. Trauma. Resusc. Emerg. Med. 2013, 21, 9–16. [Google Scholar] [CrossRef] [PubMed]
13. Grol, R. Successes and Failures in the Implementation of Evidence-Based Guidelines for Clinical Practice. Med. Care 2001, 39, II46. [Google Scholar] [CrossRef] [PubMed]
14. van der Weijden, B.M.; Achten, N.B.; Bekhof, J.; Evers, E.E.; Berk, M.; Kamps, A.W.; Rijpert, M.; Tusscher, G.W.T.; van Houten, M.A.; Plötz, F.B. Multicentre study found that adherence to national antibiotic recommendations for neonatal early-onset sepsis was low. Acta Paediatr. 2021, 110, 791–798. [Google Scholar] [CrossRef] [PubMed]
15. Dhudasia, M.B.; Mukhopadhyay, S.; Puopolo, K.M. Implementation of the Sepsis Risk Calculator at an Academic Birth Hospital. Hosp. Pediatr. 2018, 8, 243–250. [Google Scholar] [CrossRef] [PubMed]
16. Kopec, G.; Collin, M.; Das, A. Application of Kaiser Sepsis Calculator in culture-positive infants with early onset sepsis. World J. Pediatr. 2021, 17, 429–433. [Google Scholar] [CrossRef] [PubMed]
17. Snoek, L.; van Kassel, M.N.; Krommenhoek, J.F.; Achten, N.B.; Plötz, F.B.; van Sorge, N.M.; Brouwer, M.C.; van de Beek, D.; Bijlsma, M.W. Neonatal early-onset infections: Comparing the sensitivity of the neonatal early-onset sepsis calculator to the Dutch and the updated NICE guidelines in an observational cohort of culture-positive cases. EClinicalMedicine 2022, 44, 101270. [Google Scholar] [CrossRef] [PubMed]
18. Scott, P.A.; Lai, M.; Inglis, G.D.T.; Davies, M.W. Neonatal early-onset sepsis calculator safety in an Australian tertiary perinatal centre. J. Paediatr. Child Health 2022, 58, 863–867. [Google Scholar] [CrossRef]
19. Norton, W.E.; Chambers, D.A. Unpacking the complexities of de-implementing inappropriate health interventions. Implement. Sci. 2020, 15, 1–7. [Google Scholar] [CrossRef]
20. Pronovost, P.J. Enhancing Physicians’ Use of Clinical Guidelines. JAMA 2013, 310, 2501–2502. [Google Scholar] [CrossRef]
21. Damschroder, L.J.; Aron, D.C.; Keith, R.E.; Kirsh, S.R.; Alexander, J.A.; Lowery, J.C. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement. Sci. 2009, 4, 50. [Google Scholar] [CrossRef]
22. Burns, K.E.; Duffett, M.; Kho, M.E.; Meade, M.O.; Adhikari, N.K.; Sinuff, T.; Cook, D.J.; for the ACCADEMY Group. A guide for the design and conduct of self-administered surveys of clinicians. Can. Med. Assoc. J. 2008, 179, 245–252. [Google Scholar] [CrossRef] [PubMed]
23. Sharma, A.; Duc, N.T.M.; Thang, T.L.L.; Nam, N.H.; Ng, S.J.; Abbas, K.S.; Huy, N.T.; Marušić, A.; Paul, C.L.; Kwok, J.; et al. A Consensus-Based Checklist for Reporting of Survey Studies (CROSS). J. Gen. Intern. Med. 2021, 36, 3179–3187. [Google Scholar] [CrossRef] [PubMed]
24. Gong, C.L.; Dasgupta-Tsinikas, S.; Zangwill, K.M.; Bolaris, M.; Hay, J.W. Early onset sepsis calculator-based management of newborns exposed to maternal intrapartum fever: A cost benefit analysis. J. Perinatol. 2019, 39, 571–580. [Google Scholar] [CrossRef] [PubMed]
25. Cussen, A.; Guinness, L. Cost savings from use of a neonatal sepsis calculator in Australia: A modelled economic analysis. J. Paediatr. Child Health 2021, 57, 1037–1043. [Google Scholar] [CrossRef] [PubMed]
26. Zayek, M.; Bhat, J.; Bonner, K.; Blake, M.; Peevy, K.; Jha, O.P.; Gulati, R.; Bhat, R. Implementation of a Modified Neonatal Early-onset Sepsis Calculator in Well-baby Nursery: A Quality Improvement Study. Pediatr. Qual. Saf. 2020, 5, e330. [Google Scholar] [CrossRef] [PubMed]
27. Fischer, A.; Mowrer, M.C.; Shallat, S.; Walker, L.; Shallat, J. Ensuring a Locally Tailored Response to Early Onset Sepsis Screening Meets or Exceeds the Performance of Published Approaches. Hosp. Pediatr. 2020, 10, 877–883. [Google Scholar] [CrossRef] [PubMed]
28. Loughlin, L.M.; Knowles, S.; Twomey, A.; Murphy, J.F.A. The Neonatal Early Onset Sepsis Calculator; in Clinical Practice. Ir. Med. J. 2020, 113, 57. [Google Scholar] [PubMed]
29. Leonardi, B.M.; Binder, M.; Griswold, K.J.; Yalcinkaya, G.F.; Walsh, M.C. Utilization of a Neonatal Early-Onset Sepsis Calculator to Guide Initial Newborn Management. Pediatr. Qual Saf. 2019, 4, e214. [Google Scholar] [CrossRef]
30. Bridges, M.; Pesek, E.; McRae, M.; Chabra, S. Use of an Early Onset-Sepsis Calculator to Decrease Unnecessary NICU Admissions and Increase Exclusive Breastfeeding. J. Obstet. Gynecol. Neonatal. Nurs. 2019, 48, 372–382. [Google Scholar] [CrossRef]
31. Helmbrecht, A.R.; Marfurt, S.; Chaaban, H. Systematic Review of the Effectiveness of the Neonatal Early-Onset Sepsis Calculator. J. Perinat. Neonatal. Nurs. 2019, 33, 82–88. [Google Scholar] [CrossRef]
32. Strunk, T.; Buchiboyina, A.; Sharp, M.; Nathan, E.; Doherty, D.; Patole, S. Implementation of the Neonatal Sepsis Calculator in an Australian Tertiary Perinatal Centre. Neonatology 2018, 113, 379–382. [Google Scholar] [CrossRef] [PubMed]
33. Beavers, J.B.; Bai, S.; Perry, J.; Simpson, J.; Peeples, S. Implementation and Evaluation of the Early-Onset Sepsis Risk Calculator in a High-Risk University Nursery. Clin. Pediatr. 2018, 57, 1080–1085. [Google Scholar] [CrossRef] [PubMed]
34. Pettinger, K.J.; Mayers, K.; McKechnie, L.; Phillips, B. Sensitivity of the Kaiser Permanente early-onset sepsis calculator: A systematic review and meta-analysis. EClinicalMedicine 2019, 19, 100227. [Google Scholar] [CrossRef] [PubMed]
35. Kim, M.J. Utility of neonatal early-onset sepsis calculator in risk-based group B Streptococcus screening approach. Clin. Exp. Pediatr. 2020, 63, 393. [Google Scholar] [CrossRef] [PubMed]
36. Achten, N.B.; Dorigo-Zetsma, J.W.; van Rossum, A.M.C.; Oostenbrink, R.; Plötz, F.B. Risk-based maternal group B Streptococcus screening strategy is compatible with the implementation of neonatal early-onset sepsis calculator. Clin. Exp. Pediatr. 2020, 63, 406–410. [Google Scholar] [CrossRef] [PubMed]
37. Saw, C.; Kulasekaran, K.; Fernando, D.T.; Tan, D.; Dharmapuri, R.; Bulsara, M.K.; Friesen, N.D. Retrospective cohort study of neonatal early onset of sepsis and the role of the EOS calculator in a level II nursery. Pediatr. Neonatol. 2021, 62, 512–521. [Google Scholar] [CrossRef] [PubMed]
38. Riskin, A.; Bryskin, S.; Zaitoon, H.; Toropine, A.; Iofe, A.; Zoabi-Safadi, R.; Bader, D. Evaluation of Implementation of Early-Onset Sepsis Calculator in Newborns in Israel. J. Pediatr. 2021, 234, 71–76.e2. [Google Scholar] [CrossRef] [PubMed]
39. Kimpton, J.A.; Verma, A.; Thakkar, D.; Teoh, S.; Verma, A.; Piyasena, C.; Battersby, C. Comparison of NICE Guideline CG149 and the Sepsis Risk Calculator for the Management of Early-Onset Sepsis on the Postnatal Ward. Neonatology 2021, 118, 562–568. [Google Scholar] [CrossRef]
40. Cavazos, R.; Patil, M.; Gautham, K.S. A sepsis risk calculator can decrease antibiotic exposure in neonatal early-onset sepsis screening. Acta Paediatr. 2020, 109, 2166–2167. [Google Scholar] [CrossRef]
41. Achten, N.B.; Visser, D.H.; Tromp, E.; Groot, W.; van Goudoever, J.B.; Plötz, F.B. Early onset sepsis calculator implementation is associated with reduced healthcare utilization and financial costs in late preterm and term newborns. Eur. J. Pediatr. 2020, 179, 727–734. [Google Scholar] [CrossRef]
42. Stipelman, C.H.; Smith, E.R.; Diaz-Ochu, M.; Spackman, J.; Stoddard, G.; Kawamoto, K.; Shakib, J.H. Early-Onset Sepsis Risk Calculator Integration Into an Electronic Health Record in the Nursery. Pediatrics 2019, 144, e20183464. [Google Scholar] [CrossRef] [PubMed]
43. Qualitative Data—The Consolidated Framework for Implementation Research. Available online: https://cfirguide.org/evaluation-design/qualitative-data/ (accessed on 30 October 2022).
44. Zavalkoff, S.R.; Razack, S.I.; Lavoie, J.; Dancea, A.B. Handover after pediatric heart surgery: A simple tool improves information exchange*. Pediatr. Crit. Care Med. 2011, 12, 309–313. [Google Scholar] [CrossRef] [PubMed]
45. Karakaya, A.; Moerman, A.T.; Peperstraete, H.; François, K.; Wouters, P.F.; de Hert, S.G. Implementation of a structured information transfer checklist improves postoperative data transfer after congenital cardiac surgery. Eur. J. Anaesthesiol. 2013, 30, 764–769. [Google Scholar] [CrossRef] [PubMed]
46. U.S. Human Services; National Institutes Health; National Cancer Institute. Theory at a Glance A Guide for Health Promotion Practice, 2nd ed.; Createspace Independent Publishing Platform: Scotts Valley, CA, USA, 2012.
47. Greenhalgh, T.; Robert, G.; Macfarlane, F.; Bate, P.; Kyriakidou, O. Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. Milbank Q. 2004, 82, 581–629. [Google Scholar] [CrossRef]
48. Grol, R.; Wensing, M. Implementatie: Effectieve Verandering in de Patiëntenzorg; Elsevier Gezondheidszorg: Maarssen, The Netherlands, 2001. [Google Scholar]
49. Peters, S.; Sukumar, K.; Blanchard, S.; Ramasamy, A.; Malinowski, J.; Ginex, P.; Senerth, E.; Corremans, M.; Munn, Z.; Kredo, T.; et al. Trends in guideline implementation: An updated scoping review. Implement. Sci. 2022, 17, 1–17. [Google Scholar] [CrossRef] [PubMed]
50. Soong, C.; Shojania, K.G. Education as a low-value improvement intervention: Often necessary but rarely sufficient. BMJ Qual. Saf. 2020, 29, 353–357. [Google Scholar] [CrossRef] [PubMed]
51. Edmondson, A.C. Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy; Jossey-Bass Inc. Publisher: San Francisco, CA, USA, 2012. [Google Scholar]
52. Carey, R.N.; Connell, L.E.; Johnston, M.; Rothman, A.J.; de Bruin, M.; Kelly, M.P.; Michie, S. Behavior Change Techniques and Their Mechanisms of Action: A Synthesis of Links Described in Published Intervention Literature. Ann. Behav. Med. 2019, 53, 693–707. [Google Scholar] [CrossRef] [PubMed]
53. Dy, S.M.; Ashok, M.; Wines, R.C.; Smith, L.R. A Framework to Guide Implementation Research for Care Transitions Interventions. J. Health Qual. 2015, 37, 41–54. [Google Scholar] [CrossRef]
54. Brown, B.; Gude, W.T.; Blakeman, T.; van der Veer, S.N.; Ivers, N.; Francis, J.J.; Lorencatto, F.; Presseau, J.; Peek, N.; Daker-White, G. Clinical Performance Feedback Intervention Theory (CP-FIT): A new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement. Sci. 2019, 14, 1–25. [Google Scholar] [CrossRef]
55. Kawamoto, K.; Houlihan, C.A.; Balas, E.A.; Lobach, D.F. Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ 2005, 330, 765. [Google Scholar] [CrossRef]
56. van Wyk, J.T.; van Wijk, M.A.; Sturkenboom, M.C.; Mosseveld, M.; Moorman, P.W.; van der Lei, J. Electronic Alerts Versus On-Demand Decision Support to Improve Dyslipidemia Treatment. Circulation 2008, 117, 371–378. [Google Scholar] [CrossRef] [PubMed]
57. Lukas, C.V.; Holmes, S.K.; Cohen, A.B.; Restuccia, J.; Cramer, I.E.; Shwartz, M.; Charns, M.P. Transformational change in health care systems. Health Care Manag. Rev. 2007, 32, 309–320. [Google Scholar] [CrossRef] [PubMed]
58. Gustafson, D.H.; Sainfort, F.; Eichler, M.; Adams, L.; Bisognano, M.; Steudel, H. Developing and Testing a Model to Predict Outcomes of Organizational Change. Health Serv. Res. 2003, 38, 751–776. [Google Scholar] [CrossRef] [PubMed]
59. Eccles, M.; Grimshaw, J.; Walker, A.; Johnston, M.; Pitts, N. Changing the behavior of healthcare professionals: The use of theory in promoting the uptake of research findings. J. Clin. Epidemiol. 2005, 58, 107–112. [Google Scholar] [CrossRef] [PubMed]
60. Grol, R.; Buchan, H. Clinical guidelines: What can we do to increase their use? Med. J. Aust. 2006, 185, 301–302. [Google Scholar] [CrossRef] [PubMed]
61. Damschroder, L.J. Clarity out of chaos: Use of theory in implementation research. Psychiatry Res. 2020, 283, 112461. [Google Scholar] [CrossRef] [PubMed]
62. Damschroder, L.J.; Reardon, C.M.; Widerquist, M.A.O.; Lowery, J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement. Sci. 2022, 17, 1–16. [Google Scholar] [CrossRef] [PubMed]
63. Sammut, R.; Griscti, O.; Norman, I.J. Strategies to improve response rates to web surveys: A literature review. Int. J. Nurs. Stud. 2021, 123, 104058. [Google Scholar] [CrossRef]
64. Cook, J.V.; Dickinson, H.O.; Eccles, M.P. Response rates in postal surveys of healthcare professionals between 1996 and 2005: An observational study. BMC Health Serv. Res. 2009, 9, 160. [Google Scholar] [CrossRef]
65. Badger, F.; Werrett, J. Room for improvement? Reporting response rates and recruitment in nursing research in the past decade. J. Adv. Nurs. 2005, 51, 502–510. [Google Scholar] [CrossRef]
66. Timmins, F.; Ottonello, G.; Napolitano, F.; Musio, M.E.; Calzolari, M.; Gammone, M.; Catania, G.; Zanini, M.; Aleo, G.; Sasso, L.; et al. The state of the science—The impact of declining response rates by nurses in nursing research projects. J. Clin. Nurs. 2023, 32, E9–E11. [Google Scholar] [CrossRef]