Evidence-based Ultrasound Education? – A Systematic Literature Review of Undergraduate Ultrasound Training Studies

Abstract

Despite a large number of studies previously conducted on educational concepts of ultrasound teaching in medical school, various controversial issues remain. Currently, recommendations for ultrasound curricula that aim to standardize ultrasound teaching are frequently based on expert panel advice instead of relying on valid evidence-based literature. In December 2022, a systematic literature review on the subject of undergraduate ultrasound education was conducted. All ultrasound studies listed in the PubMed and Google Scholar medical databases were filtered and analyzed with respect to various aspects of their methodological conduct and curricular implementation. A total of 259 publications were considered in the data synthesis, including 145 teaching studies in the field of undergraduate ultrasound education. The latter encompassed 58 (40%) studies that did not compare their ultrasound training to a control group. Furthermore, 84 (58%) of these studies did not assess knowledge prior to the applied ultrasound training, neglecting this factor’s potential influence on study outcomes. Despite a great interest in the development and further implementation of ultrasound education during medical school, this process is still compromised by significant deficiencies in studies that have been conducted in the past. In order to provide a valid basis for curricular decisions, teaching studies should fulfill essential methodological requirements despite the multifactorial framework in which they are conducted. In the future, a guide for the design of ultrasound studies could be a useful aid for ultrasound enthusiasts and promote scientific knowledge gain.

Keywords

methods & techniques - ultrasound - themes - education - medical student

Introduction

The development of the concept of point-of-care ultrasound (POCUS) has contributed immensely to ultrasound becoming a fundamental diagnostic tool for many disciplines. As a radiation-free and noninvasive modality, ultrasound meets the ideal conditions to provide medical students with the opportunity to practice a commonly used clinical imaging tool during medical school. The growing interest in ultrasound training during medical school is reflected by the increasing number of scientific publications on undergraduate ultrasound education throughout the last 20 years [1]. Numerous studies with different training approaches investigating teaching contents, the use of different ultrasound devices, and didactic tools have been conducted and published around the world. With increasing training opportunities in ultrasound education at both undergraduate and postgraduate levels, the growing number of program concepts is very heterogeneous in terms of curricular structure, content, and didactic approaches [2] [3] [4]. In order to achieve a broad impact through ultrasound training and to raise the minimum level of competency for students at the end of their studies, the standardization of ultrasound training using structured curricula could resolve current major discrepancies [5]. With regard to this objective, several recommendations have been published by international expert panels and ultrasound societies. Both the European and World Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB, WFUMB) recommend the vertical implementation of ultrasound in medical studies, starting as an integrated component in preclinical basic sciences and increasingly including clinical aspects from the very beginning until the end of medical school [5] [6]. In order to contribute to a more consistent ultrasound education, both societies published instructions on how to develop, establish, and evaluate an ultrasound curriculum. By founding a student committee, organizing student congresses during the annual EUROSON congresses, and providing free online resources for student ultrasound training, the EFSUMB created crucial resources that local universities and educational institutions can draw on [6]. As also stated in the consensus conference recommendations of the Society of Ultrasound in Medical Education (SUSME) and the World Interactive Network Focused on Critical Ultrasound (WINFOCUS), it is noticeable that most recommendations regarding ultrasound education are solely based on consensus decisions by expert panels, since - despite the high volume of publications - only a few studies have used a sufficient methodology to draw valid conclusions [1]. Altogether, it seems paradoxical that in a highly dynamic field in terms of conducted and published studies, recommendations still rely on eminence rather than evidence. Consequently, it is necessary to investigate which methodological requirements teaching studies should fulfill and which of these are currently already implemented.

Objectives

In assessing the quality of the study implementation, it is important to analyze which study design was chosen and whether a control group would be necessary. In addition, the curricular framework into which ultrasound training is embedded (such as elective, mandatory, or extracurricular courses) should be considered as the context of the teaching situation. As the final evaluation of the ultrasound course’s teaching intervention, an appropriate assessment is important to measure the extent to which the objectives of the ultrasound course were met [7].

Aiming to provide an overview of the methodological quality of current studies on ultrasound education in the following systematic literature review and subsequent analysis, the PICOS scheme was used to define characteristics that publications should meet for inclusion in the literature selection [8] ([Table 1]).

Table 1 Study characteristics for included studies according to PICOS.

Participants	Medical students
Intervention	Ultrasound education
Comparator	Different approaches and ideas
Outcomes	Didactic approaches, integration of ultrasound curricula in preclinical and clinical years, assessment and evaluation methods, assessment of long-term retention
Study design	Original studies, reports on prospective and retrospective studies, interventional studies, observational studies, and cross-sectional studies

Materials and Methods

Search strategy

The medical databases PubMed and Google Scholar were searched for relevant publications on undergraduate ultrasound education during December 2022. All ultrasound studies published by December 18, 2022 were taken into account.

The search was filtered for relevant results using different combinations of the keywords [ultrasound] plus [undergraduate], [didactics], [teaching], [curriculum], [medical student] and [education]. First, publications were selected which, according to their title, promised to contain information about the relevant topic. Then, those publications’ abstracts were reviewed by two authors (R.N. and F.R.) and checked for compliance with the inclusion criteria. Any discrepancies between the authors regarding inclusion were resolved by direct discussion. After removal of duplicates, full-text versions of the initially included publications were obtained, read, and analyzed for data extraction. In the course of the systematic literature search, we followed the updated PRISMA statement for reporting systematic reviews and meta-analyses of studies [8] [9].

Inclusion and exclusion criteria

All publications with content related to ultrasound education of medical students, including ultrasound studies or reports of conducted studies, reports of ultrasound curricula, surveys, recommendations and guidelines, systematic and non-systematic literature reviews, and statements in the form of editorials or letters to the editor were included. In this way, we strived to get the best possible impression of the current situation, research questions, and potential issues of ultrasound education studies. Due to different educational structures, we limited this literature review to medical students. Unfortunately, we had to exclude a few publications due to the lack of accessibility of their full text versions despite contacting the authors. Furthermore, we could only consider literature entries written in English or German language.

Data extraction and analysis

All included studies that examined ultrasound teaching interventions were analyzed for their study characteristics using the PICOS scheme, as well as further parameters which were related to the course structure and curricular integration of the ultrasound intervention. For this purpose, the ultrasound training’s timing (preclinical or clinical) and form (mandatory, elective, or extracurricular) of curricular integration were examined. Furthermore, the type of instructors that were involved was investigated, differentiating between postgraduate faculty, near-peer and peer tutors. Finally, it was checked whether any form of assessment quantifying and evaluating the effect achieved by the ultrasound training was applied, both shortly after the intervention and with regards to long-term maintenance of the acquired skills. As an easily applied taxonomy, Kirkpatrick’s Four Levels [10] for evaluation of training was used to provide a structured overview despite the wide variety of methods for measuring training effects. In his work, Kirkpatrick has defined four different assessment levels in training program evaluation, which reflect increasing training program evaluation quality as the level increases (see [Fig. 1]). The first stage (as the lowest level) involves recording the participants’ reaction to the training program. In this regard, the general satisfaction of the participants with the planning and implementation of the program can be surveyed. Programs reach the next evaluation level when verifying the extent to which the training has achieved learning success among participants in accordance with the learning objectives previously set. Even though incorporation of a pre-assessment and a control group are parts of Kirkpatrick’s recommendations, it was not included as a mandatory requirement in this level assignment following Kirkpatrick’s levels. Nonetheless, in order to take these differences into account, these aspects were still examined individually. Kirkpatrick Level 3 is devoted to potential changes in student behavior or associated skills. Regarding student ultrasound teaching, studies might examine an ultrasound course’s effect on transfer skills such as understanding of human anatomy and physical examination [11]. At the highest level, the results of a training program are examined on a larger scale, e. g., by investigating the long-term effectiveness or the overarching question regarding the extent to which a certain ultrasound teaching approach does in fact benefit the students’ future capabilities of clinical ultrasound application and thereby improve patient care. Other factors, such as potential costs of the program in terms of funding, time, personnel resources should also be considered.

Fig. 1 Literature selection process.

All statistical analyses (descriptive statistics) were performed in the factory version of Microsoft Excel for Mac (Version 16.77.1. Microsoft Corporation, Redmond, Washington, USA) without further add-ins.

Results

Search results

The literature search revealed 3356 records; 690 of these were relevant as identified by title. After further exclusion of duplicates (n=379), records with inappropriate abstract content (n=16), publications with no full text version available (n=14), and records that turned out to be duplicates or non-compliant with the inclusion criteria after full text review (n=22), a total of 259 publications dedicated to ultrasound education of medical students resulted from the overall literature search (see [Fig. 2]). While 115 out of 259 publications did not feature original study data but recommendations, guidelines, surveys, and more (see [Fig. 3]), 144 literature entries of those 259 represented 145 original studies (one publication included two studies conducted by Cawthorn et al. [12]). The original studies incorporated a huge variety of different course concepts examined regarding different didactic approaches, of discussed ultrasound examinations and pathologies, of professional levels of the course instructors involved, and several curricular formats of ultrasound training programs. An overview of all studies and the collected parameters can be found in supplementary tab. 1.

Fig. 2 Kirkpatrick’s Four Levels for Evaluating Training Programs.

Fig. 3 Number of included studies by type of publication.

Study design and control of the educational intervention

Analysis of the included studies revealed that 92 (64%) of them did not clearly specify their study type for publication (see [Fig. 4]). As such we considered publications that either did not address their study design at all or were incomplete (by e. g., only stating that the study was prospectively conducted, a proof-of-concept, pilot, or feasibility study). [Fig. 4] illustrates the distribution of the different study types used. All studies classified as such or as a subgroup including case control, cross-sectional, and prospective cohort studies fell under the category “prospective observational studies” [13]. The category “other prospective interventional studies” included those that declared themselves as such or as quasi-experimental studies. As most informed readers might also be capable of inferring the study design from the process description (e. g., whether a study was conducted prospectively or retrospectively), these inferences were also included in the first sub-column of the characteristic “Study type” within the list of studies (see supplementary tab. 1), regardless of whether the study type was explicitly stated in the respective publication or not.

Fig. 4 Distribution of different study designs if indicated. Most studies did not sufficiently specify their formal study format, possibly due to difficulties of transferring the educational setting into a rigid framework.

By choosing a certain type of study, investigators also determine whether the investigated intervention is to be compared with a (non-interventional or interventional) control. Among the studies we examined, 58 (40%) compared their intervention with a control group (see [Fig. 5]).

Fig. 5 Distribution of studies regarding the inclusion of a control group.

Curricular integration – mandatory, elective, or extracurricular course?

With a total of 70 out of 145 (48%) investigations, the highest proportion of studies examined ultrasound training within mandatory courses (although it is important to distinguish between the mandatory course completion setting and the voluntary participation in the data collection of the study). This was followed by studies offering ultrasound training as electives (24%) or extracurricular courses (5%), as a facultative option for students to deepen their individual interests. Among the studies reviewed, 32 (22%) studies did not report how ultrasound training was embedded in the students’ curriculum (see [Fig. 6]).

Fig. 6 Form of curricular integration of ultrasound training. As the setting of the educational intervention influences its outcome based on different levels of motivation, studies should clarify the curricular framework.

Evaluation of educational approaches and the assessment of their effects

Two of the reviewed studies did not report any evaluation of their ultrasound training outcome. 23 (16%) studies performed an evaluation regarding the students´ satisfaction with the delivery and approach of ultrasound training alone. Seven (5%) studies conducted a measurement of potentially improved secondary skills in reaction to the ultrasound training. Most studies (78%) examined their intervention’s effect with regard to acquired knowledge, skills, and/or changes in students’ personal attitudes towards ultrasound. For this purpose, in addition to self-assessment queries, direct feedback by the instructors, written tests such as multiple-choice questionnaires, and practical tests, e. g., in the form of Objective Structured Clinical Examinations (OSCE), were performed. Furthermore, in some studies a quality review of acquired ultrasound images, for example according to the Brightness mode quality ultrasound imaging examination technique (B-QUIET), was part of the assessment. The number of studies that used each measurement tool is shown in [Fig. 7]. [Fig. 8] illustrates the training evaluation method-dependent quality level of all studies according to Kirkpatrick’s Four Levels. 19 studies investigated the maintenance of skills after a predefined time interval (study-specific time intervals are listed in supplementary tab. 1).

Fig. 7 Assessment time and modality in ultrasound education (total number may exceed 145, as some training programs also used a combination of different methods). Although ultrasound is a competence taught on a multidisciplinary basis with a flat learning curve, preexisting skills and their long-term retention were not tested. It is remarkable that despite ultrasound being a practical skill, theoretical assessment techniques are used more frequently.

Fig. 8 What Kirkpatrick level was reached by the ultrasound studies through assessment and evaluation?

Out of a total of 61 (42%) studies that conducted some form of pre-interventional testing, it was based on self-assessing preexisting knowledge in seven (11%) studies. At 13%, a comparable proportion of the post-interventional assessments conducted were also exclusively based on subjective parameters like self-perceived skill improvement (see [Fig. 9]).

Fig. 9 Objectivity of assessment modalities. Purely subjective test procedures can be positively biased. At the same time, a realistic self-evaluation of individual expertise and its limitations is a core competence of a physician and should also be strengthened in ultrasound studies.

Discussion

This systematic literature review analyzed ultrasound training studies published by December 18, 2022 and listed in PubMed or Google Scholar with the aim of identifying strengths and weaknesses in the design and methodology of current ultrasound education studies.

While ultrasound enthusiasts and medical educators intensively deal with the question of how medical teaching can be best examined and improved on the basis of the most valid scientific evidence possible, there is still fundamental controversy regarding whether and how didactic interventions can be investigated and improved by the outcomes of medical study formats: To what extent can study formats – as we know them from medical research – accurately represent the multifactorial nature of student learning [14] [15] [16]. A common way to determine the benefit of a teaching program is measuring its effectiveness with respect to achieving predefined learning objectives by assuming a linear correlation between the training as a cause and the acquired skills as a subsequent effect [15]. However, ultrasound education takes place in a complex framework of social structures embedded in a diverse curriculum. Furthermore, each student´s learning process is influenced by individual experiences, interests, and learning style [17]. The high number of studies in our analysis that did not define and state their study design within their publication shows this difficulty of compressing ultrasound courses into conventional study types. Nevertheless, defining the study design is an important step in study conceptualization to set the framework for an accurate methodological approach and for recipients in order to properly evaluate the validity of its results [18]. After all, it is feasible to achieve a pragmatic compromise between the claims of evidence-generating, stringent research and real-world teaching, particularly by eliminating common confounders via appropriate measures. Especially in elective settings, the randomization of students into an experimental and a control group as well as the repeated investigation of ultrasound training with different student cohorts and subsequent subgroup analyses could reduce the impact of potential misleading influences. Within all studies examined by this systematic literature review, only 87 studies (60%) involved a control group, stating that in the remaining 40% there is no certainty that the learning outcome is caused by the examined teaching intervention and did not benefit from parallel courses or previous skills.

Furthermore, regarding the transferability of educational concepts to other teaching sites, the context in which the original study was conducted plays an important role [19]. A simple and applicable means of understanding the context in which ultrasound training took place in medical school is the type of curricular integration (which may include curricular integration as part of the mandatory curriculum, as an elective or extracurricular event). Especially in a time-dense setting such as medical school, student motivation is higher for self-selected study content than for mandatory courses in the curriculum. Previous publications have underlined the significant influence of the form of implementation on student motivation [20] which subsequently has a strong impact on their individual learning outcome. Thus, the results of studies conducted as part of electives or voluntary extracurricular events are expected to be more positive than those that took place in the ordinary course of study [20]. Unfortunately, the curricular integration background was not reported by more than a fifth of the examined studies.

Assessment of the learning outcome is an important tool of evaluation to determine the effectiveness of applied didactic methods. Therefore, the choice of appropriate assessment tools suitable for the previously set learning objectives and instructional methods is crucial [7]. Particularly in ultrasound, which is used and taught within many disciplines in a variety of ways and where students’ prior experience levels often differ greatly, it is essential to differentiate between preexisting and newly acquired ultrasound skills. In our analysis, less than half of all studies applied a baseline assessment prior to the ultrasound course (see [Fig. 9]). Out of these, seven (5%) studies exclusively relied on students’ self-assessment and did not include any objective examination of prior knowledge. A further 16 (11%) studies solely based their conclusions on the students’ subjectively self-perceived change in ultrasound skills in the post-interventional assessment. The bias susceptibility of this metric is evident. In a comparison of students’ self-perceived ultrasound skills with the level of skills objectively observed by instructors, Steinmetz et al. showed that, on average, students rated their ultrasound skills 68% better than they actually performed [21]. This may be influenced by the students’ own perceptions of how much time and learning effort they have personally invested during the course [7]. However, these investigations show how much the results of ultrasound training studies can be distorted by missing objective parameters [22]. Still, as ultrasound is a practical skill, whose quality of application and interpretation strongly influences patient care [23], it is important to assure certain levels of competence by the use of objective parameters when evaluating ultrasound training interventions. By categorizing all analyzed studies according to Kirkpatrick levels [10], we identified 23 teaching studies that evaluated their implementation using feedback forms only (see [Fig. 8]). In addition to self-assessment, collecting the students’ perspective in the form of an evaluation of planning and implementation offers significant opportunities for the further development of the training format, but should not be the only indicator for evaluating a training format. In accordance with this, the majority of studies additionally assessed the feasibility and effectiveness of their approach by measuring effect parameters through various metrics and assessment tools (see [Fig. 7]). One aspect that has been little studied to date is how previously taught ultrasound skills can be reinforced during the course of medical school and when refresher training would be useful to maintain them [24] [25]. In our analysis, 19 (13%) studies investigated the retention of knowledge and/or skills after different time intervals (see supplementary table 1). In curricular planning, this data is helpful to introduce and reinforce ultrasound training at the right time in the curriculum.

Without questioning the fact itself, the landscape of ultrasound education studies is repetitive in drawing the conclusion that ultrasound teaching is positively received by students and that hands-on teaching helps students to improve their ultrasound skills [24]. Consistent with the findings of Davis et al., in our analysis, the number of ultrasound studies that reports broader effects of ultrasound teaching on secondary skills and behaviors (Kirkpatrick Level 3), postgraduate residency, and clinical patient care (Kirkpatrick Level 4) represents a vanishing minority (see [Fig. 8]) [24] [26]. Since the clinical benefits of ultrasound are undeniable, it seems intuitive that early and longitudinal teaching as recommended by EFSUMB and WFSUMB will predominantly have positive effects. Nevertheless, the optimal format and way of curricular implementation has yet to be defined. Both societies (EFSUMB and WFSUMB) offer a broad range of resources in order to promote student ultrasound education, including student committees congresses, guidelines on how to best implement undergraduate ultrasound training, and free e-learning materials for courses and self-directed practice [5] [6] [27]. Furthermore, the EFSUMB even established a student committee led by medical students. Regarding ultrasound training studies, the Educational and Professional Standard Committee (EPSC) of the EFSUMB pursues the task of reviewing studies and reporting experiences in student ultrasound training [6]. All of these efforts are in line with the purpose of this systematic literature review that pursued the extraction of key learnings from all previous literature on ultrasound education to shape future research with the ultimate goal of optimizing student ultrasound training. The learnings from this systematic literature review culminate in the proposal of five basic principles which educators should consider when conducting ultrasound training studies, in order to avoid future unnecessary duplication of feasibility studies and to maximize the validity and informative value of study results:

1. Define the methodological approach of the study.

2. Use a control group to compare the effectiveness of the training intervention with the previous established way of teaching ultrasound.

3. Describe the educational setting and curricular integration of the training intervention into the existing curriculum.

4. Prefer structured, objective, and practical examination tools to assess the achievement of learning objectives over self-assessment and training implementation evaluation only.

5. Evaluate long-term effects of the ultrasound training.

Building on these five core principles, the development of guidelines for the design and conduction of ultrasound training studies could provide standardization and a foundation for higher-level scientific evidence to finally answer open questions in ultrasound education. Meta-analyses of those studies would then allow inference of trends that could provide information moving from a general "[…] ´what works´ towards what works for whom” [14] and under which circumstances [28], and thus be an important aid to educators worldwide in the development and implementation of optimal ultrasound teaching.

Limitations

This review has certain limitations. In general, when conducting a literature search, despite a careful review, there is a risk of missing relevant articles and thus not including them in the data synthesis. Certainly, this risk is compounded by the fact that only the databases PubMed and Google Scholar were searched. Considering the dynamic study situation, further teaching studies on the topic of undergraduate ultrasound training may have been published between the literature search and the publication of this paper. In the comparative analysis of the approaches to ultrasound teaching, it must also be noted that the studies were carried out at very different times and places. The curricular integration of ultrasound teaching has evolved greatly over the years and continues to do so globally. Depending on the local level of previous ultrasound training, the studies target different outcome parameters. The investigation of the didactic implementation of ultrasound itself clearly requires a different methodology than a study merely examining innovative teaching concepts of an already established learning tool. This limits the comparability of the studies with each other and makes it difficult to draw general conclusions. In addition, the various subspecialties of ultrasound applications can place very different demands on the learner. Depending on the type of ultrasound, certain didactic approaches may be more effective than others [29], making it necessary to differentiate the studies into subgroups for a qualitative review.

Conclusion

The review of current literature on undergraduate ultrasound education revealed several deficiencies in the methodological conduct of ultrasound training studies. In order to build valid evidence, studies should meet certain minimal standards that protect their results from being biased by confounding factors. From now on, rather than producing duplicate studies on the overall feasibility of ultrasound teaching, specific questions should be addressed that could add value for students, universities, and ultrasound societies, such as the timing and form of curricular integration of ultrasound teaching, how acquired skills can be maintained and reinforced, as well as the impact of ultrasound teaching on other skills, residency training, and patient care on a larger scale. This review aims to draw attention to the low methodological quality of many studies in undergraduate ultrasound education. Starting with five fundamental principles, it aims to build a foundation for further recommendations on the conduction of educational studies regarding the acquisition of practical skills in medical school.

Conflict of Intrest

The authors declare that they have no conflict of interest.

• References

• 1 Hoppmann RA, Mladenovic J, Melniker L, Badea R, Blaivas M, Montorfano M. et al. International consensus conference recommendations on ultrasound education for undergraduate medical students. Ultrasound J 2022; 14: 31
• 2 Nicholas E, Ly AA, Prince AM, Klawitter PF, Gaskin K, Prince LA. The current status of ultrasound education in United States medical schools. J Ultrasound Med 2021; 40: 2459-2465
• 3 Welle R, Seufferlein T, Kratzer W. [Current state of under- and postgraduate education in abdominal ultrasonography at German university hospitals. A panel study over 20 years]. Z Gastroenterol 2021; 59: 225-240
• 4 Russell FM, Zakeri B, Herbert A. et al. The State of Point-of-Care Ultrasound Training in Undergraduate Medical Education: Findings From a National Survey. Acad Med 2022; 97: 723-727
• 5 Dietrich CF, Hoffmann B, Abramowicz J. et al. Medical Student Ultrasound Education: A WFUMB Position Paper, Part I. Ultrasound Med Biol 2019; 45: 271-281
• 6 Cantisani V, Dietrich CF, Badea R. et al. EFSUMB Statement on Medical Student Education in Ultrasound [long version]. Ultrasound Int Open 2016; 2: E2-E7
• 7 Höhne E, Recker F, Dietrich CF, Schäfer VS. Assessment Methods in Medical Ultrasound Education. Front Med (Lausanne) 2022; 9: 871957
• 8 Liberati A, Altman DG, Tetzlaff J. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009; 339: b2700
• 9 Page MJ, McKenzie JE, Bossuyt PM. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372: n71
• 10 Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. 3rd ed. San Francisco, CA: Berrett-Koehler; 2006: 372 p.
• 11 Tarique U, Tang B, Singh M, Kulasegaram KM, Ailon J. Ultrasound Curricula in Undergraduate Medical Education: A Scoping Review. J Ultrasound Med 2018; 37: 69-82
• 12 Cawthorn TR, Nickel C, O’Reilly M. et al. Development and evaluation of methodologies for teaching focused cardiac ultrasound skills to medical students. J Am Soc Echocardiogr 2014; 27: 302-309
• 13 Thiese MS. Observational and interventional study design types; an overview. Biochem Med 2014; 24: 199-210
• 14 Connolly P, Keenan C, Urbanska K. The trials of evidence-based practice in education: a systematic review of randomised controlled trials in education research 1980–2016. Educational Research 2018; 60: 276-291
• 15 Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher 2012; 34: e288-e299
• 16 Biesta G. WHY “WHAT WORKS” WON’T WORK: EVIDENCE-BASED PRACTICE AND THE DEMOCRATIC DEFICIT IN EDUCATIONAL RESEARCH. Educational Theory 2007; 57: 1-22
• 17 Cremerius C, Gradl-Dietsch G, Beeres FJP. et al. Team-based learning for teaching musculoskeletal ultrasound skills: a prospective randomised trial. Eur J Trauma Emerg Surg 2021; 47: 1189-1199
• 18 Röhrig B, Prel JBD, Blettner M. Study Design in Medical Research. Deutsches Ärzteblatt international [Internet]. 2009 [cited 2023 Sep 10]; Available from: https://www.aerzteblatt.de/10.3238/arztebl.2009.0184
• 19 Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Social Science & Medicine 2018; 210: 2-21
• 20 Darby JA. The effects of the elective or required status of courses on student evaluations. Journal of Vocational Education & Training 2006; 58: 19-29
• 21 Steinmetz P, Oleskevich S, Lewis J. Acquisition and Long-term Retention of Bedside Ultrasound Skills in First-Year Medical Students. J Ultrasound Med 2016; 35: 1967-1975
• 22 Birrane J, Misran H, Creaney M, Shorten G, Nix CM. A Scoping Review of Ultrasound Teaching in Undergraduate Medical Education. MedSciEduc 2018; 28: 45-56
• 23 Adhikari S, Stolz L, Amini R, Blaivas M. Impact of point-of-care ultrasound on quality of care in clinical practice. RMI 2014; 7: 81-93
• 24 Davis JJ, Wessner CE, Potts J. et al. Ultrasonography in Undergraduate Medical Education: A Systematic Review. J Ultrasound Med 2018; 37: 2667-2679
• 25 Menegozzo CAM, Cazolari PG, Novo FDCF. et al. Prospective Analysis of Short- and Mid-term Knowledge Retention after a Brief Ultrasound Course for Undergraduate Medical Students. Clinics (Sao Paulo) 2016; 74: e1087
• 26 Feilchenfeld Z, Dornan T, Whitehead C, Kuper A. Ultrasound in undergraduate medical education: a systematic and critical review. Med Educ 2017; 51: 366-378
• 27 Hoffmann B, Blaivas M, Abramowicz J. et al. Medical Student Ultrasound Education, a WFUMB Position Paper, Part II. A consensus statement of ultrasound societies. Med Ultrason 2020; 22: 220-229
• 28 Dekker I, Meeter M. Evidence-based education: Objections and future directions. Front Educ 2022; 7: 941410
• 29 Rappaport CA, McConomy BC, Arnold NR. et al. A Prospective Analysis of Motor and Cognitive Skill Retention in Novice Learners of Point of Care Ultrasound. Crit Care Med 2019; 47: e948-e952

Publication History

Received: 10 October 2023

Accepted after revision: 15 February 2024

Article published online:
26 March 2024