Validation of the Simulation Effectiveness Tool – Modified scale in Poland

Introduction

Simulation education in medicine revolutionized the education of all medical professions, including nursing. It enabled students to acquire practical knowledge in controlled conditions. Thanks to realistic simulation scenarios, students have the opportunity to develop clinical skills, make decisions in difficult situations and gain self-confidence before starting clinical practice [1]. In Poland, simulation education is a relatively new didactic approach that is becoming increasingly common in curricula. To assess the effectiveness of this type of education, appropriate assessment tools are necessary, such as the Simulation Effectiveness Tool – Modified (SET-M) scale, which was designed to measure the effectiveness of simulation in the context of acquisition of practical skills by students of medical fields such as nursing, medicine, midwifery or emergency medical services [2].
Validation of the SET-M scale is a key step to ensure its reliability and validity in the Polish educational context. This process included several stages, consistent with international standards for the adaptation of psychometric tools, such as the World Health Organization (WHO) guidelines and best practices described in the medical education literature [3]. First, the scale was translated into Polish by experienced translators who took into account the specificity of medical terminology and cultural context [4]. After the translation stage, a back-translation process was carried out, which involved the Polish version being translated back into English by an independent translator for whom English was the native language, which allowed for the comparison of both versions and the identification of potential inconsistencies [4, 5].
The aim of this study was to validate the SET-M scale to confirm its usefulness as a tool for assessing the effectiveness of simulation education among nursing students and potentially other health professions. The scale validation aims to determine whether the SET-M reliably and accurately measures students’ perceived effectiveness of educational simulations, taking into account various aspects of their learning, including clinical decision-making skills, emergency response, and teamwork skills. The study will include an analysis of both subjective and objective assessment of students’ skills, which will allow for a comprehensive verification of the scale’s effectiveness. Subjective assessment involves students’ self-evaluation, while objective measures include task accuracy and speed in simulations. The results of the study will provide valuable information on the usefulness of the SET-M scale in the Polish educational context and indicate possible areas that may require adaptation of the scale to local conditions.

Material and methods

In the process of validating the SET-M scale, it was planned to include 300 nursing students in the study. After obtaining the consent of the bioethics committee of the Collegium Medicum in Bydgoszcz, 256 participants took part in the study (85.3% of the originally planned number). The exclusion of 44 people resulted from meeting the inclusion and exclusion criteria, ensuring the homogeneity and credibility of the study group (Table 1).
In the validation study of the SET-M scale, the following inclusion and exclusion criteria were established to ensure the reliability and validity of the results and the homogeneity of the study group.
Inclusion criteria: nursing students from the Nicolaus Copernicus University in Toruń, participation in high-fidelity simulation sessions, age 18–35, informed consent to participate in the study. Exclusion criteria: no experience with simulation, health problems limiting participation, studies in fields unrelated to nursing (i.e., students enrolled in non-medical fields such as disciplines outside nursing, medicine, midwifery, or emergency medical services, whose educational goals differ from the clinical focus of medical simulations).
As a result, a representative sample of nursing students was obtained, which increased the credibility of the results. The study included 256 students, mostly women, with an average age of 22.3 years. All of them participated in simulation sessions during which their competences in realistic clinical scenarios were assessed using the SET-M scale. The results enabled an assessment of the effectiveness of simulation education in Polish educational conditions.

Validation process

The SET-M scale was validated according to international standards, adapted to the Polish medical education context. Due to the unification of didactic methods in the education of medical professions, there was no need to modify the structure of the tool. The validation process included three key stages [5, 6]:
1. Translation equivalence. The scale was translated from English into Polish using the back-translation method, which involved independent translation into Polish and re-translation into English by another person. Comparison of the English versions (original and back) made it possible to ensure that the content of the instrument retained its original meaning, eliminating potential semantic distortions [5, 7, 8].
2. Functional equivalence medical simulation experts assessed the scale for its usability in Polish educational settings. The language adequacy and compliance of the tool with local standards and expectations regarding simulation teaching methods were verified [9].
3. Psychometric equivalence, internal consistency (Cronbach’s a = 0.88) and construct validity were analyzed, confirming that the Polish version of the scale retains the structure of the original. Correlation analysis showed strong correlations between the scale items, which indicates its reliability and precision in assessing the effectiveness of medical simulations [7, 10].
Validation in accordance with WHO guidelines and medical education literature confirmed that the Polish version of SET-M is both a reliable tool for assessing medical simulations and comparable to the original version. This enables international comparative studies and supports the development of educational methodologies [7, 11].

Results

The SET-M scale demonstrated high reliability, with Cronbach’s a = 0.88, which exceeds international psychometric standards (acceptable: > 0.70, good: > 0.80) [12, 13]. This reliability aligns with the results obtained in studies on the English version of the tool, confirming the effective adaptation of the scale to the Polish context [14].
Additionally, the analysis of individual scale items revealed strong correlations within subscales, confirming their structural coherence [15]. This coherence indicates that the scale items consistently measure the same construct, which strengthens the credibility of the results obtained using the tool [16]. High internal correlations are particularly significant in assessing medical simulations, ensuring consistency across subscales such as participant perception, simulation effectiveness, and educational potential [17].
However, a detailed analysis of individual questions showed that item no. 2 („pre-briefing had a positive effect on my knowledge”) and item no. 8 („I had the opportunity to practice my clinical decision-making skills”) were the least correlated with other items. This suggests that these items reflect the accuracy of the terms used in the original to a weaker degree.
Figure 1 presents SET-M validation results. The validation study involved a diverse group of students, primarily women. The average age of the group was 22.3 ±6.60 years. The standard deviation, constituting 29% of the mean, indicates moderate age variability. Men were slightly older (mean: 23.0 years) than women (mean: 22.2 years), but this difference was not statistically significant. The minimum age was similar for both genders (18 years), while the maximum age was higher among women (52 years). Participants were divided into two age groups: ≤ 20 years and > 20 years. Most participants (50.4%) were ≤ 20 years old.
The largest proportion of respondents resided in rural areas, while urban participants were evenly divided between smaller and larger cities (32.4% each). Additionally, 22 participants (8.6%) reported other medical education backgrounds, including 11 with emergency medical services qualifications, 3 with midwifery degrees, and 1 with a background in physiotherapy. Twenty-one respondents (8.2%) had prior education in nursing from medical high schools or colleges before entering university. Experience in nursing work before starting studies was reported by 26 participants (10.2%).
The general statistics are presented in Table 2.
Of all the items on the SET-M scale, the highest-rated ones were:
• item 19: “Debriefing enabled a constructive assessment of the simulation,”
• item 18: “Debriefing was an opportunity for self-reflection on my actions during the simulation,”
• item 2: “Prebriefing had a positive impact on my knowledge,” and
• item 15: “Debriefing had a positive impact on my knowledge.”
In contrast, the lowest-rated items were:
• item 4: “I have greater knowledge of pathophysiology,”
• item 7: “I understand the effects of drugs better,” and
• item 6: “I felt that I could make clinical decisions.”
These results support the hypothesis that medical simulations primarily facilitate skill acquisition through repetition rather than teaching clinical reasoning [12]. The standard deviation exceeded 14% of the mean value, indicating moderate variability in responses. Figure 2 presents the SET-M scale correlation matrix. Figure 3 presents mean scores on the SET-M scale.
No significant differences were observed in SET-M scores between genders. While men scored higher on most items, exceptions included items 2, 5, 7, 9, and 10. Among the 19 items, a statistically significant difference was found only for item 14: „I am more confident in using evidence-based medicine to provide nursing care.” Men scored nonsignificantly higher overall but had a broader range of scores.
Similarly, no significant differences were noted between age groups (≤ 20 years and > 20 years). Participants ≤ 20 years old scored higher on most items except for items 1, 6, 7, 8, 9, and 12.
Scores also varied slightly based on participants’ place of residence. Rural respondents scored higher on almost all items except items 7 and 11, where participants from cities up to 350,000 inhabitants scored the highest. Conversely, respondents from cities with over 350,000 inhabitants scored the lowest on most items except for items 2, 4, 5, 7, 10, 11, and 13. Significant differences were found for item 4 („I have greater knowledge of pathophysiology,” p = 0.017) and item 8 („I had the opportunity to practice my clinical decision-making skills,” p = 0.040), favoring rural participants.
The distribution of mean SET-M scores is presented in Figure 4.

Discussion

The aim of this study was to validate the Polish version of SET-M, focusing on assessing the reliability, psychometric consistency, and translation quality to ensure its usability in Polish educational programs. The adaptation process followed rigorous standards, ensuring functional equivalence and statistical integrity [11].
The findings confirm the successful adaptation of the scale, with reliability (Cronbach’s a = 0.88) consistent with the original version [14, 15]. High internal correlations indicate strong structural coherence, validating the scale’s ability to measure simulation effectiveness in a Polish context [16, 18].
This study has certain limitations. First, it did not assess the tool’s test-retest reliability, which could provide insights into measurement stability over time [18, 19]. Second, the sample comprised only Polish medical students, limiting generalizability to other populations. Expanding validation studies to include more diverse groups is recommended to explore broader applications [20–22].
In conclusion, the validated Polish SET-M scale is a reliable tool for assessing medical simulations in Poland, with high internal consistency and significant subscale correlations. Future studies should address its test-retest reliability and extend validation to more diverse groups. This will further establish SET-M as a key instrument in advancing medical education in Poland [23, 24].

Conclusions

The SET-M scale has proven to be a highly reliable tool for assessing the effectiveness of simulation education in nurse education, which is of great importance both for nursing practice and the entire process of educating medical staff in Poland. Its use provides valuable information on the practical skills of students, enabling both subjective and objective assessment of acquired competences [25]. Thanks to this, educators can precisely identify areas requiring improvement and optimize simulation exercise programs, leading to better preparation of future nurses for real clinical challenges.
The high Cronbach’s a coefficient and significant item correlations in the scale additionally confirm its reliability and validity in Polish educational conditions. Validation of the Polish version of the SET-M scale not only enables standardization of clinical skills assessment in nursing programs, but also contributes to the increase in the quality of simulation education in Poland, supporting the development of highly qualified medical staff able to work effectively in a demanding medical environment.

Disclosure

1. The study was approved by the Bioethics Committee. Approval number: 721/2022.
2. Assistance with the article: None.
3. Financial support and sponsorship: None.
4. Conflicts of interest: None

References