The correct training and evaluation of medical intern-resident (MIR) trainees in paediatrics is one of the chief concerns of university children’s hospitals. In Spain, for many years, the education of medical students and MIR trainees has focused on the acquisition of theoretical knowledge with little emphasis on its subsequent application in clinical practice. In 1990, Miller proposed a model defining the different stages of clinical knowledge and skill acquisition in a pyramid with different levels.1 The base corresponds to theoretical knowledge (‘knows’) providing the foundation for the next level, competence (‘knows how’), which progresses to performance (‘shows how’) and finally action in real-world practice (‘does’), which constitutes what we understand as clinical competence. Thus, clinical competence can be defined as the capacity to exercise good judgment and having the required knowledge, skills and attitudes to solve complex problems that may emerge in clinical practice in various contexts.2 The evaluation of clinical competence poses a challenge to medical educators, which has led to the development in recent years of assessment methods focused on the top level of Miller’s pyramid. In particular, the mini-CEX is one of the most widely used instruments of the kind, and facilitates the assessment of trainees’ clinical competence through direct observation using a structured questionnaire and the provision of feedback to trainees.3 This instrument assesses 6 components of competence (medical interviewing skills, physical examination skills, professionalism, clinical judgment, counselling skills and organization/efficiency) through a standardised questionnaire that has been validated in different clinical settings and different observers/evaluators.4 Descriptors for each of the assessed competencies are provided to promote a homogeneous approach to the assessment of trainees by different evaluators. This tool was developed in the 1990s by the American Board of Internal Medicine,5,6 and several studies have confirmed its validity, reliability, feasibility and user satisfaction.6–10 There is also evidence of its positive impact on the education of medical students and MIR trainees assessed with this approach.11

There is a dearth of data on the application of the mini-CEX in the field of paediatrics2,12,13 and, to our knowledge, only one study on the subject has been conducted in Spain in a hospital serving adults,7 and there are no data from studies conducted in children’s hospitals, despite the fact that the Asociación Española de Pediatría (Spanish Association of Pediatrics) has recommended its use for assessment of clinical skills learning outcomes in MIR trainees through Continuum, its virtual training platform.14

For this reason, the objectives of our study were to assess the evolution in paediatric emergency care skill acquisition in paediatrics MIR trainees by means of the mini-CEX as well as the satisfaction of educators and trainees with the use of this instrument.

We conducted an observational descriptive study in the paediatric emergency department of a university tertiary care children’s hospital that manages approximately 105 000 emergency visits a year.

We assessed trainees in in years 1 and 2 of the paediatrics speciality MIR programme during their rotations and on-call shifts in the emergency department between May 1, 2019 and April 30, 2023. The number of assessed MIR trainees varied over the 3 years under study, as hospital staffing increased (12 year 1 and 12 year 2 trainees in the 2019−2020 academic year, 16 year 1 and 12 year 2 in the 2020−2021 academic year, and 16 year 1 and 16 year 2 in the 2021−2022 and 2022−2023 academic years). Before the study began, a meeting was held to inform MIR trainees of the objectives of the assessment and that their skills would be evaluated for training purposes. The assessment consisted of the direct observation of a visit with the MIR trainee and subsequent feedback after the trainee had finished providing care to the patient. The evaluators were 7 adjunct doctors employed full-time in the emergency department who had been in the department for a mean of 12 years, were accredited as simulation instructors by the Boston Children's Hospital simulation programme and trained in crisis resource management and communication skills in in-house courses. The assessments were performed by any of these physicians on a convenience basis and adhered to the direct observation and evaluation criteria defined by the American Board of Internal Medicine.4,15

Each assessment focused on one of the following skill sets: medical interviewing and physical examination; communication and professionalism; clinical judgement, overall clinical competence and organisation. Following the recommendations found in the previous literature,2,15,16 a maximum of 6 assessments (2 per skill set) were performed per resident per year using a standardised data collection form (Appendix B). The assessment form was developed according to the criteria established by the Sociedad Española de Urgencias Pediátricas (SEUP, Spanish Society of Paediatric Emergency Medicine) in their proposed emergency care education curriculum plans and the new European Union training requirements for the specialty of paediatrics.17

The assessment form included fields to collect data on the clinical setting where the trainee-patient interaction takes place (emergency department, which was always the case of patients with triage levels 1, 2 or 3, or ambulatory, reserved exclusively for patients with triage levels 4 or 5), characteristics of the patient (age, sex), the complexity of the case (low, moderate or high) based on criteria previously defined by the evaluators (triage level, presenting complaint, difficulties in communication with the family) and which of the skills/competencies the assessment focused on. The evaluator rated each assessed skill on a numerical scale: 1–3, unsatisfactory; 4–6, satisfactory; 7–9, superior. The form also had fields to document the time devoted to observation and to providing feedback and the satisfaction of both the evaluator and the trainee, rated on a scale from 1 to 9 (low to high satisfaction).

To guarantee the right to autonomy of the trainees and independence from the compulsory annual evaluation, the emergency department physician who asked the trainees to participate in the study and who performed the mini-CEX evaluations was never the physician appointed as the supervisor of the trainee during the emergency care rotation or the physician in charge of the annual evaluation conducted by the teaching department. In cases in which one of the researchers was the supervisor of one of the trainees to be assessed, this researcher was not the one to carry out the mini-CEX assessment of the given trainee.

The collected data were stored and processed in a Microsoft Access relational database developed for the purpose. Data for quantitative and categorical variables were entered in tables to be subsequently analysed with the software package IBM SPSS Statistics, version 29 for Windows. We summarised quantitative data as median and range and categorical data as absolute frequency and percentage distributions. The distribution of the data was assessed by means of the Kolmogorov-Smirnov test and quantitative data compared with the Student t test or Mann-Whitney U test as applicable. We conducted a comparative analysis of the differences in scores based on the year of the MIR programme. Statistical significance was defined as a P value of less than .05.

The study was conducted in compliance with the current ethical standards, laws and regulations in Europe and Spain. It was approved by the Ethics Committee for Research with Medicines of our hospital (code PIC-78−19). Patients and their relatives were verbally informed that a direct assessment of the skills of MIR trainees was taking place.

The study included a total of 237 assessments of 56 trainees: 132 (55.7%) assessments corresponded to trainees in year 1 of the MIR programme and 105 (44.3%) year 2 (Table 1). During the study period, each trainee underwent between 1 and 9 assessments (median, 4): 95 (40.1%) focused on the history-taking and physical examination, 80 (33.8%) on communication and professionalism and 62 (26.2%) on clinical judgement, organisation and overall clinical competence. One hundred and forty-eight (62.4%) trainee-patient interactions took place in emergency care, 83 (35%) in urgent care and 6 (2.5%) in outpatient settings; 54.9% were of low complexity and 38% of moderate complexity. The median age of the patients was 4 years (IQR, 1.5–10) and 50.2% were male.

The median duration of the observation was 15 min per assessment (IQR, 12–25) and the median duration of feedback was 10 min (IQR, 7–15). Table 2 compares the scores obtained by first-year versus second-year trainees in the different clinical competencies. There was a significant difference between year 1 and year 2 MIR trainees in medical interviewing, physical examination, organisation and overall clinical competence. The median satisfaction score was 9 (minimum, 5; maximum, 9) for both evaluators and trainees.

Usually, the emergency department constitutes an unfavourable environment for teaching on account of the high workloads and time constraints, but having tested the use of the mini-CEX, we found that it was also feasible in a high-volume care setting. The mini-CEX proved to be a useful tool that allowed the monitoring of clinical skill acquisition in MIR trainees throughout the residency period. In our hospital, we observed positive learning outcomes in paediatrics interns and residents who did rotations in the emergency department, especially in the core competencies established in the curricula for early residency years,17 such as medical interviewing, physical examination and communication with families and coworkers. On the other hand, the fact that MIR trainees did not improve in areas such as organisation, clinical judgement and global clinical competence suggests that these skills may be more difficult to achieve or that they are achieved with a greater level of knowledge in the speciality. In light of this, the mini-CEX could also be used as a tool to analyse and adapt paediatrics MIR programme learning objectives over the years.

Our findings confirmed that the implementation of the mini-CEX was a very satisfactory intervention for the MIR trainee team, as participants reported that it helped them improve aspects of their work life that could not be assessed or corrected using traditional assessment approaches. The experience was also very satisfactory for evaluators, who believed they acquired knowledge that they could apply in their daily work, although they considered that more time should be carved out from their schedules to be able to provide adequate feedback. This was consistent with the findings of previous studies18–20 and could explain the scarcity of the literature on this teaching tool in emergency care settings, where heavy workloads usually do not allow breaks for teaching purposes. Another factor that could explain the dearth of data for the emergency care setting is that the mini-CEX is a tool that is well-known and widely used in Spanish universities in the training of medical students,9,21 but not widely implemented in hospitals offering MIR programmes. With this study, in agreement with the current literature,19 we confirm that the use of the mini-CEX as an educational tool in an emergency care department is feasible and has a positive impact on learning in MIR trainees.

The main limitation of the study is the potential for bias due to the Hawthorne effect during direct observation, as the attitudes and performance of MIR trainees during the visit may change due to their awareness of being observed. Since each trainee undergoes several assessments, this effect may become smaller over time.

In conclusion, we want to highlight that the use of the mini-CEX in the emergency care setting allowed us to observe a positive trend in the development of most of the assessed clinical skills in paediatrics MIR trainees in a tertiary care hospital and that both trainees and educators expressed a high level of satisfaction with its implementation. The mini-CEX could be used as a tool to evaluate outcomes of improvement strategies in the educational curricula of paediatrics residency training.