The retinopathy of prematurity (ROP) is a potentially avoidable cause of blindness in children. The advances in neonatal care make the survival of extremely premature infants, who show a greater incidence of the disease, possible. The aim of the study is to evaluate the reliability of ROP screening using retinography imaging with the RetCam 3 wide-angle camera and also to study the variability of ROP diagnosis depending on the evaluator.
Materials and methodsThe indirect ophthalmoscopy exam was performed by a paediatric ROP-expert ophthalmologist. The same ophthalmologist and a technician specialised in digital image capture took retinal images using the RetCam 3 wide-angle camera. A total of 30 image sets were analysed by 3 masked groups: group A (8 ophthalmologists), group B (5 experts in vision), and group C (2 ROP-expert ophthalmologists).
ResultsAccording to the diagnosis using indirect ophthalmoscopy, the sensitivity (26–93), Kappa (0.24–0.80), and the percentage agreement were statistically significant in group C for the diagnosis of ROP type 1. In the diagnosis of ROP type 1+type 2, Kappa (0.17–0.33) and the percentage agreement (58–90) were statistically significant, with higher values in group C.
ConclusionThe diagnosis, carried out by ROP-expert ophthalmologists, using the wide-angle camera RetCam 3 has proven to be a reliable method.
La retinopatía del prematuro (ROP) es una de las principales causas de ceguera infantil potencialmente evitable. Los avances en los cuidados neonatales consiguen la supervivencia de niños de bajo peso extremo asociado con una mayor incidencia de la enfermedad. El objetivo de este estudio es evaluar la fiabilidad en el diagnóstico de la ROP a través de retinografías obtenidas con la cámara de campo amplio RetCam 3 y estudiar la variabilidad en el diagnóstico de ROP en función del evaluador.
Material y métodosEl examen con oftalmoscopio indirecto fue realizado por un oftalmólogo pediátrico experto en ROP. Las imágenes retinianas fueron obtenidas por el mismo oftalmólogo y un técnico especializado en la captura de imágenes digitales. Un total de 30sets de imágenes fueron analizados por 3 grupos enmascarados: grupo A (8 oftalmólogos), grupoB (5 expertos en visión) y grupo C (2 oftalmólogos expertos en ROP).
ResultadosAcorde con el diagnóstico mediante oftalmoscopia indirecta, la sensibilidad (26-93), el índice Kappa (0,24-0,80) y el porcentaje de aciertos fueron estadísticamente significativos en el grupoC para el diagnóstico de ROP tipo1. En el diagnóstico de ROP tipo1+tipo2 el índice Kappa (0,17-0,33) y el porcentaje de aciertos (58-90) fueron estadísticamente significativos, obteniendo valores superiores en el grupo C.
ConclusionesEl diagnóstico realizado por oftalmólogos expertos en ROP con el retinógrafo móvil RetCam 3 ha demostrado ser un método fiable.
Despite advances in retinopathy of prematurity (ROP), this disorder is one of the main potentially avoidable causes of childhood blindness.1 ROP is a vasoproliferative retinopathy, characterised by the presence of an immature and partially vascularised retina, affecting low birth-weight preterm infants. The spectrum of possible results for patients with ROP ranges from mild cases, where the sequelae are minimal and do not affect vision, to more aggressive, bilateral cases, with total and irreversible blindness.2 Prompt intervention is critical to maximise the chance of a favourable outcome3; without treatment, at least 50% of eyes with severe ROP will suffer serious and permanent visual loss.3
Recent studies show that in the last 20 years the number of children requiring monitoring has risen constantly.1,4–7 Advances in neonatal care enable very low weight newborns to survive; low weight and gestational age are associated with a higher incidence of severe ROP.4,8–10 The incidence of the disorder in developed countries varies compared to developing countries,11,12 partly due to greater use of assisted conception, increasing maternal age, possible genetic aetiologies and other socioeconomic issues.13 Prevalence in Spain, during the last decade and the beginning of the present one,11 was similar to that of other developed countries in the European Community and the United States.7,14 Currently, according to the available data from the Instituto Nacional de Estadística (INE: National Statistics Institute), the number of newborns whose weight was less than 1500g in Spain in 2011 was 4178, of whom 682 were in the Autonomous Community of Madrid.
The current method of diagnosis, binocular indirect ophthalmoscopy (BIO), has serious limitations.5 The examiner's interpretations are transcribed onto ophthalmological examination sheets, where the interpretation is assumed to be correct and cannot be revised.15 Another worrying factor is the shortage of trained ophthalmologists capable of performing examinations for diagnosis of ROP.5,6,10 As a result, a large number of untrained ophthalmologists are carrying out screening and treatment of ROP.10,16,17 Several studies show that inexperienced ophthalmologists are less skilled than trained ophthalmologists at identifying clinically significant ROP by analysing digital images.10,17,18 Because of the shortage of trained ophthalmologists, long delays can occur in screening; another alternative is for the infants to be transferred, with the considerable cost this entails and the potential risk to which the premature baby is subjected.3
Telemedicine makes it possible to capture images and send them for subsequent interpretation by a remote expert,6 thereby solving many of the limitations in diagnosis of ROP performed by means of the “gold standard”. By using wide-angle cameras, telemedicine has demonstrated high precision and reliability in the diagnosis of ROP,6,19–26 and may be more cost-effective than the current method of diagnosis.6,27
The main objectives of this study are:
- •
To assess the reliability of diagnosis of ROP through ocular fundus images obtained with a RetCam 3 mobile retinal camera (Clarity Medical Systems Inc.) via telemedicine.
- •
To explore the variability in diagnosis of ROP according to the training profile of the evaluator.
This study followed the principles of the Helsinki Declaration and was approved by the Ethics Committee of the Hospital Clínico San Carlos and the Hospital Universitario La Paz in Madrid. All the parents of the preterm infants at risk of ROP who participated in the study signed an informed consent document.
PatientsDigital images were taken of children at risk of ROP admitted to the neonatal intensive care unit (NICU) of the Hospital Universitario La Paz in Madrid. The screening criterion applied was that laid down in the NICU protocol, which includes all infants with a gestational age of less than 31 weeks (≤30+6), those of less than 34 weeks (31–33) with unstable clinical course or a weight of less than 1250g. Thirty retinal photographs were captured (15 right eyes and 15 left eyes) of 9 preterm infants between January 2012 and April 2012. A total of 24 preterm infants were examined, but only the cases with ROP were included in the study. Excluded from the study were the preterm infants who, in the neonatologist's opinion, had an unstable clinical evolution and those whose retinopathy had previously been treated by laser. In three low birth-weight preterm infant the initial examination could not be performed owing to the poor quality of the images obtained, two of them because of corneal oedema and one because of insufficient pupillary dilation.
Schedule of careThe first ocular fundus examination is carried out at the age of 4 weeks for newborns with a gestational age at birth of 27 weeks or more. For newborns with a gestational age of less than 27 weeks, the first examination is performed at the postmenstrual age of 31 weeks. Depending on the results found in the ocular fundus exploration by BIO, the scheduling of the follow-up examinations will adhere to the guidelines published by the Academy of Paediatrics, the American Association for Paediatric Ophthalmology and Strabismus and the American Academy of Ophthalmology.28 The check-ups are continued until (a) the retina is fully vascularised; (b) the vessels reach zone III without previous ROP in zone I or II (a confirmatory examination would be indicated in very low-weight infants) or (c) absence of “pre-threshold” disease by 45 weeks post-conceptional age (zone I: any stage with plus, stage 3 without plus; zone II: stage 2 “plus” or stage 3).
Examination techniquesPupillary dilation is necessary to perform an ophthalmological examination. Mydriasis is achieved by instillation of one drop of cyclopentolate 0.25% and one drop of phenylephrine 1% 1h before the examination, repeated after 10min. The presence or absence of ROP, its location, its extent and plus disease were documented according to the criteria established in the International Classification of ROP.29 During the ophthalmological examination a nurse immobilises the premature infant and monitors his or her vital signs; if there is any sudden alteration in the low birth-weight preterm infant cardiorespiratory indicators the examination is interrupted until the infant stabilises. To avoid retinal phototoxicity, ambient lighting is kept to a minimum and the lowest light intensity that allows a reliable examination is used in the ophthalmoscope.
Indirect ophthalmoscopic examination (the “gold standard”) was performed by a paediatric ROP-expert ophthalmologist. The eyelids are kept open with special blepharostats for premature infants. Viewing of the peripheral retina is carried out using scleral indentation and a 28-D lens. Immediately after the BIO examination, the same ophthalmologist and a technician specialising in capture of digital images (Certificate of Completion. Training Module Series: RetCam Practice, Ocular Anatomy and Imaging ROP Presented by Clarity Medical Systems Inc., Pleasanton, CA) captured the images of each eye using the RetCam 3 wide-angle camera (Clarity Medical Systems, Pleasanton, CA, USA) equipped with a special 130° lens for premature infants. In the methodology employed the ophthalmologist holds the camera while the technician controls the lighting and focus. A minimum of 3 photos per eye was captured (posterior pole, nasal retina and temporal retina); a group of images of the same eye in the same session was called a “set”.
Reading and coding of digital imagesThe retinal photographs obtained were sent to the storage centre, where they were coded. A total of 30 sets were interpreted independently by 3 masked groups: group A, comprising 8 ophthalmologists (3 anterior pole surgeons, 3 retinologists, and 2 general ophthalmologists); group B, composed of 5 vision experts from the Universidad Complutense in Madrid (graduates with official master's degrees in optics, optometry and vision) and group C, consisting of 2 ophthalmologist experts in diagnosis and treatment of ROP. Groups A and B were informed about the diagnosis of ROP by means of a 4-page guide compiled by the authors. All those surveyed were asked about the diagnosis of ROP; each set of photos was classified by ROP, stage, zone, plus, referral to paediatric ophthalmologist (type 1+type 2) and observations. The interpretation of the retinal photographs was based on the criteria laid down in the study Early Treatment for Retinopathy of Prematurity (ETROP).30 The images transmitted were coded at the storage centre and classified using a system of 4 levels, as in the study by Myung et al.17:
- (1)
Type 1 ROP: zone I, any stage with plus disease or stage 3 without plus disease; zone II, stages 2 or 3 with plus disease.
- (2)
Type 2 ROP: zone I, stages 1 or 2 without plus disease; zone II, stage 3 without plus disease.
- (3)
Moderate ROP: defined as ROP less severe than type 2.
- (4)
No ROP.
No time limit was set for interpreting the images. The diagnosis and recommendations for control examinations were returned by email to the storage centre.
Analysis of dataThe validity of the new diagnosis system was determined by comparing the results obtained by the three groups, and the diagnosis performed using BIO by a paediatric ophthalmologist specialising in ROP was taken as a reference. The results obtained with BIO and RetCam for each set were collected in separate columns using statistical software (Microsoft Excel 2010, Redmond, WA, USA). The calculations for the statistical analysis were performed with SAS 9.2 software. Sensitivity, specificity, Kappa index and percent success rate of each group were calculated for detection of type 1 ROP and referral (type 1+type 2 ROP). We used ANOVA and Duncan's test to compare the results between groups.
ResultsA total of 30 sets of images were analysed by 15 observers (8 ophthalmologists, 5 vision experts and 2 paediatric ophthalmologists specialising in ROP). According to the diagnosis with BIO performed by the reference observer (the “gold standard”), 47% of the digital images showed type 1 ROP, 3% type 2 ROP, and 50% moderate ROP.
Table 1 shows sensitivity, specificity, Kappa index and percent agreement in the diagnosis of type 1 ROP and type 1+type 2 (referral) by group C, comprising two ophthalmologists expert in the diagnosis of ROP.
Sensitivity, specificity, Kappa index and percent agreement for diagnosis of type 1 ROP and type 1+type 2 ROP (referral) in group C (ROP-expert ophthalmologist).
| Sensitivity (%) (standard error) | Specificity (%) (standard error) | Kappa (standard error) | CD (%) (maximum–minimum) | |
| Type 1 | 93 (0) | 88 (13) | 0.80 (0.14) | 88 (95–81) |
| Type 1+type 2 | 90 (3) | 90 (10) | 0.80 (0.09) | 90 (95–85) |
Tables 2 and 3 show sensitivity, specificity, Kappa index and percent agreement in the diagnosis of type 1 ROP and type 1+type 2 (referral) by three groups of specialists compared with the reference diagnosis performed by a paediatric ROP-expert ophthalmologist. Sensitivity, Kappa index and percent success rate were statistically significant in group C for diagnosis of type 1 ROP. In the diagnosis of type 1+type 2 ROP the Kappa index and percent success rate were statistically significant, producing higher values in group C. No statistically significant differences were found between groups A and B in the diagnosis of type 1 ROP and type 1+type 2 ROP.
Sensitivity, specificity, Kappa index and percent agreement for diagnosis of type 1 ROP by 3 groups of specialists compared with the reference diagnosis performed by a paediatric ROP-expert ophthalmologist.
| Type 1 | Sensitivity (%) (standard error) | Specificity (%) (standard error) | Kappa (standard error) | CD (%) (maximum–minimum) |
| Group A | 26 (10) | 93 (2) | 0.24 (0.20) | 57 (68–40) |
| Group B | 39 (5) | 92 (3) | 0.29 (0.12) | 56 (65–47) |
| Group C | 93 (0) | 88 (13) | 0.80 (0.14) | 88 (95–81) |
| Value of p | 0.0114a | 0.6944 | 0.0046a | 0.0053a |
CD, correct diagnoses.
Sensitivity, specificity, Kappa index and percent agreement for diagnosis of type 1+type 2 ROP (referral) by 3 groups of specialists compared with the reference diagnosis performed by a paediatric ROP-expert ophthalmologist.
| Type 1+type 2 | Sensitivity (%) (standard error) | Specificity (%) (standard error) | Kappa (standard error) | CD (%) (maximum–minimum) |
| Group A | 47 (10) | 86 (7) | 0.33 (0.23) | 66 (78–55) |
| Group B | 59 (11) | 59 (13) | 0.17 (0.07) | 58 (61–55) |
| Group C | 90 (3) | 90 (10) | 0.80 (0.09) | 90 (95–85) |
| Value of p | 0.1358 | 0.1101 | 0.0048a | 0.0045a |
CD, correct diagnoses.
The results of this study confirm, firstly, the efficacy of the RetCam 3 wide-angle camera through images obtained by trained staff. In addition, the study demonstrates the viability of telemedicine for the diagnosis of type 1 ROP and type 1+type 2 ROP (referral) when performed by paediatric ROP-expert ophthalmologists.
The viability of telemedicine in the assessment of low birth-weight preterm infant is a point of great interest given the gradually increasing number of children at risk of ROP4,8 in developed countries. Digital images offer an innovative strategy for detecting infants with severe ROP,31,32 and several studies show excellent sensitivity and specificity in the diagnosis of ROP.5,18–20,24,25,33–36 In most research studies an ophthalmologist5,19,26,32,33,36–38 or a nurse17,21,22,25,34,35,39,40 carried out the test, although occasionally an ophthalmological photographer18,20,23 obtained the images. Our conclusions coincide with the results published by Skalet et al.,40 in which two trained nurses captured the images with the NIDEK NM200-D camera and subsequently sent them to five ROP experts. The conclusions of Skalet et al. show that images obtained by trained staff and sent to specialists can be useful for identifying infants who require evaluation by a ROP-expert ophthalmologist.
In 2008 Kemper et al.16 showed that there was a high proportion of untrained ophthalmologists carrying out screening and treatment of ROP. Moreover, Paul Chan et al.,18 in 2010, assessed the reliability of inexperienced ophthalmologists in the diagnosis of ROP; they demonstrated a clinically significant underestimation of ROP, which raises important questions about the quality of ROP screening performed by non-expert ophthalmologists. We therefore agree with the suggestion of Wong et al.10 that an ophthalmologists’ training should include a minimum number of ROP examinations and treatments.
The results of this research are similar to those obtained in the study by Myung et al.,17 which shows a sensitivity of approximately 50% in the detection of type 1 and type 2 ROP by paediatric ophthalmologists inexperienced in ROP; it is equivalent, in this study, to the sensitivity in referral (type 1+type 2) of groups A and B, consisting of non-ROP experts. The results also coincide for type 1 ROP, where low sensitivity and high specificity were obtained in groups A and B. Unlike the study by Myung et al.,17 in which the diagnosis made by a paediatric retina expert using digital images is taken as a reference, the “gold standard” used here is the diagnosis obtained using indirect ophthalmoscopy performed by an expert in ROP, following the recommendations of the Section on Ophthalmology of the American Academy of Pediatrics, the American Academy of Ophthalmology and the American Association for Pediatric Ophthalmology and Strabismus.7
The advantages of telemedicine have already been described by several authors. It has been shown to be reliable for the diagnosis of ROP and to save costs compared with the conventional method of diagnosis,27 but this is not enough; the opinion of the parents must be taken into account. The research of Lee6 concludes that parents have a positive impression of ROP diagnosis by telemedicine, but express a preference for face-to-face contact. The current benefits of new technologies are indisputable, but it should not be forgotten that information must be as widely available as possible, both to patients and to those accompanying them.
In our review of the literature we found few studies that evaluated the training of ophthalmologists who carry out the examination, assessment and treatment of ROP.10,16–18 As has been shown, differences in the diagnosis of ROP lead to different outcomes of the disease, and therefore future research should propose a minimum number of examinations for untrained ophthalmologists. We also wish to highlight the potential risk involved in untrained staff carrying out the diagnosis and treatment of ROP. This study found no significant differences in the results between ophthalmologists (group A) and vision experts (group B). Future projects could analyse the role of opticians and optometrists for screening of ROP, which would save the cost of transferring the premature infant and avoid having specialists perform the test.
We recognise that one of the limitations of this study is the number of premature infants examined and the small number of experts performing the diagnosis of ROP. In this phase, only retinal photographs of premature infants with some stage of retinopathy were captured. This was done to avoid the stress for the infant caused by the additional test with the RetCam. Future research could be conducted, as in the study by Skarlet et al.,40 with several experts on premature infants, comparing them with a series of ophthalmologists without experience of ROP. As in previous studies, the principal limitations were the assessment of zone III and the quality of the images;17 in particular, serious difficulties were encountered in the examination of patients with CPAP both by BIO and by RetCam; the view of the retina is slightly better with digital images.
In conclusion, use of the RetCam 3 mobile retinal camera by ophthalmologist experts in ROP has proved to be reliable for diagnosis.
Conflicts of interestThe authors have no conflicts of interest to declare.
The authors thank the Neonatal Intensive Care Unit (NICU) of the Hospital Universitario La Paz in Madrid and the whole team at the Clínica Rementería, Madrid, Spain for their collaboration.
Please cite this article as: Navarro-Blanco C, Peralta-Calvo J, Pastora-Salvador N, Álvarez-Rementería L, Chamorro E, Sánchez-Ramos C. Fiabilidad en el cribado de la retinopatía del prematuro mediante el análisis de retinografías. An Pediatr (Barc). 2014;81:149–154.
