Performance of ChatGPT-4 Omni and Gemini 1.5 Pro on Ophthalmology-Related Questions in the Turkish Medical Specialty Exam
Turkish Journal of Ophthalmology, cilt.55, sa.4, ss.177-185, 2025 (Scopus, TRDizin)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 55 Sayı: 4
- Basım Tarihi: 2025
- Doi Numarası: 10.4274/tjo.galenos.2025.27895
- Dergi Adı: Turkish Journal of Ophthalmology
- Derginin Tarandığı İndeksler: Scopus, TR DİZİN (ULAKBİM)
- Sayfa Sayıları: ss.177-185
- Anahtar Kelimeler: Artificial intelligence, ChatGPT-4 Omni, e-learning, Gemini 1.5 Pro, large language model, medical education, ophthalmology
- Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
- Bilecik Şeyh Edebali Üniversitesi Adresli: Evet
Özet
Objectives: To evaluate the response and interpretative capabilities of two pioneering artificial intelligence (AI)-based large language model (LLM) platforms in addressing ophthalmology-related multiple-choice questions (MCQs) from Turkish Medical Specialty Exams. Materials and Methods: MCQs from a total of 37 exams held between 2006-2024 were reviewed. Ophthalmology-related questions were identified and categorized into sections. The selected questions were asked to the ChatGPT-4o and Gemini 1.5 Pro AI-based LLM chatbots in both Turkish and English with specific prompts, then re-asked without any interaction. In the final step, feedback for incorrect responses were generated and all questions were posed a third time. Results: A total of 220 ophthalmology-related questions out of 7312 MCQs were evaluated using both AI-based LLMs. A mean of 6.47±2.91 (range: 2-13) MCQs was taken from each of the 33 parts (32 full exams and the pooled 10% of exams shared between 2022 and 2024). After the final step, ChatGPT-4o achieved higher accuracy in both Turkish (97.3%) and English (97.7%) compared to Gemini 1.5 Pro (94.1% and 93.2%, respectively), with a statistically significant difference in English (p=0.039) but not in Turkish (p=0.159). There was no statistically significant difference in either the inter-AI comparison of sections or interlingual comparison. Conclusion: While both AI platforms demonstrated robust performance in addressing ophthalmology-related MCQs, ChatGPT-4o was slightly superior. These models have the potential to enhance ophthalmological medical education, not only by accurately selecting the answers to MCQs but also by providing detailed explanations.