Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study

Soshi Takagi; Takashi Watari; Ayano Erabi; Kota Sakaguchi

doi:10.2196/48002

Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study

JMIR Med Educ. 2023 Jun 29:9:e48002. doi: 10.2196/48002.

Authors

Soshi Takagi^#¹, Takashi Watari^{1

2

3

4}, Ayano Erabi¹, Kota Sakaguchi²

Affiliations

¹ Faculty of Medicine, Shimane University, Izumo, Japan.
² General Medicine Center, Shimane University Hospital, Izumo, Japan.
³ Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI, United States.
⁴ Medicine Service, VA Ann Arbor Healthcare System, Ann Arbor, MI, United States.

^# Contributed equally.

PMID: 37384388
PMCID: PMC10365615
DOI: 10.2196/48002

Abstract

Background: The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied.

Objective: This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages.

Methods: This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions.

Results: The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages.

Conclusions: GPT-4 could become a valuable tool for medical education and clinical support in non-English-speaking regions, such as Japan.

Keywords: AI; Chat Generative Pre-trained Transformer; ChatGPT; GPT-4; Generative Pre-trained Transformer 4; Japanese Medical Licensing Examination; artificial intelligence; clinical support; learning model; medical education; medical licensing.

©Soshi Takagi, Takashi Watari, Ayano Erabi, Kota Sakaguchi. Originally published in JMIR Medical Education (https://mededu.jmir.org), 29.06.2023.