Polycystic Ovarian Syndrome (PCOS) is an endocrine disorder affecting 5-15% females (1). It is a heterogeneous disease, with each woman’s symptomology being vastly different. (2-5). Consequently, diagnosis is delayed and treatment difficult to find, so women self-manage their condition, seeking information from sources such as ChatGPT (2-3). There is, however, little data on the accuracy of evidence about PCOS (6-7). This study, therefore aims to assess the quality of recommendations provided by ChatGPT for PCOS management.
Common PCOS questions were gathered from various social media platforms. Meetings with PCOS and fertility experts were conducted where questions were verified. The questions were then categorised into groups using NVivo. The questions were typed into ChatGPT using a non-institutional email in a new conversation each time. After 14-21 days, questions were re-queried by another reviewer. The accuracy of the answers was then analysed by comparing its recommendations to PCOS guidelines. 6-point and 3-point likert scales were used to assess the accuracy and completeness of answers, respectively.
Across all questions (n=36), the mean accuracy score was 4.2 and the mean completeness score was 1.8. All questions were re-queried and re-graded approximately 14-21 days later with a mean accuracy score of 4.4 and mean completeness score of 2.5. The scores differed between reviewers because ChatGPT is constantly evolving to provide more accurate information. However, there were limitations, including providing outdated and wrong advice and not using clinical guidelines as an information source. Answers to many of the questions were also the same, highlighting the lack of creativity of ChatGPT.
ChatGPT generated mostly accurate and complete answers to a range of questions regarding PCOS, although with some limitations. ChatGPT may be improved by including its references in its responses. It would also benefit from more robust training where it utilises information from credible, evidence-based sources.