Researchers evaluating the performance of ChatGPT-4 Vision found that the model performed well on text-based radiology exam questions but struggled to answer image-related questions accurately. The study’s results were published today in Radiology, a journal of the Radiological Society of North America (RSNA).

Chat GPT-4 Vision is the first version of the large language model that can interpret both text and images.

“ChatGPT-4 has shown promise for assisting radiologists in tasks such as simplifying patient-facing radiology reports and identifying the appropriate protocol for imaging exams,” said Chad Klochko, M.D., musculoskeletal radiologist and artificial intelligence (AI) researcher at Henry Ford Health in Detroit, Michigan. “With image processing capabilities, GPT-4 Vision allows for new potential applications in radiology.”

For the study, Dr. Klochko’s research team used retired questions from the American College of Radiology’s Diagnostic Radiology In-Training Examinations, a series of tests used to benchmark the progress of radiology residents. After excluding duplicates, the researchers used 377 questions across 13 domains, including 195 questions that were text-only and 182 that contained an image.

GPT-4 Vision answered 246 of the 377 questions correctly, achieving an overall score of 65.3%. The model correctly answered 81.5% (159) of the 195 text-only queries and 47.8% (87) of the 182 questions with images.

“The 81.5% accuracy for text-only questions mirrors the performance of the model’s predecessor,” he said. “This consistency on text-based questions may suggest that the model has a degree of textual understanding in radiology.”

Genitourinary radiology was the only subspecialty for which GPT-4 Vision performed better on questions with images (67%, or 10 of 15) than text-only questions (57%, or 4 of 7). The model performed better on text-only questions in all other subspecialties.

The model performed best on image-based questions in the chest and genitourinary subspecialties, correctly answering 69% and 67% of the image-containing questions, respectively. The model performed lowest on image-containing questions in the nuclear medicine domain, correctly answering only 2 of 10 questions.

The study also evaluated the impact of various prompts on the performance of GPT-4 Vision.

• Original: You are taking a radiology board exam. Images of the questions will be uploaded. Choose the correct answer for each question.

• Basic: Choose the single best answer in the following retired radiology board exam question.

• Short instruction: This is a retired radiology board exam question to gauge your medical knowledge. Choose the single best answer letter and do not provide any reasoning for your answer.

• Long instruction: You are a board-certified diagnostic radiologist taking an examination. Evaluate each question carefully and if the question additionally contains an image, please evaluate the image carefully in order to answer the question. Your response must include a single best answer choice. Failure to provide an answer choice will count as incorrect.

• Chain of thought: You are taking a retired board exam for research purposes. Given the provided image, think step by step for the provided question.

Although the model correctly answered 183 of 265 questions with a basic prompt, it declined to answer 120 questions, most of which contained an image.

Although the model correctly answered 183 of 265 questions with a basic prompt, it declined to answer 120 questions, most of which contained an image.

“The phenomenon of declining to answer questions was something we hadn’t seen in our initial exploration of the model,” Dr. Klochko said.

The short instruction prompt yielded the lowest accuracy (62.6%).

On text-based questions, chain-of-thought prompting outperformed long instruction by 6.1%, basic by 6.8%, and original prompting style by 8.9%. There was no evidence to suggest performance differences between any two prompts on image-based questions.

“Our study showed evidence of hallucinatory responses when interpreting image findings,” Dr. Klochko said. “We noted an alarming tendency for the model to provide correct diagnoses based on incorrect image interpretations, which could have significant clinical implications.”

Dr. Klochko said his study’s findings underscore the need for more specialized and rigorous evaluation methods to assess large language model performance in radiology tasks.

“Given the current challenges in accurately interpreting key radiologic images and the tendency for hallucinatory responses, the applicability of GPT-4 Vision in information-critical fields such as radiology is limited in its current state,” he said.

Hospital admissions for heart attacks increase after exposure to lower air temperature and cold spells, according to a study published today in JACC, the flagship journal of the American College of Cardiology, and presented at the European Society of Cardiology (ESC) Congress 2024. The findings underscore the need to understand further the physiological effects of global warming’s contribution to colder weather in specific regions and cold-related cardiac risks.

“This nationwide study reveals that short-term exposures to lower air temperature and cold spells are associated with an increased risk of myocardial infarction (MI) hospitalization after two -six days, suggesting that individuals may be particularly vulnerable to acute cardiac events during periods of cold stress,” said Wenli Ni, PhD, lead author of the study and a postdoctoral research fellow at Harvard University.

JACC Editor-in-Chief Harlan M. Krumholz, MD, SM, FACC, said the study reveals a crucial link between cold weather exposure and heart attack risk, highlighting a delayed effect that peaks days after the cold spell.

“These findings also underscore the urgent need for targeted interventions to protect vulnerable populations during and, particularly, after cold stress,” Krumholz said.

Heart attacks, or MI, occur when blood flow to a part of the heart is blocked, usually by a blood clot. This blockage prevents oxygen from reaching the heart muscle, causing damage or death to that part of the muscle.

Previous studies have revealed low temperatures had a greater cardiovascular burden than high temperatures worldwide. Due to a lack of data from colder regions with extreme weather conditions, researchers based this study in Sweden, a region known for its cold climate where cold spells are common.

Tracking 120,380 individuals from the SWEDEHEART registry, researchers examined how short-term exposure to lower air temperatures and cold spells influenced the risk of heart attack hospital admissions during Sweden’s cold season (October to March) from 2005 to 2019.

For this study, researchers defined cold spells as a period of at least two consecutive days where the average daily temperature was colder than the 10th percentile of temperatures recorded across the study duration.

Results revealed that lower air temperature exposure was associated with an increased risk of total MI, NSTEMI and STEMI after two to six days. Cold spell exposure after two to six days was associated with the same increased risks.

Researchers also found exposure from day zero to one decreased the risk of heart attack hospitalizations. They said that this temporary protective effect could be due to behavior modifications during cold weather, like staying indoors to reduce exposure or delaying healthcare due to service disruptions; however, those behaviors are not sustainable and could account for the delay in hospitalizations to two to six days later.

“This temporal pattern may indicate a delayed onset of cold-related impacts on MI risk, aligning with prior research underscoring delayed cardiovascular effects of cold exposure,” Ni said. “Understanding this cold-to-MI risk lag sequence may be important for designing and implementing targeted preventive interventions.”

Effects on first-time and recurrent MI were also observed independently but the difference in these effects between first-time and recurrent MI was not statistically significant.

In an accompanying editorial comment, Kai Chen, PhD, an associate professor of Epidemiology at the Yale School of Public Health and Khurram Nasir, MD, a professor of Cardiology at the Houston Methodist, said the study’s findings call for reevaluating how health care professionals approach the intersection of environmental factors and cardiovascular health.

“Our approaches must be reevaluated in the context of increasingly unpredictable climate patterns,” Chen and Nasir said. “Addressing both ends of the temperature spectrum will ensure our healthcare systems are well equipped to manage and mitigate these challenges, ultimately fostering a more sustainable and resilient cardiovascular future.”