As an NVIDIA launch partner, Nolla Health is sharing new results from post-training NVIDIA Nemotron 3.5 Super VL to interpret clinical images and conduct simulated patient consultations. This work is part of NollaMD, our effort to build models that can gather a patient’s history, examine images, and produce a useful clinical assessment.
Over approximately one week of training with TrainLoop, visual accuracy rose from 35.0% to 66.8% on Nolla’s 500-question internal evaluation, ahead of all three comparison models. In a separate evaluation of 75 simulated consultations, the model went from completing 50 encounters to completing all 75, while top-diagnosis accuracy increased from 48% to 64%.
Clinical evaluations and training environments
A patient’s first photo and message rarely contain everything needed for an assessment. The image might show the border of a lesion or the pattern of a rash. The conversation reveals when it appeared, how it has changed, and what else the patient is experiencing. The model has to connect those details and decide what to ask next.
We measure progress through Nolla’s internal clinical evaluations, which test how a model interprets images, gathers missing information, reasons through possible diagnoses, and documents an encounter. Our physicians help define the criteria for a good assessment, including whether the questions are useful and whether the final note stays faithful to the conversation.
For this training run, we used a focused environment built around a single clinical encounter: an image, a patient with a defined history, and a conversation ending in an assessment and note. This let us measure how well Nemotron gathered information and completed the encounter.
Our training environments let models practice these skills through conversations with simulated patients. We calibrated the patient simulator against behavior observed in real conversations: patients give short answers, leave out relevant details, and become impatient with repetitive questions. The model has to learn how to gather the information it needs under those conditions.
Physicians reviewed generated conversations and helped develop the scoring rubric. It covers diagnosis, interaction, clinical documentation, and image reasoning, with penalties for unsupported facts and a safety gate that can reduce the reward to zero for a serious error.
Training
We trained Nemotron in two stages. Starting with approximately 2.5 million images of skin conditions, we filtered and balanced the data to produce 115,396 multiple-choice questions spanning 2,864 conditions, reserving 500 questions for the model comparison. We also generated 10,000 synthetic patient profiles and backgrounds for the conversational environment, using images from Nolla’s data.
Working with TrainLoop, we ran the training in NVIDIA NeMo RL using Group Relative Policy Optimization, or GRPO, on 32 NVIDIA B200 GPUs provided by Nebius. The first stage consisted of 400 steps on visual questions. Held-out validation performance continued to improve toward the end of the run, when we moved on to conversational training.
The second stage consisted of 69 steps on full simulated consultations. Each training example now required a sequence of questions and answers, ending in an assessment and a SOAP note: a structured record of the patient’s history, findings, assessment, and plan. Gemini 3.8 Flash graded the conversations and notes using the physician-informed rubric.
Results
At the 50-step conversational checkpoint, the model completed all 75 held-out consultations, up from 50 before conversational training. Top-diagnosis accuracy rose from 48% to 64%, and tokens used per completed note fell by 47%. Completing an encounter and reaching the right diagnosis are separate outcomes; we measure both.
The model also retained much of its visual performance after conversational training. It scored 64.0% on the same 500 visual questions, compared with 66.8% immediately after the visual-training stage and 35.0% for the base model. The visual questions and simulated consultations are separate evaluations, even though the conversational checkpoint scored 64% on both.
Comparison with frontier models
We compared the trained model with GPT-6 Astra across the conversational rubric. Nemotron scored higher on differential-diagnosis reasoning, at 95.6% versus 83.9%, and on SOAP-note quality, at 86.3% versus 75.1%. It also scored higher on image observation and photo handling.
The differential-diagnosis score assesses the quality of the reasoning rather than whether the first diagnosis is correct. On top-diagnosis accuracy, GPT-6 Astra scored 65.3%, compared with Nemotron’s 64.0%. Astra also scored slightly higher on question strategy, treatment planning, and tone.
The largest gap in Astra’s favor was triage urgency: 99.2% versus 89.5%. Recognizing when a patient needs more urgent care is a specific priority for further training and evaluation. A strong clinical note does not compensate for missing an urgent situation.
Limitations and next steps
These results come from internal visual evaluations and simulated patient encounters. They show improvements on those tasks; they do not establish how reliably the model will perform in patient care. The conversational results also depend on the patient simulator and an automated judge applying our clinical rubric.
Our next task is to evaluate how well these behaviors carry into real encounters, including when the model should ask for more information, request another image, or recognize the need for urgent care. Nolla’s clinical evaluations and training environments give us a way to identify those weaknesses and measure whether further training addresses them.
Thank you to NVIDIA for early access to Nemotron and for NeMo RL, to TrainLoop for the training collaboration, and to Nebius for the B200 compute.
