ClinSeekAgent

Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning

From passive evidence consumption to active evidence acquisition. Real-world clinical workflows require agents to actively seek, iteratively plan, and synthesize multimodal evidence from heterogeneous sources.

Juncheng Wu1*, Letian Zhang1*, Yuhan Wang1*, Haoqin Tu1, Hardy Chen1, Zijun Wang1, Cihang Xie1, Yufan He2, Can Zhao2, Daguang Xu2, Yucheng Tang2, Yuyin Zhou1

1University of California, Santa Cruz    2NVIDIA

*Equal technical contribution  ·  Corresp. jwu418@ucsc.edu

61.7 → 63.9
Text-only F1 of Claude Opus 5: Curated Input → ClinSeekAgent
60.0 → 62.5
Multimodal F1 of GPT-5.6 Sol: Curated Input → ClinSeekAgent
22
Tools: 16 EHR + 6 medical-image
35.6 (+16.1)
ClinSeek-35B-A3B average F1 on AgentEHR-Bench (gain over its base)
ClinSeekAgent overview: the agent seeks evidence from raw EHR data and medical imaging. Asked to identify diseases in the right upper lung zone or upper mediastinum, it retrieves the radiology report from the local EHR database and analyzes the chest X-ray to answer pneumonia, whereas the curated-context setting lacks the needed information.
ClinSeekAgent overview. An automated agentic pipeline that interacts with heterogeneous data sources for multimodal evidence seeking. Example: asked for the diseases in the right upper lung zone or upper mediastinum, ClinSeekAgent retrieves the study's radiology report (right upper-lobe opacity) and analyzes the chest X-ray to answer pneumonia; the same model with curated context fails for lack of critical evidence.

Abstract

Large language models (LLMs) and agentic systems have shown promise for clinical decision support, but existing works largely assume that evidence has already been curated and handed to the model. Real-world clinical workflows instead require agents to actively seek, iteratively plan, and synthesize multimodal evidence from heterogeneous sources. In this paper, we introduce ClinSeekAgent, an automated agentic framework for dynamic multimodal evidence seeking that shifts the paradigm from passive evidence consumption to active evidence acquisition. Given a clinical task and access to raw patient data, ClinSeekAgent queries longitudinal EHR records and invokes imaging analysis tools when relevant; refines its hypotheses as new information emerges; and integrates the collected evidence into grounded clinical decisions. We validate ClinSeekAgent in two roles: as an inference-time agent for frontier LLMs and as a training pipeline for open-source models.

For inference-time validation, we construct ClinSeek-Bench, which covers 13 text-only EHR task categories and 6 multimodal EHR–chest X-ray task categories, and pairs each question with both a Curated Input of pre-selected evidence and the raw patient data for automated evidence seeking. Without any benchmark-specific curation, ClinSeekAgent significantly outperforms curated input for both Claude Opus 5 and GPT-5.6 Sol, improving overall F1 from 61.7 to 63.9 and from 57.8 to 59.3 on text-only EHR tasks, and from 66.5 to 68.8 and from 60.0 to 62.5 on multimodal tasks. The largest gains occur in text-based decision-making and multimodal phenotype prediction, and ablations show that they come from multi-round evidence seeking rather than from access to more of the patient record.

For training-time validation, we fine-tune Qwen3.5-35B-A3B on ClinSeekAgent trajectories generated by GPT-5.6 Sol. The resulting ClinSeek-35B-A3B reaches 35.6 average F1 on AgentEHR-Bench (+16.1 over its base model), outperforming all evaluated open-source models, including ones more than ten times larger, as well as Claude Opus 5.

Key findings

What changes when the agent seeks its own evidence instead of receiving a curated package.

(a)

Evidence seeking beats curated input

Without benchmark-specific curation, ClinSeekAgent significantly raises overall F1 for both frontier models on both benchmarks: Claude Opus 5 61.7 → 63.9 (text) and 66.5 → 68.8 (multimodal); GPT-5.6 Sol 57.8 → 59.3 and 60.0 → 62.5. The largest gains are on text decision making and multimodal phenotype prediction.

(b)

Multi-round seeking is the source

Putting the full record in the prompt adds only +0.8 / +1.9 (text) and +0.1 / +0.3 (multimodal) over Curated Input (Opus 5 / Sol), and single-pass retrieval gives no significant gain. Making retrieval iterative adds +5.4 / +1.7 (text) and +6.1 / +1.4 (multimodal) over single-pass retrieval.

(c)

The gain grows with record length

Below 30k tokens of patient record, ClinSeekAgent is on par with both prompt-based settings. On records above 250k tokens, its margin over Curated Input reaches +4.6 (Opus 5) and +5.3 (Sol).

(d)

Distillable into an open model

Fine-tuned on GPT-5.6 Sol trajectories, ClinSeek-35B-A3B reaches 35.6 average F1 on AgentEHR-Bench (+16.1 over its base), outperforming all evaluated open-source models and Claude Opus 5 (31.6), at 92% of its teacher (38.8).

Method: agentic evidence seeking over raw patient data

Each task gives the agent a patient identifier, a prediction time, and an instruction, but not the curated patient context. The agent then runs an open-ended trajectory τ = (x, (ak, ok)k=1..K, ŷ) of tool calls and observations, for up to 200 rounds, before calling finish with its answer. ClinSeekAgent standardizes the environment and tool interface; it imposes no ordering over evidence sources, so the evidence-seeking policy comes from the model.

ehr.* 16 tools

Longitudinal EHR retrieval

Patient-specific access to the raw record through schema exploration; time-, keyword-, and value-based retrieval; SQL; and candidate-term lookup, plus think and finish.

Schema exploration
load_ehr get_table_names get_column_names get_table_description get_unique_values
Record retrieval
get_records_by_time get_event_counts_by_time get_latest_records get_records_by_keyword get_records_by_value
SQL · candidate lookup · control
run_sql_query get_candidates_by_keyword get_candidates_by_fuzzy_matching get_candidates_by_semantic_similarity think finish
image.* 6 tools

Medical-image analysis

Visual evidence from chest radiographs through DICOM preprocessing, image visualization, chest X-ray classification, report generation, phrase grounding, and anatomical segmentation.

Image tools
dicom_processor image_visualizer chest_xray_classifier chest_xray_report_generator xray_phrase_grounding chest_xray_segmentation

ClinSeek-Bench-Text uses the 16 EHR tools; ClinSeek-Bench-MM uses all 22.

Temporal policy

Each question gets a question-specific view of the record that keeps only what was available before the prediction time, and every EHR tool (including SQL) and every Curated Input prompt reads from this same view. Diagnosis and procedure codes become available only 24 hours after their encounter ends, and the codes of the encounter being predicted are always hidden. A linked chest X-ray is visible only if acquired before the cutoff, and its report only if finalized by then.

Two roles

Inference-time agent

A frontier LLM runs ClinSeekAgent directly on raw patient data; we compare it with the same model on Curated Input on ClinSeek-Bench.

Training pipeline

ClinSeekAgent trajectories from a strong teacher supervise an open-source model; we evaluate the student on AgentEHR-Bench.

ClinSeek-Bench: paired curated and agentic settings

Every question appears in two settings with the same task definition and answer label: Curated Input, the evidence package of the source benchmark, and Automated Evidence-Seeking, where that context is removed and the model must retrieve evidence from the raw data with ClinSeekAgent tools.

ClinSeek-Bench-Text
1,800 questions

40 per subtask from the 45 EHR-Bench subtasks over MIMIC-IV, grouped into 13 categories: 4 risk prediction, 9 decision making.

ClinSeek-Bench-MM
989 questions

Reconstructed from EHRXQA and MedMod, which pair MIMIC-IV EHRs with MIMIC-CXR chest radiographs, in 6 categories.

Text categoryNPredict
Risk prediction
Mortality280In-hospital death, or death within 1, 2, 3, 7, or 14 days in the ICU
Readmission160ED return within 72 hours, ICU readmission, or hospital readmission within 30 or 60 days
Length of stay160ICU stay over 7 or 14 days, or hospital stay over 3 or 7 days
Escalation of care120Hospitalization from the ED, ICU transfer within 12 hours, or a critical outcome
Decision making
Diagnosis160Next diagnoses, as ICD titles or CCS categories
Medication320Next administered medications, prescriptions, ED medication items, and ICU inputs
Procedure120Next procedures and ICU procedure items
Laboratory & microbiology80Next laboratory and microbiology tests
Vitals & measurements160Next ICU charted and output items, and outpatient measurements
Imaging report40Next radiology examination
Orders40Type of the next provider order
Encounter & flow120Next admission type, clinical service, and transfer care unit
Next event40EHR table of the next event
Multimodal categoryNTask (source)
CXR finding presence215Is a finding present in a chest X-ray study? (EHRXQA)
CXR finding enumeration182List the findings of a study (EHRXQA, MedMod)
CXR change comparison222How findings change between two studies (EHRXQA)
Clinical deterioration125ICU death within the next 24 hours (MedMod)
Mortality125Death during the current ICU stay (MedMod)
Phenotype120Acute and chronic conditions of an ICU stay (MedMod)
Metric

Sample-wise F1 (%) over the predicted and gold label sets, averaged over questions; for single-label questions it reduces to accuracy. All comparisons are paired on questions, with paired-bootstrap 95% confidence intervals.

Inference-time results: Curated Input vs. ClinSeekAgent

Same questions, same labels; only the way evidence reaches the model changes. Both modes use the same reasoning budget and one run per question.

Curated Input source-benchmark evidence package

The pre-selected evidence is rendered into the prompt (for EHR-Bench, up to 100 events from the past 24 hours) and the model answers directly, without tools.

ClinSeekAgent (ours) automated evidence seeking

Given only the patient identifier and the prediction cutoff, the model retrieves evidence from the raw record with the tools above for up to 200 rounds before calling finish.

ClinSeekAgent improves both frontier models on both benchmarks

Claude Opus 5 · Text
61.7 → 63.9 +2.2
Claude Opus 5 · Multimodal
66.5 → 68.8 +2.3
GPT-5.6 Sol · Text
57.8 → 59.3 +1.6
GPT-5.6 Sol · Multimodal
60.0 → 62.5 +2.5

Overall F1 (%), Curated Input → ClinSeekAgent; Δ is the mean per-question difference, and every paired-bootstrap 95% CI excludes zero.

Dumbbell plot of F1 by task category for Claude Opus 5 (top) and GPT-5.6 Sol (bottom) on ClinSeek-Bench-Text (left) and ClinSeek-Bench-MM (right); hollow markers are Curated Input, filled markers ClinSeekAgent, and green labels mark categories where ClinSeekAgent improves.
ClinSeekAgent vs. Curated Input by task category (sample-wise F1, %) for Claude Opus 5 (top) and GPT-5.6 Sol (bottom) on ClinSeek-Bench-Text (left) and ClinSeek-Bench-MM (right). Hollow markers: Curated Input; filled: ClinSeekAgent; the label above each pair is their difference, green where ClinSeekAgent improves.

Per-category results

Click a tab to switch models. Full-Record Input and Single-Pass Retrieval are the ablation settings described below.

Where does the gain come from?

ClinSeekAgent differs from Curated Input in two ways: it can access the whole record instead of a curated window, and it gathers evidence over multiple rounds of tool calls and reasoning. Two ablation settings extend record access without the multi-round loop; all four settings share the same temporal policy.

Full-Record Input

Curated Input with the patient's full pre-cutoff record in the prompt, truncated oldest-first when it does not fit the 100k-token context window.

Single-Pass Retrieval

ClinSeekAgent with the same tools and record access, but all retrieval calls must be issued in one turn before answering.

F1 gain of ClinSeekAgent over Curated Input and over Full-Record Input, by length of the patient's full pre-cutoff record, for Claude Opus 5 and GPT-5.6 Sol on ClinSeek-Bench-Text. The gain over Curated Input reaches +4.6 and +5.3 above 250k tokens.
The gain of ClinSeekAgent grows with the length of the patient record (ClinSeek-Bench-Text, F1 points). Below 30k tokens it is on par with both prompt-based settings; above 250k tokens its margin reaches +4.6 / +5.3 over Curated Input and +2.1 / +1.3 over Full-Record Input (Opus 5 / Sol). In the shaded bins the record does not fit in the 100k-token window of Full-Record Input and is truncated.

The improvements are stable across runs

Over five runs of GPT-5.6 Sol, the standard deviation of overall F1 is at most 0.4 points within each mode, ClinSeekAgent beats Curated Input in every run, and Krippendorff's α of per-question F1 across runs is 0.87–0.91.

More models

Models with stronger agentic capability benefit more. For Claude Opus 4.6, GLM-5.2, and Kimi K2.6, all three stay within two points of Curated Input overall on text; on multimodal, Claude Opus 4.6 gains +5.1 (mostly on phenotype) and Kimi K2.6 +2.0, and only the gain of Opus 4.6 is significant.

Case studies

Two questions where ClinSeekAgent answers correctly and an ablation setting does not.

ClinSeekAgent vs. Single-Pass Retrieval

Next eMAR medication question. ClinSeekAgent's SQL queries find a prior anticoagulant (apixaban) and a pending CT order, then retrieve the CT report showing a large intraventricular hemorrhage, and answer the reversal agent Andexxa. Single-Pass Retrieval plans eight queries upfront, never retrieves the radiology report, and answers apixaban.
Early rounds return the patient's prior anticoagulant and a pending CT order, so the agent requests the CT report, finds a large intracranial bleed, and predicts the reversal agent. Single-Pass Retrieval had to plan all of its queries before seeing any result; it never retrieved the report and predicted the anticoagulant that the bleed contraindicates.

ClinSeekAgent vs. Full-Record Input (GPT-5.6 Sol, next transfer care unit)

Next transfer care unit question. ClinSeekAgent sees that the admission transfer is still pending, widens its time window, finds that a comparable prior orthopedic stay was admitted to Med/Surg/GYN, and answers Med/Surg/GYN. Full-Record Input has the precedent at 26% of the prompt but answers Observation, a string near the end of the prompt that is not a care unit in the record.
Having the record in the prompt is not the same as finding the deciding row: the unit to which the patient was admitted after a comparable orthopedic visit two weeks earlier sits at 26% of the untruncated Full-Record prompt, yet the model answers with a string near the end of the prompt that is not a care unit in the record. ClinSeekAgent notices that the admission transfer is still pending, widens its time window, and retrieves the precedent.

Training-time validation: ClinSeek-35B-A3B

The gain of ClinSeekAgent is consistent only for models with strong agentic capability, so we use it as a training pipeline to transfer this capability to a smaller open model. GPT-5.6 Sol runs ClinSeekAgent on the 3,000 questions of the AgentEHR-Bench training split, and Qwen3.5-35B-A3B is fully fine-tuned on the resulting trajectories, with the teacher's reasoning in the student's native thinking block and tool calls in its native format.

2,628 training / 53 validation trajectories 2× H200 · tensor 2 / expert 2 parallel 52K-token sequences 3 epochs, epoch-2 checkpoint (lowest validation loss) lr 2×10−5, cosine decay

Open-source state of the art on AgentEHR-Bench

ClinSeek-35B-A3B improves on all five tasks, from +7.6 on Microbiology to +23.9 on Labs; it beats Qwen3.5-397B-A17B, with about eleven times more parameters, by +7.5, exceeds Claude Opus 4.6 by +2.4 and Claude Opus 5 by +4.0, and reaches 92% of its teacher (35.6 vs. 38.8), matching it on Transfers (27.4 vs. 27.3). Evaluation uses 100 test questions per task, none of which appears in the training data; all models run ClinSeekAgent with the same tools and temporal policy.

Average F1 on AgentEHR-Bench versus model size: ClinSeek-35B-A3B (35.6) is above every open-source model, including Qwen3.5-397B-A17B (28.1), and above Claude Opus 5 (31.6); GPT-5.6 Sol scores 38.8 and GPT-5.6 Luna 33.8.
Performance–model size comparison on AgentEHR-Bench. ClinSeek-35B-A3B outperforms all open-source models, as well as Claude Opus 5.

What does ClinSeek-35B-A3B learn?

Over the 500 AgentEHR-Bench trajectories, both precision (18.6 → 32.4) and recall (31.1 → 53.3) improve over the base model, accompanied by two changes in behavior.

More complex SQL queries

SQL with joins, aggregation, or time filters; it typically first summarizes what the record contains and then drills into the current admission.

Questions with a complex SQL query
0.8% → 53%

Reasoning grounded on the patient record

The student surveys the record before narrowing down.

Available tables read
36% → 72%
Targeted follow-up queries
13% → 36%
Questions reading imaging or discharge notes
12% → 74%
Reasoning refers to the current admission
39% → 80%

Base model (Qwen3.5-35B-A3B) → ClinSeek-35B-A3B. The student learns the evidence-seeking process, not only its final answers.

Resources

Code, model, benchmark, and results.

Cite

@misc{wu2026clinseekagent,
  title         = {ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning},
  author        = {Wu, Juncheng and Zhang, Letian and Wang, Yuhan and Tu, Haoqin and Chen, Hardy and Wang, Zijun and Xie, Cihang and He, Yufan and Zhao, Can and Xu, Daguang and Tang, Yucheng and Zhou, Yuyin},
  year          = {2026},
  eprint        = {2605.20176},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2605.20176}
}

Preprint: arXiv:2605.20176.