Serving and evaluating open-weight models for health on Ortet’s Orion cluster
Health is one of the most important applications of AI, yet it gets limited attention in how new models are evaluated. AI model releases mostly cite coding, math, and general reasoning, and general leaderboards rank models on the same skills. Health-specific evaluations do exist, but they rarely provide comprehensive, in-depth analyses of the latest open models.
Ortet is building a health model that connects biological, clinical, and operational dimensions. Starting with evaluation, we show what today’s models can already do, where they fail, and which capabilities remain unmeasured. How we measure performance also shapes how we train models. We focus on open-weight models because they let us run the whole loop ourselves: evaluation, data gathering, training on our own infrastructure, and re-evaluating those gains on healthcare tasks.
As a first step towards continuously improving intelligence for health, we evaluated 34 recent open-weight models on 50 tasks drawn from 15 public health benchmarks. As of the release date of this post, October 26, 25 of the 34 models—nearly 75%—had been released in the previous six months, including 18 in the previous three months and four in the previous two weeks. We self-host all 34 models and serve them through our own API on Orion, our cluster of NVIDIA Blackwell GPUs. We built this infrastructure and API in September to gain visibility, control, and security—it lets us verify which model version we are running, configure inference settings, and track how data is handled, so we know exactly what we are evaluating and can preserve the setup needed to reproduce the results.
For comparison, we also evaluated eight closed frontier models, all released in September 2026—GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5, Claude Sonnet 5.5, Claude Fable 5.1, Gemini 3.8 Flash, and Gemini 4 Argon—and two sovereign models, Mistral Large 4,1 released in October 2026, and Upstage’s Solar Pro 4, released in August 2026.2 We chose public benchmarks that anyone can download and rerun, covering clinical medicine, biological research, and medical coding. Here is what we found.
Muse and Kimi lead health benchmarks among open models
These are the top six models among 34 recent open-weight models on 50 health tasks.
General rankings are not enough for health
Among 32 matched open models, Muse Spark Open: #9 → #1, Inkling: #20 → #6.
Open models know the basics and are more willing research collaborators
On cloning questions, DeepSeek Flash V4.1 1017 scores 73%; all 33 Opus requests were blocked.
Newer can beat bigger
GLM-5.3 rises from #21 to #8 among open models at the same 753B parameters (best observed).
More reasoning helps for most open models, but the highest setting isn’t always best
Up to ~50× as many output tokens at the best observed settings.
Health rankings can be estimated with a small subset of questions
We would like to support organizations with evaluating models on various healthcare problems. See Collaborate with us for more details.
- As of October 26, 2026, Mistral Large 4’s weights had not been released.
- We also attempted to evaluate MuseWonder, but our account was flagged for a content policy violation during evaluation, preventing us from continuing.
Comparing open health models
The table ranks models using aggregated benchmark results. Weighted Bradley–Terry compares task wins and losses, with default / lowest or best observed reasoning. Both overall and category rankings give each benchmark family equal weight, divided equally among its tasks.
We group the tasks into clinical medicine (16 tasks on medical knowledge, patient care, and clinical prediction), biological research (19 on genomics and scientific research), and medical coding (15 on drug, diagnosis, and procedure codes). Under best-observed settings, Muse Spark Open leads open models in clinical medicine, DeepSeek V4.1 Flash 1017 in biological research, and Kimi K3 in medical coding.
Ortet Health Index
MedicineBiological
ResearchMedical
Coding
No matching models.
- A run needs at least 80% completed, non-truncated responses; refusals count as wrong. “Best observed” takes each model’s best reasoning setting separately for each task. Full method in the appendix.
- Ratings are weighted Bradley–Terry abilities fitted from task-level wins, ties, and losses; the size of a score gap does not count. Each benchmark family gets equal total weight; a category ranking uses only that category’s families.
General rankings are not enough for health
Could a general intelligence leaderboard stand in for a health evaluation? Overall, the rankings agree: across 32 matched open models, the rank correlation between the Artificial Analysis Intelligence Index and our Health Index is 0.77.
Among the largest models, general rankings tell us much less about health. For models under 500B parameters, rank correlation is 0.78 across 20 models. For models at or above 500B, it falls to 0.35 across 12 models. Medical coding shows even less agreement: correlation is −0.17 among 12 large models. This benchmark alone is insufficient to conclude that higher general ability predicts worse coding performance, but overall, these low correlations support dedicated health benchmarks rather than relying on the general ranking.
Inkling and Muse Spark Open are health standouts. Among the matched models, Inkling ranks 20th on the AA Intelligence Index and 6th on Ortet Health Index, while Muse Spark Open ranks 9th and 1st, respectively. A general leaderboard would make it easy to overlook models that perform near the top on health tasks.
Open models are competitive on basic medical knowledge—and more willing research collaborators
We evaluated eight closed frontier models—GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5, Claude Sonnet 5.5, Claude Fable 5.1, Gemini 3.8 Flash, and Gemini 4 Argon—alongside two sovereign models, Mistral Large 4 and Upstage’s Solar Pro 4. All ten used the same fixed random sample from each task: approximately 10% of its questions, with a minimum of 50 questions per task.
Competitive on medical exams, weaker on difficult clinical cases. Leading open models are competitive on medical exam questions: Kimi K3 scores 86.1% and Muse Spark Open scores 85.6% on MedMCQA, compared with 86.2% for GPT-6 Astra, 86.4% for Claude Opus 5.5, and 88.8% for Gemini 4 Argon. The gap widens on MedXpertQA Text, which tests difficult clinical cases: the best open model, Gemma 4 31B, scores 58.7%, compared with 68.2% for Astra, 71.4% for Opus, and 75.9% for Argon.
More willing research collaborators. Open models answer research questions that closed providers block. As a result, Kimi K3 scores 85.8% on DNA and protein sequence questions, while Claude Opus 5.5 scores 31.7%, with 60% of its requests blocked by the provider. On CloningScenarios, DeepSeek V4.1 Flash 1017 scores 72.7%, while all 33 Opus requests were blocked. GPT-6 Astra reaches 93.9% on cloning, demonstrating the limitation is specific to the model and provider rather than a refusal pattern shared by all closed models.
Neither open nor closed models are good at patient prediction. On a task that asks the model to place a person in one of 11 age groups based on DNA methylation, the best open model scores 24.4%, versus 6.0% for Astra, 24.0% for Opus, and 26.0% for Argon; random guessing gives 9.1%. Survival prediction is also difficult: given clinical measurements, models must choose one of four survival-time groups: 0–5, 5–10, 10–15, or 15+ years after examination. The best open model scores 38.7%, versus 37.2% for Astra, 35.0% for Opus, and 40.1% for Argon, against a 25% random baseline. Astra explicitly declines to choose an age group in 43 of 50 methylation questions, saying the information is insufficient. Yet when asked which of two people is older, it abstained from 11 out of 50 questions and scores 62.0%.
Newer models raise the bar without growing
Larger models remain strong performers, but newer releases can achieve more with less. Muse Spark Open, one of the newest releases, leads our open-model ranking despite being smaller than Kimi K3 and Qwen3.8 2.4T. Under best-observed settings, Inkling also outranks the larger DeepSeek V4 Pro, while GLM-5.3 Flash edges out both despite having just 320B parameters. Among the leading models, more parameters do not necessarily translate into better performance.
Within the same model family, larger versions of Qwen and Inkling outperform their smaller counterparts. Yet newer releases can improve without growing: GLM-5.3 beats GLM-5.2 at the same size, and Qwen3.8 27B beats the older, larger Qwen3.5 35B. Under best-observed settings, DeepSeek V4.1 Flash also outperforms the larger V4 Pro. We hypothesized that benchmark-specific optimization (“Benchmaxxing”) might explain these gains, but similar improvements on the very recently released Longevity Bench suggest that they may extend beyond established benchmarks.
Newer models improve without getting bigger
More reasoning helps, but the highest setting isn’t always best
We ran each model at the reasoning strengths it offers, from off or low up to its maximum.3 Across the seven tasks (MedMCQA, MedXpertQA Text, PubMedQA, GeneGPT, LAB-Bench (LitQA2, ProtocolQA, SuppQA)), 30 of 31 open models score higher on average at their highest setting than at their lowest. From the lowest setting to the best observed setting on each task, Inkling shows the largest average gain at 12.3 percentage points, while DeepSeek V4 Pro shows the smallest gain at 1.1 points.
The highest setting can overshoot the best observed result. Inkling loses 1.4 percentage points and Muse Spark Open loses 2.8 when moving from each task’s best observed setting to its highest. In the cases we checked, Muse sometimes switches from a correct answer to “Insufficient information,” while Inkling’s medical-exam decline simply comes from choosing more wrong answers.
Min effort → Best-scoring effortFurther increasing:
Best-scoring effort → Max effort
- Command A Plus and the two MedGemma models do not support a reasoning mode and are therefore excluded from this analysis.
Reasoning changes whether models answer or refuse
On a Longevity Bench task predicting whether a person survived ten years after an examination, turning on reasoning raises Nemotron 3.5 Lightning’s refusal rate from 0.2% to 87.0%. In an inspected response, it points out that the clinical measurements contain no follow-up records and declines to predict the outcome. HY3 moves in the opposite direction: its refusal rate falls from 19.3% without reasoning to 0.1% at low reasoning. More reasoning also reduces refusal for Mistral Small 4 (93.7% → 31.9%), Gemma 4 26B (20.5% → 0%), and Qwen3.5 35B (14.2% → 0.7%).
Reasoning can increase or decrease refusal
On the 10-year survival prediction task (Longevity Bench LB-0042)
Refusal rate (%) as lower effort → higher effort
Finding the cost-optimal model
At modest token cost, open models outperform closed models at comparable estimated spending. At costs below about $0.30 per 1,000 questions,1 open-weight models offer the best scores for the price; on health conversations, an open-weight model achieves the highest score overall. At ten to a hundred times the per-question cost, closed models lead the cost–performance frontier on most tasks. Open models combine competitive performance at affordable costs with control over deployment, giving health systems the freedom to balance cost, accuracy, and scale according to their own needs.
Medical exam questions (MedMCQA)
- Cost is tokens per question × OpenRouter list price as of October 9, 2026, for 1,000 questions.
Comparable model rankings with 10x less evaluation data
Collecting data to evaluate model capabilities is expensive in general. For health-related tasks, the costs can be even higher—preparing each example may require access to sensitive patient information, careful privacy review, and annotation by clinical experts whose time is limited. These costs prompted us to ask how much data an evaluation needs to produce meaningful results. Could a smaller, randomly selected sample capture the findings of the full evaluation?
We randomly sampled questions within each of the 50 tasks and repeated the experiment 30 times, holding the closed-model reference scores fixed. We found that using just 10% of the evaluation data reproduced the full-evaluation ranking with an average change of 1.32 places across 34 open-weight models. This result suggests a way to reduce evaluation costs and make repeated assessments more practical.
How many evaluation items do we need to rank models?
What comes next
Over the past few weeks, we set up a cluster and began self-hosting models at Ortet. Self-hosting gives us clear control over where data is stored and how models are run, and supports closer inspection of their behavior. That visibility matters for health-related tasks, where understanding how sensitive information is handled is part of building trust. Through this evaluation, we took a first step towards our goal—to build continuously improving intelligence across the clinical, biological, and operational aspects of health.
We began by surveying public benchmarks to understand what they measure, which capabilities they leave out, and how existing open models perform. Our survey identified gaps in tasks that require models to use tools, reason across a patient’s history, and understand difficult clinical cases. We also found limited coverage of patient trajectory prediction in both open and closed models.
These gaps will guide how we expand the evaluation suite. We plan to include medical images, patient histories spanning multiple timepoints, and more tasks that require tool use. We also want to assess whether models can identify which interventions would change a patient’s outcome. In a follow-up post, we will examine agentic systems, including how different base models interact with different harnesses—the software that provides tools and manages execution. That evaluation will help us understand how the model and its surrounding software contribute to performance on agentic, long-horizon healthcare tasks.
Collaborate with us
We imagine a future where everyone has access to affordable, precise, and better care. We believe that to build this future, we must partner across health organizations—providers, researchers, administrators, and operators.
We want to support organizations with running scaled evaluation campaigns to evaluate the performance of open-weight models on your specific healthcare problems. We can bring together your expertise and our infrastructure to understand the strengths and limitations of AI in real-world use.
If you have a workflow, question, or problem where you are curious about how the latest models perform, contact us.
Appendix
Models evaluated
The evaluation covers 34 open-weight models and ten models accessed through APIs. The open-weight models were served on the Orion cluster. Each model version is listed separately below, with its recorded release date.
| Open-weight model | Release date |
|---|---|
| Beam 1.0 | 2026-10-16 |
| Command A Plus | 2026-05-20 |
| DeepSeek V4 Flash | 2026-07-31 |
| DeepSeek V4 Pro | 2026-08-13 |
| DeepSeek V4.1 Flash | 2026-09-10 |
| DeepSeek V4.1 Flash 1017 | 2026-10-17 |
| GLM-5.2 | 2026-06-16 |
| GLM-5.3 | 2026-08-18 |
| GLM-5.3 Flash | 2026-08-26 |
| GPT-OSS 120B | 2025-08-05 |
| GPT-OSS 20B | 2025-08-05 |
| Gemma 4 26B | 2026-04-02 |
| Gemma 4 31B | 2026-04-02 |
| HY3 | 2026-07-06 |
| Inkling | 2026-07-15 |
| Inkling Small | 2026-07-30 |
| Kimi K3 | 2026-07-16 |
| Open-weight model | Release date |
|---|---|
| MedGemma 1.5 4B | 2026-01-13 |
| MedGemma 27B | 2025-05-20 |
| MiMo V2.6 Flash | 2026-09-21 |
| MiMo V2.6 Pro | 2026-09-21 |
| MiniMax M3 | 2026-06-01 |
| Mistral Small 4 119B | 2026-03-16 |
| Muse Glimmer 30B | 2026-08-10 |
| Muse Spark Open | 2026-10-15 |
| Nemotron 3 Ultra | 2026-06-04 |
| Nemotron 3.5 Lightning | 2026-08-11 |
| Nemotron 3.5 Super VL | 2026-10-15 |
| Qwen3.5 122B | 2026-02-24 |
| Qwen3.5 35B | 2026-02-24 |
| Qwen3.8 2.4T | 2026-08-12 |
| Qwen3.8 27B | 2026-08-14 |
| Qwen3.8 Flash Next | 2026-08-26 |
| Solar Open 2 250B | 2026-08-12 |
| API-accessed model | Recorded evaluation access dates | Service provider |
|---|---|---|
| GPT-6 Astra | October 2026 | OpenAI API |
| GPT-6.1 Sol | October 2026 | OpenAI API |
| GPT-6 Luna | October 2026 | OpenAI API |
| Claude Opus 5.5 | October 2026 | Anthropic API / OpenRouter |
| Claude Sonnet 5.5 | October 2026 | Anthropic API / OpenRouter |
| Claude Fable 5.1 | October 2026 | Anthropic API / OpenRouter |
| Gemini 3.8 Flash | October 2026 | Gemini API / OpenRouter |
| Gemini 4 Argon | October 2026 | Gemini API / OpenRouter |
| Mistral Large 4 | October 2026 | OpenRouter |
| Solar Pro 4 | October 2026 | OpenRouter |
The open-weight table lists release dates from the linked model listings and developer documentation. Mistral Large 4’s weights were unavailable at the evaluation cutoff.1
Benchmark reference
There are 15 families of benchmarks that span 50 tasks, covering clinical medicine, biological research, and medical coding.
For open-ended answers, we use four open-weight models as judges: Gemma 4 26B, GLM-5.2, Inkling, and Qwen3.8 27B. Each judge grades the response against the benchmark’s rubric, which specifies what a correct answer should contain. We prevent self-judging; the responses from one of those four models are graded by the other three judges. We combine valid judgments using each benchmark’s scoring method.
| Benchmark | Description & tasks | Example |
|---|---|---|
| Clinical Medicine | ||
| ASCENT rationales 2026-06-29–30 Metric: Micro-F1 from averaged judge counts Judge: Four open-weight LLM judges; diagnosis and rationale rubric. | Generate a diagnosis and supporting rationale from a clinical case. | Question: A 5-year-old boy presents with altered mental status a difficulty breathing for the past couple of hours. The patient's father, a mechanic, says the boy accidentally ingested an unknown amount of radiator fluid. Based on the provided information and medical knowledge, consider the name(s) of the disorder(s) or the related system(s) that should be suspected at the final Findings stage. Possible answer: Ethylene glycol poisoning; the reported ingestion of radiator fluid supports it. Question source |
| Bedside Bench 2026-09-22 Metric: Mean rubric score, equally weighted across categories Judge: Four open-weight LLM judges; case-specific rubric. | Clinical question answering across physician-reviewed cases, graded against case-specific rubrics. | Question: What obesity endpoints were reported for dexranib in the MIRVALDEX trial? Possible answer: The trial and drug cannot be verified; do not invent endpoint results. Question source |
| HealthBench Consensus 2025-05-12 Metric: HealthBench rubric score Judge: Four open-weight LLM judges; physician-consensus rubric. | Health conversations scored on the subset of HealthBench criteria with stronger agreement among clinicians. | Question: I found my 70 year old neighbor lying on the floor unresponsive, but they have a pulse and they’re breathing slowly. Should I just wait for them to come around, or do I need to do something right away, given they have no known health issues? Please give details. Possible answer: Call emergency services immediately, monitor breathing, and begin CPR if breathing or pulse stops. Question source |
| HealthBench Professional 2026-04-22 Metric: Length-adjusted rubric score Judge: Four open-weight LLM judges; clinician-written rubric. | Tasks clinicians bring to an assistant at work, evaluated with clinician-written grading criteria. | Question: What is the guideline on the use of short-term corticosteroids for a mild asthma exacerbation that is unresponsive to bronchodilators? Possible answer: Consider a short course of systemic corticosteroids when symptoms do not respond adequately to bronchodilators; assess for urgent red flags. Question source |
| MedLFQA 2024-05-21 Metric: Factuality score Judge: Four open-weight LLM judges; factuality rubric. | Long-form biomedical questions that require a detailed, factual answer rather than a choice letter. | Question: Is blepharitis contagious? Possible answer: No. The reference answer says blepharitis is not contagious. Question source |
| MentalHealthBench 2026-09-23 Metric: Provisional rubric score Judge: Four open-weight LLM judges; expert-written rubric. | Responses to realistic mental-health conversations, assessed against expert-informed rubrics. | Question: I invited her even though I had a gut feeling not to.. and she has been so distant in this friendship... Possible answer: Acknowledge that the distance has been hard, and ask what help the user wants with the birthday trip. Question source |
| OGCaReBench 2026-05-20 Metric: Answer equivalence rate Judge: Four open-weight LLM judges; reference-recommendation rubric. | Rare, off-guideline clinical cases that call for case-specific reasoning and evidence. | Question: Given these findings concerning for ocular trauma, what would be the most suitable urgent diagnostic approach to assess for traumatic ophthalmic damage? Possible answer: Perform an ocular ultrasound with an ocular preset and linear transducer. Question source |
| MedMCQA 2022-03-27 Metric: Accuracy | Multiple-choice medical knowledge questions adapted from entrance examinations. | Question: Which vitamin is required for glycogen Phosphorylase? Options: A. PLP B. TPP C. Riboflavin D. Lipoic acid. Possible answer: A. PLP (the active form of vitamin B6). Question source |
| MedXpertQA Text 2025-02-09 Metric: Accuracy | Difficult text-only, multiple-choice questions designed to test expert-level medical reasoning. | Question: A 4-year-old boy with Klippel-Feil syndrome has elevation of the left scapula since birth. Spine radiographs shows no evidence of scoliosis. What shoulder motion is likely to be most limited? Possible answer: D. Abduction. Question source |
| PubMedQA 2019-09-13 Metric: Macro-F1 over yes, no, and maybe | Answer a biomedical research question from a PubMed abstract with yes, no, or maybe. | Question: Is halofantrine ototoxic? Possible answer: Yes, according to the reference abstract (PubMed 20537205). Question source |
| Longevity Bench 2026-01-13 Metric: Accuracy for choice tasks; mean absolute error for numeric tasks, with answer coverage reported separately | Seventeen prediction tasks using clinical, genetic, epigenomic, transcriptomic, and proteomic measurements. 6 tasks: LB-0030 (clinical multiclass), LB-0034 (clinical pairwise), LB-0038 (clinical regression), LB-0042 (clinical binary), LB-0046 (clinical multiclass), LB-0050 (clinical pairwise) | Question: How long did this participant survive post-examination? Options: A. 0-5 years B. 15+ years C. 5-10 years D. 10-15 years Clinical profile: Demographic information: 54-year-old female with less than 9th grade of Mexican American ethnicity. Body measurements: BMI 26.5 kg/m2, height 166.6 cm, waist circumference 97.0 cm, body weight 73.5 kg, diastolic blood pressure 70.7 mm Hg, systolic blood pressure 110.7 mm Hg, pulse 40.0 bpm. … Possible answer: C. 5–10 years (recorded survival: 7 years 3 months). Question source |
| Biological Research | ||
| GeneTuring (GeneGPT) 2023-04-19 Metric: Mean extracted-answer score across question groups Judge: Rule-based match to the reference answer. | GeneTuring questions about gene names, locations, associations, and DNA sequences, using the dataset released with GeneGPT. | Question: What is the official gene symbol of LMP10? Possible answer: PSMB10. Question source |
| OpenBioRQ 2026-06-20 Metric: Checklist score Judge: Four open-weight LLM judges; weighted research rubric. | Open biomedical research questions that ask for a reasoned research response, not a textbook fact. | Question: What is the optimal adjuvant therapy strategy for biliary tract cancers after resection? Possible answer: The reference discusses six months of capecitabine as the current standard and notes that optimal intensification remains unresolved. Question source |
| LAB-Bench 2024-07-16 Metric: Accuracy | Six selected tasks covering biology literature, databases, protocols, sequences, and cloning scenarios. 6 tasks: CloningScenarios, DBQA, LitQA2, ProtocolQA, SeqQA, SuppQA | Question: Which of the following genes is associated with Currarino syndrome (Currarino triad) according to DisGeNet but not according to OMIM? Possible answer: PCSK5 (the keyed answer in DBQA). Question source |
| Longevity Bench 2026-01-13 Metric: Accuracy for choice tasks; mean absolute error for numeric tasks, with answer coverage reported separately | Seventeen prediction tasks using clinical, genetic, epigenomic, transcriptomic, and proteomic measurements. 11 tasks: LB-0002 (epigenomics multiclass), LB-0006 (epigenomics pairwise), LB-0010 (epigenomics regression), LB-0014 (transcriptomics multiclass), LB-0018 (transcriptomics pairwise), LB-0074 (proteomics multiclass), LB-0082 (proteomics regression), LB-0086 (genetics binary), LB-0122 (genetics pairwise), LB-0126 (genetics regression), LB-0299 (proteomics pairwise) | Question: Which intervention results in longer mean lifespan? Options: A. p38a(df);p38b(df) B. p38a(df). Species: Drosophila melanogaster. Previously observed effects: p38a(df): −30.6% lifespan (34 vs 49 days wild type); p38b(df): −61.2%, +38.8%, mean −11.2% (43 vs 49 days wild type). Possible answer: B. p38a(df). Question source |
| Medical Coding | ||
| MedConceptsQA 2024-05-12 Metric: Accuracy | Multiple-choice questions asking for the description of a given medical code across ATC, ICD-9, and ICD-10, at three difficulty levels. 15 tasks: ATC (Easy, Hard, Medium), ICD-10-CM (Easy, Hard, Medium), ICD-10-PCS (Easy, Hard, Medium), ICD-9-CM (Easy, Hard, Medium), ICD-9 procedure (Easy, Hard, Medium) | Question: What is the description of the medical code R91.1 in ICD10CM? Possible answer: A. Solitary pulmonary nodule. Question source |
Evaluation configurations
Reasoning and sampling settings
For models hosted on Orion, we used greedy decoding. For API-accessed models, we generally used provider-default sampling; medical-coding runs for GPT-6 Luna, Mistral Large 4, and Solar Pro 4 used greedy decoding.
We varied reasoning effort on the medical-exam and research tasks to measure how additional reasoning changes performance. The settings depend on the model: some support only reasoning off/on, while others offer none, minimal, low, medium, high, xhigh, or max, depending on the model. Some models instead use settings named full or adaptive.
| Benchmarks | How we evaluated reasoning |
|---|---|
| MedMCQA, MedXpertQA Text, PubMedQA, GeneGPT, LAB-Bench, and Longevity Bench | Swept the model-specific reasoning settings, from the lowest available through the highest tested level, including xhigh or max where supported. Coverage varies by model. Models without adjustable reasoning used their default setting. |
| MedConceptsQA: medical coding | Scored answer choices by likelihood where supported, selecting the most likely choice. Generated-answer runs used no reasoning or the lowest available setting. |
| LLM-judged benchmarks: ASCENT rationales, Bedside Bench, HealthBench Consensus, HealthBench Professional, MedLFQA, MentalHealthBench, OGCaReBench, and OpenBioRQ | Selected lower-reasoning or default configurations for the reported comparison. |
In early evaluations, some models continued reasoning for hundreds of thousands of tokens without producing a final answer. To prevent this, we limited reasoning to 8,192 tokens and total output, including the final answer, to 16,384 tokens. For models that did not support a reasoning cap directly, we used a two-stage procedure. If a model reached the 8,192-token limit, we appended “I have to give the answer based on my reasoning now.” to its response and sent a second request to generate the final answer. The total output across both requests was limited to 16,384 tokens.
Answer extraction
We first used the benchmark’s official answer parser where available. When a parser was missing or failed to recognize valid answer formats, we inspected saved responses, refined the extraction rules, and replayed them to review changed predictions.
For Longevity Bench, DeepSeek V4.1 Flash extracted choices when the deterministic parser failed and extracted numeric predictions from completed responses. The extractor never received the correct answer. Explicit numeric ranges were represented by their midpoint. Refusals and abstentions received no credit on accuracy tasks; numeric tasks reported prediction error and answer coverage separately. Failed or truncated generations retained their failure status.
For open-ended tasks, we combined valid judgments from the four open-weight judges described in the benchmark reference, following each benchmark’s scoring rule. For ASCENT, we averaged each answer’s true-positive, false-positive, and false-negative counts across valid judges, summed those counts across answers, and calculated micro-F1.
How we calculate the rankings
To rank the models, we use a Bradley–Terry model. Each task produces a win, loss, or tie for each pair of models. For binary accuracy tasks, higher accuracy is better; for age prediction, lower mean absolute error is better; and for tasks scored with F1, higher F1 is better. Equal scores produce a tie. For numeric predictions, we convert prediction error into a score and multiply it by the fraction of questions answered, so missing answers reduce the score. Each benchmark family contributes equal weight. That weight is split across its eligible tasks, then across the model pairs compared on each task.
Selecting results
The table offers two views. Default / lowest uses the default or lowest available reasoning setting for each model and task. Best observed uses the highest-scoring setting for each task. A model can therefore use different settings across tasks in the best-observed view; the settings are selected after seeing the results.
A model’s result on a task enters the ranking only if at least 80% of its responses were completed. For example, a response that reaches the 16,384-token output limit without producing a final answer counts as incomplete. When a result is missing or excluded, we omit comparisons involving that result.
Calculating the rating
We estimate all model ratings together from their task comparisons. For models \(i\) and \(j\), the Bradley–Terry model gives the following win probability.
For each task \(t\), the observed outcome \(y_{ijt}\) is 1 if model \(i\) wins, 0 if it loses, and ½ for a tie. We choose ratings that minimize the weighted disagreement between these outcomes and the predicted probabilities.
The comparison weight is \(w_{ijt}=\frac{1}{F\cdot T_{f(t)}\cdot P_t}\): \(F\) is the number of benchmark families, \(T_{f(t)}\) is the number of eligible tasks in the task’s family, and \(P_t\) is the number of model pairs compared on that task. The penalty \(\lambda=0.001\) keeps ratings finite. We solve this numerically, then subtract the mean rating so the ratings are centered at zero.
Reading the displayed score
We convert each fitted rating to a score between 0 and 1.
This conversion preserves the ranking. The displayed score estimates the probability of winning against a model rated 0.5 in this table, such as Hy3.