Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Lasik, Antoni; Pokrywka, Jakub; Grzybowski, Łukasz; Kaczmarek, Jeremi Ignacy; Korzańska, Gabriela; Świeczkowski-Feiz, Janusz; Pastuszek, Oskar; Hoffman, Paulina; Dąbrowski, Jakub Tomasz; Kusa, Wojciech

Abstract

TL;DROn the same Polish medical exam questions, the average accuracy of general-purpose LLMs falls from 62.6% to 25.7% once the question structure is altered to limit elimination and guessing. Standard MCQA scores overstate medical competence.

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases.

To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test medical competence.

We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence, due to format saturation and models' ability to exploit it. To facilitate further research, we make our benchmark publicly available.

Motivation

What We Set Out to Answer

Q1
Does the prompting protocol used in earlier evaluations bias model predictions, and does removing that bias change scores and rankings?
Q2
How much of a model's MCQA accuracy remains when the question format limits elimination, pattern matching and guessing?
Q3
Can models answer expert-level medical questions without answer options, and can an LLM judge grade such answers as medical experts would?
Q4
Can exposure of public exam questions in training data explain high benchmark scores?

Benchmark

An Extended Polish Medical Exams Benchmark

We extend the Polish Medical Exams benchmark (Grzybowski et al., 2025) with two specialist examinations not previously used for LLM evaluation: the Pharmaceutical Specialist Examination (PESF) and the Examination for Laboratory Diagnosticians' Specialization (PESDL). We also add the most recent editions of the existing exams. All items are official five-option questions from the Centre of Medical Examination (CEM) and the Supreme Medical Chamber (NIL), mostly clinical vignettes that require interpreting symptoms, laboratory findings, imaging and treatment options.

38,431exam questions

15k+newly added questions

7exam sets (5 Polish, 2 English)

68%converted to harder formats

21LLMs evaluated

Exam	Editions	Questions	Altered	Coverage
LEKMedical Final Examination	26 (2008–15, 2020–26)	4,728	2,875	61%
LDEKMedical–Dental Final Examination	26 (2008–15, 2020–26)	4,773	3,401	71%
PESBoard Certification Examination (physicians, dentists)	12 (2008, 2012, 2015–20, 2023, 2024)	9,965	7,070	71%
PESDLnewExamination for Laboratory Diagnosticians' Specialization	25 (2008–20)	10,908	7,627	70%
PESFnewPharmaceutical Specialist Examination	22 (2008–20)	1,710	982	57%
LEK-ENOfficial English version of LEK	18 (2013–15, 2020–26)	3,151	1,899	60%
LDEK-ENOfficial English version of LDEK	18 (2013–15, 2020–26)	3,196	2,223	70%
All		38,431	26,077	68%

Table 1. Questions per exam set and the share converted into at least one altered format.

Composition of question types across exam sets — Figure 2. Composition of question types in each exam set after alteration. The share of each format depends on how the questions of a given exam are structured.

from datasets import load_dataset

ds = load_dataset("NASK-PIB/Reassess-Polish-Medical-Exams")

Code 1. Quickstart: loading the raw exam dataset with the datasets library.

Methodology

Evaluation Design

We separate two sources of inflated scores: bias introduced by the prompt, and shortcuts offered by the multiple-choice format itself.

Figure 1. Overview of the methodology: exam scraping, question alteration, constrained generation for closed formats, and LLM-as-a-judge grading of open-ended answers, validated by medical experts.

1Removing prompt-induced bias

The earlier protocol showed the model a sample answer. Comparing predicted and ground-truth answer distributions showed that models favour the example option, and that even the order in which options are listed in the instruction affects predictions. Since the question already lists options A–E, we use a plain instruction with constrained decoding (when applicable).

Previous protocol (PM)

Your task is to provide answers to a medical test for doctors. From all the provided answers A, B, C, D, E select only one. If you are not sure, choose the most probable one. Answer in a manner: Correct answer is B

The sample answer and the listed option order skew predictions.

Bias-mitigated protocol (OM)

Your task is to provide answers to a medical test for doctors. From the provided answers, select the correct one and respond only with that answer. End your answer with a period.

Output constrained to A–E (or statement indices); temperature 0.

Table 4: average accuracy under OM and PM — Table 4. Average accuracy (%) under the previous MCQA protocol (PM) and the bias-mitigated protocol (OM), over all seven exam sets.

2Four structural alterations

Eligible questions are identified with regular expressions on the Polish text and mirrored on the parallel English sets. When a question qualifies for several formats, the open-ended conversion takes priority.

Multiple Statements

Composite options such as "1, 3" are removed; the model returns the exact set of true statements.

Multiple Answers

Meta-options such as "A and B are correct" are removed; the model selects every correct option.

Correct Answer Substitution

The correct option is replaced with "none of the answers is correct", testing abstention.

Open-Ended

Options are removed entirely; the free-text answer is graded by an LLM judge.

3Scoring and models

MS and MA answers count as correct only on an exact set match. Open-ended answers are graded by DeepSeek-V4-Pro prompted as an oral examiner, and the judge is validated against six medical experts (see below). All generations use temperature 0; runs used vLLM on NVIDIA A100 GPUs.

General-purpose LLMs

Llama-3-8BQwen3.5-9BGemma-3-12BGemma-3-27BQwen3.5-35B-A3BLlama-3.3-70BQwen3.5-122B-A10B

Medical LLMs

MedGemma-4BMediPhi-InstructBioMistral-7BBioMistral-7B-DAREMedGemma-27BOpenBioLLM-70BMeditron3-70B

Polish LLMs

Bielik-4.5BBielik-11BPLLuM-12B-InstPLLuM-12B-ChatPLLuM-12B-NC-InstPLLuM-12B-NC-ChatPLLuM-70B-Inst

Results

Key Findings

General-purpose LLMs

Original

62.6

Altered

25.7

Medical LLMs

Original

44.9

Altered

17.3

Polish LLMs

Original

41.0

Altered

11.4

Original MCQA Altered (MS, MA, AS)

Average accuracy (%) on the same questions before and after structural alteration, by model group (seven models each). Values from Table 5.

Table 5: accuracy on original and altered questions — Table 5. Average accuracy on altered questions and on the original versions of the same questions, with the change on English (LEK-EN, LDEK-EN) and Polish (LEK, LDEK, PES, PESDL, PESF) exams.

The prompt protocol biases MCQA scores. With a plain instruction and constrained decoding, 18 of 21 models score higher than under the earlier protocol, by up to 38.9 pp (Meditron3-70B). The ranking of Polish models also changes: PLLuM-70B-Inst overtakes Bielik-11B.
Altered questions reveal large gaps. Each model group keeps only 28% to 41% of its original accuracy on the same questions. Even the strongest model, Qwen3.5-122B, loses 28.4 pp on English and 31 pp on Polish exams.
Open-ended answering remains difficult. Only two models exceed 50% average accuracy without options (Qwen3.5-122B 63.1%, Qwen3.5-35B-A3B 52.3%). Medical fine-tuned models perform on par with general Polish LLMs: Meditron3-70B 46.5% and MedGemma-27B 43.3%, against PLLuM-70B-Inst 46.4% and Bielik-11B 42.6%.
One apparent improvement is an artefact. MediPhi-Instruct is the only model that gains under alteration, and only on Polish exams. The gain comes from Correct Answer Substitution, where it reaches 57–61% on Polish exams but 16–20% on English ones, which points to a bias towards "none of the answers is correct" rather than better knowledge.
General-purpose models lead. Qwen3.5-122B has the best average accuracy on original questions (75.1%), altered questions (47.0%) and open-ended questions (63.1%).

Table 6: open-ended accuracy per exam — Table 6. Accuracy on open-ended questions per exam, graded by DeepSeek-V4-Pro.

Validation

Can an LLM Grade Expert-Level Medical Answers?

Six medical experts each judged 200 (question, generated answer, reference answer) triplets from four models, with 15 control items each. After quality control, including removal of one outlier annotator, 743 judgements remained.

6medical experts

743valid expert judgements

0.921Fleiss' κ, expert vs expert

0.649Cohen's κ, experts vs LLM judge

Agreement with the LLM judge is substantial but well below agreement among experts. A senior physician reviewed 74 cases in which an expert contradicted a unanimous panel of seven LLM judges and identified recurring patterns:

Pattern in the generated answer	Medical experts	LLM judges
Errors in redundant content	Rejected all 14 affected answers	Accepted if the expected content was present
Valid alternative interpretation of the open-ended question	Accepted 19 of 20	Usually rejected if the answer key was not matched
Non-existent medical terms, including Polish–English hybrids	Rejected 12 of 14	Usually not penalised
Incorrect spelling or notation of medical terms	Rejected 9 of 10	Usually not penalised

Experts weigh clinical and terminological errors; LLM judges rely on overlap with the reference answer. Underspecified detail and misuse of existing terms showed no consistent tendency.

Data contamination

Is It Memorisation?

The exams are public, so we tested for training-data exposure in three ways. Contamination estimates do not track the performance drops under alteration, so memorisation does not explain the gap.

Data Contamination Quiz

For six models on LEK and LDEK (2015, 2020 and 2026 editions), estimated contamination ranges from 3.3% to 26.5%. The 2026 edition, which could not have leaked into training data, is not estimated lower than the older ones.

Web search audit

For LEK (autumn 2023), 169 of 194 retrieved pages contained the question, but only 36 also showed the answer. For PES, no top result contained both question and answer.

Temporal analysis

Five models evaluated on LEK-EN and LDEK-EN editions from 2020 to 2026 show no decline on editions released after their training cut-off.

Estimated contamination ranges by model across exam editions — Figure 3. Estimated contamination ranges (Data Contamination Quiz) for six models on the 2015, 2020 and 2026 spring editions of LEK and LDEK.

Contributions

Summary

Extended benchmark

38,431 questions from seven exam sets, including the first use of the pharmacy (PESF) and laboratory diagnostics (PESDL) specialist exams for LLM evaluation.

Harder evaluation protocol

A bias-mitigated prompting setup, four structural alterations covering 68% of questions, and an open-ended setting graded by an expert-validated LLM judge.

Evaluation of 21 LLMs

General-purpose, medical and Polish models compared on MCQA and underlying knowledge, with three complementary contamination analyses.

Citation

BibTeX

If you use this benchmark or evaluation protocol, please cite:

@article{lasik2026reassessinghighperformingllmspolish,
  title     = {Reassessing High-Performing {LLMs} on {Polish} Medical Exams: True Competence or Bias-Driven Performance?},
  author    = {Antoni Lasik and Jakub Pokrywka and Łukasz Grzybowski and Jeremi Ignacy Kaczmarek and Gabriela Korzańska and Janusz Świeczkowski-Feiz and Oskar Pastuszek and Paulina Hoffman and Jakub Tomasz Dąbrowski and Wojciech Kusa},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  url       = {https://arxiv.org/abs/2606.12250}
}

This benchmark extends Polish Medical Exams (Grzybowski et al., 2025). If you use the original LEK, LDEK or PES splits, please also cite it.

@inproceedings{grzybowski2025polish,
  title     = {{Polish-English} Medical Knowledge Transfer: A New Benchmark and Results},
  author    = {Grzybowski, {\L}ukasz and Pokrywka, Jakub and Ciesi{\'o}{\l}ka, Micha{\l} and Kaczmarek, Jeremi Ignacy and Kubis, Marek},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  pages     = {9042--9063},
  year      = {2025},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2025.findings-emnlp.480/}
}