Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical Consultation
ILlama builds a searchable collection of medical subgraphs and retrieves relevant evidence for each question.
Figure 1 from Park et al. (EMNLP 2025), cropped from the paper and converted to WebP. Source ↗ · License ↗
Abstract
ILlama uses a medical knowledge graph as a source of evidence for retrieval-augmented language models. It converts local graph neighborhoods into searchable text, retrieves and reranks the documents for each patient-style question, and gives the selected evidence to the generator. In the paper’s evaluation, ILlama 8B reached semantic similarity F1 of 0.884 on HealthCareMagic and 0.871 on iCliniq. Paper
Why use a medical knowledge graph?
Clinical questions often describe a combination of symptoms in everyday language. ILlama studies whether explicitly retrieving relationships among symptoms, conditions, and treatments can improve a model’s responses to these questions. Paper
How ILlama works
- Build searchable documents from a graph. The system takes a relationship from a UMLS-based medical knowledge graph, gathers its nearby connections, and turns the resulting subgraph into a short document.
- Retrieve and rerank the evidence. A domain-tuned embedding model retrieves candidate documents through FAISS. A cross-encoder scores the question and each candidate together.
- Generate a response with the retrieved context. A fine-tuned language model uses the top-ranked documents as context for its answer.
The authors also generate patient-style questions from the subgraph documents and filter them for relevance before training the retriever and reranker. Method, §3
Main results
The paper compares generated answers against reference responses on two medical consultation datasets using semantic similarity F1. Paper, Table 2
| Model | HealthCareMagic F1 | iCliniq F1 |
|---|---|---|
| ChatDoctor | 0.846 | 0.845 |
| ILlama 8B | 0.884 | 0.871 |
| IGemma 27B | 0.897 | 0.881 |
What changes when retrieval changes?
Graph neighborhoods and reranking
In the paper’s component analysis, removing reranking from the UMLS-based ILlama 8B pipeline lowered F1 from 0.884 to 0.857. The paper also found that retrieving focused subgraphs outperformed its full-graph retrieval comparison. Paper, Table 3 and Figure 7
Scope
The paper identifies narrow disease and language coverage, the risk of an outdated knowledge graph, and the need for evaluation of clinical safety and decision quality beyond answer-similarity metrics. Limitations
Resources and citation
BibTeX
@inproceedings{park-etal-2025-leveraging-knowledge,
title = {Leveraging Knowledge Graph-Enhanced {LLM}s for Context-Aware Medical Consultation},
author = {Park, Su-Hyeong and Kim, Ho-Beom and Park, Seong-Jin
and Aliyeva, Dinara and Kim, Kang-Min},
booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
month = nov,
year = {2025},
address = {Suzhou, China},
publisher = {Association for Computational Linguistics},
pages = {30459--30475},
doi = {10.18653/v1/2025.emnlp-main.1549},
url = {https://aclanthology.org/2025.emnlp-main.1549/}
}