Evaluating an On-Premises Retrieval-Augmented Generation System for Health Records Retrieval: Retrieval Performance, Response Latency and Perceived Usability in Kenyan Level 5 Hospitals

Author

Kevin Comba Gatimu, James Mwikya, Stanley Mogaka


Abstract

 Retrieval-Augmented Generation (RAG) systems are increasingly proposed for clinical information retrieval, yet evaluations are overwhelmingly conducted on cloud infrastructure in high-income settings, reported gains from advanced retrieval techniques are seldom subjected to ablation, and automated judges are routinely drawn from the same model as the system under test. This article reports the third objective of a design science study: evaluation of the retrieval performance, response latency and perceived usability of a developed On-Premises RAG system for health records retrieval in Level 5 hospitals in Kenya. Twelve curated queries spanning six classes, each paired with a ground-truth reference computed by structured query over a synthetic corpus of 1,603 indexed chunks, were executed through the live pipeline under three configurations, being the full advanced pipeline, a naive vector-search baseline and a no-rerank ablation, with retrieval depth held at eight chunks throughout. Two deterministic, judge-free measures were computed for all three configurations, and three judged measures were computed for the naive configuration by Qwen2.5-7B-Instruct, a judge independent of the Phi-4 Mini Instruct (3.8B) generator in both instance and model family and served locally, so that no health-record content left the machine. Contrary to expectation, the naive baseline outperformed both advanced configurations on key-fact context precision, at 0.687 against 0.633 and 0.591 over the nine queries for which the measure is defined, and on key-fact accuracy, at 0.491 against 0.400 and 0.275 over all twelve. It was also the only configuration to meet the thirty-second design target, at a mean of 13.99 seconds against 34.33 seconds for the no-rerank ablation. Judged context recall of 0.533 over ten defined scores and faithfulness of 0.599 over eleven, the latter below the criterion of 0.80 adopted by the study, together indicate that incomplete retrieval rather than unsupported generation was the binding constraint. Performance varied sharply by class: catalogue lookup recorded key-fact accuracy of 0.875 while aggregate counting recorded 0.167 and identifier-keyed per-patient retrieval 0.244, the last with judged context recall of 0.000. Thirteen intended users at the two study hospitals recorded completion of between 84.6 and 100.0 per cent of six task scenarios, rated source verifiability highest at a median of 5, and returned a mean System Usability Scale total of 80.6; every one of them identified per-patient retrieval as routine, which is the class on which the system performed worst. The study concludes that added pipeline complexity conferred no measured benefit at this corpus scale, and that aggregate and identifier-keyed questions require structured and exact-match retrieval alongside semantic search.


Keywords

retrieval-augmented generation; RAGAS; ablation analysis; response latency; System Usability Scale; on-premises deployment; health records retrieval; Kenya



Full Text:

Download Paper PDF


References


  • Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025). Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6), e0000877. https://doi.org/10.1371/journal.pdig.0000877
  • Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
  • Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.
  • Data Protection (General) Regulations, 2021, Legal Notice No. 263. (2021). Republic of Kenya. Kenya Gazette Supplement No. 186. https://www.odpc.go.ke/wp-content/uploads/2023/05/Kenya-Gazette-Supplement-No.-186-Data-Protection-General-Regulations-2021.pdf
  • Data Protection Act, No. 24 of 2019. (2019). Republic of Kenya. Kenya Gazette Supplement. https://www.odpc.go.ke/wp-content/uploads/2023/01/Kenya-Data-Protection-Act-2019.pdf
  • Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3900–3906). https://arxiv.org/abs/2309.15217
  • Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2024). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. https://arxiv.org/abs/2312.10997
  • Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024). Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 517–545). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.29
  • Korom, R., Kiptinness, S., Adan, N., Said, K., Ithuli, C., Rotich, O., Kimani, B., King'ori, I., Kamau, S., Atemba, E., Aden, M., Bowman, P., Sharman, M., Soskin Hicks, R., Distler, R., Heidecke, J., Arora, R. K., & Singhal, K. (2025). AI-based clinical decision support for primary care: A real-world study. arXiv preprint arXiv:2507.16947. https://arxiv.org/abs/2507.16947
  • Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in neural information processing systems 33 (pp. 9459–9474). Curran Associates.
  • Oche, A. J., Folashade, A. G., Ghosal, T., & Biswas, A. (2025). A systematic review of key retrieval-augmented generation (RAG) systems: Progress, gaps, and future directions. arXiv preprint arXiv:2507.18910. https://arxiv.org/abs/2507.18910
  • Park, S., Kim, J., Lee, H., & Choi, Y. (2025). Enhancing clinical decision support with adaptive iterative self-query retrieval for retrieval-augmented large language models. PMC. https://pmc.ncbi.nlm.nih.gov/articles/PMC12383471/
  • Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302
  • Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025). Play favorites: A statistical method to measure self-bias in LLM-as-a-judge. arXiv preprint arXiv:2508.06709. https://arxiv.org/abs/2508.06709
  • Wampler, D., Nielson, D., & Seddighi, A. (2025). Engineering the RAG stack: A comprehensive review of the architecture and trust frameworks for retrieval-augmented generation systems. arXiv preprint arXiv:2601.05264. https://arxiv.org/abs/2601.05264
  • Wataoka, K., Takahashi, T., & Ri, R. (2024). Self-preference bias in LLM-as-a-judge. arXiv preprint arXiv:2410.21819. https://arxiv.org/abs/2410.21819
  • Wołk, K. (2025). Evaluating retrieval-augmented generation variants for clinical decision support: Hallucination mitigation and secure on-premises deployment. Electronics, 14(21), Article 4227. https://doi.org/10.3390/electronics14214227
  • Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024). Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178. https://arxiv.org/abs/2402.13178
  • Yang, X., Sun, K., Xin, H., Sun, Y., Bhatt, N., Jiang, X., Sun, S., Dong, L., & Choi, Y. (2024). CRAG: Comprehensive RAG benchmark. arXiv preprint arXiv:2406.04744. https://arxiv.org/abs/2406.04744
  • Zhang, Y., Li, X., & Chen, H. (2025). Retrieval augmented generation evaluation in the era of large language models: A comprehensive survey. arXiv preprint arXiv:2504.14891. https://arxiv.org/abs/2504.14891
  • Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in neural information processing systems 36 (pp. 46595–46623). Curran Associates.

Share your valuable work from Social Media Buttons