AIKDAP (AI Knowledge Discovery And Analytics Platform) : A Review of Retrieval-Augmented, Multi-Agent and Provenance-Aware AI Research Assistants

Author

Prof. Ms. Madhuri Bhaisare, Harshita Dayma, Vineet Singh, Thoravi Daf, Divya Bhange

Abstract

 Large language model (LLM) assistants are increasingly used to read papers, analyse data and draft reports, yet most ask the model to stay grounded through a prompt instruction, which is a request rather than a guarantee. This paper reviews the literature on making such assistants verifiable. It surveys work on hallucination, retrieval-augmented generation, two-stage retrieval and reranking, attribution and citation checking, multi-agent orchestration, natural-language data analysis, explainability and provenance, and the reliability engineering of LLM services. From this body of work it derives a five-level taxonomy of grounding enforcement that separates prompt-level requests from guarantees implemented in backend control flow, and it compares representative research assistants against that taxonomy. AIKDAP, a project-scoped multi-agent platform built by the authors, serves as a worked case study: its calibrated relevance gate (threshold −2.0) reached a precision and recall of 0.818 on a 70-pair fixture, and four of five out-of-corpus queries were reduced to zero evidence. The paper closes with open problems: multi-domain calibration, benchmark and human evaluation, the gap between attribution and correctness, latency, and the semantic binding of numbers.

Keywords

retrieval-augmented generation, hallucination, grounding, attribution, multi-agent systems, reranking, explainable AI, provenance, research assistants.

Full Text:

Download Paper PDF

References

[1] Z. Ji et al., “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, 2023.

[2] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems (NeurIPS), 2020.

[3] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017.

[4] L. Huang et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv:2311.05232, 2023.

[5] S. Lin, J. Hilton and O. Evans, “TruthfulQA: Measuring how models mimic human falsehoods,” in Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2022.

[6] P. Manakul, A. Liusie and M. J. F. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” in Proc. EMNLP, 2023.

[7] S. Min et al., “FActScore: Fine-grained atomic evaluation of factual precision in long form text generation,” in Proc. EMNLP, 2023.

[8] K. Guu et al., “REALM: Retrieval-augmented language model pre-training,” in Proc. International Conference on Machine Learning (ICML), 2020.

[9] G. Izacard et al., “Atlas: Few-shot learning with retrieval augmented language models,” Journal of Machine Learning Research, vol. 24, 2023.

[10] V. Karpukhin et al., “Dense passage retrieval for open-domain question answering,” in Proc. EMNLP, 2020.

[11] Y. Gao et al., “Retrieval-augmented generation for large language models: A survey,” arXiv:2312.10997, 2023.

[12] N. F. Liu et al., “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, 2024.

[13] A. Asai et al., “Self-RAG: Learning to retrieve, generate, and critique through self-reflection,” in Proc. International Conference on Learning Representations (ICLR), 2024.

[14] S. Es et al., “RAGAs: Automated evaluation of retrieval augmented generation,” in Proc. EACL (System Demonstrations), 2024.

[15] R. Nogueira and K. Cho, “Passage re-ranking with BERT,” arXiv:1901.04085, 2019.

[16] O. Khattab and M. Zaharia, “ColBERT: Efficient and effective passage search via contextualized late interaction over BERT,” in Proc. ACM SIGIR, 2020.

[17] K. Santhanam et al., “ColBERTv2: Effective and efficient retrieval via lightweight late interaction,” in Proc. NAACL, 2022.

[18] N. Thakur et al., “BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models,” in NeurIPS Datasets and Benchmarks Track, 2021.

[19] J. Chen et al., “BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” arXiv:2402.03216, 2024.

[20] C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. ICML, 2017.

 

 

 

Share your valuable work from Social Media Buttons