A Comparative AI-Based Performance Analysis Framework for Large Language Models

Author

V. Petchi Kamala, E Julie Ruth

Abstract

 The AI Benchmark Analyzer is a web-based benchmarking system designed to evaluate and compare multiple Large Language Models under standardized conditions. The platform allows users to provide prompts, select multiple AI models, and execute repeated benchmark tests through different AI providers. The system evaluates model performance using metrics such as Time to First Token (TTFT), response duration, token count, and Tokens Per Second (TPS). The results are aggregated across successful benchmark iterations and presented through dashboards, tables, charts, and recommendation components. The framework supports multi-provider model routing, real-time response streaming, retry handling, and persistent storage, providing a unified approach for analyzing the practical performance of different Large Language Models.

Keywords

Large Language Models, AI Benchmarking, Performance Analysis, TTFT, Token Throughput, Server-Sent Events, Multi-Provider Routing, AI Model Evaluation

Full Text:

Download Paper PDF

References

[1] Wang, S., Long, Z., Fan, Z., Wei, Z., & Huang, X. (2025). Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation. Proceedings of COLING 2025. This work focuses on dynamically evaluating LLMs across different capabilities and making benchmark evaluation more robust.

[2] Qu, X., Heng, Q., Zeng, H., & Liu, X. (2026). An Interpretable and Scalable Framework for Evaluating Large Language Models. arXiv. The study proposes a scalable framework for evaluating LLM capabilities and addresses limitations of simple average-accuracy evaluation.

[3] Siedler, P. D., & Sassoon, J. (2026). Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation. arXiv. This research examines benchmark datasets at the sample level and discusses how different benchmark characteristics affect LLM evaluation.

[4] Mitra, S. (2026). Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation. arXiv. The framework focuses on large-scale LLM evaluation, parallel processing, statistical evaluation, caching, and model comparison.

[5] Benchmarking Large Language Models for Automated Labeling: The Case of Issue Report Classification. (2025). Information and Software Technology. The study compares multiple LLMs using performance, response time, hardware requirements, and response quality, which is particularly close to your project's comparison approach.

[6] Benchmarking Large Language Models for MIMIC-IV Clinical Note Summarization. (2025). Journal of Healthcare Informatics Research. The research compares LLMs using response time, computational cost, and model performance.

[7] Latency-Aware Benchmarking of Large Language Models for Natural-Language Robot Navigation in ROS 2. (2026). This work evaluates LLM inference latency, task success, token efficiency, and overall performance, making it relevant to your latency and efficiency metrics.

[8] Benchmarking Large Language Model-Based Agent Systems for Clinical Decision Tasks. (2026). npj Digital Medicine. The study evaluates LLM systems using accuracy, latency, token usage, and computational/resource requirements.

[9] Large Language Model Benchmarks: A Taxonomy of Capabilities, Scientific Quality Assessment, and Saturation Analysis. (2026). Machine Learning and Knowledge Extraction, 8(6), 141. This review discusses LLM benchmark capabilities and limitations of traditional static, accuracy-focused evaluation.

[10] Benchmarking Terminology for Large Language Model Serving. (2026). This work provides terminology for LLM serving benchmarks, including latency, throughput, resource utilization, and quality metrics.

Share your valuable work from Social Media Buttons