LLM Bench – AI Powered Local LLM Hardware Compatibility and Performance Benchmarking Platform

Author

Ms. Perachiselvi M, Kamaleswaran S N


Abstract

 Running open-weight Large Language Models (LLMs) on consumer hardware is hampered by trial and error: users download 8–30 GB of weights only to meet Out-Of-Memory (OOM) failures, unbudgeted Key-Value (KV) cache overheads, or unsuitable quantization choices. This paper presents LLM Bench, an open-source platform that verifies hardware compatibility and predicts token generation speed before any model is downloaded. A standalone Windows scanner profiles the GPU, VRAM, system RAM, CPU cores, and storage in about 7 s without administrative privileges. A backend engine converts GGUF file sizes (Q4_K_M, Q5_K_M, Q8_0, FP16) into loaded memory footprints with a 15% safety margin and assigns a Recommended, Minimum, or Not Recommended verdict. A physics-informed constrained linear regression model, PhysicsLinear, predicts tokens per second under full GPU offload, CPU fallback, and partial offloading. Under Leave-One-Group-Out (LOGO) cross-validation it reached a Mean Absolute Percentage Error (MAPE) of 42.01%, the lowest among the candidates evaluated. The platform combines a React 18 front end, a FastAPI back end, PostgreSQL on Supabase, and ephemeral 15-minute sessions that avoid persisting identifying user information.


Keywords

LLM inference, GGUF quantization, hardware compatibility, KV-cache, token generation speed prediction, constrained linear regression



Full Text:

Download Paper PDF


References


  • Dettmers, T., et al. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems (NeurIPS), 36.
  • Gerganov, G. (2023). llama.cpp: Port of Facebook’s LLaMA model in C/C++. GitHub Repository.https://github.com/ggerganov/llama.cpp
  • Hendrycks, D., et al. (2021). Measuring massive multitask language understanding (MMLU). International Conference on Learning Representations (ICLR).
  • Meta Open Source. (n.d.). React documentation.https://react.dev
  • Ollama Development Team. (2024). Ollama: Get up and running with large language models locally.https://ollama.com
  • Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12,2825-2830.
  • Ramírez, S. (n.d.). FastAPI documentation.https://fastapi.tiangolo.com
  • (n.d.). Supabase documentation (PostgreSQL and Storage).https://supabase.com/docs
  • Touvron, H., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
  • Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4),65-76.

Share your valuable work from Social Media Buttons