ADAPTIVE FAULT-TOLERANT SCHEDULING FOR THROUGHPUT AND ENERGY EFFICIENCY IN TRAINING DISTRIBUTED MIXTURE-OF-EXPERTS LARGE AI MODELS

Author

KIRORI ZACHARY, JAMES MWIKYA


Abstract

 The rapid growth of large artificial intelligence models has led to an increased need for distributed GPU systems for model training. At the same time, failures, stragglers, network congestion, and imbalanced workloads significantly reduce the efficiency of training and increase energy consumption. These issues are particularly acute in the case of Mixture-of-Experts (MoE) due to dynamic routing which causes uneven workloads and large amounts of communication between GPUs. This desktop review aims to assess how adaptive fault tolerant scheduling can optimize throughput and energy in the context of distributed MoE training. The literature search was focused on articles published in 2021-2026, including peer-reviewed manuscripts, conferences, and technical reports. Overall, the reviewed literature indicates that adaptive scheduling, dynamic recovery, and energy-aware resource allocation can make large-scale AI training more reliable and efficient. However, most of the aforementioned literature discusses specific aspects of the problem. Therefore, this desktop review concludes that more research is needed in order to develop scheduling algorithms which simultaneously address workload variations, failures, throughput, and energy efficiency.



Keywords

Distributed training, Mixture-of-Experts, adaptive scheduling, fault tolerance, training throughput, energy efficiency.



Full Text:

Download Paper PDF


References


  • Athlur, S., Saran, N., Sivathanu, M., Ramjee, R., & Kwatra, N. (2022). Varuna: Scalable, low-cost training of massive deep learning models. Proceedings of the 17th European Conference on Computer Systems (EuroSys 2022).
  • Chung, J.-W., Gu, Y., Jang, I., Meng, L., Bansal, N., & Chowdhury, M. (2024). Reducing energy bloat in large model training. Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP 2024).
  • Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 1–39.
  • Gupta, T., Krishnan, S., Kumar, R., Vijeev, A., Gulavani, B., Kwatra, N., Ramjee, R., & Sivathanu, M. (2024). Just-In-Time checkpointing: Low cost error recovery from deep learning training failures. Proceedings of the 19th European Conference on Computer Systems (EuroSys 2024).
  • Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., Chau, J., Cheng, P., Yang, F., & Xiong, Y. (2022). Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 4, 269–287.
  • Jang, I., Yang, Z., Zhang, Z., Jin, X., & Chowdhury, M. (2023). Oobleck: Resilient distributed training of large models using pipeline templates. Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP 2023).
  • Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V. A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M. (2021). Efficient large-scale language model training on GPU clusters using Megatron-LM. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2021).
  • Nie, X., Zhao, P., Miao, X., Zhao, T., & Cui, B. (2022). HetuMoE: An efficient trillion-scale mixture-of-expert distributed training system. arXiv preprint arXiv:2203.14685.
  • Qiao, A., Choe, S. K., Subramanya, S. K., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., & Xing, E. P. (2021). Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning. Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2021).
  • Shi, S., Pan, X., Wang, Q., Liu, C., Ren, X., Hu, Z., Yang, Y., Li, B., & Chu, X. (2024). ScheMoE: An extensible mixture-of-experts distributed training system with tasks scheduling. Proceedings of the 19th European Conference on Computer Systems (EuroSys 2024), 236–249.
  • Thorpe, J., Zhao, P., Eyolfson, J., Qiao, Y., Jia, Z., Zhang, M., Netravali, R., & Xu, G. H. (2023). Bamboo: Making preemptible instances resilient for affordable training of large DNNs. Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023).
  • You, J., Chung, J.-W., & Chowdhury, M. (2023). Zeus: Understanding and optimizing GPU energy consumption of DNN training. Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023), 119–139.
  • Zeng, Y., Huang, C., Mei, Y., Zhang, L., Su, T., Ye, W., Shi, W., & Wang, S. (2025). EfficientMoE: Optimizing mixture-of-experts model training with adaptive load balance. IEEE Transactions on Parallel and Distributed Systems, 36(4), 677–688.

 

Share your valuable work from Social Media Buttons