The Research on Algorithms, Algorithm Optimization, and Applications of Large-Scale AI Models

Abstract

In recent years, artificial intelligence systems, such as Large Language Models (LLMs) and multimodal large models, have developed rapidly. With the exponential increase in model parameters, computational complexity, algorithm designs, and engineering implementations face incomparable challenges. How to train models with parameters in the range of hundred billion to trillion with restricted computing capacity and storage resources, efficient fine-tuning, deployment, further latency reduction, and decreased memory footprint have become the most fundamental problems in modern studies. Benefiting from the systematic analysis on respective reviews, different papers, and studies in these years, based on above analysis, in this article, there would be analysis on three different aspects, separately: Firstly, at algorithm foundations, there’s investigation on evolution in optimizers, network architectures, and training patterns. Secondly, in algorithm levels, there’s summary on core areas like memory, resource, efficient parameter fine-tuning, model compression, and other directions. Thirdly, in levels concerning applications, problems, challenges, there’s observation on applications, their respective feasibilities, limitations in modern models with optimized algorithms. By comparison on different studies on these areas, there’s illustration on trends in algorithm optimizations for large-scale AI models, identification on key problems in modern studies, and illustration on different possibilities in new studies. This article looks for providing a systematic literature analysis for researchers, engineers, participants in studies concerning algorithms in large models, optimizations, in AI studies, for necessary references in their studies.