Comparative study on large model training methods based on distributed parallel and memory-saving mechanisms

The integration of deep learning—particularly Transformer-based models—has profoundly reshaped scientific research paradigms. In domains such as drug discovery, weather forecasting, and protein structure prediction, these models enable the extraction of complex patterns from large-scale data. However, as model resolution and parameter counts continue to grow, GPU memory consumption increases rapidly, creating a “memory wall” that confines large-scale training to costly, high-end hardware and limits broader adoption in scientific research.

To address this challenge, the authors present a comprehensive optimization framework spanning multiple layers. At the algorithmic level, approaches such as mixed-precision training and model compression techniques, including quantization and pruning, reduce memory usage while preserving model accuracy. At the system level, distributed training strategies and memory management frameworks, such as Zero Redundancy Optimizer (ZeRO) and memory swapping, improve resource utilization. In addition, hardware–software co-design further exploits architectural characteristics to enhance efficiency for scientific workloads.

AlphaFold 2 serves as a representative case study, illustrating how techniques like chunking and gradient recomputation alleviate extreme memory pressure when modeling long protein sequences. Together, these strategies offer a practical roadmap for training large, high-resolution scientific models under realistic hardware constraints, lowering the entry barrier and supporting scalable, cost-effective AI for science.

Title:

A survey on memory-efficient transformer-based model training in AI for science

Journal Name: Frontiers of Computer Science

DOI: 10.1007/s11704-025-50302-6

URL: https://scholarcast.cast.org.cn/top-news/2104400142280921088