GPU Memory Bandwidth and the MemoryWall on Computing Performance
DOI:
https://doi.org/10.61173/stecep16262Keywords:
Graphics processing unit (GPU), memory wall, hierarchical memory architecture, memory bandwidth, high-performance computingAbstract
Graphics processing units (GPUs) have evolved from specialized graphics accelerators into essential computing platforms for artificial intelligence (AI) and high-performance computing (HPC). However, continued improvements in computational throughput have intensified the memory-wall problem, as the growth of data-transfer capability increasingly lags behind that of processing performance. This review examines the hierarchical memory architecture of modern GPUs, including registers, caches, shared memory, and global memory, with particular emphasis on the role of data locality and memory-access efficiency in overall system performance. The influence of memory behavior on representative workloads, including general matrix multiplication (GEMM) and convolutional neural networks (CNNs), is further discussed in terms of arithmetic intensity, data reuse, bandwidth utilization, and access patterns. Common optimization approaches, such as tiling, memory coalescing, data reuse, and hierarchical memory management, are also reviewed. Overall, mitigating the GPU memory wall requires coordinated optimization of memory architecture, data-access patterns, and software strategies to improve data movement efficiency and sustain computational throughput in increasingly data-intensive workloads.
References
Gholami Amir, Yao Zhewei, Kim Sehoon, et al. AI and Memory Wall. IEEE Micro, 2024. 44(3): 33-39. https://doi.org/10.1109/MM.2024.3373763.
Maurya A, Ye J, Rafique M M,et al. Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers. arXiv: 2024. https://doi.org/10.48550/arXiv.2406.10728.
Bitalebi H, Safaei F, Ebrahimi M. Nearest data processing in GPU. Microelectronics Journal, 2024, 44(000). https://doi.org/10.1016/j.suscom.2024.101047.
Jia Shiwei, Tian Ze, Ma Yueyuan, et al. A Survey of GPGPU Parallel Processing Architecture Performance Optimization. 2021 IEEE/ACIS 20th International Fall Conference on Computer and Information Science (ICIS Fall), Xi'an, China, 2021: 75-82. https://doi.org/10.1109/ICISFall51598.2021.9627400.
Davies M, Crago N, Sankaralingam K, et al. Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need. arXiv:2507.14397v1, 2025. https://doi.org/10.48550/arXiv.2507.14397.
Hobbhahn M, Heim L, Aydos G. Trends in machine learning hardware. Published online at epoch.ai. Retrieved from 'https://epoch.ai/publications/trends-in-machine-learning-hardware' [online resource], Accessed 1 Sept 2026.
Hanindhito B, John L K, Accelerating ML Workloads using GPU Tensor Cores: The Good, the Bad, and the Ugly. ICPE '24: Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering, 2024: 178-189. https://doi.org/10.1145/3629526.3653835.
Zhao Xia, Wang Limin, Zhang Yufei, et al. A review of convolutional neural networks in computer vision. Artif Intell Rev, 2024, 57: 99. https://doi.org/10.1007/s10462-024-10721-6.
Liu Ze, Lin Yutong, Cao Yue, et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, 9992-10002. https://doi.org/10.1109/ICCV48922.2021.00986.
Wang Xueying, Li Shigang, Qian Hao, et al. OptiFX: Automatic Optimization for Convolutional Neural Networks with Aggressive Operator Fusion on GPUs. ACM Transactions on Architecture and Code Optimization, 2025, 22(2): 1-27. https://doi.org/10.1145/3716876.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 by the authors.
This work is licensed under a Creative Commons Attribution 4.0 International License.
