Energy efficiency in AI data operations with dual-IMC

Investigating Solution to the Von Neumann Bottleneck in AI Models

Introduction to the Research

AI models like ChatGPT are driven by complex algorithms and an insatiable need for data, which they interpret through machine learning. But what are the boundaries of their data-processing capacity? Led by Professor Sun Zhong, a team from Peking University is investigating solutions to the Von Neumann bottleneck, a key barrier to data-processing performance.

Dual-IMC Scheme for Enhanced Machine Learning

In their September 12, 2024 publication in Device, the research team introduced a dual-IMC (in-memory computing) scheme that enhances machine learning speed while significantly boosting the energy efficiency of conventional data operations.

Matrix-Vector Multiplication in Neural Networks

Software engineers and computer scientists utilize matrix-vector multiplication (MVM) operations when designing algorithms to power neural networks, a computational architecture resembling the structure and function of the human brain in AI models.

Understanding the Von Neumann Bottleneck

As datasets expand at an accelerated rate, computing performance frequently encounters bottlenecks due to the disparity between data transfer speeds and processing capabilities, commonly referred to as the Von Neumann Bottleneck. A traditional approach to this issue is the single in-memory computing (single-IMC) scheme, where neural network weights reside in memory, while input data (e.g., images) is provided externally.

Limitations of the Single-IMC Model

The drawback of the single-IMC model is the necessity of switching between on-chip and off-chip data transport, along with the dependence on digital-to-analog converters (DACs), which contribute to increased circuit size and elevated power demands.

Introducing the Dual-IMC Approach

To maximize the capabilities of the IMC principle, the research team introduced a dual-IMC approach, which integrates both the weights and inputs of a neural network within the memory array, enabling fully in-memory data operations.

Testing the Dual-IMC on RAM Devices

The researchers then conducted tests of the dual-IMC on resistive random-access memory (RRAM) devices, focusing on signal recovery and image processing applications.

Key Benefits of the Dual-IMC Scheme

Below are some key benefits of the dual-IMC scheme when utilized for MVM operations:

1. Enhanced efficiency results from conducting computations entirely within memory, reducing time and energy expenditures associated with off-chip dynamic random-access memory (DRAM) and on-chip static random-access memory (SRAM).
2. Computing performance is enhanced by eliminating data movement, which has historically been a limiting factor, through comprehensive in-memory operations.
3. By eliminating digital-to-analog converters (DACs) required in the single-IMC scheme, production costs are reduced, leading to savings in chip area, computing latency, and power consumption.

Conclusion: Implications for Future Computing Architecture

Given the surging demand for data processing in the contemporary digital landscape, the findings from this research may lead to significant advancements in computing architecture and artificial intelligence.

Source