In-memory Vector Symbolic Architectures

Geethan Karunaratne

Abstract: The field of Artificial Intelligence (AI) has achieved enormous progress in the past decade thanks primarily to deep neural network architectures and specialized hardware that support training the models within a reasonable time. However, since then a trend has emerged where, for solving increasingly difficult cognitive tasks, the model complexity in terms of the number of parameters and energy spent on training the models and the size of the datasets used for benchmarking has grown steadily every year. Yet, the capabilities of each model are limited to a narrow task such as classification or translation. The validity of this approach in building bigger and more power-hungry models needs to be critically questioned. For example, the human brain originally conceived from a tip of a 3-millimeter-long neural tube slowly grows over a period of approximately 20 years into a device that can perform a wide variety of complex cognitive tasks. It operates at a frugal power budget as low as 20W, requiring fewer examples than an AI model to learn new concepts. The emerging brain-inspired computing paradigm known as vector symbolic architecture (VSA) offers interesting avenues to advance the field of AI along the path of how the human brain works. For starters, it only requires a few examples for training and does not entail computationally expensive iterative gradient updates. It however requires data to be represented in extremely high dimensional vectors, having dimensions typically in the order of thousands, certainly larger than the size of the data path of any classical computer. Because of its necessity to manipulate these high dimensional (HD) vectors, the intensity of memory accesses dominates the computational intensity, creating a bottleneck in conventional von Neumann computing architectures. In-memory computing (IMC) on the other hand is a type of so-called non-von Neumann computing architecture that can bring down the cost of data movement between the processing unit and memory unit by keeping the majority of data stationary in memory and executing parallel computations using enhanced peripheral circuits. At the core of IMC, computations are performed on a noisy analog fabric by exploiting the laws of physics. Thus results are not guaranteed to be deterministic, as they are reproducible only in a probabilistic sense, leading to an interesting set of opportunities but at the same time challenges that need to be carefully considered. This doctoral thesis investigates the usability of vector symbolic architectures on IMC hardware. It entails the following key contributions: 1) Design of VSA and neuro-VSA architectures for several applications, namely categorical and numerical sequence encoding, few-shot learning, and few-shot continual learning. In each application, the models are engineered by taking the strengths and limitations of in-memory computing architecture into consideration. 2) Simulation of the architectures using IMC models and experiment with IMC prototype chips to characterize the model performance in terms of accuracy and robustness. 3) Design of full systems with CMOS peripherals complementing the IMC tiles and estimate the power, performance, and area of the proposed systems. The following is a brief overview of what is presented in each chapter. Chapter 1 provides the background and motivation for implementing VSA models on IMC hardware. Chapter 2 introduces the concept of in-memory hyperdimensional computing, where it is shown how VSA modules can be mapped to IMC hardware taking categorical symbol sequence classification as an example. Chapter 3 generalizes this idea to include encoding for spatio-temporal signals which, encompass not only categorical but also numerical symbols. To directly work with raw data, in Chapter 4, a novel neuro-VSA architecture and training methodology are proposed with an application for the few-shot learning problem, backed up with simulation and experimental results from IMC models and real IMC hardware, respectively. Chapter 5 and Chapter 6 present an extended version of the neuro-VSA architecture for the few-shot continual learning problem with an emphasis on software and hardware innovations respectively. Finally, Chapter 7, provides conclusions to the thesis and outlook for future research directions.