Neural Computers

Mingchen Zhuge, Changsheng Zhao, Haozhe Liu, Zijian Zhou, Shuming Liu, Wenyi Wang, Ernie Chang, Gael Le Lan, Junjie Fei, Wenxuan Zhang, Yasheng SUN, Yunyang Xiong, Zechun Liu, Zhipeng Cai, Yining Yang, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber

Keywords: Neural Computers, World Models, Computer Use, Agents, I/O Traces

TL;DR: We propose Neural Computers as a practical step toward Completely Neural Computers (CNCs), prototyped with today’s video models and evaluated on CLI and GUI interaction.

Abstract: We train neural networks (NNs) to learn the dynamics of traditional computers (TCs). The NN's inputs include screenshots, instructions (natural language or program commands), and user actions such as keyboard typing and mouse movements/clicks. Without relying on internal program state, we investigate whether a unified neural system can learn, simulate, and operate a computer interface directly from raw I/O data. We present initial steps towards a completely neural computer (CNC): in our first experiments, our NNs learn to emulate TCs in both CLI and GUI settings. We derive practical design considerations for building more and more general CNCs.

Submission Number: 97