There is no one-to-one mapping from objects to tool concepts. In fact, objects can support diversified tool uses, enabling compositional and flexible functionalities. These tool-like functionalities are mostly context-dependent and vary across scenes. To address this unique property of tool concepts, we propose the Bongard-Tool challenge and formulate the context-dependent tool understanding as a few-shot concept induction problem. Specifically, to build Bongard-Tool, we employ large language models for knowledge building, web crawling, and vision-language models for content retrieval and filtering. We also perform extensive experiments on recent few-shot and meta-learning methods to show the hardship of understanding compositional tool concepts from pure visual perception. We hope to shed light on future studies by introducing Bongard-Tool benchmark as a testbed for building machines that can flexibly understand and use tools. 

Bongard-Tool: Tool Concept Induction from Few-Shot Visual Exemplars

Deep neural networks have achieved high accuracy in many computer vision and natural language processing tasks through their complicated structures and operations. But their formulation is numerical and the computation process cannot be interpreted by humans directly. Therefore they are often criticized for their inability to align with human language. In this project, we aim at discovering emergent language and how it evolves in multi-agent communication games. We design a visual referential game combining the ideas of the referential game (Lazaridou et al. [11]) and the sketch drawing game (Qiu et al. [16]). We consider both continuous and discrete communication types and propose a model which can be efficiently optimized by back-propagation. Experiments conducted on MNIST and CIFAR-10 have shown the efficiency and interpretability of this communication, with t-SNE of visual embedding as visualization. Our code is available at https://github.com/ran1812/pkucore-nonverbal_communication. Our video is available at https://disk.pku.edu.cn:443/link/5C38F27C4323A6AE8B28389A5C851212

A preliminary study of interpretable emergent language in visual referential games

In recent years, artificial agents have made drastic advances in multi-player games such as Go and poker. However, cooperative games with imperfect information are relatively underexplored, despite that such setting is common in daily human-robot interaction. The card game Hanabi is an example, where reasoning about other agents' mental states (\eg, belief and intention) is brought to the foreground. It is a meaningful benchmark because it requires Theory of Mind (ToM) reasoning and challenges an agent’s decision-making ability in a partially observable and cooperative setting. Existing work on Hanabi achieves great self-play results. However, they fail to address these essential challenges in the other-play setting. To fill in the gap, we propose two innovative plug-in modules that can be applied to general RL agents. The Hand Card Information Completion module is designed to model other agents’ mental states and complement environment information. The Goal-Oriented Intrinsic Reward module encourages agents' exploration and collaboration. We believe such attempts will boost performance in this particular game and facilitate human-robot cooperation in a broader range of interactive scenarios. Our code is available at \url{https://github.com/LoYuXr/Hanabi_Plugins}.

Playing Hanabi with ToM and Intrinsic Rewards

Visual grounded \emph{Violation of expectations} (VoE) paradigm is widely used to evaluate the physics learning capability of both humans and machines. It does this by measuring the prediction error, or \emph{surprise}, of a physics learning model in a given scene. Despite intuitive formulation and perfect alignment with developmental psychology, the design of evaluation protocol based on \textit{surprise} score is empirical. We point out the potential risks behind the traditional \textit{surprise} score design and provide a probabilistic explanation of VoE paradigm based on \textit{likelihood ratio theory}. Guided by the theoretical framework, we propose two novel and extensible surprise scores that are theoretically sounded. Furthermore, we implement a simple yet novel baseline based on PredRNN~\cite{wang2017predrnn} that demonstrates the ability to perform physical reasoning through direct \emph{pixel-level prediction}. Our model outperforms a strong \emph{object-level prediction} baseline PLATO, achieving an overall accuracy of 90.0\% on the \texttt{Probe} dataset, compared to 73.4\% for PLATO (with overall accuracy $73.4\%$). Additionally, we conduct experiments using our newly proposed metric.

A Probabilistic Explanation for VoE-based Evaluation

Causal reasoning is considered by many to be the cornerstone of human intelligence. Humans have the ability to develop causal schemas to solve novel problems and establish explanations of latent constraints while interacting with the environment. In this paper, we explore and examine agents' ability to discover causal schemas and generalize to more complex environments based on the OpenLock task, in which participants are required to escape a room by moving levers, and the sequential movements of the correct levers form a causal sequence beginning with either a common-cause (CC) or a common-effect (CE) structure. We develop a deep Q-learning model to solve the simplified OpenLock problem by learning the transition probability between solutions. We argue that the simplification is reasonable since the original task is a complex combination of several diverse sub-tasks and it's difficult to solve all sub-tasks in one single model. We focus on the agents' exploring the latent causal structure of levers, which is challenging and meaningful enough to show the potential of the Reinforcement Learning in causal reasoning. Experiments show that our method learns the underlying causal mechanisms perfectly and pertains a great generalization ability that can transfer the knowledge trained in simple conditions to more complex tasks.

OpenLock Grid: Learning Causal Structure with Deep RL Models

Social norms are a significant part of human society.  It is impossible for our agents to interact with humans in social environments if they are not able to understand and conform to social norms.  Ownership is an essential part of social norms.  The concept of territory, company, and property is all based on ownership.  Recent works represent and infer ownership using simple inference, or represent it with a probabilistic graph and infer it using predicate-based norms.  However, they all link ownership with action rather than intent, ignoring agents' mental states.  This paper presents a Theory-of-Mind (ToM) based model capable of simulating agents' change of intention based on ownership. With the given intention, our agents can choose different actions based on the state of the world.  By combining ownership with intention rather than directly with action, we can study how agents' mental states will change with the understanding of ownership, which can give our agents a stronger generalization ability. 

Learning Ownership based on Theory of Mind

The shape abstraction task is to learn the primitive-based representation of 3D objects. The past work usually learns semantic shape abstraction from manually labeled part annotation. Unsupervised learning for 3D semantic shape abstraction remains difficult because the model cannot understand the affordance of each part and the relationship between parts.   In this project, we reproduce two off-the-shelf well-performing shape abstract models, Cuboid Shape Abstraction via Joint Segmentation and Neural Parts, on the PartNet dataset, and analyze their respective task perspectives and methodical characteristics.  The former method uses the traditional cuboid primitive and tries to map the point cloud to a compact cuboid representation. And the latter proposes a new 3D primitive representation, which realizes homomorphic mapping between the sphere and the target object, and is more flexible than the traditional primitive representation. Our intuition is that Neural Parts will perform better on semantic shape abstraction. We test our intuition experimentally, but only in the chair category. Based on the experimental results, we analyze the advantages and disadvantages of the two.