AgentMario: A Multitask Agent for Robotic Interaction With Locker Systems

Haimo Zhang, Ting Lyu, Hong Li, Yishan Liu, Zibo Gao, Yan Yu, Can Wang, Lindsay Wang, Yuejia Zhang, Kunlun He, Kaigui Bian

Abstract: A robotic locker system is needed where automated storage and retrieval of items are required without the need for staff presence. For example, a robot can provide 7/24 available services of medical items pick-up and return, during the COVID-19 pandemic (or under other emergencies). A robotic locker system is usually equipped with a user-friendly intuitive interface (e.g., a touchscreen); meanwhile, the robot desires a multitask agent that can observe, understand, and operate the locker’s interface to complete many tasks of storing/accessing/shipping items. In this article, we study building a multitask agent for interacting with robotic locker systems, called AgentMario. Without human intervention for a specific task, AgentMario decomposes solving a task into learning basic skills (states or user interfaces) and planning over the skills (finding the next state/interface). When the agent is solving a task, our search algorithm walks on the finite state machine graph and generates the proper plans (operation sequence) for the agent. In experiments, our method accomplishes four diverse tasks of picking-up/storing/dropping-off/shipping items. By employing image recognition and mechanical automation technologies, we implement AgentMario with a robot arm to enable contactless operation over the locker’s interface. Experimental results show that our method outperforms baselines in most tasks by a large margin.