A Conceptual Approach to Explainable Object and Action Classification

Stephanie Stacy, Alfredo Gabaldon, Guy Ben-Yosef, Idan Tankel, Marc Tomlinson, Sharon G. Small, Ting Liu, Peter Tu

Abstract: The Automated Domain-Understanding and Collaborative Agency (ADUCA) system allows for the acquisition of conceptual models of both objects and actions. With exposure to a small number of training exemplars, the ADUCA system constructs a canonical model based on semantically meaningful attributes and relationships that are common to the training data. At test time, the ADUCA system is first tasked with describing an image or video in generic semantic terms which constitutes a bottom up description of the input. The ADUCA system then attempts to determine which if any of its canonical class models is most consistent with the bottom up description. When prompted, the ADUCA system can provide a grounded explanation for its decision as well as answer questions such as why a different conclusion was not drawn. When a mistake has been identified, the ADUCA system attempts to reason over why the mistake was made as well as propose how best to modify its canonical model so as to avoid making similar mistakes in the future. This allows for a form of continuous learning.