Epi-attention : Adaptive Context-Aware Attention for Dynamic Feature Relevance in Neural Networks
Keywords: Epi-Attention Mechanism, Context-Aware Attention, Dynamic Feature Relevance, Explainability in Machine Learning, Local Correlation Analysis, Contextual Information Integration, Neural Network Interpretability, Feature Importance Re-evaluation, Domain-Specific Knowledge Alignment, Transparent Decision-Making, Context-Driven Feature Selection, Class-Specific Feature Characteristics
TL;DR: This paper presents Epi-Attention, a novel context-aware attention mechanism that dynamically adjusts feature relevance based on contextual information, enhancing model explainability aligned with local correlations.
Abstract: In this paper, we introduce Epi-Attention, a novel context-aware attention mechanism designed to enhance the relevance of features in neural networks by incorporating external contextual information. Unlike traditional attention mechanisms that rely solely on the input sequence, Epi-Attention dynamically adjusts the significance of features based on additional evidence provided by external contexts. This approach allows the model to emphasize or diminish the relevance of specific features, leading to better capture and reflect the internal properties of specific classes. This mechanism provides a nuanced interpretation of feature relevance that aligns with domain knowledge, enabling the model to focus on contextually significant features in a way that resonates with expert understanding. We formalize the problem and present two variants of the proposed mechanism: Scaled Dot-Product Epi-Attention and Self-Epi-Attention, both of which re-evaluate feature importance considering either external or internal information, respectively. By leveraging the dynamic aspect of Epi-Attention, models can highlight local correlations that are characteristic of certain classes, offering a more transparent and interpretable decision-making process compared to global correlations favorized by classical approaches such as Decision trees, Logistic regression and Neural Networks. We demonstrate the efficency of Epi-Attention through three different applications (dynamic feature relevance, processing mixed datatypes and multi-source datasets) with respectively benchmark datasets, including the Wisconsin Breast Cancer, Bank Marketing and ABIDE-II datasets. Our results show significant improvements in model interpretability over traditional models that aligns with domain knowledge. Furthermore, we discuss the potential of Epi-Attention for enhancing explainability in complex machine learning tasks, paving the way for more robust and transparent neural network architectures.
Primary Area: interpretability and explainable AI
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10799
Loading