To Transfer or Not to Transfer: Suppressing Concepts from Source Representations
Abstract: With the proliferation of large pre-trained models in various domains, transfer learning has gained prominence where intermediate representations from these models can be leveraged to train better (target) task-specific models, with possibly limited labeled data. Although transfer learning can be beneficial in many applications, it can transfer undesirable information to target tasks that may severely curtail its performance in the target domain or raise ethical concerns related to privacy and/or fairness. In this paper, we propose a novel approach for suppressing the transfer of user-determined semantic concepts (viz. color, glasses, etc.) in intermediate source representations to target tasks without retraining the source model which can otherwise be expensive or even infeasible. Notably, we tackle a bigger challenge in the input data as a given intermediate source representation is biased towards the source task, thus possibly further entangling the desired concepts. We evaluate our approach qualitatively and quantitatively in the visual domain showcasing its efficacy for classification and generative source models. Finally, we provide a concept selection approach that automatically suppresses the undesirable concepts.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: We have updated the manuscript to incorporate changes suggested by the Action Editor. Specifically,
Q1. I would suggest that the authors include a section discussing the limitations or weaknesses (2-3) and possible future actions.
A1. We have added a separate section discussing limitations/weaknesses and potential future directions
Q2. Regarding weakness (3), I would suggest a proof-of-concept experiment on tabular data. Specifically, it may not be necessary to consider representation learning, but only the operation on the concepts.
A2. We thank the AE for suggesting this experiment. We demonstrate such transfer using a proof of concept experiment on a tabular dataset in Appendix A.4.
Assigned Action Editor: ~changjian_shui1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1620
Loading