Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Learning Invariance with Compact Transforms
Anna T. Thomas, Albert Gu, Tri Dao, Atri Rudra, Christopher Ré
Feb 12, 2018 (modified: Feb 20, 2018)ICLR 2018 Workshop Submissionreaders: everyone
Abstract:The problem of building machine learning models that admit efficient representations and also capture an appropriate inductive bias for the domain has recently attracted significant interest. Existing work for compressing deep learning pipelines has explored classes of structured matrices that exhibit forms of shift-invariance akin to convolutions. We leverage the displacement rank framework to automatically learn the structured class, allowing for adaptation to the invariances required for a given dataset while preserving asymptotically efficient multiplication and storage. In a setting with a small fixed parameter budget, our broad classes of structured matrices improve final accuracy by 5-7% on standard image classification datasets compared to conventional parameter constraining methods.
TL;DR:We leverage the displacement rank framework to automatically learn compact models, adapting to invariances in a given dataset while preserving efficient operations.
Keywords:structured matrices, low displacement rank, learning invariance, model compression
Enter your feedback below and we'll get back to you as soon as possible.