Wavelet and Optical Features Sparkling NLP
Abstract: Computational resources are vital in natural language processing (NLP) development. Since the physical limit of transistors is approaching a saturation point due to the outspace of Moore's Law and Dennard scaling, we look for alternative computing power from optical devices.
As an initial step in this research direction, we facilitate feature extraction using optical computing and integrate optical extracted features to enhance NLP baselines on conventional electronic GPUs. Unlike another one of a kind of features extracted from Transformer, such as lexical embeddings, we extend the feature space beyond traditional embeddings using Wavelet functions that can run on optical toolkits. These extracted features, alongside the original input text, provide additional information that enhances model performance in NLP tasks. We employ two different feature extraction methods: a direct approach involving Wavelet or FFT transformations, and a novel method employing optical computing for NLP feature extraction. Our evaluation encompasses fice GLUE tasks - CoLA, SST-2, STSB, MRPC, and RTE - and reveals a notable improvement of up to +2.8% in classification accuracy.
Paper Type: Short
Research Area: Machine Learning for NLP
Research Area Keywords: NLP; machine learning; optical computing; Wavelet function; FFT
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models
Languages Studied: English
Submission Number: 1511
Loading