Cross Attention for Oddly Shaped Data and Applications in Ionospheric Modeling

Liam Bruce Smith, Morris Cohen

Keywords: Ionosphere, Cross Attention, Attention, Transformers, Electron Density, Total Electron Content, TEC, Misshapen Data, Variable Sequence Length

Abstract:

It is desirable to have models of many physical phenomena, yet often data for these phenomena are oddly structured. These structures, such as ungridded and arbitrary length data prevents the use of many types of machine learning techniques, such as feed-forward neural networks. It is thus quite desirable to be able to move this data into a fixed size and shape for easier data ingest. We propose a method of using cross attention to do this. An example of oddly shaped data is Total Electron Content (TEC), or the vertical integral of electron density in the atmosphere. TEC data is calculated using both the position of a satellite and a position on the surface of Earth, giving a non-fixed location per sample. This leads to a splattering of points on the globe where measurements exist that change in shape and amount each time step. We apply our technique to TEC in an autoregressive approach. This allows us to both obtain an embedding describing the global TEC and create completed TEC maps, filling in where measurements are not taken. The global embedding can then be further used in other models.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13180