\title{CrysMMNet -  Supplementary Materials}
\onecolumn %% Turn this off if single column is desired for the supplement
\maketitle
\appendix
\section{Qualitative analysis of the attention layers}
\label{supp}uai2023-supplimentary
In this section, we aim to visualize and understand attention in different tokens in the material description and we perform a qualitative analysis of the attention layer in MatSciBert.  We utilized the standard BertViz tool \footnote{https://github.com/jessevig/bertviz} to analyze and visualize the attention scores in the MatSciBert Model. We present a case study of  the textual data of $FeH_8(ClO_2)_2$ in figure \ref{fig:attn_analysis_1} and \ref{fig:attn_analysis_2}, where we have examined the attention score of the [CLS] token at the 5th layer of MatSciBert. From figure \ref{fig:attn_analysis_1} and \ref{fig:attn_analysis_2}, it is  clearly observed,  MatSciBert allocates higher attention scores to tokens that define global features of the crystal, such as   `Formula', `Mineral', `Crystal System', `Space Group Number', and `Dimensionality'. Further, we observe for local information corresponding to Fe, H, and O atoms. MatScibert provides more attention score to tokens related to bond types (octahedral geometry, equivalent bond, distorted water-like geometry etc) and bond lengths (2.08 Å, 2.10 Å and 2.53 Å bond length). It is evident from these observations that MatSciBert is attending the important tokens related to global and local material information, to generate more expressive multimodal representation.
% \begin{figure}
% 	\centering
%         \subfloat{\includegraphics[width=0.35\columnwidth, height=110mm]{UAI 2023 CrysmmNet/Figure/Global_Line.png}}
% 	\subfloat{\includegraphics[width=0.35\columnwidth, height=110mm]{UAI 2023 CrysmmNet/Figure/Local_l1.png}}\\
%          \subfloat{\includegraphics[width=0.35\columnwidth, height=110mm]{UAI 2023 CrysmmNet/Figure/Local_l2.png}}
%          \subfloat{\includegraphics[width=0.35\columnwidth, height=110mm]{UAI 2023 CrysmmNet/Figure/Local_l3.png}}
% 	\caption{Attention weights of 5th Layer at MatSciBert between [CLS] token and other tokens in material's description for $\mathbf{FeH_8(ClO_2)_2}$.}
% 	\label{fig:attn_analysis}
% \end{figure}
\begin{figure}
	\centering
        \subfloat{\includegraphics[scale=0.8]{UAI 2023 CrysmmNet/Figure/Global_Line.png}}
	\subfloat{\includegraphics[scale=0.8]{UAI 2023 CrysmmNet/Figure/Local_l1.png}}
	\caption{Attention weights of 5th Layer at MatSciBert between [CLS] token and other tokens in material's description for $\mathbf{FeH_8(ClO_2)_2}$.}
	\label{fig:attn_analysis_1}
\end{figure}
\begin{figure}
	\centering
         \subfloat{\includegraphics[scale=0.8]{UAI 2023 CrysmmNet/Figure/Local_l2.png}}
         \subfloat{\includegraphics[scale=0.8]{UAI 2023 CrysmmNet/Figure/Local_l3.png}}
	\caption{Attention weights of 5th Layer at MatSciBert between [CLS] token and other tokens in material's description for $\mathbf{FeH_8(ClO_2)_2}$.}
	\label{fig:attn_analysis_2}
\end{figure}

