% Title: Graph Contrastive Learning for Skeleton-based Action Recognition

% ArXiv ID: 2301.10900




\maketitle

\begin{abstract}
In the field of skeleton-based action recognition, current top-performing graph convolutional networks (GCNs) exploit intra-sequence context to construct adaptive graphs for feature aggregation. However, we argue that such context is still \textit{local} since the rich cross-sequence relations have not been explicitly investigated. In this paper, we propose a graph contrastive learning framework for skeleton-based action recognition (\textit{SkeletonGCL}) to explore the \textit{global} context across all sequences. In specific, SkeletonGCL associates graph learning across sequences by enforcing graphs to be class-discriminative, \emph{i.e.,} intra-class compact and inter-class dispersed, which improves the GCN capacity to distinguish various action patterns. Besides, two memory banks are designed to enrich cross-sequence context from two complementary levels, \emph{i.e.,} instance and semantic levels, enabling graph contrastive learning in multiple context scales. Consequently, SkeletonGCL establishes a new training paradigm, and it can be seamlessly incorporated into current GCNs. Without loss of generality, we combine SkeletonGCL with three GCNs (2S-ACGN, CTR-GCN, and InfoGCN), and achieve consistent improvements on NTU60, NTU120, and NW-UCLA benchmarks. The source code will be available at \url{https://github.com/OliverHxh/SkeletonGCL}.
\end{abstract}

\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\footnotetext[1]{Work done when Xiaohu Huang was an intern at Baidu VIS.}
\footnotetext[2]{Corresponding authors.}

\section{Introduction}
\label{sec:intro}
Graph convolutional networks (GCNs) have been widely applied in skeleton-based action recognition since they can naturally process non-grid skeleton sequences. For GCN-based methods, how to effectively learn the graphs remains a core and challenging problem. In particular, ST-GCN \citep{stgcn} is a milestone work, using pre-defined graphs to extract the action patterns. However, the pre-defined graphs only enable each joint to access the fixed neighboring joints but are hard to capture long-range dependency adaptively. Therefore, a mainstream of subsequent works \citep{ASGCN, 2SAGCN, CAGCN, SGN, DynamicGCN, CTRGCN, INFOGCN} take efforts to solve this issue by generating adaptive graphs. The adaptive graphs can dynamically aggregate the features within each sequence and thus show significant advantages in performance comparison.

Generally, adaptive graphs are constructed by using intra-sequence context. However, such context will still be ``local'' when viewing the cross-sequence information as an available context. Therefore, we wonder: \textit{Is it possible to involve the cross-sequence context in graph learning?} To find out the answer, in Fig. \ref{fig:motivation}, we visualize the adaptive graphs learned from sequences of two easily confused classes (``point to something'' and ``take a selfie''). The graphs are learned by a strong GCN, \emph{i.e.,} CTR-GCN \citep{CTRGCN}. From the visualization, we find that 
(1) For sequences that are correctly classified in Fig. \ref{fig:motivation} (a) and Fig. \ref{fig:motivation} (b), the learned graphs in the same class look similar, while graphs in different classes have distinct differences. (2) For a misclassified sequence in Fig. \ref{fig:motivation} (c), the learned graph resembles the graphs from the misclassified class more than those from the ground truth class. These observations remind us that graph learning in current adaptive GCNs can implicitly learn class-specific graph representations to some extent. But without explicit constraints, it leads to class-ambiguous representations in some cases, thereby affecting the GCN capacity to discriminate classes (in Tab. \ref{table:Graph Distance} of Sec. \ref{exp:dia}, we provide quantitative results to further support our hypothesis). Therefore, we speculate that if the cross-sequence semantic relations are incorporated as guidance in graph learning, the class-ambiguity issue will be alleviated and the graph representations will better express individual characteristics of actions.


In recent years, contrastive learning has achieved great success in self-supervised representation learning \citep{moco, simclr, mocov3}. It studies cross-sample relations in the dataset. The essence of contrastive learning is ``comparing", which pulls together the feature embedding from positive pairs and pushes away the feature embedding from negative pairs.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{fig/motivation.pdf}
    \caption{\textbf{Graph visualization of sequences from two easily confused classes }(``point to something'' and ``take a selfie''). The graphs are learned by CTR-GCN \citep{CTRGCN}. We take the tip of the hand that does the action as the anchor. The size of the red circles and the width of the blue lines both denote the strengths of connections between joints. For simplicity, only representative frames are visualized. (a) Three sequences from class ``point to something'' are correctly classified, where the graphs contain connections to the body joints. (b) Three sequences from class ``take a selfie'' are correctly classified, where the graphs highly emphasize the connections to the hands, while the connections to the body are suppressed. (c) A sequence from class ``point to something'' is misclassified as ``take a selfie'', whose graph resembles the graphs in (b), but is dissimilar from graphs in (a). Hence, we realize that the class-ambiguous graph representations would make negative impacts on recognition performance.}
    \label{fig:motivation}
\end{figure}

Based on the analysis above and the advances in contrastive learning, we propose a graph contrastive learning framework for skeleton-based action recognition in the fully-supervised setting, dubbed \textbf{SkeletonGCL}. Instead of just using the local information within each sequence, SkeletonGCL explores the cross-sequence global context to improve graph learning. The core idea is to pull together the learned graphs from the same class while pushing away the learned graphs from different classes. Since graphs can reveal the action patterns of sequences, enforcing graph consistency in the same class and inconsistency among different classes helps the model understand various motion modes. In addition, to enrich the cross-sequence context, we build memory banks to store the graphs from historical sequences. In specific, an instance-level memory bank stores the sequence-wise graphs, which hold the individual properties of each sequence. In contrast, a semantic-level memory bank stores the aggregation of graphs from each class, which contains the class-level representation. The two banks are complementary to each other, enabling us to leverage more samples. SkeletonGCL can be seamlessly combined with existing GCNs. 
Eventually, we combine SkeletonGCL with three GCNs (2S-AGCN \citep{2SAGCN}, CTR-GCN \citep{CTRGCN}, and InfoGCN \citep{INFOGCN}), and conduct experiments on three popular datasets (NTU60 \citep{ntu60}, NTU120 \citep{ntu120} and NW-UCLA \citep{nw-ucla}). SkeletonGCL achieves consistent improvements with these models using different testing protocols (single-modal or multi-modal) on all three datasets, which widely demonstrates the effectiveness of our design. Notably, SkeletonGCL only introduces a small amount of training consumption but has no impact at the test stage.

Though there exist some works that apply contrastive learning in skeleton-based action recognition \citep{crossclr, Aimclr, CMD}, our method differs from them as follows: (1) The previous methods took pooled feature vectors to conduct contrastive learning as in \citep{moco,simclr}, where the structural properties in skeletons are lost. In contrast, SkeletonGCL uses graphs to contrast, which maintains the structure details of skeletons and offers high-order connection information between joints. (2) The previous methods used memory banks to store instance-level representations only. Differently, our memory banks store both instance-level and semantic-level representations, allowing us to leverage context from individual sequences and class-specific aggregations, which are complementary to each other. (3) The previous methods were used in the pre-training stage, while SkeletonGCL is incorporated into the fully-supervised setting without extra pre-training cost.

Summarily, the contribution of this paper can be concluded as follows:
\begin{itemize}
    \item We present a new perspective for graph learning of GCN models in skeleton-based action recognition. In specific, we propose to make use of the cross-sequence context to guide graph learning, whose goal is to enforce graphs to be intra-class compact and inter-class dispersed.
    \item Motivated by the advances in contrastive learning, we smoothly combine the ideas of contrastive learning and cross-sequence graph learning together, then propose a new training paradigm for skeleton-based action recognition, called SkletonGCL. SkeletonGCL incorporates an instance-level and a semantic-level memory bank to enrich the cross-sequence context comprehensively. Besides, it can be seamlessly incorporated into current GCNs.
    \item SkeletonGCL achieves consistent improvements combined with three GCNs (2S-AGCN, CTR-GCN, and InfoGCN) on three popular benchmarks (NTU60, NTU120, NW-UCLA) using both single-modal and multi-modal testing protocols. In addition, SkeletonGCL is training-efficient and has no impact at the test stage.
\end{itemize}

\section{Related Works}
\subsection{Skeleton-based Action Recognition}
Skeleton-based action recognition is to classify actions from sequences of estimated key points. The early deep-learning methods applied convolution neural networks (CNNs) \citep{PCNN, enhancedcnn} or recurrent neural networks (RNNs) \citep{rnnskeleton,lev2016rnn, twostreamrnn, contextrnn} to model the skeletons, but they could not explicitly explore the topological structure of skeletons, thus the performances were limited. Recently, PoseC3D \citep{posec3d} revisited the CNN-based method by stacking the heatmaps as 3D volumes, which maintained the spatial-temporal properties of skeletons and obtained marginal performance improvements. In the past few years, the mainstream works in skeleton-based action recognition were GCN models. ST-GCN \citep{stgcn} was the first work that adopted GCN as the feature extractor, which heuristically designed fixed graphs to model the skeletons. The follow-up methods proposed spatial-temporal graphs \citep{MSG3D}, multi-scale graph convolutions \citep{MSGCN}, channel-decoupled graphs \citep{CTRGCN, decouplinggcn} and adaptive graphs \citep{ASGCN,2SAGCN,DynamicGCN,CAGCN,CTRGCN,INFOGCN} to improve the capacity of GCNs. Tracking the development of GCN-based methods, we find that graph learning has always been a core problem and now the adaptive GCNs are leading since they can model the intrinsic topology of skeletons.

However, current adaptive GCNs generated the graphs based on the local context within each sequence, where the cross-sequence relations have been neglected. In contrast, we propose to explore the cross-sequence global context to shape graph representations. In this way, the learned graphs can not only describe the individual characteristics within each sequence but also emphasize the similarity and dissimilarity of motion patterns across sequences.

\subsection{Contrastive Learning}
In recent years, numerous representation learning methods \citep{unsupervisedID, oord2018representation,moco,simclr, cross-sequence-semantic} with contrastive learning have emerged, especially in self-supervised representation learning. The key idea is to pull together the  positive pairs and push away the negative pairs in the feature space. Generally, the features are vectors obtained from feature extractors followed by a pooling layer. A standard approach to obtaining the positive pairs is augmenting an original sample into two different views. The negative samples are selected randomly or using hard mining strategies \citep{hard, hard1, hard2}. To increase the capacity of negative samples, the memory bank mechanism was devised in \citep{moco, memorybank} to store more negative instances. By contrasting positive pairs against negative pairs, the model can learn to focus on semantic representations.

In the field of skeleton-based action recognition, prior works \citep{crossclr, CMD, Aimclr} proposed to apply contrastive learning in the pre-training stage by roughly following the frameworks mentioned above. CrossCLR \citep{crossclr} mined positive pairs in the data space and explored the cross-modal distribution relationships. Further, CMD \citep{CMD} transferred the cross-modal knowledge in a distillation manner. And AimCLR \citep{Aimclr} used extreme augmentations to improve the representation universality. 

Compared with the above methods, we use graph representations to contrast instead of using pooled feature vectors. Meanwhile, we establish two different memory banks at complementary levels, \emph{i.e.,} instance and semantic levels, to enrich the context scales. Besides, the proposed method is used with GCNs under the fully-supervised setting, which requires no pre-training procedure.


\section{Method}
\subsection{Preliminary}
\label{section:preliminary}
We denote a human skeleton as a vertex set $\mathcal{V}=\{v_1, v_2, ..., v_N\}$, where $N$ denotes the number of vertices. For each vertex $v_i$, the feature dimension is set as $C$. Hence, a skeleton sequence with $T$ frames can be denoted as $\mathbf{X} \in \mathbb{R}^{T \times N \times C}$. Graph topology is used to represent the correlations between joints, formulated as $\mathbf{g}$.

\noindent \textbf{GCNs in Skeleton-Based Action Recognition.} Generally, GCN models alternatively apply graph convolutions and temporal convolutions to extract the spatial configuration and motion pattern of skeletons, respectively. The graph $\mathbf{g}$ is vital for graph convolutions since it determines the message passing among joints. In current adaptive GCNs, $\mathbf{g}$ is learned within each sequence and has different sizes, \emph{e.g.,} $\mathbf{g} \in \mathbb{R}^{K_S \times N \times N}$ in 2S-AGCN \citep{2SAGCN} and $\mathbf{g} \in \mathbb{R}^{K_S \times C \times N \times N}$ in CTR-GCN \citep{CTRGCN}. The $K_S$ denotes the number of sub-graphs, normally set as 3.
In general, the graph convolution is defined as:
\begin{equation}
    \mathbf{X_S} = \sum_{k=1}^{K_S}{\mathbf{g}^{k}\mathbf{X}\mathbf{W^{k}_S}},
    \label{eq:graph conv}
\end{equation}
where $\mathbf{X_S} \in \mathbb{R}^{T \times N \times C'}$ denotes the spatial extracted feature with $C'$ channels, and $\mathbf{W_S} \in \mathbb{R}^{K_S \times C \times C'}$ denotes the spatial feature transformation filters. Next, temporal convolutions are applied on $\mathbf{X_S}$, producing motion extracted feature $\mathbf{X_T} \in \mathbb{R}^{T \times N \times C'}$. After stacking layers of graph convolutions and temporal convolutions, a global average pooling (GAP) layer summarizes the global features, then a classification head (fully-connected layer) followed by a Softmax activation function is applied to obtain the class prediction $\mathbf{\mathbf{\hat{y}}} \in \mathbb{R}^{C_k}$, where $C_k$ denotes the number of classes. Finally, a cross-entropy loss $\mathcal{L}_\text{CE}$ supervises the class prediction with the ground truth label $\mathbf{y}$ as follows:
\begin{equation}
    \mathcal{L}_\text{CE} = -\sum_i{\mathbf{y_i}\log{\mathbf{\hat{y}_i}}}
\end{equation}
\noindent \textbf{Self-Supervised Contrastive Learning.} In the context of self-supervised contrastive learning, each input sample is processed by data augmentations to produce a positive pair: $I$ and $I^+$. Through a feature extraction network, $I$ and $I^+$ are transformed into feature vectors $\mathbf{f} \in \mathbb{R}^{D}$ and $\mathbf{f^+} \in \mathbb{R}^{D}$. As for the negative samples, they are selected from the dataset excluding $I$ and $I^+$, represented as a negative set $\mathcal{N^-}$. Each negative in $\mathcal{N^-}$ is denoted as $\mathbf{f^-} \in \mathbb{R}^{D}$. The similarity between two feature vectors is calculated as $sim(\mathbf{f^+}, \mathbf{f^-})=\frac{\mathbf{f^+}\mathbf{f^-}}{\Vert \mathbf{f^+} \Vert \Vert \mathbf{f^-} \Vert}$. InfoNCE \citep{nce, oord2018representation} is widely adopted for contrastive learning, which is formulated as:
\begin{equation}
    \label{eq:contrastive loss}
    \mathcal{L}_\text{NCE} = -\log{\frac{sim(\mathbf{f}, \mathbf{f^+})/\tau}{sim(\mathbf{f}, \mathbf{f^+})/\tau+\sum_{\mathbf{f^-} \in {\mathcal{N^-}}}{sim(\mathbf{f}, \mathbf{f^-})/\tau}}},
\end{equation}
where temperature $\tau > 0$ is a hyper-parameter.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{fig/framework-crop.pdf}
    \caption{\textbf{Overview of SkeletonGCL.} An input skeleton sequence $I$ is fed into a GCN encoder, producing a feature vector $\mathbf{f}$ for classification and a learned graph $\mathbf{g}$ for graph contrastive learning. The graph $\mathbf{g}$ is embedded into a vector by a projection head. And two memory banks are built to store the embedded graphs. From the memory banks, we sample the positives and negatives according to the labels, then perform contrastive loss. The memory banks are only used in the training stage but will be removed during the testing stage.}
    \label{fig:framework}
\end{figure}
\subsection{Graph Contrastive Learning} The proposed SkeletonGCL is illustrated in Fig. \ref{fig:framework}. The framework consists of two branches, where the classification branch is parallel to the graph contrast branch. Taking a skeleton sequence $I$ as input, the GCN encoder outputs a feature vector $\mathbf{f}$ for classification and a graph $\mathbf{g}$ for graph contrast. 

\noindent \textbf{Graph Projection Head.} In order to contrast the graphs in a common feature space, we embed the graphs into vectors by a graph projection head. The projection heads for different GCNs are similar (see App. \ref{appx:projection head} for details). In Fig. \ref{fig:framework}, taking the graph $\mathbf{g} \in \mathbb{R}^{K_S \times C \times N \times N}$ learned in CTR-GCN \citep{CTRGCN} as an example, we first squeeze $\mathbf{g}$ along the channel dimension by an average pooling layer into $\mathbf{\overline{g}} \in \mathbb{R}^{K_S \times N \times N}$. 
Then, we flatten graph $\overline{\mathbf{g}}$ into a 1D vector as $\mathbf{g'} \in \mathbb{R}^{K_S N^2}$ and project $\mathbf{g'}$ into a vector $\mathbf{v} \in \mathbb{R}^{C_g}$ by an FC layer $\mathbf{W_G} \in \mathbb{R}^{K_S N^2 \times C_g}$. Since different channels in $\mathbf{W_G}$ are specific to different vertices in the graph, the graph projection is vertex-aware and thus can encode the structures of skeletons. Afterward, we update two memory banks with $\mathbf{v}$. The memory banks are illustrated in Fig. \ref{fig:framework}, and detailed next.

\noindent \textbf{Memory Bank.} To enrich the cross-sequence context, we build memory banks to store the cross-batch graphs. In specific, two memory banks are constructed, \emph{i.e.,} an instance-level memory bank $\mathcal{M}_\text{Ins} \in \mathbb{R}^{C_k \times P \times C_g}$ and a semantic-level memory bank $\mathcal{M}_\text{Sem} \in \mathbb{R}^{C_k \times C_g}$. $P$ denotes the number of instances stored for each class in $\mathcal{M}_\text{Ins}$. Particularly, each element in $\mathcal{M}_\text{Ins}$ denotes a graph instance from a class. In contrast, each element in $\mathcal{M}_\text{Sem}$ denotes the graph aggregation of a class. Therefore, the two memory banks are on complementary levels, where the instance-level memory bank emphasizes the instance discrimination of each sequence, while the semantic-level memory bank covers the class properties across sequences.

We update $\mathcal{M}_\text{Ins}$ in a first-in-first-out manner, which maintains the number of instances for each class as $P$. As for $\mathcal{M}_\text{Sem}$, we use a momentum update strategy, which integrates the graphs of the same class from the current timestamp and all previous timestamps, regarded as a long-term representation. The momentum update is defined as follows:
\begin{equation}
    \label{eq:momentum}
    \mathbf{m}_\text{sem}^{c_*} \gets \alpha \mathbf{m}_\text{sem}^{c_*} + (1-\alpha)\mathbf{v},
\end{equation}
where $\mathbf{m}_\text{sem}^{c_*}$ is the representation for class $c_*$, $c_*$ is the class label for the input $I$ and $0\!<\!\alpha\!<\!1$ is a hyper-parameter. 

\noindent \textbf{Loss.} To achieve the graph contrast, we sample positives and negatives from the memory banks $\mathcal{M}_\text{Ins}$ and $\mathcal{M}_\text{Sem}$. For $\mathcal{M}_\text{Ins}$, vector $\mathbf{v}$ is set as the anchor, hence samples in the positive set $\mathcal{N}_\text{Ins}^+$ are with label $c_*$, and samples in the negative set $\mathcal{N}_\text{Ins}^-$ are with different labels. Consequently, the InfoNCE loss in Eq. \ref{eq:contrastive loss} can be rewritten as:
\begin{equation}
    \mathcal{L}_\text{NCE}^\text{Ins} = -\sum_{\mathbf{v^+} \in \mathcal{N}_\text{Ins}^+}{\log{\frac{sim(\mathbf{v}, \mathbf{v^+})/\tau}{sim(\mathbf{v}, \mathbf{v^+})/\tau+\sum_{\mathbf{v^-} \in {\mathcal{N}_\text{Ins}^-}}{sim(\mathbf{v}, \mathbf{v^-})/\tau}}}},
\end{equation}
\begin{equation}
\mathcal{L}_\text{NCE}^\text{Sem} = -\sum_{\mathbf{v^+} \in \mathcal{N}_\text{Sem}^+}{\log{\frac{sim(\mathbf{v}, \mathbf{v^+})/\tau}{sim(\mathbf{v}, \mathbf{v^+})/\tau+\sum_{\mathbf{v^-} \in {\mathcal{N}_\text{Sem}^-}}{sim(\mathbf{v}, \mathbf{v^-})/\tau}}}}.
\end{equation}

$\mathcal{L}_\text{NCE}^\text{Ins}$ leverages multiple positives compared with Eq. \ref{eq:contrastive loss} by using label information, which mines more semantic-related samples. Similarly, we can define the InfoNCE loss $\mathcal{L}_\text{NCE}^\text{Sem}$, which is specific for the memory bank $\mathcal{M}_\text{Sem}$. Summarily, the overall contrastive loss is written as follows:
\begin{equation}
    \mathcal{L}_\text{NCE} = \mathcal{L}_\text{NCE}^\text{Ins} + \mathcal{L}_\text{NCE}^\text{Sem}.
\end{equation}
And the overall loss function is defined as follows:
\begin{equation}
    \mathcal{L} = \mathcal{L}_\text{NCE} + \mathcal{L}_\text{CE}.
\end{equation}
\noindent \textbf{Hard Sampling.} As the training continues, most samples become too easy, which contribute less to the training. Therefore, methods in \citep{tabassum2022hard, hard1, hard2, cross-sequence-semantic} are proposed to use hard mining strategies to focus on informative samples. In this paper, considering the massive number of instances in $\mathcal{M}_\text{Ins}$, contrasting with all these instances naturally leads to redundancy and hinders the training. To alleviate this issue, we propose to mine hard examples in $\mathcal{M}_\text{Ins}$. Specifically, we take the similarity calculation $sim(\mathbf{v},\mathbf{v'})$ as a criterion to evaluate hardness. Harder positives are with lower similarities, and harder negatives are with higher similarities. In total, for $\mathcal{M}_\text{Ins}$, we select $K_\text{H}^+$ hardest positive examples, $K_\text{H}^-$ hardest negative examples, and $K_\text{R}^-$ random negative examples.

\section{Experiments}
\subsection{Datasets}
\textbf{NTU RGB+D}. NTU RGB+D (NTU60) \citep{ntu60} is a large-scale skeleton-based action recognition dataset, which contains 60 action classes and 56,880 sequences. Each sequence is annotated as skeletons with 25 joints. All the sequences are performed by 40 subjects and filmed by 3 Kinect cameras from three different views. Generally, two protocols are used to evaluate the performances: (1) cross-subject (X-Sub): train data are performed by 20 subjects, and test data are performed by other 20 subjects. (2) cross-view (X-View): train data from cameras 2 and 3, and test data from camera 1.

\textbf{NTU RGB+D 120.} NTU RGB+D 120 (NTU120) \citep{ntu120} is an extension of NTU RGB+D dataset, which newly includes 57,367 skeletons of 60 extra classes. All the sequences are performed by 106 subjects and filmed by three cameras from three different views. In addition, NTU RGB+D 120 has 32 setups, where each denotes a unique location. Generally, two protocols are used to evaluate the performances: (1) cross-subject (X-Sub): train data are performed by 53 subjects, and test data are performed by other 53 subjects. (2) cross-setup (X-Set): train data are samples with even setup IDs, and test data are samples with odd setup IDs.

\textbf{Northwestern-UCLA}. Northwestern-UCLA (NW-UCLA) dataset \citep{nw-ucla} contains 1494 sequences from 10 action classes. Each sequence is annotated as skeletons with 20 joints. All sequences are performed by 10 subjects and filmed by three Kinect cameras from different views. We follow the official evaluation protocol: train data are captured by the first two cameras, and test data are captured by the third camera.

\subsection{Implementation Details}
To thoroughly validate SkeletonGCL, we take three GCNs (2S-AGCN \citep{2SAGCN}, CTR-GCN \citep{CTRGCN}, and InfoGCN \citep{INFOGCN}) as baseline models. For CTR-GCN and InfoGCN, we follow their training recipes. Particularly, for 2S-AGCN, since its training recipe is out of date, we borrow the training recipe from CTR-GCN, which effectively improves its baseline performance. $P$, the number of stored instances for each class in $\mathcal{M}_\text{Ins}$, is set as 684 on NTU60 and NTU120, and 342 on NW-UCLA. The dimension of graph vector $C_g$ is set to 256. For all datasets, the number of sampling examples $K_\text{H}^+$, $K_\text{H}^-$, and $K_\text{R}^-$ are set as 128, 512, and 512, respectively. For different models used in different modalities, we experiment with temperature $\tau$ of 0.5, 0.8, 1.0, and 1.5, and choose the best one. The hyper-parameter $\alpha$ for momentum updating is set as 0.85. Besides, we fix the random seed to ensure experiment reproducibility. All experiments are conducted using a single NVIDIA V100 GPU.

\subsection{Compared with the state-of-the-art}
In this section, we combine our method with three GCNs, and compare them with the state-of-the-art (SoTA) methods. In Tab. \ref{tab:state-of-the-art ntu} and Tab. \ref{tab:sota nwucla}, we list current SoTA methods in skeleton-based action recognition except PoseC3D \citep{posec3d}. PoseC3D is a promising CNN-based method, but it uses non-official skeleton data and applies a multi-crop test protocol (GCN methods typically use one crop), which are unfair for comparison here. In evaluation, four modalities are used: ``joint stream'' (\textit{J}) denotes the joint coordinates, ``bone stream'' (\textit{B}) denotes the coordinate difference between spatially connected joints, ``joint motion'' (\textit{J-M}) denotes the coordinate difference between temporally adjacent frames, and ``bone motion'' (\textit{B-M}) denotes the bone difference between temporally adjacent frames. The 4-stream ensemble (\textit{4S}) denotes using the four modalities together. 
Following the widely-adopted protocol, we evaluate models using $J$, $B$, $J+B$, and $4S$ modalities.

\noindent \textbf{NTU60 and NTU120.} Tab. \ref{tab:state-of-the-art ntu} lists the results on NTU60 and NTU120. From the results, we find that: (1) Combined with SkeletonGCL, all three baseline models achieve solid improvements on these two benchmarks over different settings and modalities. Taking the \textit{J} modality on NTU60 X-Sub as an example, 2S-AGCN improves by 1.0\% (88.9\% to 89.9\%), CTR-GCN improves by 1.0\% (89.8\% to 90.8 \%), and InfoGCN improves by 0.7\% (89.4\% to 90.1\%). Considering NTU60 is an extensively-benchmarked dataset, such improvements are quite hard. (2) With SkeletonGCL, CTR-GCN and InfoGCN can set new SoTA performance.

\noindent \textbf{NW-UCLA.} Tab. \ref{tab:sota nwucla} lists the results on NW-UCLA. SkeletonGCL can still achieve consistent improvements based on the three models. And new state-of-the-art performances are achieved by combining SkeletonGCL with CTR-GCN and InfoGCN.

\begin{table}[t]
    \scriptsize
    \setlength\tabcolsep{2.5pt}
    \caption{Top-1 accuracy comparison ($\%$) with the state-of-the-art methods on NTU 60 and NTU120 datasets. The numbers in gray indicate the results reported in their papers. * indicates that we retrain the models using their officially released code. Particularly, 2S-AGCN is retrained using a stronger train recipe from CTR-GCN.}
    \begin{tabular}{c||c|c|c|c|c|c|c|c||c|c|c|c|c|c|c|c}
       \rowcolor{gray!30} Dataset & \multicolumn{8}{c||}{NTU 60} & \multicolumn{8}{c}{NTU 120} \\\hline
       \rowcolor{gray!30} Setting & \multicolumn{4}{c|}{X-Sub} & \multicolumn{4}{c||}{X-View} & \multicolumn{4}{c|}{X-Sub} & \multicolumn{4}{c}{X-Set} \\\hline
       \rowcolor{gray!30} Method/Modality & \textit{J} & \textit{B} & \textit{J+B} & \textit{4S} & \textit{J} & \textit{B} & \textit{J+B} & \textit{4S} & \textit{J} & \textit{B} & \textit{J+B} & \textit{4S} & \textit{J} & \textit{B} & \textit{J+B} & \textit{4S} \\
       \hline \hline
       SGCN \citep{SGN} & - & - & 89.0 & - & - & - & 94.5 & - & - & - & 79.2 & - & - & - & 81.5 & - \\
       ST-TR-GCN \citep{sttragcn} & 89.2 & - & 90.3 & - & 95.8 & - & 96.3 & - & 82.7 & - & 85.1 & - & 85.0 & - & 87.1 & - \\
       Shift-GCN \citep{shiftgcn} & 87.8 & - & 89.7 & 90.7 & 95.1 & - & 96.0 & 96.5 & 80.9 & - & 85.3 & 85.9 & 83.2 & - & 86.6 & 87.6 \\
       DC-GCN+ADG \citep{decouplinggcn} & - & - & 90.8 & - & - & - & 96.6 & - & - & - & 86.5 & - & - & - & 88.1 & - \\
       Dynamic GCN \citep{DynamicGCN} & - & - & - & 91.5 & - & - & - & 96.0 & - & - & - & 87.3 & - & - & - & 88.6 \\
       MS-G3D \citep{MSG3D} & 89.4 & 90.1 & 91.5 & - & 95.0 & 95.3 & 96.2 & - & - & - & 86.9 & - & - & - & 88.4 & - \\ 
       MST-GCN \citep{MSGCN} & 89.0 & 89.5 & 91.1 & 91.5 & 95.1 & 95.2 & 96.4 & 96.6 & 82.8 & 84.8 & 87.0 & 87.5 & 84.5 & 86.3 & 88.3 & 88.8 \\\hline \hline
       {\color{gray}2S-AGCN \citep{2SAGCN}} & - & - & {\color{gray}88.5} & - & {\color{gray}93.7} & {\color{gray}93.2} & {\color{gray}95.1} & - & - & - & - & - & - & - & - & - \\
       2S-AGCN* \citep{2SAGCN} & 88.9 & 89.2 & 91.0 & 91.5 & 94.5 & 94.1 & 95.7 & 95.9 & 84.0 & 85.1 & 87.8 & 88.2 & 85.3 & 86.3 & 89.0 & 89.6 \\
       2S-AGCN* \textit{w}/SkeletonGCL & \textbf{89.9} & \textbf{90.0} & \textbf{91.6} & \textbf{92.2} & \textbf{95.0} & \textbf{94.4} & \textbf{96.1} & \textbf{96.4} & \textbf{84.7} & \textbf{86.0} & \textbf{88.4} & \textbf{88.7} & \textbf{86.1} & \textbf{86.8} & \textbf{89.7} & \textbf{90.2} \\
       \hline \hline 
       {\color{gray}CTR-GCN \citep{CTRGCN}} & - & - & - & {\color{gray}92.4} & - & - & - & {\color{gray}96.8} & - & {\color{gray}85.7} & {\color{gray}88.7} & {\color{gray}88.9} & - & {\color{gray}87.5} & {\color{gray}90.1} & {\color{gray}90.6} \\
       CTR-GCN* \citep{CTRGCN} & 89.8 & 90.2 & 92.0 & 92.4 & 94.8 & 94.8 & 96.3 & 96.8 & 84.9 & 85.7 & 88.7 & 88.9 & 86.7 & 87.5 & 90.1 & 90.5 \\
       CTR-GCN* \textit{w}/SkeletonGCL & \textbf{90.8} & \textbf{91.1} & \textbf{92.6} & \textbf{93.1} & \textbf{95.3} & \textbf{95.4} & \textbf{96.6} & \textbf{97.0} & \textbf{85.6} & \textbf{86.9} & \textbf{89.2} & \textbf{89.5} & \textbf{87.3} & \textbf{88.2} & \textbf{90.5} & \textbf{91.0} \\
       \hline \hline
       {\color{gray} InfoGCN \citep{INFOGCN}} & {\color{gray}89.8} & {\color{gray}90.6} & {\color{gray}91.6} & {\color{gray}92.7} & {\color{gray}95.2} & {\color{gray}95.5} & {\color{gray}96.5} & {\color{gray}96.9} & {\color{gray}85.1} & {\color{gray}87.3} & {\color{gray}88.5} & {\color{gray}89.4} & {\color{gray}86.3} & {\color{gray}88.5} & {\color{gray}89.7} & {\color{gray}90.7} \\
       InfoGCN* \citep{INFOGCN} & 89.4 & 90.6 & 91.3 & 92.3 & 95.2 & 95.4 & 96.2 & 96.7 & 84.2 & 86.9 & 88.2 & 89.2 & 86.3 & 88.5 & 89.4 & 90.7 \\
       InfoGCN* \textit{w}/SkeletonGCL & \textbf{90.1} & \textbf{91.0} & \textbf{91.9} & \textbf{92.8} & \textbf{95.5} & \textbf{95.7} & \textbf{96.6} & \textbf{97.1} & \textbf{85.2} & \textbf{87.4} & \textbf{88.8} & \textbf{89.8} & \textbf{87.2} & \textbf{88.7} & \textbf{90.0} & \textbf{91.2} \\
       \hline
    \end{tabular}
    \label{tab:state-of-the-art ntu}
\end{table}


\begin{table}[h]
    \centering
    \scriptsize
    \caption{Top-1 accuracy Comparison ($\%$) with the state-of-the-art methods on the NW-UCLA dataset. Numbers in gray denote the results reported in their papers. * indicates that we retrain the models using their officially released codes. Particularly, 2S-AGCN is retrained using a stronger train recipe from CTR-GCN.}
    \begin{tabular}{c||c|c|c|c}
        \rowcolor{gray!30} Dataset & \multicolumn{4}{c}{NW-UCLA} \\\hline
        \rowcolor{gray!30} Method/Modality & \textit{J} & \textit{B} & \textit{J+B} & \textit{4S} \\
        \hline \hline
        AGC-LSTM \citep{AGC-LSTM} & 93.3 & - & - & - \\
        DC-GCN+ADG \citep{decouplinggcn} & - & - & 95.3 & - \\ 
        Shift-GCN \citep{shiftgcn} & 92.5 & - & 94.2 & 94.6 \\
        \hline \hline 
        {\color{gray}2S-AGCN \citep{2SAGCN}} & - & - & - & - \\
        2S-AGCN* \citep{2SAGCN} & 92.0 & 92.2 & 95.0 & 95.5 \\
        2S-AGCN* \textit{w}/SkeletonGCL & \textbf{92.6} & \textbf{93.0} & \textbf{95.7} & \textbf{96.3} \\
        \hline \hline
        {\color{gray} CTR-GCN \citep{CTRGCN}} & - & - & - & {\color{gray}96.5} \\
        CTR-GCN* \citep{CTRGCN} & 94.6 & 91.8 & 94.2 & 96.5 \\\
        CTR-GCN* \textit{w}/SkeletonGCL & \textbf{95.1} & \textbf{95.0} & \textbf{95.9 }& \textbf{96.8} \\
        \hline \hline
        {\color{gray}InfoGCN \citep{INFOGCN}} & {\color{gray}94.0} & {\color{gray}95.3} & {\color{gray}96.3} & {\color{gray}96.6} \\
        InfoGCN* \citep{INFOGCN} & 93.8 & 94.2 & 95.5 & 96.1 \\
        InfoGCN* \textit{w}/SkeletonGCL & \textbf{94.8} & \textbf{94.6} & \textbf{96.1} & \textbf{96.8} \\
        \hline
    \end{tabular}
    \label{tab:sota nwucla}
\end{table}

\subsection{Diagnostic Experiments}
\label{exp:dia}
In this section, we conduct diagnostic experiments to verify the design of SkeletonGCL. Otherwise stated, we use CTR-GCN as the GCN encoder to perform the experiments on the NTU60 dataset under the X-Sub setting using the joint modality (\textit{J}). See App. \ref{app:diagnostic experiments} for more diagnostic experiments. 

\begin{minipage}{\textwidth}
\begin{minipage}[t]{0.3\textwidth}
\centering
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
\scriptsize
\caption{Comparison of intra-batch and inter-batch contrast.}
    \begin{tabular}{c||c}
       \rowcolor{gray!30} Model (CTR-GCN) & Acc (\%) \\
       \hline \hline
        Baseline (\textit{w}/o contrast) & 89.8 \\\hline
        Intra-batch Contrast (No Bank) & 90.2 \\
        Inter-batch Contrast & \textbf{90.8} \\\hline
    \end{tabular}
    \label{tab: intra and inter batch}
\end{minipage}
\hspace{3mm}
\begin{minipage}[t]{0.3\textwidth}
\centering
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Comparison of feature and graph contrast.}
    \begin{tabular}{c||c}
       \rowcolor{gray!30} Model (CTR-GCN) & Acc (\%) \\
       \hline \hline
        Baseline & 89.8 \\\hline
        Feature Contrast & 90.2 \\
        Graph contrast & \textbf{90.8} \\\hline
    \end{tabular}
\label{table:feature contrast and graph contrast}
\end{minipage}
\hspace{3mm}
\begin{minipage}[t]{0.3\textwidth}
\centering
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Impact of memory banks.}
    \begin{tabular}{c||c}
       \rowcolor{gray!30} Model (CTR-GCN) & Acc (\%) \\
       \hline \hline
        Baseline & 89.8 \\\hline
        Instance Memory & 90.3 \\
        Semantic Memory & 90.2 \\
        \hline
        Instance + Semantic & \textbf{90.8} \\
        \hline
    \end{tabular}
\label{table:memory banks}
\end{minipage}

\begin{minipage}[t]{0.3\textwidth}
\centering
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Impact of sampling strategies. R: Random; H: Hard.}
    \begin{tabular}{c|c||c}
       \multicolumn{2}{c||}{\cellcolor{gray!30} Sampling} & \cellcolor{gray!30} \\\cline{1-2}
       \cellcolor{gray!30} Positive & \cellcolor{gray!30} Negative & \multirow{-2}{*}{\cellcolor{gray!30} Acc(\%)} \\
       \hline \hline
        \multirow{3}{*}{R} & R & 90.1 \\
         & H & 90.2 \\
         & R+H & 90.5 \\
        \hline
        \multirow{3}{*}{H} & R & 90.4 \\
         & H & 90.6 \\
         & R+H & \textbf{90.8} \\
        \hline
        R + H & R + H & 90.3 \\\hline
    \end{tabular}
    \label{table:sampling}
\end{minipage}
\hspace{3mm}
\begin{minipage}[t]{0.25\textwidth}
\centering
\setlength\tabcolsep{2pt}
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Comparison with using triplet loss.}
    \begin{tabular}{c||c}
       \rowcolor{gray!30} Model (CTR-GCN) & Acc (\%) \\
       \hline \hline
        Baseline & 89.8 \\\hline
        Triplet loss & 90.7 \\
        InfoNCE loss & \textbf{90.8} \\
        \hline
    \end{tabular}
    \label{table:info_triplet_loss}
\end{minipage}
\hspace{3mm}
\begin{minipage}[t]{0.43\textwidth}
\centering
\setlength\tabcolsep{2pt}
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Training consumption on NTU60.}
    \begin{tabular}{c||c}
       \rowcolor{gray!30} Model & Time (hours) \\
       \hline \hline
        2S-AGCN & 3.4 \\
        2S-AGCN \textit{w}/ours & 3.5 \color{red}{$\uparrow$ 2.9\%} \\\hline
        CTR-GCN & 11.4  \\
        CTR-GCN \textit{w}/ours & 11.7 \color{red}{$\uparrow$ 2.6\%} \\\hline
        InfoGCN & 4.3 \\
        InfoGCN \textit{w}/ours & 4.6 \color{red}{$\uparrow$ 7.0\%} \\
        \hline
    \end{tabular}
    \label{table:train comsumption}
\end{minipage}

\begin{minipage}[t]{0.5\textwidth}
\centering
\setlength\tabcolsep{1pt}
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Graph Distance (dis.) comparison using Euclidean distance ($10^{- 5}$).}
    \begin{tabular}{c||c||c||c}
       \cellcolor{gray!30} & \cellcolor{gray!30} Average dis. to & \cellcolor{gray!30} Dis. to & \cellcolor{gray!30} Dis. to \\
        \multirow{-2}{*}{\cellcolor{gray!30} Sample} & \cellcolor{gray!30} all classes & \cellcolor{gray!30} correct class & \cellcolor{gray!30} misclassified class \\
       \hline \hline
        Incorrectly-Classified & \multirow{2}{*}{1.70} & \multirow{2}{*}{0.74} & \multirow{2}{*}{0.68} \\
        Samples & & & \\\hline
Correctly-Classified & \multirow{2}{*}{2.20} & \multirow{2}{*}{0.47} & \multirow{2}{*}{-} \\
Samples & & & \\
        \hline
    \end{tabular}
    \label{table:Graph Distance}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[t]{0.5\textwidth}
\centering
\setlength\tabcolsep{2pt}
\makeatletter\def\@captype{table}
\setlength\tabcolsep{2.5pt}
\captionsetup{font={small,stretch=1}}
   \scriptsize
   \caption{Performance (\%) of samples with different graph distance ranks to the correct class.}
    \begin{tabular}{c||c||c||c}
       \rowcolor{gray!30} & CTR & CTR \textit{w/Ours} & +/- \\
       \hline \hline
        rank1 &	97.2 & 97.4 & +0.2 \\
        rank2-5 & 92.1 & 92.9 & +0.8 \\
        rank6-10 & 88.3 & 89.3 & +1.0 \\
        rank11-20 & 84.9 & 86.3 & +1.4 \\
        rank21-40 & 83.2 & 84.8 & +1.6 \\
        rank41-60 & 80.8 & 82.9 & +2.1 \\
        \hline
    \end{tabular}
    \label{table:rank}
\end{minipage}
\end{minipage}

\noindent \textbf{Intra-batch vs. Inter-batch Graph Contrast.} 
In Tab. \ref{tab: intra and inter batch}, the effectiveness of introducing cross-sequence context is investigated. We find that only contrasting the graphs within one batch can bring improvement with 0.4\% (89.8\% to 90.2\%), which owes to the cross-sequence relation mining. And further exploring the inter-batch relations can bring more improvements to 1.0\% (89.8\% to 90.8\%), which explains that different batches provide richer context than a single batch.

\noindent \textbf{Graph Contrast vs. Feature Contrast.}
In Tab. \ref{table:feature contrast and graph contrast}, the comparison of using features $f$ to contrast and using graphs $g$ to contrast is investigated. We find that feature contrast can improve the performance on the baseline with 0.4\% (89.8\% to 90.2\%). But graph contrast can obviously outperform it by 0.6\% (90.2\% to 90.8\%). The results suggest that, due to the high-order structural information in graphs, graph contrast can better benefit graph convolution learning in GCNs.

\noindent \textbf{Memory Banks.} In Tab. \ref{table:memory banks}, the effectiveness of instance-level and semantic-level memory banks is investigated. We find that both memory banks benefit the recognition, and using them together achieves much higher performance, which proves their complementary properties.

\noindent \textbf{Sampling Strategy.} 
In Tab. \ref{table:sampling}, we compare different sampling strategies for SkeletonGCL. We find that selecting hard positive/negative examples can generally improve recognition. And also random negative samples are meaningful to recognition, which allows the contrastive loss to involve more negative samples.

\noindent \textbf{InfoNCE Loss vs. Triplet Loss.}
In Tab. \ref{table:info_triplet_loss}, we compare the performance of using another popular metric learning loss, \emph{i.e.,} triplet loss \citep{triplet}. We find that using triplet loss can achieve similar performance compared to InfoNCE loss. The results indicate the generality of our idea that it does not depend on a certain loss but can boost the performance using different losses.

\noindent \textbf{Traning Comsumption.}
In Tab. \ref{table:train comsumption}, we report the training consumption on NTU60. With our method, the training time only slightly increases with different baseline models, ranging from 2.6\% to 7.0\%, which proves the efficiency of the design. 

\noindent \textbf{Quantitative Results about Graph Similarities.} As shown in Tab. \ref{table:Graph Distance}, we statistically calculate the graph distances between each sample and all classes (detailed in App. \ref{app. graph distance}). For incorrectly-classified samples, we find that: (1) The graph distance to the misclassified class (0.68) is much lower than the average distance (1.70) to all classes. (2) The graph distance to the misclassified class (0.68) is indeed slightly lower than the distance to the correct class (0.74), which explains that not learning class-specific graphs could truly degrade recognition performance. In addition, for correctly-classified samples, we notice that: (1) The average graph distance (2.20) is higher than that for the misclassified ones (1.70), which indicates that the inter-class graph representations are more dispersed for the correctly-classified samples. (2) The distance to the correct class (0.47) is lower than that for the misclassified ones (0.74), which reveals that the intra-class representations are more compact for the correctly classified samples. To sum up, these quantitative results illustrate the strong correlation between recognition performance and class-specific graph representation.

\noindent \textbf{Performance vs. Graph Quality.} In Tab. \ref{table:rank}. we first calculate the graph distances between each sample and all classes (detailed in App. \ref{app. graph distance}) for CTR-GCN. Then, we rank the distances from low to high. In Tab. \ref{table:rank}, we report the recognition accuracies of samples according to their distance ranks to the correct class. Here, higher ranks indicate that graphs are of higher quality (intra-class compact and inter-class dispersed), while lower ranks indicate that graphs are of lower quality (intra-class dispersed and inter-class aliasing). We note that: (1) Considering samples from lower ranks to higher ranks, performances improve monotonically, revealing the significant correlations between graph quality and recognition performance. (2) Combined with the proposed method, we improve performances in all cases, where larger improvements are obtained on the samples with lower-quality graphs. These results prove that our method can alleviate the problem caused by learning low-quality graphs.


\section{Conclusion}
In this paper, we establish a new training paradigm for skeleton-based action recognition, called SkeletonGCL, which explicitly explores the rich semantic context across sequences. Concretely, SkeletonGCL contrasts the learned graphs among sequences, guiding the graph representations to be class-associated, hence improving GCN capacity to recognize different actions. We improve the current methods significantly to achieve SoTA on three benchmarks. \\
\noindent \textbf{Limitation.} In this paper, we push away the negative pairs from different classes in the same way without considering their intrinsic relations. Therefore, a comprehensive contrasting manner may be more suitable by delicately involving cross-class relations. We leave this for future work.

\section*{Acknowledgements}
This research is supported by the NSFC (grants No. 61773176 and No. 61733007).