Dear all reviewers,

Here is what we discussed,

1. An additional evaluation metric with PRAUC
2. The complexity discussion of SoM-TP with big-O.
3. An additional ablation study on SoM-TP with a perspective loss with histogram.
   1. DPLN pulls the classification network’s output while finding the optimal classification result.
4. The extended experiment on a large real-world dataset and NLP.

and additional ablation study on SoM-TP with SVM and kernel,

5. With pooled vectors(before fully-connected layers), we draw the decision boundary of SVM with various kernels and found that SoM-TP’s decision boundary classifies the features more than other temporal poolings.

As the discussion period is ending nearest, we sincerely thank all reviewers for their valuable comments. It was a great honor to have discussions with all reviewers.

Thank you.

Dear reviewers,

The discussion period is ending this week, so we look forward to more discussion. We would be happy to have further discussion with an updated manuscript.

• Additionally, we acknowledge that our first manuscript was in short supply. So, we tried hard to reflect the feedback of all reviewers during the rebuttal period. We would like all reviewers to know that we are just doing our best all the time; when we first submitted the manuscript, during the rebuttal period, and now in the discussion period.

We really appreciate all reviewers' time, effort, and valuable comments.

Thank you.

We thank again the reviewers for their thoughtful effort in their review and discussion. And here is the summary of the “Discussion” and “Extended Experiment”.

• First, as for the “Discussion”, we discussed the complexity of SoM-TP compared to other existing temporal poolings (with reviewer 2KQe).

  For complexity in the concentrate on each pooling layer and whole model optimization are as follows,

  • GTP: $\mathcal{O}(1)$ for pooling complexity, and $\mathcal{O}(N)$ for training optimization complexity.
  • STP: $\mathcal{O}(L)$ for pooling complexity, and $\mathcal{O}(N)$ for training optimization complexity.
  • DTP: $\mathcal{O}(2\cdot L_{DTW})=\mathcal{O}(L)$ for calculating soft-DTW in forward and backward by GPU, while $\mathcal{O}(L_{DTW}^3)=\mathcal{O}(L^3)$ in increment in complexity by CPU for pooling complexity, and $\mathcal{O}(N+w_{dtp})=\mathcal{O}(N)$ for training optimization complexity with additional optimization for soft-DTW layer.
  • SoM-TP: $\mathcal{O}(L_{conv}+2\cdot L_Q\cdot L_K)=\mathcal{O}(L^2)$ for pooling complexity to compute attention score to dynamic pooling selection, ($Q,K$ for query and key, respectively). And $\mathcal{O}(N+w_{dtp}+w_A+n_{DPLN})=\mathcal{O}(N)$ for training optimization complexity with additional optimization for attention weight $A$ and DPLN.
  • In the inference process, optimization complexity becomes all $\mathcal{O}(1)$ and only pooling complexity remains.
  • Here, we considered pooling complexity except for calculating the maximum or average value which has in common.

  There are trade-offs between complexity and performance because increasing complexity and capacity generally make better performance. And SoM-TP which has the highest complexity and capacity achieved the best performance on TSC. To this end, the increment in complexity is needed to reflect various poolings in the context of perspective. Therefore, SoM-TP has a little degradation in complexity but has also achieved robust performance.

  • As for the notation, $w$ is the weight of the parameter and $n$ is the weight of the layers. We are planning to add this discussion in section 3.2.2.

• Second, as for the “Extended Experiment”, we have done an additional ablation study on SpM-TP with perspective loss. We draw a histogram with several epochs of training which indicate the distribution in comparison with classification network output ($y_{pred1}$) and DPLN output ($y_{pred2}$). This is for all reviewers’ comments on the solidity and novelty of SoM-TP.

  • As for perspective loss,

    $L_{perspective}=KL(y_{pred1},y_{pred2})+L_{DPLN}$

    with $L_{DPLN}$ is cross-entropy loss with $y_{pred2}$. The reason why $L_{DPLN}$ is included in $L_{perspective}$ is that the sub-network should be trained to make the right classification.

  • We found that $y_{pred2}$ distribution pulls $y_{pred1}$ while finding optimal classification results as itself. Therefore, $y_{pred1}$ are tied to $y_{pred2}$’s distribution and indirectly reflected by classification results from all pooling output vectors. And here, the updated distribution with every epoch shows that perspective loss works rightly for diverse perspective learning.

  • We are planning to add this experiment to Appendix A.

We would again like to thank all reviewers for their time and feedback, and we hope that our discussion adequately addresses all concerns. Please let us know if you have any further questions, and we are very happy to follow up!

To all reviewers,

We thank again the reviewers for their review. In response to feedback, we provide an additional general response to “novelty” and an updated manuscript.

• First, we updated the manuscript to effectively deliver SoM-TP’s purpose and goal to diverse perspective learning with detailed explanations and captions with the newly designed figures. Also, the Appendix section is added for additional experiments and results to support SoM-TP for effective comprehension. And here is the summarization of the updated manuscript as follows,
  • Section 1 “Introduction” is updated with the definition of “perspective” of pooling, followed by the meaning of “fixed-perspective learning” and “diverse-perspective learning”.
  • Section 2 “Background” is updated to explain the distinct perspective of existing temporal poolings and also with detailed examples of LRP.
  • Section 3 “SoM-TP: Towards Diverse Perspective Learning” is updated with a detailed notation of the formula. We specifically tried to explain the role of DPLN and perspective loss.
  • Also in section 3 with performance results, the analysis is updated with a quantitative way of tabling average performance and a qualitative way of LRP in Figure 2. Furthermore, we tried to objectively analyze SoM-TP’s robustness with other evaluation metrics, such as histogram and rank bar chart.
  • Table 2 is updated with the average performance of the UCR/UEA repository with various cases of temporal poolings, compared to SoM-TP. Also, ResNet architecture is added.
  • Figure 1 is updated with a detailed notation of vectors and matrixes, and the two fully-connected layers are detailly defined. Also the detailed caption with the flow of the SoM-TP process.
  • Figure 2 is updated by adding the accuracy performance and color to the figure. We circled to where each pooling is focusing on, and samely provide performance below. With detailed explanations in the caption with example-based analysis, we can get through why SoM-TP’s diverse perspective is necessary.
  • Figure 3 is updated with a detailed explanation in the caption. We tried to explain how SoM-TP finds optimal attention scores to reach diverse perspective learning.
  • Figure 4 is added, which is to show an objective analysis of SoM-TP. The histogram and bar chart is selected to see SoM-TP’s outperformance.
• Second for the updated Appendix as follows,
  • In A.1. a detailed experimental setting, specifically for the convolutional stack is explained in the figure, containing embedding dimensions and overall architecture.
  • In A.2. an ablation study in $\lambda$ is added. We designed the experiment to show how perspective loss with $\lambda$ decay affects the SoM-TP in performance. And we found optimal $\lambda$ for the datasets.
  • In A.3. an extended analysis of SoM-TP with an individual dataset with varies time length, data size, and dimension, to see if SoM-TP robustly works in any characteristic of the dataset. And figure shows that SoM-TP dynamically selects each data with appropriate pooling in any dataset characteristics.
  • In B. the DTP algorithm is introduced.
  • In C. the extended related work is introduced.

To all reviewers,

We thank the reviewers for their thoughtful and constructive review. In response to feedback, we provide a general response to points raised by multiple reviewers and an updated manuscript.

• Experiment Result and Performance: SoM-TP has a robust and highest performance overall.

• We updated more clear experiment results through both quantitative analysis and qualitative analysis. For the quantitative analysis, three evaluation methods are used to compare SoM-TP to other pooling methods: 1) Average performance comparison with Table2, 2) histogram comparison between SoM-TP and DTP which is SOTA of temporal pooling, and 3) rank bar chart.

  • First, the Average performance of SoM-TP outperforms all the temporal poolings in Table 2 as mentioned in the second bullet point.
  • Second, we calculated histograms under-area to see how many datasets cases in which SoM-TP and DTP perform better than each other. and SoM-TP also outperformed DTP with a big gap in Figure 4.
  • Finally, the bar chart (Figure 4) shows SoM-TP has more robust results than other temporal poolings from the fact that SoM-TP has the most number of rank 1 and the least number of rank 4.
  • A detailed explanation of these three evaluations is in section 3.2.3.

• The optimal $\lambda$ is applied through the hyperparameter search, and SoM-TP outperforms all the other temporal pooling methods.

  • In Table 2, you can check that SoM-TP outperforms all the other temporal pooling methods (including DTP) based on the average performance of all datasets.
  • In detail, for the repository of 112 univariate datasets, SoM-TP MAX shows the best performance in FCN (acc: 0.7503 / f1macro: 0.7212) and ResNet (acc: 0.7690 / f1macro: 0.7398).
  • For the repository of 21 multivariate datasets, SoM-TP also shows better performance than the other pooling method in FCN (acc: 0.6969 / f1macro: 0.6648) and ResNet (acc: 0.6766 / f1macro: 0.6542).
  • If you would like to check more detailed results, please refer to Table 2, Figure 4, and Section 3.2.

• For the qualitative analysis (Figure 2), we can check the LRP results of GTP, STP, DTP, and SoM-TP with a different perspective of each pooling.

  • In the example datasets (CricketZ, Fungi, and WordSynonyms) with input attribution results, we can identify how the diverse perspectives of SoM-TP focus differently with other temporal poolings.

• Furthermore, we are planning to update the Appendix section with additional experiment explanations and results. (e.g. detailed explanation of experimental setting, $\lambda$ ablation study, extended study on large dataset, and NLP).

We would again like to thank all reviewers for their time and feedback, and we hope that our changes adequately address all concerns.

The paper proposes a attention based method that can dynamically select data-specific temporal pooling method from (1) global temporal pooling (GTP), (2) static temporal pooling (STP), and (3) dynamic temporal pooling (DTP). Experiments can show the effectiveness of the proposed method and demonstrate the approach indeed selects different pooling methods for different batches.

Strengths:

• The work is well-motivated given the sufficient analysis on the limitations of each existing temporal pooling method (Section 2.2).
• Experiments can show how different pooling methods vary while changing different batches of data.

Weaknesses:

• The novelty and solidity of the work are insufficient. The work is incremental on top of Lee, Dongha, Seonghyeon Lee, and Hwanjo Yu. "Learnable dynamic temporal pooling for time series classification." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021. given that it aims to select different pooling methods from the above mentioned work based on attention mechanism.
• The proposed method SoM-TP does not perform better than DTP according to Table 2.
• The paper need to be proofread more carefully. Many algorithms and figures are not referenced correctly (many ? reference throughout the paper).
• Experimental results are not clearly explained.

Both clarity and novelty need to be improved:

• Clarity: The motivation is clearly illustrated given the extensive analysis and comparisons in Section 2.2. on the constraints of existing temporal pooling methods. However, the clarity of experiments is not sufficient: (1) what's the embedding dimension? This also relates to the limited reproducibility; (2) which specific dataset (UCR or UEA) is used for a specific result (e.g., dataset for Table 2 is not mentioned). This also relates to the limited reproducibility.

• Novelty: As explained above, the novelty of this work is incremental.

Despite some strengths of this paper (e.g., good motivations, sufficient motivating analysis), there are a few weaknesses that need to be addressed for acceptance: (1) novelty, clarity; (2) experimental effectiveness of the proposed method; (3) writing readiness.

In this paper, a pooling architecture for diverse perspective learning is proposed. The architecture is referred to as switch over multiple pooling (SoM-TP). To motivate their method, the authors argue that one pooling cannot be dominant in various time series data characteristics and thus there is a need for a more robust pooling that encompasses multiple diverse perspectives. Experiments were conducted on UCR/UEA public datasets, the results of which demonstrate the effectiveness and robustness of SoM-TP in comparison with three other pooling methods.

Strengths:

• Existing temporal poolings cannot capture dependencies that may exist within time series data through a fixed perspective learning since a single pooling method considers only one perspective, i.e. has only a single viewpoint. In contrast, SoM-TP is designed to address this problem by leveraging a diverse perspective learning (DPLN) sub-network in its training process that captures various (including both global and local) viewpoints.

• SoM-TP is capable of identifying the characteristics of the data in a specific mini-batch and leverages attention weights to dynamically select suitable temporal poolings based on the identified data characteristics. In a downstream classification scenario, however, the output class is predicted considering all perspectives of pooling in SoM-TP.

• The authors have compared SoM-TP with three pooling methods (GTP, STP and DTP) on 112 univariate and 21 multivariate time series datasets from the UCR/UEA repository stemming from various domains. The results (1) demonstrate that SoM-TP tends to be more robust to data that requires capturing different perspectives; and (2) suggest that SoM-TP is more accurate than fixed perspective learning with GTP and STP, while having similar accuracy to that of DTP.

• The authors have conducted an ablation study that provides an insight into the inner-workings of the diverse perspective learning process.

Weaknesses:

• The proposed SoM-TP method appears to simply combine several components including different pooling methods (GTP, STP and DTP) and an attention mechanism, which are all already well-established in the literature. Even the DPLN sub-network is a simple classification network that considers all pooling perspectives to make a final decision, which limits the methodological novelty of this work.

• SoM-TP is designed to dynamically select suitable temporal poolings based on certain characteristics that were identified in a certain portion (mini-batch) of a dataset. Nevertheless, neither have the characteristics (identified in each of the considered datasets) been discussed in the paper; nor have the authors discussed how the descriptive statistics (such as size, dimensionality, time series length, etc.) of different datasets impact the dynamic selection of the temporal poolings. I would encourage the authors to discuss and provide more clarity on the aforementioned two points in their repose.

• For one to select the most suitable pooling method for a certain mini-batch, naturally, all three considered pooling operations would need to be applied to the hidden features with temporal position information in each feed-forward step of the overall architecture. This increases the complexity of the training as well as the inference stage, however, the complexity of neighter is discussed by the authors. I would suggest that such a discussion is included in both the authors’ response as well as in the paper.

• The tradeoff parameter (or decay) $\lambda$ in Eq. (6) appears to have an important role as it controls the influence of the perspective loss. That being said, I am wondering why the authors have not analyzed the effect that different values of $\lambda$ may have on the downstream time series classification performance.

• Average accuracy is considered as a classification performance metric, however, to my knowledge some of the considered UCR/UEA datasets are rather imbalanced, thus the authors should have considered metrics that are less sensitive to class imbalance (e.g., area under the precision-recall curve (AUPRC)).

Minor weaknesses: There are also certain grammatical and typographical errors and remarks that require attention. Some of them are summarized as follows:

• In the next-to-last paragraph of the Introduction section on page 2, “all learnable weight” should be replaced with “all learnable weights”.
• In the “Static Temporal Pooling” paragraph on page 3, replace “segmentation” with “segments”.
• In the “Dynamic Temporal Pooling” paragraph on page 3, there is a missing reference to an Algorithm and another one to an Appendix.
• At the beginning of page 6, there is a reference missing right after “class weight matrix”.
• In the caption of Table 1, “detail” should be replaced with “detailed”.
• In the “Robustness” paragraph on page 7, “Figure 4 (a)” should be corrected to “Figure 5 (a)”. Later in the same sentence, “GTP” should be included between the words “than” and “as”.
• The caption of Figure 5 should be more descriptive instead of simply being “Performance graph”.

Clarity: The paper is decently written, but there is certainly room for improvement when writing is concerned. However, in my view, the paper is not organized and presented well, while being difficult-to-follow, particularly when one reads the experimental section.

Quality: The design and justifications for the proposed SoM-TP method seem to be technically sound. The observations made regarding the robustness and accuracy of SoM-TP seem to hold in the considered settings, however, these observations need to be further supported by conducting additional ablation studies (mentioned in the “Weaknesses” part of this review). Overall, the paper has merit but needs more work as it does not appear to be fully developed in terms of quality.

Novelty: From a methodological perspective, the contribution of this work can be considered rather incremental. In essence, the authors leveraged several existing pooling methods and integrated them into a single architecture that learns attention weights for each of the poolings in other to select the most suitable pooling given a certain batch of data samples (please refer to the first point of the “Weaknesses” part of this review, where this is explained in more detail).

Reproducibility: The experiments were conducted on widely-used time series classification datasets which are publicly available. On the other hand, the code for the proposed SoM-TP method is not made available by the authors in the present anonymized version, however, with some effort one might be able to implement SoM-TP by following Section 3.

This work has merit but does not appear to be well developed. Although the authors provided interesting insights regarding the need for diverse perspective learning for pooling methods, the work, in its current state, lacks methodological novelty; certain key aspects (such as the impact of data size, dimensionality, time series length, etc., on the dynamic pooling selection) are not addressed in the paper; the role of the perspective loss (which can be considered central to this work) has not been analyzed; among other weak points outlined in the “Weaknesses” part of this review. Overall, these weak points of the paper seem to outweigh its strengths. Therefore, I am not convinced that this work is a good fit for ICLR. Nevertheless, I am looking forward to the authors’ response and I would be willing to adjust my score in case I have misunderstood or misinterpreted certain aspects of the work.

Not applicable.

The paper proposes a new method to pool convolutional feature maps for time series. The work proposes to add an addition diverse perspective learning network used only for training and also introduces a new loss. The evaluation is done on over 100 dataset from the UCR/UEA repositories. In addition, a quantitative analysis using LRP is done.

Strengths:

• The idea of using interpretability methods for evaluating is interesting
• The pooling operation is an essential operation – improving it could therefore have a large impact.

Weakness:

• The captions of Table 1, 2 and Figure 4,5 are too short (only a few words). They fail to describe the content. I especially do not understand Figures 4 and 5.

• What is the definition of "diverse perspective learning"?

• There seems to be no improvement over dynamic temporal pooling (DTP) in Table 2. However, Table 2 is not cited in the text, and I am unsure if I understand it correctly.

• The LRP evaluation is incomplete. First, in 3.2.2 it is stated: "A quantitative analysis using LRP is performed to examine how the perspectives of SoM-TP are different from those of other methods." However, in section 3.3 no such comparison between the methods is made.

• Furthermore, the experimental design is not clear and misses details:

  The relevance score, which is input attribution, is the LRP result from the best-performed individual pooling.

  On which pooling method is this based? Is the pooling method selected per individual sample or not? Which LRP rule was used to compute the attribution values?

• The evaluation is done on UCR/UEA datasets only. The proposed pooling method could also be interesting in an NLP setting, and an evaluation of larger datasets/models would be needed.

• The paper lacks substantial clarity and quality (see weaknesses).
• As many important details are not reported, I also question the reproducibility.

The paper is not ready for publication. The main reasons are the incomplete evaluation (LRP and missing larger datasets), and the unfinished manuscript (missing captions, ...), which do not allow accessing this work's merits.