Stochastic gradient descent (SGD) is the main approach for training deep networks: it moves towards the optimum of the cost function by  iteratively updating the parameters of a model in the direction of the gradient of the loss evaluated on a minibatch. Several variants of SGD have been proposed to make adaptive step sizes for each parameter (adaptive gradient) and take into account the previous updates (momentum). Among several alternative of SGD the most popular are AdaGrad, AdaDelta, RMSProp and Adam which scale coordinates of the gradient by square roots of some form of averaging of the squared coordinates in the past gradients and automatically adjust the learning rate on a parameter basis. In this work, we compare Adam based variants based on the difference between the present and the past gradients, the step size is adjusted for each parameter. We run several tests benchmarking proposed methods using medical image data. The experiments are performed using ResNet50 architecture neural network. Moreover, we have tested ensemble of networks and the fusion with ResNet50 trained with stochastic gradient descent. To combine the set of ResNet50 the simple sum rule has been applied. Proposed ensemble obtains very high performance, it obtains accuracy comparable or better than actual state of the art. To improve reproducibility and research efficiency the MATLAB source code used for this research is available at GitHub: https://github.com/LorisNanni.

Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks

"Too Long; Didn't Read": a short sentence describing your paper

Abstract of paper. Add TeX formulas using the following formats: $In-line Formula$ or $$Block Formula$$

Search author profile by first, middle and last name or email address. If the profile is not found, you can add the author by completing first, middle, and last names as well as author email address.

Similarly, please either provide links to the datasets used (comma-separated if necessary) or an explanation why they cannot be published.

In addition to original work, we also specifically accept short papers discussing recently published or submitted journal contributions to give authors the opportunity to present their work and obtain feedback from conference attendees that they would not otherwise receive. Double submissions to other conferences are NOT acceptable. Please select the status of your paper below.

What best describes the contribution of your paper?

Upload a PDF file that ends with .pdf. Please carefully read the author instructions at https://2021.midl.io/author-instructions.html and in the call for papers at https://2021.midl.io/call-for-papers.html. Short papers are subject to a page limit of 3 PAGES TOTAL including references, with no additional allowance for appendix nor supplementary material. PDFs exceeding 3 pages or not using the official style will be desk-rejected. Submissions are single-blind: please include full author names and affiliations in the paper.

MIDL highly encourages open access for the data and code, and reviewers are instructed to take availability of source code and data as well as reproducibility into account. We therefore highly recommend to publish the source code together with the paper and provide the URL in the field below. If you feel that you cannot publish the source code at this point or this does not apply to your submission (e.g., your paper is theory-only), please provide an explanation of the specific reasons instead. 

Title of paper. Add TeX formulas using the following formats: $In-line Formula$ or $$Block Formula$$

Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks

Loris Nanni, Gianluca Maguolo, Alessandra Lumini