Multimedia COMMONS Workshop 2016 (MMCommons 2016): Datasets, Evaluation, and Reproducibility

Bart Thomee, Damian Borth, Julia Bernd

Abstract: Leveraged wisely, new datasets can inspire new multimedia methods and algorithms, as well as catalyze innovations in how their efficacy, efficiency, and generalizability can be evaluated. The availability of very large multimedia datasets like the Yahoo-Flickr Creative Commons 100 Million has offered unique opportunities for advancing the state of the art in multimedia processing, analysis, search, and visualization. The Multimedia Commons Initiative has been developing a community around the YFCC100M, including associated annotation and evaluation efforts. In addition to research in several multimedia subfields, including computer vision, image processing, and video content analysis, the YFCC100M and Multimedia Commons resources have been used in various competitions and benchmarks, such as the MediaEval Placing Task and the ACM Multimedia Grand Challenge competition. With additional annotation and curation, the data has the potential to enable major leaps forward in research. As use of the YFCC100M and the Multimedia Commons resources broadens across the multimedia community, the MMCommons'16 workshop offered an opportunity for researchers to share new research results, compare approaches, and coordinate efforts to maximize the scientific benefit of the initiative. In particular, the Multimedia Commons data has the potential to provide both inspiration and concrete resources to pursue some important "meta-research" questions, such as how to measure the scalability, generalizability, and reproducibility of methods across datasets, whether we need to rethink our evaluation paradigms as the field moves in new directions, and how annotation strategies affect the impact of benchmarks and data challenges using that data.