Identifying related landmark tags in urban scenes using spatial and semantic clustering

Phil J. Bartie, William A. Mackaness, Philipp Petrenz, Anna Dickinson

Abstract: Highlights • Web based experiment to identify useful landmarks for navigation from urban images. • Development of a spatial-semantic clustering technique to generate network graphs. • Use of trigram phrase matching with stop word and synonym lists. • Two phase bag-of-words query expansion linked to different types of landmarks. Abstract There is considerable interest in developing landmark saliency models as a basis for describing urban landscapes, and in constructing wayfinding instructions, for text and spoken dialogue based systems. The challenge lies in knowing the truthfulness of such models; is what the model considers salient the same as what is perceived by the user? The method developed in this research identifies related annotated tags supplied from a web based experiment in which users were asked to tag the most salient features on urban images for the purposes of navigation and exploration. The tag collections may be used to rank landmark popularity in each scene, but the challenge is in determining which tags relate to the same object (e.g. tags relating to a particular café). Existing clustering techniques did not perform well for this task, and it was therefore necessary to develop a new spatial-semantic clustering method which considered the proximity of nearby tags and the similarity of their label content. The annotation similarity was initially calculated using trigrams in conjunction with a synonym list, generating a set of networks formed from the links between related tags. These networks were used to build related word lists encapsulating conceptual connections (e.g. church tower related to clock) so that during a secondary pass of the data, related network segments could be merged. This approach gives interesting insight into the partonomic relationships between the constituent parts of landmarks and the range and frequency of terms used to describe them.