Speed Labeling: Non-stop Scrolling for Fast Image Labeling

Chia-Ming Chang, Yi Tang, Xi Yang, Xiang Chen, Takeo Igarashi

Letter Of Changes:

We appreciate your careful review and vulnerable feedback. Below is a summary of the main changes in the camera-ready version.

1. For the formative study, we have modified the following points:

a. We added more explanation for the reasons behind selecting the image layout based on the results of the formative study. The main reason is that the formative study results showed that a single-line layout was faster than a single-image layout and comparable to the grid layout, making it more space-efficient (i.e., the single-line layout is more efficient than the single-image layout).

b. We added more information about the formative study results (captions for Figure 2 and Figure 3). The results present the average time from the formative study (N=4).

c. In the formative study, we also compared different display refresh rates (60 and 120), but it didn't affect the result in a small-and-easy labeling task. Therefore, we did not discuss this part in the previous version. In the camera-ready version. We have mentioned this aspect in the "Limitations and Future Work" section.

2. For the “Speed Labeling” algorithm, we have modified the following points:

a. We removed unnecessary sentences in Section 4.2 (as indicated in the meta review).

b. We corrected mistakes in the algorithm as pointed out by the reviewers (revised to “3at^2 + 2bt + c”).

3. Others

a. In the user study, users were not allowed to modify their results (selected labels). Nevertheless, the question of "how users could modify their results with automatic transition" is very interesting. We have added it as a topic for future work.

b. We corrected mistakes in the captions of the result figures.

c. We carefully reviewed the writing and corrected some small typographical errors.

Keywords: Manual Image Labeling, Non-stop Scrolling, Labeling Efficiency, Human Processor

Abstract: This study presents “speed labeling”, an image-labeling technique to increase the efficiency of easy binary labeling tasks where an annotator can choose a label instantly. We first conduct a formative study to identify the factors affecting the efficiency of easy image labeling: image layout and image transition. Based on these results, we designed a novel labeling technique using non-stop scrolling. In conventional image labeling, the system moves to the next image only after the user assigns a label to the previous image. To maximize efficiency, our technique continuously scrolls images without waiting for the completion of labeling, assuming that the user gives labels at a mostly constant speed. The system dynamically adjusts the scrolling speed based on the labeling speed. Subsequently, we conduct a user study to compare the proposed “non-stop scrolling” technique to the conventional “stop-and-go scrolling” technique in an easy image-labeling task. The results showed that speed labeling requires less time (faster by 7%, 305 more images labeled per man-hour) to complete the labeling task than the conventional technique without a significant increase in errors. In addition, the results showed that speed labeling makes the labeling task more enjoyable for crowd workers and makes them feel more attentive during tasks.

Submission Number: 12