Pen+Touch+Midair: Cross-Space Hybrid Bimanual Interaction on Horizontal Surfaces in Virtual Reality

Fabrice Matulic, Daniel Vogel

Keywords: Pen and touch input, bimanual interaction, hand gestures, VR

TL;DR: We present a design space for hybrid bimanual pen + touch input extended to midair interaction in desktop-based virtual reality

Abstract: We present and explore a design space for hybrid bimanual pen and touch input extended to midair interaction in desktop-based virtual reality (VR). The investigation focuses on asymmetric interaction patterns combining the pen with the other hand when interacting in the same “space” (either surface or midair), across both spaces, and with cross-space transitions (from surface to midair and vice versa). To show how these interactions and associated gestures can work in context, we create three testbed applications for modelling, volumetric rendering, and terrain editing. A qualitative evaluation with 16 participants provides insights into hand and space preferences for key tasks including object manipulation, navigation, and menu invocation. From the results, design implications are identified for VR systems that combine pen, touch, and midair input, and extensions to other forms of extended reality are discussed.

Track: HCI/visualization

Summary Of Changes:

We thank the reviewers for their comments. Here are the revisions we made based on their feedback as well as responses to some of the comments.

*Metareview*

- Explain the design reason for choosing these particular interaction techniques.

We made sure that each application and interaction technique has a justification for inclusion as demonstrating example of the design space. We emphasised when a particular interaction was filling a gap unexplored by prior work, cited the literature when it was inspired by prior techniques, and stated when interactions were included as alternative for comparison. We expanded subsection 4.6 to clearly summarise how our techniques cover the design space.

- Explain how users could make use of the design space.

We think the 23 techniques already provide very concrete examples of how the design space can be used, but the discussion covers more general considerations to apply the design space in other scenarios and we clarified the main takeaways, which include design recommendations that the reader can quickly refer to, via a numbered list.

- Make references to prior work and explain how they inspired the design.

The related work section and the introduction of the design space provide the general context of how prior work inspired the design space (pen+touch, continuous interaction space etc.). Furthermore, in the description of the applications, for each technique we referenced relevant prior work and explicitly stated when a gesture was novel, when we couldn't find a similar prior technique.

*Comments by other reviewers*

- Reference to ErgoDesk

We added that reference

- Explicitly describe desktop VR setup

In the introduction, we clearly specify our context: "bimanual pen and barehand input on and above a horizontal desk-bound interactive surface in XR" and in section 4.1 we describe the actual setup. Figure 1 further shows "how the 2D surface was placed and aimed for inside the VR view".

- Not clear how interactions referenced in section 2.1 can be used in VR.

This small section is just to explain how pen, touch and midair input has been considered outside of XR. We added that the main difference with XR is that in XR objects can be manipulated in 3D space. When describing our interactions, we referred to relevant non-XR techniques to include that context.

- Discussion of display space

We touch on this in point 6 of the discussion, but the question of where to display content generally depends on the type of content. For 2D objects, such as UIs, the problem of display space is related to direct and indirect input, which we address. We further emphasise the precision benefits of a hard surface, which translate to smooth UI operation.

- Why evaluating these techniques makes sense for the evaluation of the design space.

We do not aim to "evaluate the design space" as this is impossible, but to "form a broader analysis and discussion on [its] practical aspects" and "to identify issues of interest at a more granular level for future focused investigations" as stated in the introductory paragraph of section 5. The bimanual techniques cover important types of interactions in VR, such as object manipulation, menu invocation and navigation, and through our qualitative evaluation, we gain a better understanding of issues surrounding some of the practical implications of design decisions, which we report in the results and the discussion, but of course it's impossible to be exhaustive, and that's not our goal.

- Subsection 4.6 just a single sentence and s table

We expanded that subsection by adding a summary of how our techniques covered the design space.

- Inclusion of participant quotes

The paper is already quite long and we feel adding participant quotes would further increase its length for little added value, as we already summarised the most salient observations and participant feedback.

- difficult to map the gestures of to the names used in section 5.2.

We improved clarity by using the same names and italics to refer to the gestures in that section.

- Integrate subsection 6.1 into section 3

Done.

- Et al -> et al. - Add spaces between the numbers in the references

Done.

*Other edits*

We made numerous other cosmetic edits and style improvements to make the paper more readable.