“There is no reason anybody should be using 1D anymore”: Design and Evaluation of 2D Jupyter Notebooks

Jesse Harden, Elizabeth Christman, Nurit Kirshenbaum, Mahdi Belcaid, Jason Leigh, Chris North

Keywords: computational notebooks, space to think, visualization, HCI

TL;DR: The design and evaluation of a Jupyter extension using 2D space to organize computational notebook cells into multiple columns leads the authors to evidence suggesting such 2D computational notebooks may be preferable to current 1D notebooks.

Abstract: Current computational notebooks, such as Jupyter, are a popular tool for data science and analysis. However, they use a 1D list structure for cells that introduces and exacerbates user issues, such as messiness, tedious navigation, inefficient use of large screen space, performance of non-linear analyses, and presentation of non-linear narratives. To ameliorate these issues, we designed a prototype extension for Jupyter Notebooks that enables 2D organization of computational notebook cells into multiple columns. In this paper, we present two evaluative studies to determine whether such “2D computational notebooks” provide advantages over the current computational notebook structure. From these studies, we found empirical evidence that our multi-column 2D computational notebooks provide enhanced efficiency and usability. We also gathered design feedback which may inform future works. Overall, the prototype was positively received, with some users expressing a clear preference for 2D computational notebooks even at this early stage of development.

Track: HCI/visualization

Revision: No

Summary Of Changes:

Qualitative methodology section added to Study 2's methodology section as Data Analysis Process
- open coding was used by the second author, and the other authors gave feedback to refine the themes found; once refinement was done, a second pass to group the themes was performed by the second author.

____________________
Claims about 2D notebooks generally are too strong [R1]
- Softened claims by clarifying their application to multi-column layouts specifically in the discussion section (Section 8) and in the abstract.
No real justification for this specific notebook design [R1,R3]
- Justified notebook design using Harden et al. earlier work which found multi-column to be the most consistent, often used pattern, as well as a pattern used in the more complex grouped combinations workboard subpattern. This is done in Section 3 and reiterated in Section 4.3.
High level of study attrition not explained [R1,R3]
- Clarified that only 31 participants out of the 62 who passed the online screening questionnaire actually scheduled and attended a study session, meaning that there was not really a high attrition rate of participants who were eligible and decided to participate.
Statistical tests need further justification [R2]
- Checked the normality assumption and found the data to be quite non-normal, so replaced the 2-Factor ANOVA with the non-parametric version (Scheirer-Ray-Hare test) and its results. The same tasks were found to be significant for the layout, which is noted in the paper. Also, the code for both the Scheirer-Ray-Hare test and the ANOVA notebook are in supplemental materials in the Github repository, whose link has been updated in the paper.
Study tasks and metrics aren't sufficiently clear [R1,R3]
- Clarified that time to completion was the time to answer the question and press the "Next" button, and that accuracy was the percentage correct across all participants and questions.
Images of the system are not high-resolution or legible enough [R1, R3]
- Updated the images and used more space for the images; these images can also be found in the supplemental materials. We also uploaded the notebooks used in Study 1 to the supplemental materials.
Heatmap figures need to be reworked [R1, R2]
- Reworked heatmap figures with colorblind-friendly palette; an Excel workbook with the heatmaps is available in the supplemental materials on the Github repository for those who wish to highlight the numbers.
Importance of notebooks for reproducible science could be more strongly emphasized [R2]
- Added section to related work on this (Section 2.1.1)
Some small writing issues [R1]
- Fixed the headings issue and moved the study 2 full prompt to supplemental materials in the Github repository.
- Expanded Sections 7.2 and 7.3 with more robust qualitative analysis.
- Reformatted the quotes tables to take up less space.
- Noted that participants were allowed to use Control + F, although it wasn't suggested unless asked about it.
- Clarified relationship between 2D layout and Run All execution order in Section 3.

Supplemental materials are also included in this revision submission for ease of access for reviewers. This also includes the Study 1 questionnaire and the correct answers sheet, but not the data collected at this time. Still, there should be enough in the supplemental materials to allow replication of our study.