Tracking and Mainstreaming Replications across the Social, Behavioral and Cognitive Sciences

Helena Hartmann, Leticia Micheli, Flavio Azevedo

Funding Area: Critical shared infrastructure / Infraestructura compartida critica

Problem Statement: Replication studies test the same hypotheses using comparable methods and statistics and are pivotal to verifying previous results, accumulating knowledge, and accelerating scientific progress. Despite its significance, replications are cited at a lower rate than original studies, producing a skewed view of the literature. This disparity creates consequential challenges for the scientific community and society at large, potentially misleading policymakers, funding agencies, and decision-makers who may allocate resources to areas lacking robust evidence. The lack of visibility of replication efforts is exacerbated for people from low and middle-income economies (LMIEs), who often face barriers to accessing scientific papers hidden behind prohibitive paywalls. Limited access to scientific work hampers the progress of replication studies and perpetuates a cycle of information disparity. The limited access to scientific work obstructs the progress of replication studies, perpetuating a cycle of information disparity. Consequently, LMIE researchers and educators are disadvantaged by additional barriers in staying up-to-date with the latest developments, which may inadvertently lead them to base their research and teaching on outdated or disproven studies, hindering progress and perpetuating misconceptions. This not only impairs the quality of research and education but also inhibits the growth of local scientific communities and the generation of context-specific knowledge.

Proposed Activities: In 2020, FORRT initiated a crowd-sourced collection of replication effects (forrt.org/reversals). Our vision is to create a culture in academia where replications are embedded and valued activities in both research and education. The IOI Open Infrastructure fund will help expand the reach and impact of our database through the following activities in 2024. 1) Building & maintaining the FORRT Replication database: Around 100 scholars have already contributed over 440 replication studies in 20+ disciplines across the social, cognitive, and behavioral sciences. To maximize the database’s value and impact, we aim to expand it to 800+ completed effects across several disciplines at the start of 2024. To increase interoperability, we will integrate data from existing projects (Replication Database & Curate Science) into a consolidated open platform. We will conduct hackathons in January and March to crowdsource contributions from the community (improving existing entries and adding new ones). These events will be advertised across underrepresented fields and in LMIEs, aiming to increase the interdisciplinarity and reach of the database. 2) Building accessible, interactive online apps: Two online, interactive, and user-friendly apps using the Shiny package in the R infrastructure will be created. The first app, Replicability Annotator, supports educators by allowing the uploading of reference lists and identifying replicated studies and their status. The second app, Replication Explorer, enables the exploration and visualization of replication effects and allows meta-scientific questions to be investigated. We will hold two focus groups in May to discuss initial versions of the apps, and users’ needs, paying special attention to the needs of scholars in LMIEs. Following a development stage, we will run two hackathons to stress-test the apps, identify bugs, collect community feedback, and prepare them for release by December 2024 (including the associated R package). 3) Dissemination - Teaching, community interaction, and feedback: We will publish and disseminate the FORRT Replication database and apps within and outside of the scientific community to promote wider recognition of replication studies and broader adoption of replications as a tool to evaluate scientific results critically. We will further disseminate through workshops, YouTube tutorials, teaching, and community interactions from mid to end of 2024, focusing on low-barrier educational events and the active involvement of scholars from LMIEs. Together, these activities will produce valuable research and pedagogical tools to advance Open Scholarship practices for multiple target audiences. Researchers can find updated information on replication and generate new meta-analytic knowledge. Educators can incorporate replications into their teaching. Consumers of science can interact with updated scientific output and learn about the most credible evidence for interventions and policy-making.

Openness: The proposed work aligns with the principle of openness in four ways aiming to create equity and social justice, thus leveling the playing field for people from LMIEs. First, FORRT’s Replication database is made fully openly available on FORRT’s website & GitHub, to be continuously updated. The two ensuing apps will also be openly accessible. We will ensure through workshops and video tutorials (in different languages) that the apps are user-friendly and understandable to both scholars and the lay public. Second, the resource is community-driven, with anybody interested in contributing being able to join. We have an openly available Slack workspace to connect people working together, currently including 880+ members and a regular newsletter with information on how to get involved. In addition, we encourage contributions from scholars in all career stages from varied countries and disciplines by hosting multiple online hackathons in different time zones. Third, contributors to the resource are formally recognized on the website and have the opportunity to become co-authors in subsequent publications related to the project. We also strive to pay contributions going forward to incentivize contributions to the resource. Fourth, by fostering international partnerships and facilitating collaboration and resource-sharing, our initiative can empower scientists from LMIEs to conduct replication studies and contribute more effectively to the global scientific discourse.

Challenges: As FORRT is a volunteer-based organization, a major challenge is to oversee contributions from all over the world and reply to queries in a timely manner. We also aim to alleviate this challenge by paying for project managers’ time to facilitate the process of overseeing and managing contributions. An important challenge is related to the community building and engagement aimed at LMIE. As active involvement from underrepresented scholars is crucial, we will be particularly keen on using effective and kind communication, flexible incentives, and tailored outreach efforts and strategies. Similarly, understanding and addressing the specific needs of educators, researchers, and consumers of science, particularly those from LMIEs, is crucial for the success of the apps. Conducting focus groups and obtaining feedback from target users will require significant efforts toward effective and kind coordination. Another challenge regards the sustainability and long-term support for the two apps (and FORRT in general). After the initial development phase, maintaining the database, apps, and associated infrastructure over the long term requires sustained funding and institutional support. Ensuring the continuity and accessibility of the tools is essential for lasting impact. Lastly, measuring the impact of the project on academic culture, community building, replication practices, and open scholarship is a tall task requiring thoughtful evaluation metrics to gauge its effectiveness.

Neglectedness: Funding FORRT and its initiatives toward a sustainable path is our organization's single, hardest challenge. FORRT has tried and failed to obtain funds for several years as support for an organization like FORRT is virtually non-existent (IOI being a welcome exception). Many of the pathways we tried have dismissed our funding needs based on the very characteristics we are proud of: ‘too’ interdisciplinary, international, educational, de-centralized, non-institutionalized at a given University, volunteer-based, and comprised of early-career scholars. Recently, FORRT secured €50k funding through the Dutch Research Council (NWO) Open Science Fund. Despite being rated 1st among other applications, we faced a significant setback when we realized the funding didn't cover (hidden!) overhead at our University, affecting the hiring of the project manager (which now needs to be internal from the University) and teaching relief (discounted in 60%), hindering our ability to progress. The grant proposal details can be found here: https://doi.org/10.53962/r4qf-7peg. Most importantly, the present application to IOI requests funds for *unique* aspects of the execution of our plans, in line with a focus on LMIEs and the general sustainability of the project. We are obtaining a legal status so that we are no longer beholden to resource-hungry institutions and instead partner with similarly aligned organizations for fiscal sponsorship (eg, OLS) to achieve our goals more efficiently.

Success: The success will be measured in the following deliverables: Building and maintaining the FORRT Replication database: 1) Organization of three hackathons to extend FORRT’s Replication database, 2) Reaching at least 800 effects in the FORRT’s Replication database. Building accessible, interactive online apps: 1) Organization of two focus groups to test the two Shiny apps, 2) Launch of the two shiny apps to engage with FORRT’s Replication database. Dissemination through teaching, community interaction, and feedback: 1) Publication of an open-access manuscript presenting FORRT’s Replication database and its associated Shiny apps, 2) Participate in at least three conferences to discuss FORRT’s Replication database and associated Shiny apps and gather community feedback, 3) Create a YouTube Tutorial in six languages (English, Portuguese, Spanish, Chinese, Hindi, and German) on how to interact with the Shiny Apps and FORRT’s Replication database, 4) Give at least three presentations/workshops on FORRT’s Replication database, which will be made openly available.

Total Budget: US$ 21.500

Affiliations: This proposal is affiliated with FORRT, the Framework for Open and Reproducible Research Training (www.forrt.org; [e-mail]).

LMIE Carveout: The project fits into the category “services that are widely adopted by communities in LMIEs”, as we actively strive to include and promote work from individuals of LMIEs. Moreover, many of our active FORRT members and project contributors come from LMIE, and our final resource will be designed and distributed in a way that benefits these communities most.

Team Skills: Helena Hartmann, a German community manager at FORRT, is leading the project by managing the extension of the database and its contributors. She will contribute to event organization and dissemination due to her long-standing expertise in public outreach. Leticia Micheli, a Brazilian community manager of FORRT, will aid in organizing and leading workshops. In that, she will leverage her extensive with teaching and outreach at FORRT (e.g. initiatives to integrate Open Science practices into teaching and mentoring and to lower barriers to the adoption of OS practices). Flavio Azevedo, a Brazilian researcher in the Netherlands, is the co-founder and director of FORRT and has extensive experience in project management and leading large teams. He is a pioneer in developing tools and practices in OS and an advocate for a more diverse, equitable, and inclusive open science movement. He has received prestigious awards for OS, including the UKRN Dorothy Bishop Prize, Hidden-REF, and JISC. We also leverage the international FORRT community, which, since 2018, has demonstrated a strong track record of completing big-team science projects with high-impact publications and resource dissemination. FORRT includes diverse researchers and educators across all career stages with a wide array of expertise. Lastly, we will collaborate with other replication databases to integrate and merge efforts (e.g., https://metaanalyses.shinyapps.io/replicationdatabase/ & https://curatescience.org/).

Submission Number: 128