Motivating a preregistration (especially in experimental linguistics)
The disorder
Studies in the cognitive sciences often feature multiple experimental conditions and other independent variables. Once the study progresses, the analytical liberties associated with all the conditions and variables are compounded by the myriad possible steps in data processing and analysis. All these combinations lead to a garden of forking paths, and the researcher degrees of freedom soar to unexpected highs.
The system of professional incentives in academia largely ignores the issue of researcher degrees of freedom. When CVs are assessed, quantity weighs more than quality of research. When studies are assessed, statistical significance weighs more than methodological rigour. Thus, the low replication rates in various fields, including linguistics (Kobrock & Roettger, 2023), are hardly surprising (cf. Barsalou, 2019).
The treatment
The number of researcher degrees of freedom, and their unforeseen influence, can be reduced a priori by publishing a preregistration before data collection has begun (or before the analysis in the case of meta-analyses and secondary-data analyes). The preregistration takes a bit of time, which poses a challenge because funding systems often require doing things as soon and as fast as possible, driven by a questionable notion of scientific productivity.
How to gather the strength
The best argument to motivate a preregistration may be that does not incur any extra time. It only requires frontloading an important portion of the work. This effort will be rewarded when the study is published, as preregistered analyses are received with greater trust by peer-reviewers and other readers.
If any contributors of a project can gather just enough time and interest to initiate a preregistration in time, the researchers can attempt to pocket this important asset for their study. They will reap the reward in time, and so will their field of research at large.
What to do
Initially, rather than detailed analytical steps, it is the broader theoretical aspects that should be laid out. An abstract and an introduction should be written, describing why the study has been designed in such a way, what analyses will be performed, and what hypotheses are afforded by the literature. Next, some methodological details regarding the materials and the analyses should be added.
The degree of detail in the preregistration can be determined by the researchers alone. Yet, ceteris paribus, the greater the detail in a preregistration, the greater the trustworthiness of the analyses and the results. Multiple preregistration guidelines exist by now, including some field-specific ones.
Solace
Preregistration is not perfect, but is a lesser evil that reduces the misuse of statistical analysis in science (Mertzen et al., 2021; Roettger, 2023).
Even the strongest blizzards start with a single snowflake. (Sara Raasch)
Rules are made to be broken—at least as and when necessary. That is, deviations from the preregistration are possible, and indeed very frequent (Bakker et al., 2020; van den Akker et al., 2023).
References
Bakker, M., Veldkamp, C. L. S., Assen, M. A. L. M. van, Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Barsalou, L. W. (2019). Establishing generalizable mechanisms. Psychological Inquiry, 30(4), 220–230. https://doi.org/10.1080/1047840X.2019.1693857
Kobrock, K., & Roettger, T. B. (2023). Assessing the replication landscape in experimental linguistics. Glossa Psycholinguistics, 2(1). https://doi.org/10.5070/G6011135
Mertzen, D., Lago, S., & Vasishth, S. (2021). The benefits of preregistration for hypothesis-driven bilingualism research. Bilingualism: Language and Cognition, 24(5), 807–812. https://doi.org/10.1017/S1366728921000031
Roettger, T. B. (2021). Preregistration in experimental linguistics: Applications, challenges, and limitations. Linguistics, 59(5), 1227–1249. https://doi.org/10.1515/ling-2019-0048
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023). Selective hypothesis reporting in psychology: Comparing preregistrations and corresponding publications. Advances in Methods and Practices in Psychological Science, 6(3), 25152459231187988. https://doi.org/10.1177/25152459231187988