Pablo Bernabeu

You shall know a word by the company it keeps — so choose your prompts wisely

Tue, 24 Mar 2026 00:00:00 +0000

In computational linguistics, word meanings are shaped by their contexts. As the British linguist John Rupert Firth put it in 1957, ‘You shall know a word by the company it keeps’ (see Brunila & LaViolette, 2022, for a re-examination of the intellectual history). It sounds almost like life advice, but Firth meant something technical: words that habitually appear alongside each other tend to share semantic territory. The adjective ‘good’, for instance, is far more likely to appear near ‘kind’, ‘genuine’, ‘fair’ and ‘quality’ than near ‘broken’ or ‘fraud’ – and a model that tracks those neighbours can learn what ‘good’ means without ever being told. The principle extends to polysemy: ‘bank’ means something entirely different in the company of ‘river’ and ‘fishing rod’ than in the company of ‘loan’ and ‘account’. Context is everything.

This deceptively simple insight is the bedrock on which generative AI was built. The earliest computational implementations of Firth’s principle, distributional semantic models such as Latent Semantic Analysis (LSA; Landauer & Dumais, 1997) and the Hyperspace Analogue to Language (Lund & Burgess, 1996), were modest by today’s standards: a matrix of word co-occurrence counts, a few hundred latent dimensions and a vocabulary of perhaps tens of thousands of words. Yet even these pocket-sized models captured real-world structure with startling fidelity. Louwerse and Zwaan (2009) showed that the frequency with which city names co-occur in English text predicts their actual geographical distances: cities close together on a map tend to be mentioned together more often, and an LSA model trained on text alone can reconstruct approximate maps of the United States without ever seeing one.

Distributional models can also track sensorimotor properties of concepts (Bernabeu, 2022; Louwerse, 2011; Louwerse & Connell, 2011), though unaided text captures only part of the picture – grounding sensorimotor features more fully may require fine-tuning on human ratings (Wu et al., 2026) or other forms of supervision (Xu et al., 2025). Language, then, does not merely label the world – it encodes its structure, and even a simple co-occurrence model can read much of that encoding back.

We can see this for ourselves. The R code included below (click ‘Expand’ to view it) applies LSA – one of the simplest distributional models – to three text collections, projects the resulting word vectors into two dimensions via PCA (principal component analysis) and plots them. In brief, LSA builds a term-document matrix (a large table recording how often each word appears in each document), weights it with TF-IDF (term frequency–inverse document frequency, which highlights words distinctive to particular documents rather than ubiquitous everywhere) and then compresses it via truncated SVD (singular value decomposition, a form of dimensionality reduction). Each corpus is split into two groups: the most distinctive words per group (selected by the difference in mean TF-IDF weight between groups) are plotted in the group’s colour, while the most frequent shared words appear in purple. Words that co-occur in similar contexts cluster together; words from different domains drift apart.

PCA works by finding new axes, principal components, that capture the maximum variance in the data. Each word receives a loading on each component: a number ranging from −1 to +1 that indicates how strongly that word contributes to that axis of variation (a gentle introduction to PCA in R is available in an earlier post on this blog). High absolute loadings on a component mean that the word is a strong marker of the distinction that component captures.

How are the thematic groups decided? The code computes the mean TF-IDF weight of every word in each group of documents and then takes the difference. Words whose weight is much higher in group A than in group B are classified as distinctive to A, and vice versa. The top 15 words at each extreme become the coloured labels in the plot, while the most frequent words that do not belong to either extreme are labelled ‘Shared’. The grouping is therefore entirely data-driven: no human decides which words are ‘finance’ or ‘energy’ – the corpus statistics do. Above each plot, a table shows the mean loading of each thematic group on the first two principal components, with the highest positive loading per group highlighted in bold. A high absolute loading tells us that a given group of words is strongly aligned with that component; in other words, the component captures precisely the distinction between those groups. When one group loads heavily on PC1 while another does not, the first principal component is essentially the axis that separates them.

Reuters Newswire: Finance vs Energy

The first corpus uses two classic newswire collections from the tm package (Feinerer et al., 2008): acq (50 Reuters articles on corporate acquisitions) and crude (20 articles on crude oil markets). Both have been standard NLP benchmarks since the 1980s (Lewis, 1997). The code builds a TF-IDF weighted term-document matrix, reduces it to a 20-dimensional LSA space via truncated SVD, and computes pairwise cosine similarities – a standard measure of how close two word vectors sit, on a scale from –1 (opposite) to +1 (identical) – using LSAfun::Cosine() (Günther et al., 2016). The PCA loadings table and word-vector plot below show the results.

pkgs <- c("LSAfun", "tm", "ggplot2", "plotly")
invisible(lapply(pkgs, function(p)
if (!requireNamespace(p, quietly = TRUE)) install.packages(p)))
library(LSAfun)
library(tm)
library(ggplot2)
library(plotly)
# --- Reusable helper: LSA + PCA plot ------------------------------------
# Builds a TF-IDF term-document matrix, computes a truncated SVD,
# selects the most distinctive and most shared words and projects them
# to 2D via PCA *on the selected words only* for maximum spread.
lsa_pipeline <- function(doc_list, labels, grp_a, grp_b,
lab_a, lab_b, colour_a, colour_b,
top_n = 15, n_shared = 10,
k = 20, min_docs = 4) {
corp <- VCorpus(VectorSource(doc_list))
corp <- tm_map(corp, content_transformer(tolower))
corp <- tm_map(corp, removePunctuation)
corp <- tm_map(corp, removeNumbers)
corp <- tm_map(corp, removeWords, stopwords("en"))
corp <- tm_map(corp, stripWhitespace)
tdm <- as.matrix(TermDocumentMatrix(corp,
control = list(weighting = weightTfIdf,
bounds = list(global = c(min_docs, Inf)))))
k_use <- min(as.integer(k), nrow(tdm) - 1L, ncol(tdm) - 1L)
sv <- svd(tdm, nu = k_use, nv = k_use)
wlsa <- sv$u %*% diag(sv$d[1:k_use])
rownames(wlsa) <- rownames(tdm)
idx_a <- which(labels == grp_a)
idx_b <- which(labels == grp_b)
mean_a <- rowMeans(tdm[, idx_a, drop = FALSE])
mean_b <- rowMeans(tdm[, idx_b, drop = FALSE])
total <- mean_a + mean_b
spec <- mean_a - mean_b # positive = distinctive to A
top_a <- names(sort(spec, decreasing = TRUE))[1:top_n]
top_b <- names(sort(spec, decreasing = FALSE))[1:top_n]
shared_pool <- setdiff(names(sort(total, decreasing = TRUE)),
c(top_a, top_b))
shared <- head(shared_pool, n_shared)
hl <- unique(c(top_a, top_b, shared))
hl <- hl[hl %in% rownames(wlsa)]
# PCA on the selected words only, for better spatial spread
wlsa_hl <- wlsa[hl, , drop = FALSE]
pca <- prcomp(wlsa_hl, scale. = FALSE)
cd <- data.frame(PC1 = pca$x[, 1], PC2 = pca$x[, 2],
word = rownames(wlsa_hl))
cd$topic <- ifelse(cd$word %in% top_a & !cd$word %in% top_b, lab_a,
ifelse(cd$word %in% top_b & !cd$word %in% top_a, lab_b,
"Shared"))
p <- ggplot(cd, aes(PC1, PC2, colour = topic,
text = paste0(word, " (", topic, ")"))) +
geom_point(size = 0, alpha = 0) +
scale_colour_manual(values = setNames(c(colour_a, colour_b, "#7B2D8E"),
c(lab_a, lab_b, "Shared")),
guide = guide_legend(override.aes = list(size = 3, alpha = 1))) +
labs(x = "Principal Component 1", y = "Principal Component 2",
colour = NULL) +
theme_minimal(base_size = 12) +
theme(legend.position = "bottom",
legend.margin = margin(t = -5),
axis.title.x = element_text(margin = margin(t = 12)),
axis.title.y = element_text(margin = margin(r = 12)),
plot.margin = margin(0, 0, 0, 0))
# Map each word to its group colour for label text
col_map <- setNames(c(colour_a, colour_b, "#7B2D8E"),
c(lab_a, lab_b, "Shared"))
cd$label_col <- col_map[cd$topic]
# Trim spatial outliers so the dense cluster is readable.
# Words beyond the IQR fence are dropped from the plot (not from LSA).
q1 <- quantile(cd$PC1, 0.25); q3 <- quantile(cd$PC1, 0.75)
iqr <- q3 - q1; fence <- 2.5
keep <- cd$PC1 >= (q1 - fence * iqr) & cd$PC1 <= (q3 + fence * iqr)
q1y <- quantile(cd$PC2, 0.25); q3y <- quantile(cd$PC2, 0.75)
iqry <- q3y - q1y
keep <- keep & cd$PC2 >= (q1y - fence * iqry) & cd$PC2 <= (q3y + fence * iqry)
cd <- cd[keep, , drop = FALSE]
pp <- ggplotly(p, tooltip = "text")
# Hide all ggplot traces from plot AND legend
for (k in seq_along(pp$x$data)) {
pp$x$data[[k]]$marker$size <- 0.1
pp$x$data[[k]]$marker$opacity <- 0
pp$x$data[[k]]$showlegend <- FALSE
}
# Constrain axes to the data range (with a small pad)
pad_x <- diff(range(cd$PC1)) * 0.06
pad_y <- diff(range(cd$PC2)) * 0.06
# Add text traces per group (toggleable via legend)
legend_groups <- c(lab_a, lab_b, "Shared")
legend_cols <- c(colour_a, colour_b, "#7B2D8E")
offscreen_x <- max(cd$PC1) + pad_x * 50
offscreen_y <- max(cd$PC2) + pad_y * 50
for (i in seq_along(legend_groups)) {
grp <- legend_groups[i]
grp_data <- cd[cd$topic == grp, , drop = FALSE]
if (nrow(grp_data) == 0) next
# Text trace at actual positions (no legend entry)
pp <- pp %>% add_trace(
x = grp_data$PC1, y = grp_data$PC2,
type = "scatter", mode = "text",
text = grp_data$word,
textfont = list(size = 11, color = legend_cols[i]),
name = grp, legendgroup = grp, showlegend = FALSE,
hoverinfo = "text",
hovertext = paste0(grp_data$word, " (", grp, ")"),
inherit = FALSE
)
# Legend-only marker trace (off-screen, linked via legendgroup)
pp <- pp %>% add_trace(
x = offscreen_x, y = offscreen_y, type = "scatter", mode = "markers",
marker = list(size = 12, color = legend_cols[i], opacity = 1,
symbol = "circle"),
name = grp, legendgroup = grp, showlegend = TRUE,
hoverinfo = "skip", inherit = FALSE
)
}
pp <- pp %>% layout(
legend = list(orientation = "h", x = 0.5, xanchor = "center",
y = -0.12, yanchor = "top",
tracegroupgap = 2, itemwidth = 30,
itemsizing = "constant",
font = list(size = 11),
bordercolor = "#CCCCCC", borderwidth = 1,
bgcolor = "#FAFAFA",
xpad = 4, ypad = 4),
xaxis = list(title = list(text = "Principal Component 1",
standoff = 8),
range = c(min(cd$PC1) - pad_x, max(cd$PC1) + pad_x)),
yaxis = list(title = list(text = "Principal Component 2",
standoff = 8),
range = c(min(cd$PC2) - pad_y, max(cd$PC2) + pad_y)),
margin = list(b = 55)
)
list(plot = pp, lsa = wlsa, tdm = tdm, pca = pca, words = cd)
}
# --- 1. Reuters newswire ------------------------------------------------
data(acq)
data(crude)
docs <- c(lapply(acq, content), lapply(crude, content))
labels <- c(rep("acq", length(acq)), rep("crude", length(crude)))
res1 <- lsa_pipeline(docs, labels,
grp_a = "acq", grp_b = "crude",
lab_a = "Finance", lab_b = "Energy",
colour_a = "#D55E00", colour_b = "#0072B2",
min_docs = 4)
# Cosine similarities in the 20-dimensional LSA space
pairs <- list(
c("oil", "barrel"), c("shares", "acquisition"),
c("price", "barrel"), c("price", "shares"),
c("shares", "oil"), c("acquisition", "barrel"))
pairs <- Filter(function(p) all(p %in% rownames(res1$lsa)), pairs)
sims <- sapply(pairs, function(p)
round(Cosine(p[1], p[2], tvectors = res1$lsa), 3))
names(sims) <- sapply(pairs, paste, collapse = " ~ ")
sims
#> oil ~ barrel shares ~ acquisition price ~ barrel
#> 0.675 0.236 0.938
#> price ~ shares shares ~ oil acquisition ~ barrel
#> 0.129 -0.014 -0.043

Table 1: *Mean PCA Loadings on the First Two Components (Highest Positive Loading per Group in Bold, Excluding Shared)*
Group	PC1	PC2
Energy	.439	-.321
Finance	-.265	.24
Shared	.172	.069

Figure 1: Word Vectors from Reuters Newswire Articles (Finance vs Energy) Projected to Two Dimensions via PCA on a 20-Dimensional LSA Space. Finance terms (vermilion) cluster in a distinct region from energy terms (blue); shared vocabulary occupies intermediate positions. Select an area of the plot to zoom in; double-click to reset.

The cosine similarities confirm what Figure 1 shows geometrically. Within-domain pairs cluster tightly – oil ~ barrel and price ~ barrel have high positive cosines because these words habitually appear together in oil-market dispatches – while cross-domain pairs like shares ~ oil and acquisition ~ barrel sit near zero: they simply never keep each other’s company. Notice, too, that price ~ shares is far lower than price ~ barrel. The same word, ‘price’, lands in a different region of the space depending on the context in which it predominantly occurs. Firth’s principle, made numerical.

State of the Union: Pre-War vs Post-War

From newswire to politics. The sotu package provides the full text of every US State of the Union address. Splitting at 1945, the end of the Second World War, reveals how American political vocabulary has shifted over two centuries: from the constitutional and agrarian language of the early republic to the geopolitical and welfare-state vocabulary of the modern era. The loadings table and figure below present the results.

if (!requireNamespace("sotu", quietly = TRUE)) install.packages("sotu")
sotu_texts <- sotu::sotu_text
sotu_years <- sotu::sotu_meta$year
sotu_labels <- ifelse(sotu_years < 1945, "Pre-1945", "Post-1945")
res2 <- lsa_pipeline(as.list(sotu_texts), sotu_labels,
grp_a = "Pre-1945", grp_b = "Post-1945",
lab_a = "Pre-1945", lab_b = "Post-1945",
colour_a = "#E69F00", colour_b = "#009E73",
min_docs = 5)

Table 2: *Mean PCA Loadings on the First Two Components (Highest Positive Loading per Group in Bold, Excluding Shared)*
Group	PC1	PC2
Pre-1945	-.845	-.499
Post-1945	-.492	.385
Shared	-.379	.132

Figure 2: Word Vectors from US State of the Union Addresses Projected to Two Dimensions, Split at 1945. Pre-war speeches (amber) feature constitutional and agrarian vocabulary; post-war speeches (green) shift to geopolitical and welfare-state terms. Select an area of the plot to zoom in; double-click to reset.

The separation is striking. Table 2 reveals that both groups have negative mean loadings on PC1, so the first component does not cleanly separate them; it mainly captures variance shared across eras (general political vocabulary that appears throughout the full 200-year span). The real separation lives on PC2: pre-war words load negatively while post-war words load positively, confirming that the vertical axis in Figure 2 is the one that distinguishes the two eras. Pre-war presidents address ‘gentlemen’ (the formal salutation of a different era) and discuss ‘vessels’, ‘militia’, ‘commerce’ and ‘treasury’ – the vocabulary of a young republic preoccupied with trade, territorial expansion and the mechanics of governance. Modern presidents speak of ‘tonight’ (State of the Union addresses have been televised since the 1960s), ‘jobs’, ‘nuclear’ and ‘program’ – the vocabulary of a superpower managing a welfare state and a global military presence. Words like ‘congress’, ‘government’, ‘war’ and ‘people’ anchor both eras, sitting in the shared middle ground.

IMDB Film Reviews: Positive vs Negative

Now a harder test. The text2vec package includes 5,000 IMDB film reviews labelled as positive or negative – a classic sentiment-analysis benchmark. Unlike the two corpora above, the split here is not by topic but by evaluative tone. Both positive and negative reviews discuss films, characters, plots and acting; the difference lies in the adjectives and evaluative phrasing. This makes the separation task far harder for a simple co-occurrence model – and the result is instructive. The loadings table and figure below present the results.

if (!requireNamespace("text2vec", quietly = TRUE)) install.packages("text2vec")
data("movie_review", package = "text2vec")
mv_labels <- ifelse(movie_review$sentiment == 1, "Positive", "Negative")
res3 <- lsa_pipeline(as.list(movie_review$review), mv_labels,
grp_a = "Positive", grp_b = "Negative",
lab_a = "Positive", lab_b = "Negative",
colour_a = "#009E73", colour_b = "#D55E00",
min_docs = 50)

Table 3: *Mean PCA Loadings on the First Two Components (Highest Positive Loading per Group in Bold, Excluding Shared)*
Group	PC1	PC2
Negative	.329	-.255
Positive	-.081	.192
Shared	.103	.281

Figure 3: Word Vectors from 5,000 IMDB Film Reviews (Positive vs Negative) Projected to Two Dimensions. Unlike the clean topic-based separations in the Reuters and SOTU corpora, the sentiment-based distinction is much muddier: positive and negative reviews share most of their vocabulary, and evaluative words overlap heavily. Select an area of the plot to zoom in; double-click to reset.

What the Plots Capture – and What They Miss

Taken together, Figures 1–3 illustrate both the power and the limits of distributional models. Figure 1 captures the real-world distinction between financial and energy markets with striking clarity: domain-specific vocabulary clusters tightly, and a polysemous word like ‘price’ lands in different positions depending on its dominant context – precisely the kind of structure that Louwerse and colleagues have documented at larger scale. Figure 2 captures genuine historical change: pre-war addresses use the vocabulary of a young republic (‘gentlemen’, ‘militia’, ‘vessels’); modern ones use the vocabulary of a superpower (‘tonight’, ‘jobs’, ‘nuclear’), reflecting two centuries of political evolution.

Figure 3, however, reveals a clear limitation. Because both positive and negative reviews discuss the same subject – films – the topical vocabulary is largely shared, and the evaluative words that do separate them (‘excellent’ vs ‘worst’, for instance) form only a thin layer atop a large common vocabulary. A 20-dimensional LSA space simply lacks the resolution to untangle sentiment from topic. The model captures what people write about more easily than how they feel about it.

These imprecisions are not accidental; they reflect a fundamental constraint: model capacity.

From Toy Models to Titans

The LSA spaces above used just 20 latent dimensions, trained on corpora of a few dozen to a few thousand documents. The surviving vocabulary numbers in the low thousands. Under these conditions, the model does a remarkable job of sorting finance from energy or 19th-century language from modern, but it lacks the capacity to encode the subtler distributional cues that distinguish evaluative tone, sarcasm or register.

The history of distributional models is, in large part, a history of scale. As Connell and Lynott (2024) illustrate, growth in model size over the past three decades has been staggering. The LSA models of the late 1990s (Landauer & Dumais, 1997) had a few hundred latent dimensions and were trained on roughly 30,000 documents, already enough to pass synonym tests at near-human levels. Word2Vec (Mikolov et al., 2013) moved to shallow neural networks with a few million learnable parameters trained on billions of words. Then came the Transformer-based models, and the scale exploded: BERT (Devlin et al., 2019) had 340 million parameters, GPT-3 (Brown et al., 2020) reached 175 billion, and today’s largest models are estimated at well over a trillion parameters, trained on text corpora so vast they encompass a substantial fraction of everything ever written on the internet.

The core principle has not changed: predict the next word on the basis of the company it keeps. What changed is scale. A model with 20 dimensions and 10,000 words can distinguish finance from energy; a model with billions of parameters and trillions of training tokens can distinguish a Shakespearean sonnet from a legal brief, track the implications of a subordinate clause across a 3,000-word passage and generate fluent prose in dozens of languages. Generative AI was not built on a fundamentally new idea about language – it was built by scaling Firth’s old idea up by many orders of magnitude and combining it with a crucial algorithmic innovation.

The Transformer Revolution

That algorithmic innovation was the Transformer, introduced by Vaswani et al. in their 2017 paper ‘Attention Is All You Need’. Earlier language models relied on recurrent or convolutional neural networks, which processed words sequentially – reading a sentence one word at a time while trying to hold everything so far in memory. The approach worked, but imperfectly: it was slow and struggled with long-range dependencies.

The Transformer replaced all of that with multi-head self-attention: a mechanism that lets the model weigh every word in a passage simultaneously, comparing each one directly with every other. In plain terms, attention allows the model to ask, for each word, ‘which other words here matter most for understanding me?’ The idea is simple but transformative. It outperformed existing models on translation and a host of other tasks, without any recurrence or convolution, and was far faster to train in parallel.

With Transformers in hand, NLP entered a new era. Large pretrained models like BERT (Devlin et al., 2019) and the GPT series (Brown et al., 2020) set successive benchmarks for language understanding and generation. The combination of the Transformer architecture with the massive scale described above – hundreds of billions of parameters trained on essentially the whole internet – is what made generative AI possible. From Firth’s insight about co-occurrence, through LSA’s matrix decompositions and Word2Vec’s neural embeddings, to the attention-powered behemoths of today, the thread is continuous. But despite their extraordinary power, these models remain predictors of text, not infallible oracles of truth. The Transformer revolution made the storyteller more eloquent; it did not make the storyteller more honest.

Fluency Is Not Truth

Crucially, LLMs are optimised for fluency, not truth. They have no built-in fact-checking; they simply predict plausible continuations. As Radford et al. (2019) showed with GPT-2, the training objective is straightforward: learn to predict the next token in a sequence, given all preceding tokens. The loss function rewards fluent, likely text – but it never rewards the model for replying ‘I don’t know.’ Lin et al. (2022) demonstrated with their TruthfulQA benchmark that models frequently produce confident but false answers rather than admitting uncertainty, and that larger models can actually perform worse on truthfulness because they are better at reproducing convincing-sounding misinformation from their training data. The upshot: models tend to guess when unsure, and they guess with alarming confidence.

Consider what this means in practice. Ask an LLM about a niche historical event, and it may cheerfully invent plausible-sounding details – dates, names, citations – that are entirely fabricated. Ask it about a scientific finding at the edges of its training data, and it may blend two real studies into one fictional hybrid, complete with a convincing journal name. This phenomenon, known as hallucination, is not a bug that will eventually be patched away. Xu et al. (2024) demonstrated formally that if an LLM cannot reliably distinguish true from false statements in its training data, hallucinations are mathematically inevitable – a structural consequence of the architecture itself. The model’s very fluency becomes its greatest liability: it weaves a convincing narrative whether or not the underlying facts support it. In short, current LLMs are trained to be good storytellers, not guaranteed truth-tellers.

Why Prompts Matter – and Why One Is Rarely Enough

Because of this predictive nature, prompt engineering is essential. A vague or generic question will often yield a superficial, off‑target or simply wrong answer. One guide defines prompt engineering as ‘the art and science of designing and optimising prompts to guide AI models towards generating the desired responses’. That sounds rather grand, but in practice it often means something as prosaic as adding context, specifying a format, giving an example or two, and then refining iteratively until the output is actually useful.

The sensitivity of LLMs to phrasing is remarkable – and, on first encounter, humbling. Asking ‘What are some criticisms of capitalism?’ and ‘What are the main drawbacks of market economies?’ can elicit strikingly different responses, even though the questions are conceptually near‑identical. Sclar et al. (2024) showed that even tiny changes – swapping a single word, reordering a clause, adding an explicit instruction to be concise – can dramatically alter what a model produces, with performance varying by up to 76 percentage points across prompt formats for the same task. Wang et al. (2024) found that well-engineered prompts can yield ‘ideal and stable answers’, but that different formulations can have very different effects on performance. Researchers testing LLMs rigorously often try dozens of prompt variants to achieve reliable output. A single query is, in most cases, simply not enough.

There is also the matter of role, tone and constraints. Instructing the model to respond as a sceptical scientist, a sympathetic teacher or a meticulous copy‑editor changes its behaviour markedly. Asking it to respond in plain English, to avoid jargon, to stay under 150 words or to number its assumptions shapes the answer in ways a bare question never could. Each of these additions is, in Firth’s terms, part of the ‘company’ the prompt keeps – and consequently part of what determines the model’s response.

What Good Prompting Looks Like

Treat an LLM as a collaborator requiring careful, iterative guidance – not a search engine that delivers verdicts on demand. Provide ample background in your prompt, and invite the model to clarify anything before responding. Consider the first reply a draft, not a conclusion. Push back on suspect claims, request sources or alternative views, and rephrase when the model goes off track. Press further: ask it to explain its reasoning, consider counter-arguments and flag what it is uncertain about. Each move draws more of the model’s latent capability to the surface.

This iterative approach mirrors good intellectual practice more generally. A scientist does not run one experiment and publish; they replicate, vary conditions and triangulate across methods. A journalist does not accept a single source; they seek corroboration. A doctor does not diagnose on one symptom; they gather a fuller picture. Using an LLM well requires the same instinct: treat each exchange as one data point in an ongoing investigation, not as the final word.

A Powerful Tool, Not an Oracle

Using an LLM is a bit like navigating a foreign city without a map: you will stumble upon genuinely useful places, but you will also take wrong turns, end up in dead ends, and occasionally find yourself confidently heading in exactly the wrong direction. These models will often produce accurate information, because language genuinely encodes reality – words cluster around what they describe, and texts about geography, commodity markets or sensory properties track how the world actually works (Louwerse & Zwaan, 2009; Louwerse, 2011). But an LLM is trained to be fluent, not truthful: hallucinations are a mathematical inevitability of the architecture (Xu et al., 2024). The underlying mechanism is still word co-occurrence – Firth’s old principle, scaled up. Neither the brute force of massive training data (Connell & Lynott, 2024) nor the ingenious attention mechanisms of modern architectures (Vaswani et al., 2017) has yet tamed this heuristic machine into a reliable truth-teller.

Good results take deliberate effort. A well-crafted prompt, with specific context, clear constraints, iterative refinement and healthy scepticism, does not transform the model into a truth engine. What it does is steer its predictions towards the regions of language that most faithfully reflect the world. Skip that effort, and arriving at the right destination becomes a matter of luck rather than design.

Firth’s insight about words applies equally to prompts: you shall know an answer by the company the question keeps (He et al., 2024; Wang et al., 2024).

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power [Doctoral thesis, Lancaster University]. https://doi.org/10.17635/lancaster/thesis/1795

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (Vol. 33, pp. 1877–1901). https://doi.org/10.48550/arXiv.2005.14165

Brunila, M., & LaViolette, J. (2022). What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris. Proceedings of NAACL 2022. https://doi.org/10.18653/v1/2022.naacl-main.327

Connell, L., & Lynott, D. (2024). What can language models tell us about human cognition? Current Directions in Psychological Science. https://doi.org/10.1177/09637214241242746

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423

Feinerer, I., Hornik, K., & Meyer, D. (2008). Text mining infrastructure in R. Journal of Statistical Software, 25(5), 1–54. https://doi.org/10.18637/jss.v025.i05

Firth, J. R. (1957). Studies in Linguistic Analysis. Basil Blackwell.

Günther, F., Dudschig, C., & Kaup, B. (2016). LSAfun: An R package for computations based on Latent Semantic Analysis. Behavior Research Methods, 48(2), 409-421. https://doi.org/10.3758/s13428-015-0662-x

He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., & Hasan, S. (2024). Does prompt formatting have any impact on LLM performance? arXiv. https://doi.org/10.48550/arXiv.2411.10541

Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240. https://doi.org/10.1037/0033-295X.104.2.211

Lewis, D. D. (1997). Reuters-21578 text categorization test collection, distribution 1.0 [Dataset]. AT&T Bell Laboratories. http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html

Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (pp. 3214–3252). https://doi.org/10.18653/v1/2022.acl-long.229

Louwerse, M. M. (2011). Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, 3(2), 273–302. https://doi.org/10.1111/j.1756-8765.2010.01106.x

Louwerse, M., & Connell, L. (2011). A taste of words: Linguistic context and perceptual simulation predict the modality of words. Cognitive Science, 35(2), 381–398. https://doi.org/10.1111/j.1551-6709.2010.01157.x

Louwerse, M. M., & Zwaan, R. A. (2009). Language encodes geographical information. Cognitive Science, 33(1), 51–73. https://doi.org/10.1111/j.1551-6709.2008.01003.x

Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. https://doi.org/10.3758/BF03204766

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv. https://doi.org/10.48550/arXiv.1301.3781

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design. In Proceedings of ICLR 2024. https://doi.org/10.48550/arXiv.2310.11324

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). https://doi.org/10.48550/arXiv.1706.03762

Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024). Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. NPJ Digital Medicine, 7, Article 41. https://doi.org/10.1038/s41746-024-01029-4

Wu, M., Conde, J., Reviriego, P., & Brysbaert, M. (2026). How does fine-tuning improve sensorimotor representations in large language models? arXiv. https://doi.org/10.48550/arXiv.2603.03313

Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is inevitable: An innate limitation of large language models. arXiv. https://doi.org/10.48550/arXiv.2401.11817

Xu, Q., Peng, Y., Nastase, S. A., Chodorow, M., Wu, M., & Li, P. (2025). Large language models without grounding recover non-sensorimotor but not sensorimotor features of human concepts. Nature Human Behaviour, 9(9), 1871–1886. https://doi.org/10.1038/s41562-025-02203-8

R functions for checking and fixing vmrk files from BrainVision

Sun, 30 Jun 2024 00:00:00 +0000

Electroencephalography (EEG) has become a cornerstone for understanding the intricate workings of the human brain in the field of neuroscience. However, EEG software and hardware come with their own set of constraints, particularly in the management of markers, also known as triggers. This article aims to shed light on these limitations and future prospects of marker management in EEG studies, while also introducing R functions that can help deal with vmrk files from BrainVision.

Markers, serving as timestamps that indicate specific events during data collection, play a crucial role in EEG studies. These events could range from the onset of a stimulus to the participant’s response. However, one of the major constraints in current EEG systems is the limitation of markers to numbers between 1 and 255. This limitation is due to the fact that markers are typically stored as 8-bit unsigned integers in computer memory, which can only represent numbers in the range of 0 to 255. However, the number 0 is usually reserved for system use, leaving only the numbers 1 to 255 available for markers.

This numerical constraint can pose significant challenges in the interpretation of markers, especially in complex experimental designs where a multitude of events need to be marked and differentiated. It necessitates careful documentation of each marker’s purpose prior to running the study. This means that researchers must meticulously map each number to a specific event or condition in their experiment, which can be a daunting task, especially for complex studies with numerous conditions and events.

Looking towards the future, one might wonder if it will become possible to send markers with semantic information, instead of being constrained to numbers between 1 and 255. This would allow researchers to encode more detailed information in each marker, such as the type of stimulus presented or the specific condition being tested. Such a development could revolutionize the way we conduct and analyze EEG studies, offering greater flexibility in experimental design and more nuanced insights into brain activity.

Below, we’ll demonstrate some functions from this GitHub repository that help inspect and fix vmrk files. We’ll work with the vmrk file shown below.

# Read the file into a vector of lines
readLines('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/example.vmrk')

## [1] "Brain Vision Data Exchange Marker File, Version 1.0"
## [2] ""
## [3] "[Common Infos]"
## [4] "DataFile=3.eeg"
## [5] "Codepage=utf-8"
## [6] ""
## [7] "[Marker Infos]"
## [8] "; Each entry: Mk<Marker number>=<Type>,<Description>,<Position in data points>,"
## [9] "; <Size in data points>, <Channel number (0 = marker is related to all channels)>,"
## [10] "; <Date (YYYYMMDDhhmmssuuuuuu)>"
## [11] "; Fields are delimited by commas, some fields might be omited (empty)."
## [12] "; Commas in type or description text are coded as \"\\1\"."
## [13] "Mk1=New Segment,,1,1,0,20240318111955090000"
## [14] "Mk2=Stimulus,S254,20419,1,0"
## [15] "Mk3=Stimulus,S 5,22332,1,0"
## [16] "Mk4=Stimulus,S 42,23095,1,0"
## [17] "Mk5=Stimulus,S143,23100,1,0"
## [18] "Mk6=Stimulus,S 2,23106,1,0"
## [19] "Mk7=Stimulus,S102,23111,1,0"
## [20] "Mk8=Stimulus,S 6,25882,1,0"
## [21] "Mk9=Stimulus,S 5,28106,1,0"
## [22] "Mk10=Stimulus,S 50,29053,1,0"
## [23] "Mk11=Stimulus,S241,29058,1,0"
## [24] "Mk12=Stimulus,S 1,29063,1,0"
## [25] "Mk13=Stimulus,S101,29069,1,0"
## [26] "Mk14=Stimulus,S 6,31830,1,0"
## [27] "Mk15=Stimulus,S 5,34056,1,0"
## [28] "Mk16=Stimulus,S 49,35055,1,0"
## [29] "Mk17=Stimulus,S226,35060,1,0"
## [30] "Mk18=Stimulus,S 2,35066,1,0"
## [31] "Mk19=Stimulus,S103,35071,1,0"
## [32] "Mk20=Stimulus,S 6,37242,1,0"
## [33] "Mk21=Stimulus,S 5,39436,1,0"
## [34] "Mk22=Stimulus,S 43,40417,1,0"
## [35] "Mk23=Stimulus,S155,40423,1,0"
## [36] "Mk24=Stimulus,S 2,40429,1,0"
## [37] "Mk25=Stimulus,S103,40434,1,0"
## [38] "Mk26=Stimulus,S 6,42481,1,0"
## [39] "Mk27=Stimulus,S 5,44662,1,0"
## [40] "Mk28=Stimulus,S 40,45678,1,0"
## [41] "Mk29=Stimulus,S118,45683,1,0"
## [42] "Mk30=Stimulus,S 1,45688,1,0"
## [43] "Mk31=Stimulus,S103,45693,1,0"
## [44] "Mk32=Stimulus,S 6,47621,1,0"
## [45] "Mk33=Stimulus,S 5,49809,1,0"
## [46] "Mk34=Stimulus,S 50,50808,1,0"
## [47] "Mk35=Stimulus,S237,50813,1,0"
## [48] "Mk36=Stimulus,S 3,50818,1,0"
## [49] "Mk37=Stimulus,S101,50823,1,0"
## [50] "Mk38=Stimulus,S 6,53823,1,0"
## [51] "Mk39=Stimulus,S 5,56042,1,0"
## [52] "Mk40=Stimulus,S 40,57129,1,0"
## [53] "Mk41=Stimulus,S114,57134,1,0"
## [54] "Mk42=Stimulus,S 3,57140,1,0"
## [55] "Mk43=Stimulus,S103,57145,1,0"
## [56] "Mk44=Stimulus,S 6,59661,1,0"

Numbering trials

The antediluvian limitation of markers often prevents us from storing the order of trials using markers. Thus, when we need to inspect or fix vmrk files, we must mentally divide the lines into trials. For instance, in our example vmrk file, all trials begin with the marker S 5. The function below allows us to temporarily number trials by appending the number to the first marker of each trial. The parameters of the function allow us to select the start_line to skip the metadata at the top of the vmrk file, as well as to select the number of lines_per_trial and the first_number to use in the first trial.

# Create temporary vmrk file with numbered trials
source('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/number_trials.R')
number_trials('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/example.vmrk',
start_line = 15, lines_per_trial = 6, first_number = 1)

## [1] "Brain Vision Data Exchange Marker File, Version 1.0"
## [2] ""
## [3] "[Common Infos]"
## [4] "DataFile=3.eeg"
## [5] "Codepage=utf-8"
## [6] ""
## [7] "[Marker Infos]"
## [8] "; Each entry: Mk<Marker number>=<Type>,<Description>,<Position in data points>,"
## [9] "; <Size in data points>, <Channel number (0 = marker is related to all channels)>,"
## [10] "; <Date (YYYYMMDDhhmmssuuuuuu)>"
## [11] "; Fields are delimited by commas, some fields might be omited (empty)."
## [12] "; Commas in type or description text are coded as \"\\1\"."
## [13] "Mk1=New Segment,,1,1,0,20240318111955090000"
## [14] "Mk2=Stimulus,S254,20419,1,0"
## [15] "Mk3=Stimulus,S 5,22332,1,0 1"
## [16] "Mk4=Stimulus,S 42,23095,1,0"
## [17] "Mk5=Stimulus,S143,23100,1,0"
## [18] "Mk6=Stimulus,S 2,23106,1,0"
## [19] "Mk7=Stimulus,S102,23111,1,0"
## [20] "Mk8=Stimulus,S 6,25882,1,0"
## [21] "Mk9=Stimulus,S 5,28106,1,0 2"
## [22] "Mk10=Stimulus,S 50,29053,1,0"
## [23] "Mk11=Stimulus,S241,29058,1,0"
## [24] "Mk12=Stimulus,S 1,29063,1,0"
## [25] "Mk13=Stimulus,S101,29069,1,0"
## [26] "Mk14=Stimulus,S 6,31830,1,0"
## [27] "Mk15=Stimulus,S 5,34056,1,0 3"
## [28] "Mk16=Stimulus,S 49,35055,1,0"
## [29] "Mk17=Stimulus,S226,35060,1,0"
## [30] "Mk18=Stimulus,S 2,35066,1,0"
## [31] "Mk19=Stimulus,S103,35071,1,0"
## [32] "Mk20=Stimulus,S 6,37242,1,0"
## [33] "Mk21=Stimulus,S 5,39436,1,0 4"
## [34] "Mk22=Stimulus,S 43,40417,1,0"
## [35] "Mk23=Stimulus,S155,40423,1,0"
## [36] "Mk24=Stimulus,S 2,40429,1,0"
## [37] "Mk25=Stimulus,S103,40434,1,0"
## [38] "Mk26=Stimulus,S 6,42481,1,0"
## [39] "Mk27=Stimulus,S 5,44662,1,0 5"
## [40] "Mk28=Stimulus,S 40,45678,1,0"
## [41] "Mk29=Stimulus,S118,45683,1,0"
## [42] "Mk30=Stimulus,S 1,45688,1,0"
## [43] "Mk31=Stimulus,S103,45693,1,0"
## [44] "Mk32=Stimulus,S 6,47621,1,0"
## [45] "Mk33=Stimulus,S 5,49809,1,0 6"
## [46] "Mk34=Stimulus,S 50,50808,1,0"
## [47] "Mk35=Stimulus,S237,50813,1,0"
## [48] "Mk36=Stimulus,S 3,50818,1,0"
## [49] "Mk37=Stimulus,S101,50823,1,0"
## [50] "Mk38=Stimulus,S 6,53823,1,0"
## [51] "Mk39=Stimulus,S 5,56042,1,0 7"
## [52] "Mk40=Stimulus,S 40,57129,1,0"
## [53] "Mk41=Stimulus,S114,57134,1,0"
## [54] "Mk42=Stimulus,S 3,57140,1,0"
## [55] "Mk43=Stimulus,S103,57145,1,0"
## [56] "Mk44=Stimulus,S 6,59661,1,0"

Removing trial numbers

Now, we will unnumber the trials. This can be especially useful after fixing any errors in the markers. For this demo, we’ll use an example file with numbered trials that looks just like the output from number_trials() shown above.

# Remove trial numbers from temporary vmrk file
source('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/remove_trial_numbers.R')
remove_trial_numbers('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/example_numbered_trials.vmrk')

## [1] "Brain Vision Data Exchange Marker File, Version 1.0"
## [2] ""
## [3] "[Common Infos]"
## [4] "DataFile=3.eeg"
## [5] "Codepage=utf-8"
## [6] ""
## [7] "[Marker Infos]"
## [8] "; Each entry: Mk<Marker number>=<Type>,<Description>,<Position in data points>,"
## [9] "; <Size in data points>, <Channel number (0 = marker is related to all channels)>,"
## [10] "; <Date (YYYYMMDDhhmmssuuuuuu)>"
## [11] "; Fields are delimited by commas, some fields might be omited (empty)."
## [12] "; Commas in type or description text are coded as \"\\1\"."
## [13] "Mk1=New Segment,,1,1,0,20240318111955090000"
## [14] "Mk2=Stimulus,S254,20419,1,0"
## [15] "Mk3=Stimulus,S 5,22332,1,0"
## [16] "Mk4=Stimulus,S 42,23095,1,0"
## [17] "Mk5=Stimulus,S143,23100,1,0"
## [18] "Mk6=Stimulus,S 2,23106,1,0"
## [19] "Mk7=Stimulus,S102,23111,1,0"
## [20] "Mk8=Stimulus,S 6,25882,1,0"
## [21] "Mk9=Stimulus,S 5,28106,1,0"
## [22] "Mk10=Stimulus,S 50,29053,1,0"
## [23] "Mk11=Stimulus,S241,29058,1,0"
## [24] "Mk12=Stimulus,S 1,29063,1,0"
## [25] "Mk13=Stimulus,S101,29069,1,0"
## [26] "Mk14=Stimulus,S 6,31830,1,0"
## [27] "Mk15=Stimulus,S 5,34056,1,0"
## [28] "Mk16=Stimulus,S 49,35055,1,0"
## [29] "Mk17=Stimulus,S226,35060,1,0"
## [30] "Mk18=Stimulus,S 2,35066,1,0"
## [31] "Mk19=Stimulus,S103,35071,1,0"
## [32] "Mk20=Stimulus,S 6,37242,1,0"
## [33] "Mk21=Stimulus,S 5,39436,1,0"
## [34] "Mk22=Stimulus,S 43,40417,1,0"
## [35] "Mk23=Stimulus,S155,40423,1,0"
## [36] "Mk24=Stimulus,S 2,40429,1,0"
## [37] "Mk25=Stimulus,S103,40434,1,0"
## [38] "Mk26=Stimulus,S 6,42481,1,0"
## [39] "Mk27=Stimulus,S 5,44662,1,0"
## [40] "Mk28=Stimulus,S 40,45678,1,0"
## [41] "Mk29=Stimulus,S118,45683,1,0"
## [42] "Mk30=Stimulus,S 1,45688,1,0"
## [43] "Mk31=Stimulus,S103,45693,1,0"
## [44] "Mk32=Stimulus,S 6,47621,1,0"
## [45] "Mk33=Stimulus,S 5,49809,1,0"
## [46] "Mk34=Stimulus,S 50,50808,1,0"
## [47] "Mk35=Stimulus,S237,50813,1,0"
## [48] "Mk36=Stimulus,S 3,50818,1,0"
## [49] "Mk37=Stimulus,S101,50823,1,0"
## [50] "Mk38=Stimulus,S 6,53823,1,0"
## [51] "Mk39=Stimulus,S 5,56042,1,0"
## [52] "Mk40=Stimulus,S 40,57129,1,0"
## [53] "Mk41=Stimulus,S114,57134,1,0"
## [54] "Mk42=Stimulus,S 3,57140,1,0"
## [55] "Mk43=Stimulus,S103,57145,1,0"
## [56] "Mk44=Stimulus,S 6,59661,1,0"

Counting markers

source('https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/count_markers.R')
count_markers(file = 'https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/example.vmrk',
marker = 'S 2')

## Count of "S 2": 3 instances

count_markers(file = 'https://raw.githubusercontent.com/pablobernabeu/EEG-tools-and-tips/main/example.vmrk',
marker = 'S 3')

## Count of "S 3": 2 instances

rscopus_plus: An extension of the rscopus package

Wed, 21 Feb 2024 00:00:00 +0000

Sometimes it’s useful to do a bibliometric analysis. To this end, the rscopus_plus functions (Bernabeu, 2024) extend the R package rscopus (Muschelli, 2022) to administer the search quota and enable specific searches and comparisons.

scopus_search_plus runs rscopus::scopus_search as many times as necessary based on the number of results and the search quota.
scopus_search_DOIs gets DOIs from scopus_search_plus, which can then be imported into a reference manager, such as Zotero, to create a list of references.
scopus_search_additional_DOIs searches for additional DOIs.
scopus_comparison compares counts of publications on various topics throughout a certain period. The comparison terms are shown in the legend and in the lines, and they all include the reference query.
plot_scopus_comparison draws a line plot with the output from scopus_comparison.

Note. Before using any of the first four functions, the user must set their Scopus API key confidentially (see rscopus guidelines).

Example of use

As an example (also available on GitHub), we’ll visualise the prevalence of three executive functions in the literatures on second and third language throughout the past two decades.

First, we’ll use scopus_comparison() (fragment shown below).

Next, we’ll use plot_scopus_comparison() (fragment shown below).

library(rscopus)
# set_api_key('your_key_here') # (see https://cran.r-project.org/web/packages/rscopus/vignettes/api_key.html)
# I'll read in my personal Scopus key from a file.
# If you do this, make sure not to share your file.
api_key = readLines('scopus_key.txt')
set_api_key(api_key)
library(dplyr)
library(patchwork)
# Load in Scopus search functions from https://github.com/pablobernabeu/rscopus_plus
source('https://raw.githubusercontent.com/pablobernabeu/rscopus_plus/main/scopus_comparison.R')
source('https://raw.githubusercontent.com/pablobernabeu/rscopus_plus/main/plot_scopus_comparison.R')
# General parameters
search_period = 1990:2023
quota = 20
# Prevalence of three executive functions in second language studies from 2000 to 2023.
# In addition to "second language", the reference query includes the terms
# "learning" and "cognition" to make the scope of the search more relevant to
# the topic of interest.
reference_query = '"second language"'
comparison_terms = c( '"working memory"', 'inhibit*', '"implicit learning"' )
N_comparison_terms = length(comparison_terms)
L2_EF =
scopus_comparison(reference_query, comparison_terms,
search_period, quota, verbose = FALSE,
reference_query_field_tag = 'TITLE-ABS-KEY')
saveRDS(L2_EF, 'L2_EF.rds')
L2_EF = readRDS('L2_EF.rds') # it's possible to load results directly
plot_L2_EF =
plot_scopus_comparison(L2_EF,
pub_count_in_legend = FALSE,
pub_count_in_lines = TRUE) +
scale_color_manual(
values = c( "[ref.] + '\"working memory\"'" = scales::hue_pal()(N_comparison_terms)[1],
"[ref.] + 'inhibit*'" = scales::hue_pal()(N_comparison_terms)[2],
"[ref.] + '\"implicit learning\"'" = scales::hue_pal()(N_comparison_terms)[3] )
) +
guides(colour = guide_legend(override.aes = list(alpha = 1))) +
# Prepare layout for the multi-plot combination
theme(axis.text.x = element_blank(), axis.title.x = element_blank(),
legend.position = 'none', plot.margin = margin(0, 0, 15, 0))
# Prevalence of three executive functions in third language studies from 2000 to 2023.
# In addition to "third language", the reference query includes the terms "learning"
# and "cognition" to make the scope of the search more relevant to the topic of
# interest.
reference_query = '"third language"'
# Other parameters identical to those used in the query above.
L3_EF =
scopus_comparison(reference_query, comparison_terms,
search_period, quota, verbose = FALSE,
reference_query_field_tag = 'TITLE-ABS-KEY')
saveRDS(L3_EF, 'L3_EF.rds')
L3_EF = readRDS('L3_EF.rds') # it's possible to load results directly
plot_L3_EF =
plot_scopus_comparison(L3_EF,
pub_count_in_legend = FALSE,
pub_count_in_lines = TRUE) +
scale_color_manual(
values = c( "[ref.] + '\"working memory\"'" = scales::hue_pal()(N_comparison_terms)[1],
"[ref.] + 'inhibit*'" = scales::hue_pal()(N_comparison_terms)[2],
"[ref.] + '\"implicit learning\"'" = scales::hue_pal()(N_comparison_terms)[3] )
) +
guides(colour = guide_legend(override.aes = list(alpha = 1))) +
# Prepare layout for the multi-plot combination
theme(axis.text.x = element_text(margin = margin(7, 0, 0, 0, 'pt')),
axis.title.x = element_text(margin = margin(8, 0, 0, 0, 'pt')),
legend.position = 'inside', legend.position.inside = c(.82, .8))
# Combine plots
plot_L2_EF + plot_L3_EF +
plot_layout(ncol = 1, axes = 'collect') &
theme(axis.text = element_text(size = 10),
axis.title = element_text(vjust = 0.5, size = 13),
plot.title = element_markdown(hjust = 0.5, size = 12),
legend.text = element_text(size = 11,),
legend.background = element_rect(color = 'grey80', fill = 'grey99'),
legend.margin = margin(0, 5, 2, 0))

The total number of publications over the current period is shown between brackets after each query.

References

Bernabeu, P. (2024). rscopus_plus. OSF. https://doi.org/10.17605/OSF.IO/BUZQ6

Muschelli, J. (2022). Package ’rscopus’. CRAN. https://cran.r-project.org/web/packages/rscopus/rscopus.pdf

FAQs on mixed-effects models

Sat, 04 Nov 2023 00:00:00 +0000

I am dealing with nested data, and I remember from an article by Clark (1973) that nested should be analysed using special models. I’ve looked into mixed-effects models, and I’ve reached a structure with random intercepts by subjects and by items. Is this fine?

In early days, researchers would aggregate the data across these repeated measures to prevent the violation of the assumption of independence of observations, which is one of the most important assumptions in statistics. With the advent of mixed-effects models, researchers began accounting for these repeated measures using random intercepts and slopes. However, problems of convergence led many researchers to remove random slopes. This became widespread until, over the past few years, we have realised that random slopes are necessary to prevent an inflation of the Type I error due to the violation of the assumption of independence (Brauer & Curtin, 2018; Singmann & Kellen, 2019). Please see Table 17 in Brauer and Curtin (2018). Due to the present reasons, the models in the current article are anti-conservative. To redress this problem, please consider the inclusion of random slopes by participant for all between-items variables [e.g., (stimulus_condition | participant)], and random slopes by item for all between-participants variables [e.g., (extraversion | item)]. Interaction terms should also have the corresponding slopes, except when the variables in the interaction vary within different units, that is, one between participants and one between items (Brauer & Curtin, 2018). Each of the random intercepts and random slopes included in the model should be noted in the main text, for instance using footnotes in the results table (see example).

I calculated the p values by comparing minimally-different models using the anova function. Is this fine?

Luke (2017) warns that the p values calculated by model comparison—which are based on likelihood ratio tests—can be anti-conservative. Therefore, the Kenward-Roger and the Satterthwaite methods are recommended instead (both available in other packages, such as lmerTest and afex).

The lme4 package only runs on one thread (CPU) but the computer has 8. Do you have any advice on making the model run using more of the threads? It’s taking a very long time. I’ve seen these two possible solutions online from 2018 (here and here) but would like some advice if they have any or have attempted either of these solutions.

From the information I have seen in the past as well as right now, parallelising (g)lmer intentionally would be very involved. There is certainly interest in it, as your resources show (also see here). However, the current information suggests to me that it is not possible.

Interestingly, some isolated cases of unintentional parallelisation have been documented, and the developers of the lme4 package were surprised about them because they have not created this feature (see here and here).

I think the best approach may be running your model(s) in a high-performance computing (HPC) cluster. Although this would not reduce the amount of time required for each model, it would have two advantages. First, your own computers wouldn’t be busy for days, and second, you could even run several models at the same time without exhausting your own computers. I still have access to the HPC at my previous university, and it would be fine for me to send your model(s) there if that would help you. Feel free to let me know. Otherwise I can see that your university has this facility too.

We took your advice and ran the model on a supercomputer - it took roughly 2.5 days, which is what it took for the model to run on my iMac and a gaming laptop Vivienne has.

The model, however, didn’t converge. We have read that you can use allFit() to try the fit with all available optimizers. Do you have any experience using this? If you did, I wondered where this would sit in the code for the model? How and where do I add this in to check all available optimizers, please?

I have attached my code in a txt file and the data in excel for you to see, in case it is of any use.

The multi-optimizer check is indeed a way (albeit tentative) to probe into the convergence. Convergence has long been a fuzzy subject, as there are different standpoints depending on the degree of conservativeness that is sought after by the analysts.

On Page 124 in my thesis (https://osf.io/97u5c), you can find this multi-optimizer check (also see this blog post). All the code is available on OSF. More generally, I discuss the issue of convergence throughout the thesis.

I have run the model with optimizer="nloptwrap" and algorithm="NLOPT_LN_BOBYQA" and received the following warning message (once the model ran) -
In optwrap(optimizer, devfun, start, rho$lower, control = control, :
convergence code 5 from nloptwrap: NLOPT_MAXEVAL_REACHED: optimization stopped because maxeval (above) was reached.
Does this mean that the model didn’t converge? I’m only asking because I wasn’t given a statement saying it didn’t converge, as it did with Nelder_Mead. It was stated (at the end of summary table)
Optimizer (Nelder_Mead) convergence code: 4 (failure to converge in 10000 evaluations)
failure to converge in 10000 evaluations

Please try increasing the max number of iterations.

We increased the max number of iterations to 1e6 and then 1e7, and the model didn’t converge. But it has converged with maxeval=1e8.

I wanted to ask please, do you know of any issues with the max iterations being this high and effecting the interpretability of the model? Or is it completely fine?

There are no side-effects to increasing the number of iterations (see Remedy 6 in Brauer & Curtin, 2018).

References

Brauer, M., & Curtin, L. A. (2018). Mixed models in psychology: An introduction to the world of multilevel models. Psychological Methods, 23(3), 389-411. https://doi.org/10.1037/met0000132

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335-359. https://doi.org/10.1016/S0022-5371(73)80014-3

Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y

Singmann, H., & Kellen, D. (2019). An introduction to mixed models for experimental psychology. In D. H. Spieler & E. Schumacher (Eds.), New Methods in Cognitive Psychology (pp. 4–31). Hove, UK: Psychology Press. http://singmann.org/download/publications/singmann_kellen-introduction-mixed-models.pdf

FAIR standards for the creation of research materials, with examples

Thu, 02 Nov 2023 00:00:00 +0000

In the fast-paced world of scientific research, establishing minimum standards for the creation of research materials is essential. Whether it’s stimuli, custom software for data collection, or scripts for statistical analysis, the quality and transparency of these materials significantly impact the reproducibility and credibility of research. This blog post explores the importance of adhering to FAIR (Findable, Accessible, Interoperable, Reusable) principles, and offers practical examples for researchers, with a focus on the cognitive sciences.

Notwithstanding the need for speed in most scientific projects, what should be the minimum acceptable standards in the creation of research materials such as stimuli, custom software for data collection (e.g., experiment in jsPsych, OpenSesame or psychoPy), or scripts for statistical analysis?

The answer to this question is contingent upon the field of research, the purpose and the duration of the project, and many other contextual factors. So, to narrow down the scope and come at a general answer, let’s suppose we asked a researcher in the cognitive sciences (e.g., a linguist, a psychologist or a neuroscientist) who values open science. Perhaps, such a researchers would be satisfied with a method for the creation of materials that allows the creators of the materials, as well as their collaborators and any other stakeholders (e.g., any fellow scientists working in the same field), to explore, understand, reproduce, modify, and reuse the materials following their completion and thereafter. Let’s review some of the implements that can help fulfil these standards.

FAIRness

The FAIR Guiding Principles for scientific data management and stewardship exhaustively describe a protocol for making materials Findable, Accessible, Interoperable and Reusable. These terms cover the five allowances listed above, along with other important aspects.

Let’s look at some instantiations of the FAIR principles.

Reproducibility

It is convenient to allow others, and our future selves, to reproduce the materials throughout their preparation and at any time thereafter. For this purpose, R can be used to register in scripts as many as possible of the steps followed throughout the preparation of the materials. Far from being only a software for data analysis, R allows the preparation of texts, images, audios, etc. Humans err, by definition. That can be counted on. Conveniently, registering the steps followed during weeks or months of preparation allows us to offload part of the documentation efforts. It’s a way of video-recording, as it were, all the additions, subtractions, replacements, transformations and calculations performed with the raw materials, for the creation of the final materials.

Generous documentation

Under the curse of knowledge, the creators of research materials may believe that their materials are self-explanatory. Often they are more obscure than they think. To allow any other stakeholders, including their future selves, to exercise the five allowances listed above—i.e., explore, understand, reproduce, modify, and reuse the materials—, the preparation process and the end materials should be documented with enough detail. This can be done using README.txt files throughout the project. Using the .txt format/extension is recommended because other formats, such as Microsoft Word, may not be (fully) available in some computers. To exemplify the format and the content of readme files, below is an excerpt from a longitudinal study on which I’ve been working.

-- Post-training test --
In Sessions 2, 3, 4 and 6, if the test is failed in the first attempt, the training and the test are
repeated (following González Alonso et al., 2020). In such cases, the result is shown at the end
of the second attempt. The session advances if the accuracy achieved in the second attempt exceeds
80%, whereas the session stops if the accuracy is lower. In the latter situation, an 'End of session'
message is presented, flanked by two orange circles, and followed by an acknowledgement for the
participant. Once the participant has read this screen, the experimenter quits the session by
pressing 'ESC' and then 'Q'.
== Stimuli ==
The stimulus lists are described in the R functions that were used to create the stimuli, as well as
in the 'list' column in the stimulus files.
== Participant-specific parameters for lab-based sessions ==
Each participant was assigned certain parameters in advance, including the mini-language, the order
of the resting-state parts, and the stimulus lists. The code that was used to create this assignment
is available in the 'stimulus_preparation' folder.
Due to the pre-assignment of the parameters, there is a fixed set of participant IDs that can be
used in OpenSesame. These identification numbers range between 1 and 144. If an ID outside of this
range is used, the OpenSesame session does not run.
== General procedure for lab-based sessions ==
At the beginning of lab-based Sessions 2, 3, 4 and 6, the experimenter will first signal the lab is
busy using a light or a sign. Next, they will ascertain what participant and what session applies.
This is done using the session logbook that is shared among all session conductors. This session
logbook is instantly updated online using a cloud service, such as OneDrive.
Next, the experimenter starts OpenSesame by opening the program directly (not by opening the
session-specific file), and then opens the appropriate session within OpenSesame. This procedure
helps prevent the opening of a standalone Python window, the closing of which would result in the
closing of OpenSesame. Next, the experimenter opens BrainVision Recorder.
When the participant arrives in the lab, they are informed that they can use the toilet outside.
The participant is also offered some water.
Next, the size of their head is measured, and an appropriate cap is tried on the participant’s
head. Next, the cap is placed on a dummy head, and the electrodes are attached to the cap. At
that point, to prevent signal interference, the participant is kindly asked to either put their
mobile devices (phone, tablet, smartwatch) in flight mode, or to leave them outside of the booth.
Next, to protect the participant's clothes from any drops of gel, a towel is placed on their
upper back, covering shoulders and upper torso. Both ends of the towel are clipped together at
the front using two or three clothes pegs. Next, the cap is fitted on the participant's head. To
prevent the cap from being pulled back during the session, the splitter box is attached to the
towel on the participant's back, right below their head. Next, measures are taken to adjust the
position of the cap evenly, first from the nasion to the inion, and then from the tip of an ear
to the other ear.
Next, the experimenter returns to OpenSesame and runs the session in full screen by clicking on
the full green triangle at the top left. Then, a file explorer window opens, in which the
experimenter must assign a subject number consistent with the session logbook, and must select
the destination folder for the logfile. The destination folder is called 'logfiles'. Any prompts
to overwrite a logfile must normally be refused, or considered carefully, due to the risk of
losing data from previous sessions.
In the first screen, the experimenter can disable some of the tasks. This option can be used if a
session has ended abruptly, in which case the session can be resumed from a near checkpoint. In
such a case, the experimenter must first note this incident in their logbook, and rename the log
file that was produced on the first run, by appending '_first_run' to the name. This prevents
overwriting the file on the second run. Next, they must open a new session, enter the same
participant ID, and select the appropriate part from which to begin. This part must be the part
immediately following the last part that was completed in full. For instance, if a session ended
abruptly during the experiment, the beginning selected on the second run would be the experiment.
Once the session has finished completely, the first log file and the second log file must be
safely merged into a single file, keeping only the fully completed tasks.
In the first instructional screen, participants are asked to refrain from asking any questions
unless it is necessary, so that all participants can receive the same instructions.
At the beginning of the resting-state part in Session 2, and at the beginning of the Experiment
part, instructions are presented on the screen that ask participants to stay as still as possible
during the following task. The screen contains an orange-coloured square with the letters 'i.s.r',
that remind the experimenter to check the impedance and the signal, and finally to begin recording
the EEG signal. If the impedance of any electrodes is poor, the experimenter may enter the booth
to lower the impedance of the electrodes affected. Otherwise, after validating the signal and the
impedance, the experimenter can begin the recording in BrainVision, and press the letter 'C' twice
in the stimulus computer. At that point, a green circle will appear, along with instructions for
the participant.
Similarly, at the end of the eyes-closed resting-state measurement (which is five minutes long),
the experimenter must intervene when they see the screen with the orange stripes, by knocking on
the door to let the participant open their eyes.
Furthermore, at the end of the resting-state part and at the end of the Experiment part, a screen
with a crossed-out R appears to remind the experimenter to stop recording the EEG.
Notice that the above-mentioned stages, characterised by screens with orange stripes, require the
experimenter's intervention. The experimenter must allow the participant to read any text on these
screens. Next, the experimenter must press the letter 'C' twice to let the session continue. This
protocol provides the experimenter with control when necessary. The experimenter should be aware
of the use of the letter 'C' at these points, as the requirement is not signalled on the screen
to prevent participants from pressing the letter themselves.
During the experiment, it is important to monitor the EEG signal. If it ever becomes very noisy,
the experimenter must wait until the next break and the participant to stop, so that the signal
can be verified. If the noise in the signal is due to the participant's movement, they should be
asked again to please stay as still possible. If the noise is due to an increase in the
impedance of some electrodes, the impedance of those electrodes should be revised.
The experiment in Session 2 contains breaks every 40 trials, whereas the experiments in subsequent
sessions contain breaks every 50 trials. During these breaks, the number of the current trial
appears in grey on the bottom right corner of the screen.
If a session ends abruptly during the experiment, but there is not enough time to restart the
session from the experiment, then the data should be uploaded to the repository.
== Definition of items in OpenSesame (only for programming purposes, not for in-session use) ==
-- Each major part of the session is contained in a sequence item that is named in capital
letters (e.g., 'PRETRAINING', 'TRAINING', 'TEST', 'EXPERIMENT').
-- 'continue_space': allows proceeding to the following screen after pressing the space bar,
which should be done by the participant. In most cases, two presses are required, as
detailed on the screen.
-- 'continue_c': allows proceeding to the following screen after pressing the letter 'C',
which should be done by the experimenter. In most cases, two presses are required, as
detailed on the screen.
== Variables in the OpenSesame log files ==
In the log files produced by OpenSesame, each part of the session (e.g., Test, Experiment) is
identified in the variable 'session_part'. The names of the response variables are 'response',
'response_time' and 'correct'. Item-specific response variables follow the formats of
'response_[item_name]', 'response_time_[item_name]' and 'correct_[item_name]'
(see https://osdoc.cogsci.nl/3.3/manual/variables/#response-variables).
The output is verbose and requires preprocessing of the data. For instance, the last response
in each loop may appear twice in the output, due to the processing of the response. These
duplicates can--and must--be cleaned up by discarding the rows that have the same trial number
as the preceding row.
== EEG triggers ==
Triggers are sent to the EEG recorder throughout the experiment. The system for sending
triggers is set up in OpenSesame script within the inline script 'EEG_trigger_setup'.
The key to the triggers is provided below.
0: reset trigger port in BrainVision Recorder. This trigger is integrated in the
trigger-sending function.
-- Resting-state EEG part --
10: beginning of eyes-open resting-state EEG
11: end of eyes-open resting-state EEG
12: beginning of eyes-closed resting-state EEG
13: end of eyes-closed resting-state EEG
-- Experiment part --
5: fixation mark
-- ID of each target sentence (only applicable to target trials) --
110--253: triggers ranging between 110 and 253, time-locked to the onset of the
word of interest in each trial.

Comments in code scripts

It is helpful for our future selves, for our collaborators, and for any other stakeholders associated with a project—which includes any fellow researchers worldwide—to include comments in code scripts. These comments should introduce the purpose of the script at the top, and the purpose of various components of the code. Some excerpts are shown below as examples.

Open-source software

Where possible, open-source software should be used. Open-source software is free, and hence more accessible. Open-source software can be classified in various dimensions, such as the size of the user base. The more users, the greater the support, because the core developers have more resources, and the users will often help each other in public forums such as Stack Exchange. For instance, a programming language such as R boasts millions of users worldwide who count on support in public forums and in R-specific forums such as the Posit Community.

Other software are not as large. For instance, open-source software for psychological research (e.g., OpenSesame, psychoPy) are far smaller than R in terms of community. Yet, these software too can count on substantial support. For the more basic uses, most of the way has already been paved, and the existing documentation suffices. For more advanced uses, the smaller size of the community can become more obvious, as one needs to spend more time looking for solutions.

Regardless of the size of the community, all else being equal, open-source software is the right choice to ensure access to one’s work for all (potential) stakeholders in the future. The other option, proprietary software, entails dependence on the services of a private company.

Tidiness and parsimony in computer code

Code scripts should be as tidy and parsimonious as possible. For instance, to prevent overly long scripts that would impair the comprehension of the materials, it is useful to break down large projects into nested scripts, and source (i.e., run) the smaller scripts in the larger scripts.

# Compose all stimuli for Sessions 2, 3, 4 and 6
# Create participant-specific parameters
source('stimulus_preparation/participant_parameters.R')
# Frame base images
source('stimulus_preparation/base_images.R')
# Session 2
source('stimulus_preparation/Session 2/Session2_compile_all_stimuli.R')
# Session 3
source('stimulus_preparation/Session 3/Session3_compile_all_stimuli.R')
# Session 4
source('stimulus_preparation/Session 4/Session4_compile_all_stimuli.R')
# Session 6
source('stimulus_preparation/Session 6/Session6_compile_all_stimuli.R')

Tidiness and parsimony in project directories

A directory tree is useful to display all the folders in a project. The tree can be produced in the RStudio ‘Terminal’ console using the following one-line command.

find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/| - \1/"

The output will look like the following (excerpt from https://osf.io/gt5uf/wiki).

.
| - bayesian_priors
| | - plots
| - semanticpriming
| | - analysis_with_visualsimilarity
| | | - model_diagnostics
| | | | - results
| | | | - plots
| | | - results
| | | - plots
| | | - correlations
| | | | - plots
| | - frequentist_bayesian_plots
| | | - plots
| | - frequentist_analysis
| | | - model_diagnostics
| | | | - results
| | | | - plots
| | | - lexical_covariates_selection
| | | | - results
| | | | - plots
| | | - results
| | | - plots

Conclusion

Adhering to best practices—including the FAIR principles—in the creation of research materials enhances transparency, accessibility and reproducibility in scientific research. These standards facilitate researchers’ work beyond the short term, and increase the reliability of scientific work, thus contributing to the best use of resources.

Preprocessing the Norwegian Web as Corpus (NoWaC) in R

Thu, 28 Sep 2023 00:00:00 +0000

The present script can be used to preprocess data from a frequency list of the Norwegian as Web Corpus (NoWaC; Guevara, 2010).

Before using the script, the frequency list should be downloaded from this URL. The list is described as ‘frequency list sorted primary alphabetic and secondary by frequency within each character’, and this is the direct URL. The download requires signing in to an institutional network. Last, the downloaded file should be unzipped.

The script is shown below.

References

Guevara, E. R. (2010). NoWaC: A large web-based corpus for Norwegian. In Proceedings of the NAACL HLT 2010 Sixth Web as Corpus Workshop (pp. 1-7). https://aclanthology.org/W10-1501

Rodina, Y., & Westergaard, M. (2015). Grammatical gender in Norwegian: Language acquisition and language change. Journal of Germanic Linguistics, 27(2), 145–187. https://doi.org/10.1017/S1470542714000245

Rodina, Y., & Westergaard, M. (2021). Grammatical gender and declension class in language change: A study of the loss of feminine gender in Norwegian. Journal of Germanic Linguistics, 33(3), 235–263. https://doi.org/10.1017/S1470542719000217

ggplotting power curves from the simr package

Tue, 27 Jun 2023 00:00:00 +0000

The R package ‘simr’ has greatly facilitated power analysis for mixed-effects models using Monte Carlo simulation (which involves running hundreds or thousands of tests under slight variations of the data). The powerCurve function is used to estimate the statistical power for various sample sizes in one go. Since the tests are run serially, they can take a VERY long time; approximately, the time it takes to run the model supplied once (say, a few hours) times the number of simulations (nsim, which should be higher than 200), and times the number of different sample sizes examined. While there isn’t a built-in parallel method, the power curves for different sample sizes can be run separately, and the results can be progressively combined as each component finishes running (see tutorial). The power curves produced by simr are so good they deserve ‘ggplot2’ rendering. So, here’s a function for it.

A usage example

library(lme4)
library(simr)
library(ggplot2)
# Toy model with data from 'simr' package
fit = lmer(y ~ x + (x | g), data = simdata)
# Extend sample size of `g`
fit_extended_g = extend(fit, along = 'g', n = 12)
fit_powercurve =
powerCurve(fit_extended_g, fixed('x'),
along = 'g', breaks = c(4, 6, 8, 10, 12),
nsim = 50, seed = 123, progress = FALSE)
# Read in custom function to ggplot results from simr::powerCurve
source('https://raw.githubusercontent.com/pablobernabeu/powercurvePlot/main/powercurvePlot.R')
powercurvePlot(fit_powercurve, number_x_axis_levels = 6) +
# Change some defaults
xlab("Number of levels in 'g'") +
theme(plot.title = element_blank(),
axis.title.x = element_text(size = 18),
axis.title.y = element_text(size = 18),
axis.text = element_text(size = 17))

How to discretise the colour variable in sjPlot::plot_model into equally-sized intervals

Sat, 24 Jun 2023 16:54:46 +0200

library(lme4)
#> Loading required package: Matrix
library(sjPlot)
#> Learn more about sjPlot with 'browseVignettes("sjPlot")'.
library(ggplot2)
theme_set(theme_sjplot())
# Create data partially based on code by Ben Bolker
# from https://stackoverflow.com/a/38296264/7050882
set.seed(101)
spin = runif(800, 1, 24)
trait = rep(1:40, each = 20)
ID = rep(1:80, each = 10)
testdata <- data.frame(spin, trait, ID)
testdata$fatigue <-
testdata$spin * testdata$trait /
rnorm(800, mean = 6, sd = 2)
# Model
fit = lmer(fatigue ~ spin * trait + (1|ID),
data = testdata, REML = TRUE)
#> boundary (singular) fit: see help('isSingular')
plot_model(fit, type = 'pred', terms = c('spin', 'trait'))
#> Warning: Ignoring unknown parameters: linewidth

^{Created on 2023-06-24 with reprex v2.0.2}

However, I needed an extra feature, as sjPlot by default breaks down the colour (fill) variable into few levels that do not include the minimum or the maximum values in my variable. What I would like to do is to stratify the colour variable into equally-sized levels that include the minimum and the maximum values.

Furthermore, in the legend, I would also like to display the number of levels of a grouping variable (ID) that are contained in each level of the colour variable.

Below is a solution using custom functions called deciles_interaction_plot and sextiles_interaction_plot.

library(lme4)
#> Loading required package: Matrix
library(sjPlot)
library(ggplot2)
theme_set(theme_sjplot())
# Create data partially based on code by Ben Bolker
# from https://stackoverflow.com/a/38296264/7050882
set.seed(101)
spin = runif(800, 1, 24)
trait = rep(1:40, each = 20)
ID = rep(1:80, each = 10)
testdata <- data.frame(spin, trait, ID)
testdata$fatigue <-
testdata$spin * testdata$trait /
rnorm(800, mean = 6, sd = 2)
# Model
fit = lmer(fatigue ~ spin * trait + (1|ID),
data = testdata, REML = TRUE)
#> boundary (singular) fit: see help('isSingular')
# plot_model(fit, type = 'pred', terms = c('spin', 'trait'))
# Binning the colour variable into ten levels (deciles)
# Read in function from GitHub
source('https://raw.githubusercontent.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/main/R_functions/deciles_interaction_plot.R')
deciles_interaction_plot(
model = fit,
x = 'spin',
fill = 'trait',
fill_nesting_factor = 'ID'
)
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
#> Loading required package: RColorBrewer
#> Loading required package: ggtext
#> Loading required package: Cairo
#> Warning in RColorBrewer::brewer.pal(n, pal): n too large, allowed maximum for palette Set1 is 9
#> Returning the palette you asked for with that many colors
#> Warning: Ignoring unknown parameters: linewidth
#> Scale for 'y' is already present. Adding another scale for 'y', which will
#> replace the existing scale.
#> Scale for 'colour' is already present. Adding another scale for 'colour',
#> which will replace the existing scale.


# If you wanted or needed to make six levels (sextiles) instead
# of ten, you could use the function sextiles_interaction_plot.
# Read in function from GitHub
source('https://raw.githubusercontent.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/main/R_functions/sextiles_interaction_plot.R')
sextiles_interaction_plot(
model = fit,
x = 'spin',
fill = 'trait',
fill_nesting_factor = 'ID'
)
#> Warning: Ignoring unknown parameters: linewidth
#> Scale for 'y' is already present. Adding another scale for 'y', which will
#> replace the existing scale.
#> Scale for 'colour' is already present. Adding another scale for 'colour',
#> which will replace the existing scale.

^{Created on 2023-06-24 with reprex v2.0.2}

How to map more informative values onto fill argument of sjPlot::plot_model

Sat, 24 Jun 2023 16:51:11 +0200

Whereas the direction of main effects can be interpreted from the sign of the estimate, the interpretation of interaction effects often requires plots. This task is facilitated by the R package sjPlot. For instance, using the plot_model function, I plotted the interaction between a continuous variable and a categorical variable. The categorical variable was passed to the fill argument of plot_model.

library(lme4)
#> Loading required package: Matrix
library(sjPlot)
#> Install package "strengejacke" from GitHub (`devtools::install_github("strengejacke/strengejacke")`) to load all sj-packages at once!
library(ggplot2)
theme_set(theme_sjplot())
cake$recipe_recoded = ifelse(cake$recipe == 'A', -0.5,
ifelse(cake$recipe == 'B', 0,
ifelse(cake$recipe == 'C', 0.5,
NA)))
fit = lmer(angle ~ recipe_recoded * temp +
(1|recipe_recoded:replicate),
cake, REML= FALSE)
plot_model(fit, type = 'pred', terms = c('temp', 'recipe_recoded'))
#> Warning: Ignoring unknown parameters: linewidth

^{Created on 2023-06-24 with reprex v2.0.2}

However, I needed an extra feature, as the categorical variable was not quite informative because it was a sum-coded transformation. Thus, I wanted the legend of the plot to show the values of the original variable (i.e., A, B and C), instead of those of the sum-coded variable that had been used in the model (i.e., -0.5, 0 and 0.5).

Below is a solution using a custom function called alias_interaction_plot.

library(lme4)
#> Loading required package: Matrix
library(sjPlot)
library(ggplot2)
theme_set(theme_sjplot())
cake$recipe_recoded = ifelse(cake$recipe == 'A', -0.5,
ifelse(cake$recipe == 'B', 0,
ifelse(cake$recipe == 'C', 0.5,
NA)))
fit = lmer(angle ~ recipe_recoded * temp +
(1|recipe_recoded:replicate),
cake, REML= FALSE)
# plot_model(fit, type = 'pred', terms = c('temp', 'recipe_recoded'))
# Displaying the original variable instead
# Read in function from GitHub
source('https://raw.githubusercontent.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/main/R_functions/alias_interaction_plot.R')
alias_interaction_plot(
model = fit,
dataset = cake,
x = 'temp',
fill = 'recipe_recoded',
fill_alias = 'recipe',
fill_title = 'recipe'
)
#> Loading required package: rlang
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
#> Loading required package: RColorBrewer
#> Loading required package: ggtext
#> Loading required package: Cairo
#> Warning: Ignoring unknown parameters: linewidth
#> Scale for 'y' is already present. Adding another scale for 'y', which will
#> replace the existing scale.
#> Scale for 'colour' is already present. Adding another scale for 'colour',
#> which will replace the existing scale.

^{Created on 2023-06-24 with reprex v2.0.2}

How to visually assess the convergence of a mixed-effects model by plotting various optimizers

Sat, 24 Jun 2023 16:42:34 +0200

To assess whether convergence warnings render the results invalid, or on the contrary, the results can be deemed valid in spite of the warnings, Bates et al. (2023) suggest refitting models affected by convergence warnings with a variety of optimizers. The authors argue that, if the different optimizers produce practically-equivalent results, the results are valid. The allFit function from the ‘lme4’ package allows the refitting of models using a number of optimizers. To use the seven optimizers listed above, two extra packages must be installed: ‘dfoptim’ and ‘optimx’ (see lme4 manual). The output from allFit contains several statistics on the fixed and the random effects fitted by each optimizer.

library(lme4)
#> Loading required package: Matrix
library(dfoptim)
library(optimx)
# Create data using code by Ben Bolker from
# https://stackoverflow.com/a/38296264/7050882
set.seed(101)
spin = runif(600, 1, 24)
reg = runif(600, 1, 15)
ID = rep(c("1","2","3","4","5", "6", "7", "8", "9", "10"))
day = rep(1:30, each = 10)
testdata <- data.frame(spin, reg, ID, day)
testdata$fatigue <- testdata$spin * testdata$reg/10 * rnorm(30, mean=3, sd=2)
# Model
fit = lmer(fatigue ~ spin * reg + (1|ID),
data = testdata, REML = TRUE)
# Refit model using all available algorithms
multi_fit = allFit(fit)
#> bobyqa : [OK]
#> Nelder_Mead : [OK]
#> nlminbwrap : [OK]
#> nmkbw : [OK]
#> optimx.L-BFGS-B : [OK]
#> nloptwrap.NLOPT_LN_NELDERMEAD : [OK]
#> nloptwrap.NLOPT_LN_BOBYQA : [OK]
# Show results
summary(multi_fit)$fixef
#> (Intercept) spin reg spin:reg
#> bobyqa -2.975678 0.5926561 0.1437204 0.1834016
#> Nelder_Mead -2.975675 0.5926559 0.1437202 0.1834016
#> nlminbwrap -2.975677 0.5926560 0.1437203 0.1834016
#> nmkbw -2.975678 0.5926561 0.1437204 0.1834016
#> optimx.L-BFGS-B -2.975680 0.5926562 0.1437205 0.1834016
#> nloptwrap.NLOPT_LN_NELDERMEAD -2.975666 0.5926552 0.1437196 0.1834017
#> nloptwrap.NLOPT_LN_BOBYQA -2.975678 0.5926561 0.1437204 0.1834016
# Read in function from GitHub
source('https://raw.githubusercontent.com/pablobernabeu/plot.fixef.allFit/main/plot.fixef.allFit.R')
plot.fixef.allFit(multi_fit,
select_predictors = c('spin', 'reg', 'spin:reg'),
# Increase padding at top and bottom of Y axis
multiply_y_axis_limits = 1.3,
y_title = 'Fixed effect (*b*)')
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
#> Loading required package: reshape2
#> Loading required package: stringr
#> Loading required package: scales
#> Loading required package: ggplot2
#> Loading required package: ggtext
#> Loading required package: patchwork


# Alternative using plot-specific Y axes and other modified settings
plot.fixef.allFit(multi_fit,
select_predictors = c('spin', 'spin:reg'),
# Use plot-specific Y axis limits
shared_y_axis_limits = FALSE,
decimal_places = 7,
# Move up Y axis title
y_title_hjust = 4.5,
y_title = 'Fixed effect (*b*)')

^{Created on 2023-06-26 with reprex v2.0.2}

Table joins with conditional "fuzzy" string matching in R

Sat, 24 Jun 2023 00:00:00 +0000

Here’s an example of fuzzy-matching strings in R that I shared on StackOverflow. In stringdist_join, the max_dist argument is used to constrain the degree of fuzziness.

library(fuzzyjoin)
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(knitr)
small_tab = data.frame(Food.Name = c('Corn', 'Squash', 'Peppers'),
Food.Code = c(NA, NA, NA))
large_tab = data.frame(Food.Name = c('Sweet Corn', 'Red Corn', 'Baby Corns',
'Squash', 'Long Squash', 'Red Pepper',
'Green Pepper', 'Red Peppers'),
Food.Code = c(532, 532, 944, 111, 123, 654, 655, 654))
joined_tab = stringdist_join(small_tab, large_tab, by = 'Food.Name',
ignore_case = TRUE, method = 'cosine',
max_dist = 0.5, distance_col = 'dist') %>%
# Tidy columns
select(Food.Name = Food.Name.x, -Food.Name.y,
Food.Code = Food.Code.y, -dist) %>%
# Only keep most frequent food code per food name
group_by(Food.Name) %>% count(Food.Name, Food.Code) %>%
slice(which.max(n)) %>% select(-n) %>%
# Order food names as in the small table
arrange(factor(Food.Name, levels = small_tab$Food.Name))
# Show table with columns renamed
joined_tab %>%
rename('Food Name' = Food.Name,
'Food Code' = Food.Code) %>%
kable()

Food Name	Food Code
Corn	532
Squash	111
Peppers	654

^{Created on 2023-05-31 with reprex v2.0.2}

A new function to plot convergence diagnostics from lme4::allFit()

Thu, 22 Jun 2023 00:00:00 +0000

Linear mixed-effects models (LMM) offer a consistent way of performing regression and analysis of variance tests which allows accounting for non-independence in the data. Over the past decades, LMMs have subsumed most of the General Linear Model, with a steady increase in popularity (Meteyard & Davies, 2020). Since their conception, LMMs have presented the challenge of model convergence. In essence, the issue of convergence boils down to the widespread tension between parsimony and completeness in data analysis. That is, on the one hand, a good model must allow an accurate, parsimonious analysis of each predictor, and thus, it must not be overfitted with too many parameters. Yet, on the other hand, the model must be complete enough to account for a sufficient amount of variation in the data. In LMMs, any predictors that entail non-independent observations (also known as repeated measures) will normally bring both fixed and random effects into the model. Where a few of these predictors coexist, models often struggle to find enough information in the data to account for every predictor—and especially, for every random effect. This difficulty translates into convergence warnings (Brauer & Curtin, 2018; Singmann & Kellen, 2019). In this article, I review the issue of convergence before presenting a new plotting function in R that facilitates the diagnosis of convergence by visualising the fixed effects fitted by different optimization algorithms (also dubbed optimizers).

Completeness versus parsimony

Both fixed and random effects comprise intercepts and slopes. The pressure exerted by each of those types of effects on the model is determined by the number of data points involved by each. First, slopes are more demanding than intercepts, as they involve a (far) larger number of data points. Second, random effects are more demanding than fixed effects, as random effects entail the number of estimates required for fixed effects times the number of levels in the grouping factor. As a result, on the most lenient end of the scale lies the fixed intercept, and on the heaviest end lie the random slopes. Convergence warnings in LMMs are often due to the random slopes alone.

Sounds easy, then! Not inviting the random slopes to the party should solve the problem. Indeed, since random slopes involve the highest number of estimates by far, removing them does often remove convergence warnings. This, however, leads to a different problem. Surrendering the information provided by random slopes can result in the violation of the assumption of independence of observations. For years, the removal of random slopes due to convergence warnings was standard practice. Currently, in contrast, proposals increasingly consider other options, such as removing random effects if they do not significantly improve the fit of the model (Matuschek et al., 2017), and keeping the random slopes in the model in spite of the convergence warnings to safeguard the assumption of independence (see Table 17 in Brauer & Curtin, 2018; Singmann & Kellen, 2019).

The multiple-optimizers sanity check from `lme4::allFit()`

Framed within the drive to maintain random slopes wherever possible, the developers of the ‘lme4’ package propose a sanity check that uses a part of the ‘lme4’ engine called ‘optimizer’. Every model has a default optimizer, unless a specific one is chosen through control = lmerControl(optimizer = '...') (in lmer models) or control = glmerControl(optimizer = '...') (in glmer models). The seven widely-available optimizers are:

bobyqa
Nelder_Mead
nlminbwrap
nmkbw
optimx.L-BFGS-B
nloptwrap.NLOPT_LN_NELDERMEAD
nloptwrap.NLOPT_LN_BOBYQA

To assess whether convergence warnings render the results invalid, or on the contrary, the results can be deemed valid in spite of the warnings, Bates et al. (2022) suggest refitting models affected by convergence warnings with a variety of optimizers. The authors argue that if the different optimizers produce practically-equivalent results, the results are valid. The allFit function from the ‘lme4’ package allows the refitting of models using a number of optimizers. To use the seven optimizers listed above, two extra packages must be installed: ‘dfoptim’ and ‘optimx’ (see lme4 manual). The output from allFit() contains several statistics on the fixed and the random effects fitted by each optimizer (see example).

Plotting the fixed effects from allFit()

Several R users have ventured into plotting the allFit() output but there is not a function in ‘lme4’ yet at the time of writing. Bernabeu (2022) developed a function that takes the output from allFit(), tidies it, selects the fixed effects and plots them using ‘ggplot2’. The function is shown below, and can be copied through the Copy Code button at the top right corner. It can be renamed by changing plot.fixef.allFit to another valid name.

# Plot the results from the fixed effects produced by different optimizers. This function
# takes the output from lme4::allFit(), tidies it, selects fixed effects and plots them.
plot.fixef.allFit = function(allFit_output,
# Set the same Y axis limits in every plot
shared_y_axis_limits = TRUE,
# Multiply Y axis limits by a factor (only
# available if shared_y_axis_limits = TRUE)
multiply_y_axis_limits = 1,
# Number of decimal places
decimal_places = NULL,
# Select predictors
select_predictors = NULL,
# Number of rows
nrow = NULL,
# Y axis title
y_title = 'Fixed effect',
# Alignment of the Y axis title
y_title_hjust = NULL,
# Add number to the names of optimizers
number_optimizers = TRUE,
# Replace colon in interactions with x
interaction_symbol_x = TRUE) {
require(lme4)
require(dfoptim)
require(optimx)
require(dplyr)
require(reshape2)
require(stringr)
require(scales)
require(ggplot2)
require(ggtext)
require(patchwork)
library(Cairo)
# Tidy allFit output
# Extract fixed effects from the allFit() output
allFit_fixef = summary(allFit_output)$fixef %>% # Select fixed effects in the allFit results
reshape2::melt() %>% # Structure the output as a data frame
rename('Optimizer' = 'Var1', 'fixed_effect' = 'Var2') # set informative names
# If number_optimizers = TRUE, assign number to each optimizer and place it before its name
if(number_optimizers == TRUE) {
allFit_fixef$Optimizer = paste0(as.numeric(allFit_fixef$Optimizer), '. ', allFit_fixef$Optimizer)
}
# If select_predictors was supplied, select them along with the intercept (the latter required)
if(!is.null(select_predictors)) {
allFit_fixef = allFit_fixef %>% dplyr::filter(fixed_effect %in% c('(Intercept)', select_predictors))
}
# Order variables
allFit_fixef = allFit_fixef[, c('Optimizer', 'fixed_effect', 'value')]
# PLOT. The overall plot is formed of a first row containing the intercept and the legend
# (intercept_plot), and a second row containing the predictors (predictors_plot),
# which may in turn occupy several rows.
# If multiply_y_axis_limits was supplied but shared_y_axis_limits = FALSE,
# warn that shared_y_axis_limits is required.
if(!multiply_y_axis_limits == 1 & shared_y_axis_limits == FALSE) {
message('The argument `multiply_y_axis_limits` has not been used because \n it requires `shared_y_axis_limits` set to TRUE.')
}
# If extreme values were entered in y_title_hjust, show warning
if(!is.null(y_title_hjust)) {
if(y_title_hjust < 0.5 | y_title_hjust > 6) {
message('NOTE: For y_title_hjust, a working range of values is between 0.6 and 6.')
}
}
# If decimal_places was supplied, convert number to the format used in 'scales' package
if(!is.null(decimal_places)) {
decimal_places =
ifelse(decimal_places == 1, 0.1,
ifelse(decimal_places == 2, 0.01,
ifelse(decimal_places == 3, 0.001,
ifelse(decimal_places == 4, 0.0001,
ifelse(decimal_places == 5, 0.00001,
ifelse(decimal_places == 6, 0.000001,
ifelse(decimal_places == 7, 0.0000001,
ifelse(decimal_places == 8, 0.00000001,
ifelse(decimal_places == 9, 0.000000001,
ifelse(decimal_places == 10, 0.0000000001,
ifelse(decimal_places == 11, 0.00000000001,
ifelse(decimal_places == 12, 0.000000000001,
ifelse(decimal_places == 13, 0.0000000000001,
ifelse(decimal_places == 14, 0.00000000000001,
ifelse(decimal_places >= 15, 0.000000000000001,
0.001
)))))))))))))))
}
# First row: intercept_plot
# Select intercept data only
intercept = allFit_fixef %>% dplyr::filter(fixed_effect == '(Intercept)')
intercept_plot = intercept %>%
ggplot(., aes(fixed_effect, value, colour = Optimizer)) +
geom_point(position = position_dodge(1)) +
facet_wrap(~fixed_effect, scale = 'free') +
guides(colour = guide_legend(title.position = 'left')) +
theme_bw() +
theme(axis.title = element_blank(), axis.ticks.x = element_blank(),
axis.text.x = element_blank(),
strip.text = element_text(size = 10, margin = margin(t = 4, b = 6)),
strip.background = element_rect(fill = 'grey96'),
legend.margin = margin(0.3, 0, 0.8, 1, 'cm'),
legend.title = element_text(size = unit(15, 'pt'), angle = 90, hjust = 0.5))
# Second row: predictors_plot
# Select all predictors except intercept
predictors = allFit_fixef %>% dplyr::filter(!fixed_effect == '(Intercept)')
# If interaction_symbol_x = TRUE (default), replace colon with times symbol x between spaces
if(interaction_symbol_x == TRUE) {
# Replace colon in interactions with \u00D7, i.e., x; then set factor class
predictors$fixed_effect = predictors$fixed_effect %>%
str_replace_all(':', ' \u00D7 ') %>% factor()
}
# Order predictors as in the original output from lme4::allFit()
predictors$fixed_effect = factor(predictors$fixed_effect,
levels = unique(predictors$fixed_effect))
# Set number of rows for the predictors excluding the intercept.
# First, if nrow argument supplied, use it
if(!is.null(nrow)) {
predictors_plot_nrow = nrow - 1 # Subtract 1 as intercept row not considered
# Else, if nrow argument not supplied, calculate sensible number of rows: i.e., divide number of
# predictors (exc. intercept) by 2 and round up the result. For instance, 7 predictors --> 3 rows
} else predictors_plot_nrow = (length(unique(predictors$fixed_effect)) / 2) %>% ceiling()
predictors_plot = ggplot(predictors, aes(fixed_effect, value, colour = Optimizer)) +
geom_point(position = position_dodge(1)) +
facet_wrap(~fixed_effect, scale = 'free',
# Note that predictors_plot_nrow was defined a few lines above
nrow = predictors_plot_nrow,
# Wrap names of predictors with more than 54 characters into new lines
labeller = labeller(fixed_effect = label_wrap_gen(width = 55))) +
labs(y = y_title) +
theme_bw() +
theme(axis.title.x = element_blank(), axis.text.x = element_blank(),
axis.ticks.x = element_blank(),
axis.title.y = ggtext::element_markdown(size = 14, margin = margin(0, 15, 0, 0)),
strip.text = element_text(size = 10, margin = margin(t = 4, b = 6)),
strip.background = element_rect(fill = 'grey96'), legend.position = 'none')
# Below, the function scale_y_continuous is applied conditionally to avoid overriding settings. First,
# if shared_y_axis_limits = TRUE and decimal_places was supplied, set the same Y axis limits in
# every plot and set decimal_places. By default, also expand limits by a seventh of its original
# limit, and allow further multiplication of limits through multiply_y_axis_limits.
if(shared_y_axis_limits == TRUE & !is.null(decimal_places)) {
intercept_plot = intercept_plot +
scale_y_continuous(limits = c(min(allFit_fixef$value) - allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits,
max(allFit_fixef$value) + allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits),
# Set number of decimal places
labels = scales::label_number(accuracy = decimal_places))
predictors_plot = predictors_plot +
scale_y_continuous(limits = c(min(allFit_fixef$value) - allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits,
max(allFit_fixef$value) + allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits),
# Set number of decimal places
labels = scales::label_number(accuracy = decimal_places))
# Else, if shared_y_axis_limits = TRUE but decimal_places were not supplied, do as above but without
# setting decimal_places.
} else if(shared_y_axis_limits == TRUE & is.null(decimal_places)) {
intercept_plot = intercept_plot +
scale_y_continuous(limits = c(min(allFit_fixef$value) - allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits,
max(allFit_fixef$value) + allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits),
# Set number of decimal places
labels = scales::label_number(accuracy = decimal_places))
predictors_plot = predictors_plot +
scale_y_continuous(limits = c(min(allFit_fixef$value) - allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits,
max(allFit_fixef$value) + allFit_fixef$value %>% abs %>%
max / 7 * multiply_y_axis_limits),
# Set number of decimal places
labels = scales::label_number(accuracy = decimal_places))
# Else, if shared_y_axis_limits = FALSE and decimal_places was supplied, set decimal_places.
} else if(shared_y_axis_limits == FALSE & !is.null(decimal_places)) {
# Set number of decimal places in both plots
intercept_plot = intercept_plot +
scale_y_continuous(labels = scales::label_number(accuracy = decimal_places))
predictors_plot = predictors_plot +
scale_y_continuous(labels = scales::label_number(accuracy = decimal_places))
}
# Plot matrix: based on number of predictors_plot_nrow, adjust height of Y axis title
# (unless supplied), and assign space to intercept_plot and predictors_plot
if(predictors_plot_nrow == 1) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 3.6))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 11, l = 0) # predictors row(s)
)
} else if(predictors_plot_nrow == 2) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 1.4))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 16, l = 0) # predictors row(s)
)
} else if(predictors_plot_nrow == 3) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 0.92))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 21, l = 0) # predictors row(s)
)
} else if(predictors_plot_nrow == 4) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 0.8))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 26, l = 0) # predictors row(s)
)
} else if(predictors_plot_nrow == 5) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 0.73))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 31, l = 0) # predictors row(s)
)
} else if(predictors_plot_nrow > 5) {
# If y_title_hjust supplied, use it
if(!is.null(y_title_hjust)) {
predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = y_title_hjust))
# Otherwise, set a sensible height
} else predictors_plot = predictors_plot +
theme(axis.title.y = ggtext::element_markdown(hjust = 0.65))
layout = c(
patchwork::area(t = 1.5, r = 8.9, b = 6.8, l = 0), # intercept row
patchwork::area(t = 7.3, r = 9, b = 36, l = 0) # predictors row(s)
)
# Also, advise user to consider distributing predictors into several plots
message(' Many rows! Consider distributing predictors into several plots \n using argument `select_predictors`')
}
# Add margin
predictors_plot = predictors_plot + theme(plot.margin = margin(15, 15, 15, 15))
# Return matrix of plots
wrap_plots(intercept_plot, predictors_plot, design = layout,
# The 2 below corresponds to intercept_plot and predictors_plot
nrow = 2)
}

Optional arguments

Below are the optional arguments allowed by the function, with their default values.

# Set the same Y axis limits in every plot
shared_y_axis_limits = TRUE,
# Multiply Y axis limits by a factor (only
# available if shared_y_axis_limits = TRUE)
multiply_y_axis_limits = 1,
# Number of decimal places
decimal_places = NULL,
# Select predictors
select_predictors = NULL,
# Number of rows
nrow = NULL,
# Y axis title
y_title = 'Fixed effect',
# Alignment of the Y axis title
y_title_hjust = NULL,
# Add number to the names of optimizers
number_optimizers = TRUE,
# Replace colon in interactions with x
interaction_symbol_x = TRUE

The argument shared_y_axis_limits deserves a comment. It allows using the same Y axis limits (i.e., range) in all plots or, alternatively, using plot-specific limits. The parameter is TRUE by default to prevent overinterpretations of small differences across optimizers (see the first figure below). In contrast, when shared_y_axis_limits = FALSE, plot-specific limits are used, which results in a narrower range of values in the Y axis (see the second figure below). Since data points will span the entire Y axis in that case, any difference across optimizers—regardless of its relative importance—might be perceived as large, unless the specific range of values in each plot is noticed.

Use case

Let’s test the function with a minimal model.

# Create data using code by Ben Bolker from
# https://stackoverflow.com/a/38296264/7050882
set.seed(101)
spin = runif(600, 1, 24)
reg = runif(600, 1, 15)
ID = rep(c("1","2","3","4","5", "6", "7", "8", "9", "10"))
day = rep(1:30, each = 10)
testdata <- data.frame(spin, reg, ID, day)
testdata$fatigue <- testdata$spin * testdata$reg/10 * rnorm(30, mean=3, sd=2)
# Model
library(lme4)
fit = lmer(fatigue ~ spin * reg + (1|ID),
data = testdata, REML = TRUE)
# Refit model using all available algorithms
multi_fit = allFit(fit)

## bobyqa : [OK]
## Nelder_Mead : [OK]
## nlminbwrap : [OK]
## nmkbw : [OK]
## optimx.L-BFGS-B : [OK]
## nloptwrap.NLOPT_LN_NELDERMEAD : [OK]
## nloptwrap.NLOPT_LN_BOBYQA : [OK]

summary(multi_fit)$fixef

## (Intercept) spin reg spin:reg
## bobyqa -2.975678 0.5926561 0.1437204 0.1834016
## Nelder_Mead -2.975675 0.5926559 0.1437202 0.1834016
## nlminbwrap -2.975677 0.5926560 0.1437203 0.1834016
## nmkbw -2.975678 0.5926561 0.1437204 0.1834016
## optimx.L-BFGS-B -2.975680 0.5926562 0.1437205 0.1834016
## nloptwrap.NLOPT_LN_NELDERMEAD -2.975666 0.5926552 0.1437196 0.1834017
## nloptwrap.NLOPT_LN_BOBYQA -2.975678 0.5926561 0.1437204 0.1834016

The effects to be visualised are selected below using the argument select_predictors. Notice that the intercept is plotted by default on the first row, along with the legend that lists all the optimizers used.

# Read in function from GitHub
source('https://raw.githubusercontent.com/pablobernabeu/plot.fixef.allFit/main/plot.fixef.allFit.R')
plot.fixef.allFit(multi_fit,
select_predictors = c('spin', 'reg', 'spin:reg'),
# Increase padding at top and bottom of Y axis
multiply_y_axis_limits = 1.3,
y_title = 'Fixed effect (*b*)',
# Align y title
y_title_hjust = .9)

## Warning: package 'dplyr' was built under R version 4.5.2

## Warning: package 'ggplot2' was built under R version 4.5.2

The plot produced by plot.fixef.allFit() by default replaces the colons in interaction effects (e.g., spin:reg) with ’ × ’ to facilitate the visibility (this can be overriden by setting interaction_symbol_x = FALSE). Yet, it is important to note that any interactions passed to select_predictors must have the colon, as that is the symbol present in the lme4::allFit() output.

The output of plot.fixef.allFit() is a ggplot2 object that can be stored for further use, as in the example below, in which new parameters are used.

library(ggplot2)
plot_fit_convergence =
plot.fixef.allFit(multi_fit,
select_predictors = c('spin', 'spin:reg'),
# Use plot-specific Y axis limits
shared_y_axis_limits = FALSE,
decimal_places = 7,
# Move up Y axis title
y_title_hjust = -20,
y_title = 'Fixed effect (*b*)')
# Print
plot_fit_convergence

# Plot can be saved to disk as pdf, png, etc. through `ggplot2::ggsave()`
# ggsave('plot_fit_convergence.pdf', plot_fit_convergence,
# device = cairo_pdf, width = 9, height = 9, dpi = 900)

References

Bates, D., Maechler, M., Bolker, B., Walker, S., Christensen, R. H. B., Singmann, H., Dai, B., Scheipl, F., Grothendieck, G., Green, P., Fox, J., Bauer, A., & Krivitsky, P. N. (2022). Package ‘lme4’. CRAN. https://cran.r-project.org/web/packages/lme4/lme4.pdf

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Brauer, M., & Curtin, J. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411. https://doi.org/10.1037/met0000159

Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing type 1 error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001

Meteyard, L., & Davies, R. A. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112, 104092. https://doi.org/10.1016/j.jml.2020.104092

Singmann, H., & Kellen, D. (2019). An introduction to mixed models for experimental psychology. In D. H. Spieler & E. Schumacher (Eds.), New methods in cognitive psychology (pp. 4–31). Psychology Press.

A table of results for Bayesian mixed-effects models: Grouping variables and specifying random slopes

Thu, 29 Dec 2022 00:00:00 +0000

Here I share the format applied to tables presenting the results of Bayesian models in Bernabeu (2022; the table for frequentist models is covered in this other post). The sample table presents a Bayesian mixed-effects model that was fitted using the R package brms (Bürkner et al., 2022). The mixed effects were driven by the maximal principle (Brauer & Curtin, 2018). The format of the table resembles one of the examples published by the American Psychological Association. However, there are also deviations from those examples. For instance, in the present table, the effects are grouped under informative labels to facilitate the readers’ comprehension, using the kableExtra package (Zhu, 2022). Furthermore, the random slopes are specified using superscript letters and a footnote. The table can be reproduced using the materials at https://osf.io/gt5uf.

Loading packages and the results of the models

library(dplyr)
library(knitr)
library(tibble)
library(stringr)
library(lmerTest)
library(kableExtra)
# Load Bayesian results summary
semanticpriming_summary_weaklyinformativepriors_exgaussian =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/bayesian_analysis/results/semanticpriming_summary_weaklyinformativepriors_exgaussian.rds?raw=true')))

Adjusting the names of the effects

First, to facilitate the understanding of the results, the original names of the effects will be adjusted in the brms summary.

# Rename effects in plain language and specify the random slopes
# (if any) for each effect, in the footnote. For this purpose,
# superscripts are added to the names of the appropriate effects.
#
# In the interactions below, word-level variables are presented
# first for the sake of consistency (the order does not affect
# the results in any way). Also in the interactions, double
# colons are used to inform the 'bayesian_model_table'
# function that the two terms in the interaction must be split
# into two lines.
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_attentional_control'] = 'Attentional control'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_attentional_control'] = 'Attentional control'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_vocabulary_size'] = 'Vocabulary size <sup>a</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_recoded_participant_gender'] = 'Gender <sup>a</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_target_word_frequency'] = 'Word frequency'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_target_number_syllables'] = 'Number of syllables'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_word_concreteness_diff'] = 'Word-concreteness difference'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_cosine_similarity'] = 'Language-based similarity <sup>b</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_visual_rating_diff'] = 'Visual-strength difference <sup>b</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_recoded_interstimulus_interval'] = 'Stimulus onset asynchrony (SOA) <sup>b</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_word_concreteness_diff:z_vocabulary_size'] =
'Word-concreteness difference :: Vocabulary size'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_word_concreteness_diff:z_recoded_interstimulus_interval'] =
'Word-concreteness difference : SOA'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_word_concreteness_diff:z_recoded_participant_gender'] =
'Word-concreteness difference : Gender'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_attentional_control:z_cosine_similarity'] =
'Language-based similarity :: Attentional control'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_attentional_control:z_visual_rating_diff'] =
'Visual-strength difference :: Attentional control'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_vocabulary_size:z_cosine_similarity'] =
'Language-based similarity :: Vocabulary size'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_vocabulary_size:z_visual_rating_diff'] =
'Visual-strength difference :: Vocabulary size'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_recoded_participant_gender:z_cosine_similarity'] =
'Language-based similarity : Gender'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_recoded_participant_gender:z_visual_rating_diff'] =
'Visual-strength difference : Gender'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_cosine_similarity:z_recoded_interstimulus_interval'] =
'Language-based similarity : SOA <sup>b</sup>'
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed)[
rownames(semanticpriming_summary_weaklyinformativepriors_exgaussian$fixed) ==
'z_visual_rating_diff:z_recoded_interstimulus_interval'] =
'Visual-strength difference : SOA <sup>b</sup>'

`bayesian_model_table()`

In Bernabeu (2022), the following custom function was used.

The above function was used to render a PDF output. In the current scenario, however, we have an HTML output. In the above function, the code used for the $\widehat{R}$ tailored to the HTML output (&Rcirc;) does not render properly.

Instead, the LaTeX code $\\widehat{R}$ must be used. Therefore, we’ll correct this error and load the function below.

# Function used in the manuscript to present summaries from 'brms' models
# in APA-formatted tables. The only obligatory argument to be supplied is
# a summary of a 'brms' model.
bayesian_model_table =
function(model_summary, show_intercept = TRUE, select_effects = NULL,
order_effects = NULL, format = NULL,
# If interaction_symbol_x = TRUE, replace double colons with
# times symbols followed by line breaks and indentation.
# Then, replace single colons with times symbols.
interaction_symbol_x = FALSE,
caption = 'Summary of the lmerTest model.') {
require(dplyr)
require(knitr)
require(tibble)
require(stringr)
require(lmerTest)
require(kableExtra)
# Create data frame
model_summary =
data.frame(Effect = rownames(model_summary$fixed),
Estimate = model_summary$fixed$Estimate,
SE = model_summary$fixed$Est.Error,
CrI_2.5 = model_summary$fixed$`l-95% CI`,
CrI_97.5 = model_summary$fixed$`u-95% CI`,
Rhat = model_summary$fixed$Rhat,
row.names = NULL)
# Process credible intervals and present both inside square brackets
model_summary$CrI_2.5 = model_summary$CrI_2.5 %>%
# Round off and keep trailing zeros
sprintf('%.2f', .) %>%
# Remove minus sign from pure zeros
sub('-0.00', '0.00', .)
model_summary$CrI_97.5 = model_summary$CrI_97.5 %>%
# Round off and keep trailing zeros
sprintf('%.2f', .) %>%
# Remove minus sign from pure zeros
sub('-0.00', '0.00', .)
model_summary$CrI_95 = paste0('[', model_summary$CrI_2.5, ', ',
model_summary$CrI_97.5, ']')
# If show_intercept = FALSE, remove it
if(isFALSE(show_intercept)) {
model_summary = model_summary %>% filter(!grepl('Intercept', Effect))
# Put 'Intercept' in parentheses
} else if(!is.null(model_summary[model_summary$Effect == 'Intercept', 'Effect'])) {
model_summary[model_summary$Effect == 'Intercept', 'Effect'] = '(Intercept)'
}
# If select_effects was supplied, apply it and order effects accordingly
if(!is.null(select_effects)) {
model_summary = model_summary %>% filter(Effect %in% select_effects) %>%
arrange(factor(Effect, levels = select_effects))
}
# If order_effects was supplied, apply order
if(!is.null(order_effects)) {
model_summary = model_summary %>%
arrange(factor(Effect, levels = order_effects))
}
# Round other values
model_summary$Estimate = model_summary$Estimate %>% as.numeric %>%
# Round off and keep trailing zeros
sprintf('%.2f', .) %>%
# Remove minus sign from pure zeros
sub('-0.00', '0.00', .)
model_summary$SE = model_summary$SE %>% as.numeric %>%
# Round off and keep trailing zeros
sprintf('%.2f', .)
model_summary$Rhat = model_summary$Rhat %>% as.numeric %>%
# Round off and keep trailing zeros
sprintf('%.2f', .)
# Order columns
model_summary = model_summary %>% select(Effect, Estimate, SE, CrI_95, Rhat)
# Right-align all columns after first one
align = c('l', 'r', 'r', 'r', 'r')
# Establish latex or HTML format: if no format supplied,
# try to obtain it from knitr, or apply HTML
if(missing(format) || is.null(format)) {
if(knitr::is_latex_output()) {
format = 'latex'
} else format = 'html'
}
# HTML format
if(format == 'html') {
# If interaction_symbol_x = TRUE, replace double colons with times
# symbols followed by line breaks and indentation. Then, replace
# single colons with times symbols.
if(interaction_symbol_x) {
model_summary$Effect = model_summary$Effect %>%
gsub('::', ' &times; <br> &nbsp;&nbsp;', .) %>%
gsub(':', ' &times; ', .)
}
# Output table
model_summary %>%
# Remove header of first column and rename other headers
rename(' ' = 'Effect', '&beta;' = 'Estimate', '<i>SE</i>' = 'SE',
'95% CrI' = 'CrI_95', '$\\widehat{R}$' = 'Rhat') %>%
# Present table
kbl(digits = 2, booktabs = TRUE, escape = FALSE, align = align,
# Caption of the table (default unless specified)
caption = caption,
# Disable occasional line gap (https://stackoverflow.com/a/49018919/7050882)
linesep = '') %>%
# Apply nice kableExtra format
kable_styling() %>%
# Center-align header row
row_spec(0, align = 'c')
# LaTeX format
} else {
# If interaction_symbol_x = TRUE, replace double colons with times
# symbols followed by line breaks and indentation. Then, replace
# single colons with times symbols.
if(interaction_symbol_x) {
model_summary$Effect = model_summary$Effect %>%
gsub('::', ' $\\\\times$ \n \\\\hspace{0.3cm}', .) %>%
gsub(':', ' $\\\\times$ ', .)
}
model_summary$Effect = model_summary$Effect %>%
# Escape underscores to avoid error in table
str_replace_all('_', '\\\\_') %>%
# Allow line breaks in the names of the effects
# (used in the interactions)
kableExtra::linebreak(align = 'l')
# Output table
model_summary %>%
# Remove header of first column and rename other headers
rename(' ' = 'Effect', '$\\upbeta$' = 'Estimate', '$SE$' = 'SE',
'95\\% CrI' = 'CrI_95', '$\\widehat R$' = 'Rhat') %>%
# Present table
kbl(digits = 2, booktabs = TRUE, escape = FALSE, align = align,
# Caption of the table (default unless specified)
caption = caption,
# Disable occasional line gap (https://stackoverflow.com/a/49018919/7050882)
linesep = '') %>%
# Apply nice kableExtra format
kable_styling() %>%
# Center-align header row
row_spec(0, align = 'c')
}
}

The function in use

# Create table (using custom function from the 'R_functions' folder)
bayesian_model_table(
semanticpriming_summary_weaklyinformativepriors_exgaussian,
order_effects = c('(Intercept)',
'Attentional control',
'Vocabulary size <sup>a</sup>',
'Gender <sup>a</sup>',
'Word frequency',
'Number of syllables',
'Word-concreteness difference',
'Language-based similarity <sup>b</sup>',
'Visual-strength difference <sup>b</sup>',
'Stimulus onset asynchrony (SOA) <sup>b</sup>',
'Word-concreteness difference :: Vocabulary size',
'Word-concreteness difference : SOA',
'Word-concreteness difference : Gender',
'Language-based similarity :: Attentional control',
'Visual-strength difference :: Attentional control',
'Language-based similarity :: Vocabulary size',
'Visual-strength difference :: Vocabulary size',
'Language-based similarity : Gender',
'Visual-strength difference : Gender',
'Language-based similarity : SOA <sup>b</sup>',
'Visual-strength difference : SOA <sup>b</sup>'),
# Replace colons in the names of interactions with times symbols
interaction_symbol_x = TRUE,
# No title
caption = NULL) %>%
# Group predictors under headings
pack_rows('Individual differences', 2, 4) %>%
pack_rows('Target-word lexical covariates', 5, 6) %>%
pack_rows('Prime--target relationship', 7, 9) %>%
pack_rows('Task condition', 10, 10) %>%
pack_rows('Interactions', 11, 21) %>%
# Apply white background to override default shading in HTML output
row_spec(1:21, background = 'white') %>%
# Highlight covariates
row_spec(c(2, 5:7, 11:15), background = '#FFFFF1') %>%
# Format
kable_classic(full_width = FALSE, html_font = 'Cambria') %>%
# Footnote describing abbreviations, random slopes, etc.
# LaTeX code used to format the text.
footnote(escape = FALSE, threeparttable = TRUE, general_title = '<br>',
general = paste('*Note*. &beta; = Estimate based on $z$-scored predictors; *SE* = standard error;',
'CrI = credible interval. Yellow rows contain covariates. Some interactions are ',
'split over two lines, with the second line indented. <br>',
'<sup>a</sup> By-word random slopes were included for this effect.',
'<sup>b</sup> By-participant random slopes were included for this effect.'))

	β	SE	95% CrI	$\widehat{R}$
(Intercept)	0.00	0.00	[0.00, 0.01]	1.00
Individual differences
Attentional control	0.00	0.00	[0.00, 0.01]	1.00
Vocabulary size ^a	-0.01	0.00	[-0.01, 0.00]	1.00
Gender ^a	0.00	0.00	[0.00, 0.01]	1.00
Target-word lexical covariates
Word frequency	-0.11	0.00	[-0.12, -0.11]	1.00
Number of syllables	0.07	0.00	[0.06, 0.07]	1.00
Prime–target relationship
Word-concreteness difference	0.01	0.00	[0.00, 0.01]	1.00
Language-based similarity ^b	-0.06	0.00	[-0.07, -0.06]	1.00
Visual-strength difference ^b	0.01	0.00	[0.01, 0.01]	1.00
Task condition
Stimulus onset asynchrony (SOA) ^b	0.03	0.01	[0.02, 0.04]	1.00
Interactions
Word-concreteness difference × Vocabulary size	0.00	0.00	[0.00, 0.00]	1.00
Word-concreteness difference × SOA	0.00	0.00	[0.00, 0.00]	1.00
Word-concreteness difference × Gender	0.00	0.00	[0.00, 0.00]	1.00
Language-based similarity × Attentional control	0.00	0.00	[-0.01, 0.00]	1.00
Visual-strength difference × Attentional control	0.00	0.00	[0.00, 0.00]	1.00
Language-based similarity × Vocabulary size	0.00	0.00	[-0.01, 0.00]	1.00
Visual-strength difference × Vocabulary size	0.00	0.00	[0.00, 0.00]	1.00
Language-based similarity × Gender	0.00	0.00	[-0.01, 0.00]	1.00
Visual-strength difference × Gender	0.00	0.00	[0.00, 0.00]	1.00
Language-based similarity × SOA ^b	0.00	0.00	[0.00, 0.00]	1.00
Visual-strength difference × SOA ^b	0.00	0.00	[0.00, 0.00]	1.00

Note. β = Estimate based on $z$-scored predictors; SE = standard error; CrI = credible interval. Yellow rows contain covariates. Some interactions are split over two lines, with the second line indented. ^a By-word random slopes were included for this effect. ^b By-participant random slopes were included for this effect.

Shading specific rows

Shading specific rows is done differently when the output is PDF, as shown below (see p. 170 in Bernabeu, 2022).

Session info

If you encounter any blockers while reproducing the table using the materials at https://osf.io/gt5uf, my current session info may be useful.

sessionInfo()

## R version 4.3.2 (2023-10-31 ucrt)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 11 x64 (build 22621)
##
## Matrix products: default
##
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
##
## time zone: Europe/Oslo
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] kableExtra_1.3.4 lmerTest_3.1-3 lme4_1.1-35.1
## [4] Matrix_1.6-4 stringr_1.5.1 tibble_3.2.1
## [7] dplyr_1.1.4 knitr_1.45 xaringanExtra_0.7.0
##
## loaded via a namespace (and not attached):
## [1] sass_0.4.8 utf8_1.2.4 generics_0.1.3
## [4] xml2_1.3.6 blogdown_1.18 stringi_1.8.3
## [7] lattice_0.22-5 digest_0.6.33 magrittr_2.0.3
## [10] evaluate_0.23 grid_4.3.2 bookdown_0.37
## [13] fastmap_1.1.1 jsonlite_1.8.8 httr_1.4.7
## [16] rvest_1.0.3 fansi_1.0.6 viridisLite_0.4.2
## [19] scales_1.3.0 numDeriv_2016.8-1.1 jquerylib_0.1.4
## [22] cli_3.6.2 rlang_1.1.2 munsell_0.5.0
## [25] splines_4.3.2 withr_2.5.2 cachem_1.0.8
## [28] yaml_2.3.8 tools_4.3.2 uuid_1.1-1
## [31] nloptr_2.0.3 minqa_1.2.6 colorspace_2.1-0
## [34] ggplot2_3.4.4 webshot_0.5.5 boot_1.3-28.1
## [37] vctrs_0.6.5 R6_2.5.1 lifecycle_1.0.4
## [40] MASS_7.3-60 pkgconfig_2.0.3 pillar_1.9.0
## [43] bslib_0.6.1 gtable_0.3.4 glue_1.6.2
## [46] Rcpp_1.0.11 systemfonts_1.0.5 xfun_0.41
## [49] tidyselect_1.2.0 rstudioapi_0.15.0 htmltools_0.5.7
## [52] nlme_3.1-164 svglite_2.1.3 rmarkdown_2.25
## [55] compiler_4.3.2

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Bürkner, P.-C., Gabry, J., Weber, S., Johnson, A., Modrak, M., Badr, H. S., Weber, F., Ben-Shachar, M. S., & Rabel, H. (2022). Package ’brms’. CRAN. https://cran.r-project.org/web/packages/brms/brms.pdf

Zhu, H. (2022). Package ’kableExtra’. CRAN. https://cran.r-project.org/web/packages/kableExtra/kableExtra.pdf

A table of results for frequentist mixed-effects models: Grouping variables and specifying random slopes

Thu, 29 Dec 2022 00:00:00 +0000

Here I share the format applied to tables presenting the results of frequentist models in Bernabeu (2022; the table for Bayesian models is covered in this other post). The sample table presents a mixed-effects model that was fitted using the R package lmerTest (Kuznetsova et al., 2022). The mixed effects were driven by the maximal principle (Brauer & Curtin, 2018). The format of the table resembles one of the examples published by the American Psychological Association. However, there are also deviations from those examples. For instance, in the present table, the effects are grouped under informative labels to facilitate the readers’ comprehension, using the kableExtra package (Zhu, 2022). Furthermore, the random slopes are specified using superscript letters and a footnote. The table can be reproduced using the materials at https://osf.io/gt5uf.

Loading packages and the results of the models

library(dplyr)
library(knitr)
library(tibble)
library(stringr)
library(lmerTest)
library(kableExtra)
# Load frequentist coefficients (estimates and confidence intervals)
KR_summary_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/KR_summary_semanticpriming_lmerTest.rds?raw=true')))
confint_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/confint_semanticpriming_lmerTest.rds?raw=true')))

Adjusting the names of the effects

First, to facilitate the understanding of the results, the original names of the effects will be adjusted in the lmerTest summary and in the confidence intervals object.

Incidentally, the confidence intervals were obtained using the confint.merMod function from the lme4 package, as neither lmerTest nor lme4 currently provide confidence intervals in their default results output. However, computing the confidence intervals is uncomplicated (see code).

# Rename effects in plain language and specify the random slopes
# (if any) for each effect, in the footnote. For this purpose,
# superscripts are added to the names of the appropriate effects.
#
# In the interactions below, word-level variables are presented
# first for the sake of consistency (the order does not affect
# the results in any way). Also in the interactions, double
# colons are used to inform the 'frequentist_model_table'
# function that the two terms in the interaction must be split
# into two lines.
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_attentional_control'] = 'Attentional control'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_vocabulary_size'] = 'Vocabulary size <sup>a</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_participant_gender'] = 'Gender <sup>a</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_target_word_frequency'] = 'Word frequency'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_target_number_syllables'] = 'Number of syllables'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_word_concreteness_diff'] = 'Word-concreteness difference'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_cosine_similarity'] = 'Language-based similarity <sup>b</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_visual_rating_diff'] = 'Visual-strength difference <sup>b</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_interstimulus_interval'] = 'Stimulus onset asynchrony (SOA) <sup>b</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_word_concreteness_diff:z_vocabulary_size'] =
'Word-concreteness difference :: Vocabulary size'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_word_concreteness_diff:z_recoded_interstimulus_interval'] =
'Word-concreteness difference : SOA'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_word_concreteness_diff:z_recoded_participant_gender'] =
'Word-concreteness difference : Gender'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_attentional_control:z_cosine_similarity'] =
'Language-based similarity :: Attentional control'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_attentional_control:z_visual_rating_diff'] =
'Visual-strength difference :: Attentional control'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_vocabulary_size:z_cosine_similarity'] =
'Language-based similarity :: Vocabulary size'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_vocabulary_size:z_visual_rating_diff'] =
'Visual-strength difference :: Vocabulary size'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_participant_gender:z_cosine_similarity'] =
'Language-based similarity : Gender'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_participant_gender:z_visual_rating_diff'] =
'Visual-strength difference : Gender'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_interstimulus_interval:z_cosine_similarity'] =
'Language-based similarity : SOA <sup>b</sup>'
rownames(KR_summary_semanticpriming_lmerTest$coefficients)[
rownames(KR_summary_semanticpriming_lmerTest$coefficients) ==
'z_recoded_interstimulus_interval:z_visual_rating_diff'] =
'Visual-strength difference : SOA <sup>b</sup>'
# Next, change the names in the confidence intervals object
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_attentional_control'] = 'Attentional control'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_vocabulary_size'] = 'Vocabulary size <sup>a</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_participant_gender'] = 'Gender <sup>a</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_target_word_frequency'] = 'Word frequency'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_target_number_syllables'] = 'Number of syllables'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_word_concreteness_diff'] = 'Word-concreteness difference'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_cosine_similarity'] = 'Language-based similarity <sup>b</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_visual_rating_diff'] = 'Visual-strength difference <sup>b</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_interstimulus_interval'] = 'Stimulus onset asynchrony (SOA) <sup>b</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_word_concreteness_diff:z_vocabulary_size'] =
'Word-concreteness difference :: Vocabulary size'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_word_concreteness_diff:z_recoded_interstimulus_interval'] =
'Word-concreteness difference : SOA'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_word_concreteness_diff:z_recoded_participant_gender'] =
'Word-concreteness difference : Gender'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_attentional_control:z_cosine_similarity'] =
'Language-based similarity :: Attentional control'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_attentional_control:z_visual_rating_diff'] =
'Visual-strength difference :: Attentional control'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_vocabulary_size:z_cosine_similarity'] =
'Language-based similarity :: Vocabulary size'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_vocabulary_size:z_visual_rating_diff'] =
'Visual-strength difference :: Vocabulary size'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_participant_gender:z_cosine_similarity'] =
'Language-based similarity : Gender'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_participant_gender:z_visual_rating_diff'] =
'Visual-strength difference : Gender'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_interstimulus_interval:z_cosine_similarity'] =
'Language-based similarity : SOA <sup>b</sup>'
rownames(confint_semanticpriming_lmerTest)[
rownames(confint_semanticpriming_lmerTest) ==
'z_recoded_interstimulus_interval:z_visual_rating_diff'] =
'Visual-strength difference : SOA <sup>b</sup>'

`frequentist_model_table()`

The following custom function was used.

Loading the function from GitHub

source('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/raw/main/R_functions/frequentist_model_table.R')

The function in use

# Create table (using custom function from the 'R_functions' folder)
frequentist_model_table(
KR_summary_semanticpriming_lmerTest,
confint_semanticpriming_lmerTest,
order_effects = c('(Intercept)',
'Attentional control',
'Vocabulary size <sup>a</sup>',
'Gender <sup>a</sup>',
'Word frequency',
'Number of syllables',
'Word-concreteness difference',
'Language-based similarity <sup>b</sup>',
'Visual-strength difference <sup>b</sup>',
'Stimulus onset asynchrony (SOA) <sup>b</sup>',
'Word-concreteness difference :: Vocabulary size',
'Word-concreteness difference : SOA',
'Word-concreteness difference : Gender',
'Language-based similarity :: Attentional control',
'Visual-strength difference :: Attentional control',
'Language-based similarity :: Vocabulary size',
'Visual-strength difference :: Vocabulary size',
'Language-based similarity : Gender',
'Visual-strength difference : Gender',
'Language-based similarity : SOA <sup>b</sup>',
'Visual-strength difference : SOA <sup>b</sup>'),
# Replace colons in the names of interactions with times symbols
interaction_symbol_x = TRUE,
# No title
caption = NULL) %>%
# Group predictors under headings
pack_rows('Individual differences', 2, 4) %>%
pack_rows('Target-word lexical covariates', 5, 6) %>%
pack_rows('Prime--target relationship', 7, 9) %>%
pack_rows('Task condition', 10, 10) %>%
pack_rows('Interactions', 11, 21) %>%
# Apply white background to override default shading in HTML output
row_spec(1:21, background = 'white') %>%
# Highlight covariates
row_spec(c(2, 5:7, 11:15), background = '#FFFFF1') %>%
# Format
kable_classic(full_width = FALSE, html_font = 'Cambria') %>%
# Footnote describing abbreviations, random slopes, etc.
# LaTeX code used to format the text.
footnote(escape = FALSE, threeparttable = TRUE, general_title = '<br>',
general = paste('*Note*. &beta; = Estimate based on $z$-scored predictors; *SE* = standard error;',
'CI = confidence interval. Yellow rows contain covariates. Some interactions are ',
'split over two lines, with the second line indented. <br>',
'<sup>a</sup> By-word random slopes were included for this effect.',
'<sup>b</sup> By-participant random slopes were included for this effect.'))

	β	SE	95% CI	t	p
(Intercept)	0.00	0.00	[0.00, 0.01]	1.59	.112
Individual differences
Attentional control	0.00	0.00	[0.00, 0.00]	-0.56	.577
Vocabulary size ^a	0.00	0.00	[0.00, 0.00]	0.02	.987
Gender ^a	0.00	0.00	[0.00, 0.00]	-0.03	.979
Target-word lexical covariates
Word frequency	-0.16	0.00	[-0.16, -0.15]	-49.40	<.001
Number of syllables	0.07	0.00	[0.07, 0.08]	22.81	<.001
Prime–target relationship
Word-concreteness difference	0.01	0.00	[0.01, 0.02]	3.48	.001
Language-based similarity ^b	-0.08	0.00	[-0.08, -0.07]	-22.44	<.001
Visual-strength difference ^b	0.01	0.00	[0.01, 0.02]	4.18	<.001
Task condition
Stimulus onset asynchrony (SOA) ^b	0.06	0.01	[0.04, 0.07]	7.47	<.001
Interactions
Word-concreteness difference × Vocabulary size	0.00	0.00	[0.00, 0.01]	1.31	.189
Word-concreteness difference × SOA	0.00	0.00	[0.00, 0.01]	2.57	.010
Word-concreteness difference × Gender	0.00	0.00	[-0.01, 0.00]	-0.97	.332
Language-based similarity × Attentional control	-0.01	0.00	[-0.01, 0.00]	-2.46	.014
Visual-strength difference × Attentional control	0.00	0.00	[0.00, 0.00]	0.24	.810
Language-based similarity × Vocabulary size	-0.01	0.00	[-0.01, 0.00]	-2.34	.020
Visual-strength difference × Vocabulary size	0.00	0.00	[-0.01, 0.00]	-1.37	.172
Language-based similarity × Gender	0.00	0.00	[-0.01, 0.00]	-0.79	.433
Visual-strength difference × Gender	0.00	0.00	[0.00, 0.01]	1.46	.144
Language-based similarity × SOA ^b	0.01	0.00	[0.00, 0.01]	3.22	.001
Visual-strength difference × SOA ^b	0.00	0.00	[-0.01, 0.00]	-2.25	.025

Note. β = Estimate based on $z$-scored predictors; SE = standard error; CI = confidence interval. Yellow rows contain covariates. Some interactions are split over two lines, with the second line indented. ^a By-word random slopes were included for this effect. ^b By-participant random slopes were included for this effect.

Shading specific rows

Shading specific rows is done differently when the output is PDF, as shown below (see p. 62 in Bernabeu, 2022).

Session info

If you encounter any blockers while reproducing the table using the materials at https://osf.io/gt5uf, my current session info may be useful.

sessionInfo()

## R version 4.3.2 (2023-10-31 ucrt)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 11 x64 (build 22621)
##
## Matrix products: default
##
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
##
## time zone: Europe/Oslo
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] kableExtra_1.3.4 lmerTest_3.1-3 lme4_1.1-35.1
## [4] Matrix_1.6-4 stringr_1.5.1 tibble_3.2.1
## [7] dplyr_1.1.4 knitr_1.45 xaringanExtra_0.7.0
##
## loaded via a namespace (and not attached):
## [1] sass_0.4.8 utf8_1.2.4 generics_0.1.3
## [4] xml2_1.3.6 blogdown_1.18 stringi_1.8.3
## [7] lattice_0.22-5 digest_0.6.33 magrittr_2.0.3
## [10] evaluate_0.23 grid_4.3.2 bookdown_0.37
## [13] fastmap_1.1.1 jsonlite_1.8.8 httr_1.4.7
## [16] rvest_1.0.3 fansi_1.0.6 viridisLite_0.4.2
## [19] scales_1.3.0 numDeriv_2016.8-1.1 jquerylib_0.1.4
## [22] cli_3.6.2 rlang_1.1.2 munsell_0.5.0
## [25] splines_4.3.2 withr_2.5.2 cachem_1.0.8
## [28] yaml_2.3.8 tools_4.3.2 uuid_1.1-1
## [31] nloptr_2.0.3 minqa_1.2.6 colorspace_2.1-0
## [34] ggplot2_3.4.4 webshot_0.5.5 boot_1.3-28.1
## [37] vctrs_0.6.5 R6_2.5.1 lifecycle_1.0.4
## [40] MASS_7.3-60 pkgconfig_2.0.3 pillar_1.9.0
## [43] bslib_0.6.1 gtable_0.3.4 glue_1.6.2
## [46] Rcpp_1.0.11 systemfonts_1.0.5 xfun_0.41
## [49] tidyselect_1.2.0 rstudioapi_0.15.0 htmltools_0.5.7
## [52] nlme_3.1-164 svglite_2.1.3 rmarkdown_2.25
## [55] compiler_4.3.2

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2022). Package ’lmerTest’. CRAN. https://cran.r-project.org/web/packages/lmerTest/lmerTest.pdf

Zhu, H. (2022). Package ’kableExtra’. CRAN. https://cran.r-project.org/web/packages/kableExtra/kableExtra.pdf

Plotting two-way interactions from mixed-effects models using alias variables

Mon, 26 Dec 2022 00:00:00 +0000

Whereas the direction of main effects can be interpreted from the sign of the estimate, the interpretation of interaction effects often requires plots. This task is facilitated by the R package sjPlot (Lüdecke, 2022). In Bernabeu (2022), the sjPlot function called plot_model served as the basis for the creation of some custom functions. One of these functions is alias_interaction_plot, which allows the plotting of interactions between a continuous variable and a categorical variable. Importantly, the categorical variable is replaced with an alias variable. This feature allows the back-transformation of the categorical variable to facilitate the communication of the results, for instance, when the categorical variable was sum-coded, which has been recommended for mixed-effects models (Brauer & Curtin, 2018).

Below, we’ll use the function with a model fitted using lmerTest (Kuznetsova et al., 2022), although the function also works with several other models (see sjPlot manual). The plot can be reproduced using the materials at https://osf.io/gt5uf.

Alias interaction plot

The function

The function in use

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2022). Package ’lmerTest’. CRAN. https://cran.r-project.org/web/packages/lmerTest/lmerTest.pdf

Lüdecke, D. (2022). Package ’sjPlot’. CRAN. https://cran.r-project.org/web/packages/sjPlot/sjPlot.pdf

Plotting two-way interactions from mixed-effects models using ten or six bins

Mon, 26 Dec 2022 00:00:00 +0000

Whereas the direction of main effects can be interpreted from the sign of the estimate, the interpretation of interaction effects often requires plots. This task is facilitated by the R package sjPlot (Lüdecke, 2022). In Bernabeu (2022), the sjPlot function called plot_model served as the basis for the creation of some custom functions. Two of these functions are deciles_interaction_plot and sextiles_interaction_plot. These functions allow the plotting of interactions between two continuous variables. In the case of deciles_interaction_plot, one of the variables is divided into ten bins, known as deciles, and the other variable is unchanged. In the case of sextiles_interaction_plot, one of the variables is divided into six bins, or sextiles, and the other variable is unchanged. Sample size per bin can be printed in the legend, which is particularly useful for research involving individual differences.

Below, we’ll use these functions with models fitted using lmerTest (Kuznetsova et al., 2022), although the functions also work with several other models (see sjPlot manual). The plots can be reproduced using the materials at https://osf.io/gt5uf.

Deciles interaction plot

The function

The function in use

Sextiles interaction plot

The function

The function in use

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2022). Package ‘lmerTest’. CRAN. https://cran.r-project.org/web/packages/lmerTest/lmerTest.pdf

Lüdecke, D. (2022). Package ‘sjPlot’. CRAN. https://cran.r-project.org/web/packages/sjPlot/sjPlot.pdf

Why can't we be friends? Plotting frequentist (lmerTest) and Bayesian (brms) mixed-effects models

Fri, 23 Dec 2022 00:00:00 +0000

Frequentist and Bayesian statistics are sometimes regarded as fundamentally different philosophies. Indeed, can both methods qualify as philosophies, or is one of them just a pointless ritual? Is frequentist statistics about $p$ values only? Are frequentist estimates diametrically opposed to Bayesian posterior distributions? Are confidence intervals and credible intervals irreconcilable? Will R crash if lmerTest and brms are simultaneously loaded? If only we could fit frequentist and Bayesian models to the same data and plot the results together, we might get a glimpse into these puzzles.

All the analyses shown below can be reproduced using the materials at https://osf.io/gt5uf. The combination of the frequentist and the Bayesian estimates in the same plot is achieved using the following custom function from Bernabeu (2022).

Visualising frequentist and Bayesian estimates in one plot

Both frequentist and Bayesian statistics offer the options of hypothesis testing and parameter estimation (Cumming, 2014; Kruschke & Liddell, 2018; Rouder et al., 2018; Schmalz et al., 2022; Tendeiro & Kiers, 2019, 2022; van Ravenzwaaij & Wagenmakers, 2022). In the statistical analyses conducted by Bernabeu (2022), hypothesis testing was performed within the frequentist framework, whereas parameter estimation was performed within both the frequentist and the Bayesian frameworks (for other examples of the estimation approach, see Milek et al., 2018; Pregla et al., 2021; Rodríguez-Ferreiro et al., 2020).

Let’s load the function from GitHub and put it to the test.

# Presenting the frequentist and the Bayesian estimates in the same plot.
# For this purpose, the frequentist results are merged into a plot from
# brms::mcmc_plot()
source('https://raw.githubusercontent.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/main/R_functions/frequentist_bayesian_plot.R')
# install.packages('devtools')
# library(devtools)
# install_version('tidyverse', '1.3.1') # Due to breaking changes, Version 1.3.1 is required.
# install_version('ggplot2', '3.3.5') # Due to breaking changes, Version 3.3.5 is required.
library(tidyverse)
library(ggplot2)
library(Cairo)
# Load frequentist coefficients (estimates and confidence intervals)
KR_summary_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/KR_summary_semanticpriming_lmerTest.rds?raw=true')))
confint_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/confint_semanticpriming_lmerTest.rds?raw=true')))
# Below are the default names of the effects
# rownames(KR_summary_semanticpriming_lmerTest$coefficients)
# rownames(confint_semanticpriming_lmerTest)
# Load Bayesian posterior distributions
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/bayesian_analysis/results/semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian.rds?raw=true')))
# Below are the default names of the effects
# levels(semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian$data$parameter)
# Reorder the components of interactions in the frequentist results to match
# with the order present in the Bayesian results.
rownames(KR_summary_semanticpriming_lmerTest$coefficients) =
rownames(KR_summary_semanticpriming_lmerTest$coefficients) %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_cosine_similarity',
replacement = 'z_cosine_similarity:z_recoded_interstimulus_interval') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_visual_rating_diff',
replacement = 'z_visual_rating_diff:z_recoded_interstimulus_interval')
rownames(confint_semanticpriming_lmerTest) =
rownames(confint_semanticpriming_lmerTest) %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_cosine_similarity',
replacement = 'z_cosine_similarity:z_recoded_interstimulus_interval') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_visual_rating_diff',
replacement = 'z_visual_rating_diff:z_recoded_interstimulus_interval')
# Create a vector containing the names of the effects. This vector will be passed
# to the plotting function.
new_labels =
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian$data$parameter %>%
unique %>%
# Remove the default 'b_' from the beginning of each effect
str_remove('^b_') %>%
# Put Intercept in parentheses
str_replace(pattern = 'Intercept', replacement = '(Intercept)') %>%
# First, adjust names of variables (both in main effects and in interactions)
str_replace(pattern = 'z_target_word_frequency',
replacement = 'Target-word frequency') %>%
str_replace(pattern = 'z_target_number_syllables',
replacement = 'Number of target-word syllables') %>%
str_replace(pattern = 'z_word_concreteness_diff',
replacement = 'Word-concreteness difference') %>%
str_replace(pattern = 'z_cosine_similarity',
replacement = 'Language-based similarity') %>%
str_replace(pattern = 'z_visual_rating_diff',
replacement = 'Visual-strength difference') %>%
str_replace(pattern = 'z_attentional_control',
replacement = 'Attentional control') %>%
str_replace(pattern = 'z_vocabulary_size',
replacement = 'Vocabulary size') %>%
str_replace(pattern = 'z_recoded_participant_gender',
replacement = 'Gender') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval',
replacement = 'SOA') %>%
# Show acronym in main effect of SOA
str_replace(pattern = '^SOA$',
replacement = 'Stimulus onset asynchrony (SOA)') %>%
# Second, adjust order of effects in interactions. In the output from the model,
# the word-level variables of interest (i.e., 'z_cosine_similarity' and
# 'z_visual_rating_diff') sometimes appeared second in their interactions. For
# better consistency, the code below moves those word-level variables (with
# their new names) to the first position in their interactions. Note that the
# order does not affect the results in any way.
sub('(\\w+.*):(Language-based similarity|Visual-strength difference)',
'\\2:\\1',
.) %>%
# Replace colons denoting interactions with times symbols
str_replace(pattern = ':', replacement = ' &times; ')
# Create plot
plot_semanticpriming_frequentist_bayesian_plot_weaklyinformativepriors_exgaussian =
frequentist_bayesian_plot(KR_summary_semanticpriming_lmerTest,
confint_semanticpriming_lmerTest,
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian,
labels = new_labels, interaction_symbol_x = TRUE,
vertical_line_at_x = 0, x_title = 'Effect size (&beta;)',
legend_ncol = 1) +
theme(legend.position = 'bottom')

plot_semanticpriming_frequentist_bayesian_plot_weaklyinformativepriors_exgaussian

Frequentist and Bayesian estimates are not so polar opposites, are they? What is more, the larger differences between some estimates are the result of the priors that were set on the corresponding effects. With uninformative priors, the frequentist and the Bayesian estimates are virtually identical.

Now it’s time to consider in earnest:

Is frequentist statistics about $p$ values only? Are frequentist estimates diametrically opposed to Bayesian posterior distributions? Are confidence intervals and credible intervals irreconcilable? Will R crash if lmerTest and brms are simultaneously loaded?

Session info

If you encounter any blockers while reproducing the above analyses using the materials at https://osf.io/gt5uf, my current session info may be useful. For instance, the legend of the plot may not show if the latest versions of the ggplot2 and the tidyverse packages are used. Instead, ggplot2 3.3.5 and tidyverse 1.3.1 should be installed using install_version('ggplot2', '3.3.5') and install_version('tidyverse', '1.3.1').

sessionInfo()

## R version 4.2.3 (2023-03-15 ucrt)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 22621)
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggtext_0.1.2 Cairo_1.6-0 forcats_1.0.0
## [4] stringr_1.5.0 dplyr_1.1.1 purrr_1.0.1
## [7] readr_2.1.4 tidyr_1.3.0 tibble_3.2.1
## [10] ggplot2_3.3.5 tidyverse_1.3.1 knitr_1.42
## [13] xaringanExtra_0.7.0
##
## loaded via a namespace (and not attached):
## [1] Rcpp_1.0.10 lubridate_1.9.2 digest_0.6.31 utf8_1.2.3
## [5] plyr_1.8.8 R6_2.5.1 cellranger_1.1.0 ggridges_0.5.4
## [9] backports_1.4.1 reprex_2.0.2 evaluate_0.21 highr_0.10
## [13] httr_1.4.6 blogdown_1.16 pillar_1.9.0 rlang_1.1.0
## [17] uuid_1.1-0 readxl_1.4.2 rstudioapi_0.14 jquerylib_0.1.4
## [21] rmarkdown_2.21 labeling_0.4.2 munsell_0.5.0 gridtext_0.1.5
## [25] broom_1.0.4 compiler_4.2.3 modelr_0.1.11 xfun_0.38
## [29] pkgconfig_2.0.3 htmltools_0.5.5 tidyselect_1.2.0 bookdown_0.33.3
## [33] fansi_1.0.4 crayon_1.5.2 tzdb_0.4.0 dbplyr_2.3.2
## [37] withr_2.5.0 commonmark_1.9.0 grid_4.2.3 jsonlite_1.8.4
## [41] gtable_0.3.3 lifecycle_1.0.3 DBI_1.1.3 magrittr_2.0.3
## [45] scales_1.2.1 cli_3.4.1 stringi_1.7.12 cachem_1.0.7
## [49] farver_2.1.1 fs_1.6.1 xml2_1.3.3 bslib_0.4.2
## [53] generics_0.1.3 vctrs_0.6.1 tools_4.2.3 glue_1.6.2
## [57] markdown_1.5 hms_1.1.3 fastmap_1.1.1 yaml_2.3.7
## [61] timechange_0.2.0 colorspace_2.1-0 rvest_1.0.3 haven_2.5.2
## [65] sass_0.4.6

References

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206.

Milek, A., Butler, E. A., Tackman, A. M., Kaplan, D. M., Raison, C. L., Sbarra, D. A., Vazire, S., & Mehl, M. R. (2018). “Eavesdropping on happiness” revisited: A pooled, multisample replication of the association between life satisfaction and observed daily conversation quantity and quality. Psychological Science, 29(9), 1451–1462. https://doi.org/10.1177/0956797618774252

Pregla, D., Lissón, P., Vasishth, S., Burchert, F., & Stadie, N. (2021). Variability in sentence comprehension in aphasia in German. Brain and Language, 222, 105008. https://doi.org/10.1016/j.bandl.2021.105008

Rodríguez-Ferreiro, J., Aguilera, M., & Davies, R. (2020). Semantic priming and schizotypal personality: Reassessing the link between thought disorder and enhanced spreading of semantic activation. PeerJ, 8, e9511. https://doi.org/10.7717/peerj.9511

Rouder, J. N., Haaf, J. M., & Vandekerckhove, J. (2018). Bayesian inference for psychology, part IV: Parameter estimation and Bayes factors. Psychonomic Bulletin & Review, 25(1), 102–113. https://doi.org/10.3758/s13423-017-1420-7

Schmalz, X., Biurrun Manresa, J., & Zhang, L. (2021). What is a Bayes factor? Psychological Methods. https://doi.org/10.1037/met0000421

Tendeiro, J. N., & Kiers, H. A. L. (2019). A review of issues about null hypothesis Bayesian testing. Psychological Methods, 24(6), 774–795. https://doi.org/10.1037/met0000221

Tendeiro, J. N., & Kiers, H. A. L. (2022). On the white, the black, and the many shades of gray in between: Our reply to van Ravenzwaaij and Wagenmakers (2021). Psychological Methods, 27(3), 466–475. https://doi.org/10.1037/met0000505

van Ravenzwaaij, D., & Wagenmakers, E.-J. (2022). Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019). Psychological Methods, 27(3), 451–465. https://doi.org/10.1037/met0000415

Bayesian workflow: Prior determination, predictive checks and sensitivity analyses

Thu, 22 Dec 2022 00:00:00 +0000

library(dplyr)
library(ggplot2)
library(ggridges)
library(ggtext)
library(patchwork)
library(papaja)

This post presents a code-through of a Bayesian workflow in R, which can be reproduced using the materials at https://osf.io/gt5uf. The content is closely based on Bernabeu (2022), which was in turn based on lots of other references. In addition to those, you may wish to consider Nicenboim et al. (n.d.), a book in preparation that is already available online (https://vasishth.github.io/bayescogsci/book).

In Bernabeu (2022), a Bayesian analysis was performed to complement the estimates that had been obtained in the frequentist analysis. Whereas the goal of the frequentist analysis had been hypothesis testing, for which $p$ values were used, the goal of the Bayesian analysis was parameter estimation. Accordingly, we estimated the posterior distribution of every effect, without calculating Bayes factors (for other examples of the same estimation approach, see Milek et al., 2018; Pregla et al., 2021; Rodríguez-Ferreiro et al., 2020; for comparisons between estimation and hypothesis testing, see Cumming, 2014; Kruschke & Liddell, 2018; Rouder et al., 2018; Schmalz et al., 2021; Tendeiro & Kiers, 2019, in press; van Ravenzwaaij & Wagenmakers, 2021). In the estimation approach, the estimates are interpreted by considering the position of their credible intervals in relation to the expected effect size. That is, the closer an interval is to an effect size of 0, the smaller the effect of that predictor. For instance, an interval that is symmetrically centred on 0 indicates a very small effect, whereas—in comparison—an interval that does not include 0 at all indicates a far larger effect.

This analysis served two purposes: first, to ascertain the interpretation of the smaller effects—which were identified as unreliable in the power analyses—, and second, to complement the estimates obtained in the frequentist analysis. The latter purpose was pertinent because the frequentist models presented convergence warnings—even though it must be noted that a previous study found that frequentist and Bayesian estimates were similar despite convergence warnings appearing in the frequentist analysis (Rodríguez-Ferreiro et al., 2020). Furthermore, the complementary analysis was pertinent because the frequentist models presented residual errors that deviated from normality—even though mixed-effects models are fairly robust to such a deviation (Knief & Forstmeier, 2021; Schielzeth et al., 2020). Owing to these precedents, we expected to find broadly similar estimates in the frequentist analyses and in the Bayesian ones. Across studies, each frequentist model has a Bayesian counterpart, with the exception of the secondary analysis performed in Study 2.1 (semantic priming) that included vision-based similarity as a predictor. The R package ‘brms’, Version 2.17.0, was used for the Bayesian analysis (Bürkner, 2018; Bürkner et al., 2022).

Priors

Priors are one of the hardest nuts to crack in Bayesian statistics. First, it can be useful to inspect what priors can be set in the model. Second, it is important to visualise a reasonable set of priors based on the available literature or any other available sources. Third, just before fitting the model, the adequacy of a range of priors should be assessed using prior predictive checks. Fourth, posterior predictive checks were performed to assess the consistency between the observed data and new data predicted by the posterior distributions. Fifth, the influence of the priors on the results should be assessed through a prior sensitivity analysis (Lee & Wagenmakers, 2014; Van de Schoot et al., 2021; also see Bernabeu, 2022; Pregla et al., 2021; Rodríguez-Ferreiro et al., 2020; Stone et al., 2020; Stone et al., 2021).

1. Checking what priors can be set

The brms::get_prior function can be used to check what effects in the model can be assigned a prior. The output (see example) will include the current (perhaps default) prior on each effect.

2. Determining the priors

The priors were established by inspecting the effect sizes obtained in previous studies as well as the effect sizes obtained in our frequentist analyses of the present data (reported in Studies 2.1, 2.2 and 2.3). In the first regard, the previous studies that were considered were selected because the experimental paradigms, variables and analytical procedures they had used were similar to those used in our current studies. Specifically, regarding paradigms, we sought studies that implemented: (I) semantic priming with a lexical decision task—as in Study 2.1—, (II) semantic decision—as in Study 2.2—, or (III) lexical decision—as in Study 2.3. Regarding analytical procedures, we sought studies in which both the dependent and the independent variables were $z$-scored. We found two studies that broadly matched these criteria: Lim et al. (2020) (see Table 5 therein) and Pexman & Yap (2018) (see Tables 6 and 7 therein). Out of these studies, Pexman & Yap (2018) contained the variables that were most similar to ours, which included vocabulary size (labelled ‘NAART’) and word frequency.

Based on both these studies and on the frequentist analyses, a range of effect sizes was identified that spanned between β = -0.30 and β = 0.30. This range was centred around 0 as the variables were $z$-scored. The bounds of this range were determined by the largest effects, which appeared in Pexman & Yap (2018). Pexman et al. conducted a semantic decision study, and split the data set into abstract and concrete words. The two largest effects they found were—first—a word concreteness effect in the concrete-words analysis of β = -0.41, and—second—a word concreteness effect in the abstract-words analysis of β = 0.20. Unlike Pexman et al., we did not split the data set into abstract and concrete words, but analysed these sets together. Therefore, we averaged between the aforementioned values, obtaining a range between β = -0.30 and β = 0.30.

In the results of Lim et al. (2020) and Pexman & Yap (2018), and in our frequentist results, some effects consistently presented a negative polarity (i.e., leading to shorter response times), whereas some other effects were consistently positive. We incorporated the direction of effects into the priors only in cases of large effects that had presented a consistent direction (either positive or negative) in previous studies and in our frequentist analyses in the present studies. These criteria were matched by the following variables: word frequency—with a negative direction, as higher word frequency leads to shorter RTs (Brysbaert et al., 2016; Brysbaert et al., 2018; Lim et al., 2020; Mendes & Undorf, 2021; Pexman & Yap, 2018)—, number of letters and number of syllables—both with positive directions (Barton et al., 2014; Beyersmann et al., 2020; Pexman & Yap, 2018)—, and orthographic Levenshtein distance—with a positive direction (Cerni et al., 2016; Dijkstra et al., 2019; Kim et al., 2018; Yarkoni et al., 2008). We did not incorporate information about the direction of the word concreteness effect, as this effect can follow different directions in abstract and concrete words (Brysbaert et al., 2014; Pexman & Yap, 2018), and we analysed both sets of words together. In conclusion, the four predictors that had directional priors were covariates. All the other predictors had priors centred on 0. Last, as a methodological matter, it is noteworthy that most of the psycholinguistic studies applying Bayesian analysis have not incorporated any directional information in priors (e.g., Pregla et al., 2021; Rodríguez-Ferreiro et al., 2020; Stone et al., 2020; cf. Stone et al., 2021).

Prior distributions

The choice of priors can influence the results in consequential ways. To assess the extent of this influence, prior sensitivity analyses have been recommended. These analyses are performed by comparing the effect of more and less strict priors—or, in other words, priors varying in their degree of informativeness. The degree of variation is adjusted through the standard deviation, and the means are not varied (Lee & Wagenmakers, 2014; Stone et al., 2020; Van de Schoot et al., 2021).

In this way, we compared the results obtained using ‘informative’ priors ($SD$ = 0.1), ‘weakly-informative’ priors ($SD$ = 0.2) and ‘diffuse’ priors ($SD$ = 0.3). These standard deviations were chosen so that around 95% of values in the informative priors would fall within our initial range of effect sizes that spanned from -0.30 to 0.30. All priors are illustrated in the figure below. These priors resembled others from previous psycholinguistic studies (Pregla et al., 2021; Stone et al., 2020; Stone et al., 2021). For instance, Stone et al. (2020) used the following priors: $Normal$(0, 0.1), $Normal$(0, 0.3) and $Normal$(0, 1). The range of standard deviations we used—i.e., 0.1, 0.2 and 0.3—was narrower than those of previous studies because our dependent variable and our predictors were $z$-scored, resulting in small estimates and small $SD$s (see Lim et al., 2020; Pexman & Yap, 2018). These priors were used on the fixed effects and on the standard deviation parameters of the fixed effects. For the correlations among the random effects, an $LKJ$(2) prior was used (Lewandowski et al., 2009). This is a ‘regularising’ prior, as it assumes that high correlations among random effects are rare (also used in Rodríguez-Ferreiro et al., 2020; Stone et al., 2020; Stone et al., 2021; Vasishth et al., 2018).

# Set seed number to ensure exact reproducibility
# of the random distributions
set.seed(123)
# The code below plots all our types of priors. Each distribution
# contains 10,000 simulations, resulting in 90,000 rows.
# The green vertical rectangle shows the range of plausible effect
# sizes based on previous studies that applied a similar analysis
# (Lim et al., 2020, https://doi.org/10.1177/1747021820906566;
# Pexman & Yap, 2018, https://doi.org/10.1037/xlm0000499) as
# well as on the frequentist analyses of the current data.
priors = data.frame(
informativeness =
as.factor(c(rep('Informative priors (*SD* = 0.1)', 30000),
rep('Weakly-informative priors (*SD* = 0.2)', 30000),
rep('Diffuse priors (*SD* = 0.3)', 30000))),
direction = as.factor(c(rep('negative', 10000),
rep('neutral', 10000),
rep('positive', 10000),
rep('negative', 10000),
rep('neutral', 10000),
rep('positive', 10000),
rep('negative', 10000),
rep('neutral', 10000),
rep('positive', 10000))),
direction_and_distribution =
as.factor(c(rep('Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.1)', 10000),
rep('Neutral (*M* = 0)<br>*Normal*(0, 0.1)', 10000),
rep('Positive (*M* = 0.1)<br>*Normal*(0.1, 0.1)', 10000),
rep('Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.2)', 10000),
rep('Neutral (*M* = 0)<br>*Normal*(0, 0.2)', 10000),
rep('Positive (*M* = 0.1)<br>*Normal*(0.1, 0.2)', 10000),
rep('Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.3)', 10000),
rep('Neutral (*M* = 0)<br>*Normal*(0, 0.3)', 10000),
rep('Positive (*M* = 0.1)<br>*Normal*(0.1, 0.3)', 10000))),
estimate = c(rnorm(10000, m = -0.1, sd = 0.1),
rnorm(10000, m = 0, sd = 0.1),
rnorm(10000, m = 0.1, sd = 0.1),
rnorm(10000, m = -0.1, sd = 0.2),
rnorm(10000, m = 0, sd = 0.2),
rnorm(10000, m = 0.1, sd = 0.2),
rnorm(10000, m = -0.1, sd = 0.3),
rnorm(10000, m = 0, sd = 0.3),
rnorm(10000, m = 0.1, sd = 0.3))
)
# Order factor levels
priors$informativeness =
ordered(priors$informativeness,
levels = c('Informative priors (*SD* = 0.1)',
'Weakly-informative priors (*SD* = 0.2)',
'Diffuse priors (*SD* = 0.3)'))
priors$direction =
ordered(priors$direction,
levels = c('negative', 'neutral', 'positive'))
priors$direction_and_distribution =
ordered(priors$direction_and_distribution,
levels = c('Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.1)',
'Neutral (*M* = 0)<br>*Normal*(0, 0.1)',
'Positive (*M* = 0.1)<br>*Normal*(0.1, 0.1)',
'Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.2)',
'Neutral (*M* = 0)<br>*Normal*(0, 0.2)',
'Positive (*M* = 0.1)<br>*Normal*(0.1, 0.2)',
'Negative (*M* = -0.1)<br>*Normal*(-0.1, 0.3)',
'Neutral (*M* = 0)<br>*Normal*(0, 0.3)',
'Positive (*M* = 0.1)<br>*Normal*(0.1, 0.3)'))
# PLOT zone
colours = c('#7276A2', 'black', '#A27272')
fill_colours = c('#CCCBE7', '#D7D7D7', '#E7CBCB')
# Initialise plot (`aes` specified separately to allow
# use of `geom_rect` at the end)
ggplot() +
# Turn to the distributions
stat_density_ridges(data = priors,
aes(x = estimate, y = direction_and_distribution,
color = direction, fill = direction),
geom = 'density_ridges_gradient', alpha = 0.7,
jittered_points = TRUE, quantile_lines = TRUE,
quantiles = c(0.025, 0.975), show.legend = F) +
scale_color_manual(values = colours) +
scale_fill_manual(values = fill_colours) +
# Adjust X axis to the random distributions obtained
scale_x_continuous(limits = c(min(priors$estimate),
max(priors$estimate)),
n.breaks = 6, expand = c(0.04, 0.04)) +
scale_y_discrete(expand = expansion(add = c(0.18, 1.9))) +
# Facets containing the three models varying in informativeness
facet_wrap(vars(informativeness), scales = 'free', dir = 'v') +
# Vertical line at x = 0
geom_vline(xintercept = 0, linetype = 'dashed', color = 'grey50') +
xlab('Effect size (&beta;)') +
ylab('Direction of the prior and corresponding distribution') +
theme_minimal() +
theme(axis.title.x = ggtext::element_markdown(size = 12, margin = margin(t = 9)),
axis.text.x = ggtext::element_markdown(size = 11, margin = margin(t = 4)),
axis.title.y = ggtext::element_markdown(size = 12, margin = margin(r = 9)),
axis.text.y = ggtext::element_markdown(lineheight = 1.6, colour = colours),
strip.background = element_rect(fill = 'grey98', colour = 'grey90',
linetype = 'solid'),
strip.text = element_markdown(size = 11, margin = margin(t = 7, b = 7)),
panel.spacing.y = unit(9, 'pt'), panel.grid.minor = element_blank(),
plot.margin = margin(8, 8, 9, 8)
) +
# Shaded rectangle containing range of previous effects
geom_rect(data = data.frame(x = 1), xmin = -0.3, xmax = 0.3,
ymin = -Inf, ymax = Inf, fill = 'darkgreen', alpha = .3)

Priors used in the three studies. The green vertical rectangle shows the range of plausible effect sizes based on previous studies and on our frequentist analyses. In the informative priors, around 95% of the values fall within the range.

3. Prior predictive checks

The adequacy of each of these priors was assessed by performing prior predictive checks, in which we compared the observed data to the predictions of the model (Van de Schoot et al., 2021). Furthermore, in these checks we also tested the adequacy of two model-wide distributions: the traditional Gaussian distribution (default in most analyses) and an exponentially modified Gaussian—dubbed ‘ex-Gaussian’—distribution (Matzke & Wagenmakers, 2009). The ex-Gaussian distribution was considered because the residual errors of the frequentist models were not normally distributed (Lo & Andrews, 2015), and because this distribution was found to be more appropriate than the Gaussian one in a previous, related study (see supplementary materials of Rodríguez-Ferreiro et al., 2020). The ex-Gaussian distribution had an identity link function, which preserves the interpretability of the coefficients, as opposed to a transformation applied directly to the dependent variable (Lo & Andrews, 2015). The results of these prior predictive checks revealed that the priors were adequate, and that the ex-Gaussian distribution was more appropriate than the Gaussian one, converging with Rodríguez-Ferreiro et al. (2020). Therefore, the ex-Gaussian distribution was used in the final models.

Models with a Gaussian distribution

The figures below show the prior predictive checks for the Gaussian models. These plots show the maximum, mean and minimum values of the observed data ($y$) and those of the predicted distribution ($y_{rep}$, which stands for replications of the outcome). The way of interpreting these plots is by comparing the observed data to the predicted distribution. The specifics of this comparison vary across the three plots. First, in the upper plot, which shows the maximum values, the ideal scenario would show the observed maximum value ($y$) overlapping with the maximum value of the predicted distribution ($y_{rep}$). Second, in the middle plot, showing the mean values, the ideal scenario would show the observed mean value ($y$) overlapping with the mean value of the predicted distribution ($y_{rep}$). Last, in the lower plot, which shows the minimum values, the ideal scenario would have the observed minimum value ($y$) overlapping with the minimum value of the predicted distribution ($y_{rep}$). While the overlap need not be absolute, the closer the observed and the predicted values are on the X axis, the better. As such, the three predictive checks below—corresponding to models that used the default Gaussian distribution—show that the priors fitted the data acceptably but not very well.

Prior predictive checks for the Gaussian, informative-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

Prior predictive checks for the Gaussian, weakly-informative-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

Prior predictive checks for the Gaussian, diffuse-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

Models with an exponentially-modified Gaussian (i.e., ex-Gaussian) distribution

In contrast to the above results, the figures below demonstrate that, when an ex-Gaussian distribution was used, the priors fitted the data far better, which converged with the results of a similar comparison performed by Rodríguez-Ferreiro et al. (2020; see supplementary materials of the latter study).

Prior predictive checks for the ex-Gaussian, informative-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

Prior predictive checks for the ex-Gaussian, weakly-informative-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

Prior predictive checks for the ex-Gaussian, diffuse-prior model from the semantic priming study. $y$ = observed data; $y_{rep}$ = predicted data.

4. Posterior predictive checks

Based on the results from the prior predictive checks, the ex-Gaussian distribution was used in the final models. Next, posterior predictive checks were performed to assess the consistency between the observed data and new data predicted by the posterior distributions (Van de Schoot et al., 2021). The figure below presents the posterior predictive checks for the latter models. The interpretation of these plots is simple: the distributions of the observed ($y$) and the predicted data ($y_{rep}$) should be as similar as possible. As such, the plots below suggest that the results are trustworthy.

Posterior predictive checks for the (ex-Gaussian) models from the semantic priming study. The observed data ($y$) and the predicted data ($y_{rep}$) almost entirely overlap with each other, demonstrating a very good fit.

5. Prior sensitivity analysis

In the main analysis, the informative, weakly-informative and diffuse priors were used in separate models. In other words, in each model, all priors had the same degree of informativeness (as done in Pregla et al., 2021; Rodríguez-Ferreiro et al., 2020; Stone et al., 2020; Stone et al., 2021). In this way, a prior sensitivity analysis was performed to acknowledge the likely influence of the priors on the posterior distributions—that is, on the results (Lee & Wagenmakers, 2014; Stone et al., 2020; Van de Schoot et al., 2021).

We’ll first load a custom function (frequentist_bayesian_plot) from GitHub.

source('https://raw.githubusercontent.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/main/R_functions/frequentist_bayesian_plot.R')

# Presenting the frequentist and the Bayesian estimates in the same plot.
# For this purpose, the frequentist results are merged into a plot from
# brms::mcmc_plot()
# install.packages('devtools')
# library(devtools)
# install_version('tidyverse', '1.3.1') # Due to breaking changes, Version 1.3.1 is required.
# install_version('ggplot2', '5.3.5') # Due to breaking changes, Version 5.3.5 is required.
library(tidyverse)
library(ggplot2)
library(Cairo)
# Load frequentist coefficients (estimates and confidence intervals)
KR_summary_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/KR_summary_semanticpriming_lmerTest.rds?raw=true')))
confint_semanticpriming_lmerTest =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/frequentist_analysis/results/confint_semanticpriming_lmerTest.rds?raw=true')))
# Below are the default names of the effects
# rownames(KR_summary_semanticpriming_lmerTest$coefficients)
# rownames(confint_semanticpriming_lmerTest)
# Load Bayesian posterior distributions
semanticpriming_posteriordistributions_informativepriors_exgaussian =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/bayesian_analysis/results/semanticpriming_posteriordistributions_informativepriors_exgaussian.rds?raw=true')))
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/bayesian_analysis/results/semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian.rds?raw=true')))
semanticpriming_posteriordistributions_diffusepriors_exgaussian =
readRDS(gzcon(url('https://github.com/pablobernabeu/language-sensorimotor-simulation-PhD-thesis/blob/main/semanticpriming/bayesian_analysis/results/semanticpriming_posteriordistributions_diffusepriors_exgaussian.rds?raw=true')))
# Below are the default names of the effects
# levels(semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian$data$parameter)
# Reorder the components of interactions in the frequentist results to match
# with the order present in the Bayesian results.
rownames(KR_summary_semanticpriming_lmerTest$coefficients) =
rownames(KR_summary_semanticpriming_lmerTest$coefficients) %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_cosine_similarity',
replacement = 'z_cosine_similarity:z_recoded_interstimulus_interval') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_visual_rating_diff',
replacement = 'z_visual_rating_diff:z_recoded_interstimulus_interval')
rownames(confint_semanticpriming_lmerTest) =
rownames(confint_semanticpriming_lmerTest) %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_cosine_similarity',
replacement = 'z_cosine_similarity:z_recoded_interstimulus_interval') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval:z_visual_rating_diff',
replacement = 'z_visual_rating_diff:z_recoded_interstimulus_interval')
# Create a vector containing the names of the effects. This vector will be passed
# to the plotting function.
new_labels =
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian$data$parameter %>%
unique %>%
# Remove the default 'b_' from the beginning of each effect
str_remove('^b_') %>%
# Put Intercept in parentheses
str_replace(pattern = 'Intercept', replacement = '(Intercept)') %>%
# First, adjust names of variables (both in main effects and in interactions)
str_replace(pattern = 'z_target_word_frequency',
replacement = 'Target-word frequency') %>%
str_replace(pattern = 'z_target_number_syllables',
replacement = 'Number of target-word syllables') %>%
str_replace(pattern = 'z_word_concreteness_diff',
replacement = 'Word-concreteness difference') %>%
str_replace(pattern = 'z_cosine_similarity',
replacement = 'Language-based similarity') %>%
str_replace(pattern = 'z_visual_rating_diff',
replacement = 'Visual-strength difference') %>%
str_replace(pattern = 'z_attentional_control',
replacement = 'Attentional control') %>%
str_replace(pattern = 'z_vocabulary_size',
replacement = 'Vocabulary size') %>%
str_replace(pattern = 'z_recoded_participant_gender',
replacement = 'Gender') %>%
str_replace(pattern = 'z_recoded_interstimulus_interval',
replacement = 'SOA') %>%
# Show acronym in main effect of SOA
str_replace(pattern = '^SOA$',
replacement = 'Stimulus onset asynchrony (SOA)') %>%
# Second, adjust order of effects in interactions. In the output from the model,
# the word-level variables of interest (i.e., 'z_cosine_similarity' and
# 'z_visual_rating_diff') sometimes appeared second in their interactions. For
# better consistency, the code below moves those word-level variables (with
# their new names) to the first position in their interactions. Note that the
# order does not affect the results in any way.
sub('(\\w+.*):(Language-based similarity|Visual-strength difference)',
'\\2:\\1',
.) %>%
# Replace colons denoting interactions with times symbols
str_replace(pattern = ':', replacement = ' &times; ')
# Create plots, beginning with the informative-prior model
plot_semanticpriming_frequentist_bayesian_plot_informativepriors_exgaussian =
frequentist_bayesian_plot(KR_summary_semanticpriming_lmerTest,
confint_semanticpriming_lmerTest,
semanticpriming_posteriordistributions_informativepriors_exgaussian,
labels = new_labels, interaction_symbol_x = TRUE,
vertical_line_at_x = 0, x_title = 'Effect size (&beta;)',
x_axis_labels = 3, note_frequentist_no_prior = TRUE) +
ggtitle('Prior *SD* = 0.1')
#####
plot_semanticpriming_frequentist_bayesian_plot_weaklyinformativepriors_exgaussian =
frequentist_bayesian_plot(KR_summary_semanticpriming_lmerTest,
confint_semanticpriming_lmerTest,
semanticpriming_posteriordistributions_weaklyinformativepriors_exgaussian,
labels = new_labels, interaction_symbol_x = TRUE,
vertical_line_at_x = 0, x_title = 'Effect size (&beta;)',
x_axis_labels = 3, note_frequentist_no_prior = TRUE) +
ggtitle('Prior *SD* = 0.2') +
theme(axis.text.y = element_blank())
#####
plot_semanticpriming_frequentist_bayesian_plot_diffusepriors_exgaussian =
frequentist_bayesian_plot(KR_summary_semanticpriming_lmerTest,
confint_semanticpriming_lmerTest,
semanticpriming_posteriordistributions_diffusepriors_exgaussian,
labels = new_labels, interaction_symbol_x = TRUE,
vertical_line_at_x = 0, x_title = 'Effect size (&beta;)',
x_axis_labels = 3, note_frequentist_no_prior = TRUE) +
ggtitle('Prior *SD* = 0.3') +
theme(axis.text.y = element_blank())

The figure below presents the posterior distribution of each effect in each model. The frequentist estimates are also shown to facilitate the comparison.

plot_semanticpriming_frequentist_bayesian_plot_informativepriors_exgaussian +
plot_semanticpriming_frequentist_bayesian_plot_weaklyinformativepriors_exgaussian +
plot_semanticpriming_frequentist_bayesian_plot_diffusepriors_exgaussian +
plot_layout(ncol = 3, guides = 'collect') & theme(legend.position = 'bottom')

Estimates from the frequentist analysis (in red) and from the Bayesian analysis (in blue) for the semantic priming study, in each model. The frequentist means (represented by points) are flanked by 95% confidence intervals. The Bayesian means (represented by vertical lines) are flanked by 95% credible intervals in light blue (in some cases, the interval is occluded by the bar of the mean)

A blog post on the frequentist-Bayesian plots is also available.

Session info

If you encounter any blockers while reproduce the above analyses using the materials at https://osf.io/gt5uf, my current session info may be useful. For instance, the legend of the last plot may not show if the latest versions of the ggplot2 and the tidyverse packages are used. Instead, ggplot2 3.3.5 and tidyverse 1.3.1 should be installed using install_version('ggplot2', '3.3.5') and install_version('tidyverse', '1.3.1').

sessionInfo()

## R version 4.2.3 (2023-03-15 ucrt)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 22621)
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] Cairo_1.6-0 forcats_1.0.0 stringr_1.5.0
## [4] purrr_1.0.1 readr_2.1.4 tidyr_1.3.0
## [7] tibble_3.2.1 tidyverse_1.3.1 papaja_0.1.1
## [10] tinylabels_0.2.3 patchwork_1.1.2 ggtext_0.1.2
## [13] ggridges_0.5.4 ggplot2_3.3.5 dplyr_1.1.1
## [16] knitr_1.42 xaringanExtra_0.7.0
##
## loaded via a namespace (and not attached):
## [1] fs_1.6.1 lubridate_1.9.2 insight_0.19.2 httr_1.4.6
## [5] tools_4.2.3 backports_1.4.1 bslib_0.4.2 utf8_1.2.3
## [9] R6_2.5.1 DBI_1.1.3 colorspace_2.1-0 withr_2.5.0
## [13] tidyselect_1.2.0 emmeans_1.8.7 compiler_4.2.3 cli_3.4.1
## [17] rvest_1.0.3 xml2_1.3.3 sandwich_3.0-2 labeling_0.4.2
## [21] bookdown_0.33.3 bayestestR_0.13.1 sass_0.4.6 scales_1.2.1
## [25] mvtnorm_1.1-3 commonmark_1.9.0 digest_0.6.31 rmarkdown_2.21
## [29] pkgconfig_2.0.3 htmltools_0.5.5 dbplyr_2.3.2 fastmap_1.1.1
## [33] highr_0.10 rlang_1.1.0 readxl_1.4.2 rstudioapi_0.14
## [37] jquerylib_0.1.4 farver_2.1.1 generics_0.1.3 zoo_1.8-11
## [41] jsonlite_1.8.4 magrittr_2.0.3 parameters_0.21.1 Matrix_1.6-1
## [45] Rcpp_1.0.10 munsell_0.5.0 fansi_1.0.4 lifecycle_1.0.3
## [49] stringi_1.7.12 multcomp_1.4-23 yaml_2.3.7 MASS_7.3-60
## [53] plyr_1.8.8 grid_4.2.3 crayon_1.5.2 lattice_0.21-8
## [57] haven_2.5.2 splines_4.2.3 gridtext_0.1.5 hms_1.1.3
## [61] pillar_1.9.0 uuid_1.1-0 markdown_1.5 estimability_1.4.1
## [65] effectsize_0.8.3 codetools_0.2-19 reprex_2.0.2 glue_1.6.2
## [69] evaluate_0.21 blogdown_1.16 modelr_0.1.11 vctrs_0.6.1
## [73] tzdb_0.4.0 cellranger_1.1.0 gtable_0.3.3 datawizard_0.8.0
## [77] cachem_1.0.7 xfun_0.38 xtable_1.8-4 broom_1.0.4
## [81] survival_3.5-7 timechange_0.2.0 TH.data_1.1-1

References

Barton, J. J. S., Hanif, H. M., Eklinder Björnström, L., & Hills, C. (2014). The word-length effect in reading: A review. Cognitive Neuropsychology, 31(5-6), 378–412. https://doi.org/10.1080/02643294.2014.895314

Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power. Lancaster University. https://doi.org/10.17635/lancaster/thesis/1795

Beyersmann, E., Grainger, J., & Taft, M. (2020). Evidence for embedded word length effects in complex nonwords. Language, Cognition and Neuroscience, 35(2), 235–245. https://doi.org/10.1080/23273798.2019.1659989

Brysbaert, M., Mandera, P., & Keuleers, E. (2018). The word frequency effect in word processing: An updated review. Current Directions in Psychological Science, 27(1), 45–50. https://doi.org/10.1177/0963721417727521

Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). The impact of word prevalence on lexical decision times: Evidence from the Dutch Lexicon Project 2. Journal of Experimental Psychology: Human Perception and Performance, 42(3), 441–458. https://doi.org/10.1037/xhp0000159

Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46, 904–911. https://doi.org/10.3758/s13428-013-0403-5

Bürkner, P.-C. (2018). Advanced Bayesian multilevel modeling with the R package brms. The R Journal, 10(1), 395–411. https://journal.r-project.org/archive/2018/RJ-2018-017/index.html

Cerni, T., Velay, J.-L., Alario, F.-X., Vaugoyeau, M., & Longcamp, M. (2016). Motor expertise for typing impacts lexical decision performance. Trends in Neuroscience and Education, 5(3), 130–138. https://doi.org/10.1016/j.tine.2016.07.007

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

Dijkstra, T., Wahl, A., Buytenhuijs, F., Halem, N. V., Al-Jibouri, Z., Korte, M. D., & Rekké, S. (2019). Multilink: A computational model for bilingual word recognition and word translation. Bilingualism: Language and Cognition, 22(4), 657–679. https://doi.org/10.1017/S1366728918000287

Kim, M., Crossley, S. A., & Skalicky, S. (2018). Effects of lexical features, textual properties, and individual differences on word processing times during second language reading comprehension. Reading and Writing, 31(5), 1155–1180. https://doi.org/10.1007/s11145-018-9833-x

Knief, U., & Forstmeier, W. (2021). Violating the normality assumption may be the lesser of two evils. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01587-5

Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge University Press. https://doi.org/10.1017/CBO9781139087759

Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001. https://doi.org/10.1016/j.jmva.2009.04.008

Lim, R. Y., Yap, M. J., & Tse, C.-S. (2020). Individual differences in Cantonese Chinese word recognition: Insights from the Chinese Lexicon Project. Quarterly Journal of Experimental Psychology, 73(4), 504–518. https://doi.org/10.1177/1747021820906566

Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6, 1171. https://doi.org/10.3389/fpsyg.2015.01171

Matzke, D., & Wagenmakers, E.-J. (2009). Psychological interpretation of the ex-Gaussian and shifted Wald parameters: A diffusion model analysis. Psychonomic Bulletin & Review, 16(5), 798–817. https://doi.org/10.3758/PBR.16.5.798

Mendes, P. S., & Undorf, M. (2021). On the pervasive effect of word frequency in metamemory. Quarterly Journal of Experimental Psychology, 17470218211053329. https://doi.org/10.1177/17470218211053329

Milek, A., Butler, E. A., Tackman, A. M., Kaplan, D. M., Raison, C. L., Sbarra, D. A., Vazire, S., & Mehl, M. R. (2018). “Eavesdropping on happiness” revisited: A pooled, multisample replication of the association between life satisfaction and observed daily conversation quantity and quality. Psychological Science, 29(9), 1451–1462. https://doi.org/10.1177/0956797618774252

Nicenboim, B., Schad, D., & Vasishth, S. (n.d.). An introduction to Bayesian data analysis for cognitive science. Chapman and Hall/CRC Statistics in the Social and Behavioral Sciences Series.

Pexman, P. M., & Yap, M. J. (2018). Individual differences in semantic processing: Insights from the Calgary semantic decision project. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(7), 1091–1112. https://doi.org/10.1037/xlm0000499

Pregla, D., Lissón, P., Vasishth, S., Burchert, F., & Stadie, N. (2021). Variability in sentence comprehension in aphasia in German. Brain and Language, 222, 105008. https://doi.org/10.1016/j.bandl.2021.105008

Rouder, J. N., Haaf, J. M., & Vandekerckhove, J. (2018). Bayesian inference for psychology, part IV: Parameter estimation and Bayes factors. Psychonomic Bulletin & Review, 25(1), 102–113. https://doi.org/10.3758/s13423-017-1420-7

Schielzeth, H., Dingemanse, N. J., Nakagawa, S., Westneat, D. F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N. A., Garamszegi, L. Z., & Araya‐Ajoy, Y. G. (2020). Robustness of linear mixed‐effects models to violations of distributional assumptions. Methods in Ecology and Evolution, 11(9), 1141–1152. https://doi.org/10.1111/2041-210X.13434

Schmalz, X., Biurrun Manresa, J., & Zhang, L. (2021). What is a Bayes factor? Psychological Methods. https://doi.org/10.1037/met0000421

Stone, K., Malsburg, T. von der, & Vasishth, S. (2020). The effect of decay and lexical uncertainty on processing long-distance dependencies in reading. PeerJ, 8, e10438. https://doi.org/10.7717/peerj.10438

Stone, K., Veríssimo, J., Schad, D. J., Oltrogge, E., Vasishth, S., & Lago, S. (2021). The interaction of grammatically distinct agreement dependencies in predictive processing. Language, Cognition and Neuroscience, 36(9), 1159–1179. https://doi.org/10.1080/23273798.2021.1921816

Tendeiro, J. N., & Kiers, H. A. L. (2019). A review of issues about null hypothesis Bayesian testing. Psychological Methods, 24(6), 774–795. https://doi.org/10.1037/met0000221

Tendeiro, J. N., & Kiers, H. A. L. (in press). On the white, the black, and the many shades of gray in between: Our reply to van Ravenzwaaij and Wagenmakers (2021). Psychological Methods.

Van de Schoot, R., Depaoli, S., Gelman, A., King, R., Kramer, B., Märtens, K., Tadesse, M. G., Vannucci, M., Willemsen, J., & Yau, C. (2021). Bayesian statistics and modelling. Nature Reviews Methods Primers, 1, 3. https://doi.org/10.1038/s43586-020-00003-0

van Ravenzwaaij, D., & Wagenmakers, E.-J. (2021). Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019). Psychological Methods. https://doi.org/10.1037/met0000415

Vasishth, S., Nicenboim, B., Beckman, M. E., Li, F., & Kong, E. J. (2018). Bayesian data analysis in the phonetic sciences: A tutorial introduction. Journal of Phonetics, 71, 147–161. https://doi.org/10.1016/j.wocn.2018.07.008

Yarkoni, T., Balota, D., & Yap, M. J. (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin & Review, 15(5), 971–979. https://doi.org/10.3758/PBR.15.5.971

Avoiding (R) Markdown knitting errors using knit_deleting_service_files()

Fri, 17 Dec 2021 21:46:06 +0000

The function knit_deleting_service_files() helps avoid (R) Markdown knitting errors caused by files and folders remaining from previous knittings (e.g., manuscript.tex, ZHJhZnQtYXBhLlJtZA==.Rmd, manuscript.synctex.gz). The only obligatory argument for this function is the name of a .Rmd or .md file. The optional argument is a path to a directory containing this file.

The function first offers deleting potential service files and folders, for which the user’s approval is requested in the console (see screenshot below). Next, the document is knitted. Last, the function offers deleting potential service files and folders again.

NOTE: The deletions, if accepted, are irreversible as they are made through unlink(). Therefore, our familiar adage truly applies: this function comes with ABSOLUTELY NO WARRANTY. Please ensure you understand the source code before using the function.

Screenshot of the function in use.

The function

Walking the line between reproducibility and efficiency in R Markdown: Three methods

Sun, 14 Nov 2021 00:10:08 +0000

As technology and research methods advance, the data sets tend to be larger and the methods more exhaustive. Consequently, the analyses take longer to run. This poses a challenge when the results are to be presented using R Markdown. One has to balance reproducibility and efficiency. On the one hand, it is desirable to keep the R Markdown document as self-contained as possible, so that those who may later examine the document can easily test and edit the code. On the other hand, it would be inefficient to create a document that is very slow to run or very long. The context of the task will determine how how time-consuming and long the code in an Rmd file can be. For instance, one could decide that the knitting can take up to 15 minutes, and each code chunk can span up to 30 lines.

Several methods can be used in each document to accommodate different types of code. Three methods are presented below, ordered from easier-to-reproduce to easier-to-knit.

For fast- and concise-enough code: Provide the original code in the Rmd file. The code is run as the document is knitted. Example:
```
 nrow(myData)
```
For fast-enough but very long code: Store the code in a separate script and source it in the Rmd file. The code is run as the document is knitted. Example:
```
 source('analysis/model_diagnostics.R')
```
For very slow and/or long code: Store the code in a separate script and run it prior to knitting the Rmd file, so that the output from the code (e.g., a model, a plot) is saved and can be read into the Rmd. Example:
```
 model_1 = readRDS('results/model_1.rds')
```

Importantly, even the third method allows the reproducibility of the code. It just requires a bit of additional documentation to ensure that the end user can also access the script in which the result was produced (e.g., ‘analysis/model_1.R’).

Tackling knitting errors in R Markdown

Sat, 13 Nov 2021 18:11:22 +0000

When knitting an R Markdown document after the first time, errors may sometimes appear. Three tips are recommended below.

1. Close PDF reader window

When the document is knitted through the ‘Knit’ button, a PDF reader window opens to present the result. Closing this window can help resolve errors.

2. Delete service files

Every time the Rmd is knitted, some service files are created. Some of these files have the ‘.tex’ extension (e.g., index.tex, Appendix-B.tex), whereas others do not (e.g., index.log, ZHJhZnQtYXBhLlJtZA==.Rmd, index.synctex.gz). Deleting these files can help resolve errors (see a possible function).

3. Delete code chunks related to the appendices

When knitting papaja documents containing any appendices, some code chunks related to the appendices will automatically appear at the end of the primary Rmd file—e.g.:

```{r echo = FALSE, results = 'asis', cache = FALSE}
papaja::render_appendix('Appendix-A.Rmd')
```

Normally, these code chunks are automatically removed as the knitting finishes. However, if the chunks remain in place, deleting them manually can help resolve errors.

Parallelizing simr::powercurve() in R

Fri, 23 Jul 2021 16:46:54 +0100

The powercurve function from the R package ‘simr’ (Green & MacLeod, 2016) can incur very long running times when the method used for the calculation of p values is Kenward-Roger or Satterthwaite (see Luke, 2017). Here I suggest three ways for cutting down this time.

Where possible, use a high-performance (or high-end) computing cluster. This removes the need to use personal computers for these long jobs.
In case you’re using the fixed() parameter of the powercurve function, and calculating the power for different effects, run these at the same time (‘in parallel’) on different machines, rather than one after another.
Parallelize the breaks argument. The breaks argument of the powercurve function allows the calculation of power for different levels of the grouping factor passed to along. Some grouping factors are participant, trial and item. The breaks argument sets the different sample sizes for which power will be calculated. Parallelizing breaks is done by running each number of levels in a separate function. When each has been run and saved, they are combined to allow the plotting. This procedure is demonstrated below.

Parallelizing `breaks`

Let’s do a minimal example using a toy lmer model. A power curve will be created for the fixed effect of x along different sample sizes of the grouping factor g.

Notice that the six sections of the power curve below are serially arranged, one after another. In contrast, to enable parallel processing, each power curve would be placed in a single script, and they would all be run at the same time.

Although the power curves below run in a few minutes, the settings that are often used (e.g., a larger model; fixed('x', 'sa') instead of fixed('x'); nsim = 500 instead of nsim = 50) take far longer. That is where parallel processing becomes useful.¹

library(lme4)
library(simr)
# Toy model with data from 'simr' package
fm = lmer(y ~ x + (x | g), data = simdata)
# Extend sample size of `g`
fm_extended_g = extend(fm, along = 'g', n = 12)
# Parallelize `breaks` by running each number of levels in a separate function.
# 4 levels of g
pwcurve_4g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 4,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 6 levels of g
pwcurve_6g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 6,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 8 levels of g
pwcurve_8g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 8,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 10 levels of g
pwcurve_10g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 10,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 12 levels of g
pwcurve_12g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 12,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)

Having saved each section of the power curve, we must now combine them to be able to plot them together (if you wish to automatise this procedure, consider this function).

# Create a destination object using any of the power curves above.
all_pwcurve = pwcurve_4g
# Combine results
all_pwcurve$ps = c(pwcurve_4g$ps[1], pwcurve_6g$ps[1], pwcurve_8g$ps[1],
pwcurve_10g$ps[1], pwcurve_12g$ps[1])
# Combine the different numbers of levels.
all_pwcurve$xval = c(pwcurve_4g$nlevels, pwcurve_6g$nlevels, pwcurve_8g$nlevels,
pwcurve_10g$nlevels, pwcurve_12g$nlevels)
print(all_pwcurve)

## Power for predictor 'x', (95% confidence interval),
## by number of levels in g:
## 4: 46.00% (31.81, 60.68) - 40 rows
## 6: 74.00% (59.66, 85.37) - 60 rows
## 8: 92.00% (80.77, 97.78) - 80 rows
## 10: 98.00% (89.35, 99.95) - 100 rows
## 12: 100.0% (92.89, 100.0) - 120 rows
##
## Time elapsed: 0 h 0 m 7 s

plot(all_pwcurve, xlab = 'Levels of g')

# For reproducibility purposes
sessionInfo()

## R version 4.5.1 (2025-06-13 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
##
## Matrix products: default
## LAPACK version 3.12.1
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
##
## time zone: Europe/London
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices datasets utils methods base
##
## other attached packages:
## [1] simr_1.0.7 lme4_1.1-37 Matrix_1.7-3
## [4] knitr_1.50 xaringanExtra_0.8.0
##
## loaded via a namespace (and not attached):
## [1] tidyr_1.3.1 sass_0.4.10 generics_0.1.4 renv_1.1.5
## [5] stringi_1.8.7 blogdown_1.21 lattice_0.22-7 digest_0.6.37
## [9] magrittr_2.0.3 evaluate_1.0.4 grid_4.5.1 bookdown_0.43
## [13] iterators_1.0.14 fastmap_1.2.0 plyr_1.8.9 jsonlite_2.0.0
## [17] backports_1.5.0 Formula_1.2-5 mgcv_1.9-3 purrr_1.1.0
## [21] jquerylib_0.1.4 abind_1.4-8 reformulas_0.4.1 Rdpack_2.6.4
## [25] cli_3.6.5 rlang_1.1.6 rbibutils_2.3 binom_1.1-1.1
## [29] splines_4.5.1 plotrix_3.8-4 cachem_1.1.0 yaml_2.3.10
## [33] RLRsim_3.1-8 parallel_4.5.1 tools_4.5.1 pbkrtest_0.5.5
## [37] uuid_1.2-1 nloptr_2.2.1 minqa_1.2.8 dplyr_1.1.4
## [41] boot_1.3-31 broom_1.0.9 vctrs_0.6.5 R6_2.6.1
## [45] lifecycle_1.0.4 stringr_1.5.1 car_3.1-3 MASS_7.3-65
## [49] pkgconfig_2.0.3 bslib_0.9.0 pillar_1.11.0 glue_1.8.0
## [53] Rcpp_1.1.0 tidyselect_1.2.1 tibble_3.3.0 xfun_0.52
## [57] htmltools_0.5.8.1 nlme_3.1-168 rmarkdown_2.29 carData_3.0-5
## [61] compiler_4.5.1

Just the code


library(lme4)
library(simr)
# Toy model
fm = lmer(y ~ x + (x | g), data = simdata)
# Extend sample size of `g`
fm_extended_g = extend(fm, along = 'g', n = 12)
# Parallelize `breaks` by running each number of levels in a separate function.
# 4 levels of g
pwcurve_4g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 4,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 6 levels of g
pwcurve_6g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 6,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 8 levels of g
pwcurve_8g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 8,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 10 levels of g
pwcurve_10g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 10,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# 12 levels of g
pwcurve_12g = powerCurve(fm_extended_g, fixed('x'), along = 'g', breaks = 12,
nsim = 50, seed = 123,
# No progress bar
progress = FALSE)
# Create a destination object using any of the power curves above.
all_pwcurve = pwcurve_4g
# Combine results
all_pwcurve$ps = c(pwcurve_4g$ps[1], pwcurve_6g$ps[1], pwcurve_8g$ps[1],
pwcurve_10g$ps[1], pwcurve_12g$ps[1])
# Combine the different numbers of levels.
all_pwcurve$xval = c(pwcurve_4g$nlevels, pwcurve_6g$nlevels, pwcurve_8g$nlevels,
pwcurve_10g$nlevels, pwcurve_12g$nlevels)
print(all_pwcurve)
plot(all_pwcurve, xlab = 'Levels of g')

References

Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1), 9. http://doi.org/10.5334/joc.10

Green, P., & MacLeod, C. J. (2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution 7(4), 493–498, https://doi.org/10.1111/2041-210X.12504

Kumle, L., Vo, M. L. H., & Draschkow, D. (2021). Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behavior Research Methods, 1–16. https://doi.org/10.3758/s13428-021-01546-0

Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y

The number of simulations set by nsim should be larger (Brysbaert & Stevens, 2018; Green & MacLeod, 2016). In addition, the effect size for x should be adjusted to the value that best fits with the planned study (Kumle et al., 2021).↩︎

WebVTT caption transcription app

Fri, 01 Jan 2021 00:00:00 +0000

How it works

graph TD A["User uploads WebVTT file
(vtt or txt)"] --> B["Regular expressions remove
metadata such as timestamps"] B --> C["Text formatted
into a paragraph"] C --> D["Result displayed
on the website"] D --> E["Download as docx
or txt document"]

This open-source, R-based web application converts video captions (subtitles) from the Web Video Text Tracks (WebVTT) Format into plain text. For this purpose, users upload a WebVTT file with the extension .vtt or .txt (examples available here and here).

Download VTT file

When the caption file is uploaded to the web application, metadata such as timestamps are automatically removed, and the text is formatted into a paragraph. The result is displayed on the website, and can be downloaded as .docx and .txt documents. Overall, this application serves to improve the accessibility of video captions.

The data is only available to the user, and is deleted when the website is closed.

Web application

The application can be used below, or it can be opened separately here.

Programming details

The source code is available on Github, where the app can be extended via pull requests. Questions and suggestions can be submitted as issues or emailed to pcbernabeu@gmail.com. The licence is Creative Commons Attribution 4.0 International.

The core of the application is in the index.Rmd script, which uses ‘regular expressions’ to process the VTT file.

In turn, the script above draws on the one below to enable the download of .docx documents. Last, the latter script draws on this Word template.

R Markdown amidst Madison parks

Fri, 14 Aug 2020 00:00:00 +0000

This document is part of teaching materials created for the workshop ‘Open data and reproducibility v2.1: R Markdown, dashboards and Binder’, delivered at the CarpentryCon 2020 conference. The purpose of this specific document is to practise R Markdown, including basic features such as Markdown markup and code chunks, along with more special features such as cross-references for figures, tables, code chunks, etc. The code is on GitHub.

Since this conference was originally going to take place in Madison, let’s look at some open data from the City of Madison.

Park types

[Placeholder text] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Figure 1 shows the number of parks within each type.

# Showing the code because echo = TRUE
dat = read.csv('https://opendata.arcgis.com/datasets/9e00ff81868e49b7ba65d4e628b9e14f_6.csv')
dat =
dat %>%
group_by(Type) %>%
mutate(parks_number = n())
ggplotly(
ggplot(dat, aes(x=reorder(Type, parks_number), y=parks_number,
text = paste('Number of parks =', parks_number))) +
theme(axis.title.y=element_blank()) + stat_identity(geom='bar') +
labs(x='Type', y='Number of parks') + coord_flip(),
tooltip = 'text'
)

Figure 1: Number of parks within each type.

Twenty largest parks

[Placeholder text] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Table 1 shows the twenty largest parks types, along with their type and acreage. The code doesn’t show below because echo = FALSE.

Table 1: The twenty largest parks in Madison.
Name	Type	Acreage
Cherokee Marsh - North Unit	CONSERVATION	946.58
Cherokee Marsh - South Unit (School Road Unit)	CONSERVATION	261.27
Elver Park	COMMUNITY	250.82
Northeast Park	COMMUNITY	237.76
Warner Park	COMMUNITY	213.49
Door Creek Park	COMMUNITY	159.97
Cherokee Marsh - Mendota Unit	CONSERVATION	122.08
Owen Conservation Park	CONSERVATION	96.79
Reindahl (Amund) Park	COMMUNITY	90.74
Olbrich Park	COMMUNITY	90.01
Yahara Hills Park (West)	COMMUNITY	82.20
Sycamore Park	COMMUNITY	71.42
Turville Point Conservation Park	CONSERVATION	64.28
Edna Taylor Conservation Park	CONSERVATION	60.27
Quann Park	COMMUNITY	55.43
Demetral Park	COMMUNITY	49.18
Prairie Ridge Conservation Park	CONSERVATION	48.76
Olin Park	COMMUNITY	47.12
Hiestand Park	COMMUNITY	46.27
Vilas (Henry) Park	COMMUNITY	45.67

We could decide to display the code for the table at this point (or any other), which can be done using the parameters ref.label = 'largest-parks', echo = TRUE, eval = FALSE in the chunk options (Xie, Dervieux, & Riederer, 2020).

dat %>%
summarize(Name = Park_Name, Type, Acreage) %>%
arrange(-Acreage, Name, Type) %>%
select(Name, Type, Acreage) %>%
head(20) %>%
kable(caption = 'The twenty largest parks in Madison.')

Web application for the simulation of experimental data

Mon, 01 Jun 2020 00:00:00 +0000

Purposes

This open-source, R-based web application is suitable for educational and research purposes in experimental and quantitative sciences. It allows the creation of varied data sets with specified structures, such as between-group and within-participant variables, that can be categorical or continuous. These parameters can be set throughout the various tabs (sections) from the top menu. In the last tab, the data set can be downloaded. The benefits of this application include time-saving and flexibility in the control of parameters.

How it works

graph TD A["Set parameters across tabs"] --> B["Define between-group
variables"] A --> C["Define within-participant
variables"] B --> D["Choose categorical
or continuous"] C --> D D --> E["Simulated data set"] E --> F["Build custom summary
(penultimate tab)"] F --> G["Download parameters and
data set (last tab)"]

Guidelines

General guidelines include the following:

In the names of variables, it is recommended to use only alphanumeric characters and underscore signs. The latter can be used to separate characters or words (e.g., variable_name). Different names should be used for each variable.
In the levels of categorical variables, alphanumeric, special characters and spaces are allowed.
In numeric fields (e.g., ‘Mean’, ‘Standard deviation’, ‘Relative probability [0, 1]’), only numbers and decimal points are allowed.
As the data set increases, so does the processing time.

More specific guidelines are available in each section.

The web application can be launched here.

Screenshot of the Dependent tab (view larger)

Reference

Bernabeu, P., & Lynott, D. (2020). Web application for the simulation of experimental data (Version 1.4). https://github.com/pablobernabeu/Experiment-simulation-app/

Code

This web application was developed in R (R Core Team, 2020). The code is available on Github, where contributions may be made. The initial code for this application was influenced by Section 5.7 (Simulating data for multi-factor designs) in Crump (2017). The R packages used include ‘dplyr’ (Wickham, François, Henry, & Müller, 2018), ‘DT’ (Xie, 2020), ‘flexdashboard’ (Iannone, Allaire, & Borges, 2020), ‘shiny’ (Chang, Cheng, Allaire, Xie, & McPherson, 2020) and ‘stringr’ (Wickham, 2019).

Options for development and local use of the app

Option A) Using local R/RStudio or RStudio Cloud project or Binder RStudio environment

[Step only necessary in R/RStudio] Install the packages in the versions used in the latest release of this application, by running:

install.packages('devtools')
library(devtools)
install_version('dplyr', '1.0.2', 'http://cran.us.r-project.org')
install_version('DT', '0.15', 'http://cran.us.r-project.org')
install_version('flexdashboard', '0.5.2', 'http://cran.us.r-project.org')
install_version('htmltools', '0.5.0', 'http://cran.us.r-project.org')
install_version('knitr', '1.30', 'http://cran.us.r-project.org')
install_version('ngram', '3.0.4', 'http://cran.us.r-project.org')
install_version('purrr', '0.3.4', 'http://cran.us.r-project.org')
install_version('shiny', '1.5.0', 'http://cran.us.r-project.org')
install_version('stringr', '1.4.0', 'http://cran.us.r-project.org')
install_version('tidyr', '1.1.2', 'http://cran.us.r-project.org')

Open the index.Rmd script.
Run the application by clicking on ▶️ Run document at the top left, or by running rmarkdown::run('index.Rmd') in the console.
Click on Open in Browser at the top left.

Option B) Using Dockerfile (see instructions)

Acknowledgements

Thank you to RStudio for the free hosting server used by this application, shinyapps.io.

References

Chang, W., Cheng, J., Allaire, J., Xie, Y., & McPherson, J. (2020). shiny: Web Application Framework for R. R package version 1.4.0. Available at http://CRAN.R-project.org/package=shiny.

Crump, M. J. C. (2017). Programming for Psychologists: Data Creation and Analysis (Version 1.1). https://crumplab.github.io/programmingforpsych/.

Iannone, R., Allaire, J. J., & Borges, B. (2020). Flexdashboard: R Markdown Format for Flexible Dashboards. http://rmarkdown.rstudio.com/flexdashboard.

R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

Wickham, H. (2019). stringr: Simple, Consistent Wrappers for Common String Operations. R package version 1.4.0. https://CRAN.R-project.org/package=stringr.

Wickham, H., François, R., Henry, L., & Müller, K. (2018). dplyr: A Grammar of Data Manipulation. R package version 0.7.6. https://CRAN.R-project.org/package=dplyr.

Xie, Y. (2020). DT: A Wrapper of the JavaScript Library “DataTables”. R package version 0.14. Available at https://CRAN.R-project.org/package=DT.

Contact

To submit any questions or feedback, please post an issue, or email Pablo Bernabeu at pcbernabeu@gmail.com.

Data dashboard: Butterfly species richness in Los Angeles

Wed, 01 Jan 2020 00:00:00 +0000

How it works

graph TD A["Open data from
Prudic et al. (2018)"] --> B["iNaturalist (citizen science)"] A --> C["BioScan (traditional traps)"] B --> D["Transform, merge,
and wrangle in R"] C --> D D --> E["Dashboard: butterfly species
richness in Los Angeles"]

This dashboard presents open data (iNaturalist and BioScan) from Prudic et al. (2018). In their study, Prudic et al. compared citizen science with traditional methods in the measurement of butterfly populations.

I developed this dashboard after reproducing the analyses of the original study in a Reprohack session.

My coding tasks included transforming the data to a long format,

# There are pseudovariables, that is, observations entered as variables.
# Since most R processes need the tidy format, convert below
# (see https://r4ds.had.co.nz/tidy-data.html). The specific numbers
# found through traps and crowdsourcing methods are preserved.
BioScan = BioScan %>% pivot_longer(
cols = Anthocharis_sara:Vanessa_cardui, names_to = "Species",
values_to = "Number", values_drop_na = TRUE
)
# Compare
#str(BioScan)
#str(dat)
# 928 rows now; the result of 29 pseudo-variables being transposed
# into rows, interacting with 32 previous rows, i.e., 29 * 32 = 928.

merging three data sets,

# The iNaturalist data set presents a challenge that differs from the
# pseudovariables found above. The number of animals of each species
# must be computed from repeated entries, per site.
iNaturalist = merge(iNaturalist,
iNaturalist %>%
count(species, site, name = 'Number'))

and, as ever, wrangling with the format of the dashboard pages to preserve the format of a table.

Species details {style="background-color: #FCFCFC;"}
=======================================================================
Column {style="data-width:100%; position:static; height:1000px;"}
-----------------------------------------------------------------------

Reference

Prudic, K. L., Oliver, J. C., Brown, B. V., & Long, E. C. (2018). Comparisons of citizen science data-gathering approaches to evaluate urban butterfly diversity. Insects, 9(4), 186. https://doi.org/10.3390/insects9040186

Data is present: Workshops and datathons

Wed, 01 Jan 2020 00:00:00 +0000

Enhanced data presentation using reproducible documents and dashboards

Calendar

Date	Title	Event and location	Registration
26 Nov 2020	Mixed-effects models in R, and a new tool for data simulation	New Tricks Seminars, Dept. Psychology, Lancaster University [online]
8 Oct 2020	Reproducibilidad en torno a una aplicación web	Reprohack en español, LatinR Conference 2020 [online]	Link
13 Aug 2020	Open data and reproducibility: R Markdown, data dashboards and Binder v2.1 (co-led with Florencia D’Andrea)	CarpentryCon@Home, The Carpentries [online]	Link
26 July 2020	Open data and reproducibility: R Markdown, data dashboards and Binder (co-led with Eirini Zormpa)	UK Cognitive Linguistics Conference, University of Birmingham [online]	Link
6 May 2020	R Markdown	Lancaster University [online]
Event cancelled	Open data and reproducibility 2.0	SatRday Newcastle upon Tyne, Newcastle University	Link

Background

This project offers free activities to learn and practise reproducible data presentation. Pablo Bernabeu organises these events in the context of a Software Sustainability Institute Fellowship.

Open-source software

Programming languages such as R and Python offer free, powerful resources for data processing, visualisation and analysis. Experience in these programs is highly valued in data-intensive disciplines.

Open data

Original data has become a public good in many research fields thanks to cultural and technological advances. On the internet, we can find innumerable data sets from sources such as scientific journals and repositories (e.g., OSF), local and national governments (e.g., London, UK [1, 2]), non-governmental organisations (e.g., data.world), etc. Researchers inside and outside academia nowadays share a lot of their data under attribution licences (e.g., Creative Commons, the UK Open Government Licence, etc.). This allows any external analysts to access these raw data, create (additional) visualisations and analyses, and share these. In society, making data more accessible can demonstrably benefit citizens (despite limitations).

Activities

Activities comprise free workshops and datathons.

Workshops

R is a programming language greatly equipped for the creation of reproducible documents and dashboards. Four workshops are offered that cover a suite of interrelated tools—R, R Markdown, data dashboards and Binder environments—, all underlain by reproducible workflows and open-source software.

Each workshop includes taught and practical sections. The practice provides a chance for participants to experience and address common issues with the code. The level of taught sections is largely tailored to participants; similarly, practice sections are individually adaptable by means of easier and tougher tasks. The duration is also flexible, and some of the workshops can be combined into the same session.

The RStudio interface is used in all workshops. Multi-levelled, real code examples are used. Throughout the workshops, and especially in the practice sections, individual questions will be encouraged.

Workshop 1: Introduction to R

This workshop can serve as an introduction to R or a revision. It demonstrates what can be done in R, and provides resources for individual training. Since the duration is limited, online courses are also recommended (see examples and fee waivers for full content).

Data structures
Packages: general-purpose examples (e.g., tidyverse) and more specific ones (e.g., for statistics or geography)
Loading and writing data, in native and foreign formats
Tidy format
Wide versus long format. For most processes in R, data needs to be in a tidy, long format.

Illustration from Postma and Goedhart (2019).

Combining data sets
Data summaries
Plots with ggplot2::ggplot()
Interactive plots with plotly::ggplotly()
Statistics
Linear mixed-effects models (see also a review of practices)
How functions work
Debugging. Code errors are known as bugs. They can tiresome, but also interesting sometimes! :sweat_smile: Some tips for the first many years of experience include: reading and investigating error messages, in both source and console windows; controlling letter case and typos; closing parentheses and inverted commas; ensuring to have the necessary packages installed and loaded; following the format required by each function. To debug, break up code into subcomponents and test each of those to find out the source of the error. Once we act on that, the best outcome is seeing the code work, but sometimes different errors overlap, in which case we may see one error disappearing before another one appears. Debugging soon leads to proficient information seeking. The search process often begins on an internet search engine and extends to user communities, package documentation, tutorials, blogs… (see video explanation). Advanced debugging tools are also available.
Vast availability of free resources on the internet, from Coursera and other MOOC sites, RStudio, University of Glasgow, Carpentries, etc.
Community: StackOverflow, RStudio Community, Github issues (e.g., for R packages), etc. Using and contributing back.
RStudio Cloud: a personal RStudio environment on the internet

Workshop 2: R Markdown documents

Set your input and output in stone using R Markdown. The analysis reports may be enriched with website features (HTML/CSS) and published as HTML, PDF or Word documents. Moreover, with R packages such as bookdown, bookdownplus, blogdown and flexdashboard, documents can be formatted as websites, digital papers and books and data dashboards. Other useful packages include rmarkdown, knitr, DT, kableExtra and ggplot2. Further background: presentation by Michael Frank, slides by Ed Berry.

As well as facilitating the reproducibility of analyses and results to third parties, R Markdown is helpful during the creation of a report. In particular, it reduces the chances of errors and the number of repetitive tasks. For instance, any part of the data can be inputted in the text directly from the source, rather than manually copying it (e.g., `r mean(dat[dat$location=='Havana', 'measure'])` (expand). Thus, if and when the analysis needs to be changed or updated, the report can be automatically updated at the click of a button. In another area, the captions for figures and tables can be automatised using cross-reference labels (e.g., Table \@ref(tab:mtcars)). This secures the match between the text and the captions of figures and tables, and it automatically updates the numbering whenever and wherever a new figure or table is introduced.

Image from bookdownplus package (https://bookdownplus.netlify.com/portfolio/).

Workshop 3: Introduction to data dashboards

Data dashboards are web applications used to visualise data in detail through tables and plots. They assist in explaining and accounting for our data processing and analysis. They don’t require any coding from the end user. While most dashboards and web applications present existing data, a few of them serve the purpose of creating or simulating new data (see example).

These all-reproducible dashboards are published as websites, and thus, they can include hyperlinks and downloadable files. Some of the R packages used are knitr, DT, kableExtra, reactable, ggplot2, plotly, rmarkdown, flexdashboard and shiny. The aim of this workshop is to practise creating different forms of dashboards—Flexdashboard and Shiny—the latter of which offers greater features, and to practise also with the hosting platforms fitting each type—such as personal websites, RPubs, Binder, Shinyapps and custom servers.

A great thing about dashboards is that they may be made very simple, but they can also be taken to the next level using some HTML, CSS or Javascript code (on top of the back-end code present in the R packages used), which is addressed in the next workshop.

Workshop 4: Binder environments and improving data dashboards

Binder

Binder is a tool to facilitate public access to software environments—for instance, by publishing an RStudio environment on the internet. Binder can also host Shiny apps. It is generously free for users. After looking at the nuts and bolts of a deployment, participants will be able to deploy their own Binder environments and check the result by the end of the workshop. For this purpose, it’s recommended to have data and R code ready, ideally in a GitHub repository.

Improving data dashboards

We will practise how to improve the functionality of dashboards using some HTML, CSS and Javascript code, which is the basis of websites.

<!-- Javascript function to enable a hovering tooltip -->
<script>
$(document).ready(function(){
$('[data-toggle="tooltip1"]').tooltip();
});
</script>

Trade-offs among dashboards

Next, we will practise with three dashboard types—Flexdashboard, Shiny and Flexdashboard-Shiny—and with the suitable hosting platforms. Firstly, the strength of Flexdashboard (example) is its basis on R Markdown, yielding an unmatched user interface (front-end). Secondly, the strength of Shiny (example) is the input reactivity (back-end) it offers, allowing users to download sections of data they select, in various formats. Last, Flexdashboard-Shiny (example) combines the best of both worlds.

★ Flexdashboard ★

★ ★ Shiny ★ ★

★ ★ ★ Flexdashboard-Shiny ★ ★ ★

Flexdashboard types are rendered as an HTML document—simple websites—, and can therefore be easily published on personal sites or RPubs. This is convenient because no special hosting is required. In contrast, Shiny and Flexdashboard-Shiny types offer greater features, but require Shiny servers. Fortunately, the shinyapps.io server is available for free, up to some usage limit. This server can host any of the three dashboards mentioned here. Another good option is presented by Binder environments, which can host the Shiny-type dashboards with no (explicit) limit. Yet, the Flexdashboard-Shiny type cannot be hosted in this server (as of January 2020, at least). Consequently, greater functionality may come at a cost for dashboards that have any considerable traffic, whereas dashboards with low traffic may do well on shinyapps.io. Knowing these trade-offs can help navigate usage limits, save on web hosting fees, and increase the availability of our dashboards online, as we can offer fall-back versions on different platforms, as in the example below:

… preferred-dashboard (in case of downtime, please visit this alternative)

Transforming dashboards into the different versions can be as easy as enabling or disabling some features, especially input reactivity. For instance, if we want to downgrade a Flexdashboard-Shiny to a Flexdashboard, to publish it outside of a Shiny server (see example), we must delete runtime:shiny from the header, and disable reactive features, as below.


``` r
# Number of words selected on sidebar
# reactive(cat(paste0('Words selected below: ', nrow(selected_props()))))
```

Free accounts and tips

Hosting sites have specific terms of use. For instance, shinyapps.io has a free starter license with limited use. Free apps can handle a large but limited amount of data, and up to five apps may be created. Beyond this, RStudio offers a wide range of subscriptions starting at $9/month.

Memory and traffic limits of the free shinyapps.io account can sometimes present problems when heavy data data sets are used, or there are many visits to the app. The memory overload issue is often flagged as Shiny cannot use on-disk bookmarking, whereas excessive traffic may see the app not loading. Fortunately, usage limits need not always require a paid subscription or a custom server, thanks to the following workarounds:

develop app locally as far as possible, and only deploy to shinyapps.io only at the last stage;
prune data set, leaving only the necessary data;
if necessary, unlink data by splitting it into different sets, reducing computational demands;
if necessary, use various apps (five are allowed in each free shinyapps.io account);
if necessary, link from the app to a PDF with visualisations requiring heavy, interlinked data. High-resolution plots can be rendered into a PDF document in a snap, using code such as below.

pdf('List of plots per page', width = 13, height = 5)
print(plot1)
print(plot2)
# ...
print(plot150)
dev.off()

Conveniently, all text in a PDF—even in plots—is indexed, so it can be searched [ Ctrl+f / Cmd+f / 🔍 ] (see example). Furthermore, you may also merge the rendered PDF with any other documents.

Prerequisites and suggestions for participation in each workshop

Necessary: laptop or computer with R and RStudio installed, or access to RStudio Cloud; familiarity with the content of the preceding workshops through the web links herein.

Suggested: having your own data and R code ready (on a Github repository if participating in Workshop 4); participation in some of the preceding workshops.

Datathons: creating reproducible documents and dashboards

In these coding meetups, participants collaborate to create reproducible documents or dashboards using the data and software they prefer (see examples). Since the work can be split across different people and sections, some nice products may be achieved within a session. Any programming languages may be used.

Data used: academic or non-academic data of your own or from open-access sources such as OSF, scientific journals, governments, international institutions, NGOs, etc.
- Inspired by the great Reprohacks, content suggestions are encouraged. That is, if you’d like to have a reproducible document or dashboard created for a certain, open-access data set, please let us know, and some participants may take it on. Suggestions may be posted as issues or emailed to p.bernabeu@lancaster.ac.uk.
Purposes
- collaborating to visualise data in novel ways using reproducible documents or interactive dashboards. For this purpose, participants sometimes draw on additional data to look at a bigger picture;
- reflecting on the process by reviewing the techniques applied and challenges encountered.
Output: A key aspect of datathons is the creation of output. Documents and dashboards are (co-)authored by the participants who work on them, who can then publish them on their websites, or on RPubs, Binder, Shinyapps or custom servers. Time constraints notwithstanding, a lot of this output may be very enticing for further development by the same participants, or even by other people if the code is shared online. Just like with data, an attribution licence can be attached to the code.

Prerequisites and suggestions for participation in datathons

Necessary: basic knowledge of reproducible documents or dashboards.

Suggested: familiarity with the development of reproducible documents or dashboards; an idea about the data you’d like to work with and the kind of document or dashboard you want to create.

Contact

Please submit any queries or requests by posting an issue or emailing p.bernabeu@lancaster.ac.uk.

Dutch modality exclusivity norms

Mon, 01 Jan 2018 00:00:00 +0000

How it works

graph TD A["Dutch modality
exclusivity norms data"] --> B["Flexdashboard front-end
(R Markdown)"] B --> C["Shiny back-end
(reactive selection and download)"] C --> D["Info tab: HTML and CSS text
plus rmarkdown output"] C --> E["Table tab: reactable
(colours, bar charts)"] C --> F["Plot tab: plotly
(PCA scatter, tooltips)"] B --> G["Static Flexdashboard-only version
on RPubs (Shiny removed)"]

This web application presents linguistic data over several tabs. The code combines the great front-end of Flexdashboard—based on R Markdown and yielding an unmatched user interface—, with the great back-end of Shiny—allowing users to download sections of data they select, in various formats.

A nice find was the 'reactable' package, which implements Javascript under the hood to allow the use of colours, bar charts, etc.

Auditory = colDef(header = with_tooltip('Auditory Rating',
'Mean rating of each word on the auditory modality across participants.'),
cell = function(value) {
width <- paste0(value / max(table_data$Auditory) * 100, "%")
value = sprintf("%.2f", round(value,2)) # Round to two digits, keeping trailing zeros
bar_chart(value, width = width, fill = '#ff3030')
},
align = 'left'),

One of the hardest nuts to crack was allowing the full functionality of tables—i.e., scaling to screen, frozen header, and vertical and horizontal scrolling—whilst having tweaked the vertical/horizontal orientation of the dashboard sections. Initial clashes were resolved by adjusting the section's CSS styles

Table {#table style="background-color:#FCFCFC;"}
=======================================================================
Inputs {.sidebar style='position:fixed; padding-top: 65px; padding-bottom:30px;'}
-----------------------------------------------------------------------

and by also adjusting the reactable settings.

renderReactable({
reactable(selected_words(),
defaultSorted = list(cat = 'desc', word = 'asc'),
defaultColDef = colDef(footerStyle = list(fontWeight = "bold")),
height = 840, striped = TRUE, pagination = FALSE, highlight = TRUE,

A nice feature, especially suited to Flexdashboard, was the use of different formats across tabs. Whereas the Info tab presents long text using HTML and CSS styling, along with rmarkdown code output, the other tabs rely more strongly on Javascript features, enabled by R packages such as ‘shiny’ and sweetalert (e.g., allowing modal dialogs—pop-ups), reactable and plotly (e.g., allowing information opened by hovering—tooltips).

```{r}
# reactive for the word bar
highlighted_properties = reactive(input$highlighted_properties)
renderPlotly({
ggplotly(
ggplot( selected_props(), aes(RC1, RC2, label = as.character(word), color = main,
# Html tags below used for format. Decimals rounded to two.
text = paste0(' ', '<span style="padding-top:3px; padding-bottom:3px; font-size:2.2em; color:#EEEEEE">', capitalize(word), '</span> ', '<br>',
'</b><br><span style="color:#EEEEEE"> Dominant modality: </span><b style="color:#EEEEEE">', main, ' ',
' ', '</b><br><span style="color:#EEEEEE"> Modality exclusivity: </span><b style="color:#EEEEEE">', sprintf("%.2f", round(Exclusivity, 2)), '% ',
'</b><br><span style="color:#EEEEEE"> Perceptual strength: </span><b style="color:#EEEEEE">', sprintf("%.2f", round(Perceptualstrength, 2)),
'</b><br><span style="color:#EEEEEE"> Auditory rating: </span><b style="color:#EEEEEE">', sprintf("%.2f", round(Auditory, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Haptic rating: </span><b style="color:#EEEEEE">', sprintf("%.2f", round(Haptic, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Visual rating: </span><b style="color:#EEEEEE">', sprintf("%.2f", round(Visual, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Concreteness (Brysbaert et al., 2014): </span><b style="color:#EEEEEE">',
sprintf("%.2f", round(concrete_Brysbaertetal2014, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Number of letters: </span><b style="color:#EEEEEE">', letters, ' ',
'</b><br><span style="color:#EEEEEE"> Number of phonemes (DutchPOND): </span><b style="color:#EEEEEE">',
round(phonemes_DUTCHPOND, 2), ' ',
'</b><br><span style="color:#EEEEEE"> Contextual diversity (lg10CD SUBTLEX-NL): </span><b style="color:#EEEEEE">',
sprintf("%.2f", round(freq_lg10CD_SUBTLEXNL, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Word frequency (lg10WF SUBTLEX-NL): </span><b style="color:#EEEEEE">',
sprintf("%.2f", round(freq_lg10WF_SUBTLEXNL, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Lemma frequency (CELEX): </span><b style="color:#EEEEEE">',
sprintf("%.2f", round(freq_CELEX_lem, 2)), ' ',
'</b><br><span style="color:#EEEEEE"> Phonological neighbourhood size (DutchPOND): </span><b style="color:#EEEEEE">',
round(phon_neighbours_DUTCHPOND, 2), ' ',
'</b><br><span style="color:#EEEEEE"> Orthographic neighbourhood size (DutchPOND): </span><b style="color:#EEEEEE">',
round(orth_neighbours_DUTCHPOND, 2), ' ',
'</b><br><span style="color:#EEEEEE"> Age of acquisition (Brysbaert et al., 2014): </span><b style="color:#EEEEEE">',
sprintf("%.2f", round(AoA_Brysbaertetal2014, 2)), ' ', '<br> '
) ) ) +
geom_text(size = ifelse(selected_props()$word %in% highlighted_properties(), 7,
ifelse(is.null(highlighted_properties()), 3, 2.8)),
fontface = ifelse(selected_props()$word %in% highlighted_properties(), 'bold', 'plain')) +
geom_point(alpha = 0) + # This geom_point helps to colour the tooltip according to the dominant modality
scale_colour_manual(values = colours, drop = FALSE) + theme_bw() + ggtitle('Property words') +
labs(x = 'Varimax-rotated Principal Component 1', y = 'Varimax-rotated Principal Component 2') +
guides(color = guide_legend(title = 'Main<br>modality')) +
theme( plot.background = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), panel.border = element_blank(),
axis.line = element_line(color = 'black'), plot.title = element_text(size = 14, hjust = .5),
axis.title.x = element_text(colour = 'black', size = 12, margin = margin(15,15,0,15)),
axis.title.y = element_text(colour = 'black', size = 12, margin = margin(0,15,15,5)),
axis.text.x = element_text(size = 8), axis.text.y = element_text(size = 8),
legend.background = element_rect(size = 2), legend.position = 'none',
legend.title = element_blank(),
legend.text = element_text(colour = colours, size = 13) ),
tooltip = 'text'
)
})
# For download, save plot without the interactive 'plotly' part
properties_png = reactive({ ggplot(selected_props(), aes(RC1, RC2, color = main, label = as.character(word))) +
geom_text(show.legend = FALSE, size = ifelse(selected_props()$word %in% highlighted_properties(), 7,
ifelse(is.null(highlighted_properties()), 3, 2.8)),
fontface = ifelse(selected_props()$word %in% highlighted_properties(), 'bold', 'plain')) +
geom_point(alpha = 0) + scale_colour_manual(values = colours, drop = FALSE) + theme_bw() +
guides(color = guide_legend(title = 'Main<br>modality', override.aes = list(size = 7, alpha = 1))) +
ggtitle( paste0('Properties', ' (showing ', nrow(selected_props()), ' out of ', nrow(props), ')') ) +
labs(x = 'Varimax-rotated Principal Component 1', y = 'Varimax-rotated Principal Component 2') +
theme( plot.background = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), panel.border = element_blank(),
axis.line = element_line(color = 'black'), plot.title = element_text(size = 17, hjust = .5, margin = margin(3,3,7,3)),
axis.title.x = element_text(colour = 'black', size = 12, margin = margin(10,10,2,10)),
axis.title.y = element_text(colour = 'black', size = 12, margin = margin(10,10,10,5)),
axis.text.x = element_text(size = 8), axis.text.y = element_text(size = 8),
legend.background = element_rect(size = 2), legend.position = 'right',
legend.title = element_blank(), legend.text = element_text(size = 15))
})
```

The only instance in which I drew on JavaScript code outside R packages was to enable tooltips beyond the packages’ limits—for instance, in the sidebar. This JavaScript feature is created at the top of the script, in the head area.

<!-- Javascript function to enable a hovering tooltip -->
<script>
$(document).ready(function(){
$('[data-toggle="tooltip1"]').tooltip();
});
</script>

In the sidebar, I added a reactive mean for each variable, complementing the range selector.

reactive(cat(paste0('Mean = ',
sprintf("%.2f", round(mean(selected_words()$Exclusivity),2)))))

Static version published on RPubs

A reduced, static version was also created to increase the availability of the content. Removing some reactivity features allows the dashboard to be published as a standard website (i.e., on a personal website, on RPubs, etc.), without the need for a back-end Shiny server. Note that this type of website is dubbed 'static', but it can retain multiple interactive features thanks to Javascript-based tools under the hood, allowed by R packages such as leaflet for maps, DT for tables, plotly for plots, etc.

To create the Flexdashboard-only version departing from the Flexdashboard-Shiny version, I deleted runtime: shiny from the YAML header, and disabled Shiny reactive inputs and objects, as below.

```{r}
# Number of words selected on sidebar
# reactive(cat(paste0('Words selected below: ', nrow(selected_props()))))
```

Reference

Bernabeu, P. (2018). Dutch modality exclusivity norms for 336 properties and 411 concepts [Web application]. Retrieved from https://pablobernabeu.shinyapps.io/Dutch-Modality-Exclusivity-Norms

Naive principal component analysis in R

Mon, 01 Jan 2018 00:00:00 +0000

Principal Component Analysis (PCA) is a technique used to find the core components that underlie different variables. It comes in very useful whenever doubts arise about the true origin of three or more variables. There are two main methods for performing a PCA: naive or less naive. In the naive method, you first check some conditions in your data which will determine the essentials of the analysis. In the less-naive method, you set those yourself based on whatever prior information or purposes you had. The latter method is appropriate when you already have enough information about the intercorrelations, or when you are required to select a specific number of components. I will tackle the naive method, mainly by following the guidelines in Field, Miles, and Field (2012), with updated code where necessary. A manual by Charles M. Friel (Sam Houston State University) was also useful.

The ‘naive’ approach is characterized by a first stage that checks whether the PCA should actually be performed with your current variables, or if some should be removed. The variables that are accepted are taken to a second stage which identifies the number of principal components that seem to underlie your set of variables.

STAGE 1. Determine whether PCA is appropriate at all, considering the variables

Variables should be inter-correlated enough but not too much. Field et al. (2012) provide some thresholds, suggesting that no variable should have many correlations below .30, or any correlation at all above .90. Thus, in the example here, variable Q06 should probably be excluded from the PCA.
Bartlett’s test, on the nature of the intercorrelations, should be significant. Significance suggests that the variables are not an ‘identity matrix’ in which correlations are a sampling error.
KMO (Kaiser-Meyer-Olkin), a measure of sampling adequacy based on common variance (so similar purpose as Bartlett’s). As Field et al. review, ‘values between .5 and .7 are mediocre, values between .7 and .8 are good, values between .8 and .9 are great and values above .9 are superb’ (p. 761). There’s a general score as well as one per variable. The general one will often be good, whereas the individual scores may more likely fail. Any variable with a score below .5 should probably be removed, and the test should be run again.

Determinant: A formula about multicollinearity. The result should preferably fall below .00001. Note that some of these tests are run on the dataframe and others on a correlation matrix of the data, as distinguished below.

# Necessary libraries
library(ltm)
library(lattice)
library(psych)
library(car)
library(pastecs)
library(scales)
library(ggplot2)
library(arules)
library(plyr)
library(Rmisc)
library(GPArotation)
library(gdata)
library(MASS)
library(qpcR)
library(dplyr)
library(gtools)
library(Hmisc)
# Select variables of interest for the PCA
dataset = mydata[, c('select_var1', 'select_var1',
'select_var2', 'select_var3', 'select_var4',
'select_var5', 'select_var6', 'select_var7')]
# Create matrix: some tests will require it
data_matrix = cor(dataset, use = 'complete.obs')
# See intercorrelations
round(data_matrix, 2)
# Bartlett's
cortest.bartlett(dataset)
# KMO (Kaiser-Meyer-Olkin)
KMO(data_matrix)
# Determinant
det(data_matrix)

STAGE 2. Identify number of components (aka factors)

In this stage, principal components (formally called ‘factors’ at this stage) are identified among the set of variables.

The identification is done through a basic, ‘unrotated’ PCA. The number of components set a priori must equal the number of variables that are being tested.

# Start off with unrotated PCA
pc1 = psych::principal(dataset, nfactors = length(dataset), rotate="none")
pc1

Below is an example result:

 ## Principal Components Analysis
## Call: psych::principal(r = eng_prop, nfactors = 3, rotate = "none")
## Standardized loadings (pattern matrix) based upon correlation matrix
## PC1 PC2 PC3 h2 u2 com
## Aud_eng -0.89 0.13 0.44 1 -2.2e-16 1.5
## Hap_eng 0.64 0.75 0.15 1 1.1e-16 2.0
## Vis_eng 0.81 -0.46 0.36 1 -4.4e-16 2.0
##
## PC1 PC2 PC3
## SS loadings 1.87 0.79 0.34
## Proportion Var 0.62 0.26 0.11
## Cumulative Var 0.62 0.89 1.00
## Proportion Explained 0.62 0.26 0.11
## Cumulative Proportion 0.62 0.89 1.00
##
## Mean item complexity = 1.9
## Test of the hypothesis that 3 components are sufficient.
##
## The root mean square of the residuals (RMSR) is 0
## with the empirical chi square 0 with prob < NA
##
## Fit based upon off diagonal values = 1

Among the columns, there are first the correlations between variables and components, followed by a column (h2) with the ‘communalities’. If less factors than variables had been selected, communality values would be below 1. Then there is the uniqueness column (u2): uniqueness is equal to 1 minus the communality. Next is ‘com’, which reflects the complexity with which a variable relates to the principal components. Those components are precisely found below. The first row contains the sums of squared loadings, or eigenvalues, namely, the total variance explained by each linear component. This value corresponds to the number of units explained out of all possible factors (which were three in the above example). The rows below all cut from the same cloth. Proportion var = variance explained over a total of 1. This is the result of dividing the eigenvalue by the number of components. Multiply by 100 and you get the percentage of total variance explained, which becomes useful. In the example, 99% of the variance has been explained. Aside from the meddling maths, we should actually expect 100% there because the number of factors equaled the number of variables. Cumulative var: variance added consecutively up to the last component. Proportion explained: variance explained over what has actually been explained (only when variables = factors is this the same as Proportion var). Cumulative proportion: the actually explained variance added consecutively up to the last component (Field et al., 2012).

According to Field et al. (2012), two criteria can be used to determine the number of components that should be carried forward to the next stage:

SS (sum of squares) loadings, with two possible cut-off points. On the one hand, following Kaiser’s threshold, we should select components with SS loadings > 1. In the example result shown above, only PC1 meets this criterion. A more lenient alternative is Kaiser’s threshold, whereby SS loadings > .7 are accepted. In the example result above, PC2 meet this criterion too.
Scree plot: retain as many components as the number of points after the point of inflection. To create a scree plot, call:
```
plot(pc1$values, type = 'b')
```

Imagine a straight line from the first point on the right. Once this line bends considerably, count the points after the bend and up to the last point on the left. The number of points is the number of components to select. The example here is probably the most complicated (two components were finally chosen), but often it’s easier see examples.

Based on either or both criteria, select the definitive number of components.

STAGE 3. Run definitive PCA

Run a very similar command as you did before, but now with a more advanced method. The first PCA, a heuristic one, worked essentially on the inter-correlations. The definitive PCA, in contrast, will implement a prior shuffling known as ‘rotation’, to ensure that the result is robust enough (just like cards are shuffled). Explained variance is captured better this way. The go-to rotation method is the orthogonal, or ‘varimax’ (though others may be considered too).

# Now with varimax rotation, Kaiser-normalized by default:
pc2 = psych::principal(dataset, nfactors=2, rotate = "varimax",
scores = TRUE)
pc2
pc2$loadings
# Sanity check
pc2$residual
pc2$fit
pc2$communality

According to Field et al. (2012), we would want:

Less than half of residuals with absolute values > 0.05
Model fit > .9
All communalities > .7

If any of this fails, consider changing the number of factors. Next, the rotated components that have been ‘extracted’ from the core of the set of variables can be added to the dataset. This would enable the use of these components as new variables that might prove powerful and useful (as in this research).

dataset = cbind(dataset, pc2$scores)
summary(dataset$RC1, dataset$RC2)

STAGE 4. Determine ascription of each variable to components

Check the main summary by just calling pc2, and see how each variable correlates with the rotated components. This is essential because it reveals how variables load on each component, or in other words, to which component a variable belongs. For instance, the table shown here belongs to a study about the meaning of words (Bernabeu, 2018). These results suggest that the visual and haptic modalities of words are quite related, whereas the auditory modality is relatively unique. When the analysis works out well, a cut-off point of r = .8 may be applied for considering a variable as part of a component.

STAGE 5. Enjoy the plot

The plot is perhaps the coolest part about PCA. It really makes an awesome illustration of the power of data analysis.

ggplot(eng_props,
aes(RC1, RC2, label = as.character(main_eng))) + stat_density2d (color = "gray87") +
geom_text(size = ifelse(eng_props$word_eng %in% w_set, 12, 7),
fontface = ifelse(eng_props$word_eng %in% w_set, 'bold', 'plain')) +
geom_point(data=eng_props[eng_props$word_eng %in% w_set,], pch=21, fill=NA, size=14, stroke=2, alpha=.6) +
labs(subtitle='(Data from Lynott & Connell, 2009)', x = "Varimax-rotated Principal Component 1",
y = "Varimax-rotated Principal Component 2") + theme_bw() +
theme( plot.background = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), panel.border = element_blank(),
axis.line = element_line(color = 'black'),
axis.title.x = element_text(colour = 'black', size = 23, margin=margin(15,15,15,15)),
axis.title.y = element_text(colour = 'black', size = 23, margin=margin(15,15,15,15)),
axis.text.x = element_text(size=16), axis.text.y = element_text(size=16),
plot.title = element_text(hjust = 0.5, size = 32, face = "bold", margin=margin(15,15,15,15)),
plot.subtitle = element_text(hjust = 0.5, size = 20, margin=margin(2,15,15,15)) ) +
geom_label_repel(data = eng_props[eng_props$word_eng %in% w_set,], aes(label = word_eng), size = 8,
alpha = 0.77, color = 'black', box.padding = 1.5 )

Below is an example combining PCA plots with code similar to the above. These plots illustrate something further with regard to the relationships among modalities. In property words, the different modalities spread out more clearly than they do in concept words. This makes sense because in language, properties define concepts (Bernabeu, 2018).

An example of these analyses is available in available in this RStudio environment, in the norms.R script.

References

Bernabeu, P. (2018). Dutch modality exclusivity norms for 336 properties and 411 concepts. PsyArXiv. https://doi.org/10.31234/osf.io/s2c5h

Field, A. P., Miles, J., & Field, Z. (2012). Discovering Statistics Using R. London, UK: Sage.

Modality switch effects emerge early and increase throughout conceptual processing

Sun, 01 Jan 2017 00:00:00 +0000

How it works

graph TD A["EEG-ERP data from word
comprehension experiment"] --> B["ERP plots spanning 800 ms
of word processing"] B --> C["Groups of participants"] C --> D["Individual participants"] D --> E["Brain areas"] E --> F["Electrodes"] B --> G["Download HD plots, view
95% confidence intervals"]

Content

The data are from a psychology experiment on the comprehension of words, in which electroencephalographic (EEG) responses were measured. The data are presented in plots spanning 800 milliseconds (the duration of word processing). The aim of this Shiny app is to facilitate the exploration of the data by researchers and the public. Users can delve into the different sections of the data. In a hierarchical order, these sections are groups of participants, individual participants, brain areas, and electrodes.

Shiny apps in science

By creating this app, I tried to reach beyond the scope of current open science, which is often confined to files shared on data repositories. I believe that Shiny apps will become general practice in science within a few years ( see blog post or slides for more information).

Technical details

I used tabs on the upper area of the application page to avoid cramming the side bar with widgets. I adjusted the appearance of these tabs, and by means of 'reactivity' conditions, modified the inputs in the side bar depending on the active tab.

mainPanel(
tags$style(HTML('
.tabbable > .nav > li > a {background-color:white; color:#3E454E}
.tabbable > .nav > li > a:hover {background-color:#002555; color:white}
.tabbable > .nav > li[class=active] > a {background-color:#ECF4FF; color:black}
.tabbable > .nav > li[class=active] > a:hover {background-color:#E7F1FF; color:black}
')),
tabsetPanel(id='tabvals',
tabPanel(value=1, h4(strong('Group & Electrode')), br(), plotOutput('plot_GroupAndElectrode'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/2tpxn/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.1', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=2, h4(strong('Participant & Area')), br(), plotOutput('plot_ParticipantAndLocation'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/86ch9/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.2', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=3, h4(strong('Participant & Electrode')), br(), plotOutput('plot_ParticipantAndElectrode'),
br(), downloadButton('downloadPlot.3', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=4, h4(strong('OLD Group & Electrode')), br(), plotOutput('plot_OLDGroupAndElectrode'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/dvs2z/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.4', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000))
),

The data set was fairly large, considering the fact that it's hosted with the free plan. In order to lighten the processing, I split the data into various files, reducing the total size. Furthermore, I outsourced a particularly heavy set of plots (those with Confidence Intervals) to PDF files, to which I linked in the app.

h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/dvs2z/',
target='_blank'), style='text-decoration: underline;'),

I also used web links to the published paper and raw data, as well as to the server and ui scripts. These files, along with the data, are publicly available in this repository; they may be accessed within the "Files" section, by opening the folders "ERPs" -> "Analyses of ERPs averaged across trials" -> "Shiny app".

Another feature I added was the download button.

# From server.R script
spec_title = paste0('ERP waveforms for ', input$var.Group, ' Group, Electrode ', input$var.Electrodes.1, ' (negative values upward; time windows displayed)')
plot_GroupAndElectrode = ggplot(df2, aes(x=time, y=-microvolts, color=condition)) +
geom_rect(xmin=160, xmax=216, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=270, xmax=370, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=350, xmax=550, ymin=8, ymax=-7.5, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=500, xmax=750, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_line(size=1, alpha = 1) + scale_linetype_manual(values=colours) +
scale_y_continuous(limits=c(-8.38, 8.3), breaks=seq(-8,8,by=1), expand = c(0,0.1)) +
scale_x_continuous(limits=c(-208,808),breaks=seq(-200,800,by=100), expand = c(0.005,0), labels= c('-200','-100 ms','0','100 ms','200','300 ms','400','500 ms','600','700 ms','800')) +
ggtitle(spec_title) + theme_bw() + geom_vline(xintercept=0) +
annotate(geom='segment', y=seq(-8,8,1), yend=seq(-8,8,1), x=-4, xend=8, color='black') +
annotate(geom='segment', y=-8.2, yend=-8.38, x=seq(-200,800,100), xend=seq(-200,800,100), color='black') +
geom_segment(x = -200, y = 0, xend = 800, yend = 0, size=0.5, color='black') +
theme(legend.position = c(0.100, 0.150), legend.background = element_rect(fill='#EEEEEE', size=0),
axis.title=element_blank(), legend.key.width = unit(1.2,'cm'), legend.text=element_text(size=17),
legend.title = element_text(size=17, face='bold'), plot.title= element_text(size=20, hjust = 0.5, vjust=2),
axis.text.y = element_blank(), axis.text.x = element_text(size = 14, vjust= 2.12, face='bold', color = 'grey32', family='sans'),
axis.ticks=element_blank(), panel.border = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), plot.margin = unit(c(0.1,0.1,0,0), 'cm')) +
annotate('segment', x=160, xend=216, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=270, xend=370, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=350, xend=550, y=-7.5, yend=-7.5, colour = 'grey75', size = 1.5) +
annotate('segment', x=500, xend=750, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
scale_fill_manual(name = 'Context / Target trial', values=colours) +
scale_color_manual(name = 'Context / Target trial', values=colours) +
guides(linetype=guide_legend(override.aes = list(size=1.2))) +
guides(color=guide_legend(override.aes = list(size=2.5))) +
# Print y axis labels within plot area:
annotate('text', label = expression(bold('\u2013' * '3 ' * '\u03bc' * 'V')), x = -29, y = 3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('+3 ' * '\u03bc' * 'V')), x = -29, y = -3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('\u2013' * '6 ' * '\u03bc' * 'V')), x = -29, y = 6, size = 4.5, color = 'grey32', family='sans')
print(plot_GroupAndElectrode)
output$downloadPlot.1 <- downloadHandler(
filename <- function(file){
paste0(input$var.Group, ' group, electrode ', input$var.Electrodes.1, ', ', Sys.Date(), '.png')},
content <- function(file){
png(file, units='in', width=13, height=5, res=900)
print(plot_GroupAndElectrode)
dev.off()},
contentType = 'image/png')
} )

# From ui.R script
downloadButton('downloadPlot.1', 'Download HD plot')

Rising to the challenge

My experience with R Shiny has been so good I've been sharing it. Yet, on my first crawling days, I spent an eternity stuck with this elephant in my room: "μ". This μ letter (micro-souvenir from hell, as I later knew it), was part of the labels of my plots. All I knew was that I could not deploy the app online, even while I could perfectly launch it locally in my laptop. So, I wondered what use it was to deploy locally if I couldn't publish the app?! Eventually, I read about UTF-8 encoding in one forum. Bless them forums. All I had to do was use "Âμ" instead of the single "μ". A better option I found later was: expression("\u03bc").

Beyond encoding issues, I had a tough time embedding images. You know, the 'www' folder... To be honest, I still haven't handled the 'www' way--but where there's a will, there's a way. I managed to include my images by uploading them to a website and then entering their URL in "img(src", avoiding the use of folder paths.

img(src="https://preview.ibb.co/n7qiYR/EEG_montage.png 1", height=500, width=1000)

Long after I had built the app, I added another image--the favicon (the little icon on the browser tab).

tags$head(tags$link(rel="shortcut icon", href="https://image.ibb.co/fXUwzb/favic.png")), # web favicon

Reference

Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). Modality switch effects emerge early and increase throughout conceptual processing: Evidence from ERPs [Web application]. Retrieved from https://pablobernabeu.shinyapps.io/ERP-waveform-visualization_CMS-experiment

The case for data dashboards: First steps in R Shiny

Sun, 01 Jan 2017 00:00:00 +0000

Dashboards for data visualisation, such as R Shiny and Tableau, allow the interactive exploration of data by means of drop-down lists and checkboxes, with no coding required from the final users. These web applications can be useful for both the data analyst and the public at large.

Visualisation apps run on internet browsers. This allows for three options: private viewing (useful during analysis), selective sharing (used within work groups), or internet publication. Among the available platforms, R Shiny and Tableau stand out due to being relatively accessible to new users. Apps serve a broad variety of purposes (see this gallery and this one). In science and beyond, these apps allow us to go the extra mile in sharing data. Alongside files and code shared in repositories, we can present the data in a website, in the form of plots or tables. This facilitates the public exploration of each section of the data (groups, participants, trials…) to anyone interested, and allows researchers to account for their proceeding in the analysis.

Publishers and journals highly encourage authors to make the most of their data by facilitating its easy exploration by the readership–even though they don’t normally offer the possibility of hosting any web applications yet.

Apps can also prove valuable to those analysing the data. For instance, my app helped me to identify the extent of noise in a section of the data. Instead of running through a heavy score of code, the drop-down lists of the app let me seamlessly surf through the different sections.

At a certain point, I found a data section that was consistently noisier than the rest, and eventually I had to discard it from further statistical analyses. Yet, instead of removing that from the app, I maintained it with a note attached. This particular trait in the data was rather salient.

Beyond such a salient feature in the data, a visualisation app may also help to spot subtler patterns such as third variables or individual differences.

There are several platforms for creating apps (e.g., Tableau, D3.js, and R Shiny). I focus on R Shiny here for three reasons: it is affordable to use, fairly accessible to new users, and well suited for science as it is based on the R language (see for instance this article).

How to Shiny

Shiny apps draw on any standard R code that you may already have. This is most commonly plots or tables, but other stuff such as images or Markdown texts are valid too. This is a nice thing to keep in mind when having to create a new app. Part of the job may already be done! The app is distributed among five different areas.

Data file(s): These are any data files you’re using (e.g., with csv or rds extensions).

1a. `server.R` script

The server script contains the central processes: plots, tables, etc. Code that existed independently of the app app may be brought into this script by slightly adapting it. At the top, call the shiny library and any others used (e.g., ‘ggplot2’), and also read in the data. The snippet below shows the beginning of an example server.R script.


# server
library(shiny)
library(ggplot2)
EEG.ParticipantAndElectrode = readRDS('EEG.ParticipantAndElectrode.rds')
EEG.ParticipantAndBrainArea = readRDS('EEG.ParticipantAndBrainArea.rds')
EEG.GroupAndElectrode = readRDS('EEG.GroupAndElectrode.rds')
EEG.OLDGroupAndElectrode = readRDS('EEG.OLDGroupAndElectrode.rds')
server =
shinyServer(
function(input, output) {
# plot_GroupAndElectrode:
output$plot_GroupAndElectrode <- renderPlot({
dfelectrode <- aggregate(microvolts ~ electrode*time*condition,
EEG.GroupAndElectrode[EEG.GroupAndElectrode$RT.based_Groups==input$var.Group,], mean)
df2 <- subset(dfelectrode, electrode == input$var.Electrodes.1)
df2$condition= as.factor(df2$condition)
df2$condition <- gsub('visual2visual', ' Visual / Visual', df2$condition)
df2$condition <- gsub('haptic2visual', ' Haptic / Visual', df2$condition)
df2$condition <- gsub('auditory2visual', ' Auditory / Visual', df2$condition)
df2$time <- as.integer(as.character(df2$time))
colours <- c('firebrick1', 'dodgerblue', 'forestgreen')
# green:visual2visual, blue:haptic2visual, red:auditory2visual
spec_title = paste0('ERP waveforms for ', input$var.Group, ' Group, Electrode ', input$var.Electrodes.1, ' (negative values upward; time windows displayed)')
plot_GroupAndElectrode = ggplot(df2, aes(x=time, y=-microvolts, color=condition)) +
geom_rect(xmin=160, xmax=216, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=270, xmax=370, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=350, xmax=550, ymin=8, ymax=-7.5, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=500, xmax=750, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_line(size=1, alpha = 1) + scale_linetype_manual(values=colours) +
scale_y_continuous(limits=c(-8.38, 8.3), breaks=seq(-8,8,by=1), expand = c(0,0.1)) +
scale_x_continuous(limits=c(-208,808),breaks=seq(-200,800,by=100), expand = c(0.005,0), labels= c('-200','-100 ms','0','100 ms','200','300 ms','400','500 ms','600','700 ms','800')) +
ggtitle(spec_title) + theme_bw() + geom_vline(xintercept=0) +
annotate(geom='segment', y=seq(-8,8,1), yend=seq(-8,8,1), x=-4, xend=8, color='black') +
annotate(geom='segment', y=-8.2, yend=-8.38, x=seq(-200,800,100), xend=seq(-200,800,100), color='black') +
geom_segment(x = -200, y = 0, xend = 800, yend = 0, size=0.5, color='black') +
theme(legend.position = c(0.100, 0.150), legend.background = element_rect(fill='#EEEEEE', size=0),
axis.title=element_blank(), legend.key.width = unit(1.2,'cm'), legend.text=element_text(size=17),
legend.title = element_text(size=17, face='bold'), plot.title= element_text(size=20, hjust = 0.5, vjust=2),
axis.text.y = element_blank(), axis.text.x = element_text(size = 14, vjust= 2.12, face='bold', color = 'grey32', family='sans'),
axis.ticks=element_blank(), panel.border = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), plot.margin = unit(c(0.1,0.1,0,0), 'cm')) +
annotate('segment', x=160, xend=216, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=270, xend=370, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=350, xend=550, y=-7.5, yend=-7.5, colour = 'grey75', size = 1.5) +
annotate('segment', x=500, xend=750, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
scale_fill_manual(name = 'Context / Target trial', values=colours) +
scale_color_manual(name = 'Context / Target trial', values=colours) +
guides(linetype=guide_legend(override.aes = list(size=1.2))) +
guides(color=guide_legend(override.aes = list(size=2.5))) +
# Print y axis labels within plot area:
annotate('text', label = expression(bold('\u2013' * '3 ' * '\u03bc' * 'V')), x = -29, y = 3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('+3 ' * '\u03bc' * 'V')), x = -29, y = -3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('\u2013' * '6 ' * '\u03bc' * 'V')), x = -29, y = 6, size = 4.5, color = 'grey32', family='sans')
print(plot_GroupAndElectrode)
output$downloadPlot.1 <- downloadHandler(
filename <- function(file){
paste0(input$var.Group, ' group, electrode ', input$var.Electrodes.1, ', ', Sys.Date(), '.png')},
content <- function(file){
png(file, units='in', width=13, height=5, res=900)
print(plot_GroupAndElectrode)
dev.off()},
contentType = 'image/png')
} )
# ...

— Whole script

1b. `ui.R` script

The ui script defines the user interface. For instance, a factor column in the data that has multiple categories may be neatly displayed with a drop-down list on the side bar of the website. The interface may present a central plot before by a legend key below. The snippet below shows the beginning of an example ui.R script.


# UI
library(shiny)
library(ggplot2)
EEG.GroupAndElectrode = readRDS('EEG.GroupAndElectrode.rds')
EEG.ParticipantAndBrainArea = readRDS('EEG.ParticipantAndBrainArea.rds')
EEG.ParticipantAndElectrode = readRDS('EEG.ParticipantAndElectrode.rds')
EEG.OLDGroupAndElectrode = readRDS('EEG.OLDGroupAndElectrode.rds')
ui =
shinyUI(
fluidPage(
tags$head(tags$link(rel='shortcut icon', href='https://image.ibb.co/fXUwzb/favic.png')), # web favicon
tags$meta(charset='UTF-8'),
tags$meta(name='description', content='This R Shiny visualisation dashboard presents data from a psycholinguistic ERP experiment (Bernabeu et al., 2017).'),
tags$meta(name='keywords', content='R, Shiny, ggplot2, visualisation, data, psycholinguistics, conceptual processing, modality switch, embodied cognition'),
tags$meta(name='viewport', content='width=device-width, initial-scale=1.0'),
titlePanel(h3(strong('Waveforms in detail from an ERP experiment on the Conceptual Modality Switch'), a('(Bernabeu et al., 2017)',
href='https://figshare.com/articles/EEG_study_on_conceptual_modality-switching_Bernabeu_et_al_in_prep_/4210863', target='_blank',
style = 'color:#3E454E; text-decoration:underline; font-weight:normal'), align = 'center', style = 'color:black'),
windowTitle = 'Visualization of ERP waveforms from experiment on Conceptual Modality Switch (Bernabeu et al., 2017)'),
sidebarLayout(
sidebarPanel(width = 2,
# Condition 1 for reactivity between tabs and sidebars
conditionalPanel(
condition = 'input.tabvals == 1',
h5(a(strong('See paper, statistics, all data.'), 'Plots by group and brain area shown in paper.',
href='https://figshare.com/articles/EEG_study_on_conceptual_modality-switching_Bernabeu_et_al_in_prep_/4210863',
target='_blank'), align = 'center'),
br(),
selectInput('var.Group', label = 'Group', choices = list('Quick','Slow'), selected = 'Quick'),
h6('Quick G.: 23 participants'),
h6('Slow G.: 23 participants'),
br(),
selectInput('var.Electrodes.1', label = h5(strong('Electrode'), br(), '(see montage below)'),
choices = list('1','2','3','4','5','6','7','8','9','10',
'11','12','13','14','15','16','17','18','19','20','21',
'22','23','24','25','26','27','28','29','30','31','33',
'34','35','36','37','38','39','40','41','42','43','44',
'45','46','47','48','49','50','51','52','53','54','55',
'56','57','58','59','60'), selected = '30' ),
br(), br(),
h6('Source code:'),
h6(strong('- '), a('server.R', href='https://osf.io/uj8z4/', target='_blank', style = 'text-decoration: underline;')),
h6(strong('- '), a('ui.R', href='https://osf.io/8bwcx/', target='_blank', style = 'text-decoration: underline;')),
br(),
h6(a('CC-By 4.0 License', href='https://osf.io/97unm/', target='_blank'), align = 'center', style = 'text-decoration: underline;'),
br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(),
br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(), br(),
h5(a(strong('See paper, statistics, all data.'),
href='https://figshare.com/articles/EEG_study_on_conceptual_modality-switching_Bernabeu_et_al_in_prep_/4210863',
target='_blank'), align = 'center'),
br(), br(), br(), br(), br(), br(), br(), br()
),
# ...

— Whole script

2. Deployment and logs

This script contains the commands for deploying the app on- or off-line, and for checking the session logs in case of any errors.

3. Automatically created folder

When the app is first deployed on the internet, a subfolder is automatically created with the name ‘rsconnect’. This folder contains a text file which can be used to modify the URL and the title of the webpage.

Steps to create a Shiny app from scratch:

1. Tutorials (link). Being open-source software, excellent directions are available through a Google search.

The core ideas are:

As mentioned above, create a ui.R script for the code containing the user interface, and create a server.R script for the code containing the main content (your plots / tables, etc).

At the top of both ui.R and server.R scripts, enter the command library(shiny) and also load any other libraries you’re using (e.g., ggplot2).

Test your app by deploying it locally, before launching online. For this purpose, first save the ui and server parts independently, as in:


ui =
shinyUI(
fluidPage(
# ...

Then deploy locally by running:

shinyApp(ui, server)

Managing to run the app locally is a great first step before launching online (which may sometimes prove a bit trickier).

2. User token (link). Sign up and read in your private key—just to be done once in a computer.

3. Go for it. After locally testing and saving the two main scripts (ui.R and server.R), run deployApp() to launch the app online.

4. Bugs and session logs. Most often they won’t be bugs actually, but fancies, as it were. For instance, some special characters have to get even more special (technically, UTF-8 encoding). For a character such as ‘μ’, Shiny prefers ‘Âμ’, or better, the Unicode expression("\u03bc").

Cling to your logs by calling the line below, which you may keep at hand in your ‘Shiny deployer.R’ script.

showLogs(appPath = getwd(), appFile = NULL, appName = NULL, account = NULL,
entries = 50, streaming = FALSE)

At best, the log output will mention any typos and unaccepted characters, pointing to specific lines in your code.

It may take a couple of intense days to get a first Shiny app running. Although the usual rabbit holes do exist, years of Shiny have already yielded a sizeable body of free resources online (tutorials, blogs, vlogs). Moreover, there’s also the RStudio Community, and then StackOverflow etc., where you can post any needs/despair. Post your code, log, and explanation, and you’ll be rescued out in a couple of days. Long live those contributors.

It’s sometimes enough to upload a bare app, but you might then think it can look better.

5 (optional). Advance. Use tabs to combine multiple apps on one webpage, use different widgets, include a download option, etc. Tutorials like this one on Youtube can take you there, especially those that provide the code, as in the description of that video. Use those scripts as templates. For example, I made use of tabs on the top of the dashboard in order to keep the side bar from having too many widgets. The appearance of these tabs can be adjusted. More importantly, the inputs in the sidebar can be modified depending on the active tab, by means of ‘reactivity’ conditions.


mainPanel(
tags$style(HTML('
.tabbable > .nav > li > a {background-color:white; color:#3E454E}
.tabbable > .nav > li > a:hover {background-color:#002555; color:white}
.tabbable > .nav > li[class=active] > a {background-color:#ECF4FF; color:black}
.tabbable > .nav > li[class=active] > a:hover {background-color:#E7F1FF; color:black}
')),
tabsetPanel(id='tabvals',
tabPanel(value=1, h4(strong('Group & Electrode')), br(), plotOutput('plot_GroupAndElectrode'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/2tpxn/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.1', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=2, h4(strong('Participant & Area')), br(), plotOutput('plot_ParticipantAndLocation'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/86ch9/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.2', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=3, h4(strong('Participant & Electrode')), br(), plotOutput('plot_ParticipantAndElectrode'),
br(), downloadButton('downloadPlot.3', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000)),
tabPanel(value=4, h4(strong('OLD Group & Electrode')), br(), plotOutput('plot_OLDGroupAndElectrode'),
h5(a(strong('See plots with 95% Confidence Intervals'), href='https://osf.io/dvs2z/',
target='_blank'), style='text-decoration: underline;'),
downloadButton('downloadPlot.4', 'Download HD plot'), br(), br(),
# EEG montage
img(src='https://preview.ibb.co/n7qiYR/EEG_montage.png', height=500, width=1000))
),

The official Shiny gallery offers a great array of apps including their code (e.g., basic example). Another feature you may add is the option to download your plots, tables, data…


# In ui.R script
downloadButton('downloadPlot.1', 'Download HD plot')
#___________________________________________________
# In server.R script
spec_title = paste0('ERP waveforms for ', input$var.Group, ' Group, Electrode ', input$var.Electrodes.1, ' (negative values upward; time windows displayed)')
plot_GroupAndElectrode = ggplot(df2, aes(x=time, y=-microvolts, color=condition)) +
geom_rect(xmin=160, xmax=216, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=270, xmax=370, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=350, xmax=550, ymin=8, ymax=-7.5, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_rect(xmin=500, xmax=750, ymin=7.5, ymax=-8, color = 'grey75', fill='black', alpha=0, linetype='longdash') +
geom_line(size=1, alpha = 1) + scale_linetype_manual(values=colours) +
scale_y_continuous(limits=c(-8.38, 8.3), breaks=seq(-8,8,by=1), expand = c(0,0.1)) +
scale_x_continuous(limits=c(-208,808),breaks=seq(-200,800,by=100), expand = c(0.005,0), labels= c('-200','-100 ms','0','100 ms','200','300 ms','400','500 ms','600','700 ms','800')) +
ggtitle(spec_title) + theme_bw() + geom_vline(xintercept=0) +
annotate(geom='segment', y=seq(-8,8,1), yend=seq(-8,8,1), x=-4, xend=8, color='black') +
annotate(geom='segment', y=-8.2, yend=-8.38, x=seq(-200,800,100), xend=seq(-200,800,100), color='black') +
geom_segment(x = -200, y = 0, xend = 800, yend = 0, size=0.5, color='black') +
theme(legend.position = c(0.100, 0.150), legend.background = element_rect(fill='#EEEEEE', size=0),
axis.title=element_blank(), legend.key.width = unit(1.2,'cm'), legend.text=element_text(size=17),
legend.title = element_text(size=17, face='bold'), plot.title= element_text(size=20, hjust = 0.5, vjust=2),
axis.text.y = element_blank(), axis.text.x = element_text(size = 14, vjust= 2.12, face='bold', color = 'grey32', family='sans'),
axis.ticks=element_blank(), panel.border = element_blank(), panel.grid.major = element_blank(),
panel.grid.minor = element_blank(), plot.margin = unit(c(0.1,0.1,0,0), 'cm')) +
annotate('segment', x=160, xend=216, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=270, xend=370, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
annotate('segment', x=350, xend=550, y=-7.5, yend=-7.5, colour = 'grey75', size = 1.5) +
annotate('segment', x=500, xend=750, y=-8, yend=-8, colour = 'grey75', size = 1.5) +
scale_fill_manual(name = 'Context / Target trial', values=colours) +
scale_color_manual(name = 'Context / Target trial', values=colours) +
guides(linetype=guide_legend(override.aes = list(size=1.2))) +
guides(color=guide_legend(override.aes = list(size=2.5))) +
# Print y axis labels within plot area:
annotate('text', label = expression(bold('\u2013' * '3 ' * '\u03bc' * 'V')), x = -29, y = 3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('+3 ' * '\u03bc' * 'V')), x = -29, y = -3, size = 4.5, color = 'grey32', family='sans') +
annotate('text', label = expression(bold('\u2013' * '6 ' * '\u03bc' * 'V')), x = -29, y = 6, size = 4.5, color = 'grey32', family='sans')
print(plot_GroupAndElectrode)
output$downloadPlot.1 <- downloadHandler(
filename <- function(file){
paste0(input$var.Group, ' group, electrode ', input$var.Electrodes.1, ', ', Sys.Date(), '.png')},
content <- function(file){
png(file, units='in', width=13, height=5, res=900)
print(plot_GroupAndElectrode)
dev.off()},
contentType = 'image/png')
} )

Apps can include any text, such as explanations of any length and web links. For instance, we can link back to the data repository, where the code for the app can be found.

An example of a Shiny app is available, which may also be edited and run in this RStudio environment, inside the ‘Shiny-app’ folder.

The Shiny server (shinyapps.io) allows publishing dashboards built with various frameworks besides Shiny proper. Flexdashboard and Shinydashboard are two of these frameworks, which have visible advantages over basic Shiny, in terms of layout. An example with Flexdashboard is available.

★ Flexdashboard ★

★ ★ Shiny ★ ★

★ ★ ★ Flexdashboard-Shiny ★ ★ ★

Logistics

Memory capacity can become an issue as you go on, which will be flagged in the error logs as: ‘Shiny cannot use on-disk bookmarking’. This doesn’t necessarily lead you to a paid subscription or to host the website on a custom server. Try pruning the data file, outsourcing data sections across the five available apps.

App providers have specific terms of use. To begin, Shiny has a free starter license with limited use, where free apps can handle a certain amount of data, and up to five apps may be created. Beyond that, RStudio offers a wide range of subscriptions starting at $9/month. For its part, Tableau in principle deals only with subscriptions from $35/month on. While they offer 1-year licenses to students and instructors for free, these don’t include web hosting, unlike Shiny’s free plan. Further comparisons of these platforms are available online. Last, I’ll just mention a third language, D3, which is powerful, and may also be used through R.

In the case of very heavy data or frequent public use, if you don’t want to host your Shiny app externally, you might consider rendering a PDF with your visualisations instead.


pdf("List of plots per page", width=13, height=5)
print(plot1)
print(plot2)
# ...
print(plot150)
dev.off()

High-resolution plots can be rendered into a PDF document in a snap. Conveniently, all text is indexed, so it can be searched (Ctrl+f / Cmd+f / 🔍) (see example). Furthermore, you may also merge the rendered PDF with any other documents.

Your browser does not support embedded PDFs. Download the PDF instead.

Summary in slides

Presenting data interactively online using R Shiny

Pablo Bernabeu

You shall know a word by the company it keeps — so choose your prompts wisely

Reuters Newswire: Finance vs Energy

State of the Union: Pre-War vs Post-War

IMDB Film Reviews: Positive vs Negative

What the Plots Capture – and What They Miss

From Toy Models to Titans

The Transformer Revolution

Fluency Is Not Truth

Why Prompts Matter – and Why One Is Rarely Enough

What Good Prompting Looks Like

A Powerful Tool, Not an Oracle

References

R functions for checking and fixing vmrk files from BrainVision

Numbering trials

Removing trial numbers

Counting markers

rscopus_plus: An extension of the rscopus package

Example of use

References

FAQs on mixed-effects models

References

FAIR standards for the creation of research materials, with examples

FAIRness

Sharing the materials

Reproducibility

Generous documentation

Comments in code scripts

Open-source software

Tidiness and parsimony in computer code

Tidiness and parsimony in project directories

Conclusion

Preprocessing the Norwegian Web as Corpus (NoWaC) in R

The present script can be used to preprocess data from a frequency list of the Norwegian as Web Corpus (NoWaC; Guevara, 2010).

References

ggplotting power curves from the simr package

A usage example

How to discretise the colour variable in sjPlot::plot_model into equally-sized intervals

How to map more informative values onto fill argument of sjPlot::plot_model

How to visually assess the convergence of a mixed-effects model by plotting various optimizers

Table joins with conditional "fuzzy" string matching in R

A new function to plot convergence diagnostics from lme4::allFit()

Completeness versus parsimony

The multiple-optimizers sanity check from lme4::allFit()

Plotting the fixed effects from allFit()

Optional arguments

Use case

References

A table of results for Bayesian mixed-effects models: Grouping variables and specifying random slopes

Loading packages and the results of the models

Adjusting the names of the effects

bayesian_model_table()

The function in use

Shading specific rows

Session info

References

A table of results for frequentist mixed-effects models: Grouping variables and specifying random slopes

Loading packages and the results of the models

Adjusting the names of the effects

frequentist_model_table()

Loading the function from GitHub

The function in use

Shading specific rows

Session info

References

Plotting two-way interactions from mixed-effects models using alias variables

Alias interaction plot

The function

The function in use

References

Plotting two-way interactions from mixed-effects models using ten or six bins

Deciles interaction plot

The function

The function in use

Sextiles interaction plot

The function

The function in use

References

Why can't we be friends? Plotting frequentist (lmerTest) and Bayesian (brms) mixed-effects models

Visualising frequentist and Bayesian estimates in one plot

The multiple-optimizers sanity check from `lme4::allFit()`

`bayesian_model_table()`

`frequentist_model_table()`

Parallelizing `breaks`