Done with convergence warnings? Publications that used allFit to compare optimisers

Development

A function name is a poor search term for bibliographic databases, because it appears in the methods section of an article and almost never in its title, abstract or keywords. Three sources with different coverage were therefore searched with a script in the website’s repository.

Scopus, which I queried through my scopusflow package (Bernabeu, 2026), indexes titles, abstracts, keywords, references and a few other fields but not the body of an article. A search for the function name in any indexed field can only find the articles that mention it in one of those fields. A second Scopus query serves as a proxy. It looks for titles, abstracts and keywords that mention lme4, lmerTest or brms alongside maximal random slopes and convergence. The assumption is that an abstract discussing convergence in that much detail probably belongs to a study that dealt with it. Europe PMC holds the full text of a large body of life-science articles, most but not all of them open access, and its search accepts full-text queries. OpenAlex offers a full-text search over the articles whose text it holds. Both full-text queries pair the function name with a mixed-model term, which removes some unrelated uses of the string without removing all of them.

library(scopusflow)   # reads the key from SCOPUS_API_KEY
scopus_any_field <- scopus_fetch('ALL("allFit")', page_size = 25)
scopus_proxy <- scopus_fetch(
  paste('("lme4" OR "lmerTest" OR "brms") AND maximal AND "random slopes"',
        'AND (convergence OR converge OR converged OR converging)'),
  field = 'TITLE-ABS-KEY', page_size = 25)

The Scopus any-field query and the two full-text queries are read from the files saved with the results, so they are exactly the queries that ran. A query retyped into the post could drift from the one that produced the counts. If Scopus rejects the quoted form of the any-field query, the script tries other forms, and the file records the one that ran.

cat(readLines('searches/scopus_allfit_any_field_query.txt'), sep = '\n')

ALL("allFit")

cat(readLines('searches/europepmc_query.txt'), sep = '\n')

"allFit" AND (lme4 OR "mixed-effects" OR "mixed effects" OR "mixed model" OR "mixed models" OR "multilevel")

cat(readLines('searches/openalex_filter.txt'), sep = '\n')

fulltext.search:allFit AND (lme4 OR lmer OR "mixed effects" OR "mixed-effects" OR "mixed model" OR "mixed models" OR multilevel)

Results

Each search saved its results to a comma-separated values file, and Tables 1 and 2 are built from those files. Nothing in this section queries a database, so the counts are those of the search date rather than of today.

library(dplyr)
library(ggplot2)

read_if_present <- function(file, source) {
  path <- file.path('searches', file)
  if (!file.exists(path)) return(NULL)
  x <- read.csv(path, stringsAsFactors = FALSE)
  x$source <- source
  x
}

scopus_any   <- read_if_present('scopus_allfit_any_field.csv', 'Scopus, any indexed field')
scopus_proxy <- read_if_present('scopus_convergence_proxy.csv', 'Scopus, convergence proxy')
europepmc    <- read_if_present('europepmc_allfit_fulltext.csv', 'Europe PMC, full text')
openalex     <- read_if_present('openalex_allfit_fulltext.csv', 'OpenAlex, full text')

# Each table has a stamp beside it with the time of its search, named after it
# with _retrieved.txt in place of .csv. Tables saved before those stamps
# existed share the single one in retrieved.txt.
retrieved_on <- function(file) {
  stamp <- file.path('searches', sub('\\.csv$', '_retrieved.txt', file))
  if (!file.exists(stamp)) stamp <- file.path('searches', 'retrieved.txt')
  as.Date(substr(readLines(stamp, n = 1), 1, 10))
}
queried <- sapply(c('Scopus (any indexed field)' = 'scopus_allfit_any_field.csv',
                    'Scopus (proxy query)' = 'scopus_convergence_proxy.csv',
                    'Europe PMC' = 'europepmc_allfit_fulltext.csv',
                    'OpenAlex' = 'openalex_allfit_fulltext.csv'),
                  function(f) trimws(format(retrieved_on(f), '%e %B %Y')))

# One sentence for all the sources if they were searched on the same day, and
# otherwise one for each date.
when_queried <- function(queried) {
  if (queried[[1]] == queried[[2]])
    queried <- c(Scopus = queried[[1]], queried[-(1:2)])
  if (length(unique(queried)) == 1)
    return(paste0('The three sources were queried on ', queried[[1]], '.'))
  paste(sapply(unique(queried), function(d) {
    who <- names(queried)[queried == d]
    paste0(knitr::combine_words(who, oxford_comma = FALSE),
           if (length(who) == 1) ' was' else ' were', ' queried on ', d, '.')
  }), collapse = ' ')
}

standardise <- function(x) {
  if (is.null(x)) return(NULL)
  tibble(source = x$source,
         doi = tolower(sub('^https?://doi.org/', '', x$doi)),
         year = as.integer(x$year),
         authors = x$authors,
         title = x$title,
         venue = if ('journal' %in% names(x)) x$journal else x$publication)
}
records <- bind_rows(lapply(list(scopus_any, scopus_proxy, europepmc, openalex),
                            standardise))
records |> count(source, name = 'records')

#> # A tibble: 4 × 2
#>   source                    records
#>   <chr>                       <int>
#> 1 Europe PMC, full text          30
#> 2 OpenAlex, full text            68
#> 3 Scopus, any indexed field     176
#> 4 Scopus, convergence proxy       1

# Count the Scopus any-field titles that mention a mixed model at all.
mixed_terms <- 'mixed|lme4|multilevel|random effect|convergence|optimi[sz]'
sum(grepl(mixed_terms, scopus_any$title, ignore.case = TRUE))

#> [1] 0

head(sort(table(scopus_any$publication), decreasing = TRUE), 4)

#> 
#> European Journal of Pharmacology   Journal of Medicinal Chemistry 
#>                                8                                7 
#>                Neuropharmacology               Psychopharmacology 
#>                                7                                7

The three sources were queried on 2 September 2026. The Scopus search for the function name in any indexed field shows why a bare string makes a poor query. It returned 176 records, the largest group of them in pharmacology and medicinal chemistry, and not one of their titles mentions a mixed model. Scopus appears to match the string loosely against unrelated text, so those records are left out of the list below. The proxy query returned one record, which shows how rarely an abstract describes the treatment of convergence in that much detail. The two full-text sources come closest to answering the question, because they reach the methods section, where the function name usually appears. Their results are merged into one record per work. Records that share a digital object identifier (DOI) are merged, and so are records whose titles match once case and punctuation are set aside, since a preprint and the article published from it carry different DOIs. The merged record keeps the DOI of the published article. The merged list is then screened once, by dropping any record published before allFit existed, as a match in such an article cannot be a use of the function. The screen spares the records of R packages on the Comprehensive R Archive Network (CRAN), because OpenAlex dates them by the first release of the package. The record of afex, first released in 2012, would otherwise be dropped, although any of its text that mentions allFit must have been written in 2014 or later.

# allFit appeared in lme4's GitHub repository in April 2014 and first shipped
# with lme4 1.1-8 in June 2015, so a record published before 2014 cannot be a
# use of it, whatever matched the string.
allfit_year <- 2014L

# OpenAlex dates the record of a CRAN package by the package's first release,
# not by the version whose text matched, so the screen leaves these alone.
cran_package <- '^10\\.32614/cran\\.package\\.'

# A preprint and its published article are one work under two DOIs. Records
# are taken as one work when their titles match once case, markup and
# punctuation are set aside, and the published DOI is kept over that of a
# preprint server: PsyArXiv, bioRxiv and medRxiv, arXiv, OSF Preprints, SSRN.
preprint <- '^10\\.(31234|1101|48550|31219|2139)/'
same_title <- function(title) {
  trimws(gsub('[^a-z0-9]+', ' ', tolower(gsub('<[^>]+>', '', title))))
}

used_allfit <- records |>
  filter(source %in% c('Europe PMC, full text', 'OpenAlex, full text')) |>
  mutate(key = ifelse(is.na(doi) | doi == '', paste(source, title), doi)) |>
  group_by(key) |>
  mutate(work = same_title(first(title)),
         work = ifelse(is.na(work) | work == '', key, work)) |>
  group_by(work) |>
  arrange(grepl(preprint, doi), .by_group = TRUE) |>
  summarise(year = first(na.omit(year)), authors = first(authors),
            title = first(title), venue = first(na.omit(venue)),
            other_dois = paste(setdiff(na.omit(doi), c(first(doi), '')), collapse = ' '),
            doi = first(doi), sources = paste(sort(unique(source)), collapse = '; '),
            .groups = 'drop') |>
  arrange(desc(year), authors)

too_early <- used_allfit |>
  filter(!is.na(year), year < allfit_year, !grepl(cran_package, doi))
used_allfit <- used_allfit |> anti_join(too_early, by = 'work')
c(matched = nrow(used_allfit) + nrow(too_early),
  before_allfit_existed = nrow(too_early), kept = nrow(used_allfit))

#>               matched before_allfit_existed                  kept 
#>                    76                     5                    71

table(used_allfit$sources)

#> 
#>                      Europe PMC, full text Europe PMC, full text; OpenAlex, full text 
#>                                         12                                         17 
#>                        OpenAlex, full text 
#>                                         42

# The works found under more than one DOI, to be checked by eye.
used_allfit |>
  filter(other_dois != '') |>
  transmute(year, doi, other_dois)

#> # A tibble: 6 × 3
#>    year doi                            other_dois                  
#>   <int> <chr>                          <chr>                       
#> 1  2025 10.1504/ijamechs.2025.10069510 10.1504/ijamechs.2025.144590
#> 2  2024 10.1162/opmi_a_00130           10.31234/osf.io/3mfcu       
#> 3  2024 10.1111/cdev.14099             10.31234/osf.io/8rdmj       
#> 4  2023 10.1111/1365-2745.14225        10.48550/arxiv.2303.15712   
#> 5  2022 10.1242/dmm.048025             10.1101/2020.12.03.411058   
#> 6  2021 10.1007/s10531-021-02232-5     10.1101/2020.12.21.422503

Figure 1 shows how many of the kept publications, all of them from the two full-text sources, appeared in each year. It leaves out the CRAN packages in the list, since the year of a package record is that of its first release.

# The year of a CRAN package is that of its first release, which says nothing
# about when its text came to mention allFit.
per_year <- used_allfit |>
  filter(!is.na(year), !grepl(cran_package, doi)) |>
  count(year)

# The year of the search was still under way, so its label says where the
# count stops: at the later of the two full-text searches.
searched <- max(retrieved_on('europepmc_allfit_fulltext.csv'),
                retrieved_on('openalex_allfit_fulltext.csv'))
year_label <- function(y) {
  ifelse(y == as.integer(format(searched, '%Y')),
         paste0(y, '\nto ', trimws(format(searched, '%e %b'))), y)
}

ggplot(per_year, aes(year, n)) +
  geom_col(fill = '#0072B2', width = 0.8) +
  geom_text(aes(label = n), vjust = -0.5, size = 3.4) +
  scale_x_continuous(breaks = min(per_year$year):max(per_year$year),
                     labels = year_label) +
  scale_y_continuous(expand = expansion(mult = c(0, 0.1))) +
  labs(x = NULL, y = NULL) +
  theme_minimal(base_size = 12) +
  theme(axis.text.y = element_blank(), panel.grid = element_blank())

Figure 1: Publications per Year Whose Full Text Mentions allFit, According to Europe PMC and OpenAlex. The last bar stops at the date of the search.

Table 1 lists all the kept publications, the CRAN packages included, most recent first, with the sources in which each was found. Authors are given as the sources returned them. The two sources overlap in a minority of records, which reflects how differently they cover the literature.

short_authors <- function(a) {
  a <- ifelse(is.na(a), '', a)
  first <- sub('[;,].*$', '', a)
  ifelse(grepl('[;,]', a), paste(first, 'et al.'), first)
}
used_allfit |>
  transmute(Year = year,
            Authors = short_authors(authors),
            Title = title,
            Venue = venue,
            DOI = ifelse(is.na(doi) | doi == '', '',
                         sprintf('[%s](https://doi.org/%s)', doi, doi)),
            Sources = sources) |>
  knitr::kable(caption = 'Candidate Publications Whose Full Text Mentions allFit, Most Recent First, With the Sources in Which Each Was Found.')

The only hit of the proxy query, in Table 2, is an erratum notice that carries the abstract of the original article. A wider proxy query would be a reasonable place to look for further precedents, since studies that compared optimisers rarely name the function in a field that Scopus indexes.

records |>
  filter(source == 'Scopus, convergence proxy') |>
  transmute(Year = year, Authors = short_authors(authors), Title = title, Venue = venue,
            DOI = ifelse(is.na(doi) | doi == '', '',
                         sprintf('[%s](https://doi.org/%s)', doi, doi))) |>
  knitr::kable(caption = 'The Single Record Returned by the Scopus Convergence Proxy Query.')

Discussion

The list is incomplete, because the searches miss part of the literature. Europe PMC holds full text mostly for open-access articles in journals within its scope, so a paywalled study in a linguistics or psychology journal is usually invisible to it unless a preprint or an accepted manuscript was deposited. OpenAlex holds the text of a subset of the literature that it does not fully document, and Scopus cannot see the methods section at all. None of the three can find a study that compared optimisers with its own loop or by calling the optimx package directly, or one that reported the comparison only in supplementary material. At the same time, a full-text match can be an article that uses the function for something other than a convergence check, or that mentions it only in a footnote. Records published before the function existed have been dropped, but the table has not otherwise been screened by hand, so some of the publications it lists may mention the function without having used it. A preprint and its published article are merged only when their titles match, so a study whose title changed on publication can still appear twice. The list is therefore a starting point for the reader who needs a precedent.

References

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. (2015). Parsimonious mixed models [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1506.04967

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

Bernabeu, P. (2026). scopusflow: A reproducible workflow layer for Scopus bibliographic searches (Version 0.4.0) [Computer software]. Comprehensive R Archive Network (CRAN). https://doi.org/10.32614/CRAN.package.scopusflow

Brauer, M., & Curtin, J. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411. https://doi.org/10.1037/met0000159

Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001

Singmann, H., & Kellen, D. (2019). An introduction to mixed models for experimental psychology. In D. Spieler & E. Schumacher (Eds.), New methods in cognitive psychology (pp. 4–31). Routledge. https://doi.org/10.4324/9780429318405-2