Matching, dimensions and designs • lexsync

Choosing stimuli is a constrained selection problem. One lexical property is manipulated, several others must not vary with it, and the words that satisfy every constraint at once are a small and awkward subset of the language. This vignette covers how lexsync states those constraints and how it resolves them: the dimensions it computes, the filters that narrow a lexicon to a candidate pool, the four matching methods and the situations each one suits, the tolerance windows that bound an acceptable match, the continuous alternative to a matched dichotomy, and the report that tells you what control the selection actually achieved.

Everything below runs on the English example lexicon bundled with the package, a 3000-word slice that needs no download.

library(lexsync)
schema <- yaml::read_yaml(
  system.file("extdata", "schema.yaml", package = "lexsync")
)
lex <- load_lexicon(
  system.file("extdata", "en_example.csv", package = "lexsync"),
  schema, language = "english"
)
head(
  lex[, c("word", "frequency", "length", "n_syllables", "n_density", "old20")],
  4
)

   word frequency length n_syllables n_density old20
1   aaa      3.77      3           1        19   1.0
2   aac      3.00      3           1        22   1.0
3   aap      3.47      3           1        21   1.0
4 aaron      4.23      5           2         3   1.8

The lexical dimensions

A dimension is any numeric column the matcher can equate. lexsync distinguishes those read straight from the lexicon file from those computed from the orthographic form at load time, and the schema records the distinction along with each dimension’s unit. Units matter when reading a report: a raw difference of 1 means one letter on length but a factor of ten in corpus occurrence on frequency, which is why the comparisons are standardised.

dims <- schema$dimensions
knitr::kable(data.frame(
  dimension = names(dims),
  type = vapply(dims, function(d) d$type, character(1)),
  unit = vapply(dims, function(d) d$unit, character(1)),
  row.names = NULL
), caption = "The dimensions declared in schema.yaml")

The dimensions declared in schema.yaml
dimension	type	unit
length	derived	characters
frequency	column	Zipf
n_density	derived	neighbours
old20	derived	mean Levenshtein distance
n_syllables	derived	syllables (orthographic estimate)
bigram_freq	derived	mean bigram probability (type-based, non-positional)

length is the character count of the orthographic form. frequency is the Zipf value read from the lexicon’s freq_zipf column, a log-scaled measure on which roughly 1 is a word occurring once per hundred million tokens and 7 a word occurring ten thousand times per million. The value is the base-10 logarithm of the frequency per billion words (van Heuven et al., 2014). It is preferred to raw counts because it is linear in the log space in which frequency effects are approximately linear, and because it is comparable across corpora of different sizes. n_density is Coltheart’s N, the count of same-length words differing by one letter substitution (Coltheart et al., 1977). old20 is the mean Levenshtein distance to the twenty nearest words in the reference set, which unlike Coltheart’s N is defined for long words, where substitution neighbours run out (Yarkoni et al., 2008). A larger OLD20 means a more isolated word, so it runs in the opposite direction to n_density.

n_syllables also arrives at load time, counting maximal runs of vowels, an orthographic estimate with no pronunciation model behind it. Only bigram_freq is computed on request, by add_bigram_frequency(): it is the mean corpus probability of a word’s adjacent letter bigrams, standing proxy for phonotactic probability.

count_syllables(c("cat", "table", "rhythm", "being", "chocolate"))

[1] 1 2 1 1 4

The estimate is worth its caveat, and the errors run in both directions. It handles rhythm because y is treated as a vowel letter. It undercounts being, whose ei is one run of vowel letters spanning two nuclei, and it overcounts chocolate, whose silent final e forms a run of its own. The measure is adequate for equating two conditions on gross syllabic complexity and it is not a substitute for a pronunciation dictionary.

Computing dimensions on a pool

The bundled lexicon arrives with n_density and old20 already present, but neighbourhood measures depend on the reference set they are computed against, so a lexicon derived elsewhere may need them recomputed. add_neighbourhood() does that. Both measures are quadratic in the size of the reference set, which is why the pipeline computes them on the pool alone, and why the reference should still be the whole lexicon: a word’s neighbours are its neighbours in the language, not among the handful of words that survived your filters.

sample_words <- lex[lex$word %in% c("cat", "dog", "house", "rhythm"), ]
add_neighbourhood(
  sample_words, reference = lex$word
)[, c("word", "n_density", "old20")]

      word n_density old20
1287 house         0  2.55

add_bigram_frequency() works the same way, and merge_norms() joins any word-keyed norm table, which is how dimensions the corpus does not carry (concreteness, age of acquisition, behavioural measures from a megastudy) become matchable. Once a column is present, the matcher treats it like any other.

aoa <- data.frame(word = c("cat", "dog", "house"), aoa = c(3.1, 3.0, 3.4))
merged <- merge_norms(lex, aoa)
merged[merged$word %in% aoa$word, c("word", "frequency", "aoa")]

      word frequency aoa
1287 house      5.71 3.4

The join is case- and whitespace-insensitive on both sides, keeps the lexicon’s own row and column order, and refuses a norm column whose name already exists on the lexicon. A silent rename would be the worst available outcome: it leaves the dimension the design matches on under a name nothing looks for.

In a design you do not call this yourself. Name the tables in a norms: block and the pipeline joins them before the pool is built, so a norm column can be filtered on, matched on or spanned like any other dimension.

norms:
  - path: norms/en_concreteness.csv
    on: word                  # optional join key, default 'word'
    columns: [concreteness]   # optional; default every column but the key

pool_filters:
  concreteness: [2.0, 5.0]
match_on: [length, concreteness]

lexsync ships no norm data: licences vary, and the citation is yours to honour. What it does do is record what you joined. Every table appears in the materials datasheet with its checksum and its per-column coverage, because a norm table can supply the very variable a design manipulates, and a selection over columns of unstated origin is not reproducible from the record that exists to make it so. Coverage is recorded for a related reason: a word the table does not cover gets a missing value, and the tolerance windows then drop it from the pool.

Pool filters

A pool filter narrows the lexicon before any matching happens. It states what counts as an eligible word at all, independently of condition, and it is the right place for constraints you would apply to every item in the study.

pool_filters <- list(length = c(3, 7), frequency = c(3.8, 7.0))
pool <- build_pool(lex, pool_filters)
c(lexicon = nrow(lex), pool = nrow(pool))

lexicon    pool 
   3000     782

A numeric pair is read as an inclusive range. Any other vector is read as a set of permitted values, so list(pos = c("noun", "verb")) would keep nouns and verbs if the lexicon carried a pos column. A filter naming a column the lexicon does not have is skipped silently, which keeps one schema usable across corpora that differ in their optional columns, and which is worth knowing about when a filter appears not to bite. Rows missing a value on a filtered column are always dropped.

Filtering is a separate step from matching. match_stimuli() reads a design’s conditions and match_on fields but never its pool_filters, so passing a raw lexicon where a pool was intended silently widens the selection, with no error to warn you. The pipeline calls build_pool() first and the examples here do the same.

Supplied item pools

Narrowing a whole lexicon is the right way round when any word of the language will do. It is the wrong way round when the candidate set is itself a research decision: a list from a previous study, from a norming session, or one restricted to a semantic category that no lexical filter can express. Such a list used to have to masquerade as a lexicon, which meant inventing a freq_zipf column for it, or else skip matching entirely by going in as items.source: table.

items.source: pool takes the list as it is.

items:
  source: pool
  path: items/pool_en_concrete_nouns.csv
  lexicon: corpora/derived/en.csv

The list needs only a word column. Length and the syllable estimate are derived from the form, and items.lexicon supplies the rest, frequency above all, by lookup. A word the lexicon does not have raises a hard error, because the tolerance windows drop rows with missing values silently and the pool would then be smaller than it appears to be.

One subtlety would quietly invalidate the controls if it went the other way. n_density and old20 are properties of a word in its language, not among the 131 words of a supplied list, so they are computed against the lexicon’s words. A supplied pool with no lexicon falls back to itself, which is why one is given above. load_pool() returns both parts of that, the pool and the reference word list, if you call it directly outside a design.

Everything downstream is unchanged: the same conditions, the same match_on, the same report, the same datasheet. What differs is only where the candidates came from, and the datasheet records the list’s checksum so that stays answerable later. design_en_supplied_pool.yaml is a worked example.

Relational designs, where the item is a pair

A priming study’s theoretical variable is a property of the pair, such as distributional similarity or associative strength, while its nuisance variables are properties of each member: frequency, length, neighbourhood. A design could previously have matching, or its own item table, but not both, so a relational design had to be assembled by hand.

items.members promotes an ordinary item table to a pair-keyed one.

items:
  source: table
  path: items/priming_pairs_en.csv
  members: [prime, target]          # the two word fields
  lexicon: corpora/derived/en.csv   # where each member's norms are looked up
  anchor_condition: related         # whose row represents the pair during selection

Each member’s word-level norms are joined from the lexicon and prefixed, giving prime.frequency, target.length and so on. The member name leads and the dimension follows, for a reason specific to R: prime.frequency is safe because df$prime still exact-matches the bare prime column, whereas frequency.prime would be a partial-matching hazard. Those prefixed names are usable wherever a dimension is: in pool_filters, in match_on, as a continuous predictor or as a control.

pair.lev is the Levenshtein distance between the two members and pair.overlap is 1 - pair.lev / length of the longer form, a normalised orthographic similarity. It scores edit operations, so it is not a count of shared material: abc and cba share every letter but score 0.33. Orthographic overlap is the standard confound control in a priming design, since a related pair that also shares letters confounds semantic relatedness with orthographic similarity. The pipeline adds both automatically when the two members are named prime and target. Under any other member names, call add_pair_overlap() on the item table yourself. A predictor that cannot be computed from the forms alone, such as cosine similarity or forward associative strength, arrives as an extra column on the item table and is used like any other.

n_per_condition: 8                  # counts PAIRS: 8 pairs, 16 rows, 8 per list
pool_filters:
  target.length: [3, 7]
  target.frequency: [3.0, 6.0]
continuous:
  predictor: target.frequency
  controls: [target.length, pair.overlap]
match_on: [target.length, pair.overlap]

Two rules follow from the item being a pair. A filter applies to a pair only if every one of its rows passes, and selection runs over one row per pair, the anchor_condition, defaulting to the byte-first condition, with the result re-expanded afterwards so every condition row of every chosen pair survives. Neither is an optimisation. A filter on prime.frequency applied row by row would keep a pair’s related row and drop its unrelated one, leaving a set the Latin-square counterbalancer cannot complete. Only a dimension that varies within the pair can split it that way, which the prime-level and pair-level ones do and the target-level ones do not, since the target is the same word in both rows.

A member name may not be one of word, id, set, list, trial, condition, replicate or item, which the engines read directly, and a design using one is refused. The datasheet gains a relational block recording the members, the pair count, the member lexicon and its checksum, and the member-level dimensions separately from the relational ones. design_en_priming_continuous.yaml is the worked example, and its own comments go further into why each rule is there.

Conditions, the anchor and matched sets

A condition is a named region of the pool, given by define_by, which is read exactly as pool_filters is. The first condition listed is the anchor. Its items are chosen by an even spread across the sorted candidate subpool, so the anchor covers the whole of its region on the manipulated dimension and does not cluster at one end. Every later condition is then matched to the anchor item by item, and the set column records the pairing.

design <- list(
  name = "vignette_matching", language = "english", n_per_condition = 20,
  pool_filters = pool_filters,
  conditions = list(
    list(name = "high", define_by = list(frequency = c(5.2, 7.0))),
    list(name = "low",  define_by = list(frequency = c(3.8, 4.4)))
  ),
  match_on = list("length", "n_density", "old20")
)
stim <- match_stimuli(pool, design, schema)
head(
  stim[
    stim$set %in% 1:3,
    c("set", "condition", "word", "frequency", "length", "old20")
  ],
  6
)

   set condition    word frequency length old20
1    1      high    knew      5.20      4  1.75
2    2      high  energy      5.23      6  2.60
3    3      high running      5.23      7  1.80
21   1       low    myth      4.08      4  1.75
22   2       low  arctic      4.03      6  2.40
23   3       low blowing      4.20      7  1.80

Because the anchor is spread across its own subpool, the order of the conditions is a design decision with consequences. The anchor’s distribution on the manipulated dimension is inherited from the pool, and every other condition’s distribution is whatever matching to that anchor produces.

The four matching methods

The method is set with matching.method in a design, or left to the schema default. All four minimise a distance in standardised space, computed by z-scoring each match_on dimension against the whole pool, so that a letter and a Zipf unit become comparable. They differ in what they minimise it over.

standardised_euclidean, the default, walks the anchor items in order and gives each one its nearest unused counterpart, greedily. Because the anchor is selected before any matching happens, it preserves the anchor’s even spread exactly. Being anchored, it generalises past two conditions, as mahalanobis does, while joint and optimal score cross-condition pairs and are refused with any number of conditions but two.

joint abandons the anchor. It scores every cross-condition pair, then takes the best-matched pairs in ascending order of distance until it has enough, so an item with no good counterpart is simply never selected. This equates the controls more tightly than per-anchor matching when the manipulation is confounded with a control, and the price is that the manipulated dimension’s realised distribution is no longer under your control. It requires exactly two conditions.

mahalanobis replaces the Euclidean metric with one weighted by the inverse of the pool’s correlation matrix, so that dimensions sharing variance are not counted twice. It suits the case where the matched dimensions are strongly intercorrelated, which is the normal case for orthographic measures.

round(cor(pool[, c("length", "n_density", "old20")]), 2)

          length n_density old20
length      1.00     -0.69  0.76
n_density  -0.69      1.00 -0.73
old20       0.76     -0.73  1.00

Length, neighbourhood density and OLD20 are far from independent here. Under a plain Euclidean metric, matching on all three counts the shared length-neighbourhood variance three times over and lets an independent dimension drift. Mahalanobis distance discounts that overlap (Rubin, 1980; Stuart, 2010).

optimal keeps the Euclidean metric but solves the pairing globally, minimising the summed distance over all pairings at once (Gu & Rosenbaum, 1993; Hansen & Klopfer, 2006). Greedy matching can spend the one good counterpart of a hard item on an easy item that had alternatives. Optimal matching cannot. It requires exactly two conditions and the clue package.

methods <- c("standardised_euclidean", "joint", "mahalanobis")
if (requireNamespace("clue", quietly = TRUE)) methods <- c(methods, "optimal")

summarise_method <- function(method) {
  d <- design
  d$matching <- list(method = method)
  s <- match_stimuli(pool, d, schema)
  cmp <- match_report(
    s, c("frequency", "length", "n_density", "old20"), schema
  )$comparisons
  controls <- cmp[cmp$dimension != "frequency", ]
  high <- s$frequency[s$condition == "high"]
  low  <- s$frequency[s$condition == "low"]
  data.frame(
    method = method,
    worst_control_d = round(max(abs(controls$cohens_d)), 3),
    freq_d = round(cmp$cohens_d[cmp$dimension == "frequency"], 2),
    freq_raw_diff = round(mean(high) - mean(low), 2),
    freq_sd_high = round(sd(high), 2)
  )
}
do.call(rbind, lapply(methods, summarise_method))

                  method worst_control_d freq_d freq_raw_diff freq_sd_high
1 standardised_euclidean           0.038   5.12          1.51         0.39
2                  joint           0.000   6.94          1.40         0.23
3            mahalanobis           0.052   5.07          1.49         0.39
4                optimal           0.000   6.94          1.40         0.23

The comparison rewards a careful reading, and it is a good illustration of why a standardised difference should not be read as an effect size on its own. The two globally pairing methods drive the control differences to zero, because a free choice over all pairs can find counterparts that match exactly on all three dimensions. Their standardised frequency separation is also the larger of the four, yet their raw separation in Zipf units is the smaller. Both facts have one cause: those methods take whichever pairs match best, and those pairs turn out to cluster, which shrinks the SD of the high condition and so shrinks the pooled SD that freq_d divides by. A narrower manipulation can raise d.

Whether that is a gain depends on the study. The anchor methods keep the high condition spread across its whole band by construction, which supports a claim about the band rather than about a cluster inside it, and it is variance you may want if the manipulation is later modelled as continuous. The globally pairing methods trade some of that spread for exact control of the nuisance dimensions. These numbers describe this pool and this design. The point is how to make the comparison, not which method wins.

The parity caveat on two methods

The R and Python engines select byte-identical stimuli under standardised_euclidean and joint. They do not guarantee it under mahalanobis or optimal. Both of the latter route the distance through an operation whose last bits are decided by the platform’s linear-algebra implementation, a covariance-matrix inverse in one case and an assignment solver in the other, and once two candidate distances differ in their final bits the tie-break that would otherwise make the engines agree never fires. mahalanobis usually still agrees exactly, and optimal often selects an equally-optimal but different set. Each run’s datasheet records which case applies, under selection.cross_engine. If cross-engine reproducibility is a requirement of your project, the two default methods are the ones that carry the guarantee. The reproducibility vignette sets out why the exception cannot simply be engineered away.

Tolerance windows

Matching by distance alone will always return the nearest available candidate, however far away it is. The tolerance window is the guard against that. Before the distance ranking, each later condition’s candidates are filtered to those lying within the anchor’s mean plus or minus k standard deviations on every matched dimension, with k set per dimension.

str(schema$matching$tolerance_k)

List of 4
 $ length   : num 2
 $ frequency: num 1
 $ n_density: num 2
 $ old20    : num 2

Smaller k is stricter. The default of 2.0 on the orthographic dimensions is permissive, admitting most of a normal distribution, and it is meant as a backstop against a wild counterpart. The distance ranking within the window is what does the work of control. A design tightens the window per dimension, and only the named dimensions are overridden.

The window belongs to the anchored path, so it is what standardised_euclidean and mahalanobis apply. joint and optimal score cross-condition pairs directly and never read tolerance_k at all, which is worth knowing before tightening a window and finding the selection unmoved.

tight <- design
tight$matching <- list(tolerance_k = list(old20 = 0.25))
tight_stim <- match_stimuli(pool, tight, schema, verbose = TRUE)
round(match_report(tight_stim, "old20", schema)$comparisons$cohens_d, 3)

[1] -0.066

There is a behaviour here that will bite you if you do not know it. If a window leaves a condition with fewer candidates than there are anchor items, the matcher neither fails nor returns a short set. It relaxes that condition’s window and carries on. Tightening the window past what the pool can supply therefore looks exactly like a window that was never set, and under the default policy verbose = TRUE is the only thing that tells you which happened.

too_tight <- design
too_tight$matching <- list(tolerance_k = list(old20 = 0.05))
relaxed_stim <- match_stimuli(pool, too_tight, schema, verbose = TRUE)

lexsync: condition 'low' has 0 candidates within tolerance (< 20 needed); relaxing the window.

Two details of the relaxation matter. It is per condition and it is all-or-nothing: the fallback drops that condition’s windows on every matched dimension at once, not merely the one that proved unsatisfiable, so an over-tightened old20 also discards the length and n_density windows for that condition. And it does not degrade the match as much as you might fear, because the distance ranking still runs over the whole subpool and still prefers near counterparts. Only when the relaxed subpool is still too small does the matcher stop, with an error naming the condition and the counts.

Both behaviours are policies, and the schema’s matching block names them. on_insufficient_tolerance governs the fallback just described. Its default of relax keeps the behaviour above, while error makes the matcher refuse and say so, which is the honest setting when the window is itself the criterion, as it is when reproducing a published study’s stated windows. shortfall governs what happens when fewer sets than n_per_condition can be selected at all, and it defaults to error deliberately: the datasheet and the generated methods paragraph state the requested n, so a set that silently shrank would leave the record misstating the materials. A design that can live with fewer sets says so with shortfall: allow, and either key may be overridden under the design’s own matching: block, exactly as tolerance_k was overridden above.

Windows are the place to reproduce a published study’s stated criteria. The bundled design_es_gender_repro.yaml sets frequency to 0.111..., which is the mean plus or minus SD/9 used by González Alonso et al. (2025), alongside length at the mean plus or minus 2 SD.

One asymmetry is worth knowing. Tolerance windows are derived from the anchor, so they apply to the anchor-based methods, standardised_euclidean and mahalanobis, and to continuous designs. joint and optimal never build an anchor and never consult tolerance_k. Both instead cap each subpool to the 1200 candidates nearest the other condition’s centroid before pairing, which bounds the cost of scoring every pair.

Continuous designs

A matched dichotomy is not the only way to study a graded property, and for a property like frequency it is often not the best one. Splitting a continuum into a high and a low band discards the variation within each band, spends statistical power to do it, and leaves the estimate dependent on where the split was placed. The selection can also manufacture its own artefacts: Liben-Nowell et al. (2019) show that choosing a controlled subset from a large candidate set can produce an apparent effect from a set where none exists, because the selection is itself a search over the space of possible item sets. Kuperman (2015) makes the constructive case, that a megastudy’s variation can be used directly by sampling across a predictor’s range and modelling it.

A continuous block does that. Rather than conditions, it names one predictor to span and the controls to hold still, and the two must agree with match_on.

cont_design <- list(
  name = "vignette_continuous", language = "english", n_per_condition = 60,
  pool_filters = pool_filters,
  continuous = list(
    predictor = "frequency",
    controls = c("length", "n_density", "old20")
  ),
  match_on = list("length", "n_density", "old20"),
  matching = list(
    tolerance_k = list(length = 1.5, n_density = 1.5, old20 = 1.5)
  )
)
cont <- select_continuous_stimuli(pool, cont_design, schema)
nrow(cont)

[1] 60

range(cont$frequency)

[1] 3.80 6.85

Selection runs in two passes. An even spread over the pool defines the control windows from its own distribution, the pool is filtered to those windows, and a second even spread over what survives is the selection. The result covers the predictor’s range while the controls stay near-constant, and therefore near-uncorrelated with the predictor, which is what the regression needs.

cont_report <- match_report_continuous(
  cont, "frequency",
  c("length", "n_density", "old20"), schema
)
cont_report$comparisons

  dimension      role pearson_r predictor_span
1 frequency predictor        NA           3.05
2    length   control    -0.191           3.05
3 n_density   control     0.112           3.05
4     old20   control    -0.228           3.05

The report is read differently from a matched one. predictor_span is the realised range of the predictor in the selected set and should be wide, since it is the variation the regression has to work with. Each control’s pearson_r is its correlation with the predictor and should be near zero, since a control that still correlates with the predictor is still a confound. Tightening tolerance_k pushes those correlations down at the cost of a smaller eligible pool, and that is the dial to turn when a control will not come loose.

The datasheet for a continuous design reports the span, the correlations and a suggested regression model in which the controls enter as covariates.

Reading the validation output

match_report() returns two data frames. The descriptives give per-condition n, mean, SD, minimum, median and maximum for each dimension, which is the table that belongs in a paper’s materials section.

report <- match_report(
  stim, c("frequency", "length", "n_density", "old20"), schema
)
knitr::kable(
  report$descriptives,
  caption = "Descriptive statistics per condition"
)

Descriptive statistics per condition
group	dimension	n	mean	sd	min	median	max
high	frequency	20	5.573	0.387	5.20	5.480	6.85
high	length	20	5.400	0.995	4.00	5.000	7.00
high	n_density	20	2.950	2.911	0.00	2.500	12.00
high	old20	20	1.780	0.318	1.15	1.750	2.60
low	frequency	20	4.059	0.159	3.85	4.035	4.32
low	length	20	5.400	0.995	4.00	5.000	7.00
low	n_density	20	2.850	2.323	0.00	2.500	8.00
low	old20	20	1.773	0.272	1.20	1.750	2.40

The comparisons contrast every other condition against the anchor on every dimension, and they carry the four statistics that answer whether the matching worked.

knitr::kable(
  report$comparisons,
  caption = paste(
    "Realised control: standardised differences with 90% intervals,",
    "variance ratios and TOST verdicts"
  )
)

Realised control: standardised differences with 90% intervals, variance ratios and TOST verdicts
condition	reference	dimension	cohens_d	d_ci_low	d_ci_high	var_ratio	tost_p	equivalent
low	high	frequency	5.119	4.586	5.653	0.168	1.0000	FALSE
low	high	length	0.000	-0.533	0.533	1.000	0.0611	FALSE
low	high	n_density	0.038	-0.495	0.571	0.637	0.0761	FALSE
low	high	old20	0.025	-0.508	0.559	0.734	0.0708	FALSE

cohens_d is the standardised mean difference against the anchor, using the pooled SD. On the manipulated dimension it should be large and on a controlled dimension it should be near zero.

d_ci_low and d_ci_high bound it with a 90% confidence interval. The interval, not the point estimate, is the primary summary of a controlled dimension. Its upper limit is the largest imbalance still consistent with the stimuli you have, and with few items that limit can be substantial even when the point estimate is almost zero. Reporting the interval keeps the dependence on set size visible, where a bare ‘no significant difference’ hides it. The width is a property of the number of items, so a small set cannot demonstrate tight control however good its point estimates look (Sassenhagen & Alday, 2016).

var_ratio is the condition’s variance over the anchor’s. Two conditions can share a mean and still confound if one is far more spread out than the other, which every mean-based statistic misses, so this is the distributional check. A ratio near 1 is balanced and a common heuristic flags ratios outside roughly 0.5 to 2. See ?variance_ratio for the sources behind that heuristic.

tost_p and equivalent are the two one-sided tests. A conventional non-significant difference test is not evidence of equivalence, because failing to reject a null is not the same as supporting it, and the failure is easiest to achieve with few items and noisy measures. TOST inverts this by testing against an explicit bound: the reported p is the larger of two one-sided p-values against a difference of plus or minus bound_d pooled SDs, and a value below alpha supports equivalence within that bound (Lakens, 2017). The bound and alpha are the schema’s, so the verdict is only as meaningful as the bound you chose.

str(schema$equivalence)

List of 2
 $ bound_d: num 0.5
 $ alpha  : num 0.05

A bound of 0.5 SD is the schema default and it is a convention rather than a finding. It is a permissive bound for a nuisance dimension in a design where that dimension has a strong effect on the response, and a study that cares about a particular confound should set a bound it can defend. The 90% interval is the 1 - 2 x alpha interval precisely so that it corresponds to the TOST decision at alpha: the test declares equivalence exactly when the interval falls entirely inside the bounds, which means the interval and the verdict are two readings of one thing.

The individual statistics are exported and can be called directly on any two vectors.

high <- stim$old20[stim$condition == "high"]
low  <- stim$old20[stim$condition == "low"]
str(cohens_d_ci(high, low))

List of 3
 $ d      : num 0.0254
 $ ci_low : num -0.508
 $ ci_high: num 0.559

str(tost_equiv(high, low, bound_d = 0.5))

List of 2
 $ p         : num 0.0708
 $ equivalent: logi FALSE

round(variance_ratio(low, high), 3)

[1] 0.734

balance_check() is the last piece and answers a different question: whether the levels of a column occur equally often. It returns human-readable warnings and an empty vector when all is well, and the pipeline logs whatever it returns.

balance_check(stim, "condition")

character(0)

balance_check(rbind(stim, stim[1, ]), "condition")

[1] "Column 'condition' is unbalanced: high=21, low=20"

Pseudowords

Lexical decision needs non-words, and the difficulty is that they must be orthographically plausible without being real. build_lexdec_stimuli() draws real words by an even spread across the byte-ordered pool, then generates a length-matched pseudoword for each.

lexdec_pool <- build_pool(lex, list(length = c(4, 7), frequency = c(3.5, 6.0)))
stim_lexdec <- build_lexdec_stimuli(
  lexdec_pool, n = 8, reference_words = lex$word
)
head(stim_lexdec[, c("target", "condition", "length", "set")], 4)

   target condition length set
1   aaron      word      5   1
2 carrier      word      7   2
3 erected      word      7   3
4   house      word      5   4

The presented string is target, the conditions are word and pseudoword, and set pairs each word with its non-word twin. reference_words should be the full lexicon: it supplies the bigram statistics and the list of real forms a pseudoword must avoid, and it defaults to the pool when omitted, which is rarely what you want.

Two methods are available, chosen per design under items.generation.method.

letter_substitution, the default, changes as few letters as possible subject to three constraints: every resulting bigram must be attested in the corpus, the form must not be a real word or one already generated, and the length must be preserved exactly. Single substitutions are searched first, then pairs as a fallback. Among the legal candidates the most bigram-plausible wins, with a byte-order tie-break.

subsyllabic splits each word into onset, nucleus and coda constituents and swaps whole constituents for attested alternatives of the same role and length, so the pseudowords keep their syllabic structure. Codas and nuclei are changed before onsets, which carry the most identifying orthography. Roughly two thirds of a word’s constituents are targeted, each swap preserves length, and a word with no legal swap falls back to letter substitution, so every word yields a pseudoword. It is a deterministic orthographic approximation of Wuggy (Keuleers & Brysbaert, 2010), trading Wuggy’s phonological model for exact length matching and cross-engine reproducibility.

sub_stim <- build_lexdec_stimuli(
  lexdec_pool, n = 8, reference_words = lex$word, method = "subsyllabic"
)
# The real words come first in the frame, so pair each with its twin by `set`.
merge(
  sub_stim[sub_stim$condition == "word", c("set", "target")],
  sub_stim[sub_stim$condition == "pseudoword", c("set", "target")],
  by = "set", suffixes = c("_word", "_pseudoword")
)[1:4, ]

  set target_word target_pseudoword
1   1       aaron             ierer
2   2     carrier           cinrees
3   3     erected           aristar
4   4       house             reesi

Both generators process base words in byte order, so the set of already-used pseudowords evolves identically in the two engines and both select byte-identical stimuli. Both are also orthographic models defined for a-z words: build_lexdec_stimuli() filters the pool to those, and the subsyllabic segmenter returns nothing for a word with an accent, a hyphen or a digit, which sends it down the letter-substitution path. That is why lexical decision is demonstrated on the English designs and never on the Chinese one.

generate_pseudowords() and make_pseudoword() are available directly if you want the forms without the lexical-decision scaffolding around them.

generate_pseudowords(c("house", "table"), lex$word)

  base_word pseudoword
1     house      hoese
2     table      talle

In a design this is items.source: generate, with the method under items.generation.method. design_en_lexdec.yaml and design_en_lexdec_wuggy.yaml are the worked examples of the two methods, and design_en_lexdec_blocks.yaml adds practice and filler blocks to the same generated set.

Items as a random factor

A matched set is one sample from the language, and an analysis that treats its items as fixed generalises to those items rather than to the language they were drawn from (Clark, 1973; Yarkoni, 2022). Crossed random effects are the standard answer, and the datasheet suggests a model that includes them. resample_stimuli() supports the stronger move of drawing several fully matched sets that share no items, each one an independent sample, which lets a study run different item samples across participant groups or show that an effect survives the change of materials.

reps <- resample_stimuli(pool, design, schema, n_sets = 2)
table(reps$replicate, reps$condition)

length(intersect(
  reps$word[reps$replicate == 1],
  reps$word[reps$replicate == 2]
))

[1] 0

Each replicate is matched independently against the pool with the earlier replicates’ items removed, so the sets are disjoint by construction. Later replicates draw from a depleted pool, which means match quality can degrade across replicates. The report will show it if it does.

References

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. https://doi.org/10.1016/S0022-5371(73)80014-3

Coltheart, M., Davelaar, E., Jonasson, J. T., & Besner, D. (1977). Access to the internal lexicon. In S. Dornic (Ed.), Attention and Performance VI (pp. 535–555). Erlbaum.

González Alonso, J., Bernabeu, P., Silva, G., DeLuca, V., Poch, C., Ivanova, I., & Rothman, J. (2025). Starting from the very beginning: Unraveling third language (L3) development with longitudinal data from artificial language learning and EEG. International Journal of Multilingualism, 22(1), 119–142. https://doi.org/10.1080/14790718.2024.2415993

Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2(4), 405–420. https://doi.org/10.1080/10618600.1993.10474623

Hansen, B. B., & Klopfer, S. O. (2006). Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics, 15(3), 609–627. https://doi.org/10.1198/106186006X137047

Keuleers, E., & Brysbaert, M. (2010). Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3), 627–633. https://doi.org/10.3758/BRM.42.3.627

Kuperman, V. (2015). Virtual experiments in megastudies: A case study of language and emotion. Quarterly Journal of Experimental Psychology, 68(8), 1693–1710. https://doi.org/10.1080/17470218.2014.989865

Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177

Liben-Nowell, D., Strand, J., Sharp, A., Wexler, T., & Woods, K. (2019). The danger of testing by selecting controlled subsets, with applications to spoken-word recognition. Journal of Cognition, 2(1), Article 2. https://doi.org/10.5334/joc.51

Rubin, D. B. (1980). Bias reduction using Mahalanobis-metric matching. Biometrics, 36(2), 293–298. https://doi.org/10.2307/2529981

Sassenhagen, J., & Alday, P. M. (2016). A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests. Brain and Language, 162, 42–45. https://doi.org/10.1016/j.bandl.2016.08.001

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313

van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. Quarterly Journal of Experimental Psychology, 67(6), 1176–1190. https://doi.org/10.1080/17470218.2013.850521

Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017/S0140525X20001685

Yarkoni, T., Balota, D., & Yap, M. (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin & Review, 15(5), 971–979. https://doi.org/10.3758/PBR.15.5.971