lexsync: From a word-frequency corpus to an EEG-ready experiment

Put the Design in Data

The design below asks for 40 words in each frequency condition, matched on length, neighbourhood density and orthographic Levenshtein distance 20 (OLD20). Those three controls and frequency itself are then balanced across two presentation lists. Frequency is expressed on the Zipf scale (van Heuven et al., 2014), and OLD20 measures orthographic similarity as the mean edit distance between a word and its 20 nearest neighbours (Yarkoni et al., 2008).

design <- list(
  name = 'frequency_eeg',
  language = 'english',
  n_per_condition = 40,
  pool_filters = list(length = c(3, 7), frequency = c(3.8, 7.0)),
  conditions = list(
    list(name = 'high_frequency', define_by = list(frequency = c(5.2, 7.0))),
    list(name = 'low_frequency', define_by = list(frequency = c(3.8, 4.4)))
  ),
  match_on = c('length', 'n_density', 'old20'),
  counterbalance = list(
    lists = 2,
    optimise = TRUE,
    balance_on = c('length', 'n_density', 'old20', 'frequency')
  )
)

pool <- build_pool(lexicon, design$pool_filters)
candidate_counts <- vapply(
  design$conditions,
  function(condition) nrow(build_pool(pool, condition$define_by)),
  integer(1)
)
names(candidate_counts) <- vapply(design$conditions, `[[`, character(1), 'name')
candidate_counts

#> high_frequency  low_frequency 
#>             71            480

The high-frequency band is the smaller candidate set, so it limits both the number and the quality of possible matches. Counting the candidates before matching shows whether a later shortfall stems from the size of this pool.

Select and Check the Words

match_stimuli() spreads anchor words evenly across the first condition in the design, here the smaller high-frequency band, and finds nearby words from the other condition in the standardised space of the control variables. Standardising first means that a given distance counts the same whichever control it falls on.

stimuli <- match_stimuli(pool, design, schema)
head(stimuli[, c('set', 'condition', 'word', 'frequency', 'length',
                  'n_density', 'old20')])

#>   set      condition    word frequency length n_density old20
#> 1   1 high_frequency    knew      5.20      4         3  1.75
#> 2   2 high_frequency    date      5.22      4        17  1.00
#> 3   3 high_frequency  energy      5.23      6         0  2.60
#> 4   4 high_frequency  points      5.23      6         5  1.65
#> 5   5 high_frequency running      5.23      7         3  1.80
#> 6   6 high_frequency    term      5.23      4         4  1.65

The report treats matching as a claim of equivalence. For each control, it gives Cohen’s \(d\) with a 90% confidence interval and applies the two one-sided tests (TOST) procedure against the schema’s bound of half a standard deviation (Lakens, 2017). The frequency row reflects the intended manipulation, so Figure 1 leaves it out.

report <- match_report(
  stimuli,
  c('frequency', 'length', 'n_density', 'old20'),
  schema
)
report$comparisons

#>       condition      reference dimension cohens_d d_ci_low d_ci_high var_ratio tost_p
#> 1 low_frequency high_frequency frequency    5.642    5.270     6.014     0.293 1.0000
#> 2 low_frequency high_frequency    length    0.000   -0.372     0.372     1.000 0.0141
#> 3 low_frequency high_frequency n_density    0.076   -0.296     0.448     0.632 0.0308
#> 4 low_frequency high_frequency     old20    0.016   -0.356     0.389     0.843 0.0168
#>   equivalent
#> 1      FALSE
#> 2       TRUE
#> 3       TRUE
#> 4       TRUE

controls <- subset(report$comparisons, dimension != 'frequency')
controls$dimension <- factor(
  controls$dimension,
  levels = rev(c('length', 'n_density', 'old20')),
  labels = rev(c('Word length', 'Neighbourhood density', 'OLD20'))
)
bound <- schema$equivalence$bound_d

ggplot(controls, aes(cohens_d, dimension)) +
  annotate('rect', xmin = -bound, xmax = bound, ymin = -Inf, ymax = Inf,
           fill = '#E8F1F5') +
  geom_vline(xintercept = 0, colour = '#66747B', linewidth = 0.5) +
  geom_pointrange(aes(xmin = d_ci_low, xmax = d_ci_high),
                  colour = '#006D77', linewidth = 0.8, size = 0.7) +
  scale_x_continuous(
    breaks = c(-bound, 0, bound), expand = expansion(0),
    labels = scales::label_number(accuracy = 0.1, style_negative = 'minus')
  ) +
  coord_cartesian(xlim = c(-1, 1) * bound * 1.12) +
  labs(x = 'Standardised difference (high minus low), 90% CI', y = NULL) +
  theme_minimal(base_size = 12) +
  theme(panel.grid = element_blank(),
        axis.title.x = element_text(size = 10.5, margin = margin(t = 6)),
        axis.text.x = element_text(size = 10),
        axis.text.y = element_text(size = 11, hjust = 0, colour = 'grey20',
                                   margin = margin(r = 10)),
        plot.margin = margin(8, 10, 8, 8))

Figure 1: Realised Balance Between the Frequency Conditions on Each Control Variable, With 90% Confidence Intervals Against the Prespecified Equivalence Region. Shading marks the equivalence region set in the schema, from minus to plus half a standard deviation. CI stands for confidence interval, and OLD20 for orthographic Levenshtein distance 20.

The intervals deserve more attention than the point estimates, because they carry the uncertainty. With few items per condition, an estimate near zero can still be too imprecise to support a claim of equivalence.

Carry the Items Into the Experiment

The design also describes each trial, which presents a fixation cross, the word with a condition trigger locked to its onset, a response window and a blank interval of jittered length. The jitter is derived from a keyed hash of the design and seed, so rebuilding the experiment reproduces the same timing.

design$events <- list(
  list(type = 'fixation', content = '+', duration_ms = 500),
  list(type = 'text', content = '{word}', duration_ms = 800,
       trigger = 'condition', onset_locked = TRUE),
  list(type = 'response', keys = c('f', 'j'), timeout_ms = 2000),
  list(type = 'blank', duration = list(jitter = c(400, 800), as = 'iti_ms'))
)

list_plan <- balance_lists(stimuli, design, schema)
trials <- counterbalance(stimuli, design, schema, list_plan$list_of_set)
trials <- resolve_trial_timing(trials, design, schema)
trials <- assign_triggers(trials)

c(cost_before = list_plan$report$cost_before,
  cost_after = list_plan$report$cost_after,
  swaps = list_plan$report$n_swaps)

#> cost_before  cost_after       swaps 
#>      743080       17880           5

head(trials[, c('list', 'trial', 'condition', 'word', 'iti_ms',
                'condition_trigger')])

#>   list trial      condition    word iti_ms condition_trigger
#> 1    1     1  low_frequency approve    696               101
#> 2    1     2 high_frequency    knew    504               102
#> 3    1     3 high_frequency  mother    643               102
#> 4    1     4  low_frequency    loop    688               101
#> 5    1     5 high_frequency    lost    489               102
#> 6    1     6  low_frequency   ghana    498               101

The imbalance cost sums each list’s departure from its fair share on every balanced variable, scaled to that variable’s mean. Each swap exchanged a matched word pair in one list for a pair in the other.

All three exported files read from this trial table. In the PsychoPy script, win.callOnFlip sends the trigger on the refresh that presents the word, and TRIGGER_HOLD_MS sets how long the trigger code is held before the port is reset to zero.

experiments <- export_experiments(trials, design, schema, output_dir)
basename(unlist(experiments))

#> [1] "frequency_eeg_english_psychopy.py" "frequency_eeg_english.osexp"      
#> [3] "frequency_eeg_english.html"

psychopy_code <- readLines(experiments$psychopy, warn = FALSE)
grep('^\\s*(TRIGGER_HOLD_MS\\s*=|win\\.callOnFlip\\()', psychopy_code,
     value = TRUE, perl = TRUE)

#> [1] "TRIGGER_HOLD_MS = 50"                                 
#> [2] "        win.callOnFlip(port.setData, trigger)"        
#> [3] "            win.callOnFlip(port.setData, 0)"          
#> [4] "                win.callOnFlip(port.setData, trig)"   
#> [5] "    win.callOnFlip(port.setData, BLOCK_START_TRIGGER)"
#> [6] "    win.callOnFlip(port.setData, BLOCK_END_TRIGGER)"

Limits

Matching can only draw on what the candidate corpus contains. The bundled English lexicon takes its words and Zipf frequencies from wordfreq (Speer, 2022). It serves this demonstration, but it includes proper nouns, estimates syllables from spelling and has no part-of-speech information. A study should therefore supply its own exclusions and whatever norms it needs. Equivalence also depends on the bound chosen before selection, and the default of half a standard deviation is only a starting point. Each study should derive its own bound from the smallest difference that would matter for its research question (Lakens, 2017).

The R documentation and Python documentation cover pseudoword generation, semantic norms, continuous designs, Latin-square counterbalancing and the full file-based pipeline. Across the package, the design records the experiment as intended, while the generated tables and reports record it as realised.