API reference

Every public name in lexsync is documented here, grouped along the path a study takes: reach a corpus, derive the dimensions, build a pool and match it, describe a trial, counterbalance it, report what the matching achieved, generate the experiment, and write down the provenance. The groups are the same ones the R package's reference index uses, so a name can be found in the same place on either site.

Everything listed under a group heading is importable straight from lexsync, with a handful of exceptions that are noted where they appear and are reached through their own module. Guides with worked examples are linked from the home page, and the published work that these entries cite is listed in full on the references page.

Corpora and lexica

Languages are supplied through a corpus registry, so reaching a new one takes a registry entry and no code. These functions find a corpus, download it into a per-user cache that can be cleared again, turn a corpus file in its publisher's layout into a lexicon, and read a derived lexicon or a prepared item table into the frame everything else expects.

lexsync.list_corpora(registry_path=None)

Source code in src/lexsync/corpora.py

def list_corpora(registry_path: str | None = None) -> pd.DataFrame:
    with open(_registry_path(registry_path), encoding="utf-8") as handle:
        reg = yaml.safe_load(handle)
    rows = []
    for name, entry in (reg.get("corpora") or {}).items():
        lang = entry.get("language") or {}
        rows.append(dict(
            name=name, language=lang.get("name"), iso=lang.get("iso"),
            status=entry.get("status"), connector=entry.get("connector", "openlexicon"),
            citation=entry.get("citation"),
        ))
    return pd.DataFrame(rows)

lexsync.fetch_corpus(name, registry_path=None, n_words=10000, *, strict=False, exclude_latin=False)

Fetch a registered corpus into the cache.

name must be a single corpus name: one holding '/', '\' or ':' is refused before the registry is read, since the name becomes the cached file's name.

If name is a language code supported by the wordfreq connector, a lexicon is built with wordfreq from up to n_words of its most frequent forms that are spelt with letters. A form may hold combining marks, such as the vowel signs of Hindi and Bengali, and the zero-width non-joiner of Persian. It may also hold an apostrophe inside a Ukrainian word or a middle dot between two l's in a Catalan one, but no digit and no other punctuation mark. A lexicon left with fewer forms is reported on standard output, or refused with strict, and exclude_latin drops every form holding a Latin letter. These three arguments apply to the wordfreq connector only, and lexsync.corpora.build_wordfreq_lexicon gives the rule in full.

For any other name the registry entry's file is downloaded, once its URL's scheme has been checked. An entry's derived file, a lexicon already normalised, is fetched when it has one and cached as it arrives. Otherwise its openlexicon file is fetched and converted by :func:normalise_lexicon with the entry's columns and duplicates, so fetch_corpus("subtlex_us") caches a lexicon that :func:lexsync.load_lexicon reads. The transfer lands in a sidecar file that is renamed into the cache only after the size cap, the markup sniff, any registered sha256 and the normalisation have all passed. Every refusal names the registry file it read. A registry_path that does not exist is refused.

The file lands in :func:cache_dir. The cache directory is created only once the registry entry and its URL have been accepted, or the wordfreq lexicon built, so a refused call writes nothing. Each fetch into the cache begins by deleting any .part sidecar that an interrupted transfer left there more than a day earlier. The cache persists between sessions, and one corpus may reach the 200 MB download cap, so several of them add up. :func:cache_clear removes one corpus or the whole cache, and the next call fetches the corpus again.

Source code in src/lexsync/corpora.py

def fetch_corpus(name: str, registry_path: str | None = None, n_words: int = 10000, *,
                 strict: bool = False, exclude_latin: bool = False) -> str:
    """Fetch a registered corpus into the cache.

    `name` must be a single corpus name: one holding '/', '\\' or ':' is refused
    before the registry is read, since the name becomes the cached file's name.

    If `name` is a language code supported by the wordfreq connector, a lexicon
    is built with wordfreq from up to `n_words` of its most frequent forms that
    are spelt with letters. A form may hold combining marks, such as the vowel
    signs of Hindi and Bengali, and the zero-width non-joiner of Persian. It may
    also hold an apostrophe inside a Ukrainian word or a middle dot between two
    l's in a Catalan one, but no digit and no other punctuation mark. A lexicon
    left with fewer forms is reported on standard output, or refused with
    `strict`, and `exclude_latin` drops every form holding a Latin letter. These
    three arguments apply to the wordfreq connector only, and
    ``lexsync.corpora.build_wordfreq_lexicon`` gives the rule in full.

    For any other name the registry entry's file is downloaded, once its URL's scheme
    has been checked. An entry's ``derived`` file, a lexicon already normalised, is
    fetched when it has one and cached as it arrives. Otherwise its ``openlexicon``
    file is fetched and converted by :func:`normalise_lexicon` with the entry's
    ``columns`` and ``duplicates``, so ``fetch_corpus("subtlex_us")`` caches a lexicon
    that :func:`lexsync.load_lexicon` reads. The transfer lands in a sidecar file that
    is renamed into the cache only after the size cap, the markup sniff, any registered
    sha256 and the normalisation have all passed. Every refusal names the registry
    file it read. A `registry_path` that does not exist is refused.

    The file lands in :func:`cache_dir`. The cache directory is created only once
    the registry entry and its URL have been accepted, or the wordfreq lexicon
    built, so a refused call writes nothing. Each fetch into the cache begins by
    deleting any ``.part`` sidecar that an interrupted transfer left there more
    than a day earlier. The cache persists between sessions, and one corpus may
    reach the 200 MB download cap, so several of them add up. :func:`cache_clear`
    removes one corpus or the whole cache, and the next call fetches the corpus
    again.
    """
    _check_corpus_name(name)
    reg_path, reg = _read_registry(registry_path)
    wf = reg.get("wordfreq_connector") or {}
    if name in (wf.get("languages") or []):
        df = build_wordfreq_lexicon(name, n_words, strict=strict, exclude_latin=exclude_latin)
        dest = os.path.join(_prepare_cache(), f"{name}_wordfreq.csv")
        # LF explicitly: pandas otherwise follows os.linesep, and the cached
        # lexicon's bytes (and any checksum of them) must not depend on the OS.
        df.to_csv(dest, index=False, encoding="utf-8", lineterminator="\n")
        print(f"lexsync: built '{name}' via wordfreq. Please cite: {wf.get('citation', '')}")
        return dest
    entry = (reg.get("corpora") or {}).get(name)
    if not entry:
        raise ValueError(f"lexsync: corpus '{name}' is not in the registry '{reg_path}'.")
    url, expected, normalise = _download_source(name, entry, reg_path)
    # A registry is editable, and fetch_corpus() writes wherever it points, so a
    # 'file://' or 'ftp://' entry would read a local path under the guise of a
    # download. Only the two schemes a corpus is published over are honoured.
    if not re.match(r"https?://", url, re.IGNORECASE):
        raise ValueError(
            f"lexsync: corpus '{name}' registers a non-http(s) URL ({url}) in "
            f"'{reg_path}'; refusing to download."
        )
    import urllib.error
    import urllib.request
    # Only now, with every refusal behind it, does the call touch the disk.
    dest = os.path.join(_prepare_cache(), f"{name}.csv")
    # The transfer lands in a sidecar and is renamed over `dest` only after every
    # check below has passed, so a truncated or unverified body can never sit at
    # the cache path, where a later run would trust it.
    part = dest + ".part"
    try:
        # 60 s guards against a stalled server, which urlretrieve() would wait on
        # forever; the R engine's download.file() honours options(timeout).
        with urllib.request.urlopen(url, timeout=60) as response, \
                open(part, "wb") as handle:
            received = 0
            for chunk in iter(lambda: response.read(65536), b""):
                received += len(chunk)
                if received > _max_download_bytes():
                    raise ValueError(
                        "lexsync: corpus download exceeded the 200 MB size limit. "
                        "Retrieve the delimited file manually and pass it to "
                        "load_lexicon()."
                    )
                handle.write(chunk)
    except (urllib.error.URLError, OSError) as exc:
        if os.path.exists(part):
            os.remove(part)
        raise RuntimeError(
            f"lexsync: could not download corpus '{name}' from {url} ({exc}). "
            f"Check the URL in '{reg_path}', or download the file manually, convert it "
            f"with normalise_lexicon() and pass the result to load_lexicon()."
        ) from exc
    except ValueError:
        if os.path.exists(part):
            os.remove(part)
        raise
    if _starts_with_markup(part):
        os.remove(part)
        raise ValueError(
            f"lexsync: corpus '{name}' returned an HTML page, not a delimited file "
            f"({url}); the registry URL may have rotted. Retrieve the delimited file "
            f"manually and pass it to load_lexicon()."
        )
    # 'sha256' is optional per registry entry; when present the download must
    # match it before it may enter the cache.
    if expected and sha256_file(part) != expected:
        os.remove(part)
        raise ValueError(
            f"lexsync: checksum mismatch for corpus '{name}'; the download does "
            f"not match the registry's sha256. Retry the download, or verify the "
            f"sha256 recorded in '{reg_path}'."
        )
    if normalise:
        # The corpus file is read into memory and its sidecar then holds the lexicon,
        # so a file that cannot be normalised leaves nothing behind either.
        try:
            # The file's name and extension, without any query or fragment.
            named = re.sub(r"[?#].*$", "", url)
            lexicon, message = _normalise(part, None, entry, _delimiter_for(named),
                                          os.path.basename(named))
        finally:
            os.remove(part)
        print(message)
        write_csv_utf8(lexicon, part)
    os.replace(part, dest)
    print(f"lexsync: downloaded '{name}'. Please cite: {entry.get('citation', '(see registry)')}")
    return dest

lexsync.normalise_lexicon(path, columns=None, entry=None, delimiter=None, out=None)

Turn a corpus file in its publisher's layout into a lexicon load_lexicon reads.

Reads a tab- or comma-separated file. delimiter names the separator, "," or "\t". Left as None, it is taken from the extension (.tsv and .tab are tab, .csv is comma) and otherwise from the header line, which is tab-separated when it holds a tab. A tab-separated file is read with no quoting.

columns maps lexsync's column names to the file's, as a registry entry's columns does: word, one of freq_zipf and freq_permillion, and optionally pos. entry is a registry entry, or the name of one in the registry lexsync finds by default (see :func:list_corpora), and supplies the map when columns is None and the duplicates rule. Without a map the file's own word, freq_zipf or freq_permillion, and pos columns are read.

Words are trimmed, lower-cased and put in Unicode NFC, as load_lexicon keys them, and rows missing a word or a frequency are dropped. A form that then appears more than once is resolved by duplicates: first (the default) keeps the first row in file order, and sum adds its rows' frequencies per million, as Lexique needs, since it splits a homograph's frequency between its rows. Frequencies per million are converted with Zipf = log10(per million) + 3, rounded to three places by lexsync's shared rounder. A value of 0 or below has no Zipf value and is refused. Zipf values read as they are must lie between 0 and 9.

The lexicon has the columns word, freq_zipf, then freq_permillion when Zipf was derived from it, and pos when the file has one, in the UTF-8 byte order of word. With out it is also written there, comma-separated with LF line endings. A line on standard output gives the number of forms, and for derived Zipf values the number that lay within 1e-9 of a rounding boundary after scaling. That is the only step at which the two engines' log10 could round a value apart, and the count is expected to be 0. attrs["normalisation"] holds {"n_read", "n_forms", "n_merged", "duplicates", "zipf_derived", "near_boundary"}, and a run over the lexicon records the boundary count in its run log and datasheet.

Mirrors normalise_lexicon in R_workflow/R/corpora.R, which writes the same bytes.

Source code in src/lexsync/corpora.py

def normalise_lexicon(path: str, columns=None, entry=None, delimiter=None,
                      out: str | None = None) -> pd.DataFrame:
    """Turn a corpus file in its publisher's layout into a lexicon load_lexicon reads.

    Reads a tab- or comma-separated file. ``delimiter`` names the separator, "," or
    "\\t". Left as None, it is taken from the extension (.tsv and .tab are tab, .csv is
    comma) and otherwise from the header line, which is tab-separated when it holds a
    tab. A tab-separated file is read with no quoting.

    ``columns`` maps lexsync's column names to the file's, as a registry entry's
    ``columns`` does: ``word``, one of ``freq_zipf`` and ``freq_permillion``, and
    optionally ``pos``. ``entry`` is a registry entry, or the name of one in the
    registry lexsync finds by default (see :func:`list_corpora`), and supplies the map
    when ``columns`` is None and the ``duplicates`` rule. Without a map the file's own
    ``word``, ``freq_zipf`` or ``freq_permillion``, and ``pos`` columns are read.

    Words are trimmed, lower-cased and put in Unicode NFC, as load_lexicon keys them,
    and rows missing a word or a frequency are dropped. A form that then appears more
    than once is resolved by ``duplicates``: ``first`` (the default) keeps the first
    row in file order, and ``sum`` adds its rows' frequencies per million, as Lexique
    needs, since it splits a homograph's frequency between its rows. Frequencies per
    million are converted with Zipf = log10(per million) + 3, rounded to three places
    by lexsync's shared rounder. A value of 0 or below has no Zipf value and is
    refused. Zipf values read as they are must lie between 0 and 9.

    The lexicon has the columns ``word``, ``freq_zipf``, then ``freq_permillion`` when
    Zipf was derived from it, and ``pos`` when the file has one, in the UTF-8 byte
    order of ``word``. With ``out`` it is also written there, comma-separated with LF
    line endings. A line on standard output gives the number of forms, and for derived
    Zipf values the number that lay within 1e-9 of a rounding boundary after scaling.
    That is the only step at which the two engines' log10 could round a value apart,
    and the count is expected to be 0. ``attrs["normalisation"]`` holds ``{"n_read",
    "n_forms", "n_merged", "duplicates", "zipf_derived", "near_boundary"}``, and a run
    over the lexicon records the boundary count in its run log and datasheet.

    Mirrors normalise_lexicon in R_workflow/R/corpora.R, which writes the same bytes.
    """
    frame, message = _normalise(path, columns, entry, delimiter, os.path.basename(str(path)))
    print(message)
    if out is not None:
        write_csv_utf8(frame, out)
    return frame

lexsync.corpora.build_wordfreq_lexicon(language, n_words=10000, min_len=2, max_len=15, *, strict=False, exclude_latin=False)

Build a (word, freq_zipf) lexicon for any wordfreq language.

Takes wordfreq's most frequent forms for language, lower-cased and in Unicode NFC, until n_words of them pass the letter filter and are min_len to max_len characters long. It looks no further than the first 3 * n_words forms. A form passes when it opens with a letter and every other character is a letter, a combining mark or a zero-width non-joiner or joiner. Hindi है and Bengali না, spelt with combining vowel signs, therefore pass, as do the many Persian forms spelt with a non-joiner. A Ukrainian or Belarusian form may also hold an apostrophe between two letters (п'ять), and a Catalan one a middle dot between two l's (pel·lícula). Digits never pass, and nor do underscores or any punctuation mark besides those joiners, so English contractions and French elisions stay out. Length counts code points, as the lexicon readers count it, so vowel signs, viramas and zero-width characters count. A form that wordfreq gives no frequency is dropped.

When the lexicon holds fewer than n_words forms, a line on standard output says so, and with strict the shortfall raises ValueError. exclude_latin drops every form holding a Latin letter, such as the English tokens in the Hindi list. It is not a one-script rule, so a Japanese lexicon keeps its kana and kanji together, and it leaves nothing of a language written in the Latin alphabet.

Raises ModuleNotFoundError if the optional [corpora] extra is not installed.

Source code in src/lexsync/corpora.py

def build_wordfreq_lexicon(language: str, n_words: int = 10000,
                           min_len: int = 2, max_len: int = 15, *,
                           strict: bool = False,
                           exclude_latin: bool = False) -> pd.DataFrame:
    """Build a (word, freq_zipf) lexicon for any wordfreq language.

    Takes wordfreq's most frequent forms for `language`, lower-cased and in Unicode
    NFC, until `n_words` of them pass the letter filter and are `min_len` to
    `max_len` characters long. It looks no further than the first 3 * `n_words`
    forms. A form passes when it opens with a letter and every other character is a
    letter, a combining mark or a zero-width non-joiner or joiner. Hindi है and
    Bengali না, spelt with combining vowel signs, therefore pass, as do the many
    Persian forms spelt with a non-joiner. A Ukrainian or Belarusian form may also
    hold an apostrophe between two letters (п'ять), and a Catalan one a middle dot
    between two l's (pel·lícula). Digits never pass, and nor do underscores or any
    punctuation mark besides those joiners, so English contractions and French
    elisions stay out. Length counts code points, as the lexicon readers count it,
    so vowel signs, viramas and zero-width characters count. A form that wordfreq
    gives no frequency is dropped.

    When the lexicon holds fewer than `n_words` forms, a line on standard output says
    so, and with `strict` the shortfall raises ValueError. `exclude_latin` drops every
    form holding a Latin letter, such as the English tokens in the Hindi list. It is
    not a one-script rule, so a Japanese lexicon keeps its kana and kanji together,
    and it leaves nothing of a language written in the Latin alphabet.

    Raises ModuleNotFoundError if the optional [corpora] extra is not installed.
    """
    # Deferred because wordfreq belongs to the [corpora] extra, so importing it at
    # module scope would break `import lexsync` on a default install. Guarded because
    # this is what `lexsync fetch fr` reaches, and the README advertises that command:
    # unguarded, a default install answers it with a bare ModuleNotFoundError nine
    # frames down, which names the missing module and no way to get it. Everything
    # else in this module answers a failure with the remedy, and so does this now.
    # The R twin has no counterpart here: Connector B is Python-only, and R/corpora.R
    # reads its pre-derived output as an ordinary lexicon.
    try:
        import wordfreq
    except ImportError as exc:
        raise ModuleNotFoundError(
            f"lexsync: building a lexicon for '{language}' needs the wordfreq connector, "
            f"which ships in the optional [corpora] extra and is not installed. Install "
            'it with: pip install "wordfreq>=3.0,<4".'
        ) from exc

    words: list[str] = []
    seen: set[str] = set()
    for token in wordfreq.top_n_list(language, n_words * 3):
        # NFC after lower-casing, the order in which load_lexicon normalises a word. A
        # form that wordfreq gives decomposed, as it gives some Greek ones, then keeps
        # its marks, is stored as the readers will key it and is kept only once.
        w = unicodedata.normalize("NFC", token.lower())
        if (w in seen or not min_len <= len(w) <= max_len
                or not _spelt_with_letters(w, language)
                or (exclude_latin and _has_latin_letter(w))):
            continue
        seen.add(w)
        words.append(w)
        if len(words) >= n_words:
            break
    zipf = [wordfreq.zipf_frequency(w, language) for w in words]
    df = pd.DataFrame({"word": words, "freq_zipf": np.round(zipf, 3),
                       "language": language, "source": "wordfreq"})
    df = df[df.freq_zipf > 0].sort_values("word").reset_index(drop=True)
    if len(df) < n_words:
        # A short lexicon thins every frequency band a design draws on, and a Hindi one
        # used to come back with 3,207 of the 10,000 forms asked for, and nothing said so.
        message = (f"lexsync: wordfreq supplied only {len(df)} forms for '{language}' "
                   f"that pass the letter filter (n_words was {n_words}).")
        if strict:
            raise ValueError(message)
        print(message)
    return df

lexsync.corpora.cache_dir()

Per-user cache directory for fetched corpora.

Where :func:fetch_corpus puts what it fetches, and the only place the package writes to without being handed a path. This function only reports the path. The directory is created by the first fetch into it, so until then it does not exist.

The cache persists between sessions. A registered corpus is a delimited word list, and a download is refused above 200 MB, so a cache holding several large corpora can reach a few hundred megabytes. Each fetch into the cache first deletes any partial download that an interrupted transfer left there more than a day earlier, and fetching a corpus again replaces the earlier copy. Fetched corpora otherwise stay until :func:cache_clear removes them, one corpus at a time or all together. Nothing kept there is irreplaceable, so the next call downloads afresh. The R twin documents the same contract in lexsync_cache_dir.Rd; only the location differs, since R uses tools::R_user_dir.

Source code in src/lexsync/corpora.py

def cache_dir() -> str:
    """Per-user cache directory for fetched corpora.

    Where :func:`fetch_corpus` puts what it fetches, and the only place the
    package writes to without being handed a path. This function only reports
    the path. The directory is created by the first fetch into it, so until then
    it does not exist.

    The cache persists between sessions. A registered corpus is a delimited word
    list, and a download is refused above 200 MB, so a cache holding several
    large corpora can reach a few hundred megabytes. Each fetch into the cache
    first deletes any partial download that an interrupted transfer left there
    more than a day earlier, and fetching a corpus again replaces the earlier
    copy. Fetched corpora otherwise stay until :func:`cache_clear` removes them,
    one corpus at a time or all together. Nothing kept there is irreplaceable, so
    the next call downloads afresh. The R twin documents the same contract in
    lexsync_cache_dir.Rd; only the location differs, since R uses
    tools::R_user_dir.
    """
    return os.path.join(os.path.expanduser("~"), ".lexsync", "cache")

lexsync.corpora.cache_clear(name=None)

Remove fetched corpora from the cache.

Deletes what :func:fetch_corpus put in :func:cache_dir. Given a name, that is the corpus's <name>.csv, any <name>.csv.part that an interrupted download left behind and the <name>_wordfreq.csv lexicon the wordfreq connector builds. With name left as None it is the whole cache directory, and the ~/.lexsync directory above it once nothing else is left there. A corpus needed again is fetched afresh by the next :func:fetch_corpus call. Mirrors lexsync_cache_clear() in R_workflow/R/corpora.R, whose engine has no wordfreq connector and so no _wordfreq.csv to remove.

Returns the paths of the files removed, empty when there was nothing to remove. Raises OSError, naming them, when files survive the attempt (one held open on Windows, say).

Source code in src/lexsync/corpora.py

def cache_clear(name: str | None = None) -> list[str]:
    """Remove fetched corpora from the cache.

    Deletes what :func:`fetch_corpus` put in :func:`cache_dir`. Given a `name`,
    that is the corpus's ``<name>.csv``, any ``<name>.csv.part`` that an
    interrupted download left behind and the ``<name>_wordfreq.csv`` lexicon the
    wordfreq connector builds. With `name` left as None it is the whole cache
    directory, and the ``~/.lexsync`` directory above it once nothing else is left
    there. A corpus needed again is fetched afresh by the next
    :func:`fetch_corpus` call. Mirrors lexsync_cache_clear() in
    R_workflow/R/corpora.R, whose engine has no wordfreq connector and so no
    ``_wordfreq.csv`` to remove.

    Returns the paths of the files removed, empty when there was nothing to
    remove. Raises OSError, naming them, when files survive the attempt (one held
    open on Windows, say).
    """
    cache = cache_dir()
    if name is None:
        if not os.path.isdir(cache):
            return []
        removed = sorted(os.path.join(root, fname)
                         for root, _dirs, files in os.walk(cache) for fname in files)
        targets = [cache]
        # Keep going past a file that cannot be deleted, as R's unlink() does, and
        # report what survives below.
        shutil.rmtree(cache, ignore_errors=True)
    else:
        _check_corpus_name(name)
        targets = [os.path.join(cache, f"{name}{suffix}")
                   for suffix in (".csv", ".csv.part", "_wordfreq.csv")]
        removed = [path for path in targets if os.path.isfile(path)]
        for path in removed:
            with contextlib.suppress(OSError):
                os.remove(path)
    left = [path for path in (targets if name is None else removed)
            if os.path.exists(path)]
    if left:
        raise OSError(f"lexsync: could not remove {', '.join(left)} from the cache; "
                      f"check that no other process is using it.")
    if name is None:
        # ~/.lexsync holds nothing but the cache, so an emptied one would be left
        # behind in the user's home. rmdir() removes it only when it is empty. R's
        # twin has no such step: the parent of its cache directory is shared by
        # every R package's cache.
        with contextlib.suppress(OSError):
            os.rmdir(os.path.dirname(cache))
    return removed

lexsync.load_lexicon(path, schema, language=None)

Load a derived lexicon and give it the inexpensive dimensions.

word is trimmed, lower-cased and normalised to Unicode NFC, and duplicates are then removed, keeping the first row in file order. When normalisation changed or merged any word, attrs["input_normalisation"] holds {"changed", "collapsed"}: the words it rewrote and the duplicate rows it removed.

A freq_zipf value outside 0 to 9 is refused: it is almost always a frequency per million in the wrong column (convert it with normalise_lexicon). A log count such as SUBTLEX-US's Lg10WF, log10(count + 1) and at most 6.33, passes the check and must still be converted, since it reads about 1.3 below Zipf. Mirrors load_lexicon in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def load_lexicon(path: str, schema: dict, language: str | None = None) -> pd.DataFrame:
    """Load a derived lexicon and give it the inexpensive dimensions.

    ``word`` is trimmed, lower-cased and normalised to Unicode NFC, and duplicates are
    then removed, keeping the first row in file order. When normalisation changed or
    merged any word, ``attrs["input_normalisation"]`` holds ``{"changed",
    "collapsed"}``: the words it rewrote and the duplicate rows it removed.

    A ``freq_zipf`` value outside 0 to 9 is refused: it is almost always a frequency per
    million in the wrong column (convert it with ``normalise_lexicon``). A log count such
    as SUBTLEX-US's ``Lg10WF``, log10(count + 1) and at most 6.33, passes the check and
    must still be converted, since it reads about 1.3 below Zipf. Mirrors load_lexicon
    in R_workflow/R/querying.R.
    """
    df = read_csv_utf8(path)
    validate_lexicon(df, schema)
    # `or {}` at every level: an empty `dimensions:` or `frequency:` key parses to
    # None here and to NULL in R, where `$` chains through it to the default. A
    # default argument only covers an ABSENT key, so the two engines answered the
    # same schema with a column name and an AttributeError.
    freq_col = ((schema.get("dimensions") or {}).get("frequency") or {}).get("column") or "freq_zipf"
    _require_numbers(df[freq_col], freq_col)
    # Filter before coercing: str() renders a missing word as the literal
    # string "nan", which survives both guards below, whereas the R engine (where
    # as.character(NA) stays NA) drops the row. Coercing first would therefore
    # desynchronise the two engines' row counts and shift every subsequent id.
    df = df[df["word"].notna() & df[freq_col].notna()].copy()
    df["word"] = _fold_words(df["word"])
    df = df[df["word"] != ""].copy()
    # The rows the lexicon keeps, before de-duplication, as the R engine checks them.
    zipf = df["freq_zipf"]
    if pd.api.types.is_numeric_dtype(zipf) and not pd.api.types.is_bool_dtype(zipf):
        _check_zipf_range(zipf)
    # NFC before de-duplication, so the two spellings of a word are one word and the
    # first row in file order is the one kept.
    df["word"], changed, collapsed = _nfc_count(df["word"])
    df = df.drop_duplicates(subset="word")
    # An empty selection is never what the caller meant, and it fails far from
    # here: pandas would hand back a well-formed frame of nothing, while the R
    # engine dies on base R's "replacement has 1 row, data has 0". Both engines
    # raise this message instead, while the path is still in scope.
    if df.empty:
        raise ValueError(
            f"lexsync: lexicon '{path}' has no usable rows: it is empty, "
            f"or every row is missing 'word' or '{freq_col}'."
        )
    # Sort by UTF-8 byte order so the lexicon order is locale-independent and
    # identical to the R engine (which uses a 'radix' byte-order sort).
    df = (df.assign(_k=df["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))
    df["id"] = np.arange(1, len(df) + 1)
    df["length"] = df["word"].str.len()
    df["n_syllables"] = _syllable_column(df["word"])
    df["frequency"] = df[freq_col].astype(float)
    if language is not None:
        df["language"] = language
    df = df.reset_index(drop=True)
    # Read by run_pipeline for the run log and the datasheet.
    record = _nfc_record(changed, collapsed)
    if record:
        df.attrs["input_normalisation"] = record
    return df

lexsync.load_items(path, required_fields)

Load a paradigm item table (prime-target pairs, sentences, …).

The table must carry an item identifier, a condition label and the paradigm's presented fields. Field values are validated (no control characters; bounded length) so a crafted item cannot corrupt the generated loop table or scripts. Items are mapped to a deterministic integer set id (byte order) so counterbalancing matches the corpus path and the two engines. The item id, the condition and the presented fields are normalised to Unicode NFC. When that changed any cell, attrs["input_normalisation"] holds {"changed", "collapsed"}, with collapsed always 0 since no row is removed.

Source code in src/lexsync/querying.py

def load_items(path: str, required_fields) -> pd.DataFrame:
    """Load a paradigm item table (prime-target pairs, sentences, …).

    The table must carry an ``item`` identifier, a ``condition`` label and the
    paradigm's presented fields. Field values are validated (no control
    characters; bounded length) so a crafted item cannot corrupt the generated
    loop table or scripts. Items are mapped to a deterministic integer ``set`` id
    (byte order) so counterbalancing matches the corpus path and the two engines.
    The item id, the condition and the presented fields are normalised to Unicode
    NFC. When that changed any cell, ``attrs["input_normalisation"]`` holds
    ``{"changed", "collapsed"}``, with ``collapsed`` always 0 since no row is removed.
    """
    parts = str(path).replace("\\", "/").split("/")
    if ".." in parts:
        raise ValueError("lexsync: items path must not contain '..'.")
    # The item id, the condition label and the paradigm's presented fields stay text.
    # Typed by their values, a column of response keys such as TRUE and FALSE would
    # become logical, and an `item` column of numbers would lose the zero of '01'. The
    # reader trims every cell and reads a blank, all-whitespace or 'NA' cell as
    # missing, in both engines (see read_csv_utf8).
    df = read_csv_utf8(path, as_character=["item", "condition"] + list(required_fields))
    needed = ["item", "condition"] + list(required_fields)
    missing = [c for c in needed if c not in df.columns]
    if missing:
        raise ValueError(f"lexsync: items table '{path}' is missing column(s): {', '.join(missing)}.")
    # Missingness is tested before any string coercion: str() would render a missing
    # cell as the literal 'nan', which a blank condition then carries past the
    # hash-key guard downstream.
    for col in needed:
        if df[col].isna().any():
            raise ValueError(
                f"lexsync: the items table has missing value(s) in column '{col}'; "
                "every item, condition and presented field must be filled.")
    df = df.copy()
    # The item id, the condition and the presented text in NFC, as every other text
    # key is, so one item spelt two ways is one set and its text one string. A merged
    # spelling removes no row here, so only the changed cells are counted.
    changed = 0
    for col in dict.fromkeys(needed):
        df[col], n_changed, _ = _nfc_count(df[col].astype(str))
        changed += n_changed
    item_key = df["item"].astype(str)
    # A repeated item-condition pair is a slip that would silently duplicate a trial
    # in every generated list; the first repeat in file order is reported.
    seen = set()
    for it, cond in zip(item_key, df["condition"], strict=True):
        if (it, cond) in seen:
            raise ValueError(
                f"lexsync: the items table repeats item '{it}' for condition '{cond}'; "
                "each item and condition pair may appear once.")
        seen.add((it, cond))
    for f in [c for c in required_fields if c in df.columns]:
        df[f] = [clean_field(v, f) for v in df[f]]
    items = sorted(item_key.unique(), key=lambda s: s.encode("utf-8"))
    set_map = {it: i + 1 for i, it in enumerate(items)}
    df["set"] = item_key.map(set_map)
    df = df.reset_index(drop=True)
    record = _nfc_record(changed, 0)
    if record:
        df.attrs["input_normalisation"] = record
    return df

lexsync.load_pool(path, schema, lexicon=None, language=None)

Load a supplied candidate pool of words and give it the matcher's dimensions.

A researcher who already has a curated word list (from a previous study, a norming session, a colleague) should not have to dress it up as a corpus lexicon to get lexsync's matching, validation and datasheet. This reads such a list and returns something the matcher accepts.

The list needs only a word column. Length and the syllable estimate are derived from the form. Everything else is either supplied on the list itself or looked up: with lexicon given, the corpus dimensions (frequency above all) are joined for those words, and a word the lexicon does not have is a hard error rather than a NaN, because the tolerance windows drop missing rows silently and the pool would then be smaller than the user believes it is.

The returned reference matters as much as the pool. n_density and old20 are properties of a word in its language, not among the handful of words a study happens to use, so computing them against a 200-word supplied list would give numbers that mean nothing. When a lexicon is given, the reference is the lexicon's words; only without one does it fall back to the pool itself. A reference_member column on whichever of the two supplies the reference keeps only the rows it marks 1.

Returns {"pool": DataFrame, "reference": list}. Words are trimmed, lower-cased and normalised to Unicode NFC as in load_lexicon, and the pool carries the same attrs["input_normalisation"] record when that changed or merged any word of the pool or the lexicon. Mirrors load_pool in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def load_pool(path: str, schema: dict, lexicon=None, language=None) -> dict:
    """Load a supplied candidate pool of words and give it the matcher's dimensions.

    A researcher who already has a curated word list (from a previous study, a norming
    session, a colleague) should not have to dress it up as a corpus lexicon
    to get lexsync's matching, validation and datasheet. This reads such a list and
    returns something the matcher accepts.

    The list needs only a ``word`` column. Length and the syllable estimate are derived
    from the form. Everything else is either supplied on the list itself or looked up:
    with ``lexicon`` given, the corpus dimensions (frequency above all) are joined for
    those words, and a word the lexicon does not have is a hard error rather than a
    NaN, because the tolerance windows drop missing rows silently and the pool would
    then be smaller than the user believes it is.

    The returned ``reference`` matters as much as the pool. ``n_density`` and ``old20``
    are properties of a word in its *language*, not among the handful of words a study
    happens to use, so computing them against a 200-word supplied list would give
    numbers that mean nothing. When a lexicon is given, the reference is the lexicon's
    words; only without one does it fall back to the pool itself. A
    ``reference_member`` column on whichever of the two supplies the reference keeps
    only the rows it marks 1.

    Returns ``{"pool": DataFrame, "reference": list}``. Words are trimmed, lower-cased
    and normalised to Unicode NFC as in ``load_lexicon``, and the pool carries the same
    ``attrs["input_normalisation"]`` record when that changed or merged any word of
    the pool or the lexicon. Mirrors load_pool in R_workflow/R/querying.R.
    """
    if ".." in str(path).replace("\\", "/").split("/"):
        raise ValueError("lexsync: pool path must not contain '..'.")
    df = read_csv_utf8(path)
    if "word" not in df.columns:
        raise ValueError(
            "lexsync: supplied pool '%s' is missing the required 'word' column." % path)
    # The same normalisation load_lexicon applies, for the same reason: `word` is the
    # canonical key behind every byte-order tie-break, so the two engines must fold and
    # trim it identically before anything is sorted or numbered by it.
    df = df[df["word"].notna()].copy()
    df["word"] = _fold_words(df["word"])
    df = df[df["word"] != ""].copy()
    df["word"], changed, collapsed = _nfc_count(df["word"])
    df = df.drop_duplicates(subset="word")
    if df.empty:
        raise ValueError(
            "lexsync: supplied pool '%s' has no usable rows: it is empty, or every row "
            "is missing 'word'." % path)
    df = (df.assign(_k=df["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))

    reference = None if lexicon else _neighbourhood_reference(df)
    if lexicon:
        lex = load_lexicon(lexicon, schema, language=language)
        # The lexicon's own normalisation is reported with the pool's, since the run
        # read both.
        lex_record = lex.attrs.get("input_normalisation") or {}
        changed += lex_record.get("changed", 0)
        collapsed += lex_record.get("collapsed", 0)
        reference = _neighbourhood_reference(lex)
        # `id` is the lexicon's own row number, meaningless once the pool is a subset.
        dims = [c for c in lex.columns if c not in ("word", "language", "source", "id")]
        clash = sorted((c for c in dims if c in df.columns),
                       key=lambda s: s.encode("utf-8"))
        if clash:
            raise ValueError(
                "lexsync: supplied pool '%s' already has column(s) %s, which the lexicon "
                "would also supply. Rename or drop them, so it is unambiguous which "
                "values the matcher used."
                % (path, ", ".join("'%s'" % c for c in clash)))
        known = set(lex["word"])
        absent = sorted((w for w in df["word"] if w not in known),
                        key=lambda s: s.encode("utf-8"))
        if absent:
            shown = absent[:5]
            raise ValueError(
                "lexsync: %d word(s) of supplied pool '%s' are absent from lexicon "
                "'%s': %s%s." % (len(absent), path, lexicon,
                                 ", ".join("'%s'" % w for w in shown),
                                 ", ..." if len(absent) > len(shown) else ""))
        by_word = lex.set_index("word")
        rows = by_word.loc[df["word"].tolist()]
        for d in dims:
            df[d] = rows[d].to_numpy()
    # Derived after any join, so a lexicon cannot overwrite them with its own copies.
    df["length"] = df["word"].str.len()
    df["n_syllables"] = _syllable_column(df["word"])
    if language is not None:
        df["language"] = language
    df["id"] = np.arange(1, len(df) + 1)
    df = df.reset_index(drop=True)
    record = _nfc_record(changed, collapsed)
    if record:
        df.attrs["input_normalisation"] = record
    return {"pool": df, "reference": reference}

lexsync.merge_norms(lexicon, norms, on='word', columns=None)

Left-join a norm table (e.g. concreteness, age of acquisition, valence).

norms is a data frame or the path to a CSV with a word column and one or more norm columns. This is the connector for semantic dimensions: the norm data themselves are fetched separately (licensing varies), then merged here so the matcher can equate on them.

The result is the lexicon itself with the norm columns appended, and the key is looked up positionally rather than through merge. That is what makes the two engines agree by construction, with nothing to repair afterwards, because merge and R's merge() were measured to diverge in three ways, each of them silent: R hoists the by column to position 1 while pandas keeps the left frame's order, so the column order differed whenever on was not already first; R disambiguates a colliding column name with .x/.y and pandas with _x/_y, and either way a dimension the design matches on disappears under a name nothing looks for; and R's merge(sort = FALSE) leaves the row order unspecified. A positional lookup has none of those degrees of freedom. A colliding name is now an error instead.

The key is trimmed, case-folded and normalised to NFC on both sides. Only the norm table's side was trimmed and case-folded before, so a lexicon holding Dog matched nothing and the design carried on with an all-NaN dimension. Because both engines agreed on that wrong answer, no parity test could have caught it. The lexicon's own spelling is preserved rather than folded in place: word is the byte-order tie-break behind every selection, so the join must not rewrite it.

Mirrors merge_norms in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def merge_norms(lexicon: pd.DataFrame, norms, on: str = "word", columns=None) -> pd.DataFrame:
    """Left-join a norm table (e.g. concreteness, age of acquisition, valence).

    ``norms`` is a data frame or the path to a CSV with a word column and one or
    more norm columns. This is the connector for semantic dimensions: the norm data
    themselves are fetched separately (licensing varies), then merged here so the
    matcher can equate on them.

    The result is the lexicon itself with the norm columns appended, and the key is
    looked up positionally rather than through ``merge``. That is what makes the two
    engines agree by construction, with nothing to repair afterwards, because ``merge``
    and R's ``merge()`` were measured to diverge in three ways, each of them silent:
    R hoists the ``by`` column to position 1 while pandas keeps the left frame's
    order, so the column order differed whenever ``on`` was not already first; R
    disambiguates a colliding column name with ``.x``/``.y`` and pandas with
    ``_x``/``_y``, and either way a dimension the design matches on disappears under
    a name nothing looks for; and R's ``merge(sort = FALSE)`` leaves the row order
    unspecified. A positional lookup has none of those degrees of freedom. A
    colliding name is now an error instead.

    The key is trimmed, case-folded and normalised to NFC on *both* sides. Only the
    norm table's side was trimmed and case-folded before, so a lexicon holding
    ``Dog`` matched nothing and the design carried on with an all-``NaN`` dimension.
    Because both engines agreed on that wrong answer, no parity test could have
    caught it. The lexicon's
    own spelling is preserved rather than folded in place: ``word`` is the
    byte-order tie-break behind every selection, so the join must not rewrite it.

    Mirrors merge_norms in R_workflow/R/querying.R.
    """
    n = norms if isinstance(norms, pd.DataFrame) else read_csv_utf8(norms)
    if on not in lexicon.columns:
        raise ValueError(
            "lexsync: merge_norms needs join column '%s' on the lexicon." % on)
    if on not in n.columns:
        raise ValueError(
            "lexsync: merge_norms needs join column '%s' on the norm table." % on)
    cols = list(columns) if columns else [c for c in n.columns if c != on]
    absent = [c for c in cols if c not in n.columns]
    if absent:
        raise ValueError("lexsync: the norm table has no column(s): %s."
                         % ", ".join("'%s'" % c for c in absent))
    # Silently renaming the clash, as both merges do, is the worst outcome
    # available: a design matching on `frequency` would find neither `frequency.x`
    # nor `frequency_x` and would fail far from the cause, or match on the norm
    # table's column believing it was the lexicon's.
    clash = [c for c in cols if c in lexicon.columns]
    if clash:
        raise ValueError(
            "lexsync: norm column(s) %s already exist on the lexicon. Rename them "
            "in the norm table, or name the ones you want in `columns`."
            % ", ".join("'%s'" % c for c in clash))
    key = _norm_key(n[on])
    keep = (key.notna() & ~key.duplicated()).to_numpy()
    add = n.loc[keep, cols].copy()
    add.index = pd.Index(key.to_numpy()[keep], dtype="object")
    # reindex on a de-duplicated index is a left join: a label that is absent
    # (including a missing lexicon key) yields NaN, exactly as R's match() gives NA.
    joined = add.reindex(_norm_key(lexicon[on]).to_numpy())
    out = lexicon.copy()
    for c in cols:
        out[c] = joined[c].to_numpy()
    return out

Lexical dimensions

Two dimensions arrive with the lexicon and the rest are derived from the orthographic forms. Derive them before matching, and compute the neighbourhood measures against the full lexicon rather than the pool, since a word's neighbours do not stop existing because a design excluded them.

lexsync.add_neighbourhood(df, reference=None, n_old=20)

Coltheart's N (same-length, single substitution) and OLD20 for each word.

The Levenshtein distances to the reference come from one rapidfuzz distance matrix per block of rows, which took 0.9 s for 500 words against the 30,000-word English lexicon where the word-by-word loop it replaces took 19 s. Between two forms of the same length a distance of 1 is a single substitution, so N counts the same-length reference words at distance 1, as the Hamming distance did. The distances are integers, so N and OLD20 are the values the loop gave and the R engine gives.

Source code in src/lexsync/querying.py

def add_neighbourhood(df: pd.DataFrame, reference=None, n_old: int = 20) -> pd.DataFrame:
    """Coltheart's N (same-length, single substitution) and OLD20 for each word.

    The Levenshtein distances to the reference come from one rapidfuzz distance matrix
    per block of rows, which took 0.9 s for 500 words against the 30,000-word English
    lexicon where the word-by-word loop it replaces took 19 s. Between two forms of the same
    length a distance of 1 is a single substitution, so N counts the same-length
    reference words at distance 1, as the Hamming distance did. The distances are
    integers, so N and OLD20 are the values the loop gave and the R engine gives.
    """
    words = df["word"].astype(str).tolist()
    ref = list(dict.fromkeys(str(w) for w in (reference if reference is not None else words)))
    ref_len = np.array([len(w) for w in ref])
    n_density = np.zeros(len(words), dtype=int)
    old = np.full(len(words), np.nan)
    for start in range(0, len(words) if ref else 0, _NEIGHBOURHOOD_CHUNK):
        chunk = words[start:start + _NEIGHBOURHOOD_CHUNK]
        dist = process.cdist(chunk, ref, scorer=Levenshtein.distance, dtype=np.int32,
                             workers=-1)
        for j, w in enumerate(chunk):
            row = dist[j]
            n_density[start + j] = int(np.sum((row == 1) & (ref_len == len(w))))
            ld = row[row > 0].astype(float)
            if ld.size:
                k = min(n_old, ld.size)
                old[start + j] = float(np.mean(np.partition(ld, k - 1)[:k]))
    out = df.copy()
    out["n_density"] = n_density
    out["old20"] = old
    return out

lexsync.add_bigram_frequency(df, reference=None)

Mean bigram probability (type-based, non-positional), a phonotactic-probability proxy.

For each word, the mean over its adjacent letter bigrams of the corpus bigram probability (count divided by the total bigram count). Computed from integer counts and rounded, so it is identical in the R and Python engines.

Source code in src/lexsync/querying.py

def add_bigram_frequency(df: pd.DataFrame, reference=None) -> pd.DataFrame:
    """Mean bigram probability (type-based, non-positional), a phonotactic-probability proxy.

    For each word, the mean over its adjacent letter bigrams of the corpus bigram
    probability (count divided by the total bigram count). Computed from integer
    counts and rounded, so it is identical in the R and Python engines.
    """
    ref = [str(w) for w in (reference if reference is not None else df["word"].tolist())]
    counts: dict[str, int] = {}
    total = 0
    for w in ref:
        for i in range(len(w) - 1):
            counts[w[i:i + 2]] = counts.get(w[i:i + 2], 0) + 1
            total += 1
    total = total or 1

    def bf(w):
        w = str(w)
        bgs = [w[i:i + 2] for i in range(len(w) - 1)]
        if not bgs:
            return 0.0
        return _round_dp(sum(counts.get(b, 0) for b in bgs) / len(bgs) / total, 9)

    out = df.copy()
    out["bigram_freq"] = out["word"].map(bf)
    return out

lexsync.add_pair_overlap(df, prime='prime', target='target')

Orthographic overlap between the two members of each pair.

Adds two columns. pair.lev is the Levenshtein distance between the pair's two orthographic forms, and pair.overlap is 1 - lev / max(len), the proportion of the longer form the two share. Overlap is the standard confound control in a priming design: a related pair that also shares letters confounds semantic relatedness with orthographic similarity.

Both engines return identical values, and the reasons are worth stating because they are the constraints on any future relational dimension. The core is an integer edit distance, and rapidfuzz's Levenshtein.distance and stringdist(method = "lv") agree exactly, including on decomposed Unicode and CJK, which is the same cross-library agreement add_neighbourhood already stakes old20 on. Length is counted in code points, len() and R's nchar() default, never in bytes. The arithmetic uses only - and /, which IEEE-754 mandates be correctly rounded, and the result is rounded to nine decimal places, the constant used everywhere else in the package. A degenerate pair of two empty forms returns 0 rather than 0/0, because a NaN would be sorted and compared and would then drop the row from one engine's control window but not the other's.

Source code in src/lexsync/querying.py

def add_pair_overlap(df: pd.DataFrame, prime: str = "prime",
                     target: str = "target") -> pd.DataFrame:
    """Orthographic overlap between the two members of each pair.

    Adds two columns. ``pair.lev`` is the Levenshtein distance between the pair's
    two orthographic forms, and ``pair.overlap`` is ``1 - lev / max(len)``, the
    proportion of the longer form the two share. Overlap is the standard confound
    control in a priming design: a related pair that also shares letters confounds
    semantic relatedness with orthographic similarity.

    Both engines return identical values, and the reasons are worth stating because
    they are the constraints on any future relational dimension. The core is an
    integer edit distance, and rapidfuzz's ``Levenshtein.distance`` and
    ``stringdist(method = "lv")`` agree exactly, including on decomposed Unicode and
    CJK, which is the same cross-library agreement ``add_neighbourhood`` already
    stakes ``old20`` on. Length is counted in code points, ``len()`` and R's
    ``nchar()`` default, never in bytes. The arithmetic uses only ``-`` and ``/``,
    which IEEE-754 mandates be correctly rounded, and the result is rounded to nine
    decimal places, the constant used everywhere else in the package. A degenerate
    pair of two empty forms returns 0 rather than ``0/0``, because a NaN would be
    sorted and compared and would then drop the row from one engine's control window
    but not the other's.
    """
    for col in (prime, target):
        if col not in df.columns:
            raise ValueError("lexsync: add_pair_overlap needs column '%s'." % col)
    a = [_nfc(str(w).strip().lower()) for w in df[prime]]
    b = [_nfc(str(w).strip().lower()) for w in df[target]]
    lev = [Levenshtein.distance(x, y) for x, y in zip(a, b, strict=True)]
    den = [max(len(x), len(y)) for x, y in zip(a, b, strict=True)]
    out = df.copy()
    out["pair.lev"] = np.array(lev, dtype=int)
    out["pair.overlap"] = [0.0 if d == 0 else _round_dp(1 - l / d, 9)
                           for l, d in zip(lev, den, strict=True)]
    return out

lexsync.count_syllables(word)

An orthographic syllable estimate: the number of maximal vowel runs.

Counts the maximal runs of Latin vowel letters after decomposing the word and setting its diacritics aside, so a vowel counts whatever its accent. The vowels of Polish, Czech, Hungarian, Romanian, Turkish, Danish, Norwegian and Vietnamese count as those of English, Spanish, French and German do. Hiatus (idea counts 2) and syllabic consonants such as the Czech l in vlk and r in krk stay undercounted.

Returns None for a missing value and for a word holding a character outside the Latin script (Chinese, Cyrillic or Greek, for example), where the estimate is undefined. load_lexicon and load_pool then leave n_syllables NaN. Mirrors count_syllables in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def count_syllables(word):
    """An orthographic syllable estimate: the number of maximal vowel runs.

    Counts the maximal runs of Latin vowel letters after decomposing the word and
    setting its diacritics aside, so a vowel counts whatever its accent. The vowels
    of Polish, Czech, Hungarian, Romanian, Turkish, Danish, Norwegian and Vietnamese
    count as those of English, Spanish, French and German do. Hiatus (``idea``
    counts 2) and syllabic consonants such as the Czech l in ``vlk`` and r in ``krk``
    stay undercounted.

    Returns ``None`` for a missing value and for a word holding a character outside
    the Latin script (Chinese, Cyrillic or Greek, for example), where the estimate is
    undefined. ``load_lexicon`` and ``load_pool`` then leave ``n_syllables`` NaN.
    Mirrors count_syllables in R_workflow/R/querying.R.
    """
    # pd.isna covers None, NaN of any float width, pd.NA and NaT, which str() would
    # otherwise turn into a word ("<NA>" counts one syllable).
    if not isinstance(word, str) and pd.api.types.is_scalar(word) and pd.isna(word):
        return None
    w = unicodedata.normalize("NFD", str(word).lower())
    if not _SYLLABLE_SCRIPT.fullmatch(w):
        return None
    return len(_SYLLABLE_VOWELS.findall(_SYLLABLE_MARKS.sub("", w)))

Pools and matching

The pool is the set of candidates a design will consider at all, and the matcher works only on what it is given. match_stimuli never reads a design's pool_filters, so build_pool is a required step.

lexsync.build_pool(lexicon, filters=None)

Build an experimental candidate pool by filtering a lexicon.

filters maps a column to either a two-element numeric range or a list of permitted values. A row missing the filtered column is dropped under either kind, and a range with a reversed or non-finite bound is an error rather than an empty pool.

Two numbers make a range, and a boolean is not a number, so YAML's [true, false] is a list of permitted values. A list of numbers is compared with a numeric column as numbers, and any other list as text, the cells and the values written as R's as.character() writes them (see _r_text). A filter of 1 therefore keeps the 1.0 cells of a 0/1 column that a missing value has made float. As in the R engine, [true, false] keeps both values of a logical column and no row of a 0/1 column.

A filter naming a column the frame does not have is silently skipped, because the same function filters lexica, supplied pools and pair tables, and those carry different columns. The cost is that a misspelt key silently widens a selection, so every caller that takes its filters from a design checks the names against the frame first: run_pipeline for pool_filters, match_stimuli for a condition's define_by, and select_continuous_pairs for both. Mirrors build_pool in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def build_pool(lexicon: pd.DataFrame, filters: dict | None = None) -> pd.DataFrame:
    """Build an experimental candidate pool by filtering a lexicon.

    ``filters`` maps a column to either a two-element numeric range or a list of
    permitted values. A row missing the filtered column is dropped under either
    kind, and a range with a reversed or non-finite bound is an error rather than an
    empty pool.

    Two numbers make a range, and a boolean is not a number, so YAML's
    ``[true, false]`` is a list of permitted values. A list of numbers is compared
    with a numeric column as numbers, and any other list as text, the cells and the
    values written as R's as.character() writes them (see _r_text). A filter of 1
    therefore keeps the 1.0 cells of a 0/1 column that a missing value has made
    float. As in the R engine, ``[true, false]`` keeps both values of a logical
    column and no row of a 0/1 column.

    A filter naming a column the frame does not have is silently skipped, because the
    same function filters lexica, supplied pools and pair tables, and those carry
    different columns. The cost is that a misspelt key silently widens a
    selection, so every caller that takes its filters from a design checks the names
    against the frame first: ``run_pipeline`` for ``pool_filters``,
    ``match_stimuli`` for a condition's ``define_by``, and
    ``select_continuous_pairs`` for both. Mirrors build_pool in
    R_workflow/R/querying.R.
    """
    df = lexicon
    if filters:
        for col, rng in filters.items():
            if col not in df.columns:
                continue
            vals = list(rng) if isinstance(rng, (list, tuple)) else [rng]
            if len(vals) == 2 and all(_is_number(v) for v in vals):
                # YAML's .inf and .nan arrive as ordinary floats, and either one,
                # like a reversed range, silently empties the pool row by row.
                # Non-finite first: a NaN bound would make the reversal test
                # meaningless. Equal bounds stay legal (the zh design uses [2, 2]).
                if not all(math.isfinite(float(v)) for v in vals):
                    raise ValueError(
                        f"lexsync: filter '{col}' has a non-finite bound; "
                        "ranges need finite numbers.")
                if vals[0] > vals[1]:
                    raise ValueError(
                        f"lexsync: filter '{col}' has a reversed range; "
                        "give it as [low, high].")
                df = df[df[col].notna() & (df[col] >= vals[0]) & (df[col] <= vals[1])]
            elif (pd.api.types.is_numeric_dtype(df[col])
                  and not pd.api.types.is_bool_dtype(df[col])
                  and all(_is_number(v) for v in vals)):
                # pandas counts a bool column as numeric and R does not, hence the
                # second test. Compared as numbers, 1 matches the 1.0 that R writes '1'.
                df = df[df[col].notna() & df[col].isin([float(v) for v in vals])]
            else:
                wanted = {t for t in map(_r_text, vals) if t is not None}
                keep = np.array([_r_text(v) in wanted for v in df[col]], dtype=bool)
                df = df[df[col].notna() & keep]
    return df.reset_index(drop=True)

lexsync.match_stimuli(pool, design, schema, verbose=False)

Match stimuli across conditions on the match_on dimensions.

Two policies govern degraded selections, each read from the design's matching block with the schema as fallback: shortfall ("error", the default, refuses to return fewer sets than requested; "allow" accepts the shrink) and on_insufficient_tolerance ("relax", the default, widens an undersupplied tolerance window to the full condition subpool and records the relaxation in out.attrs["audit"]; "error" refuses instead).

match_on must name at least one dimension. The number of items per condition is n_per_condition, else n_per_cell, else 20, and must be a whole number of at least 1. A null tolerance is read as R's modifyList() reads it (see the schema's tolerance_k comment).

A candidate missing any match_on dimension cannot be scored, so every method excludes it before selecting, and the number excluded per condition is recorded in out.attrs["audit"]["incomplete_dropped"]. The shortfall policy then applies to what remains, and a condition left with no complete candidate is refused.

tie_break, read in the same way, decides between candidates at exactly the same rounded distance, which integer dimensions make common: "hash", the default, prefers the smaller SHA-256 of TIE_SALT, a bar and the word, which favours no part of the alphabet, and "bytes" prefers the spelling that sorts first and reproduces lexsync 0.1. The rule also orders the tied pairs of joint and optimal and the tied rows at their candidate cap. The anchor's even spread keeps its byte-order tie-break under both rules.

Source code in src/lexsync/matching.py

def match_stimuli(pool: pd.DataFrame, design: dict, schema: dict, verbose: bool = False) -> pd.DataFrame:
    """Match stimuli across conditions on the ``match_on`` dimensions.

    Two policies govern degraded selections, each read from the design's
    ``matching`` block with the schema as fallback: ``shortfall`` ("error", the
    default, refuses to return fewer sets than requested; "allow" accepts the
    shrink) and ``on_insufficient_tolerance`` ("relax", the default, widens an
    undersupplied tolerance window to the full condition subpool and records the
    relaxation in ``out.attrs["audit"]``; "error" refuses instead).

    ``match_on`` must name at least one dimension. The number of items per
    condition is ``n_per_condition``, else ``n_per_cell``, else 20, and must be a
    whole number of at least 1. A ``null`` tolerance is read as R's ``modifyList()``
    reads it (see the schema's ``tolerance_k`` comment).

    A candidate missing any ``match_on`` dimension cannot be scored, so every
    method excludes it before selecting, and the number excluded per condition is
    recorded in ``out.attrs["audit"]["incomplete_dropped"]``. The shortfall policy
    then applies to what remains, and a condition left with no complete candidate
    is refused.

    ``tie_break``, read in the same way, decides between candidates at exactly the same
    rounded distance, which integer dimensions make common: "hash", the default,
    prefers the smaller SHA-256 of ``TIE_SALT``, a bar and the word, which favours no
    part of the alphabet, and "bytes" prefers the spelling that sorts first and
    reproduces lexsync 0.1. The rule also orders the tied pairs of ``joint`` and
    ``optimal`` and the tied rows at their candidate cap. The anchor's even spread
    keeps its byte-order tie-break under both rules.
    """
    conditions = design.get("conditions")
    if not conditions:
        # Without this, the anchor lookup below dies with a KeyError here and a
        # bare subscript error in R, two different messages for the same mistake.
        raise ValueError("lexsync: the design has no conditions; a matched design "
                         "needs a conditions list.")
    # Refused here as well as in validate_design, for a direct call: without a
    # dimension every distance is zero and the tie-break alone would pick the words.
    match_on = _require_match_on(design)
    n = _resolve_n(design, N_DEFAULT_MATCHED)
    # Tolerance window k per dimension (window = anchor mean +/- k * SD). A design
    # may override the schema defaults per dimension, e.g. to reproduce a published
    # study's exact windows (Gonzalez Alonso et al. used SD/9 for frequency).
    tol_k = _resolve_tolerance_k(design, schema)
    shortfall = _resolve_policy(design, schema, "shortfall", "error", ("error", "allow"))
    on_tol = _resolve_policy(design, schema, "on_insufficient_tolerance", "relax",
                             ("relax", "error"))
    tie_break = _resolve_tie_break(design, schema)

    for d in match_on:
        if d not in pool.columns:
            raise ValueError(f"lexsync: dimension '{d}' is absent from the pool.")
        k = tol_k.get(d, 2)
        if isinstance(k, (int, float)) and k < 0:
            # A negative k inverts the window (upper bound below the lower), which
            # empties the candidate set and silently relaxes to the full pool,
            # which is never intended.
            raise ValueError(f"lexsync: tolerance_k for dimension '{d}' is negative; "
                             f"tolerances must be zero or positive.")
    cnames = [c["name"] for c in conditions]
    dup_name = [x for x in cnames if cnames.count(x) > 1]
    if dup_name:
        raise ValueError(f"lexsync: condition name '{dup_name[0]}' appears more than "
                         f"once; condition names must be unique.")
    for c in conditions:
        for d in (c.get("define_by") or {}):
            # build_pool skips a column it does not recognise, so a misspelt
            # define_by key would silently hand the condition the whole pool as
            # its subpool and the manipulated contrast would vanish while
            # matching proceeds.
            if d not in pool.columns:
                raise ValueError(f"lexsync: dimension '{d}' in condition '{c['name']}' "
                                 f"is absent from the pool.")

    center = np.array([_exact_mean(pool[d].dropna()) for d in match_on], dtype=float)
    scale = np.array([_exact_sd(pool[d].dropna()) for d in match_on], dtype=float)
    scale[np.isnan(scale) | (scale == 0)] = 1.0

    # .get, not ["define_by"]: a condition without define_by draws on the whole
    # pool, as the R engine's NULL does through build_pool.
    subpools = [build_pool(pool, c.get("define_by")) for c in conditions]
    cond_names = [c["name"] for c in conditions]

    method = ((design.get("matching") or {}).get("method")
              or (schema.get("matching") or {}).get("method") or "standardised_euclidean")
    if method not in _KNOWN_METHODS:
        raise ValueError(f"lexsync: unknown matching method '{method}'. "
                         f"Known methods: {', '.join(_KNOWN_METHODS)}.")
    if method in ("joint", "optimal") and len(conditions) != 2:
        # Both are pairwise matchers; falling back to the anchor matcher here would
        # make the datasheet's recorded method differ from the one actually used.
        raise ValueError(f"lexsync: matching method '{method}' requires exactly two "
                         f"conditions, got {len(conditions)}.")

    # A candidate missing a matched dimension has no distance to any counterpart, so
    # whatever a method did with it, the tie-break decided. The anchored path paired
    # such an anchor with the byte-first word in its window, or re-picked a used
    # candidate. Joint matching took it once the complete pairs ran out, and optimal
    # matching handed the missing cost to a solver that refused it. Every method
    # therefore starts from the rows complete on match_on, and the count excluded per
    # condition goes into the audit for the run log and the datasheet.
    incomplete = []
    for i, sp in enumerate(subpools):
        complete = sp[match_on].notna().all(axis=1).to_numpy()
        n_drop = int((~complete).sum())
        if not n_drop:
            continue
        if not complete.any():
            raise ValueError(f"lexsync: condition '{cond_names[i]}' has no candidates "
                             f"complete on the matched dimensions ({n_drop} lacked a "
                             f"value).")
        if verbose:
            print(f"lexsync: dropped {n_drop} candidate(s) of condition "
                  f"'{cond_names[i]}' missing a matched dimension.")
        incomplete.append({"condition": cond_names[i], "n": n_drop})
        subpools[i] = sp[complete]

    if method == "joint" and len(conditions) == 2:
        out = _match_joint(subpools, cond_names, match_on, center, scale, n,
                           tie_break=tie_break)
        _check_shortfall(out["set"].nunique(), n, shortfall)
        _assert_distinct_words(out)
        return _attach_audit(out, [], incomplete)
    if method == "optimal" and len(conditions) == 2:
        out = _match_optimal(subpools, cond_names, match_on, center, scale, n,
                             tie_break=tie_break)
        _check_shortfall(out["set"].nunique(), n, shortfall)
        _assert_distinct_words(out)
        return _attach_audit(out, [], incomplete)
    # A covariance-aware metric for Mahalanobis matching (None -> plain Euclidean).
    metric = _maha_metric(_zmat(pool, match_on, center, scale)) if method == "mahalanobis" else None

    anchor_pool = subpools[0]
    if len(anchor_pool) == 0:
        raise ValueError(f"lexsync: anchor condition '{cond_names[0]}' has no candidates.")
    # An anchor without define_by orders by the default dimension, as the R
    # engine's names(NULL)[1] falls back to "frequency".
    ord_keys = list((conditions[0].get("define_by") or {}).keys())
    ord_dim = ord_keys[0] if ord_keys else "frequency"
    if ord_dim not in anchor_pool.columns:
        ord_dim = "frequency"
    anchor_pool = (anchor_pool.assign(_k=anchor_pool["word"].map(lambda w: w.encode("utf-8")))
                   .sort_values([ord_dim, "_k"], kind="mergesort").drop(columns="_k").reset_index(drop=True))
    n_take = min(n, len(anchor_pool))
    idx1 = np.unique(np.round(np.linspace(1, len(anchor_pool), n_take)).astype(int))
    anchor = anchor_pool.iloc[idx1 - 1].copy()
    anchor["condition"] = cond_names[0]
    n_take = len(anchor)
    _check_shortfall(n_take, n, shortfall)
    if verbose and n_take < n:
        print(f"lexsync: anchor condition '{cond_names[0]}' yields only {n_take} items; "
              f"n_per_condition is {n}.")
    z_anchor = _zmat(anchor, match_on, center, scale)

    win = {}
    for d in match_on:
        m = _exact_mean(anchor[d].dropna())
        s = _exact_sd(anchor[d].dropna())
        k = tol_k.get(d, 2)
        win[d] = (m - k * s, m + k * s)

    selected = [anchor]
    used_words = set(anchor["word"])
    relaxations = []

    for ci in range(1, len(conditions)):
        cname = cond_names[ci]
        cand = subpools[ci]
        cand = cand[~cand["word"].isin(used_words)]
        if len(cand) == 0:
            raise ValueError(f"lexsync: condition '{cname}' has no candidates left to match.")
        keep = np.ones(len(cand), dtype=bool)
        for d in match_on:
            col = cand[d].to_numpy()
            keep &= (col >= win[d][0]) & (col <= win[d][1])
        cand_f = cand[keep]
        if len(cand_f) < n_take:
            if on_tol == "error":
                raise ValueError(f"lexsync: condition '{cname}' has {len(cand_f)} candidates "
                                 f"within tolerance but {n_take} are needed; raise tolerance_k "
                                 f"or set matching: on_insufficient_tolerance: relax to widen "
                                 f"the window.")
            if verbose:
                print(f"lexsync: condition '{cname}' has {len(cand_f)} candidates within tolerance "
                      f"(< {n_take} needed); relaxing the window.")
            # The relaxation changes what "matched" means for this condition, so it
            # is recorded on the result for the run log and datasheet, where a
            # console message would leave no trace.
            relaxations.append({"condition": cname,
                                "n_within_tolerance": int(len(cand_f)),
                                "n_needed": int(n_take)})
            cand_f = cand
        if len(cand_f) < n_take:
            # The assignment below gives every anchor item an unused candidate, which
            # an exhausted pool cannot supply.
            raise ValueError(f"lexsync: condition '{cname}' has only {len(cand_f)} candidate(s) "
                             f"but {n_take} are needed; widen pool_filters/define_by or lower "
                             f"n_per_condition.")
        # The subpools hold only rows complete on match_on (see the exclusion above), so
        # this repeats the guard before it. It stays as the assignment's precondition: a
        # candidate whose distance is missing cannot be matched, so n_take complete
        # unused candidates must remain however the subpool was built.
        usable = int(cand_f[match_on].notna().all(axis=1).sum())
        if usable < n_take:
            raise ValueError(f"lexsync: condition '{cname}' has only {usable} usable candidate(s) "
                             f"complete on the matched dimensions but {n_take} are needed; widen "
                             f"pool_filters/define_by or lower n_per_condition.")
        cand_f = cand_f.reset_index(drop=True)
        z_cand = _zmat(cand_f, match_on, center, scale)
        # What breaks an exact tie, computed once per candidate (see _tie_sort_keys).
        tie_keys = _tie_sort_keys(cand_f["word"], tie_break)
        ids = cand_f["id"].to_numpy()
        used = np.zeros(len(cand_f), dtype=bool)
        pick = np.empty(n_take, dtype=int)
        for a in range(n_take):
            # Rounded to 9 dp through the shared rule so the stable tie-break below
            # is itself reproducible across R and Python. np.round would pair
            # numpy's scale-rint-unscale with R's decimal algorithm, a pairing
            # io_utils documents as disagreeing at boundaries. On the mahalanobis
            # path the inputs already differ in their last bits, so the absorber
            # must be the same function in both engines.
            delta = z_cand - z_anchor[a]
            if metric is None:
                dvec = _round_dp_vec(np.sqrt((delta ** 2).sum(axis=1)), 9)
            else:
                dvec = _round_dp_vec(np.sqrt(np.maximum((delta @ metric * delta).sum(axis=1), 0.0)), 9)
            # Only unused candidates compete. Giving a used one an infinite distance
            # was not enough, because a missing distance ranks after infinity and an
            # anchor without a value then re-picked a used candidate.
            free = np.flatnonzero(~used)
            # A NaN distance still ranks last, as R's order(na.last = TRUE) ranks it: a
            # bare min() over NaN keeps whichever row it saw first, which would make the
            # selection depend on pool row order and diverge from the R engine.
            nan_last = np.isnan(dvec)
            best = min(free,
                       key=lambda j: (bool(nan_last[j]), 0.0 if nan_last[j] else dvec[j],
                                      tie_keys[j], int(ids[j])))
            pick[a] = best
            used[best] = True
        sel = cand_f.iloc[pick].copy()
        sel["condition"] = cname
        used_words |= set(sel["word"])
        selected.append(sel)

    common = [c for c in selected[0].columns if all(c in s.columns for s in selected)]
    out = pd.concat([s[common] for s in selected], ignore_index=True)
    out["set"] = list(range(1, n_take + 1)) * len(conditions)
    _assert_distinct_words(out)
    return _attach_audit(out, relaxations, incomplete)

lexsync.resample_stimuli(pool, design, schema, n_sets, verbose=False)

Produce n_sets disjoint matched item sets (a replicate column).

Each replicate is an independent, fully matched set drawn from the pool with the items of earlier replicates removed, so no item is reused. This lets a study treat its items as a random factor (running different item samples across participant groups, or showing an effect holds across samples) instead of treating them as a fixed set (Clark, 1973; Yarkoni, 2022). Deterministic: the matcher is deterministic and the used-item set evolves identically across engines.

The first replicate draws on the whole pool, so its count of the candidates excluded for a missing matched value is the one recorded in attrs["audit"]["incomplete_dropped"], as :func:match_stimuli records it. A relaxed tolerance window inside a replicate is not recorded: it reaches the console under verbose but not the run log or the datasheet.

n_sets must be a whole number of at least 1, as resample.n_sets must.

Source code in src/lexsync/matching.py

def resample_stimuli(pool: pd.DataFrame, design: dict, schema: dict,
                     n_sets: int, verbose: bool = False) -> pd.DataFrame:
    """Produce ``n_sets`` disjoint matched item sets (a ``replicate`` column).

    Each replicate is an independent, fully matched set drawn from the pool with
    the items of earlier replicates removed, so no item is reused. This lets a
    study treat its items as a random factor (running different item samples
    across participant groups, or showing an effect holds across samples) instead
    of treating them as a fixed set (Clark, 1973; Yarkoni, 2022). Deterministic: the
    matcher is deterministic and the used-item set evolves identically across engines.

    The first replicate draws on the whole pool, so its count of the candidates
    excluded for a missing matched value is the one recorded in
    ``attrs["audit"]["incomplete_dropped"]``, as :func:`match_stimuli` records it.
    A relaxed tolerance window inside a replicate is not recorded: it reaches the
    console under ``verbose`` but not the run log or the datasheet.

    ``n_sets`` must be a whole number of at least 1, as ``resample.n_sets`` must.
    """
    n_sets = _check_n_sets(n_sets)
    used: set = set()
    parts = []
    incomplete = []
    for k in range(1, n_sets + 1):
        pk = pool[~pool["word"].isin(used)].reset_index(drop=True)
        sk = match_stimuli(pk, design, schema, verbose=verbose).copy()
        if k == 1:
            incomplete = (sk.attrs.get("audit") or {}).get("incomplete_dropped") or []
        sk["replicate"] = k
        used |= set(sk["word"])
        parts.append(sk)
    # pd.concat keeps an audit only when every replicate holds the same one, and
    # rbind keeps the first replicate's whatever the others hold. The record is
    # therefore set explicitly, and both engines return the same one.
    return _attach_audit(pd.concat(parts, ignore_index=True), [], incomplete)

Continuous designs

Dichotomising a continuous predictor costs power and can introduce selection artefacts. A design may instead spread its items over the predictor's ranks, following the pool's distribution, while tolerance windows trim the tails of its controls, and be analysed by regression or a mixed model.

lexsync.select_continuous_stimuli(pool, design, schema, verbose=False, key='word', label='continuous', renumber_sets=True)

Select a set spread over a continuous predictor, with windows on the controls.

Instead of dichotomising the predictor into conditions and matching, items are spread evenly over the predictor's ranks in the pool, so the selection follows the pool's distribution of the predictor and does not cover its range evenly. The control dimensions are held within a tolerance band, which trims their tails without holding them constant, so read each control's correlation with the predictor in match_report_continuous before relying on it. Tightening tolerance_k does not reliably lower those correlations. The set is analysed by regression / mixed models rather than by between-condition contrasts, which avoids the loss of power and the selection artefacts of matched dichotomies (Kuperman, 2015; Liben-Nowell et al., 2019).

Two deterministic passes reuse the matcher's even-spread primitive, so the R and Python engines select byte-identical stimuli: an even spread over the predictor defines a tolerance window on each control; the pool is filtered to that window; a second even spread over the filtered pool is the selection. There is no per-item matching and no random number generator.

The design is checked before anything is selected, so a design that cannot be honoured raises ValueError outright. continuous.controls must be non-empty and must not name the predictor, match_on must name exactly the same dimensions as continuous.controls, every dimension named and the key column must be present in the pool, no tolerance_k may be negative, and the pool must not be empty.

A row missing the predictor or a control is excluded before the first pass, and the number excluded is recorded in attrs["audit"]["incomplete_dropped"] under the condition continuous. A pool with no complete row is refused.

attrs["audit"]["control_windows"] records, for each control, the window the first pass set (low, high) and the SD of the complete pool (pool_sd), which :func:match_report_continuous sets the realised SD against. A relaxed window is recorded as computed, and the relaxation in window_relaxations.

Returns the selected stimuli. Unless label is None the condition column is set to it, continuous by default, and unless renumber_sets is False the set column is renumbered 1..n.

Source code in src/lexsync/matching.py

def select_continuous_stimuli(pool: pd.DataFrame, design: dict, schema: dict,
                              verbose: bool = False, key: str = "word",
                              label: str | None = "continuous",
                              renumber_sets: bool = True) -> pd.DataFrame:
    """Select a set spread over a continuous predictor, with windows on the controls.

    Instead of dichotomising the predictor into conditions and matching, items are
    spread evenly over the predictor's ranks in the pool, so the selection follows
    the pool's distribution of the predictor and does not cover its range evenly.
    The control dimensions are held within a tolerance band, which trims their
    tails without holding them constant, so read each control's correlation with
    the predictor in ``match_report_continuous`` before relying on it. Tightening
    ``tolerance_k`` does not reliably lower those correlations. The set is analysed
    by regression / mixed models rather than by between-condition contrasts, which
    avoids the loss of power and the selection artefacts of matched dichotomies
    (Kuperman, 2015; Liben-Nowell et al., 2019).

    Two deterministic passes reuse the matcher's even-spread primitive, so the R
    and Python engines select byte-identical stimuli: an even spread over the
    predictor defines a tolerance window on each control; the pool is filtered to
    that window; a second even spread over the filtered pool is the selection.
    There is no per-item matching and no random number generator.

    The design is checked before anything is selected, so a design that cannot be
    honoured raises ``ValueError`` outright. ``continuous.controls`` must be non-empty
    and must not name the predictor, ``match_on`` must name exactly the same dimensions
    as ``continuous.controls``, every dimension named and the ``key`` column must be
    present in the pool, no ``tolerance_k`` may be negative, and the pool must not be
    empty.

    A row missing the predictor or a control is excluded before the first pass, and
    the number excluded is recorded in ``attrs["audit"]["incomplete_dropped"]`` under
    the condition ``continuous``. A pool with no complete row is refused.

    ``attrs["audit"]["control_windows"]`` records, for each control, the window the
    first pass set (``low``, ``high``) and the SD of the complete pool (``pool_sd``),
    which :func:`match_report_continuous` sets the realised SD against. A relaxed
    window is recorded as computed, and the relaxation in ``window_relaxations``.

    Returns the selected stimuli. Unless ``label`` is ``None`` the ``condition``
    column is set to it, ``continuous`` by default, and unless ``renumber_sets`` is
    ``False`` the ``set`` column is renumbered ``1..n``.
    """
    cfg = design["continuous"]
    predictor = cfg["predictor"]
    controls = list(cfg.get("controls") or [])
    match_on = list(design.get("match_on") or [])
    if not controls:
        raise ValueError("lexsync: a continuous design needs at least one control "
                         "dimension (continuous.controls must be non-empty).")
    if predictor in controls:
        raise ValueError(f"lexsync: the continuous predictor '{predictor}' must not "
                         "also appear in continuous.controls.")
    if sorted(match_on) != sorted(controls):
        raise ValueError("lexsync: for a continuous design, match_on must equal "
                         "continuous.controls.")
    for d in [predictor] + controls:
        if d not in pool.columns:
            raise ValueError(f"lexsync: dimension '{d}' is absent from the pool.")
    if key not in pool.columns:
        raise ValueError(
            "lexsync: the continuous tie-break column '%s' is absent from the pool." % key)
    n = _resolve_n(design, N_DEFAULT_CONTINUOUS)
    tol_k = _resolve_tolerance_k(design, schema)
    shortfall = _resolve_policy(design, schema, "shortfall", "error", ("error", "allow"))
    on_tol = _resolve_policy(design, schema, "on_insufficient_tolerance", "relax",
                             ("relax", "error"))
    for d in controls:
        k = tol_k.get(d, 2)
        if isinstance(k, (int, float)) and k < 0:
            raise ValueError(f"lexsync: tolerance_k for dimension '{d}' is negative; "
                             f"tolerances must be zero or positive.")

    # A row missing the predictor sorts last and the even spread always takes the
    # last row, so such rows were selected and then analysed without their
    # predictor. A row missing a control fails every window but comes back when the
    # window relaxes to the whole pool. Both are excluded before the first pass.
    complete = pool[[predictor] + controls].notna().all(axis=1).to_numpy()
    n_drop = int((~complete).sum())
    incomplete = []
    if n_drop:
        if not complete.any():
            raise ValueError(f"lexsync: no item in the pool is complete on the predictor "
                             f"and the controls ({n_drop} lacked a value).")
        if verbose:
            print(f"lexsync: dropped {n_drop} candidate(s) missing the predictor or a "
                  f"control.")
        incomplete.append({"condition": "continuous", "n": n_drop})
        pool = pool[complete]

    def even_spread(df):
        # `key` is the deterministic tie-break when two rows share a predictor
        # value: `word` for a corpus pool, `set` for a collapsed pair table where
        # no `word` column exists. A text key is compared as UTF-8 BYTES, which is
        # what R's radix sort does; a numeric key is compared numerically in both.
        k = df[key]
        if pd.api.types.is_object_dtype(k) or pd.api.types.is_string_dtype(k):
            k = k.map(lambda w: str(w).encode("utf-8"))
        df = (df.assign(_k=k)
              .sort_values([predictor, "_k"], kind="mergesort")
              .drop(columns="_k").reset_index(drop=True))
        if len(df) == 0:
            return df
        n_take = min(n, len(df))
        idx = np.unique(np.round(np.linspace(1, len(df), n_take)).astype(int))
        return df.iloc[idx - 1].reset_index(drop=True)

    # Pass 1: an even spread over the whole pool defines the control windows.
    spread = even_spread(pool)
    if len(spread) == 0:
        raise ValueError("lexsync: the pool is empty for the continuous design.")
    win = {}
    for d in controls:
        m = _exact_mean(spread[d].dropna())
        s = _exact_sd(spread[d].dropna())
        k = tol_k.get(d, 2)
        win[d] = (m - k * s, m + k * s)
    # The windows and the pool's spread, which the realised-control report sets the
    # selection's spread against.
    windows = [{"dimension": d, "low": win[d][0], "high": win[d][1],
                "pool_sd": _exact_sd(pool[d].dropna())} for d in controls]
    keep = np.ones(len(pool), dtype=bool)
    for d in controls:
        col = pool[d].to_numpy()
        keep &= (col >= win[d][0]) & (col <= win[d][1])
    filtered = pool[keep]
    relaxations = []
    if len(filtered) < n:
        if on_tol == "error":
            raise ValueError(f"lexsync: {len(filtered)} items lie within the control windows "
                             f"but {n} are needed; raise tolerance_k or set matching: "
                             f"on_insufficient_tolerance: relax to widen the window.")
        if verbose:
            print(f"lexsync: {len(filtered)} items within the control windows "
                  f"(< {n} needed); relaxing to the full pool.")
        relaxations.append({"condition": "continuous",
                            "n_within_tolerance": int(len(filtered)),
                            "n_needed": int(n)})
        filtered = pool
    # Pass 2: an even spread over the filtered pool is the selection.
    sel = even_spread(filtered).copy()
    _check_shortfall(len(sel), n, shortfall, continuous=True)
    # A pair table already carries its own `condition` and `set`, which the Latin
    # square and the trial-order digest depend on, so the pair path passes
    # label=None and renumber_sets=False to leave both alone.
    if label is not None:
        sel["condition"] = label
    if renumber_sets:
        sel["set"] = list(range(1, len(sel) + 1))
    return _attach_audit(sel, relaxations, incomplete, windows)

lexsync.match_report_continuous(stimuli, predictor, controls, schema, audit=None)

Realised-control report for a continuous design.

Returns the same {descriptives, comparisons} shape as :func:match_report, so the pipeline and datasheet stay uniform, but the comparisons describe a continuous predictor instead of a between-condition contrast: the predictor's realised span and, for each control, its Pearson correlation with the predictor (near zero when the control is held constant). The set is meant for regression / mixed-model analysis, not equivalence tests.

sd is each dimension's realised SD. For the predictor, sd_ratio sets it against the SD of as many values spaced evenly over the same range, (max - min) * sqrt(n * (n + 1) / 12) / (n - 1) with the n - 1 that sd uses, so an even spread has a ratio of 1 whatever its size and a selection that follows a skewed pool a ratio below 1. That benchmark tends to (max - min) / sqrt(12), the SD of a uniform distribution over the range, as n grows. For a control, sd_ratio sets the realised SD against the SD of the pool the selector drew from, and window_low and window_high give the window the selector banded it to, before any relaxation. Both come from audit, the selector's record, which is read from stimuli.attrs["audit"] when not given. Without one, a control's ratio and window are missing.

Source code in src/lexsync/validation.py

def match_report_continuous(stimuli, predictor, controls, schema, audit=None) -> dict:
    """Realised-control report for a continuous design.

    Returns the same ``{descriptives, comparisons}`` shape as :func:`match_report`,
    so the pipeline and datasheet stay uniform, but the comparisons describe a
    continuous predictor instead of a between-condition contrast: the predictor's
    realised span and, for each control, its Pearson correlation with the predictor
    (near zero when the control is held constant). The set is meant for regression /
    mixed-model analysis, not equivalence tests.

    ``sd`` is each dimension's realised SD. For the predictor, ``sd_ratio`` sets it
    against the SD of as many values spaced evenly over the same range,
    ``(max - min) * sqrt(n * (n + 1) / 12) / (n - 1)`` with the n - 1 that ``sd``
    uses, so an even spread has a ratio of 1 whatever its size and a selection that
    follows a skewed pool a ratio below 1. That benchmark tends to
    ``(max - min) / sqrt(12)``, the SD of a uniform distribution over the range, as n
    grows. For a control, ``sd_ratio`` sets the realised SD against the SD of the pool
    the selector drew from, and ``window_low`` and ``window_high`` give the window the
    selector banded it to, before any relaxation. Both come from ``audit``, the
    selector's record, which is read from ``stimuli.attrs["audit"]`` when not given.
    Without one, a control's ratio and window are missing.
    """
    if audit is None:
        audit = stimuli.attrs.get("audit")
    windows = {w["dimension"]: w for w in ((audit or {}).get("control_windows") or [])}
    desc = describe_stimuli(stimuli, [predictor] + controls)
    pv = pd.to_numeric(stimuli[predictor], errors="coerce").to_numpy(dtype=float)
    valid = pv[~np.isnan(pv)]
    # None (not NaN) when the predictor has no span, so both engines agree.
    span = _round_dp(float(valid.max() - valid.min()), 3) if len(valid) >= 2 else None
    # The SD of n values spaced evenly over the realised range. (max - min) / sqrt(12)
    # alone is its limit for large n, and against it an even spread of 4 values read
    # 1.49, as if the items crowded the ends of the range. The operations run in the
    # order of the R twin, so both engines get the same double.
    nv = len(valid)
    even = (float(valid.max() - valid.min()) * math.sqrt(nv * (nv + 1) / 12) / (nv - 1)
            if nv >= 2 else None)
    sd = _exact_sd(valid)
    rows = [dict(dimension=predictor, role="predictor", pearson_r=None, predictor_span=span,
                 sd=_round3(sd), sd_ratio=_round3(_ratio(sd, even)), window_low=None,
                 window_high=None)]
    for c in controls:
        cv = pd.to_numeric(stimuli[c], errors="coerce").to_numpy(dtype=float)
        r = _pearson(pv, cv)
        csd = _exact_sd(cv[~np.isnan(cv)])
        w = windows.get(c) or {}
        rows.append(dict(dimension=c, role="control",
                         pearson_r=_round_dp(r, 3) if r is not None else None,
                         predictor_span=span, sd=_round3(csd),
                         sd_ratio=_round3(_ratio(csd, w.get("pool_sd"))),
                         window_low=_round3(w.get("low")), window_high=_round3(w.get("high"))))
    return dict(descriptives=desc, comparisons=pd.DataFrame(rows))

Pseudoword generation

Non-words are generated deterministically, with no sampling anywhere, preserving length exactly and keeping every letter bigram attested in the corpus. Both methods select byte-identical stimuli in the R and Python engines.

lexsync.generate_pseudowords(base_words, reference_words, exclude=None)

A length-matched pseudoword for each base word.

Base words are processed in byte order so the used set evolves identically across engines. The letters substituted in are those the reference words attest. exclude (optional) lists further word forms no pseudoword may equal, such as a list of low-frequency words the reference lacks. A base word in a logographic script is refused. Returns a frame with base_word and pseudoword.

Source code in src/lexsync/generation.py

def generate_pseudowords(base_words, reference_words, exclude=None) -> pd.DataFrame:
    """A length-matched pseudoword for each base word.

    Base words are processed in byte order so the ``used`` set evolves identically
    across engines. The letters substituted in are those the reference words attest.
    ``exclude`` (optional) lists further word forms no pseudoword may equal, such as
    a list of low-frequency words the reference lacks. A base word in a logographic
    script is refused. Returns a frame with ``base_word`` and ``pseudoword``.
    """
    base = [str(w) for w in base_words]
    _refuse_logographic(base)
    lexicon = _known_set(reference_words, exclude)
    bigrams = bigram_counts(reference_words)
    alphabet = _reference_alphabet(reference_words)
    order = sorted(range(len(base)), key=lambda i: base[i].encode("utf-8"))
    used: set = set()
    pseudo = [None] * len(base)
    for i in order:
        pw = make_pseudoword(base[i], bigrams, lexicon, used, alphabet)
        if pw is None:
            raise ValueError(f"lexsync: could not generate a pseudoword for '{base[i]}'.")
        used.add(pw)
        pseudo[i] = pw
    return pd.DataFrame({"base_word": base, "pseudoword": pseudo})

lexsync.make_pseudoword(word, bigrams, lexicon, used, alphabet=None)

The most bigram-plausible legal non-word at the smallest edit distance.

Searches single-letter substitutions first, then two-letter substitutions; candidates are ranked by summed bigram frequency with a byte-order tie-break, so the choice is deterministic and identical across engines. alphabet lists the letters substituted in, in UTF-8 byte order. It defaults to the letters of the bigram table, which hold every letter a legal candidate of two letters or more can use. The generators pass the letters of the whole reference.

Source code in src/lexsync/generation.py

def make_pseudoword(word: str, bigrams: dict, lexicon: set, used: set,
                    alphabet=None) -> str | None:
    """The most bigram-plausible legal non-word at the smallest edit distance.

    Searches single-letter substitutions first, then two-letter substitutions;
    candidates are ranked by summed bigram frequency with a byte-order tie-break,
    so the choice is deterministic and identical across engines. ``alphabet`` lists
    the letters substituted in, in UTF-8 byte order. It defaults to the letters of
    the bigram table, which hold every letter a legal candidate of two letters or
    more can use. The generators pass the letters of the whole reference.
    """
    word = str(word)
    L = len(word)
    if alphabet is None:
        alphabet = _reference_alphabet(bigrams)
    # Distance 1: one substituted position.
    cands = []
    for pos in range(L):
        for c in alphabet:
            if c == word[pos]:
                continue
            cand = word[:pos] + c + word[pos + 1:]
            if cand in lexicon or cand in used:
                continue
            if _legal(cand, bigrams):
                cands.append(cand)
    if not cands:
        # Distance 2: two substituted positions (rare fallback).
        for i in range(L):
            for j in range(i + 1, L):
                for ci in alphabet:
                    if ci == word[i]:
                        continue
                    for cj in alphabet:
                        if cj == word[j]:
                            continue
                        cand = word[:i] + ci + word[i + 1:j] + cj + word[j + 1:]
                        if cand in lexicon or cand in used:
                            continue
                        if _legal(cand, bigrams):
                            cands.append(cand)
    if not cands:
        return None
    return min(cands, key=lambda c: (-_score(c, bigrams), c.encode("utf-8")))

lexsync.build_lexdec_stimuli(pool, n, reference_words=None, method='letter_substitution', exclude=None)

Assemble a word-vs-pseudoword lexical-decision set from a candidate pool.

Real words are drawn by an even spread across the byte-ordered pool (the same deterministic device as the matcher's anchor), then a length-matched pseudoword is generated for each. The pool is first filtered to the words written wholly in letters the reference attests, which the generators substitute among, so the eligible pool can be smaller than the request; the pipeline's shortfall policy then decides whether that errors. A pool word in a logographic script is refused. reference_words (the full lexicon) supplies the letters, the bigram statistics and the real-word list a pseudoword must avoid; it falls back to the pool when not given. exclude (optional) lists further known word forms a pseudoword must not equal, which the reference may lack. The presented string is the target column; conditions are word and pseudoword and set pairs them. base gives the word each pseudoword was made from, a word's own form for a word, and edit_distance the Levenshtein distance from target to base, 0 for a word.

Source code in src/lexsync/generation.py

def build_lexdec_stimuli(pool: pd.DataFrame, n: int, reference_words=None,
                         method: str = "letter_substitution", exclude=None) -> pd.DataFrame:
    """Assemble a word-vs-pseudoword lexical-decision set from a candidate pool.

    Real words are drawn by an even spread across the byte-ordered pool (the same
    deterministic device as the matcher's anchor), then a length-matched
    pseudoword is generated for each. The pool is first filtered to the words
    written wholly in letters the reference attests, which the generators
    substitute among, so the eligible pool can be smaller than the request; the
    pipeline's shortfall policy then decides whether that errors. A pool word in a
    logographic script is refused. ``reference_words`` (the full lexicon)
    supplies the letters, the bigram statistics and the real-word list a
    pseudoword must avoid; it falls back to the pool when not given. ``exclude``
    (optional) lists further known word forms a pseudoword must not equal, which
    the reference may lack. The presented string is the ``target`` column;
    conditions are ``word`` and ``pseudoword`` and ``set`` pairs them. ``base``
    gives the word each pseudoword was made from, a word's own form for a word, and
    ``edit_distance`` the Levenshtein distance from ``target`` to ``base``, 0 for a
    word.
    """
    pool = (pool.assign(_k=pool["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))
    pool = _lexdec_eligible(pool, reference_words)
    _refuse_logographic(pool["word"].tolist())
    if len(pool) == 0:
        raise ValueError("lexsync: lexical-decision pool has no word written wholly in "
                         "letters the reference attests.")
    n_take = min(n, len(pool))
    idx = np.unique(np.round(np.linspace(1, len(pool), n_take)).astype(int)) - 1
    words = pool.iloc[idx].reset_index(drop=True)
    ref = list(reference_words) if reference_words is not None else pool["word"].tolist()
    if method == "subsyllabic":
        gen = generate_pseudowords_subsyllabic(words["word"].tolist(), ref, exclude=exclude)
    elif method == "letter_substitution":
        gen = generate_pseudowords(words["word"].tolist(), ref, exclude=exclude)
    else:
        raise ValueError(f"lexsync: unknown pseudoword generation method '{method}'.")
    pw_map = dict(zip(gen["base_word"], gen["pseudoword"], strict=True))
    base = words["word"].tolist()
    forms = [pw_map[w] for w in base]

    real = pd.DataFrame({
        "target": base, "word": base,
        "condition": "word", "length": words["length"].tolist(),
        "set": list(range(1, len(words) + 1)),
        "base": base, "edit_distance": 0,
    })
    pseudo = pd.DataFrame({
        "target": forms, "word": forms,
        "condition": "pseudoword", "length": words["length"].tolist(),
        "set": list(range(1, len(words) + 1)),
        # rapidfuzz's distance equals stringdist's "lv" in the R engine, the agreement
        # add_neighbourhood already rests OLD20 on.
        "base": base,
        "edit_distance": [Levenshtein.distance(p, b) for p, b in zip(forms, base, strict=True)],
    })
    out = pd.concat([real, pseudo], ignore_index=True)
    return out

Paradigms and trial events

A trial is a list of event dictionaries rather than backend code, which is what lets one engine serve six paradigms and three presentation targets. A design either names a paradigm and inherits its event sequence, or supplies its own events.

lexsync.PARADIGMS = {'factorial': {'stimulus_fields': ['word'], 'counterbalance': 'factorial', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{word}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'lexical_decision': {'stimulus_fields': ['target'], 'counterbalance': 'factorial', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{target}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'priming': {'stimulus_fields': ['prime', 'target'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{prime}', 'duration_frames': 3, 'trigger': 20, 'onset_locked': True}, {'type': 'mask', 'content': '#####', 'duration_frames': 2}, {'type': 'text', 'content': '{target}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'masked_priming': {'stimulus_fields': ['prime', 'target'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'mask', 'content': '#', 'length_of': ['prime', 'target'], 'duration_ms': 500}, {'type': 'text', 'content': '{prime}', 'duration_frames': 3, 'trigger': 20, 'onset_locked': True, 'case': 'lower'}, {'type': 'text', 'content': '{target}', 'duration_ms': 2000, 'trigger': 'condition', 'onset_locked': True, 'case': 'upper'}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 0}, {'type': 'blank', 'duration_ms': 250}]}, 'categorisation': {'stimulus_fields': ['target', 'category', 'answer'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_ms': 500}, {'type': 'text', 'content': '{category}', 'duration_ms': 750}, {'type': 'text', 'content': '{target}', 'duration_ms': 800, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['f', 'j'], 'timeout_ms': 2500}, {'type': 'blank', 'duration_ms': 250}]}, 'self_paced_reading': {'stimulus_fields': ['sentence', 'question'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'region_by_region', 'content': '{sentence}', 'advance': 'space', 'critical_region_trigger': 'condition'}, {'type': 'question', 'content': '{question}', 'keys': ['f', 'j'], 'timeout_ms': 5000}, {'type': 'blank', 'duration_frames': _ISI}]}} module-attribute

lexsync.resolve_events(design)

Return the event list for a design: explicit events or paradigm default.

Source code in src/lexsync/paradigms.py

def resolve_events(design: dict) -> list:
    """Return the event list for a design: explicit ``events`` or paradigm default."""
    if design.get("events"):
        return [dict(e) for e in design["events"]]
    name = design.get("paradigm", "factorial")
    return [dict(e) for e in get_paradigm(name)["events"]]

lexsync.resolve_trial_timing(stimuli, design, schema)

Realise per-trial event durations onto the stimuli table.

An event may declare a duration that varies from trial to trial, either read from an item column or drawn from a range. A drawn value is a pure function of the keyed hash, so both engines realise the same milliseconds, and it is written into the stimuli table as well as the generated script, because timing that varies is a variable the analysis needs, not presentation detail.

Every value read from a column must be a finite number of milliseconds, not negative. The first that is not, a blank cell included, is refused, naming the column and the trial's set and condition. A blank or negative value used to reach the experiments, where OpenSesame and PsychoPy stopped at the blank and jsPsych ended the trial at once. When a column holds fractional milliseconds, each of its values is rounded once with the rounder both engines share and written back, so the stimuli, the loop tables and every target carry the same whole milliseconds. A column of whole numbers is left as it is. Each column rounded is recorded in the returned frame's attrs["duration_rounding"], as one {"event", "column", "n_rounded"} entry, which run_pipeline writes into the run log.

Source code in src/lexsync/scripting.py

def resolve_trial_timing(stimuli: pd.DataFrame, design: dict, schema: dict) -> pd.DataFrame:
    """Realise per-trial event durations onto the stimuli table.

    An event may declare a duration that varies from trial to trial, either read
    from an item column or drawn from a range. A drawn value is a pure function of
    the keyed hash, so both engines realise the same milliseconds, and it is
    written into the stimuli table as well as the generated script, because timing
    that varies is a variable the analysis needs, not presentation detail.

    Every value read from a column must be a finite number of milliseconds, not
    negative. The first that is not, a blank cell included, is refused, naming the
    column and the trial's set and condition. A blank or negative value used to reach
    the experiments, where OpenSesame and PsychoPy stopped at the blank and jsPsych
    ended the trial at once. When a column holds fractional milliseconds, each of its
    values is rounded once with the rounder both engines share and written back, so
    the stimuli, the loop tables and every target carry the same whole milliseconds.
    A column of whole numbers is left as it is. Each column rounded is recorded in
    the returned frame's ``attrs["duration_rounding"]``, as one
    ``{"event", "column", "n_rounded"}`` entry, which run_pipeline writes into the run
    log.
    """
    events = resolve_events(design)
    seed = (schema or {}).get("seed", 1)
    stimuli = stimuli.copy()
    # A record carried in from an earlier call describes that call, not this one.
    stimuli.attrs.pop("duration_rounding", None)
    rounding = []
    for i, ev in enumerate(events, start=1):
        spec = _duration_spec(ev, i)
        if spec is None:
            continue
        if spec["jitter"] is None:
            # from_column reads a column the items already carry, so it is checked here.
            if spec["column"] not in stimuli.columns:
                raise ValueError(
                    "lexsync: event %d reads its duration from column '%s', which the "
                    "items do not have." % (i, spec["column"]))
            n_rounded = _column_durations(stimuli, i, spec["column"])
            if n_rounded:
                rounding.append({"event": i, "column": spec["column"], "n_rounded": n_rounded})
            continue
        lo, hi = spec["jitter"]
        # The key names the column as well as the trial, so two jittered events in
        # one design draw independently, where a key naming only the trial would
        # give them one shared value.
        lists = stimuli["list"] if "list" in stimuli.columns else [1] * len(stimuli)
        keys = ["|".join([_key_part(seed), "jitter", spec["column"],
                          _key_part(l), _key_part(s), _key_part(c)])
                for l, s, c in zip(lists, stimuli["set"], stimuli["condition"], strict=True)]
        stimuli[spec["column"]] = [hash_int_range(k, lo, hi) for k in keys]
    if rounding:
        stimuli.attrs["duration_rounding"] = rounding
    return stimuli

lexsync.required_fields(design)

Trial fields a design needs present in its items (paradigm + events).

Source code in src/lexsync/paradigms.py

def required_fields(design: dict) -> list:
    """Trial fields a design needs present in its items (paradigm + events)."""
    name = design.get("paradigm", "factorial")
    base = list(get_paradigm(name)["stimulus_fields"]) if name in PARADIGMS else []
    for f in referenced_fields(resolve_events(design)):
        if f not in base:
            base.append(f)
    return base

Counterbalancing

Two recipes are available, and the paradigm chooses between them unless the design names one in counterbalance.recipe. A design's own events do not change the choice. Trial order comes from a seeded, keyed-hash shuffle, a pure function of the design with no generator behind it, so the same seed gives the same order in both engines. balance_lists is the optional search for a list assignment whose lists are equated on the item dimensions, where the plain deal goes by set rank.

lexsync.counterbalance(stimuli, design, schema, list_of_set=None)

Assign stimuli to lists and a randomised, reproducible trial order.

The recipe is counterbalance.recipe when the design names one, and otherwise the recipe of the paradigm it names. That is the factorial recipe for matched word lists, or a Latin square over conditions for the paired and sentence paradigms. A design's own events list has no say in the recipe. The factorial recipe refuses more lists than item sets, since the surplus lists would be empty.

list_of_set optionally maps each set to a list, from balance_lists. The pipeline supplies it when counterbalance.optimise is on. Without it, the factorial recipe deals sets to lists by rank.

Each list's trials are ordered by the seeded keyed hash. counterbalance.max_run repairs that order so that no run of one counterbalance.run_on value (by default the condition) is longer than the limit, and refuses a limit no order can meet. A design with fillers has the limit applied by add_blocks, to the order the fillers are interleaved in. counterbalance.orders: m adds order_1 to order_m, each trial's position in each of m orders of its list, order_1 being the order trial gives.

Returns stimuli with added list and trial columns, and the order columns when there is more than one order. Mirrors counterbalance() in R_workflow/R/counterbalancing.R.

Source code in src/lexsync/counterbalancing.py

def counterbalance(stimuli: pd.DataFrame, design: dict, schema: dict,
                   list_of_set=None) -> pd.DataFrame:
    """Assign stimuli to lists and a randomised, reproducible trial order.

    The recipe is ``counterbalance.recipe`` when the design names one, and otherwise
    the recipe of the paradigm it names. That is the factorial recipe for matched word
    lists, or a Latin square over conditions for the paired and sentence paradigms. A
    design's own ``events`` list has no say in the recipe. The factorial recipe
    refuses more lists than item sets, since the surplus lists would be empty.

    ``list_of_set`` optionally maps each ``set`` to a list, from ``balance_lists``.
    The pipeline supplies it when ``counterbalance.optimise`` is on. Without it, the
    factorial recipe deals sets to lists by rank.

    Each list's trials are ordered by the seeded keyed hash. ``counterbalance.max_run``
    repairs that order so that no run of one ``counterbalance.run_on`` value (by default
    the condition) is longer than the limit, and refuses a limit no order can meet. A
    design with fillers has the limit applied by ``add_blocks``, to the order the fillers
    are interleaved in. ``counterbalance.orders: m`` adds ``order_1`` to ``order_m``,
    each trial's position in each of m orders of its list, ``order_1`` being the order
    ``trial`` gives.

    Returns ``stimuli`` with added ``list`` and ``trial`` columns, and the order columns
    when there is more than one order. Mirrors counterbalance() in
    R_workflow/R/counterbalancing.R.
    """
    _trial_order_settings(design)
    # Resampled designs counterbalance each replicate (an independent item set)
    # on its own, so trial order is numbered within each replicate.
    if "replicate" in stimuli.columns and stimuli["replicate"].nunique() > 1:
        parts = [_counterbalance_one(g.reset_index(drop=True), design, schema, list_of_set)
                 for _, g in stimuli.groupby("replicate", sort=True)]
        return pd.concat(parts, ignore_index=True)
    return _counterbalance_one(stimuli, design, schema, list_of_set)

lexsync.balance_lists(stimuli, design, schema)

Assign item sets to lists so the lists match on the item dimensions.

The factorial recipe's default deal is by set rank, which balances nothing. This searches instead for an assignment whose lists have near-equal totals on each declared dimension, by steepest-descent pairwise swaps between lists. List sizes are preserved, because a swap exchanges one set for another.

The search is deterministic and identical in the R and Python engines: the objective is all-integer (see the notes in this module), the descent takes the single best swap each pass, and ties are broken by the seeded keyed hash rather than by position, so no list is favoured by being numbered first. Because the cost is a non-negative integer that strictly decreases, the search terminates; max_passes bounds it anyway and the report says whether the bound was reached.

counterbalance.scale sets each dimension's unit. mean, the default, divides by the dimension's mean, which suits ratio scales only. mad balances deviations from the mean in units of the mean absolute deviation, so adding a constant to a dimension, such as writing Zipf frequency as Zipf - 3, leaves the lists unchanged. The report then gives scale too, and cost_unit names the unit under either scale.

Five situations raise ValueError rather than being answered with an assignment that would mislead. A Latin-square design is refused because every item already appears in every list there, so there is nothing left to equate, and fewer than two lists leaves no pair of lists to exchange sets between. A design with no resolvable balance dimension is refused, as is one naming a dimension the stimuli do not carry, and the message names the columns. The last refusal is arithmetic: the search stops if the integer objective would leave the range a double represents exactly, since past that point the two engines could disagree.

Returns {"list_of_set": {set: list}, "report": {...}}.

Source code in src/lexsync/counterbalancing.py

def balance_lists(stimuli: pd.DataFrame, design: dict, schema: dict) -> dict:
    """Assign item sets to lists so the lists match on the item dimensions.

    The factorial recipe's default deal is by set rank, which balances nothing. This
    searches instead for an assignment whose lists have near-equal totals on each
    declared dimension, by steepest-descent pairwise swaps between lists. List sizes
    are preserved, because a swap exchanges one set for another.

    The search is deterministic and identical in the R and Python engines: the
    objective is all-integer (see the notes in this module), the descent takes the
    single best swap each pass, and ties are broken by the seeded keyed hash rather
    than by position, so no list is favoured by being numbered first. Because the cost
    is a non-negative integer that strictly decreases, the search terminates;
    ``max_passes`` bounds it anyway and the report says whether the bound was reached.

    ``counterbalance.scale`` sets each dimension's unit. ``mean``, the default,
    divides by the dimension's mean, which suits ratio scales only. ``mad`` balances
    deviations from the mean in units of the mean absolute deviation, so adding a
    constant to a dimension, such as writing Zipf frequency as Zipf - 3, leaves the
    lists unchanged. The report then gives ``scale`` too, and ``cost_unit`` names the
    unit under either scale.

    Five situations raise ``ValueError`` rather than being answered with an assignment
    that would mislead. A Latin-square design is refused because every item already
    appears in every list there, so there is nothing left to equate, and fewer than two
    lists leaves no pair of lists to exchange sets between. A design with no resolvable
    balance dimension is refused, as is one naming a dimension the stimuli do not carry,
    and the message names the columns. The last refusal is arithmetic: the search stops
    if the integer objective would leave the range a double represents exactly, since
    past that point the two engines could disagree.

    Returns ``{"list_of_set": {set: list}, "report": {...}}``.
    """
    cb = design.get("counterbalance") or {}
    n_lists = int(cb.get("lists", 1))
    seed = schema.get("seed", 1)
    if _recipe(design) == "latin_square_target":
        raise ValueError(
            "lexsync: counterbalance.optimise does not apply to a Latin-square "
            "design. Every item already appears in every list there, so the lists are "
            "balanced on the items by construction; the rotation decides only which "
            "condition each item takes.")
    if n_lists < 2:
        raise ValueError("lexsync: counterbalance.optimise needs "
                         "counterbalance.lists to be 2 or more.")
    dims = _balance_dims(design)
    if not dims:
        raise ValueError(
            "lexsync: counterbalance.optimise has no dimensions to balance. Name them "
            "in counterbalance.balance_on, or give the design a match_on.")
    absent = sorted((d for d in dims if d not in stimuli.columns),
                    key=lambda s: s.encode("utf-8"))
    if absent:
        raise ValueError("lexsync: cannot balance on column(s) the stimuli do not "
                         "have: %s." % ", ".join("'%s'" % a for a in absent))

    scale = _balance_scale(design)
    sets = sorted(stimuli["set"].unique())
    n_sets = len(sets)
    V = _balance_values(stimuli, dims, scale)
    totals = {d: sum(v) for d, v in V.items()}
    # Bounded on the sum of the absolute set totals, which bounds every list's total.
    # Under mad the values are deviations whose total is near zero, so a bound on the
    # total would bound nothing. For values of one sign the two bounds are the same.
    if (max((sum(abs(v) for v in vals) for vals in V.values()), default=0) * n_sets
            > BALANCE_MAX_MAGNITUDE):
        raise ValueError(
            "lexsync: the balance objective would exceed the exact-integer range.")

    # Start from the default deal, so the search improves on the shipped behaviour
    # rather than starting somewhere unrelated to it.
    assign = [(i % n_lists) + 1 for i in range(n_sets)]
    cost0 = _balance_cost(V, assign, n_lists, n_sets)

    pairs = [(i, j) for i in range(n_sets - 1) for j in range(i + 1, n_sets)]
    max_passes = int(cb.get("max_passes", 500))
    n_swaps = 0
    cost = cost0
    hit_bound = False
    for pass_i in range(1, max(0, max_passes) + 1):
        cand = [(i, j) for i, j in pairs if assign[i] != assign[j]]
        if not cand:
            break
        n_in = [sum(1 for a in assign if a == l) for l in range(1, n_lists + 1)]
        S = {d: [sum(v for v, a in zip(V[d], assign, strict=True) if a == l)
                 for l in range(1, n_lists + 1)] for d in dims}
        best = 0
        tied = []
        for i, j in cand:
            la, lb = assign[i], assign[j]
            delta = 0
            for d in dims:
                va, vb = V[d][i], V[d][j]
                Sa, Sb = S[d][la - 1], S[d][lb - 1]
                sha = totals[d] * n_in[la - 1]
                shb = totals[d] * n_in[lb - 1]
                delta += (abs((Sa - va + vb) * n_sets - sha)
                          + abs((Sb - vb + va) * n_sets - shb)
                          - abs(Sa * n_sets - sha)
                          - abs(Sb * n_sets - shb))
            if delta < best:
                best, tied = delta, [(i, j)]
            elif delta == best and tied:
                tied.append((i, j))
        if best >= 0 or not tied:
            break
        if len(tied) > 1:
            # Hash tie-break, not position: taking the first tied pair would
            # systematically prefer low-numbered sets, and the digest is the package's
            # established way of choosing without a generator.
            def _h(pair):
                key = "|".join([_key_part(seed), "balance",
                                _key_part(sets[pair[0]]), _key_part(sets[pair[1]])])
                return hashlib.sha256(key.encode("utf-8")).hexdigest()
            pick = sorted(tied, key=_h)[0]
        else:
            pick = tied[0]
        i, j = pick
        assign[i], assign[j] = assign[j], assign[i]
        cost += best
        n_swaps += 1
        if pass_i == max_passes:
            hit_bound = True

    report = {
        "dimensions": list(dims), "cost_before": int(cost0),
        "cost_after": int(cost), "n_swaps": int(n_swaps),
        "max_passes_reached": hit_bound,
        "cost_unit": _BALANCE_COST_UNITS[scale],
    }
    # Recorded only off the default, so the record of a design that names no scale
    # reads as it did before the option. Its cost_unit names the mean.
    if scale != "mean":
        report["scale"] = scale
    return {"list_of_set": {s: int(a) for s, a in zip(sets, assign, strict=True)},
            "report": report}

lexsync.participant_table(factors, n_participants)

Source code in src/lexsync/counterbalancing.py

def participant_table(factors: dict, n_participants: int) -> pd.DataFrame:
    keys = list(factors.keys())
    # R's expand.grid() varies the first factor fastest, itertools.product the
    # last; cross the reversed keys and unreverse each cell so a participant
    # number is allocated the same cell by either engine.
    grid = [cell[::-1] for cell in product(*[factors[k] for k in reversed(keys)])]
    rows = [dict(zip(keys, grid[i % len(grid)], strict=True)) for i in range(n_participants)]
    df = pd.DataFrame(rows)
    df["participant"] = np.arange(1, n_participants + 1)
    return df

Validation and equivalence

A matched design claims that its controls do not differ, and a non-significant test of difference does not establish that. These functions report the realised control instead: the standardised difference with its interval, an equivalence test against a declared bound, and the variance ratio that a mean-based statistic would miss.

lexsync.match_report(stimuli, dims, schema, roles=None)

Build the full match-quality report: descriptives and comparisons.

Every pair of conditions is compared on every dimension. The rows against the first condition in order of appearance, the anchor, come first, then every further pair in condition order, so a design with a single condition has nothing to compare and comparisons comes back with its columns and no rows. A dimension with no numeric value among the stimuli, such as n_syllables for Chinese or a text label, is left out of both frames.

Each row of comparisons names the earlier condition of its pair in reference and the later one in condition. The signed statistics run from the reference to the other condition: cohens_d, d_ci_low and d_ci_high are the reference mean minus the condition mean, in pooled standard deviations, so a positive d means the reference scores higher. mean_diff is the same difference in the dimension's own unit, to three places. var_ratio is the other way up, the condition variance over the reference variance.

roles (optional) is a function of (condition, reference, dimension) that returns the row's role: manipulated when the design sets the two conditions apart on the dimension, controlled when it matches them on it and uncontrolled otherwise. run_pipeline derives it from the design. Without it, role is missing. Stimuli without a condition column, or without a dimension named in dims, are refused.

Source code in src/lexsync/validation.py

def match_report(stimuli: pd.DataFrame, dims, schema: dict, roles=None) -> dict:
    """Build the full match-quality report: descriptives and comparisons.

    Every pair of conditions is compared on every dimension. The rows against the
    first condition in order of appearance, the anchor, come first, then every further
    pair in condition order, so a design with a single condition has nothing to
    compare and ``comparisons`` comes back with its columns and no rows. A dimension
    with no numeric value among the stimuli, such as ``n_syllables`` for Chinese or a
    text label, is left out of both frames.

    Each row of ``comparisons`` names the earlier condition of its pair in
    ``reference`` and the later one in ``condition``. The signed statistics run from
    the reference to the other condition: ``cohens_d``, ``d_ci_low`` and ``d_ci_high``
    are the reference mean minus the ``condition`` mean, in pooled standard
    deviations, so a positive *d* means the reference scores higher. ``mean_diff`` is
    the same difference in the dimension's own unit, to three places. ``var_ratio``
    is the other way up, the ``condition`` variance over the reference variance.

    ``roles`` (optional) is a function of ``(condition, reference, dimension)`` that
    returns the row's ``role``: ``manipulated`` when the design sets the two
    conditions apart on the dimension, ``controlled`` when it matches them on it and
    ``uncontrolled`` otherwise. ``run_pipeline`` derives it from the design. Without
    it, ``role`` is missing. Stimuli without a ``condition`` column, or without a
    dimension named in ``dims``, are refused.
    """
    _require_columns(stimuli, [], "condition")
    conds = list(dict.fromkeys(stimuli["condition"]))
    dims = _reportable(stimuli, dims)
    desc = describe_stimuli(stimuli, dims)
    bound = (schema.get("equivalence") or {}).get("bound_d", 0.5)
    alpha = (schema.get("equivalence") or {}).get("alpha", 0.05)
    rows = []
    for ref, cc in _condition_pairs(conds):
        for dim in dims:
            x = _numeric(stimuli.loc[stimuli["condition"] == ref, dim])
            y = _numeric(stimuli.loc[stimuli["condition"] == cc, dim])
            tt = tost_equiv(x, y, bound, alpha)
            ci = cohens_d_ci(x, y, alpha)
            vr = variance_ratio(y, x)
            p = tt["p"]
            # An undefined d and its interval serialise as missing cells (empty in
            # the CSV, null in the datasheet), exactly like the other missing stats.
            d = cohens_d(x, y)
            lo, hi = ci["ci_low"], ci["ci_high"]
            # The raw difference needs one value a side, where d needs two.
            xv, yv = x.dropna(), y.dropna()
            diff = _exact_mean(xv) - _exact_mean(yv) if len(xv) and len(yv) else None
            rows.append(dict(
                condition=cc, reference=ref, dimension=dim,
                cohens_d=_round_dp(d, 3) if d is not None else None,
                d_ci_low=_round_dp(lo, 3) if lo is not None and lo == lo else None,
                d_ci_high=_round_dp(hi, 3) if hi is not None and hi == hi else None,
                var_ratio=_round_dp(vr, 3) if vr is not None else None,
                tost_p=_round_dp(p, 4) if p == p else None,
                equivalent=tt["equivalent"],
                mean_diff=_round_dp(diff, 3) if diff is not None else None,
                role=roles(cc, ref, dim) if roles is not None else None,
            ))
    return dict(descriptives=desc,
                comparisons=pd.DataFrame(rows, columns=list(_COMPARISON_COLUMNS)) if rows
                else pd.DataFrame(columns=list(_COMPARISON_COLUMNS)))

lexsync.describe_stimuli(stimuli, dims, by='condition')

Per-group descriptive statistics for several dimensions.

One row per group of by, in order of first appearance, and dimension, with n, mean, sd, min, median and max. A grouping column or a dimension the stimuli do not carry is refused. Mirrors describe_stimuli in R_workflow/R/validation.R.

Source code in src/lexsync/validation.py

def describe_stimuli(stimuli: pd.DataFrame, dims, by: str = "condition") -> pd.DataFrame:
    """Per-group descriptive statistics for several dimensions.

    One row per group of ``by``, in order of first appearance, and dimension, with
    ``n``, ``mean``, ``sd``, ``min``, ``median`` and ``max``. A grouping column or a
    dimension the stimuli do not carry is refused. Mirrors describe_stimuli in
    R_workflow/R/validation.R.
    """
    _require_columns(stimuli, dims, by)
    rows = []
    for g, d in stimuli.groupby(by, sort=False):
        for dim in dims:
            x = pd.to_numeric(d[dim], errors="coerce").dropna()
            rows.append(dict(
                group=g, dimension=dim, n=int(x.size),
                mean=_round_dp(_exact_mean(x), 3), sd=_round_dp(_exact_sd(x), 3),
                min=_round_dp(float(x.min()), 3),
                # _exact_median, not pandas' .median(): the latter reduces through
                # numpy, the one reduction here that would bypass the shared exact
                # primitives.
                median=_round_dp(_exact_median(x), 3),
                max=_round_dp(float(x.max()), 3),
            ))
    # The columns even without rows, as when match_report leaves out every dimension
    # it was given, so the frame has the header the R twin's has.
    return pd.DataFrame(rows, columns=["group", "dimension", "n", "mean", "sd", "min",
                                       "median", "max"])

lexsync.balance_check(stimuli, columns)

Source code in src/lexsync/validation.py

def balance_check(stimuli: pd.DataFrame, columns) -> list:
    if isinstance(columns, str):
        columns = [columns]
    issues = []
    for col in columns:
        if col not in stimuli.columns:
            continue
        tab = stimuli[col].value_counts()
        if tab.nunique() > 1:
            detail = ", ".join(f"{k}={v}" for k, v in tab.items())
            issues.append(f"Column '{col}' is unbalanced: {detail}")
    return issues

lexsync.variance_ratio(cond, ref)

Ratio of a condition's variance to the reference's: a distributional balance check that complements the mean-based Cohen's d and TOST.

Two conditions can share a mean yet differ in spread and still confound, which a mean-based statistic misses (Armstrong, Watson & Plaut, 2012; Austin, 2009). A ratio near 1 is balanced; a common heuristic flags ratios outside about [0.5, 2] as unequal spread. Returns None when a variance is undefined.

Source code in src/lexsync/validation.py

def variance_ratio(cond, ref):
    """Ratio of a condition's variance to the reference's: a distributional
    balance check that complements the mean-based Cohen's d and TOST.

    Two conditions can share a mean yet differ in spread and still confound, which
    a mean-based statistic misses (Armstrong, Watson & Plaut, 2012; Austin, 2009).
    A ratio near 1 is balanced; a common heuristic flags ratios outside about
    [0.5, 2] as unequal spread. Returns ``None`` when a variance is undefined.
    """
    cond = np.asarray(cond, dtype=float); ref = np.asarray(ref, dtype=float)
    cond = cond[~np.isnan(cond)]; ref = ref[~np.isnan(ref)]
    if len(cond) < 2 or len(ref) < 2:
        return None
    v_ref = _exact_var(ref)
    if v_ref == 0:
        return 1.0 if _exact_var(cond) == 0 else None
    return float(_exact_var(cond) / v_ref)

lexsync.cohens_d(x, y)

Cohen's d, the pooled-SD standardised mean difference of x and y.

None when either sample has fewer than two values, since one value has no spread and the pooled SD is then undefined, and when the pooled SD is zero but the means differ. Two samples of one shared constant give 0. Mirrors cohens_d in R_workflow/R/validation.R.

Source code in src/lexsync/validation.py

def cohens_d(x, y):
    """Cohen's *d*, the pooled-SD standardised mean difference of ``x`` and ``y``.

    ``None`` when either sample has fewer than two values, since one value has no
    spread and the pooled SD is then undefined, and when the pooled SD is zero but
    the means differ. Two samples of one shared constant give 0. Mirrors cohens_d in
    R_workflow/R/validation.R.
    """
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        # Undefined, not 0: a frequency contrast of 6.5 against 3.9 Zipf with one item
        # a side was reported as a perfect balance.
        return None
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    if sp == 0 or math.isnan(sp):
        # Two constants: an exactly-zero difference is exactly zero SDs apart, but
        # unequal constants are infinitely many. That is undefined, not perfect balance.
        if float(_exact_mean(x) - _exact_mean(y)) == 0:
            return 0.0
        return None
    return float((_exact_mean(x) - _exact_mean(y)) / sp)

lexsync.cohens_d_ci(x, y, alpha=0.05)

Cohen's d with a confidence interval, complementing the TOST verdict.

The interval is the (1 - 2 * alpha) confidence interval for the standardised mean difference, so alpha = 0.05 gives the 90% interval that goes with a two one-sided tests (TOST) decision at the .05 level (Lakens, 2017). Reporting the interval, rather than only a binary "equivalent / not" verdict, makes the realised imbalance and its sampling uncertainty explicit and keeps the dependence on the number of items visible: with few items the interval is wide, whatever the point estimate. The upper limit of the interval on |d| is the largest imbalance still consistent with the stimuli. Both the interval and the TOST treat the selected items as a sample, although a test on stimuli concerns a closed population. Sassenhagen and Alday (2016) recommend describing the stimuli and entering potential confounds in the analysis as covariates.

The limits are d +/- t * SE(d), where t is the Student t quantile on nx + ny - 2 degrees of freedom and SE(d) is the large-sample standard error of d, sqrt(1/nx + 1/ny + d**2 / (2 * (nx + ny))) (Hedges & Olkin, 1985; Borenstein et al., 2009, eq. 4.20). The d**2 term carries the sampling error of the pooled standard deviation. It is negligible for a well-matched control, where d is near zero, but it dominates for a manipulated dimension and can more than double the width of that interval. TOST tests the raw mean difference, so the interval is slightly wider than the one its decision implies, and when a control's limit lies close to the bound, tost_p gives the verdict.

The estimate and both limits are None when either sample has fewer than two values, as :func:cohens_d is.

Source code in src/lexsync/validation.py

def cohens_d_ci(x, y, alpha: float = 0.05) -> dict:
    """Cohen's *d* with a confidence interval, complementing the TOST verdict.

    The interval is the ``(1 - 2 * alpha)`` confidence interval for the
    standardised mean difference, so ``alpha = 0.05`` gives the 90% interval that
    goes with a two one-sided tests (TOST) decision at the .05 level (Lakens,
    2017). Reporting the interval, rather than only a binary "equivalent / not"
    verdict, makes the realised imbalance and its sampling uncertainty explicit and
    keeps the dependence on the number of items visible: with few items the
    interval is wide, whatever the point estimate. The upper limit of the interval
    on ``|d|`` is the largest imbalance still consistent with the stimuli. Both the
    interval and the TOST treat the selected items as a sample, although a test on
    stimuli concerns a closed population. Sassenhagen and Alday (2016) recommend
    describing the stimuli and entering potential confounds in the analysis as
    covariates.

    The limits are ``d +/- t * SE(d)``, where ``t`` is the Student *t* quantile on
    ``nx + ny - 2`` degrees of freedom and ``SE(d)`` is the large-sample standard
    error of *d*, ``sqrt(1/nx + 1/ny + d**2 / (2 * (nx + ny)))`` (Hedges & Olkin,
    1985; Borenstein et al., 2009, eq. 4.20). The ``d**2`` term carries the
    sampling error of the pooled standard deviation. It is negligible for a
    well-matched control, where *d* is near zero, but it dominates for a
    manipulated dimension and can more than double the width of that
    interval. TOST tests the raw mean difference, so the interval is slightly
    wider than the one its decision implies, and when a control's limit lies
    close to the bound, ``tost_p`` gives the verdict.

    The estimate and both limits are ``None`` when either sample has fewer than two
    values, as :func:`cohens_d` is.
    """
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        return dict(d=None, ci_low=None, ci_high=None)
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    diff = float(_exact_mean(x) - _exact_mean(y))
    if sp == 0 or math.isnan(sp):
        # Equal constants carry no sampling uncertainty: a point at zero. Unequal
        # constants are infinitely many SDs apart, so the estimate and its interval
        # are undefined, not a perfect [0, 0].
        if diff == 0:
            return dict(d=0.0, ci_low=0.0, ci_high=0.0)
        return dict(d=None, ci_low=None, ci_high=None)
    d = diff / sp
    # Large-sample variance of d: 1/nx + 1/ny + d**2 / (2 * (nx + ny)) (Hedges &
    # Olkin, 1985; Borenstein et al., 2009, eq. 4.20). Without the d**2 term the
    # interval treated the pooled SD as known, which gave a manipulated dimension
    # with d = 5.6 and 40 items a side a margin of 0.37 in place of 0.83. Written
    # with d * d, not d ** 2, and summed in the same order as validation.R, so both
    # engines round every step identically.
    se = math.sqrt(1 / nx + 1 / ny + d * d / (2 * (nx + ny)))
    margin = float(stats.t.ppf(1 - alpha, nx + ny - 2) * se)
    return dict(d=float(d), ci_low=d - margin, ci_high=d + margin)

lexsync.tost_equiv(x, y, bound_d=0.5, alpha=0.05)

Source code in src/lexsync/validation.py

def tost_equiv(x, y, bound_d: float = 0.5, alpha: float = 0.05) -> dict:
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        return dict(p=float("nan"), equivalent=None)
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    if sp == 0 or math.isnan(sp):
        # Both conditions are constants (e.g. a dimension fixed by the pool, such
        # as two-character Chinese words). They are equivalent iff they share that
        # constant; the standardised difference is then exactly zero.
        if float(_exact_mean(x) - _exact_mean(y)) == 0:
            return dict(p=0.0, equivalent=True)
        return dict(p=1.0, equivalent=False)
    se = sp * math.sqrt(1 / nx + 1 / ny)
    if se == 0 or math.isnan(se):
        return dict(p=float("nan"), equivalent=None)
    bound = bound_d * sp
    dfree = nx + ny - 2
    diff = _exact_mean(x) - _exact_mean(y)
    p = max(stats.t.sf((diff + bound) / se, dfree), stats.t.cdf((diff - bound) / se, dfree))
    return dict(p=float(p), equivalent=bool(p < alpha))

Experiment generation

All three targets are rendered from the same event list. Generation imports neither PsychoPy nor pyserial, so it needs no laboratory hardware. The experiment extra is for running the result. assign_triggers is reached as lexsync.scripting.assign_triggers, and export_experiments calls it for you. Each participant is presented one presentation version of a design, and lexsync.scripting.presentation_versions lists them.

lexsync.export_experiments(stimuli, design, schema, outdir, base=None, conditions=None)

Write every presentation target (PsychoPy, OpenSesame, jsPsych).

Assigns the EEG trigger codes with assign_triggers and then writes each target. The condition codes follow conditions, which defaults to the order of the design's conditions entries, so in a design that declares its conditions the first is 101, the second 102, and so on, whatever order the counterbalanced trials put them in. A design with no conditions block (a generated lexical decision, an item table) falls back to the order of first appearance, unless conditions is given. assign_triggers is given the design's events and the schema, so the item codes of a design that sends them alongside other codes are kept apart from those codes, and a code of the design's own that the stream already uses is reported.

Each trial that names no answer of its own takes the key the design's response mapping gives its condition in its list, so the loop tables can score every trial. A lexical decision maps word to its response event's first key and pseudoword to its second unless the design gives response_mapping, and rotate_mapping_by_list: true rotates the keys across lists. A practice or filler trial keeps the answer its table gives. The stimuli passed in are not changed. Every target shows the design's instructions, or lexsync's English instructions for its task, and waits for the space bar before the first trial.

Returns the generated paths by target: psychopy, opensesame (version 1's experiment when a design has several, see :func:export_opensesame) and jspsych. files lists every file written, in a fixed order: the PsychoPy script, each OpenSesame experiment in version order, the jsPsych page, the PsychoPy conditions file, and then each OpenSesame experiment's loop table and text table.

Source code in src/lexsync/scripting.py

    names = [str(c["name"]) for c in conds if isinstance(c, dict) and c.get("name") is not None]
    return names or None


def export_experiments(stimuli, design, schema, outdir, base=None, conditions=None) -> dict:
    """Write every presentation target (PsychoPy, OpenSesame, jsPsych).

    Assigns the EEG trigger codes with ``assign_triggers`` and then writes each
    target. The condition codes follow ``conditions``, which defaults to the order
    of the design's ``conditions`` entries, so in a design that declares its
    conditions the first is 101, the second 102, and so on, whatever order the
    counterbalanced trials put them in. A design with no ``conditions`` block (a
    generated lexical decision, an item table) falls back to the order of first
    appearance, unless ``conditions`` is given. ``assign_triggers`` is given the
    design's events and the schema, so the item codes of a design that sends them
    alongside other codes are kept apart from those codes, and a code of the design's
    own that the stream already uses is reported.

    Each trial that names no ``answer`` of its own takes the key the design's response
    mapping gives its condition in its list, so the loop tables can score every trial.
    A lexical decision maps word to its response event's first key and pseudoword to
    its second unless the design gives ``response_mapping``, and
    ``rotate_mapping_by_list: true`` rotates the keys across lists. A practice or filler
    trial keeps the answer its table gives. The stimuli passed in are not changed. Every
    target shows the design's ``instructions``, or lexsync's English instructions for its
    task, and waits for the space bar before the first trial.

    Returns the generated paths by target: ``psychopy``, ``opensesame`` (version 1's
    experiment when a design has several, see :func:`export_opensesame`) and
    ``jspsych``. ``files`` lists every file written, in a fixed order: the PsychoPy
    script, each OpenSesame experiment in version order, the jsPsych page, the PsychoPy
    conditions file, and then each OpenSesame experiment's loop table and text table.
    """
    if conditions is None:
        conditions = _design_condition_names(design)
    stimuli = assign_triggers(stimuli, conditions, events=resolve_events(design), schema=schema)
    stimuli = _assign_answers(stimuli, design)[0]
    script, conditions_file = _psychopy_files(stimuli, design, schema, outdir, base)
    experiments, tables = _opensesame_files(stimuli, design, schema, outdir, base)
    page = export_jspsych(stimuli, design, schema, outdir, base)
    return {
        "psychopy": script,

lexsync.export_psychopy(stimuli, design, schema, outdir, base=None)

Write a runnable PsychoPy script and the conditions file it reads.

The conditions file, <base>_psychopy.csv, holds every presentation version (see :func:presentation_versions), and the script presents one per run: the version given by --version (or --list when there are no replicates), else the participant's by the rule ((participant - 1) mod V) + 1 from --participant, else what a start-up dialogue asks for. A version the script does not have stops the run with a message. With counterbalance.orders: m the conditions file holds each trial's position in each of m orders, and the script presents the version in order (((participant - 1) div V) mod m) + 1, or in order 1 without a participant number. The data file then records the order.

Before the first trial, the script shows the version's instructions, the design's instructions or lexsync's for its task, and waits for the space bar, which also lets the experimenter start the run. Escape there stops the run before anything is recorded. The text is embedded as JSON data, one text per version.

Each run writes data/<base>_p<participant>_v<version>_<date>-<time>.csv beside the script, without the _p part when no participant number is given, one row per trial, each written as its trial ends. A row holds the run, the trial's fields and trigger codes, the response and its reaction time from the onset of the stimulus it answers, whether the window closed without one, whether the key was the trial's answer, the self-paced reading times, the question's response and reaction time, the measured refresh rate, the trigger mode the run used, and each event's realised flips and onset.

Each trigger code goes out on its stimulus's first flip and is held for triggers.trigger_hold_ms, in whole flips at the measured refresh and never under 10 ms, through the events after it when its own is shorter, unless the next onset's code replaces it first. The session markers, 254 before the first trial and 255 after the last, are held the same way.

The script runs full screen unless presentation.fullscreen is false, when it opens a 1280 x 1024 window, and draws its stimuli presentation.text_height_px pixels high, 48 by default. It writes the codes to the parallel port at triggers.parallel_address or, with triggers.backend: serial, as single bytes to triggers.serial_port at triggers.baudrate. triggers.mode decides what happens when that device cannot be opened. Under auto, the script prints the codes instead and, before the instructions, shows a notice that the experimenter dismisses with the C key. Under hardware, it stops before its window opens, and under mock it prints the codes without trying the device. The data file's trigger_mode column records hardware or mock.

Parameters:

Name	Type	Description	Default
stimuli	DataFrame	Stimuli with trigger columns (see :func:assign_triggers).	required
design	dict	A parsed design configuration.	required
schema	dict	The parsed global schema (trigger and presentation settings).	required
outdir	str	Output directory.	required
base	str | None	Optional file-name stem, by default the design's :func:design_stem.	None

Returns:

Type	Description
str	The path to the generated .py.

Source code in src/lexsync/scripting.py

def export_psychopy(stimuli: pd.DataFrame, design: dict, schema: dict, outdir: str,
                    base: str | None = None) -> str:
    """Write a runnable PsychoPy script and the conditions file it reads.

    The conditions file, ``<base>_psychopy.csv``, holds every presentation version
    (see :func:`presentation_versions`), and the script presents one per run: the
    version given by ``--version`` (or ``--list`` when there are no replicates), else
    the participant's by the rule ``((participant - 1) mod V) + 1`` from
    ``--participant``, else what a start-up dialogue asks for. A version the script does
    not have stops the run with a message. With ``counterbalance.orders: m`` the
    conditions file holds each trial's position in each of m orders, and the script
    presents the version in order ``(((participant - 1) div V) mod m) + 1``, or in order 1
    without a participant number. The data file then records the order.

    Before the first trial, the script shows the version's instructions, the design's
    ``instructions`` or lexsync's for its task, and waits for the space bar, which also
    lets the experimenter start the run. Escape there stops the run before anything is
    recorded. The text is embedded as JSON data, one text per version.

    Each run writes ``data/<base>_p<participant>_v<version>_<date>-<time>.csv`` beside
    the script, without the ``_p`` part when no participant number is given, one row
    per trial, each written as its trial ends. A row holds the run, the trial's fields
    and trigger codes, the response and its reaction time from the onset of the
    stimulus it answers, whether the window closed without one, whether the key was
    the trial's ``answer``, the self-paced reading times, the question's response and
    reaction time, the measured refresh rate, the trigger mode the run used, and each
    event's realised flips and onset.

    Each trigger code goes out on its stimulus's first flip and is held for
    ``triggers.trigger_hold_ms``, in whole flips at the measured refresh and never under
    10 ms, through the events after it when its own is shorter, unless the next onset's
    code replaces it first. The session markers, 254 before the first trial and 255
    after the last, are held the same way.

    The script runs full screen unless ``presentation.fullscreen`` is false, when it
    opens a 1280 x 1024 window, and draws its stimuli ``presentation.text_height_px``
    pixels high, 48 by default. It writes the codes to the parallel port at
    ``triggers.parallel_address`` or, with ``triggers.backend: serial``, as single bytes
    to ``triggers.serial_port`` at ``triggers.baudrate``. ``triggers.mode`` decides what
    happens when that device cannot be opened. Under ``auto``, the script prints the
    codes instead and, before the instructions, shows a notice that the experimenter
    dismisses with the C key. Under ``hardware``, it stops before its window opens, and
    under ``mock`` it prints the codes without trying the device. The data file's
    ``trigger_mode`` column records ``hardware`` or ``mock``.

    Args:
        stimuli: Stimuli with trigger columns (see :func:`assign_triggers`).
        design: A parsed design configuration.
        schema: The parsed global schema (trigger and presentation settings).
        outdir: Output directory.
        base: Optional file-name stem, by default the design's :func:`design_stem`.

    Returns:
        The path to the generated ``.py``.
    """
    return _psychopy_files(stimuli, design, schema, outdir, base)[0]

lexsync.export_opensesame(stimuli, design, schema, outdir, base=None)

Write complete plain-text OpenSesame experiments and the two tables each reads.

<base>.osexp loops over <base>_opensesame.csv, which holds numbers only: lexsync_row (the trial's position in loop order, from 1), lexsync_trial, lexsync_list, lexsync_set, the trigger codes, critical_region and any per-trial duration. The presented text, condition, block and answer are in <base>_opensesame_text.csv, which the inline scripts read with Python's csv module, so OpenSesame never evaluates them. Copy both tables with the experiment.

Before the loop, an inline script, lexsync_instructions, shows the design's instructions, or lexsync's for its task, and waits for the space bar. Every trial is logged. A response after a text event is collected by that event's script from the stimulus's onset, which sets response, response_time, timed_out and correct. A response kept after the stimulus with from: offset is a keyboard_response, and a script after it sets timed_out and correct. Self-paced reading sets lexsync_region_rts and a question question_response and question_rt. A script clears these as each trial starts, another copies the trial's labels into lexsync_condition, lexsync_block and lexsync_answer, and a logger item ends the trial.

Every canvas a trial shows is built before the trial runs, in the prepare phase of its script, and each event is timed from the onset its display returns. An event ends half a frame before its duration is up, so that the next display lands on the refresh that ends it. The frame period is measured at start-up and logged as lexsync_frame_ms, with presentation.assumed_refresh_hz as the fallback when the measurement fails. Each trigger code is held for triggers.trigger_hold_ms, at least 10 ms, through the events after it when its own is shorter, unless the next onset's code replaces it first. The session markers, 254 and 255, are sent just before the first trial and just after the last, held the same way.

The setup writes the codes to the parallel port at triggers.parallel_address or, with triggers.backend: serial, as single bytes to triggers.serial_port at triggers.baudrate. When that device cannot be opened, triggers.mode: auto prints the codes instead. An inline script, lexsync_trigger_notice, then says so before any other screen and waits for the C key. hardware stops the experiment there, and mock prints the codes without trying the device. Every trial logs lexsync_trigger_mode, hardware or mock. Text is drawn presentation.text_height_px pixels high, OpenSesame's font_size, and OpenSesame takes full screen from the mode it is launched in.

A design with more than one presentation version (see :func:presentation_versions) gets one experiment per version instead, each with its own two tables holding only that version's trials: <base>_list<k>.osexp, or <base>_rep<r>_list<k>.osexp when the versions span several replicates. Each one checks OpenSesame's subject number at start-up and shows a warning when that participant would normally run another version. A design with counterbalance.orders: m gets one experiment per version and order, <base>_list<k>_order<j>.osexp (<base>_order<j>.osexp for a single version), each with tables sorted by its order, a lexsync_order variable that every trial logs, and a check that compares the participant's order too, order (((p - 1) div V) mod m) + 1. export_experiments lists every file written.

OpenSesame evaluates the cells of a loop table: in 4.x a cell holding { runs as a Python f-string, and in 3.3 a cell starting with = or holding [=...] runs as Python. A value in a numeric column that is not a plain number is therefore refused, naming the column. An empty cell passes. These rules were checked against the sources of OpenSesame 3.3.14 and 4.1.20.

Parameters

stimuli : pandas.DataFrame Stimuli with trigger columns (see lexsync.scripting.assign_triggers). design : dict A parsed design configuration. schema : dict The parsed global schema (trigger and presentation settings). outdir : str Output directory. base : str, optional File-name stem, by default lexsync.design.design_stem(design), built from the design's name and language.

Returns

str The path to the generated .osexp, version 1's when there are several.

Source code in src/lexsync/scripting.py

def export_opensesame(stimuli, design, schema, outdir, base=None) -> str:
    """Write complete plain-text OpenSesame experiments and the two tables each reads.

    ``<base>.osexp`` loops over ``<base>_opensesame.csv``, which holds numbers only:
    ``lexsync_row`` (the trial's position in loop order, from 1), ``lexsync_trial``,
    ``lexsync_list``, ``lexsync_set``, the trigger codes, ``critical_region`` and any
    per-trial duration. The presented text, ``condition``, ``block`` and ``answer`` are
    in ``<base>_opensesame_text.csv``, which the inline scripts read with Python's
    ``csv`` module, so OpenSesame never evaluates them. Copy both tables with the
    experiment.

    Before the loop, an inline script, ``lexsync_instructions``, shows the design's
    ``instructions``, or lexsync's for its task, and waits for the space bar. Every
    trial is logged. A response after a text event is collected by that event's
    script from the stimulus's onset, which sets ``response``, ``response_time``,
    ``timed_out`` and ``correct``. A response kept after the stimulus with
    ``from: offset`` is a keyboard_response, and a script after it sets ``timed_out``
    and ``correct``. Self-paced reading sets ``lexsync_region_rts`` and a question
    ``question_response`` and ``question_rt``. A script clears these as each trial
    starts, another copies the trial's labels into ``lexsync_condition``,
    ``lexsync_block`` and ``lexsync_answer``, and a logger item ends the trial.

    Every canvas a trial shows is built before the trial runs, in the prepare phase of
    its script, and each event is timed from the onset its display returns. An event
    ends half a frame before its duration is up, so that the next display lands on
    the refresh that ends it. The frame period is measured at start-up and logged as
    ``lexsync_frame_ms``, with ``presentation.assumed_refresh_hz`` as the fallback when
    the measurement fails. Each trigger code is held for ``triggers.trigger_hold_ms``,
    at least 10 ms, through the events after it when its own is shorter, unless the
    next onset's code replaces it first. The session markers, 254 and 255, are sent
    just before the first trial and just after the last, held the same way.

    The setup writes the codes to the parallel port at ``triggers.parallel_address`` or,
    with ``triggers.backend: serial``, as single bytes to ``triggers.serial_port`` at
    ``triggers.baudrate``. When that device cannot be opened, ``triggers.mode: auto``
    prints the codes instead. An inline script, ``lexsync_trigger_notice``, then says
    so before any other screen and waits for the C key. ``hardware`` stops the
    experiment there, and ``mock`` prints the codes without trying the device. Every
    trial logs ``lexsync_trigger_mode``, ``hardware`` or ``mock``. Text is drawn
    ``presentation.text_height_px`` pixels high, OpenSesame's ``font_size``, and
    OpenSesame takes full screen from the mode it is launched in.

    A design with more than one presentation version (see
    :func:`presentation_versions`) gets one experiment per version instead, each with
    its own two tables holding only that version's trials: ``<base>_list<k>.osexp``, or
    ``<base>_rep<r>_list<k>.osexp`` when the versions span several replicates. Each one
    checks OpenSesame's subject number at start-up and shows a warning when that
    participant would normally run another version. A design with
    ``counterbalance.orders: m`` gets one experiment per version and order,
    ``<base>_list<k>_order<j>.osexp`` (``<base>_order<j>.osexp`` for a single version),
    each with tables sorted by its order, a ``lexsync_order`` variable that every trial
    logs, and a check that compares the participant's order too, order
    ``(((p - 1) div V) mod m) + 1``. ``export_experiments`` lists every file written.

    OpenSesame evaluates the cells of a loop table: in 4.x a cell holding ``{`` runs as
    a Python f-string, and in 3.3 a cell starting with ``=`` or holding ``[=...]`` runs
    as Python. A value in a numeric column that is not a plain number is therefore
    refused, naming the column. An empty cell passes. These rules were checked against
    the sources of OpenSesame 3.3.14 and 4.1.20.

    Parameters
    ----------
    stimuli : pandas.DataFrame
        Stimuli with trigger columns (see ``lexsync.scripting.assign_triggers``).
    design : dict
        A parsed design configuration.
    schema : dict
        The parsed global schema (trigger and presentation settings).
    outdir : str
        Output directory.
    base : str, optional
        File-name stem, by default ``lexsync.design.design_stem(design)``, built
        from the design's name and language.

    Returns
    -------
    str
        The path to the generated ``.osexp``, version 1's when there are several.
    """
    return _opensesame_files(stimuli, design, schema, outdir, base)[0][0]

lexsync.export_jspsych(stimuli, design, schema, outdir, base=None)

A browser-runnable jsPsych experiment from the event list.

The same rendered events and the trial data are embedded in one HTML file, so anyone can reproduce the exact procedure online from the same materials. The jsPsych library and stylesheet are loaded from a CDN, so the machine running the file needs an internet connection; the trial data are embedded and the responses are saved locally, so no server is required either to run it or to collect them. Each of the three files carries its SHA-384 as Subresource Integrity, so a browser refuses a copy that differs from the published release. Stimuli are drawn presentation.text_height_px pixels high, 48 by default. Onset triggers are recorded in each trial's data (a browser cannot drive a parallel port); online EEG synchronisation needs WebSerial/LSL or a photodiode. A stimulus that collects its response is one trial, recorded as target: it accepts the keys from its onset, shows the stimulus for its duration and stays open for the timeout after it, a key ends it, and rt runs from the onset.

The page embeds every presentation version (see :func:presentation_versions) and runs one, named in its address: ?version=K, ?list=K when there are no replicates, or ?participant=N, whose version is ((N - 1) mod V) + 1. A page with more than one version, opened with none of these, shows a screen naming them and runs nothing. With counterbalance.orders: m the page presents the version in order (((N - 1) div V) mod m) + 1, or in order 1 without a participant number, and every data row carries the order. Its welcome screen shows the version's instructions, the design's instructions or lexsync's for its task, and waits for the space bar. The data rows carry the participant, the version, the list and the replicate, and the data file is named after the participant and the version. The page names the keys as the browser does, arrowleft for left, in its events and in each trial's answer, so a recorded response can be compared with the answer as it stands.

Source code in src/lexsync/scripting.py

def export_jspsych(stimuli, design, schema, outdir, base=None) -> str:
    """A browser-runnable jsPsych experiment from the event list.

    The same rendered events and the trial data are embedded in one HTML file, so
    anyone can reproduce the exact procedure online from the same materials. The
    jsPsych library and stylesheet are loaded from a CDN, so the machine running the
    file needs an internet connection; the trial data are embedded and the responses
    are saved locally, so no server is required either to run it or to collect them.
    Each of the three files carries its SHA-384 as Subresource Integrity, so a browser
    refuses a copy that differs from the published release. Stimuli are drawn
    ``presentation.text_height_px`` pixels high, 48 by default.
    Onset triggers are recorded in each trial's data (a browser cannot drive a
    parallel port); online EEG synchronisation needs WebSerial/LSL or a photodiode.
    A stimulus that collects its response is one trial, recorded as ``target``: it
    accepts the keys from its onset, shows the stimulus for its duration and stays
    open for the timeout after it, a key ends it, and ``rt`` runs from the onset.

    The page embeds every presentation version (see :func:`presentation_versions`) and
    runs one, named in its address: ``?version=K``, ``?list=K`` when there are no
    replicates, or ``?participant=N``, whose version is ``((N - 1) mod V) + 1``. A page
    with more than one version, opened with none of these, shows a screen naming them
    and runs nothing. With ``counterbalance.orders: m`` the page presents the version in
    order ``(((N - 1) div V) mod m) + 1``, or in order 1 without a participant number,
    and every data row carries the order. Its welcome screen shows the version's instructions, the design's
    ``instructions`` or lexsync's for its task, and waits for the space bar. The data
    rows carry the participant, the version, the list and the replicate, and the data
    file is named after the participant and the version. The page names the keys as the
    browser does, ``arrowleft`` for ``left``, in its events and in each trial's
    ``answer``, so a recorded response can be compared with the answer as it stands.
    """
    base = base or design_stem(design)
    events = resolve_events(design)
    rendered = _map_keys_for_jspsych(_sized_masks(
        render_events(events, design.get("timing") or {}, _refresh_hz(schema)), stimuli))
    versions = presentation_versions(stimuli)
    trials = [{key: _fifteen_digits(value) for key, value in row.items()}
              for row in _map_answers_for_jspsych(
                  loop_table(stimuli, events)).to_dict(orient="records")]
    presentation = schema.get("presentation") or {}
    font = design.get("font") or presentation.get("font") or "Courier New"
    with open(find_template("jspsych/experiment_template.html"), encoding="utf-8") as handle:
        tmpl = handle.read()
    # DESIGN lands in <title> and a comment, LANGUAGE_TAG in a lang attribute, WORD_FONT
    # inside a CSS font-family declaration. Only the JSON blobs, the instructions among
    # them, are escaped by _json_html; the rest are validated, since a quote or an angle
    # bracket in any of them would leave its attribute or rule and start markup in a page
    # that collects participant responses.
    subs = {
        "DESIGN": clean_meta(design["name"], "the design's `name`"),
        "LANGUAGE": clean_meta(design["language"], "the design's `language`"),
        "LANGUAGE_TAG": _language_tag(design), "WORD_FONT": clean_meta(font, "the font"),
        "TEXT_HEIGHT_PX": str(_text_height_px(schema)),
        "EVENTS_JSON": _json_html(rendered), "TRIALS_JSON": _json_html(trials),
        "VERSIONS_JSON": _json_html(versions),
        "ORDERS": str(_n_orders(stimuli)),
        "INSTRUCTIONS_JSON": _json_html(_instruction_texts(design, versions)),
    }
    for k, v in subs.items():

lexsync.scripting.assign_triggers(stimuli, conditions=None, events=None, schema=None)

A per-condition marker and a per-item marker, both 0-255 EEG codes.

condition_trigger numbers the conditions from 101 upwards and item_trigger numbers the item sets from 40, in code-point order of their labels, so with sets 1 to 20, set 10 is 41 and set 2 is 51. Decode an item code through the loop table, not from the set's label. The item range holds 200 codes (an 8-bit-port constraint), so past 200 sets the codes wrap and repeat, and a runtime notice says so.

events, the design's event list, says which codes the experiment sends. When it sends item codes together with condition codes, or with a code of its own from 40 to 239, the item codes skip 101 to 100 + k, the codes of the k conditions, and the design's own codes, so that no item takes a code that marks something else. Sets 65 and 66 of a 70-set design took 101 and 102, the condition codes. A design with more item sets than free codes is then refused, unless schema sets triggers.allow_code_overlap: true, which lets the item codes wrap and repeat within the free codes. Every other design, and a call without events, gets the item codes it had before. Each trigger in events is checked as :func:render_events checks it.

A notice is printed for a code of the design's own that the stream already uses for something else: 254 or 255, which the PsychoPy and OpenSesame experiments send when the session starts and ends, a condition code the design sends, or a code within the range of the item codes it sends.

The condition codes follow conditions when it is given: its first entry is 101, its second 102, and so on, whether or not every entry occurs in stimuli, so the same condition keeps the same code in a subset such as a practice block. A condition present in stimuli but missing from conditions takes the next free codes, in code-point order of its label. Either way the mapping depends only on the labels, never on the row order. Without conditions, the codes follow the order in which the conditions first appear in stimuli, as in lexsync 0.1.0. On a shuffled trial list that order comes from the shuffle, so a different seed can swap two conditions' codes. export_experiments passes the design's order, and the pipeline passes the order of the design's conditions, followed by any other condition in the order the item source lists it.

stim = pd.DataFrame({"condition": ["low", "high", "low", "high"], ... "set": [1, 1, 2, 2]}) assign_triggers(stim)["condition_trigger"].tolist() [101, 102, 101, 102] assign_triggers(stim, conditions=["high", "low"])["condition_trigger"].tolist() [102, 101, 102, 101]

Source code in src/lexsync/scripting.py

def assign_triggers(stimuli: pd.DataFrame, conditions=None, events=None,
                    schema=None) -> pd.DataFrame:
    """A per-condition marker and a per-item marker, both 0-255 EEG codes.

    ``condition_trigger`` numbers the conditions from 101 upwards and
    ``item_trigger`` numbers the item sets from 40, in code-point order of their
    labels, so with sets 1 to 20, set 10 is 41 and set 2 is 51. Decode an item code
    through the loop table, not from the set's label. The item range holds 200 codes
    (an 8-bit-port constraint), so past 200 sets the codes wrap and repeat, and a
    runtime notice says so.

    ``events``, the design's event list, says which codes the experiment sends. When
    it sends item codes together with condition codes, or with a code of its own from
    40 to 239, the item codes skip 101 to 100 + k, the codes of the k conditions, and
    the design's own codes, so that no item takes a code that marks something else.
    Sets 65 and 66 of a 70-set design took 101 and 102, the condition codes. A design
    with more item sets than free codes is then refused, unless ``schema`` sets
    ``triggers.allow_code_overlap: true``, which lets the item codes wrap and repeat
    within the free codes. Every other design, and a call without ``events``, gets the
    item codes it had before. Each trigger in ``events`` is checked as
    :func:`render_events` checks it.

    A notice is printed for a code of the design's own that the stream already uses
    for something else: 254 or 255, which the PsychoPy and OpenSesame experiments send
    when the session starts and ends, a condition code the design sends, or a code
    within the range of the item codes it sends.

    The condition codes follow ``conditions`` when it is given: its first entry is
    101, its second 102, and so on, whether or not every entry occurs in
    ``stimuli``, so the same condition keeps the same code in a subset such as a
    practice block. A condition present in ``stimuli`` but missing from
    ``conditions`` takes the next free codes, in code-point order of its label.
    Either way the mapping depends only on the labels, never on the row order.
    Without ``conditions``, the codes follow the order in which the conditions
    first appear in ``stimuli``, as in lexsync 0.1.0. On a shuffled trial list that
    order comes from the shuffle, so a different seed can swap two conditions'
    codes. ``export_experiments`` passes the design's order, and the pipeline
    passes the order of the design's ``conditions``, followed by any other
    condition in the order the item source lists it.

    >>> stim = pd.DataFrame({"condition": ["low", "high", "low", "high"],
    ...                      "set": [1, 1, 2, 2]})
    >>> assign_triggers(stim)["condition_trigger"].tolist()
    [101, 102, 101, 102]
    >>> assign_triggers(stim, conditions=["high", "low"])["condition_trigger"].tolist()
    [102, 101, 102, 101]
    """
    stimuli, notes = _allocate_codes(stimuli, conditions, events, schema)
    for note in notes:
        print(note)
    return stimuli

lexsync.scripting.presentation_versions(stimuli)

The presentation versions a set of trials holds, in the order they are numbered.

A version is one (replicate, list) pair of the trials, and each participant is presented one version. Version 1 is the first pair in (replicate, list) order, version 2 the next, and so on. A table without a replicate column is one replicate, numbered 1, and one without list is one list. Participant p runs version ((p - 1) mod V) + 1 of V unless another is chosen at launch, the rule participant_table allocates by. Every generated experiment carries this list, and export_opensesame writes one experiment per version when there is more than one. Mirrors .presentation_versions in scripting.R.

Parameters:

Name	Type	Description	Default
stimuli	DataFrame	The presented trials, with the list (and any replicate) column that counterbalance adds.	required

Returns:

Type	Description
list	A list of {"version", "replicate", "list", "n_trials"} dictionaries, one per
list	version, numbered from 1.

Raises:

Type	Description
ValueError	If a list or replicate value is not a whole number.

Examples:

>>> stim = pd.DataFrame({"list": [2, 1, 2], "set": [1, 1, 2]})
>>> [(v["version"], v["list"], v["n_trials"]) for v in presentation_versions(stim)]
[(1, 1, 1), (2, 2, 2)]

Source code in src/lexsync/scripting.py

def presentation_versions(stimuli: pd.DataFrame) -> list:
    """The presentation versions a set of trials holds, in the order they are numbered.

    A version is one (replicate, list) pair of the trials, and each participant is
    presented one version. Version 1 is the first pair in (replicate, list) order,
    version 2 the next, and so on. A table without a ``replicate`` column is one
    replicate, numbered 1, and one without ``list`` is one list. Participant ``p`` runs
    version ``((p - 1) mod V) + 1`` of ``V`` unless another is chosen at launch, the
    rule ``participant_table`` allocates by. Every generated experiment carries this
    list, and ``export_opensesame`` writes one experiment per version when there is
    more than one. Mirrors .presentation_versions in scripting.R.

    Args:
        stimuli: The presented trials, with the ``list`` (and any ``replicate``)
            column that ``counterbalance`` adds.

    Returns:
        A list of ``{"version", "replicate", "list", "n_trials"}`` dictionaries, one per
        version, numbered from 1.

    Raises:
        ValueError: If a ``list`` or ``replicate`` value is not a whole number.

    Examples:
        >>> stim = pd.DataFrame({"list": [2, 1, 2], "set": [1, 1, 2]})
        >>> [(v["version"], v["list"], v["n_trials"]) for v in presentation_versions(stim)]
        [(1, 1, 1), (2, 2, 2)]
    """
    keys = _version_keys(stimuli)
    return [{"version": k, "replicate": rep, "list": lst,
             "n_trials": sum(1 for key in keys if key == (rep, lst))}
            for k, (rep, lst) in enumerate(sorted(set(keys)), start=1)]

Materials datasheet

The datasheet is the provenance record that travels with a stimulus set: where the items came from, how they were selected, the realised control, the candidate-pool sizes, the checksums and the versions. run_pipeline builds and writes one for every run.

lexsync.build_datasheet(design, schema, report, stimuli, source_path, artifacts, seed, engine='python', candidate_pool=None, norms=None, balance=None, blocks=None, design_path=None, schema_path=None, selection_audit=None, neighbourhood_reference=None, counterbalance_check=None, input_normalisation=None, generation_exclude=None, exclusions=None, zipf_derivation=None, trial_order=None)

Assemble the datasheet dictionary from the pipeline's objects.

For a corpus or generated design, materials_source["provenance"] credits the lexicon that source_path names. A lexicon lexsync ships is credited by its source, a corpus fetched into the cache by the registry's citation, and any other file by its file name and checksum (see _lexicon_provenance).

candidate_pool (optional) is a list of {"condition", "n_candidates"} recording how many items satisfied each condition's window before matching, the size of the discretionary pool the selection drew from, reported so that item-selection bias is auditable (Forster, 2000; Simmons et al., 2011).

norms (optional) is a list of norm-table provenance records from the design's norms: block. Each names a file, its sha256, the join key and the per-column coverage. Recorded because a norm table can supply the very variable a design manipulates, so a record that omitted it would describe a selection over columns of unstated origin.

balance (optional) is the balance-optimiser report from balance_lists. Recorded because it decides which items each participant sees.

design_path / schema_path (optional) are the design and schema files the run read; when given, their sha256 checksums complete the reproducibility record, because those two files decide everything the seed does not. The schema's path is recorded too, in POSIX form, since an unnamed schema is chosen from several candidates.

selection_audit (optional) is the matcher's audit record; its window_relaxations entries are recorded because a relaxed window changes what "matched" means for that condition. Its incomplete_dropped entries ({"condition", "n"}) count the candidates excluded for missing a matched dimension, which candidate_pool still includes. They are recorded because the selection drew on a smaller pool than the window holds.

neighbourhood_reference (optional) records a supplied reference the derived dimensions were recomputed against ({"source", "n_words", "sha256", "applied_to"}, where applied_to lists the recomputed columns), verbatim.

counterbalance_check (optional) is the pipeline's counterbalance_check report on the counterbalanced table. Without it, the check is run here on stimuli. The record gives the recipe that ran and the lists the table holds, or for a table without a list column the lists counterbalance() would deal. It says whether a Latin-square rotation is complete and counts the trials per list and condition. It also gives the presentation versions, one per (replicate, list) pair, and the rule that assigns each participant one of them.

input_normalisation (optional) is {"changed", "collapsed"}: how many text keys the run's inputs held that Unicode NFC normalisation rewrote, and how many duplicates the rewriting created and removed. Recorded only when given, which the pipeline does only when either count is above zero, because a merged spelling changes the candidate pool.

artifacts names the files written for the design: stimuli, descriptives, comparisons and experiments, the result of export_experiments. Each file is recorded with its sha256: the three tables, then every file in experiments["files"], the loop and text tables included, or the three experiment paths of a result that has no files.

generation_exclude (optional, generated designs) lists the known-form lists of items.generation.exclude as {"source", "sha256", "n_forms"} records. A pseudoword was refused for equalling any form in them, so they decide which pseudowords a run produces, as the lexicon does.

exclusions (optional, corpus, pool and generated designs) records the design's exclude_words as {"path", "sha256", "n_listed", "n_removed"}. It gives the file, or "inline" for a list in the design, its checksum, the number of distinct words it lists and the number of candidates it removed. Recorded as materials_source["exclusions"], because the words decide which candidates the selection could draw.

zipf_derivation (optional) is {"n_values", "near_boundary"} for a lexicon carrying freq_permillion, as normalise_lexicon writes one: how many positive rates it holds, and how many put Zipf = log10(per million) + 3 within 1e-9 of a three-place rounding boundary, where the two engines' log10 could round a derived value apart. Recorded as materials_source["zipf_derivation"].

trial_order (optional) is the trial-order record of the presented trials, which the pipeline passes. Without it, the record is taken from stimuli. It gives counterbalancing["max_run"] ({"k", "run_on", "longest"}, the longest run realised in each presentation version) when the design limits runs, and counterbalancing["orders"] ({"m", "rule"}) when it asks for more than one trial order.

design["n_per_condition"] is the request the selector applied, the selector's default when the design gives none, and items["n_per_condition_realised"] counts the items each condition holds in stimuli, per replicate for resampled sets, so a selection that fell short of the request under shortfall: allow shows it. The suggested analysis keys its by-item slope on the recipe that ran and adds a matched design's matched dimensions that vary among its items as covariates.

Source code in src/lexsync/datasheet.py

def build_datasheet(design, schema, report, stimuli, source_path, artifacts,
                    seed, engine="python", candidate_pool=None, norms=None,
                    balance=None, blocks=None, design_path=None, schema_path=None,
                    selection_audit=None, neighbourhood_reference=None,
                    counterbalance_check=None, input_normalisation=None,
                    generation_exclude=None, exclusions=None, zipf_derivation=None,
                    trial_order=None) -> dict:
    """Assemble the datasheet dictionary from the pipeline's objects.

    For a corpus or generated design, ``materials_source["provenance"]`` credits the
    lexicon that ``source_path`` names. A lexicon lexsync ships is credited by its
    source, a corpus fetched into the cache by the registry's citation, and any other
    file by its file name and checksum (see ``_lexicon_provenance``).

    ``candidate_pool`` (optional) is a list of ``{"condition", "n_candidates"}``
    recording how many items satisfied each condition's window before matching,
    the size of the discretionary pool the selection drew from, reported so that
    item-selection bias is auditable (Forster, 2000; Simmons et al., 2011).

    ``norms`` (optional) is a list of norm-table provenance records from the design's
    ``norms:`` block. Each names a file, its sha256, the join key and the per-column
    coverage. Recorded because a norm table can supply the very variable a design
    manipulates, so a record that omitted it would describe a selection over columns
    of unstated origin.

    ``balance`` (optional) is the balance-optimiser report from ``balance_lists``.
    Recorded because it decides which items each participant sees.

    ``design_path`` / ``schema_path`` (optional) are the design and schema files the
    run read; when given, their sha256 checksums complete the reproducibility
    record, because those two files decide everything the seed does not. The
    schema's path is recorded too, in POSIX form, since an unnamed schema is chosen
    from several candidates.

    ``selection_audit`` (optional) is the matcher's audit record; its
    ``window_relaxations`` entries are recorded because a relaxed window changes
    what "matched" means for that condition. Its ``incomplete_dropped`` entries
    (``{"condition", "n"}``) count the candidates excluded for missing a matched
    dimension, which ``candidate_pool`` still includes. They are recorded because
    the selection drew on a smaller pool than the window holds.

    ``neighbourhood_reference`` (optional) records a supplied reference the derived
    dimensions were recomputed against (``{"source", "n_words", "sha256",
    "applied_to"}``, where ``applied_to`` lists the recomputed columns), verbatim.

    ``counterbalance_check`` (optional) is the pipeline's ``counterbalance_check``
    report on the counterbalanced table. Without it, the check is run here on
    ``stimuli``. The record gives the recipe that ran and the lists the table holds,
    or for a table without a ``list`` column the lists counterbalance() would deal. It
    says whether a Latin-square rotation is complete and counts the trials per list
    and condition. It also gives the presentation versions, one per (replicate, list)
    pair, and the rule that assigns each participant one of them.

    ``input_normalisation`` (optional) is ``{"changed", "collapsed"}``: how many text
    keys the run's inputs held that Unicode NFC normalisation rewrote, and how many
    duplicates the rewriting created and removed. Recorded only when given, which the
    pipeline does only when either count is above zero, because a merged spelling
    changes the candidate pool.

    ``artifacts`` names the files written for the design: ``stimuli``,
    ``descriptives``, ``comparisons`` and ``experiments``, the result of
    ``export_experiments``. Each file is recorded with its sha256: the three tables,
    then every file in ``experiments["files"]``, the loop and text tables included,
    or the three experiment paths of a result that has no ``files``.

    ``generation_exclude`` (optional, generated designs) lists the known-form lists
    of ``items.generation.exclude`` as ``{"source", "sha256", "n_forms"}`` records.
    A pseudoword was refused for equalling any form in them, so they decide which
    pseudowords a run produces, as the lexicon does.

    ``exclusions`` (optional, corpus, pool and generated designs) records the design's
    ``exclude_words`` as ``{"path", "sha256", "n_listed", "n_removed"}``. It gives the
    file, or "inline" for a list in the design, its checksum, the number of distinct
    words it lists and the number of candidates it removed. Recorded as
    ``materials_source["exclusions"]``, because the words decide which candidates the
    selection could draw.

    ``zipf_derivation`` (optional) is ``{"n_values", "near_boundary"}`` for a lexicon
    carrying ``freq_permillion``, as ``normalise_lexicon`` writes one: how many positive
    rates it holds, and how many put Zipf = log10(per million) + 3 within 1e-9 of a
    three-place rounding boundary, where the two engines' log10 could round a derived
    value apart. Recorded as ``materials_source["zipf_derivation"]``.

    ``trial_order`` (optional) is the trial-order record of the presented trials, which
    the pipeline passes. Without it, the record is taken from ``stimuli``. It gives
    ``counterbalancing["max_run"]`` (``{"k", "run_on", "longest"}``, the longest run
    realised in each presentation version) when the design limits runs, and
    ``counterbalancing["orders"]`` (``{"m", "rule"}``) when it asks for more than one
    trial order.

    ``design["n_per_condition"]`` is the request the selector applied, the selector's
    default when the design gives none, and ``items["n_per_condition_realised"]``
    counts the items each condition holds in ``stimuli``, per replicate for resampled
    sets, so a selection that fell short of the request under ``shortfall: allow``
    shows it. The suggested analysis keys its by-item slope on the recipe that ran and
    adds a matched design's matched dimensions that vary among its items as covariates.
    """
    source = (design.get("items") or {}).get("source", "corpus")
    is_continuous = _is_continuous(design)
    controlled = _controlled(design, source)
    relational = _relational_record(design, stimuli)
    conditions = list(dict.fromkeys(stimuli["condition"]))
    covariates, left_out = (([], []) if is_continuous
                            else _covariates(design, source, stimuli))
    # The response windows the generated experiments open, and with them where a
    # reaction time starts, which the analysis has to know.
    events = resolve_events(design)
    windows = _response_windows(events, design.get("timing") or {}, _refresh_hz(schema))
    incomplete = (selection_audit or {}).get("incomplete_dropped") or []
    dropped = {d["condition"]: int(d["n"]) for d in incomplete}

    realised = []
    if report is not None:
        # A report the pipeline wrote carries each row's role. One from a direct call
        # of match_report may not, and its roles are then taken from the design.
        role_of = _comparison_roles(design, controlled)
        for _, r in report["comparisons"].iterrows():
            if is_continuous:
                realised.append({
                    "dimension": r["dimension"], "role": r["role"],
                    "pearson_r": _num(r.get("pearson_r")),
                    "predictor_span": _num(r.get("predictor_span")),
                    "sd": _num(r.get("sd")), "sd_ratio": _num(r.get("sd_ratio")),
                    "window_low": _num(r.get("window_low")),
                    "window_high": _num(r.get("window_high")),
                })
            else:
                role = r.get("role")
                if not isinstance(role, str):
                    role = role_of(r["condition"], r["reference"], r["dimension"])
                realised.append({
                    "condition": r["condition"], "reference": r["reference"],
                    "dimension": r["dimension"], "role": role,
                    "cohens_d": _num(r["cohens_d"]),
                    "ci_low": _num(r.get("d_ci_low")), "ci_high": _num(r.get("d_ci_high")),
                    "var_ratio": _num(r.get("var_ratio")),
                    "tost_p": _num(r.get("tost_p")), "equivalent": _bool(r.get("equivalent")),
                    "mean_diff": _num(r.get("mean_diff")),
                })

    if is_continuous:
        # The controls are banded by the same tolerance windows the matcher uses, so
        # the record states them here too; without them the banding is unreproducible.
        selection = {"method": "continuous even-spread (predictor spanned, controls banded)",
                     "predictor": design["continuous"]["predictor"],
                     "controls": list(design["continuous"].get("controls") or []),
                     "tolerance_k": _resolve_tolerance_k(design, schema)}
    elif source in ("corpus", "pool"):
        method = ((design.get("matching") or {}).get("method")
                  or (schema.get("matching") or {}).get("method")
                  or "standardised_euclidean")
        selection = {"method": method, "match_on": controlled}
        if method in ("joint", "optimal"):
            # The pairwise methods rank whole pairs and never consult the tolerance
            # windows; recording tolerance_k here would claim a filter that was not
            # applied. They get the cap they do apply instead, with a per-condition
            # verdict on whether it fired. The candidates excluded for a missing
            # matched value never reach the cap, so the verdict counts the rest.
            selection["candidate_cap"] = {
                "cap": int(PAIRWISE_CAP),
                "applied": {c["condition"]: bool(c["n_candidates"]
                                                 - dropped.get(c["condition"], 0)
                                                 > PAIRWISE_CAP)
                            for c in (candidate_pool or [])
                            if c.get("condition") is not None
                            and c.get("n_candidates") is not None},
            }
        else:
            selection["tolerance_k"] = _resolve_tolerance_k(design, schema)
        # Exact ties decide most picks when the matched dimensions are whole numbers, so
        # the rule that broke them is part of the selection. The salt goes with the
        # hash, which makes every tie key recomputable from the record.
        rule = _resolve_tie_break(design, schema)
        selection["tie_break"] = ({"rule": rule, "salt": TIE_SALT} if rule == "hash"
                                  else {"rule": rule})
    elif source == "generate":
        gen_method = ((design.get("items") or {}).get("generation") or {}).get(
            "method", "letter_substitution")
        selection = {"method": _GENERATION_LABELS.get(
            gen_method, f"{gen_method} (deterministic pseudowords)"),
            "generation_method": gen_method,
            "matched_on": ["length"]}
    else:
        selection = {"method": "item table (user-supplied)"}
    if candidate_pool is not None and source in ("corpus", "pool", "generate"):
        selection["candidate_pool"] = candidate_pool
    # A pair-keyed continuous design does select, over the item table.
    selection["cross_engine"] = _cross_engine(
        selection.get("method"), source, selected=is_continuous and relational is not None)
    # A relaxed window changes what "matched" means for that condition, so the
    # matcher's audit trail belongs in the record, not only in the run narration.
    # Integers only, so both engines serialise the counts identically.
    relaxations = (selection_audit or {}).get("window_relaxations") or []
    if relaxations:
        selection["window_relaxations"] = [
            {"condition": r["condition"],
             "n_within_tolerance": int(r["n_within_tolerance"]),
             "n_needed": int(r["n_needed"])} for r in relaxations]
    # The candidates excluded for missing a matched dimension, which candidate_pool
    # still counts: the selection drew on a smaller pool than the windows hold.
    if incomplete:
        selection["incomplete_dropped"] = [
            {"condition": d["condition"], "n": int(d["n"])} for d in incomplete]
    if neighbourhood_reference is not None:
        selection["neighbourhood_reference"] = neighbourhood_reference
    if generation_exclude and source == "generate":
        selection["generation_exclude"] = [
            {"source": r["source"], "sha256": r["sha256"], "n_forms": int(r["n_forms"])}
            for r in generation_exclude]

    if source in ("corpus", "generate"):
        provenance = _lexicon_provenance(source_path)
    elif source == "pool":
        provenance = "user-supplied word pool, matched by lexsync"
    else:
        provenance = "user-supplied item table"
    materials_source = {
        "type": source, "path": _posix(source_path), "sha256": sha256_file(source_path),
        "provenance": provenance,
    }
    # A supplied pool usually draws its dimensions from a corpus lexicon, and that
    # lexicon is where every matched value came from, so it is named and checksummed
    # here: `path` above records only the word list itself.
    if source == "pool":
        dim_lex = (design.get("items") or {}).get("lexicon") or design.get("lexicon")
        materials_source["dimensions_from"] = (
            dim_lex if dim_lex else "the supplied pool's own columns (no lexicon given)")
        if dim_lex:
            materials_source["dimensions_sha256"] = sha256_file(dim_lex)
    # Added only when present, so a design with no `norms:` block gets no key at all
    # rather than a "norms": null that every datasheet would then carry. Matches the R
    # engine, where assigning NULL to a list element removes it.
    if norms:
        materials_source["norms"] = norms
    # The words the design took out of the candidates, after the norms and before the
    # pool filters. Integers as integers, so both engines write the same JSON.
    if exclusions is not None and source in ("corpus", "pool", "generate"):
        materials_source["exclusions"] = {
            "path": exclusions["path"], "sha256": exclusions["sha256"],
            "n_listed": int(exclusions["n_listed"]),
            "n_removed": int(exclusions["n_removed"])}
    if input_normalisation is not None:
        materials_source["input_normalisation"] = {
            "changed": int(input_normalisation["changed"]),
            "collapsed": int(input_normalisation["collapsed"])}
    if zipf_derivation is not None:
        materials_source["zipf_derivation"] = {
            "n_values": int(zipf_derivation["n_values"]),
            "near_boundary": int(zipf_derivation["near_boundary"])}

    # A corpus design draws on every schema dimension, and so does a pair-keyed
    # design: every lexicon dimension is joined onto each member. A generate or plain
    # table design reports only the ones it controlled, so the record does not claim
    # dimensions that played no part in the selection.
    all_dims = source in ("corpus", "pool") or relational is not None
    # `or {}`, not a default argument, for the reason load_lexicon carries the same
    # guard: an empty `dimensions:` key parses to None here, which is not iterable,
    # while the R twin's schema$dimensions[keep] returns NULL and the record simply
    # carries no dimensions.
    schema_dims = schema.get("dimensions") or {}
    dims = {d: schema_dims[d] for d in schema_dims if all_dims or d in controlled}

    # The recipe comes from the resolver counterbalance() dispatches on, and the lists
    # from the table. The record used to infer the recipe from the item source and to
    # copy counterbalance.lists with a default of 1, though a Latin square without
    # that key deals one list per condition.
    recipe = _recipe(design)
    check = (counterbalance_check if counterbalance_check is not None
             else _counterbalance_check(stimuli, design, recipe))
    versions = _version_count(stimuli, int(check["lists"]))
    counterbalancing = {
        "recipe": recipe,
        "lists": int(check["lists"]),
        "complete": bool(check["complete"]),
        "issues": list(check["issues"]),
        "cells": list(check["cells"]),
        "versions": versions,
        "assignment": (f"version = ((participant - 1) mod {versions}) + 1 unless a version "
                       "is given at launch"),
    }
    # The balance report is added only when the optimiser ran, for the same reason the
    # norms record is: a key that is null on every design that does not use the feature
    # is noise in a research artefact.
    if balance:
        counterbalancing["optimise"] = balance
    # Practice and filler trials change what a participant sees but not what is
    # analysed, so the record has to state both counts: a reader comparing the stimuli
    # file against the experiment would otherwise find them a different length with no
    # explanation.
    if blocks:
        counterbalancing["blocks"] = blocks
    # The run limit and the trial orders, recorded only when the design sets them, so a
    # design without either keeps the record it had.
    if trial_order is None:
        trial_order = _trial_order_record(stimuli, design)
    if trial_order.get("max_run"):
        counterbalancing["max_run"] = trial_order["max_run"]
    if trial_order.get("orders"):
        m = int(trial_order["orders"])
        counterbalancing["orders"] = {
            "m": m, "rule": (f"order = (((participant - 1) div {versions}) mod {m}) + 1; a "
                             "run without a participant number presents order 1")}

    # The equivalence settings the report's TOST verdicts were computed against,
    # recorded so the Methods prose can state the bound it actually ran with.
    equivalence = {"bound_d": (schema.get("equivalence") or {}).get("bound_d", 0.5),
                   "alpha": (schema.get("equivalence") or {}).get("alpha", 0.05)}

    # The key the participants pressed for each condition, recorded only when a response
    # mapping applies: the design's own, or a lexical decision's default.
    presentation = {"response_windows": windows}
    mapping = _response_mapping_record(design)
    if mapping is not None:
        presentation["response_mapping"] = mapping

    reproducibility = {"seed": seed, "versions": _versions(engine)}
    # The design and schema decide everything the seed does not, so their checksums
    # complete the reproducibility record when the pipeline names them.
    if design_path is not None:
        reproducibility["design_sha256"] = sha256_file(design_path)
    if schema_path is not None:
        # The path as well as the checksum: when no schema was named, the run chose
        # one of several files, and the record has to say which.
        reproducibility["schema_path"] = _posix(schema_path)
        reproducibility["schema_sha256"] = sha256_file(schema_path)

    return {
        "lexsync_datasheet_version": DATASHEET_VERSION,
        "design": {
            "name": design["name"], "language": design["language"],
            "paradigm": design.get("paradigm", "factorial"), "source": source,
            "description": design.get("description"),
            "n_per_condition": _requested_n(design, source, is_continuous),
        },
        "materials_source": materials_source,
        "dimensions": dims,
        "selection": selection,
        "relational": relational,
        "analysis": _analysis(design, source, _response_text(windows, events), recipe=recipe,
                              n_conditions=len(conditions), covariates=covariates,
                              left_out=left_out),
        "equivalence": equivalence,
        "realised_control": realised,
        "counterbalancing": counterbalancing,
        "presentation": presentation,
        "resampling": ({"n_sets": (design.get("resample") or {}).get("n_sets"),
                        "disjoint": True} if design.get("resample") else None),
        "items": {
            "n_total": int(len(stimuli)), "n_conditions": len(conditions),
            "conditions": conditions,
            "n_per_condition_realised": _realised_counts(stimuli, conditions),
            "stimuli_file": _posix(artifacts.get("stimuli")),
            "stimuli_sha256": sha256_file(artifacts.get("stimuli")),
        },
        "reproducibility": reproducibility,
        "artifacts": [{"file": _posix(p), "sha256": sha256_file(p)}
                      for p in _artifact_paths(artifacts) if p],
    }

lexsync.write_datasheet(ds, json_path, md_path)

Source code in src/lexsync/datasheet.py

def write_datasheet(ds: dict, json_path: str, md_path: str) -> tuple:
    import os
    os.makedirs(os.path.dirname(json_path) or ".", exist_ok=True)
    with open(json_path, "w", encoding="utf-8", newline="\n") as handle:
        json.dump(_at_15_significant_digits(ds), handle, indent=2, ensure_ascii=False,
                  sort_keys=True)
        handle.write("\n")
    with open(md_path, "w", encoding="utf-8", newline="\n") as handle:
        handle.write(render_datasheet_md(ds) + "\n")
    return json_path, md_path

lexsync.methods_paragraph(ds)

A ready-to-adapt Methods paragraph rendered from a datasheet.

Every number in the paragraph is printed by lexsync's own formatter, rounded half away from zero from the value the datasheet stores, so the R and Python engines write the same text from the same record, a record read back from its JSON included.

It states the items each condition holds, and the request beside them when they fell short of it. The realised control is described before it is tested: the largest difference on a matched dimension in that dimension's unit, with d and its interval, then the equivalence tests, then, largest first, up to five comparisons of an uncontrolled dimension whose |d| exceeds the bound (see _control_note). Every interval is labelled by the schema's alpha. A continuous design gives the predictor's spread and the control windows (see _bands_note). When a response mapping applies, it says which key the participants pressed for which condition (see _mapping_note). Mirrors methods_paragraph in datasheet.R.

Source code in src/lexsync/datasheet.py

def methods_paragraph(ds: dict) -> str:
    """A ready-to-adapt Methods paragraph rendered from a datasheet.

    Every number in the paragraph is printed by lexsync's own formatter, rounded
    half away from zero from the value the datasheet stores, so the R and Python
    engines write the same text from the same record, a record read back from its
    JSON included.

    It states the items each condition holds, and the request beside them when they
    fell short of it. The realised control is described before it is tested: the
    largest difference on a matched dimension in that dimension's unit, with d and its
    interval, then the equivalence tests, then, largest first, up to five comparisons of
    an uncontrolled dimension whose |d| exceeds the bound (see _control_note). Every
    interval is labelled by the schema's alpha. A continuous design gives the predictor's
    spread and the control windows (see _bands_note). When a response mapping applies, it
    says which key the participants pressed for which condition (see _mapping_note).
    Mirrors methods_paragraph in datasheet.R.
    """
    d = ds["design"]
    src = ds["materials_source"]["type"]
    # The first letter raised and the rest as written, as the R twin writes it.
    # str.capitalize() lowered the rest, so 'British English' became 'British english'.
    lang = d["language"][:1].upper() + d["language"][1:]
    sel = ds.get("selection") or {}
    lo, hi = _realised_range(ds)
    n = (_int_text(lo) + " to " + _int_text(hi)) if lo != hi else _int_text(lo)
    one = hi == 1
    were = "was" if one else "were"
    req = _requested_note(ds, lo)
    # What was selected. A pair design's unit is the pair, and its `n_per_condition`
    # counts pairs, so calling them "items" would misreport the size of the materials
    # by a factor of the number of conditions.
    rel = ds.get("relational")
    unit = ("item" if not rel else "-".join(rel["members"]) + " pair") + ("" if one else "s")
    if "predictor" in sel:
        predictor = sel["predictor"]
        rc = ds.get("realised_control") or []
        rs = [abs(r["pearson_r"]) for r in rc
              if r.get("pearson_r") is not None and r["pearson_r"] == r["pearson_r"]]
        # |r| is printed at three places, the precision of the comparisons report it
        # comes from (the record keeps up to four, see _num), so the record holds the
        # digit shown. _fmt_dp prints it, as every number here, because R's sprintf()
        # rounds a tie such as 0.0825 to 0.082 on Windows where this engine gives 0.083.
        corr_str = (" The largest predictor-control correlation was |r| = "
                    + _fmt_dp(max(rs), 3) + "." if rs else "")
        cb = ds["counterbalancing"]
        recipe_label = _recipe_label(cb)
        excluded = _exclusions_note(ds)
        excluded = " " + excluded + "." if excluded else ""
        return (f"{n} {lang} {unit}{req} {were} selected to span {predictor}{_spread_note(rc)}, "
                f"for analysis by regression or a mixed model rather than a between-condition "
                f"contrast (Kuperman, 2015; Liben-Nowell et al., 2019)."
                f"{_bands_note(ds, rc)}{corr_str}{_norms_note(ds)}{excluded} Materials were "
                f"counterbalanced into {cb['lists']} list(s) ({recipe_label}) and generated "
                f"for PsychoPy, OpenSesame and jsPsych.{_versions_note(ds)}"
                f"{_trial_order_note(ds)}{_response_note(ds)}"
                f"{_mapping_note(ds)} The selection is deterministic and reproducible (lexsync "
                f"{ds['reproducibility']['versions']['lexsync']}), and the seed "
                f"({ds['reproducibility']['seed']}) fixes the trial order.")
    items = "item" if one else "items"
    if src == "corpus":
        ctrl = ", ".join(ds["selection"]["match_on"]) or "the control dimensions"
        lead = (f"{n} {items} per condition{req} {were} selected from the {lang} lexicon "
                f"({ds['materials_source']['provenance']}) and matched item by item on "
                f"{ctrl} using lexsync's {ds['selection']['method']} matcher")
    elif src == "pool":
        # A supplied pool is matched exactly as a corpus is; what differs, and what the
        # Methods section has to say, is that the candidate words were chosen by the
        # researcher rather than drawn from the whole lexicon.
        ctrl = ", ".join(ds["selection"]["match_on"]) or "the control dimensions"
        lead = (f"{n} {lang} {items} per condition{req} {were} selected from a supplied "
                f"candidate pool and matched item by item on {ctrl} using lexsync's "
                f"{ds['selection']['method']} matcher, with the matched dimensions taken "
                f"from {ds['materials_source']['dimensions_from']}")
    elif src == "generate":
        lead = (f"{n} real {lang} {'word' if one else 'words'}{req} and {n} length-matched "
                f"{'pseudoword' if one else 'pseudowords'} (generated by "
                f"{ds['selection']['method']}) were assembled for a lexical-decision "
                f"contrast{_known_forms_note(ds)}")
    else:
        # An item table selects nothing, so it has the items it holds and no request. A
        # record written before the realised counts existed divides its rows.
        if not (ds.get("items") or {}).get("n_per_condition_realised"):
            lo = hi = ds["items"]["n_total"] // max(1, ds["items"]["n_conditions"])
        lead = ("%s drawn from an item table for a %s design (%s)"
                % (_count_text(lo, hi, "item") + (" was" if hi == 1 else " were"),
                   d["paradigm"], lang))
    control = _control_note(ds)
    rs = ds.get("resampling")
    resamp = (f". {rs['n_sets']} disjoint matched item sets were drawn, so items can be "
              f"treated as a random factor" if rs else "")
    cb = ds["counterbalancing"]
    recipe_label = _recipe_label(cb)
    tail = (resamp + f". Materials were counterbalanced into {cb['lists']} list(s) "
            f"({recipe_label}) and generated for PsychoPy, OpenSesame and "
            f"jsPsych.{_versions_note(ds)}{_trial_order_note(ds)}{_response_note(ds)}"
            f"{_mapping_note(ds)} The "
            f"selection is deterministic and reproducible (lexsync "
            f"{ds['reproducibility']['versions']['lexsync']}), and the seed "
            f"({ds['reproducibility']['seed']}) fixes the trial order.")
    cp = (ds.get("selection") or {}).get("candidate_pool")
    # The candidate pool counts the candidates excluded for a missing matched value,
    # which were never eligible, so they come off the count of the condition that
    # lost them. Mirrors methods_paragraph in datasheet.R.
    lost = {}
    for d in (ds.get("selection") or {}).get("incomplete_dropped") or []:
        lost[d["condition"]] = lost.get(d["condition"], 0) + int(d["n"])
    pool_note = ""
    if cp:
        sizes = [c["n_candidates"] - lost.get(c.get("condition"), 0) for c in cp
                 if c.get("n_candidates") is not None]
        if sizes:
            pool_note = (f". The smallest condition was selected from {min(sizes)} "
                         "eligible candidates, and the selection was deterministic and "
                         "blind to any outcome measure")
    cap_rec = (ds.get("selection") or {}).get("candidate_cap")
    cap_note = ""
    if cap_rec and any(cap_rec.get("applied", {}).values()):
        cap_note = (f". Each candidate pool exceeding the pairwise cap was reduced to "
                    f"the {int(cap_rec['cap'])} candidates nearest the other "
                    f"condition's centroid before pairing")
    ce_note = ""
    if str((ds.get("selection") or {}).get("cross_engine", "")).startswith("approximate"):
        ce_note = (". This design's matching method uses a covariance inverse or an "
                   "assignment solver, so the R and Python engines select equivalent "
                   "but not byte-identical materials")
    bal = (ds.get("counterbalancing") or {}).get("optimise")
    bal_note = "" if not bal else (
        ". Item sets were assigned to lists so as to equate the lists on %s rather than "
        "by an arbitrary deal, by a deterministic integer search%s (%d swap(s); "
        "imbalance reduced from %d to %d)"
        % (", ".join(bal["dimensions"]), _BALANCE_SCALE_NOTES.get(bal.get("scale"), ""),
           bal["n_swaps"], bal["cost_before"], bal["cost_after"]))
    # Placed before the pool size, which it shrank.
    excluded = _exclusions_note(ds)
    excluded = ". " + excluded if excluded else ""
    return (lead + control + excluded + pool_note + cap_note + ce_note + bal_note + tail
            + _norms_note(ds))

Pipeline and logging

run_pipeline is the orchestrator behind lexsync run, and run_all loops it over a directory of designs. validate_design is the check run_pipeline applies before anything is selected or written, and design_stem gives the stem every written file is named after. The logging functions are reached as lexsync.logging.*, which does not shadow the standard library's logging for absolute imports. All three entry points choose the schema through lexsync.paths.resolve_schema.

lexsync.run_pipeline

The orchestrator. Mirrors R_workflow/R/run_pipeline.R.

For each design it obtains stimuli from the configured item source (a corpus matched on lexical dimensions, generated pseudowords for lexical decision, or an item table for priming and self-paced reading), then counterbalances, writes any match-quality report and the run log, and exports the PsychoPy and OpenSesame scripts from the design's trial-event sequence. run_all loops over every design.

run_all(config_dir='config', schema_path=None, outdir='output', verbose=True)

Run every design_*.yaml in config_dir and return each design's paths.

schema_path left as None is resolved by :func:lexsync.paths.resolve_schema: <config_dir>/schema.yaml, then config/schema.yaml in the working directory, then the copy bundled with the package, with a line on stderr naming the file chosen. A named schema that does not exist is an error, raised before any design runs.

Source code in src/lexsync/run_pipeline.py

def run_all(config_dir="config", schema_path=None, outdir="output", verbose=True) -> dict:
    """Run every ``design_*.yaml`` in ``config_dir`` and return each design's paths.

    ``schema_path`` left as None is resolved by :func:`lexsync.paths.resolve_schema`:
    ``<config_dir>/schema.yaml``, then ``config/schema.yaml`` in the working
    directory, then the copy bundled with the package, with a line on stderr naming
    the file chosen. A named schema that does not exist is an error, raised before
    any design runs.
    """
    if schema_path is not None:
        schema_path, _ = resolve_schema(schema_path)
    designs = sorted(glob.glob(os.path.join(config_dir, "design_*.yaml")) +
                     glob.glob(os.path.join(config_dir, "design_*.yml")))
    if not designs:
        raise FileNotFoundError(f"lexsync: no design_*.yaml files in '{config_dir}'.")
    # Two designs with one stem write the same files, and the second would overwrite
    # the first's without a word, so every stem is checked before any design runs.
    # `en-lexdec` and `en_lexdec` collide this way, as names differing only outside
    # ASCII did before design_stem gave them a checksum.
    stems: dict = {}
    for design in designs:
        stem = design_stem(read_config(design))
        if stem in stems:
            raise ValueError(f"lexsync: '{os.path.basename(stems[stem])}' and "
                             f"'{os.path.basename(design)}' would write the same files "
                             f"('{stem}'); give the designs distinct names.")
        stems[stem] = design
    if schema_path is None:
        # Resolved once, after the designs are found, so a folder without designs is
        # reported on its own and every design reads the same file.
        schema_path, _ = resolve_schema(config_dir=config_dir)
    results = {}
    for design in designs:
        if verbose:
            print(f"\n=== lexsync: design '{os.path.basename(design)}' ===")
        results[os.path.basename(design)] = run_pipeline(design, schema_path, outdir, verbose=verbose)
    if verbose:
        print(f"\n[lexsync] all {len(designs)} designs complete.")
    return results

run_pipeline(design_path, schema_path=None, outdir='output', reference_words=None, verbose=True)

Run the lexsync pipeline for one design.

A design may refer to a derived dimension (n_density, old20, bigram_freq) in match_on, a condition's define_by, pool_filters, a continuous predictor or control, or counterbalance.balance_on. When the lexicon lacks it, the pipeline computes it on the pool against the neighbourhood reference, after the filters the lexicon can answer and before those on the derived dimension.

A design that refers to n_syllables in any of those places is refused when a word that passes the other filters the lexicon can answer has no syllable estimate, which is the case for every word outside the Latin script (see count_syllables).

A corpus, pool or generate design may list words to leave out under exclude_words, or name a file of them with exclude_words: {path: ...}. They are removed from the candidates after any norms: join and before the pool filters, and the datasheet records the list with its checksum. The neighbourhood reference, and the words a generated pseudoword must not equal, keep them.

The run log notes a design in a language other than English that gives no instructions, since lexsync's own are in English, and a design whose response keys lexsync's instructions only name, such as a factorial or priming design. It also notes every practice or filler answer that the response mapping contradicts. A response_mapping that would leave a trial without an answer is refused before anything is written.

Parameters:

Name	Type	Description	Default
design_path	str	Path to a design configuration (YAML).	required
schema_path	str | None	The schema to read. Left as None, it is resolved by :func:lexsync.paths.resolve_schema: config/schema.yaml in the working directory, then schema.yaml beside the design, then the copy bundled with the package, with a line on stderr naming the file chosen. A named schema that does not exist is an error. The path used is recorded in the run log and in the datasheet's reproducibility.schema_path.	None
outdir	str	Output directory, in which the subdirectories stimuli, reports and experiments are created.	'output'
reference_words	list | str | None	Optional list of words to compute the derived dimensions against. By default they are computed against the lexicon, or against its rows marked 1 in a reference_member column. For a corpus or pool design, a supplied list recomputes n_density and old20, and bigram_freq when the lexicon carries it or the design refers to it, overriding the lexicon's columns. The datasheet then records the list's size and checksum and the columns it was applied to. A generate or table design does not use it, and the run log says so. A lone string is one word. A list that is empty or holds a missing value is refused. Its words are normalised to Unicode NFC before use, like every other text key the pipeline reads.	None
verbose	bool	Whether to print progress.	True

Returns:

Type	Description
dict	A mapping of the output paths written.

Source code in src/lexsync/run_pipeline.py

@_restores_verbosity
def run_pipeline(design_path: str, schema_path: str | None = None, outdir: str = "output",
                 reference_words: list | str | None = None, verbose: bool = True) -> dict:
    """Run the lexsync pipeline for one design.

    A design may refer to a derived dimension (``n_density``, ``old20``,
    ``bigram_freq``) in ``match_on``, a condition's ``define_by``, ``pool_filters``,
    a continuous predictor or control, or ``counterbalance.balance_on``. When the
    lexicon lacks it, the pipeline computes it on the pool against the neighbourhood
    reference, after the filters the lexicon can answer and before those on the
    derived dimension.

    A design that refers to ``n_syllables`` in any of those places is refused when a
    word that passes the other filters the lexicon can answer has no syllable
    estimate, which is the case for every word outside the Latin script (see
    ``count_syllables``).

    A corpus, pool or generate design may list words to leave out under
    ``exclude_words``, or name a file of them with ``exclude_words: {path: ...}``.
    They are removed from the candidates after any ``norms:`` join and before the pool
    filters, and the datasheet records the list with its checksum. The neighbourhood
    reference, and the words a generated pseudoword must not equal, keep them.

    The run log notes a design in a language other than English that gives no
    ``instructions``, since lexsync's own are in English, and a design whose response
    keys lexsync's instructions only name, such as a factorial or priming design. It
    also notes every practice or filler answer that the response mapping contradicts. A
    ``response_mapping`` that would leave a trial without an answer is refused before
    anything is written.

    Args:
        design_path: Path to a design configuration (YAML).
        schema_path: The schema to read. Left as None, it is resolved by
            :func:`lexsync.paths.resolve_schema`: ``config/schema.yaml`` in the
            working directory, then ``schema.yaml`` beside the design, then the copy
            bundled with the package, with a line on stderr naming the file chosen.
            A named schema that does not exist is an error. The path used is recorded
            in the run log and in the datasheet's ``reproducibility.schema_path``.
        outdir: Output directory, in which the subdirectories ``stimuli``,
            ``reports`` and ``experiments`` are created.
        reference_words: Optional list of words to compute the derived dimensions
            against. By default they are computed against the lexicon, or against
            its rows marked 1 in a ``reference_member`` column. For a corpus or pool
            design, a supplied list recomputes ``n_density`` and ``old20``, and
            ``bigram_freq`` when the lexicon carries it or the design refers to it,
            overriding the lexicon's columns. The datasheet then records the list's
            size and checksum and the columns it was applied to. A generate or table
            design does not use it, and the run log says so. A lone string is one
            word. A list that is empty or holds a missing value is refused. Its words
            are normalised to Unicode NFC before use, like every other text key the
            pipeline reads.
        verbose: Whether to print progress.

    Returns:
        A mapping of the output paths written.
    """
    runlog.set_verbose(verbose)
    # The design first, so a mistyped design path is reported on its own and not
    # after a line announcing which schema the run would have used.
    design = read_config(design_path)
    schema_path, _ = resolve_schema(schema_path, design_path=design_path)
    schema = read_config(schema_path)
    # Everything a design can be refused for without reading its lexicon is checked
    # here, before the output directory is touched: the keys, the name and language,
    # n_per_condition, the metadata written into the experiments, the events, the
    # condition names and match_on. Unknown keys come back as notes for the run log.
    key_notes = validate_design(design, schema)
    items_cfg = design.get("items") or {}
    source = items_cfg.get("source", "corpus")
    paradigm = design.get("paradigm", "factorial")
    is_continuous = _is_continuous(design)
    if is_continuous and source == "table" and not list(items_cfg.get("members") or []):
        # Without members the table branch loads the rows and never selects, while
        # the log and datasheet would still record continuous mode, which is a
        # provenance lie.
        raise ValueError("lexsync: a 'continuous' block with items.source 'table' "
                         "requires items.members.")
    if source == "table" and not is_continuous and design.get("pool_filters") is not None:
        # Only the continuous pairs selector consumes pool_filters on the table
        # path; a curated item table is the researcher's selection, so a stray
        # filter here is always a mistake rather than a request to drop rows.
        raise ValueError("lexsync: pool_filters have no effect for items.source 'table' "
                         "without a 'continuous' block; remove them or use a continuous "
                         "design.")
    # counterbalance.recipe and counterbalance.incomplete take closed sets of values,
    # so a misspelt one is refused here, before any lexicon is read or file written.
    recipe = _recipe(design)
    incomplete_policy = _incomplete_policy(design, schema)
    # A lone string is one word, as R reads a character scalar, not a list of letters.
    # An empty list would set every word's N to 0 and its OLD20 to NaN, and the
    # selection would go ahead on those values. Mirrors the R twin.
    if isinstance(reference_words, str):
        reference_words = [reference_words]
    if reference_words is not None:
        reference_words = list(reference_words)
        if not reference_words or any(pd.isna(w) for w in reference_words):
            raise ValueError("lexsync: reference_words must hold at least one word and no "
                             "missing value.")
    # The words the design takes out of its candidates, read now, so that a list that
    # cannot be read is refused before any lexicon is loaded or file written.
    exclude = _exclude_words(design)
    excluded = None if exclude is None else _read_exclude_words(exclude)

    log = runlog.new_run_log(design["name"], meta={
        "design": design["name"], "language": design["language"],
        "paradigm": paradigm, "source": source, "seed": schema.get("seed"),
        "mode": "continuous" if is_continuous else "conditions",
        "schema_path": _posix(schema_path),
    })
    for note in key_notes:
        runlog.log_step(log, note)
    # What the experiments' instructions leave unsaid: lexsync's own are in English, and
    # for a task lexsync cannot describe they name the keys without saying what each means.
    for note in _instruction_notes(design):
        runlog.log_step(log, note)
    # Only a corpus or a supplied pool has derived dimensions to compute. Elsewhere the
    # argument changes nothing, so it is said here and kept out of the datasheet.
    override = reference_words is not None and source in ("corpus", "pool")
    if reference_words is not None and not override:
        runlog.log_step(log, "reference_words is not used by items.source '%s' and was "
                             "ignored" % source)

    report = None
    pair_eligible = None
    generation_exclude = None
    exclusions = None
    zipf_derivation = None
    norms: list = []
    selection_audit = None
    # What NFC normalisation did to the inputs this run read (see querying._nfc_count),
    # summed for the run log and the datasheet.
    nfc_total = {"changed": 0, "collapsed": 0}

    def tally(record):
        for k in nfc_total:
            nfc_total[k] += int((record or {}).get(k, 0))

    # Read the matcher's audit attribute straight off the returned frame (pd.concat
    # and rbind drop attrs) and put any excluded candidates and window relaxations on
    # the run log, in the order the matcher met them. The datasheet records both too.
    # Returns the frame unchanged.
    def take_audit(stim):
        nonlocal selection_audit
        selection_audit = stim.attrs.get("audit")
        for dx in (selection_audit or {}).get("incomplete_dropped", []):
            if is_continuous:
                msg = "dropped %d candidate(s) missing the predictor or a control" % dx["n"]
            else:
                msg = ("dropped %d candidate(s) of condition '%s' missing a matched "
                       "dimension" % (dx["n"], dx["condition"]))
            runlog.log_step(log, msg, {"condition": dx["condition"], "n": dx["n"]})
        for rx in (selection_audit or {}).get("window_relaxations", []):
            runlog.log_step(log,
                            "tolerance window relaxed for condition '%s'"
                            " (%d within tolerance, %d needed)"
                            % (rx["condition"], rx["n_within_tolerance"], rx["n_needed"]),
                            {"condition": rx["condition"],
                             "n_within_tolerance": rx["n_within_tolerance"],
                             "n_needed": rx["n_needed"]})
        return stim

    # A lexicon carrying freq_permillion, as normalise_lexicon writes one, gets a record
    # of how many of its rates put Zipf within 1e-9 of a rounding boundary, the one
    # place the two engines' log10 could have derived a different value. Every lexicon
    # lexsync ships lacks the column, so their runs log and record nothing new.
    def note_zipf_derivation(frame):
        record = _zipf_derivation(frame)
        if record is not None:
            runlog.log_step(log, "freq_permillion: %d of %d value(s) put Zipf within 1e-9 of "
                                 "a rounding boundary"
                                 % (record["near_boundary"], record["n_values"]), dict(record))
        return record

    # The design's `norms:` tables are joined onto the lexicon before the pool is
    # built, so a filter, a matched dimension or a continuous predictor may name a
    # semantic dimension lexsync does not compute. The records come back with the
    # lexicon and go into the datasheet: a norm table can carry the manipulated
    # variable itself, so the run is not reproducible from a record that does not name
    # the file and its checksum.
    def join_norms(lex):
        joined = apply_norms(lex, design)
        tally(joined["normalisation"])
        norms.extend(joined["provenance"])
        # A bare `on:` that was read as the join key, which the design should spell
        # `join_on:`.
        for note in joined["notes"]:
            runlog.log_step(log, note)
        for rec in joined["provenance"]:
            runlog.log_step(log, "joined %d norm column(s) from '%s'"
                                 % (len(rec["columns"]), rec["path"]),
                            {"norms": rec["path"], "sha256": rec["sha256"]})
        return joined["lexicon"]

    # A supplied reference list is a set of word keys like any other, so it is put in
    # NFC too, before any derived dimension is computed against it. The datasheet's
    # checksum below stays that of the list as supplied.
    ref_supplied = None
    if reference_words is not None:
        ref_supplied, n_changed, n_collapsed = _nfc_count(list(reference_words))
        tally({"changed": n_changed, "collapsed": n_collapsed})

    ref_words = None
    derived = []
    if source in ("corpus", "generate", "pool"):
        if source == "pool":
            # A supplied word list, given the matcher's dimensions rather than dressed
            # up as a corpus lexicon. `reference` comes back separately because the
            # neighbourhood dimensions are properties of the language, not of the list.
            pool_path = items_cfg["path"]
            lexicon = items_cfg.get("lexicon") or design.get("lexicon")
            runlog.log_step(log, f"loading supplied pool '{pool_path}'")
            lp = load_pool(pool_path, schema, lexicon=lexicon, language=design["language"])
            lex = lp["pool"]
            tally(lex.attrs.get("input_normalisation"))
            ref_words = lp["reference"]
            runlog.log_step(
                log, "supplied pool: %d words%s"
                     % (len(lex), "" if not lexicon
                        else " (dimensions from '%s')" % lexicon),
                {"words": len(lex), "lexicon": lexicon})
            zipf_derivation = note_zipf_derivation(lex)
        else:
            lexicon = items_cfg.get("lexicon") or design.get("lexicon")
            runlog.log_step(log, f"loading lexicon '{lexicon}'")
            lex = load_lexicon(lexicon, schema, language=design["language"])
            tally(lex.attrs.get("input_normalisation"))
            runlog.log_step(log, f"lexicon loaded: {len(lex)} words", {"words": len(lex)})
            zipf_derivation = note_zipf_derivation(lex)
            ref_words = _neighbourhood_reference(lex)
        lex = join_norms(lex)
        # The design's exclude_words leave the candidates here, after the norms join and
        # before the pool filters. `lex` keeps them, because an excluded word is still a
        # word of the language: the neighbourhood reference came from it above, and a
        # generated design's pseudowords are checked against it below.
        candidates = lex
        if excluded is not None:
            candidates = _apply_exclude_words(lex, excluded["words"])
            exclusions = dict(excluded["record"], n_removed=len(lex) - len(candidates))
            runlog.log_step(
                log, "exclude_words: removed %d of the %d listed word(s) from the "
                     "candidates (%s)" % (exclusions["n_removed"], exclusions["n_listed"],
                                          exclusions["path"]),
                {"exclude_words": exclusions["path"], "sha256": exclusions["sha256"],
                 "n_removed": exclusions["n_removed"]})
        filters = dict(design.get("pool_filters") or {})
        derived = _dims_to_derive(design, lex.columns, override)
        reference = ref_supplied if override else ref_words
        n_supplied = len(reference_words) if override else None
        # A filter on a dimension still to be derived waits until it is: the other
        # filters run first and the derivation, quadratic in the reference, then runs
        # on the smaller frame. The values do not depend on which rows carry them,
        # because every word is measured against the whole reference.
        deferred = {k: v for k, v in filters.items() if k in derived}
        # build_pool skips a filter column it does not recognise, so a misspelt key
        # would silently leave the pool unfiltered; the pair path carries the same
        # guard. Checked after the norms join, which legitimately adds filterable
        # columns.
        unknown = [c for c in filters if c not in lex.columns and c not in deferred]
        if unknown:
            raise ValueError("lexsync: pool_filters name column(s) the lexicon does not "
                             "have: %s." % ", ".join(unknown))
        early = {k: v for k, v in filters.items() if k not in deferred}
        # The syllable check sees the words that the other filters on lexicon columns
        # keep (a filter on a dimension still to be derived runs later), and its own
        # filter runs after it: a filter drops the rows missing its column, so a
        # filter on n_syllables over Chinese would otherwise leave an empty pool and
        # an error far from the cause. Filters commute, so the rows are those of one
        # pass.
        pool = build_pool(candidates, {k: v for k, v in early.items() if k != "n_syllables"})
        _require_syllables(pool, design)
        pool = build_pool(pool, {k: v for k, v in early.items() if k == "n_syllables"})
        pending = derived
        if deferred:
            pool = build_pool(_derive_dims(pool, derived, reference, log, n_supplied),
                              deferred)
            pending = []
        runlog.log_step(log, f"pool after filters: {len(pool)} words", {"pool": len(pool)})

    if source in ("corpus", "pool"):
        pool = _derive_dims(pool, pending, reference, log, n_supplied)
        if is_continuous:
            predictor = design["continuous"]["predictor"]
            controls = list(design["continuous"].get("controls") or [])
            stim = take_audit(select_continuous_stimuli(pool, design, schema, verbose=verbose))
            runlog.log_step(log, f"selected {len(stim)} items spanning '{predictor}' "
                                 f"(continuous design)", {"predictor": predictor})
            report = match_report_continuous(stim, predictor, controls, schema)
        else:
            resample = design.get("resample")
            if resample:
                # resample_stimuli records the first replicate's exclusions, which
                # are the whole pool's. A window relaxation inside a replicate
                # reaches only the console, via verbose.
                n_sets = resample.get("n_sets", 2)
                stim = take_audit(resample_stimuli(pool, design, schema, n_sets,
                                                   verbose=verbose))
                runlog.log_step(log, f"resampled {stim['replicate'].nunique()} disjoint matched "
                                     f"sets ({len(stim)} items total)",
                                {"conditions": ", ".join(dict.fromkeys(stim["condition"]))})
            else:
                stim = take_audit(match_stimuli(pool, design, schema, verbose=verbose))
                runlog.log_step(log,
                                f"matched {len(stim)} items across "
                                f"{stim['condition'].nunique()} conditions",
                                {"conditions": ", ".join(dict.fromkeys(stim["condition"]))})
            # Bigram frequency is reported for every design, computed against the
            # run's reference on a copy when the stimuli lack it, so the stimuli file
            # and the selection stay as they were.
            report_stim = (stim if "bigram_freq" in stim.columns
                           else add_bigram_frequency(stim, reference=reference))
            # First-occurrence order, through dict.fromkeys in _report_dims: sorted()
            # would give a collated order that drifts from the R engine. A joined norm
            # the design matches on or defines its conditions by then reaches the
            # descriptives, comparisons and realised-control record, and not only the
            # stimuli file.
            dims = [d for d in _report_dims(design, norms) if d in report_stim.columns]
            report = match_report(report_stim, dims, schema,
                                  roles=_comparison_roles(design, _controlled(design, source)))
    elif source == "generate":
        n = _resolve_n(design, N_DEFAULT_GENERATED)
        gen_cfg = items_cfg.get("generation") or {}
        gen_method = gen_cfg.get("method", "letter_substitution")
        # Further known word forms a pseudoword must not equal. The lexicon stops at
        # its frequency floor, so on its own it lets real low-frequency words through.
        exclude = None
        if gen_cfg.get("exclude") is not None:
            known = _read_known_forms(gen_cfg.get("exclude"))
            exclude = known["forms"]
            generation_exclude = known["records"]
            for rec in generation_exclude:
                runlog.log_step(log, "pseudowords checked against %d known word form(s) "
                                     "from '%s'" % (rec["n_forms"], rec["source"]),
                                {"exclude": rec["source"], "sha256": rec["sha256"]})
        stim = build_lexdec_stimuli(pool, n, reference_words=lex["word"].tolist(),
                                    method=gen_method, exclude=exclude)
        sf = _resolve_policy(design, schema, "shortfall", "error", ("error", "allow"))
        realised = int(stim["set"].nunique())
        if realised < n and sf != "allow":
            # build_lexdec_stimuli keeps only the words written in letters the lexicon
            # attests before selecting, so the eligible pool can be smaller than the
            # request without any pool_filters.
            raise ValueError(f"lexsync: {n} sets per condition were requested but only "
                             f"{realised} could be generated; the eligible pool is "
                             f"smaller than the request, so lower n_per_condition or supply "
                             f"more words, or set matching: shortfall: allow to accept a "
                             f"smaller set.")
        runlog.log_step(log, f"generated {len(stim)} items (words + pseudowords, {gen_method})",
                        {"conditions": ", ".join(dict.fromkeys(stim["condition"]))})
        # Words and pseudowords share their length by construction, so the report also
        # sets them against each other on the dimensions the generator does not hold,
        # computed against the lexicon's reference on a copy of the stimuli.
        report_stim = add_bigram_frequency(add_neighbourhood(stim, reference=ref_words),
                                           reference=ref_words)
        report = match_report(report_stim, list(LEXDEC_REPORT_DIMS), schema,
                              roles=_comparison_roles(design, _controlled(design, source)))
    elif source == "table":
        path = items_cfg["path"]
        runlog.log_step(log, f"loading items '{path}'")
        stim = load_items(path, required_fields(design))
        tally(stim.attrs.get("input_normalisation"))
        runlog.log_step(log, f"loaded {stim['set'].nunique()} items across "
                             f"{stim['condition'].nunique()} conditions",
                        {"conditions": ", ".join(dict.fromkeys(stim["condition"]))})
        members = list(items_cfg.get("members") or [])
        if members:
            # Load the member lexicon here rather than inside join_member_norms, so the
            # design's `norms:` block reaches the members too: a semantic predictor such
            # as `target.concreteness` needs the norm columns present before they are
            # prefixed.
            mem_lexicon = member_lexicon_path(items_cfg, design)
            runlog.log_step(log, f"loading member lexicon '{mem_lexicon}'")
            mem_lex = load_lexicon(mem_lexicon, schema, language=design["language"])
            tally(mem_lex.attrs.get("input_normalisation"))
            zipf_derivation = note_zipf_derivation(mem_lex)
            mem_lex = join_norms(mem_lex)
            stim = join_member_norms(stim, members, items_cfg, design, schema, lex=mem_lex)
            runlog.log_step(log, "joined word-level norms onto %s" % " and ".join(members))
            if "prime" in members and "target" in members:
                # The columns by name, not members[0]/members[1]: a member listed
                # before prime/target would silently redirect the overlap to the
                # wrong pair.
                stim = add_pair_overlap(stim, "prime", "target")
                runlog.log_step(log, "computed relational dimensions (pair.lev, pair.overlap)")
            if is_continuous:
                res = select_continuous_pairs(stim, items_cfg, design, schema, verbose)
                stim, report = take_audit(res["stim"]), res["report"]
                pair_eligible = res["n_eligible"]
                runlog.log_step(
                    log, "selected %d pairs spanning '%s' (%d eligible)"
                         % (stim["set"].nunique(), design["continuous"]["predictor"],
                            res["n_eligible"]),
                    {"sets": int(stim["set"].nunique()), "eligible": res["n_eligible"]})
    else:
        raise ValueError(f"lexsync: unknown item source '{source}'. "
                         "Known sources: corpus, pool, generate, table.")

    if report is not None:
        for msg in balance_check(stim, "condition"):
            runlog.log_step(log, "balance: " + msg)
        if is_continuous:
            for _, cr in report["comparisons"].iterrows():
                if cr["role"] == "control":
                    runlog.log_step(log, f"continuous: '{cr['dimension']}' correlation with "
                                         f"the predictor r = {cr['pearson_r']}")
        else:
            for _, cr in report["comparisons"].iterrows():
                verdict = "equivalent" if cr["equivalent"] else "not shown equivalent"
                lo, hi = cr["d_ci_low"], cr["d_ci_high"]
                # A missing d or p is None or NaN and prints "NA", as R's sprintf()
                # prints it, and a missing interval is left out, as in R. Formatting
                # None raised a TypeError when a condition had a single item.
                ci = f" [{lo:.2f}, {hi:.2f}]" if (_present(lo) and _present(hi)) else ""
                d = f"{cr['cohens_d']:.2f}" if _present(cr["cohens_d"]) else "NA"
                p = cr["tost_p"] if _present(cr["tost_p"]) else "NA"
                runlog.log_step(log, f"equivalence {cr['condition']} vs {cr['reference']} on "
                                     f"'{cr['dimension']}': d = {d}{ci}, "
                                     f"TOST p = {p} ({verdict})")

    # Balance-aware list assignment, when the design asks for it. Off by default,
    # because it changes which items a participant sees. The Latin-square recipe
    # rejects it: there every item is in every list already.
    list_of_set = None
    balance = None
    if (design.get("counterbalance") or {}).get("optimise"):
        bl = balance_lists(stim, design, schema)
        list_of_set = bl["list_of_set"]
        balance = bl["report"]
        runlog.log_step(
            log, "balanced %d item sets across %d lists on %s: cost %d -> %d in %d swap(s)"
                 % (len(list_of_set), (design.get("counterbalance") or {}).get("lists", 1),
                    ", ".join(balance["dimensions"]), balance["cost_before"],
                    balance["cost_after"], balance["n_swaps"]),
            {"cost_before": balance["cost_before"], "cost_after": balance["cost_after"],
             "swaps": balance["n_swaps"]})
        if balance["max_passes_reached"]:
            runlog.log_step(log, "balance: the pass bound was reached, so the search "
                                 "stopped before it ran out of improving swaps")
    # The trigger codes follow the design's conditions, then any other condition in
    # the order the item source lists it (a generated lexical decision's words
    # before its pseudowords, an item table's own order). Taken here, before the
    # shuffle, because the order of first appearance after it depends on the seed.
    condition_order = list(dict.fromkeys(
        (_design_condition_names(design) or []) + [str(c) for c in stim["condition"]]))
    # The conditions the table holds before it is dealt, which a Latin square with
    # fewer item sets than conditions can leave out of every list.
    cb_conditions = list(dict.fromkeys(str(c) for c in stim["condition"]))
    stim = counterbalance(stim, design, schema, list_of_set)
    # An incomplete rotation is a design decision the researcher may accept, so under
    # the default policy it is reported, in the run log here and in the datasheet,
    # and only counterbalance.incomplete: error refuses it.
    cb_check = counterbalance_check(stim, design, recipe, conditions=cb_conditions)
    for issue in cb_check["issues"]:
        runlog.log_step(log, "counterbalancing: " + issue)
    _check_incomplete(cb_check, incomplete_policy)
    # Practice and filler trials are presented but not analysed, so the frame splits
    # here: the experiment is generated from every presented trial, the stimuli file and
    # the reports from the main ones. A design declaring neither block is unaffected,
    # down to not gaining a `block` column.
    blk = add_blocks(stim, design, schema)
    blocks = blk["report"]
    tally(blk["normalisation"])
    # Logged here, once the practice and filler tables have been read too, and only when
    # normalising changed something, so a run over NFC input, every shipped design
    # included, writes the same log and datasheet as before.
    input_normalisation = _nfc_record(nfc_total["changed"], nfc_total["collapsed"])
    if input_normalisation:
        runlog.log_step(log, "normalised %d word form(s) to NFC; %d duplicate(s) removed"
                             % (input_normalisation["changed"],
                                input_normalisation["collapsed"]),
                        dict(input_normalisation))
    # Realise any per-trial duration before the stimuli are written, so a jittered
    # or item-driven interval is recorded as a variable rather than living only
    # inside the generated script.
    # Run on the presented set, so practice and filler trials get their own realised
    # durations too.
    presented = resolve_trial_timing(blk["presented"], design, schema)
    # Fractional milliseconds read from the items are rounded once, and the run log says
    # how many. The record is read straight off the frame and dropped, so no table
    # written later carries it.
    for note in presented.attrs.pop("duration_rounding", []):
        runlog.log_step(log, "rounded %d fractional duration(s) in column '%s' to whole "
                             "milliseconds" % (note["n_rounded"], note["column"]), dict(note))
    # A design whose item codes cannot be kept apart from its other trigger codes is
    # refused here, before any file is written. export_experiments allocates the same
    # codes again and reports on them.
    _allocate_codes(presented, condition_order, resolve_events(design), schema)
    stim = (presented if blocks is None
            else presented[presented["block"] == BLOCK_MAIN].reset_index(drop=True))
    if blocks is not None:
        for b in blocks["blocks"]:
            runlog.log_step(
                log, "block '%s': %d trial(s) per list%s"
                     % (b["block"], b["n_per_list"],
                        "" if not b.get("placement") else ", " + b["placement"]),
                {"block": b["block"], "n_per_list": b["n_per_list"]})
        runlog.log_step(log, "presented %d trial(s); %d analysed"
                             % (len(presented), len(stim)))
    # The trial orders and the run limit, logged only when the design sets them.
    order_record = _trial_order_record(presented, design)
    if order_record.get("orders"):
        runlog.log_step(log, "trial orders: %d per list" % order_record["orders"])
    if order_record.get("max_run"):
        limit = order_record["max_run"]
        runlog.log_step(log, "runs of '%s' limited to %d trial(s); longest realised %d"
                             % (limit["run_on"], limit["k"],
                                max((r["longest_run"] for r in limit["longest"]), default=0)))
    # The answers the experiments will carry, derived as export_experiments derives them:
    # checked here, so a mapping that leaves trials unscored is refused before anything is
    # written, and a practice or filler answer it contradicts is noted.
    for note in _assign_answers(presented, design)[1]:
        runlog.log_step(log, note)

    base = design_stem(design)
    for sub in ("stimuli", "reports", "experiments"):
        os.makedirs(os.path.join(outdir, sub), exist_ok=True)

    stim_path = os.path.join(outdir, "stimuli", f"{base}_stimuli_py.csv")
    write_csv_utf8(stim, stim_path)
    runlog.log_artefact(log, stim_path, len(stim))
    desc_path = comp_path = None
    if report is not None:
        desc_path = os.path.join(outdir, "reports", f"{base}_descriptives_py.csv")
        write_csv_utf8(report["descriptives"], desc_path)
        runlog.log_artefact(log, desc_path, len(report["descriptives"]))
        comp_path = os.path.join(outdir, "reports", f"{base}_comparisons_py.csv")
        write_csv_utf8(report["comparisons"], comp_path)
        runlog.log_artefact(log, comp_path, len(report["comparisons"]))

    # Generated from the PRESENTED set: the experiment runs the practice and filler
    # trials too, even though they are absent from the stimuli file above.
    exps = export_experiments(presented, design, schema,
                              os.path.join(outdir, "experiments"), base,
                              conditions=condition_order)
    # Every file, the loop and text tables included: they hold the trial order and the
    # trigger codes the experiments present, and the scripts alone did not record them.
    for path in exps["files"]:
        runlog.log_artefact(log, path)

    # A materials datasheet (machine + human readable) and a pre-registration
    # template: the shareable provenance record the reproducibility literature asks
    # for (Bochynska et al., 2023; Roettger, 2019).
    source_path = items_cfg.get("path") or items_cfg.get("lexicon") or design.get("lexicon")
    artifacts = {"stimuli": stim_path, "descriptives": desc_path,
                 "comparisons": comp_path, "experiments": exps}
    candidate_pool = None
    if is_continuous and source == "table":
        # A pair design has no word pool. Its candidates are the item sets that
        # passed the filters on every one of their rows, counted at set granularity.
        candidate_pool = [{"condition": "eligible pairs", "n_candidates": pair_eligible}]
    elif is_continuous:
        candidate_pool = [{"condition": "continuous", "n_candidates": int(len(pool))}]
    elif source in ("corpus", "pool"):
        # .get, not ["define_by"]: a condition without define_by counts the whole
        # pool, as the R engine's NULL does through build_pool.
        candidate_pool = [{"condition": c["name"],
                           "n_candidates": int(len(build_pool(pool, c.get("define_by"))))}
                          for c in (design.get("conditions") or [])]
    elif source == "generate":
        # The words the generator can use, which build_lexdec_stimuli draws from: the
        # band after the letter filter, not before it.
        candidate_pool = [{"condition": "words in band",
                           "n_candidates": int(len(_lexdec_eligible(pool,
                                                                    lex["word"].tolist())))}]
    # The neighbourhood reference is provenance: overriding it changes n_density,
    # old20 and bigram_freq without touching any input file the datasheet hashes. It
    # is recorded only when a recomputation ran, with the columns it was applied to.
    nref = None
    if override and derived:
        nref = {"source": "user-supplied", "n_words": len(reference_words),
                "sha256": hashlib.sha256("\n".join(reference_words).encode("utf-8")).hexdigest(),
                "applied_to": list(derived)}
    ds = build_datasheet(design, schema, report, stim, source_path, artifacts,
                         schema.get("seed"), engine="python", candidate_pool=candidate_pool,
                         norms=norms, balance=balance, blocks=blocks,
                         design_path=design_path, schema_path=schema_path,
                         selection_audit=selection_audit,
                         neighbourhood_reference=nref, counterbalance_check=cb_check,
                         input_normalisation=input_normalisation,
                         generation_exclude=generation_exclude, exclusions=exclusions,
                         zipf_derivation=zipf_derivation, trial_order=order_record)
    ds_json = os.path.join(outdir, "reports", f"{base}_datasheet_py.json")
    ds_md = os.path.join(outdir, "reports", f"{base}_datasheet_py.md")
    write_datasheet(ds, ds_json, ds_md)
    runlog.log_artefact(log, ds_json)
    runlog.log_artefact(log, ds_md)

    log_md = os.path.join(outdir, "reports", f"{base}_run_log_py.md")
    runlog.write_run_log(log, log_md, os.path.join(outdir, "reports", f"{base}_run_log_py.jsonl"))

    if verbose:
        print(f"[lexsync] design '{base}' complete.")
    return {"stimuli": stim_path, "descriptives": desc_path, "comparisons": comp_path,
            "experiments": exps, "log": log_md}

lexsync.run_all(config_dir='config', schema_path=None, outdir='output', verbose=True)

Run every design_*.yaml in config_dir and return each design's paths.

schema_path left as None is resolved by :func:lexsync.paths.resolve_schema: <config_dir>/schema.yaml, then config/schema.yaml in the working directory, then the copy bundled with the package, with a line on stderr naming the file chosen. A named schema that does not exist is an error, raised before any design runs.

Source code in src/lexsync/run_pipeline.py

def run_all(config_dir="config", schema_path=None, outdir="output", verbose=True) -> dict:
    """Run every ``design_*.yaml`` in ``config_dir`` and return each design's paths.

    ``schema_path`` left as None is resolved by :func:`lexsync.paths.resolve_schema`:
    ``<config_dir>/schema.yaml``, then ``config/schema.yaml`` in the working
    directory, then the copy bundled with the package, with a line on stderr naming
    the file chosen. A named schema that does not exist is an error, raised before
    any design runs.
    """
    if schema_path is not None:
        schema_path, _ = resolve_schema(schema_path)
    designs = sorted(glob.glob(os.path.join(config_dir, "design_*.yaml")) +
                     glob.glob(os.path.join(config_dir, "design_*.yml")))
    if not designs:
        raise FileNotFoundError(f"lexsync: no design_*.yaml files in '{config_dir}'.")
    # Two designs with one stem write the same files, and the second would overwrite
    # the first's without a word, so every stem is checked before any design runs.
    # `en-lexdec` and `en_lexdec` collide this way, as names differing only outside
    # ASCII did before design_stem gave them a checksum.
    stems: dict = {}
    for design in designs:
        stem = design_stem(read_config(design))
        if stem in stems:
            raise ValueError(f"lexsync: '{os.path.basename(stems[stem])}' and "
                             f"'{os.path.basename(design)}' would write the same files "
                             f"('{stem}'); give the designs distinct names.")
        stems[stem] = design
    if schema_path is None:
        # Resolved once, after the designs are found, so a folder without designs is
        # reported on its own and every design reads the same file.
        schema_path, _ = resolve_schema(config_dir=config_dir)
    results = {}
    for design in designs:
        if verbose:
            print(f"\n=== lexsync: design '{os.path.basename(design)}' ===")
        results[os.path.basename(design)] = run_pipeline(design, schema_path, outdir, verbose=verbose)
    if verbose:
        print(f"\n[lexsync] all {len(designs)} designs complete.")
    return results

lexsync.validate_design(design, schema)

Check a design and its schema before anything is selected or written.

run_pipeline calls this right after reading the two files, so a design that cannot be honoured is refused before any output exists, with the same message in both engines. It first refuses a key within one or two edits of a key lexsync reads at that level, naming the key it was probably meant to be (one edit for a known key of up to five characters, two for a longer one). It then refuses a missing, empty or non-text name or language, an n_per_condition, n_per_cell or resample.n_sets that is not a whole number of at least 1, a counterbalance.optimise that is not true or false, a counterbalance.scale other than mean or mad, a counterbalance.max_run or counterbalance.orders that is not a whole number of at least 1, and a counterbalance.run_on that is not the name of a column. It refuses a name, language or font that cannot be written into the generated experiments, a trigger port that is not an address, and an event list the renderer refuses, such as one with an unknown timing key, a bad response key or a response key that YAML reads as a boolean (an unquoted yes, no, on or off). It refuses a presentation.fullscreen that is not true or false and a presentation.text_height_px that is not a whole number from 8 to 400. It refuses a triggers.mode other than auto, hardware or mock, a triggers.backend other than parallel or serial, and a triggers.baudrate that is not a positive whole number. A serial backend needs a triggers.serial_port, with a name the experiments can carry. Last, it refuses a condition without a name, with a name that is not text or with a repeated name, a corpus or pool design with conditions but no match_on, and a name and language that leave no file stem. It also refuses an exclude_words that is neither a list of words nor a mapping whose path names a file, and one that a design with an item table names.

Any other key lexsync does not read is not refused yet. It is returned as a note, which run_pipeline writes into the run log, and a later release will refuse it. description and keys beginning x_ are annotations and are never noted. A schema that still sets triggers.inter_trigger_ms, which no longer does anything, gets a note too.

The experiments' instructions and response mapping are checked here too. It refuses instructions that are not text, are empty or hold a control character other than a line break. It refuses a response_mapping that is not a mapping of conditions to single keys, names a key the response event does not allow, or comes with no response event. And it refuses a rotate_mapping_by_list that is not true or false, or that has no mapping to rotate.

Parameters

design : dict A parsed design configuration. schema : dict The parsed global schema.

Returns

list of str The notes on unknown keys, in the order the keys were written, then the note on triggers.inter_trigger_ms; empty when there are none.

Source code in src/lexsync/design.py

def validate_design(design: dict, schema: dict) -> list:
    """Check a design and its schema before anything is selected or written.

    ``run_pipeline`` calls this right after reading the two files, so a design that
    cannot be honoured is refused before any output exists, with the same message in
    both engines. It first refuses a key within one or two edits of a key lexsync
    reads at that level, naming the key it was probably meant to be (one edit for a
    known key of up to five characters, two for a longer one). It then refuses a
    missing, empty or non-text ``name`` or ``language``, an ``n_per_condition``,
    ``n_per_cell`` or ``resample.n_sets`` that is not a whole number of at least 1, a
    ``counterbalance.optimise`` that is not true or false, a ``counterbalance.scale``
    other than ``mean`` or ``mad``, a ``counterbalance.max_run`` or
    ``counterbalance.orders`` that is not a whole number of at least 1, and a
    ``counterbalance.run_on`` that is not the name of a column. It refuses a name,
    language or font that cannot be written into the generated experiments, a
    trigger port that is not an address, and an event list the renderer refuses, such
    as one with an unknown timing key, a bad response key or a response key that
    YAML reads as a boolean (an unquoted ``yes``, ``no``, ``on`` or ``off``). It
    refuses a ``presentation.fullscreen`` that is not true or false and a
    ``presentation.text_height_px`` that is not a whole number from 8 to 400. It
    refuses a ``triggers.mode`` other than ``auto``, ``hardware`` or ``mock``, a
    ``triggers.backend`` other than ``parallel`` or ``serial``, and a
    ``triggers.baudrate`` that is not a positive whole number. A serial backend needs a
    ``triggers.serial_port``, with a name the experiments can carry. Last, it
    refuses a condition without a name, with a name that is not text or with a
    repeated name, a corpus or pool design with conditions but no ``match_on``, and a
    name and language that leave no file stem. It also refuses an ``exclude_words``
    that is neither a list of words nor a mapping whose ``path`` names a file, and one
    that a design with an item table names.

    Any other key lexsync does not read is not refused yet. It is returned as a note,
    which ``run_pipeline`` writes into the run log, and a later release will refuse
    it. ``description`` and keys beginning ``x_`` are annotations and are never
    noted. A schema that still sets ``triggers.inter_trigger_ms``, which no longer
    does anything, gets a note too.

    The experiments' instructions and response mapping are checked here too. It
    refuses ``instructions`` that are not text, are empty or hold a control character
    other than a line break. It refuses a ``response_mapping`` that is not a mapping of
    conditions to single keys, names a key the response event does not allow, or comes
    with no response event. And it refuses a ``rotate_mapping_by_list`` that is not
    true or false, or that has no mapping to rotate.

    Parameters
    ----------
    design : dict
        A parsed design configuration.
    schema : dict
        The parsed global schema.

    Returns
    -------
    list of str
        The notes on unknown keys, in the order the keys were written, then the note
        on ``triggers.inter_trigger_ms``; empty when there are none.
    """
    # Imported here: scripting names its files through design_stem, and counterbalancing
    # quotes values through _shown, so a module-level import would be circular.
    from .counterbalancing import _balance_scale, _trial_order_settings
    from .scripting import (
        _check_instructions_and_mapping,
        _fullscreen,
        _refresh_hz,
        _text_height_px,
        _trigger_settings,
        render_events,
    )

    schema = schema or {}
    if not isinstance(design, dict):
        raise ValueError("lexsync: a design must be a mapping of keys to values.")
    notes = (_check_keys(design, DESIGN_KEYS, "design")
             + _check_keys(schema, SCHEMA_KEYS, "schema"))
    # The key held the self-paced reading critical-region code, which is now held for
    # trigger_hold_ms like every other code. A schema written for an earlier release
    # still sets it, so it is accepted and noted.
    triggers = schema.get("triggers")
    if isinstance(triggers, dict) and triggers.get("inter_trigger_ms") is not None:
        notes.append("lexsync: triggers.inter_trigger_ms is no longer used; the "
                     "self-paced reading critical-region code is held for "
                     "triggers.trigger_hold_ms, as every other code is.")
    _name_and_language(design)
    _resolve_n(design, None)
    resample = design.get("resample")
    if isinstance(resample, dict) and resample.get("n_sets") is not None:
        _check_n_sets(resample["n_sets"])
    counterbalance = design.get("counterbalance")
    if isinstance(counterbalance, dict):
        _require_bool(counterbalance.get("optimise"), "counterbalance.optimise")
        _balance_scale(design)
    # counterbalance.max_run, run_on and orders, refused here before anything is
    # selected, by the rule counterbalance() applies them with.
    _trial_order_settings(design)

    clean_meta(design["name"], "the design's `name`")
    clean_meta(design["language"], "the design's `language`")
    font = design.get("font")
    if font is not None and not isinstance(font, str):
        raise ValueError("lexsync: the design's `font` must be text; quote it in the "
                         "design file.")
    presentation = schema.get("presentation") or {}
    # The PsychoPy and jsPsych targets fall back to presentation.font, OpenSesame to
    # its own font token, so both effective fonts are checked.
    clean_meta(font or presentation.get("font") or "Courier New", "the font")
    clean_meta(font or presentation.get("opensesame_font") or "mono", "the font")
    clean_port((schema.get("triggers") or {}).get("parallel_address", "0x0378"))
    _fullscreen(schema)
    _text_height_px(schema)
    _trigger_settings(schema)
    render_events(resolve_events(design), design.get("timing") or {}, _refresh_hz(schema))
    _check_instructions_and_mapping(design)

    _check_conditions(design)
    _exclude_words(design)
    source = (design.get("items") or {}).get("source", "corpus")
    if (source in ("corpus", "pool") and not design.get("continuous")
            and design.get("conditions")):
        _require_match_on(design)
    design_stem(design)
    return notes

lexsync.design_stem(design)

The stem every file a design writes is named after.

The slug of the design's name and language, which keeps letters and digits only. Two designs whose names differ only outside ASCII (词频 and 笔画, both chinese) would share a slug and overwrite each other's files, and an all-Chinese name and language would give no slug at all. Whenever the name or the language holds a non-ASCII character, the stem therefore gains _ and the first eight hexadecimal digits of the SHA-256 of the NFC-normalised UTF-8 text name|language. An empty slug is replaced by those eight digits. NFC keeps a precomposed and a decomposed é on one stem. A design whose name and language hold no letter or digit at all is refused. Mirrors design_stem() in R_workflow/R/design.R.

Parameters

design : dict A parsed design configuration.

Returns

str The file stem, such as en_lexdec_english or chinese_14aadfa7.

Source code in src/lexsync/design.py

def design_stem(design: dict) -> str:
    """The stem every file a design writes is named after.

    The slug of the design's ``name`` and ``language``, which keeps letters and
    digits only. Two designs whose names differ only outside ASCII (``词频`` and
    ``笔画``, both ``chinese``) would share a slug and overwrite each other's files,
    and an all-Chinese name and language would give no slug at all. Whenever the
    name or the language holds a non-ASCII character, the stem therefore gains ``_``
    and the first eight hexadecimal digits of the SHA-256 of the NFC-normalised UTF-8
    text ``name|language``. An empty slug is replaced by those eight digits. NFC keeps a
    precomposed and a decomposed ``é`` on one stem. A design whose name and language
    hold no letter or digit at all is refused. Mirrors design_stem() in
    R_workflow/R/design.R.

    Parameters
    ----------
    design : dict
        A parsed design configuration.

    Returns
    -------
    str
        The file stem, such as ``en_lexdec_english`` or ``chinese_14aadfa7``.
    """
    name, language = _name_and_language(design)
    name = unicodedata.normalize("NFC", name)
    language = unicodedata.normalize("NFC", language)
    stem = slugify(name, language)
    joined = unicodedata.normalize("NFC", name + "|" + language)
    if not joined.isascii():
        digest = hashlib.sha256(joined.encode("utf-8")).hexdigest()[:8]
        stem = f"{stem}_{digest}" if stem else digest
    if not stem:
        raise ValueError("lexsync: the design's `name` and `language` hold no letter or "
                         "digit to name its files after; use letters or digits in the "
                         "name.")
    return stem

lexsync.paths.resolve_schema(schema_path=None, design_path=None, config_dir=None)

Resolve the schema for a run, returning (path, origin).

A named schema_path is used as given and must exist; it is never replaced by another file, since that would run a different schema from the one the caller chose. With no schema named, the candidates are tried in order:

• for a folder of designs (config_dir given): <config_dir>/schema.yaml, then config/schema.yaml in the working directory;
• for a single design: config/schema.yaml in the working directory, then schema.yaml in the design's own directory;
• in both cases, last, the copy bundled with the package.

The first candidate in each list is the default these entry points had before, so a call that worked then reads the same file now. Whenever no schema was named, the chosen path is printed to stderr as "lexsync: using the schema at ''".

origin is one of "named", "config_dir", "working_directory", "design_directory" or "bundled".

Source code in src/lexsync/paths.py

def resolve_schema(schema_path=None, design_path=None, config_dir=None) -> tuple:
    """Resolve the schema for a run, returning ``(path, origin)``.

    A named ``schema_path`` is used as given and must exist; it is never replaced by
    another file, since that would run a different schema from the one the caller
    chose. With no schema named, the candidates are tried in order:

    - for a folder of designs (``config_dir`` given): ``<config_dir>/schema.yaml``,
      then ``config/schema.yaml`` in the working directory;
    - for a single design: ``config/schema.yaml`` in the working directory, then
      ``schema.yaml`` in the design's own directory;
    - in both cases, last, the copy bundled with the package.

    The first candidate in each list is the default these entry points had before,
    so a call that worked then reads the same file now. Whenever no schema was named,
    the chosen path is printed to stderr as "lexsync: using the schema at '<path>'".

    ``origin`` is one of "named", "config_dir", "working_directory",
    "design_directory" or "bundled".
    """
    if schema_path is not None:
        if not os.path.isfile(schema_path):
            raise FileNotFoundError(
                f"lexsync: schema not found: '{schema_path}'. The schema bundled with "
                f"lexsync is {_BUNDLED_CALL}, at '{bundled_schema()}'.")
        return schema_path, "named"
    cwd_config = (os.path.join("config", "schema.yaml"), "working_directory")
    if config_dir is not None:
        candidates = [(os.path.join(config_dir, "schema.yaml"), "config_dir"), cwd_config]
    else:
        candidates = [cwd_config]
        if design_path is not None:
            # `or "."`, as R's dirname() gives for a bare file name, so both engines
            # record the same path.
            design_dir = os.path.dirname(design_path) or "."
            candidates.append((os.path.join(design_dir, "schema.yaml"), "design_directory"))
    candidates.append((bundled_schema(), "bundled"))
    for path, origin in candidates:
        if os.path.isfile(path):
            # stderr, so a run's printed output and any redirected stdout stay clean.
            # The run log and the datasheet record the path as well, for a caller who
            # never sees this line. In POSIX form, as they record it and as the R
            # engine prints it.
            print(f"lexsync: using the schema at '{_posix(path)}'", file=sys.stderr)
            return path, origin
    raise FileNotFoundError("lexsync: no schema found, and the copy bundled with "
                            "lexsync is missing; reinstall the package.")

lexsync.paths.bundled_schema()

The schema shipped inside the package.

It is byte-identical to the repository's config/schema.yaml (test_config.py enforces the identity), so an installed package runs anywhere.

Source code in src/lexsync/paths.py

def bundled_schema() -> str:
    """The schema shipped inside the package.

    It is byte-identical to the repository's config/schema.yaml (test_config.py
    enforces the identity), so an installed package runs anywhere.
    """
    from importlib.resources import files
    return str(files("lexsync") / "data" / "schema.yaml")

lexsync.logging.new_run_log(name, meta=None)

Source code in src/lexsync/logging.py

def new_run_log(name: str, meta: dict | None = None) -> dict:
    return {
        "name": name,
        "started": _now(),
        "engine": f"Python {platform.python_version()}",
        "meta": meta or {},
        "steps": [],
    }

lexsync.logging.log_step(log, message, data=None)

Source code in src/lexsync/logging.py

def log_step(log: dict, message: str, data: dict | None = None) -> dict:
    log["steps"].append({"time": _now(), "message": message, "data": data})
    if _VERBOSE:
        print(f"[lexsync] {message}")
    return log

lexsync.logging.log_artefact(log, path, rows=None)

Source code in src/lexsync/logging.py

def log_artefact(log: dict, path: str, rows=None) -> dict:
    return log_step(log, f"wrote '{os.path.basename(path)}'",
                    {"path": path, "rows": rows, "md5": hash_file(path)})

lexsync.logging.write_run_log(log, md_path, jsonl_path=None)

Source code in src/lexsync/logging.py

def write_run_log(log: dict, md_path: str, jsonl_path: str | None = None) -> str:
    os.makedirs(os.path.dirname(md_path) or ".", exist_ok=True)
    lines = [
        f"# lexsync run log: {log['name']}", "",
        f"- Engine: {log['engine']}",
        f"- Started: {log['started']}",
        f"- Finished: {_now()}",
    ]
    if log["meta"]:
        lines += ["", "## Run metadata", ""]
        for key, value in log["meta"].items():
            lines.append(f"- {key}: {value}")
    lines += ["", "## Steps", ""]
    for step in log["steps"]:
        lines.append(f"- **{step['time']}**: {step['message']}")
        if step["data"]:
            for key, value in step["data"].items():
                lines.append(f"    - {key}: {value}")
    with open(md_path, "w", encoding="utf-8", newline="\n") as handle:
        handle.write("\n".join(lines) + "\n")
    if jsonl_path:
        with open(jsonl_path, "w", encoding="utf-8", newline="\n") as handle:
            for step in log["steps"]:
                handle.write(json.dumps(step, ensure_ascii=False) + "\n")
    return md_path