API reference

Every public name in lexsync is documented here, grouped along the path a study takes: reach a corpus, derive the dimensions, build a pool and match it, describe a trial, counterbalance it, report what the matching achieved, generate the experiment, and write down the provenance. The groups are the same ones the R package's reference index uses, so a name can be found in the same place on either site.

Everything listed under a group heading is importable straight from lexsync, with a handful of exceptions that are noted where they appear and are reached through their own module. Guides with worked examples are linked from the home page, and the published work that these entries cite is listed in full on the references page.

Corpora and lexica

Languages are supplied through a corpus registry, so reaching a new one takes a registry entry and no code. These functions find a corpus, fetch it if it is not already local, and read a derived lexicon or a prepared item table into the frame everything else expects.

lexsync.list_corpora(registry_path=None)

Source code in src/lexsync/corpora.py

def list_corpora(registry_path: str | None = None) -> pd.DataFrame:
    with open(_registry_path(registry_path), encoding="utf-8") as handle:
        reg = yaml.safe_load(handle)
    rows = []
    for name, entry in (reg.get("corpora") or {}).items():
        lang = entry.get("language") or {}
        rows.append(dict(
            name=name, language=lang.get("name"), iso=lang.get("iso"),
            status=entry.get("status"), connector=entry.get("connector", "openlexicon"),
            citation=entry.get("citation"),
        ))
    return pd.DataFrame(rows)

lexsync.fetch_corpus(name, registry_path=None, n_words=10000)

Fetch a registered corpus into the cache.

If name is a language code supported by the wordfreq connector, a lexicon is built with wordfreq. Otherwise the corpus's registered URL is downloaded, once its scheme has been checked; the transfer lands in a sidecar file that is renamed into the cache only after the size cap, the markup sniff and any registered sha256 have all passed.

The file lands in :func:cache_dir. That cache persists between sessions and the package never prunes it; one corpus may reach the 200 MB download cap, so several of them add up. Nothing kept there is irreplaceable, so the directory may be deleted at any time and the next call downloads the corpus again.

Source code in src/lexsync/corpora.py

def fetch_corpus(name: str, registry_path: str | None = None, n_words: int = 10000) -> str:
    """Fetch a registered corpus into the cache.

    If `name` is a language code supported by the wordfreq connector, a lexicon
    is built with wordfreq. Otherwise the corpus's registered URL is downloaded,
    once its scheme has been checked; the transfer lands in a sidecar file that
    is renamed into the cache only after the size cap, the markup sniff and any
    registered sha256 have all passed.

    The file lands in :func:`cache_dir`. That cache persists between sessions and
    the package never prunes it; one corpus may reach the 200 MB download cap, so
    several of them add up. Nothing kept there is irreplaceable, so the directory
    may be deleted at any time and the next call downloads the corpus again.
    """
    with open(_registry_path(registry_path), encoding="utf-8") as handle:
        reg = yaml.safe_load(handle)
    wf = reg.get("wordfreq_connector") or {}
    if name in (wf.get("languages") or []):
        df = build_wordfreq_lexicon(name, n_words)
        dest = os.path.join(cache_dir(), f"{name}_wordfreq.csv")
        # LF explicitly: pandas otherwise follows os.linesep, and the cached
        # lexicon's bytes (and any checksum of them) must not depend on the OS.
        df.to_csv(dest, index=False, encoding="utf-8", lineterminator="\n")
        print(f"lexsync: built '{name}' via wordfreq. Please cite: {wf.get('citation', '')}")
        return dest
    entry = (reg.get("corpora") or {}).get(name)
    if not entry:
        raise ValueError(f"lexsync: corpus '{name}' is not in the registry.")
    # Only 'openlexicon' names a delimited file; 'url' is the landing page, and
    # downloading that would silently cache an HTML document as <name>.csv.
    url = entry.get("openlexicon")
    if not url:
        raise ValueError(
            f"lexsync: corpus '{name}' registers only a landing page "
            f"({entry.get('url', 'see registry.yaml')}); lexsync cannot download it "
            f"automatically. Retrieve the delimited file manually and pass it to "
            f"load_lexicon()."
        )
    # A registry is editable, and fetch_corpus() writes wherever it points, so a
    # 'file://' or 'ftp://' entry would read a local path under the guise of a
    # download. Only the two schemes a corpus is published over are honoured.
    if not re.match(r"https?://", url, re.IGNORECASE):
        raise ValueError(
            f"lexsync: corpus '{name}' registers a non-http(s) URL ({url}); "
            f"refusing to download."
        )
    import urllib.error
    import urllib.request
    dest = os.path.join(cache_dir(), f"{name}.csv")
    # The transfer lands in a sidecar and is renamed over `dest` only after every
    # check below has passed, so a truncated or unverified body can never sit at
    # the cache path, where a later run would trust it.
    part = dest + ".part"
    try:
        # 60 s guards against a stalled server, which urlretrieve() would wait on
        # forever; the R engine's download.file() honours options(timeout).
        with urllib.request.urlopen(url, timeout=60) as response, \
                open(part, "wb") as handle:
            received = 0
            for chunk in iter(lambda: response.read(65536), b""):
                received += len(chunk)
                if received > _max_download_bytes():
                    raise ValueError(
                        "lexsync: corpus download exceeded the 200 MB size limit. "
                        "Retrieve the delimited file manually and pass it to "
                        "load_lexicon()."
                    )
                handle.write(chunk)
    except (urllib.error.URLError, OSError) as exc:
        if os.path.exists(part):
            os.remove(part)
        raise RuntimeError(
            f"lexsync: could not download corpus '{name}' from {url} ({exc}). "
            f"Check the URL in registry.yaml, or download the file manually and "
            f"pass it to load_lexicon()."
        ) from exc
    except ValueError:
        if os.path.exists(part):
            os.remove(part)
        raise
    if _starts_with_markup(part):
        os.remove(part)
        raise ValueError(
            f"lexsync: corpus '{name}' returned an HTML page, not a delimited file "
            f"({url}); the registry URL may have rotted. Retrieve the delimited file "
            f"manually and pass it to load_lexicon()."
        )
    # 'sha256' is optional per registry entry; when present the download must
    # match it before it may enter the cache.
    expected = entry.get("sha256")
    if expected and sha256_file(part) != expected:
        os.remove(part)
        raise ValueError(
            f"lexsync: checksum mismatch for corpus '{name}'; the download does "
            f"not match the registry's sha256. Retry the download, or verify the "
            f"sha256 recorded in registry.yaml."
        )
    os.replace(part, dest)
    print(f"lexsync: downloaded '{name}'. Please cite: {entry.get('citation', '(see registry)')}")
    return dest

lexsync.corpora.cache_dir()

Per-user cache directory for fetched corpora, created on first use.

Where :func:fetch_corpus puts a download unless told otherwise, and the only place the package writes to without being handed a path.

The cache persists between sessions and lexsync never prunes it. A registered corpus is a delimited word list, and a download is refused above 200 MB, so a cache holding several large corpora can reach a few hundred megabytes. It holds nothing that cannot be fetched again, so it may be deleted at any time, whole or file by file, and the next call downloads afresh. The R twin documents the same contract in lexsync_cache_dir.Rd; only the location differs, since R uses tools::R_user_dir.

Source code in src/lexsync/corpora.py

def cache_dir() -> str:
    """Per-user cache directory for fetched corpora, created on first use.

    Where :func:`fetch_corpus` puts a download unless told otherwise, and the only
    place the package writes to without being handed a path.

    The cache persists between sessions and lexsync never prunes it. A registered
    corpus is a delimited word list, and a download is refused above 200 MB, so a
    cache holding several large corpora can reach a few hundred megabytes. It holds
    nothing that cannot be fetched again, so it may be deleted at any time, whole or
    file by file, and the next call downloads afresh. The R twin documents the same
    contract in lexsync_cache_dir.Rd; only the location differs, since R uses
    tools::R_user_dir.
    """
    path = os.path.join(os.path.expanduser("~"), ".lexsync", "cache")
    os.makedirs(path, exist_ok=True)
    return path

lexsync.load_lexicon(path, schema, language=None)

Source code in src/lexsync/querying.py

def load_lexicon(path: str, schema: dict, language: str | None = None) -> pd.DataFrame:
    df = read_csv_utf8(path)
    validate_lexicon(df, schema)
    # `or {}` at every level: an empty `dimensions:` or `frequency:` key parses to
    # None here and to NULL in R, where `$` chains through it to the default. A
    # default argument only covers an ABSENT key, so the two engines answered the
    # same schema with a column name and an AttributeError.
    freq_col = ((schema.get("dimensions") or {}).get("frequency") or {}).get("column") or "freq_zipf"
    # Filter before coercing: astype(str) renders a missing word as the literal
    # string "nan", which survives both guards below, whereas the R engine (where
    # as.character(NA) stays NA) drops the row. Coercing first would therefore
    # desynchronise the two engines' row counts and shift every subsequent id.
    df = df[df["word"].notna() & df[freq_col].notna()].copy()
    df["word"] = df["word"].astype(str).str.strip().str.lower()
    df = df[df["word"] != ""]
    df = df.drop_duplicates(subset="word")
    # An empty selection is never what the caller meant, and it fails far from
    # here: pandas would hand back a well-formed frame of nothing, while the R
    # engine dies on base R's "replacement has 1 row, data has 0". Both engines
    # raise this message instead, while the path is still in scope.
    if df.empty:
        raise ValueError(
            f"lexsync: lexicon '{path}' has no usable rows: it is empty, "
            f"or every row is missing 'word' or '{freq_col}'."
        )
    # Sort by UTF-8 byte order so the lexicon order is locale-independent and
    # identical to the R engine (which uses a 'radix' byte-order sort).
    df = (df.assign(_k=df["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))
    df["id"] = np.arange(1, len(df) + 1)
    df["length"] = df["word"].str.len()
    df["n_syllables"] = df["word"].map(count_syllables)
    df["frequency"] = df[freq_col].astype(float)
    if language is not None:
        df["language"] = language
    return df.reset_index(drop=True)

lexsync.load_items(path, required_fields)

Load a paradigm item table (prime-target pairs, sentences, …).

The table must carry an item identifier, a condition label and the paradigm's presented fields. Field values are validated (no control characters; bounded length) so a crafted item cannot corrupt the generated loop table or scripts. Items are mapped to a deterministic integer set id (byte order) so counterbalancing matches the corpus path and the two engines.

Source code in src/lexsync/querying.py

def load_items(path: str, required_fields) -> pd.DataFrame:
    """Load a paradigm item table (prime-target pairs, sentences, …).

    The table must carry an ``item`` identifier, a ``condition`` label and the
    paradigm's presented fields. Field values are validated (no control
    characters; bounded length) so a crafted item cannot corrupt the generated
    loop table or scripts. Items are mapped to a deterministic integer ``set`` id
    (byte order) so counterbalancing matches the corpus path and the two engines.
    """
    parts = str(path).replace("\\", "/").split("/")
    if ".." in parts:
        raise ValueError("lexsync: items path must not contain '..'.")
    # The item id, the condition label and the paradigm's presented fields are read as
    # text, never type-guessed. This engine happens to keep "f" a string anyway, but the
    # R engine's reader turns a column of `f` or `t` into a LOGICAL, so a design coding
    # its two response keys as f and j had `answer` become FALSE there and "f" here; and
    # `item` left to inference float-promoted a numeric id column, so '01' became '1'
    # (or, with a missing cell, '1.0'). Forcing the type on both sides makes the
    # agreement structural, where it would otherwise be coincidental.
    df = read_csv_utf8(path, as_character=["item", "condition"] + list(required_fields))
    needed = ["item", "condition"] + list(required_fields)
    missing = [c for c in needed if c not in df.columns]
    if missing:
        raise ValueError(f"lexsync: items table '{path}' is missing column(s): {', '.join(missing)}.")
    # Missingness is tested before any string coercion: a missing cell in a dtype=str
    # column still arrives as NaN, and str() would render it as the literal 'nan',
    # which a blank condition then carries past the hash-key guard downstream.
    for col in needed:
        if df[col].isna().any():
            raise ValueError(
                f"lexsync: the items table has missing value(s) in column '{col}'; "
                "every item, condition and presented field must be filled.")
    df = df.copy()
    for f in [c for c in required_fields if c in df.columns]:
        df[f] = [str(v).strip(_ASCII_WS) for v in df[f]]
    item_key = df["item"].astype(str).str.strip(_ASCII_WS)
    df["condition"] = df["condition"].astype(str).str.strip(_ASCII_WS)
    # An all-whitespace cell is already missing in the R engine's reader (readr trims
    # before matching its na strings), so refusing the trimmed-empty value here keeps
    # the two engines refusing the same tables, with the same message.
    for col in needed:
        vals = item_key if col == "item" else df[col]
        if (vals == "").any():
            raise ValueError(
                f"lexsync: the items table has missing value(s) in column '{col}'; "
                "every item, condition and presented field must be filled.")
    # A repeated item-condition pair is a slip that would silently duplicate a trial
    # in every generated list; the first repeat in file order is reported.
    seen = set()
    for it, cond in zip(item_key, df["condition"], strict=True):
        if (it, cond) in seen:
            raise ValueError(
                f"lexsync: the items table repeats item '{it}' for condition '{cond}'; "
                "each item and condition pair may appear once.")
        seen.add((it, cond))
    for f in [c for c in required_fields if c in df.columns]:
        df[f] = [clean_field(v, f) for v in df[f]]
    items = sorted(item_key.unique(), key=lambda s: s.encode("utf-8"))
    set_map = {it: i + 1 for i, it in enumerate(items)}
    df["set"] = item_key.map(set_map)
    return df.reset_index(drop=True)

lexsync.load_pool(path, schema, lexicon=None, language=None)

Load a supplied candidate pool of words and give it the matcher's dimensions.

A researcher who already has a curated word list (from a previous study, a norming session, a colleague) should not have to dress it up as a corpus lexicon to get lexsync's matching, validation and datasheet. This reads such a list and returns something the matcher accepts.

The list needs only a word column. Length and the syllable estimate are derived from the form. Everything else is either supplied on the list itself or looked up: with lexicon given, the corpus dimensions (frequency above all) are joined for those words, and a word the lexicon does not have is a hard error rather than a NaN, because the tolerance windows drop missing rows silently and the pool would then be smaller than the user believes it is.

The returned reference matters as much as the pool. n_density and old20 are properties of a word in its language, not among the handful of words a study happens to use, so computing them against a 200-word supplied list would give numbers that mean nothing. When a lexicon is given, the reference is the lexicon's words; only without one does it fall back to the pool itself.

Returns {"pool": DataFrame, "reference": list}. Mirrors load_pool in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def load_pool(path: str, schema: dict, lexicon=None, language=None) -> dict:
    """Load a supplied candidate pool of words and give it the matcher's dimensions.

    A researcher who already has a curated word list (from a previous study, a norming
    session, a colleague) should not have to dress it up as a corpus lexicon
    to get lexsync's matching, validation and datasheet. This reads such a list and
    returns something the matcher accepts.

    The list needs only a ``word`` column. Length and the syllable estimate are derived
    from the form. Everything else is either supplied on the list itself or looked up:
    with ``lexicon`` given, the corpus dimensions (frequency above all) are joined for
    those words, and a word the lexicon does not have is a hard error rather than a
    NaN, because the tolerance windows drop missing rows silently and the pool would
    then be smaller than the user believes it is.

    The returned ``reference`` matters as much as the pool. ``n_density`` and ``old20``
    are properties of a word in its *language*, not among the handful of words a study
    happens to use, so computing them against a 200-word supplied list would give
    numbers that mean nothing. When a lexicon is given, the reference is the lexicon's
    words; only without one does it fall back to the pool itself.

    Returns ``{"pool": DataFrame, "reference": list}``. Mirrors load_pool in
    R_workflow/R/querying.R.
    """
    if ".." in str(path).replace("\\", "/").split("/"):
        raise ValueError("lexsync: pool path must not contain '..'.")
    df = read_csv_utf8(path)
    if "word" not in df.columns:
        raise ValueError(
            "lexsync: supplied pool '%s' is missing the required 'word' column." % path)
    # The same normalisation load_lexicon applies, for the same reason: `word` is the
    # canonical key behind every byte-order tie-break, so the two engines must fold and
    # trim it identically before anything is sorted or numbered by it.
    df = df[df["word"].notna()].copy()
    df["word"] = df["word"].astype(str).str.strip().str.lower()
    df = df[df["word"] != ""].drop_duplicates(subset="word")
    if df.empty:
        raise ValueError(
            "lexsync: supplied pool '%s' has no usable rows: it is empty, or every row "
            "is missing 'word'." % path)
    df = (df.assign(_k=df["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))

    reference = df["word"].tolist()
    if lexicon:
        lex = load_lexicon(lexicon, schema, language=language)
        reference = lex["word"].tolist()
        # `id` is the lexicon's own row number, meaningless once the pool is a subset.
        dims = [c for c in lex.columns if c not in ("word", "language", "source", "id")]
        clash = sorted((c for c in dims if c in df.columns),
                       key=lambda s: s.encode("utf-8"))
        if clash:
            raise ValueError(
                "lexsync: supplied pool '%s' already has column(s) %s, which the lexicon "
                "would also supply. Rename or drop them, so it is unambiguous which "
                "values the matcher used."
                % (path, ", ".join("'%s'" % c for c in clash)))
        known = set(lex["word"])
        absent = sorted((w for w in df["word"] if w not in known),
                        key=lambda s: s.encode("utf-8"))
        if absent:
            shown = absent[:5]
            raise ValueError(
                "lexsync: %d word(s) of supplied pool '%s' are absent from lexicon "
                "'%s': %s%s." % (len(absent), path, lexicon,
                                 ", ".join("'%s'" % w for w in shown),
                                 ", ..." if len(absent) > len(shown) else ""))
        by_word = lex.set_index("word")
        rows = by_word.loc[df["word"].tolist()]
        for d in dims:
            df[d] = rows[d].to_numpy()
    # Derived after any join, so a lexicon cannot overwrite them with its own copies.
    df["length"] = df["word"].str.len()
    df["n_syllables"] = df["word"].map(count_syllables)
    if language is not None:
        df["language"] = language
    df["id"] = np.arange(1, len(df) + 1)
    return {"pool": df.reset_index(drop=True), "reference": reference}

lexsync.merge_norms(lexicon, norms, on='word', columns=None)

Left-join a norm table (e.g. concreteness, age of acquisition, valence).

norms is a data frame or the path to a CSV with a word column and one or more norm columns. This is the connector for semantic dimensions: the norm data themselves are fetched separately (licensing varies), then merged here so the matcher can equate on them.

The result is the lexicon itself with the norm columns appended, and the key is looked up positionally rather than through merge. That is what makes the two engines agree by construction, with nothing to repair afterwards, because merge and R's merge() were measured to diverge in three ways, each of them silent: R hoists the by column to position 1 while pandas keeps the left frame's order, so the column order differed whenever on was not already first; R disambiguates a colliding column name with .x/.y and pandas with _x/_y, and either way a dimension the design matches on disappears under a name nothing looks for; and R's merge(sort = FALSE) leaves the row order unspecified. A positional lookup has none of those degrees of freedom. A colliding name is now an error instead.

The key is trimmed and case-folded on both sides. Only the norm table's side was normalised before, so a lexicon holding Dog matched nothing and the design carried on with an all-NaN dimension. Because both engines agreed on that wrong answer, no parity test could have caught it. The lexicon's own spelling is preserved rather than folded in place: word is the byte-order tie-break behind every selection, so the join must not rewrite it.

Mirrors merge_norms in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def merge_norms(lexicon: pd.DataFrame, norms, on: str = "word", columns=None) -> pd.DataFrame:
    """Left-join a norm table (e.g. concreteness, age of acquisition, valence).

    ``norms`` is a data frame or the path to a CSV with a word column and one or
    more norm columns. This is the connector for semantic dimensions: the norm data
    themselves are fetched separately (licensing varies), then merged here so the
    matcher can equate on them.

    The result is the lexicon itself with the norm columns appended, and the key is
    looked up positionally rather than through ``merge``. That is what makes the two
    engines agree by construction, with nothing to repair afterwards, because ``merge``
    and R's ``merge()`` were measured to diverge in three ways, each of them silent:
    R hoists the ``by`` column to position 1 while pandas keeps the left frame's
    order, so the column order differed whenever ``on`` was not already first; R
    disambiguates a colliding column name with ``.x``/``.y`` and pandas with
    ``_x``/``_y``, and either way a dimension the design matches on disappears under
    a name nothing looks for; and R's ``merge(sort = FALSE)`` leaves the row order
    unspecified. A positional lookup has none of those degrees of freedom. A
    colliding name is now an error instead.

    The key is trimmed and case-folded on *both* sides. Only the norm table's side
    was normalised before, so a lexicon holding ``Dog`` matched nothing and the
    design carried on with an all-``NaN`` dimension. Because both engines agreed on
    that wrong answer, no parity test could have caught it. The lexicon's
    own spelling is preserved rather than folded in place: ``word`` is the
    byte-order tie-break behind every selection, so the join must not rewrite it.

    Mirrors merge_norms in R_workflow/R/querying.R.
    """
    n = norms if isinstance(norms, pd.DataFrame) else read_csv_utf8(norms)
    if on not in lexicon.columns:
        raise ValueError(
            "lexsync: merge_norms needs join column '%s' on the lexicon." % on)
    if on not in n.columns:
        raise ValueError(
            "lexsync: merge_norms needs join column '%s' on the norm table." % on)
    cols = list(columns) if columns else [c for c in n.columns if c != on]
    absent = [c for c in cols if c not in n.columns]
    if absent:
        raise ValueError("lexsync: the norm table has no column(s): %s."
                         % ", ".join("'%s'" % c for c in absent))
    # Silently renaming the clash, as both merges do, is the worst outcome
    # available: a design matching on `frequency` would find neither `frequency.x`
    # nor `frequency_x` and would fail far from the cause, or match on the norm
    # table's column believing it was the lexicon's.
    clash = [c for c in cols if c in lexicon.columns]
    if clash:
        raise ValueError(
            "lexsync: norm column(s) %s already exist on the lexicon. Rename them "
            "in the norm table, or name the ones you want in `columns`."
            % ", ".join("'%s'" % c for c in clash))
    key = _norm_key(n[on])
    keep = (key.notna() & ~key.duplicated()).to_numpy()
    add = n.loc[keep, cols].copy()
    add.index = pd.Index(key.to_numpy()[keep], dtype="object")
    # reindex on a de-duplicated index is a left join: a label that is absent
    # (including a missing lexicon key) yields NaN, exactly as R's match() gives NA.
    joined = add.reindex(_norm_key(lexicon[on]).to_numpy())
    out = lexicon.copy()
    for c in cols:
        out[c] = joined[c].to_numpy()
    return out

Lexical dimensions

Two dimensions arrive with the lexicon and the rest are derived from the orthographic forms. Derive them before matching, and compute the neighbourhood measures against the full lexicon rather than the pool, since a word's neighbours do not stop existing because a design excluded them.

lexsync.add_neighbourhood(df, reference=None, n_old=20)

Coltheart's N (same-length, single substitution) and OLD20 for each word.

Source code in src/lexsync/querying.py

def add_neighbourhood(df: pd.DataFrame, reference=None, n_old: int = 20) -> pd.DataFrame:
    """Coltheart's N (same-length, single substitution) and OLD20 for each word."""
    words = df["word"].astype(str).tolist()
    ref = list(dict.fromkeys(str(w) for w in (reference if reference is not None else words)))
    ref_len = np.array([len(w) for w in ref])
    n_density = np.zeros(len(words), dtype=int)
    old = np.full(len(words), np.nan)
    for i, w in enumerate(words):
        same_len = [r for r, L in zip(ref, ref_len, strict=True) if L == len(w)]
        if same_len:
            hd = np.array([Hamming.distance(w, s) for s in same_len])
            n_density[i] = int(np.sum(hd == 1))
        ld = np.array([Levenshtein.distance(w, r) for r in ref], dtype=float)
        ld = ld[ld > 0]
        if ld.size:
            k = min(n_old, ld.size)
            old[i] = float(np.mean(np.partition(ld, k - 1)[:k]))
    out = df.copy()
    out["n_density"] = n_density
    out["old20"] = old
    return out

lexsync.add_bigram_frequency(df, reference=None)

Mean bigram probability (type-based, non-positional), a phonotactic-probability proxy.

For each word, the mean over its adjacent letter bigrams of the corpus bigram probability (count divided by the total bigram count). Computed from integer counts and rounded, so it is identical in the R and Python engines.

Source code in src/lexsync/querying.py

def add_bigram_frequency(df: pd.DataFrame, reference=None) -> pd.DataFrame:
    """Mean bigram probability (type-based, non-positional), a phonotactic-probability proxy.

    For each word, the mean over its adjacent letter bigrams of the corpus bigram
    probability (count divided by the total bigram count). Computed from integer
    counts and rounded, so it is identical in the R and Python engines.
    """
    ref = [str(w) for w in (reference if reference is not None else df["word"].tolist())]
    counts: dict[str, int] = {}
    total = 0
    for w in ref:
        for i in range(len(w) - 1):
            counts[w[i:i + 2]] = counts.get(w[i:i + 2], 0) + 1
            total += 1
    total = total or 1

    def bf(w):
        w = str(w)
        bgs = [w[i:i + 2] for i in range(len(w) - 1)]
        if not bgs:
            return 0.0
        return _round_dp(sum(counts.get(b, 0) for b in bgs) / len(bgs) / total, 9)

    out = df.copy()
    out["bigram_freq"] = out["word"].map(bf)
    return out

lexsync.add_pair_overlap(df, prime='prime', target='target')

Orthographic overlap between the two members of each pair.

Adds two columns. pair.lev is the Levenshtein distance between the pair's two orthographic forms, and pair.overlap is 1 - lev / max(len), the proportion of the longer form the two share. Overlap is the standard confound control in a priming design: a related pair that also shares letters confounds semantic relatedness with orthographic similarity.

Both engines return identical values, and the reasons are worth stating because they are the constraints on any future relational dimension. The core is an integer edit distance, and rapidfuzz's Levenshtein.distance and stringdist(method = "lv") agree exactly, including on decomposed Unicode and CJK, which is the same cross-library agreement add_neighbourhood already stakes old20 on. Length is counted in code points, len() and R's nchar() default, never in bytes. The arithmetic uses only - and /, which IEEE-754 mandates be correctly rounded, and the result is rounded to nine decimal places, the constant used everywhere else in the package. A degenerate pair of two empty forms returns 0 rather than 0/0, because a NaN would be sorted and compared and would then drop the row from one engine's control window but not the other's.

Source code in src/lexsync/querying.py

def add_pair_overlap(df: pd.DataFrame, prime: str = "prime",
                     target: str = "target") -> pd.DataFrame:
    """Orthographic overlap between the two members of each pair.

    Adds two columns. ``pair.lev`` is the Levenshtein distance between the pair's
    two orthographic forms, and ``pair.overlap`` is ``1 - lev / max(len)``, the
    proportion of the longer form the two share. Overlap is the standard confound
    control in a priming design: a related pair that also shares letters confounds
    semantic relatedness with orthographic similarity.

    Both engines return identical values, and the reasons are worth stating because
    they are the constraints on any future relational dimension. The core is an
    integer edit distance, and rapidfuzz's ``Levenshtein.distance`` and
    ``stringdist(method = "lv")`` agree exactly, including on decomposed Unicode and
    CJK, which is the same cross-library agreement ``add_neighbourhood`` already
    stakes ``old20`` on. Length is counted in code points, ``len()`` and R's
    ``nchar()`` default, never in bytes. The arithmetic uses only ``-`` and ``/``,
    which IEEE-754 mandates be correctly rounded, and the result is rounded to nine
    decimal places, the constant used everywhere else in the package. A degenerate
    pair of two empty forms returns 0 rather than ``0/0``, because a NaN would be
    sorted and compared and would then drop the row from one engine's control window
    but not the other's.
    """
    for col in (prime, target):
        if col not in df.columns:
            raise ValueError("lexsync: add_pair_overlap needs column '%s'." % col)
    a = [str(w).strip().lower() for w in df[prime]]
    b = [str(w).strip().lower() for w in df[target]]
    lev = [Levenshtein.distance(x, y) for x, y in zip(a, b, strict=True)]
    den = [max(len(x), len(y)) for x, y in zip(a, b, strict=True)]
    out = df.copy()
    out["pair.lev"] = np.array(lev, dtype=int)
    out["pair.overlap"] = [0.0 if d == 0 else _round_dp(1 - l / d, 9)
                           for l, d in zip(lev, den, strict=True)]
    return out

lexsync.count_syllables(word)

An orthographic syllable estimate: the number of maximal vowel runs.

Source code in src/lexsync/querying.py

def count_syllables(word) -> int:
    """An orthographic syllable estimate: the number of maximal vowel runs."""
    return len(_VOWELS.findall(str(word).lower()))

Pools and matching

The pool is the set of candidates a design will consider at all, and the matcher works only on what it is given. match_stimuli never reads a design's pool_filters, so build_pool is a required step.

lexsync.build_pool(lexicon, filters=None)

Build an experimental candidate pool by filtering a lexicon.

filters maps a column to either a two-element numeric range or a list of permitted values. A row missing the filtered column is dropped under either kind, and a range with a reversed or non-finite bound is an error rather than an empty pool.

A filter naming a column the frame does not have is silently skipped, because the same function filters lexica, supplied pools and pair tables, and those carry different columns. The cost is that a misspelt key silently widens a selection, so every caller that takes its filters from a design checks the names against the frame first: run_pipeline for pool_filters, match_stimuli for a condition's define_by, and select_continuous_pairs for both. Mirrors build_pool in R_workflow/R/querying.R.

Source code in src/lexsync/querying.py

def build_pool(lexicon: pd.DataFrame, filters: dict | None = None) -> pd.DataFrame:
    """Build an experimental candidate pool by filtering a lexicon.

    ``filters`` maps a column to either a two-element numeric range or a list of
    permitted values. A row missing the filtered column is dropped under either
    kind, and a range with a reversed or non-finite bound is an error rather than an
    empty pool.

    A filter naming a column the frame does not have is silently skipped, because the
    same function filters lexica, supplied pools and pair tables, and those carry
    different columns. The cost is that a misspelt key silently widens a
    selection, so every caller that takes its filters from a design checks the names
    against the frame first: ``run_pipeline`` for ``pool_filters``,
    ``match_stimuli`` for a condition's ``define_by``, and
    ``select_continuous_pairs`` for both. Mirrors build_pool in
    R_workflow/R/querying.R.
    """
    df = lexicon
    if filters:
        for col, rng in filters.items():
            if col not in df.columns:
                continue
            vals = list(rng) if isinstance(rng, (list, tuple)) else [rng]
            if len(vals) == 2 and all(isinstance(v, (int, float)) for v in vals):
                # YAML's .inf and .nan arrive as ordinary floats, and either one,
                # like a reversed range, silently empties the pool row by row.
                # Non-finite first: a NaN bound would make the reversal test
                # meaningless. Equal bounds stay legal (the zh design uses [2, 2]).
                if not all(math.isfinite(float(v)) for v in vals):
                    raise ValueError(
                        f"lexsync: filter '{col}' has a non-finite bound; "
                        "ranges need finite numbers.")
                if vals[0] > vals[1]:
                    raise ValueError(
                        f"lexsync: filter '{col}' has a reversed range; "
                        "give it as [low, high].")
                df = df[df[col].notna() & (df[col] >= vals[0]) & (df[col] <= vals[1])]
            else:
                df = df[df[col].notna() & df[col].astype(str).isin([str(v) for v in vals])]
    return df.reset_index(drop=True)

lexsync.match_stimuli(pool, design, schema, verbose=False)

Match stimuli across conditions on the match_on dimensions.

Two policies govern degraded selections, each read from the design's matching block with the schema as fallback: shortfall ("error", the default, refuses to return fewer sets than requested; "allow" accepts the shrink) and on_insufficient_tolerance ("relax", the default, widens an undersupplied tolerance window to the full condition subpool and records the relaxation in out.attrs["audit"]; "error" refuses instead).

Source code in src/lexsync/matching.py

def match_stimuli(pool: pd.DataFrame, design: dict, schema: dict, verbose: bool = False) -> pd.DataFrame:
    """Match stimuli across conditions on the ``match_on`` dimensions.

    Two policies govern degraded selections, each read from the design's
    ``matching`` block with the schema as fallback: ``shortfall`` ("error", the
    default, refuses to return fewer sets than requested; "allow" accepts the
    shrink) and ``on_insufficient_tolerance`` ("relax", the default, widens an
    undersupplied tolerance window to the full condition subpool and records the
    relaxation in ``out.attrs["audit"]``; "error" refuses instead).
    """
    conditions = design.get("conditions")
    if not conditions:
        # Without this, the anchor lookup below dies with a KeyError here and a
        # bare subscript error in R, two different messages for the same mistake.
        raise ValueError("lexsync: the design has no conditions; a matched design "
                         "needs a conditions list.")
    match_on = list(design["match_on"])
    n = design.get("n_per_condition") or design.get("n_per_cell") or 20
    # Tolerance window k per dimension (window = anchor mean +/- k * SD). A design
    # may override the schema defaults per dimension, e.g. to reproduce a published
    # study's exact windows (Gonzalez Alonso et al. used SD/9 for frequency).
    tol_k = dict(schema["matching"].get("tolerance_k") or {})
    tol_k.update((design.get("matching") or {}).get("tolerance_k") or {})
    shortfall = _resolve_policy(design, schema, "shortfall", "error", ("error", "allow"))
    on_tol = _resolve_policy(design, schema, "on_insufficient_tolerance", "relax",
                             ("relax", "error"))

    for d in match_on:
        if d not in pool.columns:
            raise ValueError(f"lexsync: dimension '{d}' is absent from the pool.")
        k = tol_k.get(d, 2)
        if isinstance(k, (int, float)) and k < 0:
            # A negative k inverts the window (upper bound below the lower), which
            # empties the candidate set and silently relaxes to the full pool,
            # which is never intended.
            raise ValueError(f"lexsync: tolerance_k for dimension '{d}' is negative; "
                             f"tolerances must be zero or positive.")
    cnames = [c["name"] for c in conditions]
    dup_name = [x for x in cnames if cnames.count(x) > 1]
    if dup_name:
        raise ValueError(f"lexsync: condition name '{dup_name[0]}' appears more than "
                         f"once; condition names must be unique.")
    for c in conditions:
        for d in (c.get("define_by") or {}):
            # build_pool skips a column it does not recognise, so a misspelt
            # define_by key would silently hand the condition the whole pool as
            # its subpool and the manipulated contrast would vanish while
            # matching proceeds.
            if d not in pool.columns:
                raise ValueError(f"lexsync: dimension '{d}' in condition '{c['name']}' "
                                 f"is absent from the pool.")

    center = np.array([_exact_mean(pool[d].dropna()) for d in match_on], dtype=float)
    scale = np.array([_exact_sd(pool[d].dropna()) for d in match_on], dtype=float)
    scale[np.isnan(scale) | (scale == 0)] = 1.0

    # .get, not ["define_by"]: a condition without define_by draws on the whole
    # pool, as the R engine's NULL does through build_pool.
    subpools = [build_pool(pool, c.get("define_by")) for c in conditions]
    cond_names = [c["name"] for c in conditions]

    method = ((design.get("matching") or {}).get("method")
              or (schema.get("matching") or {}).get("method") or "standardised_euclidean")
    if method not in _KNOWN_METHODS:
        raise ValueError(f"lexsync: unknown matching method '{method}'. "
                         f"Known methods: {', '.join(_KNOWN_METHODS)}.")
    if method in ("joint", "optimal") and len(conditions) != 2:
        # Both are pairwise matchers; falling back to the anchor matcher here would
        # make the datasheet's recorded method differ from the one actually used.
        raise ValueError(f"lexsync: matching method '{method}' requires exactly two "
                         f"conditions, got {len(conditions)}.")
    if method == "joint" and len(conditions) == 2:
        out = _match_joint(subpools, cond_names, match_on, center, scale, n)
        _check_shortfall(out["set"].nunique(), n, shortfall)
        _assert_distinct_words(out)
        return out
    if method == "optimal" and len(conditions) == 2:
        out = _match_optimal(subpools, cond_names, match_on, center, scale, n)
        _check_shortfall(out["set"].nunique(), n, shortfall)
        _assert_distinct_words(out)
        return out
    # A covariance-aware metric for Mahalanobis matching (None -> plain Euclidean).
    metric = _maha_metric(_zmat(pool, match_on, center, scale)) if method == "mahalanobis" else None

    anchor_pool = subpools[0]
    if len(anchor_pool) == 0:
        raise ValueError(f"lexsync: anchor condition '{cond_names[0]}' has no candidates.")
    # An anchor without define_by orders by the default dimension, as the R
    # engine's names(NULL)[1] falls back to "frequency".
    ord_keys = list((conditions[0].get("define_by") or {}).keys())
    ord_dim = ord_keys[0] if ord_keys else "frequency"
    if ord_dim not in anchor_pool.columns:
        ord_dim = "frequency"
    anchor_pool = (anchor_pool.assign(_k=anchor_pool["word"].map(lambda w: w.encode("utf-8")))
                   .sort_values([ord_dim, "_k"], kind="mergesort").drop(columns="_k").reset_index(drop=True))
    n_take = min(n, len(anchor_pool))
    idx1 = np.unique(np.round(np.linspace(1, len(anchor_pool), n_take)).astype(int))
    anchor = anchor_pool.iloc[idx1 - 1].copy()
    anchor["condition"] = cond_names[0]
    n_take = len(anchor)
    _check_shortfall(n_take, n, shortfall)
    if verbose and n_take < n:
        print(f"lexsync: anchor condition '{cond_names[0]}' yields only {n_take} items; "
              f"n_per_condition is {n}.")
    z_anchor = _zmat(anchor, match_on, center, scale)

    win = {}
    for d in match_on:
        m = _exact_mean(anchor[d].dropna())
        s = _exact_sd(anchor[d].dropna())
        k = tol_k.get(d, 2)
        win[d] = (m - k * s, m + k * s)

    selected = [anchor]
    used_words = set(anchor["word"])
    relaxations = []

    for ci in range(1, len(conditions)):
        cname = cond_names[ci]
        cand = subpools[ci]
        cand = cand[~cand["word"].isin(used_words)]
        if len(cand) == 0:
            raise ValueError(f"lexsync: condition '{cname}' has no candidates left to match.")
        keep = np.ones(len(cand), dtype=bool)
        for d in match_on:
            col = cand[d].to_numpy()
            keep &= (col >= win[d][0]) & (col <= win[d][1])
        cand_f = cand[keep]
        if len(cand_f) < n_take:
            if on_tol == "error":
                raise ValueError(f"lexsync: condition '{cname}' has {len(cand_f)} candidates "
                                 f"within tolerance but {n_take} are needed; raise tolerance_k "
                                 f"or set matching: on_insufficient_tolerance: relax to widen "
                                 f"the window.")
            if verbose:
                print(f"lexsync: condition '{cname}' has {len(cand_f)} candidates within tolerance "
                      f"(< {n_take} needed); relaxing the window.")
            # The relaxation changes what "matched" means for this condition, so it
            # is recorded on the result for the run log and datasheet, where a
            # console message would leave no trace.
            relaxations.append({"condition": cname,
                                "n_within_tolerance": int(len(cand_f)),
                                "n_needed": int(n_take)})
            cand_f = cand
        if len(cand_f) < n_take:
            # The assignment below would otherwise re-pick an exhausted pool's first
            # item (every remaining distance is Inf, so the tie-break decides), and
            # emit the same word in several sets.
            raise ValueError(f"lexsync: condition '{cname}' has only {len(cand_f)} candidate(s) "
                             f"but {n_take} are needed; widen pool_filters/define_by or lower "
                             f"n_per_condition.")
        # Relaxing the window re-admits rows missing a matched dimension. Their distance
        # is NaN and they rank last, so they are never assigned; counting them would let an
        # NA-depleted pool past the guard above and back into re-picking used rows.
        usable = int(cand_f[match_on].notna().all(axis=1).sum())
        if usable < n_take:
            raise ValueError(f"lexsync: condition '{cname}' has only {usable} usable candidate(s) "
                             f"complete on the matched dimensions but {n_take} are needed; widen "
                             f"pool_filters/define_by or lower n_per_condition.")
        cand_f = cand_f.reset_index(drop=True)
        z_cand = _zmat(cand_f, match_on, center, scale)
        words = cand_f["word"].to_numpy()
        ids = cand_f["id"].to_numpy()
        used = np.zeros(len(cand_f), dtype=bool)
        pick = np.empty(n_take, dtype=int)
        for a in range(n_take):
            # Rounded to 9 dp through the shared rule so the stable tie-break below
            # is itself reproducible across R and Python. np.round would pair
            # numpy's scale-rint-unscale with R's decimal algorithm, a pairing
            # io_utils documents as disagreeing at boundaries. On the mahalanobis
            # path the inputs already differ in their last bits, so the absorber
            # must be the same function in both engines.
            delta = z_cand - z_anchor[a]
            if metric is None:
                dvec = _round_dp_vec(np.sqrt((delta ** 2).sum(axis=1)), 9)
            else:
                dvec = _round_dp_vec(np.sqrt(np.maximum((delta @ metric * delta).sum(axis=1), 0.0)), 9)
            dvec = np.where(used, np.inf, dvec)
            # A relaxed window can admit a row whose matched dimension is missing, and
            # its distance is NaN. Rank those last, as R's order(na.last = TRUE) does:
            # a bare min() over NaN keeps whichever row it saw first, so the selection
            # would otherwise depend on pool row order and diverge from the R engine.
            nan_last = np.isnan(dvec)
            best = min(range(len(cand_f)),
                       key=lambda j: (bool(nan_last[j]), 0.0 if nan_last[j] else dvec[j],
                                      words[j].encode("utf-8"), int(ids[j])))
            pick[a] = best
            used[best] = True
        sel = cand_f.iloc[pick].copy()
        sel["condition"] = cname
        used_words |= set(sel["word"])
        selected.append(sel)

    common = [c for c in selected[0].columns if all(c in s.columns for s in selected)]
    out = pd.concat([s[common] for s in selected], ignore_index=True)
    out["set"] = list(range(1, n_take + 1)) * len(conditions)
    _assert_distinct_words(out)
    if relaxations:
        # pd.concat and rbind both drop attrs, so the pipeline reads this
        # immediately after the call, before any reshaping.
        out.attrs["audit"] = {"window_relaxations": relaxations}
    return out

lexsync.resample_stimuli(pool, design, schema, n_sets, verbose=False)

Produce n_sets disjoint matched item sets (a replicate column).

Each replicate is an independent, fully matched set drawn from the pool with the items of earlier replicates removed, so no item is reused. This lets a study treat its items as a random factor (running different item samples across participant groups, or showing an effect holds across samples) instead of treating them as a fixed set (Clark, 1973; Yarkoni, 2022). Deterministic: the matcher is deterministic and the used-item set evolves identically across engines.

The replicates are concatenated, which drops the attrs["audit"] entry :func:match_stimuli uses to report a relaxed tolerance window, so a relaxation inside a replicate reaches the console under verbose but not the run log or the datasheet.

Source code in src/lexsync/matching.py

def resample_stimuli(pool: pd.DataFrame, design: dict, schema: dict,
                     n_sets: int, verbose: bool = False) -> pd.DataFrame:
    """Produce ``n_sets`` disjoint matched item sets (a ``replicate`` column).

    Each replicate is an independent, fully matched set drawn from the pool with
    the items of earlier replicates removed, so no item is reused. This lets a
    study treat its items as a random factor (running different item samples
    across participant groups, or showing an effect holds across samples) instead
    of treating them as a fixed set (Clark, 1973; Yarkoni, 2022). Deterministic: the
    matcher is deterministic and the used-item set evolves identically across engines.

    The replicates are concatenated, which drops the ``attrs["audit"]`` entry
    :func:`match_stimuli` uses to report a relaxed tolerance window, so a relaxation
    inside a replicate reaches the console under ``verbose`` but not the run log or
    the datasheet.
    """
    used: set = set()
    parts = []
    for k in range(1, int(n_sets) + 1):
        pk = pool[~pool["word"].isin(used)].reset_index(drop=True)
        sk = match_stimuli(pk, design, schema, verbose=verbose).copy()
        sk["replicate"] = k
        used |= set(sk["word"])
        parts.append(sk)
    return pd.concat(parts, ignore_index=True)

Continuous designs

Dichotomising a continuous predictor costs power and can introduce selection artefacts. A design may instead span the predictor evenly while holding its controls near-constant, and be analysed by regression or a mixed model.

lexsync.select_continuous_stimuli(pool, design, schema, verbose=False, key='word', label='continuous', renumber_sets=True)

Select a set that spans a continuous predictor, holding controls constant.

Instead of dichotomising the predictor into conditions and matching, items are chosen to cover the predictor's range evenly while the control dimensions are held within a tolerance band, so they stay near-constant and near-uncorrelated with the predictor. The set is analysed by regression / mixed models rather than by between-condition contrasts, which avoids the loss of power and the selection artefacts of matched dichotomies (Kuperman, 2015; Liben-Nowell et al., 2019).

Two deterministic passes reuse the matcher's even-spread primitive, so the R and Python engines select byte-identical stimuli: an even spread over the predictor defines a tolerance window on each control; the pool is filtered to that window; a second even spread over the filtered pool is the selection. There is no per-item matching and no random number generator.

The design is checked before anything is selected, so a design that cannot be honoured raises ValueError outright. continuous.controls must be non-empty and must not name the predictor, match_on must name exactly the same dimensions as continuous.controls, every dimension named and the key column must be present in the pool, no tolerance_k may be negative, and the pool must not be empty.

Returns the selected stimuli. Unless label is None the condition column is set to it, continuous by default, and unless renumber_sets is False the set column is renumbered 1..n.

Source code in src/lexsync/matching.py

def select_continuous_stimuli(pool: pd.DataFrame, design: dict, schema: dict,
                              verbose: bool = False, key: str = "word",
                              label: str | None = "continuous",
                              renumber_sets: bool = True) -> pd.DataFrame:
    """Select a set that spans a continuous predictor, holding controls constant.

    Instead of dichotomising the predictor into conditions and matching, items are
    chosen to cover the predictor's range evenly while the control dimensions are
    held within a tolerance band, so they stay near-constant and near-uncorrelated
    with the predictor. The set is analysed by regression / mixed models rather
    than by between-condition contrasts, which avoids the loss of power and the
    selection artefacts of matched dichotomies (Kuperman, 2015; Liben-Nowell et
    al., 2019).

    Two deterministic passes reuse the matcher's even-spread primitive, so the R
    and Python engines select byte-identical stimuli: an even spread over the
    predictor defines a tolerance window on each control; the pool is filtered to
    that window; a second even spread over the filtered pool is the selection.
    There is no per-item matching and no random number generator.

    The design is checked before anything is selected, so a design that cannot be
    honoured raises ``ValueError`` outright. ``continuous.controls`` must be non-empty
    and must not name the predictor, ``match_on`` must name exactly the same dimensions
    as ``continuous.controls``, every dimension named and the ``key`` column must be
    present in the pool, no ``tolerance_k`` may be negative, and the pool must not be
    empty.

    Returns the selected stimuli. Unless ``label`` is ``None`` the ``condition``
    column is set to it, ``continuous`` by default, and unless ``renumber_sets`` is
    ``False`` the ``set`` column is renumbered ``1..n``.
    """
    cfg = design["continuous"]
    predictor = cfg["predictor"]
    controls = list(cfg.get("controls") or [])
    match_on = list(design.get("match_on") or [])
    if not controls:
        raise ValueError("lexsync: a continuous design needs at least one control "
                         "dimension (continuous.controls must be non-empty).")
    if predictor in controls:
        raise ValueError(f"lexsync: the continuous predictor '{predictor}' must not "
                         "also appear in continuous.controls.")
    if sorted(match_on) != sorted(controls):
        raise ValueError("lexsync: for a continuous design, match_on must equal "
                         "continuous.controls.")
    for d in [predictor] + controls:
        if d not in pool.columns:
            raise ValueError(f"lexsync: dimension '{d}' is absent from the pool.")
    if key not in pool.columns:
        raise ValueError(
            "lexsync: the continuous tie-break column '%s' is absent from the pool." % key)
    n = design.get("n_per_condition") or design.get("n_per_cell") or 60
    tol_k = dict(schema["matching"].get("tolerance_k") or {})
    tol_k.update((design.get("matching") or {}).get("tolerance_k") or {})
    shortfall = _resolve_policy(design, schema, "shortfall", "error", ("error", "allow"))
    on_tol = _resolve_policy(design, schema, "on_insufficient_tolerance", "relax",
                             ("relax", "error"))
    for d in controls:
        k = tol_k.get(d, 2)
        if isinstance(k, (int, float)) and k < 0:
            raise ValueError(f"lexsync: tolerance_k for dimension '{d}' is negative; "
                             f"tolerances must be zero or positive.")

    def even_spread(df):
        # `key` is the deterministic tie-break when two rows share a predictor
        # value: `word` for a corpus pool, `set` for a collapsed pair table where
        # no `word` column exists. A text key is compared as UTF-8 BYTES, which is
        # what R's radix sort does; a numeric key is compared numerically in both.
        k = df[key]
        if pd.api.types.is_object_dtype(k) or pd.api.types.is_string_dtype(k):
            k = k.map(lambda w: str(w).encode("utf-8"))
        df = (df.assign(_k=k)
              .sort_values([predictor, "_k"], kind="mergesort")
              .drop(columns="_k").reset_index(drop=True))
        if len(df) == 0:
            return df
        n_take = min(n, len(df))
        idx = np.unique(np.round(np.linspace(1, len(df), n_take)).astype(int))
        return df.iloc[idx - 1].reset_index(drop=True)

    # Pass 1: an even spread over the whole pool defines the control windows.
    spread = even_spread(pool)
    if len(spread) == 0:
        raise ValueError("lexsync: the pool is empty for the continuous design.")
    win = {}
    for d in controls:
        m = _exact_mean(spread[d].dropna())
        s = _exact_sd(spread[d].dropna())
        k = tol_k.get(d, 2)
        win[d] = (m - k * s, m + k * s)
    keep = np.ones(len(pool), dtype=bool)
    for d in controls:
        col = pool[d].to_numpy()
        keep &= (col >= win[d][0]) & (col <= win[d][1])
    filtered = pool[keep]
    relaxations = []
    if len(filtered) < n:
        if on_tol == "error":
            raise ValueError(f"lexsync: {len(filtered)} items lie within the control windows "
                             f"but {n} are needed; raise tolerance_k or set matching: "
                             f"on_insufficient_tolerance: relax to widen the window.")
        if verbose:
            print(f"lexsync: {len(filtered)} items within the control windows "
                  f"(< {n} needed); relaxing to the full pool.")
        relaxations.append({"condition": "continuous",
                            "n_within_tolerance": int(len(filtered)),
                            "n_needed": int(n)})
        filtered = pool
    # Pass 2: an even spread over the filtered pool is the selection.
    sel = even_spread(filtered).copy()
    _check_shortfall(len(sel), n, shortfall, continuous=True)
    # A pair table already carries its own `condition` and `set`, which the Latin
    # square and the trial-order digest depend on, so the pair path passes
    # label=None and renumber_sets=False to leave both alone.
    if label is not None:
        sel["condition"] = label
    if renumber_sets:
        sel["set"] = list(range(1, len(sel) + 1))
    if relaxations:
        sel.attrs["audit"] = {"window_relaxations": relaxations}
    return sel

lexsync.match_report_continuous(stimuli, predictor, controls, schema)

Realised-control report for a continuous design.

Returns the same {descriptives, comparisons} shape as :func:match_report, so the pipeline and datasheet stay uniform, but the comparisons describe a continuous predictor instead of a between-condition contrast: the predictor's realised span and, for each control, its Pearson correlation with the predictor (near zero when the control is held constant). The set is meant for regression / mixed-model analysis, not equivalence tests.

Source code in src/lexsync/validation.py

def match_report_continuous(stimuli, predictor, controls, schema) -> dict:
    """Realised-control report for a continuous design.

    Returns the same ``{descriptives, comparisons}`` shape as :func:`match_report`,
    so the pipeline and datasheet stay uniform, but the comparisons describe a
    continuous predictor instead of a between-condition contrast: the predictor's
    realised span and, for each control, its Pearson correlation with the predictor
    (near zero when the control is held constant). The set is meant for regression /
    mixed-model analysis, not equivalence tests.
    """
    desc = describe_stimuli(stimuli, [predictor] + controls)
    pv = pd.to_numeric(stimuli[predictor], errors="coerce").to_numpy(dtype=float)
    valid = pv[~np.isnan(pv)]
    # None (not NaN) when the predictor has no span, so both engines agree.
    span = _round_dp(float(valid.max() - valid.min()), 3) if len(valid) >= 2 else None
    rows = [dict(dimension=predictor, role="predictor", pearson_r=None, predictor_span=span)]
    for c in controls:
        cv = pd.to_numeric(stimuli[c], errors="coerce").to_numpy(dtype=float)
        r = _pearson(pv, cv)
        rows.append(dict(dimension=c, role="control",
                         pearson_r=_round_dp(r, 3) if r is not None else None,
                         predictor_span=span))
    return dict(descriptives=desc, comparisons=pd.DataFrame(rows))

Pseudoword generation

Non-words are generated deterministically, with no sampling anywhere, preserving length exactly and keeping every letter bigram attested in the corpus. Both methods select byte-identical stimuli in the R and Python engines.

lexsync.generate_pseudowords(base_words, reference_words)

A length-matched pseudoword for each base word.

Base words are processed in byte order so the used set evolves identically across engines. Returns a frame with base_word and pseudoword.

Source code in src/lexsync/generation.py

def generate_pseudowords(base_words, reference_words) -> pd.DataFrame:
    """A length-matched pseudoword for each base word.

    Base words are processed in byte order so the ``used`` set evolves identically
    across engines. Returns a frame with ``base_word`` and ``pseudoword``.
    """
    base = [str(w) for w in base_words]
    lexicon = set(str(w) for w in reference_words)
    bigrams = bigram_counts(reference_words)
    order = sorted(range(len(base)), key=lambda i: base[i].encode("utf-8"))
    used: set = set()
    pseudo = [None] * len(base)
    for i in order:
        pw = make_pseudoword(base[i], bigrams, lexicon, used)
        if pw is None:
            raise ValueError(f"lexsync: could not generate a pseudoword for '{base[i]}'.")
        used.add(pw)
        pseudo[i] = pw
    return pd.DataFrame({"base_word": base, "pseudoword": pseudo})

lexsync.make_pseudoword(word, bigrams, lexicon, used)

The most bigram-plausible legal non-word at the smallest edit distance.

Searches single-letter substitutions first, then two-letter substitutions; candidates are ranked by summed bigram frequency with a byte-order tie-break, so the choice is deterministic and identical across engines.

Source code in src/lexsync/generation.py

def make_pseudoword(word: str, bigrams: dict, lexicon: set, used: set) -> str | None:
    """The most bigram-plausible legal non-word at the smallest edit distance.

    Searches single-letter substitutions first, then two-letter substitutions;
    candidates are ranked by summed bigram frequency with a byte-order tie-break,
    so the choice is deterministic and identical across engines.
    """
    word = str(word)
    L = len(word)
    # Distance 1: one substituted position.
    cands = []
    for pos in range(L):
        for c in _LETTERS:
            if c == word[pos]:
                continue
            cand = word[:pos] + c + word[pos + 1:]
            if cand in lexicon or cand in used:
                continue
            if _legal(cand, bigrams):
                cands.append(cand)
    if not cands:
        # Distance 2: two substituted positions (rare fallback).
        for i in range(L):
            for j in range(i + 1, L):
                for ci in _LETTERS:
                    if ci == word[i]:
                        continue
                    for cj in _LETTERS:
                        if cj == word[j]:
                            continue
                        cand = word[:i] + ci + word[i + 1:j] + cj + word[j + 1:]
                        if cand in lexicon or cand in used:
                            continue
                        if _legal(cand, bigrams):
                            cands.append(cand)
    if not cands:
        return None
    return min(cands, key=lambda c: (-_score(c, bigrams), c.encode("utf-8")))

lexsync.build_lexdec_stimuli(pool, n, reference_words=None, method='letter_substitution')

Assemble a word-vs-pseudoword lexical-decision set from a candidate pool.

Real words are drawn by an even spread across the byte-ordered pool (the same deterministic device as the matcher's anchor), then a length-matched pseudoword is generated for each. The pool is first filtered to lower-case a-z forms, the only ones the pseudoword generators are defined for, so the eligible pool can be smaller than the request; the pipeline's shortfall policy then decides whether that errors. reference_words (the full lexicon) supplies the bigram statistics and the real-word list a pseudoword must avoid; it falls back to the pool when not given. The presented string is the target column; conditions are word and pseudoword and set pairs them.

Source code in src/lexsync/generation.py

def build_lexdec_stimuli(pool: pd.DataFrame, n: int, reference_words=None,
                         method: str = "letter_substitution") -> pd.DataFrame:
    """Assemble a word-vs-pseudoword lexical-decision set from a candidate pool.

    Real words are drawn by an even spread across the byte-ordered pool (the same
    deterministic device as the matcher's anchor), then a length-matched
    pseudoword is generated for each. The pool is first filtered to lower-case
    a-z forms, the only ones the pseudoword generators are defined for, so the
    eligible pool can be smaller than the request; the pipeline's shortfall
    policy then decides whether that errors. ``reference_words`` (the full lexicon)
    supplies the bigram statistics and the real-word list a pseudoword must avoid;
    it falls back to the pool when not given. The presented string is the
    ``target`` column; conditions are ``word`` and ``pseudoword`` and ``set``
    pairs them.
    """
    pool = (pool.assign(_k=pool["word"].map(lambda w: w.encode("utf-8")))
            .sort_values("_k").drop(columns="_k").reset_index(drop=True))
    # The pseudoword generators are orthographic and defined for Latin a-z words;
    # any word with another character is skipped, keeping the two engines in step.
    pool = pool[pool["word"].astype(str).str.fullmatch(r"[a-z]+")].reset_index(drop=True)
    if len(pool) == 0:
        raise ValueError("lexsync: lexical-decision pool has no a-z words.")
    n_take = min(n, len(pool))
    idx = np.unique(np.round(np.linspace(1, len(pool), n_take)).astype(int)) - 1
    words = pool.iloc[idx].reset_index(drop=True)
    ref = list(reference_words) if reference_words is not None else pool["word"].tolist()
    if method == "subsyllabic":
        gen = generate_pseudowords_subsyllabic(words["word"].tolist(), ref)
    elif method == "letter_substitution":
        gen = generate_pseudowords(words["word"].tolist(), ref)
    else:
        raise ValueError(f"lexsync: unknown pseudoword generation method '{method}'.")
    pw_map = dict(zip(gen["base_word"], gen["pseudoword"], strict=True))

    real = pd.DataFrame({
        "target": words["word"].tolist(), "word": words["word"].tolist(),
        "condition": "word", "length": words["length"].tolist(),
        "set": list(range(1, len(words) + 1)),
    })
    pseudo = pd.DataFrame({
        "target": [pw_map[w] for w in words["word"]], "word": [pw_map[w] for w in words["word"]],
        "condition": "pseudoword", "length": words["length"].tolist(),
        "set": list(range(1, len(words) + 1)),
    })
    out = pd.concat([real, pseudo], ignore_index=True)
    return out

Paradigms and trial events

A trial is a list of event dictionaries rather than backend code, which is what lets one engine serve five paradigms and three presentation targets. A design either names a paradigm and inherits its event sequence, or supplies its own events.

lexsync.PARADIGMS = {'factorial': {'stimulus_fields': ['word'], 'counterbalance': 'factorial', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{word}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'lexical_decision': {'stimulus_fields': ['target'], 'counterbalance': 'factorial', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{target}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'priming': {'stimulus_fields': ['prime', 'target'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'text', 'content': '{prime}', 'duration_frames': 3, 'trigger': 20, 'onset_locked': True}, {'type': 'mask', 'content': '#####', 'duration_frames': 2}, {'type': 'text', 'content': '{target}', 'duration_frames': _WORD, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['left', 'right'], 'timeout_ms': 2000}, {'type': 'blank', 'duration_frames': _ISI}]}, 'categorisation': {'stimulus_fields': ['target', 'category', 'answer'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_ms': 500}, {'type': 'text', 'content': '{category}', 'duration_ms': 750}, {'type': 'text', 'content': '{target}', 'duration_ms': 800, 'trigger': 'condition', 'onset_locked': True}, {'type': 'response', 'keys': ['f', 'j'], 'timeout_ms': 2500}, {'type': 'blank', 'duration_ms': 250}]}, 'self_paced_reading': {'stimulus_fields': ['sentence', 'question'], 'counterbalance': 'latin_square_target', 'events': [{'type': 'fixation', 'content': '+', 'duration_frames': _FIX}, {'type': 'region_by_region', 'content': '{sentence}', 'advance': 'space', 'critical_region_trigger': 'condition'}, {'type': 'question', 'content': '{question}', 'keys': ['f', 'j'], 'timeout_ms': 5000}, {'type': 'blank', 'duration_frames': _ISI}]}} module-attribute

lexsync.resolve_events(design)

Return the event list for a design: explicit events or paradigm default.

Source code in src/lexsync/paradigms.py

def resolve_events(design: dict) -> list:
    """Return the event list for a design: explicit ``events`` or paradigm default."""
    if design.get("events"):
        return [dict(e) for e in design["events"]]
    name = design.get("paradigm", "factorial")
    return [dict(e) for e in get_paradigm(name)["events"]]

lexsync.resolve_trial_timing(stimuli, design, schema)

Realise per-trial event durations onto the stimuli table.

An event may declare a duration that varies from trial to trial, either read from an item column or drawn from a range. A drawn value is a pure function of the keyed hash, so both engines realise the same milliseconds, and it is written into the stimuli table as well as the generated script, because timing that varies is a variable the analysis needs, not presentation detail.

Source code in src/lexsync/scripting.py

def resolve_trial_timing(stimuli: pd.DataFrame, design: dict, schema: dict) -> pd.DataFrame:
    """Realise per-trial event durations onto the stimuli table.

    An event may declare a duration that varies from trial to trial, either read
    from an item column or drawn from a range. A drawn value is a pure function of
    the keyed hash, so both engines realise the same milliseconds, and it is
    written into the stimuli table as well as the generated script, because timing
    that varies is a variable the analysis needs, not presentation detail.
    """
    events = resolve_events(design)
    seed = (schema or {}).get("seed", 1)
    stimuli = stimuli.copy()
    for i, ev in enumerate(events, start=1):
        spec = _duration_spec(ev, i)
        if spec is None or spec["jitter"] is None:
            # from_column reads a column the items already carry; nothing to realise.
            if spec is not None and spec["column"] not in stimuli.columns:
                raise ValueError(
                    "lexsync: event %d reads its duration from column '%s', which the "
                    "items do not have." % (i, spec["column"]))
            continue
        lo, hi = spec["jitter"]
        # The key names the column as well as the trial, so two jittered events in
        # one design draw independently, where a key naming only the trial would
        # give them one shared value.
        lists = stimuli["list"] if "list" in stimuli.columns else [1] * len(stimuli)
        keys = ["|".join([_key_part(seed), "jitter", spec["column"],
                          _key_part(l), _key_part(s), _key_part(c)])
                for l, s, c in zip(lists, stimuli["set"], stimuli["condition"], strict=True)]
        stimuli[spec["column"]] = [hash_int_range(k, lo, hi) for k in keys]
    return stimuli

lexsync.required_fields(design)

Trial fields a design needs present in its items (paradigm + events).

Source code in src/lexsync/paradigms.py

def required_fields(design: dict) -> list:
    """Trial fields a design needs present in its items (paradigm + events)."""
    name = design.get("paradigm", "factorial")
    base = list(get_paradigm(name)["stimulus_fields"]) if name in PARADIGMS else []
    for f in referenced_fields(resolve_events(design)):
        if f not in base:
            base.append(f)
    return base

Counterbalancing

Two recipes are available, and the paradigm chooses between them. Trial order comes from a seeded, keyed-hash shuffle, a pure function of the design with no generator behind it, so the same seed gives the same order in both engines. balance_lists is the optional search for a list assignment whose lists are equated on the item dimensions, where the plain deal goes by set rank.

lexsync.counterbalance(stimuli, design, schema, list_of_set=None)

Source code in src/lexsync/counterbalancing.py

def counterbalance(stimuli: pd.DataFrame, design: dict, schema: dict,
                   list_of_set=None) -> pd.DataFrame:
    # Resampled designs counterbalance each replicate (an independent item set)
    # on its own, so trial order is numbered within each replicate.
    if "replicate" in stimuli.columns and stimuli["replicate"].nunique() > 1:
        parts = [_counterbalance_one(g.reset_index(drop=True), design, schema, list_of_set)
                 for _, g in stimuli.groupby("replicate", sort=True)]
        return pd.concat(parts, ignore_index=True)
    return _counterbalance_one(stimuli, design, schema, list_of_set)

lexsync.balance_lists(stimuli, design, schema)

Assign item sets to lists so the lists match on the item dimensions.

The factorial recipe's default deal is by set rank, which balances nothing. This searches instead for an assignment whose lists have near-equal totals on each declared dimension, by steepest-descent pairwise swaps between lists. List sizes are preserved, because a swap exchanges one set for another.

The search is deterministic and identical in the R and Python engines: the objective is all-integer (see the notes in this module), the descent takes the single best swap each pass, and ties are broken by the seeded keyed hash rather than by position, so no list is favoured by being numbered first. Because the cost is a non-negative integer that strictly decreases, the search terminates; max_passes bounds it anyway and the report says whether the bound was reached.

Five situations raise ValueError rather than being answered with an assignment that would mislead. A Latin-square design is refused because every item already appears in every list there, so there is nothing left to equate, and fewer than two lists leaves no pair of lists to exchange sets between. A design with no resolvable balance dimension is refused, as is one naming a dimension the stimuli do not carry, and the message names the columns. The last refusal is arithmetic: the search stops if the integer objective would leave the range a double represents exactly, since past that point the two engines could disagree.

Returns {"list_of_set": {set: list}, "report": {...}}.

Source code in src/lexsync/counterbalancing.py

def balance_lists(stimuli: pd.DataFrame, design: dict, schema: dict) -> dict:
    """Assign item sets to lists so the lists match on the item dimensions.

    The factorial recipe's default deal is by set rank, which balances nothing. This
    searches instead for an assignment whose lists have near-equal totals on each
    declared dimension, by steepest-descent pairwise swaps between lists. List sizes
    are preserved, because a swap exchanges one set for another.

    The search is deterministic and identical in the R and Python engines: the
    objective is all-integer (see the notes in this module), the descent takes the
    single best swap each pass, and ties are broken by the seeded keyed hash rather
    than by position, so no list is favoured by being numbered first. Because the cost
    is a non-negative integer that strictly decreases, the search terminates;
    ``max_passes`` bounds it anyway and the report says whether the bound was reached.

    Five situations raise ``ValueError`` rather than being answered with an assignment
    that would mislead. A Latin-square design is refused because every item already
    appears in every list there, so there is nothing left to equate, and fewer than two
    lists leaves no pair of lists to exchange sets between. A design with no resolvable
    balance dimension is refused, as is one naming a dimension the stimuli do not carry,
    and the message names the columns. The last refusal is arithmetic: the search stops
    if the integer objective would leave the range a double represents exactly, since
    past that point the two engines could disagree.

    Returns ``{"list_of_set": {set: list}, "report": {...}}``.
    """
    cb = design.get("counterbalance") or {}
    n_lists = int(cb.get("lists", 1))
    seed = schema.get("seed", 1)
    if _recipe(design) == "latin_square_target":
        raise ValueError(
            "lexsync: counterbalance.optimise does not apply to a Latin-square "
            "design. Every item already appears in every list there, so the lists are "
            "balanced on the items by construction; the rotation decides only which "
            "condition each item takes.")
    if n_lists < 2:
        raise ValueError("lexsync: counterbalance.optimise needs "
                         "counterbalance.lists to be 2 or more.")
    dims = _balance_dims(design)
    if not dims:
        raise ValueError(
            "lexsync: counterbalance.optimise has no dimensions to balance. Name them "
            "in counterbalance.balance_on, or give the design a match_on.")
    absent = sorted((d for d in dims if d not in stimuli.columns),
                    key=lambda s: s.encode("utf-8"))
    if absent:
        raise ValueError("lexsync: cannot balance on column(s) the stimuli do not "
                         "have: %s." % ", ".join("'%s'" % a for a in absent))

    sets = sorted(stimuli["set"].unique())
    n_sets = len(sets)
    V = _balance_values(stimuli, dims)
    totals = {d: sum(v) for d, v in V.items()}
    if max((abs(t) for t in totals.values()), default=0) * n_sets > BALANCE_MAX_MAGNITUDE:
        raise ValueError(
            "lexsync: the balance objective would exceed the exact-integer range.")

    # Start from the default deal, so the search improves on the shipped behaviour
    # rather than starting somewhere unrelated to it.
    assign = [(i % n_lists) + 1 for i in range(n_sets)]
    cost0 = _balance_cost(V, assign, n_lists, n_sets)

    pairs = [(i, j) for i in range(n_sets - 1) for j in range(i + 1, n_sets)]
    max_passes = int(cb.get("max_passes", 500))
    n_swaps = 0
    cost = cost0
    hit_bound = False
    for pass_i in range(1, max(0, max_passes) + 1):
        cand = [(i, j) for i, j in pairs if assign[i] != assign[j]]
        if not cand:
            break
        n_in = [sum(1 for a in assign if a == l) for l in range(1, n_lists + 1)]
        S = {d: [sum(v for v, a in zip(V[d], assign, strict=True) if a == l)
                 for l in range(1, n_lists + 1)] for d in dims}
        best = 0
        tied = []
        for i, j in cand:
            la, lb = assign[i], assign[j]
            delta = 0
            for d in dims:
                va, vb = V[d][i], V[d][j]
                Sa, Sb = S[d][la - 1], S[d][lb - 1]
                sha = totals[d] * n_in[la - 1]
                shb = totals[d] * n_in[lb - 1]
                delta += (abs((Sa - va + vb) * n_sets - sha)
                          + abs((Sb - vb + va) * n_sets - shb)
                          - abs(Sa * n_sets - sha)
                          - abs(Sb * n_sets - shb))
            if delta < best:
                best, tied = delta, [(i, j)]
            elif delta == best and tied:
                tied.append((i, j))
        if best >= 0 or not tied:
            break
        if len(tied) > 1:
            # Hash tie-break, not position: taking the first tied pair would
            # systematically prefer low-numbered sets, and the digest is the package's
            # established way of choosing without a generator.
            def _h(pair):
                key = "|".join([_key_part(seed), "balance",
                                _key_part(sets[pair[0]]), _key_part(sets[pair[1]])])
                return hashlib.sha256(key.encode("utf-8")).hexdigest()
            pick = sorted(tied, key=_h)[0]
        else:
            pick = tied[0]
        i, j = pick
        assign[i], assign[j] = assign[j], assign[i]
        cost += best
        n_swaps += 1
        if pass_i == max_passes:
            hit_bound = True

    return {
        "list_of_set": {s: int(a) for s, a in zip(sets, assign, strict=True)},
        "report": {
            "dimensions": list(dims), "cost_before": int(cost0),
            "cost_after": int(cost), "n_swaps": int(n_swaps),
            "max_passes_reached": hit_bound,
            "cost_unit": ("summed absolute deviation of each list's dimension total "
                          "from its fair share, in milli-units of the dimension's "
                          "mean, scaled by the number of item sets"),
        },
    }

lexsync.participant_table(factors, n_participants)

Source code in src/lexsync/counterbalancing.py

def participant_table(factors: dict, n_participants: int) -> pd.DataFrame:
    keys = list(factors.keys())
    # R's expand.grid() varies the first factor fastest, itertools.product the
    # last; cross the reversed keys and unreverse each cell so a participant
    # number is allocated the same cell by either engine.
    grid = [cell[::-1] for cell in product(*[factors[k] for k in reversed(keys)])]
    rows = [dict(zip(keys, grid[i % len(grid)], strict=True)) for i in range(n_participants)]
    df = pd.DataFrame(rows)
    df["participant"] = np.arange(1, n_participants + 1)
    return df

Validation and equivalence

A matched design claims that its controls do not differ, and a non-significant test of difference does not establish that. These functions report the realised control instead: the standardised difference with its interval, an equivalence test against a declared bound, and the variance ratio that a mean-based statistic would miss.

lexsync.match_report(stimuli, dims, schema)

Build the full match-quality report: descriptives and comparisons.

Every comparison is against the first condition in order of appearance, so a design with a single condition has nothing to compare and comparisons comes back with its columns and no rows.

Source code in src/lexsync/validation.py

def match_report(stimuli: pd.DataFrame, dims, schema: dict) -> dict:
    """Build the full match-quality report: descriptives and comparisons.

    Every comparison is against the first condition in order of appearance, so a
    design with a single condition has nothing to compare and ``comparisons`` comes
    back with its columns and no rows.
    """
    conds = list(dict.fromkeys(stimuli["condition"]))
    anchor = conds[0]
    desc = describe_stimuli(stimuli, dims)
    bound = (schema.get("equivalence") or {}).get("bound_d", 0.5)
    alpha = (schema.get("equivalence") or {}).get("alpha", 0.05)
    rows = []
    for cc in conds[1:]:
        for dim in dims:
            x = pd.to_numeric(stimuli.loc[stimuli["condition"] == anchor, dim], errors="coerce")
            y = pd.to_numeric(stimuli.loc[stimuli["condition"] == cc, dim], errors="coerce")
            tt = tost_equiv(x, y, bound, alpha)
            ci = cohens_d_ci(x, y, alpha)
            vr = variance_ratio(y, x)
            p = tt["p"]
            # An undefined d and its interval serialise as missing cells (empty in
            # the CSV, null in the datasheet), exactly like the other missing stats.
            d = cohens_d(x, y)
            lo, hi = ci["ci_low"], ci["ci_high"]
            rows.append(dict(
                condition=cc, reference=anchor, dimension=dim,
                cohens_d=_round_dp(d, 3) if d is not None else None,
                d_ci_low=_round_dp(lo, 3) if lo is not None and lo == lo else None,
                d_ci_high=_round_dp(hi, 3) if hi is not None and hi == hi else None,
                var_ratio=_round_dp(vr, 3) if vr is not None else None,
                tost_p=_round_dp(p, 4) if p == p else None,
                equivalent=tt["equivalent"],
            ))
    return dict(descriptives=desc,
                comparisons=pd.DataFrame(rows) if rows
                else pd.DataFrame(columns=list(_COMPARISON_COLUMNS)))

lexsync.describe_stimuli(stimuli, dims, by='condition')

Source code in src/lexsync/validation.py

def describe_stimuli(stimuli: pd.DataFrame, dims, by: str = "condition") -> pd.DataFrame:
    rows = []
    for g, d in stimuli.groupby(by, sort=False):
        for dim in dims:
            x = pd.to_numeric(d[dim], errors="coerce").dropna()
            rows.append(dict(
                group=g, dimension=dim, n=int(x.size),
                mean=_round_dp(_exact_mean(x), 3), sd=_round_dp(_exact_sd(x), 3),
                min=_round_dp(float(x.min()), 3),
                # _exact_median, not pandas' .median(): the latter reduces through
                # numpy, the one reduction here that would bypass the shared exact
                # primitives.
                median=_round_dp(_exact_median(x), 3),
                max=_round_dp(float(x.max()), 3),
            ))
    return pd.DataFrame(rows)

lexsync.balance_check(stimuli, columns)

Source code in src/lexsync/validation.py

def balance_check(stimuli: pd.DataFrame, columns) -> list:
    if isinstance(columns, str):
        columns = [columns]
    issues = []
    for col in columns:
        if col not in stimuli.columns:
            continue
        tab = stimuli[col].value_counts()
        if tab.nunique() > 1:
            detail = ", ".join(f"{k}={v}" for k, v in tab.items())
            issues.append(f"Column '{col}' is unbalanced: {detail}")
    return issues

lexsync.variance_ratio(cond, ref)

Ratio of a condition's variance to the reference's: a distributional balance check that complements the mean-based Cohen's d and TOST.

Two conditions can share a mean yet differ in spread and still confound, which a mean-based statistic misses (Armstrong, Watson & Plaut, 2012; Austin, 2009). A ratio near 1 is balanced; a common heuristic flags ratios outside about [0.5, 2] as unequal spread. Returns None when a variance is undefined.

Source code in src/lexsync/validation.py

def variance_ratio(cond, ref):
    """Ratio of a condition's variance to the reference's: a distributional
    balance check that complements the mean-based Cohen's d and TOST.

    Two conditions can share a mean yet differ in spread and still confound, which
    a mean-based statistic misses (Armstrong, Watson & Plaut, 2012; Austin, 2009).
    A ratio near 1 is balanced; a common heuristic flags ratios outside about
    [0.5, 2] as unequal spread. Returns ``None`` when a variance is undefined.
    """
    cond = np.asarray(cond, dtype=float); ref = np.asarray(ref, dtype=float)
    cond = cond[~np.isnan(cond)]; ref = ref[~np.isnan(ref)]
    if len(cond) < 2 or len(ref) < 2:
        return None
    v_ref = _exact_var(ref)
    if v_ref == 0:
        return 1.0 if _exact_var(cond) == 0 else None
    return float(_exact_var(cond) / v_ref)

lexsync.cohens_d(x, y)

Source code in src/lexsync/validation.py

def cohens_d(x, y):
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        return 0.0
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    if sp == 0 or math.isnan(sp):
        # Two constants: an exactly-zero difference is exactly zero SDs apart, but
        # unequal constants are infinitely many. That is undefined, not perfect balance.
        if float(_exact_mean(x) - _exact_mean(y)) == 0:
            return 0.0
        return None
    return float((_exact_mean(x) - _exact_mean(y)) / sp)

lexsync.cohens_d_ci(x, y, alpha=0.05)

Cohen's d with a confidence interval, complementing the TOST verdict.

The interval is the (1 - 2 * alpha) confidence interval for the standardised mean difference; for alpha = 0.05 this is the 90% interval that corresponds exactly to a two one-sided tests (TOST) decision at the .05 level (Lakens, 2017). Reporting the interval, rather than only a binary "equivalent / not" verdict, makes the realised imbalance and its sampling uncertainty explicit and keeps the dependence on the number of items visible rather than hidden: with few items the interval is wide, so a small point estimate cannot be over-read as evidence of a small true difference (Sassenhagen & Alday, 2016). The upper limit of the interval on |d| is the largest imbalance still consistent with the stimuli.

Source code in src/lexsync/validation.py

def cohens_d_ci(x, y, alpha: float = 0.05) -> dict:
    """Cohen's *d* with a confidence interval, complementing the TOST verdict.

    The interval is the ``(1 - 2 * alpha)`` confidence interval for the
    standardised mean difference; for ``alpha = 0.05`` this is the 90% interval
    that corresponds exactly to a two one-sided tests (TOST) decision at the .05
    level (Lakens, 2017). Reporting the interval, rather than only a binary
    "equivalent / not" verdict, makes the realised imbalance and its sampling
    uncertainty explicit and keeps the dependence on the number of items visible
    rather than hidden: with few items the interval is wide, so a small point
    estimate cannot be over-read as evidence of a small true difference
    (Sassenhagen & Alday, 2016). The upper limit of the interval on ``|d|`` is the
    largest imbalance still consistent with the stimuli.
    """
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        return dict(d=0.0, ci_low=float("nan"), ci_high=float("nan"))
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    diff = float(_exact_mean(x) - _exact_mean(y))
    if sp == 0 or math.isnan(sp):
        # Equal constants carry no sampling uncertainty: a point at zero. Unequal
        # constants are infinitely many SDs apart, so the estimate and its interval
        # are undefined, not a perfect [0, 0].
        if diff == 0:
            return dict(d=0.0, ci_low=0.0, ci_high=0.0)
        return dict(d=None, ci_low=None, ci_high=None)
    d = diff / sp
    margin = float(stats.t.ppf(1 - alpha, nx + ny - 2) * math.sqrt(1 / nx + 1 / ny))
    return dict(d=float(d), ci_low=d - margin, ci_high=d + margin)

lexsync.tost_equiv(x, y, bound_d=0.5, alpha=0.05)

Source code in src/lexsync/validation.py

def tost_equiv(x, y, bound_d: float = 0.5, alpha: float = 0.05) -> dict:
    x = np.asarray(x, dtype=float); y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]; y = y[~np.isnan(y)]
    nx, ny = len(x), len(y)
    if nx < 2 or ny < 2:
        return dict(p=float("nan"), equivalent=None)
    sp = math.sqrt(((nx - 1) * _exact_var(x) + (ny - 1) * _exact_var(y)) / (nx + ny - 2))
    if sp == 0 or math.isnan(sp):
        # Both conditions are constants (e.g. a dimension fixed by the pool, such
        # as two-character Chinese words). They are equivalent iff they share that
        # constant; the standardised difference is then exactly zero.
        if float(_exact_mean(x) - _exact_mean(y)) == 0:
            return dict(p=0.0, equivalent=True)
        return dict(p=1.0, equivalent=False)
    se = sp * math.sqrt(1 / nx + 1 / ny)
    if se == 0 or math.isnan(se):
        return dict(p=float("nan"), equivalent=None)
    bound = bound_d * sp
    dfree = nx + ny - 2
    diff = _exact_mean(x) - _exact_mean(y)
    p = max(stats.t.sf((diff + bound) / se, dfree), stats.t.cdf((diff - bound) / se, dfree))
    return dict(p=float(p), equivalent=bool(p < alpha))

Experiment generation

All three targets are rendered from the same event list. Generation imports neither PsychoPy nor pyserial, so it needs no laboratory hardware. The experiment extra is for running the result. assign_triggers is reached as lexsync.scripting.assign_triggers, and export_experiments calls it for you.

lexsync.export_experiments(stimuli, design, schema, outdir, base=None)

Source code in src/lexsync/scripting.py

def export_experiments(stimuli, design, schema, outdir, base=None) -> dict:
    stimuli = assign_triggers(stimuli)
    return {
        "psychopy": export_psychopy(stimuli, design, schema, outdir, base),
        "opensesame": export_opensesame(stimuli, design, schema, outdir, base),

lexsync.export_psychopy(stimuli, design, schema, outdir, base=None)

Source code in src/lexsync/scripting.py

def export_psychopy(stimuli, design, schema, outdir, base=None) -> str:
    base = base or slugify(design["name"], design["language"])
    events = resolve_events(design)
    rendered = render_events(events, design.get("timing") or {}, _refresh_hz(schema))
    csv_name = f"{base}_psychopy.csv"
    write_csv_utf8(loop_table(stimuli, events), os.path.join(outdir, csv_name))
    with open(find_template("psychopy/trial_runner_template.py"), encoding="utf-8") as handle:
        tmpl = handle.read()
    triggers = schema.get("triggers") or {}
    presentation = schema.get("presentation") or {}
    # These land in CODE positions -- a module docstring, a bare assignment, a string
    # literal, so they are validated on the way in. Escaping would need three
    # different rules for three targets in two engines. See clean_meta in io_utils.
    subs = {
        "DESIGN": clean_meta(design["name"], "the design's `name`"),
        "LANGUAGE": clean_meta(design["language"], "the design's `language`"),
        "CONDITIONS_FILE": csv_name,
        "TRIGGER_ADDRESS": clean_port(triggers.get("parallel_address", "0x0378")),
        "TRIGGER_HOLD_MS": "%.17g" % _trigger_hold_ms(schema),
        # Through %.17g like its neighbours, never default stringification: str()
        # and R's as.character() part company on a non-integer quotient
        # (str(16.65 / 1000) gives "0.016649999999999998", as.character gives
        # "0.01665"), and this value lands in the generated script.
        "INTER_TRIGGER_S": "%.17g" % (triggers.get("inter_trigger_ms", 10) / 1000),
        "WORD_FONT": clean_meta(
            design.get("font") or presentation.get("font") or "Courier New", "the font"),
        "FULLSCREEN": "False",
        # The fallback used only when the script cannot measure the display.
        "ASSUMED_REFRESH_HZ": "%.17g" % _refresh_hz(schema),
        "EVENTS_JSON": _json_r(rendered),
    }
    for key, value in subs.items():
        tmpl = tmpl.replace("{{" + key + "}}", str(value))
    return _write_text(tmpl.rstrip("\n"), os.path.join(outdir, f"{base}_psychopy.py"))

lexsync.export_opensesame(stimuli, design, schema, outdir, base=None)

Source code in src/lexsync/scripting.py

def export_opensesame(stimuli, design, schema, outdir, base=None) -> str:
    base = base or slugify(design["name"], design["language"])
    events = resolve_events(design)
    rendered = render_events(events, design.get("timing") or {}, _refresh_hz(schema))
    csv_name = f"{base}_opensesame.csv"
    write_csv_utf8(loop_table(stimuli, events), os.path.join(outdir, csv_name))
    presentation = schema.get("presentation") or {}
    font = design.get("font") or presentation.get("opensesame_font") or "mono"
    text = build_osexp(design, csv_name, schema, rendered, font=font)
    return _write_text(text, os.path.join(outdir, f"{base}.osexp"))

lexsync.export_jspsych(stimuli, design, schema, outdir, base=None)

A browser-runnable jsPsych experiment from the event list.

The same rendered events and the trial data are embedded in one HTML file, so anyone can reproduce the exact procedure online from the same materials. The jsPsych library and stylesheet are loaded from a CDN, so the machine running the file needs an internet connection; the trial data are embedded and the responses are saved locally, so no server is required either to run it or to collect them. Onset triggers are recorded in each trial's data (a browser cannot drive a parallel port); online EEG synchronisation needs WebSerial/LSL or a photodiode.

Source code in src/lexsync/scripting.py

def export_jspsych(stimuli, design, schema, outdir, base=None) -> str:
    """A browser-runnable jsPsych experiment from the event list.

    The same rendered events and the trial data are embedded in one HTML file, so
    anyone can reproduce the exact procedure online from the same materials. The
    jsPsych library and stylesheet are loaded from a CDN, so the machine running the
    file needs an internet connection; the trial data are embedded and the responses
    are saved locally, so no server is required either to run it or to collect them.
    Onset triggers are recorded in each trial's data (a browser cannot drive a
    parallel port); online EEG synchronisation needs WebSerial/LSL or a photodiode.
    """
    base = base or slugify(design["name"], design["language"])
    events = resolve_events(design)
    rendered = _map_keys_for_jspsych(
        render_events(events, design.get("timing") or {}, _refresh_hz(schema)))
    trials = loop_table(stimuli, events).to_dict(orient="records")
    presentation = schema.get("presentation") or {}
    font = design.get("font") or presentation.get("font") or "Courier New"
    with open(find_template("jspsych/experiment_template.html"), encoding="utf-8") as handle:
        tmpl = handle.read()
    # DESIGN lands in <title> and a comment, LANGUAGE_TAG in a lang attribute, WORD_FONT
    # inside a CSS font-family declaration. Only the two JSON blobs are escaped by
    # _json_html; the rest are validated, since a quote or an angle bracket in any of
    # them would leave its attribute or rule and start markup in a page that collects
    # participant responses.
    subs = {
        "DESIGN": clean_meta(design["name"], "the design's `name`"),
        "LANGUAGE": clean_meta(design["language"], "the design's `language`"),
        "LANGUAGE_TAG": _language_tag(design), "WORD_FONT": clean_meta(font, "the font"),
        "EVENTS_JSON": _json_html(rendered), "TRIALS_JSON": _json_html(trials),
    }
    for k, v in subs.items():

lexsync.scripting.assign_triggers(stimuli)

A per-condition marker and a per-item marker, both 0-255 EEG codes.

The item range holds 200 codes (an 8-bit-port constraint), so past 200 sets the codes wrap and repeat, and a runtime notice says so.

Source code in src/lexsync/scripting.py

def assign_triggers(stimuli: pd.DataFrame) -> pd.DataFrame:
    """A per-condition marker and a per-item marker, both 0-255 EEG codes.

    The item range holds 200 codes (an 8-bit-port constraint), so past 200 sets
    the codes wrap and repeat, and a runtime notice says so.
    """
    stimuli = stimuli.copy()
    conds = list(dict.fromkeys(stimuli["condition"]))
    stimuli["condition_trigger"] = stimuli["condition"].map(lambda c: 101 + conds.index(c))
    sets = sorted(stimuli["set"].unique(), key=lambda s: str(s))
    # A wrapped code no longer identifies its item one to one, which the analyst
    # must hear about at generation time, not at decode time.
    if len(sets) > 200:
        print(f"lexsync: {len(sets)} item sets exceed the 200-code trigger range; "
              "item codes wrap and repeat.")
    set_code = {s: 40 + (i % 200) for i, s in enumerate(sets)}
    stimuli["item_trigger"] = stimuli["set"].map(set_code)
    return stimuli

Materials datasheet

The datasheet is the provenance record that travels with a stimulus set: where the items came from, how they were selected, the realised control, the candidate-pool sizes, the checksums and the versions. run_pipeline builds and writes one for every run.

lexsync.build_datasheet(design, schema, report, stimuli, source_path, artifacts, seed, engine='python', candidate_pool=None, norms=None, balance=None, blocks=None, design_path=None, schema_path=None, selection_audit=None, neighbourhood_reference=None)

Assemble the datasheet dictionary from the pipeline's objects.

candidate_pool (optional) is a list of {"condition", "n_candidates"} recording how many items satisfied each condition's window before matching, the size of the discretionary pool the selection drew from, reported so that item-selection bias is auditable (Forster, 2000; Simmons et al., 2011).

norms (optional) is a list of norm-table provenance records from the design's norms: block. Each names a file, its sha256, the join key and the per-column coverage. Recorded because a norm table can supply the very variable a design manipulates, so a record that omitted it would describe a selection over columns of unstated origin.

balance (optional) is the balance-optimiser report from balance_lists. Recorded because it decides which items each participant sees.

design_path / schema_path (optional) are the design and schema files the run read; when given, their sha256 checksums complete the reproducibility record, because those two files decide everything the seed does not.

selection_audit (optional) is the matcher's audit record; its window_relaxations entries are recorded because a relaxed window changes what "matched" means for that condition.

neighbourhood_reference (optional) records the lexicon the neighbourhood dimensions were computed against ({"source", "n_words", "sha256"}), verbatim.

Source code in src/lexsync/datasheet.py

def build_datasheet(design, schema, report, stimuli, source_path, artifacts,
                    seed, engine="python", candidate_pool=None, norms=None,
                    balance=None, blocks=None, design_path=None, schema_path=None,
                    selection_audit=None, neighbourhood_reference=None) -> dict:
    """Assemble the datasheet dictionary from the pipeline's objects.

    ``candidate_pool`` (optional) is a list of ``{"condition", "n_candidates"}``
    recording how many items satisfied each condition's window before matching,
    the size of the discretionary pool the selection drew from, reported so that
    item-selection bias is auditable (Forster, 2000; Simmons et al., 2011).

    ``norms`` (optional) is a list of norm-table provenance records from the design's
    ``norms:`` block. Each names a file, its sha256, the join key and the per-column
    coverage. Recorded because a norm table can supply the very variable a design
    manipulates, so a record that omitted it would describe a selection over columns
    of unstated origin.

    ``balance`` (optional) is the balance-optimiser report from ``balance_lists``.
    Recorded because it decides which items each participant sees.

    ``design_path`` / ``schema_path`` (optional) are the design and schema files the
    run read; when given, their sha256 checksums complete the reproducibility
    record, because those two files decide everything the seed does not.

    ``selection_audit`` (optional) is the matcher's audit record; its
    ``window_relaxations`` entries are recorded because a relaxed window changes
    what "matched" means for that condition.

    ``neighbourhood_reference`` (optional) records the lexicon the neighbourhood
    dimensions were computed against (``{"source", "n_words", "sha256"}``),
    verbatim.
    """
    source = (design.get("items") or {}).get("source", "corpus")
    is_continuous = _is_continuous(design)
    controlled = _controlled(design, source)
    relational = _relational_record(design, stimuli)
    conditions = list(dict.fromkeys(stimuli["condition"]))

    realised = []
    if report is not None:
        for _, r in report["comparisons"].iterrows():
            if is_continuous:
                realised.append({
                    "dimension": r["dimension"], "role": r["role"],
                    "pearson_r": _num(r.get("pearson_r")),
                    "predictor_span": _num(r.get("predictor_span")),
                })
            else:
                realised.append({
                    "dimension": r["dimension"],
                    "role": "controlled" if r["dimension"] in controlled else "manipulated/free",
                    "cohens_d": _num(r["cohens_d"]),
                    "ci_low": _num(r.get("d_ci_low")), "ci_high": _num(r.get("d_ci_high")),
                    "var_ratio": _num(r.get("var_ratio")),
                    "tost_p": _num(r.get("tost_p")), "equivalent": _bool(r.get("equivalent")),
                })

    if is_continuous:
        # The controls are banded by the same tolerance windows the matcher uses, so
        # the record states them here too; without them the banding is unreproducible.
        selection = {"method": "continuous even-spread (predictor spanned, controls banded)",
                     "predictor": design["continuous"]["predictor"],
                     "controls": list(design["continuous"].get("controls") or []),
                     "tolerance_k": _resolve_tolerance_k(design, schema)}
    elif source in ("corpus", "pool"):
        method = ((design.get("matching") or {}).get("method")
                  or (schema.get("matching") or {}).get("method")
                  or "standardised_euclidean")
        selection = {"method": method, "match_on": controlled}
        if method in ("joint", "optimal"):
            # The pairwise methods rank whole pairs and never consult the tolerance
            # windows; recording tolerance_k here would claim a filter that was not
            # applied. They get the cap they do apply instead, with a per-condition
            # verdict on whether it fired.
            selection["candidate_cap"] = {
                "cap": int(PAIRWISE_CAP),
                "applied": {c["condition"]: bool(c["n_candidates"] > PAIRWISE_CAP)
                            for c in (candidate_pool or [])
                            if c.get("condition") is not None
                            and c.get("n_candidates") is not None},
            }
        else:
            selection["tolerance_k"] = _resolve_tolerance_k(design, schema)
    elif source == "generate":
        gen_method = ((design.get("items") or {}).get("generation") or {}).get(
            "method", "letter_substitution")
        selection = {"method": _GENERATION_LABELS.get(
            gen_method, f"{gen_method} (deterministic pseudowords)"),
            "generation_method": gen_method,
            "matched_on": ["length"]}
    else:
        selection = {"method": "item table (user-supplied)"}
    if candidate_pool is not None and source in ("corpus", "pool", "generate"):
        selection["candidate_pool"] = candidate_pool
    # A pair-keyed continuous design does select, over the item table.
    selection["cross_engine"] = _cross_engine(
        selection.get("method"), source, selected=is_continuous and relational is not None)
    # A relaxed window changes what "matched" means for that condition, so the
    # matcher's audit trail belongs in the record, not only in the run narration.
    # Integers only, so both engines serialise the counts identically.
    relaxations = (selection_audit or {}).get("window_relaxations") or []
    if relaxations:
        selection["window_relaxations"] = [
            {"condition": r["condition"],
             "n_within_tolerance": int(r["n_within_tolerance"]),
             "n_needed": int(r["n_needed"])} for r in relaxations]
    if neighbourhood_reference is not None:
        selection["neighbourhood_reference"] = neighbourhood_reference

    if source in ("corpus", "generate"):
        provenance = ("wordfreq (Speer, 2022), data CC BY-SA 4.0; full corpus licence "
                      "and citation at https://github.com/pablobernabeu/lexsync/blob/"
                      "main/corpora/ATTRIBUTION.md")
    elif source == "pool":
        provenance = "user-supplied word pool, matched by lexsync"
    else:
        provenance = "user-supplied item table"
    materials_source = {
        "type": source, "path": _posix(source_path), "sha256": sha256_file(source_path),
        "provenance": provenance,
    }
    # A supplied pool usually draws its dimensions from a corpus lexicon, and that
    # lexicon is where every matched value came from, so it is named and checksummed
    # here: `path` above records only the word list itself.
    if source == "pool":
        dim_lex = (design.get("items") or {}).get("lexicon") or design.get("lexicon")
        materials_source["dimensions_from"] = (
            dim_lex if dim_lex else "the supplied pool's own columns (no lexicon given)")
        if dim_lex:
            materials_source["dimensions_sha256"] = sha256_file(dim_lex)
    # Added only when present, so a design with no `norms:` block gets no key at all
    # rather than a "norms": null that every datasheet would then carry. Matches the R
    # engine, where assigning NULL to a list element removes it.
    if norms:
        materials_source["norms"] = norms

    # A corpus design draws on every schema dimension, and so does a pair-keyed
    # design: every lexicon dimension is joined onto each member. A generate or plain
    # table design reports only the ones it controlled, so the record does not claim
    # dimensions that played no part in the selection.
    all_dims = source in ("corpus", "pool") or relational is not None
    # `or {}`, not a default argument, for the reason load_lexicon carries the same
    # guard: an empty `dimensions:` key parses to None here, which is not iterable,
    # while the R twin's schema$dimensions[keep] returns NULL and the record simply
    # carries no dimensions.
    schema_dims = schema.get("dimensions") or {}
    dims = {d: schema_dims[d] for d in schema_dims if all_dims or d in controlled}

    # The balance report is added only when the optimiser ran, for the same reason the
    # norms record is: a key that is null on every design that does not use the feature
    # is noise in a research artefact.
    counterbalancing = {
        "recipe": "latin_square_target" if source == "table" else "factorial",
        "lists": (design.get("counterbalance") or {}).get("lists", 1),
    }
    if balance:
        counterbalancing["optimise"] = balance
    # Practice and filler trials change what a participant sees but not what is
    # analysed, so the record has to state both counts: a reader comparing the stimuli
    # file against the experiment would otherwise find them a different length with no
    # explanation.
    if blocks:
        counterbalancing["blocks"] = blocks

    # The equivalence settings the report's TOST verdicts were computed against,
    # recorded so the Methods prose can state the bound it actually ran with.
    equivalence = {"bound_d": (schema.get("equivalence") or {}).get("bound_d", 0.5),
                   "alpha": (schema.get("equivalence") or {}).get("alpha", 0.05)}

    reproducibility = {"seed": seed, "versions": _versions(engine)}
    # The design and schema decide everything the seed does not, so their checksums
    # complete the reproducibility record when the pipeline names them.
    if design_path is not None:
        reproducibility["design_sha256"] = sha256_file(design_path)
    if schema_path is not None:
        reproducibility["schema_sha256"] = sha256_file(schema_path)

    return {
        "lexsync_datasheet_version": DATASHEET_VERSION,
        "design": {
            "name": design["name"], "language": design["language"],
            "paradigm": design.get("paradigm", "factorial"), "source": source,
            "description": design.get("description"),
            "n_per_condition": design.get("n_per_condition") or design.get("n_per_cell"),
        },
        "materials_source": materials_source,
        "dimensions": dims,
        "selection": selection,
        "relational": relational,
        "analysis": _analysis(design, source),
        "equivalence": equivalence,
        "realised_control": realised,
        "counterbalancing": counterbalancing,
        "resampling": ({"n_sets": (design.get("resample") or {}).get("n_sets"),
                        "disjoint": True} if design.get("resample") else None),
        "items": {
            "n_total": int(len(stimuli)), "n_conditions": len(conditions),
            "conditions": conditions,
            "stimuli_file": _posix(artifacts.get("stimuli")),
            "stimuli_sha256": sha256_file(artifacts.get("stimuli")),
        },
        "reproducibility": reproducibility,
        "artifacts": [{"file": _posix(p), "sha256": sha256_file(p)}
                      for p in _artifact_paths(artifacts) if p],
    }

lexsync.write_datasheet(ds, json_path, md_path)

Source code in src/lexsync/datasheet.py

def write_datasheet(ds: dict, json_path: str, md_path: str) -> tuple:
    import os
    os.makedirs(os.path.dirname(json_path) or ".", exist_ok=True)
    with open(json_path, "w", encoding="utf-8", newline="\n") as handle:
        json.dump(_at_15_significant_digits(ds), handle, indent=2, ensure_ascii=False,
                  sort_keys=True)
        handle.write("\n")
    with open(md_path, "w", encoding="utf-8", newline="\n") as handle:
        handle.write(render_datasheet_md(ds) + "\n")
    return json_path, md_path

lexsync.methods_paragraph(ds)

Source code in src/lexsync/datasheet.py

def methods_paragraph(ds: dict) -> str:
    d = ds["design"]
    src = ds["materials_source"]["type"]
    n = d["n_per_condition"]
    lang = d["language"].capitalize()
    sel = ds.get("selection") or {}
    # What was selected. A pair design's unit is the pair, and its `n_per_condition`
    # counts pairs, so calling them "items" would misreport the size of the materials
    # by a factor of the number of conditions.
    rel = ds.get("relational")
    unit = "items" if not rel else "-".join(rel["members"]) + " pairs"
    if "predictor" in sel:
        predictor = sel["predictor"]
        controls = ", ".join(sel.get("controls") or []) or "the control dimensions"
        rc = ds.get("realised_control") or []
        span = next((r.get("predictor_span") for r in rc if r.get("predictor_span") is not None), None)
        rs = [abs(r["pearson_r"]) for r in rc
              if r.get("pearson_r") is not None and r["pearson_r"] == r["pearson_r"]]
        span_str = f" (a span of {span:.2f})" if span is not None else ""
        # Report |r| at 3 dp (its stored precision) so the text is identical across
        # engines; a 2-dp format of, say, 0.165 rounds to 0.17 in Python and 0.16 in R.
        corr_str = (f", and the largest predictor-control correlation was |r| = {max(rs):.3f}"
                    if rs else "")
        cb = ds["counterbalancing"]
        recipe_label = {"latin_square_target": "a Latin-square rotation",
                        "factorial": "a factorial split"}.get(cb["recipe"], cb["recipe"])
        return (f"{n} {lang} {unit} were selected to span {predictor}{span_str} continuously "
                f"while holding {controls} near-constant{corr_str}, for analysis by regression "
                f"or a mixed model rather than a between-condition contrast (Kuperman, 2015; "
                f"Liben-Nowell et al., 2019).{_norms_note(ds)} Materials were counterbalanced "
                f"into {cb['lists']} list(s) ({recipe_label}) and generated for PsychoPy, "
                f"OpenSesame and jsPsych. The selection is deterministic and reproducible "
                f"(seed {ds['reproducibility']['seed']}; lexsync "
                f"{ds['reproducibility']['versions']['lexsync']}).")
    if src == "corpus":
        ctrl = ", ".join(ds["selection"]["match_on"]) or "the control dimensions"
        lead = (f"{n} items per condition were selected from the {lang} lexicon "
                f"({ds['materials_source']['provenance']}) and matched item by item on "
                f"{ctrl} using lexsync's {ds['selection']['method']} matcher")
    elif src == "pool":
        # A supplied pool is matched exactly as a corpus is; what differs, and what the
        # Methods section has to say, is that the candidate words were chosen by the
        # researcher rather than drawn from the whole lexicon.
        ctrl = ", ".join(ds["selection"]["match_on"]) or "the control dimensions"
        lead = (f"{n} {lang} items per condition were selected from a supplied candidate "
                f"pool and matched item by item on {ctrl} using lexsync's "
                f"{ds['selection']['method']} matcher, with the matched dimensions taken "
                f"from {ds['materials_source']['dimensions_from']}")
    elif src == "generate":
        lead = (f"{n} real {lang} words and {n} length-matched pseudowords "
                f"(generated by {ds['selection']['method']}) were assembled for a "
                f"lexical-decision contrast")
    else:
        lead = (f"{ds['items']['n_total'] // max(1, ds['items']['n_conditions'])} items "
                f"were drawn from an item table for a {d['paradigm']} design ({lang})")
    ctrl_rows = [r for r in ds["realised_control"] if r["role"] == "controlled"]
    defined = [r for r in ctrl_rows if r["cohens_d"] is not None]
    control = ""
    if ctrl_rows:
        # Affirmative only when the stored verdicts support it: every controlled row
        # has a defined d and every one of them passed the TOST. An undefined d
        # (constant dimensions at different constants) is the worst possible failure
        # of the matching, not an excludable row.
        all_equivalent = (len(defined) == len(ctrl_rows)
                          and all(r.get("equivalent") is True for r in defined))
        if all_equivalent and defined:
            worst = max(defined, key=lambda r: abs(r["cohens_d"]))
            control = (f". The realised control was close. The largest standardised difference on "
                       f"any matched dimension was {worst['cohens_d']:.2f} "
                       f"(90% CI [{worst['ci_low']:.2f}, {worst['ci_high']:.2f}]), within the "
                       f"{ds['equivalence']['bound_d']}-SD equivalence bound")
        else:
            control = (". Equivalence was not confirmed on every matched dimension; "
                       "the per-dimension differences are reported in the "
                       "realised-control table")
    rs = ds.get("resampling")
    resamp = (f". {rs['n_sets']} disjoint matched item sets were drawn, so items can be "
              f"treated as a random factor" if rs else "")
    cb = ds["counterbalancing"]
    recipe_label = {"latin_square_target": "a Latin-square rotation",
                    "factorial": "a factorial split"}.get(cb["recipe"], cb["recipe"])
    tail = (resamp + f". Materials were counterbalanced into {cb['lists']} list(s) "
            f"({recipe_label}) and generated for PsychoPy, OpenSesame and jsPsych. The "
            f"selection is deterministic and reproducible (seed "
            f"{ds['reproducibility']['seed']}; lexsync "
            f"{ds['reproducibility']['versions']['lexsync']}).")
    cp = (ds.get("selection") or {}).get("candidate_pool")
    pool_note = ""
    if cp:
        sizes = [c["n_candidates"] for c in cp if c.get("n_candidates") is not None]
        if sizes:
            pool_note = (f". The smallest condition was selected from {min(sizes)} "
                         "eligible candidates, and the selection was deterministic and "
                         "blind to any outcome measure")
    cap_rec = (ds.get("selection") or {}).get("candidate_cap")
    cap_note = ""
    if cap_rec and any(cap_rec.get("applied", {}).values()):
        cap_note = (f". Each candidate pool exceeding the pairwise cap was reduced to "
                    f"the {int(cap_rec['cap'])} candidates nearest the other "
                    f"condition's centroid before pairing")
    ce_note = ""
    if str((ds.get("selection") or {}).get("cross_engine", "")).startswith("approximate"):
        ce_note = (". This design's matching method uses a covariance inverse or an "
                   "assignment solver, so the R and Python engines select equivalent "
                   "but not byte-identical materials")
    bal = (ds.get("counterbalancing") or {}).get("optimise")
    bal_note = "" if not bal else (
        ". Item sets were assigned to lists so as to equate the lists on %s rather than "
        "by an arbitrary deal, by a deterministic integer search (%d swap(s); imbalance "
        "reduced from %d to %d)"
        % (", ".join(bal["dimensions"]), bal["n_swaps"],
           bal["cost_before"], bal["cost_after"]))
    return lead + control + pool_note + cap_note + ce_note + bal_note + tail + _norms_note(ds)

Pipeline and logging

run_pipeline is the orchestrator behind lexsync run, and run_all loops it over a directory of designs. The logging functions are reached as lexsync.logging.*, which does not shadow the standard library's logging for absolute imports.

lexsync.run_pipeline

The orchestrator. Mirrors R_workflow/R/run_pipeline.R.

For each design it obtains stimuli from the configured item source (a corpus matched on lexical dimensions, generated pseudowords for lexical decision, or an item table for priming and self-paced reading), then counterbalances, writes any match-quality report and the run log, and exports the PsychoPy and OpenSesame scripts from the design's trial-event sequence. run_all loops over every design.

lexsync.run_all(config_dir='config', schema_path=None, outdir='output', verbose=True)

Source code in src/lexsync/run_pipeline.py

def run_all(config_dir="config", schema_path=None, outdir="output", verbose=True) -> dict:
    schema_path = schema_path or os.path.join(config_dir, "schema.yaml")
    designs = sorted(glob.glob(os.path.join(config_dir, "design_*.yaml")) +
                     glob.glob(os.path.join(config_dir, "design_*.yml")))
    if not designs:
        raise FileNotFoundError(f"lexsync: no design_*.yaml files in '{config_dir}'.")
    results = {}
    for design in designs:
        if verbose:
            print(f"\n=== lexsync: design '{os.path.basename(design)}' ===")
        results[os.path.basename(design)] = run_pipeline(design, schema_path, outdir, verbose=verbose)
    if verbose:
        print(f"\n[lexsync] all {len(designs)} designs complete.")
    return results

lexsync.logging.new_run_log(name, meta=None)

Source code in src/lexsync/logging.py

def new_run_log(name: str, meta: dict | None = None) -> dict:
    return {
        "name": name,
        "started": _now(),
        "engine": f"Python {platform.python_version()}",
        "meta": meta or {},
        "steps": [],
    }

lexsync.logging.log_step(log, message, data=None)

Source code in src/lexsync/logging.py

def log_step(log: dict, message: str, data: dict | None = None) -> dict:
    log["steps"].append({"time": _now(), "message": message, "data": data})
    if _VERBOSE:
        print(f"[lexsync] {message}")
    return log

lexsync.logging.log_artefact(log, path, rows=None)

Source code in src/lexsync/logging.py

def log_artefact(log: dict, path: str, rows=None) -> dict:
    return log_step(log, f"wrote '{os.path.basename(path)}'",
                    {"path": path, "rows": rows, "md5": hash_file(path)})

lexsync.logging.write_run_log(log, md_path, jsonl_path=None)

Source code in src/lexsync/logging.py

def write_run_log(log: dict, md_path: str, jsonl_path: str | None = None) -> str:
    os.makedirs(os.path.dirname(md_path) or ".", exist_ok=True)
    lines = [
        f"# lexsync run log: {log['name']}", "",
        f"- Engine: {log['engine']}",
        f"- Started: {log['started']}",
        f"- Finished: {_now()}",
    ]
    if log["meta"]:
        lines += ["", "## Run metadata", ""]
        for key, value in log["meta"].items():
            lines.append(f"- {key}: {value}")
    lines += ["", "## Steps", ""]
    for step in log["steps"]:
        lines.append(f"- **{step['time']}**: {step['message']}")
        if step["data"]:
            for key, value in step["data"].items():
                lines.append(f"    - {key}: {value}")
    with open(md_path, "w", encoding="utf-8", newline="\n") as handle:
        handle.write("\n".join(lines) + "\n")
    if jsonl_path:
        with open(jsonl_path, "w", encoding="utf-8", newline="\n") as handle:
            for step in log["steps"]:
                handle.write(json.dumps(step, ensure_ascii=False) + "\n")
    return md_path