Skip to content

The literature layer

A theory does not stand on its own. Before claiming that a construct or proposition adds something new, an author needs a view of the field it enters: which questions are already crowded, which fronts are thin, and where the proposed theory overlaps with existing accounts. The literature layer in theoryforge provides that view from a corpus of records, using deterministic computations so that the same corpus always yields the same map.

This layer comprises four entry points. tf.read_corpus loads a corpus from disk. tf.litmap summarises the corpus through keyword co-occurrence, themes and co-citation. Theory.landscape positions a theory and its registered alternatives against those themes. tf.lit_diagram exports any of these structures as Graphviz DOT. A fifth function, tf.fetch_corpus, builds a corpus from the OpenAlex API. It is optional and network-dependent, and is described at the end.

A corpus

A corpus is a mapping with a records list. Each record may carry an id, a title, a year, a list of keywords and a list of references. The keyword and reference lists drive the analysis. Other fields are carried through but not required. A small corpus stored as YAML looks like this. The examples on this page all read the bundled fixture fixtures/panic-corpus.yaml, of which the first and the third of its eight records appear below, so the counts in the output are those of the whole file rather than of this excerpt.

schema_version: "1.0"
id: "panic-corpus-demo"
records:
  - id: r1
    title: "Interoceptive accuracy and panic"
    year: 2018
    keywords: ["arousal", "interoception"]
    references: ["clark1986", "barlow2002"]
  # ... r2 omitted here, and r4 to r8 below
  - id: r3
    title: "Catastrophic cognitions in panic"
    year: 2017
    keywords: ["appraisal", "catastrophic misinterpretation"]
    references: ["clark1986", "barlow2002"]

Read it with tf.read_corpus, which accepts YAML or JSON and returns a plain dictionary. The examples below read the repository's sample corpus and theory from the fixture directory named fixtures, as on the Getting started page, and every result shown is produced by running the code when this page is built.

import theoryforge as tf

corpus = tf.read_corpus(fixtures / "panic-corpus.yaml")

Mapping the field with litmap

tf.litmap reduces a corpus to a set of deterministic summaries. It counts how often pairs of keywords appear together across records, retains the pairs that meet a minimum link count, and groups the resulting keyword network into themes by connected component. It applies the same procedure to the references field to produce a co-citation network.

m = tf.litmap(corpus)

print("records read:", m["n_records"])
print("keywords:", m["keywords"])
print("keyword co-occurrence:", m["keyword_cooccurrence"])
print("themes:", m["themes"])
print("co-citation:", m["co_citation"])

records read: 8
keywords: ['appraisal', 'arousal', 'avoidance', 'catastrophic misinterpretation', 'exposure', 'genetics', 'heritability', 'interoception']
keyword co-occurrence: [{'a': 'appraisal', 'b': 'catastrophic misinterpretation', 'count': 2}, {'a': 'arousal', 'b': 'interoception', 'count': 2}, {'a': 'avoidance', 'b': 'exposure', 'count': 2}, {'a': 'genetics', 'b': 'heritability', 'count': 2}]
themes: [{'id': 'theme_1', 'keywords': ['appraisal', 'catastrophic misinterpretation'], 'size': 2}, {'id': 'theme_2', 'keywords': ['arousal', 'interoception'], 'size': 2}, {'id': 'theme_3', 'keywords': ['avoidance', 'exposure'], 'size': 2}, {'id': 'theme_4', 'keywords': ['genetics', 'heritability'], 'size': 2}]
co-citation: [{'a': 'barlow2002', 'b': 'clark1986', 'count': 3}, {'a': 'bouton2001', 'b': 'craske2008', 'count': 2}]

Each theme is a dictionary with an id (for example theme_1), the sorted keywords it contains, and its size. Themes are ordered by their smallest keyword, so the output is stable across runs.

The threshold for retaining an edge defaults to two co-occurrences. A pair of keywords or references that appear together only once is dropped, which keeps incidental overlaps out of the map. Lower the threshold to include sparser links, or raise it to keep only the strongest. The count of retained keyword edges shows what each threshold does to this small corpus.

m_sparse = tf.litmap(corpus, min_link=1)   # keep single co-occurrences
m_strict = tf.litmap(corpus, min_link=3)   # keep only frequent pairs

print("edges at min_link=1:", len(m_sparse["keyword_cooccurrence"]))
print("edges at min_link=3:", len(m_strict["keyword_cooccurrence"]))

edges at min_link=1: 4
edges at min_link=3: 0

Positioning a theory with landscape

Theory.landscape takes the themes from litmap and places a theory on them. It matches the theory's title and construct labels, and the labels and key constructs of any registered alternatives, against each theme's keywords. A theme that no account touches is flagged as an under-theorised front. A theme that two or more accounts touch is flagged as a redundancy risk.

t = tf.read(fixtures / "panic-network.theory.yaml")
ls = t.landscape(corpus)

print("under-theorised fronts:", ls["under_theorised_fronts"])
print("redundancy risk:", ls["redundancy_risk"])
print("themes:", ls["themes"])

under-theorised fronts: ['theme_4']
redundancy risk: ['theme_2']
themes: [{'id': 'theme_1', 'keywords': ['appraisal', 'catastrophic misinterpretation'], 'alternatives': ['alt_cognitive'], 'focal': False, 'status': 'covered'}, {'id': 'theme_2', 'keywords': ['arousal', 'interoception'], 'alternatives': ['alt_biological'], 'focal': True, 'status': 'crowded'}, {'id': 'theme_3', 'keywords': ['avoidance', 'exposure'], 'alternatives': [], 'focal': True, 'status': 'covered'}, {'id': 'theme_4', 'keywords': ['genetics', 'heritability'], 'alternatives': [], 'focal': False, 'status': 'under_theorised'}]

Each entry in ls["themes"] reports the theme id, its keywords, the alternatives that map onto it, whether the focal theory is focal on it, and a status of under_theorised, covered or crowded. The under-theorised fronts point to questions a new theory could claim, and the redundancy risks point to ground where it would need to justify a further account.

The same function is available at module level as tf.landscape(theory, corpus), which is convenient when the theory is held as a plain dictionary rather than a Theory object. The min_link argument is passed through to the underlying litmap call.

ls = tf.landscape(t, corpus, min_link=2)

Diagrams

tf.lit_diagram exports the literature structures as Graphviz DOT text. It accepts the output of litmap for the two network views, and the output of landscape for the theme map. The default type is keyword_cooccurrence.

m = tf.litmap(corpus)
ls = t.landscape(corpus)

The keyword co-occurrence view is an undirected graph over the retained keyword pairs, each edge labelled with the number of records in which the pair appears.

print(tf.lit_diagram(m, type="keyword_cooccurrence"))

graph keyword_cooccurrence {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  node [shape=ellipse, style="filled", fillcolor="#E4F1F1", color="#1E7B7B"];
  "appraisal";
  "arousal";
  "avoidance";
  "catastrophic misinterpretation";
  "exposure";
  "genetics";
  "heritability";
  "interoception";
  "appraisal" -- "catastrophic misinterpretation" [label="2"];
  "arousal" -- "interoception" [label="2"];
  "avoidance" -- "exposure" [label="2"];
  "genetics" -- "heritability" [label="2"];
}

Passed to Graphviz, that text reads as the figure below. The four themes are visible as four disconnected pairs, which is what makes this corpus separate so cleanly into components.

keyword_cooccurrence appraisal appraisal catastrophic misinterpretation catastrophic misinterpretation appraisal--catastrophic misinterpretation 2 arousal arousal interoception interoception arousal--interoception 2 avoidance avoidance exposure exposure avoidance--exposure 2 genetics genetics heritability heritability genetics--heritability 2

The co-citation view has the same shape, drawn over pairs of cited works rather than pairs of keywords. At the default threshold this corpus retains only two disjoint pairs, so the text carries the whole picture and no figure is needed.

print(tf.lit_diagram(m, type="co_citation"))

graph co_citation {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  node [shape=ellipse, style="filled", fillcolor="#E7EDF5", color="#33567A"];
  "barlow2002";
  "bouton2001";
  "clark1986";
  "craske2008";
  "barlow2002" -- "clark1986" [label="3"];
  "bouton2001" -- "craske2008" [label="2"];
}

The theme landscape is a directed graph instead. It links the focal theory and its registered alternatives to the themes each of them addresses, and labels every theme node with its status, so one figure carries what the prose above described a piece at a time.

print(tf.lit_diagram(ls, type="theme_landscape"))

digraph theme_landscape {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  "theme_1" [label="theme_1\nappraisal, catastrophic\nmisinterpretation\n(covered)", fillcolor="#F1F1F1", color="#8A8A8A"];
  "theme_2" [label="theme_2\narousal, interoception\n(crowded)", fillcolor="#FBF1DC", color="#9C6B14"];
  "theme_3" [label="theme_3\navoidance, exposure\n(covered)", fillcolor="#F1F1F1", color="#8A8A8A"];
  "theme_4" [label="theme_4\ngenetics, heritability\n(under_theorised)", fillcolor="#E4F1F1", color="#1E7B7B"];
  "alt_cognitive" [label="alt_cognitive", shape=ellipse, fillcolor="#F1F1F1", color="#8A8A8A"];
  "alt_biological" [label="alt_biological", shape=ellipse, fillcolor="#F1F1F1", color="#8A8A8A"];
  "focal" [label="focal", shape=ellipse, fillcolor="#12283A", color="#12283A", fontcolor="#FFFFFF"];
  "alt_cognitive" -> "theme_1";
  "alt_biological" -> "theme_2";
  "focal" -> "theme_2";
  "focal" -> "theme_3";
}
theme_landscape theme_1 theme_1 appraisal, catastrophic misinterpretation (covered) theme_2 theme_2 arousal, interoception (crowded) theme_3 theme_3 avoidance, exposure (covered) theme_4 theme_4 genetics, heritability (under_theorised) alt_cognitive alt_cognitive alt_cognitive->theme_1 alt_biological alt_biological alt_biological->theme_2 focal focal focal->theme_2 focal->theme_3

Write the returned text to a .dot file and render it with Graphviz, or pass it to any tool that reads DOT.

from pathlib import Path

Path("keywords.dot").write_text(tf.lit_diagram(m), encoding="utf-8")

Fetching a corpus from OpenAlex

tf.fetch_corpus builds a corpus by querying the OpenAlex API. It is an optional convenience adapter. Because it depends on a network service whose results change over time, it sits outside the deterministic core of the package. Use it to assemble a starting corpus, then save the result and work from the saved file so that later analysis stays reproducible.

corpus = tf.fetch_corpus("panic disorder theory", per_page=25, mailto="you@example.org")

The mailto argument is optional and identifies the caller to OpenAlex, as that service requests. The returned mapping has the same shape as a corpus read from disk, so it flows straight into litmap and landscape. Supplying a corpus file remains the recommended path for any analysis that needs to be repeated exactly.

Locating a corpus is only one use of a literature search. A second, recurring need is narrower: checking whether a search has turned up any source the theory does not already cite. new_evidence_dois answers that question deterministically, from a theory and a plain list of candidate DOIs, regardless of where the DOIs came from.

t = tf.new_theory("demo", "A demonstration theory")
t.data["evidence"] = [{"supports": "p1", "source_doi": "10.1016/j.brat.2015.10.002"}]

candidates = [
    "10.1016/j.brat.2015.10.002",                  # already cited
    "https://doi.org/10.1037/0033-2909.99.1.20",   # not yet cited
]

print(t.new_evidence_dois(candidates))

['https://doi.org/10.1037/0033-2909.99.1.20']

The comparison is on a normalised form of each DOI (lowercased, with a doi.org/dx.doi.org URL prefix stripped), so a plain DOI and a resolvable URL for the same work are recognised as the same source. The function takes no network dependency itself: the search is left entirely to whichever tool supplies the candidate list.

Combining it with scopusflow-py

scopusflow-py is a companion package, by the same author, for querying the Elsevier Scopus Search API. It is a natural source of candidate DOIs: fetch_plan retrieves records for a search plan, and extract_dois reduces them to a plain list of DOIs, which is exactly the input new_evidence_dois expects. scopusflow-py is not a dependency of theoryforge, so the snippet below is illustrative rather than a tested example. Install scopusflow-py, and configure pybliometrics with a Scopus API key, to run it.

from scopusflow import scopus_query, SearchPlan, fetch_plan, extract_dois

query = scopus_query("panic disorder", "interoception", op="AND")
plan = SearchPlan(query, years=range(2015, 2027))
records = fetch_plan(plan)
candidates = extract_dois(records)

t.new_evidence_dois(candidates)

diff_dois extends this to tracking a search over time: it compares an earlier and a later retrieval and returns a frame of DOIs marked added, removed or unchanged. Re-running a saved plan and passing the added DOIs into new_evidence_dois gives a routine for revisiting a theory's evidence base as the literature grows.

from scopusflow import diff_dois

records_2025 = fetch_plan(SearchPlan(query, years=range(2015, 2026)))
records_2026 = fetch_plan(SearchPlan(query, years=range(2015, 2027)))
diff = diff_dois(records_2025, records_2026)
added = diff.loc[diff["status"] == "added", "doi"].tolist()

t.new_evidence_dois(added)

scopusflow-py also offers compare_topics, which tracks the relative publication share of several comparison terms against a reference term across a range of years. This suits comparing a theory against its registered alternatives on their standing in the literature, using the alternatives' labels as the comparison terms.

from scopusflow import compare_topics

compare_topics(
    reference_query="panic disorder",
    comparison_terms=["cognitive appraisal account", "biological account"],
    years=range(2015, 2027),
)

Using scopusflow for a Scopus-based corpus

litmap and landscape read the keywords and references fields of each corpus record, and scopusflow-py supplies both. Its corpus builder takes the records from fetch_plan and enriches them, through Abstract Retrieval, into a frame of id, title, year, keywords (one list of author keywords per row) and references (one DataFrame of cited works per row, with id, doi, title and other fields).

fetch_corpus (OpenAlex) stays the built-in default because OpenAlex is free and keyless, so the literature layer works with no setup. Scopus needs an institutional subscription and an API key, so scopusflow-py is an opt-in source rather than a dependency. The two packages exchange plain data, a DOI list or a corpus written to a file, with no coupling in either direction; that keeps theoryforge dependency-light and usable out of the box, and lets a reader reach for whichever index they have access to.

The corpus format expects a top-level {schema_version, id, records} mapping and, within each record, references as a flat list of id strings, whereas the corpus builder returns a frame whose references entries are DataFrames. So build the mapping explicitly, reducing each references frame to one id string per cited work (the DOI where present, the Scopus id otherwise), write the result to disk and read it back with read_corpus:

# illustrative: needs scopusflow-py and a configured Scopus API key
import json

from scopusflow import SearchPlan, corpus, fetch_plan, scopus_query

import theoryforge as tf

records = fetch_plan(SearchPlan(scopus_query("panic disorder"), years=range(2015, 2027)))
frame = corpus(records)                   # id, title, year, keywords, references

corpus_records = []
for row in frame.itertuples(index=False):
    refs = row.references                 # one DataFrame of cited works
    ref_ids = refs["doi"].fillna(refs["id"]).dropna().tolist()
    corpus_records.append({
        "id": row.id,
        "title": row.title,
        "year": row.year,
        "keywords": list(row.keywords),
        "references": ref_ids,
    })

with open("corpus.json", "w", encoding="utf-8") as fh:
    json.dump({"schema_version": "1.0",
               "id": "scopus:panic disorder",
               "records": corpus_records}, fh)

lit = tf.read_corpus("corpus.json")
tf.litmap(lit)