Mapping the literature

A theory is rarely built in isolation. It sits within a field that already has crowded areas, sparse areas and recurring sets of references. The literature layer of theoryforge gives a deterministic, machine-checkable account of that field, so the placement of a theory within it can be argued from evidence rather than from impression. This article reads a small corpus, computes a bibliometric map, positions a theory against the map, and emits a diagram. The analysis functions are deterministic and require no optional dependency.

library(theoryforge)

Reading a corpus

A corpus is a simple object of the form {schema_version, id, records}, where each record carries an id and may carry title, year, keywords and references. The package ships a small fixture corpus around panic and anxiety research. The format is chosen by file extension, so the same reader handles YAML and JSON.

corpus_path <- system.file("fixtures/panic-corpus.yaml", package = "theoryforge")
corpus <- tf_read_corpus(corpus_path)

corpus$id
#> [1] "panic-corpus-demo"
length(corpus$records)   # number of works in the corpus
#> [1] 8
corpus$records[[1]]       # first record
#> $id
#> [1] "r1"
#> 
#> $title
#> [1] "Interoceptive accuracy and panic"
#> 
#> $year
#> [1] 2018
#> 
#> $keywords
#> [1] "arousal"       "interoception"
#> 
#> $references
#> [1] "clark1986"  "barlow2002"

Building a literature map

tf_litmap() computes three things from the records. It counts keyword co-occurrence within records, groups co-occurring keywords into thematic components, and counts reference co-citation. An edge is kept only when its co-occurrence count reaches min_link (default 2), which suppresses incidental pairings. Records are read in file order and every result is sorted, so the map is fully deterministic.

lm <- tf_litmap(corpus, min_link = 2)

lm$n_records
#> [1] 8
unlist(lm$keywords)                 # the vocabulary of the corpus
#> [1] "appraisal"                      "arousal"                       
#> [3] "avoidance"                      "catastrophic misinterpretation"
#> [5] "exposure"                       "genetics"                      
#> [7] "heritability"                   "interoception"
lm$keyword_cooccurrence[[1]]        # first kept co-occurrence edge
#> $a
#> [1] "appraisal"
#> 
#> $b
#> [1] "catastrophic misinterpretation"
#> 
#> $count
#> [1] 2

The thematic components are the connected groups of co-occurring keywords. The fixture is constructed so that the threshold-two graph separates cleanly into four themes.

for (theme in lm$themes) {
  cat(theme$id, ":", paste(unlist(theme$keywords), collapse = ", "), "\n")
}
#> theme_1 : appraisal, catastrophic misinterpretation 
#> theme_2 : arousal, interoception 
#> theme_3 : avoidance, exposure 
#> theme_4 : genetics, heritability

Co-citation is computed the same way over the references field. Two cited sources are linked when at least min_link records in the corpus cite them together.

lm$co_citation
#> [[1]]
#> [[1]]$a
#> [1] "barlow2002"
#> 
#> [[1]]$b
#> [1] "clark1986"
#> 
#> [[1]]$count
#> [1] 3
#> 
#> 
#> [[2]]
#> [[2]]$a
#> [1] "bouton2001"
#> 
#> [[2]]$b
#> [1] "craske2008"
#> 
#> [[2]]$count
#> [1] 2

Positioning a theory against the field

tf_landscape() maps a theory’s focal constructs and its registered alternatives onto the themes from tf_litmap(). A theme is tagged "under_theorised" when neither the focal theory nor any alternative touches it, "covered" when exactly one does, and "crowded" when two or more do. Matching is by shared tokens between the theme keywords and the text of the construct labels, the theory title and the alternatives.

We build a small theory whose constructs speak to arousal and to appraisal, and register one alternative that also speaks to appraisal. This leaves the arousal theme covered, the appraisal theme crowded and the remaining themes untouched.

theory <- tf_theory("panic-appraisal", "Arousal and appraisal in panic") |>
  tf_add_construct("c_arousal", "Physiological arousal",
                   "Bodily activation in response to a stressor.") |>
  tf_add_construct("c_appraisal", "Threat appraisal",
                   "Catastrophic appraisal of bodily sensations.") |>
  tf_add_alternative("alt_cog", "Cognitive appraisal account",
                     key_constructs = c("appraisal"))

landscape <- tf_landscape(theory, corpus, min_link = 2)
landscape$theory_id
#> [1] "panic-appraisal"

Each theme in the result records which alternatives land on it, whether the focal theory lands on it, and the resulting status.

for (theme in landscape$themes) {
  cat(theme$id,
      "| status:", theme$status,
      "| focal:", theme$focal,
      "| alternatives:", paste(unlist(theme$alternatives), collapse = ", "),
      "\n")
}
#> theme_1 | status: crowded | focal: TRUE | alternatives: alt_cog 
#> theme_2 | status: covered | focal: TRUE | alternatives:  
#> theme_3 | status: under_theorised | focal: FALSE | alternatives:  
#> theme_4 | status: under_theorised | focal: FALSE | alternatives:

Two summaries draw out the practical reading. The under-theorised fronts are themes that the field around this theory has not yet engaged, and so are candidates for new work. The redundancy risk lists crowded themes, where a new contribution would have to distinguish itself from existing accounts.

unlist(landscape$under_theorised_fronts)
#> [1] "theme_3" "theme_4"
unlist(landscape$redundancy_risk)
#> [1] "theme_1"

Emitting a literature diagram

tf_lit_diagram() renders a deterministic Graphviz DOT string for the literature layer. The keyword co-occurrence and co-citation maps render as undirected graphs, and a landscape result renders as a directed theme map. The output is a plain string, so it can be written to a .dot file or passed to a Graphviz renderer.

cat(tf_lit_diagram(lm, type = "keyword_cooccurrence"))

graph keyword_cooccurrence {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  node [shape=ellipse, style="filled", fillcolor="#E4F1F1", color="#1E7B7B"];
  "appraisal";
  "arousal";
  "avoidance";
  "catastrophic misinterpretation";
  "exposure";
  "genetics";
  "heritability";
  "interoception";
  "appraisal" -- "catastrophic misinterpretation" [label="2"];
  "arousal" -- "interoception" [label="2"];
  "avoidance" -- "exposure" [label="2"];
  "genetics" -- "heritability" [label="2"];
}

tf_render_diagram() accepts a DOT string as well as a theory, so the literature diagrams render the same way as the theory views.

cat('<div class="tf-figure tf-diagram">', tf_render_diagram(tf_lit_diagram(lm, type = "keyword_cooccurrence"), as = "svg"), '</div>', sep = "")

The co-citation map links the references that the corpus cites together, so tightly co-cited work stands out as the intellectual base the field shares.

cat(tf_lit_diagram(lm, type = "co_citation"))

graph co_citation {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  node [shape=ellipse, style="filled", fillcolor="#E7EDF5", color="#33567A"];
  "barlow2002";
  "bouton2001";
  "clark1986";
  "craske2008";
  "barlow2002" -- "clark1986" [label="3"];
  "bouton2001" -- "craske2008" [label="2"];
}

cat('<div class="tf-figure tf-diagram">', tf_render_diagram(tf_lit_diagram(lm, type = "co_citation"), as = "svg"), '</div>', sep = "")

The theme landscape diagram is produced from the tf_landscape() result rather than from the map, since it carries the focal theory and the alternatives.

cat(tf_lit_diagram(landscape, type = "theme_landscape"))

digraph theme_landscape {
  graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
  node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
  edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
  "theme_1" [label="theme_1\nappraisal, catastrophic\nmisinterpretation\n(crowded)", fillcolor="#FBF1DC", color="#9C6B14"];
  "theme_2" [label="theme_2\narousal, interoception\n(covered)", fillcolor="#F1F1F1", color="#8A8A8A"];
  "theme_3" [label="theme_3\navoidance, exposure\n(under_theorised)", fillcolor="#E4F1F1", color="#1E7B7B"];
  "theme_4" [label="theme_4\ngenetics, heritability\n(under_theorised)", fillcolor="#E4F1F1", color="#1E7B7B"];
  "alt_cog" [label="alt_cog", shape=ellipse, fillcolor="#F1F1F1", color="#8A8A8A"];
  "focal" [label="focal", shape=ellipse, fillcolor="#12283A", color="#12283A", fontcolor="#FFFFFF"];
  "alt_cog" -> "theme_1";
  "focal" -> "theme_1";
  "focal" -> "theme_2";
}

cat('<div class="tf-figure tf-diagram">', tf_render_diagram(tf_lit_diagram(landscape, type = "theme_landscape"), as = "svg"), '</div>', sep = "")

Building a corpus from OpenAlex

The corpus above was read from a file. A corpus can also be assembled from the OpenAlex works API with tf_fetch_corpus(). This adapter is assistive. It makes a live network call and its result depends on an external service that changes over time, so it sits outside the deterministic core of the package. The call is shown here but not run, and the supplied email enters the OpenAlex polite pool.

fetched <- tf_fetch_corpus("panic disorder interoception",
                           per_page = 25,
                           mailto = "me@example.org")
fetched_map <- tf_litmap(fetched)

Once fetched, the corpus object has the same shape as a file-read corpus, so it flows into tf_litmap(), tf_landscape() and tf_lit_diagram() without any further change. The file-read corpus used throughout this article shows that shape.

str(corpus, max.level = 2, list.len = 4)
#> List of 3
#>  $ schema_version: chr "1.0"
#>  $ id            : chr "panic-corpus-demo"
#>  $ records       :List of 8
#>   ..$ :List of 5
#>   ..$ :List of 5
#>   ..$ :List of 5
#>   ..$ :List of 5
#>   .. [list output truncated]

The recommended pattern is to fetch once, write the result to disk, and then work from the saved file so that later analyses remain reproducible.

Tracking new evidence with an external search

Locating a corpus is only one use of a literature search. A second, recurring need is narrower: checking whether a search has turned up any source the theory does not already cite. tf_new_evidence_dois() answers that question deterministically, from a theory object and a plain vector of candidate DOIs, regardless of where the DOIs came from.

theory <- tf_theory("demo", "A demonstration theory") |>
  tf_add_construct("c_arousal", "Arousal", "Bodily activation.")
theory$evidence <- list(list(supports = "p1",
                              source_doi = "10.1016/j.brat.2015.10.002"))

candidates <- c("10.1016/j.brat.2015.10.002",       # already cited
                "https://doi.org/10.1037/0033-2909.99.1.20")  # not yet cited

tf_new_evidence_dois(theory, candidates)
#> [1] "https://doi.org/10.1037/0033-2909.99.1.20"

The comparison is on a normalised form of each DOI (lowercased, with a doi.org/dx.doi.org URL prefix stripped), so a plain DOI and a resolvable URL for the same work are recognised as the same source. The function takes no network dependency itself: the search is left entirely to whichever tool supplies the candidate list.

Combining it with scopusflow

scopusflow is a companion R package, by the same author, for querying the Elsevier Scopus Search API. It is a natural source of candidate DOIs: scopus_fetch() retrieves records for a query, and scopus_extract_dois() reduces them to a plain DOI vector, which is exactly the input tf_new_evidence_dois() expects.

scopusflow is not a dependency of theoryforge, so the snippet below is illustrative rather than executed by this vignette. Install scopusflow and supply a Scopus API key (see its scopus_has_key()) to run it.

library(scopusflow)

query <- scopus_query("panic disorder", "interoception", .op = "AND")
records <- scopus_fetch(query, years = 2015:2026)
candidates <- scopus_extract_dois(records)

tf_new_evidence_dois(theory, candidates)

scopus_diff_dois() extends this to tracking a search over time: it compares an earlier and a later retrieval (either scopus_records objects or DOI vectors) and reports which DOIs were added, removed or unchanged. Re-running a saved query and passing the "added" DOIs into tf_new_evidence_dois() gives a routine for revisiting a theory’s evidence base as the literature grows.

records_2025 <- scopus_fetch(query, years = 2015:2025)
records_2026 <- scopus_fetch(query, years = 2015:2026)
diff <- scopus_diff_dois(records_2025, records_2026)
added <- diff$doi[diff$status == "added"]

tf_new_evidence_dois(theory, added)

The hand-off itself needs nothing from scopusflow: scopus_diff_dois() returns one row per DOI with its status, which we reproduce here so the last step runs. Only the newly added DOI comes back as new evidence.

diff <- data.frame(
  doi = c("10.1016/j.brat.2015.10.002", "10.1037/0033-2909.99.1.20"),
  status = c("unchanged", "added")
)
added <- diff$doi[diff$status == "added"]

tf_new_evidence_dois(theory, added)
#> [1] "10.1037/0033-2909.99.1.20"

scopusflow also offers scopus_compare_topics(), which tracks the relative publication share of several comparison terms against a reference term across a range of years. This suits comparing a theory against its registered alternatives on their standing in the literature, using the alternatives’ labels as the comparison terms.

scopus_compare_topics(
  reference_query = "panic disorder",
  comparison_terms = c("cognitive appraisal account", "biological account"),
  years = 2015:2026
)

The comparison and its plot, with per-year stability bands, are shown in scopusflow’s Comparing topics article.

Using scopusflow for a Scopus-based corpus

tf_litmap() and tf_landscape() read the keywords and references fields of each corpus record, and scopusflow supplies both. scopus_corpus() takes the records from scopus_fetch() and enriches them, through Abstract Retrieval, into a tibble of id, title, year, keywords (one character vector of author keywords per record) and references (one data frame of cited works per record, with id, doi, title and other fields).

tf_fetch_corpus() (OpenAlex) stays the built-in default because OpenAlex is free and keyless, so the literature layer works with no setup. Scopus needs an institutional subscription and an API key, so scopusflow is an opt-in source rather than a dependency. The two packages exchange plain data, a DOI vector or a corpus written to a file, with no coupling in either direction; that keeps theoryforge dependency-light and usable out of the box, and lets a reader reach for whichever index they have access to.

The corpus format expects a top-level {schema_version, id, records} envelope and, within each record, references as a flat list of id strings, whereas scopus_corpus() returns a tibble whose references entries are data frames. So build the envelope explicitly, reducing each references data frame to one id string per cited work (the DOI where present, the Scopus id otherwise), write the result to disk and read it back with tf_read_corpus():

# illustrative: needs scopusflow and a configured Scopus API key
recs <- scopusflow::scopus_fetch(query, years = 2015:2026)
sc   <- scopusflow::scopus_corpus(recs)   # id, title, year, keywords, references

records <- lapply(seq_len(nrow(sc)), function(i) {
  refs <- sc$references[[i]]              # one data frame of cited works
  ref_ids <- ifelse(is.na(refs$doi), refs$id, refs$doi)
  list(id         = sc$id[[i]],
       title      = sc$title[[i]],
       year       = sc$year[[i]],
       keywords   = as.list(sc$keywords[[i]]),
       references = as.list(ref_ids[!is.na(ref_ids)]))
})
corpus <- list(schema_version = "1.0",
               id = "scopus:panic disorder",
               records = records)
jsonlite::write_json(corpus, "corpus.json", auto_unbox = TRUE)

lit <- tf_read_corpus("corpus.json")
tf_litmap(lit)