Developing and testing a theory
Source:vignettes/articles/developing-and-testing.Rmd
developing-and-testing.RmdThe companion vignette covers building a theory and checking its
rigour. This article continues the workflow into development and
testing. It shows how to operationalise severity, derive what the causal
graph forbids in data, export a preregistration, appraise an amendment
in Lakatosian terms, compile a measurement model for lavaan
and assemble a reviewer-facing audit bundle. The worked example reads
the panic-network fixture shipped with the package, joined in one
section by the shipped modality-switching fixture, so every chunk runs
with only theoryforge loaded.
theory <- tf_read(
system.file("fixtures/panic-network.theory.yaml", package = "theoryforge")
)
theory$id[1] "panic-network-2026"
Operationalised severity
A prediction earns evidential weight in proportion to the risk it
takes. A point prediction stakes more on the data than a directional
one, and that asymmetry should be made explicit before any test is run.
tf_severity() reports, for each prediction in file order,
the riskiness of its claim form and the resulting computed severity.
tf_severity(theory) prediction_id type risk_score computed_severity
1 pred1 point 0.9 1.0
2 pred2 interval 0.7 0.7
3 pred3 directional 0.4 0.3
The risk_score reflects the prediction type, while
computed_severity applies the discounts and bonuses defined
in the specification. Reading these values before data collection guards
against the temptation to read a weak, hard-to-fail prediction as though
it were a stringent test.
What the causal graph commits you to
Severity scores the predictions a theory states. A causal theory also
commits itself to claims it never states, because the propositions taken
together are a directed graph, and a directed acyclic graph entails a
set of conditional independencies. tf_implications()
derives them. Every pair of constructs with no causal relation between
them yields one claim: the two are independent once the parents of both
are held fixed. That collection is the basis set, from which every other
independence the graph entails follows, and each member is something a
study can be designed to look for and fail to find.
The panic-network example does not get that far. Its third proposition returns from perceived threat to arousal, closing the feedback loop the theory is about, and a graph with a cycle has no basis set at all.
tf_implications(theory)Error:
! implications requires an acyclic causal graph; cycle found: c_arousal -> c_perceived_threat -> c_arousal
The refusal names the cycle it found, so nobody has to hunt for it. Where a theory does hold a feedback loop, this is the point at which the loop has to be modelled over time, since a single graph cannot carry it.
The package ships a second example whose causal graph is acyclic, a theory of modality switching in grounded conceptual processing. Sensorimotor experience with a concept drives activation of the modality-specific perceptual system. That activation both raises the cost of switching modality between consecutive trials and eases conceptual access, which lexical familiarity with the word form eases as well.
switching <- tf_read(
system.file("fixtures/modality-switching.theory.yaml", package = "theoryforge")
)
implied <- tf_implications(switching)
implied$n_edges[1] 4
implied$n_implications[1] 6
writeLines(vapply(implied$implications, function(i) i$statement, character(1)))c_sensorimotor_experience _||_ c_switch_cost | c_modality_activation
c_sensorimotor_experience _||_ c_conceptual_access | c_modality_activation, c_lexical_familiarity
c_sensorimotor_experience _||_ c_lexical_familiarity
c_modality_activation _||_ c_lexical_familiarity | c_sensorimotor_experience
c_switch_cost _||_ c_conceptual_access | c_modality_activation, c_lexical_familiarity
c_switch_cost _||_ c_lexical_familiarity | c_modality_activation
Five constructs and four causal propositions leave six non-adjacent pairs, and so six claims, each written in the notation dagitty prints. Two of them are worth dwelling on. Switch cost and ease of conceptual access are the two children of a fork at modality activation, so the theory says the correlation between them is entirely due to that shared cause and vanishes once it is held fixed. Sensorimotor experience and lexical familiarity meet only at the collider at conceptual access, and the theory says they are unrelated to each other with nothing held fixed, which is what an account deriving perceptual knowledge from distributional word statistics would deny. Neither claim appears among the theory’s own predictions, and either could sink it.
Preregistration
tf_preregister() renders a preregistration document as a
single markdown string. It records the predictions, each with its claim
type and derivation, plus the severity values, in a fixed order, so the
output is stable across runs and suitable for deposit. Because the
string is markdown, it renders below as the document a registry would
receive.
prereg <- tf_preregister(theory)
cat(prereg)Preregistration: Network theory of panic disorder
- Theory ID: panic-network-2026
- Schema version: 1.0
- Maturity: developing
- Derivation chain verified: yes
Hypotheses
- [point] An interoceptive challenge raises arousal to a specified level within 90 seconds. (derives from: p1, p3)
- [interval] Avoidance frequency falls within a 20-35% band after exposure therapy. (derives from: p2)
- [directional] Higher perceived threat is associated with more avoidance. (derives from: p2)
Severity
- pred1: severity 1.0, risk 0.9
- pred2: severity 0.7, risk 0.7
- pred3: severity 0.3, risk 0.4
Passing a path writes the same markdown to disk with LF
line endings, which is convenient when the document is committed to a
repository or uploaded to a registry.
prereg_path <- tempfile(fileext = ".md")
invisible(tf_preregister(theory, path = prereg_path))
file.exists(prereg_path)[1] TRUE
Appraising an amendment
Theories change. The question is whether a change is progressive, in
that it predicts and then corroborates something new, or degenerating,
in that it only adds assumptions to absorb anomalies.
tf_appraise_amendment() compares an amended theory against
its prior version and returns a verdict in those terms.
Here the prior is the panic-network fixture and the amended version is its 2026 revision, which adds one new prediction that is later corroborated and introduces no new ad-hoc assumptions.
v2 <- tf_read(
system.file(
"fixtures/panic-network-2026-v2.theory.yaml",
package = "theoryforge"
)
)
appraisal <- tf_appraise_amendment(v2, theory)
appraisal$verdict[1] "progressive"
appraisal$new_predictions[1] "pred4"
appraisal$corroborated_new[1] "pred4"
appraisal$ad_hoc_assumptionscharacter(0)
A progressive verdict reflects new content that survived
a test. Had the amendment instead added assumptions tied to specific
anomalies without generating corroborated predictions, the verdict would
record that as degenerating.
Compiling a measurement model
When constructs carry measurement indicators,
tf_compile_sem() compiles them into lavaan
model syntax. Constructs with indicators become latent measurement
models with =~, and propositions become structural paths
with ~, covariances with ~~ or annotated
moderation lines.
cat(tf_compile_sem(theory))# lavaan model generated by theoryforge for panic-network-2026
# Measurement model
c_arousal =~ heart_rate_variability + self_reported_arousal
c_perceived_threat =~ threat_appraisal_questionnaire
c_avoidance =~ behavioural_avoidance_task + diary_counts
# Structural model
c_perceived_threat ~ c_arousal
c_avoidance ~ c_perceived_threat
c_arousal ~ c_perceived_threat
The function returns a plain string and does not require
lavaan to be installed. The output can be passed directly
to lavaan::sem() once the data are available.
The reviewer audit bundle
tf_dossier() assembles a single markdown document for a
reviewer or editor. It collects the header, the rigour-checklist table,
the severity list, the provenance trail and the preregistration into one
bundle, so a reader can follow the theory from its claims through to its
planned tests in one place. The opening of the bundle, through the
rigour-checklist table, renders below.
dossier <- tf_dossier(theory)
cat(strsplit(dossier, "\n## Severity", fixed = TRUE)[[1]][1])theoryforge dossier: Network theory of panic disorder
- Theory ID: panic-network-2026
- Maturity: developing
- Checklist version: 1.0
- Aggregate rigour score: 84.8/100
- Gate: pass
- Blockers failed: 0
Rigour checklist
| item | status | score | weight |
|---|---|---|---|
| falsifiability | pass | 1.0 | 0.15 |
| precision | pass | 0.667 | 0.1 |
| risk_severity | pass | 0.567 | 0.1 |
| parsimony | pass | 0.667 | 0.08 |
| non_redundancy | pass | 0.909 | 0.1 |
| construct_clarity | pass | 1.0 | 0.08 |
| scope | pass | 1.0 | 0.06 |
| logical_why | pass | 1.0 | 0.08 |
| causal_testability | pass | 1.0 | 0.06 |
| diagnosticity | pass | 0.333 | 0.06 |
| formalisation | pass | 1.0 | 0.05 |
| derivation_chain | pass | 1.0 | 0.08 |
The severity list, the provenance trail and the preregistration follow in the full document, which is deterministic, so the same theory always produces the same dossier.
Rendering and depositing
tf_render_report() writes the dossier as a Quarto source
document, ready to render to HTML or PDF. With
render = FALSE, which is the default, it writes the
.qmd without calling Quarto, so the step runs anywhere. It
returns the path of the file it wrote.
report_path <- tf_render_report(theory, tempfile(fileext = ".qmd"))
basename(report_path)[1] "file1faf104f4873.qmd"
tf_osf_push() deposits the audit dossier on an Open
Science Framework node, as <id>.dossier.md. It
defaults to dry_run = TRUE, which assembles the request and
returns it without sending anything, so the planned deposit can be
inspected offline and without a token. What goes up is the dossier,
since that is the machine-checkable bundle a reviewer needs. The report
written just above stays on your own machine.
str(tf_osf_push(theory, node = "ab12c"))List of 3
$ dry_run: logi TRUE
$ request:List of 4
..$ method : chr "PUT"
..$ url : chr "https://files.osf.io/v1/resources/ab12c/providers/osfstorage/?kind=file&name=panic-network-2026.dossier.md"
..$ filename : chr "panic-network-2026.dossier.md"
..$ content_bytes: int 1669
$ note : chr "set dry_run=FALSE with a valid token and node to perform the upload"
The returned request records the method, the destination URL, the
filename the deposit would create and the size of the dossier in bytes,
so the step can be checked before anything leaves the machine. A live
deposit needs dry_run = FALSE together with a token and the
node id, and that is the only path that touches the network.
tf_osf_push(
theory,
node = "ab12c",
token = Sys.getenv("OSF_PAT"),
dry_run = FALSE
)Visualising the theory
tf_diagram() exports several views of the same object.
The workflow view traces the lifecycle from constructs,
through propositions and predictions, to the recorded test outcomes, so
the path from a claim to its planned test is visible in one figure.
cat(tf_diagram(theory, "workflow"))digraph workflow {
graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
subgraph cluster_build {
label="building";
style="rounded";
color="#C4D1D9";
fontcolor="#5B7285";
"c_arousal" [label="Physiological\narousal", fillcolor="#E4F1F1", color="#1E7B7B"];
"c_perceived_threat" [label="Perceived threat", fillcolor="#E4F1F1", color="#1E7B7B"];
"c_avoidance" [label="Avoidance\nbehaviour", fillcolor="#E4F1F1", color="#1E7B7B"];
}
subgraph cluster_relate {
label="propositions";
style="rounded";
color="#C4D1D9";
fontcolor="#5B7285";
"prop_p1" [label="p1\nincreases", fillcolor="#FBF1DC", color="#9C6B14"];
"prop_p2" [label="p2\nincreases", fillcolor="#FBF1DC", color="#9C6B14"];
"prop_p3" [label="p3\ncauses", fillcolor="#FBF1DC", color="#9C6B14"];
}
subgraph cluster_predict {
label="predictions";
style="rounded";
color="#C4D1D9";
fontcolor="#5B7285";
"pred_pred1" [label="pred1\npoint", fillcolor="#E7EDF5", color="#33567A"];
"pred_pred2" [label="pred2\ninterval", fillcolor="#E7EDF5", color="#33567A"];
"pred_pred3" [label="pred3\ndirectional", fillcolor="#E7EDF5", color="#33567A"];
}
subgraph cluster_test {
label="testing";
style="rounded";
color="#C4D1D9";
fontcolor="#5B7285";
"outcome_pred1" [label="pred1\npassed", fillcolor="#E5F2E7", color="#3E7A46"];
}
"c_arousal" -> "prop_p1";
"c_perceived_threat" -> "prop_p2";
"c_perceived_threat" -> "prop_p3";
"prop_p1" -> "pred_pred1";
"prop_p3" -> "pred_pred1";
"prop_p2" -> "pred_pred2";
"prop_p2" -> "pred_pred3";
"pred_pred1" -> "outcome_pred1";
}
Rendered with tf_render_diagram(), which passes the DOT
to the DiagrammeR engine, the same view reads as a figure.
cat(
'<div class="tf-figure tf-diagram">',
tf_render_diagram(theory, "workflow", as = "svg"),
'</div>',
sep = ""
)The context view places the theory among the scope
conditions under which it is claimed to hold and the registered rivals
it is meant to outpredict.
cat(tf_diagram(theory, "context"))digraph context {
graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
"theory" [shape=ellipse, label="Network theory of\npanic disorder", fillcolor="#12283A", color="#12283A", fontcolor="#FFFFFF"];
"c_arousal" [label="Physiological\narousal", fillcolor="#E4F1F1", color="#1E7B7B"];
"theory" -> "c_arousal";
"c_perceived_threat" [label="Perceived threat", fillcolor="#E4F1F1", color="#1E7B7B"];
"theory" -> "c_perceived_threat";
"c_avoidance" [label="Avoidance\nbehaviour", fillcolor="#E4F1F1", color="#1E7B7B"];
"theory" -> "c_avoidance";
"scope1" [shape=note, style="filled", label="adults", fillcolor="#FBF7EA", color="#B49B55"];
"scope1" -> "theory" [style=dotted, label="holds within"];
"scope2" [shape=note, style="filled", label="non-clinical\nbaseline", fillcolor="#FBF7EA", color="#B49B55"];
"scope2" -> "theory" [style=dotted, label="holds within"];
"scope3" [shape=note, style="filled", label="no beta-blocker\nmedication", fillcolor="#FBF7EA", color="#B49B55"];
"scope3" -> "theory" [style=dotted, label="holds within"];
"scope4" [shape=note, style="filled", label="waking hours", fillcolor="#FBF7EA", color="#B49B55"];
"scope4" -> "theory" [style=dotted, label="holds within"];
"scope5" [shape=note, style="filled", label="community sample", fillcolor="#FBF7EA", color="#B49B55"];
"scope5" -> "theory" [style=dotted, label="holds within"];
"alt_cognitive" [style="rounded,filled,dashed", label="Cognitive model of\npanic", fillcolor="#F1F1F1", color="#8A8A8A"];
"theory" -> "alt_cognitive" [style=dashed, label="contrasts with"];
"alt_biological" [style="rounded,filled,dashed", label="Biological model\nof panic", fillcolor="#F1F1F1", color="#8A8A8A"];
"theory" -> "alt_biological" [style=dashed, label="contrasts with"];
}
cat(
'<div class="tf-figure tf-diagram">',
tf_render_diagram(theory, "context", as = "svg"),
'</div>',
sep = ""
)The pipeline view links each prediction to its recorded
test outcome, so predictions still awaiting a test are visible as loose
ends.
cat(tf_diagram(theory, "pipeline"))digraph pipeline {
graph [rankdir=LR, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
"pred1" [label="pred1\npoint", fillcolor="#E7EDF5", color="#33567A"];
"pred2" [label="pred2\ninterval", fillcolor="#E7EDF5", color="#33567A"];
"pred3" [label="pred3\ndirectional", fillcolor="#E7EDF5", color="#33567A"];
"result_pred1" [label="passed", fillcolor="#E5F2E7", color="#3E7A46"];
"pred1" -> "result_pred1";
}
cat(
'<div class="tf-figure tf-diagram">',
tf_render_diagram(theory, "pipeline", as = "svg"),
'</div>',
sep = ""
)The provenance view draws the build log as a digraph, so
the record of how the theory reached its current state travels with the
object itself.
cat(tf_diagram(theory, "provenance"))digraph provenance {
graph [rankdir=TB, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
"n1" [label="tf_construct\nRegistered three\nconstructs."];
"n2" [label="tf_proposition\nLinked constructs into a\nfeedback network."];
"n3" [label="tf_predict\nDerived three predictions\nfrom the propositions."];
"n1" -> "n2";
"n2" -> "n3";
}
cat(
'<div class="tf-figure tf-diagram">',
tf_render_diagram(theory, "provenance", as = "svg"),
'</div>',
sep = ""
)The venn view takes each construct’s boundary conditions
as a set and returns an SVG showing where those scopes coincide and
where they part company, with each region labelled by the number of
conditions in it. It renders inline because the output is a
self-contained image.
cat('<div class="tf-figure">', tf_diagram(theory, "venn"), '</div>', sep = "")Two further views summarise the scoring. The rigour view
draws the checklist as a status grid, colouring each item by its result
and reporting the aggregate score and the gate.
cat('<div class="tf-figure">', tf_diagram(theory, "rigour"), '</div>', sep = "")The severity view draws one bar per prediction, scaled
by its computed severity, so the riskier tests stand out.
cat(
'<div class="tf-figure">',
tf_diagram(theory, "severity"),
'</div>',
sep = ""
)The development_roadmap view turns the same checklist
into a worklist by keeping only the items that still fail or warn, and
it orders that worklist the way the work should be done. Whatever gates
the theory comes first, heaviest check first, and the advisory items
follow. Each step carries its number, the criterion it is measured
against and whether missing it blocks the gate or is merely advisory, so
the figure amounts to a plan of work. The hub names the theory and its
current standing. The panic network passes every check, so its roadmap
reduces to that hub and an all checks pass node, and the
deliberately weak theory shipped alongside it shows the worklist in
full.
weak <- tf_read(
system.file("fixtures/weak-theory.theory.yaml", package = "theoryforge")
)
cat(tf_diagram(theory, "development_roadmap"))digraph development_roadmap {
graph [rankdir=TB, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
"roadmap" [shape=ellipse, label="Network theory of\npanic disorder\nscore 84.8, gate pass", fillcolor="#12283A", color="#12283A", fontcolor="#FFFFFF"];
"all_checks_pass" [label="all checks pass", fillcolor="#E5F2E7", color="#3E7A46"];
"roadmap" -> "all_checks_pass";
}
cat(tf_diagram(weak, "development_roadmap"))digraph development_roadmap {
graph [rankdir=TB, bgcolor="transparent", fontname="Helvetica", fontsize=11, pad="0.2", nodesep="0.3", ranksep="0.45"];
node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", color="#33567A", fillcolor="#F2F6F9", fontcolor="#12283A", penwidth=1.1, margin="0.16,0.1"];
edge [fontname="Helvetica", fontsize=10, color="#7B909F", fontcolor="#0F6E6E", arrowsize=0.7];
"roadmap" [shape=ellipse, label="An underspecified\nmotivation theory\nscore 12.0, gate blocked", fillcolor="#12283A", color="#12283A", fontcolor="#FFFFFF"];
"falsifiability" [label="1. falsifiability\nAt least one\nprediction forbids an\nobservation\nblocks the gate", fillcolor="#F9E5E4", color="#B2453C"];
"derivation_chain" [label="2. derivation_chain\nEach prediction is\ngraph-reachable from\npropositions\n(reachability only)\nblocks the gate", fillcolor="#F9E5E4", color="#B2453C"];
"precision" [label="3. precision\nPredictions are\npoint/interval, not\nmerely directional\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"risk_severity" [label="4. risk_severity\nMean prediction\nseverity above\nthreshold\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"construct_clarity" [label="5. construct_clarity\nEvery construct has\ndefinition +\nmeasurement + boundary\nconditions\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"logical_why" [label="6. logical_why\nEach proposition\nstates a mechanism,\nnot just a correlation\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"scope" [label="7. scope\nBoundary conditions\nexplicitly stated\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"causal_testability" [label="8. causal_testability\nAt least one\nproposition states a\ncausal relation\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"diagnosticity" [label="9. diagnosticity\nAt least one\nprediction\ndiscriminates from a\nregistered alternative\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"formalisation" [label="10. formalisation\nA formal-model stub\nexists (warn-only at\nbuilding stage)\nadvisory", fillcolor="#FBF1DC", color="#9C6B14"];
"roadmap" -> "falsifiability";
"falsifiability" -> "derivation_chain";
"derivation_chain" -> "precision";
{ rank=same; "precision" -> "risk_severity" -> "construct_clarity" [style=invis]; }
"precision" -> "logical_why" [style=invis];
{ rank=same; "logical_why" -> "scope" -> "causal_testability" [style=invis]; }
"logical_why" -> "diagnosticity" [style=invis];
{ rank=same; "diagnosticity" -> "formalisation" [style=invis]; }
}
cat(
'<div class="tf-figure tf-diagram">',
tf_render_diagram(weak, "development_roadmap", as = "svg"),
'</div>',
sep = ""
)All of these views are deterministic. The DOT strings render with any
Graphviz engine, and the SVG views open in any browser. Together with
the nomological_net and causal_dag views
introduced in the Get started vignette, they complete the set of diagram
types that tf_diagram() exports, each documented in
?tf_diagram.