You shall know a word by the company it keeps — so choose your prompts wisely
In computational linguistics, word meanings are shaped by their contexts. As the British linguist John Rupert Firth (1957, p. 11) put it, ‘You shall know a word by the company it keeps’ (see Brunila & LaViolette, 2022, for a re-examination of the intellectual history). It sounds almost like life advice, but Firth meant something technical: words that habitually appear alongside each other tend to share semantic territory. The adjective ‘good’, for instance, is far more likely to appear near ‘kind’, ‘genuine’, ‘fair’ and ‘quality’ than near ‘broken’ or ‘fraud’, and a model that tracks those neighbours can learn what ‘good’ means without ever being told. The principle extends to polysemy, as the ‘bank’ that keeps company with ‘river’ and ‘fishing rod’ is a different thing altogether from the ‘bank’ found beside ‘loan’ and ‘account’.
This deceptively simple insight is the bedrock on which generative AI was built. Its earliest computational implementations were distributional semantic models such as Latent Semantic Analysis (LSA; Landauer & Dumais, 1997) and the Hyperspace Analogue to Language (Lund & Burgess, 1996). By today’s standards, they were modest, built on a matrix of word co-occurrence counts, with a few hundred latent dimensions and a vocabulary of perhaps tens of thousands of words. Yet even these pocket-sized models captured real-world structure with startling fidelity. Louwerse and Zwaan (2009) applied LSA to newspaper texts and used the resulting similarities between the 50 largest cities in the United States to place the cities in a two-dimensional space. The coordinates correlated with the cities’ actual longitude and latitude, because cities that lie close together share similar contexts in text, and a simple count of how often the city names co-occur replicated the result. A model trained on text alone had drawn an approximate map of the country.
Distributional models can also track sensorimotor properties of concepts (Bernabeu, 2022; Louwerse, 2011; Louwerse & Connell, 2011). Text alone captures only part of that picture, though. Q. Xu et al. (2025) found that large language models trained without grounding aligned with human representations of concepts far less closely in sensory and motor dimensions than in non-sensorimotor ones. Models that also learned from visual input aligned better in the visual dimensions. Fine-tuning on human sensorimotor ratings can likewise steer a model’s representations towards more grounded patterns (Wu et al., 2026). Language, then, encodes much of the structure of the world, and even a simple co-occurrence model can read a good deal of it back.
We can see this for ourselves. The R code below (click ‘Expand’ to view it) applies LSA, one of the simplest distributional models, to three text collections, then projects the resulting word vectors into two dimensions using principal component analysis (PCA) and plots them. LSA starts from a term-document matrix, a large table recording how often each word appears in each document. It weights these counts with TF-IDF (term frequency–inverse document frequency), which boosts words that are distinctive to particular documents and discounts words that appear everywhere. Truncated SVD (singular value decomposition, a form of dimensionality reduction) then compresses the weighted matrix into a small number of latent dimensions. In the resulting plots, words that occur in similar contexts cluster together, and words from different domains drift apart.
PCA finds new axes, the principal components, that capture as much of the variance in the data as possible. Each word receives a loading on each component, a number from −1 to +1 that indicates how strongly the word contributes to that axis of variation. A word with a high absolute loading on a component is a strong marker of the distinction that the component captures. An earlier post on this blog offers a gentle introduction to PCA in R.
Each corpus is split into two groups of documents. The code computes the mean TF-IDF weight of every word in each group of documents and takes the difference. Words weighted much more heavily in group A than in group B count as distinctive to A, and vice versa. The 15 words at each extreme are plotted in their group’s colour, and the ten words with the highest combined weight outside both extremes appear in purple as ‘Shared’. Which words count as ‘finance’ or ‘energy’ is thus decided by the corpus statistics alone. Words lying far outside the main cluster are trimmed from the plot, though not from the LSA space, to keep the dense region readable. Above each plot, a table gives the mean loading of each group on the first two principal components, with each group’s highest positive loading in bold. When one group loads heavily on a component and the other does not, that component is essentially the axis that separates them.
Reuters Newswire: Finance vs Energy
The first corpus combines two newswire samples from the tm package (Feinerer et al., 2008): acq, with 50 Reuters articles on corporate acquisitions, and crude, with 20 articles on crude oil markets. Both samples come from Reuters-21578, a collection of stories from the 1987 Reuters newswire that has served as a standard text categorisation benchmark since the 1990s (Lewis, 1997). The code builds a TF-IDF-weighted term-document matrix and reduces it to a 20-dimensional LSA space with truncated SVD. It then uses LSAfun::Cosine() (Günther et al., 2015) to compute pairwise cosine similarities, a standard measure of how closely two word vectors align, on a scale from −1 (opposite) to +1 (identical). The similarities are printed at the end of the code, and Table 1 and Figure 1 show the PCA projection.
pkgs <- c("LSAfun", "tm", "ggplot2", "plotly")
invisible(lapply(pkgs, function(p)
if (!requireNamespace(p, quietly = TRUE)) install.packages(p)))
library(LSAfun)
library(tm)
library(ggplot2)
library(plotly)
# --- Label placement ----------------------------------------------------
# Printed at their own coordinates, the words in a dense cluster land on
# top of one another. This finds each label the nearest free spot around
# its point, in screen pixels for a plot area of the given size. Points in
# crowded regions are placed first, and their labels try the directions
# that face away from their neighbours first. A label that ends up away
# from its point gets a leader line.
repel_labels <- function(x, y, words, xlim, ylim, width, height, font_px,
clearance = 3, step = 5, rings = 50) {
sx <- width / diff(xlim)
sy <- height / diff(ylim)
px <- (x - xlim[1]) * sx
py <- (y - ylim[1]) * sy
hw <- nchar(words) * font_px * 0.28 + 1 # half width of each label
hh <- font_px * 0.6 # half height
angles <- seq(0, 345, by = 15) * pi / 180
# Offsets that put a label's nearest edge or corner at a given distance
ox <- sign(cos(angles)) * pmin(1, abs(cos(angles)) * sqrt(2))
oy <- sign(sin(angles)) * pmin(1, abs(sin(angles)) * sqrt(2))
near <- abs(outer(px, px, "-")) < 80 & abs(outer(py, py, "-")) < 40
diag(near) <- FALSE
lx <- ly <- rep(NA_real_, length(words))
for (i in order(-rowSums(near))) {
away <- if (any(near[i, ])) {
atan2(py[i] - mean(py[near[i, ]]), px[i] - mean(px[near[i, ]]))
} else pi / 2
turn <- abs(atan2(sin(angles - away), cos(angles - away)))
cand <- expand.grid(a = order(turn), ring = 0:rings)
gap <- clearance + 0.5 + cand$ring * step
cx <- px[i] + ox[cand$a] * (hw[i] + gap)
cy <- py[i] + oy[cand$a] * (hh + gap)
# A candidate must stay inside the plot and clear of every point and
# of every label placed so far
ok <- cx >= hw[i] & cx <= width - hw[i] & cy >= hh & cy <= height - hh &
rowSums(abs(outer(cx, px, "-")) < hw[i] + clearance &
abs(outer(cy, py, "-")) < hh + clearance) == 0
placed <- which(!is.na(lx))
if (length(placed)) {
ok <- ok & rowSums(abs(outer(cx, lx[placed], "-")) <
rep(hw[i] + hw[placed], each = length(cx)) &
abs(outer(cy, ly[placed], "-")) < 2 * hh) == 0
}
pick <- if (any(ok)) which(ok)[1] else 1
lx[i] <- cx[pick]
ly[i] <- cy[pick]
}
ex <- pmin(pmax(px, lx - hw), lx + hw) # nearest point of the label box
ey <- pmin(pmax(py, ly - hh), ly + hh)
data.frame(x = xlim[1] + lx / sx, y = ylim[1] + ly / sy,
end_x = xlim[1] + ex / sx, end_y = ylim[1] + ey / sy,
leader = sqrt((ex - px)^2 + (ey - py)^2) > clearance + step + 1)
}
# --- Interactive word map -----------------------------------------------
# Points coloured by group, each with its word label. The labels are laid
# out twice: for the plot area of the 720 x 540 px desktop figure, and for
# that of a 350 px wide phone figure, which is less than half as wide (the
# CSS for this post sets both aspect ratios). A snippet of JavaScript swaps
# in the phone layout, with larger text on phones that have room to spare,
# when the figure is drawn narrower than 500 px. It makes that choice again
# whenever the figure is resized, as when a phone or tablet is rotated.
word_map <- function(cd, groups, colours) {
names(colours) <- groups
phone_area <- c(280, 320)
pad_x <- diff(range(cd$PC1)) * 0.06
pad_y <- diff(range(cd$PC2)) * 0.06
xlim <- range(cd$PC1) + c(-pad_x, pad_x)
ylim <- range(cd$PC2) + c(-pad_y, pad_y)
wide <- repel_labels(cd$PC1, cd$PC2, cd$word, xlim, ylim,
width = 650, height = 424, font_px = 12)
narrow <- repel_labels(cd$PC1, cd$PC2, cd$word, xlim, ylim,
width = phone_area[1], height = phone_area[2],
font_px = 10)
# One segment per word, of zero length where the label sits by its point,
# so that both layouts give every group a path of the same shape
leader_path <- function(lab, rows) {
end_x <- ifelse(lab$leader, lab$end_x, cd$PC1)
end_y <- ifelse(lab$leader, lab$end_y, cd$PC2)
list(x = as.vector(rbind(cd$PC1[rows], end_x[rows], NA)),
y = as.vector(rbind(cd$PC2[rows], end_y[rows], NA)))
}
p <- ggplot(transform(cd, topic = factor(topic, levels = groups)),
aes(PC1, PC2, colour = topic,
text = paste0(word, " (", topic, ")"))) +
geom_point(size = 1.4) +
scale_colour_manual(values = colours) +
labs(x = "Principal Component 1", y = "Principal Component 2",
colour = NULL) +
theme_minimal(base_size = 12) +
theme(axis.text = element_text(size = 9),
axis.title = element_text(size = 10.5))
pp <- ggplotly(p, tooltip = "text")
# Leader lines and labels share their group's legend entry, so clicking
# a group in the legend hides or shows its points, lines and labels
for (grp in groups) {
rows <- cd$topic == grp
if (!any(rows)) next
wide_path <- leader_path(wide, rows)
narrow_path <- leader_path(narrow, rows)
pp <- pp %>% add_trace(
x = wide_path$x, y = wide_path$y, type = "scatter", mode = "lines",
line = list(color = colours[[grp]], width = 0.8), opacity = 0.6,
meta = list(narrow = list(x = I(narrow_path$x), y = I(narrow_path$y))),
legendgroup = grp, showlegend = FALSE, hoverinfo = "skip",
inherit = FALSE)
pp <- pp %>% add_trace(
x = wide$x[rows], y = wide$y[rows], type = "scatter", mode = "text",
text = cd$word[rows], textfont = list(size = 12, color = colours[[grp]]),
meta = list(narrow = list(x = I(narrow$x[rows]), y = I(narrow$y[rows]))),
cliponaxis = FALSE, legendgroup = grp, showlegend = FALSE,
hoverinfo = "text", hovertext = paste0(cd$word[rows], " (", grp, ")"),
inherit = FALSE)
}
pp <- pp %>%
layout(
legend = list(orientation = "h", x = 0.5, xanchor = "center",
y = 0, yref = "container", yanchor = "bottom",
itemsizing = "constant", font = list(size = 13)),
xaxis = list(tickmode = "auto", nticks = 6, range = xlim,
title = list(text = "Principal Component 1", standoff = 8)),
yaxis = list(tickmode = "auto", nticks = 6, range = ylim,
title = list(text = "Principal Component 2", standoff = 8)),
margin = list(t = 32, r = 8, b = 84)
) %>%
config(displaylogo = FALSE,
modeBarButtonsToRemove = c("select2d", "lasso2d"))
pp$x$config$modeBarButtonsToAdd <- NULL # hover-mode buttons plotly adds
htmlwidgets::onRender(pp, "
function(el, x, phoneArea) {
var traces = [], texts = [], wide = {x: [], y: []}, narrow = {x: [], y: []};
el.data.forEach(function(trace, i) {
if (!trace.meta || !trace.meta.narrow) return;
traces.push(i);
wide.x.push(trace.x);
wide.y.push(trace.y);
narrow.x.push(trace.meta.narrow.x);
narrow.y.push(trace.meta.narrow.y);
if (trace.mode === 'text') texts.push(i);
});
var current = 'wide 12';
function fit() {
var size = el._fullLayout._size;
var isWide = el.getBoundingClientRect().width >= 500;
var font = isWide ? 12 : Math.max(10, Math.min(12, Math.floor(10 *
Math.min(size.w / phoneArea[0], size.h / phoneArea[1]))));
var next = (isWide ? 'wide ' : 'narrow ') + font;
if (next === current) return;
current = next;
var xy = isWide ? wide : narrow;
Plotly.restyle(el, {x: xy.x, y: xy.y}, traces);
Plotly.restyle(el, {'textfont.size': font}, texts);
}
fit();
el.on('plotly_relayout', fit);
}", data = phone_area)
}
# --- Reusable helper: LSA + PCA plot ------------------------------------
# Builds a TF-IDF term-document matrix, computes a truncated SVD,
# selects the most distinctive and most shared words, and projects them
# to 2D via PCA *on the selected words only* for maximum spread.
lsa_pipeline <- function(doc_list, labels, grp_a, grp_b,
lab_a, lab_b, colour_a, colour_b,
top_n = 15, n_shared = 10,
k = 20, min_docs = 4) {
corp <- VCorpus(VectorSource(doc_list))
corp <- tm_map(corp, content_transformer(tolower))
corp <- tm_map(corp, removePunctuation)
corp <- tm_map(corp, removeNumbers)
corp <- tm_map(corp, removeWords, stopwords("en"))
corp <- tm_map(corp, stripWhitespace)
tdm <- as.matrix(TermDocumentMatrix(corp,
control = list(weighting = weightTfIdf,
bounds = list(global = c(min_docs, Inf)))))
k_use <- min(as.integer(k), nrow(tdm) - 1L, ncol(tdm) - 1L)
sv <- svd(tdm, nu = k_use, nv = k_use)
wlsa <- sv$u %*% diag(sv$d[1:k_use])
rownames(wlsa) <- rownames(tdm)
idx_a <- which(labels == grp_a)
idx_b <- which(labels == grp_b)
mean_a <- rowMeans(tdm[, idx_a, drop = FALSE])
mean_b <- rowMeans(tdm[, idx_b, drop = FALSE])
total <- mean_a + mean_b
spec <- mean_a - mean_b # positive = distinctive to A
top_a <- names(sort(spec, decreasing = TRUE))[1:top_n]
top_b <- names(sort(spec, decreasing = FALSE))[1:top_n]
shared_pool <- setdiff(names(sort(total, decreasing = TRUE)),
c(top_a, top_b))
shared <- head(shared_pool, n_shared)
hl <- unique(c(top_a, top_b, shared))
hl <- hl[hl %in% rownames(wlsa)]
# PCA on the selected words only, for better spatial spread
wlsa_hl <- wlsa[hl, , drop = FALSE]
pca <- prcomp(wlsa_hl, scale. = FALSE)
cd <- data.frame(PC1 = pca$x[, 1], PC2 = pca$x[, 2],
word = rownames(wlsa_hl))
cd$topic <- ifelse(cd$word %in% top_a & !cd$word %in% top_b, lab_a,
ifelse(cd$word %in% top_b & !cd$word %in% top_a, lab_b,
"Shared"))
# Trim spatial outliers so the dense cluster is readable.
# Words beyond the IQR fence are dropped from the plot (not from LSA).
q1 <- quantile(cd$PC1, 0.25); q3 <- quantile(cd$PC1, 0.75)
iqr <- q3 - q1; fence <- 2.5
keep <- cd$PC1 >= (q1 - fence * iqr) & cd$PC1 <= (q3 + fence * iqr)
q1y <- quantile(cd$PC2, 0.25); q3y <- quantile(cd$PC2, 0.75)
iqry <- q3y - q1y
keep <- keep & cd$PC2 >= (q1y - fence * iqry) & cd$PC2 <= (q3y + fence * iqry)
cd <- cd[keep, , drop = FALSE]
pp <- word_map(cd, groups = c(lab_a, lab_b, "Shared"),
colours = c(colour_a, colour_b, "#7B2D8E"))
# The full plotly bundle is about 3.5 MB, most of it chart types these figures
# do not use. partial_bundle() substitutes the smallest build that still covers
# the traces present. It fetches that build from the plotly CDN, so the render
# keeps the full bundle when it has no network access.
pp <- tryCatch(plotly::partial_bundle(pp), error = function(e) {
message('Keeping the full plotly bundle: ', conditionMessage(e))
pp
})
list(plot = pp, lsa = wlsa, tdm = tdm, pca = pca, words = cd)
}
# --- 1. Reuters newswire ------------------------------------------------
data(acq)
data(crude)
docs <- c(lapply(acq, content), lapply(crude, content))
labels <- c(rep("acq", length(acq)), rep("crude", length(crude)))
res1 <- lsa_pipeline(docs, labels,
grp_a = "acq", grp_b = "crude",
lab_a = "Finance", lab_b = "Energy",
colour_a = "#D55E00", colour_b = "#0072B2",
min_docs = 4)
# Cosine similarities in the 20-dimensional LSA space
pairs <- list(
c("oil", "barrel"), c("shares", "acquisition"),
c("price", "barrel"), c("price", "shares"),
c("shares", "oil"), c("acquisition", "barrel"))
pairs <- Filter(function(p) all(p %in% rownames(res1$lsa)), pairs)
sims <- sapply(pairs, function(p)
round(Cosine(p[1], p[2], tvectors = res1$lsa), 3))
names(sims) <- sapply(pairs, paste, collapse = " ~ ")
sims
#> oil ~ barrel shares ~ acquisition price ~ barrel
#> 0.675 0.236 0.938
#> price ~ shares shares ~ oil acquisition ~ barrel
#> 0.129 -0.014 -0.043| Group | PC1 | PC2 |
|---|---|---|
| Energy | .439 | -.321 |
| Finance | -.265 | .24 |
| Shared | .172 | .069 |
Figure 1: Word Vectors from Reuters Newswire Articles (Finance vs Energy) Projected to Two Dimensions via PCA on a 20-Dimensional LSA Space. Finance terms (vermilion) cluster in a distinct region from energy terms (blue); shared vocabulary occupies intermediate positions. Select an area of the plot to zoom in; double-click to reset.
The cosine similarities confirm what Figure 1 shows geometrically. The pairs oil ~ barrel and price ~ barrel have high positive cosines because these words habitually appear together in oil-market dispatches. Cross-domain pairs such as shares ~ oil and acquisition ~ barrel sit near zero, since these words seldom keep each other’s company. ‘Price’ could belong to either domain, yet price ~ shares is far lower than price ~ barrel. In this corpus, ‘price’ is far more common in reports on the oil market than in those on acquisitions, and its position in the space follows that dominant company, which is Firth’s principle made numerical.
State of the Union: Pre-War vs Post-War
The second corpus moves from newswire to politics. The sotu package (Arnold, 2022) provides the full text of every US State of the Union address from 1790 to 2020. Splitting the addresses at 1945, the end of the Second World War, shows how American political vocabulary has shifted from the constitutional and agrarian language of the early republic to the geopolitical and welfare-state vocabulary of the modern era. Table 2 and Figure 2 present the results.
if (!requireNamespace("sotu", quietly = TRUE)) install.packages("sotu")
sotu_texts <- sotu::sotu_text
sotu_years <- sotu::sotu_meta$year
sotu_labels <- ifelse(sotu_years < 1945, "Pre-1945", "Post-1945")
res2 <- lsa_pipeline(as.list(sotu_texts), sotu_labels,
grp_a = "Pre-1945", grp_b = "Post-1945",
lab_a = "Pre-1945", lab_b = "Post-1945",
colour_a = "#E69F00", colour_b = "#009E73",
min_docs = 5)| Group | PC1 | PC2 |
|---|---|---|
| Pre-1945 | -.845 | -.499 |
| Post-1945 | -.492 | .385 |
| Shared | -.379 | .132 |
Figure 2: Word Vectors from US State of the Union Addresses Projected to Two Dimensions, Split at 1945. Pre-war speeches (amber) feature constitutional and agrarian vocabulary; post-war speeches (green) shift to geopolitical and welfare-state terms. Select an area of the plot to zoom in; double-click to reset.
The two eras separate clearly, though not along the first component, on which both groups have negative mean loadings (Table 2). The separation lies on PC2, where pre-war words load negatively and post-war words positively, so the vertical axis of Figure 2 is the one that distinguishes the eras. Pre-war presidents address ‘gentlemen’, the formal salutation of a different era, and discuss ‘vessels’, ‘militia’, ‘commerce’ and ‘treasury’. This is the vocabulary of a young republic preoccupied with trade, territorial expansion and the mechanics of governance. Modern presidents speak of ‘jobs’, ‘budget’, ‘nuclear’ and ‘soviet’, the vocabulary of a superpower managing a welfare state and a global military presence. The most distinctive post-war word of all, ‘tonight’, lies outside the plotted region. It reflects the setting of the modern address, delivered in the evening to a national television audience. The purple ‘Shared’ words, such as ‘america’, ‘tax’, ‘spending’ and ‘workers’, carry heavy TF-IDF weights but rank just below the 15 most distinctive post-war words, and on PC2 they sit among the post-war vocabulary.
IMDB Film Reviews: Positive vs Negative
The third corpus sets a harder test. The text2vec package (Selivanov et al., 2025) includes 5,000 IMDB film reviews labelled as positive or negative, a classic sentiment-analysis benchmark. The groups in the two corpora above differed in topic. Here, positive and negative reviews alike discuss films, characters, plots and acting, and what sets them apart is mainly their adjectives and evaluative phrasing. That makes the two groups far harder for a simple co-occurrence model to separate, and the result, shown in Table 3 and Figure 3, is instructive.
if (!requireNamespace("text2vec", quietly = TRUE)) install.packages("text2vec")
data("movie_review", package = "text2vec")
mv_labels <- ifelse(movie_review$sentiment == 1, "Positive", "Negative")
res3 <- lsa_pipeline(as.list(movie_review$review), mv_labels,
grp_a = "Positive", grp_b = "Negative",
lab_a = "Positive", lab_b = "Negative",
colour_a = "#009E73", colour_b = "#D55E00",
min_docs = 50)| Group | PC1 | PC2 |
|---|---|---|
| Negative | .329 | -.255 |
| Positive | -.081 | .192 |
| Shared | .103 | .281 |
Figure 3: Word Vectors from 5,000 IMDB Film Reviews (Positive vs Negative) Projected to Two Dimensions. Unlike the clean topic-based separations in the Reuters and SOTU corpora, the sentiment-based distinction is much muddier: positive and negative reviews share most of their vocabulary, and evaluative words overlap heavily. Select an area of the plot to zoom in; double-click to reset.
What the Plots Capture and What They Miss
Taken together, Figures 1–3 illustrate both the power and the limits of distributional models. Figure 1 captures the real-world distinction between financial and energy markets with striking clarity. Domain-specific vocabulary clusters tightly, and a word such as ‘price’, which could belong to either market, is placed according to the company it predominantly keeps. This is the kind of structure that Louwerse and colleagues have documented at a larger scale. Figure 2 captures historical change, as the vocabulary of a young republic (‘gentlemen’, ‘militia’, ‘vessels’) gives way to that of a superpower (‘jobs’, ‘nuclear’, ‘soviet’) over two centuries of political evolution.
Figure 3, however, reveals a clear limitation. Positive and negative reviews discuss the same subject, so most of their vocabulary is shared, and the evaluative words that do separate them (‘excellent’ and ‘worst’, for instance) form only a thin layer atop that common vocabulary. A 20-dimensional LSA space lacks the resolution to untangle sentiment from topic, and the model captures what people write about more easily than how they feel about it. These imprecisions stem from a fundamental constraint on model capacity.
From Toy Models to Titans
The LSA spaces above used just 20 latent dimensions and were built from corpora of 70 to 5,000 documents, with vocabularies of a few hundred to about 11,000 words after filtering. Under these conditions, the model does a remarkable job of sorting finance from energy, or pre-war political language from post-war, but it lacks the capacity to encode the subtler distributional cues that distinguish evaluative tone, sarcasm or register.
The history of distributional models is, in large part, a history of scale. As Connell and Lynott (2024) illustrate, the amount of text used to train language models has grown from less than a single person encounters in a lifetime to far more than any human could. The LSA model of Landauer and Dumais (1997) had about 300 latent dimensions and was trained on some 30,000 encyclopaedia articles. That was already enough to match the average score of non-native applicants to US colleges on a synonym test. Word2Vec (Mikolov et al., 2013) moved to shallow neural networks trained on billions of words, with vocabularies of up to a million words. The Transformer-based models that followed took scale much further. BERT-Large (Devlin et al., 2019) had 340 million parameters, and GPT-3 (Brown et al., 2020) reached 175 billion. Today’s largest models are estimated at well over a trillion parameters, trained on text corpora so vast that they encompass a substantial fraction of everything ever written on the internet.
Through all of this, the core principle has stayed the same: each of these models learns about a word from the company it keeps. A model with 20 dimensions and a few hundred words can distinguish finance from energy. A model with billions of parameters and trillions of training tokens can distinguish a Shakespearean sonnet from a legal brief, track the implications of a subordinate clause across a 3,000-word passage and generate fluent prose in dozens of languages. Generative AI grew out of Firth’s old idea, scaled up by many orders of magnitude and combined with a decisive algorithmic innovation.
The Transformer Revolution
That innovation was the Transformer, introduced by Vaswani et al. (2017) in a paper titled ‘Attention Is All You Need’. The leading sequence models of the time relied on recurrent or convolutional neural networks. Recurrent networks read a sentence one word at a time while trying to hold everything so far in memory. The approach worked, but it was slow to train and struggled with long-range dependencies.
The Transformer replaced recurrence with multi-head self-attention, a mechanism that lets the model weigh every word in a passage simultaneously by comparing each one directly with every other. In plain terms, attention allows the model to ask, for each word, ‘Which other words here matter most for understanding me?’ Dispensing with recurrence and convolution entirely, the Transformer outperformed the best existing models on machine translation and generalised well to English constituency parsing. Because its computations run in parallel, it also took far less time to train.
With Transformers in hand, NLP entered a new era. Large pretrained models such as BERT (Devlin et al., 2019) and the GPT series (Brown et al., 2020) set successive benchmarks for language understanding and generation. It was the Transformer architecture, combined with the massive scale described above, that made generative AI possible. The thread runs unbroken from Firth’s insight about co-occurrence, through the matrix decompositions of LSA and the neural embeddings of Word2Vec, to the attention-powered behemoths of today. For all their power, these models remain predictors of text. The Transformer revolution made the storyteller more eloquent without making it more honest.
Fluency Is Not Truth
LLMs are optimised for fluency. They have no built-in fact-checking and simply predict plausible continuations. As Radford et al. (2019) describe for GPT-2, the training objective is to predict the next token in a sequence from all the tokens that precede it. That objective rewards fluent, likely text, and nothing in it rewards the model for replying ‘I don’t know’. Using their TruthfulQA benchmark, Lin et al. (2022) found that models generated many false answers that mimic popular misconceptions. The largest models were generally the least truthful, which is what one would expect if models learn false answers from their training data and larger models imitate that data more closely. Models therefore tend to guess when unsure, and they guess with alarming confidence.
Ask an LLM about a niche historical event, and it may cheerfully invent plausible-sounding dates, names and citations. Ask it about a scientific finding at the edges of its training data, and it may blend two real studies into one fictional hybrid, complete with a convincing journal name. This phenomenon, known as hallucination, cannot be patched away entirely. Z. Xu et al. (2024) proved, in a formal setting, that any computable LLM used as a general problem solver will inevitably hallucinate, because no such model can learn every computable function. Since that formal setting is part of the real world, they argue, the same limit applies to real-world LLMs. The model’s fluency thus becomes its greatest liability, as it weaves a convincing narrative whether or not the facts support it.
Why Prompts Matter, and Why One Is Rarely Enough
Because LLMs work by prediction, prompt engineering is essential. A vague or generic question will often yield a superficial, off‑target or simply wrong answer. A guide from Google Cloud defines prompt engineering as ‘the art and science of designing and optimizing prompts to guide AI models, particularly LLMs, towards generating the desired responses’. That sounds rather grand, but it often comes down to something as prosaic as adding context, specifying a format and giving an example or two, then refining the prompt until the output is useful.
The sensitivity of LLMs to phrasing is remarkable and, on first encounter, humbling. Asking ‘What are some criticisms of capitalism?’ and ‘What are the main drawbacks of market economies?’ can elicit strikingly different responses, even though the two questions are conceptually near‑identical. Changes that leave the meaning untouched can matter too. Sclar et al. (2024) varied only the formatting of few-shot prompts, such as separators, spacing and capitalisation, and found accuracy differences of up to 76 points on the same task. They recommend that evaluations report performance across a range of plausible prompt formats. Wang et al. (2024) asked several LLMs the same clinical guideline questions in different prompt styles, five times each. The effect of a prompt style varied from one model to another, and answers to repeated questions were not always consistent. A single query is seldom enough.
There is also the matter of role, tone and constraints. Instructing the model to respond as a sceptical scientist, a sympathetic teacher or a meticulous copy‑editor changes its behaviour markedly. Asking it to respond in plain English, to avoid jargon, to stay under 150 words or to number its assumptions shapes the answer in ways a bare question never could. In Firth’s terms, each of these additions is part of the ‘company’ the prompt keeps, and so part of what determines the model’s response.
What Good Prompting Looks Like
A search engine returns results on demand, whereas an LLM works best as a collaborator given careful, iterative guidance. Provide ample background in your prompt, and invite the model to ask about anything unclear before it responds. Regard the first reply as a draft. Push back on suspect claims, request sources or alternative views, and rephrase when the model goes off track. Ask it, too, to explain its reasoning, to consider counter-arguments and to flag what it is uncertain about. Each of these moves draws more of the model’s latent capability to the surface.
This iterative approach mirrors good intellectual practice more generally. Scientists replicate their experiments, vary the conditions and triangulate across methods before they publish. Journalists seek corroboration beyond a single source, and doctors gather a fuller picture before settling on a diagnosis. Using an LLM well calls for the same instinct, treating each exchange as one data point in an ongoing investigation.
A Powerful Tool, Not an Oracle
Using an LLM is a bit like navigating a foreign city without a map. You will stumble upon useful places, but you will also take wrong turns, end up in dead ends and occasionally find yourself heading confidently in exactly the wrong direction. These models often produce accurate information because language encodes much of reality. Words cluster around what they describe, and texts about geography, commodity markets or sensory properties track how the world works (Louwerse, 2011; Louwerse & Zwaan, 2009). Yet an LLM is trained to be fluent, and some hallucination is formally inevitable (Z. Xu et al., 2024). The underlying mechanism is still word co-occurrence, Firth’s old principle scaled up. Neither the brute force of massive training data (Connell & Lynott, 2024) nor the ingenious attention mechanisms of modern architectures (Vaswani et al., 2017) has yet tamed this heuristic machine into a reliable truth-teller.
Getting good results takes deliberate effort. A well-crafted prompt will not turn the model into a truth engine, even with specific context, clear constraints, iterative refinement and healthy scepticism behind it. What it does is steer the model’s predictions towards the regions of language that most faithfully reflect the world. Without that effort, reaching the right destination is largely a matter of luck.
Firth’s insight about words applies equally to prompts: you shall know an answer by the company the question keeps (He et al., 2024; Wang et al., 2024).
References
Arnold, T. B. (2022). sotu: United States presidential State of the Union addresses (Version 1.0.4) [Computer software]. CRAN. https://doi.org/10.32614/CRAN.package.sotu
Bernabeu, P. (2022). Language and sensorimotor simulation in conceptual processing: Multilevel analysis and statistical power [Doctoral thesis, Lancaster University]. https://doi.org/10.17635/lancaster/thesis/1795
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (Vol. 33, pp. 1877–1901). https://doi.org/10.48550/arXiv.2005.14165
Brunila, M., & LaViolette, J. (2022). What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris. In Proceedings of NAACL 2022 (pp. 4403–4417). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.327
Connell, L., & Lynott, D. (2024). What can language models tell us about human cognition? Current Directions in Psychological Science, 33(3), 181–189. https://doi.org/10.1177/09637214241242746
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
Feinerer, I., Hornik, K., & Meyer, D. (2008). Text mining infrastructure in R. Journal of Statistical Software, 25(5), 1–54. https://doi.org/10.18637/jss.v025.i05
Firth, J. R. (1957). A synopsis of linguistic theory, 1930–1955. In Studies in linguistic analysis (pp. 1–32). Basil Blackwell.
Günther, F., Dudschig, C., & Kaup, B. (2015). LSAfun: An R package for computations based on latent semantic analysis. Behavior Research Methods, 47(4), 930–944. https://doi.org/10.3758/s13428-014-0529-0
He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., & Hasan, S. (2024). Does prompt formatting have any impact on LLM performance? arXiv. https://doi.org/10.48550/arXiv.2411.10541
Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240. https://doi.org/10.1037/0033-295X.104.2.211
Lewis, D. D. (1997). Reuters-21578 text categorization test collection, distribution 1.0 [Data set]. AT&T Labs–Research. http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html
Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 3214–3252). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.229
Louwerse, M. M. (2011). Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, 3(2), 273–302. https://doi.org/10.1111/j.1756-8765.2010.01106.x
Louwerse, M., & Connell, L. (2011). A taste of words: Linguistic context and perceptual simulation predict the modality of words. Cognitive Science, 35(2), 381–398. https://doi.org/10.1111/j.1551-6709.2010.01157.x
Louwerse, M. M., & Zwaan, R. A. (2009). Language encodes geographical information. Cognitive Science, 33(1), 51–73. https://doi.org/10.1111/j.1551-6709.2008.01003.x
Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. https://doi.org/10.3758/BF03204766
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv. https://doi.org/10.48550/arXiv.1301.3781
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In Proceedings of ICLR 2024. https://doi.org/10.48550/arXiv.2310.11324
Selivanov, D., Bickel, M., & Wang, Q. (2025). text2vec: Modern text mining framework for R (Version 0.6.6) [Computer software]. CRAN. https://doi.org/10.32614/CRAN.package.text2vec
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). https://doi.org/10.48550/arXiv.1706.03762
Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024). Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. npj Digital Medicine, 7, Article 41. https://doi.org/10.1038/s41746-024-01029-4
Wu, M., Conde, J., Reviriego, P., & Brysbaert, M. (2026). How does fine-tuning improve sensorimotor representations in large language models? arXiv. https://doi.org/10.48550/arXiv.2603.03313
Xu, Q., Peng, Y., Nastase, S. A., Chodorow, M., Wu, M., & Li, P. (2025). Large language models without grounding recover non-sensorimotor but not sensorimotor features of human concepts. Nature Human Behaviour, 9(9), 1871–1886. https://doi.org/10.1038/s41562-025-02203-8
Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is inevitable: An innate limitation of large language models. arXiv. https://doi.org/10.48550/arXiv.2401.11817
Comments are provided by Disqus and are not loaded automatically. Loading them connects your browser to Disqus, which may use cookies and process data under its privacy policy. See this site's privacy notice for details.