Designing queries¶
A retrieval is only as good as the query behind it. This guide shows how to compose correct, field-tagged Scopus queries with scopus_query, which spares you pasting fragments together by hand, where a missing bracket or a mistyped tag quietly returns the wrong records. Everything here is string construction, so it all runs offline without an API key. Each call returns the query as a literal string, which means you can read it before you ever spend quota on it. A count, a trend or a harvest sends that string with any year limit appended, as From a query to a plan shows.
import scopusflow as sf
Field tags decide where to look¶
A field tag restricts a query to one part of a record, so a search for a topic does not drown in incidental full-text matches. The FIELD_TAGS dictionary maps the common tags to a short description of what each one searches.
out([f"{tag:<20} {meaning}" for tag, meaning in sf.FIELD_TAGS.items()])
TITLE Words in the document title
TITLE-ABS-KEY Title, abstract and keywords
TITLE-ABS-KEY-AUTH Title, abstract, keywords and author names
ABS Abstract text
KEY Indexed and author keywords
AUTH Author names
AUTHKEY Author-supplied keywords
AFFIL Affiliation, any part
AFFILORG Affiliation organisation name
SRCTITLE Source (publication) title
DOI Digital Object Identifier
ALL All available fields
The most generally useful tag is TITLE-ABS-KEY, which searches the title, abstract and keywords together. It is broad enough to catch a topic without the noise of a full-text match. Because FIELD_TAGS is an ordinary dictionary, you can also check membership before building a query, for example "AUTHKEY" in sf.FIELD_TAGS.
Wrapping a single term¶
wrap_field applies one tag to one term and is the building block the rest of the API uses. Passing None as the field leaves the term untouched, which is the right choice when the term already carries its own tag.
out(sf.wrap_field("graphene", "TITLE-ABS-KEY"))
out(sf.wrap_field("graphene", None))
TITLE-ABS-KEY(graphene)
graphene
The tag is upper-cased and validated, so a lower-case or stray entry is normalised or rejected, and never passed through untouched. A tag that contains anything other than letters and hyphens raises a ValueError, which catches a typo at the point of construction.
out(sf.wrap_field("graphene", "title-abs-key")) # upper-cased to TITLE-ABS-KEY
try:
sf.wrap_field("graphene", "TITLE_ABS") # underscore is not allowed
except ValueError as exc:
print(exc)
TITLE-ABS-KEY(graphene)
Invalid field tag 'TITLE_ABS'; use letters and hyphens only.
One term, many disciplines¶
scopus_query builds on wrap_field and serves any field. Each call below returns the exact query string, with the field argument applied to every term.
out([
# molecular biology
sf.scopus_query("CRISPR", field="TITLE-ABS-KEY"),
sf.scopus_query("gravitational waves", field="TITLE-ABS-KEY"), # physics
# environmental science
sf.scopus_query("microplastics", field="TITLE-ABS-KEY"),
# computer science
sf.scopus_query("blockchain", field="TITLE-ABS-KEY"),
sf.scopus_query("digital humanities", field="AUTHKEY"), # humanities
])
TITLE-ABS-KEY(CRISPR)
TITLE-ABS-KEY(gravitational waves)
TITLE-ABS-KEY(microplastics)
TITLE-ABS-KEY(blockchain)
AUTHKEY(digital humanities)
The last example uses AUTHKEY, the author-supplied keywords, which isolates work that self-identifies with a field and so cuts incidental mentions.
Combining terms with boolean operators¶
Passing several terms joins them into one query. The default operator is AND, and OR or AND NOT are available through the op argument. The same field is wrapped around each term before the terms are joined. Without a field, a term of several words is put in brackets, so each term stays one operand of the join (see Phrases below).
# Two concepts that must co-occur (materials science).
out(sf.scopus_query("perovskite", "solar cell", field="TITLE-ABS-KEY"))
# Two names for one condition, either of which will do (medicine).
out(sf.scopus_query('"heart attack"', '"myocardial infarction"', op="OR"))
# A family of related tools (molecular biology).
out(sf.scopus_query("CRISPR", "Cas9", "Cas12", op="OR"))
# Exclude a neighbouring literature (medicine).
out(sf.scopus_query(
"hypertension", "pulmonary", op="AND NOT", field="TITLE-ABS-KEY"
))
TITLE-ABS-KEY(perovskite) AND TITLE-ABS-KEY(solar cell)
"heart attack" OR "myocardial infarction"
CRISPR OR Cas9 OR Cas12
TITLE-ABS-KEY(hypertension) AND NOT TITLE-ABS-KEY(pulmonary)
An operator outside the permitted set raises a ValueError, so op="NOT" or a typo is caught before the string is built, well short of a rejection from the API.
Phrases¶
Scopus joins the words of an unquoted term with AND, so heart attack finds records that mention both words anywhere in the searched field. Double quotation marks ask for a loose phrase, whose words must sit next to each other, although punctuation is ignored and plurals are included. Braces ask for an exact phrase, including any stop words, spaces and punctuation. Elsevier's search tips describe all three.
out([
sf.scopus_query('"heart attack"', field="TITLE-ABS-KEY"), # loose phrase
sf.scopus_query("{heart attack}", field="TITLE-ABS-KEY"), # exact phrase
])
TITLE-ABS-KEY("heart attack")
TITLE-ABS-KEY({heart attack})
scopus_query leaves both kinds of phrase as written. When terms are joined without a field tag, a term of several unquoted words is put in brackets, so it stays one operand of the join. Joined bare, the first query below would read as machine AND (learning OR deep) AND learning.
out([
sf.scopus_query("machine learning", "deep learning", op="OR"),
sf.scopus_query('"machine learning"', '"deep learning"', op="OR"),
])
(machine learning) OR (deep learning)
"machine learning" OR "deep learning"
Brackets in a query written by hand¶
A query that mixes operators needs brackets to say what it means. The search tips state that Scopus applies OR first, then AND, then AND NOT, so A AND NOT B AND C reads as A AND NOT (B AND C). Elsevier has announced a new order, with AND NOT first, then AND, then OR (Feldner, 2025), under which A OR B AND C reads as A OR (B AND C). Brackets give the same reading under either order.
The package brackets the strings it composes. A query built by an earlier scopus_query call keeps its own join together when it is joined again, as below, and compare_topics and scopus_intersections bracket each side of their joins. Plans, counts and trends append the year limit to the query, bracketing the query first wherever an operator would otherwise take the limit from part of it, as From a query to a plan shows.
young = sf.scopus_query("children", "adolescents", op="OR")
out(sf.scopus_query("vaccine", young))
vaccine AND (children OR adolescents)
Searching by affiliation¶
Field tags reach beyond topics. AFFILORG searches the affiliation organisation name, which turns a query into an institution-level view of output.
out(sf.scopus_query("Max Planck", field="AFFILORG"))
AFFILORG(Max Planck)
When a term is empty¶
The builder validates its input, so a stray empty term is caught at construction, well before it can produce a malformed query that fails downstream. An empty or whitespace-only term raises a ValueError.
try:
sf.scopus_query("graphene", "")
except ValueError as exc:
print(exc)
All terms must be non-empty.
Checks before any request¶
SearchPlan, scopus_count, scopus_trend, compare_topics and scopus_intersections check each query offline before anything is sent, and refuse three mistakes. Refinement syntax copied from the web interface, such as LIMIT-TO(DOCTYPE, "ar"), is ignored by the Search API, so the search would run unfiltered with no sign of a problem. A field tag wrapped around a query that already opens with one is rejected by the API, and wrap_field refuses it too. A bracket, quotation mark or brace left unclosed changes what the rest of the query means. Each raises a ValueError whose message says what to write instead. The messages are those of the R twin, word for word.
import textwrap
try:
sf.SearchPlan('working memory AND (LIMIT-TO (DOCTYPE, "ar"))')
except ValueError as exc:
print(textwrap.fill(str(exc), 76))
The query contains LIMIT-TO() or EXCLUDE(), which belong to the Scopus web interface's refinement panel. The Search API does not apply them, so the search would run unfiltered (pybliometrics documents LIMIT-TO() as affecting the display only). Write the limit as a search field instead, such as DOCTYPE(ar), LANGUAGE(english), SUBJAREA(PSYC), SRCTYPE(j) or PUBYEAR > 2015.
Written as search fields, the same limits pass the check.
out(sf.SearchPlan("working memory AND DOCTYPE(ar) AND LANGUAGE(english)").wrapped_query)
working memory AND DOCTYPE(ar) AND LANGUAGE(english)
A strategy pasted from the advanced search with its own tags, such as TITLE-ABS-KEY ( "working memory" ), is sent as written when field is left as None. INDEXTERMS(), which pybliometrics documents as not working through the Search API, issues a scopusflow.query.QuerySyntaxWarning, and KEY() searches the index terms together with author keywords, trade names and chemical names.
From a query to a plan¶
A composed query drops straight into the rest of the workflow. The same string anchors a SearchPlan, and partitioning by year fetches and checkpoints it one year at a time. Note that the plan can apply the field tag itself through its own field argument, so you pass the bare topic and let the plan wrap it once.
plan = sf.SearchPlan(
"gut microbiome",
years=range(2015, 2023),
field="TITLE-ABS-KEY",
partition="year",
)
# the expression every cell starts from
out(plan.wrapped_query)
out([(c.cell, c.year) for c in plan.cells()]) # one cell per year
TITLE-ABS-KEY(gut microbiome)
[(1, 2015), (2, 2016), (3, 2017), (4, 2018), (5, 2019), (6, 2020), (7, 2021), (8, 2022)]
Each cell sends that expression with its own year appended, so the first cell here asks for TITLE-ABS-KEY(gut microbiome) AND PUBYEAR IS 2015. An expression with a top-level OR, AND NOT or proximity operator is put in brackets before the year is added, so the limit applies to all of it. Any other expression is sent exactly as shown. The R twin sends the year as a separate request parameter, so its query strings never carry the limit.
Sizing and running the plan both contact the Scopus API, so the two calls below need your key and are the only step here that goes online. pybliometrics keeps the key in its configuration file, ~/.config/pybliometrics.cfg, and reads it only when pybliometrics.init() is called, once in every session. scopus_count reports how many records the query matches without downloading them, which is the cheap way to check a search before committing quota to it.
import pybliometrics
pybliometrics.init() # once per session, before the first search
sf.scopus_count(
"gut microbiome", years=range(2015, 2023), field="TITLE-ABS-KEY"
)
records = sf.fetch_plan(plan)
Checking recall against known records¶
A search strategy can be tested against records already known to be relevant, for instance the studies included in an earlier review or those named by experts, collected without the strategy's help. The share of them the strategy retrieves is its relative recall (Sampson et al., 2006), and Bramer et al. (2018) make the check a step in developing a search. It takes two searches per chain of DOIs, run under the same year limit. The chain alone finds which known records Scopus indexes, and the strategy joined to the chain finds which of those it retrieves. A known record the chain does not find, because Scopus does not index it or holds it without that DOI, cannot show whether the strategy finds it. It is left out of the denominator and listed, so it can be checked by hand.
The sketch below contacts the API, so it is not run here. The DOIs are placeholders for your own, and extract_dois cleans them first, so a resolver prefix or a difference in case does no harm. Chains of at most 25 DOIs keep each request short. How the API treats much longer chains has not been checked.
known = sf.extract_dois(
["10.xxxx/known-1", "https://doi.org/10.xxxx/known-2", "doi:10.xxxx/known-3"]
)
strategy = sf.scopus_query("working memory", "training", field="TITLE-ABS-KEY")
years = range(2010, 2025)
chains = [
" OR ".join(f'DOI("{doi}")' for doi in known[i:i + 25])
for i in range(0, len(known), 25)
]
def found(query):
records = sf.fetch_plan(sf.SearchPlan(query, years=years))
return {doi.lower() for doi in sf.extract_dois(records)}
indexed = set().union(*(found(chain) for chain in chains))
retrieved = set().union(*(found(f"({strategy}) AND ({chain})") for chain in chains))
not_found = sorted({doi.lower() for doi in known} - indexed)
relative_recall = len(retrieved) / len(indexed)
The figure describes the strategy against this reference set alone, so report the set, its source and the DOIs left out alongside it.