Skip to content

API reference

Every public name in scopusflow is documented here, grouped along the path a search takes: describe it, size it, run it as a resumable harvest, normalise what comes back, see what a later retrieval changed, and summarise and plot the result. The groups are the ones the R package's reference index uses, and they come in the same order, so a function documented in both languages is filed under the same heading on either site, except where the next paragraph says otherwise.

One group name differs from the R one because the membership differs. The R group 'Plan and size' also holds the counting function, which is documented under Retrieve here alongside the other calls that spend quota, so the first group is called 'Plan and query' instead. Records is wider than its R namesake, because it absorbs the R groups 'Export and I/O' and 'Data'. The record writer and reader, the BibTeX and RIS writers and the bundled example harvest all work on the record table, and the corpus writers on a corpus built from it. Reporting holds the same one function in both languages. The two remaining R groups have no counterpart below. 'App' documents a function that launches the interface. The Python app is started from the command line as scopusflow-gui, or from Python with scopusflow.app.launch(), which takes the host and port to serve on, whether to open a browser tab and whether to reload on code changes, and The code-free app covers both. 'Keys' documents a key check that scopusflow does not need in Python, since pybliometrics holds the key configuration.

Every name with an entry of its own under a group heading is importable straight from scopusflow, conventionally as sf. The guides linked from the home page work the same functions through end-to-end examples.

Plan and query

Describe a search before running it, and build field-tagged Boolean queries.

scopus_query

scopus_query(*terms, op='AND', field=None)

Combine terms into one Scopus query, optionally field-wrapping each.

Each term is trimmed and, given field, wrapped in that tag, whose brackets hold the term together. Without a field tag, a term of several words is put in brackets before two or more terms are joined, because Scopus joins the words of an unquoted term with AND. Joined bare, machine learning OR deep learning would read as machine AND (learning OR deep) AND learning. scopus_query("machine learning", "deep learning", op="OR") therefore gives (machine learning) OR (deep learning). A term that is already one operand is left as written: a quoted "..." phrase, a braced {...} phrase, a bracketed group or a single field-tagged group such as TITLE(deep learning). Single words are never bracketed, so scopus_query("CRISPR", "Cas9", op="OR") stays CRISPR OR Cas9, and a query composed by an earlier call keeps its meaning when it is joined again.

Parameters:

Name Type Description Default
*terms str

One or more non-empty search terms.

()
op str

The boolean operator joining the terms: "AND", "OR" or "AND NOT".

'AND'
field str | None

An optional field tag applied to every term (see FIELD_TAGS).

None

Returns:

Type Description
str

The combined query.

Raises:

Type Description
ValueError

For an op outside the three, for no terms or an empty one and for a field that is not a tag of letters and hyphens. Given field, a term that already opens with a field tag, such as TITLE(x), raises it too, because the API rejects one tag nested inside another (see wrap_field).

Source code in src/scopusflow/query.py
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
def scopus_query(*terms: str, op: str = "AND", field: str | None = None) -> str:
    """Combine terms into one Scopus query, optionally field-wrapping each.

    Each term is trimmed and, given ``field``, wrapped in that tag, whose
    brackets hold the term together. Without a field tag, a term of several
    words is put in brackets before two or more terms are joined, because
    Scopus joins the words of an unquoted term with AND. Joined bare,
    ``machine learning OR deep learning`` would read as
    ``machine AND (learning OR deep) AND learning``.
    ``scopus_query("machine learning", "deep learning", op="OR")`` therefore
    gives ``(machine learning) OR (deep learning)``. A term that is already one
    operand is left as written: a quoted ``"..."`` phrase, a braced ``{...}``
    phrase, a bracketed group or a single field-tagged group such as
    ``TITLE(deep learning)``. Single words are never bracketed, so
    ``scopus_query("CRISPR", "Cas9", op="OR")`` stays ``CRISPR OR Cas9``, and a
    query composed by an earlier call keeps its meaning when it is joined again.

    Parameters
    ----------
    *terms:
        One or more non-empty search terms.
    op:
        The boolean operator joining the terms: ``"AND"``, ``"OR"`` or ``"AND NOT"``.
    field:
        An optional field tag applied to every term (see
        [`FIELD_TAGS`][scopusflow.query.FIELD_TAGS]).

    Returns
    -------
    str
        The combined query.

    Raises
    ------
    ValueError
        For an ``op`` outside the three, for no terms or an empty one and for
        a ``field`` that is not a tag of letters and hyphens. Given ``field``,
        a term that already opens with a field tag, such as ``TITLE(x)``,
        raises it too, because the API rejects one tag nested inside another
        (see [`wrap_field`][scopusflow.query.wrap_field]).
    """
    if op not in {"AND", "OR", "AND NOT"}:
        raise ValueError("op must be one of 'AND', 'OR', 'AND NOT'.")
    cleaned = [t.strip() for t in terms]
    if not cleaned or any(not t for t in cleaned):
        raise ValueError("All terms must be non-empty.")
    if field is None and len(cleaned) > 1:
        cleaned = [_group_term(t) for t in cleaned]
    return f" {op} ".join(wrap_field(t, field) for t in cleaned)

wrap_field

wrap_field(query, field)

Wrap query in a field tag, e.g. TITLE-ABS-KEY(graphene).

field=None returns query unchanged. Otherwise, field is stripped and upper-cased, and a tag of anything but letters and hyphens raises ValueError.

A query that already opens with a field tag raises ValueError, because the API rejects one tag nested inside another, as in TITLE-ABS-KEY(TITLE(x)).

Source code in src/scopusflow/query.py
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
def wrap_field(query: str, field: str | None) -> str:
    """Wrap ``query`` in a field tag, e.g. ``TITLE-ABS-KEY(graphene)``.

    ``field=None`` returns ``query`` unchanged. Otherwise, ``field`` is stripped
    and upper-cased, and a tag of anything but letters and hyphens raises
    ``ValueError``.

    A query that already opens with a field tag raises ``ValueError``, because
    the API rejects one tag nested inside another, as in
    ``TITLE-ABS-KEY(TITLE(x))``.
    """
    if field is None:
        return query
    field = field.strip().upper()
    if not _FIELD_RE.match(field):
        raise ValueError(f"Invalid field tag {field!r}; use letters and hyphens only.")
    if _opens_with_tag(query):
        raise ValueError(_QUERY_MESSAGES["double_field"])
    return f"{field}({query})"

FIELD_TAGS module-attribute

FIELD_TAGS = {'TITLE': 'Words in the document title', 'TITLE-ABS-KEY': 'Title, abstract and keywords', 'TITLE-ABS-KEY-AUTH': 'Title, abstract, keywords and author names', 'ABS': 'Abstract text', 'KEY': 'Indexed and author keywords', 'AUTH': 'Author names', 'AUTHKEY': 'Author-supplied keywords', 'AFFIL': 'Affiliation, any part', 'AFFILORG': 'Affiliation organisation name', 'SRCTITLE': 'Source (publication) title', 'DOI': 'Digital Object Identifier', 'ALL': 'All available fields'}

SearchPlan dataclass

A fully specified, inspectable description of a Scopus search.

Splitting describing a search from executing it makes a workflow reproducible and lets a large retrieval be partitioned by year, so it can be cached and resumed.

years are the publication years. With partition="year" each distinct year is a cell of its own, so years=[2015, 2020] searches those two years alone, as scopus_trend and compare_topics do. With partition="none" the single cell sends one inclusive range from the earliest year to the latest, so years=[2015, 2020] covers every year from 2015 to 2020, as in scopus_count and scopus_intersections. years must be whole numbers from 1700 to 2200, or None for no year limit, and they are stored as a sorted tuple without repeats.

page_size is the number of records asked for per request, None (the default) meaning the largest the view allows: 200 for STANDARD, 25 for COMPLETE. It is stored on the plan, and sent, because the search record scopus_search_report writes has to state how the harvest was paged, and a figure taken from anywhere but the plan would be a guess.

view is the Scopus Search view: "STANDARD" (the default) or "COMPLETE", which also carries the author list and the author keywords (see to_records) but in pages of at most 25 records. partition is "none" (the default) or "year". Any other view or partition raises ValueError, and so does partition="year" without years.

field is stripped and upper-cased when the plan is built, and a value that is not a string of letters and hyphens raises ValueError. The query is checked offline at the same time: LIMIT-TO() or EXCLUDE() pasted from the web interface's refinement panel, a field tag around a query that already opens with one, and unbalanced brackets, quotation marks or braces raise ValueError, and INDEXTERMS() issues a scopusflow.query.QuerySyntaxWarning.

Source code in src/scopusflow/plan.py
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
@dataclass(frozen=True)
class SearchPlan:
    """A fully specified, inspectable description of a Scopus search.

    Splitting *describing* a search from *executing* it makes a workflow
    reproducible and lets a large retrieval be partitioned by year, so it can be
    cached and resumed.

    ``years`` are the publication years. With ``partition="year"`` each
    distinct year is a cell of its own, so ``years=[2015, 2020]`` searches
    those two years alone, as [`scopus_trend`][scopusflow.trend.scopus_trend]
    and [`compare_topics`][scopusflow.compare.compare_topics] do. With
    ``partition="none"`` the single cell sends one inclusive range from the
    earliest year to the latest, so ``years=[2015, 2020]`` covers every year
    from 2015 to 2020, as in
    [`scopus_count`][scopusflow.count.scopus_count] and
    [`scopus_intersections`][scopusflow.intersections.scopus_intersections].
    ``years`` must be whole numbers from 1700 to 2200, or ``None`` for no year
    limit, and they are stored as a sorted tuple without repeats.

    ``page_size`` is the number of records asked for per request, ``None``
    (the default) meaning the largest the view allows: 200 for ``STANDARD``,
    25 for ``COMPLETE``. It is stored on the plan, and sent, because the search
    record [`scopus_search_report`][scopusflow.report.scopus_search_report]
    writes has to state how the harvest was paged, and a figure taken from
    anywhere but the plan would be a guess.

    ``view`` is the Scopus Search view: ``"STANDARD"`` (the default) or
    ``"COMPLETE"``, which also carries the author list and the author keywords
    (see [`to_records`][scopusflow.records.to_records]) but in pages of at
    most 25 records. ``partition`` is ``"none"`` (the default) or ``"year"``.
    Any other ``view`` or ``partition`` raises ``ValueError``, and so does
    ``partition="year"`` without ``years``.

    ``field`` is stripped and upper-cased when the plan is built, and a value
    that is not a string of letters and hyphens raises ``ValueError``. The
    query is checked offline at the same time: LIMIT-TO() or EXCLUDE() pasted
    from the web interface's refinement panel, a field tag around a query that
    already opens with one, and unbalanced brackets, quotation marks or braces
    raise ``ValueError``, and INDEXTERMS() issues a
    ``scopusflow.query.QuerySyntaxWarning``.
    """

    query: str
    years: Sequence[int] | None = None
    field: str | None = None
    view: str = "STANDARD"
    partition: str = "none"  # "none" or "year"
    page_size: int | None = None

    def __post_init__(self) -> None:
        if not self.query or not self.query.strip():
            raise ValueError("query must be a non-empty string.")
        # Normalised here, as the R twin's scopus_plan() normalises it, so
        # " title-abs-key " and "TITLE-ABS-KEY", which send the same query,
        # compare equal and the search record prints the canonical tag.
        object.__setattr__(self, "field", _check_field(self.field))
        # Offline checks, so a query the API cannot run as written is refused
        # before fetch_plan() spends a request on it. stacklevel 4 reaches the
        # caller of SearchPlan() through __init__ and __post_init__.
        _check_query(self.query, self.field, stacklevel=4)
        if self.view not in {"STANDARD", "COMPLETE"}:
            raise ValueError("view must be 'STANDARD' or 'COMPLETE'.")
        if self.partition not in {"none", "year"}:
            raise ValueError("partition must be 'none' or 'year'.")
        object.__setattr__(self, "page_size",
                           _check_page_size(self.page_size, self.view))
        # Store the validated integers, never the values as passed: cells() renders the
        # year into the cell's date, and str(2015.0) would reach the API as
        # "2015.0". A tuple, since a list would leave the frozen dataclass unhashable.
        # object.__setattr__ is how a frozen dataclass normalises a field.
        #
        # Sorted and de-duplicated, as every other caller of _check_years already
        # does with its result and as cells() does with this one, so that the
        # stored years say what the search will actually do. Without it two plans
        # describing the same search compared unequal on the order the years
        # happened to be typed in, and the reproduction snippet
        # scopus_search_report() emits (which renders them canonically) rebuilt a
        # plan that ran identically but failed an equality check against its own
        # original.
        checked = _check_years(self.years)
        object.__setattr__(self, "years",
                           None if checked is None else tuple(sorted(set(checked))))
        if self.partition == "year" and not self.years:
            raise ValueError("partition='year' requires years.")

    def to_dict(self) -> dict:
        """The plan as a plain dict, which
        [`from_dict`][scopusflow.plan.SearchPlan.from_dict] turns back into it.

        This is the form [`fetch_plan`][scopusflow.fetch.fetch_plan] attaches
        as ``attrs["plan"]``. A dataclass instance cannot be written as JSON,
        and pandas 2.1 and later write a frame's attributes as JSON when it is
        saved to parquet, so a harvest carrying the plan itself could not be
        saved. ``schema`` numbers the layout, and the years are a sorted list.

        Examples
        --------
        >>> import scopusflow as sf
        >>> plan = sf.SearchPlan("graphene", years=[2020, 2019], partition="year")
        >>> plan.to_dict()["years"]
        [2019, 2020]
        >>> sf.SearchPlan.from_dict(plan.to_dict()) == plan
        True
        """
        return {
            "schema": _PLAN_SCHEMA,
            "query": self.query,
            "years": None if self.years is None else list(self.years),
            "field": self.field,
            "view": self.view,
            "partition": self.partition,
            "page_size": int(self.page_size),  # type: ignore[arg-type]
        }

    @classmethod
    def from_dict(cls, d: Mapping) -> SearchPlan:
        """Rebuild a plan from the dict
        [`to_dict`][scopusflow.plan.SearchPlan.to_dict] wrote.

        The values pass through the same validation as a plan built by hand.
        A dict recorded under a layout this version does not know is refused,
        since reading it as this one could rebuild a different search.
        """
        if not isinstance(d, Mapping):
            raise ValueError(
                "A plan is rebuilt from a dict, as SearchPlan.to_dict() writes it."
            )
        schema = d.get("schema")
        if schema != _PLAN_SCHEMA:
            raise ValueError(
                f"This plan was recorded under schema {schema!r}, and this version "
                f"of scopusflow reads schema {_PLAN_SCHEMA} only."
            )
        return cls(
            query=d.get("query"),
            years=d.get("years"),
            field=d.get("field"),
            view=d.get("view", "STANDARD"),
            partition=d.get("partition", "none"),
            page_size=d.get("page_size"),
        )

    @property
    def wrapped_query(self) -> str:
        """The query wrapped in the plan's ``field``, or the query itself when
        ``field`` is ``None``. Every cell sends it, followed by the cell's year
        limit when it has one. A query with a top-level OR, AND NOT or
        proximity operator is put in brackets before the limit is added, so the
        limit applies to all of it."""
        return wrap_field(self.query, self.field)

    def cells(self) -> list[PlanCell]:
        """Expand the plan into the cells that will be fetched."""
        q = self.wrapped_query
        size = int(self.page_size)  # type: ignore[arg-type]
        if self.partition == "year":
            years = sorted(set(self.years))  # type: ignore[arg-type]
            return [
                PlanCell(i + 1, q, str(y), y, self.view, size)
                for i, y in enumerate(years)
            ]
        date = None
        if self.years:
            lo, hi = min(self.years), max(self.years)
            date = str(lo) if lo == hi else f"{lo}-{hi}"
        return [PlanCell(1, q, date, None, self.view, size)]

wrapped_query property

wrapped_query

The query wrapped in the plan's field, or the query itself when field is None. Every cell sends it, followed by the cell's year limit when it has one. A query with a top-level OR, AND NOT or proximity operator is put in brackets before the limit is added, so the limit applies to all of it.

to_dict

to_dict()

The plan as a plain dict, which from_dict turns back into it.

This is the form fetch_plan attaches as attrs["plan"]. A dataclass instance cannot be written as JSON, and pandas 2.1 and later write a frame's attributes as JSON when it is saved to parquet, so a harvest carrying the plan itself could not be saved. schema numbers the layout, and the years are a sorted list.

Examples:

>>> import scopusflow as sf
>>> plan = sf.SearchPlan("graphene", years=[2020, 2019], partition="year")
>>> plan.to_dict()["years"]
[2019, 2020]
>>> sf.SearchPlan.from_dict(plan.to_dict()) == plan
True
Source code in src/scopusflow/plan.py
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
def to_dict(self) -> dict:
    """The plan as a plain dict, which
    [`from_dict`][scopusflow.plan.SearchPlan.from_dict] turns back into it.

    This is the form [`fetch_plan`][scopusflow.fetch.fetch_plan] attaches
    as ``attrs["plan"]``. A dataclass instance cannot be written as JSON,
    and pandas 2.1 and later write a frame's attributes as JSON when it is
    saved to parquet, so a harvest carrying the plan itself could not be
    saved. ``schema`` numbers the layout, and the years are a sorted list.

    Examples
    --------
    >>> import scopusflow as sf
    >>> plan = sf.SearchPlan("graphene", years=[2020, 2019], partition="year")
    >>> plan.to_dict()["years"]
    [2019, 2020]
    >>> sf.SearchPlan.from_dict(plan.to_dict()) == plan
    True
    """
    return {
        "schema": _PLAN_SCHEMA,
        "query": self.query,
        "years": None if self.years is None else list(self.years),
        "field": self.field,
        "view": self.view,
        "partition": self.partition,
        "page_size": int(self.page_size),  # type: ignore[arg-type]
    }

from_dict classmethod

from_dict(d)

Rebuild a plan from the dict to_dict wrote.

The values pass through the same validation as a plan built by hand. A dict recorded under a layout this version does not know is refused, since reading it as this one could rebuild a different search.

Source code in src/scopusflow/plan.py
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
@classmethod
def from_dict(cls, d: Mapping) -> SearchPlan:
    """Rebuild a plan from the dict
    [`to_dict`][scopusflow.plan.SearchPlan.to_dict] wrote.

    The values pass through the same validation as a plan built by hand.
    A dict recorded under a layout this version does not know is refused,
    since reading it as this one could rebuild a different search.
    """
    if not isinstance(d, Mapping):
        raise ValueError(
            "A plan is rebuilt from a dict, as SearchPlan.to_dict() writes it."
        )
    schema = d.get("schema")
    if schema != _PLAN_SCHEMA:
        raise ValueError(
            f"This plan was recorded under schema {schema!r}, and this version "
            f"of scopusflow reads schema {_PLAN_SCHEMA} only."
        )
    return cls(
        query=d.get("query"),
        years=d.get("years"),
        field=d.get("field"),
        view=d.get("view", "STANDARD"),
        partition=d.get("partition", "none"),
        page_size=d.get("page_size"),
    )

cells

cells()

Expand the plan into the cells that will be fetched.

Source code in src/scopusflow/plan.py
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
def cells(self) -> list[PlanCell]:
    """Expand the plan into the cells that will be fetched."""
    q = self.wrapped_query
    size = int(self.page_size)  # type: ignore[arg-type]
    if self.partition == "year":
        years = sorted(set(self.years))  # type: ignore[arg-type]
        return [
            PlanCell(i + 1, q, str(y), y, self.view, size)
            for i, y in enumerate(years)
        ]
    date = None
    if self.years:
        lo, hi = min(self.years), max(self.years)
        date = str(lo) if lo == hi else f"{lo}-{hi}"
    return [PlanCell(1, q, date, None, self.view, size)]

PlanCell dataclass

One unit of work in a SearchPlan, as SearchPlan.cells lists them.

cell numbers it from 1. query is the plan's wrapped_query, without the year limit. date is the year limit as text ("2015" or "2015-2020", or None for none), which fetch_plan appends to the query as a PUBYEAR clause. year is the cell's year under partition="year" and None under partition="none". view and page_size are the plan's.

Source code in src/scopusflow/plan.py
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
@dataclass(frozen=True)
class PlanCell:
    """One unit of work in a [`SearchPlan`][scopusflow.plan.SearchPlan], as
    [`SearchPlan.cells`][scopusflow.plan.SearchPlan.cells] lists them.

    ``cell`` numbers it from 1. ``query`` is the plan's
    [`wrapped_query`][scopusflow.plan.SearchPlan.wrapped_query], without the
    year limit. ``date`` is the year limit as text (``"2015"`` or
    ``"2015-2020"``, or ``None`` for none), which
    [`fetch_plan`][scopusflow.fetch.fetch_plan] appends to the query as a
    ``PUBYEAR`` clause. ``year`` is the cell's year under ``partition="year"``
    and ``None`` under ``partition="none"``. ``view`` and ``page_size`` are the
    plan's.
    """

    cell: int
    query: str
    date: str | None
    year: int | None
    view: str
    page_size: int = 200

SearchPlan, scopus_count, scopus_trend, compare_topics and scopus_intersections check a query before any request. For a query that uses INDEXTERMS(), which pybliometrics documents as not working through the Search API, they issue a QuerySyntaxWarning, a UserWarning subclass reached through its module as scopusflow.query.QuerySyntaxWarning.

Retrieve

Size a search cheaply first, then execute a plan as a resumable, checkpointed harvest, and pull fuller records.

scopus_count

scopus_count(query, years=None, field=None, view='STANDARD', **kwargs)

Return how many records the (optionally year-filtered) query matches.

A single cheap request that does not download the records, so it is the right way to size a search before committing quota to a harvest.

years are sent as one inclusive range from the earliest year to the latest, so years=[2015, 2020] counts every year from 2015 to 2020. scopus_intersections and a SearchPlan with partition="none" do the same, while scopus_trend, compare_topics and partition="year" take each listed year.

field wraps the query in a Scopus field tag (see FIELD_TAGS) before the year limit is appended, as a plan's field does. Left None, the query is sent as written. view and any further keyword argument go to pybliometrics' ScopusSearch.

The request is sent with refresh=True unless you pass refresh. pybliometrics keys its own response cache on the query string and view alone. Without refresh=True, it would answer a query an earlier harvest downloaded with the number of rows the harvest left there, and make no request. refresh=False, or a number of days, opts in to that cache, and a count it answers is warned about. The key ignores keyword filters such as subj, so fold any filter into the query before you opt in.

The query is checked offline first, as SearchPlan checks it, so an empty query or one the checks refuse raises ValueError before any request.

Raises ScopusFlowConfigError before any request when pybliometrics.init() has not been called in the session.

Source code in src/scopusflow/count.py
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
def scopus_count(query: str, years: Sequence[int] | None = None,
                 field: str | None = None, view: str = "STANDARD",
                 **kwargs) -> int:
    """Return how many records the (optionally year-filtered) query matches.

    A single cheap request that does not download the records, so it is the right
    way to size a search before committing quota to a harvest.

    ``years`` are sent as one inclusive range from the earliest year to the
    latest, so ``years=[2015, 2020]`` counts every year from 2015 to 2020.
    [`scopus_intersections`][scopusflow.intersections.scopus_intersections] and
    a [`SearchPlan`][scopusflow.plan.SearchPlan] with ``partition="none"`` do
    the same, while [`scopus_trend`][scopusflow.trend.scopus_trend],
    [`compare_topics`][scopusflow.compare.compare_topics] and
    ``partition="year"`` take each listed year.

    ``field`` wraps the query in a Scopus field tag (see
    [`FIELD_TAGS`][scopusflow.query.FIELD_TAGS]) before the year limit is
    appended, as a plan's ``field`` does. Left ``None``, the query is sent as
    written. ``view`` and any further keyword argument go to pybliometrics'
    ``ScopusSearch``.

    The request is sent with ``refresh=True`` unless you pass ``refresh``.
    pybliometrics keys its own response cache on the query string and view
    alone. Without ``refresh=True``, it would answer a query an earlier harvest
    downloaded with the number of rows the harvest left there, and make no
    request. ``refresh=False``, or a number of days, opts in to that cache, and
    a count it answers is warned about. The key ignores keyword filters such as
    ``subj``, so fold any filter into the query before you opt in.

    The query is checked offline first, as
    [`SearchPlan`][scopusflow.plan.SearchPlan] checks it, so an empty query or
    one the checks refuse raises ``ValueError`` before any request.

    Raises [`ScopusFlowConfigError`][scopusflow.exceptions.ScopusFlowConfigError]
    before any request when ``pybliometrics.init()`` has not been called in the
    session.
    """
    if not query or not str(query).strip():
        raise ValueError("query must be a non-empty string.")
    # Checked offline first (see scopusflow.plan.SearchPlan), so a query the
    # API cannot run as written costs no request.
    _check_query(str(query).strip(), field)
    q = _count_query(str(query).strip(), years, field)
    kwargs.setdefault("refresh", True)
    require_init()

    from pybliometrics.scopus import ScopusSearch  # imported lazily; needs a key

    t0 = time.time()
    search = ScopusSearch(q, view=view, download=False, **kwargs)
    return result_size(search, t0, kwargs["refresh"], q, stacklevel=2)

fetch_plan

fetch_plan(plan, cache_dir=None, resume=True, format='parquet', should_stop=None, *, cursor=None, fields='core', **kwargs)

Run every cell of plan and return one normalised DataFrame.

With cache_dir set, each cell is written to disk as it completes, so an interrupted or quota-limited run resumes without re-fetching finished cells. A cache_dir belongs to one plan. Checkpoints are keyed by cell number, and each is written with a manifest beside it, cell-NNN.json, recording the query as sent, the view, the page size and the paging mode, with the cell's row count, reported total, retrieval time and scopusflow version. On resume a checkpoint is served only when its manifest's query, view, page size and paging match the cell's, an empty checkpoint included. The query and view its rows record must match as well, and it must hold as many rows as the manifest records, so a checkpoint replaced while its manifest was left in place is not served under the old manifest. One written by a different plan is warned about and refetched, never silently returned. Point each plan at its own directory all the same, since that refetch spends quota. A checkpoint without a manifest, as earlier versions wrote them, is judged by the query and view its rows record, and an empty one, whose rows record nothing, is refetched at one request each, with a single warning for the harvest. A manifest without its checkpoint is ignored. A checkpoint that cannot be read back is likewise treated as a miss, warned about and refetched, and the harvest carries on. A cache_dir that is not a directory and cannot be created as one, or in which no file can be created, raises ValueError before any request. A checkpoint or manifest that cannot be written all the same, for example on a full disk, is warned about, and the cell's records are kept in the result without a checkpoint. resume (True by default) serves a cell from its checkpoint in cache_dir when that checkpoint answers the cell, as described above. resume=False reads no checkpoint: every cell is fetched again, and the checkpoint and manifest of each cell fetched are overwritten. Without cache_dir, resume has no effect. format selects the checkpoint format ("parquet" or "csv"); parquet silently falls back to CSV when no parquet engine is installed. Pass a zero-argument should_stop callable to allow co-operative cancellation: it is checked before each cell and the harvest stops (returning what it has) when it returns True. A harvest stopped that way records attrs["stopped_early"] = True, with the number of cells the plan holds as attrs["cells_planned"] and the number that ran as attrs["cells_run"]. Its total_results is None, since the cells it never reached were never counted. scopus_search_report then says how many cells ran, which years they covered and that the harvest is incomplete. A harvest that ran every cell carries none of the three. Per-cell progress is emitted on the "scopusflow" logger.

cursor chooses how each cell is paged. Left unset, or True, it pages by cursor, which retrieves a cell of any size. False pages by offset, the route pybliometrics documents for keys without a Scopus subscription. Offset paging cannot retrieve a cell that matches more than 5000 records. pybliometrics refuses such a cell after one request, and the ScopusQueryError it raises is raised again naming the cell, so the search has to be partitioned more finely. pybliometrics' own subscriber keyword is taken as the same choice, and passing both with different values raises ValueError before any request. The manifest records the mode, so a checkpoint fetched under one mode is warned about and refetched when the other is asked for. A checkpoint without a manifest counts as paged by cursor, since every earlier version paged that way. attrs["paging"] is "cursor" or "offset" accordingly. The R twin's scopus_fetch_plan() pages by offset unless given cursor = TRUE.

fields is "core" by default. "extended" also keeps the document type, source type, volume, issue, pages, ISSN, PubMed identifier and open-access status the Search API sends with every record, and the abstract under a COMPLETE plan (see to_records). They arrive in the responses each cell already pays for, so they cost no extra quota. Like cursor, it is an argument of the harvest, since it changes how the records are read while the search returns the same records. Each manifest records the fields its checkpoint was fetched with. A checkpoint that lacks a column an extended harvest asks for is warned about and refetched, one written by an earlier version included. An extended checkpoint serves a core harvest with the extended columns dropped. Any value other than "core" or "extended" raises ValueError before any request.

Each cell's row count is compared against the total the API reports for that cell's query, and a shortfall is warned about, since a truncated or failed download otherwise arrives as a merely small result. The per-cell accounting is attached as result.attrs["cell_totals"], a list of dicts with the keys cell, date, n_records and reported_total (pandas.DataFrame(result.attrs["cell_totals"]) tabulates it), and their sum as result.attrs["total_results"], the attribute the R twin's scopus_fetch() also attaches. The sum is None unless every cell reported a total, since a partial sum would understate the search while looking like a real figure. A cell resumed from a checkpoint reports the total its manifest recorded, and one resumed from a checkpoint without a manifest reports none.

Further keyword arguments go to pybliometrics' ScopusSearch for every cell, refresh among them (see below). subscriber is the exception: it is read as the paging choice, as described under cursor. A count among them replaces the plan's page_size as the number of records asked for per request, and is the page size each manifest records.

Every cell is fetched with refresh=True unless you pass refresh yourself, so a run spends quota on each cell it does not resume from a checkpoint. pybliometrics would otherwise answer from its own response cache, keyed on the query string and view alone. A re-run into a fresh cache_dir would then receive the earlier pull, dated now, with the cached row count standing in for the API's total. refresh=False, or a number of days, opts in to that cache, and a cell it serves is dated by its cache file, reports no total and is warned about. The cache key ignores keyword arguments such as subj or date, so a filtered search would be served the unfiltered one's records. Fold any filter into the query before you opt in.

The harvest also carries its provenance: the originating plan, as the dict SearchPlan.to_dict writes (SearchPlan.from_dict rebuilds the plan), retrieved_at and retrieved_until, as ISO 8601 text in UTC to the second ("2026-07-22T09:15:00+00:00"), scopusflow_version and paging. These are what scopus_search_report reads back. They are combined over the cells as the R twin combines them. retrieved_at is the earliest cell's time, since a set is only as fresh as its oldest cell. retrieved_until is the latest, the last search, which the record of a harvest resumed over several days gives for PRISMA-S item 13. The version is a sorted list when cells resumed from an earlier release's checkpoints make more than one. A resumed cell contributes the time and version its manifest recorded. Each attribute is omitted, never approximated, when any cell cannot supply it, as a cell resumed from a checkpoint without a manifest cannot, since dating the whole from the cells that can be dated would date it later than part of what it holds.

Every attribute is a value json.dumps writes, so the harvest can be saved with DataFrame.to_parquet and passed through merge, groupby(...).apply and concat. write_records saves it with its provenance whatever the pandas version.

attrs["view"] records the plan's view, which decides what authors holds: the first author under STANDARD and the author list under COMPLETE (see to_records), and top reads it back. A cell resumed from a checkpoint written before the authors were joined with "; " has them joined as it is served, so it costs no request.

When plan.view == "COMPLETE", the output gains an authkeywords column (see to_records) at no extra request cost beyond COMPLETE's own smaller page size, which already means more requests, and so more quota, for the same number of records. A plan with view="STANDARD" (the default) never carries this column, and a checkpoint written under the other view is refetched like any other different-plan checkpoint. Resuming a cache written before this column existed is safe in the COMPLETE direction: pandas.concat fills the older cells' missing column with NA rather than erroring.

Raises ScopusFlowConfigError before any request when pybliometrics.init() has not been called in the session. A cell served from its checkpoint sends no request, so a harvest resumed wholly from checkpoints runs without init().

Source code in src/scopusflow/fetch.py
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
def fetch_plan(
    plan: SearchPlan,
    cache_dir: str | None = None,
    resume: bool = True,
    format: str = "parquet",
    should_stop=None,
    *,
    cursor: bool | None = None,
    fields: str = "core",
    **kwargs,
) -> pd.DataFrame:
    """Run every cell of ``plan`` and return one normalised DataFrame.

    With ``cache_dir`` set, each cell is written to disk as it completes, so an
    interrupted or quota-limited run resumes without re-fetching finished cells.
    A cache_dir belongs to one plan. Checkpoints are keyed by cell number, and
    each is written with a manifest beside it, ``cell-NNN.json``, recording
    the query as sent, the view, the page size and the paging mode, with the
    cell's row count, reported total, retrieval time and scopusflow version.
    On resume a checkpoint is served only when its manifest's query, view,
    page size and paging match the cell's, an empty checkpoint included. The
    query and view its rows record must match as well, and it must hold as
    many rows as the manifest records, so a checkpoint replaced while its
    manifest was left in place is not served under the old manifest. One
    written by a different plan is warned about and refetched, never silently
    returned. Point each plan at its own directory all the same, since that
    refetch spends quota. A checkpoint without a manifest, as earlier versions
    wrote them, is judged by the query and view its rows record, and an empty
    one, whose rows record nothing, is refetched at one request each, with a
    single warning for the harvest. A manifest without its checkpoint is
    ignored. A checkpoint that cannot be read back is likewise treated as a
    miss, warned about and refetched, and the harvest carries on. A
    ``cache_dir`` that is not a directory and cannot be created as one, or in
    which no file can be created, raises ``ValueError`` before any request. A
    checkpoint or manifest that cannot be written all the same, for example on
    a full disk, is warned about, and the cell's records are kept in the
    result without a checkpoint. ``resume`` (``True`` by default) serves a cell
    from its checkpoint in ``cache_dir`` when that checkpoint answers the cell,
    as described above. ``resume=False`` reads no checkpoint: every cell is
    fetched again, and the checkpoint and manifest of each cell fetched are
    overwritten. Without ``cache_dir``, ``resume`` has no effect.
    ``format`` selects the checkpoint format ("parquet" or "csv"); parquet
    silently falls back to CSV when no parquet engine is installed. Pass a
    zero-argument ``should_stop`` callable to allow co-operative cancellation: it
    is checked before each cell and the harvest stops (returning what it has) when
    it returns ``True``. A harvest stopped that way records
    ``attrs["stopped_early"] = True``, with the number of cells the plan holds as
    ``attrs["cells_planned"]`` and the number that ran as ``attrs["cells_run"]``.
    Its ``total_results`` is ``None``, since the cells it never reached were
    never counted.
    [`scopus_search_report`][scopusflow.report.scopus_search_report] then says
    how many cells ran, which years they covered and that the harvest is
    incomplete. A harvest that ran every cell carries none of the three.
    Per-cell progress is emitted on the ``"scopusflow"`` logger.

    ``cursor`` chooses how each cell is paged. Left unset, or ``True``, it
    pages by cursor, which retrieves a cell of any size. ``False`` pages by
    offset, the route pybliometrics documents for keys without a Scopus
    subscription. Offset paging cannot retrieve a cell that matches more than
    5000 records. pybliometrics refuses such a cell after one request, and the
    ``ScopusQueryError`` it raises is raised again naming the cell, so the
    search has to be partitioned more finely. pybliometrics' own
    ``subscriber`` keyword is taken as the same choice, and passing both with
    different values raises ``ValueError`` before any request. The manifest
    records the mode, so a checkpoint fetched under one mode is warned about
    and refetched when the other is asked for. A checkpoint without a manifest
    counts as paged by cursor, since every earlier version paged that way.
    ``attrs["paging"]`` is ``"cursor"`` or ``"offset"`` accordingly. The R
    twin's ``scopus_fetch_plan()`` pages by offset unless given
    ``cursor = TRUE``.

    ``fields`` is ``"core"`` by default. ``"extended"`` also keeps the document
    type, source type, volume, issue, pages, ISSN, PubMed identifier and
    open-access status the Search API sends with every record, and the abstract
    under a ``COMPLETE`` plan (see
    [`to_records`][scopusflow.records.to_records]). They arrive in the
    responses each cell already pays for, so they cost no extra quota. Like
    ``cursor``, it is an argument of the harvest, since it changes how the
    records are read while the search returns the same records. Each manifest
    records the fields its checkpoint was fetched with. A checkpoint that lacks
    a column an extended harvest asks for is warned about and refetched, one
    written by an earlier version included. An extended checkpoint serves a
    core harvest with the extended columns dropped. Any value other than
    ``"core"`` or ``"extended"`` raises ``ValueError`` before any request.

    Each cell's row count is compared against the total the API reports for
    that cell's query, and a shortfall is warned about, since a truncated or
    failed download otherwise arrives as a merely small result. The per-cell
    accounting is attached as ``result.attrs["cell_totals"]``, a list of dicts
    with the keys ``cell``, ``date``, ``n_records`` and ``reported_total``
    (``pandas.DataFrame(result.attrs["cell_totals"])`` tabulates it), and their sum as
    ``result.attrs["total_results"]``, the attribute the R twin's
    ``scopus_fetch()`` also attaches. The sum is ``None`` unless every cell
    reported a total, since a partial sum would understate the search while
    looking like a real figure. A cell resumed from a checkpoint reports the
    total its manifest recorded, and one resumed from a checkpoint without a
    manifest reports none.

    Further keyword arguments go to pybliometrics' ``ScopusSearch`` for every
    cell, ``refresh`` among them (see below). ``subscriber`` is the exception:
    it is read as the paging choice, as described under ``cursor``. A ``count``
    among them replaces the plan's ``page_size`` as the number of records asked
    for per request, and is the page size each manifest records.

    Every cell is fetched with ``refresh=True`` unless you pass ``refresh``
    yourself, so a run spends quota on each cell it does not resume from a
    checkpoint. pybliometrics would otherwise answer from its own response
    cache, keyed on the query string and view alone. A re-run into a fresh
    ``cache_dir`` would then receive the earlier pull, dated now, with the
    cached row count standing in for the API's total. ``refresh=False``, or a
    number of days, opts in to that cache, and a cell it serves is dated by its
    cache file, reports no total and is warned about. The cache key ignores
    keyword arguments such as ``subj`` or ``date``, so a filtered search would
    be served the unfiltered one's records. Fold any filter into the query
    before you opt in.

    The harvest also carries its provenance: the originating ``plan``, as the
    dict [`SearchPlan.to_dict`][scopusflow.plan.SearchPlan.to_dict] writes
    ([`SearchPlan.from_dict`][scopusflow.plan.SearchPlan.from_dict] rebuilds
    the plan), ``retrieved_at`` and ``retrieved_until``, as ISO 8601 text in
    UTC to the second (``"2026-07-22T09:15:00+00:00"``), ``scopusflow_version``
    and ``paging``. These are what
    [`scopus_search_report`][scopusflow.report.scopus_search_report] reads
    back. They are combined over the cells as the R twin combines them.
    ``retrieved_at`` is the earliest cell's time, since a set is only as fresh
    as its oldest cell. ``retrieved_until`` is the latest, the last search,
    which the record of a harvest resumed over several days gives for PRISMA-S
    item 13. The version is a sorted list when cells resumed from an earlier
    release's checkpoints make more than one. A resumed cell contributes the
    time and version its manifest recorded. Each attribute is omitted, never
    approximated, when any cell cannot supply it, as a cell resumed from a
    checkpoint without a manifest cannot, since dating the whole from the cells
    that can be dated would date it later than part of what it holds.

    Every attribute is a value ``json.dumps`` writes, so the harvest can be
    saved with ``DataFrame.to_parquet`` and passed through ``merge``,
    ``groupby(...).apply`` and ``concat``.
    [`write_records`][scopusflow.io.write_records] saves it with its provenance
    whatever the pandas version.

    ``attrs["view"]`` records the plan's view, which decides what ``authors``
    holds: the first author under ``STANDARD`` and the author list under
    ``COMPLETE`` (see [`to_records`][scopusflow.records.to_records]), and
    [`top`][scopusflow.records.top] reads it back. A cell resumed from a
    checkpoint written before the authors were joined with ``"; "`` has them
    joined as it is served, so it costs no request.

    When ``plan.view == "COMPLETE"``, the output gains an ``authkeywords``
    column (see [`to_records`][scopusflow.records.to_records]) at no extra
    request cost beyond ``COMPLETE``'s own smaller page size, which already
    means more requests, and so more quota, for the same number of records. A
    plan with ``view="STANDARD"`` (the default) never carries this column, and
    a checkpoint written under the other view is refetched like any other
    different-plan checkpoint. Resuming a cache written before this column
    existed is safe in the ``COMPLETE`` direction: ``pandas.concat`` fills the
    older cells' missing column with ``NA`` rather than erroring.

    Raises [`ScopusFlowConfigError`][scopusflow.exceptions.ScopusFlowConfigError]
    before any request when ``pybliometrics.init()`` has not been called in the
    session. A cell served from its checkpoint sends no request, so a harvest
    resumed wholly from checkpoints runs without ``init()``.
    """
    if not isinstance(plan, SearchPlan):
        raise ValueError("plan must be a SearchPlan.")
    if format not in _FORMATS:
        raise ValueError("format must be 'parquet' or 'csv'.")
    cursor = _resolve_cursor(cursor, kwargs)
    paging = "cursor" if cursor else "offset"
    _check_fields(fields)

    from pybliometrics.scopus import ScopusSearch  # imported lazily; needs a key

    # Only offset paging meets pybliometrics' size refusal, so only then is its
    # error caught and given the cell. The import is guarded, since a
    # pybliometrics without this module, or a stand-in such as the offline
    # tests install, should still run a harvest, its errors passing unchanged.
    # An empty tuple in an except clause catches nothing.
    refusals: tuple[type[BaseException], ...] = ()
    if not cursor:
        try:
            from pybliometrics.exception import ScopusQueryError
        except ImportError:
            pass
        else:
            refusals = (ScopusQueryError,)

    cache = _writable_cache(cache_dir)

    # setdefault, so a caller who passes refresh, opting in to pybliometrics'
    # cache, still wins. Only then can an answer come from that cache, so only
    # then is each answer checked.
    kwargs.setdefault("refresh", True)
    check_cache = may_use_cache(kwargs["refresh"])

    # Imported inside the function, and never at module scope: this module is
    # imported while the package's own __init__ is still executing, and
    # __version__ is not bound until after that import returns.
    from . import __version__

    cells = plan.cells()
    total = len(cells)
    frames: list[pd.DataFrame] = []
    accounting: list[dict] = []
    stamps: list[datetime | None] = []
    versions: list[str | None] = []
    legacy_empty: list[int] = []
    stopped = False
    for cell in cells:
        if should_stop is not None and should_stop():
            logger.info("Stopped before cell %d/%d.", cell.cell, total)
            stopped = True
            break

        query = _cell_query(cell.query, cell.year, cell.date)
        # What the cell's request is made of, as its manifest records it. The
        # page size is the one that will be sent, which a caller's own count
        # overrides (see below). A count given as a NumPy integer is stored as
        # a plain one, which json.dumps can write.
        page_size = kwargs.get("count", cell.page_size)
        if isinstance(page_size, numbers.Integral) and not isinstance(page_size, bool):
            page_size = int(page_size)
        identity = {"query": query, "view": cell.view, "page_size": page_size,
                    "paging": paging, "fields": fields}
        if cache is not None and resume:
            existing = _find_checkpoint(cache, cell.cell)
            cached = _read_checkpoint(existing) if existing is not None else None
            manifest = (_read_manifest(_manifest_path(cache, cell.cell))
                        if cached is not None else None)
            if existing is not None and cached is None:
                warnings.warn(
                    f"The checkpoint {existing} could not be read back, so it "
                    "was discarded and the cell refetched. An interrupted run "
                    "can leave a checkpoint half-written.",
                    stacklevel=2,
                )
            elif manifest is _UNREADABLE:
                warnings.warn(
                    f"The manifest beside the checkpoint {existing} could not be "
                    "read back, so the cell was refetched.",
                    stacklevel=2,
                )
            elif cached is not None and manifest is None and len(cached) == 0:
                # Without a manifest an empty checkpoint has nothing to say
                # which search wrote it, since its rows carry no query, so it
                # would be served to any plan pointed at this directory. It
                # costs one request, and the harvest warns once for all such
                # cells after the loop.
                legacy_empty.append(cell.cell)
            elif cached is not None:
                # Checkpoints are keyed by cell number alone, so a cache_dir
                # reused for a different plan would otherwise hand back the
                # wrong records silently. The rows record the query and view
                # they were fetched with, and the manifest the whole request,
                # so both are compared, as the R twin compares both. A
                # checkpoint without a manifest, written before manifests
                # existed or by a run stopped between the two writes, is judged
                # by its rows alone. The row count ties a manifest to its own
                # checkpoint: an earlier version replacing the checkpoint
                # leaves the manifest in place, and an empty replacement has no
                # rows to betray it. A checkpoint without a manifest was paged
                # by cursor, the only mode earlier versions used.
                mismatch = _rows_mismatch(cached, query, cell.view)
                if mismatch is None and manifest is not None:
                    mismatch = _manifest_mismatch(manifest, identity)
                recorded = _LEGACY_PAGING if manifest is None else manifest.get("paging")
                # Every version that writes manifests records the mode in
                # them, so a manifest naming none is treated as another plan's.
                if mismatch is None and not isinstance(recorded, str):
                    mismatch = f"paging {recorded!r}, not {paging!r}"
                if mismatch is not None:
                    warnings.warn(
                        f"Checkpoint for cell {cell.cell} in {cache} was written "
                        f"by a different plan ({mismatch}); refetching this cell. "
                        "Use one cache_dir per plan.",
                        stacklevel=2,
                    )
                elif recorded != paging:
                    # This plan wrote the checkpoint under the other mode,
                    # which a harvest switching modes meets in every cell, so
                    # the warning names both modes and gives no advice about
                    # cache directories.
                    if manifest is not None:
                        text = (f"The checkpoint {existing} was fetched under "
                                f"{recorded} paging, not the {paging} paging asked "
                                "for, so it was discarded and the cell refetched.")
                    else:
                        text = (f"The checkpoint {existing} has no manifest, so it "
                                f"counts as fetched under {recorded} paging, the "
                                "only mode earlier versions used, and the cell was "
                                f"refetched under the {paging} paging asked for.")
                    warnings.warn(text, stacklevel=2)
                elif manifest is not None and manifest.get("n_records") != len(cached):
                    warnings.warn(
                        f"The checkpoint {existing} holds {len(cached)} record(s), "
                        f"but the manifest beside it records "
                        f"{manifest.get('n_records')!r}, so the two were not "
                        "written together and the cell was refetched.",
                        stacklevel=2,
                    )
                elif _lacks_requested_columns(cached, cell.view, fields):
                    # This plan wrote the checkpoint with the core fields, so
                    # the warning gives no advice about cache directories.
                    warnings.warn(
                        f"The checkpoint {existing} was fetched with the core "
                        "fields, not the extended fields asked for, so it was "
                        "discarded and the cell refetched.",
                        stacklevel=2,
                    )
                else:
                    logger.info("Cell %d/%d: loaded from cache.", cell.cell, total)
                    dropped = ["view"]
                    if fields == "core":
                        # Served from an extended checkpoint, a core harvest
                        # holds what a core fetch returns.
                        dropped += [*EXTENDED_COLUMNS, "abstract"]
                    served = _with_joined_authors(
                        cached.drop(columns=dropped, errors="ignore")
                    )
                    frames.append(served)
                    # A manifest restores the cell's total, time and version. A
                    # checkpoint without one has no record of them, and they
                    # are left out, never guessed.
                    provenance = manifest or {}
                    reported = provenance.get("reported_total")
                    version = provenance.get("scopusflow_version")
                    accounting.append({
                        "cell": cell.cell, "date": cell.date,
                        "n_records": len(served),
                        "reported_total": (reported if isinstance(reported, int)
                                           and not isinstance(reported, bool) else None),
                    })
                    stamps.append(_manifest_stamp(provenance))
                    versions.append(version if isinstance(version, str) else None)
                    continue

        logger.info("Cell %d/%d: fetching %s", cell.cell, total, query)
        # The page size is the plan's, so the harvest pages the way the plan (and
        # the search record built from it) says it does. setdefault, so a caller
        # passing count of their own still wins.
        kwargs.setdefault("count", cell.page_size)
        # Checked before each request, so a harvest resumed wholly from
        # checkpoints, which sends none, still runs in a fresh session that
        # has not called init().
        require_init()
        t0 = time.time()
        try:
            search = ScopusSearch(query, view=cell.view, subscriber=cursor, **kwargs)
        except refusals as error:
            raise type(error)(_offset_refusal(cell, error)) from error
        frame = to_records(search.results, query=query, view=cell.view, fields=fields)

        cached_at = cache_hit_time(search, t0) if check_cache else None
        if cached_at is None:
            # A download that returned nothing yields a zero-row frame and never
            # an error, so without this comparison a truncated or failed cell is
            # indistinguishable from one that matched only a few records.
            cell_total = _reported_total(search)
            stamp = datetime.now(timezone.utc)
        else:
            # A cached answer gives its own row count as the total, so any
            # shortfall would vanish, and its rows date from when the cache
            # file was written.
            warnings.warn(cached_cell_warning(cell.cell, cached_at), stacklevel=2)
            cell_total = None
            stamp = None if cached_at is UNDATED else cached_at
        accounting.append({"cell": cell.cell, "date": cell.date,
                           "n_records": len(frame), "reported_total": cell_total})
        stamps.append(stamp)
        versions.append(__version__)
        if cell_total is not None:
            if len(frame) < cell_total:
                warnings.warn(
                    f"Cell {cell.cell} retrieved {len(frame)} record(s), but the "
                    f"Scopus API reports {cell_total} for this query, so the "
                    "harvest may be incomplete. Check the key's remaining quota, "
                    "and consider partitioning the plan by year so each cell is "
                    "smaller.",
                    stacklevel=2,
                )

        if cache is not None:
            # The order keeps a checkpoint from ever being paired with a
            # manifest that describes another. The old manifest goes first, so
            # a run stopped before the new one is written leaves a checkpoint
            # without a manifest, which resume treats as written by an earlier
            # version. The new checkpoint replaces any of the other format,
            # which would otherwise be found beside the new manifest. A write
            # that fails, on a full disk or where a directory has taken a
            # checkpoint's name, is warned about, and the records the cell
            # paid for are kept in the result.
            manifest_path = _manifest_path(cache, cell.cell)
            written = None
            try:
                manifest_path.unlink(missing_ok=True)
                # The view also travels in a column of the checkpoint (stripped
                # again on resume), so an earlier version reading it can still
                # tell the views apart.
                written = _write_checkpoint(frame.assign(view=cell.view), cache,
                                            cell.cell, format)
                for suffix in (".parquet", ".csv"):
                    sibling = written.with_suffix(suffix)
                    if sibling != written:
                        sibling.unlink(missing_ok=True)
                written_manifest = {
                    "schema": _MANIFEST_SCHEMA,
                    **identity,
                    "n_records": len(frame),
                    "reported_total": cell_total,
                    "retrieved_at": None if stamp is None else iso_utc(stamp),
                    "scopusflow_version": __version__,
                }
                _atomic_write(
                    lambda path, m=written_manifest: path.write_text(
                        json.dumps(m, indent=2) + "\n", encoding="utf-8"
                    ),
                    manifest_path,
                )
            except OSError:
                # A checkpoint whose manifest could not follow it is removed
                # as well, so the warning holds and resume never meets it.
                if written is not None:
                    try:
                        written.unlink(missing_ok=True)
                    except OSError:
                        pass
                warnings.warn(
                    _unwritable_warning(f"cell {cell.cell}", cache, "its records are"),
                    stacklevel=2,
                )
        frames.append(frame)

    if legacy_empty:
        warnings.warn(_legacy_empty_warning(legacy_empty, cache), stacklevel=2)

    if not frames:
        out = pd.DataFrame(columns=_record_columns(plan.view, fields))
    else:
        out = pd.concat(frames, ignore_index=True)
        out["entry_number"] = range(1, len(out) + 1)
        logger.info("Retrieved %d records.", len(out))

    # The plan travels with its own harvest, as it does in the R twin, so the
    # search record can be written from the records alone. Every attribute is
    # held in a form json.dumps writes: pandas 2.1 and later write attrs as JSON
    # when a frame is saved to parquet, and pandas compares the inputs' attrs in
    # merge, groupby-apply and concat, where a DataFrame held there raised.
    out.attrs["plan"] = plan.to_dict()
    out.attrs["cell_totals"] = accounting
    reported = [row["reported_total"] for row in accounting]
    out.attrs["total_results"] = (
        sum(reported) if accounting and all(n is not None for n in reported) else None
    )
    if stopped:
        # A stopped harvest says so, and how far it got. It claims no total
        # even when every cell that ran reported one, since that sum would
        # stand for the whole search while leaving out the cells never reached.
        out.attrs["total_results"] = None
        out.attrs["stopped_early"] = True
        out.attrs["cells_planned"] = total
        out.attrs["cells_run"] = len(accounting)
    # True of every cell: a checkpoint is served only to a harvest in its own
    # paging mode, and one without a manifest only to a cursor harvest.
    out.attrs["paging"] = paging
    # The view decides whether authors are first authors or author lists, so
    # top() can say what a tally by author counts, as the R twin does.
    out.attrs["view"] = plan.view
    # Combined as the R twin combines its cells. The earliest time is kept,
    # since a set is only as fresh as its oldest cell, and so is the latest,
    # the last search, which PRISMA-S item 13 asks for. Every version that
    # built a cell is listed, since a cache written by an earlier release means
    # more than one did. Each is claimed only when every cell supplies it:
    # dating the set from the cells that can be dated would date it later than
    # part of what it holds.
    if stamps and all(s is not None for s in stamps):
        out.attrs["retrieved_at"] = iso_utc(min(stamps))
        out.attrs["retrieved_until"] = iso_utc(max(stamps))
    if versions and all(v is not None for v in versions):
        distinct = sorted(set(versions))
        out.attrs["scopusflow_version"] = distinct[0] if len(distinct) == 1 else distinct
    return out

scopus_abstract

scopus_abstract(ids, by='doi', view='META_ABS', include=(), cache_dir=None, resume=True, **kwargs)

Retrieve abstracts for one or many ids, resilient per id.

ids is one identifier, or a list, tuple or pandas Series of them. They are checked before any request, as in the R twin. A missing or blank identifier raises ValueError naming its position, counting from 0. corpus drops such records itself. Surrounding whitespace is trimmed, the SCOPUS_ID: prefix is dropped when by="scopus_id", and whole numbers are sent as text. Any other value, a float or bytes among them, raises TypeError.

by selects the lookup type ("doi", "eid" or "scopus_id"). Any id that fails is warned about and yields an all-NA row that still records the id. view defaults to "META_ABS" (pybliometrics' own default), which carries the abstract text; the lighter "META" omits it and leaves the abstract column empty. Left empty, include adds no column, so the frame holds the ABSTRACT_COLUMNS alone.

include names extra fields to retrieve in the same request: any of "references", "keywords" (the author keywords) and "index_terms" (the index keywords Scopus assigns). "references" needs Abstract Retrieval's "FULL" or "REF" view, and "keywords" needs the "FULL" view, since the other views carry no author keywords. "index_terms" needs "FULL" as well. Any other combination raises ValueError before a request. Both views are an entitlement separate from ordinary abstract access and from Scopus Search access. Per Elsevier's own documentation, some fields (notably author keywords) may need to be requested from your Scopus/Elsevier account contact even when the view itself is otherwise accessible. "FULL" returns the whole bibliography in one request. "REF" serves it about 40 references per request, and pybliometrics pages REF itself, following the view's startref parameter until the list is complete, so a document with N references costs ceil(N / 40) requests under "REF". "FULL" is therefore the cheaper view when your entitlement allows it. A mismatch between the number of references returned and the document's own reported reference count (refcount) is warned about, since the list may then be incomplete.

When "keywords" is included, an authkeywords column is added: the document's author-supplied keywords, joined with "; ", or NA when the document has none, or when the API omits the field for a given key's entitlement. In this package's own development testing, against a live, otherwise fully-entitled key, this field did not populate for documents that do carry author keywords in Scopus itself, so an all-NA column is more likely an entitlement gap worth raising with your Scopus/Elsevier account contact than genuinely absent data. A keyword entry that pybliometrics cannot parse (its 4.4.1 release raises KeyError on an entry without text) leaves that document's keywords NA with a warning, and the rest of its row is kept. From pybliometrics 4.5, an author keyword without text is dropped and the document's other keywords are kept.

When "index_terms" is included, an idxterms column is added after any authkeywords: the document's index keywords, read from pybliometrics' idxterms and joined with "; ", or NA when the response carries none. Authors choose their own keywords, whereas Scopus chooses the index keywords and standardises them to vocabularies derived from thesauri Elsevier owns or licenses, with synonyms, spellings and plurals brought together. The two kinds therefore give different keyword co-occurrence results, and a study should say which it used. Elsevier's guide lists index keywords in the "FULL" view as not available by default, so whether they arrive depends on the key's entitlement as well. In this package's development testing, they did arrive on the same key whose author keywords came back empty. An index term pybliometrics cannot parse (its 4.4.1 release raises KeyError on one without text, as 4.5 still does) leaves that document's index keywords NA with a warning, and the rest of its row is kept. The R twin drops such a term and keeps the others.

When "references" is included, a references column is added: one DataFrame per document (one row per cited work), using pybliometrics' own native field set (position, id, doi, title, authors, authors_auid, authors_affiliationid, sourcetitle, publicationyear, coverDate, volume, issue, first, last, citedbycount, type, text, fulltext). A document with no resolvable references yields a zero-row DataFrame, so the column can always be unnested.

cache_dir and resume checkpoint per identifier, the way fetch_plan checkpoints per cell, pickled rather than written as parquet/csv, since a row here can carry a nested DataFrame. Only a successful retrieval is checkpointed: a failed identifier still yields its warned-about NA row in the returned frame, but is retried on the next resumed run, never read back as data. Worth setting whenever include is used: Abstract Retrieval draws on its own weekly quota, smaller than and separate from Search's, and every identifier costs its own request, so re-running an interrupted batch without a cache re-spends quota already spent. resume=False reads no checkpoint and retrieves every identifier again, overwriting the checkpoint of each identifier retrieved successfully. Distinct identifiers can share a checkpoint file, as "10.1/a.b" and "10.1/a-b" do, and resume does not check which identifier a checkpoint holds, so such a pair should not share a cache_dir. A cache_dir that cannot be written raises ValueError before any request, and a checkpoint that cannot be written all the same is warned about, the row kept in the result, as in fetch_plan. Relying on pybliometrics' own on-disk response cache (its refresh parameter, keyed by identifier and view under its configured cache directory) is enough to avoid repeat network calls for the same identifier across script runs; this checkpoint is for batch-level progress and resumability across many identifiers, a separate concern. Further keyword arguments, refresh among them, go to pybliometrics' AbstractRetrieval.

The number of Abstract Retrieval requests made, and the most recently parsed remaining-quota figure (from pybliometrics' get_key_remaining_quota()), are attached as result.attrs["n_requests"] and result.attrs["quota"], since this is a materially more expensive operation than a search call. n_requests counts each document once, however many REF pages pybliometrics requested for it, so it is a count of documents fetched. A failed retrieval counts once, since the API has usually answered it with an error. A row read from a cache_dir checkpoint sent no request and is not counted. Nor is a row pybliometrics answered from its own cache, recognised by a cache file older than the call or by a response with no headers kept. Such a row's citations and other fields date from when that cache file was written. Pass refresh=True to fetch every identifier from the API.

A 403 (an entitlement gate, most often on the requested view or field) raises ScopusFlowForbiddenError and stops the batch immediately, naming the view and identifier, where a generic failure would leave the caller guessing. Repeating the identical failure for every remaining identifier would serve nobody: entitlement is a property of the account, so a retry cannot succeed.

A spent weekly quota stops the requests too, but returns what the batch holds, since it usually arrives part-way through. pybliometrics raises Scopus429Error once every configured key has been refused. The rows already retrieved are kept, each identifier not retrieved gets an NA row that is not checkpointed, and one ScopusFlowQuotaWarning names the identifier reached and how many were not retrieved. A checkpoint in cache_dir is still read after the stop.

Raises ScopusFlowConfigError before any request when pybliometrics.init() has not been called in the session. An identifier served from its checkpoint sends no request, so a batch resumed wholly from checkpoints runs without init().

Source code in src/scopusflow/abstract.py
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
def scopus_abstract(
    ids,
    by: str = "doi",
    view: str = "META_ABS",
    include: tuple[str, ...] = (),
    cache_dir: str | None = None,
    resume: bool = True,
    **kwargs,
) -> pd.DataFrame:
    """Retrieve abstracts for one or many ids, resilient per id.

    ``ids`` is one identifier, or a list, tuple or pandas Series of them.
    They are checked before any request, as in the R twin. A missing or blank
    identifier raises ``ValueError`` naming its position, counting from 0.
    [`corpus`][scopusflow.corpus.corpus] drops such records itself. Surrounding
    whitespace is trimmed, the ``SCOPUS_ID:`` prefix is dropped when
    ``by="scopus_id"``, and whole numbers are sent as text. Any other value,
    a float or bytes among them, raises ``TypeError``.

    ``by`` selects the lookup type ("doi", "eid" or "scopus_id"). Any id that
    fails is warned about and yields an all-NA row that still records the id.
    ``view`` defaults to "META_ABS" (pybliometrics' own default), which
    carries the abstract text; the lighter "META" omits it and leaves the
    ``abstract`` column empty. Left empty, ``include`` adds no column, so the
    frame holds the [`ABSTRACT_COLUMNS`][scopusflow.abstract.ABSTRACT_COLUMNS]
    alone.

    ``include`` names extra fields to retrieve in the same request: any of
    "references", "keywords" (the author keywords) and "index_terms" (the
    index keywords Scopus assigns). "references" needs Abstract Retrieval's
    "FULL" or "REF" view, and "keywords" needs the "FULL" view, since the
    other views carry no author keywords. "index_terms" needs "FULL" as well.
    Any other combination raises ``ValueError`` before a request. Both views
    are an entitlement separate from ordinary abstract access and from Scopus
    Search access. Per
    Elsevier's own documentation, some fields (notably author keywords) may
    need to be requested from your Scopus/Elsevier account contact even when
    the view itself is otherwise accessible. "FULL" returns the whole bibliography in
    one request. "REF" serves it about 40 references per request, and
    pybliometrics pages REF itself, following the view's ``startref``
    parameter until the list is complete, so a document with N references
    costs ``ceil(N / 40)`` requests under "REF". "FULL" is therefore the
    cheaper view when your entitlement allows it. A mismatch between the
    number of references returned and the document's own reported reference
    count (``refcount``) is warned about, since the list may then be
    incomplete.

    When "keywords" is included, an ``authkeywords`` column is added: the
    document's author-supplied keywords, joined with "; ", or ``NA`` when the
    document has none, or when the API omits the field for a given key's
    entitlement. In this package's own development testing, against a live,
    otherwise fully-entitled key, this field did not populate for documents
    that do carry author keywords in Scopus itself, so an all-``NA`` column
    is more likely an entitlement gap worth raising with your Scopus/Elsevier
    account contact than genuinely absent data. A keyword entry that
    pybliometrics cannot parse (its 4.4.1 release raises ``KeyError`` on an
    entry without text) leaves that document's keywords ``NA`` with a
    warning, and the rest of its row is kept. From pybliometrics 4.5, an
    author keyword without text is dropped and the document's other keywords
    are kept.

    When "index_terms" is included, an ``idxterms`` column is added after any
    ``authkeywords``: the document's index keywords, read from
    pybliometrics' ``idxterms`` and joined with "; ", or ``NA`` when the
    response carries none. Authors choose their own keywords, whereas Scopus
    chooses the index keywords and standardises them to vocabularies derived
    from thesauri Elsevier owns or licenses, with synonyms, spellings and
    plurals brought together. The two kinds therefore give different keyword
    co-occurrence results, and a study should say which it used. Elsevier's
    guide lists index keywords in the "FULL" view as not available by
    default, so whether they arrive depends on the key's entitlement as well.
    In this package's development testing, they did arrive on the same key
    whose author keywords came back empty. An index term pybliometrics cannot
    parse (its 4.4.1 release raises ``KeyError`` on one without text, as 4.5
    still does) leaves that document's index keywords ``NA`` with a warning,
    and the rest of its row is kept. The R twin drops such a term and keeps
    the others.

    When "references" is included, a ``references`` column is added: one
    DataFrame per document (one row per cited work), using pybliometrics' own
    native field set (``position``, ``id``, ``doi``, ``title``, ``authors``,
    ``authors_auid``, ``authors_affiliationid``, ``sourcetitle``,
    ``publicationyear``, ``coverDate``, ``volume``, ``issue``, ``first``,
    ``last``, ``citedbycount``, ``type``, ``text``, ``fulltext``). A document
    with no resolvable references yields a zero-row DataFrame, so
    the column can always be unnested.

    ``cache_dir`` and ``resume`` checkpoint per identifier, the way
    [`fetch_plan`][scopusflow.fetch.fetch_plan] checkpoints per cell, pickled
    rather than written as parquet/csv, since a row here can carry a nested
    DataFrame. Only a successful retrieval is checkpointed: a failed identifier
    still yields its warned-about NA row in the returned frame, but is retried
    on the next resumed run, never read back as data. Worth setting whenever
    ``include`` is used: Abstract Retrieval draws on its own weekly quota,
    smaller than and separate from Search's, and every identifier costs its own
    request, so re-running an interrupted batch without a cache re-spends quota
    already spent. ``resume=False`` reads no checkpoint and retrieves every
    identifier again, overwriting the checkpoint of each identifier retrieved
    successfully. Distinct identifiers can share a checkpoint file, as
    "10.1/a.b" and "10.1/a-b" do, and ``resume`` does not check which
    identifier a checkpoint holds, so such a pair should not share a
    ``cache_dir``. A ``cache_dir`` that cannot be written raises ``ValueError``
    before any request, and a checkpoint that cannot be written all the same is
    warned about, the row kept in the result, as in
    [`fetch_plan`][scopusflow.fetch.fetch_plan]. Relying on pybliometrics' own
    on-disk response cache (its ``refresh`` parameter, keyed by identifier and
    view under its configured cache directory) is enough to avoid repeat
    network calls for the *same* identifier across script runs; this checkpoint
    is for batch-level progress and resumability across *many* identifiers, a
    separate concern. Further keyword arguments, ``refresh`` among them, go to
    pybliometrics' ``AbstractRetrieval``.

    The number of Abstract Retrieval requests made, and the most recently
    parsed remaining-quota figure (from pybliometrics'
    ``get_key_remaining_quota()``), are attached as ``result.attrs["n_requests"]``
    and ``result.attrs["quota"]``, since this is a materially more expensive
    operation than a search call. ``n_requests`` counts each document once,
    however many REF pages pybliometrics requested for it, so it is a count
    of documents fetched. A failed retrieval counts once, since the API has
    usually answered it with an error. A row read from a ``cache_dir``
    checkpoint sent no request and is not counted. Nor is a row pybliometrics
    answered from its own cache, recognised by a cache file older than the
    call or by a response with no headers kept. Such a row's ``citations``
    and other fields date from when that cache file was written. Pass
    ``refresh=True`` to fetch every identifier from the API.

    A 403 (an entitlement gate, most often on the requested view or field)
    raises
    [`ScopusFlowForbiddenError`][scopusflow.exceptions.ScopusFlowForbiddenError]
    and stops the batch immediately, naming the view and identifier, where a
    generic failure would leave the caller guessing. Repeating the identical
    failure for every remaining identifier would serve nobody: entitlement is a
    property of the account, so a retry cannot succeed.

    A spent weekly quota stops the requests too, but returns what the batch
    holds, since it usually arrives part-way through. pybliometrics raises
    ``Scopus429Error`` once every configured key has been refused. The rows
    already retrieved are kept, each identifier not retrieved gets an NA row
    that is not checkpointed, and one
    [`ScopusFlowQuotaWarning`][scopusflow.exceptions.ScopusFlowQuotaWarning]
    names the identifier reached and how many were not retrieved. A checkpoint
    in ``cache_dir`` is still read after the stop.

    Raises [`ScopusFlowConfigError`][scopusflow.exceptions.ScopusFlowConfigError]
    before any request when ``pybliometrics.init()`` has not been called in the
    session. An identifier served from its checkpoint sends no request, so a
    batch resumed wholly from checkpoints runs without ``init()``.
    """
    if by not in _ID_TYPES:
        raise ValueError("by must be one of 'doi', 'eid', 'scopus_id'.")
    include = tuple(include) if include else ()
    if not set(include) <= _KNOWN_INCLUDE:
        raise ValueError(
            "include must name only 'references', 'keywords' and 'index_terms'."
        )
    if "references" in include and view not in _VIEWS_WITH_REFERENCES:
        raise ValueError('include="references" needs view="FULL" or view="REF".')
    # The REF response carries no author keywords and META/META_ABS none
    # either, so accepting them would only give a silently all-NA column.
    if "keywords" in include and view != "FULL":
        raise ValueError('include="keywords" needs view="FULL".')
    # Index keywords come only with FULL too (Scopus API Guide, V1, p. 29).
    if "index_terms" in include and view != "FULL":
        raise ValueError('include="index_terms" needs view="FULL".')
    id_type = _ID_TYPES[by]
    id_column = "doi" if by == "doi" else "scopus_id"

    ids = _clean_ids(ids, by)

    columns = [
        *ABSTRACT_COLUMNS,
        *(column for name, column in _INCLUDE_COLUMNS.items() if name in include),
    ]

    cache = _writable_cache(cache_dir)

    # Resolve the dependency once: a missing pybliometrics is a setup error and
    # must be raised in its own right, where masquerading as every id failing
    # to retrieve would hide the account-level cause.
    from pybliometrics.scopus import AbstractRetrieval  # lazy; needs a key
    try:
        # A separate, defensive import: pybliometrics.exception is an internal
        # module, outside pybliometrics.scopus's own public surface, so a
        # minimal test double or an unexpected future reorganisation should
        # degrade to generic exception handling below, which stops it breaking
        # every call to this function.
        from pybliometrics.exception import Scopus403Error
    except ImportError:
        class Scopus403Error(Exception):  # never actually raised; see above
            pass
    try:
        # Imported apart from Scopus403Error, so that a module carrying only
        # one of the two still has the other caught.
        from pybliometrics.exception import Scopus429Error
    except ImportError:
        class Scopus429Error(Exception):  # never actually raised; see above
            pass

    n_requests = 0
    quota = None
    rows = []
    # The identifier at which the quota ran out, and how many identifiers were
    # left without a row from the API from then on.
    quota_spent_at = None
    not_retrieved = 0
    for i, ident in enumerate(ids, start=1):
        checkpoint = (
            _find_abstract_checkpoint(cache, view, include, ident)
            if cache is not None else None
        )
        if checkpoint is not None and resume:
            cached = _read_abstract_checkpoint(checkpoint)
            if cached is None:
                warnings.warn(
                    f"The checkpoint {checkpoint} could not be read back, so it "
                    "was discarded and the identifier retrieved again. An "
                    "interrupted run can leave a checkpoint half-written.",
                    UserWarning,
                    stacklevel=2,
                )
            else:
                logger.info("%d/%d: %s loaded from cache.", i, len(ids), ident)
                rows.append(cached)
                continue

        if quota_spent_at is not None:
            # No request after the quota ran out. Checkpoints above are still
            # read, since they cost nothing, and the NA row is not
            # checkpointed, so a run after the reset retrieves the identifier.
            not_retrieved += 1
            rows.append(_na_row(columns, id_column, ident, include))
            continue

        logger.info("Retrieving %d/%d: %s", i, len(ids), ident)
        # Checked before the first request, outside the per-identifier handler
        # below: an uninitialised pybliometrics would fail every identifier in
        # turn, each failure warned about as if that identifier alone were at
        # fault. An identifier served from its checkpoint sends no request, so
        # a batch resumed wholly from checkpoints needs no init().
        require_init()
        # Each identifier adds to n_requests exactly once, and only when a
        # request went out, so a step that fails after the response arrived
        # does not count the same request twice. A failed construction counts
        # as one: the API has usually answered it with an error.
        ab = None
        t0 = time.time()
        try:
            ab = AbstractRetrieval(ident, id_type=id_type, view=view, **kwargs)
            if _served_from_cache(ab, t0, kwargs.get("refresh", False)):
                logger.info("%d/%d: %s read from pybliometrics' cache.", i, len(ids), ident)
            else:
                n_requests += 1
            # Optional: an object without these methods (a plain dict, or a
            # minimal stand-in) simply reports no quota, and does not fail
            # the whole retrieval over a metadata nicety. Both lookups are
            # guarded, since an object carrying only the first would otherwise
            # fail here after a successful retrieval and record an NA row.
            get_quota = getattr(ab, "get_key_remaining_quota", None)
            get_reset = getattr(ab, "get_key_reset_time", None)
            remaining = get_quota() if callable(get_quota) else None
            if remaining is not None:
                reset = get_reset() if callable(get_reset) else None
                quota = {"remaining": remaining, "reset": reset}
            row = _abstract_row(ab, include=include, ident=ident)
        except Scopus403Error as exc:
            n_requests += 1
            remaining_ids = len(ids) - i
            # The FULL/REF alternative is only sensible advice when the failed
            # view was one of the two reference-carrying views; a 403 on a
            # plain META/META_ABS retrieval is an ordinary entitlement gap.
            alternative = (
                ' or, if you have not already, try the other of "FULL"/"REF"'
                if view in _VIEWS_WITH_REFERENCES else ""
            )
            raise ScopusFlowForbiddenError(
                f'Abstract Retrieval refused view="{view}" (HTTP 403) for {ident!r}. '
                "This usually means your Scopus API key's entitlement does not cover "
                "the requested view or field; contact your Scopus/Elsevier account "
                f"holder or institutional administrator to request access{alternative}. "
                "Stopping rather "
                f"than repeating the same failure for the remaining {remaining_ids} "
                "identifier(s) (this entitlement is an account-level property, not a "
                "per-document one, so it will not succeed on retry)."
            ) from exc
        except Scopus429Error:
            # pybliometrics raises this only once every configured key has
            # been refused, and documents it as a depleted weekly quota, so no
            # later identifier could succeed until the reset. The batch stops
            # requesting but keeps its rows: unlike a 403, a spent quota
            # usually arrives part-way through, and raising would discard the
            # rows already paid for when cache_dir is not set.
            if ab is None:
                n_requests += 1
            quota_spent_at = ident
            not_retrieved += 1
            rows.append(_na_row(columns, id_column, ident, include))
            continue
        except Exception:  # one bad id must not sink the batch
            if ab is None:  # a response that arrived was counted above
                n_requests += 1
            warnings.warn(
                f"Could not retrieve abstract for {ident!r}; recording NA row.",
                stacklevel=2,
            )
            # Never checkpointed: a timeout or a 5xx may pass, and a persisted
            # NA row would be read back as data on every later resume. The
            # identifier is retried instead, at the cost of one request.
            rows.append(_na_row(columns, id_column, ident, include))
            continue

        if cache is not None:
            # A failed write keeps the row its request paid for, as a cell's
            # does in fetch_plan().
            try:
                _write_abstract_checkpoint(row, cache, view, include, ident)
            except OSError:
                warnings.warn(
                    _unwritable_warning(repr(ident), cache, "its record is"),
                    stacklevel=2,
                )
        rows.append(row)

    if quota_spent_at is not None:
        warnings.warn(
            _quota_warning(quota_spent_at, not_retrieved, quota),
            ScopusFlowQuotaWarning,
            stacklevel=2,
        )

    out = pd.DataFrame(rows, columns=columns)
    out.attrs["n_requests"] = n_requests
    out.attrs["quota"] = quota
    return out

ABSTRACT_COLUMNS module-attribute

ABSTRACT_COLUMNS = ['scopus_id', 'doi', 'title', 'abstract', 'publication', 'date', 'year', 'citations']

corpus

corpus(records, by='doi', view='FULL', cache_dir=None, resume=True, keywords='author', **kwargs)

Enrich records (from fetch_plan) with keywords (the author keywords by default, or the index keywords Scopus assigns) and structured references via Abstract Retrieval, returning a minimal, uniform shape close to what OpenAlex's works API already returns: id, scopus_id, title, year, keywords (a list of strings per row) and references (a DataFrame of cited works per row). write_corpus writes the result in theoryforge's corpus schema. This is meant for downstream tools that want to consume Scopus output without writing their own parsing layer, for example for keyword co-occurrence or citation-network analysis. It does not replace to_bibtex/ to_ris, which keep their own established interchange formats.

by selects which column of records ("doi" or "scopus_id") to look identifiers up by. view is passed to scopus_abstract and defaults to "FULL", which returns a document's whole bibliography in one request. "REF" serves it about 40 references per request, and pybliometrics pages REF itself, so the list still arrives whole, at a cost of ceil(N / 40) requests for N references (see scopus_abstract's documentation for the entitlement each view needs). cache_dir and resume are passed through unchanged, and are worth setting for anything beyond a handful of records, since this performs at least one Abstract Retrieval request per record, against its own, smaller weekly quota, separate from Search's. What that cost came to is carried through from scopus_abstract: the number of requests made, counting each document once, and the most recently parsed remaining-quota figure are attached as result.attrs["n_requests"] and result.attrs["quota"]. Three more attrs carry what write_corpus records of the corpus's provenance. query is a list of the field-wrapped search expression of each cell of the records' plan, or, for records without a plan, of the distinct values of their own query column. retrieved_at is when the search that found the records ran, as the records hold it, and keywords is the kind of keyword chosen. query and retrieved_at are None when records carries neither.

scopus_id is the document's own Scopus identifier, as Abstract Retrieval gives it, or the scopus_id of records where Abstract Retrieval gives none, as for a failed lookup, and missing when neither has one.

A record whose identifier is missing (NA/None) or blank is dropped, with a warning naming how many.

The keywords column here is list[str] per row, split out of scopus_abstract's joined authkeywords and idxterms strings, empty when the document has none or the field is unavailable. Only the "FULL" view carries author keywords, so under "REF" the references alone are requested and every keywords entry is empty. references carries pybliometrics' own native reference field set (see scopus_abstract's documentation).

keywords chooses which keywords fill that column: "author" (the default) for the author keywords, "index" for the index keywords Scopus assigns, or "both" for the author keywords followed by each index keyword not already among them. The match is on the exact text, so a term that differs only in case appears twice. "index" and "both" need view="FULL", since the "REF" response carries no keywords of either kind, and raise ValueError under "REF" before any request. Index keywords and author keywords give different co-occurrence results, and scopus_abstract describes how each is chosen and the entitlement each depends on. Either choice costs the same request per record.

Source code in src/scopusflow/corpus.py
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
def corpus(
    records: pd.DataFrame,
    by: str = "doi",
    view: str = "FULL",
    cache_dir: str | None = None,
    resume: bool = True,
    keywords: str = "author",
    **kwargs,
) -> pd.DataFrame:
    """Enrich ``records`` (from [`fetch_plan`][scopusflow.fetch.fetch_plan])
    with keywords (the author keywords by default, or the index keywords Scopus
    assigns) and structured references via Abstract Retrieval, returning a
    minimal, uniform shape close to what OpenAlex's ``works`` API already
    returns: ``id``, ``scopus_id``, ``title``, ``year``, ``keywords`` (a list
    of strings per row) and ``references`` (a DataFrame of cited works per
    row). [`write_corpus`][scopusflow.corpus.write_corpus] writes the result in
    theoryforge's corpus schema. This is meant for downstream tools that want
    to consume Scopus output without writing their own parsing layer, for
    example for keyword co-occurrence or citation-network analysis. It does not
    replace [`to_bibtex`][scopusflow.export.to_bibtex]/
    [`to_ris`][scopusflow.export.to_ris], which keep their own established
    interchange formats.

    ``by`` selects which column of ``records`` ("doi" or "scopus_id") to look
    identifiers up by. ``view`` is passed to
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] and defaults to
    "FULL", which returns a document's whole bibliography in one request. "REF"
    serves it about 40 references per request, and pybliometrics pages REF
    itself, so the list still arrives whole, at a cost of ``ceil(N / 40)``
    requests for N references (see
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract]'s documentation
    for the entitlement each view needs). ``cache_dir`` and ``resume`` are
    passed through unchanged, and are worth setting for anything beyond a
    handful of records, since this performs at least one Abstract Retrieval
    request per record, against its own, smaller weekly quota, separate from
    Search's. What that cost came to is carried through from
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract]: the number of
    requests made, counting each document once, and the most recently parsed
    remaining-quota figure are attached as ``result.attrs["n_requests"]`` and
    ``result.attrs["quota"]``. Three more ``attrs`` carry what
    [`write_corpus`][scopusflow.corpus.write_corpus] records of the corpus's
    provenance. ``query`` is a list of the field-wrapped search expression of
    each cell of the records' plan, or, for records without a plan, of the
    distinct values of their own ``query`` column. ``retrieved_at`` is when the
    search that found the records ran, as the records hold it, and ``keywords``
    is the kind of keyword chosen. ``query`` and ``retrieved_at`` are ``None``
    when ``records`` carries neither.

    ``scopus_id`` is the document's own Scopus identifier, as Abstract
    Retrieval gives it, or the ``scopus_id`` of ``records`` where Abstract
    Retrieval gives none, as for a failed lookup, and missing when neither
    has one.

    A record whose identifier is missing (``NA``/``None``) or blank is
    dropped, with a warning naming how many.

    The `keywords` column here is `list[str]` per row, split out of
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract]'s joined
    `authkeywords` and `idxterms` strings, empty when the document has none or
    the field is unavailable. Only the "FULL" view carries author keywords, so
    under "REF" the references alone are requested and every `keywords` entry
    is empty. `references` carries pybliometrics' own native reference field
    set (see [`scopus_abstract`][scopusflow.abstract.scopus_abstract]'s
    documentation).

    ``keywords`` chooses which keywords fill that column: "author" (the
    default) for the author keywords, "index" for the index keywords Scopus
    assigns, or "both" for the author keywords followed by each index keyword
    not already among them. The match is on the exact text, so a term that
    differs only in case appears twice. "index" and "both" need
    ``view="FULL"``, since the "REF" response carries no keywords of either
    kind, and raise ``ValueError`` under "REF" before any request. Index
    keywords and author keywords give different co-occurrence results, and
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] describes how each
    is chosen and the entitlement each depends on. Either choice costs the same
    request per record.
    """
    if keywords not in _KEYWORD_SOURCES:
        raise ValueError(
            f"keywords must be one of 'author', 'index' or 'both', not {keywords!r}."
        )
    # Checked here, so that the error names this function's own argument, where
    # scopus_abstract() would name the include value it was passed.
    if keywords != "author" and view != "FULL":
        raise ValueError(
            f'keywords="{keywords}" needs view="FULL", since the REF response '
            "carries no index keywords."
        )
    required = {by, "title", "year"}
    if not required.issubset(records.columns):
        raise ValueError(
            f"records must have {sorted(required)} columns "
            "(as fetch_plan() returns)."
        )

    # The search behind the records, read before they are subset: the
    # field-wrapped query each cell sent, as the search record states it, and
    # when the search ran.
    expression = _expression(records, as_plan(records.attrs.get("plan")))
    retrieved_at = records.attrs.get("retrieved_at")

    ids = records[by]
    # Blank text counts as missing too, since scopus_abstract() refuses both.
    keep = ~ids.map(_is_missing_identifier).astype(bool)
    n_dropped = int((~keep).sum())
    if n_dropped:
        warnings.warn(
            f"Dropped {n_dropped} record(s) with no usable {by}.",
            stacklevel=2,
        )
    if not keep.any():
        raise ValueError("records has no usable identifiers to look up.")
    records = records.loc[keep].reset_index(drop=True)

    # The REF response carries no author keywords (scopus_abstract() refuses
    # the combination), so under that view only the references are requested
    # and every record's keywords come back empty, as in the R twin. Under
    # FULL, only the kinds of keyword asked for are included, so a default
    # corpus keeps the checkpoint names, and so the checkpoints, that
    # releases up to 0.3.0 wrote.
    include = ("references",)
    if view == "FULL":
        if keywords in ("author", "both"):
            include += ("keywords",)
        if keywords in ("index", "both"):
            include += ("index_terms",)
    ab = scopus_abstract(
        list(records[by]), by=by, view=view, include=include,
        cache_dir=cache_dir, resume=resume, **kwargs,
    )

    def _split(column):
        if column not in ab:
            return [[] for _ in range(len(ab))]
        return [[] if pd.isna(kw) else [k.strip() for k in kw.split(";")]
                for kw in ab[column]]

    author, index = _split("authkeywords"), _split("idxterms")
    if keywords == "author":
        terms = author
    elif keywords == "index":
        terms = index
    else:
        # dict.fromkeys() keeps the first of each repeated text, as R's
        # unique() does, so the author keywords come first and an index
        # keyword they already hold is not repeated.
        terms = [list(dict.fromkeys(a + i)) for a, i in zip(author, index, strict=True)]

    # The document's own identifier, from Abstract Retrieval. A failed lookup
    # leaves it missing there, and the records' own identifier fills the gap
    # where they carry one, as in the R twin.
    scopus_id = [_text(v) for v in ab["scopus_id"]]
    if "scopus_id" in records.columns:
        scopus_id = [s if s is not None else _text(r)
                     for s, r in zip(scopus_id, records["scopus_id"], strict=True)]

    out = pd.DataFrame({
        # The identifier used to look each record up, taken from `records`
        # directly, and never read back from `ab`: scopus_abstract() only
        # echoes the input identifier verbatim in its "doi"/"scopus_id"
        # column on a failed lookup (to keep the failing row identifiable);
        # on success that column holds whatever Scopus itself returns, which
        # is usually but not guaranteedly identical to the input.
        "id": list(records[by]),
        "scopus_id": pd.Series([pd.NA if s is None else s for s in scopus_id], dtype=object),
        "title": records["title"],
        "year": records["year"],
        "keywords": terms,
        "references": list(ab["references"]),
    })
    # A freshly constructed frame carries no attrs, so the wrapper that spends
    # the most quota would otherwise be the one reporting none of it.
    out.attrs.update(ab.attrs)
    # The provenance write_corpus() records, as JSON-safe values: the
    # expressions as a list, the time as the records hold it.
    out.attrs["query"] = expression
    out.attrs["retrieved_at"] = retrieved_at
    out.attrs["keywords"] = keywords
    return out

ScopusFlowForbiddenError

Bases: Exception

Raised when the Scopus API refuses a request (HTTP 403), most often because the configured key's entitlement does not cover the requested Abstract Retrieval view or field.

Source code in src/scopusflow/exceptions.py
25
26
27
28
class ScopusFlowForbiddenError(Exception):
    """Raised when the Scopus API refuses a request (HTTP 403), most often
    because the configured key's entitlement does not cover the requested
    Abstract Retrieval view or field."""

ScopusFlowConfigError

Bases: RuntimeError

Raised before any request when pybliometrics has not been initialised in this Python session.

pybliometrics 4 reads its configuration only when pybliometrics.init() is called, so a configuration file on disk is not enough by itself. Run import pybliometrics; pybliometrics.init() once per session before the first search. It reads the configuration file, or creates it and asks for your key when there is none. scopusflow never calls init() itself.

Source code in src/scopusflow/exceptions.py
31
32
33
34
35
36
37
38
39
40
class ScopusFlowConfigError(RuntimeError):
    """Raised before any request when pybliometrics has not been initialised
    in this Python session.

    pybliometrics 4 reads its configuration only when ``pybliometrics.init()``
    is called, so a configuration file on disk is not enough by itself. Run
    ``import pybliometrics; pybliometrics.init()`` once per session before the
    first search. It reads the configuration file, or creates it and asks for
    your key when there is none. scopusflow never calls ``init()`` itself.
    """

ScopusFlowQuotaWarning

Bases: UserWarning

Warned once when scopus_abstract stops a batch because the API key's weekly quota is spent.

pybliometrics raises Scopus429Error only after it has tried every configured key, and documents the error as a depleted quota, so no later identifier in the batch could succeed. The rows already retrieved are returned, and each identifier not retrieved gets a row of NA. It is the counterpart of the R twin's scopus_warning_quota_exceeded.

Source code in src/scopusflow/exceptions.py
43
44
45
46
47
48
49
50
51
52
53
class ScopusFlowQuotaWarning(UserWarning):
    """Warned once when
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] stops a batch
    because the API key's weekly quota is spent.

    pybliometrics raises ``Scopus429Error`` only after it has tried every
    configured key, and documents the error as a depleted quota, so no later
    identifier in the batch could succeed. The rows already retrieved are
    returned, and each identifier not retrieved gets a row of ``NA``. It is the
    counterpart of the R twin's ``scopus_warning_quota_exceeded``.
    """

Records

Normalise results into one stable schema, save a record set and read it back, export to reference-manager formats, and write a corpus in theoryforge's schema.

to_records

to_records(results, query=None, view=None, fields='core')

Normalise a pybliometrics ScopusSearch().results list (named tuples) or a list of dicts into a tidy RECORD_COLUMNS DataFrame.

Whatever the query type, the columns are the same, so the downstream DOI, diff and analysis helpers can rely on them.

query is written into the query column of every row, which holds None when no query is given. fetch_plan passes each cell's query as sent, its year limit included. The core columns are entry_number (the row's position, from 1), scopus_id (the EID without its 2-s2.0- prefix), doi, title, authors (see below), year (the leading four digits of the cover date, as a whole number), date (the cover date as Scopus gives it), publication (the source title), citations (the cited-by count, as a whole number) and query. A field the record lacks is a missing value that pandas.isna recognises (None, NA or NaN, depending on the column's dtype), and an empty results gives a zero-row frame with the same columns.

What authors holds depends on the view. The Search API's dc:creator, pybliometrics' creator, is the first author alone under either view, in the indexed form Scopus uses ("Zhang F."). The author list arrives only under view="COMPLETE", and pybliometrics builds author_names from it as "Surname, Given" names. Those names are joined here with "; ", as the R twin joins them, and an author whose given name is null or empty is written as the surname alone. A row without the list falls back to creator. A STANDARD record therefore names its first author, and top says so once when it tallies a STANDARD harvest by author. The API lists at most 100 authors per record. The R twin builds the same string from the same entry in the common case, and the two part company at two edges. pybliometrics builds author_names only when every author object carries both a surname and a given-name key, so where one does not, this twin keeps the first author while the R twin lists every author. Where a surname is null or empty, pybliometrics writes the given name after a bare comma, and the R twin uses the indexed name.

When view="COMPLETE", an authkeywords column is added: the author- supplied keywords the Scopus Search API returns under that view, as a single string in Scopus's own " | "-delimited form (missing when the document has none, or when the API omits the field for a given key's entitlement: this was observed directly against a live, otherwise fully-entitled key during development, on documents that do carry author keywords in Scopus itself, so a column missing throughout is more likely an entitlement gap worth raising with your Scopus/Elsevier account contact than genuinely absent data). Under any other view, including the default None, no authkeywords column is added.

fields is "core" by default, which builds the columns above and no others. "extended" also keeps fields the Search API sends with every record, at no extra quota, which pybliometrics already parses: the EXTENDED_COLUMNS and, under view="COMPLETE", an abstract column. Elsevier's Search views table lists every one of them under both views except the abstract (dc:description), which only COMPLETE carries. A COMPLETE harvest therefore returns up to 25 abstracts with each request, while scopus_abstract spends one request per record from the separate Abstract Retrieval quota.

doc_type and doc_type_description hold the document-type code and name Scopus gives the record, such as "ar" and "Article". The code is the one the DOCTYPE() search field takes, and Elsevier's search tips list the codes with their names: ar for an article, re a review, cp a conference paper, er an erratum and others. A retracted item carries tb, which that list leaves out. The Scopus Content Coverage Guide (Elsevier, 2023, section 3.1) defines an erratum as a report of an error, correction or retraction of a previously published paper, so er covers retraction notices as well as corrections. The guide gives an article the type Retracted once a retraction notice for it is published. These codes are the labels Scopus assigns, and they do not screen a set for retractions, since Scopus does not label every retracted item (Ortega and Delgado-Quirós, 2024). Check a set against the Retraction Watch database, which Crossref makes openly available, before relying on it. Errata can also be left out of the search itself with AND NOT DOCTYPE(er), joined by scopus_query, which brackets the rest of the query first, since Scopus applies AND NOT last.

source_type holds the source type, such as "Journal". volume, issue, pages and pubmed_id are text, and so is issn, the print ISSN as Scopus writes it, without the hyphen, so its leading zeros survive. open_access holds the code Scopus sends in its openaccess field, as a whole number. Elsevier's Scopus APIs Getting Started Guide (Elsevier, 2023, section 17) illustrates its values with 0 for a record that is not open access, 1 for gold and 2 for green. The column therefore holds a status code, which can take values other than 0 and 1. A fields value other than "core" or "extended" raises ValueError.

References

Elsevier. (2023). Scopus APIs getting started guide (Version 1). https://dev.elsevier.com/guides/Scopus%20API%20Guide_V1_20230907.pdf

Elsevier. (2023). Scopus content coverage guide. https://supportcontent.elsevier.com/Support%20Hub/Scopus/Scopus_ContentCoverage_Guide_WEB.pdf

Ortega, J. L., & Delgado-Quirós, L. (2024). The indexation of retracted literature in seven principal scholarly databases: A coverage comparison of Dimensions, OpenAlex, PubMed, Scilit, Scopus, The Lens and Web of Science. Scientometrics, 129(7), 3769–3785. https://doi.org/10.1007/s11192-024-05034-y

Source code in src/scopusflow/records.py
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
def to_records(
    results, query: str | None = None, view: str | None = None, fields: str = "core"
) -> pd.DataFrame:
    """Normalise a pybliometrics ``ScopusSearch().results`` list (named tuples)
    or a list of dicts into a tidy
    [`RECORD_COLUMNS`][scopusflow.records.RECORD_COLUMNS] DataFrame.

    Whatever the query type, the columns are the same, so the downstream DOI,
    diff and analysis helpers can rely on them.

    ``query`` is written into the ``query`` column of every row, which holds
    ``None`` when no query is given. [`fetch_plan`][scopusflow.fetch.fetch_plan]
    passes each cell's query as sent, its year limit included. The core columns
    are ``entry_number`` (the row's position, from 1), ``scopus_id`` (the EID
    without its ``2-s2.0-`` prefix), ``doi``, ``title``, ``authors`` (see
    below), ``year`` (the leading four digits of the cover date, as a whole
    number), ``date`` (the cover date as Scopus gives it), ``publication`` (the
    source title), ``citations`` (the cited-by count, as a whole number) and
    ``query``. A field the record lacks is a missing value that ``pandas.isna``
    recognises (``None``, ``NA`` or ``NaN``, depending on the column's dtype),
    and an empty ``results`` gives a zero-row frame with the same columns.

    What ``authors`` holds depends on the view. The Search API's
    ``dc:creator``, pybliometrics' ``creator``, is the first author alone under
    either view, in the indexed form Scopus uses (``"Zhang F."``). The author
    list arrives only under ``view="COMPLETE"``, and pybliometrics builds
    ``author_names`` from it as ``"Surname, Given"`` names. Those names are
    joined here with ``"; "``, as the R twin joins them, and an author whose
    given name is null or empty is written as the surname alone. A row without
    the list falls back to ``creator``. A ``STANDARD`` record therefore names
    its first author, and [`top`][scopusflow.records.top] says so once when it
    tallies a ``STANDARD`` harvest by author. The API lists at most 100 authors
    per record. The R twin builds the same string from the same entry in the
    common case, and the two part company at two edges. pybliometrics builds
    ``author_names`` only when every author object carries both a ``surname``
    and a ``given-name`` key, so where one does not, this twin keeps the first
    author while the R twin lists every author. Where a surname is null or
    empty, pybliometrics writes the given name after a bare comma, and the R
    twin uses the indexed name.

    When ``view="COMPLETE"``, an ``authkeywords`` column is added: the author-
    supplied keywords the Scopus Search API returns under that view, as a
    single string in Scopus's own ``" | "``-delimited form (missing when the
    document has none, or when the API omits the field for a given key's
    entitlement: this was observed directly against a live, otherwise
    fully-entitled key during development, on documents that do carry author
    keywords in Scopus itself, so a column missing throughout is more likely an
    entitlement gap worth raising with your Scopus/Elsevier account contact
    than genuinely absent data). Under any other ``view``, including the
    default ``None``, no ``authkeywords`` column is added.

    ``fields`` is ``"core"`` by default, which builds the columns above and no
    others. ``"extended"`` also keeps fields the Search API sends with every
    record, at no extra quota, which pybliometrics already parses: the
    [`EXTENDED_COLUMNS`][scopusflow.records.EXTENDED_COLUMNS] and, under
    ``view="COMPLETE"``, an ``abstract`` column. Elsevier's Search views table
    lists every one of them under both views except the abstract
    (``dc:description``), which only ``COMPLETE`` carries. A ``COMPLETE``
    harvest therefore returns up to 25 abstracts with each request, while
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] spends one request
    per record from the separate Abstract Retrieval quota.

    ``doc_type`` and ``doc_type_description`` hold the document-type code and
    name Scopus gives the record, such as ``"ar"`` and ``"Article"``. The code
    is the one the ``DOCTYPE()`` search field takes, and Elsevier's search tips
    list the codes with their names: ``ar`` for an article, ``re`` a review,
    ``cp`` a conference paper, ``er`` an erratum and others. A retracted item
    carries ``tb``, which that list leaves out. The Scopus Content Coverage
    Guide (Elsevier, 2023, section 3.1) defines an erratum as a report of an
    error, correction or retraction of a previously published paper, so ``er``
    covers retraction notices as well as corrections. The guide gives an
    article the type Retracted once a retraction notice for it is published.
    These codes are the labels Scopus assigns, and they do not screen a set for
    retractions, since Scopus does not label every retracted item (Ortega and
    Delgado-Quirós, 2024). Check a set against the Retraction Watch database,
    which Crossref makes openly available, before relying on it. Errata can
    also be left out of the search itself with ``AND NOT DOCTYPE(er)``, joined
    by [`scopus_query`][scopusflow.query.scopus_query], which brackets the rest
    of the query first, since Scopus applies AND NOT last.

    ``source_type`` holds the source type, such as ``"Journal"``. ``volume``,
    ``issue``, ``pages`` and ``pubmed_id`` are text, and so is ``issn``, the
    print ISSN as Scopus writes it, without the hyphen, so its leading zeros
    survive. ``open_access`` holds the code Scopus sends in its ``openaccess``
    field, as a whole number. Elsevier's Scopus APIs Getting Started Guide
    (Elsevier, 2023, section 17) illustrates its values with 0 for a record
    that is not open access, 1 for gold and 2 for green. The column therefore
    holds a status code, which can take values other than 0 and 1. A
    ``fields`` value other than ``"core"`` or ``"extended"`` raises
    ``ValueError``.

    References
    ----------
    Elsevier. (2023). *Scopus APIs getting started guide* (Version 1).
    https://dev.elsevier.com/guides/Scopus%20API%20Guide_V1_20230907.pdf

    Elsevier. (2023). *Scopus content coverage guide*.
    https://supportcontent.elsevier.com/Support%20Hub/Scopus/Scopus_ContentCoverage_Guide_WEB.pdf

    Ortega, J. L., & Delgado-Quirós, L. (2024). The indexation of retracted
    literature in seven principal scholarly databases: A coverage comparison of
    Dimensions, OpenAlex, PubMed, Scilit, Scopus, The Lens and Web of Science.
    *Scientometrics*, *129*(7), 3769–3785.
    https://doi.org/10.1007/s11192-024-05034-y
    """
    _check_fields(fields)
    add_keywords = view == "COMPLETE"
    extended = fields == "extended"
    columns = _record_columns(view, fields)
    rows = []
    for i, r in enumerate(results or [], start=1):
        date = _get(r, "coverDate")
        row = {
            "entry_number": i,
            "scopus_id": _scopus_id(_get(r, "eid")),
            "doi": _get(r, "doi"),
            "title": _get(r, "title"),
            # author_names exists only under COMPLETE, and creator is the first author.
            "authors": _join_authors(_get(r, "author_names")) or _get(r, "creator"),
            "year": _year(date),
            "date": date,
            "publication": _get(r, "publicationName"),
            "citations": _citations(_get(r, "citedby_count")),
            "query": query,
        }
        if add_keywords:
            row["authkeywords"] = _get(r, "authkeywords") or pd.NA
        if extended:
            for column, source in _EXTENDED_SOURCES.items():
                row[column] = _get(r, source)
            row["open_access"] = _open_access(_get(r, "openaccess"))
            if add_keywords:
                row["abstract"] = _get(r, "description")
        rows.append(row)
    return pd.DataFrame(rows, columns=columns)

RECORD_COLUMNS module-attribute

RECORD_COLUMNS = ['entry_number', 'scopus_id', 'doi', 'title', 'authors', 'year', 'date', 'publication', 'citations', 'query']

EXTENDED_COLUMNS module-attribute

EXTENDED_COLUMNS = ['doc_type', 'doc_type_description', 'source_type', 'volume', 'issue', 'pages', 'issn', 'pubmed_id', 'open_access']

scopus_combine

scopus_combine(*sets, dedupe=False)

Bind several record frames into one, renumbering entry_number.

This is the safe way to merge separate harvests: a plain concat leaves duplicate entry numbers, and nothing then records how many records went in.

De-duplication compares DOIs as extract_dois does, without resolver prefixes such as https://doi.org/ or doi: and ignoring case. It keeps the first record of each group, in the order the sets were given, and a record with neither an identifier nor a DOI cannot be matched and is kept.

Under the default rule, records with the same Scopus identifier are one record, and a record without an identifier joins the identified record that carries its DOI. Records with different identifiers are never merged, even when they share a DOI. Scopus holds duplicate records of one paper (Valderrama-Zurián et al., 2015), but it also gives one DOI to distinct papers (Franceschini et al., 2015), and a review loses a wrongly removed record for good where a surviving duplicate is caught at screening. Such DOIs are counted in doi_conflicts, which the search record reports, so that they can be checked by hand. A record without an identifier whose DOI two identified records carry is kept once, apart from both. Records without an identifier that share a DOI no identified record carries are one record.

"id_or_doi" merges every record linked to another by a shared identifier or a shared DOI, directly or through a chain of such links. It suits a set whose shared DOIs have been checked and found to be duplicates. "id_then_doi" keys each record on its identifier, failing that on its DOI lower-cased but not otherwise cleaned. That is the rule the R twin's scopus_combine() applied in its 0.4.0 release and earlier, so it reproduces a set those releases de-duplicated.

Parameters:

Name Type Description Default
*sets

Two or more record frames, or a single list of them.

()
dedupe bool | str

Whether, and by which rule, to keep once the records that are one record. False, the default, keeps every row. True, also spelled "id_or_shared_doi", treats records as one when they share a Scopus identifier, or when they share a DOI and one of them lacks an identifier. "id_or_doi" also merges records with different identifiers that share a DOI, and "id_then_doi" applies the rule of the R twin's 0.4.0 and earlier releases.

False

Returns:

Type Description
DataFrame

The merged records. Attributes describing a single retrieval, among them plan, total_results and cell_totals, are not carried over, since a set built from several harvests is none of them. Five survive a merge, each only when every input carries it. retrieved_at is the earliest of the inputs' times, retrieved_until the latest (an input without one counting by its retrieved_at) and scopusflow_version a sorted list of every contributing version, while paging and view are kept only where the inputs agree on them. The search record of a merged set is then dated from its first retrieval to its last, as the R twin's is. The merge itself is recorded in attrs["combined"], a dict of n_in (records supplied), n_out (records kept), n_removed, deduplicated, rule (the rule applied, None when none was) and doi_conflicts (how many DOIs records with different identifiers share, None when no rule was applied). That count exists only at the moment of the merge, and PRISMA-S asks for it (item 16), so scopus_search_report reads it back from there and states how many duplicates were removed and how.

Raises:

Type Description
ValueError

When dedupe is neither a bool nor the name of a rule, or an input is not a record frame.

References

Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Errors in DOI indexing by bibliometric databases. Scientometrics, 102(3), 2181–2186. https://doi.org/10.1007/s11192-014-1503-4

Valderrama-Zurián, J.-C., Aguilar-Moya, R., Melero-Fuentes, D., & Aleixandre-Benavent, R. (2015). A systematic analysis of duplicate records in Scopus. Journal of Informetrics, 9(3), 570–576. https://doi.org/10.1016/j.joi.2015.05.002

Examples:

>>> import scopusflow as sf
>>> baseline = sf.example_records()
>>> later = sf.example_records()
>>> merged = sf.scopus_combine(baseline, later, dedupe=True)
>>> merged.attrs["combined"]["n_in"]
276
>>> len(merged)
149
Source code in src/scopusflow/combine.py
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
def scopus_combine(*sets, dedupe: bool | str = False) -> pd.DataFrame:
    """Bind several record frames into one, renumbering ``entry_number``.

    This is the safe way to merge separate harvests: a plain ``concat`` leaves
    duplicate entry numbers, and nothing then records how many records went in.

    De-duplication compares DOIs as
    [`extract_dois`][scopusflow.diff.extract_dois] does, without resolver
    prefixes such as ``https://doi.org/`` or ``doi:`` and ignoring case. It
    keeps the first record of each group, in the order the sets were given, and
    a record with neither an identifier nor a DOI cannot be matched and is
    kept.

    Under the default rule, records with the same Scopus identifier are one
    record, and a record without an identifier joins the identified record
    that carries its DOI. Records with different identifiers are never merged,
    even when they share a DOI. Scopus holds duplicate records of one paper
    (Valderrama-Zurián et al., 2015), but it also gives one DOI to distinct
    papers (Franceschini et al., 2015), and a review loses a wrongly removed
    record for good where a surviving duplicate is caught at screening. Such
    DOIs are counted in ``doi_conflicts``, which the search record reports, so
    that they can be checked by hand. A record without an identifier whose DOI
    two identified records carry is kept once, apart from both. Records without
    an identifier that share a DOI no identified record carries are one record.

    ``"id_or_doi"`` merges every record linked to another by a shared
    identifier or a shared DOI, directly or through a chain of such links. It
    suits a set whose shared DOIs have been checked and found to be
    duplicates. ``"id_then_doi"`` keys each record on its identifier, failing
    that on its DOI lower-cased but not otherwise cleaned. That is the rule the
    R twin's ``scopus_combine()`` applied in its 0.4.0 release and earlier, so
    it reproduces a set those releases de-duplicated.

    Parameters
    ----------
    *sets:
        Two or more record frames, or a single list of them.
    dedupe:
        Whether, and by which rule, to keep once the records that are one
        record. ``False``, the default, keeps every row. ``True``, also spelled
        ``"id_or_shared_doi"``, treats records as one when they share a Scopus
        identifier, or when they share a DOI and one of them lacks an
        identifier. ``"id_or_doi"`` also merges records with different
        identifiers that share a DOI, and ``"id_then_doi"`` applies the rule of
        the R twin's 0.4.0 and earlier releases.

    Returns
    -------
    pandas.DataFrame
        The merged records. Attributes describing a single retrieval, among
        them ``plan``, ``total_results`` and ``cell_totals``, are not carried
        over, since a set built from several harvests is none of them. Five
        survive a merge, each only when every input carries it.
        ``retrieved_at`` is the earliest of the inputs' times,
        ``retrieved_until`` the latest (an input without one counting by its
        ``retrieved_at``) and ``scopusflow_version`` a sorted list of every
        contributing version, while ``paging`` and ``view`` are kept only where
        the inputs agree on them. The search record of a merged set is then
        dated from its first retrieval to its last, as the R twin's is. The
        merge itself is recorded in ``attrs["combined"]``, a dict of ``n_in``
        (records supplied), ``n_out`` (records kept), ``n_removed``,
        ``deduplicated``, ``rule`` (the rule applied, ``None`` when none was)
        and ``doi_conflicts`` (how many DOIs records with different
        identifiers share, ``None`` when no rule was applied). That count
        exists only at the moment of the merge, and PRISMA-S asks for it (item
        16), so
        [`scopus_search_report`][scopusflow.report.scopus_search_report] reads
        it back from there and states how many duplicates were removed and how.

    Raises
    ------
    ValueError
        When ``dedupe`` is neither a bool nor the name of a rule, or an input
        is not a record frame.

    References
    ----------
    Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Errors in DOI
    indexing by bibliometric databases. *Scientometrics*, *102*(3), 2181–2186.
    https://doi.org/10.1007/s11192-014-1503-4

    Valderrama-Zurián, J.-C., Aguilar-Moya, R., Melero-Fuentes, D., &
    Aleixandre-Benavent, R. (2015). A systematic analysis of duplicate records
    in Scopus. *Journal of Informetrics*, *9*(3), 570–576.
    https://doi.org/10.1016/j.joi.2015.05.002

    Examples
    --------
    >>> import scopusflow as sf
    >>> baseline = sf.example_records()
    >>> later = sf.example_records()
    >>> merged = sf.scopus_combine(baseline, later, dedupe=True)
    >>> merged.attrs["combined"]["n_in"]
    276
    >>> len(merged)
    149
    """
    rule = _rule(dedupe)
    frames = list(sets)
    if len(frames) == 1 and isinstance(frames[0], (list, tuple)):
        frames = list(frames[0])
    if not frames or not all(isinstance(f, pd.DataFrame) for f in frames):
        raise ValueError("All inputs to scopus_combine() must be record frames.")

    out = pd.concat([_without_attrs(f) for f in frames], ignore_index=True)
    n_in = len(out)
    conflicts = None
    if rule is not None:
        keys, conflicts = _record_groups(out, rule)
        out = out[~pd.Series(keys, dtype=object).duplicated().to_numpy()].reset_index(
            drop=True)
    if len(out):
        out["entry_number"] = range(1, len(out) + 1)
    else:
        out = out.reindex(columns=list(out.columns) or list(RECORD_COLUMNS))
    # The rule is recorded beside the counts, since the search record has to
    # say how records were matched.
    out.attrs["combined"] = {
        "n_in": n_in,
        "n_out": len(out),
        "n_removed": n_in - len(out),
        "deduplicated": rule is not None,
        "rule": rule,
        "doi_conflicts": conflicts,
    }
    _bind_provenance(out, frames)
    return out

write_records

write_records(records, path)

Save a record set, choosing the format from the file extension.

A .parquet file keeps the provenance in records.attrs: the plan, the retrieval time, the per-cell totals, the paging mode, the software version and any merge record, all of which scopus_search_report reads back. They are stored as a JSON block under the scopusflow key of the file's schema metadata, so they survive whatever the pandas version. pandas 1.5 to 2.0 drop attrs when writing parquet, and later versions keep them under a key of their own. Attributes set by hand as a SearchPlan, a datetime or a DataFrame are written in the JSON forms fetch_plan uses, and the frame passed in is left as it was. Parquet needs pyarrow, which the parquet extra installs.

A .csv file is portable plain text, written with LF line endings on every platform, and holds the columns only. Save a baseline you will compare against later as parquet, so that it keeps the record of how it was retrieved.

Parameters:

Name Type Description Default
records DataFrame

A record frame, such as fetch_plan returns.

required
path

The file to write, ending in .parquet or .csv. It is written at exactly this path, and its directory must already exist.

required

Returns:

Type Description
DataFrame

records, unchanged.

Examples:

>>> import tempfile, pathlib
>>> import scopusflow as sf
>>> path = pathlib.Path(tempfile.mkdtemp()) / "baseline.csv"
>>> _ = sf.write_records(sf.example_records(), path)
>>> len(sf.read_records(path))
138
Source code in src/scopusflow/io.py
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
def write_records(records: pd.DataFrame, path) -> pd.DataFrame:
    """Save a record set, choosing the format from the file extension.

    A ``.parquet`` file keeps the provenance in ``records.attrs``: the plan,
    the retrieval time, the per-cell totals, the paging mode, the software
    version and any merge record, all of which
    [`scopus_search_report`][scopusflow.report.scopus_search_report] reads
    back. They are stored as a JSON block under the ``scopusflow`` key of the
    file's schema metadata, so they survive whatever the pandas version. pandas
    1.5 to 2.0 drop ``attrs`` when writing parquet, and later versions keep
    them under a key of their own. Attributes set by hand as a ``SearchPlan``,
    a ``datetime`` or a DataFrame are written in the JSON forms
    [`fetch_plan`][scopusflow.fetch.fetch_plan] uses, and the frame passed in
    is left as it was. Parquet needs pyarrow, which the ``parquet`` extra
    installs.

    A ``.csv`` file is portable plain text, written with LF line endings on
    every platform, and holds the columns only. Save a baseline you will
    compare against later as parquet, so that it keeps the record of how it
    was retrieved.

    Parameters
    ----------
    records:
        A record frame, such as [`fetch_plan`][scopusflow.fetch.fetch_plan]
        returns.
    path:
        The file to write, ending in ``.parquet`` or ``.csv``. It is written
        at exactly this path, and its directory must already exist.

    Returns
    -------
    pandas.DataFrame
        ``records``, unchanged.

    Examples
    --------
    >>> import tempfile, pathlib
    >>> import scopusflow as sf
    >>> path = pathlib.Path(tempfile.mkdtemp()) / "baseline.csv"
    >>> _ = sf.write_records(sf.example_records(), path)
    >>> len(sf.read_records(path))
    138
    """
    if not isinstance(records, pd.DataFrame):
        raise ValueError("records must be a record frame.")
    fmt = _format(path)
    if fmt == "csv":
        records.to_csv(path, index=False, lineterminator="\n")
        return records
    pa, pq = _pyarrow()
    safe = json_safe(records.attrs)
    block = json.dumps({"schema": _SCHEMA, "attrs": safe}, allow_nan=False)
    # Recent pyarrow copies attrs into its own pandas metadata, and warns and
    # drops them when they are not JSON. A shallow copy carrying the safe form
    # gives it something it can write and leaves the caller's frame alone.
    shallow = records.copy(deep=False)
    shallow.attrs = safe
    table = pa.Table.from_pandas(shallow, preserve_index=False)
    metadata = dict(table.schema.metadata or {})
    metadata[_METADATA_KEY] = block.encode("utf-8")
    pq.write_table(table.replace_schema_metadata(metadata), path)
    return records

read_records

read_records(path)

Read a record set saved by write_records.

Both formats come back with the record schema imposed: the text columns, scopus_id among them, as strings, and year and citations as nullable integers. A CSV left to pandas' own inference reads an all-digit identifier as a number, or as a float once one identifier is missing, and a de-duplicating scopus_combine then matches none of its records against a fresh harvest. The columns fields="extended" adds are read the same way, open_access as a nullable integer and the rest as strings, so an ISSN such as "00368075" keeps its leading zeros.

A parquet file comes back with the provenance it was saved with in attrs, so its search record reads as the original's did. A parquet file written by DataFrame.to_parquet under pandas 2.1 or later returns the attrs pandas stored. A CSV file carries none.

Parameters:

Name Type Description Default
path

The file to read, ending in .parquet or .csv.

required

Returns:

Type Description
DataFrame

The records, with any provenance in attrs.

Source code in src/scopusflow/io.py
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
def read_records(path) -> pd.DataFrame:
    """Read a record set saved by
    [`write_records`][scopusflow.io.write_records].

    Both formats come back with the record schema imposed: the text columns,
    ``scopus_id`` among them, as strings, and ``year`` and ``citations`` as
    nullable integers. A CSV left to pandas' own inference reads an all-digit
    identifier as a number, or as a float once one identifier is missing, and
    a de-duplicating [`scopus_combine`][scopusflow.combine.scopus_combine] then
    matches none of its records against a fresh harvest. The columns
    ``fields="extended"`` adds are read the same way, ``open_access`` as a
    nullable integer and the rest as strings, so an ISSN such as ``"00368075"``
    keeps its leading zeros.

    A parquet file comes back with the provenance it was saved with in
    ``attrs``, so its search record reads as the original's did. A parquet
    file written by ``DataFrame.to_parquet`` under pandas 2.1 or later
    returns the ``attrs`` pandas stored. A CSV file carries none.

    Parameters
    ----------
    path:
        The file to read, ending in ``.parquet`` or ``.csv``.

    Returns
    -------
    pandas.DataFrame
        The records, with any provenance in ``attrs``.
    """
    fmt = _format(path)
    if fmt == "parquet":
        _, pq = _pyarrow()
    if not Path(path).exists():
        raise FileNotFoundError(f"File not found: {path}")
    if fmt == "csv":
        frame = _read_csv_records(path)
        attrs: dict = {}
    else:
        table = pq.read_table(path)
        frame = table.to_pandas()
        attrs = _stored_attrs(table.schema.metadata, frame.attrs, path)
        for column in _CHECKPOINT_TEXT:
            if column in frame.columns:
                frame[column] = frame[column].astype("string")
        frame = _as_whole_numbers(frame)
    frame.attrs = attrs
    return frame

example_records

example_records()

Return the bundled example harvest as a records frame.

A fresh copy is returned on each call, so a caller may edit the result without disturbing anyone else's.

Returns:

Type Description
DataFrame

138 records with the standard RECORD_COLUMNS schema, covering 2015 to 2024. The harvest is complete, so the rows per year are the real publications per year for its search. scopus_id is empty throughout, these records not having come from Scopus; eleven records carry no DOI and two no source title, exactly as they arrive.

Examples:

>>> import scopusflow as sf
>>> records = sf.example_records()
>>> len(records)
138
>>> sorted(records.columns) == sorted(sf.RECORD_COLUMNS)
True
Source code in src/scopusflow/data.py
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def example_records() -> pd.DataFrame:
    """Return the bundled example harvest as a records frame.

    A fresh copy is returned on each call, so a caller may edit the result
    without disturbing anyone else's.

    Returns
    -------
    pandas.DataFrame
        138 records with the standard ``RECORD_COLUMNS`` schema, covering 2015
        to 2024. The harvest is complete, so the rows per
        year are the real publications per year for its search. ``scopus_id`` is
        empty throughout, these records not having come from Scopus; eleven
        records carry no DOI and two no source title, exactly as they arrive.

    Examples
    --------
    >>> import scopusflow as sf
    >>> records = sf.example_records()
    >>> len(records)
    138
    >>> sorted(records.columns) == sorted(sf.RECORD_COLUMNS)
    True
    """
    return _load().copy()

to_bibtex

to_bibtex(records)

Render records as a BibTeX string, one entry per row, with citation keys made unique within the export.

A set retrieved with fields="extended" (see to_records) carries each record's Scopus document type, and the entry is typed by it. A conference paper (cp) or conference review (cr) becomes an @inproceedings, a book chapter (ch) an @incollection and a book (bk) a @book. Every other record, and every record of a set retrieved with the core fields, is an @article. The source title of a conference paper or chapter goes under booktitle, braced as the title is, and a journal article's under journal. A book in a book series gives the series under series, and a book outside a series, whose source title Scopus gives as the book's own, names no source. The entries also carry volume, number, pages (101--117), issn, doi, the author keywords under keywords, joined by commas, the abstract under abstract and the Scopus identifier in note (Scopus ID: <scopus_id>). A set retrieved under view="COMPLETE" carries the keywords, and the abstract only when it was also retrieved with fields="extended". A table from scopus_abstract carries its abstracts, and its keywords under include="keywords". A value the record lacks is left out, never filled in.

Names are written family name first, with a comma. A STANDARD-view name such as Zhang F., which BibTeX would read as given name Zhang and family name F., is written Zhang, F., and run-together initials are spaced (Kitchin, J. R.), since BibTeX takes J.R. for one given name. Names that already carry a comma, as under COMPLETE, and names given first are written as they stand. The authors column and the citation keys are unchanged. In the title, every word with a capital after its first letter is braced ({DNA}, {CRISPR-Cas9}), so bibliography styles cannot lower-case it.

Source code in src/scopusflow/export.py
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
def to_bibtex(records: pd.DataFrame) -> str:
    """Render records as a BibTeX string, one entry per row, with citation keys
    made unique within the export.

    A set retrieved with ``fields="extended"`` (see
    [`to_records`][scopusflow.records.to_records]) carries each record's Scopus
    document type, and the entry is typed by it. A conference paper (``cp``)
    or conference review (``cr``) becomes an ``@inproceedings``, a book chapter
    (``ch``) an ``@incollection`` and a book (``bk``) a ``@book``. Every other
    record, and every record of a set retrieved with the core fields, is an
    ``@article``. The source title of a conference paper or chapter goes under
    ``booktitle``, braced as the title is, and a journal article's under
    ``journal``. A book in a book series gives the series under ``series``, and
    a book outside a series, whose source title Scopus gives as the book's own,
    names no source. The entries also carry ``volume``, ``number``, ``pages``
    (``101--117``), ``issn``, ``doi``, the author keywords under ``keywords``,
    joined by commas, the abstract under ``abstract`` and the Scopus identifier
    in ``note`` (``Scopus ID: <scopus_id>``). A set retrieved under
    ``view="COMPLETE"`` carries the keywords, and the abstract only when it was
    also retrieved with ``fields="extended"``. A table from
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] carries its
    abstracts, and its keywords under ``include="keywords"``. A value the
    record lacks is left out, never filled in.

    Names are written family name first, with a comma. A STANDARD-view name
    such as ``Zhang F.``, which BibTeX would read as given name ``Zhang`` and
    family name ``F.``, is written ``Zhang, F.``, and run-together initials are
    spaced (``Kitchin, J. R.``), since BibTeX takes ``J.R.`` for one given name.
    Names that already carry a comma, as under COMPLETE, and names given first
    are written as they stand. The ``authors`` column and the citation keys are
    unchanged. In the title, every word with a capital after its first letter
    is braced (``{DNA}``, ``{CRISPR-Cas9}``), so bibliography styles cannot
    lower-case it."""
    if not isinstance(records, pd.DataFrame):
        raise ValueError("records must be a pandas DataFrame.")
    rows = [row for _, row in records.iterrows()]
    keys = _disambiguate(
        [_bibtex_key(r.get("authors"), r.get("year"), r.get("scopus_id")) for r in rows]
    )
    return "\n\n".join(_bibtex_entry(r, k) for r, k in zip(rows, keys, strict=True))

to_ris

to_ris(records)

Render records as an RIS string, one record per row.

A name ending in initials is written with a comma (Zhang, F.), as for to_bibtex, but run-together initials keep Scopus's spelling. The source title goes under T2, the tag Scopus's own RIS export uses and reference managers read as the publication title. Each record names the database (DB - Scopus) and, when it has an identifier, gives the EID as the accession number (AN - 2-s2.0-<scopus_id>).

Each record is a JOUR, unless a set retrieved with fields="extended" gives its Scopus document type. A conference paper is then a CPAPER, a conference review a CONF, a book chapter a CHAP and a book a BOOK. A book outside a series has no T2, since Zotero reads a book's T2 as its series, and a book in a book series gives the series there. The record also carries VL, IS, SP and EP from the page range, SN, the DOI as DO, the document type as M3, the abstract as AB, one KW line per author keyword and the Scopus identifier as N1 - Scopus ID: <scopus_id>. The keywords and the abstract come from the same sets as for to_bibtex. Keywords are split at the " | " the Search API joins them with and at the "; " scopus_abstract joins its own with. A value the record lacks is left out, never filled in.

Source code in src/scopusflow/export.py
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
def to_ris(records: pd.DataFrame) -> str:
    """Render records as an RIS string, one record per row.

    A name ending in initials is written with a comma (``Zhang, F.``), as for
    [`to_bibtex`][scopusflow.export.to_bibtex], but run-together initials keep
    Scopus's spelling. The source title goes under ``T2``, the tag Scopus's own
    RIS export uses and reference managers read as the publication title. Each
    record names the database (``DB  - Scopus``) and, when it has an
    identifier, gives the EID as the accession number
    (``AN  - 2-s2.0-<scopus_id>``).

    Each record is a ``JOUR``, unless a set retrieved with ``fields="extended"``
    gives its Scopus document type. A conference paper is then a ``CPAPER``, a
    conference review a ``CONF``, a book chapter a ``CHAP`` and a book a
    ``BOOK``. A book outside a series has no ``T2``, since Zotero reads a
    book's ``T2`` as its series, and a book in a book series gives the series
    there. The record also carries ``VL``, ``IS``, ``SP`` and ``EP`` from the
    page range, ``SN``, the DOI as ``DO``, the document type as ``M3``, the
    abstract as ``AB``, one ``KW`` line per author keyword and the Scopus
    identifier as ``N1  - Scopus ID: <scopus_id>``. The keywords and the
    abstract come from the same sets as for
    [`to_bibtex`][scopusflow.export.to_bibtex]. Keywords are split at the
    ``" | "`` the Search API joins them with and at the ``"; "``
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract] joins its own
    with. A value the record lacks is left out, never filled in."""
    if not isinstance(records, pd.DataFrame):
        raise ValueError("records must be a pandas DataFrame.")
    return "\n\n".join(_ris_entry(row) for _, row in records.iterrows())

write_corpus

write_corpus(corpus, path, id)

Write the corpus to path as JSON in theoryforge's corpus schema.

The document is to_theoryforge_corpus's, which describes the keys, the keywords and the provenance block, and it is returned too. The file is UTF-8, with no escaping beyond what JSON requires, sorted keys, no spaces between tokens and one line feed at the end, so the same corpus gives the same bytes from either twin. allow_nan=False stops the write before any file is opened should a NaN reach the document, since JSON has no such value. path is written exactly as given, and its parent directory is assumed to exist already.

Source code in src/scopusflow/corpus.py
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
def write_corpus(corpus: pd.DataFrame, path, id: str) -> dict:
    """Write the corpus to ``path`` as JSON in theoryforge's corpus schema.

    The document is
    [`to_theoryforge_corpus`][scopusflow.corpus.to_theoryforge_corpus]'s, which
    describes the keys, the keywords and the ``provenance`` block, and it is
    returned too. The file is UTF-8, with no escaping beyond what JSON
    requires, sorted keys, no spaces between tokens and one line feed at the
    end, so the same corpus gives the same bytes from either twin.
    ``allow_nan=False`` stops the write before any file is opened should a
    ``NaN`` reach the document, since JSON has no such value. ``path`` is
    written exactly as given, and its parent directory is assumed to exist
    already.
    """
    doc = to_theoryforge_corpus(corpus, id)
    text = json.dumps(doc, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False, allow_nan=False)
    Path(path).write_bytes((text + "\n").encode("utf-8"))
    return doc

to_theoryforge_corpus

to_theoryforge_corpus(corpus, id)

The corpus as a document in theoryforge's corpus schema, as a dict.

Turns a corpus result into the literature corpus that theoryforge's read_corpus, litmap and landscape read, following the corpus schema theoryforge publishes (schema/corpus.schema.json in its repository). write_corpus writes the same document to a JSON file. Neither imports theoryforge, and neither makes a request.

theoryforge compares references as exact strings, so each cited work gets one key across the whole corpus, its Scopus identifier, as scopus:<id>. Scopus gives a cited work's DOI in some bibliographies only, and in different cases. A key built from the DOI wherever there is one would therefore split one work into several nodes and lose the co-citation edges between them. A reference that carries only a DOI takes the Scopus identifier that a document or reference elsewhere in the corpus pairs with the same DOI, the documents' own identifiers counting first. A DOI that nothing pairs with is keyed as doi:<doi>, without its resolver prefix and with its ASCII letters lower-cased, since DOIs are case-insensitive over ASCII (DOI Handbook, section 2.4). A reference with neither is dropped, and a UserWarning says how many. Each document is keyed the same way, from its scopus_id or else from an id that is a DOI or a Scopus identifier, so a citation of a document in the corpus and the document itself share one key. A document with neither raises ValueError.

Keywords are trimmed, lower-cased with str.lower() and made unique, keeping the first of each, and empty ones are dropped, so "Graphene" and "graphene " count as one keyword. The R twin lower-cases with tolower(). The two agree on the Latin, Greek and Cyrillic letters in common use and differ on a few special cases, such as the Turkish capital I with a dot and a capital sigma ending a word, which Python writes as a final sigma. A missing year is written as null, whether it is None, NaN or pd.NA, and a whole-number year of any dtype as a plain integer. A missing title is left out, since the schema admits a null year and not a null title.

The document also carries a provenance object, which the schema admits beside its own fields: source ("Scopus via scopusflow <version>"), query, retrieved_at (in UTC, as YYYY-MM-DDTHH:MM:SSZ), keywords, n_records, n_references_dropped and n_records_without_keywords. query, retrieved_at and keywords come from the attrs that corpus attaches, and are None (null) when the frame lacks them. query is a string, or a list of strings when the records were searched with several expressions. Author keywords can come back empty under some entitlements (see scopus_abstract), which leaves theoryforge's keyword themes empty whatever the writer does, and n_records_without_keywords shows when that happened.

id is a non-empty string naming the corpus, written as its top-level id. It is required, since a corpus result carries no name of its own. Returns a dict with id, provenance, records and schema_version ("1.0").

Source code in src/scopusflow/corpus.py
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
def to_theoryforge_corpus(corpus: pd.DataFrame, id: str) -> dict:
    """The corpus as a document in theoryforge's corpus schema, as a dict.

    Turns a [`corpus`][scopusflow.corpus.corpus] result into the literature
    corpus that theoryforge's ``read_corpus``, ``litmap`` and ``landscape``
    read, following the corpus schema theoryforge publishes
    (``schema/corpus.schema.json`` in its repository).
    [`write_corpus`][scopusflow.corpus.write_corpus] writes the same document
    to a JSON file. Neither imports theoryforge, and neither makes a request.

    theoryforge compares references as exact strings, so each cited work gets
    one key across the whole corpus, its Scopus identifier, as
    ``scopus:<id>``. Scopus gives a cited work's DOI in some bibliographies
    only, and in different cases. A key built from the DOI wherever there is
    one would therefore split one work into several nodes and lose
    the co-citation edges between them. A reference that carries only a DOI
    takes the Scopus identifier that a document or reference
    elsewhere in the corpus pairs with the same DOI, the documents' own
    identifiers counting first. A DOI that nothing pairs with is keyed as
    ``doi:<doi>``, without its resolver prefix and with its ASCII letters
    lower-cased, since DOIs are case-insensitive over ASCII (DOI Handbook,
    section 2.4). A reference with neither is dropped, and a ``UserWarning``
    says how many. Each document is keyed the same way, from its
    ``scopus_id`` or else from an ``id`` that is a DOI or a Scopus
    identifier, so a citation of a document in the corpus and the document
    itself share one key. A document with neither raises ``ValueError``.

    Keywords are trimmed, lower-cased with ``str.lower()`` and made unique,
    keeping the first of each, and empty ones are dropped, so ``"Graphene"``
    and ``"graphene "`` count as one keyword. The R twin lower-cases with
    ``tolower()``. The two agree on the Latin, Greek and Cyrillic letters in
    common use and differ on a few special cases, such as the Turkish capital
    I with a dot and a capital sigma ending a word, which Python writes as a
    final sigma. A missing year is written as ``null``,
    whether it is ``None``, ``NaN`` or ``pd.NA``, and a whole-number year of
    any dtype as a plain integer. A missing title is left out, since the
    schema admits a null year and not a null title.

    The document also carries a ``provenance`` object, which the schema
    admits beside its own fields: ``source`` (``"Scopus via scopusflow
    <version>"``), ``query``, ``retrieved_at`` (in UTC, as
    ``YYYY-MM-DDTHH:MM:SSZ``), ``keywords``, ``n_records``,
    ``n_references_dropped`` and ``n_records_without_keywords``. ``query``,
    ``retrieved_at`` and ``keywords`` come from the ``attrs`` that
    [`corpus`][scopusflow.corpus.corpus] attaches, and are ``None`` (``null``)
    when the frame lacks them. ``query`` is a string, or a list of strings when
    the records were searched with several expressions. Author keywords can
    come back empty under some entitlements (see
    [`scopus_abstract`][scopusflow.abstract.scopus_abstract]), which leaves
    theoryforge's keyword themes empty whatever the writer does, and
    ``n_records_without_keywords`` shows when that happened.

    ``id`` is a non-empty string naming the corpus, written as its top-level
    ``id``. It is required, since a [`corpus`][scopusflow.corpus.corpus] result
    carries no name of its own. Returns a dict with ``id``, ``provenance``,
    ``records`` and ``schema_version`` (``"1.0"``).
    """
    if not isinstance(id, str) or not id.strip():
        raise ValueError(
            "id must be a non-empty string naming the corpus, since a corpus() "
            "result carries no name of its own."
        )
    if not isinstance(corpus, pd.DataFrame) or not set(_CORPUS_COLUMNS) <= set(corpus.columns):
        raise ValueError(
            f"corpus must be a DataFrame with {list(_CORPUS_COLUMNS)} columns "
            "(as corpus() returns)."
        )
    n = len(corpus)

    # Each document's own identifiers: its scopus_id, else an id that is a
    # Scopus identifier (a corpus looked up by scopus_id), and an id that is
    # a DOI (a corpus looked up by DOI).
    ids = [_text(v) for v in corpus["id"]]
    record_doi = [d if d is not None and _DOI_SHAPE.match(d) else None
                  for d in map(_doi, ids)]
    given = corpus["scopus_id"] if "scopus_id" in corpus.columns else [None] * n
    record_sid = [
        _sid(s) or (_sid(i) if d is None else None)
        for s, i, d in zip(given, ids, record_doi, strict=True)
    ]
    cited = [_cited(refs) for refs in corpus["references"]]

    # Every document and reference carrying both says which identifier a DOI
    # belongs to. The first pair met wins, the documents' own coming first.
    sid_of_doi: dict[str, str] = {}
    pairs = list(zip(record_sid, record_doi, strict=True))
    pairs += [pair for refs in cited for pair in refs]
    for sid, doi in pairs:
        if sid is not None and doi is not None:
            sid_of_doi.setdefault(doi, sid)

    def key(sid, doi):
        sid = sid or sid_of_doi.get(doi)
        if sid is not None:
            return f"scopus:{sid}"
        return f"doi:{doi}" if doi is not None else None

    record_key = [key(s, d) for s, d in zip(record_sid, record_doi, strict=True)]
    unkeyed = [i for i, k in enumerate(record_key) if k is None]
    if unkeyed:
        shown = ", ".join(str(i) for i in unkeyed[:10])
        raise ValueError(
            f"{len(unkeyed)} record(s) of corpus (row {shown}, counting from 0) "
            "carry neither a Scopus identifier nor a DOI to key on."
        )

    dropped = 0
    records = []
    for row, own_key, refs in zip(corpus.itertuples(index=False), record_key, cited,
                                  strict=True):
        keys = [key(s, d) for s, d in refs]
        dropped += sum(k is None for k in keys)
        record = {
            "id": own_key,
            "keywords": _keywords(row.keywords),
            "references": list(dict.fromkeys(k for k in keys if k is not None)),
        }
        if isinstance(row.title, str) and row.title.strip():
            record["title"] = row.title.strip()
        record["year"] = _year(row.year)
        records.append(record)
    if dropped:
        warnings.warn(
            f"Dropped {dropped} reference(s) with neither a Scopus identifier nor "
            "a DOI, which no key can name.",
            UserWarning,
            stacklevel=2,
        )

    # Imported here: __version__ is bound only once the package's own imports,
    # this module's among them, have returned.
    from . import __version__

    kind = corpus.attrs.get("keywords")
    provenance = {
        "keywords": kind if isinstance(kind, str) else None,
        "n_records": n,
        "n_records_without_keywords": sum(not r["keywords"] for r in records),
        "n_references_dropped": dropped,
        "query": _query(corpus.attrs.get("query")),
        "retrieved_at": _utc(corpus.attrs.get("retrieved_at")),
        "source": f"Scopus via scopusflow {__version__}",
    }
    return {"id": id, "provenance": provenance, "records": records, "schema_version": "1.0"}

Reporting

Write the search up for a methods section, to the PRISMA-S standard, from what the plan and the harvest already record.

scopus_search_report

scopus_search_report(x, plan=None, file=None)

Assemble a reproducible record of a Scopus search.

Turns a harvest, or a plan not yet run, into the search-strategy record a systematic review has to report: what was searched, exactly how, when, how much came back, and how much the API said there was. The record prints as a readable report, formats as a methods paragraph fit to paste into a manuscript, and writes as Markdown. The reporting standard it follows is PRISMA-S (Rethlefsen et al., 2021), together with the identification counts of the PRISMA 2020 flow diagram.

Everything in the record comes from the objects handed to it. The dates of the search are the retrieved_at and retrieved_until attributes fetch_plan attaches, never the current time. A harvest resumed over several days, or a baseline merged with its update, is dated from its first retrieval to its last. PRISMA-S item 13 then gives the date of the last search, as the checklist asks. Stamps that fall on one UTC day give that single date. The years are those the cells' limits sent, and years with gaps between them are listed, since a span would claim years no cell searched. The number of records the API reported as matching is the per-cell accounting in cell_totals, or, for a set from a single search, its total_results, never an inference from the number of rows; and the duplicates removed are those scopus_combine recorded removing, matched by the rule it recorded. Where an attribute is absent, as it is for a frame read back from CSV, for the bundled corpus, and for a harvest with a cell resumed from a checkpoint written without a manifest by an earlier version of scopusflow, the record says the field is unrecorded and fills nothing in. This matters most for completeness: a harvest whose reported total is unknown is never described as exhaustive.

A harvest stopped through should_stop, as the app's Cancel stops one, says so. The record gives how many of the planned cells ran and which years they covered, marks the rest "not run" and calls the harvest incomplete, keeping the totals the cells that ran recorded. The R twin's scopus_fetch_plan() can also cap the records per cell, and its record then names the cap in the cells it stopped, the completeness line and item 9. A frame carrying attrs["max_results"] is described the same way here, so the two records stay one rendering, although fetch_plan takes no cap and so never records one.

The PRISMA-S map is decided the same way. Items the package holds evidence for (the database and platform, that Scopus was searched on its own, the full strategy and the limits where a plan records them, the date, the totals, and de-duplication where it was performed) are listed as supplied. Without a plan, the year limit is unrecorded, not absent, and the record says so. The rest, among them peer review of the strategy, grey literature and citation searching, are listed as the author's to supply, because the package has no way to know them.

Parameters:

Name Type Description Default
x

A records frame, which supplies the counts and the retrieval provenance and, through attrs["plan"], the plan; or a bare SearchPlan for a search not yet run. The attributes are read in the JSON-safe forms fetch_plan writes (the plan as a dict, the time as ISO 8601 text, the cell totals as a list of dicts) and in the richer forms a guide may set by hand (a SearchPlan, a datetime, a DataFrame).

required
plan SearchPlan | None

The plan describing x. Supply it when a records frame does not carry one, for instance one read back from CSV or a merged set. An explicit plan takes precedence over the one x carries. A one-cell plan supplied for a frame without per-cell counts takes the frame's own row count and total_results.

None
file

Path at which to write the record as Markdown. A file is written only when this is supplied, and only to the exact path given, so nothing is written to the working directory unless asked.

None

Returns:

Type Description
SearchReport

The record. print shows it, report.format(style="paragraph") gives the methods paragraph.

References

Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. (2021). PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews, 10, 39. https://doi.org/10.1186/s13643-020-01542-z

Examples:

A search described but not yet run. The record says so throughout rather than implying figures it cannot have.

>>> import scopusflow as sf
>>> plan = sf.SearchPlan('"graphene supercapacitor"', years=range(2015, 2025),
...                      field="TITLE-ABS-KEY", partition="year")
>>> report = sf.scopus_search_report(plan)
>>> report.n_records is None
True

The same search after a harvest. The bundled corpus stands in for one, since Scopus records may not be redistributed, so the attributes a live retrieval records are set here by hand. Their values are illustrative, and the records came from OpenAlex, so the record below describes no search that was run.

>>> from datetime import datetime, timezone
>>> records = sf.example_records()
>>> records.attrs["plan"] = plan
>>> records.attrs["retrieved_at"] = datetime(2026, 7, 22, 9, 15,
...                                          tzinfo=timezone.utc)
>>> records.attrs["scopusflow_version"] = "0.3.0"
>>> report = sf.scopus_search_report(records)
>>> report.n_records
138
>>> "22 July 2026" in report.format(style="paragraph")
True
Source code in src/scopusflow/report.py
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
def scopus_search_report(x, plan: SearchPlan | None = None, file=None) -> SearchReport:
    """Assemble a reproducible record of a Scopus search.

    Turns a harvest, or a plan not yet run, into the search-strategy record a
    systematic review has to report: what was searched, exactly how, when, how
    much came back, and how much the API said there was. The record prints as a
    readable report, formats as a methods paragraph fit to paste into a
    manuscript, and writes as Markdown. The reporting standard it follows is
    PRISMA-S (Rethlefsen et al., 2021), together with the identification counts
    of the PRISMA 2020 flow diagram.

    Everything in the record comes from the objects handed to it. The dates of
    the search are the ``retrieved_at`` and ``retrieved_until`` attributes
    [`fetch_plan`][scopusflow.fetch.fetch_plan] attaches, never the current
    time. A harvest resumed over several days, or a baseline merged with its
    update, is dated from its first retrieval to its last. PRISMA-S item 13
    then gives the date of the last search, as the checklist asks. Stamps that
    fall on one UTC day give that single date. The years are those the cells'
    limits sent, and years with gaps between them are listed, since a span
    would claim years no cell searched. The number of records the API reported
    as matching is the per-cell accounting in ``cell_totals``, or, for a set
    from a single search, its ``total_results``, never an inference from the
    number of rows; and the duplicates removed are those
    [`scopus_combine`][scopusflow.combine.scopus_combine] recorded removing,
    matched by the rule it recorded. Where an attribute is absent, as it is for
    a frame read back from CSV, for the bundled corpus, and for a harvest with
    a cell resumed from a checkpoint written without a manifest by an earlier
    version of scopusflow, the record says the field is unrecorded and fills
    nothing in. This matters most for completeness: a harvest whose reported
    total is unknown is never described as exhaustive.

    A harvest stopped through ``should_stop``, as the app's Cancel stops one,
    says so. The record gives how many of the planned cells ran and which
    years they covered, marks the rest "not run" and calls the harvest
    incomplete, keeping the totals the cells that ran recorded. The R twin's
    ``scopus_fetch_plan()`` can also cap the records per cell, and its record
    then names the cap in the cells it stopped, the completeness line and
    item 9. A frame carrying ``attrs["max_results"]`` is described the same
    way here, so the two records stay one rendering, although
    [`fetch_plan`][scopusflow.fetch.fetch_plan] takes no cap and so never
    records one.

    The PRISMA-S map is decided the same way. Items the package holds evidence
    for (the database and platform, that Scopus was searched on its own, the
    full strategy and the limits where a plan records them, the date, the
    totals, and de-duplication where it was performed) are listed as supplied.
    Without a plan, the year limit is unrecorded, not absent, and the record
    says so. The rest, among them peer review of the strategy, grey literature
    and citation searching, are listed as the author's to supply, because the
    package has no way to know them.

    Parameters
    ----------
    x:
        A records frame, which supplies the counts and the retrieval provenance
        and, through ``attrs["plan"]``, the plan; or a bare
        [`SearchPlan`][scopusflow.plan.SearchPlan] for a search not yet run.
        The attributes are read in the JSON-safe forms
        [`fetch_plan`][scopusflow.fetch.fetch_plan] writes (the plan as a dict,
        the time as ISO 8601 text, the cell totals as a list of dicts) and in
        the richer forms a guide may set by hand (a ``SearchPlan``, a
        ``datetime``, a DataFrame).
    plan:
        The plan describing ``x``. Supply it when a records frame does not carry
        one, for instance one read back from CSV or a merged set. An explicit
        plan takes precedence over the one ``x`` carries. A one-cell plan
        supplied for a frame without per-cell counts takes the frame's own row
        count and ``total_results``.
    file:
        Path at which to write the record as Markdown. A file is written only
        when this is supplied, and only to the exact path given, so nothing is
        written to the working directory unless asked.

    Returns
    -------
    SearchReport
        The record. ``print`` shows it, ``report.format(style="paragraph")``
        gives the methods paragraph.

    References
    ----------
    Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D.,
    Page, M. J., & Koffel, J. B. (2021). PRISMA-S: an extension to the PRISMA
    Statement for Reporting Literature Searches in Systematic Reviews.
    *Systematic Reviews*, *10*, 39. https://doi.org/10.1186/s13643-020-01542-z

    Examples
    --------
    A search described but not yet run. The record says so throughout rather
    than implying figures it cannot have.

    >>> import scopusflow as sf
    >>> plan = sf.SearchPlan('"graphene supercapacitor"', years=range(2015, 2025),
    ...                      field="TITLE-ABS-KEY", partition="year")
    >>> report = sf.scopus_search_report(plan)
    >>> report.n_records is None
    True

    The same search after a harvest. The bundled corpus stands in for one, since
    Scopus records may not be redistributed, so the attributes a live retrieval
    records are set here by hand. Their values are illustrative, and the records
    came from OpenAlex, so the record below describes no search that was run.

    >>> from datetime import datetime, timezone
    >>> records = sf.example_records()
    >>> records.attrs["plan"] = plan
    >>> records.attrs["retrieved_at"] = datetime(2026, 7, 22, 9, 15,
    ...                                          tzinfo=timezone.utc)
    >>> records.attrs["scopusflow_version"] = "0.3.0"
    >>> report = sf.scopus_search_report(records)
    >>> report.n_records
    138
    >>> "22 July 2026" in report.format(style="paragraph")
    True
    """
    if plan is not None and not isinstance(plan, SearchPlan):
        raise ValueError("The plan must be a search plan.")
    if isinstance(x, SearchPlan):
        records = None
        plan = plan or x
    elif isinstance(x, pd.DataFrame):
        records = x
        if plan is None:
            plan = as_plan(x.attrs.get("plan"))
    else:
        raise ValueError("A search report needs a record set or a search plan.")

    report = _build(records, plan)

    if file is not None:
        if not isinstance(file, (str, bytes)) and not hasattr(file, "__fspath__"):
            raise ValueError("The file must be a single non-empty path.")
        if isinstance(file, (str, bytes)) and not str(file).strip():
            raise ValueError("The file must be a single non-empty path.")
        # Newline fixed to LF, so the record is byte-identical on every platform
        # and against the R twin, which writes its text artefacts the same way.
        with open(file, "w", encoding="utf-8", newline="\n") as fh:
            fh.write(_render_markdown(report) + "\n")
    return report

SearchReport dataclass

The fields a search record is built from, and its three renderings.

database and platform are always "Scopus" and "Elsevier Scopus Search API". Every other attribute but the frames cells and prisma holds what the objects recorded, or None where they recorded nothing, which the renderings spell out as unrecorded. In that case, stopped_early is False, cells_reported 0 and not_run empty.

cells is a frame of cell, limit, n_records and reported_total; prisma a frame of item, name, source ("record" or "author") and note. searched_at and searched_until are the earliest and latest retrieval times. max_results is the cap on records per cell a set records. stopped_early, cells_planned and cells_run describe a harvest stopped before its last cell, not_run lists the cells it never reached and years_run the years of those it ran. dedupe_rule and doi_conflicts are the rule scopus_combine matched records by and the DOIs it left shared across identifiers.

query, field, view, page_size and partition are the plan's, and n_cells is the number of its cells. expression lists the distinct queries the cells send, before their year limits, or, without a plan, the distinct values of the records' query column. years names the years the cells' limits cover, as text such as "2015 to 2024". paging is the harvest's paging mode, "cursor" or "offset", and version the scopusflow version that retrieved it, or the comma-separated versions when there were several. n_records counts the records and n_with_doi those with a DOI. reported_total is the sum of the totals the API reported, given only when every cell reported one, and cells_reported is how many cells did. records_combined, duplicates_removed and deduplicated are the number of records scopus_combine received, the number it removed and whether it de-duplicated. snippet is Python code that rebuilds the plan and runs it through fetch_plan, or None without a plan.

Source code in src/scopusflow/report.py
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
@dataclass(eq=False)
class SearchReport:
    """The fields a search record is built from, and its three renderings.

    ``database`` and ``platform`` are always ``"Scopus"`` and
    ``"Elsevier Scopus Search API"``. Every other attribute but the frames
    ``cells`` and ``prisma`` holds what the objects recorded, or ``None`` where
    they recorded nothing, which the renderings spell out as unrecorded. In
    that case, ``stopped_early`` is ``False``, ``cells_reported`` ``0`` and
    ``not_run`` empty.

    ``cells`` is a frame of ``cell``, ``limit``, ``n_records`` and
    ``reported_total``; ``prisma`` a frame of ``item``, ``name``, ``source``
    (``"record"`` or ``"author"``) and ``note``. ``searched_at`` and
    ``searched_until`` are the earliest and latest retrieval times.
    ``max_results`` is the cap on records per cell a set records.
    ``stopped_early``, ``cells_planned`` and ``cells_run`` describe a harvest
    stopped before its last cell, ``not_run`` lists the cells it never reached
    and ``years_run`` the years of those it ran. ``dedupe_rule`` and
    ``doi_conflicts`` are the rule
    [`scopus_combine`][scopusflow.combine.scopus_combine] matched records by
    and the DOIs it left shared across identifiers.

    ``query``, ``field``, ``view``, ``page_size`` and ``partition`` are the
    plan's, and ``n_cells`` is the number of its cells. ``expression`` lists
    the distinct queries the cells send, before their year limits, or, without
    a plan, the distinct values of the records' ``query`` column. ``years``
    names the years the cells' limits cover, as text such as
    ``"2015 to 2024"``. ``paging`` is the harvest's paging mode, ``"cursor"``
    or ``"offset"``, and ``version`` the scopusflow version that retrieved it,
    or the comma-separated versions when there were several. ``n_records``
    counts the records and ``n_with_doi`` those with a DOI. ``reported_total``
    is the sum of the totals the API reported, given only when every cell
    reported one, and ``cells_reported`` is how many cells did.
    ``records_combined``, ``duplicates_removed`` and ``deduplicated`` are the
    number of records [`scopus_combine`][scopusflow.combine.scopus_combine]
    received, the number it removed and whether it de-duplicated. ``snippet``
    is Python code that rebuilds the plan and runs it through
    [`fetch_plan`][scopusflow.fetch.fetch_plan], or ``None`` without a plan.
    """

    database: str = "Scopus"
    platform: str = "Elsevier Scopus Search API"
    query: str | None = None
    field: str | None = None
    expression: list[str] | None = None
    view: str | None = None
    page_size: int | None = None
    paging: str | None = None
    max_results: int | None = None
    partition: str | None = None
    n_cells: int | None = None
    years: str | None = None
    years_run: str | None = None
    cells: pd.DataFrame = dataclass_field(default_factory=pd.DataFrame)
    searched_at: datetime | None = None
    searched_until: datetime | None = None
    version: str | None = None
    n_records: int | None = None
    n_with_doi: int | None = None
    reported_total: int | None = None
    cells_reported: int = 0
    stopped_early: bool = False
    cells_planned: int | None = None
    cells_run: int | None = None
    not_run: list[int] = dataclass_field(default_factory=list)
    records_combined: int | None = None
    duplicates_removed: int | None = None
    deduplicated: bool | None = None
    dedupe_rule: str | None = None
    doi_conflicts: int | None = None
    snippet: str | None = None
    prisma: pd.DataFrame = dataclass_field(default_factory=pd.DataFrame)

    def format(self, style: str = "report") -> str:
        """Render the record: ``"report"`` for the readable record ``print``
        shows, ``"paragraph"`` for the methods paragraph, or ``"markdown"`` for
        the whole record as Markdown, which is what ``file`` writes."""
        if style == "report":
            return _render_report(self)
        if style == "paragraph":
            return _render_paragraph(self)
        if style == "markdown":
            return _render_markdown(self)
        raise ValueError("style must be 'report', 'paragraph' or 'markdown'.")

    def __str__(self) -> str:
        return _render_report(self)

    __repr__ = __str__

format

format(style='report')

Render the record: "report" for the readable record print shows, "paragraph" for the methods paragraph, or "markdown" for the whole record as Markdown, which is what file writes.

Source code in src/scopusflow/report.py
167
168
169
170
171
172
173
174
175
176
177
def format(self, style: str = "report") -> str:
    """Render the record: ``"report"`` for the readable record ``print``
    shows, ``"paragraph"`` for the methods paragraph, or ``"markdown"`` for
    the whole record as Markdown, which is what ``file`` writes."""
    if style == "report":
        return _render_report(self)
    if style == "paragraph":
        return _render_paragraph(self)
    if style == "markdown":
        return _render_markdown(self)
    raise ValueError("style must be 'report', 'paragraph' or 'markdown'.")

PRISMA_S_ITEMS module-attribute

PRISMA_S_ITEMS = ['Database name', 'Multi-database searching', 'Study registries', 'Online resources and browsing', 'Citation searching', 'Contacts', 'Other methods', 'Full search strategies', 'Limits and restrictions', 'Search filters', 'Prior work', 'Updates', 'Dates of searches', 'Peer review', 'Total records', 'Deduplication']

DOIs and change tracking

Extract clean DOIs and compare two retrievals to see exactly what changed.

extract_dois

extract_dois(records, dedupe=True)

Pull cleaned, optionally de-duplicated DOIs from records or a list.

records is a record frame, whose doi column is read, or any iterable of DOIs. Each DOI is stripped of surrounding whitespace, of a resolver prefix such as https://doi.org/ or http://dx.doi.org/ and of a leading doi: label, and a missing or empty one is dropped. With dedupe (the default), a DOI that repeats an earlier one, ignoring case, is dropped, so the first spelling is kept. Returns a list of strings, in the order and the case the DOIs came in.

Source code in src/scopusflow/diff.py
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def extract_dois(records, dedupe: bool = True) -> list[str]:
    """Pull cleaned, optionally de-duplicated DOIs from records or a list.

    ``records`` is a record frame, whose ``doi`` column is read, or any
    iterable of DOIs. Each DOI is stripped of surrounding whitespace, of a
    resolver prefix such as ``https://doi.org/`` or ``http://dx.doi.org/`` and
    of a leading ``doi:`` label, and a missing or empty one is dropped. With
    ``dedupe`` (the default), a DOI that repeats an earlier one, ignoring case,
    is dropped, so the first spelling is kept. Returns a list of strings, in
    the order and the case the DOIs came in.
    """
    dois = _clean(_as_dois(records))
    if not dedupe:
        return dois
    seen: set[str] = set()
    result: list[str] = []
    for d in dois:
        key = d.lower()
        if key not in seen:
            seen.add(key)
            result.append(d)
    return result

diff_dois

diff_dois(old, new)

Compare the DOIs of two retrievals.

A record without a DOI cannot be compared this way, and Scopus gives no DOI to a share of what it indexes, more of it in the arts and humanities and among books and proceedings (Gorraiz et al., 2016). Such records are left out of the rows and counted in attrs. diff_records compares the records themselves, matching on the Scopus identifier.

Parameters:

Name Type Description Default
old

Record frames or lists of DOIs, the earlier (old) and the later (new) retrieval.

required
new

Record frames or lists of DOIs, the earlier (old) and the later (new) retrieval.

required

Returns:

Type Description
DataFrame

A doi column and a status column, where the status is added (in new only), removed (in old only) or unchanged (in both), compared without resolver prefixes and ignoring case. Rows are sorted by status then DOI, comparing code points, which the R twin's scopus_diff_dois() matches. The ints attrs["n_old_without_doi"] and attrs["n_new_without_doi"] count the records (or list entries) on each side that had no DOI and so were not compared.

References

Gorraiz, J., Melero-Fuentes, D., Gumpenberger, C., & Valderrama-Zurián, J.-C. (2016). Availability of digital object identifiers (DOIs) in Web of Science and Scopus. Journal of Informetrics, 10(1), 98–109. https://doi.org/10.1016/j.joi.2015.11.008

Source code in src/scopusflow/diff.py
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
def diff_dois(old, new) -> pd.DataFrame:
    """Compare the DOIs of two retrievals.

    A record without a DOI cannot be compared this way, and Scopus gives no
    DOI to a share of what it indexes, more of it in the arts and humanities
    and among books and proceedings (Gorraiz et al., 2016). Such records are
    left out of the rows and counted in ``attrs``.
    [`diff_records`][scopusflow.diff.diff_records] compares the records
    themselves, matching on the Scopus identifier.

    Parameters
    ----------
    old, new:
        Record frames or lists of DOIs, the earlier (``old``) and the later
        (``new``) retrieval.

    Returns
    -------
    pandas.DataFrame
        A ``doi`` column and a ``status`` column, where the status is
        ``added`` (in ``new`` only), ``removed`` (in ``old`` only) or
        ``unchanged`` (in both), compared without resolver prefixes and
        ignoring case. Rows are sorted by status then DOI, comparing code
        points, which the R twin's ``scopus_diff_dois()`` matches. The ints
        ``attrs["n_old_without_doi"]`` and ``attrs["n_new_without_doi"]``
        count the records (or list entries) on each side that had no DOI and
        so were not compared.

    References
    ----------
    Gorraiz, J., Melero-Fuentes, D., Gumpenberger, C., & Valderrama-Zurián,
    J.-C. (2016). Availability of digital object identifiers (DOIs) in Web of
    Science and Scopus. *Journal of Informetrics*, *10*(1), 98–109.
    https://doi.org/10.1016/j.joi.2015.11.008
    """
    old_raw, new_raw = _as_dois(old), _as_dois(new)
    old_d, new_d = extract_dois(old_raw), extract_dois(new_raw)
    old_keys = {d.lower() for d in old_d}
    new_keys = {d.lower() for d in new_d}
    rows = (
        [(d, "added") for d in new_d if d.lower() not in old_keys]
        + [(d, "removed") for d in old_d if d.lower() not in new_keys]
        + [(d, "unchanged") for d in new_d if d.lower() in old_keys]
    )
    # Sorted here, comparing code points, so that the order cannot depend on
    # how a pandas version or its Arrow backend sorts text.
    rows.sort(key=lambda row: (_DOI_STATUSES.index(row[1]), row[0]))
    df = pd.DataFrame(rows, columns=["doi", "status"])
    df.attrs["n_old_without_doi"] = sum(_clean_one(d) is None for d in old_raw)
    df.attrs["n_new_without_doi"] = sum(_clean_one(d) is None for d in new_raw)
    return df

diff_records

diff_records(old, new)

Compare two retrievals record by record.

Identifies which records were added, removed or unchanged between an earlier and a later retrieval, and gives each record's citation count in both. diff_dois compares DOIs alone, and this function compares the records themselves, matched as scopus_combine(dedupe=True) matches them. A record without a DOI is therefore still tracked by its Scopus identifier, and two records under one DOI are not taken for one.

Two records are one record when they share a Scopus identifier, or when they share a DOI and one of them lacks an identifier. DOIs are compared as extract_dois compares them, without resolver prefixes and ignoring case. Records with different identifiers are never matched, even under one DOI, since Scopus gives one DOI to distinct papers as well as holding duplicate records of one paper (scopus_combine gives the sources).

A record found in one retrieval only is a conflict where the other retrieval gives its DOI to a different record. It may have been re-indexed under a new identifier, or it may be a distinct paper sharing the DOI. Only the two records themselves can tell, so it is counted neither as added nor as removed. A record without an identifier whose DOI two identified records carry is in the same position. A record with neither an identifier nor a DOI cannot be matched at all, and is left out of the rows and counted.

scopus_id is the identifier the record was matched on, missing for a record matched on its DOI alone. doi is the DOI without its resolver prefix, in the case the record gives it. doi, title and year come from the later retrieval where it holds the record and gives a value, and from the earlier one otherwise, while citations_old and citations_new come from each retrieval in turn. Where one retrieval holds a record more than once, the first value it gives is used.

Parameters:

Name Type Description Default
old

Record frames with a scopus_id or a doi column, the earlier (old) and the later (new) retrieval. The title, year and citations columns are read where present.

required
new

Record frames with a scopus_id or a doi column, the earlier (old) and the later (new) retrieval. The title, year and citations columns are read where present.

required

Returns:

Type Description
DataFrame

One row per record, with the columns status, scopus_id, doi, title, year, citations_old and citations_new. The status is added (in new only), removed (in old only), conflict or unchanged (in both). Rows are sorted in that order of status, then by DOI, identifier and title, comparing code points, which the R twin's scopus_diff_records() matches. The ints attrs["n_old_without_id_or_doi"] and attrs["n_new_without_id_or_doi"] count the records on each side that carried neither key and so were not compared.

Raises:

Type Description
ValueError

When old or new is not a frame, or has neither a scopus_id nor a doi column.

Examples:

>>> import pandas as pd
>>> import scopusflow as sf
>>> old = pd.DataFrame({"scopus_id": ["85000000001", "85000000002"],
...                     "doi": ["10.5555/a", "10.5555/c"]})
>>> new = pd.DataFrame({"scopus_id": ["85000000001", "85000000033"],
...                     "doi": ["10.5555/a", "10.5555/c"]})
>>> list(sf.diff_records(old, new)["status"])
['conflict', 'conflict', 'unchanged']
Source code in src/scopusflow/diff.py
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
def diff_records(old, new) -> pd.DataFrame:
    """Compare two retrievals record by record.

    Identifies which records were added, removed or unchanged between an
    earlier and a later retrieval, and gives each record's citation count in
    both. [`diff_dois`][scopusflow.diff.diff_dois] compares DOIs alone, and
    this function compares the records themselves, matched as
    ``scopus_combine(dedupe=True)`` matches them. A record without a DOI is
    therefore still tracked by its Scopus identifier, and two records under one
    DOI are not taken for one.

    Two records are one record when they share a Scopus identifier, or when
    they share a DOI and one of them lacks an identifier. DOIs are compared as
    [`extract_dois`][scopusflow.diff.extract_dois] compares them, without
    resolver prefixes and ignoring case. Records with different identifiers are
    never matched, even under one DOI, since Scopus gives one DOI to distinct
    papers as well as holding duplicate records of one paper
    ([`scopus_combine`][scopusflow.combine.scopus_combine] gives the sources).

    A record found in one retrieval only is a ``conflict`` where the other
    retrieval gives its DOI to a different record. It may have been re-indexed
    under a new identifier, or it may be a distinct paper sharing the DOI.
    Only the two records themselves can tell, so it is counted neither as
    added nor as removed. A record without an identifier whose DOI two
    identified records carry is in the same position. A record with neither an
    identifier nor a DOI cannot be matched at all, and is left out of the rows
    and counted.

    ``scopus_id`` is the identifier the record was matched on, missing for a
    record matched on its DOI alone. ``doi`` is the DOI without its resolver
    prefix, in the case the record gives it. ``doi``, ``title`` and ``year``
    come from the later retrieval where it holds the record and gives a value,
    and from the earlier one otherwise, while ``citations_old`` and
    ``citations_new`` come from each retrieval in turn. Where one retrieval
    holds a record more than once, the first value it gives is used.

    Parameters
    ----------
    old, new:
        Record frames with a ``scopus_id`` or a ``doi`` column, the earlier
        (``old``) and the later (``new``) retrieval. The ``title``, ``year``
        and ``citations`` columns are read where present.

    Returns
    -------
    pandas.DataFrame
        One row per record, with the columns ``status``, ``scopus_id``,
        ``doi``, ``title``, ``year``, ``citations_old`` and
        ``citations_new``. The status is ``added`` (in ``new`` only),
        ``removed`` (in ``old`` only), ``conflict`` or ``unchanged`` (in
        both). Rows are sorted in that order of status, then by DOI,
        identifier and title, comparing code points, which the R twin's
        ``scopus_diff_records()`` matches. The ints
        ``attrs["n_old_without_id_or_doi"]`` and
        ``attrs["n_new_without_id_or_doi"]`` count the records on each side
        that carried neither key and so were not compared.

    Raises
    ------
    ValueError
        When ``old`` or ``new`` is not a frame, or has neither a ``scopus_id``
        nor a ``doi`` column.

    Examples
    --------
    >>> import pandas as pd
    >>> import scopusflow as sf
    >>> old = pd.DataFrame({"scopus_id": ["85000000001", "85000000002"],
    ...                     "doi": ["10.5555/a", "10.5555/c"]})
    >>> new = pd.DataFrame({"scopus_id": ["85000000001", "85000000033"],
    ...                     "doi": ["10.5555/a", "10.5555/c"]})
    >>> list(sf.diff_records(old, new)["status"])
    ['conflict', 'conflict', 'unchanged']
    """
    # Imported here because combine imports this module for _clean_one.
    from .combine import _record_groups

    old_rows, new_rows = _diff_frame(old, "old"), _diff_frame(new, "new")
    rows = old_rows + new_rows
    later = [False] * len(old_rows) + [True] * len(new_rows)

    # The default de-duplication rule, applied across the two retrievals, so
    # the diff and a merge of the same two sets agree on which records are one.
    keys, _ = _record_groups(
        pd.DataFrame({"scopus_id": [r["scopus_id"] for r in rows],
                      "doi": [r["doi"] for r in rows]}, dtype=object),
        "id_or_shared_doi",
    )
    dois = [_clean_one(r["doi"]) for r in rows]
    doi_keys = [None if d is None else d.lower() for d in dois]
    keyed = [not key.startswith("row:") for key in keys]

    groups: dict[str, dict] = {}
    for i, key in enumerate(keys):
        if not keyed[i]:
            continue
        group = groups.setdefault(key, {"old": [], "new": []})
        group["new" if later[i] else "old"].append(i)

    old_dois = {k for k, side in zip(doi_keys, later, strict=True) if k and not side}
    new_dois = {k for k, side in zip(doi_keys, later, strict=True) if k and side}
    titles = [r["title"] for r in rows]
    years = [r["year"] for r in rows]
    citations = [r["citations"] for r in rows]

    def first(values, indices):
        """The first value the rows at ``indices`` give, or ``None``."""
        return next((values[i] for i in indices if _given(values[i])), None)

    def latest(values, group):
        """The later retrieval's value where it gives one, the earlier one's
        if not."""
        given = first(values, group["new"])
        return given if given is not None else first(values, group["old"])

    out = []
    for key, group in groups.items():
        if group["old"] and group["new"]:
            status = "unchanged"
        else:
            # A record in one retrieval only whose DOI the other retrieval
            # gives to a record of its own. That record is a different one, or
            # the two would share a key and the first would be in both.
            other = new_dois if group["old"] else old_dois
            indices = group["old"] or group["new"]
            clash = any(doi_keys[i] in other for i in indices if doi_keys[i])
            status = "conflict" if clash else ("removed" if group["old"] else "added")
        out.append({
            "status": status,
            # The identifier the record was matched on, which its key carries.
            "scopus_id": key[3:] if key.startswith("id:") else None,
            "doi": latest(dois, group),
            "title": latest(titles, group),
            "year": latest(years, group),
            "citations_old": first(citations, group["old"]),
            "citations_new": first(citations, group["new"]),
        })

    out.sort(key=lambda r: (_RECORD_STATUSES.index(r["status"]), _byte_order(r["doi"]),
                            _byte_order(r["scopus_id"]), _byte_order(r["title"])))
    frame = pd.DataFrame({
        "status": [r["status"] for r in out],
        "scopus_id": pd.array([r["scopus_id"] for r in out], dtype="string"),
        "doi": pd.array([r["doi"] for r in out], dtype="string"),
        "title": pd.array([r["title"] for r in out], dtype="string"),
        "year": pd.array([r["year"] for r in out], dtype="Int64"),
        "citations_old": pd.array([r["citations_old"] for r in out], dtype="Int64"),
        "citations_new": pd.array([r["citations_new"] for r in out], dtype="Int64"),
    }, columns=_RECORD_DIFF_COLUMNS)
    frame.attrs["n_old_without_id_or_doi"] = sum(
        not k and not side for k, side in zip(keyed, later, strict=True))
    frame.attrs["n_new_without_id_or_doi"] = sum(
        not k and side for k, side in zip(keyed, later, strict=True))
    return frame

Analyse and visualise

Summarise a literature over time and by source or author, compare topics within it, and turn the summaries into figures.

The plotting functions below are pure over the frame they are given, so the record examples in this section run at build time over the bundled example harvest, and the topic comparison over a frame of the documented shape. Nothing here contacts the Scopus API or needs a key. The functions that do retrieve counts are marked as such and are shown but not run. The guides linked from each one demonstrate them against a live key.

scopus_trend

scopus_trend(query, years, field=None, view='STANDARD', **kwargs)

Count Scopus hits for query in each of years without downloading them.

Each year is a cheap result-size lookup, so this gives a publication trend far faster than harvesting every record. field wraps the query in a Scopus field tag (see FIELD_TAGS), the way scopus_count and the R twin's scopus_trend() do; left None, the query is sent as given.

Returns a TREND_COLUMNS frame, one row per year, sorted by year. view and any further keyword argument go to pybliometrics' ScopusSearch.

Each count is sent with refresh=True unless you pass refresh. pybliometrics keys its own response cache on the query string and view alone. Without refresh=True, it could answer a year with the number of rows an earlier harvest of the same query left there. refresh=False, or a number of days, opts in to that cache, and a year it answers is warned about. The key ignores keyword filters such as subj, so fold any filter into the query before you opt in.

The result records when the counts were taken, in attrs["retrieved_at"] as ISO 8601 text in UTC, as a harvest records its own. Scopus goes on indexing a year's publications after the year has begun, so a count for the year of retrieval is a count so far, and plot_trend says so from this time. A count answered from pybliometrics' cache is dated by its cache file, and the earliest time is kept. When a cached count's file time cannot be read, the attribute is left out, since a trend dated by its other counts would be dated later than that one.

Raises ScopusFlowConfigError before any request when pybliometrics.init() has not been called in the session.

Source code in src/scopusflow/trend.py
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
def scopus_trend(
    query: str,
    years: Sequence[int],
    field: str | None = None,
    view: str = "STANDARD",
    **kwargs,
) -> pd.DataFrame:
    """Count Scopus hits for ``query`` in each of ``years`` without downloading them.

    Each year is a cheap result-size lookup, so this gives a publication trend
    far faster than harvesting every record. ``field`` wraps the query in a
    Scopus field tag (see [`FIELD_TAGS`][scopusflow.query.FIELD_TAGS]), the way
    [`scopus_count`][scopusflow.count.scopus_count] and the R twin's
    ``scopus_trend()`` do; left ``None``, the query is sent as given.

    Returns a [`TREND_COLUMNS`][scopusflow.trend.TREND_COLUMNS] frame, one row
    per year, sorted by year. ``view`` and any further keyword argument go to
    pybliometrics' ``ScopusSearch``.

    Each count is sent with ``refresh=True`` unless you pass ``refresh``.
    pybliometrics keys its own response cache on the query string and view
    alone. Without ``refresh=True``, it could answer a year with the number of
    rows an earlier harvest of the same query left there. ``refresh=False``,
    or a number of days, opts in to that cache, and a year it answers is
    warned about. The key ignores keyword filters such as ``subj``, so fold any
    filter into the query before you opt in.

    The result records when the counts were taken, in ``attrs["retrieved_at"]``
    as ISO 8601 text in UTC, as a harvest records its own. Scopus goes on
    indexing a year's publications after the year has begun, so a count for
    the year of retrieval is a count so far, and
    [`plot_trend`][scopusflow.plots.plot_trend] says so from this time. A count
    answered from pybliometrics' cache is dated by its cache file, and the
    earliest time is kept. When a cached count's file time cannot be read, the
    attribute is left out, since a trend dated by its other counts would be
    dated later than that one.

    Raises [`ScopusFlowConfigError`][scopusflow.exceptions.ScopusFlowConfigError]
    before any request when ``pybliometrics.init()`` has not been called in the
    session.
    """
    if not query or not query.strip():
        raise ValueError("query must be a non-empty string.")
    # Checked offline first (see scopusflow.plan.SearchPlan), so a query the
    # API cannot run as written costs no request.
    _check_query(query, field)
    years = list(years)
    if not years:
        raise ValueError("years must be a non-empty sequence.")
    # Validated before it reaches the query, so a whole-number float such as
    # 2015.0 is sent as PUBYEAR IS 2015, the year its count is filed under.
    years = _check_years(years)
    # Wrapped once, before the loop: the tag applies to the query alone, never
    # to the year
    # filter folded in beside it.
    query = wrap_field(query, field)
    kwargs.setdefault("refresh", True)
    require_init()

    from pybliometrics.scopus import ScopusSearch  # imported lazily; needs a key

    counts: dict[int, int] = {}
    cached: list = []
    for y in years:
        # Bracketed first when a top-level OR or AND NOT would otherwise take
        # the year filter from part of the query.
        year_query = _and_clause(query, f"PUBYEAR IS {y}")
        t0 = time.time()
        search = ScopusSearch(year_query, view=view, download=False, **kwargs)
        counts[y], written = result_size_and_time(
            search, t0, kwargs["refresh"], year_query, stacklevel=2
        )
        if written is not None:
            cached.append(written)
    out = _trend_frame(counts)
    # Dated as fetch_plan() dates its cells: a count the API gave by the time
    # the counting finished, a count from pybliometrics' cache by its file,
    # and the earliest kept, so the trend never claims fresher counts than
    # it holds.
    if UNDATED not in cached:
        out.attrs["retrieved_at"] = iso_utc(min([datetime.now(timezone.utc), *cached]))
    return out

This asks the API for a count per year, so it cannot run at build time. See Analysing a literature for the worked example, and year_counts below for the offline equivalent over records you already hold.

year_counts

year_counts(records, fill=True)

Count records per publication year, dropping rows with a missing year.

Returns a TREND_COLUMNS frame sorted ascending by year, with both columns as plain integers.

With fill (the default), every year the records' searches covered is counted, a year without records as zero. plot_trend therefore draws the line down to zero there. The years searched are read from the plan in records.attrs["plan"], which fetch_plan attaches: the year of each cell under partition="year", and every year of the range under partition="none". A year no cell searched is not added, so a plan over 2015, 2016, 2019 and 2020 gives no row for 2017 or 2018. A harvest stopped before its last cell covers only the cells it ran. Without a plan, or with one that sent no year limit, the years run from the first year in the records to the last. A year that holds records keeps its row in every case, and a set in which no record has a year gives an empty frame. The R twin's autoplot() counts a record set's years by the same rule.

fill=False counts only the years that hold records.

The records' attrs["retrieved_at"] is carried to the result, so plot_trend can say when the last year plotted was still being indexed.

Source code in src/scopusflow/trend.py
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
def year_counts(records: pd.DataFrame, fill: bool = True) -> pd.DataFrame:
    """Count records per publication year, dropping rows with a missing year.

    Returns a [`TREND_COLUMNS`][scopusflow.trend.TREND_COLUMNS] frame sorted
    ascending by year, with both columns as plain integers.

    With ``fill`` (the default), every year the records' searches covered is
    counted, a year without records as zero.
    [`plot_trend`][scopusflow.plots.plot_trend] therefore draws the line down
    to zero there. The years searched are read from the plan in
    ``records.attrs["plan"]``, which
    [`fetch_plan`][scopusflow.fetch.fetch_plan] attaches: the year of each cell
    under ``partition="year"``, and every year of the range under
    ``partition="none"``. A year no cell searched is not added, so a plan over
    2015, 2016, 2019 and 2020 gives no row for 2017 or 2018. A harvest stopped
    before its last cell covers only the cells it ran. Without a plan, or with
    one that sent no year limit, the years run from the first year in the
    records to the last. A year that holds records keeps its row in every case,
    and a set in which no record has a year gives an empty frame. The R twin's
    ``autoplot()`` counts a record set's years by the same rule.

    ``fill=False`` counts only the years that hold records.

    The records' ``attrs["retrieved_at"]`` is carried to the result, so
    [`plot_trend`][scopusflow.plots.plot_trend] can say when the last year
    plotted was still being indexed.
    """
    years = pd.to_numeric(records["year"], errors="coerce").dropna()
    counts = {int(y): int(n) for y, n in years.astype(int).value_counts().items()}
    if fill and counts:
        span = _searched_years(records) or range(min(counts), max(counts) + 1)
        for year in span:
            counts.setdefault(int(year), 0)
    out = _trend_frame(counts)
    stamp = records.attrs.get("retrieved_at")
    if stamp is not None:
        out.attrs["retrieved_at"] = stamp
    return out
records = sf.example_records()
out(sf.year_counts(records))
year n
2015 15
2016 9
2017 10
2018 15
2019 19
2020 13
2021 13
2022 15
2023 15
2024 14

TREND_COLUMNS module-attribute

TREND_COLUMNS = ['year', 'n']

top

top(records, by='source', n=10)

Tally the most frequent sources or authors in a record set.

Author strings holding several names are split on ";", so each contributor is counted once per record. Only a view="COMPLETE" harvest holds several names, since STANDARD records carry the first author alone (see to_records). A tally by author of a harvest whose attrs["view"] is "STANDARD" therefore counts first authorships, and says so with a UserWarning, once per process, in the R twin's words.

Each value is stripped of the white space R's trimws() removes, which is spaces, tabs, carriage returns and line feeds. A value left empty is dropped, as is a missing one, so a blank source title is not counted as a journal.

The result has the columns value and n. It is sorted by descending count, with ties broken by value in code-point order. For UTF-8 text that is the byte order the R twin's scopus_top() sorts by, in which capitals come before lower case. The first n rows are returned, or every row when there are fewer distinct values. Where several values tie at the cut-off, the ones that sort first are kept, the same ones on every machine and in both twins, whatever the order of the records.

n must be one positive whole number. A whole float such as 5.0 and a numpy integer are taken, as the R twin takes any finite whole number, and an n beyond the number of distinct values returns them all. A bool, NaN, infinity, a fraction or a value below 1 raises ValueError, as does a by other than "source" or "author".

Source code in src/scopusflow/records.py
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
def top(records: pd.DataFrame, by: str = "source", n: int = 10) -> pd.DataFrame:
    """Tally the most frequent sources or authors in a record set.

    Author strings holding several names are split on ``";"``, so each
    contributor is counted once per record. Only a ``view="COMPLETE"`` harvest
    holds several names, since ``STANDARD`` records carry the first author alone
    (see [`to_records`][scopusflow.records.to_records]). A tally by author of a
    harvest whose ``attrs["view"]`` is ``"STANDARD"`` therefore counts first
    authorships, and says so with a ``UserWarning``, once per process, in the R
    twin's words.

    Each value is stripped of the white space R's ``trimws()`` removes, which is
    spaces, tabs, carriage returns and line feeds. A value left empty is dropped,
    as is a missing one, so a blank source title is not counted as a journal.

    The result has the columns ``value`` and ``n``. It is sorted by descending
    count, with ties broken by ``value`` in code-point order. For UTF-8 text that
    is the byte order the R twin's ``scopus_top()`` sorts by, in which capitals
    come before lower case. The first ``n`` rows are returned, or every row when
    there are fewer distinct values. Where several values tie at the cut-off,
    the ones that sort first are kept, the same ones on every machine and in
    both twins, whatever the order of the records.

    ``n`` must be one positive whole number. A whole float such as ``5.0`` and a
    numpy integer are taken, as the R twin takes any finite whole number, and an
    ``n`` beyond the number of distinct values returns them all. A bool, NaN,
    infinity, a fraction or a value below 1 raises ``ValueError``, as does a
    ``by`` other than ``"source"`` or ``"author"``.
    """
    if by not in ("source", "author"):
        raise ValueError("by must be 'source' or 'author'.")
    n = _top_n(n)
    if by == "source":
        values = [str(value) for value in records["publication"].dropna()]
    else:
        _warn_first_author_only(records)
        values = [name for joined in records["authors"].dropna()
                  for name in str(joined).split(";")]
    # Python's own str methods throughout, since pandas 3 hands a text column's
    # .str methods to Arrow, and an object column sorts by Python's comparison,
    # which is code-point order under every pandas version and storage.
    values = [value.strip(_R_WHITESPACE) for value in values]
    tally = (
        pd.Series([value for value in values if value], dtype=object)
        .value_counts()
        .rename_axis("value")
        .reset_index(name="n")
    )
    # value_counts() orders the counts with an unstable sort, so its order among
    # tied values followed the row order and numpy's sort implementation. Each
    # value appears once in the tally, so count and value together order every
    # row, and the R twin orders its rows on the same two keys.
    tally = tally.sort_values(["n", "value"], ascending=[False, True])
    return tally.head(n).reset_index(drop=True)
out(sf.top(records, by="source", n=5))
value n
ACS Applied Materials & Interfaces 8
Journal of Power Sources 5
Synthetic Metals 5
Electrochimica Acta 4
Journal of Materials Chemistry A 4

compare_topics

compare_topics(reference_query, comparison_terms, years, field=None, view='STANDARD', **kwargs)

Compare comparison topics against a reference topic over the years.

Each comparison query is (reference) AND (term), after field has wrapped each side. A reference containing OR or AND NOT therefore keeps its meaning, and every comparison set is a subset of the reference. The year limit is appended to each query as scopus_count appends it.

comparison_terms is one term or a list of them. view and any further keyword argument go to pybliometrics' ScopusSearch.

Returns a COMPARISON_COLUMNS frame with one row per query and year. query is the query counted, before its year limit: the reference, or (reference) AND (term), with each side wrapped in field when one is given. query_type is "reference" or "comparison", and abridged_query is the reference or the term as given, trimmed, which labels the topic in plot_comparison. n is that query's count for the year and reference_n the reference's. comparison_percentage is 100 * n / reference_n, so the reference rows read 100, and it is missing where reference_n is 0. average_comparison_percentage is the same ratio over the whole period: the query's counts summed over the years divided by the reference's summed over the same years, missing where the reference has no records in any year. The reference rows come first, then the topics by descending average share, with ties broken by abridged_query, and each topic's rows run in year order.

One count request per term per year, plus one per year for the reference topic, so keep the term and year counts modest to stay within quota.

Every count is sent with refresh=True unless you pass refresh. pybliometrics keys its own response cache on the query string and view alone, so without refresh=True a reference harvested earlier could be counted from the rows the harvest left there. The terms would still be counted by the API, and a share of the two could pass 100%. refresh=False, or a number of days, opts in to that cache, and a count it answers is warned about. The key ignores keyword filters such as subj, so fold any filter into the queries before you opt in.

Raises ScopusFlowConfigError before any request when pybliometrics.init() has not been called in the session.

Source code in src/scopusflow/compare.py
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def compare_topics(reference_query: str, comparison_terms, years: Sequence[int],
                   field: str | None = None, view: str = "STANDARD",
                   **kwargs) -> pd.DataFrame:
    """Compare comparison topics against a reference topic over the years.

    Each comparison query is ``(reference) AND (term)``, after ``field`` has
    wrapped each side. A reference containing OR or AND NOT therefore keeps its
    meaning, and every comparison set is a subset of the reference. The year
    limit is appended to each query as
    [`scopus_count`][scopusflow.count.scopus_count] appends it.

    ``comparison_terms`` is one term or a list of them. ``view`` and any further
    keyword argument go to pybliometrics' ``ScopusSearch``.

    Returns a [`COMPARISON_COLUMNS`][scopusflow.compare.COMPARISON_COLUMNS]
    frame with one row per query and year. ``query`` is the query counted,
    before its year limit: the reference, or ``(reference) AND (term)``, with
    each side wrapped in ``field`` when one is given. ``query_type`` is
    ``"reference"`` or ``"comparison"``, and ``abridged_query`` is the
    reference or the term as given, trimmed, which labels the topic in
    [`plot_comparison`][scopusflow.plots.plot_comparison]. ``n`` is that
    query's count for the year and ``reference_n`` the reference's.
    ``comparison_percentage`` is ``100 * n / reference_n``, so the reference
    rows read 100, and it is missing where ``reference_n`` is 0.
    ``average_comparison_percentage`` is the same ratio over the whole period:
    the query's counts summed over the years divided by the reference's summed
    over the same years, missing where the reference has no records in any
    year. The reference rows come first, then the topics by descending average
    share, with ties broken by ``abridged_query``, and each topic's rows run in
    year order.

    One count request per term per year, plus one per year for the reference
    topic, so keep the term and year counts modest to stay within quota.

    Every count is sent with ``refresh=True`` unless you pass ``refresh``.
    pybliometrics keys its own response cache on the query string and view
    alone, so without ``refresh=True`` a reference harvested earlier could be
    counted from the rows the harvest left there. The terms would still be
    counted by the API, and a share of the two could pass 100%.
    ``refresh=False``, or a number of days, opts in to that cache, and a count
    it answers is warned about. The key ignores keyword filters such as
    ``subj``, so fold any filter into the queries before you opt in.

    Raises [`ScopusFlowConfigError`][scopusflow.exceptions.ScopusFlowConfigError]
    before any request when ``pybliometrics.init()`` has not been called in the
    session.
    """
    if not reference_query or not str(reference_query).strip():
        raise ValueError("reference_query must be a non-empty string.")
    if isinstance(comparison_terms, str):
        comparison_terms = [comparison_terms]
    terms = [str(t).strip() for t in comparison_terms]
    if not terms or any(not t for t in terms):
        raise ValueError("comparison_terms must be a non-empty list of non-empty terms.")
    if years is None or not list(years):
        raise ValueError("years must be a non-empty sequence.")
    ys = sorted(set(_check_years(years)))
    # Every query is checked offline before the first request, so a bad term
    # cannot fail the comparison after the reference counts have been paid for.
    _check_query(str(reference_query).strip(), field)
    for term in terms:
        _check_query(term, field)
    kwargs.setdefault("refresh", True)
    require_init()

    from pybliometrics.scopus import ScopusSearch  # imported lazily; needs a key

    def size(query: str, year: int) -> int:
        full = _and_clause(query, f"PUBYEAR IS {year}")
        t0 = time.time()
        search = ScopusSearch(full, view=view, download=False, **kwargs)
        # stacklevel 3 reaches compare_topics' caller from inside size() on
        # Python 3.12 and later. Earlier versions give each comprehension that
        # calls size() a frame of its own (PEP 709), so there the warning names
        # this module.
        return result_size(search, t0, kwargs["refresh"], full, stacklevel=3)

    ref_query = wrap_field(str(reference_query).strip(), field)
    # One count step per term, plus the reference; logged as "Cell k/N:" so the
    # app's progress parser can drive a bar (mirrors the R verbose output).
    total = len(terms) + 1
    logger.info("Cell 1/%d: counting reference across %d year(s)", total, len(ys))
    ref_counts = {y: size(ref_query, y) for y in ys}

    comparison = []
    for i, term in enumerate(terms):
        logger.info("Cell %d/%d: counting '%s'", i + 2, total, term)
        # Both sides are bracketed, as in scopus_intersections(). Joined bare, an
        # AND NOT reference took the term into its negation, and the comparison
        # set outgrew the reference.
        cmp_query = f"({ref_query}) AND ({wrap_field(term, field)})"
        comparison.append((term, cmp_query, {y: size(cmp_query, y) for y in ys}))

    return _assemble(str(reference_query).strip(), ref_query, ref_counts, comparison, ys)

This makes one count request per year for the reference topic and one per term per year, (terms + 1) × years in all, so it cannot run at build time. See Comparing topics for the worked example.

COMPARISON_COLUMNS module-attribute

COMPARISON_COLUMNS = ['query', 'query_type', 'abridged_query', 'year', 'n', 'reference_n', 'comparison_percentage', 'average_comparison_percentage']

scopus_intersections

scopus_intersections(concepts, intersections=None, abbrev=None, sep=' × ', years=None, field=None, view='STANDARD', verbose=False, **kwargs)

Count each concept and each requested intersection of concepts.

Parameters:

Name Type Description Default
concepts Mapping[str, str]

A mapping whose keys are display labels and whose values are search terms (wrapped in field when one is given) or complete field-tagged query expressions such as "TITLE(virtual reality)" (used as-is). The labels must be unique.

required
intersections Sequence[Sequence[str]] | None

An optional sequence of sequences, each naming two or more distinct concept labels whose intersection should be counted, for example [["A", "B"], ["A", "B", "C"]]. A single flat sequence of labels is taken as one intersection.

None
abbrev Mapping[str, str] | None

An optional mapping of short labels, keyed by concept label and used only when composing intersection labels, so those rows stay readable while the concept rows keep their full names.

None
sep str

The separator joining member labels in an intersection label; defaults to a multiplication sign between spaces.

' × '
years Sequence[int] | None

Optional publication years, applied to every count as one inclusive range from the earliest year to the latest, as scopus_count applies them, so [2015, 2020] covers every year from 2015 to 2020.

None
field str | None

An optional Scopus field tag wrapped around each concept value that is not already a complete field-tagged expression (see FIELD_TAGS).

None
view str

The Scopus search view, "STANDARD" or "COMPLETE".

'STANDARD'
verbose bool

When True, progress is reported on the scopusflow logger.

False

Returns:

Type Description
DataFrame

One row per concept and per intersection, with columns label, query, n (the count, as a nullable integer), type ("concept" or "intersection"), size (the number of member concepts) and members (the member labels, joined by "; "). The years restriction, when given, is stored in df.attrs["years"].

Notes

This performs one count request per concept and per intersection, so it needs a valid API key and internet access, exactly as scopus_count does.

Further keyword arguments reach each of those counts, and so does its default of refresh=True. pybliometrics' own response cache, keyed on the query string and view alone, would otherwise answer a concept that an earlier harvest downloaded with the number of rows that harvest left there. refresh=False, or a number of days, opts in to that cache, and a count it answers is warned about. The key ignores keyword filters such as subj, so fold any filter into each concept's query before you opt in.

Source code in src/scopusflow/intersections.py
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
def scopus_intersections(
    concepts: Mapping[str, str],
    intersections: Sequence[Sequence[str]] | None = None,
    abbrev: Mapping[str, str] | None = None,
    sep: str = " × ",
    years: Sequence[int] | None = None,
    field: str | None = None,
    view: str = "STANDARD",
    verbose: bool = False,
    **kwargs,
) -> pd.DataFrame:
    """Count each concept and each requested intersection of concepts.

    Parameters
    ----------
    concepts:
        A mapping whose keys are display labels and whose values are search terms
        (wrapped in ``field`` when one is given) or complete field-tagged query
        expressions such as ``"TITLE(virtual reality)"`` (used as-is). The labels
        must be unique.
    intersections:
        An optional sequence of sequences, each naming two or more distinct
        concept labels whose intersection should be counted, for example
        ``[["A", "B"], ["A", "B", "C"]]``. A single flat sequence of labels is
        taken as one intersection.
    abbrev:
        An optional mapping of short labels, keyed by concept label and used only
        when composing intersection labels, so those rows stay readable while the
        concept rows keep their full names.
    sep:
        The separator joining member labels in an intersection label; defaults to
        a multiplication sign between spaces.
    years:
        Optional publication years, applied to every count as one inclusive
        range from the earliest year to the latest, as
        [`scopus_count`][scopusflow.count.scopus_count] applies them, so
        ``[2015, 2020]`` covers every year from 2015 to 2020.
    field:
        An optional Scopus field tag wrapped around each concept value that is not
        already a complete field-tagged expression (see
        [`FIELD_TAGS`][scopusflow.query.FIELD_TAGS]).
    view:
        The Scopus search view, ``"STANDARD"`` or ``"COMPLETE"``.
    verbose:
        When ``True``, progress is reported on the ``scopusflow`` logger.

    Returns
    -------
    pandas.DataFrame
        One row per concept and per intersection, with columns ``label``,
        ``query``, ``n`` (the count, as a nullable integer), ``type``
        (``"concept"`` or ``"intersection"``), ``size`` (the number of member
        concepts) and ``members`` (the member labels, joined by ``"; "``). The
        ``years`` restriction, when given, is stored in ``df.attrs["years"]``.

    Notes
    -----
    This performs one count request per concept and per intersection, so it needs
    a valid API key and internet access, exactly as
    [`scopus_count`][scopusflow.count.scopus_count] does.

    Further keyword arguments reach each of those counts, and so does its
    default of ``refresh=True``. pybliometrics' own response cache, keyed on the
    query string and view alone, would otherwise answer a concept that an
    earlier harvest downloaded with the number of rows that harvest left there.
    ``refresh=False``, or a number of days, opts in to that cache, and a count
    it answers is warned about. The key ignores keyword filters such as
    ``subj``, so fold any filter into each concept's query before you opt in.
    """
    out = _intersection_rows(concepts, intersections, abbrev, sep, field)

    ys = sorted(set(_check_years(years))) if years else None
    if verbose:
        n_inter = int((out["type"] == "intersection").sum())
        logger.info(
            "Counting %d queries (%d intersection%s).",
            len(out), n_inter, "" if n_inter == 1 else "s",
        )

    counts: list[int] = []
    for label, query in zip(out["label"], out["query"], strict=True):
        if verbose:
            logger.info("Counting %s", label)
        counts.append(scopus_count(query, years=ys, view=view, **kwargs))
    out["n"] = pd.array(counts, dtype="Int64")

    out = out[["label", "query", "n", "type", "size", "members"]]
    out.attrs["years"] = ys
    return out

This makes one count request per concept and per intersection, so it cannot run at build time. See Analysing a literature for the worked example.

plot_trend

plot_trend(trend, ax=None)

Plot publication counts over time as a filled area, line and points.

trend has columns ["year", "n"], as scopus_trend and year_counts return them. Returns the matplotlib Axes. An empty frame raises ValueError("The trend has no years to plot."), as the R twin's plot_scopus_trend() does. ax is the matplotlib Axes to draw into, and a new figure is made when it is None.

When the years plotted reach the year recorded in trend.attrs["retrieved_at"], a note below the axis label says that the year was incomplete when the counts were taken, as in "2026 was incomplete when retrieved on 2026-09-29." Scopus goes on indexing a year's publications after the year has begun. The note is read from the recorded time alone, never from the clock of the session drawing the figure, so a trend that records none is drawn without it. The R twin's figures carry the same note in their caption.

Source code in src/scopusflow/plots.py
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def plot_trend(trend: pd.DataFrame, ax=None):
    """Plot publication counts over time as a filled area, line and points.

    ``trend`` has columns ``["year", "n"]``, as
    [`scopus_trend`][scopusflow.trend.scopus_trend] and
    [`year_counts`][scopusflow.trend.year_counts] return them. Returns the
    matplotlib ``Axes``. An empty frame raises
    ``ValueError("The trend has no years to plot.")``, as the R twin's
    ``plot_scopus_trend()`` does. ``ax`` is the matplotlib ``Axes`` to draw
    into, and a new figure is made when it is ``None``.

    When the years plotted reach the year recorded in
    ``trend.attrs["retrieved_at"]``, a note below the axis label says that the
    year was incomplete when the counts were taken, as in "2026 was incomplete
    when retrieved on 2026-09-29." Scopus goes on indexing a year's
    publications after the year has begun. The note is read from the recorded
    time alone, never from the clock of the session drawing the figure, so a
    trend that records none is drawn without it. The R twin's figures carry
    the same note in their caption.
    """
    import matplotlib.pyplot as plt

    from .trend import _incomplete_note

    # An empty frame would otherwise fail inside numpy, or draw an empty axes.
    if len(trend) == 0:
        raise ValueError("The trend has no years to plot.")

    if ax is None:
        _, ax = plt.subplots()

    years = trend["year"]
    counts = trend["n"]
    ax.fill_between(years, counts, alpha=0.16, color=_TREND_COLOUR)
    ax.plot(years, counts, color=_TREND_COLOUR, linewidth=2)
    ax.scatter(years, counts, color=_TREND_COLOUR, s=18, zorder=3)
    ax.set_ylim(bottom=0)
    ax.set_xlabel("Year")
    ax.set_ylabel("Records")
    note = _incomplete_note(years, trend.attrs.get("retrieved_at"))
    if note is not None:
        # Left-aligned with the axes and placed beneath the axis label, whose
        # drawn extent sets the height, so the note covers neither the label
        # nor the tick labels at any figure size.
        ax.annotate(
            note, xy=(0, 0), xycoords=("axes fraction", ax.xaxis.label),
            xytext=(0, -4), textcoords="offset points", ha="left", va="top",
            fontsize=7.5, color="#737373", annotation_clip=False,
        )
    _clean_axes(ax)
    return ax
sf.plot_trend(sf.year_counts(records))
show()
2026-11-15T17:25:42.825567 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/

plot_top

plot_top(top, ax=None)

Plot a horizontal bar chart of the most frequent values, largest on top.

top has columns ["value", "n"] (from top); each bar carries its count at its end, matching the R plot_scopus_top. ax is the matplotlib Axes to draw into, and a new figure is made when it is None. Returns the matplotlib Axes. An empty tally raises ValueError.

Source code in src/scopusflow/plots.py
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
def plot_top(top: pd.DataFrame, ax=None):
    """Plot a horizontal bar chart of the most frequent values, largest on top.

    ``top`` has columns ``["value", "n"]`` (from
    [`top`][scopusflow.records.top]); each bar carries its count at its end,
    matching the R ``plot_scopus_top``. ``ax`` is the matplotlib ``Axes`` to
    draw into, and a new figure is made when it is ``None``. Returns the
    matplotlib ``Axes``. An empty tally raises ``ValueError``.
    """
    import matplotlib.pyplot as plt

    # An all-missing publication column tallies to no rows at all, which would
    # otherwise surface as an opaque max() error on the label widths below.
    if len(top) == 0:
        raise ValueError("The tally has no rows to plot.")

    if ax is None:
        _, ax = plt.subplots()

    # Reverse so the largest count sits at the top of the chart.
    ordered = top.iloc[::-1]
    bars = ax.barh(ordered["value"].astype(str), ordered["n"], color=_TOP_COLOUR)
    labels = [f"{int(n):,}" for n in ordered["n"]]
    ax.bar_label(bars, labels=labels, padding=3, fontsize=8, color="#4d4d4d")
    # Headroom derived from the widest label, so the count on the longest bar
    # stays inside the axes, where a wide one would otherwise clip at the
    # right edge. Mirrors the
    # R plot's label-width-derived axis expansion.
    ax.margins(x=0.04 + 0.024 * max(len(lab) for lab in labels))
    ax.set_xlim(left=0)
    ax.set_xlabel("Records")
    ax.set_ylabel("")
    _clean_axes(ax)
    return ax
sf.plot_top(sf.top(records, by="source"))
show()
2026-11-15T17:25:43.007081 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/

plot_comparison

plot_comparison(comparison, highlight=None, interval=True, counts_in_legend=True, ax=None, legend_inside=False)

Plot each comparison topic's share of the reference literature over time.

comparison is the frame from compare_topics. With interval a shaded Wilson band shows how stable each yearly share is (illustrative, never a confidence interval, since Scopus counts are exact). highlight names one topic to draw in an accent colour, the rest in grey. With counts_in_legend (the default) each label carries the topic's total record count, for example machine learning (n = 1,204).

A legend is only drawn when there are too many topics to label the lines directly. legend_inside governs where it then sits: with the default False matplotlib chooses the least-obtrusive spot automatically (loc="best"); with True the legend is placed inside the axes, in whichever corner has the most free space, saving the width an outside legend would otherwise take. Returns the matplotlib Axes.

ax is the matplotlib Axes to draw into, and a new figure is made when it is None.

Source code in src/scopusflow/plots.py
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
def plot_comparison(comparison: pd.DataFrame, highlight=None, interval: bool = True,
                    counts_in_legend: bool = True, ax=None, legend_inside: bool = False):
    """Plot each comparison topic's share of the reference literature over time.

    ``comparison`` is the frame from
    [`compare_topics`][scopusflow.compare.compare_topics]. With ``interval`` a
    shaded Wilson band shows how stable each yearly share is (illustrative,
    never a confidence interval, since Scopus counts are exact). ``highlight``
    names one topic to draw in an accent colour, the rest in grey. With
    ``counts_in_legend`` (the default) each label carries the topic's total
    record count, for example ``machine learning (n = 1,204)``.

    A legend is only drawn when there are too many topics to label the lines
    directly. ``legend_inside`` governs where it then sits: with the default
    ``False`` matplotlib chooses the least-obtrusive spot automatically
    (``loc="best"``); with ``True`` the legend is placed inside the axes, in
    whichever corner has the most free space, saving the width an outside legend
    would otherwise take. Returns the matplotlib ``Axes``.

    ``ax`` is the matplotlib ``Axes`` to draw into, and a new figure is made
    when it is ``None``.
    """
    import matplotlib.pyplot as plt
    import matplotlib.ticker as mticker

    required = {"query_type", "abridged_query", "year", "comparison_percentage",
                "average_comparison_percentage"}
    if not required.issubset(comparison.columns):
        raise ValueError("comparison must be a topic-comparison frame.")

    comp_all = comparison[comparison["query_type"] == "comparison"]
    # A year whose reference has no records carries no defined share; it is
    # dropped and noted in the caption, mirroring the R plot.
    n_missing = int(comp_all["comparison_percentage"].isna().sum())
    df = comp_all[comp_all["comparison_percentage"].notna()].copy()
    if df.empty:
        raise ValueError("comparison has no comparison topics with a finite share to plot.")

    # The reference topic names the subtitle (it is the 100% denominator).
    ref_rows = comparison[comparison["query_type"] == "reference"]
    ref_names = ref_rows["abridged_query"].dropna().unique() if len(ref_rows) else []
    ref_label = str(ref_names[0]) if len(ref_names) == 1 else None

    order = (df.groupby("abridged_query")["average_comparison_percentage"].first()
             .reset_index()
             .sort_values(["average_comparison_percentage", "abridged_query"],
                          ascending=[False, True]))
    topics = list(order["abridged_query"])
    if highlight is not None and highlight not in topics:
        raise ValueError(f"highlight must be one of: {', '.join(topics)}.")

    # Optionally append each topic's total record count to its label.
    has_counts = counts_in_legend and "n" in df.columns
    totals = df.groupby("abridged_query")["n"].sum() if has_counts else None

    def _label(topic):
        return f"{topic} (n = {int(totals[topic]):,})" if has_counts else topic

    if ax is None:
        _, ax = plt.subplots()
    cmap = plt.get_cmap("viridis")
    spread = max(len(topics) - 1, 1)
    has_band = interval and {"n", "reference_n"}.issubset(df.columns)
    label_points = []

    for i, topic in enumerate(topics):
        sub = df[df["abridged_query"] == topic].sort_values("year")
        is_hi = highlight == topic
        if highlight is not None:
            colour = "#BB5566" if is_hi else "#BFBFBF"
            width = 1.6 if is_hi else 0.8
        else:
            colour = cmap(0.05 + 0.8 * i / spread)
            width = 1.4
        if has_band and (highlight is None or is_hi):
            lo, up = _wilson(sub["n"].to_numpy(), sub["reference_n"].to_numpy())
            ax.fill_between(sub["year"], lo, up, color=colour, alpha=0.16, linewidth=0)
        ax.plot(sub["year"], sub["comparison_percentage"], color=colour,
                linewidth=width, label=_label(topic))
        ax.scatter(sub["year"], sub["comparison_percentage"], color=colour, s=14, zorder=3)
        last = sub.iloc[-1]
        label_points.append((float(last["year"]), float(last["comparison_percentage"]),
                             _label(topic), colour, is_hi))

    # Cap the y-axis at the next 5% above the data (and bands), as the R plot
    # does, to remove dead headroom.
    if has_band:
        _, band_upper = _wilson(df["n"].to_numpy(), df["reference_n"].to_numpy())
        top = max(float(df["comparison_percentage"].max()), float(band_upper.max()))
    else:
        top = float(df["comparison_percentage"].max())
    ymax = min(100, math.ceil(top / 5) * 5)
    ax.set_ylim(0, ymax)

    # Label the lines directly when they fit legibly; otherwise fall back to a
    # legend. The gap is the minimum vertical separation between labels.
    gap = ymax * 0.055
    if highlight is not None:
        to_label = [p for p in label_points if p[4]]
    elif len(topics) <= 8 and (len(topics) - 1) * gap <= ymax:
        to_label = label_points
    else:
        to_label = []
        if legend_inside:
            legend_loc = _free_corner(
                df["year"], df["comparison_percentage"],
                (float(df["year"].min()), float(df["year"].max())), (0.0, ymax))
        else:
            legend_loc = "best"
        ax.legend(fontsize=8, loc=legend_loc, frameon=False,
                  ncol=2 if len(topics) > 8 else 1)

    # Place each label just past its line's endpoint. The labels are spread
    # apart at the end (see _decollide_once), once the final layout is known,
    # so none overlaps another.
    anns, label_xs, label_y_true = [], [], []
    if to_label:
        years = df["year"]
        dx = (float(years.max()) - float(years.min())) * 0.015 + 0.1
        for x, y_true, topic, colour, _is_hi in to_label:
            # No leader line: where labels converge they are nudged apart, and a
            # leader to a nudged label then cuts across the neighbouring labels'
            # text. The colour match and shared top-to-bottom order keep each
            # label tied to its line without one.
            anns.append(ax.annotate(
                topic, xy=(x, y_true), xytext=(x + dx, y_true), textcoords="data",
                va="center", ha="left", fontsize=8, color=colour,
                annotation_clip=False,
            ))
            label_xs.append(x + dx)
            label_y_true.append(y_true)
    ax.set_xlabel("")
    ax.set_ylabel("Share of reference records")
    ax.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=100, decimals=0))
    ax.set_title("Topic share within a reference literature, over time",
                 loc="left", fontsize=12, pad=22)
    if ref_label:
        ax.annotate(
            f"Each line: % of '{ref_label}' records that also match the topic",
            xy=(0, 1), xycoords="axes fraction", xytext=(0, 6),
            textcoords="offset points", ha="left", va="bottom",
            fontsize=9, color="#555555",
        )
    # A caption that names the source and guards against reading the illustrative
    # Wilson band as an inferential confidence interval (it is not).
    caption = (f"Source: 'Scopus' Search API. Years {int(df['year'].min())} "
               f"to {int(df['year'].max())}.")
    if has_band:
        caption += ("\nShaded band: illustrative Wilson stability range "
                    "(not a confidence interval), wider where the reference set is small.")
    if n_missing > 0:
        plural = "" if n_missing == 1 else "s"
        caption += (f"\n{n_missing} year-topic value{plural} omitted for want "
                    "of reference records.")
    ax.annotate(caption, xy=(0, 0), xycoords="axes fraction", xytext=(0, -24),
                textcoords="offset points", ha="left", va="top",
                fontsize=7.5, color="#737373")
    _clean_axes(ax)

    if anns:
        # Re-spread the labels on every draw, measuring the rendered text height in
        # the layout that actually applies, so they never overlap however the lines
        # converge or the figure is sized. Running on the draw means it stays
        # correct after the caller tightens the layout. A guard stops the redraw it
        # requests from recursing once the positions have settled.
        _busy = {"on": False}

        def _on_draw(_event=None):
            if _busy["on"]:
                return
            _busy["on"] = True
            try:
                if _decollide_once(ax, anns, label_xs, label_y_true, ymax, gap):
                    ax.figure.canvas.draw_idle()
            finally:
                _busy["on"] = False

        ax.figure.canvas.mpl_connect("draw_event", _on_draw)
        # An initial draw so a one-shot render (savefig without a prior draw) still
        # gets de-collided labels.
        try:
            ax.figure.canvas.draw()
        except Exception:
            pass
    return ax
years = list(range(2013, 2022))
ref_n = [400, 550, 700, 850, 1000, 1150, 1300, 1450, 1600]
shares = {"computer vision": 34.0, "natural language processing": 24.0,
          "medical imaging": 11.0, "drug discovery": 6.0}

rows = [{"query": "deep learning", "query_type": "reference",
         "abridged_query": "deep learning", "year": year, "n": n,
         "reference_n": n, "comparison_percentage": 100.0,
         "average_comparison_percentage": 100.0}
        for year, n in zip(years, ref_n)]
for topic, end in shares.items():
    for i, (year, n) in enumerate(zip(years, ref_n)):
        pct = end * (0.45 + 0.55 * i / (len(years) - 1))
        rows.append({"query": f"(deep learning) AND ({topic})",
                     "query_type": "comparison",
                     "abridged_query": topic, "year": year,
                     "n": int(pct * n / 100), "reference_n": n,
                     "comparison_percentage": pct,
                     "average_comparison_percentage": end})

comparison = pd.DataFrame(rows, columns=sf.COMPARISON_COLUMNS)
sf.plot_comparison(comparison)
show()
2026-11-15T17:25:43.397107 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/

plot_scopus_intersections

plot_scopus_intersections(x, highlight=None, highlight_label=None, ax=None)

Plot concept and intersection counts as a horizontal lollipop chart.

x is a frame from scopus_intersections (columns label, n and type). Counts run along a logarithmic x axis, so fields spanning orders of magnitude sit together with their smaller intersections. Rows named in highlight are drawn in the focal colour, and highlight_label names them in the legend (derived from their type when unset). Returns the matplotlib Axes. Mirrors the R plot_scopus_intersections.

ax is the matplotlib Axes to draw into, and a new figure is made when it is None. A row without a positive count cannot sit on a log axis and is dropped with a warning. ValueError is raised for a frame without rows or without a positive count, for duplicate labels and for a highlight naming a label the frame does not hold.

Source code in src/scopusflow/plots.py
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
def plot_scopus_intersections(x, highlight=None, highlight_label=None, ax=None):
    """Plot concept and intersection counts as a horizontal lollipop chart.

    ``x`` is a frame from
    [`scopus_intersections`][scopusflow.intersections.scopus_intersections]
    (columns ``label``, ``n`` and ``type``). Counts run along a logarithmic x
    axis, so fields spanning orders of magnitude sit together with their
    smaller intersections. Rows named in ``highlight`` are drawn in the focal
    colour, and ``highlight_label`` names them in the legend (derived from
    their type when unset). Returns the matplotlib ``Axes``. Mirrors the R
    ``plot_scopus_intersections``.

    ``ax`` is the matplotlib ``Axes`` to draw into, and a new figure is made
    when it is ``None``. A row without a positive count cannot sit on a log
    axis and is dropped with a warning. ``ValueError`` is raised for a frame
    without rows or without a positive count, for duplicate labels and for a
    ``highlight`` naming a label the frame does not hold.
    """
    import warnings

    import matplotlib.pyplot as plt
    from matplotlib.lines import Line2D
    from matplotlib.ticker import FuncFormatter, NullFormatter

    required = {"label", "n", "type"}
    if not required.issubset(x.columns):
        raise ValueError(
            "x must have columns 'label', 'n' and 'type' (see scopus_intersections())."
        )
    df = x.copy()
    if len(df) == 0:
        raise ValueError("The intersections frame has no rows to plot.")
    if df["label"].duplicated().any():
        raise ValueError(
            "The 'label' column must be unique to place each row on its own line."
        )

    if highlight is None:
        highlight = []
    elif isinstance(highlight, str):
        highlight = [highlight]
    else:
        highlight = list(highlight)
    known = set(df["label"])
    if highlight and any(h not in known for h in highlight):
        raise ValueError(
            "highlight must name rows to accent, among: "
            + ", ".join(map(str, df["label"])) + "."
        )

    counts = pd.to_numeric(df["n"], errors="coerce")
    keep = counts.notna() & (counts > 0)
    n_dropped = int((~keep).sum())
    if n_dropped:
        warnings.warn(
            f"{n_dropped} row(s) without a positive count cannot sit on a log axis "
            "and were dropped.",
            stacklevel=2,
        )
    df = df.loc[keep].copy()
    df["n"] = counts[keep].astype(float)
    if len(df) == 0:
        raise ValueError("No row has a positive count to place on the log axis.")

    if highlight_label is None:
        hi_types = set(df.loc[df["label"].isin(highlight), "type"])
        highlight_label = {
            frozenset({"intersection"}): "Focal intersection",
            frozenset({"concept"}): "Focal concept",
        }.get(frozenset(hi_types), "Focal set")

    df["grp"] = [
        "highlight" if lab in highlight else typ
        for lab, typ in zip(df["label"], df["type"], strict=True)
    ]
    df = df.sort_values("n", kind="stable").reset_index(drop=True)

    colours = {
        "concept": _CONCEPT_COLOUR,
        "intersection": _INTERSECTION_COLOUR,
        "highlight": _HIGHLIGHT_COLOUR,
    }
    legend_names = {
        "concept": "Concept",
        "intersection": "Intersection",
        "highlight": highlight_label,
    }

    lo = max(1.0, df["n"].min()) * 0.55  # the smallest point clears the axis
    hi = df["n"].max() * 4               # headroom for the widest count label
    # A constant ratio (not increment) beyond each point renders as a constant
    # pixel gap on a log axis, mirroring the R plot's label placement.
    gap_mult = 10 ** (0.024 * math.log10(hi / lo))

    if ax is None:
        _, ax = plt.subplots()

    for y, (_, row) in enumerate(df.iterrows()):
        colour = colours[row["grp"]]
        ax.hlines(y, lo, row["n"], color=colour, linewidth=1.6, zorder=1)
        ax.scatter(row["n"], y, color=colour, s=42, zorder=2)
        ax.text(row["n"] * gap_mult, y, f"{int(row['n']):,}",
                va="center", ha="left", fontsize=8, color="#4d4d4d")

    ax.set_yticks(range(len(df)))
    ax.set_yticklabels(df["label"])
    ax.set_xscale("log")
    ax.set_xlim(lo, hi)
    ax.xaxis.set_major_formatter(
        FuncFormatter(lambda v, _pos: f"{int(v):,}" if v >= 1 else f"{v:g}")
    )
    ax.xaxis.set_minor_formatter(NullFormatter())
    ax.set_xlabel("Records (log scale)")
    ax.set_title("Records matching each concept and intersection")
    _clean_axes(ax)

    present = [g for g in ("concept", "intersection", "highlight")
               if (df["grp"] == g).any()]
    handles = [
        Line2D([0], [0], marker="o", linestyle="", markersize=7,
               color=colours[g], label=legend_names[g])
        for g in present
    ]
    if handles:
        ax.legend(handles=handles, loc="upper left", frameon=False, fontsize=8)
    return ax
sets = pd.DataFrame({
    "label": ["semantic priming", "mental simulation",
              "semantic priming × mental simulation"],
    "query": ["TITLE-ABS-KEY(semantic priming)",
              "TITLE-ABS-KEY(mental simulation)",
              "(TITLE-ABS-KEY(semantic priming)) AND "
              "(TITLE-ABS-KEY(mental simulation))"],
    "n": pd.array([6600, 2600, 18], dtype="Int64"),
    "type": ["concept", "concept", "intersection"],
    "size": [1, 1, 2],
    "members": ["semantic priming", "mental simulation",
                "semantic priming; mental simulation"],
})
sf.plot_scopus_intersections(
    sets, highlight=["semantic priming × mental simulation"]
)
show()
2026-11-15T17:25:43.698484 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/