Search plans and quota-aware retrieval

The Scopus Search API is generous but bounded. A weekly quota limits how many requests you may make, a rate limit caps how fast you may make them, and no single query will page past its first few thousand records. This guide shows how scopusflow works within those bounds so that a large retrieval stays cheap to plan, honest about its size and resumable when it stops. Every example assumes import scopusflow as sf. The steps that count or fetch records contact the API and need a Scopus key configured for pybliometrics, so they are shown but not run here. Building queries and plans is offline and runs anywhere.

Size before you spend

Counting is cheap. scopus_count issues a single request that asks the API how many records a query matches without downloading any of them, which makes it the right way to size a search before committing quota to a harvest. It takes the same years and field arguments as a plan, so you can size exactly what you intend to fetch.

import scopusflow as sf

q = sf.scopus_query("language learning", "effect size", field="TITLE-ABS-KEY")

# One cheap request; returns an int. Needs a Scopus key.
n = sf.scopus_count(q, years=range(2010, 2021))
n

Because the result is a plain integer you can branch on it before paying for anything heavier, asking whether the search is small enough to fetch in one piece or large enough to need partitioning.

if sf.scopus_count(q) > 5000:
    print("Too large for a single pull; partition by year.")

Why the offset ceiling forces a partition

A single query cannot be paged indefinitely. The API stops serving results once the start offset reaches a few thousand records, so a query matching more than that ceiling can never be retrieved in full from one uninterrupted search. The remedy is to split the search into pieces that each stay under the ceiling, and the year of publication is the natural facet to split on, because it is recorded on every record and divides a literature without leaving gaps or overlaps.

A SearchPlan with partition="year" does exactly this, turning one oversized search into one cell per year. Each cell carries the same wrapped query and a single year, so each contacts the API as its own bounded search. The plan is a plain object that describes the search before it is run, so it can be printed and version-controlled alongside the analysis.

plan = sf.SearchPlan(q, years=range(2010, 2021), partition="year")

# One cell per year, each well under the offset ceiling.
out([(c.cell, c.year) for c in plan.cells()])

[(1, 2010), (2, 2011), (3, 2012), (4, 2013), (5, 2014), (6, 2015), (7, 2016), (8, 2017), (9, 2018), (10, 2019), (11, 2020)]

The string each cell will send is the wrapped_query, with the field tag already folded in. Inspecting it is offline and shows precisely what the API receives before any request is made.

out(plan.wrapped_query)

TITLE-ABS-KEY(language learning) AND TITLE-ABS-KEY(effect size)

Counting and partitioning compose. Run scopus_count over the same years first, and if the total clears the ceiling you already know the year partition is needed, well before a fetch is part way through.

total = sf.scopus_count(q, years=range(2010, 2021))
plan = sf.SearchPlan(
    q,
    years=range(2010, 2021),
    partition="year" if total > 5000 else "none",
)

A resumable, checkpointed harvest

fetch_plan runs the cells in turn and returns one normalised frame. Given a cache_dir it writes each cell to disk as soon as that cell completes, so a run interrupted halfway, or stopped by the quota, resumes from where it left off and never pays twice for a cell that already finished. Resuming is the default, so a second call against the same directory reads the finished cells back from disk and only fetches what is missing.

records = sf.fetch_plan(plan, cache_dir="language-harvest", resume=True)
records.shape

A cache directory belongs to one plan. Checkpoints are keyed by cell number, so on resume each checkpoint's own recorded query and view are compared against the cell's, and a checkpoint written by a different plan is warned about and refetched, so it can never be returned silently. Give each plan its own directory all the same, since that refetch spends the quota the cache was meant to save.

The checkpoint format is parquet by default and falls back to CSV when no parquet engine is installed. You can ask for CSV explicitly when you want checkpoints you can open in any tool.

records = sf.fetch_plan(plan, cache_dir="language-harvest", format="csv")

For a long harvest you can hand fetch_plan a zero-argument should_stop callable. It is checked before each cell, and when it returns True the harvest stops early and returns what it has gathered so far. Because every completed cell is already on disk, stopping this way costs nothing for the work already done, and the next call picks up from the first unfinished cell.

import time

deadline = time.monotonic() + 600  # stop politely after ten minutes

records = sf.fetch_plan(
    plan,
    cache_dir="language-harvest",
    should_stop=lambda: time.monotonic() > deadline,
)

Whatever the query was, the result is one tidy frame with the stable RECORD_COLUMNS schema, the same shape a single fetch would return, with entry_number renumbered across the combined cells.

Writing the search up

A harvest is rarely the end of the work. A systematic review has to report the search itself, in enough detail that a reader can repeat it, and the reporting standard for that is PRISMA-S (Rethlefsen et al., 2021). scopus_search_report assembles the record from what the plan and the harvest already carry, so the methods section is written from the objects and never from memory.

A plan on its own can be reported before it is run, which is useful when a protocol has to be registered in advance. The plan below describes the search that produced the bundled corpus, so that the record and the records match.

graphene = sf.SearchPlan("graphene supercapacitor", years=range(2015, 2025),
                         field="TITLE-ABS-KEY", partition="year")
out(sf.scopus_search_report(graphene))

Search strategy record (PRISMA-S)

Database: Scopus, on the Elsevier Scopus Search API
Search expression: TITLE-ABS-KEY(graphene supercapacitor)
Field tag: TITLE-ABS-KEY
Years: 2015 to 2024
Partition: one cell per year, 10 cells
View: STANDARD
Page size: 200 records per request
Paging: unrecorded
Date searched: unrecorded, this plan has not been run
Software: unrecorded
Records retrieved: none, this plan has not been run
Records reported as matching: unrecorded, this plan has not been run
Completeness: unrecorded, this plan has not been run
Duplicates removed: unrecorded, this plan has not been run
Records carrying a DOI: unrecorded, this plan has not been run

Cells
  1 (2015): retrieved records unrecorded
  2 (2016): retrieved records unrecorded
  3 (2017): retrieved records unrecorded
  4 (2018): retrieved records unrecorded
  5 (2019): retrieved records unrecorded
  6 (2020): retrieved records unrecorded
  7 (2021): retrieved records unrecorded
  8 (2022): retrieved records unrecorded
  9 (2023): retrieved records unrecorded
  10 (2024): retrieved records unrecorded

PRISMA 2020 identification
  Records identified from Scopus: unrecorded, this plan has not been run
  Duplicate records removed before screening: unrecorded, this plan has not been run

PRISMA-S items this record supplies
  1 Database name. Scopus, searched on the Elsevier Scopus Search API.
  8 Full search strategies. The search expression, field tag and year limit of every cell, as the plan sends them.
  9 Limits and restrictions. Publication years 2015 to 2024. scopusflow applies no document type, language or subject area limit of its own.

PRISMA-S items only you can supply
  2 Multi-database searching. Whether any database besides Scopus was searched, and how the strategy was translated for it.
  3 Study registries. Any trial or study registry searched.
  4 Online resources and browsing. Any web site, table of contents or other source searched or browsed by hand.
  5 Citation searching. Any backward or forward citation searching.
  6 Contacts. Any authors or organisations contacted for studies.
  7 Other methods. Any further method used to identify records.
  10 Search filters. Any published or validated search filter used, and where it came from. scopusflow applies none of its own.
  11 Prior work. Any earlier review or strategy this search was adapted from.
  12 Updates. Whether the search was re-run or updated, and when.
  13 Dates of searches. The date of each search, which this plan has not been run to produce.
  14 Peer review. Whether the strategy was peer reviewed, and by whom.
  15 Total records. The number of records identified, which this plan has not been run to produce.
  16 Deduplication. How duplicate records were removed, which this plan has not been run to produce.

The reporting standard is PRISMA-S (Rethlefsen et al., 2021, Systematic Reviews, 10, 39, https://doi.org/10.1186/s13643-020-01542-z), with the identification counts of the PRISMA 2020 flow diagram.

Notice how much of it says "unrecorded". Nothing has been retrieved yet, so there is nothing to state, and the record says so in words, since a blank there would be read as a zero. That is the governing rule throughout: the record states only what the objects hold. It never substitutes the current time for a retrieval that did not record one, never gives a completeness figure for a harvest whose reported total is unknown, and never counts duplicates unless a merge recorded removing them.

After a harvest the picture fills in. fetch_plan attaches the plan, the retrieval time, the version, the paging mode and the per-cell accounting, so the record has everything it needs and you never set any of it yourself. The bundled corpus stands in for a harvest here, since Scopus records may not be redistributed, so those attributes are written out below to show what each one contributes.

from datetime import datetime, timezone

records = sf.example_records()
records.attrs["plan"] = graphene
records.attrs["retrieved_at"] = datetime(2026, 7, 22, 9, 15, tzinfo=timezone.utc)
records.attrs["scopusflow_version"] = "0.3.0"
records.attrs["paging"] = "offset"
per_year = records.groupby("year").size()
records.attrs["cell_totals"] = pd.DataFrame({
    "cell": range(1, len(per_year) + 1),
    "date": [str(y) for y in per_year.index],
    "n_records": per_year.values,
    "reported_total": per_year.values,
})

report = sf.scopus_search_report(records)
out(report)

Search strategy record (PRISMA-S)

Database: Scopus, on the Elsevier Scopus Search API
Search expression: TITLE-ABS-KEY(graphene supercapacitor)
Field tag: TITLE-ABS-KEY
Years: 2015 to 2024
Partition: one cell per year, 10 cells
View: STANDARD
Page size: 200 records per request
Paging: offset
Date searched: 2026-07-22 09:15:00 UTC
Software: scopusflow 0.3.0
Records retrieved: 138
Records reported as matching: 138
Completeness: every record the API reported as matching was retrieved
Duplicates removed: unrecorded, no de-duplication step was recorded for this set
Records carrying a DOI: 127 of 138

Cells
  1 (2015): 15 retrieved, 15 reported, complete
  2 (2016): 9 retrieved, 9 reported, complete
  3 (2017): 10 retrieved, 10 reported, complete
  4 (2018): 15 retrieved, 15 reported, complete
  5 (2019): 19 retrieved, 19 reported, complete
  6 (2020): 13 retrieved, 13 reported, complete
  7 (2021): 13 retrieved, 13 reported, complete
  8 (2022): 15 retrieved, 15 reported, complete
  9 (2023): 15 retrieved, 15 reported, complete
  10 (2024): 14 retrieved, 14 reported, complete

PRISMA 2020 identification
  Records identified from Scopus: 138
  Duplicate records removed before screening: unrecorded, no de-duplication step was recorded for this set

PRISMA-S items this record supplies
  1 Database name. Scopus, searched on the Elsevier Scopus Search API.
  8 Full search strategies. The search expression, field tag and year limit of every cell, as the plan sends them.
  9 Limits and restrictions. Publication years 2015 to 2024. scopusflow applies no document type, language or subject area limit of its own.
  13 Dates of searches. 2026-07-22 09:15:00 UTC.
  15 Total records. 138 records retrieved from Scopus, of 138 the API reported as matching.

PRISMA-S items only you can supply
  2 Multi-database searching. Whether any database besides Scopus was searched, and how the strategy was translated for it.
  3 Study registries. Any trial or study registry searched.
  4 Online resources and browsing. Any web site, table of contents or other source searched or browsed by hand.
  5 Citation searching. Any backward or forward citation searching.
  6 Contacts. Any authors or organisations contacted for studies.
  7 Other methods. Any further method used to identify records.
  10 Search filters. Any published or validated search filter used, and where it came from. scopusflow applies none of its own.
  11 Prior work. Any earlier review or strategy this search was adapted from.
  12 Updates. Whether the search was re-run or updated, and when.
  14 Peer review. Whether the strategy was peer reviewed, and by whom.
  16 Deduplication. How duplicate records were removed, which this set does not record.

The reporting standard is PRISMA-S (Rethlefsen et al., 2021, Systematic Reviews, 10, 39, https://doi.org/10.1186/s13643-020-01542-z), with the identification counts of the PRISMA 2020 flow diagram.

The completeness lines are worth a moment. Each cell is shown against the number of records the API reported for it, so a cell that came back short stays visible where a total would have hidden it, and the overall figure is given only because every cell reported one. Drop any of those attributes and the corresponding line says so instead.

The methods paragraph is the same record as prose, ready to paste into a manuscript and edit.

out(report.format(style="paragraph"))

The literature was searched in Scopus, on the Elsevier Scopus Search API, on 22 July 2026. The search expression was TITLE-ABS-KEY(graphene supercapacitor), limited to publication years 2015 to 2024. It was partitioned into 10 cells, one per year, each retrieved through the STANDARD view in pages of 200 records under offset paging. The search retrieved 138 records, matching the 138 the API reported, so every reported record was retrieved. Of the records retrieved, 127 carry a DOI. No de-duplication step was recorded for this set. The search was run with scopusflow 0.3.0. The PRISMA-S items this record cannot supply, among them peer review of the strategy, grey literature and any other database searched, remain yours to report.

Supplying a file writes the whole record as Markdown, including a runnable snippet that rebuilds the plan, which makes a natural supplementary file.

sf.scopus_search_report(records, file="search-record.md")

Five of the sixteen PRISMA-S items are answered here from the objects, and a sixth, de-duplication, would be too had these records been merged with scopus_combine. The rest, among them peer review of the strategy, grey literature and any other database searched, are listed as yours to supply, because the package has no way to know them.

Watching progress

Per-cell progress is emitted on the scopusflow logger, which is silent by default. Attaching a handler prints a line as each cell is fetched or loaded from cache, which is worth doing for a harvest that spans many years.

import logging

logging.getLogger("scopusflow").addHandler(logging.StreamHandler())
logging.getLogger("scopusflow").setLevel(logging.INFO)

records = sf.fetch_plan(plan, cache_dir="language-harvest")

Re-running and tracking change

Pointing a later run at a fresh directory fetches the plan again, and comparing the two DOI sets with diff_dois shows what the literature gained or lost in between. The earlier checkpoints stay untouched, so the original harvest remains exactly as it was.

later = sf.fetch_plan(plan, cache_dir="language-harvest-2")
sf.diff_dois(old=records, new=later)