Execute a 'Scopus' search plan, with optional caching and resume
Source:R/fetch_plan.R
scopus_fetch_plan.RdRuns every cell of a scopus_plan() in turn, optionally caching each cell's
result so that an interrupted or quota-limited retrieval can resume without
re-spending quota on the cells already fetched. Results are accumulated and
bound once into a single scopus_records tibble.
Usage
scopus_fetch_plan(
plan,
max_results = Inf,
cache_dir = NULL,
resume = TRUE,
api_key = NULL,
inst_token = NULL,
verbose = FALSE
)Arguments
- plan
A
scopus_planobject fromscopus_plan().- max_results
Maximum records to retrieve per cell (default
Inf).- cache_dir
Optional directory for per-cell cache files. The default of
NULLperforms no caching. Pass an explicit path you control, orscopus_cache_dir()to use a managed, clearable cache undertools::R_user_dir(). Caching happens only when you opt in through this argument. A cache directory serves one plan: cells are checkpointed by their position in the plan and their year, so give each distinct plan its own directory. As a safeguard, a checkpoint is served only when the query, year, view and page size it was fetched under all match the plan cell, and only when its ownmax_resultsdid not truncate it below what is being asked for now; anything else is a cache miss, refetched and overwritten. A checkpoint holding more records than the currentmax_resultsasks for is served trimmed to that cap, the fuller set staying on disk. A checkpoint that cannot be read back, for example one left half-written by an interrupted run, is also treated as a miss, and never aborts the harvest.- resume
Logical. When
TRUEandcache_diris set, a cell whose cache file already exists is loaded from disk, sparing a second request.- api_key, inst_token
Optional credentials (see
scopus_has_key()).- verbose
Logical. When
TRUE, per-cell progress is reported.
Value
A scopus_records tibble combining all cells, with the originating
plan attached as the plan attribute and the per-cell accounting as
cell_totals, a tibble of cell, date, n_records and
reported_total (the count the API gave for that cell, NA where it gave
none). total_results is their sum, and is NA unless every cell
reported one, since a partial sum would understate the search while
looking like a real figure. A cell that comes back shorter than the API
said it should warns, because a truncated or refused download otherwise
arrives as a merely small result set; a cell stopped by max_results is
short by request and does not warn. scopus_search_report() reads all of
this back. The retrieved_at and
scopusflow_version attributes described in scopus_fetch() are carried
across from the cells: the time is the earliest of them, since a combined
set is only as fresh as its oldest cell, and every version that
contributed is listed, since resuming an older cache means more than one
did. Both are omitted when any cell cannot supply them, as a checkpoint
written before they existed cannot. Dating the whole from the part of it
that can be dated would misreport the set.
API access
Any cell not served from cache requires a valid API key and internet access.
The API access section of scopus_count() gives the details.
Examples
if (FALSE) { # scopusflow::scopus_has_key()
plan <- scopus_plan("graphene supercapacitor", years = 2015:2024,
field = "TITLE-ABS-KEY", partition = "year")
dir <- file.path(tempdir(), "graphene-cache")
# `max_results` caps each yearly cell, so the example stays small and
# quota-light; drop it to harvest every record in the plan.
recs <- scopus_fetch_plan(plan, max_results = 25, cache_dir = dir, resume = TRUE)
}
# The offline companion, which needs no key: a record set with the plan
# that describes it attached. 'Scopus' records may not be redistributed, so
# the bundled corpus of real articles stands in for the harvest, and the
# plan describes the same search, one cell per year.
plan <- scopus_plan("graphene supercapacitor", years = 2015:2024,
field = "TITLE-ABS-KEY", partition = "year")
recs <- example_records
attr(recs, "plan") <- plan
recs
#> <scopus_records> 138 records
#> query: "graphene supercapacitor"
#> # A tibble: 138 × 9
#> entry_number scopus_id doi title authors year date publication citations
#> <int> <chr> <chr> <chr> <chr> <int> <chr> <chr> <int>
#> 1 1 NA 10.15… Enha… Jianhu… 2015 2015… Journal of… 1
#> 2 2 NA NA Fabr… Patric… 2015 2015… DigitalCom… 0
#> 3 3 NA 10.10… Flex… Zhiwei… 2015 2015… ACS Applie… 469
#> 4 4 NA 10.10… Heav… Vikran… 2015 2015… Electrochi… 195
#> 5 5 NA 10.10… Grap… Chih-T… 2015 2015… Small 108
#> 6 6 NA 10.10… Nano… Hao Ya… 2015 2015… Journal of… 47
#> 7 7 NA 10.11… Capa… Maxwel… 2015 2015… Physical R… 32
#> 8 8 NA 10.13… Grap… Nurbek… 2015 2015… Optics Let… 48
#> 9 9 NA 10.10… Ultr… Zhong-… 2015 2015… Advanced M… 275
#> 10 10 NA 10.10… Etch… Matthe… 2015 2015… Nanotechno… 24
#> # ℹ 128 more rows
attr(recs, "plan")
#> <scopus_plan> (10 cells, view "STANDARD", partition "year")
#> # A tibble: 10 × 6
#> cell query date year view page_size
#> * <int> <chr> <chr> <int> <chr> <int>
#> 1 1 TITLE-ABS-KEY(graphene supercapacitor) 2015 2015 STANDARD 200
#> 2 2 TITLE-ABS-KEY(graphene supercapacitor) 2016 2016 STANDARD 200
#> 3 3 TITLE-ABS-KEY(graphene supercapacitor) 2017 2017 STANDARD 200
#> 4 4 TITLE-ABS-KEY(graphene supercapacitor) 2018 2018 STANDARD 200
#> 5 5 TITLE-ABS-KEY(graphene supercapacitor) 2019 2019 STANDARD 200
#> 6 6 TITLE-ABS-KEY(graphene supercapacitor) 2020 2020 STANDARD 200
#> 7 7 TITLE-ABS-KEY(graphene supercapacitor) 2021 2021 STANDARD 200
#> 8 8 TITLE-ABS-KEY(graphene supercapacitor) 2022 2022 STANDARD 200
#> 9 9 TITLE-ABS-KEY(graphene supercapacitor) 2023 2023 STANDARD 200
#> 10 10 TITLE-ABS-KEY(graphene supercapacitor) 2024 2024 STANDARD 200