Tracking change over time¶
A literature is a moving target. Run the same search a few months apart and the result will have grown, and it may also have lost a record that was re-indexed under a different identifier. This guide shows how to compare two harvests of the same plan and see exactly which records the literature gained or lost in between. Both harvests here are cut from the corpus bundled with the package, 138 real articles on graphene supercapacitors that stand in for a Scopus retrieval, since retrieved records may not be redistributed and no key is assumed. The guide closes with the live form, where the later harvest comes from fetch_plan instead.
Two harvests to compare¶
A harvest is the normalised frame that fetch_plan returns, with the stable RECORD_COLUMNS schema. The baseline below is that same search as it stood at the start of 2022, which is the bundled harvest cut off after 2021.
import scopusflow as sf
records = sf.example_records()
baseline = records[records["year"] <= 2021].reset_index(drop=True)
out(baseline[["entry_number", "doi", "title", "year"]].tail(3))
| entry_number | doi | title | year |
|---|---|---|---|
| 92 | 10.47540/ijias.v1i3.303 | Comparative Analytical Modeling and Performance Investigation of Graphene-Based Super Capacitor with Four Traditional Batteries | 2021 |
| 93 | 10.1016/j.petrol.2021.109829 | An innovative graphene- supercapacitor for the treatment of crude oil viscosity at low temperatures | 2021 |
| 94 | 10.1039/d1ta09222g | Enhancing the energy storage capacity of graphene supercapacitors via solar heating | 2021 |
That leaves 94 of the 138 records. Every row carries the entry_number it held
in the harvest, which is what makes a partial fetch resumable, and a year
parsed from the cover date. A live pull arrives in the same shape, because
fetch_plan passes the pybliometrics result
through to_records, which strips the eid
back to a bare scopus_id and parses the coverDate. That identifier column is
empty in the bundled set, these records not having come from Scopus, so the
comparison below rests on the DOI, which is what it would key on in a live
setting anyway.
Some months on the search is repeated, and by then the three later years have been indexed. This second pull also loses one record from the earlier one, which is what happens when a paper is re-indexed under a different identifier, so we drop the first row to stand for it.
later = records.drop(index=records.index[0]).reset_index(drop=True)
out(later[["entry_number", "doi", "title", "year"]].tail(3))
| entry_number | doi | title | year |
|---|---|---|---|
| 136 | 10.1016/j.surfin.2024.105581 | Laser repeat scanning preparation of nitrogen sulfur doped graphene supercapacitor | 2024 |
| 137 | 10.1016/j.isci.2024.111696 | Boosting flexible laser-induced graphene supercapacitors performance through double pass laser processing | 2024 |
| 138 | 10.1016/j.jpowsour.2024.236149 | High-yield liquid phase production of high-quality graphene via dimethylacetamide-ethanol mixed solvent system | 2024 |
Pulling the DOIs out¶
The comparison runs on DOIs, so it helps to see what a harvest reduces to first. extract_dois reads the doi column from a record frame and returns a cleaned list. It strips a resolver prefix such as https://doi.org/ and a leading doi: label, and by default it de-duplicates case-insensitively, since the same DOI can arrive in different letter cases.
dois = sf.extract_dois(baseline)
out((len(dois), dois[:3]))
(84, ['10.15541/jim20140527', '10.1021/am509065d', '10.1016/j.electacta.2015.02.019'])The 94 baseline records reduce to 84 DOIs, the ten records that carry none being dropped, since a blank is no use to anything reading the list.
It also accepts a plain list, which is handy when the DOIs come from somewhere other than a harvest, for instance a column you read from a file. The cleaning and de-duplication apply either way.
# 10.5555 is the reserved DOI test prefix, so none of these resolve.
out(sf.extract_dois(["https://doi.org/10.5555/AB-1", "doi: 10.5555/ab-1",
"10.5555/cd-2"]))
['10.5555/AB-1', '10.5555/cd-2']That list collapses to two entries because the first two are the same DOI in different cases. Pass dedupe=False to keep every entry as it came in, including repeats.
What changed¶
diff_dois compares two harvests and returns a DataFrame with a doi column and a status column, where the status is added, removed or unchanged. It calls extract_dois on each side for you, so you can hand it the record frames directly, and the comparison is case-insensitive throughout.
changes = sf.diff_dois(old=baseline, new=later)
out(changes.head())
| doi | status |
|---|---|
| 10.1002/adfm.202315137 | added |
| 10.1002/asia.202400548 | added |
| 10.1002/ente.202201120 | added |
| 10.1002/slct.202302535 | added |
| 10.1002/smll.202301533 | added |
The 43 papers indexed since the baseline come back as added, the 83 present in
both pulls as unchanged, and the one that fell out of the later pull as
removed. The frame is sorted by status then DOI, so the categories group
together.
To act on one category, filter the frame on status as you would any pandas DataFrame.
out(changes[changes["status"] == "removed"])
| doi | status |
|---|---|
| 10.15541/jim20140527 | removed |
A count per category gives a quick read on how much moved between the two pulls.
out(changes["status"].value_counts())
| status | count |
|---|---|
| unchanged | 83 |
| added | 43 |
| removed | 1 |
Merging without duplicates¶
To carry a cumulative set forward across pulls, scopus_combine binds the harvests, renumbers entry_number across the result and, with dedupe=True, keeps one copy of each record. Records retrieved from Scopus are keyed on scopus_id. These carry none, so the key falls back to the DOI, compared case-insensitively.
combined = sf.scopus_combine(baseline, later, dedupe=True)
out((len(combined), len(sf.extract_dois(combined))))
(148, 127)That takes 231 concatenated rows down to 148: the 127 distinct DOIs, plus the 21 records carrying none, which cannot be matched this way and are kept, where collapsing them into one another would be a silent guess.
The merge records itself, which matters because the count exists only while it
happens: afterwards nothing in the result says how many rows went in. PRISMA-S
asks for exactly that figure, and scopus_search_report
reads it back from here, so the item is answered.
out(combined.attrs["combined"])
{'n_in': 231, 'n_out': 148, 'n_removed': 83, 'deduplicated': True}Keeping a record of each pull¶
Comparing against a past harvest only works if you kept it, so it is worth saving each pull as you go. A record frame is an ordinary pandas DataFrame, which means the usual pandas writers and readers round-trip it. Parquet preserves the column types exactly, which matters for the nullable integer columns in the schema.
import pandas as pd
baseline.to_parquet("baseline.parquet")
restored = pd.read_parquet("baseline.parquet")
sf.diff_dois(old=restored, new=later)
If you would rather have a plain-text artefact to commit alongside the analysis, baseline.to_csv("baseline.csv", index=False) works too, with pandas.read_csv to read it back.
In a live setting¶
Everything above runs offline because both harvests were cut from the bundled corpus. In a live setting the later harvest comes from the API, and that call needs a configured Scopus API key, which pybliometrics reads from its own configuration. The shape of the comparison does not change. You re-run the same SearchPlan through fetch_plan, read back the harvest you saved earlier, and diff the two.
import pandas as pd
q = sf.scopus_query("graphene", "supercapacitor", field="TITLE-ABS-KEY")
plan = sf.SearchPlan(q, years=range(2015, 2025), partition="year")
later = sf.fetch_plan(plan, cache_dir="graphene-harvest-2")
baseline = pd.read_parquet("baseline.parquet")
sf.diff_dois(old=baseline, new=later)
Run that on a schedule against a saved baseline and the added and removed rows tell you, harvest after harvest, precisely how the literature is shifting under your search.