Skip to content

Tracking change over time

A literature is a moving target. Run the same search a few months apart and the result will have grown, and it may also have lost a record that was re-indexed under a different identifier. This guide shows how to compare two harvests of the same plan and see exactly which records the literature gained or lost in between. Both harvests here are cut from the corpus bundled with the package, 138 real articles on graphene supercapacitors that stand in for a Scopus retrieval, since retrieved records may not be redistributed and no key is assumed. The guide closes with the live form, where the later harvest comes from fetch_plan instead.

Two harvests to compare

A harvest is the normalised frame that fetch_plan returns, with the stable RECORD_COLUMNS schema. The baseline below is that same search as it stood at the start of 2022, which is the bundled harvest cut off after 2021.

import scopusflow as sf

records = sf.example_records()
baseline = records[records["year"] <= 2021].reset_index(drop=True)
out(baseline[["entry_number", "doi", "title", "year"]].tail(3))
entry_number doi title year
92 10.47540/ijias.v1i3.303 Comparative Analytical Modeling and Performance Investigation of Graphene-Based Super Capacitor with Four Traditional Batteries 2021
93 10.1016/j.petrol.2021.109829 An innovative graphene- supercapacitor for the treatment of crude oil viscosity at low temperatures 2021
94 10.1039/d1ta09222g Enhancing the energy storage capacity of graphene supercapacitors via solar heating 2021

That leaves 94 of the 138 records. Every row keeps the entry_number it held in the full harvest, and a year parsed from the cover date. A live pull arrives in the same shape, because fetch_plan passes the pybliometrics result through to_records, which strips the eid back to a bare scopus_id and parses the coverDate. That identifier column is empty in the bundled set, these records not having come from Scopus, so the comparison below rests on the DOI, which is what it would key on in a live setting anyway.

Some months on the search is repeated, and by then the three later years have been indexed. This second pull also loses one record from the earlier one, which is what happens when a paper is re-indexed under a different identifier, so we drop the first row to stand for it.

later = records.drop(index=records.index[0]).reset_index(drop=True)
out(later[["entry_number", "doi", "title", "year"]].tail(3))
entry_number doi title year
136 10.1016/j.surfin.2024.105581 Laser repeat scanning preparation of nitrogen sulfur doped graphene supercapacitor 2024
137 10.1016/j.isci.2024.111696 Boosting flexible laser-induced graphene supercapacitors performance through double pass laser processing 2024
138 10.1016/j.jpowsour.2024.236149 High-yield liquid phase production of high-quality graphene via dimethylacetamide-ethanol mixed solvent system 2024

Pulling the DOIs out

The comparison runs on DOIs, so it helps to see what a harvest reduces to first. extract_dois reads the doi column from a record frame and returns a cleaned list. It strips a resolver prefix such as https://doi.org/ and a leading doi: label, and by default it de-duplicates case-insensitively, since the same DOI can arrive in different letter cases.

dois = sf.extract_dois(baseline)
out((len(dois), dois[:3]))
(84, ['10.15541/jim20140527', '10.1021/am509065d', '10.1016/j.electacta.2015.02.019'])

The 94 baseline records reduce to 84 DOIs, the ten records that carry none being dropped, since a blank is no use to anything reading the list.

It also accepts a plain list, which is handy when the DOIs come from somewhere other than a harvest, for instance a column you read from a file. The cleaning and de-duplication apply either way.

# 10.5555 is the reserved DOI test prefix, so none of these resolve.
out(sf.extract_dois(["https://doi.org/10.5555/AB-1", "doi: 10.5555/ab-1",
                     "10.5555/cd-2"]))
['10.5555/AB-1', '10.5555/cd-2']

That list collapses to two entries because the first two are the same DOI in different cases. Pass dedupe=False to keep every entry as it came in, including repeats.

What changed

diff_dois compares two harvests and returns a DataFrame with a doi column and a status column, where the status is added, removed or unchanged. It calls extract_dois on each side for you, so you can hand it the record frames directly, and the comparison is case-insensitive throughout.

changes = sf.diff_dois(old=baseline, new=later)
out(changes.head())
doi status
10.1002/adfm.202315137 added
10.1002/asia.202400548 added
10.1002/ente.202201120 added
10.1002/slct.202302535 added
10.1002/smll.202301533 added

The papers indexed since the baseline come back as added, those present in both pulls as unchanged, and the one that fell out of the later pull as removed. The later pull holds 44 papers from 2022 on, 43 of them with a DOI, so 43 are added. The two pulls share 93 records, 83 of them with a DOI, so 83 are unchanged. The records without a DOI, ten in the baseline and eleven in the later pull, are not compared. The frame counts them in attrs["n_old_without_doi"] and attrs["n_new_without_doi"], and it is sorted by status then DOI, so the categories group together.

out((changes.attrs["n_old_without_doi"], changes.attrs["n_new_without_doi"]))
(10, 11)

To act on one category, filter the frame on status as you would any pandas DataFrame.

out(changes[changes["status"] == "removed"])
doi status
10.15541/jim20140527 removed

A count per category gives a quick read on how much moved between the two pulls.

out(changes["status"].value_counts())
status count
unchanged 83
added 43
removed 1

Comparing records

diff_records compares the records themselves, matching them as scopus_combine de-duplicates them by default: on a shared scopus_id, or on a shared DOI where one of the two records lacks an identifier. A record without a DOI is then tracked by its identifier, and each row carries the record's title and year and its citation counts in both pulls. Where a record is found in one pull only and the other pull gives its DOI to a different record, it is marked conflict. It may have been re-indexed under a new identifier or may be a distinct paper sharing the DOI, and only the records themselves can tell. The bundled records carry no identifier, so here they are matched on their DOIs, and the records without one cannot be compared. The table below leaves out the scopus_id and title columns to fit the page.

records_changed = sf.diff_records(baseline, later)
out(records_changed[["status", "doi", "year", "citations_old", "citations_new"]].head())
status doi year citations_old citations_new
added 10.1002/adfm.202315137 2024 <NA> 32
added 10.1002/asia.202400548 2024 <NA> 6
added 10.1002/ente.202201120 2022 <NA> 3
added 10.1002/slct.202302535 2024 <NA> 2
added 10.1002/smll.202301533 2023 <NA> 21

The counts agree with the DOI comparison above, and the records left out are counted in attrs["n_old_without_id_or_doi"] and attrs["n_new_without_id_or_doi"]. The citation columns show how far each record's count moved between the two pulls. They are equal here, because both pulls were cut from one snapshot.

out(records_changed["status"].value_counts())
status count
unchanged 83
added 43
removed 1

Merging without duplicates

To carry a cumulative set forward across pulls, scopus_combine binds the harvests, renumbers entry_number across the result and, with dedupe=True, keeps one copy of each record. Records with the same scopus_id are one record, and a record without one joins the identified record that carries its DOI, compared without a resolver prefix and ignoring case. These records carry no identifier, so they are matched on the DOI alone.

combined = sf.scopus_combine(baseline, later, dedupe=True)
out((len(combined), len(sf.extract_dois(combined))))
(148, 127)

That takes 231 concatenated rows down to 148: the 127 distinct DOIs, plus the 21 records carrying none, which cannot be matched this way and are kept, where collapsing them into one another would be a silent guess.

The merge records itself, which matters because the count exists only while it happens: afterwards nothing in the result says how many rows went in. PRISMA-S asks for exactly that figure, and scopus_search_report reads it back from here, so the item is answered. The record also names the rule the records were matched by and counts the DOIs that records with different identifiers share. Such records are both kept, since Scopus gives one DOI to distinct papers as well as holding duplicate records of one paper. The search record gives the count, so that they can be checked by hand, and dedupe="id_or_doi" merges them once they have been.

out(combined.attrs["combined"])
{'n_in': 231, 'n_out': 148, 'n_removed': 83, 'deduplicated': True, 'rule': 'id_or_shared_doi', 'doi_conflicts': 0}

Keeping a record of each pull

Comparing against a past harvest only works if you kept it, so it is worth saving each pull as you go. write_records saves a record set and read_records reads it back, choosing the format from the file extension. A .parquet file keeps the provenance a harvest carries in its attrs: the plan, the time it was retrieved, the per-cell totals and the software version. A baseline read back from parquet therefore still writes its own search record with scopus_search_report. Parquet needs pyarrow, which pip install "scopusflow[parquet]" installs.

sf.write_records(baseline, "baseline.parquet")
restored = sf.read_records("baseline.parquet")
sf.diff_dois(old=restored, new=later)

A .csv path gives a plain-text artefact to commit alongside the analysis. It keeps the columns and loses the provenance, since a CSV file is a table and holds nothing else. read_records reads it back with the record schema imposed, so scopus_id stays text and year and citations stay whole numbers. Read with pandas.read_csv alone, an all-digit identifier becomes a number, or a float once one identifier is missing, and a de-duplicating merge with a fresh harvest then matches none of its records.

In a live setting

Everything above runs offline because both harvests were cut from the bundled corpus. In a live setting the later harvest comes from the API, and that call needs a configured Scopus API key, which pybliometrics reads from its own configuration once pybliometrics.init() has been called in the session. The shape of the comparison does not change. You re-run the same SearchPlan through fetch_plan, read back the harvest you saved earlier, and diff the two. The earlier harvest is one of your own, since the bundled corpus came from OpenAlex and would differ from any Scopus harvest for that reason alone. Both harvests carry a Scopus identifier on every record, so diff_records can compare them all.

import pybliometrics

import scopusflow as sf

pybliometrics.init()   # once per session, before the first search

q = sf.scopus_query('"graphene supercapacitor"', field="TITLE-ABS-KEY")
plan = sf.SearchPlan(q, years=range(2015, 2025), partition="year")

later = sf.fetch_plan(plan, cache_dir="graphene-harvest-2")
sf.write_records(later, "later.parquet")
baseline = sf.read_records("baseline.parquet")
sf.diff_dois(old=baseline, new=later)
sf.diff_records(baseline, later)

That second pull asks the API afresh and spends quota, since scopusflow sends refresh=True to pybliometrics unless you pass refresh yourself. Without it, pybliometrics' own response cache, which is keyed on the query string and view alone, would hand back the earlier pull dated now, and the diff would compare a harvest with itself. Search plans and quota-aware retrieval explains the two caches.

Run that on a schedule against a saved baseline and the added and removed rows tell you, harvest after harvest, precisely how the literature is shifting under your search.