Author keywords and references

Keyword co-occurrence and citation-network analysis both need something the Search API alone does not return: a document's author-supplied keywords, and its own reference list. This walks through retrieving both, what each costs, and what your Scopus entitlement needs to cover. The live-API examples are shown but not run, since they need a configured key. The shapes they return are put together offline from the corpus bundled with the package, 138 real articles on graphene supercapacitors that stand in for a retrieval because Scopus records may not be redistributed.

Author keywords from a search

The Search API's COMPLETE view carries an authkeywords field alongside the usual title, DOI and date. Requesting it costs no extra request beyond COMPLETE's own smaller page size (25 records per page, against 200 for STANDARD), which already means more requests, and so more quota, for the same number of records.

import scopusflow as sf

plan = sf.SearchPlan("DOI(10.1038/nature14539)", view="COMPLETE")
recs = sf.fetch_plan(plan)
recs["authkeywords"]

In development, this field came back None even on a live, otherwise fully-entitled key, for documents that do carry author keywords in Scopus itself. Those documents do have keywords, so what this points to is an entitlement gap specific to this one field. If your own keywords come back all empty, it is worth raising with your Scopus/Elsevier account contact.

References via Abstract Retrieval

The reference list is not available from Search under any view. It needs Abstract Retrieval's FULL or REF view, an entitlement separate from ordinary abstract access and from Search access. This is a per-document endpoint, so retrieving references for n documents costs n requests against Abstract Retrieval's own, smaller weekly quota, separate from Search's.

ab = sf.scopus_abstract(
    "10.1038/nature14539",
    view="FULL", include=("references", "keywords"),
)
ab.loc[0, "references"][["title", "authors", "sourcetitle", "publicationyear"]]

view="FULL" is the recommended default. In development, it returned a complete, correctly counted reference list for every document tried, while view="REF" returned an inconsistent, sometimes-truncated subset, on an otherwise identical request made moments apart. scopus_abstract() warns when the number of references returned does not match the document's own reported count, so a partial list never arrives unannounced.

The shape it returns is one DataFrame per document with pybliometrics' own native reference fields. To show that shape offline, the frame below re-labels three bundled records into those fields, as though they were works cited by a fourth. The id column is a Scopus identifier in a live result, which the bundled records do not carry, and citedbycount comes back empty often enough that it is left so here.

cited = sf.example_records().head(3)
references = pd.DataFrame({
    "position": [str(i) for i in range(1, len(cited) + 1)],
    "id": pd.NA,
    "doi": cited["doi"].to_numpy(),
    "title": cited["title"].to_numpy(),
    "authors": cited["authors"].to_numpy(),
    "sourcetitle": cited["publication"].to_numpy(),
    "publicationyear": cited["year"].astype(str).to_numpy(),
    "citedbycount": pd.NA,
})
out(references[["title", "authors", "sourcetitle", "publicationyear"]])

title	authors	sourcetitle	publicationyear
Enhanced Capacitive Properties of All-solid-state Symmetric Graphene Supercapacitors by Incorporating Nitrogen-doping and SnO2 Nanoparticles	Jianhua Yu	Journal of Inorganic Materials	2015
Fabrication and Characterization of a Vertically-Oriented Graphene Supercapacitor	Patrick R Rice	DigitalCommons - CalPoly (California State Polytechnic University)	2015
Flexible and Stackable Laser-Induced Graphene Supercapacitors	Zhiwei Peng	ACS Applied Materials & Interfaces	2015

ab.attrs["n_requests"]   # requests spent so far
ab.attrs["quota"]["remaining"]

A key or subscription tier that does not cover the requested view raises scopusflow.ScopusFlowForbiddenError naming the view, where a generic HTTP failure would leave you guessing. It stops the whole batch too, since entitlement is a property of the account, so the same refusal would only recur on every remaining identifier.

For more than a handful of identifiers, pass cache_dir so an interrupted or quota-limited batch resumes without re-spending quota already spent. Relying on pybliometrics' own on-disk response cache (its refresh parameter, keyed by identifier and view under its configured cache directory) already avoids repeat network calls for the same identifier across script runs. What cache_dir adds here is batch-level progress and resumability across many identifiers, a separate concern.

dois = sf.extract_dois(recs)

ab = sf.scopus_abstract(
    dois, view="FULL", include=("references", "keywords"),
    cache_dir="abstract-cache",
)

A minimal, cross-tool corpus

corpus combines a search result with this Abstract Retrieval step, returning a minimal shape close to what OpenAlex's works API already returns: id, title, year, keywords (a list of strings per row) and references (a DataFrame of cited works per row), where a bibliometrics export would give you a semicolon-joined citation string. It does not replace to_bibtex/to_ris, which keep their own established interchange formats.

recs = sf.fetch_plan(sf.SearchPlan("DOI(10.1038/nature14539)"))
corpus = sf.corpus(recs, view="FULL")
corpus.loc[0, "keywords"]
len(corpus.loc[0, "references"])

This costs one Abstract Retrieval request per record in recs, on top of whatever retrieved recs in the first place. The keywords column, split from the joined authkeywords field, is a list per row, which is the shape co-occurrence analysis wants. Counting every unordered pair within each document gives the co-occurrence table the guide opened on, and standard library tools are enough for it.

Author keywords are one of the fields that cannot travel with the package, so the frame below stands them in from each bundled record's own title, keeping a fixed vocabulary of terms and recording those a title mentions. That is a cruder signal than a real keyword list, but the counting is identical and it runs over 138 published titles.

import itertools
from collections import Counter

vocabulary = ["graphene", "supercapacitor", "electrode", "energy storage",
              "flexible", "electrochemical"]

corpus = (
    sf.example_records()[["doi", "title", "year"]]
    .rename(columns={"doi": "id"})
)
corpus["keywords"] = [
    [term for term in vocabulary if term in title.lower()]
    for title in corpus["title"]
]
out(corpus.loc[corpus["keywords"].str.len() > 1, ["title", "keywords"]].head(3))

title	keywords
Enhanced Capacitive Properties of All-solid-state Symmetric Graphene Supercapacitors by Incorporating Nitrogen-doping and SnO2 Nanoparticles	[graphene, supercapacitor]
Fabrication and Characterization of a Vertically-Oriented Graphene Supercapacitor	[graphene, supercapacitor]
Flexible and Stackable Laser-Induced Graphene Supercapacitors	[graphene, supercapacitor, flexible]

counts = Counter(
    pair
    for keywords in corpus["keywords"]
    for pair in itertools.combinations(sorted(keywords), 2)
)
pairs = pd.DataFrame(
    [(a, b, n) for (a, b), n in counts.most_common()],
    columns=["keyword A", "keyword B", "documents"],
)
out(pairs.head())

keyword A	keyword B	documents
graphene	supercapacitor	105
electrode	supercapacitor	23
electrode	graphene	22
flexible	supercapacitor	18
flexible	graphene	17

Each row is one pair of keywords and the number of documents carrying both, which is the edge list a co-occurrence network is built from.