Author keywords and references¶
Keyword co-occurrence and citation-network analysis both need something the Search API alone does not return: a document's author-supplied keywords, and its own reference list. This walks through retrieving both, what each costs, and what your Scopus entitlement needs to cover. The live-API examples are shown but not run, since they need a configured key. The shapes they return are put together offline from the corpus bundled with the package, 138 real articles on graphene supercapacitors that stand in for a retrieval because Scopus records may not be redistributed.
Author keywords from a search¶
The Search API's COMPLETE view carries an authkeywords field alongside the usual title, DOI and date. Requesting it costs no extra request beyond COMPLETE's own smaller page size (25 records per page, against 200 for STANDARD), which already means more requests, and so more quota, for the same number of records.
import pybliometrics
import scopusflow as sf
pybliometrics.init() # once per session, before the first search
plan = sf.SearchPlan("DOI(10.1038/nature14539)", view="COMPLETE")
recs = sf.fetch_plan(plan)
recs["authkeywords"]
In development, this field came back None even on a live, otherwise fully-entitled key, for documents that do carry author keywords in Scopus itself. Those documents do have keywords, so what this points to is an entitlement gap specific to this one field. If your own keywords come back all empty, it is worth raising with your Scopus/Elsevier account contact.
References via Abstract Retrieval¶
The reference list is not available from Search under any view. It needs Abstract Retrieval's FULL or REF view, an entitlement separate from ordinary abstract access and from Search access. This is a per-document endpoint, so retrieving references for n documents under FULL costs n requests against Abstract Retrieval's own, smaller weekly quota, separate from Search's.
ab = sf.scopus_abstract(
"10.1038/nature14539",
view="FULL", include=("references", "keywords"),
)
ab.loc[0, "references"][["title", "authors", "sourcetitle", "publicationyear"]]
Author keywords come only with FULL, so include=("keywords",) under any other view raises ValueError before a request. view="FULL" is the recommended default, since it returns the whole bibliography in one request. view="REF" serves it about 40 references at a time. pybliometrics pages REF itself, following the view's startref parameter until the list is complete, so a document with N references costs ceil(N / 40) requests under REF. n_requests still counts each document once, however many pages pybliometrics requested for it. scopus_abstract() warns when the number of references returned does not match the document's own reported count, so a partial list never arrives unannounced.
The shape it returns is one DataFrame per document with pybliometrics' own native reference fields. To show that shape offline, the frame below re-labels three bundled records into those fields, as though they were works cited by a fourth. The id column is a Scopus identifier in a live result, which the bundled records do not carry, and citedbycount comes back empty often enough that it is left so here.
cited = sf.example_records().head(3)
references = pd.DataFrame({
"position": [str(i) for i in range(1, len(cited) + 1)],
"id": pd.NA,
"doi": cited["doi"].to_numpy(),
"title": cited["title"].to_numpy(),
"authors": cited["authors"].to_numpy(),
"sourcetitle": cited["publication"].to_numpy(),
"publicationyear": cited["year"].astype(str).to_numpy(),
"citedbycount": pd.NA,
})
out(references[["title", "authors", "sourcetitle", "publicationyear"]])
| title | authors | sourcetitle | publicationyear |
|---|---|---|---|
| Enhanced Capacitive Properties of All-solid-state Symmetric Graphene Supercapacitors by Incorporating Nitrogen-doping and SnO2 Nanoparticles | Jianhua Yu | Journal of Inorganic Materials | 2015 |
| Fabrication and Characterization of a Vertically-Oriented Graphene Supercapacitor | Patrick R Rice | DigitalCommons - CalPoly (California State Polytechnic University) | 2015 |
| Flexible and Stackable Laser-Induced Graphene Supercapacitors | Zhiwei Peng | ACS Applied Materials & Interfaces | 2015 |
ab.attrs["n_requests"] # requests spent so far
ab.attrs["quota"]["remaining"]
A key or subscription tier that does not cover the requested view raises scopusflow.ScopusFlowForbiddenError naming the view, where a generic HTTP failure would leave you guessing. It stops the whole batch too, since entitlement is a property of the account, so the same refusal would only recur on every remaining identifier.
A spent weekly quota stops the requests as well, but keeps what the batch holds. pybliometrics raises Scopus429Error once every configured key has been refused, and scopus_abstract() then returns the rows already retrieved, gives each identifier not retrieved a row of NA and warns once with scopusflow.ScopusFlowQuotaWarning. The NA rows are not checkpointed, so with cache_dir set a run after the reset requests only the identifiers still missing.
For more than a handful of identifiers, pass cache_dir so an interrupted or quota-limited batch resumes without re-spending quota already spent. Relying on pybliometrics' own on-disk response cache (its refresh parameter, keyed by identifier and view under its configured cache directory) already avoids repeat network calls for the same identifier across script runs. What cache_dir adds here is batch-level progress and resumability across many identifiers, a separate concern. A row read from either cache sent no request, so n_requests leaves it out, and its citation count dates from when the cache was written. Pass refresh=True to fetch every identifier afresh.
The identifiers are checked before the first request. A missing or blank one raises ValueError naming its position, so drop those records first, or let corpus() drop them. Surrounding whitespace is trimmed and a SCOPUS_ID: prefix is dropped under by="scopus_id".
dois = sf.extract_dois(recs)
ab = sf.scopus_abstract(
dois, view="FULL", include=("references", "keywords"),
cache_dir="abstract-cache",
)
Index keywords¶
Scopus records a second kind of keyword. Authors choose their own keywords, whereas Scopus chooses the index keywords and standardises them to vocabularies derived from thesauri Elsevier owns or licenses, with synonyms, spellings and plurals brought together. include=("index_terms",) adds them as an idxterms column, read from pybliometrics' idxterms and joined with "; " as the author keywords are. Like the author keywords, they come only with view="FULL", in the same request.
ab = sf.scopus_abstract(
"10.1038/nature14539",
view="FULL", include=("keywords", "index_terms"),
)
ab.loc[0, "idxterms"]
Elsevier's guide lists index keywords in the FULL view as not available by default, so whether they arrive depends on your key's entitlement. In this package's development testing, they did arrive on the same key whose author keywords came back empty. Where that happens, the index keywords may be the only keyword signal a Scopus harvest can give. The two kinds are not interchangeable, though. A standard vocabulary merges terms that authors phrase differently, so the two give different co-occurrence networks from the same documents, and a study should say which it used.
A minimal, cross-tool corpus¶
corpus combines a search result with this Abstract Retrieval step, returning a minimal shape close to what OpenAlex's works API already returns: id, scopus_id, title, year, keywords (a list of strings per row) and references (a DataFrame of cited works per row), where a bibliometrics export would give you a semicolon-joined citation string. It does not replace to_bibtex/to_ris, which keep their own established interchange formats.
recs = sf.fetch_plan(sf.SearchPlan("DOI(10.1038/nature14539)"))
corpus = sf.corpus(recs, view="FULL")
corpus.loc[0, "keywords"]
len(corpus.loc[0, "references"])
This costs one Abstract Retrieval request per record in recs, on top of whatever retrieved recs in the first place. Under view="REF" only the references are requested, since that view carries no author keywords, and every keywords entry is empty. The keywords column, split from the joined authkeywords field, is a list per row, which is the shape co-occurrence analysis wants. keywords="index" fills it with the index keywords instead, and keywords="both" with the author keywords followed by each index keyword not already among them. Either costs what the default costs, and both need view="FULL".
corpus = sf.corpus(recs, view="FULL", keywords="both")
The match is on the exact text, so "graphene" from the author and "Graphene" from the index both stay.
Counting every unordered pair within each document gives the co-occurrence table the guide opened on, and standard library tools are enough for it.
Author keywords are one of the fields that cannot travel with the package, so the frame below stands them in from each bundled record's own title, keeping a fixed vocabulary of terms and recording those a title mentions. That is a cruder signal than a real keyword list, but the counting is identical and it runs over 138 published titles.
import itertools
from collections import Counter
vocabulary = ["graphene", "supercapacitor", "electrode", "energy storage",
"flexible", "electrochemical"]
title_terms = (
sf.example_records()[["doi", "title", "year"]]
.rename(columns={"doi": "id"})
)
title_terms["keywords"] = [
[term for term in vocabulary if term in title.lower()]
for title in title_terms["title"]
]
out(title_terms.loc[title_terms["keywords"].str.len() > 1, ["title", "keywords"]].head(3))
| title | keywords |
|---|---|
| Enhanced Capacitive Properties of All-solid-state Symmetric Graphene Supercapacitors by Incorporating Nitrogen-doping and SnO2 Nanoparticles | [graphene, supercapacitor] |
| Fabrication and Characterization of a Vertically-Oriented Graphene Supercapacitor | [graphene, supercapacitor] |
| Flexible and Stackable Laser-Induced Graphene Supercapacitors | [graphene, supercapacitor, flexible] |
counts = Counter(
pair
for keywords in title_terms["keywords"]
for pair in itertools.combinations(sorted(keywords), 2)
)
pairs = pd.DataFrame(
[(a, b, n) for (a, b), n in counts.most_common()],
columns=["keyword A", "keyword B", "documents"],
)
out(pairs.head())
| keyword A | keyword B | documents |
|---|---|---|
| graphene | supercapacitor | 105 |
| electrode | supercapacitor | 23 |
| electrode | graphene | 22 |
| flexible | supercapacitor | 18 |
| flexible | graphene | 17 |
Each row is one pair of keywords and the number of documents carrying both, which is the edge list a co-occurrence network is built from.
Handing the corpus to theoryforge¶
theoryforge maps a literature from a corpus file in its own schema, and write_corpus writes one from a corpus() result. The file needs a name for the corpus, since the result carries none.
sf.write_corpus(corpus, "corpus.json", "graphene-supercapacitors")
theoryforge compares cited works as exact strings, so the writer gives each one a single key across the corpus, its Scopus identifier, as scopus:<id>. Scopus gives a cited work's DOI in some bibliographies only, and in different cases. A key built from the DOI wherever there is one would therefore split the work into several nodes and lose its co-citation edges. A reference that carries only a DOI takes the identifier that another reference pairs with the same DOI, and a DOI that nothing pairs with is keyed as doi:<doi>, lower-cased. Keywords are lower-cased as well, so the "graphene" and "Graphene" that keywords="both" keeps apart become one. The block below runs to_theoryforge_corpus, which returns the same document as a dict, on two bundled records, each made to cite the same two others, the second with the DOIs in capitals. Both cite the same two keys.
docs = sf.example_records().dropna(subset=["doi"]).head(4)
cited = list(docs["doi"].iloc[2:4])
def cites(dois):
return pd.DataFrame({"id": [None] * len(dois), "doi": dois})
frame = pd.DataFrame({
"id": list(docs["doi"].iloc[:2]),
"title": list(docs["title"].iloc[:2]),
"year": list(docs["year"].iloc[:2]),
"keywords": [["Graphene", "supercapacitor"], ["graphene"]],
"references": [cites(cited), cites([d.upper() for d in cited])],
})
doc = sf.to_theoryforge_corpus(frame, "graphene-supercapacitors")
out([record["references"] for record in doc["records"]])
[['doi:10.1016/j.electacta.2015.02.019', 'doi:10.1002/smll.201403383'], ['doi:10.1016/j.electacta.2015.02.019', 'doi:10.1002/smll.201403383']]
The file also carries a provenance block recording the search expression, when the search ran and the kind of keyword, with counts of the references dropped and of the records without keywords. An entitlement that returns no author keywords shows up in that last count, and it leaves theoryforge's keyword themes empty whatever the writer does.