API reference¶
Every public name in scopusflow is documented here, grouped along the path a
search takes: describe it, size it, run it as a resumable harvest, normalise
what comes back, see what a later retrieval changed, and summarise and plot the
result. The groups are the ones the
R package's reference index
uses, and they come in the same order, so a function documented in both
languages is filed under the same heading on either site, except where the
next paragraph says otherwise.
One group name differs from the R one because the membership differs.
The R group 'Plan and size' also holds the counting function, which is
documented under Retrieve here alongside the other calls that spend quota, so
the first group is called 'Plan and query' instead. Records is wider than its R
namesake, because it absorbs the R groups 'Export and I/O' and 'Data'. The
record writer and reader, the BibTeX and RIS writers and the bundled example
harvest all work on the record table, and the corpus writers on a corpus built
from it. Reporting holds the same one function in both languages. The two
remaining R groups have no counterpart
below. 'App' documents a function that launches the
interface. The Python app is started from the command line as
scopusflow-gui, or from Python with scopusflow.app.launch(), which takes
the host and port to serve on, whether to open a browser tab and whether to
reload on code changes, and
The code-free app covers both. 'Keys' documents a key
check that scopusflow does not need in Python, since pybliometrics holds the
key configuration.
Every name with an entry of its own under a group heading is importable
straight from scopusflow, conventionally as sf. The guides linked from the
home page work the same functions through end-to-end examples.
Plan and query¶
Describe a search before running it, and build field-tagged Boolean queries.
scopus_query ¶
scopus_query(*terms, op='AND', field=None)
Combine terms into one Scopus query, optionally field-wrapping each.
Each term is trimmed and, given field, wrapped in that tag, whose
brackets hold the term together. Without a field tag, a term of several
words is put in brackets before two or more terms are joined, because
Scopus joins the words of an unquoted term with AND. Joined bare,
machine learning OR deep learning would read as
machine AND (learning OR deep) AND learning.
scopus_query("machine learning", "deep learning", op="OR") therefore
gives (machine learning) OR (deep learning). A term that is already one
operand is left as written: a quoted "..." phrase, a braced {...}
phrase, a bracketed group or a single field-tagged group such as
TITLE(deep learning). Single words are never bracketed, so
scopus_query("CRISPR", "Cas9", op="OR") stays CRISPR OR Cas9, and a
query composed by an earlier call keeps its meaning when it is joined again.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*terms
|
str
|
One or more non-empty search terms. |
()
|
op
|
str
|
The boolean operator joining the terms: |
'AND'
|
field
|
str | None
|
An optional field tag applied to every term (see
|
None
|
Returns:
| Type | Description |
|---|---|
str
|
The combined query. |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an |
Source code in src/scopusflow/query.py
301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 | |
wrap_field ¶
wrap_field(query, field)
Wrap query in a field tag, e.g. TITLE-ABS-KEY(graphene).
field=None returns query unchanged. Otherwise, field is stripped
and upper-cased, and a tag of anything but letters and hyphens raises
ValueError.
A query that already opens with a field tag raises ValueError, because
the API rejects one tag nested inside another, as in
TITLE-ABS-KEY(TITLE(x)).
Source code in src/scopusflow/query.py
177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 | |
FIELD_TAGS
module-attribute
¶
FIELD_TAGS = {'TITLE': 'Words in the document title', 'TITLE-ABS-KEY': 'Title, abstract and keywords', 'TITLE-ABS-KEY-AUTH': 'Title, abstract, keywords and author names', 'ABS': 'Abstract text', 'KEY': 'Indexed and author keywords', 'AUTH': 'Author names', 'AUTHKEY': 'Author-supplied keywords', 'AFFIL': 'Affiliation, any part', 'AFFILORG': 'Affiliation organisation name', 'SRCTITLE': 'Source (publication) title', 'DOI': 'Digital Object Identifier', 'ALL': 'All available fields'}
SearchPlan
dataclass
¶
A fully specified, inspectable description of a Scopus search.
Splitting describing a search from executing it makes a workflow reproducible and lets a large retrieval be partitioned by year, so it can be cached and resumed.
years are the publication years. With partition="year" each
distinct year is a cell of its own, so years=[2015, 2020] searches
those two years alone, as scopus_trend
and compare_topics do. With
partition="none" the single cell sends one inclusive range from the
earliest year to the latest, so years=[2015, 2020] covers every year
from 2015 to 2020, as in
scopus_count and
scopus_intersections.
years must be whole numbers from 1700 to 2200, or None for no year
limit, and they are stored as a sorted tuple without repeats.
page_size is the number of records asked for per request, None
(the default) meaning the largest the view allows: 200 for STANDARD,
25 for COMPLETE. It is stored on the plan, and sent, because the search
record scopus_search_report
writes has to state how the harvest was paged, and a figure taken from
anywhere but the plan would be a guess.
view is the Scopus Search view: "STANDARD" (the default) or
"COMPLETE", which also carries the author list and the author keywords
(see to_records) but in pages of at
most 25 records. partition is "none" (the default) or "year".
Any other view or partition raises ValueError, and so does
partition="year" without years.
field is stripped and upper-cased when the plan is built, and a value
that is not a string of letters and hyphens raises ValueError. The
query is checked offline at the same time: LIMIT-TO() or EXCLUDE() pasted
from the web interface's refinement panel, a field tag around a query that
already opens with one, and unbalanced brackets, quotation marks or braces
raise ValueError, and INDEXTERMS() issues a
scopusflow.query.QuerySyntaxWarning.
Source code in src/scopusflow/plan.py
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 | |
wrapped_query
property
¶
wrapped_query
The query wrapped in the plan's field, or the query itself when
field is None. Every cell sends it, followed by the cell's year
limit when it has one. A query with a top-level OR, AND NOT or
proximity operator is put in brackets before the limit is added, so the
limit applies to all of it.
to_dict ¶
to_dict()
The plan as a plain dict, which
from_dict turns back into it.
This is the form fetch_plan attaches
as attrs["plan"]. A dataclass instance cannot be written as JSON,
and pandas 2.1 and later write a frame's attributes as JSON when it is
saved to parquet, so a harvest carrying the plan itself could not be
saved. schema numbers the layout, and the years are a sorted list.
Examples:
>>> import scopusflow as sf
>>> plan = sf.SearchPlan("graphene", years=[2020, 2019], partition="year")
>>> plan.to_dict()["years"]
[2019, 2020]
>>> sf.SearchPlan.from_dict(plan.to_dict()) == plan
True
Source code in src/scopusflow/plan.py
203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 | |
from_dict
classmethod
¶
from_dict(d)
Rebuild a plan from the dict
to_dict wrote.
The values pass through the same validation as a plan built by hand. A dict recorded under a layout this version does not know is refused, since reading it as this one could rebuild a different search.
Source code in src/scopusflow/plan.py
232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 | |
cells ¶
cells()
Expand the plan into the cells that will be fetched.
Source code in src/scopusflow/plan.py
269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 | |
PlanCell
dataclass
¶
One unit of work in a SearchPlan, as
SearchPlan.cells lists them.
cell numbers it from 1. query is the plan's
wrapped_query, without the
year limit. date is the year limit as text ("2015" or
"2015-2020", or None for none), which
fetch_plan appends to the query as a
PUBYEAR clause. year is the cell's year under partition="year"
and None under partition="none". view and page_size are the
plan's.
Source code in src/scopusflow/plan.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
SearchPlan, scopus_count, scopus_trend, compare_topics and
scopus_intersections check a query before any request. For a query that uses
INDEXTERMS(), which pybliometrics documents as not working through the Search
API, they issue a QuerySyntaxWarning, a UserWarning subclass reached
through its module as scopusflow.query.QuerySyntaxWarning.
Retrieve¶
Size a search cheaply first, then execute a plan as a resumable, checkpointed harvest, and pull fuller records.
scopus_count ¶
scopus_count(query, years=None, field=None, view='STANDARD', **kwargs)
Return how many records the (optionally year-filtered) query matches.
A single cheap request that does not download the records, so it is the right way to size a search before committing quota to a harvest.
years are sent as one inclusive range from the earliest year to the
latest, so years=[2015, 2020] counts every year from 2015 to 2020.
scopus_intersections and
a SearchPlan with partition="none" do
the same, while scopus_trend,
compare_topics and
partition="year" take each listed year.
field wraps the query in a Scopus field tag (see
FIELD_TAGS) before the year limit is
appended, as a plan's field does. Left None, the query is sent as
written. view and any further keyword argument go to pybliometrics'
ScopusSearch.
The request is sent with refresh=True unless you pass refresh.
pybliometrics keys its own response cache on the query string and view
alone. Without refresh=True, it would answer a query an earlier harvest
downloaded with the number of rows the harvest left there, and make no
request. refresh=False, or a number of days, opts in to that cache, and
a count it answers is warned about. The key ignores keyword filters such as
subj, so fold any filter into the query before you opt in.
The query is checked offline first, as
SearchPlan checks it, so an empty query or
one the checks refuse raises ValueError before any request.
Raises ScopusFlowConfigError
before any request when pybliometrics.init() has not been called in the
session.
Source code in src/scopusflow/count.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | |
fetch_plan ¶
fetch_plan(plan, cache_dir=None, resume=True, format='parquet', should_stop=None, *, cursor=None, fields='core', **kwargs)
Run every cell of plan and return one normalised DataFrame.
With cache_dir set, each cell is written to disk as it completes, so an
interrupted or quota-limited run resumes without re-fetching finished cells.
A cache_dir belongs to one plan. Checkpoints are keyed by cell number, and
each is written with a manifest beside it, cell-NNN.json, recording
the query as sent, the view, the page size and the paging mode, with the
cell's row count, reported total, retrieval time and scopusflow version.
On resume a checkpoint is served only when its manifest's query, view,
page size and paging match the cell's, an empty checkpoint included. The
query and view its rows record must match as well, and it must hold as
many rows as the manifest records, so a checkpoint replaced while its
manifest was left in place is not served under the old manifest. One
written by a different plan is warned about and refetched, never silently
returned. Point each plan at its own directory all the same, since that
refetch spends quota. A checkpoint without a manifest, as earlier versions
wrote them, is judged by the query and view its rows record, and an empty
one, whose rows record nothing, is refetched at one request each, with a
single warning for the harvest. A manifest without its checkpoint is
ignored. A checkpoint that cannot be read back is likewise treated as a
miss, warned about and refetched, and the harvest carries on. A
cache_dir that is not a directory and cannot be created as one, or in
which no file can be created, raises ValueError before any request. A
checkpoint or manifest that cannot be written all the same, for example on
a full disk, is warned about, and the cell's records are kept in the
result without a checkpoint. resume (True by default) serves a cell
from its checkpoint in cache_dir when that checkpoint answers the cell,
as described above. resume=False reads no checkpoint: every cell is
fetched again, and the checkpoint and manifest of each cell fetched are
overwritten. Without cache_dir, resume has no effect.
format selects the checkpoint format ("parquet" or "csv"); parquet
silently falls back to CSV when no parquet engine is installed. Pass a
zero-argument should_stop callable to allow co-operative cancellation: it
is checked before each cell and the harvest stops (returning what it has) when
it returns True. A harvest stopped that way records
attrs["stopped_early"] = True, with the number of cells the plan holds as
attrs["cells_planned"] and the number that ran as attrs["cells_run"].
Its total_results is None, since the cells it never reached were
never counted.
scopus_search_report then says
how many cells ran, which years they covered and that the harvest is
incomplete. A harvest that ran every cell carries none of the three.
Per-cell progress is emitted on the "scopusflow" logger.
cursor chooses how each cell is paged. Left unset, or True, it
pages by cursor, which retrieves a cell of any size. False pages by
offset, the route pybliometrics documents for keys without a Scopus
subscription. Offset paging cannot retrieve a cell that matches more than
5000 records. pybliometrics refuses such a cell after one request, and the
ScopusQueryError it raises is raised again naming the cell, so the
search has to be partitioned more finely. pybliometrics' own
subscriber keyword is taken as the same choice, and passing both with
different values raises ValueError before any request. The manifest
records the mode, so a checkpoint fetched under one mode is warned about
and refetched when the other is asked for. A checkpoint without a manifest
counts as paged by cursor, since every earlier version paged that way.
attrs["paging"] is "cursor" or "offset" accordingly. The R
twin's scopus_fetch_plan() pages by offset unless given
cursor = TRUE.
fields is "core" by default. "extended" also keeps the document
type, source type, volume, issue, pages, ISSN, PubMed identifier and
open-access status the Search API sends with every record, and the abstract
under a COMPLETE plan (see
to_records). They arrive in the
responses each cell already pays for, so they cost no extra quota. Like
cursor, it is an argument of the harvest, since it changes how the
records are read while the search returns the same records. Each manifest
records the fields its checkpoint was fetched with. A checkpoint that lacks
a column an extended harvest asks for is warned about and refetched, one
written by an earlier version included. An extended checkpoint serves a
core harvest with the extended columns dropped. Any value other than
"core" or "extended" raises ValueError before any request.
Each cell's row count is compared against the total the API reports for
that cell's query, and a shortfall is warned about, since a truncated or
failed download otherwise arrives as a merely small result. The per-cell
accounting is attached as result.attrs["cell_totals"], a list of dicts
with the keys cell, date, n_records and reported_total
(pandas.DataFrame(result.attrs["cell_totals"]) tabulates it), and their sum as
result.attrs["total_results"], the attribute the R twin's
scopus_fetch() also attaches. The sum is None unless every cell
reported a total, since a partial sum would understate the search while
looking like a real figure. A cell resumed from a checkpoint reports the
total its manifest recorded, and one resumed from a checkpoint without a
manifest reports none.
Further keyword arguments go to pybliometrics' ScopusSearch for every
cell, refresh among them (see below). subscriber is the exception:
it is read as the paging choice, as described under cursor. A count
among them replaces the plan's page_size as the number of records asked
for per request, and is the page size each manifest records.
Every cell is fetched with refresh=True unless you pass refresh
yourself, so a run spends quota on each cell it does not resume from a
checkpoint. pybliometrics would otherwise answer from its own response
cache, keyed on the query string and view alone. A re-run into a fresh
cache_dir would then receive the earlier pull, dated now, with the
cached row count standing in for the API's total. refresh=False, or a
number of days, opts in to that cache, and a cell it serves is dated by its
cache file, reports no total and is warned about. The cache key ignores
keyword arguments such as subj or date, so a filtered search would
be served the unfiltered one's records. Fold any filter into the query
before you opt in.
The harvest also carries its provenance: the originating plan, as the
dict SearchPlan.to_dict writes
(SearchPlan.from_dict rebuilds
the plan), retrieved_at and retrieved_until, as ISO 8601 text in
UTC to the second ("2026-07-22T09:15:00+00:00"), scopusflow_version
and paging. These are what
scopus_search_report reads
back. They are combined over the cells as the R twin combines them.
retrieved_at is the earliest cell's time, since a set is only as fresh
as its oldest cell. retrieved_until is the latest, the last search,
which the record of a harvest resumed over several days gives for PRISMA-S
item 13. The version is a sorted list when cells resumed from an earlier
release's checkpoints make more than one. A resumed cell contributes the
time and version its manifest recorded. Each attribute is omitted, never
approximated, when any cell cannot supply it, as a cell resumed from a
checkpoint without a manifest cannot, since dating the whole from the cells
that can be dated would date it later than part of what it holds.
Every attribute is a value json.dumps writes, so the harvest can be
saved with DataFrame.to_parquet and passed through merge,
groupby(...).apply and concat.
write_records saves it with its provenance
whatever the pandas version.
attrs["view"] records the plan's view, which decides what authors
holds: the first author under STANDARD and the author list under
COMPLETE (see to_records), and
top reads it back. A cell resumed from a
checkpoint written before the authors were joined with "; " has them
joined as it is served, so it costs no request.
When plan.view == "COMPLETE", the output gains an authkeywords
column (see to_records) at no extra
request cost beyond COMPLETE's own smaller page size, which already
means more requests, and so more quota, for the same number of records. A
plan with view="STANDARD" (the default) never carries this column, and
a checkpoint written under the other view is refetched like any other
different-plan checkpoint. Resuming a cache written before this column
existed is safe in the COMPLETE direction: pandas.concat fills the
older cells' missing column with NA rather than erroring.
Raises ScopusFlowConfigError
before any request when pybliometrics.init() has not been called in the
session. A cell served from its checkpoint sends no request, so a harvest
resumed wholly from checkpoints runs without init().
Source code in src/scopusflow/fetch.py
445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 | |
scopus_abstract ¶
scopus_abstract(ids, by='doi', view='META_ABS', include=(), cache_dir=None, resume=True, **kwargs)
Retrieve abstracts for one or many ids, resilient per id.
ids is one identifier, or a list, tuple or pandas Series of them.
They are checked before any request, as in the R twin. A missing or blank
identifier raises ValueError naming its position, counting from 0.
corpus drops such records itself. Surrounding
whitespace is trimmed, the SCOPUS_ID: prefix is dropped when
by="scopus_id", and whole numbers are sent as text. Any other value,
a float or bytes among them, raises TypeError.
by selects the lookup type ("doi", "eid" or "scopus_id"). Any id that
fails is warned about and yields an all-NA row that still records the id.
view defaults to "META_ABS" (pybliometrics' own default), which
carries the abstract text; the lighter "META" omits it and leaves the
abstract column empty. Left empty, include adds no column, so the
frame holds the ABSTRACT_COLUMNS
alone.
include names extra fields to retrieve in the same request: any of
"references", "keywords" (the author keywords) and "index_terms" (the
index keywords Scopus assigns). "references" needs Abstract Retrieval's
"FULL" or "REF" view, and "keywords" needs the "FULL" view, since the
other views carry no author keywords. "index_terms" needs "FULL" as well.
Any other combination raises ValueError before a request. Both views
are an entitlement separate from ordinary abstract access and from Scopus
Search access. Per
Elsevier's own documentation, some fields (notably author keywords) may
need to be requested from your Scopus/Elsevier account contact even when
the view itself is otherwise accessible. "FULL" returns the whole bibliography in
one request. "REF" serves it about 40 references per request, and
pybliometrics pages REF itself, following the view's startref
parameter until the list is complete, so a document with N references
costs ceil(N / 40) requests under "REF". "FULL" is therefore the
cheaper view when your entitlement allows it. A mismatch between the
number of references returned and the document's own reported reference
count (refcount) is warned about, since the list may then be
incomplete.
When "keywords" is included, an authkeywords column is added: the
document's author-supplied keywords, joined with "; ", or NA when the
document has none, or when the API omits the field for a given key's
entitlement. In this package's own development testing, against a live,
otherwise fully-entitled key, this field did not populate for documents
that do carry author keywords in Scopus itself, so an all-NA column
is more likely an entitlement gap worth raising with your Scopus/Elsevier
account contact than genuinely absent data. A keyword entry that
pybliometrics cannot parse (its 4.4.1 release raises KeyError on an
entry without text) leaves that document's keywords NA with a
warning, and the rest of its row is kept. From pybliometrics 4.5, an
author keyword without text is dropped and the document's other keywords
are kept.
When "index_terms" is included, an idxterms column is added after any
authkeywords: the document's index keywords, read from
pybliometrics' idxterms and joined with "; ", or NA when the
response carries none. Authors choose their own keywords, whereas Scopus
chooses the index keywords and standardises them to vocabularies derived
from thesauri Elsevier owns or licenses, with synonyms, spellings and
plurals brought together. The two kinds therefore give different keyword
co-occurrence results, and a study should say which it used. Elsevier's
guide lists index keywords in the "FULL" view as not available by
default, so whether they arrive depends on the key's entitlement as well.
In this package's development testing, they did arrive on the same key
whose author keywords came back empty. An index term pybliometrics cannot
parse (its 4.4.1 release raises KeyError on one without text, as 4.5
still does) leaves that document's index keywords NA with a warning,
and the rest of its row is kept. The R twin drops such a term and keeps
the others.
When "references" is included, a references column is added: one
DataFrame per document (one row per cited work), using pybliometrics' own
native field set (position, id, doi, title, authors,
authors_auid, authors_affiliationid, sourcetitle,
publicationyear, coverDate, volume, issue, first,
last, citedbycount, type, text, fulltext). A document
with no resolvable references yields a zero-row DataFrame, so
the column can always be unnested.
cache_dir and resume checkpoint per identifier, the way
fetch_plan checkpoints per cell, pickled
rather than written as parquet/csv, since a row here can carry a nested
DataFrame. Only a successful retrieval is checkpointed: a failed identifier
still yields its warned-about NA row in the returned frame, but is retried
on the next resumed run, never read back as data. Worth setting whenever
include is used: Abstract Retrieval draws on its own weekly quota,
smaller than and separate from Search's, and every identifier costs its own
request, so re-running an interrupted batch without a cache re-spends quota
already spent. resume=False reads no checkpoint and retrieves every
identifier again, overwriting the checkpoint of each identifier retrieved
successfully. Distinct identifiers can share a checkpoint file, as
"10.1/a.b" and "10.1/a-b" do, and resume does not check which
identifier a checkpoint holds, so such a pair should not share a
cache_dir. A cache_dir that cannot be written raises ValueError
before any request, and a checkpoint that cannot be written all the same is
warned about, the row kept in the result, as in
fetch_plan. Relying on pybliometrics' own
on-disk response cache (its refresh parameter, keyed by identifier and
view under its configured cache directory) is enough to avoid repeat
network calls for the same identifier across script runs; this checkpoint
is for batch-level progress and resumability across many identifiers, a
separate concern. Further keyword arguments, refresh among them, go to
pybliometrics' AbstractRetrieval.
The number of Abstract Retrieval requests made, and the most recently
parsed remaining-quota figure (from pybliometrics'
get_key_remaining_quota()), are attached as result.attrs["n_requests"]
and result.attrs["quota"], since this is a materially more expensive
operation than a search call. n_requests counts each document once,
however many REF pages pybliometrics requested for it, so it is a count
of documents fetched. A failed retrieval counts once, since the API has
usually answered it with an error. A row read from a cache_dir
checkpoint sent no request and is not counted. Nor is a row pybliometrics
answered from its own cache, recognised by a cache file older than the
call or by a response with no headers kept. Such a row's citations
and other fields date from when that cache file was written. Pass
refresh=True to fetch every identifier from the API.
A 403 (an entitlement gate, most often on the requested view or field)
raises
ScopusFlowForbiddenError
and stops the batch immediately, naming the view and identifier, where a
generic failure would leave the caller guessing. Repeating the identical
failure for every remaining identifier would serve nobody: entitlement is a
property of the account, so a retry cannot succeed.
A spent weekly quota stops the requests too, but returns what the batch
holds, since it usually arrives part-way through. pybliometrics raises
Scopus429Error once every configured key has been refused. The rows
already retrieved are kept, each identifier not retrieved gets an NA row
that is not checkpointed, and one
ScopusFlowQuotaWarning
names the identifier reached and how many were not retrieved. A checkpoint
in cache_dir is still read after the stop.
Raises ScopusFlowConfigError
before any request when pybliometrics.init() has not been called in the
session. An identifier served from its checkpoint sends no request, so a
batch resumed wholly from checkpoints runs without init().
Source code in src/scopusflow/abstract.py
346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 | |
ABSTRACT_COLUMNS
module-attribute
¶
ABSTRACT_COLUMNS = ['scopus_id', 'doi', 'title', 'abstract', 'publication', 'date', 'year', 'citations']
corpus ¶
corpus(records, by='doi', view='FULL', cache_dir=None, resume=True, keywords='author', **kwargs)
Enrich records (from fetch_plan)
with keywords (the author keywords by default, or the index keywords Scopus
assigns) and structured references via Abstract Retrieval, returning a
minimal, uniform shape close to what OpenAlex's works API already
returns: id, scopus_id, title, year, keywords (a list
of strings per row) and references (a DataFrame of cited works per
row). write_corpus writes the result in
theoryforge's corpus schema. This is meant for downstream tools that want
to consume Scopus output without writing their own parsing layer, for
example for keyword co-occurrence or citation-network analysis. It does not
replace to_bibtex/
to_ris, which keep their own established
interchange formats.
by selects which column of records ("doi" or "scopus_id") to look
identifiers up by. view is passed to
scopus_abstract and defaults to
"FULL", which returns a document's whole bibliography in one request. "REF"
serves it about 40 references per request, and pybliometrics pages REF
itself, so the list still arrives whole, at a cost of ceil(N / 40)
requests for N references (see
scopus_abstract's documentation
for the entitlement each view needs). cache_dir and resume are
passed through unchanged, and are worth setting for anything beyond a
handful of records, since this performs at least one Abstract Retrieval
request per record, against its own, smaller weekly quota, separate from
Search's. What that cost came to is carried through from
scopus_abstract: the number of
requests made, counting each document once, and the most recently parsed
remaining-quota figure are attached as result.attrs["n_requests"] and
result.attrs["quota"]. Three more attrs carry what
write_corpus records of the corpus's
provenance. query is a list of the field-wrapped search expression of
each cell of the records' plan, or, for records without a plan, of the
distinct values of their own query column. retrieved_at is when the
search that found the records ran, as the records hold it, and keywords
is the kind of keyword chosen. query and retrieved_at are None
when records carries neither.
scopus_id is the document's own Scopus identifier, as Abstract
Retrieval gives it, or the scopus_id of records where Abstract
Retrieval gives none, as for a failed lookup, and missing when neither
has one.
A record whose identifier is missing (NA/None) or blank is
dropped, with a warning naming how many.
The keywords column here is list[str] per row, split out of
scopus_abstract's joined
authkeywords and idxterms strings, empty when the document has none or
the field is unavailable. Only the "FULL" view carries author keywords, so
under "REF" the references alone are requested and every keywords entry
is empty. references carries pybliometrics' own native reference field
set (see scopus_abstract's
documentation).
keywords chooses which keywords fill that column: "author" (the
default) for the author keywords, "index" for the index keywords Scopus
assigns, or "both" for the author keywords followed by each index keyword
not already among them. The match is on the exact text, so a term that
differs only in case appears twice. "index" and "both" need
view="FULL", since the "REF" response carries no keywords of either
kind, and raise ValueError under "REF" before any request. Index
keywords and author keywords give different co-occurrence results, and
scopus_abstract describes how each
is chosen and the entitlement each depends on. Either choice costs the same
request per record.
Source code in src/scopusflow/corpus.py
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 | |
ScopusFlowForbiddenError ¶
Bases: Exception
Raised when the Scopus API refuses a request (HTTP 403), most often because the configured key's entitlement does not cover the requested Abstract Retrieval view or field.
Source code in src/scopusflow/exceptions.py
25 26 27 28 | |
ScopusFlowConfigError ¶
Bases: RuntimeError
Raised before any request when pybliometrics has not been initialised in this Python session.
pybliometrics 4 reads its configuration only when pybliometrics.init()
is called, so a configuration file on disk is not enough by itself. Run
import pybliometrics; pybliometrics.init() once per session before the
first search. It reads the configuration file, or creates it and asks for
your key when there is none. scopusflow never calls init() itself.
Source code in src/scopusflow/exceptions.py
31 32 33 34 35 36 37 38 39 40 | |
ScopusFlowQuotaWarning ¶
Bases: UserWarning
Warned once when
scopus_abstract stops a batch
because the API key's weekly quota is spent.
pybliometrics raises Scopus429Error only after it has tried every
configured key, and documents the error as a depleted quota, so no later
identifier in the batch could succeed. The rows already retrieved are
returned, and each identifier not retrieved gets a row of NA. It is the
counterpart of the R twin's scopus_warning_quota_exceeded.
Source code in src/scopusflow/exceptions.py
43 44 45 46 47 48 49 50 51 52 53 | |
Records¶
Normalise results into one stable schema, save a record set and read it back, export to reference-manager formats, and write a corpus in theoryforge's schema.
to_records ¶
to_records(results, query=None, view=None, fields='core')
Normalise a pybliometrics ScopusSearch().results list (named tuples)
or a list of dicts into a tidy
RECORD_COLUMNS DataFrame.
Whatever the query type, the columns are the same, so the downstream DOI, diff and analysis helpers can rely on them.
query is written into the query column of every row, which holds
None when no query is given. fetch_plan
passes each cell's query as sent, its year limit included. The core columns
are entry_number (the row's position, from 1), scopus_id (the EID
without its 2-s2.0- prefix), doi, title, authors (see
below), year (the leading four digits of the cover date, as a whole
number), date (the cover date as Scopus gives it), publication (the
source title), citations (the cited-by count, as a whole number) and
query. A field the record lacks is a missing value that pandas.isna
recognises (None, NA or NaN, depending on the column's dtype),
and an empty results gives a zero-row frame with the same columns.
What authors holds depends on the view. The Search API's
dc:creator, pybliometrics' creator, is the first author alone under
either view, in the indexed form Scopus uses ("Zhang F."). The author
list arrives only under view="COMPLETE", and pybliometrics builds
author_names from it as "Surname, Given" names. Those names are
joined here with "; ", as the R twin joins them, and an author whose
given name is null or empty is written as the surname alone. A row without
the list falls back to creator. A STANDARD record therefore names
its first author, and top says so once when it
tallies a STANDARD harvest by author. The API lists at most 100 authors
per record. The R twin builds the same string from the same entry in the
common case, and the two part company at two edges. pybliometrics builds
author_names only when every author object carries both a surname
and a given-name key, so where one does not, this twin keeps the first
author while the R twin lists every author. Where a surname is null or
empty, pybliometrics writes the given name after a bare comma, and the R
twin uses the indexed name.
When view="COMPLETE", an authkeywords column is added: the author-
supplied keywords the Scopus Search API returns under that view, as a
single string in Scopus's own " | "-delimited form (missing when the
document has none, or when the API omits the field for a given key's
entitlement: this was observed directly against a live, otherwise
fully-entitled key during development, on documents that do carry author
keywords in Scopus itself, so a column missing throughout is more likely an
entitlement gap worth raising with your Scopus/Elsevier account contact
than genuinely absent data). Under any other view, including the
default None, no authkeywords column is added.
fields is "core" by default, which builds the columns above and no
others. "extended" also keeps fields the Search API sends with every
record, at no extra quota, which pybliometrics already parses: the
EXTENDED_COLUMNS and, under
view="COMPLETE", an abstract column. Elsevier's Search views table
lists every one of them under both views except the abstract
(dc:description), which only COMPLETE carries. A COMPLETE
harvest therefore returns up to 25 abstracts with each request, while
scopus_abstract spends one request
per record from the separate Abstract Retrieval quota.
doc_type and doc_type_description hold the document-type code and
name Scopus gives the record, such as "ar" and "Article". The code
is the one the DOCTYPE() search field takes, and Elsevier's search tips
list the codes with their names: ar for an article, re a review,
cp a conference paper, er an erratum and others. A retracted item
carries tb, which that list leaves out. The Scopus Content Coverage
Guide (Elsevier, 2023, section 3.1) defines an erratum as a report of an
error, correction or retraction of a previously published paper, so er
covers retraction notices as well as corrections. The guide gives an
article the type Retracted once a retraction notice for it is published.
These codes are the labels Scopus assigns, and they do not screen a set for
retractions, since Scopus does not label every retracted item (Ortega and
Delgado-Quirós, 2024). Check a set against the Retraction Watch database,
which Crossref makes openly available, before relying on it. Errata can
also be left out of the search itself with AND NOT DOCTYPE(er), joined
by scopus_query, which brackets the rest
of the query first, since Scopus applies AND NOT last.
source_type holds the source type, such as "Journal". volume,
issue, pages and pubmed_id are text, and so is issn, the
print ISSN as Scopus writes it, without the hyphen, so its leading zeros
survive. open_access holds the code Scopus sends in its openaccess
field, as a whole number. Elsevier's Scopus APIs Getting Started Guide
(Elsevier, 2023, section 17) illustrates its values with 0 for a record
that is not open access, 1 for gold and 2 for green. The column therefore
holds a status code, which can take values other than 0 and 1. A
fields value other than "core" or "extended" raises
ValueError.
References
Elsevier. (2023). Scopus APIs getting started guide (Version 1). https://dev.elsevier.com/guides/Scopus%20API%20Guide_V1_20230907.pdf
Elsevier. (2023). Scopus content coverage guide. https://supportcontent.elsevier.com/Support%20Hub/Scopus/Scopus_ContentCoverage_Guide_WEB.pdf
Ortega, J. L., & Delgado-Quirós, L. (2024). The indexation of retracted literature in seven principal scholarly databases: A coverage comparison of Dimensions, OpenAlex, PubMed, Scilit, Scopus, The Lens and Web of Science. Scientometrics, 129(7), 3769–3785. https://doi.org/10.1007/s11192-024-05034-y
Source code in src/scopusflow/records.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 | |
RECORD_COLUMNS
module-attribute
¶
RECORD_COLUMNS = ['entry_number', 'scopus_id', 'doi', 'title', 'authors', 'year', 'date', 'publication', 'citations', 'query']
EXTENDED_COLUMNS
module-attribute
¶
EXTENDED_COLUMNS = ['doc_type', 'doc_type_description', 'source_type', 'volume', 'issue', 'pages', 'issn', 'pubmed_id', 'open_access']
scopus_combine ¶
scopus_combine(*sets, dedupe=False)
Bind several record frames into one, renumbering entry_number.
This is the safe way to merge separate harvests: a plain concat leaves
duplicate entry numbers, and nothing then records how many records went in.
De-duplication compares DOIs as
extract_dois does, without resolver
prefixes such as https://doi.org/ or doi: and ignoring case. It
keeps the first record of each group, in the order the sets were given, and
a record with neither an identifier nor a DOI cannot be matched and is
kept.
Under the default rule, records with the same Scopus identifier are one
record, and a record without an identifier joins the identified record
that carries its DOI. Records with different identifiers are never merged,
even when they share a DOI. Scopus holds duplicate records of one paper
(Valderrama-Zurián et al., 2015), but it also gives one DOI to distinct
papers (Franceschini et al., 2015), and a review loses a wrongly removed
record for good where a surviving duplicate is caught at screening. Such
DOIs are counted in doi_conflicts, which the search record reports, so
that they can be checked by hand. A record without an identifier whose DOI
two identified records carry is kept once, apart from both. Records without
an identifier that share a DOI no identified record carries are one record.
"id_or_doi" merges every record linked to another by a shared
identifier or a shared DOI, directly or through a chain of such links. It
suits a set whose shared DOIs have been checked and found to be
duplicates. "id_then_doi" keys each record on its identifier, failing
that on its DOI lower-cased but not otherwise cleaned. That is the rule the
R twin's scopus_combine() applied in its 0.4.0 release and earlier, so
it reproduces a set those releases de-duplicated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*sets
|
Two or more record frames, or a single list of them. |
()
|
|
dedupe
|
bool | str
|
Whether, and by which rule, to keep once the records that are one
record. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The merged records. Attributes describing a single retrieval, among
them |
Raises:
| Type | Description |
|---|---|
ValueError
|
When |
References
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Errors in DOI indexing by bibliometric databases. Scientometrics, 102(3), 2181–2186. https://doi.org/10.1007/s11192-014-1503-4
Valderrama-Zurián, J.-C., Aguilar-Moya, R., Melero-Fuentes, D., & Aleixandre-Benavent, R. (2015). A systematic analysis of duplicate records in Scopus. Journal of Informetrics, 9(3), 570–576. https://doi.org/10.1016/j.joi.2015.05.002
Examples:
>>> import scopusflow as sf
>>> baseline = sf.example_records()
>>> later = sf.example_records()
>>> merged = sf.scopus_combine(baseline, later, dedupe=True)
>>> merged.attrs["combined"]["n_in"]
276
>>> len(merged)
149
Source code in src/scopusflow/combine.py
182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 | |
write_records ¶
write_records(records, path)
Save a record set, choosing the format from the file extension.
A .parquet file keeps the provenance in records.attrs: the plan,
the retrieval time, the per-cell totals, the paging mode, the software
version and any merge record, all of which
scopus_search_report reads
back. They are stored as a JSON block under the scopusflow key of the
file's schema metadata, so they survive whatever the pandas version. pandas
1.5 to 2.0 drop attrs when writing parquet, and later versions keep
them under a key of their own. Attributes set by hand as a SearchPlan,
a datetime or a DataFrame are written in the JSON forms
fetch_plan uses, and the frame passed in
is left as it was. Parquet needs pyarrow, which the parquet extra
installs.
A .csv file is portable plain text, written with LF line endings on
every platform, and holds the columns only. Save a baseline you will
compare against later as parquet, so that it keeps the record of how it
was retrieved.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
DataFrame
|
A record frame, such as |
required |
path
|
The file to write, ending in |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
|
Examples:
>>> import tempfile, pathlib
>>> import scopusflow as sf
>>> path = pathlib.Path(tempfile.mkdtemp()) / "baseline.csv"
>>> _ = sf.write_records(sf.example_records(), path)
>>> len(sf.read_records(path))
138
Source code in src/scopusflow/io.py
66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
read_records ¶
read_records(path)
Read a record set saved by
write_records.
Both formats come back with the record schema imposed: the text columns,
scopus_id among them, as strings, and year and citations as
nullable integers. A CSV left to pandas' own inference reads an all-digit
identifier as a number, or as a float once one identifier is missing, and
a de-duplicating scopus_combine then
matches none of its records against a fresh harvest. The columns
fields="extended" adds are read the same way, open_access as a
nullable integer and the rest as strings, so an ISSN such as "00368075"
keeps its leading zeros.
A parquet file comes back with the provenance it was saved with in
attrs, so its search record reads as the original's did. A parquet
file written by DataFrame.to_parquet under pandas 2.1 or later
returns the attrs pandas stored. A CSV file carries none.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
The file to read, ending in |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
The records, with any provenance in |
Source code in src/scopusflow/io.py
155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 | |
example_records ¶
example_records()
Return the bundled example harvest as a records frame.
A fresh copy is returned on each call, so a caller may edit the result without disturbing anyone else's.
Returns:
| Type | Description |
|---|---|
DataFrame
|
138 records with the standard |
Examples:
>>> import scopusflow as sf
>>> records = sf.example_records()
>>> len(records)
138
>>> sorted(records.columns) == sorted(sf.RECORD_COLUMNS)
True
Source code in src/scopusflow/data.py
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | |
to_bibtex ¶
to_bibtex(records)
Render records as a BibTeX string, one entry per row, with citation keys made unique within the export.
A set retrieved with fields="extended" (see
to_records) carries each record's Scopus
document type, and the entry is typed by it. A conference paper (cp)
or conference review (cr) becomes an @inproceedings, a book chapter
(ch) an @incollection and a book (bk) a @book. Every other
record, and every record of a set retrieved with the core fields, is an
@article. The source title of a conference paper or chapter goes under
booktitle, braced as the title is, and a journal article's under
journal. A book in a book series gives the series under series, and
a book outside a series, whose source title Scopus gives as the book's own,
names no source. The entries also carry volume, number, pages
(101--117), issn, doi, the author keywords under keywords,
joined by commas, the abstract under abstract and the Scopus identifier
in note (Scopus ID: <scopus_id>). A set retrieved under
view="COMPLETE" carries the keywords, and the abstract only when it was
also retrieved with fields="extended". A table from
scopus_abstract carries its
abstracts, and its keywords under include="keywords". A value the
record lacks is left out, never filled in.
Names are written family name first, with a comma. A STANDARD-view name
such as Zhang F., which BibTeX would read as given name Zhang and
family name F., is written Zhang, F., and run-together initials are
spaced (Kitchin, J. R.), since BibTeX takes J.R. for one given name.
Names that already carry a comma, as under COMPLETE, and names given first
are written as they stand. The authors column and the citation keys are
unchanged. In the title, every word with a capital after its first letter
is braced ({DNA}, {CRISPR-Cas9}), so bibliography styles cannot
lower-case it.
Source code in src/scopusflow/export.py
310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 | |
to_ris ¶
to_ris(records)
Render records as an RIS string, one record per row.
A name ending in initials is written with a comma (Zhang, F.), as for
to_bibtex, but run-together initials keep
Scopus's spelling. The source title goes under T2, the tag Scopus's own
RIS export uses and reference managers read as the publication title. Each
record names the database (DB - Scopus) and, when it has an
identifier, gives the EID as the accession number
(AN - 2-s2.0-<scopus_id>).
Each record is a JOUR, unless a set retrieved with fields="extended"
gives its Scopus document type. A conference paper is then a CPAPER, a
conference review a CONF, a book chapter a CHAP and a book a
BOOK. A book outside a series has no T2, since Zotero reads a
book's T2 as its series, and a book in a book series gives the series
there. The record also carries VL, IS, SP and EP from the
page range, SN, the DOI as DO, the document type as M3, the
abstract as AB, one KW line per author keyword and the Scopus
identifier as N1 - Scopus ID: <scopus_id>. The keywords and the
abstract come from the same sets as for
to_bibtex. Keywords are split at the
" | " the Search API joins them with and at the "; "
scopus_abstract joins its own
with. A value the record lacks is left out, never filled in.
Source code in src/scopusflow/export.py
352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 | |
write_corpus ¶
write_corpus(corpus, path, id)
Write the corpus to path as JSON in theoryforge's corpus schema.
The document is
to_theoryforge_corpus's, which
describes the keys, the keywords and the provenance block, and it is
returned too. The file is UTF-8, with no escaping beyond what JSON
requires, sorted keys, no spaces between tokens and one line feed at the
end, so the same corpus gives the same bytes from either twin.
allow_nan=False stops the write before any file is opened should a
NaN reach the document, since JSON has no such value. path is
written exactly as given, and its parent directory is assumed to exist
already.
Source code in src/scopusflow/corpus.py
369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 | |
to_theoryforge_corpus ¶
to_theoryforge_corpus(corpus, id)
The corpus as a document in theoryforge's corpus schema, as a dict.
Turns a corpus result into the literature
corpus that theoryforge's read_corpus, litmap and landscape
read, following the corpus schema theoryforge publishes
(schema/corpus.schema.json in its repository).
write_corpus writes the same document
to a JSON file. Neither imports theoryforge, and neither makes a request.
theoryforge compares references as exact strings, so each cited work gets
one key across the whole corpus, its Scopus identifier, as
scopus:<id>. Scopus gives a cited work's DOI in some bibliographies
only, and in different cases. A key built from the DOI wherever there is
one would therefore split one work into several nodes and lose
the co-citation edges between them. A reference that carries only a DOI
takes the Scopus identifier that a document or reference
elsewhere in the corpus pairs with the same DOI, the documents' own
identifiers counting first. A DOI that nothing pairs with is keyed as
doi:<doi>, without its resolver prefix and with its ASCII letters
lower-cased, since DOIs are case-insensitive over ASCII (DOI Handbook,
section 2.4). A reference with neither is dropped, and a UserWarning
says how many. Each document is keyed the same way, from its
scopus_id or else from an id that is a DOI or a Scopus
identifier, so a citation of a document in the corpus and the document
itself share one key. A document with neither raises ValueError.
Keywords are trimmed, lower-cased with str.lower() and made unique,
keeping the first of each, and empty ones are dropped, so "Graphene"
and "graphene " count as one keyword. The R twin lower-cases with
tolower(). The two agree on the Latin, Greek and Cyrillic letters in
common use and differ on a few special cases, such as the Turkish capital
I with a dot and a capital sigma ending a word, which Python writes as a
final sigma. A missing year is written as null,
whether it is None, NaN or pd.NA, and a whole-number year of
any dtype as a plain integer. A missing title is left out, since the
schema admits a null year and not a null title.
The document also carries a provenance object, which the schema
admits beside its own fields: source ("Scopus via scopusflow
<version>"), query, retrieved_at (in UTC, as
YYYY-MM-DDTHH:MM:SSZ), keywords, n_records,
n_references_dropped and n_records_without_keywords. query,
retrieved_at and keywords come from the attrs that
corpus attaches, and are None (null)
when the frame lacks them. query is a string, or a list of strings when
the records were searched with several expressions. Author keywords can
come back empty under some entitlements (see
scopus_abstract), which leaves
theoryforge's keyword themes empty whatever the writer does, and
n_records_without_keywords shows when that happened.
id is a non-empty string naming the corpus, written as its top-level
id. It is required, since a corpus result
carries no name of its own. Returns a dict with id, provenance,
records and schema_version ("1.0").
Source code in src/scopusflow/corpus.py
223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 | |
Reporting¶
Write the search up for a methods section, to the PRISMA-S standard, from what the plan and the harvest already record.
scopus_search_report ¶
scopus_search_report(x, plan=None, file=None)
Assemble a reproducible record of a Scopus search.
Turns a harvest, or a plan not yet run, into the search-strategy record a systematic review has to report: what was searched, exactly how, when, how much came back, and how much the API said there was. The record prints as a readable report, formats as a methods paragraph fit to paste into a manuscript, and writes as Markdown. The reporting standard it follows is PRISMA-S (Rethlefsen et al., 2021), together with the identification counts of the PRISMA 2020 flow diagram.
Everything in the record comes from the objects handed to it. The dates of
the search are the retrieved_at and retrieved_until attributes
fetch_plan attaches, never the current
time. A harvest resumed over several days, or a baseline merged with its
update, is dated from its first retrieval to its last. PRISMA-S item 13
then gives the date of the last search, as the checklist asks. Stamps that
fall on one UTC day give that single date. The years are those the cells'
limits sent, and years with gaps between them are listed, since a span
would claim years no cell searched. The number of records the API reported
as matching is the per-cell accounting in cell_totals, or, for a set
from a single search, its total_results, never an inference from the
number of rows; and the duplicates removed are those
scopus_combine recorded removing,
matched by the rule it recorded. Where an attribute is absent, as it is for
a frame read back from CSV, for the bundled corpus, and for a harvest with
a cell resumed from a checkpoint written without a manifest by an earlier
version of scopusflow, the record says the field is unrecorded and fills
nothing in. This matters most for completeness: a harvest whose reported
total is unknown is never described as exhaustive.
A harvest stopped through should_stop, as the app's Cancel stops one,
says so. The record gives how many of the planned cells ran and which
years they covered, marks the rest "not run" and calls the harvest
incomplete, keeping the totals the cells that ran recorded. The R twin's
scopus_fetch_plan() can also cap the records per cell, and its record
then names the cap in the cells it stopped, the completeness line and
item 9. A frame carrying attrs["max_results"] is described the same
way here, so the two records stay one rendering, although
fetch_plan takes no cap and so never
records one.
The PRISMA-S map is decided the same way. Items the package holds evidence for (the database and platform, that Scopus was searched on its own, the full strategy and the limits where a plan records them, the date, the totals, and de-duplication where it was performed) are listed as supplied. Without a plan, the year limit is unrecorded, not absent, and the record says so. The rest, among them peer review of the strategy, grey literature and citation searching, are listed as the author's to supply, because the package has no way to know them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
A records frame, which supplies the counts and the retrieval provenance
and, through |
required | |
plan
|
SearchPlan | None
|
The plan describing |
None
|
file
|
Path at which to write the record as Markdown. A file is written only when this is supplied, and only to the exact path given, so nothing is written to the working directory unless asked. |
None
|
Returns:
| Type | Description |
|---|---|
SearchReport
|
The record. |
References
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. (2021). PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews, 10, 39. https://doi.org/10.1186/s13643-020-01542-z
Examples:
A search described but not yet run. The record says so throughout rather than implying figures it cannot have.
>>> import scopusflow as sf
>>> plan = sf.SearchPlan('"graphene supercapacitor"', years=range(2015, 2025),
... field="TITLE-ABS-KEY", partition="year")
>>> report = sf.scopus_search_report(plan)
>>> report.n_records is None
True
The same search after a harvest. The bundled corpus stands in for one, since Scopus records may not be redistributed, so the attributes a live retrieval records are set here by hand. Their values are illustrative, and the records came from OpenAlex, so the record below describes no search that was run.
>>> from datetime import datetime, timezone
>>> records = sf.example_records()
>>> records.attrs["plan"] = plan
>>> records.attrs["retrieved_at"] = datetime(2026, 7, 22, 9, 15,
... tzinfo=timezone.utc)
>>> records.attrs["scopusflow_version"] = "0.3.0"
>>> report = sf.scopus_search_report(records)
>>> report.n_records
138
>>> "22 July 2026" in report.format(style="paragraph")
True
Source code in src/scopusflow/report.py
185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 | |
SearchReport
dataclass
¶
The fields a search record is built from, and its three renderings.
database and platform are always "Scopus" and
"Elsevier Scopus Search API". Every other attribute but the frames
cells and prisma holds what the objects recorded, or None where
they recorded nothing, which the renderings spell out as unrecorded. In
that case, stopped_early is False, cells_reported 0 and
not_run empty.
cells is a frame of cell, limit, n_records and
reported_total; prisma a frame of item, name, source
("record" or "author") and note. searched_at and
searched_until are the earliest and latest retrieval times.
max_results is the cap on records per cell a set records.
stopped_early, cells_planned and cells_run describe a harvest
stopped before its last cell, not_run lists the cells it never reached
and years_run the years of those it ran. dedupe_rule and
doi_conflicts are the rule
scopus_combine matched records by
and the DOIs it left shared across identifiers.
query, field, view, page_size and partition are the
plan's, and n_cells is the number of its cells. expression lists
the distinct queries the cells send, before their year limits, or, without
a plan, the distinct values of the records' query column. years
names the years the cells' limits cover, as text such as
"2015 to 2024". paging is the harvest's paging mode, "cursor"
or "offset", and version the scopusflow version that retrieved it,
or the comma-separated versions when there were several. n_records
counts the records and n_with_doi those with a DOI. reported_total
is the sum of the totals the API reported, given only when every cell
reported one, and cells_reported is how many cells did.
records_combined, duplicates_removed and deduplicated are the
number of records scopus_combine
received, the number it removed and whether it de-duplicated. snippet
is Python code that rebuilds the plan and runs it through
fetch_plan, or None without a plan.
Source code in src/scopusflow/report.py
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 | |
format ¶
format(style='report')
Render the record: "report" for the readable record print
shows, "paragraph" for the methods paragraph, or "markdown" for
the whole record as Markdown, which is what file writes.
Source code in src/scopusflow/report.py
167 168 169 170 171 172 173 174 175 176 177 | |
PRISMA_S_ITEMS
module-attribute
¶
PRISMA_S_ITEMS = ['Database name', 'Multi-database searching', 'Study registries', 'Online resources and browsing', 'Citation searching', 'Contacts', 'Other methods', 'Full search strategies', 'Limits and restrictions', 'Search filters', 'Prior work', 'Updates', 'Dates of searches', 'Peer review', 'Total records', 'Deduplication']
DOIs and change tracking¶
Extract clean DOIs and compare two retrievals to see exactly what changed.
extract_dois ¶
extract_dois(records, dedupe=True)
Pull cleaned, optionally de-duplicated DOIs from records or a list.
records is a record frame, whose doi column is read, or any
iterable of DOIs. Each DOI is stripped of surrounding whitespace, of a
resolver prefix such as https://doi.org/ or http://dx.doi.org/ and
of a leading doi: label, and a missing or empty one is dropped. With
dedupe (the default), a DOI that repeats an earlier one, ignoring case,
is dropped, so the first spelling is kept. Returns a list of strings, in
the order and the case the DOIs came in.
Source code in src/scopusflow/diff.py
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | |
diff_dois ¶
diff_dois(old, new)
Compare the DOIs of two retrievals.
A record without a DOI cannot be compared this way, and Scopus gives no
DOI to a share of what it indexes, more of it in the arts and humanities
and among books and proceedings (Gorraiz et al., 2016). Such records are
left out of the rows and counted in attrs.
diff_records compares the records
themselves, matching on the Scopus identifier.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
old
|
Record frames or lists of DOIs, the earlier ( |
required | |
new
|
Record frames or lists of DOIs, the earlier ( |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
A |
References
Gorraiz, J., Melero-Fuentes, D., Gumpenberger, C., & Valderrama-Zurián, J.-C. (2016). Availability of digital object identifiers (DOIs) in Web of Science and Scopus. Journal of Informetrics, 10(1), 98–109. https://doi.org/10.1016/j.joi.2015.11.008
Source code in src/scopusflow/diff.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
diff_records ¶
diff_records(old, new)
Compare two retrievals record by record.
Identifies which records were added, removed or unchanged between an
earlier and a later retrieval, and gives each record's citation count in
both. diff_dois compares DOIs alone, and
this function compares the records themselves, matched as
scopus_combine(dedupe=True) matches them. A record without a DOI is
therefore still tracked by its Scopus identifier, and two records under one
DOI are not taken for one.
Two records are one record when they share a Scopus identifier, or when
they share a DOI and one of them lacks an identifier. DOIs are compared as
extract_dois compares them, without
resolver prefixes and ignoring case. Records with different identifiers are
never matched, even under one DOI, since Scopus gives one DOI to distinct
papers as well as holding duplicate records of one paper
(scopus_combine gives the sources).
A record found in one retrieval only is a conflict where the other
retrieval gives its DOI to a different record. It may have been re-indexed
under a new identifier, or it may be a distinct paper sharing the DOI.
Only the two records themselves can tell, so it is counted neither as
added nor as removed. A record without an identifier whose DOI two
identified records carry is in the same position. A record with neither an
identifier nor a DOI cannot be matched at all, and is left out of the rows
and counted.
scopus_id is the identifier the record was matched on, missing for a
record matched on its DOI alone. doi is the DOI without its resolver
prefix, in the case the record gives it. doi, title and year
come from the later retrieval where it holds the record and gives a value,
and from the earlier one otherwise, while citations_old and
citations_new come from each retrieval in turn. Where one retrieval
holds a record more than once, the first value it gives is used.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
old
|
Record frames with a |
required | |
new
|
Record frames with a |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per record, with the columns |
Raises:
| Type | Description |
|---|---|
ValueError
|
When |
Examples:
>>> import pandas as pd
>>> import scopusflow as sf
>>> old = pd.DataFrame({"scopus_id": ["85000000001", "85000000002"],
... "doi": ["10.5555/a", "10.5555/c"]})
>>> new = pd.DataFrame({"scopus_id": ["85000000001", "85000000033"],
... "doi": ["10.5555/a", "10.5555/c"]})
>>> list(sf.diff_records(old, new)["status"])
['conflict', 'conflict', 'unchanged']
Source code in src/scopusflow/diff.py
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 | |
Analyse and visualise¶
Summarise a literature over time and by source or author, compare topics within it, and turn the summaries into figures.
The plotting functions below are pure over the frame they are given, so the record examples in this section run at build time over the bundled example harvest, and the topic comparison over a frame of the documented shape. Nothing here contacts the Scopus API or needs a key. The functions that do retrieve counts are marked as such and are shown but not run. The guides linked from each one demonstrate them against a live key.
scopus_trend ¶
scopus_trend(query, years, field=None, view='STANDARD', **kwargs)
Count Scopus hits for query in each of years without downloading them.
Each year is a cheap result-size lookup, so this gives a publication trend
far faster than harvesting every record. field wraps the query in a
Scopus field tag (see FIELD_TAGS), the way
scopus_count and the R twin's
scopus_trend() do; left None, the query is sent as given.
Returns a TREND_COLUMNS frame, one row
per year, sorted by year. view and any further keyword argument go to
pybliometrics' ScopusSearch.
Each count is sent with refresh=True unless you pass refresh.
pybliometrics keys its own response cache on the query string and view
alone. Without refresh=True, it could answer a year with the number of
rows an earlier harvest of the same query left there. refresh=False,
or a number of days, opts in to that cache, and a year it answers is
warned about. The key ignores keyword filters such as subj, so fold any
filter into the query before you opt in.
The result records when the counts were taken, in attrs["retrieved_at"]
as ISO 8601 text in UTC, as a harvest records its own. Scopus goes on
indexing a year's publications after the year has begun, so a count for
the year of retrieval is a count so far, and
plot_trend says so from this time. A count
answered from pybliometrics' cache is dated by its cache file, and the
earliest time is kept. When a cached count's file time cannot be read, the
attribute is left out, since a trend dated by its other counts would be
dated later than that one.
Raises ScopusFlowConfigError
before any request when pybliometrics.init() has not been called in the
session.
Source code in src/scopusflow/trend.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | |
This asks the API for a count per year, so it cannot run at build time. See
Analysing a literature for the worked
example, and year_counts below for the offline equivalent over records you
already hold.
year_counts ¶
year_counts(records, fill=True)
Count records per publication year, dropping rows with a missing year.
Returns a TREND_COLUMNS frame sorted
ascending by year, with both columns as plain integers.
With fill (the default), every year the records' searches covered is
counted, a year without records as zero.
plot_trend therefore draws the line down
to zero there. The years searched are read from the plan in
records.attrs["plan"], which
fetch_plan attaches: the year of each cell
under partition="year", and every year of the range under
partition="none". A year no cell searched is not added, so a plan over
2015, 2016, 2019 and 2020 gives no row for 2017 or 2018. A harvest stopped
before its last cell covers only the cells it ran. Without a plan, or with
one that sent no year limit, the years run from the first year in the
records to the last. A year that holds records keeps its row in every case,
and a set in which no record has a year gives an empty frame. The R twin's
autoplot() counts a record set's years by the same rule.
fill=False counts only the years that hold records.
The records' attrs["retrieved_at"] is carried to the result, so
plot_trend can say when the last year
plotted was still being indexed.
Source code in src/scopusflow/trend.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | |
records = sf.example_records()
out(sf.year_counts(records))
| year | n |
|---|---|
| 2015 | 15 |
| 2016 | 9 |
| 2017 | 10 |
| 2018 | 15 |
| 2019 | 19 |
| 2020 | 13 |
| 2021 | 13 |
| 2022 | 15 |
| 2023 | 15 |
| 2024 | 14 |
top ¶
top(records, by='source', n=10)
Tally the most frequent sources or authors in a record set.
Author strings holding several names are split on ";", so each
contributor is counted once per record. Only a view="COMPLETE" harvest
holds several names, since STANDARD records carry the first author alone
(see to_records). A tally by author of a
harvest whose attrs["view"] is "STANDARD" therefore counts first
authorships, and says so with a UserWarning, once per process, in the R
twin's words.
Each value is stripped of the white space R's trimws() removes, which is
spaces, tabs, carriage returns and line feeds. A value left empty is dropped,
as is a missing one, so a blank source title is not counted as a journal.
The result has the columns value and n. It is sorted by descending
count, with ties broken by value in code-point order. For UTF-8 text that
is the byte order the R twin's scopus_top() sorts by, in which capitals
come before lower case. The first n rows are returned, or every row when
there are fewer distinct values. Where several values tie at the cut-off,
the ones that sort first are kept, the same ones on every machine and in
both twins, whatever the order of the records.
n must be one positive whole number. A whole float such as 5.0 and a
numpy integer are taken, as the R twin takes any finite whole number, and an
n beyond the number of distinct values returns them all. A bool, NaN,
infinity, a fraction or a value below 1 raises ValueError, as does a
by other than "source" or "author".
Source code in src/scopusflow/records.py
334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 | |
out(sf.top(records, by="source", n=5))
| value | n |
|---|---|
| ACS Applied Materials & Interfaces | 8 |
| Journal of Power Sources | 5 |
| Synthetic Metals | 5 |
| Electrochimica Acta | 4 |
| Journal of Materials Chemistry A | 4 |
compare_topics ¶
compare_topics(reference_query, comparison_terms, years, field=None, view='STANDARD', **kwargs)
Compare comparison topics against a reference topic over the years.
Each comparison query is (reference) AND (term), after field has
wrapped each side. A reference containing OR or AND NOT therefore keeps its
meaning, and every comparison set is a subset of the reference. The year
limit is appended to each query as
scopus_count appends it.
comparison_terms is one term or a list of them. view and any further
keyword argument go to pybliometrics' ScopusSearch.
Returns a COMPARISON_COLUMNS
frame with one row per query and year. query is the query counted,
before its year limit: the reference, or (reference) AND (term), with
each side wrapped in field when one is given. query_type is
"reference" or "comparison", and abridged_query is the
reference or the term as given, trimmed, which labels the topic in
plot_comparison. n is that
query's count for the year and reference_n the reference's.
comparison_percentage is 100 * n / reference_n, so the reference
rows read 100, and it is missing where reference_n is 0.
average_comparison_percentage is the same ratio over the whole period:
the query's counts summed over the years divided by the reference's summed
over the same years, missing where the reference has no records in any
year. The reference rows come first, then the topics by descending average
share, with ties broken by abridged_query, and each topic's rows run in
year order.
One count request per term per year, plus one per year for the reference topic, so keep the term and year counts modest to stay within quota.
Every count is sent with refresh=True unless you pass refresh.
pybliometrics keys its own response cache on the query string and view
alone, so without refresh=True a reference harvested earlier could be
counted from the rows the harvest left there. The terms would still be
counted by the API, and a share of the two could pass 100%.
refresh=False, or a number of days, opts in to that cache, and a count
it answers is warned about. The key ignores keyword filters such as
subj, so fold any filter into the queries before you opt in.
Raises ScopusFlowConfigError
before any request when pybliometrics.init() has not been called in the
session.
Source code in src/scopusflow/compare.py
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 | |
This makes one count request per year for the reference topic and one per term per year, (terms + 1) × years in all, so it cannot run at build time. See Comparing topics for the worked example.
COMPARISON_COLUMNS
module-attribute
¶
COMPARISON_COLUMNS = ['query', 'query_type', 'abridged_query', 'year', 'n', 'reference_n', 'comparison_percentage', 'average_comparison_percentage']
scopus_intersections ¶
scopus_intersections(concepts, intersections=None, abbrev=None, sep=' × ', years=None, field=None, view='STANDARD', verbose=False, **kwargs)
Count each concept and each requested intersection of concepts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
concepts
|
Mapping[str, str]
|
A mapping whose keys are display labels and whose values are search terms
(wrapped in |
required |
intersections
|
Sequence[Sequence[str]] | None
|
An optional sequence of sequences, each naming two or more distinct
concept labels whose intersection should be counted, for example
|
None
|
abbrev
|
Mapping[str, str] | None
|
An optional mapping of short labels, keyed by concept label and used only when composing intersection labels, so those rows stay readable while the concept rows keep their full names. |
None
|
sep
|
str
|
The separator joining member labels in an intersection label; defaults to a multiplication sign between spaces. |
' × '
|
years
|
Sequence[int] | None
|
Optional publication years, applied to every count as one inclusive
range from the earliest year to the latest, as
|
None
|
field
|
str | None
|
An optional Scopus field tag wrapped around each concept value that is not
already a complete field-tagged expression (see
|
None
|
view
|
str
|
The Scopus search view, |
'STANDARD'
|
verbose
|
bool
|
When |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
One row per concept and per intersection, with columns |
Notes
This performs one count request per concept and per intersection, so it needs
a valid API key and internet access, exactly as
scopus_count does.
Further keyword arguments reach each of those counts, and so does its
default of refresh=True. pybliometrics' own response cache, keyed on the
query string and view alone, would otherwise answer a concept that an
earlier harvest downloaded with the number of rows that harvest left there.
refresh=False, or a number of days, opts in to that cache, and a count
it answers is warned about. The key ignores keyword filters such as
subj, so fold any filter into each concept's query before you opt in.
Source code in src/scopusflow/intersections.py
140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 | |
This makes one count request per concept and per intersection, so it cannot run at build time. See Analysing a literature for the worked example.
plot_trend ¶
plot_trend(trend, ax=None)
Plot publication counts over time as a filled area, line and points.
trend has columns ["year", "n"], as
scopus_trend and
year_counts return them. Returns the
matplotlib Axes. An empty frame raises
ValueError("The trend has no years to plot."), as the R twin's
plot_scopus_trend() does. ax is the matplotlib Axes to draw
into, and a new figure is made when it is None.
When the years plotted reach the year recorded in
trend.attrs["retrieved_at"], a note below the axis label says that the
year was incomplete when the counts were taken, as in "2026 was incomplete
when retrieved on 2026-09-29." Scopus goes on indexing a year's
publications after the year has begun. The note is read from the recorded
time alone, never from the clock of the session drawing the figure, so a
trend that records none is drawn without it. The R twin's figures carry
the same note in their caption.
Source code in src/scopusflow/plots.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | |
sf.plot_trend(sf.year_counts(records))
show()
plot_top ¶
plot_top(top, ax=None)
Plot a horizontal bar chart of the most frequent values, largest on top.
top has columns ["value", "n"] (from
top); each bar carries its count at its end,
matching the R plot_scopus_top. ax is the matplotlib Axes to
draw into, and a new figure is made when it is None. Returns the
matplotlib Axes. An empty tally raises ValueError.
Source code in src/scopusflow/plots.py
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | |
sf.plot_top(sf.top(records, by="source"))
show()
plot_comparison ¶
plot_comparison(comparison, highlight=None, interval=True, counts_in_legend=True, ax=None, legend_inside=False)
Plot each comparison topic's share of the reference literature over time.
comparison is the frame from
compare_topics. With interval a
shaded Wilson band shows how stable each yearly share is (illustrative,
never a confidence interval, since Scopus counts are exact). highlight
names one topic to draw in an accent colour, the rest in grey. With
counts_in_legend (the default) each label carries the topic's total
record count, for example machine learning (n = 1,204).
A legend is only drawn when there are too many topics to label the lines
directly. legend_inside governs where it then sits: with the default
False matplotlib chooses the least-obtrusive spot automatically
(loc="best"); with True the legend is placed inside the axes, in
whichever corner has the most free space, saving the width an outside legend
would otherwise take. Returns the matplotlib Axes.
ax is the matplotlib Axes to draw into, and a new figure is made
when it is None.
Source code in src/scopusflow/plots.py
206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 | |
years = list(range(2013, 2022))
ref_n = [400, 550, 700, 850, 1000, 1150, 1300, 1450, 1600]
shares = {"computer vision": 34.0, "natural language processing": 24.0,
"medical imaging": 11.0, "drug discovery": 6.0}
rows = [{"query": "deep learning", "query_type": "reference",
"abridged_query": "deep learning", "year": year, "n": n,
"reference_n": n, "comparison_percentage": 100.0,
"average_comparison_percentage": 100.0}
for year, n in zip(years, ref_n)]
for topic, end in shares.items():
for i, (year, n) in enumerate(zip(years, ref_n)):
pct = end * (0.45 + 0.55 * i / (len(years) - 1))
rows.append({"query": f"(deep learning) AND ({topic})",
"query_type": "comparison",
"abridged_query": topic, "year": year,
"n": int(pct * n / 100), "reference_n": n,
"comparison_percentage": pct,
"average_comparison_percentage": end})
comparison = pd.DataFrame(rows, columns=sf.COMPARISON_COLUMNS)
sf.plot_comparison(comparison)
show()
plot_scopus_intersections ¶
plot_scopus_intersections(x, highlight=None, highlight_label=None, ax=None)
Plot concept and intersection counts as a horizontal lollipop chart.
x is a frame from
scopus_intersections
(columns label, n and type). Counts run along a logarithmic x
axis, so fields spanning orders of magnitude sit together with their
smaller intersections. Rows named in highlight are drawn in the focal
colour, and highlight_label names them in the legend (derived from
their type when unset). Returns the matplotlib Axes. Mirrors the R
plot_scopus_intersections.
ax is the matplotlib Axes to draw into, and a new figure is made
when it is None. A row without a positive count cannot sit on a log
axis and is dropped with a warning. ValueError is raised for a frame
without rows or without a positive count, for duplicate labels and for a
highlight naming a label the frame does not hold.
Source code in src/scopusflow/plots.py
393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 | |
sets = pd.DataFrame({
"label": ["semantic priming", "mental simulation",
"semantic priming × mental simulation"],
"query": ["TITLE-ABS-KEY(semantic priming)",
"TITLE-ABS-KEY(mental simulation)",
"(TITLE-ABS-KEY(semantic priming)) AND "
"(TITLE-ABS-KEY(mental simulation))"],
"n": pd.array([6600, 2600, 18], dtype="Int64"),
"type": ["concept", "concept", "intersection"],
"size": [1, 1, 2],
"members": ["semantic priming", "mental simulation",
"semantic priming; mental simulation"],
})
sf.plot_scopus_intersections(
sets, highlight=["semantic priming × mental simulation"]
)
show()