Changelog¶
All notable changes to this project are documented here.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased¶
Upgrading from 0.3.0¶
Changed results¶
fetch_plan(),scopus_count(),scopus_trend(),compare_topics()andscopus_intersections()bracket a query with a top-level OR, AND NOT or proximity operator before adding the year limit, which a top-level AND NOT took into its negation. Re-run such queries, whosefetch_plan()checkpoints are fetched again once.compare_topics()sends each comparison as(reference) AND (term), where, withoutfield, it joined them without brackets, so a reference with a top-level AND NOT took the term into its negation and shares could pass 100%. Re-run such comparisons made withoutfield. Givenfield, only thequerycolumn differs.fetch_plan(),scopus_count(),scopus_trend(),compare_topics()andscopus_intersections()sendrefresh=Trueunless you passrefresh, where pybliometrics' cache could answer a re-run with an earlier result.refresh=False, or a number of days, uses that cache again and warns when it answers.fetch_plan()records each checkpoint's query, view, page size and paging in a manifest beside it and serves it only to a matching cell, where an empty checkpoint served any plan and aCOMPLETEone aSTANDARDplan. Empty 0.3.0 checkpoints are fetched again.scopus_query()brackets each term of several words, other than a quoted or braced phrase or a bracketed or field-tagged group, when it joins terms withoutfield, soscopus_query("machine learning", "deep learning", op="OR")gives(machine learning) OR (deep learning), where Scopus read the unbracketed join asmachine AND (learning OR deep) AND learning.scopus_abstract()checkpoints only successful retrievals, so a failed identifier is retried on resume, where its NA row was read back as data. NA rows that 0.3.0 checkpointed are still read back, so delete their checkpoint files to retry them.- Checkpoints are written atomically. An unreadable one, which stopped every resume, or
one lacking a record column, which was served, is fetched again with a warning. A
0.3.0 CSV checkpoint cut short by an interrupted run can still be served, so re-run
that plan with
resume=False. - A CSV checkpoint resumes with
scopus_idas text andyearandcitationsas whole numbers, where they were read back as numbers, as floats once a value was missing, so a resumed record could carry a different identifier. - A missing EID or citation count in search or abstract results gives NA, where an
absent EID became the string
"nan"inscopus_idand a missing citation count raised an error or, inscopus_abstract(), lost the row. scopus_abstract()trims identifiers, drops theSCOPUS_ID:prefix underby="scopus_id"and sends integer Scopus IDs as text, where it sent them as given.corpus()drops blank identifiers with a warning, as it drops missing ones.scopus_abstract()keeps a document's row when pybliometrics cannot parse a keyword entry without text (pybliometrics issue 436), leaving its keywords NA with a warning, where it lost the whole row. pybliometrics 4.5 drops such an author keyword itself.- Once the quota is spent,
scopus_abstract()andcorpus()give an NA row to an identifier that pybliometrics' own cache holds, where 0.3.0 returned its cached record. corpus(view="REF")no longer asksscopus_abstract()for author keywords, which REF does not carry, so a run resumed from the checkpoints 0.3.0 wrote for that view fetches those abstracts again, spending Abstract Retrieval quota.year_counts()gives zero for each searched year without records or, for a set without a plan or year limit, each year between its first and last, where it left them out andplot_trend()drew a line across them.fill=Falsegives the 0.3.0 output.top()strips sources and authors of spaces, tabs and line breaks alone and drops blank sources, where it stripped authors of any white space and sources of none.top()orders tied values in byte order, keeping those that sort first at the cut-off, where it kept an arbitrary subset of them.fetch_plan()andto_records()join aCOMPLETEharvest's authors with"; ", where they kept pybliometrics'";", which the R twin read as one author, and drop the trailing comma of a name without a given name.scopus_abstract()counts each document it requests once inn_requests, where a step failing after the response arrived counted it twice, and leaves out the rows pybliometrics answered from its own cache.scopus_trend()sends a whole-number float year, such as 2015.0, asPUBYEAR IS 2015, where it sentPUBYEAR IS 2015.0.example_records()gives the U+2010 hyphen in five author names as the ASCII hyphen, where it kept U+2010, so those names now match the R twin's corpus.- The app trims the query before every request, so it sends the query the script panel shows, where 'Fetch records' sent it as typed.
Changed interface¶
SearchPlan,scopus_count(),scopus_trend(),compare_topics()andscopus_intersections()refuse withValueError, before any request, unbalanced brackets, quotation marks or braces and theLIMIT-TO()andEXCLUDE()refinements that the Search API ignores, with which a search ran unfiltered.wrap_field()andSearchPlangivenfieldrefuse withValueErrora query, andscopus_query()givenfielda term, that already opens with a field tag, where they nested the tags, which the API rejects.scopus_count()andcompare_topics()refuse such a query before the request.years, wherever a function takes it, must hold whole numbers from 1700 to 2200, so a fraction, a year outside that range or a numeric string such as"2015"raisesValueError, where onlycompare_topics()refused the first two and every function took the third.- scopusflow requires pybliometrics 4.4 or later, and its
appextra NiceGUI 2.14 or later. corpus()returns ascopus_idcolumn in second place, so code that selects its columns by position needs updating.scopus_trend()takesfieldthird, beforeview, so a view passed by position, as inscopus_trend(q, years, "COMPLETE"), is taken as a field tag. Passview=by name.fetch_plan(subscriber=False)pages by offset, as pybliometrics documents, where scopusflow overrode it with cursor paging, so a cell over 5000 records raisesScopusQueryErrorand 0.3.0 checkpoints are fetched again. Dropsubscriber=Falseto page by cursor as 0.3.0 did.SearchPlanstoresyearssorted and without repeats, as a tuple, andfieldstripped and upper-cased. It raisesValueError, when the plan is built, for afieldthat is not a string of letters and hyphens.scopus_abstract()raisesTypeErrorforidsthat are not one identifier, a list, a tuple, a pandas Series or Index or a NumPy array, such as a set, which it accepted.scopus_abstract()raises, before any request,ValueErrorfor a missing or blank identifier andTypeErrorfor one that is neither text nor an integer, such as a float, even a whole one.corpus()raises the sameTypeErrorfor such a value in itsbycolumn.scopus_abstract(include=("keywords",))raisesValueErrorunlessview="FULL", where other views gave an empty column.fetch_plan(),scopus_count(),scopus_trend(),compare_topics()andscopus_intersections()raiseScopusFlowConfigError, aRuntimeError, before any request, whenpybliometrics.init()has not been called in the session, where pybliometrics raisedFileNotFoundError('No configuration file found').scopus_abstract()andcorpus()raiseScopusFlowConfigError, before any request, whenpybliometrics.init()has not been called in the session, where they gave every identifier an NA row with a warning.plot_trend()raisesValueErrorfor an empty trend, where it drew empty axes or failed inside numpy, depending on the versions installed.top()raisesValueErrorfor annthat is not a positive whole number, wheren=-1returned all but the last value.fetch_plan(),scopus_abstract()andcorpus()refuse withValueError, before any request, acache_dirin which no file can be created, where 0.3.0 read its checkpoints and lost any records it fetched to an error. Copy a read-only cache to a writable directory.scopusflow-guirejects an unknown flag, where it ignored one, and gains--version,--host,--portand--no-browser. Reinstall an editable install, since its entry point has moved.
Changed output¶
to_bibtex()andto_ris()write a name ending in initials as 'Surname, I.', where BibTeX read the initials as the family name and Zotero kept each RIS author as one field. Citation keys are unchanged.to_bibtex()spaces run-together initials, which BibTeX read as one given name, and protects words such as DNA in titles from lower-casing.to_ris()gives the source title asT2in place ofJOand addsDB - Scopusand the EID asAN.to_ris()andto_bibtex()export anauthkeywordsorabstractcolumn, as aCOMPLETEharvest (the abstract withfields="extended") orscopus_abstract()gives, where they wrote neither.top(by="author")warns once per process that a set recorded as retrieved underSTANDARDnames first authors only. Sets retrieved by 0.3.0 record no view and do not warn.- When the years plotted reach the year of retrieval,
plot_trend()adds a note below the axis saying that it and any later year were incomplete when retrieved. It reads the year fromattrs["retrieved_at"], whichscopus_trend()records andyear_counts()keeps. scopus_abstract()andcorpus()stop requesting at a spent quota and give the remaining identifiers their NA rows with oneScopusFlowQuotaWarning, where they requested each and warned for each.fetch_plan()warns when a cell returns fewer records than the API reported for it.SearchPlan,scopus_count(),scopus_trend(),compare_topics()andscopus_intersections()warn withscopusflow.query.QuerySyntaxWarningforINDEXTERMS(), which pybliometrics documents as not working through the Search API.plot_top()raises aValueErrorsaying that the tally is empty, where itsValueErrorconcerned an empty sequence.- The app's generated script downloads as
scopusflow-script.py, where a script saved asscopusflow.pyimported itself in place of the package when run from its folder. - The script the app generates keeps each plan's checkpoints in a directory of its own,
where plans shared
harvestand overwrote each other's checkpoints. A script generated now does not resume from the checkpoints of one generated by 0.3.0. - The script the app generates records the scopusflow version, notes when demo mode was
on, calls
pybliometrics.init(), writes the search record toscopus-search-record.mdand includes a comparison run with 'Partition by year' off, which it left out.
Added¶
scopus_search_report()writes up a harvest, or a plan not yet run, as a PRISMA-S search record (Rethlefsen et al., 2021) with the PRISMA 2020 identification counts, returned as aSearchReport. Itsformat(style="paragraph")gives a methods paragraph, andscopus_search_report(file=)writes the record as Markdown.PRISMA_S_ITEMSnames the 16 items.- The app's 'Search record (.md)' button downloads that record. In demo mode, the record says that it describes no search that was run.
scopus_combine()binds record sets and renumbersentry_number.dedupe=True, also spelled"id_or_shared_doi", keeps one copy of each record, matched by Scopus identifier or, for a record without one, by DOI, and counts the duplicates removed.dedupe="id_or_doi"and"id_then_doi"apply other rules.fetch_plan()gainscursor.cursor=Falsepages by offset, for keys without a Scopus subscription, and raisesScopusQueryErrornaming any cell that matches more than 5000 records. The default keeps cursor paging, and a checkpoint serves only a request in its own paging mode.fetch_plan()andto_records()gainfields="extended", adding the document type, source type, volume, issue, pages, ISSN, PubMed identifier, open-access status and, underCOMPLETE, the abstract, at no extra quota.EXTENDED_COLUMNSnames the columns both views add.- With
fields="extended",to_ris()andto_bibtex()type conference papers, conference reviews, chapters and books by their Scopus document type and add the volume, issue, pages, ISSN and, in RIS, the document type asM3. scopus_abstract(include=("index_terms",))returns the index keywords Scopus assigns, in anidxtermscolumn, underview="FULL".corpus()gainskeywords, to use them alone ("index") or with the author keywords ("both"). The default,"author", is unchanged.write_corpus()writes, andto_theoryforge_corpus()returns, acorpus()result in theoryforge's corpus schema, with aprovenanceblock, keying cited works on the Scopus identifier so that a work is one node however its DOI is written.diff_records()compares two harvests record by record, matching records asscopus_combine(dedupe=True)does. It gives each record's citation counts in both and marks asconflicta record whose DOI the other harvest gives to a different record.write_records()andread_records()save a record set and read it back, the file extension choosing parquet, which keeps the provenance and needs the newparquetextra, or CSV, read back with the record schema.SearchPlangainspage_size, by default the largest page the view allows (200 forSTANDARD, 25 forCOMPLETE), whichfetch_plan()sends, andto_dict()andfrom_dict(), which turn a plan into a plain dict and back.scopus_trend()gainsfield, which wraps the query in a field tag, as inscopus_count().TREND_COLUMNS,COMPARISON_COLUMNSandABSTRACT_COLUMNSare exported from the package top level.- The app's 'Search in' selector offers '(none)', which sends the query without a field tag.
Changed¶
fetch_plan()records its provenance in the harvest'sattrs: the plan, the first and last retrieval times, the scopusflow version, the paging mode, the view and each cell's record count beside the total the API reported.- A harvest stopped through
should_stoprecordsattrs["stopped_early"],attrs["cells_planned"]andattrs["cells_run"], and the app says that it stopped after k of N cells, where it announced the records retrieved. - The app shows a harvest's warnings in its live terminal and says that a harvest that warned may be incomplete, where it announced it as a success.
- In demo mode, the app says above the records that they were replayed from the bundled example harvest and were not retrieved from Scopus. Before a harvest, the results area says that the records will appear there.
corpus()reports its requests and the quota left inattrs["n_requests"]andattrs["quota"], and records the search expression, when the search that found the records ran and the kind of keyword asattrs["query"],attrs["retrieved_at"]andattrs["keywords"].diff_dois()counts the records it could not compare for want of a DOI inattrs["n_old_without_doi"]andattrs["n_new_without_doi"].- The app draws its buttons, focus rings and progress bars in a deep teal on which the buttons' white text has the contrast it needs, each figure has a name a screen reader reads and the page has a top-level heading.
Fixed¶
- A checkpoint that
fetch_plan(),scopus_abstract()orcorpus()fails to write, as on a full disk, gives a warning, and its records are kept, where they were lost to an error. scopus_intersections()uses as given a concept that opens with a tag ofFIELD_TAGSwritten with a space before the bracket, as the web interface writesTITLE-ABS-KEY ( x ), where it wrapped it infieldagain and the API refused it.- Each app session keeps its checkpoints in a directory of its own, so closing one tab no longer deletes another's mid-harvest.
- The app runs one harvest, plan check or comparison at a time, where a harvest could start during a comparison and a comparison could start twice.
fetch_plan()runs a plan withpartition="none"and whole-number float years, such as 2015.0, where it raisedValueError.- The app's size note clears when the search whose size it gave changes, where it stayed on screen for a different search.
- The app shows a short note in place of a figure it cannot draw, where the error took the export buttons away, and shows its year range and progress bar only while they apply.
Security¶
- The app keeps a pasted API key in memory, where it wrote it to
~/.config/pybliometrics.cfgon a machine without a pybliometrics configuration. On such a machine, it caches responses in a temporary directory removed when the app stops.
Documentation¶
- The guide to designing queries describes the offline query checks and gives a recipe for testing a strategy's relative recall against records known to be relevant, following Sampson et al. (2006) and Bramer et al. (2018). It also explains loose and exact phrases and why a query mixing operators needs brackets.
- The guides no longer present the bundled corpus as the result of an unquoted search. It came from an exact-phrase search in OpenAlex, and the demonstration plan and live searches quote the phrase, since Scopus joins the words of an unquoted phrase with AND.
- The API reference gives a topic comparison's cost as (terms + 1) × years requests, where it gave one count request per term per year.
- The app guide no longer promises that a run stopped by the quota resumes in a later
session, and points to the generated script for a harvest that has to continue after
the session ends. It says that demo mode is on whenever the app opens in a tab, where
it said only the first time, and it and the
launch()docstring no longer say that the app binds to127.0.0.1whatever host it is given. - The guide to building a reference set says that a harvest under the default
STANDARDview names the first author of each record alone, so each RIS or BibTeX entry has one author, and thatview="COMPLETE"gives the author list, where it said that a live harvest returned every author. - The guides show
pybliometrics.init(), which pybliometrics 4 requires in every session, before the first live call, where they implied that a configuration file was enough. - The guides and the
scopus_abstract()andcorpus()docstrings no longer say that REF reference lists are truncated, sincescopus_abstract(view="REF")andcorpus(view="REF")return the whole list at one request per 40 references. - The tracking guide keeps its baseline with
write_records()andread_records(), where its CSV route readscopus_idback as a number, and gives the correct counts of new and shared papers, saying that records without a DOI were not compared. - The guides say that a refused entitlement stops a batch of abstracts, and the planning
guide that
scopus_count()also counts the years between those of a year-partitioned plan with gaps. Theto_records()docstring gives a missing author keyword as a value thatpandas.isnarecognises, where it saidNone. - The guides no longer say that the 5000-record offset ceiling forces a partition by
year, since
fetch_plan()pages by cursor unless givencursor=False, and give the partition's reasons as per-year checkpoints, completeness checks and progress. - Install commands in the documentation give extras in double quotes and carry no trailing comments, so they paste into any shell.
0.3.0 - 2026-07-23¶
Added¶
example_records()returns a bundled worked-example harvest of 138 real journal articles on graphene supercapacitors, 2015 to 2024, over the standardRECORD_COLUMNSschema. The records come from OpenAlex, whose metadata is released under CC0; Scopus terms do not permit redistributing retrieved records. They are the same corpus the R twin ships.
Changed¶
- Every guide and reference example now runs on that bundled harvest, where a fabricated frame used to stand in, with the live API call shown alongside. The app's demo mode replays it too, so a first visit shows real articles and a real publication curve. A demo cell for a year outside the corpus returns nothing and says so in the log, since padding it out with a neighbouring year's rows would double records up. The app's year slider opens on the corpus span, and the demo comparison, whose counts are still simulated, now says so beneath the figure.
Fixed¶
extract_doisno longer emits the literal<NA>for a record with no DOI, which a nullable string column produces where a float NaN would have been skipped.
0.2.0 - 2026-07-15¶
Added¶
- Concept intersections:
scopus_intersectionssizes a named set of concepts and their overlaps from the count endpoint (one cheap request per row, no harvest), accepting bare terms wrapped in a field tag or complete field-tagged expressions used as given, with optional short labels for the intersection rows.plot_scopus_intersectionsdraws the result as a log-scale lollipop chart with a focal set accented. This brings the Python package to parity with the Rscopus_intersections()andplot_scopus_intersections().
0.1.0 - 2026-07-10¶
First release: a reproducible workflow layer over pybliometrics for Scopus searches, and the Python twin of the R package scopusflow.
Added¶
- Search planning and queries:
SearchPlandescribes a search before running it and can be partitioned by year, a field-tagged boolean query builder (scopus_query,wrap_field) with a reference table of Scopus field tags, and cheap result-size lookups (scopus_count) for quota-aware pre-flight sizing. - Harvesting: resumable, checkpointed retrieval of a plan (
fetch_plan), with per-cell progress logging on thescopusflowlogger and co-operative cancellation (should_stop). Checkpoints are keyed by the whole plan, and on resume the recorded query is compared against the cell's, so a cache directory written by a different plan is refetched with a warning rather than silently reused. - Records: one stable, tidy record schema across query types (
to_records,RECORD_COLUMNS), with anauthkeywordscolumn added whenSearchPlan(view="COMPLETE")is used, at no request cost beyond that view's own smaller page size. - Abstract retrieval: batch, per-identifier-resilient
scopus_abstract(defaultview="META_ABS", so a plain call returns the abstract text), withinclude=("references", "keywords")for a document's own reference list and author keywords, per-identifier caching keyed by view andincludeselection (cache_dir/resume),n_requests/quotaaccounting, a warning when the references returned fall short of the document's reportedrefcount, and a clearScopusFlowForbiddenErroron an entitlement 403 that stops the batch, suggesting the other of"FULL"/"REF"only when the refused view was one of those two. - Corpus assembly:
corpuscombines a search result with abstract retrieval into a minimalid/title/year/keywords/referencesshape for downstream tools such as keyword co-occurrence or citation-network analysis, covered by the Author keywords and references guide. - Trends and comparison: annual publication trends (
scopus_trend,year_counts) computed from cheap result-size lookups, and topic-trend comparison (compare_topics) with per-step progress logging. The comparison's documented count-request cost includes the reference topic: (terms + 1) x years. - Summaries and change tracking: source and author top tallies (
top), DOI extraction and change tracking between runs (extract_dois,diff_dois). - Plots: brand-themed matplotlib figures for trends (
plot_trend), top tallies (plot_top, labelling each bar with its count and reserving x-axis headroom so the longest label stays inside the axes) and topic comparison (plot_comparison, mirroring the Rplot_scopus_comparison): illustrative Wilson stability bands with a Source/Wilson caption, a percent-formatted y-axis, record counts in the labels by default (counts_in_legend), the reference topic named in a subtitle, colour-matched end-of-line labels spread apart at draw time in the same order as the line ends, an optionallegend_insideplacing the legend in the emptiest corner, and full schema validation with a friendlyValueError. - Reference-manager export to BibTeX and RIS (
to_bibtex,to_ris), so a search carries into Zotero, EndNote, Mendeley or a LaTeX bibliography. - A local, code-free NiceGUI app (
scopusflow-gui, theappextra) that drives the whole workflow through a browser tab: a background harvest with a live progress terminal, every choice (including the chosenviewand the comparison) mirrored back as a runnable Python script, results shown as a paginated table and plots with export, a Compare topics card with highlight, stability-band and counts-in-label controls plus a count-request cost estimate and CSV export, and a demo mode that needs no key. Harvest checkpoints live under the temp directory and are removed when the tab closes, so search terms do not linger on disk. Duplicate comparison terms are dropped, a zero-record harvest is reported as a warning, and the controls carry visible labels. - Documentation guides that execute their examples, so tables and plots appear inline as on a knitted vignette.