spb/wp44_uqo

1% ============================================================================2% Part III — Data and Systematic Review Protocol3% Written by Agent L (Part III instance), 2026-11-03. Every corpus number is a macro from tables/numbers.tex4% (generated by tables/make_numbers.py from data/metadata/prisma_counts.json and linkage_report.json).5% Sources of truth: RESEARCH_PLAN.md §2–§3, protocol/PROTOCOL.md, protocol/amendments.csv,6% reports/PHASE0_FEASIBILITY.md, data/raw/README.md, ingestion/*.py, systematic_review/define_corpus.py.7% ============================================================================89\section{Information sources and conditions of access}10\label{sec:data-sources}1112A field-level mapping study stands or falls with the bibliographic databases on which it is built. The choice13of sources determines which works can be observed at all, how reliably they can be dated and typed, whether14citation links are available, and under what licence derived data may be redistributed. This part documents15these choices for WP44 and reports, stage by stage, how the mapping corpus was assembled from the raw16sources. The exposition follows the order of the pipeline (identification, linkage, deduplication,17eligibility, scope, dating) and closes with the quality indicators and the research-integrity procedures that18govern every later analysis. The reporting structure is modelled on the PRISMA 2020 items that apply to a19mapping study \citep{page2021prisma,page2021prismaee}; items concerned with risk of bias and effect measures20have no counterpart in a bibliometric design and are replaced, as PRISMA-ScR does for scoping reviews21\citep{tricco2018prismascr}, by indicators of metadata quality and coverage (Section~\ref{sec:data-quality}).22The search-reporting recommendations of PRISMA-S \citep{rethlefsen2021prismas} are followed to the extent23that a database-scope strategy without keyword queries permits.2425\subsection{Why a multi-source design}2627No single database indexes computer science completely, and the gaps are not random. The venue-centred28coverage of DBLP, the publisher-centred coverage of Crossref, the preprint-centred coverage of arXiv and the29aggregating coverage of OpenAlex each miss a distinct part of the literature and type the part they do cover30in distinct ways. Comparative studies of bibliographic data sources have documented large differences in31coverage between Web of Science, Scopus, Dimensions, Crossref and Microsoft Academic32\citep{visser2021comparison,martinmartin2021comparison}, and the journal-level coverage of the two commercial33indexes is known to under-represent conference proceedings, which carry a large share of computer-science34output \citep{mongeon2016coverage}. For the open successor of Microsoft Academic, OpenAlex, the first35independent assessments report strong coverage of works with DOIs together with weaker and uneven metadata on36affiliations, document types and references \citep{alperin2024openalex,culbert2025openalex,37scheidsteger2023openalex,delgadoquiros2024completeness}. These findings motivate the design adopted here: one38source supplies the backbone (OpenAlex), a second supplies the definition of what counts as computer science39(DBLP), a third supplies the preprint layer and its dates (arXiv), and a fourth checks the coverage of a40subfield with a dense, curated venue structure (the ACL Anthology). Crossref is used for verification only.4142\subsection{OpenAlex}4344OpenAlex \citep{priem2022openalex} is the backbone of the corpus. It supplies, for every work, the45identifiers (OpenAlex~W identifier, DOI, arXiv identifier where present), the publication date and type, the46title and an inverted-index abstract where available, the venue (source) and its type, the authorships with47institutions and countries, the topic assignment at three levels (field, subfield, topic) with a primary48topic, the list of referenced works (the citation graph), the open-access status and licence, and the49field-weighted citation impact together with yearly citation counts. OpenAlex is released under the CC050public-domain dedication, which permits redistribution of record-level derived data.5152Two access routes exist and both were used, for different purposes. The public snapshot is a versioned bulk53release in gzipped JSON Lines, published in a public S3 bucket; the release used here is dated542026-10-14 and comprises 1{,}898 part files for the \emph{works} entity, 675.3~GB compressed, with a manifest55listing the record count of each file. The snapshot was streamed from the primary compute node, each file's56content length checked against the manifest and a SHA-256 of the compressed bytes recorded in a SQLite ledger57before parsing; every one of the 1{,}898 files parsed without error and the sum of the per-file record counts58equals the manifest total of \LinkSlimTotal{} works. Two outputs were written during this single pass. The59first is the full JSON record of every work carrying \emph{any} topic in OpenAlex field~17 (Computer Science)60with a publication year between 2018 and 2026, stored as Zstandard-compressed JSON Lines and partitioned by61the snapshot's own date partitions (\LinkOaCsRecords{} records, about 20~GB). The second is a \emph{slim}62table of every work in the snapshot (identifier, DOI, year, type, primary topic, subfield, field and domain,63citation count and a few flags), kept in Parquet so that works cited by computer-science papers but published64in other fields can be resolved for the interdisciplinary analyses of Part~VIII (decision D2 of65\texttt{RESEARCH\_PLAN.md}). Records flagged by OpenAlex as XPAC expansion records are retained at this stage66with their flag so that the exclusion decision can be taken, and counted, later (Section~\ref{sec:data-eligibility}).67The smaller entity snapshots (topics, subfields, fields, domains, sources, institutions, publishers, funders,68concepts) were downloaded the same day with a per-file SHA-256 ledger; author entities were not downloaded69because the authorship data embedded in each work record suffice for the analyses planned.7071The second route is the REST API. Without an API key the public endpoint operates under a shared daily budget72of USD~0.10, list-and-filter calls are priced at USD~0.0001 each, and cursor paging is unavailable, so that a73single filtered query cannot return more than 10{,}000 results through basic paging. Single-record lookups by74identifier are free. The API was therefore used for two narrowly defined tasks. First, an incremental harvest75of works with any computer-science topic and a publication date between 2026-10-01 and 2026-11-02, intended76to capture records added after the snapshot was cut; the harvest is sliced by publication day and, where a day77exceeds the paging limit, split into records with a computer-science primary field and records without one.78Raw JSON pages are stored verbatim under a dated directory with a ledger of counts and cost, and the job79resumes on the following day when the budget is exhausted. Second, free single-record lookups by DOI serve as80one of the verification routes for the bibliography of this manuscript (Section~\ref{sec:data-integrity}).81A free API key, which would raise the daily budget, requires the principal investigator to act on an e-mail82and was deliberately not obtained without instruction; the mandate forbids assuming credentials that have not83been tested.8485\subsection{DBLP}8687The DBLP computer science bibliography \citep{ley2002dblp,ley2009dblp} is a curated, venue-centred index that88has for three decades defined, in practice, the set of journals and conference series regarded as computer89science. It is used here in that capacity: a work indexed by DBLP is in computer science by construction, and90DBLP venue streams feed the taxonomy of Part~IV. The complete XML dump (\texttt{dblp.xml.gz}, dated 2026-11-029122:39~UTC, 1.12~GB) was downloaded together with its DTD and MD5 checksum, the checksum verified, and the file92parsed with a validating XML parser into one row per publication record (articles, conference papers,93proceedings volumes, books, book chapters, doctoral and master's theses, data records; the \texttt{www}94person records are skipped). The whole archive was kept so that per-year counts are available for the95historical baseline and for context. The DBLP dump is released under the Open Data Commons Attribution licence96(ODC-BY~1.0). DBLP contributes, in addition to the scope definition, author person identifiers, venue streams,97and DOI links for most records; the \texttt{ee} fields were also scanned for arXiv URLs, with a result reported98in Section~\ref{sec:data-linkage}.99100\subsection{arXiv}101102The arXiv e-print repository \citep{ginsparg2011arxiv} is the main preprint channel of computer science and,103for the period under study, the place where a large share of the most cited work first appeared. Its104metadata are harvested through the OAI-PMH interface (\texttt{set=cs}, \texttt{metadataPrefix=arXiv}) in105monthly datestamp windows from January 2018 to 2 November 2026. The harvester waits at least 3.5~s between106requests and honours \texttt{Retry-After} headers; every raw XML page is kept verbatim as the immutable107snapshot and parsed afterwards into a single Parquet table. Two properties of OAI-PMH matter for what follows.108The \texttt{from}/\texttt{until} parameters filter on the record's \emph{datestamp}, the time of its last109update, so that a monthly window contains every record updated in that month, not every record first110submitted then; duplicates across windows are resolved at parse time with the latest datestamp winning. The111\texttt{created} element gives the first submission date, which is the date used throughout this study for112preprints. The harvest returned \LinkArxivCsRecords{} distinct records, of which113\LinkArxivCreatedTwoZeroOneEightTwoZeroTwoSix{} were first submitted in 2018--2026; the remainder are older114e-prints updated during the window. The arXiv metadata are released for reuse under the repository's metadata115terms; e-prints themselves are governed by per-paper licences and are downloaded, for the high-priority set116only, within those licences and at the repository's rate limits. The use of arXiv as a research dataset, and117the caveats attached to its category labels and versioning, are discussed by \citet{clement2019arxiv}.118119\subsection{ACL Anthology}120121The ACL Anthology is the curated archive of the computational-linguistics community and has itself served as a122bibliometric corpus \citep{radev2013acl}. Its BibTeX export with abstracts (\texttt{anthology+abstracts.bib.gz},12346.7~MB, SHA-256 recorded) was downloaded on 2026-11-02 and parsed into a table of 138{,}796 entries, of which124\LinkAclRecords{} are conference papers or journal articles and \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in1252018--2026. The Anthology plays a diagnostic role: because its venue structure is complete and hand-maintained,126the share of its entries that the backbone can be matched to is a direct measure of OpenAlex coverage for a127subfield that publishes overwhelmingly in conference proceedings. It also supplies abstracts for those entries128in which OpenAlex has none. The export is licensed CC~BY~4.0.129130\subsection{Crossref, Semantic Scholar and other sources}131132Crossref \citep{hendricks2020crossref} is the registration agency behind most DOIs in the corpus and offers133open metadata through a REST API with a polite pool. It is not used to build the corpus, whose backbone134already carries Crossref-derived metadata through OpenAlex, but for three verification purposes: resolution of135every DOI cited in this manuscript, a cross-check of publisher-assigned document types on samples, and a136cross-check of citation counts (\texttt{is-referenced-by-count}) on a sample of highly cited works in137Part~IX.138139The Semantic Scholar Academic Graph \citep{kinney2023s2} was listed in the mandate among the priority sources140because of two signals it alone provides, the influential-citation classification and citation intent. At the141time of the feasibility audit, anonymous requests to its API returned HTTP~429 with an instruction to apply for142a key, and the bulk-dataset endpoint requires a key outright. In keeping with the rule that inaccessible data143be recorded rather than circumvented, Semantic Scholar is recorded as unavailable (decision D3) and no analysis144in this manuscript uses influential-citation or citation-intent signals; where Part~VIII discusses the145strength of citation links, it does so with OpenAlex reference lists and the knowledge-graph extraction of146Part~IV, whose limits are stated there. The Microsoft Academic Graph \citep{wang2020mag}, from which OpenAlex147inherited much of its historical coverage, was discontinued at the end of 2021 and is not an independent source148for the period studied. Kaggle credentials for the arXiv snapshot distributed there were absent and were not149needed, OAI-PMH being sufficient. Benchmark repositories and code-hosting platforms enter only in the150reproducibility and benchmark analyses scoped in Part~IV and are not part of the bibliographic corpus.151152Table~\ref{tab:data-sources} summarises the sources, the access route, the licence and the role of each.153154\begin{table}[htbp]155\centering\small156\caption{Information sources of the WP44 corpus (all accessed 2026-11-02/03).}157\label{tab:data-sources}158\begin{tabularx}{\textwidth}{lXll}159\toprule160Source & Access and snapshot & Licence & Role \\161\midrule162OpenAlex works snapshot & public S3 bucket, release 2026-10-14, 1{,}898 JSONL files, \LinkSlimTotal{} works & CC0 & backbone: identifiers, dates, types, topics, authorships, references, abstracts, OA status \\163OpenAlex REST API & no key; USD~0.10 per day shared budget; increment 2026-10-01 to 2026-11-02; single lookups & CC0 & post-snapshot increment; reference verification \\164DBLP & XML dump 2026-11-02 22:39~UTC, MD5 verified & ODC-BY~1.0 & computer-science scope; venue streams; author identifiers \\165arXiv & OAI-PMH set \texttt{cs}, 2018-01 to 2026-11, raw pages kept & arXiv metadata terms & preprint layer; first-submission dates; categories \\166ACL Anthology & BibTeX export with abstracts, 2026-11-02 & CC~BY~4.0 & coverage check for NLP venues; abstracts \\167Crossref & REST API, polite pool & open metadata & verification of DOIs, types and citation counts \\168Semantic Scholar & API key required; anonymous calls refused & --- & \emph{unavailable, recorded} \\169\bottomrule170\end{tabularx}171\end{table}172173\section{Identification: what each source contributes}174\label{sec:data-identification}175176There is no keyword search in this study. A mapping of an entire discipline cannot be defined by a query177string without importing the vocabulary of whoever wrote it; it is defined instead by database scope filters178that are themselves documented and reproducible, which is the strategy recommended for mapping studies in179software engineering and adopted here for computer science as a whole180\citep{petersen2008mapping,petersen2015mapping}. The identification stage (S0 in the notation of the research181plan) therefore consists of four scope filters, one per source, applied to immutable snapshots whose dates and182hashes are recorded in \texttt{data/raw/README.md}.183184\subsection{Scope filters}185186The OpenAlex filter retains every work with at least one topic, primary or secondary, in field~17 and a187publication year in $[2018, 2026]$. The choice of \emph{any} topic rather than the primary topic is deliberate188and conservative at this stage: a work whose primary topic lies in, say, statistics or electrical engineering189but which carries a secondary computer-science topic is a candidate, and the decision whether it belongs to190the core or to the penumbra is deferred to the eligibility stage, where it is made explicit and counted191(Section~\ref{sec:data-core}). The filter yields192\PrismaIdentifiedOpenalexCsTopicTwoZeroOneEightTwoZeroTwoSixPassOne{} records in the first pass over the193snapshot. The DBLP filter retains the publication records of type article, inproceedings, incollection, book,194phdthesis and data with a year in $[2018, 2026]$: \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{} records.195The arXiv filter retains the records of set \texttt{cs} whose first submission falls in the window:196\PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{} records. The ACL filter retains the conference papers197and journal articles of the Anthology with a year in the window:198\PrismaIdentifiedAclTwoZeroOneEightTwoZeroTwoSix{} records. The four counts overlap heavily, since most DBLP199and ACL records and a majority of arXiv records also exist in OpenAlex; the overlap is resolved by record200linkage, not by summation, and no total of identified records is reported as if the sources were disjoint.201202\subsection{The OpenAlex topic classifier as a scope instrument}203204Because the OpenAlex filter relies on an automated topic classifier, its properties bear on everything205downstream. OpenAlex assigns to each work up to three topics from a fixed list of 4{,}516, organised in 252206subfields, 26 fields and four domains; the assignment is made by a model applied to the title, abstract and207venue, and the field label of a work is inherited from its topics rather than assigned directly. Two208consequences follow. A work can be labelled computer science because one of its topics belongs to field~17209even when its content is only marginally computational, and conversely a work in a computer-science venue can210receive no computer-science topic when its abstract is missing or its vocabulary is domain-specific. The first211error inflates the candidate set with what will be called the penumbra; the second removes genuine212computer-science works from the first-pass candidate set. DBLP is used to correct the second error, which is213why the linkage stage has two passes (Section~\ref{sec:data-linkage}). The literature on OpenAlex has so far214examined its coverage and metadata completeness rather than the accuracy of its topic labels215\citep{alperin2024openalex,delgadoquiros2024completeness}; the present study treats the labels as a scope216instrument whose errors are bounded by the DBLP and arXiv memberships and quantified in217Section~\ref{sec:data-quality}, not as ground truth.218219\subsection{Verification of the scale against the mandate}220221The mandate set planning targets of more than one million candidate records and more than one hundred222thousand deeply classified records, subject to availability. The feasibility audit established, before any223ingestion, that the OpenAlex API alone reported more than six million works with a computer-science primary224field in 2018--2026 and that the DBLP dump held more than four million records in the same years; the targets225were therefore attainable without recourse to any restricted source. The audit also established what was226\emph{not} attainable: a Semantic Scholar key, an OpenAlex key, Kaggle credentials and the AWS command-line227tool were all absent, and the ingestion was designed around their absence (HTTPS streaming of the public228bucket, basic paging of the public API). The record of this audit is \texttt{reports/PHASE0\_FEASIBILITY.md}.229230\section{Record linkage across sources}231\label{sec:data-linkage}232233The four candidate sets describe overlapping populations with different identifiers. Record linkage (stage S1)234establishes, for each DBLP, arXiv and ACL record, the OpenAlex work it describes, so that one node per work235can be formed with the union of the metadata. The methods are deterministic and ordered by reliability; the236probabilistic framework of \citet{fellegi1969linkage} is not needed because the available keys (DOI, arXiv237identifier) are close to exact, and the residual title-based step is restricted to blocks where it is238unambiguous. The implementation follows the standard decomposition of data matching into key normalisation,239blocking, comparison and classification \citep{christen2012matching}; Part~IV gives the formal statement of the240keys and of the precision-estimation design, and this section reports what was done and what it produced.241242\subsection{Keys and matching order}243244All keys are normalised before comparison. DOIs are lower-cased. arXiv identifiers are lower-cased and245stripped of their version suffix (\texttt{v2}, \texttt{v3}, \ldots), so that all versions of an e-print map to246one identifier. Titles are reduced to a normalised string by lower-casing and deleting every character that is247not a letter or digit; the normalised title is used only when it has at least 25 characters, is unique within248its source, and is unique within OpenAlex, which is the blocking rule that keeps the title step from249producing spurious links between generic titles.250251For DBLP, the order is: (i) DOI equality, (ii) arXiv identifier equality, restricted to records not yet linked252on either side, (iii) normalised title equality with a year difference of at most one, restricted to records253not yet linked on either side. The DOI step linked \LinkLinkDblpOaDoi{} records and the title step254\LinkLinkDblpOaTitleYear{}; the arXiv step linked none, because the extraction of arXiv URLs from the DBLP255\texttt{ee} fields returned no candidate (\LinkDblpWithArxiv{} records with an extracted identifier). The most256plausible reason is that DBLP now records the arXiv version of its CoRR entries as a DataCite DOI of the form257\texttt{10.48550/arXiv.*}, which the DOI step already handles; the step is kept in the pipeline because other258dumps may differ. In total \LinkDblpLinkedDistinct{} of the \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} DBLP259records in the window were linked to an OpenAlex work after the first pass, leaving \LinkDblpUnlinked{}260unlinked. The composition of the unlinked set is informative: \LinkDblpUnlinkedByTypeArticle{} journal261articles, \LinkDblpUnlinkedByTypeInproceedings{} conference papers, \LinkDblpUnlinkedByTypePhdthesis{}262doctoral theses, \LinkDblpUnlinkedByTypeData{} data records, \LinkDblpUnlinkedByTypeIncollection{} chapters263and \LinkDblpUnlinkedByTypeBook{} books. Of these, \LinkDblpUnlinkedWithDoiInSlim{} carry a DOI that264\emph{does} exist in the slim table of all OpenAlex works, which means the work is in OpenAlex but was not265caught by the computer-science topic filter; this is the second error of the topic classifier described above266and the reason for the second pass.267268For arXiv, the order is: (i) arXiv identifier equality against the identifier OpenAlex records for the work,269(ii) DOI equality for e-prints that carry a journal DOI, (iii) normalised title equality with a year270difference of at most two, the wider tolerance reflecting the typical lag between a preprint and its published271version \citep{lin2020preprints}. The steps linked \LinkLinkArxivOaArxivId{}, \LinkLinkArxivOaDoi{} and272\LinkLinkArxivOaTitleYear{} records respectively, for \LinkArxivLinkedDistinct{} distinct e-prints linked.273For the ACL Anthology, the order is: (i) DOI equality, (ii) equality between the Anthology identifier and the274identifier embedded in an \texttt{aclanthology.org} landing-page URL among the work's OpenAlex locations,275(iii) normalised title equality with a year difference of at most one. The steps linked276\LinkLinkAclOaDoi{} records by DOI and \LinkLinkAclOaTitleYear{} by title, with the URL step contributing the277remainder, for \LinkAclLinkedDistinctTwoZeroOneEightTwoZeroTwoSix{} distinct 2018--2026 entries linked out of278\LinkAclTwoZeroOneEightTwoZeroTwoSix{}; the residual is examined in Section~\ref{sec:data-quality} as a279measure of OpenAlex coverage of conference-heavy subfields.280281\subsection{The second snapshot pass}282283A DBLP record whose DOI resolves to an OpenAlex work outside the topic filter is, by the scope rule, a284computer-science work that the first pass did not load. The same situation arises for arXiv e-prints whose285journal DOI exists in OpenAlex without a computer-science topic. The linkage script therefore writes the list286of OpenAlex identifiers that are needed but not loaded: \LinkNeededOaIdsSecondPassDblpDoi{} identifiers from287DBLP DOIs and \LinkNeededOaIdsSecondPassArxivDoi{} from arXiv DOIs. A second pass over the works snapshot,288restricted to this identifier list, retrieves the full records of these works; they are normalised into the289same Parquet tables with a \texttt{\_p2} suffix, the DuckDB views are rebuilt over the union of both passes,290and linkage is re-run so that the DBLP-linked set gains the works that the topic classifier had missed. The291second pass is what makes the DBLP membership rule of Section~\ref{sec:data-core} operational: without it,292``DBLP-indexed'' would silently mean ``DBLP-indexed and labelled computer science by OpenAlex'', and the DBLP293correction would be void precisely where it matters. DBLP records with no OpenAlex counterpart after both294passes (\PrismaLinkedDblpOnlyNoOpenalexRecord{} records before the second pass) are kept in a separate table295with their year and type and are reported as such; they enter the volume counts of Part~IX only in the296sensitivity analysis that adds DBLP-only records, because they lack the abstracts, references and affiliations297on which every other indicator depends. Most of them are journal articles and conference papers with a DOI298that OpenAlex does not hold at all, together with doctoral theses, which DBLP indexes from university299repositories that OpenAlex covers unevenly.300301\subsection{Precision of the linkage}302303Exact-key links are correct whenever the key is correct in both sources; the known failure modes are DOIs304reassigned by publishers between versions, DOIs registered for a proceedings volume rather than a paper, and305e-prints whose OpenAlex record points to a different version. Title-based links can fail on near-identical306titles of distinct works (extended abstracts and full papers, workshop and conference versions). The protocol307pre-specifies the estimation of linkage precision on a sample of 500 DBLP--OpenAlex pairs stratified by method,308with automatic agreement checks (normalised-title similarity, year difference, first-author surname) and a309200-pair subsample reserved for the principal investigator (checkpoint H-1). The sampling script draws 250 DOI310links, 200 title links and 50 arXiv links with a fixed seed; since the arXiv step produced no links in this311dump, the realised sample has 450 pairs, and the precision estimate for the arXiv step is undefined rather312than reported as perfect. At the time of writing the sampling script exists but the sample has not been drawn and the313human checkpoint is unsigned; the automatic agreement rates and the human verdicts will be reported in Part~IX314and Appendix~B, and every conclusion that depends on the title-based links is labelled as pending human review315until H-1 is signed.316317