WP44: bibliography merge script (599 entries, 15 duplicate keys, 0 DOI conflicts)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 4 h ago (Nov 3, 2026) parent 96bb3ab Browse files

5 changed files +566 −2

modified .gitignore +2 −0

				`@@ -52,3 +52,5 @@ manuscript/*.bbl`
52	manuscript/*.blg	52	manuscript/*.blg
53	manuscript/*.synctex.gz	53	manuscript/*.synctex.gz
54	.venv/	54	.venv/
		55	+references_all.bib
		56	+manuscript/references_merge_report.txt

modified manuscript/main.tex +1 −1

				`@@ -82,7 +82,7 @@`
82	\input{sections/part11_discussion}	82	\input{sections/part11_discussion}
83		83
84	\clearpage	84	\clearpage
85	−\bibliography{references}	85	+\bibliography{references_all}
86		86
87	\begin{appendices}	87	\begin{appendices}
88	\input{appendices/appA_taxonomy}	88	\input{appendices/appA_taxonomy}

added manuscript/merge_bibs.py +31 −0

++#!/usr/bin/env python3
++"""Merge manuscript/references.bib (main, Agent B-lit) with the writers' references_part*.bib into references_all.bib.
++First occurrence wins (main bib first); duplicate keys are reported in references_merge_report.txt with a flag when the
++DOI differs (a real conflict to resolve by hand). main.tex uses \\bibliography{references_all}."""
++import glob, re, os
++H = os.path.dirname(os.path.abspath(__file__))
++files = [os.path.join(H, "references.bib")] + sorted(glob.glob(os.path.join(H, "references_part*.bib")))
++seen = {}; out = []; dups = []
++for f in files:
++    txt = open(f, encoding="utf-8").read()
++    for m in re.finditer(r"@(\w+)\s*\{\s*([^,\s]+)\s*,", txt):
++        start = m.start(); depth = 0; i = m.end() - 1
++        # find matching closing brace of the entry
++        j = txt.find("{", start); depth = 0; k = j
++        while k < len(txt):
++            if txt[k] == "{": depth += 1
++            elif txt[k] == "}":
++                depth -= 1
++                if depth == 0: break
++            k += 1
++        entry = txt[start:k + 1]; key = m.group(2)
++        doi = re.search(r"doi\s*=\s*[{\"]([^}\"]+)", entry, re.I)
++        doi = doi.group(1).strip().lower() if doi else None
++        if key in seen:
++            dups.append((key, os.path.basename(f), "DOI_CONFLICT" if (doi and seen[key] and doi != seen[key]) else "same"))
++            continue
++        seen[key] = doi; out.append(entry)
++open(os.path.join(H, "references_all.bib"), "w", encoding="utf-8").write("% merged by merge_bibs.py — do not edit; edit the source bib files\n\n" + "\n\n".join(out) + "\n")
++with open(os.path.join(H, "references_merge_report.txt"), "w") as fh:
++    fh.write(f"entries: {len(out)} from {len(files)} files\n"); [fh.write(f"DUP {k} in {f}: {st}\n") for k, f, st in dups]
++print(len(out), "entries;", len(dups), "duplicates;", sum(1 for d in dups if d[2] == "DOI_CONFLICT"), "DOI conflicts")

modified manuscript/sections/part03_data.tex +316 −1

++% ============================================================================
++% Part III — Data and Systematic Review Protocol
++% Written by Agent L (Part III instance), 2026-11-03. Every corpus number is a macro from tables/numbers.tex
++% (generated by tables/make_numbers.py from data/metadata/prisma_counts.json and linkage_report.json).
++% Sources of truth: RESEARCH_PLAN.md §2–§3, protocol/PROTOCOL.md, protocol/amendments.csv,
++% reports/PHASE0_FEASIBILITY.md, data/raw/README.md, ingestion/*.py, systematic_review/define_corpus.py.
++% ============================================================================
++
++\section{Information sources and conditions of access}
++\label{sec:data-sources}
++
++A field-level mapping study stands or falls with the bibliographic databases on which it is built. The choice
++of sources determines which works can be observed at all, how reliably they can be dated and typed, whether
++citation links are available, and under what licence derived data may be redistributed. This part documents
++these choices for WP44 and reports, stage by stage, how the mapping corpus was assembled from the raw
++sources. The exposition follows the order of the pipeline (identification, linkage, deduplication,
++eligibility, scope, dating) and closes with the quality indicators and the research-integrity procedures that
++govern every later analysis. The reporting structure is modelled on the PRISMA 2020 items that apply to a
++mapping study \citep{page2021prisma,page2021prismaee}; items concerned with risk of bias and effect measures
++have no counterpart in a bibliometric design and are replaced, as PRISMA-ScR does for scoping reviews
++\citep{tricco2018prismascr}, by indicators of metadata quality and coverage (Section~\ref{sec:data-quality}).
++The search-reporting recommendations of PRISMA-S \citep{rethlefsen2021prismas} are followed to the extent
++that a database-scope strategy without keyword queries permits.
++
++\subsection{Why a multi-source design}
++
++No single database indexes computer science completely, and the gaps are not random. The venue-centred
++coverage of DBLP, the publisher-centred coverage of Crossref, the preprint-centred coverage of arXiv and the
++aggregating coverage of OpenAlex each miss a distinct part of the literature and type the part they do cover
++in distinct ways. Comparative studies of bibliographic data sources have documented large differences in
++coverage between Web of Science, Scopus, Dimensions, Crossref and Microsoft Academic
++\citep{visser2021comparison,martinmartin2021comparison}, and the journal-level coverage of the two commercial
++indexes is known to under-represent conference proceedings, which carry a large share of computer-science
++output \citep{mongeon2016coverage}. For the open successor of Microsoft Academic, OpenAlex, the first
++independent assessments report strong coverage of works with DOIs together with weaker and uneven metadata on
++affiliations, document types and references \citep{alperin2024openalex,culbert2025openalex,
++scheidsteger2023openalex,delgadoquiros2024completeness}. These findings motivate the design adopted here: one
++source supplies the backbone (OpenAlex), a second supplies the definition of what counts as computer science
++(DBLP), a third supplies the preprint layer and its dates (arXiv), and a fourth checks the coverage of a
++subfield with a dense, curated venue structure (the ACL Anthology). Crossref is used for verification only.
++
++\subsection{OpenAlex}
++
++OpenAlex \citep{priem2022openalex} is the backbone of the corpus. It supplies, for every work, the
++identifiers (OpenAlex~W identifier, DOI, arXiv identifier where present), the publication date and type, the
++title and an inverted-index abstract where available, the venue (source) and its type, the authorships with
++institutions and countries, the topic assignment at three levels (field, subfield, topic) with a primary
++topic, the list of referenced works (the citation graph), the open-access status and licence, and the
++field-weighted citation impact together with yearly citation counts. OpenAlex is released under the CC0
++public-domain dedication, which permits redistribution of record-level derived data.
++
++Two access routes exist and both were used, for different purposes. The public snapshot is a versioned bulk
++release in gzipped JSON Lines, published in a public S3 bucket; the release used here is dated
++2026-10-14 and comprises 1{,}898 part files for the \emph{works} entity, 675.3~GB compressed, with a manifest
++listing the record count of each file. The snapshot was streamed from the primary compute node, each file's
++content length checked against the manifest and a SHA-256 of the compressed bytes recorded in a SQLite ledger
++before parsing; every one of the 1{,}898 files parsed without error and the sum of the per-file record counts
++equals the manifest total of \LinkSlimTotal{} works. Two outputs were written during this single pass. The
++first is the full JSON record of every work carrying \emph{any} topic in OpenAlex field~17 (Computer Science)
++with a publication year between 2018 and 2026, stored as Zstandard-compressed JSON Lines and partitioned by
++the snapshot's own date partitions (\LinkOaCsRecords{} records, about 20~GB). The second is a \emph{slim}
++table of every work in the snapshot (identifier, DOI, year, type, primary topic, subfield, field and domain,
++citation count and a few flags), kept in Parquet so that works cited by computer-science papers but published
++in other fields can be resolved for the interdisciplinary analyses of Part~VIII (decision D2 of
++\texttt{RESEARCH\_PLAN.md}). Records flagged by OpenAlex as XPAC expansion records are retained at this stage
++with their flag so that the exclusion decision can be taken, and counted, later (Section~\ref{sec:data-eligibility}).
++The smaller entity snapshots (topics, subfields, fields, domains, sources, institutions, publishers, funders,
++concepts) were downloaded the same day with a per-file SHA-256 ledger; author entities were not downloaded
++because the authorship data embedded in each work record suffice for the analyses planned.
++
++The second route is the REST API. Without an API key the public endpoint operates under a shared daily budget
++of USD~0.10, list-and-filter calls are priced at USD~0.0001 each, and cursor paging is unavailable, so that a
++single filtered query cannot return more than 10{,}000 results through basic paging. Single-record lookups by
++identifier are free. The API was therefore used for two narrowly defined tasks. First, an incremental harvest
++of works with any computer-science topic and a publication date between 2026-10-01 and 2026-11-02, intended
++to capture records added after the snapshot was cut; the harvest is sliced by publication day and, where a day
++exceeds the paging limit, split into records with a computer-science primary field and records without one.
++Raw JSON pages are stored verbatim under a dated directory with a ledger of counts and cost, and the job
++resumes on the following day when the budget is exhausted. Second, free single-record lookups by DOI serve as
++one of the verification routes for the bibliography of this manuscript (Section~\ref{sec:data-integrity}).
++A free API key, which would raise the daily budget, requires the principal investigator to act on an e-mail
++and was deliberately not obtained without instruction; the mandate forbids assuming credentials that have not
++been tested.
++
++\subsection{DBLP}
++
++The DBLP computer science bibliography \citep{ley2002dblp,ley2009dblp} is a curated, venue-centred index that
++has for three decades defined, in practice, the set of journals and conference series regarded as computer
++science. It is used here in that capacity: a work indexed by DBLP is in computer science by construction, and
++DBLP venue streams feed the taxonomy of Part~IV. The complete XML dump (\texttt{dblp.xml.gz}, dated 2026-11-02
++22:39~UTC, 1.12~GB) was downloaded together with its DTD and MD5 checksum, the checksum verified, and the file
++parsed with a validating XML parser into one row per publication record (articles, conference papers,
++proceedings volumes, books, book chapters, doctoral and master's theses, data records; the \texttt{www}
++person records are skipped). The whole archive was kept so that per-year counts are available for the
++historical baseline and for context. The DBLP dump is released under the Open Data Commons Attribution licence
++(ODC-BY~1.0). DBLP contributes, in addition to the scope definition, author person identifiers, venue streams,
++and DOI links for most records; the \texttt{ee} fields were also scanned for arXiv URLs, with a result reported
++in Section~\ref{sec:data-linkage}.
++
++\subsection{arXiv}
++
++The arXiv e-print repository \citep{ginsparg2011arxiv} is the main preprint channel of computer science and,
++for the period under study, the place where a large share of the most cited work first appeared. Its
++metadata are harvested through the OAI-PMH interface (\texttt{set=cs}, \texttt{metadataPrefix=arXiv}) in
++monthly datestamp windows from January 2018 to 2 November 2026. The harvester waits at least 3.5~s between
++requests and honours \texttt{Retry-After} headers; every raw XML page is kept verbatim as the immutable
++snapshot and parsed afterwards into a single Parquet table. Two properties of OAI-PMH matter for what follows.
++The \texttt{from}/\texttt{until} parameters filter on the record's \emph{datestamp}, the time of its last
++update, so that a monthly window contains every record updated in that month, not every record first
++submitted then; duplicates across windows are resolved at parse time with the latest datestamp winning. The
++\texttt{created} element gives the first submission date, which is the date used throughout this study for
++preprints. The harvest returned \LinkArxivCsRecords{} distinct records, of which
++\LinkArxivCreatedTwoZeroOneEightTwoZeroTwoSix{} were first submitted in 2018--2026; the remainder are older
++e-prints updated during the window. The arXiv metadata are released for reuse under the repository's metadata
++terms; e-prints themselves are governed by per-paper licences and are downloaded, for the high-priority set
++only, within those licences and at the repository's rate limits. The use of arXiv as a research dataset, and
++the caveats attached to its category labels and versioning, are discussed by \citet{clement2019arxiv}.
++
++\subsection{ACL Anthology}
++
++The ACL Anthology is the curated archive of the computational-linguistics community and has itself served as a
++bibliometric corpus \citep{radev2013acl}. Its BibTeX export with abstracts (\texttt{anthology+abstracts.bib.gz},
++46.7~MB, SHA-256 recorded) was downloaded on 2026-11-02 and parsed into a table of 138{,}796 entries, of which
++\LinkAclRecords{} are conference papers or journal articles and \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in
++2018--2026. The Anthology plays a diagnostic role: because its venue structure is complete and hand-maintained,
++the share of its entries that the backbone can be matched to is a direct measure of OpenAlex coverage for a
++subfield that publishes overwhelmingly in conference proceedings. It also supplies abstracts for those entries
++in which OpenAlex has none. The export is licensed CC~BY~4.0.
++
++\subsection{Crossref, Semantic Scholar and other sources}
++
++Crossref \citep{hendricks2020crossref} is the registration agency behind most DOIs in the corpus and offers
++open metadata through a REST API with a polite pool. It is not used to build the corpus, whose backbone
++already carries Crossref-derived metadata through OpenAlex, but for three verification purposes: resolution of
++every DOI cited in this manuscript, a cross-check of publisher-assigned document types on samples, and a
++cross-check of citation counts (\texttt{is-referenced-by-count}) on a sample of highly cited works in
++Part~IX.
++
++The Semantic Scholar Academic Graph \citep{kinney2023s2} was listed in the mandate among the priority sources
++because of two signals it alone provides, the influential-citation classification and citation intent. At the
++time of the feasibility audit, anonymous requests to its API returned HTTP~429 with an instruction to apply for
++a key, and the bulk-dataset endpoint requires a key outright. In keeping with the rule that inaccessible data
++be recorded rather than circumvented, Semantic Scholar is recorded as unavailable (decision D3) and no analysis
++in this manuscript uses influential-citation or citation-intent signals; where Part~VIII discusses the
++strength of citation links, it does so with OpenAlex reference lists and the knowledge-graph extraction of
++Part~IV, whose limits are stated there. The Microsoft Academic Graph \citep{wang2020mag}, from which OpenAlex
++inherited much of its historical coverage, was discontinued at the end of 2021 and is not an independent source
++for the period studied. Kaggle credentials for the arXiv snapshot distributed there were absent and were not
++needed, OAI-PMH being sufficient. Benchmark repositories and code-hosting platforms enter only in the
++reproducibility and benchmark analyses scoped in Part~IV and are not part of the bibliographic corpus.
++
++Table~\ref{tab:data-sources} summarises the sources, the access route, the licence and the role of each.
++
++\begin{table}[htbp]
++\centering\small
++\caption{Information sources of the WP44 corpus (all accessed 2026-11-02/03).}
++\label{tab:data-sources}
++\begin{tabularx}{\textwidth}{lXll}
++\toprule
++Source & Access and snapshot & Licence & Role \\
++\midrule
++OpenAlex works snapshot & public S3 bucket, release 2026-10-14, 1{,}898 JSONL files, \LinkSlimTotal{} works & CC0 & backbone: identifiers, dates, types, topics, authorships, references, abstracts, OA status \\
++OpenAlex REST API & no key; USD~0.10 per day shared budget; increment 2026-10-01 to 2026-11-02; single lookups & CC0 & post-snapshot increment; reference verification \\
++DBLP & XML dump 2026-11-02 22:39~UTC, MD5 verified & ODC-BY~1.0 & computer-science scope; venue streams; author identifiers \\
++arXiv & OAI-PMH set \texttt{cs}, 2018-01 to 2026-11, raw pages kept & arXiv metadata terms & preprint layer; first-submission dates; categories \\
++ACL Anthology & BibTeX export with abstracts, 2026-11-02 & CC~BY~4.0 & coverage check for NLP venues; abstracts \\
++Crossref & REST API, polite pool & open metadata & verification of DOIs, types and citation counts \\
++Semantic Scholar & API key required; anonymous calls refused & --- & \emph{unavailable, recorded} \\
++\bottomrule
++\end{tabularx}
++\end{table}
++
++\section{Identification: what each source contributes}
++\label{sec:data-identification}
++
++There is no keyword search in this study. A mapping of an entire discipline cannot be defined by a query
++string without importing the vocabulary of whoever wrote it; it is defined instead by database scope filters
++that are themselves documented and reproducible, which is the strategy recommended for mapping studies in
++software engineering and adopted here for computer science as a whole
++\citep{petersen2008mapping,petersen2015mapping}. The identification stage (S0 in the notation of the research
++plan) therefore consists of four scope filters, one per source, applied to immutable snapshots whose dates and
++hashes are recorded in \texttt{data/raw/README.md}.
++
++\subsection{Scope filters}
++
++The OpenAlex filter retains every work with at least one topic, primary or secondary, in field~17 and a
++publication year in $[2018, 2026]$. The choice of \emph{any} topic rather than the primary topic is deliberate
++and conservative at this stage: a work whose primary topic lies in, say, statistics or electrical engineering
++but which carries a secondary computer-science topic is a candidate, and the decision whether it belongs to
++the core or to the penumbra is deferred to the eligibility stage, where it is made explicit and counted
++(Section~\ref{sec:data-core}). The filter yields
++\PrismaIdentifiedOpenalexCsTopicTwoZeroOneEightTwoZeroTwoSixPassOne{} records in the first pass over the
++snapshot. The DBLP filter retains the publication records of type article, inproceedings, incollection, book,
++phdthesis and data with a year in $[2018, 2026]$: \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{} records.
++The arXiv filter retains the records of set \texttt{cs} whose first submission falls in the window:
++\PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{} records. The ACL filter retains the conference papers
++and journal articles of the Anthology with a year in the window:
++\PrismaIdentifiedAclTwoZeroOneEightTwoZeroTwoSix{} records. The four counts overlap heavily, since most DBLP
++and ACL records and a majority of arXiv records also exist in OpenAlex; the overlap is resolved by record
++linkage, not by summation, and no total of identified records is reported as if the sources were disjoint.
++
++\subsection{The OpenAlex topic classifier as a scope instrument}
++
++Because the OpenAlex filter relies on an automated topic classifier, its properties bear on everything
++downstream. OpenAlex assigns to each work up to three topics from a fixed list of 4{,}516, organised in 252
++subfields, 26 fields and four domains; the assignment is made by a model applied to the title, abstract and
++venue, and the field label of a work is inherited from its topics rather than assigned directly. Two
++consequences follow. A work can be labelled computer science because one of its topics belongs to field~17
++even when its content is only marginally computational, and conversely a work in a computer-science venue can
++receive no computer-science topic when its abstract is missing or its vocabulary is domain-specific. The first
++error inflates the candidate set with what will be called the penumbra; the second removes genuine
++computer-science works from the first-pass candidate set. DBLP is used to correct the second error, which is
++why the linkage stage has two passes (Section~\ref{sec:data-linkage}). The literature on OpenAlex has so far
++examined its coverage and metadata completeness rather than the accuracy of its topic labels
++\citep{alperin2024openalex,delgadoquiros2024completeness}; the present study treats the labels as a scope
++instrument whose errors are bounded by the DBLP and arXiv memberships and quantified in
++Section~\ref{sec:data-quality}, not as ground truth.
++
++\subsection{Verification of the scale against the mandate}
++
++The mandate set planning targets of more than one million candidate records and more than one hundred
++thousand deeply classified records, subject to availability. The feasibility audit established, before any
++ingestion, that the OpenAlex API alone reported more than six million works with a computer-science primary
++field in 2018--2026 and that the DBLP dump held more than four million records in the same years; the targets
++were therefore attainable without recourse to any restricted source. The audit also established what was
++\emph{not} attainable: a Semantic Scholar key, an OpenAlex key, Kaggle credentials and the AWS command-line
++tool were all absent, and the ingestion was designed around their absence (HTTPS streaming of the public
++bucket, basic paging of the public API). The record of this audit is \texttt{reports/PHASE0\_FEASIBILITY.md}.
++
++\section{Record linkage across sources}
++\label{sec:data-linkage}
++
++The four candidate sets describe overlapping populations with different identifiers. Record linkage (stage S1)
++establishes, for each DBLP, arXiv and ACL record, the OpenAlex work it describes, so that one node per work
++can be formed with the union of the metadata. The methods are deterministic and ordered by reliability; the
++probabilistic framework of \citet{fellegi1969linkage} is not needed because the available keys (DOI, arXiv
++identifier) are close to exact, and the residual title-based step is restricted to blocks where it is
++unambiguous. The implementation follows the standard decomposition of data matching into key normalisation,
++blocking, comparison and classification \citep{christen2012matching}; Part~IV gives the formal statement of the
++keys and of the precision-estimation design, and this section reports what was done and what it produced.
++
++\subsection{Keys and matching order}
++
++All keys are normalised before comparison. DOIs are lower-cased. arXiv identifiers are lower-cased and
++stripped of their version suffix (\texttt{v2}, \texttt{v3}, \ldots), so that all versions of an e-print map to
++one identifier. Titles are reduced to a normalised string by lower-casing and deleting every character that is
++not a letter or digit; the normalised title is used only when it has at least 25 characters, is unique within
++its source, and is unique within OpenAlex, which is the blocking rule that keeps the title step from
++producing spurious links between generic titles.
++
++For DBLP, the order is: (i) DOI equality, (ii) arXiv identifier equality, restricted to records not yet linked
++on either side, (iii) normalised title equality with a year difference of at most one, restricted to records
++not yet linked on either side. The DOI step linked \LinkLinkDblpOaDoi{} records and the title step
++\LinkLinkDblpOaTitleYear{}; the arXiv step linked none, because the extraction of arXiv URLs from the DBLP
++\texttt{ee} fields returned no candidate (\LinkDblpWithArxiv{} records with an extracted identifier). The most
++plausible reason is that DBLP now records the arXiv version of its CoRR entries as a DataCite DOI of the form
++\texttt{10.48550/arXiv.*}, which the DOI step already handles; the step is kept in the pipeline because other
++dumps may differ. In total \LinkDblpLinkedDistinct{} of the \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} DBLP
++records in the window were linked to an OpenAlex work after the first pass, leaving \LinkDblpUnlinked{}
++unlinked. The composition of the unlinked set is informative: \LinkDblpUnlinkedByTypeArticle{} journal
++articles, \LinkDblpUnlinkedByTypeInproceedings{} conference papers, \LinkDblpUnlinkedByTypePhdthesis{}
++doctoral theses, \LinkDblpUnlinkedByTypeData{} data records, \LinkDblpUnlinkedByTypeIncollection{} chapters
++and \LinkDblpUnlinkedByTypeBook{} books. Of these, \LinkDblpUnlinkedWithDoiInSlim{} carry a DOI that
++\emph{does} exist in the slim table of all OpenAlex works, which means the work is in OpenAlex but was not
++caught by the computer-science topic filter; this is the second error of the topic classifier described above
++and the reason for the second pass.
++
++For arXiv, the order is: (i) arXiv identifier equality against the identifier OpenAlex records for the work,
++(ii) DOI equality for e-prints that carry a journal DOI, (iii) normalised title equality with a year
++difference of at most two, the wider tolerance reflecting the typical lag between a preprint and its published
++version \citep{lin2020preprints}. The steps linked \LinkLinkArxivOaArxivId{}, \LinkLinkArxivOaDoi{} and
++\LinkLinkArxivOaTitleYear{} records respectively, for \LinkArxivLinkedDistinct{} distinct e-prints linked.
++For the ACL Anthology, the order is: (i) DOI equality, (ii) equality between the Anthology identifier and the
++identifier embedded in an \texttt{aclanthology.org} landing-page URL among the work's OpenAlex locations,
++(iii) normalised title equality with a year difference of at most one. The steps linked
++\LinkLinkAclOaDoi{} records by DOI and \LinkLinkAclOaTitleYear{} by title, with the URL step contributing the
++remainder, for \LinkAclLinkedDistinctTwoZeroOneEightTwoZeroTwoSix{} distinct 2018--2026 entries linked out of
++\LinkAclTwoZeroOneEightTwoZeroTwoSix{}; the residual is examined in Section~\ref{sec:data-quality} as a
++measure of OpenAlex coverage of conference-heavy subfields.
++
++\subsection{The second snapshot pass}
++
++A DBLP record whose DOI resolves to an OpenAlex work outside the topic filter is, by the scope rule, a
++computer-science work that the first pass did not load. The same situation arises for arXiv e-prints whose
++journal DOI exists in OpenAlex without a computer-science topic. The linkage script therefore writes the list
++of OpenAlex identifiers that are needed but not loaded: \LinkNeededOaIdsSecondPassDblpDoi{} identifiers from
++DBLP DOIs and \LinkNeededOaIdsSecondPassArxivDoi{} from arXiv DOIs. A second pass over the works snapshot,
++restricted to this identifier list, retrieves the full records of these works; they are normalised into the
++same Parquet tables with a \texttt{\_p2} suffix, the DuckDB views are rebuilt over the union of both passes,
++and linkage is re-run so that the DBLP-linked set gains the works that the topic classifier had missed. The
++second pass is what makes the DBLP membership rule of Section~\ref{sec:data-core} operational: without it,
++``DBLP-indexed'' would silently mean ``DBLP-indexed and labelled computer science by OpenAlex'', and the DBLP
++correction would be void precisely where it matters. DBLP records with no OpenAlex counterpart after both
++passes (\PrismaLinkedDblpOnlyNoOpenalexRecord{} records before the second pass) are kept in a separate table
++with their year and type and are reported as such; they enter the volume counts of Part~IX only in the
++sensitivity analysis that adds DBLP-only records, because they lack the abstracts, references and affiliations
++on which every other indicator depends. Most of them are journal articles and conference papers with a DOI
++that OpenAlex does not hold at all, together with doctoral theses, which DBLP indexes from university
++repositories that OpenAlex covers unevenly.
++
++\subsection{Precision of the linkage}
++
++Exact-key links are correct whenever the key is correct in both sources; the known failure modes are DOIs
++reassigned by publishers between versions, DOIs registered for a proceedings volume rather than a paper, and
++e-prints whose OpenAlex record points to a different version. Title-based links can fail on near-identical
++titles of distinct works (extended abstracts and full papers, workshop and conference versions). The protocol
++pre-specifies the estimation of linkage precision on a sample of 500 DBLP--OpenAlex pairs stratified by method,
++with automatic agreement checks (normalised-title similarity, year difference, first-author surname) and a
++200-pair subsample reserved for the principal investigator (checkpoint H-1). The sampling script draws 250 DOI
++links, 200 title links and 50 arXiv links with a fixed seed; since the arXiv step produced no links in this
++dump, the realised sample has 450 pairs, and the precision estimate for the arXiv step is undefined rather
++than reported as perfect. At the time of writing the sampling script exists but the sample has not been drawn and the
++human checkpoint is unsigned; the automatic agreement rates and the human verdicts will be reported in Part~IX
++and Appendix~B, and every conclusion that depends on the title-based links is labelled as pending human review
++until H-1 is signed.
++

modified manuscript/sections/part04_methodology.tex +216 −0

+ have lower recall for the keyword-dependent categories; the share of such
+ works is reported by year because it is not constant.
++\section{Bibliometric indicators}
++\label{sec:m-bibliometrics}
++
++The indicators of this section are the estimands of research questions RQ1,
++RQ3 and RQ6. They are standard in the scientometric literature, and the
++definitions follow the review of \citet{waltman2016review} where a choice had
++to be made; what is specific to this study is the citation-window discipline
++imposed by the eight-year span of the corpus and the explicit treatment of the
++two counting conventions. The scripts are \texttt{bibliometrics/01\_volume\_series.py},
++\texttt{02\_collaboration.py} and \texttt{03\_citation\_impact.py}.
++
++\subsection{Volume, composition and growth}
++
++The monthly count $N_t$ and the category counts $N_{k,t}$ of
++equation~\eqref{eq:counts} are tabulated by first month, period, date precision,
++type and source membership (DBLP-indexed, OpenAlex-primary, arXiv-linked), with
++the numbers of works having an abstract, having at least one resolvable
++reference, and being open access; the annual table adds the mean number of
++authors, references and countries per work. Composition is the share
++$s_{k,t}$; its annual counterpart uses the primary label on all eligible works.
++Growth between two dates separated by $\Delta$ years is summarised by the
++compound annual growth rate
++\begin{equation}
++  \mathrm{CAGR}(t_a, t_b) \;=\; \Bigl( \frac{N_{t_b}}{N_{t_a}} \Bigr)^{1/\Delta} - 1,
++  \qquad \Delta = (t_b - t_a)\ \text{in years},
++  \label{eq:cagr}
++\end{equation}
++applied to annual totals and to annual category totals; it is a descriptive
++summary of two end points and says nothing about the path between them, which
++is why the structural-change analysis of Section~\ref{sec:m-structural} works
++on the full monthly series instead. The CAGR is computed at table-making time
++from the annual counts rather than by a dedicated script; the plan lists it
++among the Phase-4 indicators and it is reported for complete years only, since
++2026 is right-censored (Section~\ref{sec:m-truncation}).
++
++\subsection{Fixed citation windows}
++\label{sec:m-citations}
++
++OpenAlex provides, for each work, the number of citations received in each
++calendar year (\texttt{counts\_by\_year}). Let $c_{w}(u)$ be the citations
++received by $w$ in calendar year $u$ and $y(w)$ its publication year. The
++fixed-window counts are
++\begin{equation}
++  c^{(12)}_w = \sum_{u \le y(w)+1} c_w(u), \qquad
++  c^{(36)}_w = \sum_{u \le y(w)+2} c_w(u), \qquad
++  c^{(48)}_w = \sum_{u \le y(w)+3} c_w(u),
++  \label{eq:windows}
++\end{equation}
++so that $c^{(12)}$ counts citations in the publication year and the following
++calendar year, $c^{(36)}$ in the publication year and the two following years,
++and so on. The superscripts are the names used in the code and in the tables
++and must be read as calendar-year windows rather than as exact month counts: a
++work published in January and one published in December of the same year
++receive windows that differ by eleven months in effective length. This
++granularity is imposed by the data (OpenAlex does not provide citation dates
++below the year) and is the same for the baseline and the post period, so it
++blurs comparisons within a year but does not bias comparisons between years.
++The window is the device that neutralises citation-age bias: comparing the
++total citation count of a 2019 work with that of a 2025 work compares seven
++years of exposure with one, whereas comparing $c^{(36)}$ compares equal
++exposures. The cost is that the last window-length of the corpus cannot be
++evaluated; $c^{(36)}$ is complete for publication years up to 2023 given a
++snapshot taken late in 2026, and $c^{(12)}$ up to 2024. Tables flag the years
++for which a window is incomplete, and the choice of window length follows the
++evidence in \citet{wang2013window} that short windows rank works materially
++differently from long ones in slow-moving fields, which is why two lengths are
++reported rather than one.
++
++\subsection{Field normalisation}
++
++Raw citation counts differ across subfields and document types by factors that
++have nothing to do with influence. The normalised citation score of a work is
++its window count divided by the mean window count of its reference class
++\citep{waltman2016review, waltman2019fieldnorm}:
++\begin{equation}
++  \mathrm{NCS}^{(h)}_w \;=\; \frac{c^{(h)}_w}{\bar c^{(h)}\bigl(y(w), \tau(w), f(w)\bigr)}, \qquad
++  \bar c^{(h)}(y, \tau, f) = \frac{1}{|\mathcal{W}_{y,\tau,f}|}\sum_{v \in \mathcal{W}_{y,\tau,f}} c^{(h)}_v ,
++  \label{eq:ncs}
++\end{equation}
++where $f(w)$ is the primary OpenAlex subfield of $w$ and $\mathcal{W}_{y,\tau,f}$
++the set of eligible works sharing year, type and subfield. The reference class
++is computed on the eligible corpus, not on all of OpenAlex, so the
++normalisation is relative to computer science as delimited here: an NCS of 2
++means twice the mean of comparable computer-science works, not twice the world
++mean. The score is undefined when the class mean is zero (very small classes in
++the most recent years) and is then left missing rather than set to zero.
++Normalisation by the mean of a heavy-tailed distribution is sensitive to a few
++very highly cited works in the class; the median-based alternatives discussed
++by \citet{waltman2016review} are not used because the median of $c^{(36)}$ is
++zero in many classes of this corpus, which would make them uninformative.
++
++OpenAlex's own field-weighted citation impact (FWCI) is kept as a secondary
++indicator. Its documentation defines it as ``citations received divided by
++citations expected for works of the same type, year, and subfield'', with a
++world average of one \citep{openalex2026fwci}. It differs from
++equation~\eqref{eq:ncs} in three respects: the expected count is computed over
++all of OpenAlex rather than over this corpus; the citation count is the total
++to date rather than a fixed window, so that FWCI values of recent works are
++based on a short and unequal exposure; and the computation is OpenAlex's,
++recomputed nightly, so that the value depends on the snapshot date. For these
++reasons FWCI is reported for comparison and never used to select works.
++
++\subsection{Inequality and concentration}
++
++Citation inequality within a publication year is measured by the Gini
++coefficient of $c^{(36)}$ among the $n$ eligible works of that year. With the
++works sorted so that $c_{(1)} \le c_{(2)} \le \dots \le c_{(n)}$, the estimator
++implemented in SQL is the rank form
++\begin{equation}
++  G \;=\; \frac{2\sum_{i=1}^{n} i\, c_{(i)}}{n \sum_{i=1}^{n} c_{(i)}} - \frac{n+1}{n},
++  \label{eq:gini}
++\end{equation}
++which equals one minus twice the area under the Lorenz curve and lies in
++$[0, 1 - 1/n]$; the coefficient is named after \citet{ceriani2012gini}'s
++account of its origin. Uncited works (a large share within a three-year
++window) contribute zero to the numerator and pull $G$ upward, so the share of
++uncited works is reported next to it. The protocol (H5) refers to the Gini of
++24-month citations; the implemented statistic uses the three-calendar-year
++window $c^{(36)}$, which is complete for one more publication year than the
++longer alternatives and less dominated by zeros than $c^{(12)}$. This is a
++deviation from the protocol wording and is recorded as such.
++
++Venue concentration within a year is the Herfindahl--Hirschman index of the
++distribution of works over sources,
++\begin{equation}
++  \mathrm{HHI}_y \;=\; \sum_{v} \sigma_{v,y}^2, \qquad
++  \sigma_{v,y} = \frac{N_{v,y}}{\sum_{v'} N_{v',y}},
++  \label{eq:hhi}
++\end{equation}
++where $N_{v,y}$ is the number of eligible works of year $y$ whose primary
++source is $v$ \citep{hirschman1964paternity}; works without a source are
++excluded from the denominator, and the number of distinct sources and the
++largest single share are reported with the index. Because arXiv is a single
++OpenAlex source, growth in preprints mechanically raises $\mathrm{HHI}_y$ even
++when the journal and conference landscape becomes more dispersed; this is the
++reason the protocol's H5 predicts a stable or falling HHI only ``because of
++preprint growth'' with the preprint source treated separately, and the tables
++report the index with and without the preprint type.
++
++\subsection{Collaboration, countries and institutions}
++
++Team size is summarised by the mean and median number of authors per work and
++by the shares of single-author works and of works with at least five or ten
++authors. Two counting conventions are used for countries and institutions,
++because they answer different questions \citep{waltman2016review}. Under full
++counting a work contributes one unit to every country among its authors'
++affiliations; under fractional counting it contributes $1/|K(w)|$ to each of
++its $|K(w)|$ distinct countries, so that the fractional totals sum to the
++number of works with at least one country. The same pair of conventions is
++applied to institutions, with the fraction taken over the distinct institutions
++of the work. The international collaboration share of year $y$ is
++\begin{equation}
++  \mathrm{ICS}_y \;=\; \frac{|\{w : y(w)=y,\ |K(w)| \ge 2\}|}{|\{w : y(w)=y,\ |K(w)| \ge 1\}|},
++\end{equation}
++that is, among works with at least one resolved country, the share with authors
++from two or more countries; works without affiliation data are excluded from
++the denominator and their share is reported, because affiliation completeness
++in OpenAlex varies by source and year and a change in completeness would
++otherwise masquerade as a change in collaboration. The industry share is the
++share of works with at least one affiliation of OpenAlex institution type
++\textit{company} among works with at least one typed affiliation. Country-pair
++and institution-level tables apply minimum-support thresholds (fifty joint
++works for a country pair, two hundred works for an institution within a period)
++that keep the released tables to the robustly estimated part of the
++distribution; these thresholds are reporting choices and are stated in the
++table notes.
++
++\subsection{Diversity indices}
++\label{sec:m-diversity}
++
++Interdisciplinarity is measured on the distribution of a work's references, or
++of a category's citations, over the categories of the ontology. Let
++$p = (p_1, \dots, p_K)$ be such a distribution and $\delta_{ij} \in [0,1]$ a
++dissimilarity between categories $i$ and $j$. The Rao--Stirling index
++\citep{rao1982diversity, stirling2007diversity}
++\begin{equation}
++  \Delta_{\mathrm{RS}}(p) \;=\; \sum_{i \ne j} p_i\, p_j\, \delta_{ij}
++  \label{eq:rao-stirling}
++\end{equation}
++is the expected dissimilarity between two categories drawn at random from $p$;
++it is zero when all mass sits on one category and increases with variety (the
++number of categories present), balance (the evenness of $p$) and disparity (the
++dissimilarity of the categories involved), the three components that
++\citeauthor{stirling2007diversity} argues any diversity measure should respond
++to. Its use in scientometrics follows \citet{rafols2010diversity}. Its known
++weakness is that it saturates: for very even distributions over many
++categories it approaches the mean dissimilarity and no longer discriminates
++\citep{zhang2016diversity}. The Leinster--Cobbold family
++\citep{leinster2012diversity} repairs this by defining, for a similarity matrix
++$Z = (z_{ij})$ with $z_{ij} = 1 - \delta_{ij}$ and an order $q \ge 0$,
++\begin{equation}
++  {}^{q}D^{Z}(p) \;=\; \Bigl( \sum_{i : p_i > 0} p_i\, (Zp)_i^{\,q-1} \Bigr)^{1/(1-q)},
++  \qquad (Zp)_i = \sum_j z_{ij}\, p_j ,
++  \label{eq:leinster-cobbold}
++\end{equation}
++with the limit ${}^{1}D^{Z}(p) = \prod_i (Zp)_i^{-p_i}$ at $q = 1$. The quantity
++$(Zp)_i$ is the ``ordinariness'' of category $i$ under $p$: how much of the
++distribution is similar to $i$. The index is an effective number of
++categories---it equals $K$ for the uniform distribution over $K$ mutually
++dissimilar categories---and the order $q$ sets the weight given to rare
++categories (large $q$ counts only the dominant ones). Following
++\citet{zhang2016diversity} the manuscript reports $q = 2$, which has the
++closed form ${}^{2}D^{Z}(p) = 1 / \sum_{i,j} p_i z_{ij} p_j = 1/(1 - \Delta_{\mathrm{RS}}(p))$
++when $z_{ii} = 1$, so that the two families are monotone transforms of each
++other at this order and differ only in scale and in their behaviour at the
++saturation end. The dissimilarity $\delta_{ij}$ between categories is one
++minus the cosine similarity of the category centroids of
++Section~\ref{sec:m-semantic} in the baseline period, so that the disparity
++component is estimated from the corpus rather than imposed. The current
++pipeline computes the category flow matrices from which $p$ is formed
++(Section~\ref{sec:m-networks}) and the centroid similarities, but the two
++diversity indices are not yet produced by a script; they are defined here
++because the protocol commits to them, and their values in Part IX are
++\pending{Rao--Stirling and Leinster--Cobbold indices per category and period, not yet computed}
++until the computation is added and audited.
++

spb/wp44_uqo

WP44: bibliography merge script (599 entries, 15 duplicate keys, 0 DOI conflicts)

Theme