WP44: fixes to Part II and appendices A/B/C/E/H after critique (31 issues resolved)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
12 changed files +664 −433
modified
manuscript/appendices/appA_taxonomy.tex
+4 −4
@@ -392,7 +392,7 @@ Field & Name & Mapped topics \\ | |||||
| 392 | \subsection{Coverage of the DBLP venue vocabulary} | 392 | \subsection{Coverage of the DBLP venue vocabulary} | ||
| 393 | \label{app:taxonomy-dblp} | 393 | \label{app:taxonomy-dblp} | ||
| 394 | 394 | ||||
| 395 | −DBLP streams are the venue signal. The stream table from which the mapping was built lists every DBLP | 395 | +DBLP streams are the venue signal. The counts quoted in this subsection and in Table~\ref{tab:appA-dblp-unmapped} are ontology-build statistics copied from the generated ``DBLP coverage'' section of \texttt{taxonomy/ONTOLOGY.md} (computed at build time from \texttt{dblp\_streams\_2018\_2026.csv}); they describe the mapping, are not corpus quantities, and are not regenerated by \texttt{tables/make\_numbers.py}. The stream table from which the mapping was built lists every DBLP | ||
| 396 | journal or conference series with at least one hundred records in 2018--2026 (3{,}569 streams); the ontology | 396 | journal or conference series with at least one hundred records in 2018--2026 (3{,}569 streams); the ontology | ||
| 397 | maps 771 of them. Measured on the DBLP records of 2018--2026 and excluding CoRR (the arXiv mirror stream, | 397 | maps 771 of them. Measured on the DBLP records of 2018--2026 and excluding CoRR (the arXiv mirror stream, | ||
| 398 | which carries no venue information), the mapped streams cover 2{,}017{,}910 of 3{,}319{,}884 records | 398 | which carries no venue information), the mapped streams cover 2{,}017{,}910 of 3{,}319{,}884 records | ||
@@ -672,7 +672,7 @@ are to be added when the corpus reveals them, through an amendment. And keyword | |||||
| 672 | mentions as well as on contributions, a failure mode discussed with the others in | 672 | mentions as well as on contributions, a failure mode discussed with the others in | ||
| 673 | Section~\ref{sec:m-classification}. | 673 | Section~\ref{sec:m-classification}. | ||
| 674 | 674 | ||||
| 675 | −The ontology was validated, after the classification run, on a stratified sample whose design is fixed in | 675 | +The ontology is to be validated, after the classification run, on a stratified sample whose design is fixed in | ||
| 676 | \texttt{taxonomy/VALIDATION.md}: fifty eligible works with an abstract per rule-assigned primary category | 676 | \texttt{taxonomy/VALIDATION.md}: fifty eligible works with an abstract per rule-assigned primary category | ||
| 677 | (forty strata) plus two hundred works to which the rules assigned no category, drawn with seed~44; two | 677 | (forty strata) plus two hundred works to which the rules assigned no category, drawn with seed~44; two | ||
| 678 | independent LLM raters that never see the rule-based assignment and receive only the definitions of | 678 | independent LLM raters that never see the rule-based assignment and receive only the definitions of | ||
@@ -680,5 +680,5 @@ Table~\ref{tab:appA-leaves} with the title, abstract, venue and year of each wor | |||||
| 680 | as third rater on a 200-work subsample (checkpoint H-2). The pre-specified decision rule is that a category | 680 | as third rater on a 200-work subsample (checkpoint H-2). The pre-specified decision rule is that a category | ||
| 681 | with consensus $F_1 < 0.60$ triggers a mapping revision, logged as a protocol amendment, and a re-run of the | 681 | with consensus $F_1 < 0.60$ triggers a mapping revision, logged as a protocol amendment, and a re-run of the | ||
| 682 | classification. The agreement statistics, the per-category precision and recall, the list of categories that | 682 | classification. The agreement statistics, the per-category precision and recall, the list of categories that | ||
| 683 | −met or failed the decision rule, and any amendment that resulted are reported in Part~IX; nothing about them | 683 | +met or failed the decision rule, and any amendment that resulted are reported in Part~IX; the validation had | ||
| 684 | −is asserted in this appendix. | 684 | +not been run when this appendix was written, and nothing about its outcome is asserted here. | ||
modified
manuscript/appendices/appB_search_and_sources.tex
+36 −21
@@ -82,7 +82,7 @@ option resuming an interrupted transfer), up to five outer attempts per file | |||||
| 82 | with a linearly increasing pause, and the transfer was accepted only when the | 82 | with a linearly increasing pause, and the transfer was accepted only when the | ||
| 83 | byte count on disk equalled the content length in the manifest. The measured | 83 | byte count on disk equalled the content length in the manifest. The measured | ||
| 84 | single-stream throughput from the node was about 71~MB/s and the aggregate | 84 | single-stream throughput from the node was about 71~MB/s and the aggregate | ||
| 85 | −over eight streams about 110~MB/s, which fixed the worker count at eight and | 85 | +over eight streams about 108~MB/s, which fixed the worker count at eight and | ||
| 86 | the wall time at one hour and forty-six minutes for the whole bucket | 86 | the wall time at one hour and forty-six minutes for the whole bucket | ||
| 87 | (\texttt{ingestion/openalex\_snapshot.py --workers 8}). | 87 | (\texttt{ingestion/openalex\_snapshot.py --workers 8}). | ||
| 88 | 88 | ||||
@@ -108,7 +108,7 @@ be applied later by the eligibility predicate rather than silently at | |||||
| 108 | ingestion. The retained records were re-compressed with Zstandard (level~6) | 108 | ingestion. The retained records were re-compressed with Zstandard (level~6) | ||
| 109 | into one file per part, \texttt{works/cs/}\allowbreak\texttt{updated\_date=X\_\_part\_NNNN.jsonl.zst}, | 109 | into one file per part, \texttt{works/cs/}\allowbreak\texttt{updated\_date=X\_\_part\_NNNN.jsonl.zst}, | ||
| 110 | about 20~GB in total for | 110 | about 20~GB in total for | ||
| 111 | −\PrismaIdentifiedOpenalexCsTopicTwoZeroOneEightTwoZeroTwoSixPassOne{} | 111 | +\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} | ||
| 112 | records. No text search of any kind was performed: the scope of the OpenAlex | 112 | records. No text search of any kind was performed: the scope of the OpenAlex | ||
| 113 | layer is defined entirely by the topic classifier's field assignment, which is | 113 | layer is defined entirely by the topic classifier's field assignment, which is | ||
| 114 | the search strategy appropriate to a field-level mapping and whose | 114 | the search strategy appropriate to a field-level mapping and whose | ||
@@ -178,9 +178,13 @@ works indexed by DBLP that OpenAlex labels with no computer-science topic. | |||||
| 178 | Rather than accept OpenAlex's boundary, the pipeline reads the bucket a | 178 | Rather than accept OpenAlex's boundary, the pipeline reads the bucket a | ||
| 179 | second time in \emph{identifier-set mode}. The linkage script writes | 179 | second time in \emph{identifier-set mode}. The linkage script writes | ||
| 180 | the list \texttt{data/metadata/}\allowbreak\texttt{needed\_oa\_ids.csv} of OpenAlex identifiers to | 180 | the list \texttt{data/metadata/}\allowbreak\texttt{needed\_oa\_ids.csv} of OpenAlex identifiers to | ||
| 181 | −recover (\LinkNeededOaIdsSecondPassDblpDoi{} reached through a DBLP DOI and | 181 | +recover, namely the works of the slim table outside the retained subset that | ||
| 182 | −\LinkNeededOaIdsSecondPassArxivDoi{} through an arXiv DOI), and | 182 | +an unlinked DBLP DOI or an unlinked arXiv DOI reaches. The list is re-derived | ||
| 183 | −\texttt{openalex\_snapshot.py --id-list} streams every part file again, | 183 | +at every linkage run; after the run on the union of both passes it holds only | ||
| 184 | +the residue that the second pass left outside the subset | ||||
| 185 | +(\LinkNeededOaIdsSecondPassDblpDoi{} identifiers reached through a DBLP DOI | ||||
| 186 | +and \LinkNeededOaIdsSecondPassArxivDoi{} through an arXiv DOI). Given the | ||||
| 187 | +list, \texttt{openalex\_snapshot.py --id-list} streams every part file again, | ||||
| 184 | keeping exactly the records whose identifier is in the set and writing no slim | 188 | keeping exactly the records whose identifier is in the set and writing no slim | ||
| 185 | rows. The identifier set is loaded in each worker process (macOS uses the | 189 | rows. The identifier set is loaded in each worker process (macOS uses the | ||
| 186 | \texttt{spawn} start method, so module globals are rebuilt per worker), the | 190 | \texttt{spawn} start method, so module globals are rebuilt per worker), the | ||
@@ -192,12 +196,13 @@ pass costs the same bandwidth as the first---about one hour and fifty minutes | |||||
| 192 | of streaming---because an identifier lookup cannot be pushed to the bucket; | 196 | of streaming---because an identifier lookup cannot be pushed to the bucket; | ||
| 193 | the price was accepted because the alternative, fetching the identifiers one | 197 | the price was accepted because the alternative, fetching the identifiers one | ||
| 194 | by one through the API, would have exceeded the public budget by two orders of | 198 | by one through the API, would have exceeded the public budget by two orders of | ||
| 195 | −magnitude. The PRISMA counts of Part~III were generated from the first pass | 199 | +magnitude. The PRISMA counts of Part~III are generated from the union of both passes: | ||
| 196 | −(\PrismaIdentifiedOpenalexTotalRecordsLoaded{} records loaded); the second | 200 | +the first pass retained \PrismaIdentifiedOpenalexPassOneCsTopicFilter{} | ||
| 197 | −pass adds its records to $\mathcal{W}_0$ when it completes and the counts are | 201 | +records through the topic filter, the second added | ||
| 198 | −regenerated by the same scripts, so the identification total in the final | 202 | +\PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} records reached through | ||
| 199 | −PRISMA figure is \pending{PrismaIdentifiedOpenalexTotalRecordsLoaded after the second pass} | 203 | +DBLP or arXiv identifiers, and the identification total of the PRISMA figure | ||
| 200 | −rather than the first-pass value quoted here. | 204 | +is their sum, \PrismaIdentifiedOpenalexTotalRecordsLoaded{} records loaded | ||
| 205 | +into $\mathcal{W}_0$. | ||||
| 201 | 206 | ||||
| 202 | \subsection{OpenAlex API increment} | 207 | \subsection{OpenAlex API increment} | ||
| 203 | \label{sec:appB-increment} | 208 | \label{sec:appB-increment} | ||
@@ -236,10 +241,15 @@ from the ledger. Every page is stored verbatim as | |||||
| 236 | (\texttt{api\_increment/ledger.json}) records per slice the pages, records, | 241 | (\texttt{api\_increment/ledger.json}) records per slice the pages, records, | ||
| 237 | reported count, completion and truncation flags, and globally the number of | 242 | reported count, completion and truncation flags, and globally the number of | ||
| 238 | calls and the cost. At the time of writing the harvest had used the budget | 243 | calls and the cost. At the time of writing the harvest had used the budget | ||
| 239 | −of several consecutive days and was not complete; its totals | 244 | +of several consecutive days and was not complete: \IncrementApiCalls{} calls | ||
| 240 | −(\pending{IncrementApiCalls}, \pending{IncrementRecordsRetrieved}, | 245 | +had retrieved \IncrementRecordsRetrieved{} records at a cost of | ||
| 241 | −\pending{IncrementTruncatedSlices}) will be reported from the ledger when the | 246 | +USD~\IncrementCostUsd{}, and \IncrementSlicesDone{} of the sixty-six | ||
| 242 | −last slice closes. The conversion step (\texttt{ingest\_api\_increment.py}) | 247 | +day-slices (thirty-three days times two strata) were closed. The number of | ||
| 248 | +truncated slices is \IncrementTruncatedSlices{}: the \texttt{cs\_secondary} | ||||
| 249 | +stratum of 16~October 2026 reported a count nearly four times the | ||||
| 250 | +10{,}000-record paging limit, so that roughly a quarter of its records were | ||||
| 251 | +retrieved. The remaining slices will be reported from the ledger when they | ||||
| 252 | +close. The conversion step (\texttt{ingest\_api\_increment.py}) | ||||
| 243 | rewrites the pages into the same \texttt{.jsonl.zst} part format as the | 253 | rewrites the pages into the same \texttt{.jsonl.zst} part format as the | ||
| 244 | snapshot filter (\texttt{works\_pass3/}), skipping any work whose identifier | 254 | snapshot filter (\texttt{works\_pass3/}), skipping any work whose identifier | ||
| 245 | is already in the pass-1 or pass-2 tables and writing an | 255 | is already in the pass-1 or pass-2 tables and writing an | ||
@@ -294,7 +304,7 @@ parsed; counts by type and by year for the entire archive are written to | |||||
| 294 | \texttt{dblp\_parse\_report.json}, and the records of 2018--2026 | 304 | \texttt{dblp\_parse\_report.json}, and the records of 2018--2026 | ||
| 295 | (\LinkDblpTwoZeroOneEightTwoZeroTwoSix{} after the type restriction of the | 305 | (\LinkDblpTwoZeroOneEightTwoZeroTwoSix{} after the type restriction of the | ||
| 296 | linkage step) are the ones that enter the corpus. The total number of | 306 | linkage step) are the ones that enter the corpus. The total number of | ||
| 297 | −publication records in the dump is \pending{DblpParsedRecords}. A second | 307 | +publication records in the dump is \DblpParsedRecords{}. A second | ||
| 298 | script (\texttt{dblp\_streams.py}) lists the streams with at least one hundred | 308 | script (\texttt{dblp\_streams.py}) lists the streams with at least one hundred | ||
| 299 | records of type article or inproceedings in 2018--2026, with their first and | 309 | records of type article or inproceedings in 2018--2026, with their first and | ||
| 300 | last years and a sample journal or booktitle name; this list | 310 | last years and a sample journal or booktitle name; this list | ||
@@ -362,7 +372,7 @@ booktitle, journal, month, year, address, publisher, URL, DOI, pages, | |||||
| 362 | abstract, volume, number); braces inside values are removed and whitespace | 372 | abstract, volume, number); braces inside values are removed and whitespace | ||
| 363 | normalised. Entries of type \texttt{comment}, \texttt{string} and | 373 | normalised. Entries of type \texttt{comment}, \texttt{string} and | ||
| 364 | \texttt{preamble} are skipped. The export is released under CC~BY~4.0. The | 374 | \texttt{preamble} are skipped. The export is released under CC~BY~4.0. The | ||
| 365 | −number of entries parsed is \pending{AclParsedEntries}; the linkage step uses | 375 | +number of entries parsed is \AclParsedEntries{}; the linkage step uses | ||
| 366 | only the \textit{inproceedings} and \textit{article} entries | 376 | only the \textit{inproceedings} and \textit{article} entries | ||
| 367 | (\LinkAclRecords{}), of which \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in | 377 | (\LinkAclRecords{}), of which \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in | ||
| 368 | 2018--2026. The Anthology's role is a coverage check for natural-language | 378 | 2018--2026. The Anthology's role is a coverage check for natural-language | ||
@@ -445,7 +455,7 @@ No. & Item & This study \\ | |||||
| 445 | 12 & Updates & One update by API increment to 2026-11-02; no further update is planned for this version. \\ | 455 | 12 & Updates & One update by API increment to 2026-11-02; no further update is planned for this version. \\ | ||
| 446 | 13 & Dates of searches & OpenAlex snapshot released 2026-10-14, read 2026-11-02; DBLP dump 2026-11-02; arXiv harvest 2026-11-02 to 03; ACL export 2026-11-02; API increment from 2026-11-03. \\ | 456 | 13 & Dates of searches & OpenAlex snapshot released 2026-10-14, read 2026-11-02; DBLP dump 2026-11-02; arXiv harvest 2026-11-02 to 03; ACL export 2026-11-02; API increment from 2026-11-03. \\ | ||
| 447 | 14 & Peer review of the search & No librarian review; the strategy is preregistered in \texttt{protocol/PROTOCOL.md} (P3) and audited by the independent agents of Phase~8. \\ | 457 | 14 & Peer review of the search & No librarian review; the strategy is preregistered in \texttt{protocol/PROTOCOL.md} (P3) and audited by the independent agents of Phase~8. \\ | ||
| 448 | −15 & Total records & OpenAlex \PrismaIdentifiedOpenalexCsTopicTwoZeroOneEightTwoZeroTwoSixPassOne{} (pass 1); DBLP \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{}; arXiv \PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{}; ACL \PrismaIdentifiedAclTwoZeroOneEightTwoZeroTwoSix{} (all 2018--2026). \\ | 458 | +15 & Total records & OpenAlex \PrismaIdentifiedOpenalexTotalRecordsLoaded{} (\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} in pass 1, \PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} in pass 2); DBLP \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{}; arXiv \PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{}; ACL \PrismaIdentifiedAclTwoZeroOneEightTwoZeroTwoSix{} (all 2018--2026). \\ | ||
| 449 | 16 & Deduplication & Deterministic record linkage across sources (Section~\ref{sec:appB-linkage}) and preprint--publication merging within OpenAlex (Part~III, Part~IV); OpenAlex works are the counting unit; \PrismaExcludedEFiveDuplicate{} preprint members of merged pairs are flagged E5. \\ | 459 | 16 & Deduplication & Deterministic record linkage across sources (Section~\ref{sec:appB-linkage}) and preprint--publication merging within OpenAlex (Part~III, Part~IV); OpenAlex works are the counting unit; \PrismaExcludedEFiveDuplicate{} preprint members of merged pairs are flagged E5. \\ | ||
| 450 | \bottomrule | 460 | \bottomrule | ||
| 451 | \end{tabularx} | 461 | \end{tabularx} | ||
@@ -498,8 +508,8 @@ E3 & \texttt{e3\_xpac} & \texttt{is\_xpac} and not \textit{in\_dblp} & \PrismaEx | |||||
| 498 | E4 & \texttt{e4\_no\_title} & title null or empty after trimming & \PrismaExcludedEFourNoTitle{} \\ | 508 | E4 & \texttt{e4\_no\_title} & title null or empty after trimming & \PrismaExcludedEFourNoTitle{} \\ | ||
| 499 | E5 & \texttt{e5\_duplicate} & $w$ is the preprint member of a surviving preprint--published pair (Part~IV, eq.~\eqref{eq:duppair}) & \PrismaExcludedEFiveDuplicate{} \\ | 509 | E5 & \texttt{e5\_duplicate} & $w$ is the preprint member of a surviving preprint--published pair (Part~IV, eq.~\eqref{eq:duppair}) & \PrismaExcludedEFiveDuplicate{} \\ | ||
| 500 | E6 & \texttt{e6\_retracted} & \texttt{is\_retracted} (flag only; not part of eligibility) & \PrismaExcludedESixRetracted{} \\ | 510 | E6 & \texttt{e6\_retracted} & \texttt{is\_retracted} (flag only; not part of eligibility) & \PrismaExcludedESixRetracted{} \\ | ||
| 501 | −E7 & \texttt{e7\_generic\_deposit} & primary source name $\in \mathcal{G}$ and not \textit{in\_dblp} and no linked arXiv record & \pending{PrismaExcludedESevenGenericDeposit} \\ | 511 | +E7 & \texttt{e7\_generic\_deposit} & primary source name $\in \mathcal{G}$ and not \textit{in\_dblp} and no linked arXiv record & \PrismaExcludedESevenGenericDeposit{} \\ | ||
| 502 | −core & \texttt{cs\_core} & \textit{in\_dblp} or \texttt{cs\_primary} or (\texttt{cs\_any} and linked arXiv primary category begins with \texttt{cs.}) & \PrismaCsCoreEligible{} eligible \\ | 512 | +core & \texttt{cs\_core} & \textit{in\_dblp} or \texttt{cs\_primary} or linked arXiv primary category begins with \texttt{cs.} (amendment A-006) & \PrismaCsCoreEligible{} eligible \\ | ||
| 503 | pen. & \texttt{cs\_penumbra} & \texttt{cs\_any} and not \texttt{cs\_core} & \PrismaCsPenumbraEligibleOtherwise{} otherwise eligible \\ | 513 | pen. & \texttt{cs\_penumbra} & \texttt{cs\_any} and not \texttt{cs\_core} & \PrismaCsPenumbraEligibleOtherwise{} otherwise eligible \\ | ||
| 504 | \midrule | 514 | \midrule | ||
| 505 | --- & \texttt{eligible} & $\neg$E1 $\wedge$ $\neg$E2 $\wedge$ $\neg$E3 $\wedge$ $\neg$E4 $\wedge$ $\neg$E5 $\wedge$ $\neg$E7 $\wedge$ core & \PrismaCsCoreEligible{} \\ | 515 | --- & \texttt{eligible} & $\neg$E1 $\wedge$ $\neg$E2 $\wedge$ $\neg$E3 $\wedge$ $\neg$E4 $\wedge$ $\neg$E5 $\wedge$ $\neg$E7 $\wedge$ core & \PrismaCsCoreEligible{} \\ | ||
@@ -521,7 +531,12 @@ The PRISMA counts are computed twice, marginally (the column of | |||||
| 521 | Table~\ref{tab:appB-codes}) and sequentially in the order E1, E2, E3, E4, E5, | 531 | Table~\ref{tab:appB-codes}) and sequentially in the order E1, E2, E3, E4, E5, | ||
| 522 | E7, and the script asserts that the loaded total equals the sum of the | 532 | E7, and the script asserts that the loaded total equals the sum of the | ||
| 523 | sequential exclusions and the after-exclusion count | 533 | sequential exclusions and the after-exclusion count | ||
| 524 | −(\PrismaAfterExclusions{}); the macro set of Part~III reports both. The E6 and | 534 | +(\PrismaAfterExclusions{}); the macro set of Part~III reports both. Since | ||
| 535 | +amendment A-006 the script also asserts that the after-exclusion count equals | ||||
| 536 | +the core plus the penumbra plus the records that belong to neither set | ||||
| 537 | +(\PrismaNeitherCoreNorPenumbraAfterExclusions{}, works with no | ||||
| 538 | +computer-science topic reached only through a DOI and not carrying an arXiv | ||||
| 539 | +\texttt{cs.} primary category). The E6 and | ||||
| 525 | penumbra rows are not exclusions and do not enter the sum. The numbering | 540 | penumbra rows are not exclusions and do not enter the sum. The numbering | ||
| 526 | conflict between the protocol's E7 (penumbra) and the code's E7 | 541 | conflict between the protocol's E7 (penumbra) and the code's E7 | ||
| 527 | (generic deposit) is resolved by amendment A-005 in favour of the code. | 542 | (generic deposit) is resolved by amendment A-005 in favour of the code. | ||
modified
manuscript/appendices/appC_derivations.tex
+19 −11
@@ -47,8 +47,10 @@ the two parametrisations are related by $\beta_2^{\mathrm{unc}} = \beta_2 - | |||||
| 47 | text. The eleven month dummies are linearly independent of the constant | 47 | text. The eleven month dummies are linearly independent of the constant | ||
| 48 | because January is omitted; the full-column-rank condition requires at least | 48 | because January is omitted; the full-column-rank condition requires at least | ||
| 49 | one observation in every month of the year, satisfied by any series of | 49 | one observation in every month of the year, satisfied by any series of | ||
| 50 | −twelve months or more, and at least one post-event month with $t > t_e$, | 50 | +twelve months or more, and at least two post-event months with $t > t_e$ | ||
| 51 | −guaranteed by the driver's rule that twelve months follow every event. | 51 | +(with a single one the centred post-trend column is identically zero and | ||
| 52 | +$\beta_2$ and $\beta_3$ are not separately identified), guaranteed by the | ||||
| 53 | +driver's rule that twelve months follow every event. | ||||
| 52 | The ordinary least-squares estimator is $\hat\beta = (X^{\top}X)^{-1} | 54 | The ordinary least-squares estimator is $\hat\beta = (X^{\top}X)^{-1} | ||
| 53 | X^{\top}y$ and the residuals $\hat\varepsilon = y - X\hat\beta$. | 55 | X^{\top}y$ and the residuals $\hat\varepsilon = y - X\hat\beta$. | ||
| 54 | 56 | ||||
@@ -83,8 +85,8 @@ to be positive semi-definite. The implementation calls | |||||
| 83 | $L = 6$ and $w_\ell = 1 - \ell/7$ as stated in Part~IV, without the | 85 | $L = 6$ and $w_\ell = 1 - \ell/7$ as stated in Part~IV, without the | ||
| 84 | small-sample degrees-of-freedom correction and with $p$-values from the | 86 | small-sample degrees-of-freedom correction and with $p$-values from the | ||
| 85 | normal approximation to the $t$-ratio, which is the library's default for | 87 | normal approximation to the $t$-ratio, which is the library's default for | ||
| 86 | −robust covariances. With $n$ of the order of one hundred and fourteen | 88 | +robust covariances. With $n$ of the order of one hundred and fifteen | ||
| 87 | −regressors, the difference between the normal and the $t_{n-15}$ reference | 89 | +columns in $X$, the constant included, the difference between the normal and the $t_{n-15}$ reference | ||
| 88 | distribution is immaterial at the magnitudes of the $t$-statistics reported; | 90 | distribution is immaterial at the magnitudes of the $t$-statistics reported; | ||
| 89 | what matters is the placebo rank of the next paragraph, which does not | 91 | what matters is the placebo rank of the next paragraph, which does not | ||
| 90 | depend on any reference distribution. | 92 | depend on any reference distribution. | ||
@@ -757,9 +759,17 @@ training and test cutoffs. Every quantity used to fit a model---features, | |||||
| 757 | labels and the standardisation constants of the scaler, which is fitted on | 759 | labels and the standardisation constants of the scaler, which is fitted on | ||
| 758 | training rows only---is $\mathcal{F}_{T^{\mathrm{tr}}_{\max} + h}$-measurable | 760 | training rows only---is $\mathcal{F}_{T^{\mathrm{tr}}_{\max} + h}$-measurable | ||
| 759 | with $T^{\mathrm{tr}}_{\max} = \max \mathcal{T}_{\mathrm{tr}}$. The | 761 | with $T^{\mathrm{tr}}_{\max} = \max \mathcal{T}_{\mathrm{tr}}$. The | ||
| 760 | −implementation sets $T^{\mathrm{tr}}_{\max}$ to December 2021 for $h = 12$ | 762 | +implementation takes as candidate cutoffs every third month of the panel, | ||
| 761 | −and to December 2020 for $h = 24$, so that $T^{\mathrm{tr}}_{\max} + h$ is | 763 | +which begins in January 2018 (January, April, July and October of each year), | ||
| 762 | −December 2022 in both cases, and the first test cutoff is January 2023. | 764 | +and admits as training cutoffs those from December 2019 up to December 2021 | ||
| 765 | +for $h = 12$ and up to December 2020 for $h = 24$; the realised training | ||||
| 766 | +cutoffs therefore run from January 2020 to October 2021 ($h = 12$) or to | ||||
| 767 | +October 2020 ($h = 24$), so that $T^{\mathrm{tr}}_{\max} + h$ is October | ||||
| 768 | +2022 in both cases, and the first test cutoff is January 2023 (the last is | ||||
| 769 | +October 2024 at most, subject to the censoring condition below). The realised | ||||
| 770 | +lists are written to \texttt{results/emerging/metrics\_h12.json} and | ||||
| 771 | +\texttt{metrics\_h24.json} under \texttt{train\_cutoffs} and | ||||
| 772 | +\texttt{test\_cutoffs}. | ||||
| 763 | Hence | 773 | Hence | ||
| 764 | \begin{equation} | 774 | \begin{equation} | ||
| 765 | T^{\mathrm{tr}}_{\max} + h \;<\; \min \mathcal{T}_{\mathrm{te}}, | 775 | T^{\mathrm{tr}}_{\max} + h \;<\; \min \mathcal{T}_{\mathrm{te}}, | ||
@@ -770,10 +780,8 @@ that precedes every test feature, let alone every test label, which closes | |||||
| 770 | at $\max \mathcal{T}_{\mathrm{te}} + h \le M$. Condition \eqref{eq:appC-gap} | 780 | at $\max \mathcal{T}_{\mathrm{te}} + h \le M$. Condition \eqref{eq:appC-gap} | ||
| 771 | is the formal content of ``out of time'' in the sense of | 781 | is the formal content of ``out of time'' in the sense of | ||
| 772 | \citet{bergmeir2012crossval}; it is stricter than the plan's wording, which | 782 | \citet{bergmeir2012crossval}; it is stricter than the plan's wording, which | ||
| 773 | −allowed training cutoffs through 2022, and stricter for $h = 24$ than the | 783 | +allowed training cutoffs through 2022; the code's rule is the one that | ||
| 774 | −description of Part~IV, which gives December 2021 as the last training | 784 | +guarantees \eqref{eq:appC-gap}, and Part~IV describes the same rule. The test cutoffs | ||
| 775 | −cutoff for both horizons---the code's rule is the one that guarantees | ||||
| 776 | −\eqref{eq:appC-gap}, and the manuscript follows the code. The test cutoffs | ||||
| 777 | are further restricted to those with $T + h \le M$, so that no test label is | 785 | are further restricted to those with $T + h \le M$, so that no test label is | ||
| 778 | right-censored. | 786 | right-censored. | ||
| 779 | 787 | ||||
modified
manuscript/appendices/appE_data_dictionary.tex
+2 −2
@@ -127,7 +127,7 @@ Column & Type & Definition & Source \\ | |||||
| 127 | \bottomrule | 127 | \bottomrule | ||
| 128 | \endlastfoot | 128 | \endlastfoot | ||
| 129 | \texttt{key} & string & DBLP record key, unique & element attribute \texttt{key} \\ | 129 | \texttt{key} & string & DBLP record key, unique & element attribute \texttt{key} \\ | ||
| 130 | −\texttt{rec\_\allowbreak type} & string & Record type: article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis & element name \\ | 130 | +\texttt{rec\_\allowbreak type} & string & Record type: article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis, data & element name \\ | ||
| 131 | \texttt{mdate} & string & Last modification date of the record & attribute \texttt{mdate} \\ | 131 | \texttt{mdate} & string & Last modification date of the record & attribute \texttt{mdate} \\ | ||
| 132 | \texttt{publtype} & string & Publication sub-type (e.g.\ informal, survey, withdrawn); null if absent & attribute \texttt{publtype} \\ | 132 | \texttt{publtype} & string & Publication sub-type (e.g.\ informal, survey, withdrawn); null if absent & attribute \texttt{publtype} \\ | ||
| 133 | \texttt{title} & string & Title, inner markup flattened to text & \texttt{<title>} \\ | 133 | \texttt{title} & string & Title, inner markup flattened to text & \texttt{<title>} \\ | ||
@@ -250,7 +250,7 @@ Column & Type & Definition & Source \\ | |||||
| 250 | \texttt{title} & string & Title; \texttt{display\_\allowbreak name} when \texttt{title} is absent & \texttt{title}, \texttt{display\_\allowbreak name} \\ | 250 | \texttt{title} & string & Title; \texttt{display\_\allowbreak name} when \texttt{title} is absent & \texttt{title}, \texttt{display\_\allowbreak name} \\ | ||
| 251 | \texttt{publication\_\allowbreak year} & int32 & Publication year & \texttt{publication\_\allowbreak year} \\ | 251 | \texttt{publication\_\allowbreak year} & int32 & Publication year & \texttt{publication\_\allowbreak year} \\ | ||
| 252 | \texttt{publication\_\allowbreak date} & string & Publication date; OpenAlex sets \texttt{YYYY-01-01} when only the year is known (see \texttt{date\_\allowbreak precision}, Table~\ref{tab:appE-corpus}) & \texttt{publication\_\allowbreak date} \\ | 252 | \texttt{publication\_\allowbreak date} & string & Publication date; OpenAlex sets \texttt{YYYY-01-01} when only the year is known (see \texttt{date\_\allowbreak precision}, Table~\ref{tab:appE-corpus}) & \texttt{publication\_\allowbreak date} \\ | ||
| 253 | −\texttt{created\_\allowbreak date} & string & Date the record entered OpenAlex (used for indexing-lag diagnostics) & \texttt{created\_\allowbreak date} \\ | 253 | +\texttt{created\_\allowbreak date} & string & Date the record was created in OpenAlex's current data model; for most works it reflects a re-ingestion in the second half of 2025 rather than first indexing, and it could not be used for the indexing-lag profile (Part~III, Section~\ref{sec:data-dates}) & \texttt{created\_\allowbreak date} \\ | ||
| 254 | \texttt{updated\_\allowbreak date} & string & Last update, date part only & \texttt{updated\_\allowbreak date[:10]} \\ | 254 | \texttt{updated\_\allowbreak date} & string & Last update, date part only & \texttt{updated\_\allowbreak date[:10]} \\ | ||
| 255 | \texttt{type} & string & OpenAlex work type & \texttt{type} \\ | 255 | \texttt{type} & string & OpenAlex work type & \texttt{type} \\ | ||
| 256 | \texttt{type\_\allowbreak crossref} & string & Crossref type when the record has one & \texttt{type\_\allowbreak crossref} \\ | 256 | \texttt{type\_\allowbreak crossref} & string & Crossref type when the record has one & \texttt{type\_\allowbreak crossref} \\ | ||
modified
manuscript/appendices/appH_reproducibility.tex
+17 −13
@@ -22,15 +22,18 @@ this appendix and is updated whenever a step changes; where the two differ, the | |||||
| 22 | \label{app:repro-env} | 22 | \label{app:repro-env} | ||
| 23 | 23 | ||||
| 24 | Everything described in the runbook was executed on one node of the MacLustr cluster, M3U96b (Mac Studio, | 24 | Everything described in the runbook was executed on one node of the MacLustr cluster, M3U96b (Mac Studio, | ||
| 25 | −Apple M3~Ultra, 28~cores, 96~GB of unified memory, 103{,}079{,}215{,}104 bytes as recorded by the environment | 25 | +Apple M3~Ultra, 96~GB of unified memory, 103{,}079{,}215{,}104 bytes as recorded by the environment | ||
| 26 | −capture), in the working directory \texttt{\textasciitilde/\allowbreak wp44\_\allowbreak uqo}, under Python 3.12.15 in a virtual | 26 | +capture; the capture records no core count, the runbook header states 28 cores and the cluster inventory 32, | ||
| 27 | +and the discrepancy is left on record), in the working directory \texttt{\textasciitilde/\allowbreak wp44\_\allowbreak uqo}, under Python 3.12.15 in a virtual | ||||
| 27 | environment created with \texttt{uv} from \texttt{requirements.\allowbreak txt} and then frozen to | 28 | environment created with \texttt{uv} from \texttt{requirements.\allowbreak txt} and then frozen to | ||
| 28 | \texttt{reproducibility/\allowbreak environment\_\allowbreak M3U96b.\allowbreak txt} (92 lines: 88 pinned packages, the interpreter version, the | 29 | \texttt{reproducibility/\allowbreak environment\_\allowbreak M3U96b.\allowbreak txt} (92 lines: 88 pinned packages, the interpreter version, the | ||
| 29 | operating-system version string, the processor and the memory size). The operating-system version string | 30 | operating-system version string, the processor and the memory size). The operating-system version string | ||
| 30 | captured in that file is 26.4, whereas the runbook header states macOS~27.0; the discrepancy is recorded here | 31 | captured in that file is 26.4, whereas the runbook header states macOS~27.0; the discrepancy is recorded here | ||
| 31 | rather than resolved, since the node was updated between captures and neither value affects the computation. | 32 | rather than resolved, since the node was updated between captures and neither value affects the computation. | ||
| 32 | Two further nodes were used after the incidents of Section~\ref{app:repro-incidents}: M5M36 for the second | 33 | Two further nodes were used after the incidents of Section~\ref{app:repro-incidents}: M5M36 for the second | ||
| 33 | −snapshot pass and M4M64c and M3U96a for embedding shards; they run the same locked environment. | 34 | +snapshot pass and M4M64c and M3U96a for embedding shards; they were set up from the same | ||
| 35 | +\texttt{requirements.txt}, but no environment capture exists for them under \texttt{reproducibility/}, so the | ||||
| 36 | +identity of their environments with the one frozen on M3U96b is an assumption rather than a recorded fact. | ||||
| 34 | Table~\ref{tab:appH-packages} lists the packages on which results depend; the complete list is the lock file. | 37 | Table~\ref{tab:appH-packages} lists the packages on which results depend; the complete list is the lock file. | ||
| 35 | No step requires credentials except the two LLM raters of the taxonomy validation and the knowledge-graph | 38 | No step requires credentials except the two LLM raters of the taxonomy validation and the knowledge-graph | ||
| 36 | extraction (an Anthropic API key read from a file outside the repository, never stored in it). | 39 | extraction (an Anthropic API key read from a file outside the repository, never stored in it). | ||
@@ -102,11 +105,11 @@ and normalised by the same step~6; it is not yet a numbered step of the runbook. | |||||
| 102 | \endfoot | 105 | \endfoot | ||
| 103 | \bottomrule | 106 | \bottomrule | ||
| 104 | \endlastfoot | 107 | \endlastfoot | ||
| 105 | −1 & OpenAlex works snapshot, pass~1 (CS filter + slim table): \texttt{python ingestion/\allowbreak openalex\_\allowbreak snapshot.\allowbreak py --out data/\allowbreak raw/\allowbreak openalex/\allowbreak works --manifest data/\allowbreak raw/\allowbreak openalex/\allowbreak works\_\allowbreak manifest\_\allowbreak 2026-10-14.\allowbreak json --workers 8} & public S3 bucket, release 2026-10-14 & \texttt{works/\allowbreak cs/\allowbreak *.\allowbreak jsonl.\allowbreak zst} (15{,}490{,}159 records), \texttt{works/\allowbreak slim/\allowbreak *.\allowbreak parquet} (464{,}765{,}485 rows), \texttt{ledger.\allowbreak sqlite} & 1~h 46~min at $\approx$108~MB/s \\ | 108 | +1 & OpenAlex works snapshot, pass~1 (CS filter + slim table): \texttt{python ingestion/\allowbreak openalex\_\allowbreak snapshot.\allowbreak py --out data/\allowbreak raw/\allowbreak openalex/\allowbreak works --manifest data/\allowbreak raw/\allowbreak openalex/\allowbreak works\_\allowbreak manifest\_\allowbreak 2026-10-14.\allowbreak json --workers 8} & public S3 bucket, release 2026-10-14 & \texttt{works/\allowbreak cs/\allowbreak *.\allowbreak jsonl.\allowbreak zst} (\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} records), \texttt{works/\allowbreak slim/\allowbreak *.\allowbreak parquet} (\LinkSlimTotal{} rows), \texttt{ledger.\allowbreak sqlite} & 1~h 46~min at $\approx$108~MB/s \\ | ||
| 106 | 2 & OpenAlex entities: \texttt{python ingestion/\allowbreak openalex\_\allowbreak entities.\allowbreak py --out data/\allowbreak raw/\allowbreak openalex} & S3 & \texttt{topics/\allowbreak }, \ldots, \texttt{institutions/\allowbreak }; \texttt{entities\_\allowbreak ledger.\allowbreak json} & 5~min \\ | 109 | 2 & OpenAlex entities: \texttt{python ingestion/\allowbreak openalex\_\allowbreak entities.\allowbreak py --out data/\allowbreak raw/\allowbreak openalex} & S3 & \texttt{topics/\allowbreak }, \ldots, \texttt{institutions/\allowbreak }; \texttt{entities\_\allowbreak ledger.\allowbreak json} & 5~min \\ | ||
| 107 | −3 & DBLP: \texttt{python ingestion/\allowbreak dblp\_\allowbreak parse.\allowbreak py --xml data/\allowbreak raw/\allowbreak dblp/\allowbreak dblp.\allowbreak xml.\allowbreak gz --out data/\allowbreak raw/\allowbreak dblp} & dump of 2026-11-02, MD5 verified & \texttt{dblp\_\allowbreak publications.\allowbreak parquet} (8{,}840{,}866 records), parse report & 4~min 45~s \\ | 110 | +3 & DBLP: \texttt{python ingestion/\allowbreak dblp\_\allowbreak parse.\allowbreak py --xml data/\allowbreak raw/\allowbreak dblp/\allowbreak dblp.\allowbreak xml.\allowbreak gz --out data/\allowbreak raw/\allowbreak dblp} & dump of 2026-11-02, MD5 verified & \texttt{dblp\_\allowbreak publications.\allowbreak parquet} (\DblpParsedRecords{} records), parse report & 4~min 45~s \\ | ||
| 108 | −4 & arXiv OAI-PMH: \texttt{python ingestion/\allowbreak arxiv\_\allowbreak oai.\allowbreak py --out data/\allowbreak raw/\allowbreak arxiv --from 2018-01-01 --until 2026-11-02} & OAI feed, set \texttt{cs} & raw XML pages (758), \texttt{arxiv\_\allowbreak cs.\allowbreak parquet} (915{,}115 records) & $\approx$1~h \\ | 111 | +4 & arXiv OAI-PMH: \texttt{python ingestion/\allowbreak arxiv\_\allowbreak oai.\allowbreak py --out data/\allowbreak raw/\allowbreak arxiv --from 2018-01-01 --until 2026-11-02} & OAI feed, set \texttt{cs} & raw XML pages (758), \texttt{arxiv\_\allowbreak cs.\allowbreak parquet} (\LinkArxivCsRecords{} records) & $\approx$1~h \\ | ||
| 109 | −5 & ACL Anthology: \texttt{python ingestion/\allowbreak acl\_\allowbreak parse.\allowbreak py --bib data/\allowbreak raw/\allowbreak acl/\allowbreak anthology+abstracts.\allowbreak bib.\allowbreak gz --out data/\allowbreak raw/\allowbreak acl} & BibTeX export & \texttt{acl\_\allowbreak anthology.\allowbreak parquet} (138{,}796 entries) & 10~s \\ | 112 | +5 & ACL Anthology: \texttt{python ingestion/\allowbreak acl\_\allowbreak parse.\allowbreak py --bib data/\allowbreak raw/\allowbreak acl/\allowbreak anthology+abstracts.\allowbreak bib.\allowbreak gz --out data/\allowbreak raw/\allowbreak acl} & BibTeX export & \texttt{acl\_\allowbreak anthology.\allowbreak parquet} (\AclParsedEntries{} entries) & 10~s \\ | ||
| 110 | 6 & Normalise: \texttt{python ingestion/\allowbreak build\_\allowbreak corpus.\allowbreak py --cs-dir data/\allowbreak raw/\allowbreak openalex/\allowbreak works/\allowbreak cs --out data/\allowbreak processed --workers 12} & step 1 & \texttt{tables/\allowbreak <table>/\allowbreak *.\allowbreak parquet} & 2~min 10~s \\ | 113 | 6 & Normalise: \texttt{python ingestion/\allowbreak build\_\allowbreak corpus.\allowbreak py --cs-dir data/\allowbreak raw/\allowbreak openalex/\allowbreak works/\allowbreak cs --out data/\allowbreak processed --workers 12} & step 1 & \texttt{tables/\allowbreak <table>/\allowbreak *.\allowbreak parquet} & 2~min 10~s \\ | ||
| 111 | 7 & Compact: \texttt{ulimit -n 65536; python ingestion/\allowbreak compact\_\allowbreak tables.\allowbreak py \textasciitilde/\allowbreak wp44\_\allowbreak uqo} & step 6 & \texttt{compact/\allowbreak <table>.\allowbreak parquet} & 30~s \\ | 114 | 7 & Compact: \texttt{ulimit -n 65536; python ingestion/\allowbreak compact\_\allowbreak tables.\allowbreak py \textasciitilde/\allowbreak wp44\_\allowbreak uqo} & step 6 & \texttt{compact/\allowbreak <table>.\allowbreak parquet} & 30~s \\ | ||
| 112 | 8 & Link: \texttt{python ingestion/\allowbreak link\_\allowbreak sources.\allowbreak py --root \textasciitilde/\allowbreak wp44\_\allowbreak uqo} & steps 3--7 & \texttt{wp44.\allowbreak duckdb} with \texttt{link\_\allowbreak *}; \texttt{data/\allowbreak metadata/\allowbreak linkage\_\allowbreak report.\allowbreak json}; \texttt{needed\_\allowbreak oa\_\allowbreak ids.\allowbreak csv} & 10~s \\ | 115 | 8 & Link: \texttt{python ingestion/\allowbreak link\_\allowbreak sources.\allowbreak py --root \textasciitilde/\allowbreak wp44\_\allowbreak uqo} & steps 3--7 & \texttt{wp44.\allowbreak duckdb} with \texttt{link\_\allowbreak *}; \texttt{data/\allowbreak metadata/\allowbreak linkage\_\allowbreak report.\allowbreak json}; \texttt{needed\_\allowbreak oa\_\allowbreak ids.\allowbreak csv} & 10~s \\ | ||
@@ -266,8 +269,9 @@ Nothing under \texttt{data/\allowbreak raw/\allowbreak } or \texttt{data/\allowb | |||||
| 266 | repository contains is the code, the ontology and its audit table, the manifests, ledgers and hashes, the | 269 | repository contains is the code, the ontology and its audit table, the manifests, ledgers and hashes, the | ||
| 267 | PRISMA counts and the linkage report, the aggregated result tables, the figures, the generated number macros | 270 | PRISMA counts and the linkage report, the aggregated result tables, the figures, the generated number macros | ||
| 268 | and the manuscript (protocol \S P8). Record-level derived tables may be released where the source licence | 271 | and the manuscript (protocol \S P8). Record-level derived tables may be released where the source licence | ||
| 269 | −permits: OpenAlex is CC0, DBLP is ODC-BY~1.0, arXiv metadata obtained through OAI-PMH is under the CC0 | 272 | +permits: OpenAlex is CC0, DBLP is ODC-BY~1.0, arXiv metadata obtained through OAI-PMH is used under the arXiv | ||
| 270 | −terms of that feed, and the ACL Anthology is CC~BY~4.0; a derived record table therefore carries the | 273 | +metadata terms of use (Part~III, Table~\ref{tab:data-sources}), which release the metadata itself under CC0, | ||
| 274 | +and the ACL Anthology is CC~BY~4.0; a derived record table therefore carries the | ||||
| 271 | attribution required by ODC-BY and CC~BY for the DBLP and ACL fields it contains. Full texts are never | 275 | attribution required by ODC-BY and CC~BY for the DBLP and ACL fields it contains. Full texts are never | ||
| 272 | redistributed and were never obtained through paywalls. The Semantic Scholar Academic Graph was not used: | 276 | redistributed and were never obtained through paywalls. The Semantic Scholar Academic Graph was not used: | ||
| 273 | bulk access requires an API key that was not available and anonymous calls were refused, and the mandate's | 277 | bulk access requires an API key that was not available and anonymous calls were refused, and the mandate's | ||
@@ -284,12 +288,12 @@ citation-intent signals derived from that graph are consequently absent from the | |||||
| 284 | \toprule | 288 | \toprule | ||
| 285 | Source & What was obtained, when, size & Integrity & Licence \\ | 289 | Source & What was obtained, when, size & Integrity & Licence \\ | ||
| 286 | \midrule | 290 | \midrule | ||
| 287 | −OpenAlex works snapshot, release 2026-10-14 & \texttt{data/\allowbreak jsonl/\allowbreak works/\allowbreak }: 1{,}898 part files, 464{,}765{,}485 works, 675.3~GB gzip; streamed 2026-11-02 21:19--23:05~ET over HTTPS with 8 parallel streams; retained: CS subset 20~GB (15{,}490{,}159 records) and slim table 9.3~GB (all works) & manifest; per-file content length and SHA-256 in \texttt{works/\allowbreak ledger.\allowbreak sqlite} (1{,}898/1{,}898 ok, 0 parse errors, record total = manifest total) & CC0 \\ | 291 | +OpenAlex works snapshot, release 2026-10-14 & \texttt{data/\allowbreak jsonl/\allowbreak works/\allowbreak }: 1{,}898 part files, \LinkSlimTotal{} works, 675.3~GB gzip; streamed 2026-11-02 21:19--23:05~ET over HTTPS with 8 parallel streams; retained: CS subset 20~GB (\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} records) and slim table 9.3~GB (all works) & manifest; per-file content length and SHA-256 in \texttt{works/\allowbreak ledger.\allowbreak sqlite} (1{,}898/1{,}898 ok, 0 parse errors, record total = manifest total) & CC0 \\ | ||
| 288 | OpenAlex entity snapshots & topics 4{,}516; subfields 252; fields 26; domains 4; sources 268{,}629; institutions 143{,}455; publishers 10{,}713; funders 41{,}799; concepts 65{,}026; 2026-11-02; 0.6~GB & \texttt{entities\_\allowbreak ledger.\allowbreak json} (SHA-256 per file) & CC0 \\ | 292 | OpenAlex entity snapshots & topics 4{,}516; subfields 252; fields 26; domains 4; sources 268{,}629; institutions 143{,}455; publishers 10{,}713; funders 41{,}799; concepts 65{,}026; 2026-11-02; 0.6~GB & \texttt{entities\_\allowbreak ledger.\allowbreak json} (SHA-256 per file) & CC0 \\ | ||
| 289 | OpenAlex API increment & works with any CS topic, publication date 2026-10-01 to 2026-11-02, sliced by day and CS-primary status, basic paging $\le$10{,}000 per slice; from 2026-11-03 (public budget 0.10~USD/day, several days) & raw JSON pages; \texttt{api\_\allowbreak increment/\allowbreak ledger.\allowbreak json} (counts, cost) & CC0 \\ | 293 | OpenAlex API increment & works with any CS topic, publication date 2026-10-01 to 2026-11-02, sliced by day and CS-primary status, basic paging $\le$10{,}000 per slice; from 2026-11-03 (public budget 0.10~USD/day, several days) & raw JSON pages; \texttt{api\_\allowbreak increment/\allowbreak ledger.\allowbreak json} (counts, cost) & CC0 \\ | ||
| 290 | −DBLP & \texttt{dblp.\allowbreak xml.\allowbreak gz} (2026-11-02 22:39~UTC), \texttt{dblp.\allowbreak dtd}, \texttt{.md5}; 1.12~GB & MD5 \texttt{ff1c424c\ldots27c0} verified; parse report (8{,}840{,}866 records) & ODC-BY 1.0 \\ | 294 | +DBLP & \texttt{dblp.\allowbreak xml.\allowbreak gz} (2026-11-02 22:39~UTC), \texttt{dblp.\allowbreak dtd}, \texttt{.md5}; 1.12~GB & MD5 \texttt{ff1c424c\ldots27c0} verified; parse report (\DblpParsedRecords{} records) & ODC-BY 1.0 \\ | ||
| 291 | −arXiv OAI-PMH & set \texttt{cs}, metadata format \texttt{arXiv}, datestamps 2018-01-01 to 2026-11-02 in monthly windows, raw XML pages kept; 2026-11-02/03 & 758 pages, 915{,}115 unique records; \texttt{harvest\_\allowbreak state.\allowbreak json}, parse report & CC0 (metadata) \\ | 295 | +arXiv OAI-PMH & set \texttt{cs}, metadata format \texttt{arXiv}, datestamps 2018-01-01 to 2026-11-02 in monthly windows, raw XML pages kept; 2026-11-02/03 & 758 pages, \LinkArxivCsRecords{} unique records; \texttt{harvest\_\allowbreak state.\allowbreak json}, parse report & arXiv metadata terms (CC0) \\ | ||
| 292 | −ACL Anthology & \texttt{anthology+abstracts.\allowbreak bib.\allowbreak gz}; 2026-11-02; 46.7~MB, 138{,}796 entries & SHA-256 \texttt{44c3f7e9\ldots58e3} & CC BY 4.0 \\ | 296 | +ACL Anthology & \texttt{anthology+abstracts.\allowbreak bib.\allowbreak gz}; 2026-11-02; 46.7~MB, \AclParsedEntries{} entries & SHA-256 \texttt{44c3f7e9\ldots58e3} & CC BY 4.0 \\ | ||
| 293 | \bottomrule | 297 | \bottomrule | ||
| 294 | \end{tabularx} | 298 | \end{tabularx} | ||
| 295 | \end{singlespace} | 299 | \end{singlespace} | ||
modified
manuscript/references_part02.bib
+24 −48
@@ -10,26 +10,6 @@ | |||||
| 10 | note = {verified: arxiv API 2026-11-03} | 10 | note = {verified: arxiv API 2026-11-03} | ||
| 11 | } | 11 | } | ||
| 12 | 12 | ||||
| 13 | −@misc{kinney2023s2, | ||||
| 14 | − title = {The {Semantic} {Scholar} {Open} {Data} {Platform}}, | ||||
| 15 | − author = {Rodney Kinney and Chloe Anastasiades and Russell Authur and Iz Beltagy and Jonathan Bragg and Alexandra Buraczynski and Isabel Cachola and Stefan Candra and Yoganand Chandrasekhar and Arman Cohan and others}, | ||||
| 16 | − year = {2023}, | ||||
| 17 | − eprint = {2301.10140}, | ||||
| 18 | − archiveprefix = {arXiv}, | ||||
| 19 | − howpublished = {arXiv preprint arXiv:2301.10140}, | ||||
| 20 | − note = {verified: arxiv API 2026-11-03} | ||||
| 21 | −} | ||||
| 22 | − | ||||
| 23 | −@misc{culbert2024openalex, | ||||
| 24 | − title = {Reference {Coverage} {Analysis} of {OpenAlex} compared to {Web} of {Science} and {Scopus}}, | ||||
| 25 | − author = {Jack Culbert and Anne Hobert and Najko Jahn and Nick Haupka and Marion Schmidt and Paul Donner and Philipp Mayr}, | ||||
| 26 | − year = {2024}, | ||||
| 27 | − eprint = {2401.16359}, | ||||
| 28 | − archiveprefix = {arXiv}, | ||||
| 29 | − howpublished = {arXiv preprint arXiv:2401.16359}, | ||||
| 30 | − note = {verified: arxiv API 2026-11-03} | ||||
| 31 | −} | ||||
| 32 | − | ||||
| 33 | @misc{sutskever2014seq2seq, | 13 | @misc{sutskever2014seq2seq, | ||
| 34 | title = {Sequence to {Sequence} {Learning} with {Neural} {Networks}}, | 14 | title = {Sequence to {Sequence} {Learning} with {Neural} {Networks}}, | ||
| 35 | author = {Ilya Sutskever and Oriol Vinyals and Quoc V. Le}, | 15 | author = {Ilya Sutskever and Oriol Vinyals and Quoc V. Le}, | ||
@@ -800,16 +780,6 @@ | |||||
| 800 | note = {verified: arxiv API 2026-11-03} | 780 | note = {verified: arxiv API 2026-11-03} | ||
| 801 | } | 781 | } | ||
| 802 | 782 | ||||
| 803 | −@misc{ott2022benchmarks, | ||||
| 804 | − title = {Mapping global dynamics of benchmark creation and saturation in artificial intelligence}, | ||||
| 805 | − author = {Simon Ott and Adriano Barbosa-Silva and Kathrin Blagec and Jan Brauner and Matthias Samwald}, | ||||
| 806 | − year = {2022}, | ||||
| 807 | − eprint = {2203.04592}, | ||||
| 808 | − archiveprefix = {arXiv}, | ||||
| 809 | − howpublished = {arXiv preprint arXiv:2203.04592}, | ||||
| 810 | − note = {verified: arxiv API 2026-11-03} | ||||
| 811 | −} | ||||
| 812 | − | ||||
| 813 | @misc{dehghani2021lottery, | 783 | @misc{dehghani2021lottery, | ||
| 814 | title = {The {Benchmark} {Lottery}}, | 784 | title = {The {Benchmark} {Lottery}}, | ||
| 815 | author = {Mostafa Dehghani and Yi Tay and Alexey A. Gritsenko and Zhe Zhao and Neil Houlsby and Fernando Diaz and Donald Metzler and Oriol Vinyals}, | 785 | author = {Mostafa Dehghani and Yi Tay and Alexey A. Gritsenko and Zhe Zhao and Neil Houlsby and Fernando Diaz and Donald Metzler and Oriol Vinyals}, | ||
@@ -2318,17 +2288,6 @@ | |||||
| 2318 | note = {verified: crossref 2026-11-03} | 2288 | note = {verified: crossref 2026-11-03} | ||
| 2319 | } | 2289 | } | ||
| 2320 | 2290 | ||||
| 2321 | −@article{fortunato2018sos, | ||||
| 2322 | − title = {Science of science}, | ||||
| 2323 | − author = {Fortunato, Santo and Bergstrom, Carl T. and Börner, Katy and Evans, James A. and Helbing, Dirk and Milojević, Staša and Petersen, Alexander M. and Radicchi, Filippo and Sinatra, Roberta and Uzzi, Brian and others}, | ||||
| 2324 | − year = {2018}, | ||||
| 2325 | − journal = {Science}, | ||||
| 2326 | − volume = {359}, | ||||
| 2327 | − number = {6379}, | ||||
| 2328 | − doi = {10.1126/science.aao0185}, | ||||
| 2329 | − note = {verified: crossref 2026-11-03} | ||||
| 2330 | −} | ||||
| 2331 | − | ||||
| 2332 | @book{wang2021sos, | 2291 | @book{wang2021sos, | ||
| 2333 | title = {The {Science} of {Science}}, | 2292 | title = {The {Science} of {Science}}, | ||
| 2334 | author = {Wang, Dashun and Barabási, Albert-László}, | 2293 | author = {Wang, Dashun and Barabási, Albert-László}, | ||
@@ -2424,13 +2383,7 @@ | |||||
| 2424 | howpublished = {Blog post, 30 November 2022, \url{https://openai.com/index/chatgpt/}}, | 2383 | howpublished = {Blog post, 30 November 2022, \url{https://openai.com/index/chatgpt/}}, | ||
| 2425 | note = {Accessed 3 November 2026; original URL openai.com/blog/chatgpt archived by the Internet Archive on 2 December 2022} | 2384 | note = {Accessed 3 November 2026; original URL openai.com/blog/chatgpt archived by the Internet Archive on 2 December 2022} | ||
| 2426 | } | 2385 | } | ||
| 2427 | −@misc{openalex2025topics, | 2386 | + | ||
| 2428 | − title = {Topics}, | ||||
| 2429 | − author = {{OpenAlex}}, | ||||
| 2430 | − year = {2025}, | ||||
| 2431 | − howpublished = {OpenAlex documentation, \url{https://help.openalex.org/data/topics/}}, | ||||
| 2432 | − note = {Accessed 3 November 2026} | ||||
| 2433 | −} | ||||
| 2434 | @misc{arxiv2026stats, | 2387 | @misc{arxiv2026stats, | ||
| 2435 | title = {{arXiv} monthly submissions}, | 2388 | title = {{arXiv} monthly submissions}, | ||
| 2436 | author = {{arXiv}}, | 2389 | author = {{arXiv}}, | ||
@@ -2438,3 +2391,26 @@ | |||||
| 2438 | howpublished = {\url{https://arxiv.org/stats/monthly_submissions}}, | 2391 | howpublished = {\url{https://arxiv.org/stats/monthly_submissions}}, | ||
| 2439 | note = {Accessed 3 November 2026} | 2392 | note = {Accessed 3 November 2026} | ||
| 2440 | } | 2393 | } | ||
| 2394 | + | ||||
| 2395 | +@article{culbert2025openalex, | ||||
| 2396 | + author = {Culbert, Jack H. and Hobert, Anne and Jahn, Najko and Haupka, Nick and Schmidt, Marion and Donner, Paul and Mayr, Philipp}, | ||||
| 2397 | + title = {Reference coverage analysis of {OpenAlex} compared to {Web of Science} and {Scopus}}, | ||||
| 2398 | + journal = {Scientometrics}, | ||||
| 2399 | + year = {2025}, | ||||
| 2400 | + volume = {130}, | ||||
| 2401 | + pages = {2475--2492}, | ||||
| 2402 | + doi = {10.1007/s11192-025-05293-3}, | ||||
| 2403 | + note = {verified: crossref 2026-11-03} | ||||
| 2404 | +} | ||||
| 2405 | + | ||||
| 2406 | +@article{fortunato2018science, | ||||
| 2407 | + author = {Fortunato, Santo and Bergstrom, Carl T. and B{\"o}rner, Katy and Evans, James A. and Helbing, Dirk and Milojevi{\'c}, Sta{\v{s}}a and Petersen, Alexander M. and Radicchi, Filippo and Sinatra, Roberta and Uzzi, Brian and Vespignani, Alessandro and Waltman, Ludo and Wang, Dashun and Barab{\'a}si, Albert-L{\'a}szl{\'o}}, | ||||
| 2408 | + title = {Science of science}, | ||||
| 2409 | + journal = {Science}, | ||||
| 2410 | + year = {2018}, | ||||
| 2411 | + volume = {359}, | ||||
| 2412 | + number = {6379}, | ||||
| 2413 | + pages = {eaao0185}, | ||||
| 2414 | + doi = {10.1126/science.aao0185}, | ||||
| 2415 | + note = {verified: crossref 2026-11-03} | ||||
| 2416 | +} | ||||
modified
manuscript/references_part03.bib
+0 −40
@@ -34,15 +34,6 @@ | |||||
| 34 | howpublished = {arXiv preprint arXiv:1905.00075}, | 34 | howpublished = {arXiv preprint arXiv:1905.00075}, | ||
| 35 | note = {verified: arxiv 2026-11-03} | 35 | note = {verified: arxiv 2026-11-03} | ||
| 36 | } | 36 | } | ||
| 37 | −@misc{kinney2023s2, | ||||
| 38 | − author = {Kinney, Rodney and Anastasiades, Chloe and Authur, Russell and Beltagy, Iz and Bragg, Jonathan and Buraczynski, Alexandra and Cachola, Isabel and Candra, Stefan and Chandrasekhar, Yoganand and Cohan, Arman and others}, | ||||
| 39 | − title = {The {Semantic Scholar} Open Data Platform}, | ||||
| 40 | − year = {2023}, | ||||
| 41 | − eprint = {2301.10140}, | ||||
| 42 | − archiveprefix = {arXiv}, | ||||
| 43 | − howpublished = {arXiv preprint arXiv:2301.10140}, | ||||
| 44 | − note = {verified: arxiv 2026-11-03} | ||||
| 45 | −} | ||||
| 46 | @article{culbert2025openalex, | 37 | @article{culbert2025openalex, | ||
| 47 | author = {Culbert, Jack H. and Hobert, Anne and Jahn, Najko and Haupka, Nick and Schmidt, Marion and Donner, Paul and Mayr, Philipp}, | 38 | author = {Culbert, Jack H. and Hobert, Anne and Jahn, Najko and Haupka, Nick and Schmidt, Marion and Donner, Paul and Mayr, Philipp}, | ||
| 48 | title = {Reference coverage analysis of {OpenAlex} compared to {Web of Science} and {Scopus}}, | 39 | title = {Reference coverage analysis of {OpenAlex} compared to {Web of Science} and {Scopus}}, | ||
@@ -72,16 +63,6 @@ | |||||
| 72 | doi = {10.1162/qss_a_00286}, | 63 | doi = {10.1162/qss_a_00286}, | ||
| 73 | note = {verified: crossref 2026-11-03} | 64 | note = {verified: crossref 2026-11-03} | ||
| 74 | } | 65 | } | ||
| 75 | −@article{martinmartin2021comparison, | ||||
| 76 | − author = {Mart{\'i}n-Mart{\'i}n, Alberto and Thelwall, Mike and Orduna-Malea, Enrique and Delgado L{\'o}pez-C{\'o}zar, Emilio}, | ||||
| 77 | − title = {{Google Scholar}, {Microsoft Academic}, {Scopus}, {Dimensions}, {Web of Science}, and {OpenCitations}' {COCI}: a multidisciplinary comparison of coverage via citations}, | ||||
| 78 | − journal = {Scientometrics}, | ||||
| 79 | − year = {2021}, | ||||
| 80 | − volume = {126}, | ||||
| 81 | − pages = {871--906}, | ||||
| 82 | − doi = {10.1007/s11192-020-03690-4}, | ||||
| 83 | − note = {verified: crossref 2026-11-03} | ||||
| 84 | −} | ||||
| 85 | @article{mongeon2016coverage, | 66 | @article{mongeon2016coverage, | ||
| 86 | author = {Mongeon, Philippe and Paul-Hus, Ad{\`e}le}, | 67 | author = {Mongeon, Philippe and Paul-Hus, Ad{\`e}le}, | ||
| 87 | title = {The journal coverage of {Web of Science} and {Scopus}: a comparative analysis}, | 68 | title = {The journal coverage of {Web of Science} and {Scopus}: a comparative analysis}, | ||
@@ -92,16 +73,6 @@ | |||||
| 92 | doi = {10.1007/s11192-015-1765-5}, | 73 | doi = {10.1007/s11192-015-1765-5}, | ||
| 93 | note = {verified: crossref 2026-11-03} | 74 | note = {verified: crossref 2026-11-03} | ||
| 94 | } | 75 | } | ||
| 95 | −@article{radev2013acl, | ||||
| 96 | − author = {Radev, Dragomir R. and Muthukrishnan, Pradeep and Qazvinian, Vahed and Abu-Jbara, Amjad}, | ||||
| 97 | − title = {The {ACL} anthology network corpus}, | ||||
| 98 | − journal = {Language Resources and Evaluation}, | ||||
| 99 | − year = {2013}, | ||||
| 100 | − volume = {47}, | ||||
| 101 | − pages = {919--944}, | ||||
| 102 | − doi = {10.1007/s10579-012-9211-2}, | ||||
| 103 | − note = {verified: crossref 2026-11-03} | ||||
| 104 | −} | ||||
| 105 | @article{fellegi1969linkage, | 76 | @article{fellegi1969linkage, | ||
| 106 | author = {Fellegi, Ivan P. and Sunter, Alan B.}, | 77 | author = {Fellegi, Ivan P. and Sunter, Alan B.}, | ||
| 107 | title = {A theory for record linkage}, | 78 | title = {A theory for record linkage}, | ||
@@ -152,17 +123,6 @@ | |||||
| 152 | doi = {10.1007/s11192-012-0775-9}, | 123 | doi = {10.1007/s11192-012-0775-9}, | ||
| 153 | note = {verified: crossref 2026-11-03 (online 2012, volume 94 = 2013)} | 124 | note = {verified: crossref 2026-11-03 (online 2012, volume 94 = 2013)} | ||
| 154 | } | 125 | } | ||
| 155 | −@article{fortunato2018science, | ||||
| 156 | − author = {Fortunato, Santo and Bergstrom, Carl T. and B{\"o}rner, Katy and Evans, James A. and Helbing, Dirk and Milojevi{\'c}, Sta{\v{s}}a and Petersen, Alexander M. and Radicchi, Filippo and Sinatra, Roberta and Uzzi, Brian and Vespignani, Alessandro and Waltman, Ludo and Wang, Dashun and Barab{\'a}si, Albert-L{\'a}szl{\'o}}, | ||||
| 157 | − title = {Science of science}, | ||||
| 158 | − journal = {Science}, | ||||
| 159 | − year = {2018}, | ||||
| 160 | − volume = {359}, | ||||
| 161 | − number = {6379}, | ||||
| 162 | − pages = {eaao0185}, | ||||
| 163 | − doi = {10.1126/science.aao0185}, | ||||
| 164 | − note = {verified: crossref 2026-11-03} | ||||
| 165 | −} | ||||
| 166 | @article{nosek2018preregistration, | 126 | @article{nosek2018preregistration, | ||
| 167 | author = {Nosek, Brian A. and Ebersole, Charles R. and DeHaven, Alexander C. and Mellor, David T.}, | 127 | author = {Nosek, Brian A. and Ebersole, Charles R. and DeHaven, Alexander C. and Mellor, David T.}, | ||
| 168 | title = {The preregistration revolution}, | 128 | title = {The preregistration revolution}, | ||
modified
manuscript/references_part04.bib
+0 −54
@@ -12,27 +12,6 @@ | |||||
| 12 | doi = {10.1016/j.joi.2016.02.007} | 12 | doi = {10.1016/j.joi.2016.02.007} | ||
| 13 | } | 13 | } | ||
| 14 | 14 | ||||
| 15 | −@incollection{waltman2019fieldnorm, | ||||
| 16 | − author = {Waltman, Ludo and van Eck, Nees Jan}, | ||||
| 17 | − title = {Field Normalization of Scientometric Indicators}, | ||||
| 18 | − booktitle = {Springer Handbook of Science and Technology Indicators}, | ||||
| 19 | − publisher = {Springer}, | ||||
| 20 | − year = {2019}, | ||||
| 21 | − pages = {281--300}, | ||||
| 22 | − doi = {10.1007/978-3-030-02511-3_11} | ||||
| 23 | −} | ||||
| 24 | − | ||||
| 25 | −@article{wang2013window, | ||||
| 26 | − author = {Wang, Jian}, | ||||
| 27 | − title = {Citation time window choice for research impact evaluation}, | ||||
| 28 | − journal = {Scientometrics}, | ||||
| 29 | − year = {2013}, | ||||
| 30 | − volume = {94}, | ||||
| 31 | − number = {3}, | ||||
| 32 | − pages = {851--872}, | ||||
| 33 | − doi = {10.1007/s11192-012-0775-9} | ||||
| 34 | −} | ||||
| 35 | − | ||||
| 36 | @misc{openalex2026fwci, | 15 | @misc{openalex2026fwci, | ||
| 37 | author = {{OpenAlex}}, | 16 | author = {{OpenAlex}}, | ||
| 38 | title = {Works: attributes ({F}ield-{W}eighted {C}itation {I}mpact, \texttt{fwci})}, | 17 | title = {Works: attributes ({F}ield-{W}eighted {C}itation {I}mpact, \texttt{fwci})}, | ||
@@ -237,17 +216,6 @@ | |||||
| 237 | doi = {10.1007/BF01908075} | 216 | doi = {10.1007/BF01908075} | ||
| 238 | } | 217 | } | ||
| 239 | 218 | ||||
| 240 | −@article{uzzi2013atypical, | ||||
| 241 | − author = {Uzzi, Brian and Mukherjee, Satyam and Stringer, Michael and Jones, Ben}, | ||||
| 242 | − title = {Atypical combinations and scientific impact}, | ||||
| 243 | − journal = {Science}, | ||||
| 244 | − year = {2013}, | ||||
| 245 | − volume = {342}, | ||||
| 246 | − number = {6157}, | ||||
| 247 | − pages = {468--472}, | ||||
| 248 | − doi = {10.1126/science.1240474} | ||||
| 249 | −} | ||||
| 250 | − | ||||
| 251 | @article{wang2017novelty, | 219 | @article{wang2017novelty, | ||
| 252 | author = {Wang, Jian and Veugelers, Reinhilde and Stephan, Paula}, | 220 | author = {Wang, Jian and Veugelers, Reinhilde and Stephan, Paula}, | ||
| 253 | title = {Bias against novelty in science: A cautionary tale for users of bibliometric indicators}, | 221 | title = {Bias against novelty in science: A cautionary tale for users of bibliometric indicators}, | ||
@@ -270,17 +238,6 @@ | |||||
| 270 | doi = {10.1093/ije/dyw098} | 238 | doi = {10.1093/ije/dyw098} | ||
| 271 | } | 239 | } | ||
| 272 | 240 | ||||
| 273 | −@article{bernal2018controls, | ||||
| 274 | − author = {Lopez Bernal, James and Cummins, Steven and Gasparrini, Antonio}, | ||||
| 275 | − title = {The use of controls in interrupted time series studies of public health interventions}, | ||||
| 276 | − journal = {International Journal of Epidemiology}, | ||||
| 277 | − year = {2018}, | ||||
| 278 | − volume = {47}, | ||||
| 279 | − number = {6}, | ||||
| 280 | − pages = {2082--2093}, | ||||
| 281 | − doi = {10.1093/ije/dyy135} | ||||
| 282 | −} | ||||
| 283 | − | ||||
| 284 | @article{newey1987hac, | 241 | @article{newey1987hac, | ||
| 285 | author = {Newey, Whitney K. and West, Kenneth D.}, | 242 | author = {Newey, Whitney K. and West, Kenneth D.}, | ||
| 286 | title = {A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix}, | 243 | title = {A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix}, | ||
@@ -314,17 +271,6 @@ | |||||
| 314 | doi = {10.2307/2998540} | 271 | doi = {10.2307/2998540} | ||
| 315 | } | 272 | } | ||
| 316 | 273 | ||||
| 317 | −@article{bai2003computation, | ||||
| 318 | − author = {Bai, Jushan and Perron, Pierre}, | ||||
| 319 | − title = {Computation and analysis of multiple structural change models}, | ||||
| 320 | − journal = {Journal of Applied Econometrics}, | ||||
| 321 | − year = {2003}, | ||||
| 322 | − volume = {18}, | ||||
| 323 | − number = {1}, | ||||
| 324 | − pages = {1--22}, | ||||
| 325 | − doi = {10.1002/jae.659} | ||||
| 326 | −} | ||||
| 327 | − | ||||
| 328 | @article{liu1997segmented, | 274 | @article{liu1997segmented, | ||
| 329 | author = {Liu, Jian and Wu, Shiying and Zidek, James V.}, | 275 | author = {Liu, Jian and Wu, Shiying and Zidek, James V.}, | ||
| 330 | title = {On segmented multivariate regression}, | 276 | title = {On segmented multivariate regression}, | ||
modified
manuscript/sections/part02_evolution.tex
+13 −13
@@ -21,7 +21,7 @@ The \emph{arXiv subject classes} are the second system. arXiv was created in 199 | |||||
| 21 | 21 | ||||
| 22 | The \emph{DBLP computer science bibliography} is the third. DBLP does not classify papers at all; it classifies \emph{venues}. Begun in 1993 as a database and logic programming bibliography and extended to the whole of computer science, it records the tables of contents of conferences and journals with curated author identities \citep{ley2002dblp}. Membership in DBLP is itself a judgement that a venue belongs to computer science, and the venue stream to which a paper belongs is a strong signal of community: a paper in the proceedings of OSDI is an operating-systems paper in the sociological sense whatever its subject matter. DBLP therefore encodes the discipline as a set of communities rather than as a set of topics, and the two encodings disagree precisely where this paper expects change, namely where machine-learning methods enter venues that were not built for them. | 22 | The \emph{DBLP computer science bibliography} is the third. DBLP does not classify papers at all; it classifies \emph{venues}. Begun in 1993 as a database and logic programming bibliography and extended to the whole of computer science, it records the tables of contents of conferences and journals with curated author identities \citep{ley2002dblp}. Membership in DBLP is itself a judgement that a venue belongs to computer science, and the venue stream to which a paper belongs is a strong signal of community: a paper in the proceedings of OSDI is an operating-systems paper in the sociological sense whatever its subject matter. DBLP therefore encodes the discipline as a set of communities rather than as a set of topics, and the two encodings disagree precisely where this paper expects change, namely where machine-learning methods enter venues that were not built for them. | ||
| 23 | 23 | ||||
| 24 | −The fourth system is the family of \emph{algorithmic, citation-based classifications} that underlie modern open bibliographic indexes. OpenAlex, which succeeded the Microsoft Academic Graph after the latter was discontinued at the end of 2021 \citep{wang2020mag,priem2022openalex}, assigns to every work a ranked list of \emph{topics} from a fixed vocabulary of about 4{,}500 topics grouped into subfields, fields and domains; the topics were obtained by clustering the citation network and labelling the clusters with a language model, and are assigned to new works by a classifier over title, abstract and venue \citep{openalex2025topics}. The approach descends from the publication-level clustering methodology of \citet{waltman2012classification}, refined by the Leiden community-detection algorithm \citep{traag2019leiden}, and it inherits the known properties of that family: clusters are well connected and reproducible, but their granularity is a free parameter with no natural value \citep{sjogarde2018granularity}, and citation-based taxonomies, while more accurate than text-based or journal-based ones on average \citep{klavans2017taxonomy}, lag behind the literature by construction because a cluster must accumulate citations before it can exist. The Semantic Scholar academic graph offers a comparable but differently constructed resource \citep{kinney2023s2}. Older, journal-based classifications of the kind proposed by \citet{glanzel2003classification} remain in use in evaluative scientometrics but are too coarse, and too journal-centric, for a conference-dominated discipline. | 24 | +The fourth system is the family of \emph{algorithmic, citation-based classifications} that underlie modern open bibliographic indexes. OpenAlex, which succeeded the Microsoft Academic Graph after the latter was discontinued at the end of 2021 \citep{wang2020mag,priem2022openalex}, assigns to every work a ranked list of \emph{topics} from a fixed vocabulary of about 4{,}500 topics grouped into subfields, fields and domains; the topics were obtained by clustering the citation network and labelling the clusters with a language model, and are assigned to new works by a classifier over title, abstract and venue \citep{openalex2024topics}. The approach descends from the publication-level clustering methodology of \citet{waltman2012classification}, refined by the Leiden community-detection algorithm \citep{traag2019leiden}, and it inherits the known properties of that family: clusters are well connected and reproducible, but their granularity is a free parameter with no natural value \citep{sjogarde2018granularity}, and citation-based taxonomies, while more accurate than text-based or journal-based ones on average \citep{klavans2017taxonomy}, lag behind the literature by construction because a cluster must accumulate citations before it can exist. The Semantic Scholar academic graph offers a comparable but differently constructed resource \citep{kinney2023s2ag}. Older, journal-based classifications of the kind proposed by \citet{glanzel2003classification} remain in use in evaluative scientometrics but are too coarse, and too journal-centric, for a conference-dominated discipline. | ||
| 25 | 25 | ||||
| 26 | Each of these systems is a model of the discipline with its own inductive bias. The CCS encodes expert judgement frozen in 2012; arXiv classes encode author self-positioning at submission; DBLP streams encode venue communities; OpenAlex topics encode citation neighbourhoods as of a recent snapshot. None of them is ground truth, and this study treats their disagreements as data rather than as noise. | 26 | Each of these systems is a model of the discipline with its own inductive bias. The CCS encodes expert judgement frozen in 2012; arXiv classes encode author self-positioning at submission; DBLP streams encode venue communities; OpenAlex topics encode citation neighbourhoods as of a recent snapshot. None of them is ground truth, and this study treats their disagreements as data rather than as noise. | ||
| 27 | 27 | ||||
@@ -46,22 +46,22 @@ The ontology's limitations are stated in Appendix~A and are not repeated here, e | |||||
| 46 | 46 | ||||
| 47 | The disagreements among the four systems are systematic, and they fall into three kinds. | 47 | The disagreements among the four systems are systematic, and they fall into three kinds. | ||
| 48 | 48 | ||||
| 49 | −The first is \emph{coverage}. Comparisons of bibliographic databases consistently show that no source indexes the whole of a discipline and that the gaps are not random: conference proceedings, preprints, and non-English venues are the most unevenly covered \citep{visser2021comparison,martinmartin2021coverage}. OpenAlex, the backbone of this corpus, inherits the broad but uneven coverage of Microsoft Academic and Crossref; its reference lists are less complete than those of the commercial indexes for part of the period, which affects citation-based measures more than counts (a preprint finding; \citealp{culbert2024openalex}. The linking phase of this study illustrates the scale of the problem at the level of record identity. The OpenAlex snapshot contains \LinkOaCsRecords{} works classified in computer science between 2018 and 2026; DBLP contributes \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} records over the same years, of which \LinkDblpLinkedDistinct{} could be matched to an OpenAlex work by DOI or by title and year in the first linking pass and \LinkDblpUnlinked{} could not; the arXiv \texttt{cs} harvest contributes \LinkArxivCsRecords{} preprints, of which \LinkArxivLinkedDistinct{} were matched, overwhelmingly through the arXiv identifier that OpenAlex stores. These are linkage statistics, not findings about research, but they establish that a classification applied to one source cannot simply be transferred to another: a non-trivial fraction of the venue-defined (DBLP) universe has no citation-defined (OpenAlex) counterpart, and vice versa. Part~III describes the second linking pass and the treatment of unmatched records. | 49 | +The first is \emph{coverage}. Comparisons of bibliographic databases consistently show that no source indexes the whole of a discipline and that the gaps are not random: conference proceedings, preprints, and non-English venues are the most unevenly covered \citep{visser2021comparison,martinmartin2021coverage}. OpenAlex, the backbone of this corpus, inherits the broad but uneven coverage of Microsoft Academic and Crossref; its reference lists are less complete than those of the commercial indexes for part of the period, which affects citation-based measures more than counts \citep{culbert2025openalex}. The linking phase of this study illustrates the scale of the problem at the level of record identity. The first pass over the OpenAlex snapshot retained \PrismaIdentifiedOpenalexPassOneCsTopicFilter{} works that carry a computer-science topic and a publication year between 2018 and 2026; a second pass added \PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} works that OpenAlex labels with no computer-science topic but that DBLP or arXiv identifiers reach, for \PrismaIdentifiedOpenalexTotalRecordsLoaded{} records loaded in all. DBLP contributes \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} records over the same years, of which \LinkDblpLinkedDistinct{} could be matched to an OpenAlex work by DOI or by title and year once both passes had been linked and \LinkDblpUnlinked{} could not; the arXiv \texttt{cs} harvest contributes \LinkArxivCsRecords{} preprints, of which \LinkArxivLinkedDistinct{} were matched, overwhelmingly through the arXiv identifier that OpenAlex stores. These are linkage statistics, not findings about research, but they establish that a classification applied to one source cannot simply be transferred to another: a non-trivial fraction of the venue-defined (DBLP) universe has no citation-defined (OpenAlex) counterpart, and vice versa. Part~III describes the two passes and the treatment of unmatched records. | ||
| 50 | 50 | ||||
| 51 | The second kind of disagreement is \emph{granularity and boundary}. The arXiv class \texttt{cs.LG} is one node among forty; in OpenAlex, machine learning spreads across dozens of topics in several fields, including statistics, engineering and medicine, because the citation network does not respect the boundary of the computer-science field. Conversely, OpenAlex's field~17 (``Computer Science'') contains clusters on educational technology, media studies and physics that no computer scientist would claim, and the ontology of this study deliberately leaves most of them unmapped (Appendix~A). The practical consequence is that the apparent size of ``AI'' depends on which boundary is used: a venue-based definition (DBLP streams of NeurIPS, ICML, ICLR, AAAI and the vision and language conferences) gives a different set from a topic-based definition, which gives a different set again from an arXiv-class definition, and the three sets diverge most in the applied sciences, where machine-learning methods are used without the paper belonging to any machine-learning community. The multi-label design of the ontology, and the use of all three signal types with capped weights in equation~\eqref{eq:p2-score}, is the response to this divergence: a work that is an AI paper by every criterion scores high on all of them, while a work that is an AI paper by vocabulary alone is held to a higher evidentiary standard. | 51 | The second kind of disagreement is \emph{granularity and boundary}. The arXiv class \texttt{cs.LG} is one node among forty; in OpenAlex, machine learning spreads across dozens of topics in several fields, including statistics, engineering and medicine, because the citation network does not respect the boundary of the computer-science field. Conversely, OpenAlex's field~17 (``Computer Science'') contains clusters on educational technology, media studies and physics that no computer scientist would claim, and the ontology of this study deliberately leaves most of them unmapped (Appendix~A). The practical consequence is that the apparent size of ``AI'' depends on which boundary is used: a venue-based definition (DBLP streams of NeurIPS, ICML, ICLR, AAAI and the vision and language conferences) gives a different set from a topic-based definition, which gives a different set again from an arXiv-class definition, and the three sets diverge most in the applied sciences, where machine-learning methods are used without the paper belonging to any machine-learning community. The multi-label design of the ontology, and the use of all three signal types with capped weights in equation~\eqref{eq:p2-score}, is the response to this divergence: a work that is an AI paper by every criterion scores high on all of them, while a work that is an AI paper by vocabulary alone is held to a higher evidentiary standard. | ||
| 52 | 52 | ||||
| 53 | The third kind is \emph{temporal}. The CCS was fixed in 2012 and the arXiv classes are older still; OpenAlex topics were generated from a citation network observed in the early 2020s and are reassigned when the snapshot is rebuilt; DBLP streams are stable but new venues appear and old ones are renamed. A classification that does not change cannot register the emergence of a field except as the growth of an existing node; a classification that is regenerated cannot be compared across snapshots without care. Neither property is a defect, but they pull in opposite directions, and a study of structural change must choose. This study freezes all classifications at a single snapshot (Part~III), so that any growth it reports is growth of a fixed category, and it treats the emergence of \emph{new} categories as a separate detection problem (Part~IV, Section~\ref{sec:p2-claims}). | 53 | The third kind is \emph{temporal}. The CCS was fixed in 2012 and the arXiv classes are older still; OpenAlex topics were generated from a citation network observed in the early 2020s and are reassigned when the snapshot is rebuilt; DBLP streams are stable but new venues appear and old ones are renamed. A classification that does not change cannot register the emergence of a field except as the growth of an existing node; a classification that is regenerated cannot be compared across snapshots without care. Neither property is a defect, but they pull in opposite directions, and a study of structural change must choose. This study freezes all classifications at a single snapshot (Part~III), so that any growth it reports is growth of a fixed category, and it treats the emergence of \emph{new} categories as a separate detection problem (Part~IV, Section~\ref{sec:p2-claims}). | ||
| 54 | 54 | ||||
| 55 | −A final, sociological disagreement should be named explicitly because it affects what counts as a ``paper''. Computer science is unusual among the sciences in treating selective conferences as the primary venue of record, a practice that has been defended on grounds of speed and criticised on grounds of reviewing quality and the fragmentation of the literature \citep{vardi2009conferences,fortnow2009growup}. The convention has two consequences for bibliometrics. Conference papers are indexed less uniformly than journal articles, so coverage disagreements are concentrated in exactly the venues that the discipline values most; and the conference calendar imposes a strong seasonality on publication dates, so that monthly series must be deseasonalised before any break can be read off them. The preprint culture adds a third consequence. Computer scientists came to arXiv late relative to physicists and mathematicians, and a large share of computer-science preprints are never matched to a published version, either because they are not published or because the match fails \citep{lin2020preprints}. A corpus that counts preprints and published versions as separate works will over-count the preprint-heavy subfields; a corpus that links them will under-count whenever linkage fails; and the subfields where linkage fails most often are, again, the fast-moving ones. | 55 | +A final, sociological disagreement should be named explicitly because it affects what counts as a ``paper''. Computer science is unusual among the sciences in treating selective conferences as the primary venue of record, a practice that has been defended on grounds of speed and criticised on grounds of reviewing quality and the fragmentation of the literature \citep{vardi2009conferences,fortnow2009growup}. The convention has two consequences for bibliometrics. Conference papers are indexed less uniformly than journal articles, so coverage disagreements are concentrated in exactly the venues that the discipline values most; and the conference calendar imposes a strong seasonality on publication dates, so that monthly series must be deseasonalised before any break can be read off them. The preprint culture adds a third consequence. Computer scientists came to arXiv late relative to physicists and mathematicians, and a non-trivial share of computer-science preprints are never matched to a published version, either because they are not published or because the match fails: according to its abstract, the matching study of \citet{lin2020preprints} found that about a quarter of the arXiv computer-science preprints of 2008--2017 in its sample had no identifiable published counterpart, and that a further eleven per cent had been published under a different title, which a title-based match would miss. A corpus that counts preprints and published versions as separate works will over-count the preprint-heavy subfields; a corpus that links them will under-count whenever linkage fails; and the subfields where linkage fails most often are, again, the fast-moving ones. | ||
| 56 | 56 | ||||
| 57 | \subsection{Why the growth of a field is hard to measure} | 57 | \subsection{Why the growth of a field is hard to measure} | ||
| 58 | \label{sec:p2-growth} | 58 | \label{sec:p2-growth} | ||
| 59 | 59 | ||||
| 60 | The questions of this paper are ultimately about rates: of publication, of citation, of movement between categories. Rates presuppose denominators, and the denominators of science are themselves contested quantities. | 60 | The questions of this paper are ultimately about rates: of publication, of citation, of movement between categories. Rates presuppose denominators, and the denominators of science are themselves contested quantities. | ||
| 61 | 61 | ||||
| 62 | −The literature on the growth of science offers two sobering lessons. The first is that measured growth depends heavily on the database. Using cited references as a proxy for the literature that scientists actually use, \citet{bornmann2015growth} estimated the long-run growth of science at roughly three per cent per year with a doubling time of about a quarter of a century, while estimates based on the publication counts of individual databases vary widely with the database's coverage policy and its own expansion. A later analysis with segmented growth curves across several databases found that the growth rate has itself changed over time, with faster growth in recent decades, but that the location of the segments depends on the source \citep{bornmann2021growth}. For the present study the lesson is that a change in the volume of computer-science publications between 2018 and 2026 is the sum of a change in research activity and a change in indexing, and that only the first is of scientific interest. OpenAlex grew substantially during the period as it absorbed new sources and improved its matching; the indexing lag (the delay between a work's publication and its appearance in the snapshot) is longest for the most recent months, so that any series ending in 2026 slopes downward at its end for reasons that have nothing to do with research. These are among the pre-period diagnostics and sensitivity checks required by CLAUDE.md, \S6.5 and specified in Part~IV. | 62 | +The literature on the growth of science offers two sobering lessons. The first is that measured growth depends heavily on the database. Using cited references as a proxy for the literature that scientists actually use, \citet{bornmann2015growth} identified three phases of growth, each at roughly three times the rate of the previous one: under one per cent per year up to the middle of the eighteenth century, two to three per cent up to the inter-war period, and eight to nine per cent per year from then to 2010, a doubling time of about nine years for modern science; estimates based on the publication counts of individual databases vary widely with the database's coverage policy and its own expansion. A later analysis with segmented growth curves across several databases found that the growth rate has itself changed over time, with faster growth in recent decades, but that the location of the segments depends on the source \citep{bornmann2021growth}. For the present study the lesson is that a change in the volume of computer-science publications between 2018 and 2026 is the sum of a change in research activity and a change in indexing, and that only the first is of scientific interest. OpenAlex grew substantially during the period as it absorbed new sources and improved its matching; the indexing lag (the delay between a work's publication and its appearance in the snapshot) is longest for the most recent months, so that any series ending in 2026 slopes downward at its end for reasons that have nothing to do with research. These are among the pre-period diagnostics and sensitivity checks required by CLAUDE.md, \S6.5 and specified in Part~IV. | ||
| 63 | 63 | ||||
| 64 | −The second lesson concerns the ``science of science'' as a field in its own right. The programme described by \citet{fortunato2018sos} and developed at book length by \citet{wang2021sos} has established robust regularities: citation distributions are heavy-tailed within every field; team size has grown steadily; novelty in the sense of atypical combinations of prior work is associated with higher impact at the top of the distribution; and the productivity of individual scientists is approximately stable over careers. Those regularities are the background against which any field-specific claim must be read. A rise in the citation inequality of AI papers after 2022, for instance, is interesting only if it exceeds the inequality that heavy-tailed citation distributions produce in every growing field; the hypotheses H1--H8 of the preregistered protocol (Part~III) are formulated as comparisons between the post-2022 period and the baseline for that reason. | 64 | +The second lesson concerns the ``science of science'' as a field in its own right. The programme described by \citet{fortunato2018science} and developed at book length by \citet{wang2021sos} has established robust regularities: citation distributions are heavy-tailed within every field; team size has grown steadily; novelty in the sense of atypical combinations of prior work is associated with higher impact at the top of the distribution; and the productivity of individual scientists is approximately stable over careers. Those regularities are the background against which any field-specific claim must be read. A rise in the citation inequality of AI papers after 2022, for instance, is interesting only if it exceeds the inequality that heavy-tailed citation distributions produce in every growing field; the hypotheses H1--H8 of the preregistered protocol (Part~III) are formulated as comparisons between the post-2022 period and the baseline for that reason. | ||
| 65 | 65 | ||||
| 66 | A further difficulty is specific to computer science and concerns what the discipline takes itself to be. \citet{denning2005science} argued that computer science is a science of information processes and not merely an engineering of artefacts, and \citet{wing2006computational} that ``computational thinking'' is a general intellectual skill rather than a professional one; both arguments were written at a moment when the discipline was defending its boundaries. The post-2022 period raises the converse question: whether the discipline's methods have diffused so widely that the boundary is no longer where the classifications place it. Papers that apply learned models to protein structure, materials, weather or law are classified as computer science by some systems and not others; a transformation that consists partly of such diffusion is partly invisible to a classification that keeps the boundary fixed. The ontology's eighth area, computational science and discovery, and the ``penumbra'' set defined in Part~III (works with a computer-science topic but without venue or arXiv support) exist to make that diffusion observable without inflating the core. | 66 | A further difficulty is specific to computer science and concerns what the discipline takes itself to be. \citet{denning2005science} argued that computer science is a science of information processes and not merely an engineering of artefacts, and \citet{wing2006computational} that ``computational thinking'' is a general intellectual skill rather than a professional one; both arguments were written at a moment when the discipline was defending its boundaries. The post-2022 period raises the converse question: whether the discipline's methods have diffused so widely that the boundary is no longer where the classifications place it. Papers that apply learned models to protein structure, materials, weather or law are classified as computer science by some systems and not others; a transformation that consists partly of such diffusion is partly invisible to a classification that keeps the boundary fixed. The ontology's eighth area, computational science and discovery, and the ``penumbra'' set defined in Part~III (works with a computer-science topic but without venue or arXiv support) exist to make that diffusion observable without inflating the core. | ||
| 67 | 67 | ||||
@@ -139,7 +139,7 @@ Visibility in classifications & Accommodated by existing top-level nodes & New v | |||||
| 139 | 139 | ||||
| 140 | Three conclusions follow from the comparison and shape the hypotheses tested later. First, the mechanisms of propagation differ in kind. The web changed the problems of other subfields; deep learning changed their methods; the transformer era changed the \emph{models} they use, which can be adopted without changing either problems or methods. A subfield can therefore register the third transformation at low cost, by using a pretrained model as a component, and the resulting growth in the vocabulary of the subfield need not indicate a reorganisation of its research. Distinguishing component adoption from reorganisation is the purpose of the citation-flow and embedding analyses of Parts~IV and~VIII. | 140 | Three conclusions follow from the comparison and shape the hypotheses tested later. First, the mechanisms of propagation differ in kind. The web changed the problems of other subfields; deep learning changed their methods; the transformer era changed the \emph{models} they use, which can be adopted without changing either problems or methods. A subfield can therefore register the third transformation at low cost, by using a pretrained model as a component, and the resulting growth in the vocabulary of the subfield need not indicate a reorganisation of its research. Distinguishing component adoption from reorganisation is the purpose of the citation-flow and embedding analyses of Parts~IV and~VIII. | ||
| 141 | 141 | ||||
| 142 | −Second, each wave shortened the publication cycle and shifted the version of record toward the preprint. The consequence for this study is that the arXiv harvest and the OpenAlex index measure different things: the former records when a result was first made public, the latter when it entered the citation system. A break detected in one series and not in the other is informative about the mechanism, not merely about measurement. | 142 | +Second, each wave shortened the publication cycle and shifted the version of record toward the preprint. The consequence for this study is that the arXiv harvest and the OpenAlex index measure different things: the former records when a result was first made public, the latter when it entered the citation system. A break detected in one series and not in the other may be informative about the mechanism, but it may equally reflect indexing lag, linkage failure or right truncation (Part~III, Section~\ref{sec:data-dates}); only the date diagnostics of Part~III separate the two readings. | ||
| 143 | 143 | ||||
| 144 | Third, the three waves were cumulative in infrastructure and in institutions. The industrial laboratories that produced the web systems trained the generation that produced deep learning, and the frameworks and compute of the deep-learning wave made the transformer era possible. The post-2022 period inherited all of it: the laboratories, the frameworks, the model hubs, the benchmark culture, and the preprint norm. Whatever is new about the period must be identified against that inheritance rather than attributed to a single product release, and the baseline window of 2018--2022 was chosen so that the inheritance is observed directly rather than assumed. | 144 | Third, the three waves were cumulative in infrastructure and in institutions. The industrial laboratories that produced the web systems trained the generation that produced deep learning, and the frameworks and compute of the deep-learning wave made the transformer era possible. The post-2022 period inherited all of it: the laboratories, the frameworks, the model hubs, the benchmark culture, and the preprint norm. Whatever is new about the period must be identified against that inheritance rather than attributed to a single product release, and the baseline window of 2018--2022 was chosen so that the inheritance is observed directly rather than assumed. | ||
| 145 | 145 | ||||
@@ -148,9 +148,9 @@ Third, the three waves were cumulative in infrastructure and in institutions. Th | |||||
| 148 | 148 | ||||
| 149 | The three waves left computer science, on the eve of the reference date, with a publication system unlike that of any other science, and the system's properties determine what the corpus can and cannot see. Four of them are load-bearing for the analyses that follow. | 149 | The three waves left computer science, on the eve of the reference date, with a publication system unlike that of any other science, and the system's properties determine what the corpus can and cannot see. Four of them are load-bearing for the analyses that follow. | ||
| 150 | 150 | ||||
| 151 | −The first is the dominance of the preprint. By 2022 the arXiv was the first and often the only public version of most machine-learning results, and a growing share of systems, theory and security results as well; the matching study of \citet{lin2020preprints} found that a majority of computer-science preprints in their sample could not be matched to a published version, and that the matched ones had been published with a median delay of many months. The consequence is that the arXiv series lead the indexed series, and that a corpus which dates works by their indexed publication date will place a change later than it occurred and will smear it over the publication lag. Part~III dates each work by the earliest of its preprint and publication dates when both are known, and Part~IX reports the breaks on both clocks. | 151 | +The first is the dominance of the preprint. By 2022 the arXiv was the first and often the only public version of most machine-learning results, and a growing share of systems, theory and security results as well; the matching study of \citet{lin2020preprints} found, according to its abstract, that about three quarters of the arXiv computer-science preprints of 2008--2017 in its sample had been published in a peer-reviewed venue (66 per cent under an unchanged title and 11 per cent under a different one), so that roughly a quarter had no identifiable published version. The consequence is that the arXiv series lead the indexed series, and that a corpus which dates works by their indexed publication date will place a change later than it occurred and will smear it over the publication lag. Part~III dates each work by the earliest of its preprint and publication dates when both are known, and Part~IX reports the breaks on both clocks. | ||
| 152 | 152 | ||||
| 153 | −The second is the conference calendar. The major venues of every area have fixed annual deadlines, and preprint postings cluster in the weeks before and after them; monthly series of computer-science output are therefore strongly seasonal, with a pattern that differs by area because the calendars differ. Seasonality is not a nuisance to be removed once and forgotten; a change in the \emph{amplitude} of seasonality, for instance because a venue grew or because preprints began to be posted without regard to deadlines, is itself a structural change, and the seasonal component $\mathbf{s}_c(t)$ of equation~\eqref{eq:p2-its} is modelled separately for the baseline and the research period for that reason. | 153 | +The second is the conference calendar. The major venues of every area have fixed annual deadlines, and preprint postings cluster in the weeks before and after them; monthly series of computer-science output are therefore strongly seasonal, with a pattern that differs by area because the calendars differ. Seasonality is not a nuisance to be removed once and forgotten; a change in the \emph{amplitude} of seasonality, for instance because a venue grew or because preprints began to be posted without regard to deadlines, is itself a structural change, but the seasonal component $\mathbf{s}_c(t)$ of equation~\eqref{eq:p2-its} is, in the implementation of Part~IV, a single set of eleven month dummies common to the baseline and the research period, so that a change in the amplitude of seasonality is absorbed by the residuals rather than estimated; this is a limitation of the specification, not a feature of it. | ||
| 154 | 154 | ||||
| 155 | The third is the state of peer review. The volume of submissions to the leading machine-learning conferences had grown by an order of magnitude in a decade; reviewer pools had not. The debate about whether conferences remained an adequate venue of record \citep{vardi2009conferences,fortnow2009growup} was, by 2022, less about prestige than about capacity, and the reproducibility programmes of \citet{pineau2020reproducibility} were one institutional response. The relevance to this study is twofold. Reviewing capacity constrains the number of works that can appear in venue-defined categories, so that growth under capacity constraints shows up as preprint growth and as the founding of new venues rather than as growth in existing streams; and the reviews themselves became, after 2022, one of the first places where the use of language models was measured \citep{liang2024monitoring}. | 155 | The third is the state of peer review. The volume of submissions to the leading machine-learning conferences had grown by an order of magnitude in a decade; reviewer pools had not. The debate about whether conferences remained an adequate venue of record \citep{vardi2009conferences,fortnow2009growup} was, by 2022, less about prestige than about capacity, and the reproducibility programmes of \citet{pineau2020reproducibility} were one institutional response. The relevance to this study is twofold. Reviewing capacity constrains the number of works that can appear in venue-defined categories, so that growth under capacity constraints shows up as preprint growth and as the founding of new venues rather than as growth in existing streams; and the reviews themselves became, after 2022, one of the first places where the use of language models was measured \citep{liang2024monitoring}. | ||
| 156 | 156 | ||||
@@ -187,9 +187,9 @@ The boundary development is the \emph{theory of deep learning}. The empirical ob | |||||
| 187 | \subsection{Programming languages, compilers and formal verification} | 187 | \subsection{Programming languages, compilers and formal verification} | ||
| 188 | \label{sec:p2-pl} | 188 | \label{sec:p2-pl} | ||
| 189 | 189 | ||||
| 190 | −The programming-languages literature of the baseline is marked by the success of a language. Rust, whose ownership type system gives memory safety without garbage collection \citep{matsakis2014rust}, became the first systems language to be adopted at scale on the strength of a safety guarantee; the RustBelt project supplied the semantic foundation by proving, in a separation-logic framework mechanised in Coq, that the safe fragment is sound and that the unsafe library code on which it rests respects the invariants \citep{jung2018rustbelt}. The result is characteristic of the period's formal-methods culture: a verification effort that follows an engineering artefact rather than preceding it, and that uses a proof assistant as its medium. | 190 | +The programming-languages literature of the baseline is marked by the success of a language. Rust, whose ownership type system gives memory safety without garbage collection \citep{matsakis2014rust}, became one of the first systems languages to be adopted at scale on the strength of a safety guarantee; the RustBelt project supplied the semantic foundation by proving, in a separation-logic framework mechanised in Coq, that the safe fragment is sound and that the unsafe library code on which it rests respects the invariants \citep{jung2018rustbelt}. The result is characteristic of the period's formal-methods culture: a verification effort that follows an engineering artefact rather than preceding it, and that uses a proof assistant as its medium. | ||
| 191 | 191 | ||||
| 192 | −Proof assistants themselves reached a new stage. The verified C compiler CompCert \citep{leroy2009compcert} and the verified microkernel seL4 \citep{klein2009sel4} had shown a decade earlier that production software could be proved correct; the Kepler conjecture was formally proved in 2017 after a twelve-year effort \citep{hales2017kepler}; and the Lean theorem prover \citep{demoura2015lean}, re-implemented as a programming language in its fourth version \citep{demoura2021lean4}, acquired a community-maintained mathematical library whose scale and organisation were new in formalised mathematics \citep{mathlib2020}. The relevance to the post-2022 period is direct: Lean and its library became the environment in which language models were later asked to prove theorems, so that formal verification, one of the categories predicted \emph{not} to break in H3, is also one where the contact with AI is most visible in the recent literature. The hypothesis is therefore a sharp one. | 192 | +Proof assistants themselves reached a new stage. The verified C compiler CompCert \citep{leroy2009compcert} and the verified microkernel seL4 \citep{klein2009sel4} had shown a decade earlier that production software could be proved correct; the formal proof of the Kepler conjecture, begun in 2003, was completed in 2014 and published in 2017 \citep{hales2017kepler}; and the Lean theorem prover \citep{demoura2015lean}, re-implemented as a programming language in its fourth version \citep{demoura2021lean4}, acquired a community-maintained mathematical library whose scale and organisation were new in formalised mathematics \citep{mathlib2020}. The relevance to the post-2022 period is direct: Lean and its library became the environment in which language models were later asked to prove theorems, so that formal verification, one of the categories predicted \emph{not} to break in H3, is also one where the contact with AI is most visible in the recent literature. The hypothesis is therefore a sharp one. | ||
| 193 | 193 | ||||
| 194 | Software engineering had already begun to absorb learned models before 2022. The survey of \citet{allamanis2018survey} organised a decade of ``big code'' research around the naturalness hypothesis, that source code is as statistically regular as natural language; the CodeXGLUE benchmark collection \citep{lu2021codexglue} standardised evaluation; and the Codex models \citep{chen2021codex} made program synthesis from natural language a product. The software-engineering category is thus expected to show an early and strong AI signal, and its baseline is not an AI-free one. | 194 | Software engineering had already begun to absorb learned models before 2022. The survey of \citet{allamanis2018survey} organised a decade of ``big code'' research around the naturalness hypothesis, that source code is as statistically regular as natural language; the CodeXGLUE benchmark collection \citep{lu2021codexglue} standardised evaluation; and the Codex models \citep{chen2021codex} made program synthesis from natural language a product. The software-engineering category is thus expected to show an early and strong AI signal, and its baseline is not an AI-free one. | ||
| 195 | 195 | ||||
@@ -248,7 +248,7 @@ Taken together, the baseline descriptions support a conclusion that the empirica | |||||
| 248 | \subsection{What ChatGPT was and was not} | 248 | \subsection{What ChatGPT was and was not} | ||
| 249 | \label{sec:p2-chatgpt} | 249 | \label{sec:p2-chatgpt} | ||
| 250 | 250 | ||||
| 251 | −On 30 November 2022 OpenAI released ChatGPT as a ``research preview'': a conversational interface to a language model of the GPT-3.5 series, fine-tuned by the reinforcement-learning-from-human-feedback method of InstructGPT, free to use through a web browser \citep{openai2022chatgpt}. No technical report accompanied the release; the announcement described the training procedure by reference to \citet{ouyang2022instructgpt} and listed limitations (plausible but incorrect answers, sensitivity to phrasing, verbosity, incomplete refusal of harmful requests) that the subsequent literature would study at length. It is important for the design of this paper to be precise about what was new. The underlying capabilities, few-shot prompting, instruction following, code generation, had been published between 2020 and 2022 \citep{brown2020gpt3,chen2021codex,ouyang2022instructgpt} and were available through a commercial programming interface. The architecture was five years old. What ChatGPT added was an interface that required no programming, a price of zero, and a conversational format that made the model's abilities and failures legible to anyone; its diffusion among the public was by all accounts faster than that of any previous software product, though the figures usually cited for it come from industry analysts rather than from verifiable sources and are not repeated here. | 251 | +On 30 November 2022 OpenAI released ChatGPT as a ``research preview'': a conversational interface to a language model of the GPT-3.5 series, fine-tuned by the reinforcement-learning-from-human-feedback method of InstructGPT, free to use through a web browser \citep{openai2022chatgpt}. No technical report accompanied the release; the announcement described the training procedure by reference to \citet{ouyang2022instructgpt} and listed limitations (plausible but incorrect answers, sensitivity to phrasing, verbosity, incomplete refusal of harmful requests) that the subsequent literature would study at length. It is important for the design of this paper to be precise about what was new. The underlying capabilities, few-shot prompting, instruction following, code generation, had been published between 2020 and 2022 \citep{brown2020gpt3,chen2021codex,ouyang2022instructgpt} and were available through a commercial programming interface. The architecture was five years old. What ChatGPT added was an interface that required no programming, a price of zero, and a conversational format that made the model's abilities and failures legible to anyone; the figures usually cited for the speed of its public adoption come from industry analysts rather than from verifiable sources and are not repeated here. | ||
| 252 | 252 | ||||
| 253 | For a study of the \emph{research} literature, this distinction has a consequence that is easily missed. The research community did not learn about large language models on 30 November 2022; it had been building and publishing them for three years. What changed on that date, if anything, was the attention of everyone else: funders, universities, firms, governments, and the researchers in subfields that had not previously had a reason to engage. If the date marks anything in the corpus, it should mark a change in \emph{who} publishes on language models and \emph{where}, more than a change in what the language-model community itself does. That expectation is formalised in H4 and H6 and in the reading of the cross-area citation flows in Part~VIII. | 253 | For a study of the \emph{research} literature, this distinction has a consequence that is easily missed. The research community did not learn about large language models on 30 November 2022; it had been building and publishing them for three years. What changed on that date, if anything, was the attention of everyone else: funders, universities, firms, governments, and the researchers in subfields that had not previously had a reason to engage. If the date marks anything in the corpus, it should mark a change in \emph{who} publishes on language models and \emph{where}, more than a change in what the language-model community itself does. That expectation is formalised in H4 and H6 and in the reading of the cross-area citation flows in Part~VIII. | ||
| 254 | 254 | ||||
@@ -321,7 +321,7 @@ What this literature does not establish is whether the growth after 2022 differs | |||||
| 321 | 321 | ||||
| 322 | The most direct evidence of a change in the literature after 2022 is a change in its language. In a preprint, \citet{liang2024mapping} estimated, with a distributional method that measures the frequency of words whose use the models favour, that the fraction of abstracts in computer science with substantial LLM-modified content rose rapidly from the end of 2022, with computer science leading other fields; the same group found a comparable rise in the reviews of major machine-learning conferences \citep{liang2024monitoring}, and later extended the method to writing across society \citep{liang2025widespread}; both are cited here in their preprint versions. \citet{kobak2025delving} obtained the same conclusion for biomedical abstracts by a different route, measuring the excess frequency of words such as ``delves'' and ``underscores'' relative to a pre-2023 baseline, in a study designed to need no training data from the models themselves; earlier and smaller preprint analyses of the same marker words had reached the same result \citep{geng2024writing,gray2024contamination}. Surveys of researchers confirm the adoption that the text analyses infer: a Nature survey of 1{,}600 scientists found wide use of generative tools for writing and coding, with strong concerns about reliability and misuse \citep{vannoorden2023survey}, and the multidisciplinary collection assembled by \citet{dwivedi2023sowhat} and the integrity analysis of \citet{binnashwan2023chatgpt} document the reactions of the research and education communities in the first months. | 322 | The most direct evidence of a change in the literature after 2022 is a change in its language. In a preprint, \citet{liang2024mapping} estimated, with a distributional method that measures the frequency of words whose use the models favour, that the fraction of abstracts in computer science with substantial LLM-modified content rose rapidly from the end of 2022, with computer science leading other fields; the same group found a comparable rise in the reviews of major machine-learning conferences \citep{liang2024monitoring}, and later extended the method to writing across society \citep{liang2025widespread}; both are cited here in their preprint versions. \citet{kobak2025delving} obtained the same conclusion for biomedical abstracts by a different route, measuring the excess frequency of words such as ``delves'' and ``underscores'' relative to a pre-2023 baseline, in a study designed to need no training data from the models themselves; earlier and smaller preprint analyses of the same marker words had reached the same result \citep{geng2024writing,gray2024contamination}. Surveys of researchers confirm the adoption that the text analyses infer: a Nature survey of 1{,}600 scientists found wide use of generative tools for writing and coding, with strong concerns about reliability and misuse \citep{vannoorden2023survey}, and the multidisciplinary collection assembled by \citet{dwivedi2023sowhat} and the integrity analysis of \citet{binnashwan2023chatgpt} document the reactions of the research and education communities in the first months. | ||
| 323 | 323 | ||||
| 324 | −For this study the finding is a measurement problem before it is a result. The classification of Section~\ref{sec:p2-ontology} and the embedding methods of Part~IV both read text, and text that is partly produced by a model trained on the pre-2023 literature is, in a precise sense, less informative about what is new in a paper. Two responses are built into the design. The venue and arXiv signals of equation~\eqref{eq:p2-score} are immune to prose; and the robustness checks of Appendix~D re-estimate the main breaks with classifications that use no keyword signals, so that a break present in the keyword-based series and absent in the venue-based one is flagged as possibly linguistic. | 324 | +For this study the finding is a measurement problem before it is a result. The classification of Section~\ref{sec:p2-ontology} and the embedding methods of Part~IV both read text, and text that is partly produced by a model trained on the pre-2023 literature is, in a precise sense, less informative about what is new in a paper. Two responses are built into the design. The venue and arXiv signals of equation~\eqref{eq:p2-score} are immune to prose; and a planned robustness check (Appendix~D) would re-estimate the main breaks with classifications that use no keyword signals, so that a break present in the keyword-based series and absent in the venue-based one could be flagged as possibly linguistic; the classifier does not yet offer that variant, and the check is recorded as planned rather than performed. | ||
| 325 | 325 | ||||
| 326 | The measurement of LLM-generated text in \emph{peer reviews} deserves separate mention because it bears on the quality of the venue signal. \citet{liang2024monitoring} estimated that a non-trivial share of the reviews submitted to several 2024 machine-learning conferences had been substantially modified by a language model, with the share higher among reviews submitted close to the deadline and among reviewers who did not respond to author rebuttals. If reviewing is partly delegated to models, then acceptance at a selective venue, which the ontology treats as the strongest community signal, is partly a judgement rendered by the models under study. The corpus cannot measure this directly; it is recorded here as a limitation of the venue signal in the research period that does not apply to the baseline. | 326 | The measurement of LLM-generated text in \emph{peer reviews} deserves separate mention because it bears on the quality of the venue signal. \citet{liang2024monitoring} estimated that a non-trivial share of the reviews submitted to several 2024 machine-learning conferences had been substantially modified by a language model, with the share higher among reviews submitted close to the deadline and among reviewers who did not respond to author rebuttals. If reviewing is partly delegated to models, then acceptance at a selective venue, which the ontology treats as the strongest community signal, is partly a judgement rendered by the models under study. The corpus cannot measure this directly; it is recorded here as a limitation of the venue signal in the research period that does not apply to the baseline. | ||
| 327 | 327 | ||||
@@ -333,7 +333,7 @@ A second body of work concerns not the quantity of research but its quality. \ci | |||||
| 333 | \subsection{Benchmark saturation and the reproducibility of evaluation} | 333 | \subsection{Benchmark saturation and the reproducibility of evaluation} | ||
| 334 | \label{sec:p2-claim-benchmarks} | 334 | \label{sec:p2-claim-benchmarks} | ||
| 335 | 335 | ||||
| 336 | −The benchmark culture of Section~\ref{sec:p2-dl} produced a literature on its own dynamics. In a preprint, \citet{ott2022benchmarks} mapped the creation and saturation of benchmarks across AI tasks and found that many become saturated within a few years of their creation while others are abandoned before saturation; the critiques of \citet{dehghani2021lottery} and \citet{raji2021benchmark} argued that the choice of benchmark is itself an under-examined research decision and that no benchmark can measure general capability. The post-2022 literature extended the concern to \emph{contamination}, the presence of test data in the training corpora of models trained on the web, which makes a benchmark result uninterpretable without knowledge of the training data that the model reports of Table~\ref{tab:p2-timeline} typically withhold. \citet{kapoor2023leakage} documented data leakage as a general cause of irreproducible results in machine-learning-based science, and the agenda of \citet{hendrycks2021unsolved} placed robustness, monitoring and alignment among the open problems of the field. RQ7, which asks whether the period's advances are breakthroughs or incremental benchmark gains, and RQ9, on reproducibility and evaluation standards, are the corpus-level counterparts of these claims. | 336 | +The benchmark culture of Section~\ref{sec:p2-dl} produced a literature on its own dynamics. \citet{ott2022benchmarks} mapped the creation and saturation of benchmarks across AI tasks and found that many become saturated within a few years of their creation while others are abandoned before saturation; the critiques of \citet{dehghani2021lottery} and \citet{raji2021benchmark} argued that the choice of benchmark is itself an under-examined research decision and that no benchmark can measure general capability. The post-2022 literature extended the concern to \emph{contamination}, the presence of test data in the training corpora of models trained on the web, which makes a benchmark result uninterpretable without knowledge of the training data that the model reports of Table~\ref{tab:p2-timeline} typically withhold. \citet{kapoor2023leakage} documented data leakage as a general cause of irreproducible results in machine-learning-based science, and the agenda of \citet{hendrycks2021unsolved} placed robustness, monitoring and alignment among the open problems of the field. RQ7, which asks whether the period's advances are breakthroughs or incremental benchmark gains, and RQ9, on reproducibility and evaluation standards, are the corpus-level counterparts of these claims. | ||
| 337 | 337 | ||||
| 338 | \subsection{Concentration, disruption and the pace of science} | 338 | \subsection{Concentration, disruption and the pace of science} | ||
| 339 | \label{sec:p2-claim-pace} | 339 | \label{sec:p2-claim-pace} | ||
modified
manuscript/sections/part03_data.tex
+143 −87
@@ -29,7 +29,7 @@ coverage of DBLP, the publisher-centred coverage of Crossref, the preprint-centr | |||||
| 29 | aggregating coverage of OpenAlex each miss a distinct part of the literature and type the part they do cover | 29 | aggregating coverage of OpenAlex each miss a distinct part of the literature and type the part they do cover | ||
| 30 | in distinct ways. Comparative studies of bibliographic data sources have documented large differences in | 30 | in distinct ways. Comparative studies of bibliographic data sources have documented large differences in | ||
| 31 | coverage between Web of Science, Scopus, Dimensions, Crossref and Microsoft Academic | 31 | coverage between Web of Science, Scopus, Dimensions, Crossref and Microsoft Academic | ||
| 32 | −\citep{visser2021comparison,martinmartin2021comparison}, and the journal-level coverage of the two commercial | 32 | +\citep{visser2021comparison,martinmartin2021coverage}, and the journal-level coverage of the two commercial | ||
| 33 | indexes is known to under-represent conference proceedings, which carry a large share of computer-science | 33 | indexes is known to under-represent conference proceedings, which carry a large share of computer-science | ||
| 34 | output \citep{mongeon2016coverage}. For the open successor of Microsoft Academic, OpenAlex, the first | 34 | output \citep{mongeon2016coverage}. For the open successor of Microsoft Academic, OpenAlex, the first | ||
| 35 | independent assessments report strong coverage of works with DOIs together with weaker and uneven metadata on | 35 | independent assessments report strong coverage of works with DOIs together with weaker and uneven metadata on | ||
@@ -58,7 +58,9 @@ before parsing; every one of the 1{,}898 files parsed without error and the sum | |||||
| 58 | equals the manifest total of \LinkSlimTotal{} works. Two outputs were written during this single pass. The | 58 | equals the manifest total of \LinkSlimTotal{} works. Two outputs were written during this single pass. The | ||
| 59 | first is the full JSON record of every work carrying \emph{any} topic in OpenAlex field~17 (Computer Science) | 59 | first is the full JSON record of every work carrying \emph{any} topic in OpenAlex field~17 (Computer Science) | ||
| 60 | with a publication year between 2018 and 2026, stored as Zstandard-compressed JSON Lines and partitioned by | 60 | with a publication year between 2018 and 2026, stored as Zstandard-compressed JSON Lines and partitioned by | ||
| 61 | −the snapshot's own date partitions (\LinkOaCsRecords{} records, about 20~GB). The second is a \emph{slim} | 61 | +the snapshot's own date partitions (\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} records in this first | ||
| 62 | +pass, about 20~GB; a second pass by identifier list, described in Section~\ref{sec:data-linkage}, later added | ||||
| 63 | +\PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} records). The second is a \emph{slim} | ||||
| 62 | table of every work in the snapshot (identifier, DOI, year, type, primary topic, subfield, field and domain, | 64 | table of every work in the snapshot (identifier, DOI, year, type, primary topic, subfield, field and domain, | ||
| 63 | citation count and a few flags), kept in Parquet so that works cited by computer-science papers but published | 65 | citation count and a few flags), kept in Parquet so that works cited by computer-science papers but published | ||
| 64 | in other fields can be resolved for the interdisciplinary analyses of Part~VIII (decision D2 of | 66 | in other fields can be resolved for the interdisciplinary analyses of Part~VIII (decision D2 of | ||
@@ -119,8 +121,8 @@ the caveats attached to its category labels and versioning, are discussed by \ci | |||||
| 119 | \subsection{ACL Anthology} | 121 | \subsection{ACL Anthology} | ||
| 120 | 122 | ||||
| 121 | The ACL Anthology is the curated archive of the computational-linguistics community and has itself served as a | 123 | The ACL Anthology is the curated archive of the computational-linguistics community and has itself served as a | ||
| 122 | −bibliometric corpus \citep{radev2013acl}. Its BibTeX export with abstracts (\texttt{anthology+abstracts.bib.gz}, | 124 | +bibliometric corpus \citep{radev2013aclanthology}. Its BibTeX export with abstracts (\texttt{anthology+abstracts.bib.gz}, | ||
| 123 | −46.7~MB, SHA-256 recorded) was downloaded on 2026-11-02 and parsed into a table of 138{,}796 entries, of which | 125 | +46.7~MB, SHA-256 recorded) was downloaded on 2026-11-02 and parsed into a table of \AclParsedEntries{} entries, of which | ||
| 124 | \LinkAclRecords{} are conference papers or journal articles and \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in | 126 | \LinkAclRecords{} are conference papers or journal articles and \LinkAclTwoZeroOneEightTwoZeroTwoSix{} fall in | ||
| 125 | 2018--2026. The Anthology plays a diagnostic role: because its venue structure is complete and hand-maintained, | 127 | 2018--2026. The Anthology plays a diagnostic role: because its venue structure is complete and hand-maintained, | ||
| 126 | the share of its entries that the backbone can be matched to is a direct measure of OpenAlex coverage for a | 128 | the share of its entries that the backbone can be matched to is a direct measure of OpenAlex coverage for a | ||
@@ -136,7 +138,7 @@ every DOI cited in this manuscript, a cross-check of publisher-assigned document | |||||
| 136 | cross-check of citation counts (\texttt{is-referenced-by-count}) on a sample of highly cited works in | 138 | cross-check of citation counts (\texttt{is-referenced-by-count}) on a sample of highly cited works in | ||
| 137 | Part~IX. | 139 | Part~IX. | ||
| 138 | 140 | ||||
| 139 | −The Semantic Scholar Academic Graph \citep{kinney2023s2} was listed in the mandate among the priority sources | 141 | +The Semantic Scholar Academic Graph \citep{kinney2023s2ag} was listed in the mandate among the priority sources | ||
| 140 | because of two signals it alone provides, the influential-citation classification and citation intent. At the | 142 | because of two signals it alone provides, the influential-citation classification and citation intent. At the | ||
| 141 | time of the feasibility audit, anonymous requests to its API returned HTTP~429 with an instruction to apply for | 143 | time of the feasibility audit, anonymous requests to its API returned HTTP~429 with an instruction to apply for | ||
| 142 | a key, and the bulk-dataset endpoint requires a key outright. In keeping with the rule that inaccessible data | 144 | a key, and the bulk-dataset endpoint requires a key outright. In keeping with the rule that inaccessible data | ||
@@ -189,8 +191,10 @@ and conservative at this stage: a work whose primary topic lies in, say, statist | |||||
| 189 | but which carries a secondary computer-science topic is a candidate, and the decision whether it belongs to | 191 | but which carries a secondary computer-science topic is a candidate, and the decision whether it belongs to | ||
| 190 | the core or to the penumbra is deferred to the eligibility stage, where it is made explicit and counted | 192 | the core or to the penumbra is deferred to the eligibility stage, where it is made explicit and counted | ||
| 191 | (Section~\ref{sec:data-core}). The filter yields | 193 | (Section~\ref{sec:data-core}). The filter yields | ||
| 192 | −\PrismaIdentifiedOpenalexCsTopicTwoZeroOneEightTwoZeroTwoSixPassOne{} records in the first pass over the | 194 | +\PrismaIdentifiedOpenalexPassOneCsTopicFilter{} records in the first pass over the snapshot; a second pass by | ||
| 193 | −snapshot. The DBLP filter retains the publication records of type article, inproceedings, incollection, book, | 195 | +identifier list (Section~\ref{sec:data-linkage}) adds \PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} | ||
| 196 | +records that carry no computer-science topic but are reached through DBLP and arXiv identifiers, so that | ||||
| 197 | +\PrismaIdentifiedOpenalexTotalRecordsLoaded{} OpenAlex records are loaded in all. The DBLP filter retains the publication records of type article, inproceedings, incollection, book, | ||||
| 194 | phdthesis and data with a year in $[2018, 2026]$: \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{} records. | 198 | phdthesis and data with a year in $[2018, 2026]$: \PrismaIdentifiedDblpTwoZeroOneEightTwoZeroTwoSix{} records. | ||
| 195 | The arXiv filter retains the records of set \texttt{cs} whose first submission falls in the window: | 199 | The arXiv filter retains the records of set \texttt{cs} whose first submission falls in the window: | ||
| 196 | \PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{} records. The ACL filter retains the conference papers | 200 | \PrismaIdentifiedArxivCsTwoZeroOneEightTwoZeroTwoSix{} records. The ACL filter retains the conference papers | ||
@@ -250,21 +254,26 @@ producing spurious links between generic titles. | |||||
| 250 | 254 | ||||
| 251 | For DBLP, the order is: (i) DOI equality, (ii) arXiv identifier equality, restricted to records not yet linked | 255 | For DBLP, the order is: (i) DOI equality, (ii) arXiv identifier equality, restricted to records not yet linked | ||
| 252 | on either side, (iii) normalised title equality with a year difference of at most one, restricted to records | 256 | on either side, (iii) normalised title equality with a year difference of at most one, restricted to records | ||
| 253 | −not yet linked on either side. The DOI step linked \LinkLinkDblpOaDoi{} records and the title step | 257 | +not yet linked on either side and, when a first author is recorded in both sources, to pairs whose | ||
| 258 | +first-author surnames agree. The DOI step linked \LinkLinkDblpOaDoi{} records and the title step | ||||
| 254 | \LinkLinkDblpOaTitleYear{}; the arXiv step linked none, because the extraction of arXiv URLs from the DBLP | 259 | \LinkLinkDblpOaTitleYear{}; the arXiv step linked none, because the extraction of arXiv URLs from the DBLP | ||
| 255 | \texttt{ee} fields returned no candidate (\LinkDblpWithArxiv{} records with an extracted identifier). The | 260 | \texttt{ee} fields returned no candidate (\LinkDblpWithArxiv{} records with an extracted identifier). The | ||
| 256 | \texttt{ee} fields of this dump evidently do not carry \texttt{arxiv.org/abs} URLs in the form the extractor | 261 | \texttt{ee} fields of this dump evidently do not carry \texttt{arxiv.org/abs} URLs in the form the extractor | ||
| 257 | expects; whether the CoRR entries are reached by the DOI step instead depends on DBLP's current conventions for | 262 | expects; whether the CoRR entries are reached by the DOI step instead depends on DBLP's current conventions for | ||
| 258 | the \texttt{ee} field, which the audit did not characterise, and the question is left to the linkage-precision | 263 | the \texttt{ee} field, which the audit did not characterise, and the question is left to the linkage-precision | ||
| 259 | −review. The step is kept in the pipeline because other dumps may differ. In total \LinkDblpLinkedDistinct{} of the \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} DBLP | 264 | +review. The step is kept in the pipeline because other dumps may differ. The linkage is run on the union of | ||
| 260 | −records in the window were linked to an OpenAlex work after the first pass, leaving \LinkDblpUnlinked{} | 265 | +the two snapshot passes, so that every count in this section is a count after both passes. In total | ||
| 261 | −unlinked. The composition of the unlinked set is informative: \LinkDblpUnlinkedByTypeArticle{} journal | 266 | +\LinkDblpLinkedDistinct{} of the \LinkDblpTwoZeroOneEightTwoZeroTwoSix{} DBLP records in the window were | ||
| 262 | −articles, \LinkDblpUnlinkedByTypeInproceedings{} conference papers, \LinkDblpUnlinkedByTypePhdthesis{} | 267 | +linked to an OpenAlex work, leaving \LinkDblpUnlinked{} unlinked. The composition of the unlinked set is | ||
| 263 | −doctoral theses, \LinkDblpUnlinkedByTypeData{} data records, \LinkDblpUnlinkedByTypeIncollection{} chapters | 268 | +informative: \LinkDblpUnlinkedByTypeArticle{} journal articles, \LinkDblpUnlinkedByTypeInproceedings{} | ||
| 264 | −and \LinkDblpUnlinkedByTypeBook{} books. Of these, \LinkDblpUnlinkedWithDoiInSlim{} carry a DOI that | 269 | +conference papers, \LinkDblpUnlinkedByTypePhdthesis{} doctoral theses, \LinkDblpUnlinkedByTypeData{} data | ||
| 265 | −\emph{does} exist in the slim table of all OpenAlex works, which means the work is in OpenAlex but was not | 270 | +records, \LinkDblpUnlinkedByTypeIncollection{} chapters and \LinkDblpUnlinkedByTypeBook{} books. A DBLP | ||
| 266 | −caught by the computer-science topic filter; this is the second error of the topic classifier described above | 271 | +record whose DOI exists in the slim table of all OpenAlex works but not among the loaded records describes a | ||
| 267 | −and the reason for the second pass. | 272 | +work that is in OpenAlex but was not caught by the computer-science topic filter; this is the second error of | ||
| 273 | +the topic classifier described above and the reason for the second pass. After that pass, | ||||
| 274 | +\LinkDblpUnlinkedWithDoiInSlim{} unlinked DBLP records are still in this situation; their full records were | ||||
| 275 | +not returned by the identifier scan, for a reason the pipeline has not characterised, and the list is kept | ||||
| 276 | +for the linkage-precision review. | ||||
| 268 | 277 | ||||
| 269 | For arXiv, the order is: (i) arXiv identifier equality against the identifier OpenAlex records for the work, | 278 | For arXiv, the order is: (i) arXiv identifier equality against the identifier OpenAlex records for the work, | ||
| 270 | (ii) DOI equality for e-prints that carry a journal DOI, (iii) normalised title equality with a year | 279 | (ii) DOI equality for e-prints that carry a journal DOI, (iii) normalised title equality with a year | ||
@@ -274,8 +283,8 @@ version \citep{lin2020preprints}. The steps linked \LinkLinkArxivOaArxivId{}, \L | |||||
| 274 | For the ACL Anthology, the order is: (i) DOI equality, (ii) equality between the Anthology identifier and the | 283 | For the ACL Anthology, the order is: (i) DOI equality, (ii) equality between the Anthology identifier and the | ||
| 275 | identifier embedded in an \texttt{aclanthology.org} landing-page URL among the work's OpenAlex locations, | 284 | identifier embedded in an \texttt{aclanthology.org} landing-page URL among the work's OpenAlex locations, | ||
| 276 | (iii) normalised title equality with a year difference of at most one. The steps linked | 285 | (iii) normalised title equality with a year difference of at most one. The steps linked | ||
| 277 | −\LinkLinkAclOaDoi{} records by DOI and \LinkLinkAclOaTitleYear{} by title, with the URL step contributing the | 286 | +\LinkLinkAclOaDoi{} records by DOI and \LinkLinkAclOaTitleYear{} by title; the URL step contributed no link | ||
| 278 | −remainder, for \LinkAclLinkedDistinctTwoZeroOneEightTwoZeroTwoSix{} distinct 2018--2026 entries linked out of | 287 | +in this snapshot, and is kept for the same reason as the DBLP arXiv step. The result is \LinkAclLinkedDistinctTwoZeroOneEightTwoZeroTwoSix{} distinct 2018--2026 entries linked out of | ||
| 279 | \LinkAclTwoZeroOneEightTwoZeroTwoSix{}; the residual is examined in Section~\ref{sec:data-quality} as a | 288 | \LinkAclTwoZeroOneEightTwoZeroTwoSix{}; the residual is examined in Section~\ref{sec:data-quality} as a | ||
| 280 | measure of OpenAlex coverage of conference-heavy subfields. | 289 | measure of OpenAlex coverage of conference-heavy subfields. | ||
| 281 | 290 | ||||
@@ -284,15 +293,20 @@ measure of OpenAlex coverage of conference-heavy subfields. | |||||
| 284 | A DBLP record whose DOI resolves to an OpenAlex work outside the topic filter is, by the scope rule, a | 293 | A DBLP record whose DOI resolves to an OpenAlex work outside the topic filter is, by the scope rule, a | ||
| 285 | computer-science work that the first pass did not load. The same situation arises for arXiv e-prints whose | 294 | computer-science work that the first pass did not load. The same situation arises for arXiv e-prints whose | ||
| 286 | journal DOI exists in OpenAlex without a computer-science topic. The linkage script therefore writes the list | 295 | journal DOI exists in OpenAlex without a computer-science topic. The linkage script therefore writes the list | ||
| 287 | −of OpenAlex identifiers that are needed but not loaded: \LinkNeededOaIdsSecondPassDblpDoi{} identifiers from | 296 | +of OpenAlex identifiers that are needed but not loaded, from both routes. A second pass over the works | ||
| 288 | −DBLP DOIs and \LinkNeededOaIdsSecondPassArxivDoi{} from arXiv DOIs. A second pass over the works snapshot, | 297 | +snapshot, restricted to this identifier list, retrieves the full records of these works | ||
| 289 | −restricted to this identifier list, retrieves the full records of these works; they are normalised into the | 298 | +(\PrismaIdentifiedOpenalexPassTwoDblpArxivLinked{} records); they are normalised into the same Parquet | ||
| 290 | −same Parquet tables with a \texttt{\_p2} suffix, the DuckDB views are rebuilt over the union of both passes, | 299 | +tables with a \texttt{\_p2} suffix, the DuckDB views are rebuilt over the union of both passes, and linkage | ||
| 291 | −and linkage is re-run so that the DBLP-linked set gains the works that the topic classifier had missed. The | 300 | +is re-run so that the DBLP-linked set gains the works that the topic classifier had missed. Rebuilt on the | ||
| 292 | −second pass is what makes the DBLP membership rule of Section~\ref{sec:data-core} operational: without it, | 301 | +union, the list of identifiers still needed shrinks to \LinkNeededOaIdsSecondPassDblpDoi{} from DBLP DOIs and | ||
| 293 | −``DBLP-indexed'' would silently mean ``DBLP-indexed and labelled computer science by OpenAlex'', and the DBLP | 302 | +\LinkNeededOaIdsSecondPassArxivDoi{} from arXiv DOIs: every e-print whose journal DOI exists in OpenAlex was | ||
| 294 | −correction would be void precisely where it matters. DBLP records with no OpenAlex counterpart after both | 303 | +reached, and the DBLP residual is the one discussed above. The second pass is what makes the DBLP membership | ||
| 295 | −passes (\PrismaLinkedDblpOnlyNoOpenalexRecord{} records before the second pass) are kept in a separate table | 304 | +rule of Section~\ref{sec:data-core} operational: without it, ``DBLP-indexed'' would silently mean | ||
| 305 | +``DBLP-indexed and labelled computer science by OpenAlex'', and the DBLP correction would be void precisely | ||||
| 306 | +where it matters. Because the second pass loads records by identifier without any restriction on their year, | ||||
| 307 | +it also admits a small number of works dated outside the window, which the eligibility stage removes under | ||||
| 308 | +code E1. DBLP records with no OpenAlex counterpart after both | ||||
| 309 | +passes (\PrismaLinkedDblpOnlyNoOpenalexRecord{} records) are kept in a separate table | ||||
| 296 | with their year and type and are reported as such; they enter the volume counts of Part~IX only in the | 310 | with their year and type and are reported as such; they enter the volume counts of Part~IX only in the | ||
| 297 | sensitivity analysis that adds DBLP-only records, because they lack the abstracts, references and affiliations | 311 | sensitivity analysis that adds DBLP-only records, because they lack the abstracts, references and affiliations | ||
| 298 | on which every other indicator depends. Most of them are journal articles and conference papers with a DOI | 312 | on which every other indicator depends. Most of them are journal articles and conference papers with a DOI | ||
@@ -307,12 +321,21 @@ e-prints whose OpenAlex record points to a different version. Title-based links | |||||
| 307 | titles of distinct works (extended abstracts and full papers, workshop and conference versions). The protocol | 321 | titles of distinct works (extended abstracts and full papers, workshop and conference versions). The protocol | ||
| 308 | pre-specifies the estimation of linkage precision on a sample of 500 DBLP--OpenAlex pairs stratified by method, | 322 | pre-specifies the estimation of linkage precision on a sample of 500 DBLP--OpenAlex pairs stratified by method, | ||
| 309 | with automatic agreement checks (normalised-title similarity, year difference, first-author surname) and a | 323 | with automatic agreement checks (normalised-title similarity, year difference, first-author surname) and a | ||
| 310 | −200-pair subsample reserved for the principal investigator (checkpoint H-1). The sampling script draws 250 DOI | 324 | +subsample of two pairs in five reserved for the principal investigator (checkpoint H-1). The sampling script | ||
| 311 | −links, 200 title links and 50 arXiv links with a fixed seed; since the arXiv step produced no links in this | 325 | +draws 250 DOI links, 200 title links and 50 arXiv links with a fixed seed; since the arXiv step produced no | ||
| 312 | −dump, the realised sample has 450 pairs, and the precision estimate for the arXiv step is undefined rather | 326 | +links in this dump, the realised sample (\texttt{data/metadata/linkage\_sample.csv}) has 450 pairs, of which | ||
| 313 | −than reported as perfect. At the time of writing the sampling script exists but the sample has not been drawn and the | 327 | +180 are reserved for the principal investigator, and the precision estimate for the arXiv step is undefined | ||
| 314 | −human checkpoint is unsigned; the automatic agreement rates and the human verdicts will be reported in Part~IX | 328 | +rather than reported as perfect. The automatic check passes 235 of the 250 DOI pairs (0.94) and 200 of the | ||
| 315 | −and Appendix~B, and every conclusion that depends on the title-based links is labelled as pending human review | 329 | +200 title--year pairs (1.00). The fifteen DOI pairs that fail the automatic check are not fifteen wrong links: | ||
| 330 | +inspection of the sample shows that most fail on a title that OpenAlex truncated at the subtitle, on a | ||||
| 331 | +first-author name written in a different order or without diacritics, or on a one- or two-year difference | ||||
| 332 | +between the DBLP year and the OpenAlex year; a few pairs, however, carry substantively different titles under | ||||
| 333 | +the same DOI, which is the signature of a retitled version or of a DOI that resolves to a different item than | ||||
| 334 | +the one DBLP describes, and these are genuine candidates for error. The DOI stratum is therefore not | ||||
| 335 | +error-free, contrary to what an exact-key method would suggest, and the human verdicts on the reserved pairs | ||||
| 336 | +are what will settle the stratum precision. At the time of writing no verdict has been entered and the | ||||
| 337 | +checkpoint is unsigned; the stratum-weighted estimate of Part~IV will be reported in Part~IX and Appendix~B | ||||
| 338 | +once it is, and every conclusion that depends on the title-based links is labelled as pending human review | ||||
| 316 | until H-1 is signed. | 339 | until H-1 is signed. | ||
| 317 | 340 | ||||
| 318 | \section{Deduplication of preprint and published versions} | 341 | \section{Deduplication of preprint and published versions} | ||
@@ -332,15 +355,19 @@ preprint and published versions into one node per work, in the sense required by | |||||
| 332 | 355 | ||||
| 333 | The rule is conservative by design, favouring a small number of certain merges over a larger number of | 356 | The rule is conservative by design, favouring a small number of certain merges over a larger number of | ||
| 334 | probable ones. A \emph{preprint-like} record is one whose OpenAlex type is \texttt{preprint} or whose DOI is | 357 | probable ones. A \emph{preprint-like} record is one whose OpenAlex type is \texttt{preprint} or whose DOI is | ||
| 335 | −an arXiv DataCite DOI (prefix \texttt{10.48550/arxiv}); a \emph{published-like} record is any other record. | 358 | +an arXiv DataCite DOI (prefix \texttt{10.48550/arxiv}); a \emph{published-like} record is any other record | ||
| 336 | −Two records are paired when their normalised titles are identical, both titles have at least 25 characters, | 359 | +that is itself an admissible twin: dated within the window, not an XPAC expansion record, titled, and of a | ||
| 337 | −and their publication years differ by at most two. When a published record matches several preprints, the | 360 | +research document type or DBLP-linked. The twin conditions are imposed so that a preprint is never merged | ||
| 338 | −earliest preprint is kept as its preprint; when a preprint matches several published records, the earliest | 361 | +into a record that the eligibility stage would then remove, which would lose the work altogether. A | ||
| 339 | −published record is kept; all other pairs are discarded, so that the pairing is one-to-one. The pairing is | 362 | +preprint-like and a published-like record are paired by either of two routes: (i) identical normalised titles | ||
| 340 | −confined to the loaded OpenAlex records; preprints that are not computer-science candidates cannot be paired, | 363 | +of at least 25 characters with publication years differing by at most two, or (ii) a shared arXiv identifier, | ||
| 341 | −which is immaterial since they would not enter the corpus. No fuzzy title similarity is used at this stage, so | 364 | +taken from the arXiv linkage where available and from the OpenAlex record otherwise. When a published record | ||
| 342 | −that a preprint whose title changed at publication is not merged; the consequence, a residual double count, is | 365 | +matches several preprints, the earliest preprint is kept as its preprint; when a preprint matches several | ||
| 343 | −discussed below. | 366 | +published records, the earliest published record is kept; all other pairs are discarded, so that the pairing | ||
| 367 | +is one-to-one. The pairing is confined to the loaded OpenAlex records; preprints that are not computer-science | ||||
| 368 | +candidates cannot be paired, which is immaterial since they would not enter the corpus. No fuzzy title | ||||
| 369 | +similarity is used at this stage, so that a preprint whose title changed at publication is merged only when | ||||
| 370 | +the second route identifies it; the consequence, a residual double count, is discussed below. | ||||
| 344 | 371 | ||||
| 345 | \subsection{What is kept from each member of a pair} | 372 | \subsection{What is kept from each member of a pair} | ||
| 346 | 373 | ||||
@@ -348,22 +375,28 @@ The published record is the surviving node. The preprint record receives the fla | |||||
| 348 | the identifier of the published work and is excluded with reason code E5 (Section~\ref{sec:data-eligibility}). | 375 | the identifier of the published work and is excluded with reason code E5 (Section~\ref{sec:data-eligibility}). | ||
| 349 | The published node receives \texttt{preprint\_work\_id} and, through the arXiv linkage, the first-submission | 376 | The published node receives \texttt{preprint\_work\_id} and, through the arXiv linkage, the first-submission | ||
| 350 | date of the e-print, so that for every work the \emph{first date} is defined as the earliest of its OpenAlex | 377 | date of the e-print, so that for every work the \emph{first date} is defined as the earliest of its OpenAlex | ||
| 351 | −publication date, the publication date of its preprint record and the arXiv \texttt{created} date. This first | 378 | +publication date, the publication date of its preprint record and the arXiv \texttt{created} date, with one | ||
| 379 | +exception stated in Section~\ref{sec:data-dates}: a publication date of the form \emph{YYYY}-01-01 is treated | ||||
| 380 | +as a year-only date and is overridden by the preprint or e-print date when one exists. This first | ||||
| 352 | date is the date variable of all monthly series in Part~IX; the publication date of the version of record is | 381 | date is the date variable of all monthly series in Part~IX; the publication date of the version of record is | ||
| 353 | kept alongside it. The distinction matters for structural-change analysis: dating a paper by its journal issue | 382 | kept alongside it. The distinction matters for structural-change analysis: dating a paper by its journal issue | ||
| 354 | would displace much of the research activity of 2023 into 2024 and 2025 and would blur any monthly series | 383 | would displace much of the research activity of 2023 into 2024 and 2025 and would blur any monthly series | ||
| 355 | around the November 2022 boundary, whereas the first date places the work at the moment it entered the public | 384 | around the November 2022 boundary, whereas the first date places the work at the moment it entered the public | ||
| 356 | −record. Citation counts are those of the surviving published node; OpenAlex attributes to the preprint record | 385 | +record. OpenAlex attributes to the preprint record the citations that cite the preprint DOI; the impact | ||
| 357 | −the citations that cite the preprint DOI, and these are not added to the published node, a known | 386 | +analyses (\texttt{bibliometrics/03\_citation\_impact.py}) therefore take the yearly citation counts of the | ||
| 358 | −under-count for works whose preprint remained the cited version. Part~IV states the resulting measurement | 387 | +surviving node as the union of its own counts and those of its merged preprint record, so that a work whose | ||
| 359 | −assumption and Part~IX reports, for the high-priority set, the share of citations that accrue to preprint | 388 | +preprint remained the cited version is not under-counted. The merge is exact only for pairs the pairing found; | ||
| 360 | −records. | 389 | +for the residual unmatched preprints discussed below, the citations stay divided between two nodes. Part~IV | ||
| 390 | +states the resulting measurement assumption and Part~IX reports, for the high-priority set, the share of | ||||
| 391 | +citations that accrue to preprint records. | ||||
| 361 | 392 | ||||
| 362 | \subsection{Outcome and residual duplication} | 393 | \subsection{Outcome and residual duplication} | ||
| 363 | 394 | ||||
| 364 | The pairing identified \PrismaExcludedEFiveDuplicate{} preprint records as duplicates of a published work, | 395 | The pairing identified \PrismaExcludedEFiveDuplicate{} preprint records as duplicates of a published work, | ||
| 365 | of which \PrismaExcludedSequentialEFiveDuplicate{} remained after the earlier exclusions had been applied. | 396 | of which \PrismaExcludedSequentialEFiveDuplicate{} remained after the earlier exclusions had been applied. | ||
| 366 | −Set against the \PrismaEligibleByTypePreprint{} preprints that remain in the eligible corpus as works with no | 397 | +Within the eligible corpus, \PrismaVersionRelationshipTitleYear{} works carry a preprint merged by the | ||
| 398 | +title route and \PrismaVersionRelationshipArxivId{} by the shared-identifier route (a pair that satisfies | ||||
| 399 | +both is counted once, under either route). Set against the \PrismaEligibleByTypePreprint{} preprints that remain in the eligible corpus as works with no | ||||
| 367 | published counterpart, the figure shows that most e-prints in the candidate set were not matched to a | 400 | published counterpart, the figure shows that most e-prints in the candidate set were not matched to a | ||
| 368 | published version within OpenAlex. Three explanations coexist and cannot be separated without manual review: | 401 | published version within OpenAlex. Three explanations coexist and cannot be separated without manual review: | ||
| 369 | many e-prints are genuinely unpublished, consistent with the publication rates reported for computer-science | 402 | many e-prints are genuinely unpublished, consistent with the publication rates reported for computer-science | ||
@@ -388,8 +421,10 @@ records each criterion removed given those before it. | |||||
| 388 | \subsection{Reason codes} | 421 | \subsection{Reason codes} | ||
| 389 | 422 | ||||
| 390 | \textbf{E1, outside window.} The publication date is missing, earlier than 2018-01-01 or later than | 423 | \textbf{E1, outside window.} The publication date is missing, earlier than 2018-01-01 or later than | ||
| 391 | −2026-11-02. Because the first-pass filter already restricted the publication year, this code catches only | 424 | +2026-11-02. Because the first-pass filter already restricted the publication year, this code catches, among | ||
| 392 | −records with a missing date or a date beyond the extraction date. Forward-dated records, that is works whose | 425 | +first-pass records, only those with a missing date or a date beyond the extraction date; it also removes the | ||
| 426 | +second-pass records, loaded by identifier without a year restriction, whose publication date lies outside the | ||||
| 427 | +window. Forward-dated records, that is works whose | ||||
| 393 | publication date lies in the future relative to the snapshot, are a documented feature of OpenAlex, where | 428 | publication date lies in the future relative to the snapshot, are a documented feature of OpenAlex, where | ||
| 394 | publishers register articles for forthcoming issues; the window end removes those dated after the extraction | 429 | publishers register articles for forthcoming issues; the window end removes those dated after the extraction | ||
| 395 | date, and the remaining 2026 records are discussed in Section~\ref{sec:data-dates}. | 430 | date, and the remaining 2026 records are discussed in Section~\ref{sec:data-dates}. | ||
@@ -439,16 +474,17 @@ from every impact analysis (citation rankings, landmark selection, field-normali | |||||
| 439 | \textbf{E7, generic deposit platform.} Introduced by amendment A-002, this code excludes records whose primary | 474 | \textbf{E7, generic deposit platform.} Introduced by amendment A-002, this code excludes records whose primary | ||
| 440 | source is a generic deposit platform (Zenodo, Figshare, the Japanese Institutional Repositories DataBase, | 475 | source is a generic deposit platform (Zenodo, Figshare, the Japanese Institutional Repositories DataBase, | ||
| 441 | LA~Referencia, Dialnet, the DOAJ-as-source entries and the OSTI OAI feed) unless the record is DBLP- or | 476 | LA~Referencia, Dialnet, the DOAJ-as-source entries and the OSTI OAI feed) unless the record is DBLP- or | ||
| 442 | −arXiv-linked. The number of records removed under this code is reported by the macro | 477 | +arXiv-linked. Independently of order, \PrismaExcludedESevenGenericDeposit{} records carry the flag; | ||
| 443 | −\pending{PrismaExcludedSequentialESevenGenericDeposit} once the corpus has been rebuilt under A-002. | 478 | +sequentially, after the five preceding criteria, \PrismaExcludedSequentialESevenGenericDeposit{} records are | ||
| 444 | − | 479 | +removed under this code. | ||
| 445 | −One inconsistency of numbering must be recorded. In the frozen protocol, code E7 designates the | 480 | + | ||
| 446 | −penumbra of works with a secondary computer-science topic only; the implementation reuses the number E7 for | 481 | +One change of numbering must be recorded. In the frozen protocol, code E7 designated the penumbra of works | ||
| 447 | −the generic-deposit exclusion of amendment A-002 and carries the penumbra as a separate flag, | 482 | +with a secondary computer-science topic only; the implementation reuses the number E7 for the generic-deposit | ||
| 448 | −\texttt{cs\_penumbra}. The penumbra is not an exclusion in the same sense as E1--E7: it is a scope boundary, | 483 | +exclusion of amendment A-002 and carries the penumbra as a separate flag, \texttt{cs\_penumbra}. The | ||
| 449 | −reported separately and used in sensitivity analyses, and is treated in Section~\ref{sec:data-core}. The | 484 | +penumbra is not an exclusion in the same sense as E1--E7: it is a scope boundary, reported separately and used | ||
| 450 | −conflict of numbering is flagged here for correction in the amendments log before the protocol is cited in | 485 | +in sensitivity analyses, and is treated in Section~\ref{sec:data-core}. Amendment A-005 records the | ||
| 451 | −its final form. | 486 | +redefinition, so that the code labels of the protocol, of the implementation and of this manuscript now | ||
| 487 | +agree. | ||||
| 452 | 488 | ||||
| 453 | \subsection{Amendments A-001 and A-002 and their evidence} | 489 | \subsection{Amendments A-001 and A-002 and their evidence} | ||
| 454 | 490 | ||||
@@ -500,23 +536,36 @@ least one of the following holds: | |||||
| 500 | \begin{enumerate} | 536 | \begin{enumerate} | ||
| 501 | \item it is linked to a DBLP record (venue-based membership); | 537 | \item it is linked to a DBLP record (venue-based membership); | ||
| 502 | \item its OpenAlex primary field is Computer Science (classifier-based membership); | 538 | \item its OpenAlex primary field is Computer Science (classifier-based membership); | ||
| 503 | −\item it carries a secondary computer-science topic \emph{and} is linked to an arXiv e-print whose primary | 539 | +\item it is linked to an arXiv e-print whose primary category is in the \texttt{cs} archive | ||
| 504 | − category is in the \texttt{cs} archive (author-declared membership). | 540 | + (author-declared membership). | ||
| 505 | \end{enumerate} | 541 | \end{enumerate} | ||
| 506 | 542 | ||||
| 507 | −A record that survives the exclusions but satisfies none of these, which means it carries a secondary | 543 | +The third condition is the form given by amendment A-006. As frozen, it required in addition that the record | ||
| 508 | −computer-science topic without venue or author support, belongs to the \emph{penumbra}. The three conditions | 544 | +carry a computer-science topic in OpenAlex, which made the clause unreachable for exactly the works the second | ||
| 509 | −correspond to three independent judgements about disciplinary membership: the judgement of DBLP's editors | 545 | +pass recovers (works with no computer-science topic that an arXiv e-print declares as computer science); the | ||
| 510 | −about venues, the judgement of OpenAlex's classifier about content, and the judgement of the authors who chose | 546 | +amendment removes the extra requirement and makes the corpus-definition script assert that every record | ||
| 511 | −a primary arXiv category. Requiring agreement of any one of them with the first-pass topic filter removes the | 547 | +surviving the exclusions falls into exactly one of three classes, core, penumbra or the residual class | ||
| 512 | −over-inclusion that a secondary topic alone would produce, while keeping works that any single source would | 548 | +counted below. A record that survives the exclusions, carries a | ||
| 513 | −have missed. | 549 | +computer-science topic and satisfies none of the three conditions, which means it has neither venue nor | ||
| 550 | +author support, belongs to the \emph{penumbra}. The three conditions correspond to three independent | ||||
| 551 | +judgements about disciplinary membership: the judgement of DBLP's editors about venues, the judgement of | ||||
| 552 | +OpenAlex's classifier about content, and the judgement of the authors who chose a primary arXiv category. | ||||
| 553 | +Requiring any one of them removes the over-inclusion that a secondary topic alone would produce, while keeping | ||||
| 554 | +works that any single source would have missed. | ||||
| 514 | 555 | ||||
| 515 | \subsection{Counts} | 556 | \subsection{Counts} | ||
| 516 | 557 | ||||
| 517 | After the sequential exclusions, \PrismaAfterExclusions{} loaded records remain. Of these, | 558 | After the sequential exclusions, \PrismaAfterExclusions{} loaded records remain. Of these, | ||
| 518 | −\PrismaCsCoreEligible{} satisfy the membership rule and constitute the eligible mapping corpus, and | 559 | +\PrismaCsCoreEligible{} satisfy the membership rule and constitute the eligible mapping corpus, | ||
| 519 | −\PrismaCsPenumbraEligibleOtherwise{} form the penumbra. The penumbra is thus a substantial fraction of the | 560 | +\PrismaCsPenumbraEligibleOtherwise{} form the penumbra, and \PrismaNeitherCoreNorPenumbraAfterExclusions{} | ||
| 561 | +belong to neither: they carry no computer-science topic, having been loaded by the second pass, and are | ||||
| 562 | +neither DBLP-linked nor linked to an e-print with a \texttt{cs} primary category. By construction of the | ||||
| 563 | +second-pass identifier list, such records can only have been reached through the journal DOI of an e-print | ||||
| 564 | +harvested in the \texttt{cs} set but whose primary category lies in another archive (a cross-listed | ||||
| 565 | +e-print), and they are outside the corpus by the same reasoning that excludes the penumbra; this reading | ||||
| 566 | +follows from the construction and has not been verified record by record, which the linkage review will do. | ||||
| 567 | +The three classes partition the survivors exactly, and the corpus-definition script stops if they do not. | ||||
| 568 | +The penumbra is thus a substantial fraction of the | ||||
| 520 | records that a naive ``any computer-science topic'' filter would have called computer science; its exclusion | 569 | records that a naive ``any computer-science topic'' filter would have called computer science; its exclusion | ||
| 521 | from the primary analyses is the main difference between the present corpus and what an OpenAlex query alone | 570 | from the primary analyses is the main difference between the present corpus and what an OpenAlex query alone | ||
| 522 | would return, and it is the reason why no count in this manuscript should be compared directly with the counts | 571 | would return, and it is the reason why no count in this manuscript should be compared directly with the counts | ||
@@ -536,8 +585,9 @@ PRISMA-style flow, drawn from the same JSON file that generates the macros of th | |||||
| 536 | \centering | 585 | \centering | ||
| 537 | \includegraphics[width=0.9\textwidth]{fig_prisma_flow.pdf} | 586 | \includegraphics[width=0.9\textwidth]{fig_prisma_flow.pdf} | ||
| 538 | \caption{PRISMA-style flow of record identification, exclusion and the final mapping corpus. Identification | 587 | \caption{PRISMA-style flow of record identification, exclusion and the final mapping corpus. Identification | ||
| 539 | −counts are per source and overlap; exclusions are sequential in the order E1--E7; the core and the penumbra | 588 | +counts are per source and overlap; exclusions are sequential in the order E1--E7; the core, the penumbra and | ||
| 540 | −partition the records that survive the exclusions. Source: \texttt{data/metadata/prisma\_counts.json}, | 589 | +the small residual class of second-pass records with neither venue nor author support (text) partition the | ||
| 590 | +records that survive the exclusions. Source: \texttt{data/metadata/prisma\_counts.json}, | ||||
| 541 | rendered by \texttt{figures/make\_figures.py}.} | 591 | rendered by \texttt{figures/make\_figures.py}.} | ||
| 542 | \label{fig:prisma} | 592 | \label{fig:prisma} | ||
| 543 | \end{figure} | 593 | \end{figure} | ||
@@ -579,7 +629,9 @@ section documents three distinct problems, their diagnosis in the corpus, and th | |||||
| 579 | Each eligible work carries up to three dates: the OpenAlex publication date of the surviving record, the | 629 | Each eligible work carries up to three dates: the OpenAlex publication date of the surviving record, the | ||
| 580 | publication date of its merged preprint record where one exists, and the arXiv first-submission date where the | 630 | publication date of its merged preprint record where one exists, and the arXiv first-submission date where the | ||
| 581 | work is linked to an e-print. The \emph{first date} of a work is the earliest of the three | 631 | work is linked to an e-print. The \emph{first date} of a work is the earliest of the three | ||
| 582 | −(Section~\ref{sec:data-dedup}), and the \emph{first month} is its calendar month. The first date measures | 632 | +(Section~\ref{sec:data-dedup}), except that a publication date of the form \emph{YYYY}-01-01, which is in | ||
| 633 | +most cases a year-only date, is set aside whenever a preprint or e-print date exists, so that the first date | ||||
| 634 | +is then the earlier of those two; the \emph{first month} is its calendar month. The first date measures | ||||
| 583 | the entry of the work into the public record, which is the quantity relevant to questions about the pace and | 635 | the entry of the work into the public record, which is the quantity relevant to questions about the pace and | ||
| 584 | composition of research activity; the publication date of the version of record measures the pace of the | 636 | composition of research activity; the publication date of the version of record measures the pace of the | ||
| 585 | publishing system and is kept for the analyses of venues and of preprint-to-publication lags. The two | 637 | publishing system and is kept for the analyses of venues and of preprint-to-publication lags. The two | ||
@@ -594,12 +646,13 @@ eligible corpus by year found that between about a fifth and somewhat over a qua | |||||
| 594 | \emph{YYYY}-01-01, with the proportion varying by source and type; dissertations, book chapters and records | 646 | \emph{YYYY}-01-01, with the proportion varying by source and type; dissertations, book chapters and records | ||
| 595 | from repositories are the most affected, journal articles with publisher-registered issue dates the least. | 647 | from repositories are the most affected, journal articles with publisher-registered issue dates the least. | ||
| 596 | Amendment A-002 introduced the flag \texttt{date\_precision}, equal to \texttt{year} when the publication | 648 | Amendment A-002 introduced the flag \texttt{date\_precision}, equal to \texttt{year} when the publication | ||
| 597 | −date is the first of January and \texttt{month} otherwise. The rule is a heuristic with a known cost: a work | 649 | +date is the first of January and the work has neither a merged preprint date nor an arXiv first-submission | ||
| 598 | −genuinely published on 1 January is classed as year-precision, and since the rule is applied to the | 650 | +date, and \texttt{month} otherwise. A work with a 1 January publication date and an e-print is thus | ||
| 599 | −publication date rather than to the first date, a year-precision work with an arXiv first date is correctly | 651 | +month-precision, because its first date is taken from the e-print rather than from the publication date | ||
| 600 | −dated by its e-print. The counts of eligible works by precision class will be reported by the macros | 652 | +(Section~\ref{sec:data-dedup}). The rule is a heuristic with a known cost: a work genuinely published on | ||
| 601 | −\pending{PrismaEligibleDatePrecisionMonth} and \pending{PrismaEligibleDatePrecisionYear} once the corpus has | 653 | +1 January and without a preprint is classed as year-precision. In the eligible corpus, | ||
| 602 | −been rebuilt under A-002. | 654 | +\PrismaEligibleDatePrecisionMonth{} works are month-precision and \PrismaEligibleDatePrecisionYear{} are | ||
| 655 | +year-precision. | ||||
| 603 | 656 | ||||
| 604 | All monthly series in Part~IX use month-precision records only. Year-precision records enter the annual series, | 657 | All monthly series in Part~IX use month-precision records only. Year-precision records enter the annual series, | ||
| 605 | where their date is correct, and the composition analyses that do not depend on the month. The exclusion of | 658 | where their date is correct, and the composition analyses that do not depend on the month. The exclusion of | ||
@@ -662,7 +715,8 @@ The rule adopted is conservative. The ``complete'' observation window for monthl | |||||
| 662 | June~2026; the months from July to November 2026 are reported descriptively, separately, and never enter a | 715 | June~2026; the months from July to November 2026 are reported descriptively, separately, and never enter a | ||
| 663 | breakpoint test. Within the complete window, the structural-change models include the last months with | 716 | breakpoint test. Within the complete window, the structural-change models include the last months with | ||
| 664 | their observed values and the robustness section of Part~IX re-estimates every break with the window ending | 717 | their observed values and the robustness section of Part~IX re-estimates every break with the window ending | ||
| 665 | −in December 2025, so that a result that depends on the 2026 months is identified as such. The API increment | 718 | +in March 2026 and in August 2026, so that a result that depends on the last months of the window, in either | ||
| 719 | +direction, is identified as such. The API increment | ||||
| 666 | harvested after the snapshot (Section~\ref{sec:data-sources}) adds the works registered between the snapshot | 720 | harvested after the snapshot (Section~\ref{sec:data-sources}) adds the works registered between the snapshot | ||
| 667 | date and the extraction date, which narrows but does not close the gap. Finally, forward-dated records, | 721 | date and the extraction date, which narrows but does not close the gap. Finally, forward-dated records, | ||
| 668 | works carrying a 2026 publication date for an issue not yet released, are retained when their date precedes | 722 | works carrying a 2026 publication date for an issue not yet released, are retained when their date precedes | ||
@@ -786,7 +840,7 @@ None of these biases is corrected by reweighting. The study reports them, bounds | |||||
| 786 | permits, and re-estimates its results on alternative corpora (strict core; corpus with penumbra; corpus with | 840 | permits, and re-estimates its results on alternative corpora (strict core; corpus with penumbra; corpus with | ||
| 787 | XPAC research types; corpus with DBLP-only records; corpus with generic deposits) so that the reader can see | 841 | XPAC research types; corpus with DBLP-only records; corpus with generic deposits) so that the reader can see | ||
| 788 | how far each conclusion depends on each scope decision. This is the approach that the science-of-science | 842 | how far each conclusion depends on each scope decision. This is the approach that the science-of-science | ||
| 789 | −literature recommends for databases whose coverage is known to be uneven \citep{fortunato2018science}, and it | 843 | +literature recommends for databases whose coverage is known to be uneven \citep{fortunato2018sos}, and it | ||
| 790 | is the only one compatible with a mandate that forbids the fabrication of corrections. | 844 | is the only one compatible with a mandate that forbids the fabrication of corrections. | ||
| 791 | 845 | ||||
| 792 | \section{Research-integrity procedures} | 846 | \section{Research-integrity procedures} | ||
@@ -813,11 +867,13 @@ record \citep{nosek2018preregistration}; the repository's version history and th | |||||
| 813 | the available substitute, and the release of Phase~9 will include both. Every departure from the frozen | 867 | the available substitute, and the release of Phase~9 will include both. Every departure from the frozen | ||
| 814 | protocol is an entry in \texttt{protocol/amendments.csv} with an identifier, a date, the phase, the change, | 868 | protocol is an entry in \texttt{protocol/amendments.csv} with an identifier, a date, the phase, the change, | ||
| 815 | the reason and the approver, which until the principal investigator signs is ``executing agent (pending | 869 | the reason and the approver, which until the principal investigator signs is ``executing agent (pending | ||
| 816 | −PI)''. At the time of writing the log holds five entries: A-000 records that the mandate's acceptance | 870 | +PI)''. At the time of writing the log holds seven entries: A-000 records that the mandate's acceptance | ||
| 817 | criteria were completed by the executing agent after the pasted text was truncated; A-001 and A-002 are the | 871 | criteria were completed by the executing agent after the pasted text was truncated; A-001 and A-002 are the | ||
| 818 | eligibility amendments of Section~\ref{sec:data-eligibility}; A-003 and A-004 fix the budgets and the models | 872 | eligibility amendments of Section~\ref{sec:data-eligibility}; A-003 and A-004 fix the budgets and the models | ||
| 819 | −of the LLM-based taxonomy validation and knowledge-graph extraction. Analyses added after the freeze are | 873 | +of the LLM-based taxonomy validation and knowledge-graph extraction; A-005 records the renumbering of code | ||
| 820 | −labelled exploratory in Part~IX. | 874 | +E7 and A-006 the revised arXiv clause of the membership rule (Sections~\ref{sec:data-eligibility} | ||
| 875 | +and~\ref{sec:data-core}), both prompted by the writing and critique passes of the first manuscript draft. Analyses | ||||
| 876 | +added after the freeze are labelled exploratory in Part~IX. | ||||
| 821 | 877 | ||||
| 822 | \subsection{Immutable raw snapshots and provenance} | 878 | \subsection{Immutable raw snapshots and provenance} | ||
| 823 | 879 | ||||
modified
manuscript/sections/part04_methodology.tex
+333 −140
@@ -35,21 +35,41 @@ Computer Science field, plus the second-pass records recovered through DBLP and | |||||
| 35 | arXiv identifiers, Section~\ref{sec:m-linkage}). Each record $w \in \mathcal{W}_0$ | 35 | arXiv identifiers, Section~\ref{sec:m-linkage}). Each record $w \in \mathcal{W}_0$ | ||
| 36 | carries an OpenAlex identifier, a type $\tau(w)$ drawn from the OpenAlex | 36 | carries an OpenAlex identifier, a type $\tau(w)$ drawn from the OpenAlex | ||
| 37 | controlled vocabulary, a publication date $d_{\mathrm{pub}}(w)$, a creation date | 37 | controlled vocabulary, a publication date $d_{\mathrm{pub}}(w)$, a creation date | ||
| 38 | −$d_{\mathrm{cr}}(w)$ (the date at which the record entered OpenAlex), a title, an | 38 | +$d_{\mathrm{cr}}(w)$ (nominally the date at which the record entered OpenAlex; | ||
| 39 | +Part III shows that for most records it reflects a re-ingestion in the second | ||||
| 40 | +half of 2025, so it is not used as an indexing date), a title, an | ||||
| 39 | optional abstract, authorships, a primary source (venue), OpenAlex topics with | 41 | optional abstract, authorships, a primary source (venue), OpenAlex topics with | ||
| 40 | their subfield and field, a list of referenced works, and yearly citation counts. | 42 | their subfield and field, a list of referenced works, and yearly citation counts. | ||
| 41 | Linked records from DBLP, arXiv and the ACL Anthology attach to $w$ a DBLP key, | 43 | Linked records from DBLP, arXiv and the ACL Anthology attach to $w$ a DBLP key, | ||
| 42 | an arXiv identifier with its arXiv creation date $d_{\mathrm{arx}}(w)$ and primary | 44 | an arXiv identifier with its arXiv creation date $d_{\mathrm{arx}}(w)$ and primary | ||
| 43 | category, and an ACL identifier. | 45 | category, and an ACL identifier. | ||
| 44 | 46 | ||||
| 45 | −The temporal position of a work is its \emph{first date}, | 47 | +The temporal position of a work is its \emph{first date}, computed by | ||
| 48 | +\texttt{systematic\_review/define\_corpus.py} from three candidate dates: the | ||||
| 49 | +publication date $d_{\mathrm{pub}}(w)$, the publication date | ||||
| 50 | +$d_{\mathrm{pub}}(\pi(w))$ of the preprint record $\pi(w)$ merged into $w$ by | ||||
| 51 | +the deduplication rule of Section~\ref{sec:m-dedup} when one exists, and the | ||||
| 52 | +arXiv creation date $d_{\mathrm{arx}}(w)$ of the linked arXiv record when one | ||||
| 53 | +exists. Let $A(w) \subseteq \{ d_{\mathrm{pub}}(\pi(w)),\, d_{\mathrm{arx}}(w) \}$ | ||||
| 54 | +be the set of those two dates that are available for $w$. Then | ||||
| 46 | \begin{equation} | 55 | \begin{equation} | ||
| 47 | − d(w) \;=\; \min\bigl\{\, d_{\mathrm{pub}}(w),\; d_{\mathrm{pub}}(\pi(w)),\; d_{\mathrm{arx}}(w) \,\bigr\}, | 56 | + d(w) \;=\; | ||
| 57 | + \begin{cases} | ||||
| 58 | + \min A(w) & \text{if } d_{\mathrm{pub}}(w) \text{ falls on 1 January and } A(w) \neq \emptyset,\\[2pt] | ||||
| 59 | + \min\bigl( \{ d_{\mathrm{pub}}(w) \} \cup A(w) \bigr) & \text{otherwise.} | ||||
| 60 | + \end{cases} | ||||
| 48 | \label{eq:first-date} | 61 | \label{eq:first-date} | ||
| 49 | \end{equation} | 62 | \end{equation} | ||
| 50 | −where $\pi(w)$ is the preprint record merged into $w$ by the deduplication rule of | 63 | +The second branch is the minimum of the available dates. The first branch is | ||
| 51 | −Section~\ref{sec:m-dedup} when one exists, and each missing term is replaced by | 64 | +the override that matters: OpenAlex records a year-only publication date as | ||
| 52 | −$d_{\mathrm{pub}}(w)$. The first date, not the date of the version of record, is | 65 | +the first of January of that year, and when a preprint or arXiv date exists | ||
| 66 | +the code \emph{drops} the 1~January date from the minimum instead of letting | ||||
| 67 | +a fictitious date win by construction. The first date can therefore be later | ||||
| 68 | +than the publication date, with a consequence for the period assignment | ||||
| 69 | +below: a record whose publication date is 1~January 2022 and whose linked | ||||
| 70 | +arXiv version was created in March 2023 is dated March 2023 and belongs to | ||||
| 71 | +the post period, although its OpenAlex year is 2022. The first date, not the | ||||
| 72 | +date of the version of record, is | ||||
| 53 | the quantity that places a work on the time axis, because the research question | 73 | the quantity that places a work on the time axis, because the research question | ||
| 54 | concerns when ideas entered the literature rather than when a publisher released | 74 | concerns when ideas entered the literature rather than when a publisher released | ||
| 55 | them. The publication year $y(w)$ used for citation normalisation is, in | 75 | them. The publication year $y(w)$ used for citation normalisation is, in | ||
@@ -64,11 +84,22 @@ period $B = [t_0, t^{\ast})$ and a post period $P = [t^{\ast}, t_1]$: | |||||
| 64 | \begin{equation} | 84 | \begin{equation} | ||
| 65 | \mathrm{period}(w) \;=\; | 85 | \mathrm{period}(w) \;=\; | ||
| 66 | \begin{cases} | 86 | \begin{cases} | ||
| 67 | − B & \text{if } t_0 \le d(w) < t^{\ast},\\ | 87 | + \text{pre} & \text{if } d(w) < t_0,\\ | ||
| 68 | − P & \text{if } t^{\ast} \le d(w) \le t_1 . | 88 | + B & \text{if } t_0 \le d(w) < t^{\ast},\\ | ||
| 89 | + P & \text{if } t^{\ast} \le d(w) \le t_1,\\ | ||||
| 90 | + \text{future} & \text{if } d(w) > t_1 . | ||||
| 69 | \end{cases} | 91 | \end{cases} | ||
| 70 | \end{equation} | 92 | \end{equation} | ||
| 71 | −Monthly series use the month $m(w)$ of the first date. Two facts about dates | 93 | +The eligibility predicate of Section~\ref{sec:m-eligibility} constrains | ||
| 94 | +$d_{\mathrm{pub}}(w)$, not $d(w)$, to the window, so an eligible work whose | ||||
| 95 | +preprint or arXiv date precedes 2018 receives the label \emph{pre} | ||||
| 96 | +(\PrismaEligibleByPeriodPre{} eligible works); such works are counted in the | ||||
| 97 | +annual tables, which are keyed by publication year, but belong to neither | ||||
| 98 | +period and to no monthly series, since the series start in January 2018. The | ||||
| 99 | +label \emph{future} is defined for completeness and is empty in the current | ||||
| 100 | +corpus: it could arise only if an arXiv date later than the window end | ||||
| 101 | +rescued a year-only record. Monthly series use the month $m(w)$ of the first | ||||
| 102 | +date. Two facts about dates | ||||
| 72 | constrain everything that follows. First, a substantial share of OpenAlex | 103 | constrain everything that follows. First, a substantial share of OpenAlex | ||
| 73 | records carry a publication date equal to the first of January of their year, | 104 | records carry a publication date equal to the first of January of their year, | ||
| 74 | which is the database's convention for year-only precision rather than a real | 105 | which is the database's convention for year-only precision rather than a real | ||
@@ -76,16 +107,31 @@ date. The pipeline therefore attaches to every record a flag | |||||
| 76 | \begin{equation} | 107 | \begin{equation} | ||
| 77 | \mathrm{prec}(w) \;=\; | 108 | \mathrm{prec}(w) \;=\; | ||
| 78 | \begin{cases} | 109 | \begin{cases} | ||
| 79 | − \text{year} & \text{if } d_{\mathrm{pub}}(w) \text{ falls on 1 January},\\ | 110 | + \text{year} & \text{if } d_{\mathrm{pub}}(w) \text{ falls on 1 January and } A(w) = \emptyset,\\ | ||
| 80 | \text{month} & \text{otherwise}, | 111 | \text{month} & \text{otherwise}, | ||
| 81 | \end{cases} | 112 | \end{cases} | ||
| 82 | \end{equation} | 113 | \end{equation} | ||
| 83 | and every monthly series in this paper is computed on the month-precision | 114 | and every monthly series in this paper is computed on the month-precision | ||
| 84 | subset $\mathcal{W}^{\mathrm{m}} = \{w : \mathrm{prec}(w) = \text{month}\}$ only | 115 | subset $\mathcal{W}^{\mathrm{m}} = \{w : \mathrm{prec}(w) = \text{month}\}$ only | ||
| 85 | −(amendment A-002). A genuine 1~January publication is thereby misclassified as | 116 | +(amendment A-002): \PrismaEligibleDatePrecisionMonth{} eligible works are at | ||
| 86 | −year precision and dropped from monthly series; the loss is small and, more | 117 | +month precision and \PrismaEligibleDatePrecisionYear{} at year precision. A | ||
| 87 | −importantly, it does not depend on the period, so it cannot manufacture a break. | 118 | +record whose 1~January date is overridden in equation~\eqref{eq:first-date} | ||
| 88 | −Annual series use all eligible records. Second, the right end of the window is | 119 | +counts as month precision, because the date that places it is a real one. Two | ||
| 120 | +consequences follow. A genuine 1~January publication without a linked earlier | ||||
| 121 | +version is misclassified as year precision and dropped from the monthly | ||||
| 122 | +series. More importantly, the exclusion is not period-neutral: whether a | ||||
| 123 | +year-only record is rescued depends on whether it has a linked preprint or | ||||
| 124 | +arXiv version, and that linkage rate rises over the window with the growth of | ||||
| 125 | +arXiv deposit, so the fraction of eligible works that enter the monthly series | ||||
| 126 | +differs between the baseline and the post period, and Part III reports | ||||
| 127 | +year-only shares that vary by source and type. A change in level of a monthly | ||||
| 128 | +series could in principle be produced by a change in date precision alone. | ||||
| 129 | +The manuscript treats this as a maintained caveat rather than a solved | ||||
| 130 | +problem: the year-precision share is reported by year, source and type next to | ||||
| 131 | +the monthly series (Parts III and IX), and the annual series, which use all | ||||
| 132 | +eligible records, are the check that does not depend on precision. A | ||||
| 133 | +sensitivity run that redistributes year-only records over the months of their | ||||
| 134 | +year is not implemented. Second, the right end of the window is | ||||
| 89 | censored: records published in the last months before the snapshot are | 135 | censored: records published in the last months before the snapshot are | ||
| 90 | under-represented because OpenAlex indexes them with a lag. The treatment of this | 136 | under-represented because OpenAlex indexes them with a lag. The treatment of this | ||
| 91 | right-truncation is specified in Section~\ref{sec:m-truncation}. | 137 | right-truncation is specified in Section~\ref{sec:m-truncation}. | ||
@@ -198,7 +244,10 @@ DBLP records are restricted to the years 2018--2026 and to the record types | |||||
| 198 | \item \textbf{title--year}: among records still unmatched on both sides, | 244 | \item \textbf{title--year}: among records still unmatched on both sides, | ||
| 199 | restrict each side to normalised titles that are \emph{unique} within | 245 | restrict each side to normalised titles that are \emph{unique} within | ||
| 200 | that side, join on equal $\eta$, and keep the pair if the years differ | 246 | that side, join on equal $\eta$, and keep the pair if the years differ | ||
| 201 | − by at most one. | 247 | + by at most one and the first-author surnames agree whenever both are | ||
| 248 | + available (the last whitespace-delimited token of the first author's | ||||
| 249 | + name, lower-cased, taken from the DBLP author list and from the | ||||
| 250 | + OpenAlex authorship at position zero). | ||||
| 202 | \end{enumerate} | 251 | \end{enumerate} | ||
| 203 | The uniqueness restriction in step 3 is the blocking device: it turns the join | 252 | The uniqueness restriction in step 3 is the blocking device: it turns the join | ||
| 204 | into a one-to-one map and prevents a frequent title from generating a clique of | 253 | into a one-to-one map and prevents a frequent title from generating a clique of | ||
@@ -234,10 +283,18 @@ not label with any Computer Science topic; they motivate the second snapshot | |||||
| 234 | pass, in which these identifiers (\LinkNeededOaIdsSecondPassDblpDoi{} from DBLP | 283 | pass, in which these identifiers (\LinkNeededOaIdsSecondPassDblpDoi{} from DBLP | ||
| 235 | and \LinkNeededOaIdsSecondPassArxivDoi{} from arXiv) are fetched in full and the | 284 | and \LinkNeededOaIdsSecondPassArxivDoi{} from arXiv) are fetched in full and the | ||
| 236 | linkage is rerun with the union of both passes. The corpus is therefore not the | 285 | linkage is rerun with the union of both passes. The corpus is therefore not the | ||
| 237 | −set of works that OpenAlex calls computer science; it is the union of that set | 286 | +set of works that OpenAlex calls computer science: by the membership rule of | ||
| 238 | −with what DBLP and arXiv call computer science, which is the sense in which the | 287 | +Section~\ref{sec:m-eligibility}, as amended by A-006, it is the union of the | ||
| 239 | −mandate's requirement to ``cover the complete breadth of computer science'' | 288 | +works whose primary OpenAlex topic lies in computer science with the works | ||
| 240 | −rather than only the OpenAlex topic model is operationalised. | 289 | +that DBLP indexes and the works whose authors filed them under an arXiv | ||
| 290 | +\texttt{cs.*} primary category, which is the sense in which the mandate's | ||||
| 291 | +requirement to ``cover the complete breadth of computer science'' rather than | ||||
| 292 | +only the OpenAlex topic model is operationalised. The first version of the | ||||
| 293 | +rule required an OpenAlex computer-science topic \emph{in addition to} the | ||||
| 294 | +arXiv category, which made the arXiv clause unreachable for exactly the | ||||
| 295 | +second-pass works it was meant to admit; amendment A-006 removed that | ||||
| 296 | +requirement after the critic's audit found the records falling between core | ||||
| 297 | +and penumbra. | ||||
| 241 | 298 | ||||
| 242 | \subsection{Preprint--publication deduplication} | 299 | \subsection{Preprint--publication deduplication} | ||
| 243 | \label{sec:m-dedup} | 300 | \label{sec:m-dedup} | ||
@@ -246,30 +303,48 @@ A work that exists as an arXiv preprint and as a published version is two | |||||
| 246 | OpenAlex records whenever OpenAlex has not merged them. Counting both would | 303 | OpenAlex records whenever OpenAlex has not merged them. Counting both would | ||
| 247 | double-count exactly the kind of work (AI preprints) whose growth is at issue | 304 | double-count exactly the kind of work (AI preprints) whose growth is at issue | ||
| 248 | (CLAUDE.md §6.2). The script \texttt{systematic\_review/define\_corpus.py} | 305 | (CLAUDE.md §6.2). The script \texttt{systematic\_review/define\_corpus.py} | ||
| 249 | −identifies such pairs within $\mathcal{W}_0$ by a title rule: | 306 | +identifies such pairs within $\mathcal{W}_0$ by two routes. Let | ||
| 307 | +$\mathcal{P} = \{ w : \tau(w) = \text{preprint} \text{ or } \text{DOI}(w) | ||||
| 308 | +\text{ starts with } \text{10.48550/arxiv} \}$ be the preprint-like set and | ||||
| 309 | +$\mathcal{B}$ the set of admissible published twins, | ||||
| 310 | +\begin{equation} | ||||
| 311 | + \mathcal{B} \;=\; \bigl\{ b \notin \mathcal{P} :\ d_{\mathrm{pub}}(b) \in [t_0, t_1],\; | ||||
| 312 | + \neg\,\text{is\_xpac}(b),\; \text{title}(b) \neq \emptyset,\; | ||||
| 313 | + \tau(b) \in T_{\mathrm{keep}} \vee \text{in\_dblp}(b) \bigr\}, | ||||
| 314 | +\end{equation} | ||||
| 315 | +that is, records that would themselves pass the window, XPAC, title and type | ||||
| 316 | +exclusions of Section~\ref{sec:m-eligibility} (otherwise the preprint stays | ||||
| 317 | +as the canonical record). With $|\eta(\cdot)| \ge 25$ required of both | ||||
| 318 | +normalised titles, | ||||
| 250 | \begin{equation} | 319 | \begin{equation} | ||
| 251 | − (p, b) \text{ is a candidate pair} \iff | 320 | + (p, b),\ p \in \mathcal{P},\ b \in \mathcal{B},\ \text{is a candidate pair} \iff | ||
| 252 | − \eta(p) = \eta(b),\;\; |y(p) - y(b)| \le 2,\;\; | 321 | + \bigl[\, \eta(p) = \eta(b) \wedge |y(p) - y(b)| \le 2 \,\bigr] | ||
| 253 | − p \in \mathcal{P},\;\; b \notin \mathcal{P},\;\; |\text{title}| \ge 25, | 322 | + \;\vee\; \bigl[\, \mathrm{arx}(p) = \mathrm{arx}(b) \neq \emptyset \,\bigr], | ||
| 254 | \label{eq:duppair} | 323 | \label{eq:duppair} | ||
| 255 | \end{equation} | 324 | \end{equation} | ||
| 256 | −where $\mathcal{P} = \{ w : \tau(w) = \text{preprint} \text{ or } \text{DOI}(w) | 325 | +where $\mathrm{arx}(\cdot)$ is the arXiv identifier of the record (the linked | ||
| 257 | −\text{ starts with } \text{10.48550/arxiv} \}$ is the preprint-like set. A | 326 | +one when the cascade of Section~\ref{sec:m-cascade} supplied it, otherwise the | ||
| 327 | +identifier OpenAlex itself carries). The first route is labelled | ||||
| 328 | +\textit{title\_year} and the second \textit{arxiv\_id}; among eligible works | ||||
| 329 | +they account for \PrismaVersionRelationshipTitleYear{} and | ||||
| 330 | +\PrismaVersionRelationshipArxivId{} merged pairs respectively (Part III). A | ||||
| 258 | preprint may match several published records and conversely; the rule keeps, | 331 | preprint may match several published records and conversely; the rule keeps, | ||
| 259 | for each published record, the earliest-dated preprint and, for each preprint, | 332 | for each published record, the earliest-dated preprint and, for each preprint, | ||
| 260 | the earliest-dated published record, and discards any pair that is not the | 333 | the earliest-dated published record, and discards any pair that is not the | ||
| 261 | −first in both orderings. The surviving pairs define the map $\pi$ of | 334 | +first in both orderings (a pair found by both routes is counted once, and | ||
| 262 | −equation~\eqref{eq:first-date}: the published record $b$ receives | 335 | +which of the two labels it retains is not fixed by the rule). The surviving | ||
| 263 | −$\pi(b) = p$, $\text{preprint\_date}(b) = d_{\mathrm{pub}}(p)$ and the earlier of | 336 | +pairs define the map $\pi$ of equation~\eqref{eq:first-date}: the published | ||
| 264 | −the two dates as its first date; the preprint $p$ receives the flag | 337 | +record $b$ receives $\pi(b) = p$, $\text{preprint\_date}(b) = d_{\mathrm{pub}}(p)$ | ||
| 265 | −\textit{e5\_duplicate} and leaves the eligible corpus. The version relationship | 338 | +and the first date of equation~\eqref{eq:first-date}; the preprint $p$ | ||
| 266 | −is therefore kept on the surviving node, as the plan requires, but the match is | 339 | +receives the flag \textit{e5\_duplicate} and leaves the eligible corpus. The | ||
| 267 | −by exact normalised title only. Two limitations follow. A preprint whose title | 340 | +version relationship is therefore kept on the surviving node, as the plan | ||
| 268 | −changed between versions is not merged, and the pair survives as two works; a | 341 | +requires, but a title match is exact. Two limitations follow. A preprint whose | ||
| 269 | −preprint and an unrelated published paper with an identical normalised title | 342 | +title changed between versions is merged only if the two records share an | ||
| 270 | −within two years are wrongly merged. The second error is rare for titles of at | 343 | +arXiv identifier, and otherwise survives as a second work; a preprint and an | ||
| 271 | −least 25 characters; the first is the more consequential one and biases the | 344 | +unrelated published paper with an identical normalised title within two years | ||
| 272 | −preprint share upward, which is the conservative direction for hypothesis H7. | 345 | +are wrongly merged. The second error is rare for titles of at least 25 | ||
| 346 | +characters; the first is the more consequential one and biases the preprint | ||||
| 347 | +share upward, which is the conservative direction for hypothesis H7. | ||||
| 273 | Deduplication across the OpenAlex--DBLP boundary is not needed because DBLP | 348 | Deduplication across the OpenAlex--DBLP boundary is not needed because DBLP | ||
| 274 | records are never counted as works: a DBLP record either links to an OpenAlex | 349 | records are never counted as works: a DBLP record either links to an OpenAlex | ||
| 275 | record, in which case the OpenAlex record carries the flag \textit{in\_dblp}, or | 350 | record, in which case the OpenAlex record carries the flag \textit{in\_dblp}, or | ||
@@ -299,14 +374,21 @@ flags are | |||||
| 299 | \end{align} | 374 | \end{align} | ||
| 300 | The membership flags encode the computer-science rule of the protocol: | 375 | The membership flags encode the computer-science rule of the protocol: | ||
| 301 | \begin{align} | 376 | \begin{align} | ||
| 302 | − \text{core}(w) &= \text{in\_dblp}(w) \;\vee\; \text{cs\_primary}(w) \;\vee\; | 377 | + \text{core}(w) &= \text{in\_dblp}(w) \;\vee\; \text{cs\_primary}(w) \;\vee\; \text{arxiv\_cs\_primary}(w),\\ | ||
| 303 | − \bigl(\text{cs\_any}(w) \wedge \text{arxiv\_cs\_primary}(w)\bigr),\\ | ||||
| 304 | \text{penumbra}(w) &= \text{cs\_any}(w) \wedge \neg\,\text{core}(w), | 378 | \text{penumbra}(w) &= \text{cs\_any}(w) \wedge \neg\,\text{core}(w), | ||
| 305 | \end{align} | 379 | \end{align} | ||
| 306 | where \text{cs\_primary} means that the primary OpenAlex topic belongs to field | 380 | where \text{cs\_primary} means that the primary OpenAlex topic belongs to field | ||
| 307 | 17, \text{cs\_any} that some topic does, and \text{arxiv\_cs\_primary} that the | 381 | 17, \text{cs\_any} that some topic does, and \text{arxiv\_cs\_primary} that the | ||
| 308 | −linked arXiv record's primary category starts with \texttt{cs.}. The eligibility | 382 | +linked arXiv record's primary category starts with \texttt{cs.} (amendment | ||
| 309 | −predicate is | 383 | +A-006; the earlier form of the third clause, $\text{cs\_any} \wedge | ||
| 384 | +\text{arxiv\_cs\_primary}$, is discussed in Section~\ref{sec:m-cascade}). A | ||||
| 385 | +record that survives the exclusions but is neither core nor penumbra---no | ||||
| 386 | +computer-science topic at all, not DBLP-indexed, and no \texttt{cs.} arXiv | ||||
| 387 | +primary category, typically a second-pass work reached through an arXiv DOI | ||||
| 388 | +whose arXiv primary category lies outside computer science---forms a third | ||||
| 389 | +class of \PrismaNeitherCoreNorPenumbraAfterExclusions{} records, reported in | ||||
| 390 | +the PRISMA flow of Part III; the script asserts that the three classes | ||||
| 391 | +partition the surviving records. The eligibility predicate is | ||||
| 310 | \begin{equation} | 392 | \begin{equation} | ||
| 311 | E(w) \;=\; \neg e_1 \wedge \neg e_2 \wedge \neg e_3 \wedge \neg e_4 \wedge \neg e_5 \wedge \neg e_7 \wedge \text{core}(w). | 393 | E(w) \;=\; \neg e_1 \wedge \neg e_2 \wedge \neg e_3 \wedge \neg e_4 \wedge \neg e_5 \wedge \neg e_7 \wedge \text{core}(w). | ||
| 312 | \label{eq:eligible} | 394 | \label{eq:eligible} | ||
@@ -349,10 +431,15 @@ last stratum is empty in the current run, for the reason given in | |||||
| 349 | Section~\ref{sec:m-cascade}), selected by ordering each stratum on a hash of the | 431 | Section~\ref{sec:m-cascade}), selected by ordering each stratum on a hash of the | ||
| 350 | DBLP key salted with the seed 44 and taking the first records, which is a | 432 | DBLP key salted with the seed 44 and taking the first records, which is a | ||
| 351 | reproducible simple random sample within stratum. For each pair the script | 433 | reproducible simple random sample within stratum. For each pair the script | ||
| 352 | −computes an automatic agreement vector: a string similarity between the two raw | 434 | +computes an automatic agreement vector: a string similarity between the two | ||
| 353 | −titles (Python's \texttt{difflib} ratio, twice the number of matched characters | 435 | +titles after a normalisation close to $\eta$ (NFKD decomposition with | ||
| 354 | −in the longest-matching-block alignment divided by the total length), the absolute year difference, and whether the last tokens of the two | 436 | +non-ASCII characters dropped, lower-casing, runs of non-alphanumeric | ||
| 355 | −first-author names coincide; a pair is \textit{auto\_ok} if the similarity is at | 437 | +characters collapsed to one space), namely Python's \texttt{difflib} ratio | ||
| 438 | +(twice the number of matched characters in the longest-matching-block | ||||
| 439 | +alignment divided by the total length), the absolute year difference, and | ||||
| 440 | +whether the last tokens of the two first-author names coincide (lower-cased, | ||||
| 441 | +diacritics stripped, DBLP's four-digit disambiguation suffix removed); a pair | ||||
| 442 | +is \textit{auto\_ok} if the similarity is at | ||||
| 356 | least 0.9, the years differ by at most one and the surnames agree whenever both | 443 | least 0.9, the years differ by at most one and the surnames agree whenever both | ||
| 357 | are present. Two of every five pairs are reserved for the principal | 444 | are present. Two of every five pairs are reserved for the principal | ||
| 358 | investigator (\textit{pi\_review}), whose verdicts constitute the human | 445 | investigator (\textit{pi\_review}), whose verdicts constitute the human | ||
@@ -365,8 +452,19 @@ weighted estimate | |||||
| 365 | \label{eq:linkprec} | 452 | \label{eq:linkprec} | ||
| 366 | \end{equation} | 453 | \end{equation} | ||
| 367 | where $L_s$ is the number of links of method $s$ in the full table and $L$ | 454 | where $L_s$ is the number of links of method $s$ in the full table and $L$ | ||
| 368 | −their sum, so that the title--year stratum, which is the only one where errors | 455 | +their sum, so that each stratum is weighted by its share of the links; since | ||
| 369 | −are plausible, is weighted by its small share of the links. The estimate | 456 | +the DOI step produces the great majority of them, $\hat p$ is dominated by the | ||
| 457 | +DOI stratum. This matters because the DOI stratum is not the error-free one. | ||||
| 458 | +The automatic check (\texttt{data/metadata/linkage\_sample\_summary.json}) | ||||
| 459 | +passes every title--year pair of the sample but fails a small number of DOI | ||||
| 460 | +pairs, of two kinds: a DOI that covers an entire edited volume, so that a | ||||
| 461 | +chapter is linked to the record of the book or of another chapter, and a | ||||
| 462 | +version of record whose title differs from the DBLP title under the same DOI. | ||||
| 463 | +Only the first kind is a linkage error; the second is a correct link that | ||||
| 464 | +the title test rejects. An identifier match is thus near-conclusive for the | ||||
| 465 | +existence of a relation between the two records but not for the identity of | ||||
| 466 | +the work, and the principal investigator's verdicts (H-1), not the automatic | ||||
| 467 | +rate, settle the precision of each stratum. The estimate | ||||
| 370 | \pending{linkage precision from the H-1 sample, data/metadata/linkage\_sample\_summary.json} | 468 | \pending{linkage precision from the H-1 sample, data/metadata/linkage\_sample\_summary.json} | ||
| 371 | is reported in Part III once the checkpoint is signed; until then the automatic | 469 | is reported in Part III once the checkpoint is signed; until then the automatic | ||
| 372 | agreement rate is the only available figure and is labelled as such. Recall is | 470 | agreement rate is the only available figure and is labelled as such. Recall is | ||
@@ -564,7 +662,8 @@ definitions follow the review of \citet{waltman2016review} where a choice had | |||||
| 564 | to be made; what is specific to this study is the citation-window discipline | 662 | to be made; what is specific to this study is the citation-window discipline | ||
| 565 | imposed by the eight-year span of the corpus and the explicit treatment of the | 663 | imposed by the eight-year span of the corpus and the explicit treatment of the | ||
| 566 | two counting conventions. The scripts are \texttt{bibliometrics/01\_volume\_series.py}, | 664 | two counting conventions. The scripts are \texttt{bibliometrics/01\_volume\_series.py}, | ||
| 567 | −\texttt{02\_collaboration.py} and \texttt{03\_citation\_impact.py}. | 665 | +\texttt{02\_collaboration.py}, \texttt{03\_citation\_impact.py} and | ||
| 666 | +\texttt{04\_diversity\_cagr.py}. | ||||
| 568 | 667 | ||||
| 569 | \subsection{Volume, composition and growth} | 668 | \subsection{Volume, composition and growth} | ||
| 570 | 669 | ||||
@@ -585,10 +684,17 @@ compound annual growth rate | |||||
| 585 | applied to annual totals and to annual category totals; it is a descriptive | 684 | applied to annual totals and to annual category totals; it is a descriptive | ||
| 586 | summary of two end points and says nothing about the path between them, which | 685 | summary of two end points and says nothing about the path between them, which | ||
| 587 | is why the structural-change analysis of Section~\ref{sec:m-structural} works | 686 | is why the structural-change analysis of Section~\ref{sec:m-structural} works | ||
| 588 | −on the full monthly series instead. The CAGR is computed at table-making time | 687 | +on the full monthly series instead. The CAGR is computed by | ||
| 589 | −from the annual counts rather than by a dedicated script; the plan lists it | 688 | +\texttt{bibliometrics/04\_diversity\_cagr.py} on the annual primary-label | ||
| 590 | −among the Phase-4 indicators and it is reported for complete years only, since | 689 | +counts of each category (\texttt{results/graph/category\_counts\_yearly.csv}) | ||
| 591 | −2026 is right-censored (Section~\ref{sec:m-truncation}). | 690 | +over two spans of complete years, 2018 to 2022 ($\Delta = 4$) within the | ||
| 691 | +baseline and 2023 to 2025 ($\Delta = 2$) within the post period, together with | ||||
| 692 | +the category's share of the year's classified works in 2018, 2022 and 2025; | ||||
| 693 | +it is undefined when the count of the start year is zero. The plan lists it | ||||
| 694 | +among the Phase-4 indicators; 2026 is excluded because it is right-censored | ||||
| 695 | +(Section~\ref{sec:m-truncation}). The two spans split the window at the | ||||
| 696 | +calendar year, so the whole of 2022 is counted in the baseline span, unlike the | ||||
| 697 | +periods of Section~\ref{sec:m-notation}. | ||||
| 592 | 698 | ||||
| 593 | \subsection{Fixed citation windows} | 699 | \subsection{Fixed citation windows} | ||
| 594 | \label{sec:m-citations} | 700 | \label{sec:m-citations} | ||
@@ -612,6 +718,21 @@ receive windows that differ by eleven months in effective length. This | |||||
| 612 | granularity is imposed by the data (OpenAlex does not provide citation dates | 718 | granularity is imposed by the data (OpenAlex does not provide citation dates | ||
| 613 | below the year) and is the same for the baseline and the post period, so it | 719 | below the year) and is the same for the baseline and the post period, so it | ||
| 614 | blurs comparisons within a year but does not bias comparisons between years. | 720 | blurs comparisons within a year but does not bias comparisons between years. | ||
| 721 | +Two further conventions of \texttt{03\_citation\_impact.py} govern every | ||||
| 722 | +impact statistic. The citations of a canonical node include those of its | ||||
| 723 | +merged preprint: the yearly counts of $\pi(w)$ are appended to those of $w$ | ||||
| 724 | +before the windows are summed (the \texttt{CY} union in the script), so a | ||||
| 725 | +work that circulated as a preprint and was then published is credited with | ||||
| 726 | +the citations of both records. This is deliberate, since the two records are | ||||
| 727 | +one work, but it means that $c^{(h)}_w$ can exceed what the OpenAlex record of | ||||
| 728 | +the version of record shows on its own; the union applies only to pairs | ||||
| 729 | +found by the rule of Section~\ref{sec:m-dedup}, a preprint that OpenAlex has | ||||
| 730 | +already merged with its publication being a single record. And the population | ||||
| 731 | +is the set of eligible, \emph{non-retracted} works: retracted works stay in the | ||||
| 732 | +mapping corpus (Section~\ref{sec:m-eligibility}) but are excluded from the | ||||
| 733 | +window sums, from the reference classes below and from the inequality | ||||
| 734 | +statistics, so the yearly $n$ of the citation tables falls short of the | ||||
| 735 | +volume tables by the number of retracted works. | ||||
| 615 | The window is the device that neutralises citation-age bias: comparing the | 736 | The window is the device that neutralises citation-age bias: comparing the | ||
| 616 | total citation count of a 2019 work with that of a 2025 work compares seven | 737 | total citation count of a 2019 work with that of a 2025 work compares seven | ||
| 617 | years of exposure with one, whereas comparing $c^{(36)}$ compares equal | 738 | years of exposure with one, whereas comparing $c^{(36)}$ compares equal | ||
@@ -619,7 +740,7 @@ exposures. The cost is that the last window-length of the corpus cannot be | |||||
| 619 | evaluated; $c^{(36)}$ is complete for publication years up to 2023 given a | 740 | evaluated; $c^{(36)}$ is complete for publication years up to 2023 given a | ||
| 620 | snapshot taken late in 2026, and $c^{(12)}$ up to 2024. Tables flag the years | 741 | snapshot taken late in 2026, and $c^{(12)}$ up to 2024. Tables flag the years | ||
| 621 | for which a window is incomplete, and the choice of window length follows the | 742 | for which a window is incomplete, and the choice of window length follows the | ||
| 622 | −evidence in \citet{wang2013window} that short windows rank works materially | 743 | +evidence in \citet{wang2013citationwindow} that short windows rank works materially | ||
| 623 | differently from long ones in slow-moving fields, which is why two lengths are | 744 | differently from long ones in slow-moving fields, which is why two lengths are | ||
| 624 | reported rather than one. | 745 | reported rather than one. | ||
| 625 | 746 | ||||
@@ -628,14 +749,14 @@ reported rather than one. | |||||
| 628 | Raw citation counts differ across subfields and document types by factors that | 749 | Raw citation counts differ across subfields and document types by factors that | ||
| 629 | have nothing to do with influence. The normalised citation score of a work is | 750 | have nothing to do with influence. The normalised citation score of a work is | ||
| 630 | its window count divided by the mean window count of its reference class | 751 | its window count divided by the mean window count of its reference class | ||
| 631 | −\citep{waltman2016review, waltman2019fieldnorm}: | 752 | +\citep{waltman2016review, waltman2019fieldnormalization}: | ||
| 632 | \begin{equation} | 753 | \begin{equation} | ||
| 633 | \mathrm{NCS}^{(h)}_w \;=\; \frac{c^{(h)}_w}{\bar c^{(h)}\bigl(y(w), \tau(w), f(w)\bigr)}, \qquad | 754 | \mathrm{NCS}^{(h)}_w \;=\; \frac{c^{(h)}_w}{\bar c^{(h)}\bigl(y(w), \tau(w), f(w)\bigr)}, \qquad | ||
| 634 | \bar c^{(h)}(y, \tau, f) = \frac{1}{|\mathcal{W}_{y,\tau,f}|}\sum_{v \in \mathcal{W}_{y,\tau,f}} c^{(h)}_v , | 755 | \bar c^{(h)}(y, \tau, f) = \frac{1}{|\mathcal{W}_{y,\tau,f}|}\sum_{v \in \mathcal{W}_{y,\tau,f}} c^{(h)}_v , | ||
| 635 | \label{eq:ncs} | 756 | \label{eq:ncs} | ||
| 636 | \end{equation} | 757 | \end{equation} | ||
| 637 | where $f(w)$ is the primary OpenAlex subfield of $w$ and $\mathcal{W}_{y,\tau,f}$ | 758 | where $f(w)$ is the primary OpenAlex subfield of $w$ and $\mathcal{W}_{y,\tau,f}$ | ||
| 638 | −the set of eligible works sharing year, type and subfield. The reference class | 759 | +the set of eligible, non-retracted works sharing year, type and subfield. The reference class | ||
| 639 | is computed on the eligible corpus, not on all of OpenAlex, so the | 760 | is computed on the eligible corpus, not on all of OpenAlex, so the | ||
| 640 | normalisation is relative to computer science as delimited here: an NCS of 2 | 761 | normalisation is relative to computer science as delimited here: an NCS of 2 | ||
| 641 | means twice the mean of comparable computer-science works, not twice the world | 762 | means twice the mean of comparable computer-science works, not twice the world | ||
@@ -660,7 +781,8 @@ reasons FWCI is reported for comparison and never used to select works. | |||||
| 660 | \subsection{Inequality and concentration} | 781 | \subsection{Inequality and concentration} | ||
| 661 | 782 | ||||
| 662 | Citation inequality within a publication year is measured by the Gini | 783 | Citation inequality within a publication year is measured by the Gini | ||
| 663 | −coefficient of $c^{(36)}$ among the $n$ eligible works of that year. With the | 784 | +coefficient of $c^{(36)}$ (and, as a secondary figure, of $c^{(12)}$) among | ||
| 785 | +the $n$ eligible, non-retracted works of that year. With the | ||||
| 664 | works sorted so that $c_{(1)} \le c_{(2)} \le \dots \le c_{(n)}$, the estimator | 786 | works sorted so that $c_{(1)} \le c_{(2)} \le \dots \le c_{(n)}$, the estimator | ||
| 665 | implemented in SQL is the rank form | 787 | implemented in SQL is the rank form | ||
| 666 | \begin{equation} | 788 | \begin{equation} | ||
@@ -691,8 +813,13 @@ largest single share are reported with the index. Because arXiv is a single | |||||
| 691 | OpenAlex source, growth in preprints mechanically raises $\mathrm{HHI}_y$ even | 813 | OpenAlex source, growth in preprints mechanically raises $\mathrm{HHI}_y$ even | ||
| 692 | when the journal and conference landscape becomes more dispersed; this is the | 814 | when the journal and conference landscape becomes more dispersed; this is the | ||
| 693 | reason the protocol's H5 predicts a stable or falling HHI only ``because of | 815 | reason the protocol's H5 predicts a stable or falling HHI only ``because of | ||
| 694 | −preprint growth'' with the preprint source treated separately, and the tables | 816 | +preprint growth'' with the preprint source treated separately. The | ||
| 695 | −report the index with and without the preprint type. | 817 | +implemented table (\texttt{venue\_hhi\_yearly.csv}, written by | ||
| 818 | +\texttt{02\_collaboration.py}) computes the index over all types in a single | ||||
| 819 | +pass; a variant that excludes the preprint type is not computed, so the | ||||
| 820 | +preprint contribution has to be judged from the largest single share reported | ||||
| 821 | +next to the index, which in the years when arXiv is the largest source is | ||||
| 822 | +arXiv's share. | ||||
| 696 | 823 | ||||
| 697 | \subsection{Collaboration, countries and institutions} | 824 | \subsection{Collaboration, countries and institutions} | ||
| 698 | 825 | ||||
@@ -725,10 +852,14 @@ table notes. | |||||
| 725 | \subsection{Diversity indices} | 852 | \subsection{Diversity indices} | ||
| 726 | \label{sec:m-diversity} | 853 | \label{sec:m-diversity} | ||
| 727 | 854 | ||||
| 728 | −Interdisciplinarity is measured on the distribution of a work's references, or | 855 | +Interdisciplinarity is measured on a distribution over the categories of the | ||
| 729 | −of a category's citations, over the categories of the ontology. Let | 856 | +ontology; the protocol speaks of the distribution of a work's references or of | ||
| 730 | −$p = (p_1, \dots, p_K)$ be such a distribution and $\delta_{ij} \in [0,1]$ a | 857 | +a category's citations, whereas the implemented pipeline applies the indices | ||
| 731 | −dissimilarity between categories $i$ and $j$. The Rao--Stirling index | 858 | +to the \emph{category mix} of each year, the distribution of the year's | ||
| 859 | +classified eligible works over their primary categories (the reference-based | ||||
| 860 | +variant is not computed). Let $p = (p_1, \dots, p_K)$ be such a distribution | ||||
| 861 | +and $\delta_{ij} \in [0,1]$ a dissimilarity between categories $i$ and $j$. | ||||
| 862 | +The Rao--Stirling index | ||||
| 732 | \citep{rao1982diversity, stirling2007diversity} | 863 | \citep{rao1982diversity, stirling2007diversity} | ||
| 733 | \begin{equation} | 864 | \begin{equation} | ||
| 734 | \Delta_{\mathrm{RS}}(p) \;=\; \sum_{i \ne j} p_i\, p_j\, \delta_{ij} | 865 | \Delta_{\mathrm{RS}}(p) \;=\; \sum_{i \ne j} p_i\, p_j\, \delta_{ij} | ||
@@ -760,21 +891,28 @@ categories (large $q$ counts only the dominant ones). Following | |||||
| 760 | closed form ${}^{2}D^{Z}(p) = 1 / \sum_{i,j} p_i z_{ij} p_j = 1/(1 - \Delta_{\mathrm{RS}}(p))$ | 891 | closed form ${}^{2}D^{Z}(p) = 1 / \sum_{i,j} p_i z_{ij} p_j = 1/(1 - \Delta_{\mathrm{RS}}(p))$ | ||
| 761 | when $z_{ii} = 1$, so that the two families are monotone transforms of each | 892 | when $z_{ii} = 1$, so that the two families are monotone transforms of each | ||
| 762 | other at this order and differ only in scale and in their behaviour at the | 893 | other at this order and differ only in scale and in their behaviour at the | ||
| 763 | −saturation end. The dissimilarity $\delta_{ij}$ between categories is one | 894 | +saturation end. Both indices are computed for every year up to 2025 by | ||
| 764 | −minus the cosine similarity of the category centroids of | 895 | +\texttt{bibliometrics/04\_diversity\_cagr.py} and reported in Part~IX\@. The | ||
| 765 | −Section~\ref{sec:m-semantic} in the baseline period, so that the disparity | 896 | +dissimilarity is $\delta_{ij} = 1 - z_{ij}$, where $z_{ij}$ is the cosine | ||
| 766 | −component is estimated from the corpus rather than imposed. The current | 897 | +similarity of the category centroids of Section~\ref{sec:m-novelty} when the | ||
| 767 | −pipeline computes the category flow matrices from which $p$ is formed | 898 | +centroid matrix exists (the script takes the post-period matrix first and | ||
| 768 | −(Section~\ref{sec:m-networks}) and the centroid similarities, but the two | 899 | +the baseline matrix as a fallback) and, when no centroid matrix is available, | ||
| 769 | −diversity indices are not yet produced by a script; they are defined here | 900 | +the co-classification Jaccard similarity $n_{ij} / (n_i + n_j - n_{ij})$ | ||
| 770 | −because the protocol commits to them, and their values in Part IX are | 901 | +computed from the yearly co-occurrence counts of the full label sets | ||
| 771 | −The yearly Rao--Stirling and Leinster--Cobbold indices of the category mix, | 902 | +(Section~\ref{sec:m-networks}), with $n_i$ the multi-label total of category | ||
| 772 | −with bootstrap intervals from multinomial resampling of works over categories | 903 | +$i$; the diagonal is set to one in both cases and the similarity source | ||
| 773 | −($B = 500$, seed 44), are computed by \texttt{bibliometrics/04\_diversity\_cagr.py} | 904 | +actually used is written into the output table, because values computed from | ||
| 774 | −and reported in Part~IX; the dissimilarity $d_{ij}$ is one minus the cosine | 905 | +the two sources are not on the same scale and must not be compared across | ||
| 775 | −similarity of category centroids when embeddings are available and one minus | 906 | +them. The disparity component is thus estimated from the corpus rather than | ||
| 776 | −the co-classification Jaccard similarity otherwise. | 907 | +imposed. The Rao--Stirling index carries a percentile interval from $B = 500$ | ||
| 777 | −until the computation is added and audited. | 908 | +multinomial resamples of the year's works over the categories (seed 44), the | ||
| 909 | +same resamples that give the yearly category-share intervals; the | ||||
| 910 | +Leinster--Cobbold index of order two is reported without an interval. The | ||||
| 911 | +multinomial bootstrap treats the category assignments of a year as | ||||
| 912 | +independent draws from the observed mix, so its interval reflects only that | ||||
| 913 | +sampling variability and not the classification error of | ||||
| 914 | +Section~\ref{sec:m-validation}, which is the larger source of uncertainty for | ||||
| 915 | +these indices. | ||||
| 778 | 916 | ||||
| 779 | \section{Citation and collaboration networks} | 917 | \section{Citation and collaboration networks} | ||
| 780 | \label{sec:m-networks} | 918 | \label{sec:m-networks} | ||
@@ -1017,7 +1155,7 @@ that are new; the interpretation of high novelty as ``new'' rather than | |||||
| 1017 | ``interdisciplinary'' requires the clusters and the flow statistics as | 1155 | ``interdisciplinary'' requires the clusters and the flow statistics as | ||
| 1018 | corroboration, and the literature on novelty indicators warns that | 1156 | corroboration, and the literature on novelty indicators warns that | ||
| 1019 | citation-based and text-based novelty measures do not agree well | 1157 | citation-based and text-based novelty measures do not agree well | ||
| 1020 | −\citep{uzzi2013atypical, wang2017novelty}. | 1158 | +\citep{uzzi2013novelty, wang2017novelty}. | ||
| 1021 | 1159 | ||||
| 1022 | \section{Structural change in monthly series} | 1160 | \section{Structural change in monthly series} | ||
| 1023 | \label{sec:m-structural} | 1161 | \label{sec:m-structural} | ||
@@ -1042,8 +1180,12 @@ uses no causal vocabulary about it (CLAUDE.md §6.5). | |||||
| 1042 | \label{sec:m-truncation} | 1180 | \label{sec:m-truncation} | ||
| 1043 | 1181 | ||||
| 1044 | Each series is a vector of monthly counts $N_t$ on the month-precision | 1182 | Each series is a vector of monthly counts $N_t$ on the month-precision | ||
| 1045 | −subset, from February 2018 (the default start of the driver script, which | 1183 | +subset, from February 2018 (the driver's default \texttt{--first}; the | ||
| 1046 | −drops one month and is immaterial) to the last complete month | 1184 | +script documents no reason for omitting January 2018, the omission shortens | ||
| 1185 | +every series by one baseline month, to \ItsSeriesTotalNMonths{} months for | ||||
| 1186 | +the default end, and the manuscript makes no claim about its effect beyond | ||||
| 1187 | +noting that the omitted month is the first of the baseline and enters only the | ||||
| 1188 | +pre-trend) to the last complete month | ||||
| 1047 | $t_{\mathrm{L}}$, by default June 2026, with sensitivity runs at March and | 1189 | $t_{\mathrm{L}}$, by default June 2026, with sensitivity runs at March and | ||
| 1048 | August 2026. Category series $N_{k,t}$ use the multi-label count restricted to | 1190 | August 2026. Category series $N_{k,t}$ use the multi-label count restricted to | ||
| 1049 | month-precision works, and a series is analysed only if it has at least 48 | 1191 | month-precision works, and a series is analysed only if it has at least 48 | ||
@@ -1064,10 +1206,20 @@ years assumes that the profile is stationary, and the profile is itself | |||||
| 1064 | affected by the events under study (OpenAlex's own ingestion changed over the | 1206 | affected by the events under study (OpenAlex's own ingestion changed over the | ||
| 1065 | period); a correction would import that assumption into every test. Truncating | 1207 | period); a correction would import that assumption into every test. Truncating | ||
| 1066 | to the last month at which the lag profile indicates near-complete indexing | 1208 | to the last month at which the lag profile indicates near-complete indexing | ||
| 1067 | −loses a few months of data but introduces no model. Donut windows (dropping | 1209 | +loses a few months of data but introduces no model. A donut specification is | ||
| 1068 | −the months adjacent to the event) are not implemented; the alternative event | 1210 | +also run (\texttt{--donut}, default three): the ITS of Section~\ref{sec:m-its} | ||
| 1069 | −dates of Section~\ref{sec:m-altdates} serve the related purpose of showing | 1211 | +is re-estimated for the reference date after removing the event month and the | ||
| 1070 | −whether the estimated change is tied to a particular month. | 1212 | +three months on each side of it (every month whose first day lies within | ||
| 1213 | +$3 \times 31$ days of the first day of the event month, seven months in all, | ||||
| 1214 | +leaving \ItsSeriesTotalItsChatgptDonutN{} observations for the total series), | ||||
| 1215 | +so that the level and slope changes are identified from months at least a | ||||
| 1216 | +quarter away from the marker and not from the transition months themselves. | ||||
| 1217 | +For the total series the donut estimate of the level change is | ||||
| 1218 | +\ItsSeriesTotalItsChatgptDonutLevelChange{} in log points with HAC | ||||
| 1219 | +$p$-value \ItsSeriesTotalItsChatgptDonutLevelP{}, to be read against the | ||||
| 1220 | +full-sample estimate of Part IX\@. The alternative event dates of | ||||
| 1221 | +Section~\ref{sec:m-altdates} serve the related purpose of showing whether the | ||||
| 1222 | +estimated change is tied to a particular month. | ||||
| 1071 | 1223 | ||||
| 1072 | \subsection{Interrupted time series with seasonality and HAC errors} | 1224 | \subsection{Interrupted time series with seasonality and HAC errors} | ||
| 1073 | \label{sec:m-its} | 1225 | \label{sec:m-its} | ||
@@ -1119,9 +1271,14 @@ The placebo $p$-value of the true event is its rank among these: | |||||
| 1119 | \label{eq:placebo} | 1271 | \label{eq:placebo} | ||
| 1120 | \end{equation} | 1272 | \end{equation} | ||
| 1121 | and likewise for $\beta_3$. The set $\mathcal{T}'$ contains the true event, so | 1273 | and likewise for $\beta_3$. The set $\mathcal{T}'$ contains the true event, so | ||
| 1122 | −the smallest attainable value is $1/|\mathcal{T}'|$. This statistic is the | 1274 | +the smallest attainable value is $1/|\mathcal{T}'|$. A second rank, | ||
| 1275 | +restricted to the candidate dates \emph{before} the true event (the plan's | ||||
| 1276 | +original wording), is computed alongside it; because that restriction removes | ||||
| 1277 | +the true event from the candidate set, the baseline-only rank can be exactly | ||||
| 1278 | +zero when no baseline date produces a larger $t$-statistic, and Part IX states | ||||
| 1279 | +this convention next to the two values. This statistic is the | ||||
| 1123 | ``placebo in time'' diagnostic recommended for interrupted time series | 1280 | ``placebo in time'' diagnostic recommended for interrupted time series | ||
| 1124 | −\citep{bernal2018controls}: if a level shift of comparable $t$-statistic is | 1281 | +\citep{bernal2018itscontrol}: if a level shift of comparable $t$-statistic is | ||
| 1125 | found at a quarter of all candidate dates, a conventional $p$-value from the | 1282 | found at a quarter of all candidate dates, a conventional $p$-value from the | ||
| 1126 | HAC standard error of $\beta_2$ overstates the evidence, because the series | 1283 | HAC standard error of $\beta_2$ overstates the evidence, because the series | ||
| 1127 | has enough curvature and persistence that any date looks like a break. The | 1284 | has enough curvature and persistence that any date looks like a break. The | ||
@@ -1129,8 +1286,8 @@ value is not a $p$-value under a null hypothesis in the usual sense (the | |||||
| 1129 | candidate dates are not exchangeable draws from a null distribution, and the | 1286 | candidate dates are not exchangeable draws from a null distribution, and the | ||
| 1130 | regressions share the same data), and the manuscript calls it a placebo rank. | 1287 | regressions share the same data), and the manuscript calls it a placebo rank. | ||
| 1131 | Two implementation details depart from the plan's wording. The plan speaks of | 1288 | Two implementation details depart from the plan's wording. The plan speaks of | ||
| 1132 | −placebo breaks ``at every month of the baseline''; the implementation ranges | 1289 | +placebo breaks ``at every month of the baseline''; the primary implementation | ||
| 1133 | −over the baseline \emph{and} the post period, so that a change in 2023 or | 1290 | +ranges over the baseline \emph{and} the post period, so that a change in 2023 or | ||
| 1134 | 2024 that is larger than the one at the reference date lowers the rank of the | 1291 | 2024 that is larger than the one at the reference date lowers the rank of the | ||
| 1135 | reference date, which is the conservative direction for hypothesis H1 and is | 1292 | reference date, which is the conservative direction for hypothesis H1 and is | ||
| 1136 | why the wider range was kept. And the minimum of 24 pre-months and 12 | 1293 | why the wider range was kept. And the minimum of 24 pre-months and 12 | ||
@@ -1141,10 +1298,10 @@ comparison. | |||||
| 1141 | \subsection{Multiple breaks: dynamic programming with information criteria} | 1298 | \subsection{Multiple breaks: dynamic programming with information criteria} | ||
| 1142 | \label{sec:m-bp} | 1299 | \label{sec:m-bp} | ||
| 1143 | 1300 | ||||
| 1144 | −The Bai--Perron framework \citep{bai1998breaks, bai2003computation} estimates | 1301 | +The Bai--Perron framework \citep{bai1998breaks, bai2003breaks} estimates | ||
| 1145 | $m$ unknown break dates $\tau_1 < \dots < \tau_m$ as the global minimiser of | 1302 | $m$ unknown break dates $\tau_1 < \dots < \tau_m$ as the global minimiser of | ||
| 1146 | the sum of squared residuals of a segment-wise regression. The implementation | 1303 | the sum of squared residuals of a segment-wise regression. The implementation | ||
| 1147 | −is the pure dynamic programme of \citet{bai2003computation} on the log series | 1304 | +is the pure dynamic programme of \citet{bai2003breaks} on the log series | ||
| 1148 | with a segment-wise linear trend: for a segment $[i, j)$ of at least $h$ | 1305 | with a segment-wise linear trend: for a segment $[i, j)$ of at least $h$ | ||
| 1149 | observations, | 1306 | observations, | ||
| 1150 | \begin{equation} | 1307 | \begin{equation} | ||
@@ -1159,8 +1316,9 @@ is obtained recursively, | |||||
| 1159 | with the break positions recovered by back-tracking. The minimum segment | 1316 | with the break positions recovered by back-tracking. The minimum segment | ||
| 1160 | length is $h = \max\{8, \lfloor 0.15\, n \rfloor\}$ months, a trimming of 15 per | 1317 | length is $h = \max\{8, \lfloor 0.15\, n \rfloor\}$ months, a trimming of 15 per | ||
| 1161 | cent that is the usual choice in this literature; it forbids breaks in the | 1318 | cent that is the usual choice in this literature; it forbids breaks in the | ||
| 1162 | −first and last $h$ months, which for the default window of 101 months | 1319 | +first and last $h$ months, which for the default window of | ||
| 1163 | −($h = 15$) means that no break can be placed after April 2025, and it also | 1320 | +\ItsSeriesTotalNMonths{} months ($h = 15$) means that no break can be placed | ||
| 1321 | +after April 2025, and it also | ||||
| 1164 | forbids two breaks closer than $h$ months. The number of breaks, up to three, is selected by two | 1322 | forbids two breaks closer than $h$ months. The number of breaks, up to three, is selected by two | ||
| 1165 | criteria computed on the global minima $\mathrm{SSR}_m = C_m(n)$, | 1323 | criteria computed on the global minima $\mathrm{SSR}_m = C_m(n)$, | ||
| 1166 | \begin{align} | 1324 | \begin{align} | ||
@@ -1170,7 +1328,7 @@ criteria computed on the global minima $\mathrm{SSR}_m = C_m(n)$, | |||||
| 1170 | with $p_m = 2(m+1) + m$ parameters (an intercept and a slope per segment plus | 1328 | with $p_m = 2(m+1) + m$ parameters (an intercept and a slope per segment plus | ||
| 1171 | the $m$ break dates). The BIC is Schwarz's criterion \citep{schwarz1978bic}; | 1329 | the $m$ break dates). The BIC is Schwarz's criterion \citep{schwarz1978bic}; | ||
| 1172 | the LWZ criterion of \citet{liu1997segmented}, in the form used by | 1330 | the LWZ criterion of \citet{liu1997segmented}, in the form used by | ||
| 1173 | −\citet{bai2003computation}, penalises additional breaks more heavily and is | 1331 | +\citet{bai2003breaks}, penalises additional breaks more heavily and is | ||
| 1174 | known to select fewer spurious breaks when the errors are serially correlated, | 1332 | known to select fewer spurious breaks when the errors are serially correlated, | ||
| 1175 | which they are in these series (the monthly residual autocorrelation is what | 1333 | which they are in these series (the monthly residual autocorrelation is what | ||
| 1176 | the HAC errors of the ITS correct for). Both selections are reported. The | 1334 | the HAC errors of the ITS correct for). Both selections are reported. The | ||
@@ -1178,9 +1336,13 @@ sequential sup-$F$ tests of \citeauthor{bai1998breaks} are not implemented; in | |||||
| 1178 | their absence the break dates carry no $p$-value, and their stability is | 1336 | their absence the break dates carry no $p$-value, and their stability is | ||
| 1179 | assessed instead through the agreement of the two criteria, the sensitivity to | 1337 | assessed instead through the agreement of the two criteria, the sensitivity to | ||
| 1180 | $t_{\mathrm{L}}$, and the agreement with the two methods below. The regression | 1338 | $t_{\mathrm{L}}$, and the agreement with the two methods below. The regression | ||
| 1181 | −in each segment contains no seasonal terms, so a seasonal pattern can only be | 1339 | +in each segment contains no seasonal terms; instead the driver seasonally | ||
| 1182 | −absorbed by the error; with a trimming of eight or more months per segment | 1340 | +adjusts the log series before the search, subtracting from each month the | ||
| 1183 | −this cannot create a break by itself but it inflates $\mathrm{SSR}_m$ uniformly. | 1341 | +mean deviation of its calendar month from the overall mean, both means | ||
| 1342 | +computed on the baseline months only (before December 2022) so that the | ||||
| 1343 | +adjustment uses no post-period information. The same adjusted series is the | ||||
| 1344 | +input of the PELT and BOCPD methods below; only the ITS models seasonality | ||||
| 1345 | +inside the regression. | ||||
| 1184 | 1346 | ||||
| 1185 | \subsection{PELT} | 1347 | \subsection{PELT} | ||
| 1186 | 1348 | ||||
@@ -1306,7 +1468,8 @@ retrospective evaluation with a fixed vocabulary and is stated as the first | |||||
| 1306 | limitation of the framework. | 1468 | limitation of the framework. | ||
| 1307 | 1469 | ||||
| 1308 | The panel consists of two $K \times M$ matrices over topics $k$ and months | 1470 | The panel consists of two $K \times M$ matrices over topics $k$ and months | ||
| 1309 | −$t$ from January 2018 to the last month (October 2026 by default): the volume | 1471 | +$t$ from January 2018 to the last month (June 2026 by default, the driver's | ||
| 1472 | +\texttt{--last-month}, which must be the last complete month of the panel): the volume | ||||
| 1310 | $n_{k,t}$, the number of month-precision eligible works of topic $k$ with | 1473 | $n_{k,t}$, the number of month-precision eligible works of topic $k$ with | ||
| 1311 | first month $t$, and the citation inflow $c_{k,t}$, the number of internal | 1474 | first month $t$, and the citation inflow $c_{k,t}$, the number of internal | ||
| 1312 | citation edges whose cited work belongs to topic $k$ and whose citing work has | 1475 | citation edges whose cited work belongs to topic $k$ and whose citing work has | ||
@@ -1333,9 +1496,10 @@ log growth, the second its change over a year (acceleration), the third a | |||||
| 1333 | $z$-score of the last quarter's monthly volume against the trailing two years | 1496 | $z$-score of the last quarter's monthly volume against the trailing two years | ||
| 1334 | (the ``abnormal growth'' signal, which also serves as the baseline detector), | 1497 | (the ``abnormal growth'' signal, which also serves as the baseline detector), | ||
| 1335 | the fourth citation acceleration, the fifth a size control, the sixth the | 1498 | the fourth citation acceleration, the fifth a size control, the sixth the | ||
| 1336 | −topic's age in months since it reached five works. Every window ends at $T$; | 1499 | +topic's age in months since it reached five works. Every window ends at $T$, | ||
| 1337 | −the one-month overlap between the $z$-score's trailing window and the | 1500 | +and the quarter that the $z$-score evaluates ($T-2$ to $T$) does not overlap | ||
| 1338 | −quarter it is compared with is immaterial. The log-ratio form of growth with | 1501 | +the trailing window ($T-27$ to $T-3$) against which it is standardised. The | ||
| 1502 | +log-ratio form of growth with | ||||
| 1339 | the $+1$ shift keeps the features finite for topics that were empty in the | 1503 | the $+1$ shift keeps the features finite for topics that were empty in the | ||
| 1340 | earlier window. Of the seven candidate signals of the mandate, the | 1504 | earlier window. Of the seven candidate signals of the mandate, the | ||
| 1341 | implemented set covers abnormal growth, citation acceleration and (through the | 1505 | implemented set covers abnormal growth, citation acceleration and (through the | ||
@@ -1376,20 +1540,27 @@ positives separately. | |||||
| 1376 | 1540 | ||||
| 1377 | \subsection{Cutoffs, models and metrics} | 1541 | \subsection{Cutoffs, models and metrics} | ||
| 1378 | 1542 | ||||
| 1379 | −Training cutoffs are every third month from December 2019 to December 2021 | 1543 | +Cutoffs are the months whose index from January 2018 is a multiple of three, | ||
| 1380 | −for the 12-month horizon and to December 2020 for the 24-month horizon, so | 1544 | +that is, January, April, July and October of each year. Training cutoffs are | ||
| 1381 | −that every training label is observable before the first test cutoff; | 1545 | +those from December 2019 up to a horizon-dependent last month, December 2021 | ||
| 1382 | −test cutoffs are every third month from January 2023 to October 2024 for | 1546 | +for $h = 12$ and December 2020 for $h = 24$, so that the realised training | ||
| 1383 | −which $T + h$ lies within the panel. With $h = 24$ the last training label | 1547 | +cutoffs run from January 2020 to October 2021 (eight cutoffs) at $h = 12$ and | ||
| 1384 | −uses data to December 2023, which is after the reference date but more than a | 1548 | +from January 2020 to October 2020 (four cutoffs) at $h = 24$; in both cases | ||
| 1385 | −year before the first test label window closes, and no test cutoff precedes | 1549 | +the last training label window closes in October 2022, before the reference | ||
| 1386 | −2023; the twelve months of 2022 are used by neither set as a cutoff, which | 1550 | +date and before the first test cutoff. Test cutoffs are those from January | ||
| 1387 | −is the temporal gap that separates training from test in the sense of | 1551 | +2023 to October 2024 whose label window $T + h$ lies within the panel: with | ||
| 1388 | −\citet{bergmeir2012crossval}. The plan proposed training cutoffs through 2022; | 1552 | +the default last month of June 2026, January 2023 to October 2024 at $h = 12$ | ||
| 1389 | −the implementation stops at December 2021 so that the training labels do not | 1553 | +and January 2023 to April 2024 at $h = 24$. No cutoff of either set falls in | ||
| 1390 | −extend into the test period at $h = 24$, a stricter separation than the plan. | 1554 | +2022, and the interval between the close of the last training label (October | ||
| 1391 | −Rows are the eligible (topic, cutoff) pairs; features are the six quantities | 1555 | +2022) and the first test cutoff (January 2023) is the temporal separation of | ||
| 1392 | −above with non-finite values replaced by zero. | 1556 | +training from test in the sense of \citet{bergmeir2012crossval}. The plan | ||
| 1557 | +proposed training cutoffs through 2022; the implementation stops earlier so | ||||
| 1558 | +that no training label is observed after the reference date, a stricter | ||||
| 1559 | +separation than the plan, bought at the price of few training cutoffs at | ||||
| 1560 | +$h = 24$. The realised cutoff lists are written to the metrics file of each | ||||
| 1561 | +horizon (\texttt{train\_cutoffs}, \texttt{test\_cutoffs}). Rows are the | ||||
| 1562 | +eligible (topic, cutoff) pairs; features are the six quantities above with | ||||
| 1563 | +non-finite values replaced by zero. | ||||
| 1393 | 1564 | ||||
| 1394 | Three predictors are compared: the $z$-score baseline, which ranks topics by | 1565 | Three predictors are compared: the $z$-score baseline, which ranks topics by | ||
| 1395 | $z_{k,T}$ alone and involves no fitting; an $\ell_2$-regularised logistic | 1566 | $z_{k,T}$ alone and involves no fitting; an $\ell_2$-regularised logistic | ||
@@ -1419,20 +1590,38 @@ generation, state-space sequence models, vision--language models, mechanistic | |||||
| 1419 | interpretability, speculative decoding, mixture of experts at scale, | 1590 | interpretability, speculative decoding, mixture of experts at scale, | ||
| 1420 | test-time compute and reasoning models, AI for code generation) and predicts | 1591 | test-time compute and reasoning models, AI for code generation) and predicts | ||
| 1421 | that the detector flags each at least six months before its peak growth | 1592 | that the detector flags each at least six months before its peak growth | ||
| 1422 | −month. The evaluation requires a mapping from each development to one or | 1593 | +month. Under H8 the lead time of a development mapped to a topic $k$ would be | ||
| 1423 | −more OpenAlex topics, and this mapping is the weak point of the design: it | 1594 | +the number of months between the first test cutoff at which $k$ ranks in the | ||
| 1424 | −must be made by hand, after the taxonomy work, and a development that has no | 1595 | +top 50 and the month of maximum $g^{12}_{k,t}$ over the window. That | ||
| 1425 | −topic of its own (because the OpenAlex topic model folds it into a larger | 1596 | +definition requires a mapping from each development to one or more OpenAlex | ||
| 1426 | −one) cannot be detected by a topic-level detector whatever its merits. The | 1597 | +topics, and the mapping cannot be made: the twelve developments have no | ||
| 1427 | −script reserves a table for this mapping and it is empty in the current | 1598 | +topic of their own in the OpenAlex model (there is no topic for LLM agents, | ||
| 1428 | −version; the lead-time statistic is therefore | 1599 | +vision--language models, speculative decoding or mixture of experts), so a | ||
| 1429 | −\pending{known-emergence lead times, experiments/known\_set.csv not yet populated}, | 1600 | +topic-level detector cannot be scored on them whatever its merits. The check | ||
| 1430 | −and the mapping, when made, will be listed in the appendix with the reason | 1601 | +actually run (\texttt{experiments/known\_set.py}, | ||
| 1431 | −for each choice. Lead time itself is defined, for a development mapped to | 1602 | +\texttt{results/emerging/known\_set\_lead\_times.csv}) is therefore a | ||
| 1432 | −topic $k$, as the number of months between the first test cutoff at which | 1603 | +separate lead-time check on the preregistered concepts, and it departs from | ||
| 1433 | −$k$ ranks in the top 50 and the month of maximum $g^{12}_{k,t}$ over the | 1604 | +the H8 definition in three respects that are stated wherever its results | ||
| 1434 | −window; a development whose peak growth precedes the first test cutoff has | 1605 | +appear. First, the units are keyword series: each development is defined by | ||
| 1435 | −undefined lead time and is reported as such rather than dropped. | 1606 | +a hand-written regular expression over title and abstract, listed in the | ||
| 1607 | +script, and its monthly series counts the month-precision eligible works it | ||||
| 1608 | +matches. Second, the detector applied is the $z$-score baseline rule, not the | ||||
| 1609 | +trained models: a development is \emph{detected} in the first month $t$ at | ||||
| 1610 | +which the mean of months $t-2$ to $t$ exceeds the mean of months $t-26$ to | ||||
| 1611 | +$t-3$ by more than three of their standard deviations, the trailing | ||||
| 1612 | +twelve-month volume is at least thirty works, and that volume is at least | ||||
| 1613 | +twice the volume of the twelve months before. Third, the \emph{peak growth | ||||
| 1614 | +month} is the month of maximum twelve-month log growth over the whole | ||||
| 1615 | +series, chosen with full-sample knowledge, so the lead time (peak month minus | ||||
| 1616 | +detection month) is an ex-post quantity: a detection may precede a peak that | ||||
| 1617 | +no contemporaneous observer could have identified, and a negative lead time | ||||
| 1618 | +means that the series' fastest growth preceded the rule's firing. The | ||||
| 1619 | +regular expressions are themselves 2026 knowledge, since they name each | ||||
| 1620 | +development as it came to be called. The check thus measures whether a simple | ||||
| 1621 | +abnormal-growth rule on a named concept fires early relative to that | ||||
| 1622 | +concept's peak growth; it is not the H8 test of the trained detector, and | ||||
| 1623 | +Part IX reports it under that description | ||||
| 1624 | +(\pending{known-set lead times: no macro yet for results/emerging/known\_set\_lead\_times.csv}). | ||||
| 1436 | 1625 | ||||
| 1437 | \subsection{Look-ahead safeguards and what they do not cover} | 1626 | \subsection{Look-ahead safeguards and what they do not cover} | ||
| 1438 | 1627 | ||||
@@ -1565,13 +1754,17 @@ criteria and the sensitivity to truncation. The detector metrics carry | |||||
| 1565 | per-cutoff values, whose dispersion across cutoffs is the relevant measure of | 1754 | per-cutoff values, whose dispersion across cutoffs is the relevant measure of | ||
| 1566 | stability, since the rows within a cutoff are not independent. And the | 1755 | stability, since the rows within a cutoff are not independent. And the | ||
| 1567 | cluster count and the category similarity matrices carry the split-half ARI\@. | 1756 | cluster count and the category similarity matrices carry the split-half ARI\@. | ||
| 1568 | −Where the manuscript reports a bootstrap interval for a corpus statistic (the | 1757 | +The only bootstrap in the pipeline is that of | ||
| 1569 | −Gini coefficient by year, the cross-area shares), it is the percentile | 1758 | +\texttt{bibliometrics/04\_diversity\_cagr.py} (Section~\ref{sec:m-diversity}): | ||
| 1570 | −interval of the non-parametric bootstrap \citep{efron1979bootstrap} over | 1759 | +percentile intervals \citep{efron1979bootstrap} for the yearly Rao--Stirling | ||
| 1571 | −works, with 1,000 replications and seed 44, computed by the table-making | 1760 | +index and the yearly category shares, from $B = 500$ multinomial resamples of | ||
| 1572 | −script at reporting time; the protocol's commitment to ``bootstrap or HAC | 1761 | +each year's works over the categories with seed 44. No other corpus | ||
| 1573 | −intervals'' is met by these two devices, and a reported figure without an | 1762 | +statistic---the Gini coefficients, the cross-area shares, the modularity | ||
| 1574 | −interval is a population count for which none is appropriate. The | 1763 | +values---carries a resampling interval, and the manuscript reports none for | ||
| 1764 | +them; the protocol's commitment to ``bootstrap or HAC intervals'' is met by | ||||
| 1765 | +the diversity bootstrap and by the HAC errors of the ITS, and a reported | ||||
| 1766 | +figure without an interval is a population count for which none has been | ||||
| 1767 | +computed. The | ||||
| 1575 | indicators of Section~\ref{sec:m-bibliometrics} that depend on OpenAlex's | 1768 | indicators of Section~\ref{sec:m-bibliometrics} that depend on OpenAlex's | ||
| 1576 | nightly recomputation (FWCI, total citation counts) are snapshot quantities | 1769 | nightly recomputation (FWCI, total citation counts) are snapshot quantities | ||
| 1577 | and are dated. | 1770 | and are dated. | ||
@@ -1581,7 +1774,7 @@ and are dated. | |||||
| 1581 | Every stochastic step is seeded with the constant 44: the validation and | 1774 | Every stochastic step is seeded with the constant 44: the validation and | ||
| 1582 | linkage samples (hash ordering salted with the seed), the atlas sample and the | 1775 | linkage samples (hash ordering salted with the seed), the atlas sample and the | ||
| 1583 | two UMAP fits, the split-half permutations, the gradient boosting classifier, | 1776 | two UMAP fits, the split-half permutations, the gradient boosting classifier, | ||
| 1584 | −and the bootstrap. The regular-expression classifier, the SQL stages and the | 1777 | +and the diversity bootstrap. The regular-expression classifier, the SQL stages and the | ||
| 1585 | structural-change toolkit are deterministic. Two steps are deterministic only | 1778 | structural-change toolkit are deterministic. Two steps are deterministic only | ||
| 1586 | up to the platform: embedding on Apple-silicon GPUs with float16 storage | 1779 | up to the platform: embedding on Apple-silicon GPUs with float16 storage | ||
| 1587 | produces vectors that differ at the fourth decimal from a CPU run, and UMAP's | 1780 | produces vectors that differ at the fourth decimal from a CPU run, and UMAP's | ||
@@ -1619,10 +1812,10 @@ their hashes. | |||||
| 1619 | 1812 | ||||
| 1620 | \subsection{Human checkpoints and the status of conclusions} | 1813 | \subsection{Human checkpoints and the status of conclusions} | ||
| 1621 | 1814 | ||||
| 1622 | −Five checkpoints (H-1 to H-5, \texttt{reports/HUMAN\_REVIEW\_CHECKPOINTS.md}) | 1815 | +Six checkpoints (H-0 to H-5, \texttt{reports/HUMAN\_REVIEW\_CHECKPOINTS.md}) | ||
| 1623 | −require the principal investigator's review of the linkage sample, the | 1816 | +require the principal investigator's review of the plan, protocol and | ||
| 1624 | −taxonomy validation sample, the landmark list, the structural-change results | 1817 | +amendments (H-0), the linkage sample, the taxonomy validation sample, the | ||
| 1625 | −and the conclusions. Until a checkpoint is signed, the manuscript marks the | 1818 | +landmark list, the structural-change results and the conclusions. Until a checkpoint is signed, the manuscript marks the | ||
| 1626 | conclusions that depend on it as pending human review, and the numbers that | 1819 | conclusions that depend on it as pending human review, and the numbers that | ||
| 1627 | depend on it are rendered by the \texttt{\textbackslash pending} macro rather | 1820 | depend on it are rendered by the \texttt{\textbackslash pending} macro rather | ||
| 1628 | than typed. This is the mechanism by which the mandate's requirement that no | 1821 | than typed. This is the mechanism by which the mandate's requirement that no | ||
added
reports/critic/resolutions.csv
+73 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff