| 189 | |
189 | |
| 190 | The third is the maturation of \emph{adversarial machine learning}. The observation that imperceptible perturbations can change a network's output \citep{szegedy2014intriguing} and the attacks and evaluation standards that followed \citep{carlini2017robustness} created, by 2018, a field with ten years of history and a recognised research agenda \citep{biggio2018wild}. The ontology scores adversarial work in both cybersecurity and AI safety, which reflects the baseline state: the problem belongs to both communities and neither owns it. Blockchain systems, finally, entered the period with a systematised research agenda \citep{bonneau2015sok} and left it as a mature if contested area whose publication dynamics follow cryptocurrency markets more than academic calendars. |
190 | The third is the maturation of \emph{adversarial machine learning}. The observation that imperceptible perturbations can change a network's output \citep{szegedy2014intriguing} and the attacks and evaluation standards that followed \citep{carlini2017robustness} created, by 2018, a field with ten years of history and a recognised research agenda \citep{biggio2018wild}. The ontology scores adversarial work in both cybersecurity and AI safety, which reflects the baseline state: the problem belongs to both communities and neither owns it. Blockchain systems, finally, entered the period with a systematised research agenda \citep{bonneau2015sok} and left it as a mature if contested area whose publication dynamics follow cryptocurrency markets more than academic calendars. |
| 191 | |
191 | |
| |
192 | +\subsection{Privacy-preserving computation and federated learning} |
| |
193 | +\label{sec:p2-privacy} |
| |
194 | + |
| |
195 | +Differential privacy, defined by \citet{dwork2006calibrating} and systematised in the monograph of \citet{dwork2014algorithmic}, was by 2018 the accepted formal standard for statistical disclosure control, and its combination with stochastic gradient descent \citep{abadi2016dpsgd} had made private training of deep networks practical at modest cost in accuracy. Federated learning, proposed as a way to train models on data that never leaves the devices that generated it \citep{mcmahan2017fl}, grew in five years into a field with its own open problems in optimisation, systems, privacy and incentives, catalogued in a survey of unusual breadth \citep{kairouz2021fl}. Both categories are keyword-driven in the ontology because neither has a venue of its own; both were already AI-adjacent at the baseline, so that any post-2022 growth they show cannot be read as a new contact with AI. |
| |
196 | + |
| |
197 | +\subsection{Data systems, information retrieval and networks} |
| |
198 | +\label{sec:p2-data} |
| |
199 | + |
| |
200 | +Database research at the baseline was absorbing the proposal that learned models could replace the data structures at the core of the system: the learned index \citep{kraska2018learned} argued that a model of the cumulative distribution of keys can outperform a B-tree, and the argument was extended to query optimisation, cardinality estimation and tuning. In parallel, approximate nearest-neighbour search over dense vectors, long a niche concern, acquired the data structures and the GPU implementations that would later make vector databases an industry: hierarchical navigable small-world graphs \citep{malkov2020hnsw} and billion-scale similarity search on GPUs \citep{johnson2021faiss}. Information retrieval made the corresponding move from lexical to dense retrieval, in which queries and passages are embedded by a pretrained encoder and matched by inner product \citep{karpukhin2020dpr}; the retrieval-augmented generation of \citet{lewis2020rag} is the junction of that line with language modelling. Of all the non-AI areas, information retrieval is the one whose baseline most closely anticipates the post-2022 period, and its category should be read accordingly. |
| |
201 | + |
| |
202 | +Computer networks entered the period with programmability as its organising idea. OpenFlow had separated the control plane from the forwarding plane \citep{mckeown2008openflow}; the P4 language made the forwarding plane itself programmable \citep{bosshart2014p4}; and the QUIC transport, designed and deployed at internet scale by a single company before standardisation \citep{langley2017quic}, exemplified a mode of networking research in which deployment precedes publication. Edge computing, defined as the placement of computation near the sources of data \citep{shi2016edge}, grew with the internet of things and with the latency requirements of on-device inference; the ontology treats it as a child of distributed computing and shares its IoT topics with networks. |
| |
203 | + |
| |
204 | +\subsection{Human--computer interaction and computer graphics} |
| |
205 | +\label{sec:p2-hci} |
| |
206 | + |
| |
207 | +Human--computer interaction had, by 2019, a research programme on interaction with AI systems that predates the conversational interfaces of 2022. Guidelines for human--AI interaction were derived from two decades of literature and validated against deployed products \citep{amershi2019guidelines}; the particular difficulty of designing for systems whose capabilities are uncertain and whose outputs are open-ended was analysed by \citet{yang2020reexamining}; and the programme of human-centred AI, which places reliability, safety and trust above autonomy, was articulated by \citet{shneiderman2020hcai}. The HCI category is therefore one where a post-2022 surge, if observed, would be the intensification of an existing agenda rather than the arrival of a new one. Its venues (CHI, UIST, CSCW) are also among the most clearly delimited in DBLP, which makes the category's venue signal strong. |
| |
208 | + |
| |
209 | +Computer graphics underwent a transformation of its own in 2020, when neural radiance fields showed that a multilayer perceptron could represent a scene well enough to synthesise photorealistic novel views from a handful of photographs \citep{mildenhall2020nerf}. Neural rendering had been surveyed as an emerging area in the same year \citep{tewari2020neural}; within three years, explicit Gaussian primitives optimised by gradient descent had matched the quality of radiance fields at real-time frame rates \citep{kerbl2023gaussian}. The sequence is instructive for the detection methods of Part~IV: neural radiance fields are one of the pre-registered ``known emergence'' topics against which the emerging-topic detector is validated (H8), and they emerged in a non-AI category by the adoption of an AI method, which is exactly the pattern the detector must distinguish from vocabulary drift. |
| |
210 | + |
| |
211 | +\subsection{Quantum computing} |
| |
212 | +\label{sec:p2-quantum} |
| |
213 | + |
| |
214 | +Quantum computing at the baseline is defined by a term coined to describe it. \citet{preskill2018nisq} named the ``noisy intermediate-scale quantum'' era, in which devices of fifty to a few hundred qubits without error correction might perform useful tasks, and warned that no such task had yet been demonstrated. The claim of quantum computational advantage on a sampling problem \citep{arute2019supremacy} and the review of algorithms designed for such devices \citep{bharti2022nisq} bracket the period; the demonstration that logical error rates fall as a surface code is scaled \citep{google2023surface} and, later, that they fall below threshold \citep{google2025threshold} mark the transition toward fault tolerance that the NISQ framing anticipated. The quantum-computing category is small, keyword-driven, and vetoes physics papers that lack a computing term (Appendix~A); its dynamics are driven by hardware milestones and have no obvious mechanism of contact with language models, which makes it, with post-quantum cryptography, a natural placebo. |
| |
215 | + |
| |
216 | +\subsection{Scientific computing, computational biology and sustainable computing} |
| |
217 | +\label{sec:p2-science} |
| |
218 | + |
| |
219 | +Scientific computing entered the period through the door of physics-informed learning. Physics-informed neural networks, which impose a partial differential equation as a loss term \citep{raissi2019pinn}, and neural operators, which learn mappings between function spaces rather than between vectors \citep{li2021fno}, created a literature that the review of \citet{karniadakis2021piml} already presented as a field. Computational biology had its own transformation in the baseline window and does not need the post-2022 one to explain its trajectory: the first AlphaFold had won the 2018 structure-prediction assessment \citep{senior2020alphafold}, the second effectively settled the single-chain problem in 2020--2021 \citep{jumper2021alphafold}, protein language models predicted structure from sequence alone at scale \citep{lin2023esm}, and the third AlphaFold extended prediction to complexes with nucleic acids and ligands \citep{abramson2024af3}. The review of \citet{wang2023aiscience} in 2023 took stock of ``AI for science'' across disciplines; the autonomous-scientific-discovery leaf of the ontology, which requires an AI or automation term, is the place where the corpus records the next step, in which models participate in the design of experiments rather than in the analysis of their results. |
| |
220 | + |
| |
221 | +Sustainable computing, finally, was an established concern before 2022 but a small category. The careful recalibration of global data-centre energy use \citep{masanet2020datacenter} found that efficiency gains had largely offset growth in demand through 2018, a conclusion that the training and serving of large models would later put under pressure; the energy and carbon studies cited in Section~\ref{sec:p2-transformer} \citep{strubell2019energy,schwartz2020green,patterson2021carbon,luccioni2023bloom} connected the concern directly to AI. The category is keyword-driven and is expected to grow; whether its growth is driven by AI workloads or by the broader energy agenda is a question for Part~IX. |
| |
222 | + |
| |
223 | +Taken together, the baseline descriptions support a conclusion that the empirical parts must not forget: there was no AI-free area in 2022. Every non-AI category had a point of contact with machine learning before ChatGPT, ranging from marginal (complexity theory, post-quantum cryptography) to pervasive (information retrieval, software engineering, computational biology). The post-2022 question is therefore not whether AI \emph{entered} these areas but whether the rate and the kind of contact changed, and the baseline window exists so that the rate can be estimated before the reference date. |
| |
224 | + |
| |
225 | +\section{November 2022 as a reference date} |
| |
226 | +\label{sec:p2-nov2022} |
| |
227 | + |
| |
228 | +\subsection{What ChatGPT was and was not} |
| |
229 | +\label{sec:p2-chatgpt} |
| |
230 | + |
| |
231 | +On 30 November 2022 OpenAI released ChatGPT as a ``research preview'': a conversational interface to a language model of the GPT-3.5 series, fine-tuned by the reinforcement-learning-from-human-feedback method of InstructGPT, free to use through a web browser \citep{openai2022chatgpt}. No technical report accompanied the release; the announcement described the training procedure by reference to \citet{ouyang2022instructgpt} and listed limitations (plausible but incorrect answers, sensitivity to phrasing, verbosity, incomplete refusal of harmful requests) that the subsequent literature would study at length. It is important for the design of this paper to be precise about what was new. The underlying capabilities, few-shot prompting, instruction following, code generation, had been published between 2020 and 2022 \citep{brown2020gpt3,chen2021codex,ouyang2022instructgpt} and were available through a commercial programming interface. The architecture was five years old. What ChatGPT added was an interface that required no programming, a price of zero, and a conversational format that made the model's abilities and failures legible to anyone; its diffusion among the public was by all accounts faster than that of any previous software product, though the figures usually cited for it come from industry analysts rather than from verifiable sources and are not repeated here. |
| |
232 | + |
| |
233 | +For a study of the \emph{research} literature, this distinction has a consequence that is easily missed. The research community did not learn about large language models on 30 November 2022; it had been building and publishing them for three years. What changed on that date, if anything, was the attention of everyone else: funders, universities, firms, governments, and the researchers in subfields that had not previously had a reason to engage. If the date marks anything in the corpus, it should mark a change in \emph{who} publishes on language models and \emph{where}, more than a change in what the language-model community itself does. That expectation is formalised in H4 and H6 and in the reading of the cross-area citation flows in Part~VIII. |
| |
234 | + |
| |
235 | +\subsection{The releases before and after} |
| |
236 | +\label{sec:p2-timeline} |
| |
237 | + |
| |
238 | +ChatGPT was one release in a dense sequence, and the sequence matters because any of its members is a candidate reference date. Table~\ref{tab:p2-timeline} lists the ones whose primary documents could be verified. Within the twelve months before November 2022 came the compute-optimal scaling result \citep{hoffmann2022chinchilla}, the PaLM, OPT and BLOOM models \citep{chowdhery2022palm,zhang2022opt,bigscience2022bloom}, the chain-of-thought and emergent-abilities papers \citep{wei2022cot,wei2022emergent}, the second DALL-E and Imagen \citep{ramesh2022dalle2,saharia2022imagen}, the open release of Stable Diffusion \citep{rombach2022ldm}, the Flamingo and Gato systems \citep{alayrac2022flamingo,reed2022gato}, the Whisper speech model \citep{radford2023whisper}, and the ReAct formulation of language models that interleave reasoning with actions \citep{yao2023react}, posted in October 2022 and the direct ancestor of the agent literature. Within the twelve months after came LLaMA, the first competitive open-weight model family \citep{touvron2023llama}, Toolformer \citep{schick2023toolformer}, the GPT-4 technical report \citep{openai2023gpt4}, an early and much-debated qualitative evaluation of GPT-4 \citep{bubeck2023sparks}, Llama~2 with a permissive licence \citep{touvron2023llama2}, the RT-2 vision-language-action model that transferred web-scale pretraining to robot control \citep{brohan2023rt2}, Gaussian splatting in graphics \citep{kerbl2023gaussian}, the Mamba state-space architecture as the first serious challenger to attention \citep{gu2023mamba}, and the Gemini models \citep{gemini2023}. Later milestones, Llama~3 \citep{grattafiori2024llama3}, AlphaFold~3 \citep{abramson2024af3}, the o1 reasoning models \citep{openai2024o1} and DeepSeek-R1, which reported that reasoning behaviour could be elicited by reinforcement learning with verifiable rewards and released its weights \citep{deepseek2025r1}, fall inside the observation window and are candidate dates for the \emph{later} breaks that the changepoint methods of Part~IV may find. |
| |
239 | + |
| |
240 | +\begin{table}[htbp] |
| |
241 | +\centering |
| |
242 | +\caption{Candidate reference dates around November 2022. ``Document'' is the verified primary source; dates are those of the first public version of the document or of the release it describes, as given in the source. The list is restricted to items cited in this Part and is not a complete chronology.} |
| |
243 | +\label{tab:p2-timeline} |
| |
244 | +\small |
| |
245 | +\begin{tabularx}{\textwidth}{@{}llX@{}} |
| |
246 | +\toprule |
| |
247 | +Date & Release or result & Document \\ |
| |
248 | +\midrule |
| |
249 | +2022-03 & Compute-optimal scaling (Chinchilla) & \citet{hoffmann2022chinchilla}, preprint \\ |
| |
250 | +2022-04 & PaLM; DALL-E 2; Flamingo & \citet{chowdhery2022palm,ramesh2022dalle2,alayrac2022flamingo}, preprints \\ |
| |
251 | +2022-05 & OPT; Gato; Imagen & \citet{zhang2022opt,reed2022gato,saharia2022imagen}, preprints \\ |
| |
252 | +2022-06 & Latent diffusion published (CVPR); Stable Diffusion released openly in August & \citet{rombach2022ldm} \\ |
| |
253 | +2022-10 & ReAct & \citet{yao2023react}, preprint \\ |
| |
254 | +2022-11 & BLOOM; ChatGPT (30 November) & \citet{bigscience2022bloom}, preprint; \citet{openai2022chatgpt}, web page \\ |
| |
255 | +2022-12 & Whisper & \citet{radford2023whisper}, preprint \\ |
| |
256 | +2023-02 & LLaMA; Toolformer & \citet{touvron2023llama,schick2023toolformer}, preprints \\ |
| |
257 | +2023-03 & GPT-4 & \citet{openai2023gpt4}, technical report (preprint) \\ |
| |
258 | +2023-07 & Llama 2; RT-2 & \citet{touvron2023llama2,brohan2023rt2}, preprints \\ |
| |
259 | +2023-08 & 3D Gaussian splatting (SIGGRAPH) & \citet{kerbl2023gaussian} \\ |
| |
260 | +2023-12 & Mamba; Gemini & \citet{gu2023mamba,gemini2023}, preprints \\ |
| |
261 | +2024-05 & AlphaFold 3 & \citet{abramson2024af3} \\ |
| |
262 | +2024-07 & Llama 3 & \citet{grattafiori2024llama3}, preprint \\ |
| |
263 | +2024-12 & o1 system card & \citet{openai2024o1}, preprint \\ |
| |
264 | +2025-01 & DeepSeek-R1 & \citet{deepseek2025r1}, preprint \\ |
| |
265 | +\bottomrule |
| |
266 | +\end{tabularx} |
| |
267 | +\end{table} |
| |
268 | + |
| |
269 | +Two features of the table are relevant to identification. The first is that the great majority of the documents are preprints or technical reports; none of the model releases of 2022--2025 was described in a peer-reviewed venue at the time of release, and several never were. A corpus that excluded preprints would be blind to the events it is trying to date. The second is that the events are not sparse. Between March 2022 and March 2023 there was, on average, a landmark release every month, and a time-series method that is free to choose the break date will find a break somewhere in that interval for reasons that have nothing to do with ChatGPT in particular. That is the subject of the next subsection. |
| |
270 | + |
| |
271 | +\subsection{Why the date is used descriptively and not causally} |
| |
272 | +\label{sec:p2-causal} |
| |
273 | + |
| |
274 | +The mandate of this study (CLAUDE.md~\S6.5) states that the November 2022 boundary is a historical reference point and not proof of a causal effect of ChatGPT. The statement is a constraint on interpretation, but it is also a methodological position that can be made precise. |
| |
275 | + |
| |
276 | +Let $N_{c,t}$ denote the number of works assigned to category $c$ in month $t$, and consider the simplest model that the ``post-ChatGPT'' framing suggests, an interrupted time series of the form used in programme evaluation \citep{bernal2017its}: |
| |
277 | +\begin{equation} |
| |
278 | +\log N_{c,t} \;=\; \alpha_c + \beta_c\, t + \delta_c\, \mathbf{1}[t \geq \tau] + \gamma_c\,(t-\tau)\,\mathbf{1}[t \geq \tau] + \mathbf{s}_c(t) + \varepsilon_{c,t}, |
| |
279 | +\label{eq:p2-its} |
| |
280 | +\end{equation} |
| |
281 | +with $\tau$ fixed at December 2022, $\mathbf{s}_c(t)$ a seasonal component tied to the conference calendar, $\delta_c$ a level shift and $\gamma_c$ a change in trend. Three conditions must hold for $\delta_c$ or $\gamma_c$ to be read as the effect of the event at $\tau$: the pre-period trend must be well specified, so that an acceleration already under way is not attributed to the break; nothing else that affects $N_{c,t}$ may change at $\tau$; and the date $\tau$ must have been chosen independently of the data. The first condition is testable and is tested in Part~IX with pre-period trend diagnostics. The second fails on its face: Table~\ref{tab:p2-timeline} lists a dozen co-occurring events, the compute trends of \citet{sevilla2022compute} were already exponential, and the growth of the arXiv itself, documented month by month on its statistics page \citep{arxiv2026stats}, is a confounder that applies to every category. The third condition is the subtle one. The date was chosen because of what happened afterwards, which is the definition of selecting a break on the outcome; a test of a known break at $\tau$ \citep{bernal2017its} has the wrong size when $\tau$ was in fact selected from a set of candidates, and the appropriate tools are those for a break at an \emph{unknown} date, the supremum-type tests of \citet{andrews1993tests}, the multiple-break estimation of \citet{bai1998breaks,bai2003computation}, and penalised changepoint search \citep{killick2012pelt}, which let the data locate the break and report whether its location is consistent with the reference date. Equation~\eqref{eq:p2-its} is therefore the \emph{descriptive} object of the study, a convenient summary of before and after; the inferential objects are the unknown-break estimates, their confidence sets, and the placebo breaks at other dates in the baseline, as specified in Part~IV. |
| |
282 | + |
| |
283 | +The descriptive use of the date has a positive justification that goes beyond the avoidance of a fallacy. Dividing the window at a date that the whole community recognises makes the comparison between periods legible, and legibility is a scientific virtue when the quantities compared are themselves complicated. The baseline of 2018--2022 is long enough to contain the whole transformer era and the second half of the deep-learning wave, so that the ``before'' is not an AI-free past but a period of fast change in its own right; the research period of 2022--2026 is long enough for publication lags, which in computer science run from a few weeks for a preprint to more than a year for a journal, to have worked through for all but the last months. Any claim of the form ``after November 2022, $X$ increased'' in this manuscript is to be read as a comparison of two periods so defined, with the structural-break analysis as a separate and more demanding question. |
| |
284 | + |
| |
285 | +One further confounder is specific to the present study and is treated in the next section. If language models changed how papers are \emph{written}, then every text-based measure, from keyword-driven classification to embedding-based topic detection, changed its meaning at approximately the reference date, and a break in such a measure may reflect a change in prose rather than in research. |
| |
286 | + |
| |
287 | +\section{What the literature already claims about the post-2022 period} |
| |
288 | +\label{sec:p2-claims} |
| |
289 | + |
| |
290 | +A substantial literature has formed since 2023 on the consequences of large language models for science and for computer science. Its findings are the prior against which the corpus-level results of Part~IX will be compared, and its limitations are the reason the corpus-level test is needed. Five claims recur. |
| |
291 | + |
| |
292 | +\subsection{The growth of AI research} |
| |
293 | +\label{sec:p2-claim-growth} |
| |
294 | + |
| |
295 | +That AI research is growing faster than other fields was established before ChatGPT. \citet{krenn2023forecasting} built a semantic network from the AI and machine-learning literature of the preceding three decades, found that the number of papers in it had been growing exponentially with a doubling time of about two years, and used the network to forecast which concept pairs would be studied next. The Stanford AI Index reports, which compile counts from a range of sources, report continued growth in AI publications and in the number of notable models, with the share of such models produced by industry rising to a large majority by 2023 and 2024 \citep{maslej2024aiindex,maslej2025aiindex}; the reports are compilations rather than primary studies, and their source definitions change from edition to edition, so their numbers are not comparable with those of this study and are cited here for the direction of the claim only. The arXiv's own monthly statistics show that computer-science submissions overtook those of every other archive during the 2010s and continued to accelerate after 2022 \citep{arxiv2026stats}. Industry's growing weight in the field, documented by \citet{ahmed2023industry} for the period up to 2022, is the component of this claim that the present study can test most directly, through the affiliation data of the corpus (H6). |
| |
296 | + |
| |
297 | +What this literature does not establish is whether the growth after 2022 differs from the growth before it, either in rate or in kind, because the studies either end in 2022 or do not model the pre-period trend. H1 is the corpus-level version of the question. |
| |
298 | + |
| |
299 | +\subsection{LLM-assisted writing} |
| |
300 | +\label{sec:p2-claim-writing} |
| |
301 | + |
| |
302 | +The most direct evidence of a change in the literature after 2022 is a change in its language. \citet{liang2024mapping} estimated, with a distributional method that measures the frequency of words whose use the models favour, that the fraction of abstracts in computer science with substantial LLM-modified content rose rapidly from the end of 2022, with computer science leading other fields; the same group found a comparable rise in the reviews of major machine-learning conferences \citep{liang2024monitoring}, and later extended the method to writing across society \citep{liang2025widespread}. \citet{kobak2025delving} obtained the same conclusion for biomedical abstracts by a different route, measuring the excess frequency of words such as ``delves'' and ``underscores'' relative to a pre-2023 baseline, in a study designed to need no training data from the models themselves; earlier and smaller analyses of the same marker words had reached the same result \citep{geng2024writing,gray2024contamination}. Surveys of researchers confirm the adoption that the text analyses infer: a Nature survey of 1{,}600 scientists found wide use of generative tools for writing and coding, with strong concerns about reliability and misuse \citep{vannoorden2023survey}, and the multidisciplinary collection assembled by \citet{dwivedi2023sowhat} and the integrity analysis of \citet{binnashwan2023chatgpt} document the reactions of the research and education communities in the first months. |
| |
303 | + |
| |
304 | +For this study the finding is a measurement problem before it is a result. The classification of Section~\ref{sec:p2-ontology} and the embedding methods of Part~IV both read text, and text that is partly produced by a model trained on the pre-2023 literature is, in a precise sense, less informative about what is new in a paper. Two responses are built into the design. The venue and arXiv signals of equation~\eqref{eq:p2-score} are immune to prose; and the robustness checks of Appendix~D re-estimate the main breaks with classifications that use no keyword signals, so that a break present in the keyword-based series and absent in the venue-based one is flagged as possibly linguistic. |
| |
305 | + |
| |
306 | +\subsection{Illusions of understanding and the epistemology of AI in science} |
| |
307 | +\label{sec:p2-claim-epistemic} |
| |
308 | + |
| |
309 | +A second body of work concerns not the quantity of research but its quality. \citet{messeri2024illusions} argued that the uses of AI in science that researchers find most attractive, as oracle, surrogate, quant and arbiter, each create an illusion of understanding, and that the resulting monoculture of methods and of questions is a risk to the scientific enterprise rather than merely to individual studies. \citet{birhane2023science} raised the related concern that language models, used as instruments, embed the biases and the errors of their training corpora into the record. These arguments are theoretical and, so far, untested against the publication record; the knowledge-graph analyses of Part~VIII, which track methods and datasets across papers, are the closest this study comes to a test of the monoculture claim. |
| |
310 | + |
| |
311 | +\subsection{Benchmark saturation and the reproducibility of evaluation} |
| |
312 | +\label{sec:p2-claim-benchmarks} |
| |
313 | + |
| |
314 | +The benchmark culture of Section~\ref{sec:p2-dl} produced a literature on its own dynamics. \citet{ott2022benchmarks} mapped the creation and saturation of benchmarks across AI tasks and found that many become saturated within a few years of their creation while others are abandoned before saturation; the critiques of \citet{dehghani2021lottery} and \citet{raji2021benchmark} argued that the choice of benchmark is itself an under-examined research decision and that no benchmark can measure general capability. The post-2022 literature extended the concern to \emph{contamination}, the presence of test data in the training corpora of models trained on the web, which makes a benchmark result uninterpretable without knowledge of the training data that the model reports of Table~\ref{tab:p2-timeline} typically withhold. \citet{kapoor2023leakage} documented data leakage as a general cause of irreproducible results in machine-learning-based science, and the agenda of \citet{hendrycks2021unsolved} placed robustness, monitoring and alignment among the open problems of the field. RQ7, which asks whether the period's advances are breakthroughs or incremental benchmark gains, and RQ9, on reproducibility and evaluation standards, are the corpus-level counterparts of these claims. |
| |
315 | + |
| |
316 | +\subsection{Concentration, disruption and the pace of science} |
| |
317 | +\label{sec:p2-claim-pace} |
| |
318 | + |
| |
319 | +The last claim is the broadest. \citet{park2023disruptive} found, with a citation-based disruption index applied to papers and patents over six decades, that work has become less disruptive and more consolidating on average across fields; \citet{chu2021slowed} found that in large and fast-growing fields, canonical papers become entrenched, new ideas are less likely to displace them, and progress in the sense of turnover at the top slows. Both results predict that a field growing as fast as AI after 2022 should show rising concentration of citations on a few canonical works and a lower rate of disruption, independent of any product release. Hypothesis H5, on citation inequality, is formulated with that prediction in mind, and the difference between the post-2022 period and the baseline is the quantity of interest, not the level. |
| |
320 | + |
| |
321 | +\subsection{Why the claims need a corpus-level test} |
| |
322 | +\label{sec:p2-why} |
| |
323 | + |
| |
324 | +The studies reviewed above share four limitations that the design of this paper is intended to remove. They are, almost without exception, studies of AI alone, so that they cannot say whether the growth they document came at the expense of, or in addition to, the rest of computer science; the present corpus covers all forty categories with the same instrument. They rely on a single source (arXiv, a conference's reviews, a database of abstracts), so that coverage effects of the kind described in Section~\ref{sec:p2-disagree} cannot be separated from research effects; the present corpus links four sources and reports results by source. They use vocabulary as evidence, which, as Section~\ref{sec:p2-claim-writing} shows, is the quantity most directly altered by the models; the present design includes classifications that use no vocabulary. And they date the change to the release of ChatGPT without testing the alternative dates of Table~\ref{tab:p2-timeline} or the placebo dates of the baseline; the present design does both. |
| |
325 | + |
| |
326 | +None of this guarantees that the corpus-level test will overturn the claims; it may confirm them. Its contribution is to subject them, for the first time at the scale of the whole discipline and with a preregistered protocol, to the conditions under which a claim of structural change can be distinguished from the ordinary growth of a large field. Part~III describes the corpus on which that test is run. |
| |
327 | + |