| 98 |
98 |
|
| 99 |
99 |
The propagation pattern of this wave was \emph{methodological} and sequential. Deep learning entered computer vision and speech first, natural language processing second (roughly 2014--2016, with the sequence-to-sequence results), reinforcement learning and robotics third, and then, more slowly and with more resistance, the subfields whose objects are not perceptual: program analysis \citep{allamanis2018survey}, systems (learned index structures and learned scheduling), databases, and scientific computing \citep{raissi2019pinn}. In each case the receiving subfield first treated learning as a tool for a pre-existing problem, and only later, if at all, reorganised its problems around it. That two-stage pattern, tool adoption followed by problem redefinition, is the template against which Part~IX reads the post-2022 movement of methods across the forty categories. |
| 100 |
100 |
|
|
101 |
+\subsection{The transformer and pretraining era, 2017--2022} |
|
102 |
+\label{sec:p2-transformer} |
|
103 |
+ |
|
104 |
+The third transformation began with an architecture and ended with a change in what a model is for. The Transformer of \citet{vaswani2017attention} replaced recurrence with self-attention in the encoder--decoder framework of neural machine translation; its immediate merit was parallelism in training, and its lasting one was that the same block could be stacked to any depth and applied to any sequence of tokens. Within eighteen months it had been combined with the idea, already present in ELMo's contextual word vectors \citep{peters2018elmo}, that a language model trained on unlabelled text could be \emph{pretrained} once and \emph{fine-tuned} on many tasks: BERT \citep{devlin2019bert}, with its masked-language objective, set new results on every task of the GLUE benchmark \citep{wang2018glue} and on SQuAD \citep{rajpurkar2016squad}, and its successors showed that most of its design choices were less important than the amount of data and training \citep{liu2019roberta,raffel2020t5}. The generative branch of the family, the GPT series of decoder-only models, reached its third member in 2020 with 175 billion parameters and the demonstration that a sufficiently large language model could perform new tasks from a few examples placed in its prompt, without any gradient update \citep{brown2020gpt3}. In-context learning, more than any architectural novelty, is what separates the transformer era from the deep-learning wave: the unit of research shifted from the task-specific model to the general model and the prompt. |
|
105 |
+ |
|
106 |
+The era had a quantitative theory of its own growth. \citet{kaplan2020scaling} reported that the test loss of autoregressive language models falls as a power law in parameters, data and compute over several orders of magnitude, with the implication that larger models are more sample-efficient; \citet{hoffmann2022chinchilla} corrected the allocation, showing that models of the GPT-3 generation were undertrained relative to their size and that compute-optimal training requires data and parameters to scale together. Between the two papers, laboratories built models of increasing size on the strength of the first law (Gopher, PaLM, OPT and the open multilingual BLOOM \citep{rae2021gopher,chowdhery2022palm,zhang2022opt,bigscience2022bloom}), and the systems research needed to train them, model parallelism, memory-partitioned optimisers and IO-aware attention kernels \citep{shoeybi2019megatron,rajbhandari2020zero,dao2022flashattention}, became a research area in its own right, as did sparse mixtures of experts that decouple parameter count from per-token compute \citep{fedus2022switch}. The claim that certain abilities appear discontinuously with scale \citep{wei2022emergent} became one of the era's most cited and most contested propositions; the companion observation that prompting a model to write out intermediate reasoning improves its answers on arithmetic and symbolic tasks \citep{wei2022cot} established prompting as a method. New benchmarks were built at the scale of the models: a fifty-seven-subject examination of knowledge \citep{hendrycks2021mmlu}, a harder successor to GLUE \citep{wang2019superglue}, and a collaborative suite of more than two hundred tasks explicitly designed to resist saturation \citep{srivastava2023bigbench}. |
|
107 |
+ |
|
108 |
+The transformer did not remain a language architecture. Vision transformers showed that image patches could be treated as tokens and that, with enough data, the inductive bias of convolution was dispensable \citep{dosovitskiy2021vit}; self-supervised objectives, contrastive \citep{chen2020simclr}, self-distilled \citep{caron2021dino} and masked \citep{he2022mae}, removed the dependence on labels. Contrastive language--image pretraining aligned the two modalities in a shared embedding space and produced zero-shot classifiers from natural-language prompts \citep{radford2021clip}; in the same months, autoregressive text-to-image generation \citep{ramesh2021dalle} and then diffusion models, a decade-old idea \citep{sohl2015diffusion} made practical by denoising objectives \citep{ho2020ddpm}, score-based formulations \citep{song2021score} and architectural refinements that overtook adversarial models \citep{dhariwal2021beats}, converged on text-conditioned image synthesis at photographic quality \citep{ramesh2022dalle2,saharia2022imagen}. The latent-diffusion formulation \citep{rombach2022ldm}, released with open weights in the summer of 2022 under the name Stable Diffusion, is the generative-AI counterpart of ChatGPT: a system whose public availability, more than its technical novelty, changed who could use it. Speech followed the self-supervised path \citep{baevski2020wav2vec}; few-shot visual language models \citep{alayrac2022flamingo} and a single network trained across text, images, games and robot control \citep{reed2022gato} prefigured the multimodal and agentic systems of the following period. In structural biology, a transformer-based system solved, for most single-chain proteins, a problem that had resisted fifty years of effort \citep{jumper2021alphafold}, and was quickly joined by an independent three-track network \citep{baek2021rosettafold}. |
|
109 |
+ |
|
110 |
+Two further lines of work prepared the specific object that appeared in November 2022. Models trained on public code could synthesise functions from docstrings \citep{chen2021codex}, which turned software engineering into an early adopter of large models and created the first commercial product built on one. And the problem of making a language model do what a user asks, rather than what the pretraining distribution predicts, was addressed by reinforcement learning from human preferences, a method developed for control tasks \citep{christiano2017rlhf}, applied to summarisation \citep{stiennon2020summarize}, and then used to fine-tune GPT-3 into InstructGPT \citep{ouyang2022instructgpt}, whose 1.3-billion-parameter version was preferred by raters to the 175-billion-parameter base model. The same method, framed around helpfulness and harmlessness, was being developed in parallel elsewhere \citep{bai2022hh}. Retrieval-augmented generation, which conditions a generator on documents retrieved from an external index, was proposed in the same period \citep{lewis2020rag} and would become the standard answer to the models' factual unreliability. |
|
111 |
+ |
|
112 |
+The era also produced its own critique. The report that coined the term ``foundation model'' argued that the homogenisation of research around a few pretrained models was a change in the structure of the field, with risks that were social and institutional as much as technical \citep{bommasani2021foundation}; the ``stochastic parrots'' paper questioned whether ever-larger language models were a sound research direction at all, on environmental, documentary and epistemic grounds \citep{bender2021parrots}. The environmental argument had been quantified for NLP models by \citet{strubell2019energy}, contested and refined for the largest models by \citet{patterson2021carbon}, and turned into a research agenda by \citet{schwartz2020green}; the carbon footprint of a 176-billion-parameter model was measured end to end by \citet{luccioni2023bloom}. |
|
113 |
+ |
|
114 |
+What changed in publication practice during this era can be summarised in three observations. The leading models were described in \emph{technical reports} rather than in peer-reviewed papers, often with hundreds of authors, sometimes without the training data, the code, or the weights; the preprint became the version of record for the most consequential work in the discipline. The infrastructure for sharing models became an institution in its own right: the Transformers library \citep{wolf2020transformers} and the model hub built around it made the pretrained model, not the paper, the thing that propagated. And the compute required to participate at the frontier grew beyond the reach of universities, so that the concentration documented by \citet{ahmed2023industry} is, in this era, a concentration of capability rather than merely of attention. |
|
115 |
+ |
|
116 |
+\subsection{A comparative reading of the three waves} |
|
117 |
+\label{sec:p2-compare} |
|
118 |
+ |
|
119 |
+Table~\ref{tab:p2-waves} summarises the three transformations along the dimensions that matter for the empirical design of this study. The comparison is qualitative; the corpus-level counterpart is the subject of Part~IX. |
|
120 |
+ |
|
121 |
+\begin{table}[htbp] |
|
122 |
+\centering |
|
123 |
+\caption{Three prior transformations of computer science compared on the dimensions relevant to the post-2022 analysis. Qualitative characterisation by the authors from the works cited in Sections~\ref{sec:p2-web}--\ref{sec:p2-transformer}; no corpus quantities.} |
|
124 |
+\label{tab:p2-waves} |
|
125 |
+\small |
|
126 |
+\begin{tabularx}{\textwidth}{@{}lXXX@{}} |
|
127 |
+\toprule |
|
128 |
+ & Internet and web (c.~1994--2010) & Deep learning (2012--2017) & Transformer and pretraining (2017--2022) \\ |
|
129 |
+\midrule |
|
130 |
+Originating subfields & Networks, information retrieval, distributed systems & Computer vision, speech & Natural language processing \\ |
|
131 |
+Core artefact & Protocols, search engines, warehouse-scale systems & Trained task-specific networks; frameworks & Pretrained general models; prompts; weights \\ |
|
132 |
+Locus of production & Industry and start-ups; academic systems groups & Academic groups with industry laboratories entering & Industry laboratories, with academic participation declining at the frontier \\ |
|
133 |
+Publication practice & Selective conferences; deployed-artefact papers; open-source re-implementations & arXiv-first; leaderboards; shared frameworks; reproducibility programmes & Technical reports; model hubs; very large author lists; preprint as version of record \\ |
|
134 |
+Propagation pattern & Infrastructural: new objects and subfields, methods of other areas unchanged & Methodological and sequential: tool adoption then problem redefinition, perceptual fields first & Substitutive: one architecture and one pretraining recipe across modalities; general models replace task-specific ones \\ |
|
135 |
+Visibility in classifications & Accommodated by existing top-level nodes & New venues and arXiv classes gain weight; topics split & Vocabulary outruns citation clusters; keyword-defined categories required \\ |
|
136 |
+\bottomrule |
|
137 |
+\end{tabularx} |
|
138 |
+\end{table} |
|
139 |
+ |
|
140 |
+Three conclusions follow from the comparison and shape the hypotheses tested later. First, the mechanisms of propagation differ in kind. The web changed the problems of other subfields; deep learning changed their methods; the transformer era changed the \emph{models} they use, which can be adopted without changing either problems or methods. A subfield can therefore register the third transformation at low cost, by using a pretrained model as a component, and the resulting growth in the vocabulary of the subfield need not indicate a reorganisation of its research. Distinguishing component adoption from reorganisation is the purpose of the citation-flow and embedding analyses of Parts~IV and~VIII. |
|
141 |
+ |
|
142 |
+Second, each wave shortened the publication cycle and shifted the version of record toward the preprint. The consequence for this study is that the arXiv harvest and the OpenAlex index measure different things: the former records when a result was first made public, the latter when it entered the citation system. A break detected in one series and not in the other is informative about the mechanism, not merely about measurement. |
|
143 |
+ |
|
144 |
+Third, the three waves were cumulative in infrastructure and in institutions. The industrial laboratories that produced the web systems trained the generation that produced deep learning, and the frameworks and compute of the deep-learning wave made the transformer era possible. The post-2022 period inherited all of it: the laboratories, the frameworks, the model hubs, the benchmark culture, and the preprint norm. Whatever is new about the period must be identified against that inheritance rather than attributed to a single product release, and the baseline window of 2018--2022 was chosen so that the inheritance is observed directly rather than assumed. |
|
145 |
+ |