1problem_id,statement,categories,why_unresolved,testable_direction,supporting_keys2P-01,Whether LLM capabilities are discontinuous in scale (emergent) or artefacts of discontinuous metrics.,llm;ml_theory,Both positions are preprints with different metric choices; no corpus-level or peer-reviewed adjudication.,Re-evaluate the same model families under continuous and discontinuous metrics with pre-registered thresholds and report both.,wei2022emergent;schaeffer2023mirage3P-02,Whether pure-RL reasoning training (R1-style) generalises beyond verifiable-reward domains (mathematics and code).,llm;reinforcement_learning,The peer-reviewed evidence covers verifiable tasks; open-ended domains lack ground-truth rewards.,Out-of-domain evaluation of RL-only versus SFT+RL models on tasks without programmatic verifiers with held-out judges.,deepseek2025r1nature;openai2024o14P-03,Whether alignment brittleness to jailbreaks is intrinsic to RLHF-style methods or an artefact of scale and data.,ai_safety;llm,Attacks succeed in few queries across model families; no defence has withstood adaptive attackers in the landmark abstracts.,"Adaptive red-teaming with fixed attacker budgets across alignment methods (RLHF, DPO, constitutional) at matched scale.",zou2023universal;wei2023jailbroken;chao2025pair;casper2023rlhfproblems5P-04,How to measure hallucination in open-ended generation without reference answers.,llm;multimodal_ai,Surveys agree on definitions but not metrics; human evaluation frameworks are domain-specific.,Cross-domain benchmark with expert-labelled factuality and calibrated automated judges validated against it.,ji2023hallucination;huang2025hallucination;singhal2023clinical6P-05,Whether vision-language models fuse modalities or condition on the text channel.,multimodal_ai;foundation_models,A controlled medical evaluation found image predictions driven by text; general-domain evidence is benchmark-level only.,Counterfactual perturbation of each modality on held-out tasks measuring the marginal information used from images.,buckley2026textexploit;schouten2025multimodalmedicine;zhang2026lowqualityfusion7P-06,Whether 'agentic AI' denotes a distinct system class or a relabelling of tool-augmented LLM pipelines.,ai_agents,Six taxonomies in 2025–2026 and one peer-reviewed benchmark; definitions diverge.,Operational criteria (autonomy horizon; environment interaction; self-evaluation) applied to published systems with inter-rater agreement.,sapkota2026agentsvsagentic;acharya2025agenticsurvey;abouali2025agenticparadigms;wang2024agentsurvey8P-07,Whether autonomous science agents produce validated discoveries at a rate that justifies their cost.,ai_agents;autonomous_scientific_discovery,Peer-reviewed cases (Virtual Lab; Robin) are single demonstrations; AI-scientist preprints report no external validation.,Pre-registered multi-laboratory trials comparing agent-proposed and human-proposed hypotheses on validation rate and cost.,swanson2025virtuallab;ghareeb2026robin;lu2024aiscientist;si2024novelideas9P-08,Whether deep-learning architecture research plateaued after 2022 or was relabelled under LLM and foundation categories.,deep_learning_architectures;llm,DL architectures is the only AI category without a significant post-2022 slope change; multi-label counts cannot separate relabelling from decline.,Re-estimate the series on works whose only AI category is DL architectures and on a fixed 2022 classifier vocabulary.,gu2023mamba;liu2024kan;dao2022flashattention10P-09,How to evaluate models on benchmarks that may be in their training data.,llm;ml_theory,Contamination is documented; no landmark abstract reports a contamination-controlled evaluation.,Time-split benchmarks released after model training cut-offs with canary strings and reported overlap statistics.,sainz2023contamination;dehghani2021lottery;ott2022benchmarks;chiang2024arena11P-10,Whether catastrophic-risk capabilities can be measured before deployment.,ai_safety,Risk frameworks exist (repository; safety report); no landmark abstract reports a validated dangerous-capability measurement.,Pre-registered dangerous-capability evaluations with external replication and disclosure of false-negative rates.,slattery2026riskrepository;bengio2025safetyreport;shevlane2023extremerisks;hendrycks2023catastrophic12P-11,Whether domain foundation models generalise across institutions and acquisition protocols.,foundation_models;multimodal_ai,Pathology and radiology models report multi-benchmark gains but evaluation cohorts are often internal.,External multi-site validation with distribution-shift reporting and release of evaluation splits.,chen2024uni;lu2024conch;wu2025radfm;paschali2025fmradiology13P-12,Whether Transformers are the right inductive bias for long-horizon time series.,deep_learning_architectures;foundation_models,Linear baselines beat Transformer forecasters on standard benchmarks; LLM reprogramming claims gains on others.,"Common benchmark suite with leakage-free splits comparing linear, Transformer, state-space and LLM-reprogrammed models.",zeng2023transformerstsf;zhou2021informer;jin2023timellm;gu2023mamba14P-13,Whether generative-AI use harms or helps learning outcomes.,generative_ai,A preregistered field experiment reports harm to unassisted performance; surveys report perceived benefits.,Randomised trials with delayed unassisted assessments across subjects and tutoring designs.,bastani2024harmlearning;kasneci2023chatgptgood;zhai2024overreliance15P-14,Whether citation-based landmark identification can be made reliable for AI given identifier merging in bibliographic databases.,llm;deep_learning_architectures;ml_theory,"18.8 % of the automatically selected AI landmarks are mislinked, misdated or normalisation artefacts.","Re-run selection on records whose DOI, arXiv id and title are mutually consistent; document-type-specific normalisation; report the exclusion rate.",priem2022openalex;culbert2025openalexcoverage16