One-Sentence Takeaway

The panel expects continued AI progress to make automated research and broad computer-based work plausible within years, but says rapid self-improvement still depends on overcoming weak objective-setting, imperfect real-world generalization, scarce frontier-quality feedback, and unstable continual learning.

Executive Brief

This podcast panel brings together Beren Millidge of Zyphra, John Schulman of Thinking Machines, Charlie O’Neill of Baseten, and the host to debate the technical path from current language models to automated AI research and superintelligence.

  • A technically “normal” 2036 would most likely mean that transformer-plus-RL systems asymptoted before surpassing humans at research, perhaps because they remained weak at judgment, self-checking, continual learning, or transferring simulated training to open-ended reality.
  • The apparent smoothness of AI progress conceals successive innovations—pre-training, scaling laws, reinforcement learning, and better environments—and continued progress may require another discontinuity that current systems cannot themselves discover.
  • AI can already accelerate research when the objective is explicit and measurable, but open-ended science requires choosing useful questions and objectives, which the panel treats as a harder and less automatable task.
  • Labs are betting that reinforcement learning across many simulated environments will produce persistent, long-horizon agents, while deployment data will increasingly be folded into later model generations; whether this becomes seamless live learning remains unresolved.
  • Distillation pushes against model-provider concentration because behaviors can often be copied, but successful copying depends heavily on realistic prompts and interactions, not merely benchmark tasks or access to a teacher model.
  • The panel attributes much of reinforcement learning’s recent success to strong synthetic mid-training, high-signal reward feedback, broader environment coverage, and generalization to longer task horizons rather than universal reasoning transfer.
  • Forecasts were aggressive but divergent: some form of general remote worker was placed roughly one to three years away, a tenfold uplift to AI researchers anywhere from about two to ten years away, and dominance over human experts in all computer-based fields roughly three to ten years away.

Core Argument

Central thesis or narrative: Current methods may be close enough to human-level AI research to trigger rapid capability growth, but that outcome is not guaranteed by scaling alone. The decisive questions are whether models can choose and revise their own objectives, generalize from tractable simulations to messy reality, learn continually without degradation, and obtain genuinely new signal once existing human data is exhausted.

Reasoning, evidence & mechanisms

  • Human bottlenecks persist after impressive benchmark gains. New models repeatedly appear transformative, then reveal poor judgment, weak self-verification, limited autonomy, or brittle behavior in prolonged use. Writing far more code does not translate directly into proportionate research productivity because experiment choice, interpretation, coordination, and quality control remain limiting.
  • Progress has depended on hidden discontinuities. The panel compares AI scaling with Moore’s law: a smooth aggregate trend was sustained by many discrete inventions. Pre-training began to show diminishing returns, then post-training and RL extended progress; another conceptual advance may be needed, and present models might not discover it if it lies too far outside the current deep-learning paradigm.
  • Well-specified optimization differs from open-ended discovery. AI can search for a lower training loss, improve a speed-run benchmark, identify scaling-law mistakes, or optimize a measurable environment. It is less clear that it can originate the right research objective, recognize a promising but initially unmeasurable paradigm, or maintain a long chain of proposing, optimizing, and replacing objectives without drifting off course.
  • Objective specification may be the longest-lasting human role. Schulman argues that even if technical optimization is automated, people must still decide what assistants should do, what “helpful” means, and how constitutions or model specifications should encode desired behavior. He separates alignment into choosing the objective and successfully optimizing it, expecting the former to remain human-led longer.
  • Automated AI research will probably be trained incrementally. Rather than restarting from first principles and rediscovering neural networks, labs are likely to preserve the existing technical lineage, convert recent bugs and research improvements into environments, collect researcher feedback, and train models on multi-step projects. Environments with objective scores can eventually push beyond human demonstrations, but vague research still depends on intuition and imperfect judges.
  • Simulation is necessary now but may not be sufficient indefinitely. Low sample efficiency makes it impractical to train through thousands of live human interactions, so labs simulate coding, finance, office work, and other domains. Yet running a business, managing organizational relationships, litigating a case, or performing other long-horizon work involves shifting social conditions that may not transfer cleanly from data-center environments.
  • Continual learning works better at large batch scale than through repeated micro-updates. Deployment traces can improve subsequent model generations, and specialized firms can create environments from user feedback. But repeatedly fine-tuning one model on narrow, non-stationary data can cause catastrophic forgetting and degrade general abilities; RL makes smaller, less destructive changes but is less suited to inserting large amounts of explicit new knowledge.
  • Distillation weakens—but does not eliminate—centralization. Competitors can copy demonstrated behavior, especially when they possess realistic user prompts and full interaction traces. However, benchmark-oriented distillation may reproduce performance on difficult, verifiable puzzles while missing realistic multi-turn behavior, taste, and judgment; access to authentic deployment distributions can therefore remain a significant advantage.
  • RL’s low information content can still produce large behavioral changes. Mid-training may bring a model most of the way toward a capability before RL adjusts its policy using sparse but highly relevant success signals. A small number of bits can rule out large parts of the behavioral hypothesis space, while RL avoids forcing the student to imitate every incidental reasoning token in a demonstration.
  • The strongest claimed generalization is across task horizon, not all domains. Training on mathematics does not automatically create the best coding model, so labs increasingly target many domains separately. What appears to transfer more reliably is persistence: models learn to use more tokens, sustain progress longer, and carry long-horizon behavior into new environments.
  • The core uncertainty remains whether the improvement loop becomes self-sustaining. Current systems can help generate data, run experiments, and feed progress into the next model, but humans still select signals, construct environments, and judge outcomes. The panel does not establish whether this loop will become “self-sealing” or instead keep approaching human research ability asymptotically.

Timeline

  • Early deep-learning era, around 2012: Rapid progress began under neural-network scaling, although the later importance of broad next-token prediction and predictable scaling was not yet obvious.
  • Early OpenAI period: Schulman recalls doubting that next-token loss would isolate the small fraction of data containing important intelligence-related structure; in practice, the objective generalized much better than expected.
  • Pre-training scaling phase: Better data, larger models, and scaling laws produced relatively smooth gains, while engineering advances made increasingly large training runs and context windows possible.
  • Post-training and RL phase: As pre-training returns weakened, synthetic reasoning data, RL with verifiable rewards, and increasingly diverse environments extended the capability curve.
  • Current lab workflow: Researchers identify weaknesses in deployed models, convert bugs and recent discoveries into feedback or environments, and distill those improvements into later checkpoints or model generations.
  • Emerging deployment-learning phase: Companies increasingly use product traces, user feedback, specialized environments, and modular adaptations to improve domain-specific models on a cadence of days or months rather than updating one universal model instantly.
  • Possible near-term agent phase: Panelists forecast usable remote-worker-style agents in roughly one to three years, initially strong in instrumented organizations and common domains but uneven across the long tail of human work.
  • Possible recursive-improvement phase: If agents can repeatedly choose useful experiments, analyze results, and revise plans, research productivity could accelerate sharply; estimates for broad expert dominance ranged from roughly three to ten years.
  • Alternative plateau: If objective discovery, real-world transfer, continual learning, or frontier-signal generation proves substantially harder than expected, progress could flatten before systems decisively surpass human researchers.

Key Facts & Figures

  • Mid-training’s contribution: Millidge estimates that synthetic reasoning-oriented mid-training can take a model roughly 80% of the way to the final RL checkpoint, leaving RL to make comparatively focused policy changes.
  • Research-horizon trend: The panel cites EdgeBench as finding that the duration over which models can successfully work has been doubling about every three months.
  • Data versus architecture: An investigation described by the host and Jerry Han paired training recipes and datasets from 2019 onward; at small scale, newer data reportedly produced about a 12× compute-efficiency gain, versus roughly 3.7× from architecture improvements.
  • Exploration limitation: One speaker says that if an RL model cannot solve a problem within roughly 128 rollouts, it is unlikely to receive enough positive signal to advance, motivating curricula with intermediate difficulty levels.
  • Research experience and taste: The panel notes that a PhD researcher may develop substantially better judgment over about five years and 10–30 projects, suggesting that taste need not require centuries of direct experience—though transfer to much longer AI tasks remains uncertain.
  • Model-size regime: The discussion suggests that frontier systems have operated broadly between roughly 100 billion and 2 trillion total parameters, rather than following a clean annual doubling; active parameter counts may plateau temporarily because long RL rollouts make inference expensive.
  • Forecast range: Estimates were approximately one to three years for a broadly useful remote-worker form factor, two to ten years for a tenfold AI-research productivity increase, and three to ten years for dominance across computer-based cognitive fields.

Insights

  • The pivotal bottleneck is becoming epistemic rather than mechanical. Coding experiments and executing plans are increasingly automatable; deciding which question matters, what result would be informative, and when an apparent improvement is misleading may become the scarce capability.
  • AI research could accelerate before general workplace autonomy. Research progress is comparatively cumulative: once attention, mixture-of-experts methods, or an RL technique is discovered, it can be preserved in the stack. Organizational work is more non-stationary and requires continuously relearning people, norms, and shifting constraints, so recursive AI improvement could arrive before reliable AI paralegals or managers.
  • Distillation creates a two-sided market structure. Model behavior may commoditize quickly, but realistic interaction data remains scarce and commercially sensitive. The durable moat may therefore move from model weights toward deployment access, proprietary workflows, and the ability to turn traces into robust training environments.
  • Benchmark leadership may increasingly diverge from product quality. Difficulty and realism are separate axes. A model can excel at hard, easily verified tasks yet perform poorly in multi-party, ambiguous settings where success depends on taste, restraint, and adaptation.
  • A universal hive mind is economically less likely than a layered learning system. Firms have incentives to retain proprietary experience rather than donate it to a shared base model. A more plausible progression is private memory or adapter modules, periodic consolidation into new base models, and gradually shorter release cycles.
  • Frontier data scarcity is not just a quantity problem. Once models absorb existing expert work, the missing information may not exist in a corpus waiting to be filtered. New capability then requires experiments, environments, human judgment, or real-world outcomes that create fresh bits of evidence.
  • Model monoculture is a systemic risk. Widespread distillation from the same frontier teachers and optimization against similar judges can propagate stylistic tics, blind spots, and reward-hacking strategies across nominally independent models, reducing the diversity needed for error correction.
  • The forecasts depend heavily on definitions. “Remote worker,” “10× productivity,” and “dominates experts” conceal major differences in work quality, time horizon, organizational support, and ability to learn new domains. The wide forecast ranges reflect conceptual disagreement as much as uncertainty about dates.

Practical Implications

From the source

  • Evaluate agents on realistic, multi-turn work involving changing objectives and human interaction, not only difficult but easily verified benchmarks.
  • When training with RL, use curricula that keep success probabilities high enough to generate learning signal rather than jumping directly across large capability gaps.
  • Preserve recent research progress by turning discovered bugs, improved procedures, and expert feedback into fresh environments for the next training cycle.
  • Treat domain-specific training as useful even for generally capable models: it can improve runtime efficiency, cover commercially important tasks, and teach transferable long-horizon habits.
  • Use deployment traces carefully; simple proxies such as whether a user accepted an edit can be reward-hacked and may not reflect long-run quality.
  • Consider modular memory or adaptation mechanisms when frequent base-model updates would cause forgetting or expose proprietary organizational knowledge.

Analyst implications

  • Track long-horizon reliability, experiment selection, self-correction, and performance under changing conditions as leading indicators of automated R&D—not code volume or benchmark scores alone.
  • Stress-test whether claimed model improvements survive realistic prompt distributions, delayed outcomes, and independent evaluators.
  • Organizations adopting agents should make workflows machine-legible while retaining human control over objectives, escalation rules, and ambiguous social judgments.
  • Investors and builders should distinguish moats based on frontier weights from moats based on proprietary interaction data, evaluation infrastructure, and continual-improvement loops.
  • Researchers assessing takeoff scenarios should separately model cumulative technical work and non-stationary real-world work; treating both as one generic notion of “intelligence” may obscure the most important asymmetry.
  • Preserve diversity across teachers, datasets, judges, and training objectives to reduce correlated failures and behavioral monoculture.

References

  • Moravec’s paradox: The idea that tasks humans experience as effortless may be harder for machines than formal reasoning tasks humans find difficult.
  • Reinforcement learning from human feedback (RLHF): The post-training approach associated with Schulman’s work at OpenAI and central to the discussion of behavior specification.
  • Reinforcement learning with verifiable rewards (RLVR): The panel’s main framework for training agents across tasks with automatically checkable outcomes.
  • Kaplan scaling laws: Early language-model scaling results discussed as an example of how deeper automated analysis might have detected methodological issues sooner.
  • muP, or maximal update parameterization: A method for transferring training settings across model sizes, cited as an example of research progress that better analysis could accelerate.
  • GRPO: A reinforcement-learning technique mentioned as a possible point in the technical lineage from which automated research environments might eventually rediscover improved methods.
  • Chinchilla scaling laws: The compute-optimal relationship between model size and training data used to discuss whether future systems should become larger, smaller, or more data-efficient.
  • EdgeBench: The cited evaluation reporting rapid growth in the length of tasks models can complete successfully.
  • Catastrophic forgetting: The loss of earlier capabilities when a model is repeatedly updated on new or narrow distributions, presented as a central obstacle to continual learning.
  • Mixture-of-experts and model sparsity: Architectural concepts important to the trade-off among total capacity, active parameters, data efficiency, and inference cost.