Nothing from Something: Can a Language Model Discover 0?

This paper tests whether GPT-2-scale language models can spontaneously 'discover' the mathematical concept of zero after being trained only on non-zero arithmetic, finding that while true zero-shot conceptual discovery fails, models rapidly acquire the concept from a modest number of examples, especially when pretrained on natural language.

Problem Statement

As AI systems are increasingly framed as tools for extending human mathematical knowledge, it remains unclear whether they can genuinely generalize beyond their training distribution to invent new conceptual structures rather than merely interpolate within familiar ones. The paper uses the historically significant invention of zero—a concept that required real conceptual innovation in human mathematics—as a controlled testbed for evaluating claims about autonomous mathematical discovery in neural models.

Key Novelty

First controlled empirical test of whether neural language models can spontaneously discover a historically significant mathematical concept (zero) absent from their training data
Distinguishes failure at zero-shot conceptual discovery from rapid few-shot learnability, clarifying what 'mathematical discovery' claims about LLMs actually require in practice
Quantifies how natural language pretraining scaffolds sample-efficient acquisition of a novel mathematical concept, providing empirical support for the 'language as scaffold' hypothesis in mathematical cognition

Evaluation Highlights

Zero-shot generalization: GPT-2-sized models trained only on non-zero arithmetic fail to correctly produce/use the concept of zero at test time, regardless of language pretraining
Sample-efficiency curves: models improve substantially after fine-tuning on tens to hundreds of zero examples, with language-pretrained models requiring approximately 50% fewer examples than non-pretrained models

Signal Assessment

4/10 The paper offers a clean, hypothesis-driven empirical result clarifying the limits of OOD generalization and the role of language pretraining in concept acquisition, but it uses small-scale models and a narrow single-concept case study, making it a solid scientific contribution rather than a capability breakthrough.

Methodology

Construct arithmetic training datasets that systematically exclude the number/concept zero to create an out-of-distribution test condition
Train GPT-2-sized models both from scratch and from language-pretrained initializations on this zero-excluded arithmetic data
Evaluate zero-shot (test-time) performance on tasks requiring the concept of zero to assess spontaneous discovery
Incrementally fine-tune models on tens-to-hundreds of zero-containing examples and trace learning curves
Compare learning curves across pretrained vs. non-pretrained models to isolate the scaffolding effect of language pretraining

System Components

GPT-2-scale transformer

Base neural language model architecture used for all arithmetic training and evaluation experiments

Zero-excluded arithmetic corpus

Synthetic arithmetic training data that never exposes the model to the concept of zero, creating the OOD generalization test

Language pretraining condition

Models initialized from natural-language pretraining before arithmetic training, used to test whether linguistic ability scaffolds mathematical concept acquisition

Incremental zero-exposure fine-tuning sets

Graded sets of tens-to-hundreds of zero-containing examples used to trace how quickly models acquire the concept

Zero-shot discovery evaluation suite

Held-out tests assessing whether the concept of zero emerges without any explicit training exposure to it

Results

Metric/Benchmark	No Language Pretraining	With Language Pretraining	Delta
Zero-shot discovery of zero at test time	Fails	Fails	No difference (both fail without fine-tuning)
Examples needed to learn zero via fine-tuning	Baseline (tens to hundreds)	~50% fewer examples	~50% reduction in required examples

Key Takeaways

Claims that language models can autonomously 'discover' genuinely new mathematical structures should be scrutinized: at GPT-2 scale, spontaneous zero-shot conceptual generalization does not occur, even for a concept as fundamental as zero
Acquiring truly novel mathematical concepts still requires targeted fine-tuning data; practitioners building AI-for-math systems should budget for supplementary examples rather than expecting emergent OOD leaps
Natural language pretraining measurably improves sample efficiency (~50% fewer examples) when learning new mathematical concepts, suggesting linguistic pretraining is a practical lever for bootstrapping numerical/symbolic reasoning

Abstract

AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of"zero". We show that (1) language models of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.