The Generalization Problem

There’s a conversation in AI research about generalization: can large language models truly reason abstractly, or just pattern-match within domains they’ve seen? The discussion usually focuses on architecture, data scale, and benchmarks. I think it misses something more fundamental.

Language models are trained on text. All of it. The entire written output of human civilization, more or less. At a high level, this gives a model broad exposure to the world’s knowledge. From this exposure, the argument goes, generalization emerges.

But the training doesn’t happen at the level of ideas. It happens at the level of tokens. The model learns to predict the next token in a sequence, one at a time. And when you look at the training data - at what humans actually write - we rarely write generally. A cardiology paper stays in cardiology. A legal brief stays in law. Each document is a narrow, domain-specific sequence of tokens.

The counterargument is that generalization emerges from statistical patterns across these narrow examples. That the model discovers shared structures - analogy, causation, argumentation - and learns to apply them flexibly. That the token is just the input/output interface, not the ceiling of what the model can represent.

I find that argument incomplete. The reason has to do with how humans develop the capacity to generalize in the first place.

Consider a child learning the word “hot.” They don’t learn the concept of heat from the word. They touch something, feel pain, build an intuitive awareness, and only later attach the label “hot” to an understanding that already exists. The word is a communication tool layered on top of a deeper, non-linguistic cognition. The generalization (i.e. recognizing that a stove, a candle flame, and a sunburn all share a common quality) happens before and beneath language.

This is the distinction that matters. Humans generalize at a pre-linguistic level. We have a layer of cognition that operates on raw experience, not on words. Language is the protocol we use to communicate the outputs of that deeper process. The physicist’s paper is not the thinking; it’s a compressed, lossy projection of thinking that happened in a richer cognitive space.

If that’s true, then training a model exclusively on language means training it on the communication layer - the output of generalization, not the process of it. The model sees the artifacts of human thought, not the thought itself. It reads the map, but never touches the territory.

This doesn’t mean language models are useless. They are extraordinarily capable of working within the patterns they’ve seen. But the question is whether that capability is sufficient for genuine generalization, or if it’s just a convincing approximation.

I suspect the latter. Language is a lossy projection of a higher-dimensional cognitive space. You can’t fully reconstruct the original from the projection alone. 

This points to a deeper issue than is commonly discussed. The consensus view seems to be that the gap between today’s models and a general intelligence is a training problem. That with more data and compute, the existing architectures will get there. This assumes intelligence is a database, and that accumulating more knowledge is the path to greater capability.

But what if it’s an algorithm problem? An intelligence is not a database; it’s a processing unit. While knowledge accumulation has a marginal impact, the “hardware” of intelligence - the fundamental algorithm - is far more impactful on the outcome. It’s highly unlikely that the transformer, or any of our current models, represents the final or correct algorithm for generalized intelligence. It’s more likely that we have yet to identify the fundamental equations that, when released into the world, would begin on a real trajectory of learning.

The generalization problem in AI may not be a problem of scale or training data. It may be a problem of substrate. Of trying to build a mind from the words that minds produce, rather than discovering the algorithms that produce the words.