Opposite views on language on conviction, since 1950s there is a conflict between classicist, symbolic, formal and rationalist vs connectionist distributed functional empiricist.
The crux is that the former focus on regularity while the latter on messiness.
While this “war” is still raging on, others are starting to say that maybe LLMs are a synthesis of this.
The two theses is:
- Language is both regular and messy;
- modern LLMs are a synthesis of these two approaches.
Agreement in english:
A student is crossing the street, we can’t say a student are crossing the street.
This is categorical rule-like property that expresses a systematic link between form and function:
nouns and verbs inflect for number and display agreement.
but also the same rule has messiness: agreement ad sensum:
a group of students is/are crossing the street
“group” is the subjet; should be singular, but it denotes a plurality that is why are is acceptable, there is no easy fix in symbolic system, we can’t add infinite exceptions.
Within language, some domains are more rule-like than others: e.g. grammar ,but content it’s much messier, e.g. a rule for differentiating between fast and swift.
The thing to stress is that every domain has both traits.
No scholar questions the empirical data, but they appraise them differently: formal linguistics tend to focus on regularities, e.g. generative linguistics often exclude messines from linguistics. While functional linguists reject abstract rules, e.g. cognitive linguistics often have no formal predictive accounts, can’t do predictions.
Instead we need models that natively support both.
LLM may be one such synthesis.
Non-linearity provide the potential for both continuous and near-simbolic behavior.
Using a function like a sigmoid a neuron can specialize to be more continuous like or instead focus on being more binary.
The assumptions for the syntheiss view is that language data provides different kinds of pressures and LLMs have the potential and process in both very distributed and near-simbolic ways. The hypothesis is that LLMs learn differential encoding of phenomena, like grammar being more symbolic-like while content is a more distributed encoding.
Literature on interpretability has found that indinvidual neurons and specific attention heads plus circuits are causally involved in grammar.
Individual neurons are selective for morphosyntactic properties or control syntactic relations.