TL;DR
In this talk, Marco Baroni discusses the role of compositionality in neural networks and its implications for language emergence. He presents recent work on language emergence in deep networks, where two or more networks are trained with a communication channel to solve a task jointly. Baroni argues that compositionality is not a necessary condition for good generalization in neural networks and suggests that focusing on enhancing generalization directly may be more beneficial than worrying about the compositionality of emergent neural network languages.
It turns out that compositionality means different things to different people in different contexts. This talk doesn’t provide us with an aristotelian definition of compositionality but a pragmatic one that Baroni used in investigation. The notion of language emergence in deep networks is a fascinating yet also vague. With these caveats in mind there are a numbeor of reuslts that Baroni presents that are worth considering.
If I was initially critical of this talk and speaker I soon came to realize that this is just the kind of message that simulates thought and discussion. It is a good talk and I would recommend it to anyone interested in the topic of language emergence in deep networks. 👍
My work on this subject shows that by adding compositionality to the lewis signaling game not only significantly increases the ratio of messages to simple signals but also renders the overall systems easier to master by making it more sytematic and predictable. This is greatly affected by form of aggregtion used for signal composition. e.g. using a simple template can create a morphology that is automatic and therefore easy to master.
For example if we add a parameter to the game to penalize long signals and reward early decoding of the signal we can get a more efficient signaling system. This allows agents to learn a larger signaling system faster. It also has more subtle effects- perhaps the most important one is that the pooling equilibria in the lewis game which are far more common then the non-pooling equilibria which we call a signaling system can be allow us to learn more efficient signaling systems if we reinterpret these partial pooling equilibria as leading to categories of signals which are further refined within the complex signal but can lead to an early decoding of the signal.
Questions
In this subject we need to ask ourselves what are the main reseach questions and what are the main definitions we can make.
- what is language emergence
- what is compositionality
- How are Neural networks used here
- how were they set up?
- what was the task loss function?
- what is generalization in this task?
The big question is will learning an emergent language aid agents to achive better generalization? - There is also talk about compositionality in RL where transfer learning is challenging. - Hinton discussed the idea of capsules in neural networks as a way to encode compositional structure in neural networks.
- Simple Compositionality - The idea that the meaning of a complex signal is a function of the meanings of its parts.
- Non compositionality - The idea that the meaning of a complex signal is not a function of the meanings of its parts.
- Entaglement
- By the end of the talk he more or less concludes that the compositionality of emergent neural network languages is not a necessary condition for good generalization.
, and there is no reason to expect deep networks to find compositional languages more “natural” than highly entangled ones. Baroni concludes that if fast generalization is the goal, we should focus on enhancing this property without worrying about the compositionality of emergent neural network languages.
Blurb from the talk
Compositionality is the property whereby linguistic expressions that denote new composite meanings are derived by a rule-based combination of expressions denoting their parts. Linguists agree that compositionality plays a central role in natural language, accounting for its ability to express an infinite number of ideas by finite means.
“Deep” neural networks, for all their impressive achievements, often fail to quickly generalize to unseen examples, even when the latter display a predictable composite structure with respect to examples the network is already familiar with. This has led to interest in the topic of compositionality in neural networks: can deep networks parse language compositionally? how can we make them more sensitive to compositional structure? what does “compositionality” even mean in the context of deep learning?
I would like to address some of these questions in the context of recent work on language emergence in deep networks, in which we train two or more networks endowed with a communication channel to solve a task jointly, and study the communication code they develop. I will try to be precise about what “compositionality” mean in this context, and I will report the results of proof-of-concept and larger-scale experiments suggesting that (non-circular) compositionality is not a necessary condition for good generalization (of the kind illustrated in the figure). Moreover, I will show that often there is no reason to expect deep networks to find compositional languages more “natural” than highly entangled ones. I will conclude by suggesting that, if fast generalization is what we care about, we might as well focus directly on enhancing this property, without worrying about the compositionality of emergent neural network languages.
References
https://cs.stanford.edu/people/karpathy/cnnembed/,
https://www.inverse.com/article/12664-google-s-alphago-supercomputer-wins-second-go-match-vs-lee-sedol
https://hackernoon.com/deepmind-relational-networks-demystified-b593e408b643
Lazaridou et al. ICLR 2017 Multi-Agent Cooperation and the Emergence of (Natural) Language
Bouchacourt and Baroni (2018) How agents see things: On visual representations in an emergent language game
Are emergent languages compositional?
- Andreas ICLR 2019,
- Choi et al ICLR 2018,
- Havrylov & Titov NIPS 2017,
- Kottur et al EMNLP 2017,
- Mordatch & Abbeel AAAI 2018,
- Resnick et al AAMAS 2020
A compositional language is one where it is easy to read out which parts of a linguistic expression refer to which components of the input
- Naïve compositionality
- a language is naïvely compositional if the atomic symbols in its expressions refer to single input elements, independently of either input or linguistic context
- Chaabouni, Kharitonov et al. ACL 2020 Compositionality and Generalization In Emergent Languages
Quantifying (one type of) naïve compositionality
Positional disentanglement measures strong form of naïve compositionality: to what extent do symbols in a certain position univocally refer to different values of the same attribute
note - the paper has two other measures
Do emergent languages support generalization?
Is compositionality needed for generalization?
- kind of obvious, particularly if parameters are shared not but is should help
Lazaridou et al ICLR 2018
Kharitonov and Baroni: Emergent Language Generalization and Acquisition Speed are not Tied to Compositionality Emergent Language Generalization and Acquisition Speed are not tied to Compositionality
Video
Slides & and a paper
Citation
@online{bochman2024,
author = {Bochman, Oren},
title = {Is Compositionality Overrated? {The} View from Language
Emergence},
date = {2024-09-01},
url = {https://orenbochman.github.io/posts/2024/2024-10-10-marco-baoni-composionality/summary.html},
langid = {en}
}