Talking Heads and Beyond

In the following text, I try to sketch some major aspects of the work of Luc Steels on the origins of meaning/lexicon/grammar. All the ideas are taken from the references given below. Many sentences are copied, although often I do not quote explicitely.

Introduction

The Talking Heads experiment was set up as a large-scale experiment on the occasion of an art event in Antwerp. There, it took place in the 'laboratory for cognitive robots and teleportation'. Around the same time, other sites of the experiment were installed around the world and the experiments could be accessed and watched via internet.

(for more advertisement see http://talking-heads.csl.sony.fr/ )

Luc Steels designed this experiment among others to study the processes involved in the evolution of language and meaning. The aim is to build an artificial infrastructure that shows how cognitive systems might be able to develop language and increase their expressive power by interacting with other cognitive systems and a common environment.

One of the main ideas in Steels' approach is what he calls the selfish language hypothesis. According to this paradigm "language colonises brains and recruits available cognitive capacities to satisfy its appetite for expressing ever more complex meaning with minimal effort and maximum effectiveness". Therefore, language underlies constant and purposeful changes and individual adaptations. It develops out of the desire of a group of individuals to comunicate with each other about e.g. events in and states of the environment they share.

The result of these ideas is a situated and embedded approach to cognition which does not only explain how language can bootstrap itself, but also shows how communication without complete ontological or linguistic coherence is possible.

Language and Language Games

Communication can be understood as a method (one of the few) of externalizing internal mental states (meaning) and vice versa (for the hearer). As part of this process, the one who desires to speak (the speaker) needs to verbalize whatever he wants to schmooze about. Verbalization is a mapping from meaning to utterance (form). This process consists of lexicalization, i.e. mapping the 'components' of the meaning to individual words, and the use of syntax, i.e. using word structuring and additional words or pre-/suffixes to express relations between the components. The reverse process is the interpretation of an utterance, i.e. mapping form to meaning.

The communication in the experiments discussed here follow a very simple protocol. They are so-called language games.

Language games are guessing games. Two players take part, one of them as the speaker, the other one as the hearer. The speaker starts by selecting a context and communicates this choice to the hearer. Then the speaker chooses a topic from the context and verbalizes this topic. For this purpose the speaker has to choose an utterance that distinguishes the topic from other objects in the context.

Subsequently, the hearer tries to figure out the topic and points to it. There is no winner or loser to the game, but a game may succeed or fail depending on whether the hearer 'understands' the speaker or not.

In the case the game fails, the speaker points to the topic. Using this additional hint the hearer then tries to figure out the meaning of the utterance that was used and adds corresponding word-meaning relations or syntactic constructions to his internal structures (categories + lexicon + grammar ...). The speaker, in turn, decreases his own meassure for 'reliability' of the words or syntactic structures he used. If the game succeeds on the contrary the speaker and hearer increase this meassure.

Words and Meaning - The Talking Heads Experiment

Each site of the Talking Heads experiment consisted of two robots, a white board with colored geometrical objects pinned to it (the environment) and some control monitors. The robots (consisting of a camera + a loudspeaker + a microphone + a computer) served as bodies for corresponding minds, i.e. agents. Consequently, different agents could occupy the robots in consecutive language games. Moreover, the agents could travel between the different sites by teleportation (using an old-fashioned teleportation system called internet).

In this way, there was a whole population of agents at each site, interacting with each other in the language games. Each of these games was started by assigning one of the agents to be the speaker and another one to be the hearer. The speaker then chose an area of the whiteboard (the context) and an geometric object from this area (the topic). Next, he tried to find properties of the topic that distinguishes it from other geometric objects in the context and a corresponding utterance. The task of the hearer was then to identify the topic. While playing these games, the language repertoire of the agents constantly changed. New words and categories were invented or learned from other agents, others fell into oblivion.

The architecture of the agents described here consists of 4 layers: perception, categorization+conceptualization, lexicalization and a pragmatic layer (in later versions a syntactic layer was added). The following paragraphs describe the functions of these different components.

Perception

The perception of the Talking Heads is based on a low-level visual process detecting the different objects by using edge detection and other segmentation techniques. The properties of objects that are then examined are predefined. For each such property a sensory channel exists that measures a value corresponding to the property in question (e.g. horizontal position (hpos), vertical position (vpos), average grayscale, ...).

Conceptualization

The conceptualization process divides the possible values of each sensory channel into so-called regions, i.e. value intervals. A category is then a nonempty set of regions (from different sensory channels).

Example:

• sensory channels : hpos, color
• regions: hpos: LEFT,RIGHT ; vpos: TOP,BOTTOM ; color: GREEN,ORANGE
• categories: e.g. {LEFT} ; {TOP,LEFT} ; {RIGHT,ORANGE}

This subdivision takes place as a dynamic process. Regions can be split into smaller regions or be joined together, depending on how much categories are needed for succeeding in identifying and differentiating the objects in the environment.

In particular, if a language game fails because the categories the hearer currently uses are not fine-grained enough to distinguish the topic from the other objects in the context, a new category has to be created.

The use and success of categories is meassured. If categories are not sufficiently successful or used too seldom they are omitted.

Lexicalization

The lexicon contains mappings from categories ("meaning") to words ("form") Such mappings are allowed to be n to n, i.e. they allow for ambiguity and synonymy. Additionally, for each word-category pair in the lexicon a meassure for the success rate and occurence of this mapping is stored and updated during the communication process. In the case a category for which no word is known needs to be verbalized, a new word is invented.

If a language game fails because the hearer does not know a word the speaker is using, the hearer tries to guess which discriminating properties the speaker used to describe the topic. She then adds a new entry with the corresponding category-word pair to her lexicon.

Moreover, lexicon entries are deleted if they are not used frequently.

Pragmatic Layer

The pragmatic layer contains scripts for playing the language games and for interacting with the other agents.

Example of a Language Game

context: a green square on the left, an orange triangle on the right, both objects are on the top of the scene

speaker: a1 ; hearer: a2

agent	action
a1	selects topic:	green square
a1	categorizes topic:	{LEFT}
a1	looks up category in lexicon:	"wukalo"
a2	looks up "wukalo" in lexicon:	{TOP}
	( there is more than one such object )
a2	indicates confusion:	"wukalo?"
	( game fails )
a1	points to topic:	green square
a2	categorizes topic:	{GREEN}
	( a2 adds new entry to lexicon )

Results

Even though ambiguities and synonymies exist and new words appear and others disappear, lexicons that are coherent to a certain extent develop over time in a group of agents, provided not too many agents enter or leave the group.

The Evolution of Grammar

A grammar can be seen as a "product of the ability to categorize and detect associations and to apply them to language itself". The resulting grammatical structures can be used to increase the expressive power of a language. Under these premises, it seems plausible to regard a desire to communicate with maximum effectivity as the driving force in the development of grammar. In a series of experiments succeeding the Talking Head adventure, Luc Steels studied the evolution of a grammar for case.

Case can be defined as a marking of a noun phrase (NP)(noun and adjectives, articles, … belonging to it) by a marker (case marker) for expressing semantic relations (semantic roles) of the event and the objects involved (e.g. agent, patient, time, location). Additionally, case may be used to emphasize certain parts of the communicated information. But this aspect was not considered by the experiments discussed here.

In different languages, different realizations of case marker systems (affixes, inflections, particles ...) and a different usage of cases can be observed. In Steels' aproach, a short word that proceeds or succeeds the NP is used as a case marker. New markers or new semantic roles are introduced as needed. As stated earlier, the main goal for the agents participating in the experiment is maximizing their communicative success while minimizing their effort to do so.

The language games now take place in a world of dynamically moving objects. An event that took place recently together with the objects involved form the topic. Furthermore it is assumed that the agents already have a lexicon containing words for the events and objects they are required to describe. New to this setting compared to the previous one is that the agents have to deal with events, which are corresponding to the linguistic category 'verb'. Therefore, first an explanation is needed for how verbs are conceptualized and represented in the mind of an agent. Then the mechanisms involved in developing a case system can be discussed.

Event Detection

The perception and conceptualization of the agents in this setting follows similar principles as in the Talking Heads experiment. Since the environment now contains moving objects, a new requirement for the perception system is the detection of events.

In the incomming data so-called micro-events are detected. These micro-events monitor changes of the states of or the relations between the objects involved. They report certain properties of the objects (such as VISIBLE) or actions (like TOUCH, MOVE) taking place. Micro-events are ordered by time. An event (like WALK-TO) then consists of a sequence of micro events.

Example:

event: WALK-TO

micro-events:

• agent does not move
• target does not move
• agent approaches target
• agent touches target

Formalizing Events and Markers

In an event like WALK-TO, several objects are involved like the object that is walking. Therefore, the semantic structure of events is expressed in the form of predicates taking certain variables as arguments. The variables constitute the different event-object relations involved. They need to be bound to the objects involved in an event during the interpretation process of the hearer.

Example:

WALK-TO( WALK-AGENT, WALK-TARGET)
( an agent and a target are involved in the walk-to event )

An entry for the event in the lexicon does not include these informations about event-object relations. Such relations are expressed seperately by case markers as explained below.

Example:

"walk-to" <-> WALK-TO

Markers are associated to single event-object relations of particular events (or sets of events, see: Markers for Semantic Roles). The marker and the corresponding object form a syntactic unit and therefore appear in an utterance next to each other in a defined order. Thus a syntactic rule reflecting this order as well as the event-object relation a marker expresses has to be introduced.

Example:

PU << WALK-TO.WALK-AGENT
( the marker pu preceeds the agent of an walk-to event )

In addition, a rule mapping the marker to a word needs to be inserted in the lexicon.

Markers for particular Event-Object Relations

Without markers there is no difference in utterances containing the same words in different orders. That's why ambiguity may occur if similar events with similar objects involved took place recently. In this case, the language game might fail. Another problem is that the hearer needs to map the objects in the utterance to event-object relations in order to understand the communicated meaning. This mapping is often not given by the utterance itself, but needs to be interfered from the context (see example below).

In order to avoid ambiguity and decrease inferential effort for the hearer, the agents dynamically create case markers for event-object relations. For this purpose, an utterance produced by the speaker is first fed back to his own interpretation system. By this means, possible ambiguities are detected and the speaker can estimate the amount of interferences needed to understand the utterance. If necessary, a new marker is introduced for a particular event-object relation (see example) and the utterance is reformulated. The hearer, in turn, first ignores all markers she does not know and tries to figure out the meaning of the utterance without the markers. Then she guesses the event-object relation a new marker describes by trying to find an ambiguity or interference in his own mind that can be avoided by this marker.

Example:

context: a green triangle walks to a red square
speaker: a1 ; hearer: a2

( Assumption: a1+a2 conceptualize the triangle as {GREEN}, the square as {RED} and detect the event WALK-TO( WALK-AGENT, WALK-TARGET) with WALK-AGENT beeing the triangle and WALK-TARGET the square )

agent	action
a1	verbalizes the event (arbitrary word order): "red walk-to green"
	( a2 would interpret this utterance correctly if no event causing ambiguities like a "walk-to" event with a "red" agent and a "green" target appeared recently )
a1	interprets its own verbalization :
	maps "walk-to" to recent event WALK-TO( WALK-AGENT, WALK-TARGET)
	maps "red" to category {RED} detected in the observed event
	maps "green" to {GREEN}
	identifies WALK-AGENT := {GREEN} and WALK-TARGET := {RED} since this is consistent with the observed scene
	( this identification was not determined by the lexicon/grammar but needed to be interfered. Therefore a new marker "pu" ( PU << WALK-TO.WALK-AGENT ) is introduced )
a1	new verbalization: "red walk-to pu green"
a1	runs its interpretation process again
	maps "walk-to" to WALK-TO( WALK-AGENT, WALK-TARGET), "pu" to PU
	"green" to {GREEN}, "red" to {RED}
	since "pu" preceedes "green" : WALK-AGENT := {GREEN}
	since only one free variable is left for WALK-TO: WALK-TARGET := {RED}
	( no interference is needed to interpret the utterance)

Markers for Semantic Roles

The previous section described how case marking for events is achieved within the architecture of the experiments. With this procedure, different markers are introduced for different events and consequently, each 'verb' recieves its own system of markers. For using cases for expressing semantic roles (as AGENT) rather than for specific event-object relations (like WALK-AGENT), a mechanism needs to be found that identifies such roles, i.e. finds event-object relations of different events that share certain aspects (e.g. WALK-AGENT, MOVE-AGENT: both move). In Steels' work, this mechanism is based on analogy. If it is possible to find a strong analogy between two (or more) event-object relations, a case marker is introduced that can be used for all of them. In this way, a semantic role (forming a set of event-object relations) is introduced.

Analogies are detected by analyzing the micro-event structure of the events in question. First all micro-events which do not involve the object of the event-object relation investigated are discarded. Then the agent tries to map the remaining micro events to the remaining micro-events of the other event-object relation. If a sufficient number of micro-events can be mapped, this mapping is considered a (good) analogy.

Results

The introduction of case markers to the language system helps avoiding ambiguities and reducing effort in interpreting an utterance. Semantic roles may be constructed by finding analogies.

References

L. Steels; The Talking Heads Experiment Volume 1. Words and Meaning; Laboratorium, Antwerpen. Limited Pre-edition.

L. Steels; Simulating the Evolution of a Grammar for Case; Course Material IK 2004