Imagine your GPS could see. To answer the question "Do I turn there where that big tree is?", a camera is not enough; the GPS needs to connect what you say to the portion of reality that surrounds your car. AMORE enables machines to connect language to reality, and seeks an understanding of how people make this connection when they talk. The project thus explores the phenomenon of reference in natural language via computational modeling experiments, and we are particularly interested in the interaction of language with conceptual knowledge, on the one hand, and the extralinguistic context, on the other.

The main challenges are:

1. Identifying which entities ("that big tree") are being talked about, both in the visual and in the linguistic camps;
2. Tracking the entities as they are mentioned again, retrieving and adding new information about them as needed;
3. Crucially, having the machine learn these two abilities directly from examples of how people use language. We face the machine with different tasks that require using language to talk about the world, and it progressively learns to represent both the entities and the language that we use to refer to them. Specifically, we test our computational model in referential tasks that require matching noun phrases (such as "the examined boy"  in the example below) with entity representations extracted from text and images.

This interdisciplinary project builds on two complementary semantic traditions:

1. Formal semantics, a symbolic approach that can delimit and track linguistic referents, but does not adequately match them with the descriptive content of linguistic expressions;
2. Continuous approaches to language such as deep learning models and distributional semantics, which can handle descriptive content but do not associate it to individuated referents. AMORE synthesizes the two approaches into a unified, scalable model of reference that operates with individuated referents and links them to referential expressions characterized by rich descriptive content. The model is a distributed (neural network) version of a formal semantic framework that is furthermore able to integrate perceptual (visual) and linguistic information about entities. 

AMORE advances our scientific understanding of language and its computational modeling, and contributes to the far-reaching debate between symbolic and continuous approaches to cognition with a proposal that falls clearly on the continuous camp, but integrates key insights from the symbolic camp. 

(Image credits: Hagerty Ryan, USFWS)