Author: Carles Sora i Domenjó (Universitat Pompeu Fabra)
Citation: Sora i Domenjó, Carles (2010). "The phenomenology of time in interactive visual representations". Hipertext.net, 8, http://www.upf.edu/hipertextnet/en/numero-8/time_interaction.html
Abstract: This article discusses the ways that time is represented in interactive images, and the ways that interactivity can be used to adjust both the understanding and representation of time.
Keywords: Time, metaphor, interactive design, cultural interfaces, tangible time, interactivity.
Table of contents1. Introduction
2. Metaphors of space and time
3. Learning, language and cultural contexts about time
4. Representation of movement
5. Final thoughts on digital interfaces
Khronos projector, an interactive work of art by Alvaro Cassinelli , is both the object of analysis and the point of departure for this article. The work has an unusual way of approaching the control of time with respect to images in motion [Figure 1]. A viewer of the artwork can control the passage of time in the image by pressing directly on the fabric where the images are projected, which accelerates or slows them. Time moves forward as a result of this action in different parts of the image, producing a composition of different tempos within the same image.
Figure 1. Cassinelli, Alvaro. http://www.k2.t.u-tokyo.ac.jp/members/alvaro/Khronos/
Cassinelli's work can be considered relevant because it contributes a new approach to conceptualizing visual representations of time, modifying the passage of time. In this work, depth is used as a dimension of time control in the images, exploring a new way of representing time that breaks with the classic convention of the "arrow of time".
In most visual representations, the march of time is normally presented as evolving in one dimension and in one direction, from left to right. What is proposed here is that some of the standard conventions could be revisited in the context of interactive communication. The nature of this medium could reformulate some cultural conventions, contributing new approaches in the same way that other media have reformulated them in the past. This happens with many cultural conventions and has also occurred with the visual representation of time. The current metaphor is a convention adopted de facto by Western culture that has seldom been contextualized, justified or analyzed in the digital context to date. It is the result of a "natural" adaptation to the medium. If we pay attention, we can find this convention in graphics in the print media, logic interfaces in video equipment, calendars, etc. [Hirotada, 1993]. We could ask ourselves: Could we represent time in other ways, in other dimensions"
Alvaro Cassinelli's project incites us to think about representations of time using other dimensions. The use of depth in this piece as a metaphor for control is uncommon in digital interfaces. Cultural conventions can adopt new forms depending on the context, and digital tools offer them a new space with creative friction.
The control dimension used in Khronos projector has not been very relevant in the design of digital interfaces until now [Buxton, 2007]. Using depth as a control dimension is specific to the interactive digital interfaces. No other medium exists through which we can control the dimension of time in an image by pressing with the hand directly on the projected image.
The fascination generated by some of the interfaces discussed in this article, with which we can modify time in a sequence of images, emerges from intrinsic motivations [Ribas, 2009] within the artistic interactive domain and from interaction design based on experimentation and exploration. In some cases, such as Khronos projector for example, these two factors create new modalities for interpreting and representing our culture.
We can also observe (albeit in isolated cases) examples of breaking the convention of representations of time in other disciplines, such as graphic design. Charles Joseph Minard's 1869 graphic, Napoleon's March [Figure 2], is one example. Up to 5 dimensions are presented within this one graphic, and time proceeds in two directions and is depicted in relationship to physical space [Tufte, 1983].
Figure 2. Tufte, Edward R. Visual explanations: images and quantities, evidence and narrative.
This article is an initial attempt to study the representation of time in interactive contexts using images in motion. Interactive communication contributes key elements to this study, but we cannot engage in the study until we first seek out all of the factors that gravitate to the structure of the concept of time, leaving aside for now physics and philosophy, topics that will have a fundamental role in this study in the future. What is presented here is a reflection on the representation of time and on its remediation within interactive media [Bolter, 2007].
Let us assume that we have a short film showing a sequence of images related in time but with an ambiguous sense of temporal direction. We can use the example of an object flying in the air without any other visual reference. In a film of this kind, it is impossible to determine whether the film is showing the original flight or is showing it in reverse. The arrow of time is not obvious in all processes.
Time, object of study in many disciplines, arts and sciences, attempts to describe the sequence of what happens in the world and of our physical experiences. However, some aspects of our daily experiences cannot be observed, measured or predicted. For example, does time move from left to right? Or perhaps from top to bottom? Those questions cannot be answered definitively because these are abstract concepts. Lakoff and Johnson suggest that humans construct their conceptual framework from a small group of concepts, based on experience, which serve to define all the other concepts that do not emerge directly from physical experience, and that these abstract concepts are best understood using metaphor [Lakoff and Johnson, 1980]. Therefore, in our everyday language the time domain is defined using spatial terms, a less abstract, more concrete domain, to define the more abstract one. In this case, temporal ideas are shaped by the knowledge of our spatial domain [Boroditsky, 2000].
With the goal of grasping the sequence of events, we generally conceive of time as a one-dimensional entity. Although in most languages the concept of time is expressed using spatial terminology (above, below, forward and back), visual representation applies a two-dimensional metaphor in the majority of cases, instead of a multi-dimensional representation [Clark, 1973]. There exists, then, in this domain shift (from abstract to concrete) a certain redefinition of the concept and a loss of meaning.
Multiple factors intervene in the development of our knowledge of time: how we learn the concepts, in what language, and in what context.
When do we learn what time is? Our comprehension of the abstract concepts of time and space is the result of the structuring of language in childhood through direct experience. This structuring is limited and also shaped by the cultural context of the child.
Children acquire language primarily from external stimuli: what they hear and experience in the real world [Chomsky, 1965]. Throughout the process of language acquisition, the child creates knowledge through contact between cognitive mechanisms and the physical world: motor and sensory skills. The future assimilation of time and space is grounded in the cognitive knowledge developed during this phase [Clark, 1973].
The cultural context also has a fundamental role in the development of this knowledge, and language is a major determinant. In English, for example, the adverbs "before" and "after" (denoting a horizontal dimension) are used to refer to time. Curiously, in Mandarin Chinese these same words are used in general, but spatial metaphors with a vertical dimension are also used to refer to time [Skott, 1989]. Mandarin uses the adverbs up and down to distinguish between events closer or more distant in time (a vertical line dimension). What is relevant for our discussion is that there are important differences in the way Mandarin and English speakers think about time, according to the words they use for the spatial domain [Boroditsky, 2001]. This paradigm might exist in other languages.
Beyond the question of language, we need to acknowledge that the comprehension of the concepts of time and space are also enveloped in a historical and social context. For example, in some Andean regions, if someone asks a local how long it takes to walk from one place to another, the response might be a finite number of cigarettes. They use the cigarette (i.e., the time it takes to smoke one) as a unit of measure for distances and travel, which are spatial concepts [Steger, 1991].
From these two examples we can intuit that the understanding of time and its mediation using particular metaphors are subject to the cultural context. It follows, then, that they do not have the same meaning everywhere in the world.
Finally, another factor that appears to influence the construct of the perception of time and the manner of organizing events in time is the direction used for writing. This seems to offer a coherent explanation of the fact that the arrow of time differs so much for Mandarin and English speakers. We know that some Mandarin dialects are written from top to bottom and right to left, while English is written horizontally from left to right. Other variations in script exist. We can list other examples, such as a case in the Philippines of a language that is written from bottom to top. Or we can go back to the Greek alphabet, the origin of the left to right script, but which also made use of a bi-directional script called boustrophedon, in which alternate lines of text were read in a different direction (what we would consider "forward and backward").
Today we organize events sequentially by using various formats, directions and orientations. Films, comics, literature and paintings all have particular ways of articulating the concept of time. However, all of these time structures are conventions that were created at some point. We don't come into the world with a preconceived idea about time, space or motion. Consider, for example, some of the cave paintings in Altamira (Spain) that date from 18,000 years BC, in which the movement in the scenes follows no particular direction. Cave paintings have no time-structured narrative [Wachtel, 1993]. Having arrived at this point, we can affirm that the directionality of time in visual representations is a cultural convention.
The representation of time and the study of motion has been a great challenge for scientists and artists throughout history. There are many cases in both art and science in which we can identify ad hoc inventions that have permitted the study of human beings through the study of movement and thereby coming close to a reflection about time [Cutting, 2002]. This is the case of Étienne Jules Marey, for example, and his photographic series published in La Machine Animale in 1873, in which he used a photographic revolver, designed expressly to capture movement, and other techniques such as stroboscopic images, the use of simultaneous photography to shoot different moments in time in the same space but from different perspectives, long exposures, or time lapse, used to accelerate time of some objects of study [Braun, 1992].
We can also find, in our contemporary artistic history, various attempts to codify space and time in a specific visual representation, with the goal of breaking the linear sequence of moving images. Futurist art, such as cubism or Dadaism, used two-dimensional representation of a scene within a plane. One of the best-known works that explored these ideas is Nu descendant un escalier (Nude descending a staircase), painted by Marcel Duchamp in 1912. Duchamp created a sequence of different moments within the same space from different points of view within a single canvas.
With the advent of cinema, the study of time took the form of a spatial perspective with the first opportunity to adjust the time of actual filming, through editing. This was an important moment for our study because, perhaps quite naturally, the first editing tables were organized from left to right, converting this decision into the standard for all analogue and digital video editing for all time, up to the present.
With the passage of time, cinema has intensified the relationship between time and image, creating new techniques such as bullet time, utilized in the 1999 Matrix film, which permitted navigation through space in a particular time.
Up until this point, these techniques had contributed new perspectives on our object of study, but were always used in the time of filming, never at the time of viewing. For that, it is necessary to permit the external participants to modify some of these dimensions. And so we arrive at interactive communication.
It is in the use of digital moving images controlled in real time that we can continue to formulate new relationships between the images and their temporal and spatial dimensions. Thanks to the digitalization of moving images, a participant can access any time point in the film: the time of filming, the time of editing, and the time of presentation. Of course, this always involves the use of the previously established metaphors for time and space. More recently, some research projects that use emerging digital technologies have facilitated access to these dimensions, utilizing new presentation strategies. The last clock [Figure 3], by Jussi Ängesleva and Ross Cooper , is an autonomous application that captures images from a space with the goal of accumulating time in a two-dimensional plane in the form of a clock [Jaschko, S, 2002].
Figure 3. Ängesleva, Jussi; Cooper, Ross http://www.lastclock.co.uk/
In this visualization interface, the evolution of time in the images captured by a camera placed in the space is structured in a circular manner and with different degrees of resolution by hours, minutes and seconds. This is an alive, not static, representation, which places time within the parameters of the time that has passed, time of the recording. This is a singular dimension for our study.
In the field of interactivity we can differentiate between two types of applications: those that function with dynamic systems in an autonomous manner, without the participation of external actors, and those that are designed for someone to use them. In addition, within the second classification we could differentiate between projects that interest us (where the image is an object of study related to the phenomenon of time) according to their utilization of static analogue images, moving analogue images, digital moving images, and digital moving images modified in real time, or interactive images.
Although the emergence of some of these new technologies permits a re-ordering of the dimensions of time and space in video, we still need to establish new forms of interaction, perhaps outside the interface paradigm known as WIMP (window, icon, menu, pointer device), where we could take control of both dimensions using immersion interfaces. This is the case for Khronos projector, with which we began this article. It is reasonable to think that within a short time we will have access to new and more sophisticated techniques for interaction, such as for example the use of both hands on tactile surfaces [Fitzmaurice, 1995], using one as a pointer and the other as an anchor, which allows new means of representing the temporal metaphor, more meaningful interfaces that are more relevant to the creation of meaning [Backwell, 2006].
Multi-modality and the integration of controlled moving images in interactive media can open new approaches to the object of our study, e.g., conceptualizing time, for example in multiple directions or multiple instants in time, or utilizing physical gestures to stretch or shrink time. We would point out that the actions described here are in a different domain, other than spatial, and could give rise to new conventions in the representation of time, perhaps generating new metaphors not structured within the spatial domain: a liberation of the representation of time.
It is difficult to evaluate applications that disrupt cultural conventions, which are strongly linked to our culture and therefore to our convictions. Even so, it is essential to propose this study and the development of the interactive applications that can exemplify it, with the intention of finding new paradigms of the visual representation of time and, with them, the convention of a need to observe the remediation of interactive communication more closely.
In the future, it will be interesting to apply this methodology to other cultural conventions adopted in our visual culture and observe whether they can also be reformulated in interactive communication. The digital interfaces with which we design and build interactive applications are constantly evolving, and therefore this is an ongoing project that could contribute interesting new approaches to thinking about culture.
Backwell, A. F. (2006). The reification of metaphor as a design tool. ACM transactions on Computer-Human Interaction, Vol. 13.
Bolter, D.,Grusin, R. (2000). Remediation: understanding new media. MIT Press.
Boroditsky, L. (2000). Metaphoric structuring: understanding time through spatial metaphors. Cognition 75, 1-28. Elsevier.
Boroditsky, L. (2001). Does language shape thought?: Mandarin and English speakers? Conceptions of Time. Cog. Psicology 43, 1-22.
Buxton, B. (2007). Theories, models and basic concepts. Chapter 7. Models and Thories.
Braun, M. (1992). Picturing Time: the work of Etienne-Jules Marey (1830- 1904). The University of Chicago Press, Chicago.
Cassinelli, A. and Ishikawa, M. (2005). Khronos projector. Emerging Technologies, SIGGRAPH.
Clark, H. H. (1973). Space, time, semantics and the child. In T.E. Moore. Cognitive development and the acquisition of language. New York: Academic Press.
Cutting, E. J. (2002). Representing motion in a static image: constraints and parallels in art, science, and popular culture. Perception.
Fitzmaurice , G. Ishii, H. Buxton, W. (1995). Bricks: laying the foundations for graspable user interfaces. Proceedings of the SIGCHI.
Gibson, J.J. (1986). The ecological approach to visual perception. Houghton Mifflin Company.
Hirotada Ueda (1993). Automatic Structure Visualization for Video Editing. Proceedings of the INTERACT '93.
Jaschko, S. (2002). Space-Time Correlations Focused in Film Objects and Interactive Video. ISEA Papers, Nagoya/Japan.
Lakoff, G. and Johnson, M. (1980). Metaphors we live by. Chicago, IL. University of Chicago Press.
Ribas, Joan Ignasi (2009). Integració de mitjans en el discurs interactiu: el cas de la difusió cultural. Quaderns del CACNº. 31-32. Convergència tecnològica i audiovisual.
Steger, F.H. (1991). La concepción de tiempo y espacio en el mundo andino. Ed. Vervuert Verlag, Latinoamérica student 18.
Tufte, Edward R. (1983). Visual explanations : images and quantities, evidence and narrative . Cheshire, Graphics Press, cop.
Wachtel E, (1993). The first picture show: Cinematic aspects of cave art. Leonardo 26. MIT Press.