Delusions of Dialogue: Control and Choice in Interactive Art


>>  Jim Campbell


What I want to discuss this evening is art that uses a computer during the process of viewing, and the relationship of this type of art to the structure of the computer. Though understanding this structure is clearly not relevant to viewing art that incorporates a computer, I believe that some understanding of the way that a computer functions might help us to critically analyze the state of the art and examine why the art has so clearly not reached the level of transcending the technology.


Computer system:

A computer can be thought of as an empty structure into which a concept is inserted. This concept which must be represented in a mathematical way is the program... which is made up of a series of algorithms that define the response of the system. Inputs happen, the program reacts and produces an output selected from the vocabulary of that particular system. That output could be an image, or a sound or a robot that jumps up and down or it could be a change in room lighting. They are all ways of representing the internal direction of the program.

I find it useful to put interactive work on a dynamic spectrum with controllable systems on one end and responsive systems on the other. In controllable systems the actions of the viewer correlate in a 1 to 1 way with the reaction of the system. Interactive CD ROM's are on this end of the spectrum and generally speaking so are games. In responsive systems the actions of the viewer are interpreted by the program to create the response of the system. Artificial Life works fit at the extreme end of this side of the spectrum. This spectrum is fuzzy and often subjective and more importantly changes with a person's technological proficiency. If a work is responding in a predictable way, and if the viewer becomes aware of the correlation between their action and the works response to their action then they will feel that they are in control and the possibility of dialogue is lost. The first time I walked through an automatic door at the supermarket I thought the door was smart and was responding to me. Now I step on the mat to open the door on purpose. The point is that often the first time an interface is experienced it's perceived as being responsive but if the interface is experienced again it becomes controllable. The second time it's not a question but a command.

It's very hard to avoid the theme of control in computer art because computers are fundamentally designed to be controlling devices. The computer industry's goal of making computers and programs smarter is simply to make computers more efficient at being controlled by the user to get a job done. Why should they do anything else. It's generally what we want computers for. We want them to be passive slaves. One can see this in the software, hardware and interfaces that are currently being used. This model is fine until it collides with art.

For example look at the concept of icons as an interface device. They are designed to be precise and accurate and discrete, on or off. They are designed to present a closed set of possibilities. They are not capable of subtlety, ambiguity or question. An interface of choice and control makes sense for a word processor or an information retrieval system or a game, but not as a metaphor for interactivity or dialogue.

I've often wondered why most interactive work feels contrived and designed for a calculated response, like bad art school art. I've seen so many cd roms and interactive video discs that have felt like my interaction was completely scripted and predetermined within the pretext of a few choices. A painter can create a painting without consciously thinking about the future viewers. It's much harder but I think that a filmmaker can create a film without being overly affected by what the audience's response will be. It's almost impossible for an artist creating an interactive work to not try and second guess the viewer. How else can an artist design the interface without seeing it from the other side? One of the ways that I've seen artists avoid this problem is to not put themselves in the viewer's shoes but instead to take the point of view of the work itself. Instead of saying as viewer what can I trigger saying as program what can I measure. And then what can I reflect and what can I express based on some interpretation of the viewers responses. This way the work becomes a momentary, but dynamic reflection of a thinking process. Because the artist doesn't write the viewer's side of the interaction the viewer can respond in a more open way.

One of the consequences of this approach is that the work like a painting and like a film exists on its own. There is no attract mode. The work is not waiting for a person to complete it. In a way the work becomes interactive not with people but with its environment. This is particularly important with work that exists in a public space.

The degree to which a work feels like a game instead of a dialogue or the degree to which a work feels like an answer instead of a question is the choice of the artist and not a limitation of the medium or the technology.


The Program:

The program has 3 main functions: one is to interpret the sensory input devices (the mouse, the keyboard, the microphone etc.) the second is to control the memory, what to store, what not to store and what to retrieve (??or associate with the input??) and the third is to select and control a response based on the interpretation of the sensory devices and the memories. I press the letter k on the keyboard arepresentation of the letter k.

Programs are mathematical representations... They have to be defined mathematically. This brings interesting questions to the artistic process when an artist is forced to transform an idea from a concept or an emotion or an intuition into a logical representation. A difficult thing to do without trivializing the original concept. What often happens during this reductive and transformational process is that the subtlety in the work is lost simply because of the fact that things have to be defined with mathematical precision. A different approach is to start with an idea from technology, and let the work flow from the set of technological possibilities. This avoids the problem of finding a mathematical equivalent by starting with one, but certainly a problem with this approach is that it is difficult to take the work beyond self referentiality. Often these works are only about the technology that they use and their processes and effects. Yet another way of avoiding this issue of needing to be precise is to use third party programs to make this transformation. This process incorporates a whole new set of problems, the main one being that it is usually the third party software that becomes the soul of the work. The photoshop effect. Software that is written to create the same effect or response from the viewers over and over again. It becomes not a tool but a palate of cliched symbols. For third party software that is specifically written for a particular work the unique program will have unique expression in the context of the work because Software is subjective in this transforming process. The way that a program is written has meaningful expression unless the program is performing a trivial function. For example there will never be a universal program that understands a sentence, because sentence comprehension clearly has a subjective element to it. Any sentence comprehension program will take on the biases of its programmer within its interpretations. (photoshop question when does a tool become a cliché???)

There is no good way of defining what a program is. A mathematical description might say that it is a series of algorithms that choose a new state based on the current state, the past states, and the current set of inputs. It has direction. An anthropomorphic analogy might say that a program controls its own time by responding to its senses. It has motivation. Another aspect of a program is that it is completely invisible to the viewer. The viewer can only infer meaning from the program. This trait of invisibility is where the power of illusion lies, and invisibility associated with direction or motivation is the combination of characteristics that cause us to project attributes of life onto or into a computer. It's difficult for a viewer not to project intelligence and will into a program that has meaningful responses to their actions or even responses that are only perceived as meaningful. Where this viewer's projection actually ends up can often be inconsistent and confusing within a particular work. Sometimes it might be onto a physical thing, sometimes into an image of a physical thing and sometimes the projection might exist but have no physical embodiment at all.

I did an experiment a while ago to try to show how the simplest of meaningless processes could be combined to imply meaning. I created a second cursor on the computer screen that was like a shadow to the regular cursor. I then added delay to this second cursor and noise to its coordinate position on the screen. It then seemed as though this second cursor was following my cursor around, implying that it was alive. The simplest interpretation would suggest that delay implies thinking or intelligence and that adding randomness to delay implies volition. Of course there was no life, there was only the shell of some meaningless characteristics of life. This suggests that the characteristic of following used to be, but is no longer a characteristic of something alive. It is now a characteristic of the behavior of something alive, but it also the characteristic of the behavior of a meaningless computer algorithm. We are still in the illusion stages of this technology. I have wondered what the extrapolation of the willing suspension of disbelief will mean in our relationship to computers.

Expressive meaning within the program is an important part of an interactive work. To try to illustrate this, I go back to the anthropomorphic analogy that a computer is like the brain and the computer program is like the mind. I do this not to suggest that a computer is capable of life but that it has traits that I've already mentioned that if thought about can be used for expression, namely that it has direction that is hidden that can have meaning based on the present. If you're having a conversation with someone, their words and facial expressions and tone of voice and type of eye contact etc. all add up to pointing to the ideas and feelings that this person is attempting to communicate to you. Your window to their conscious and subconscious ideas and memories and motivations is through their words and behavior. These underlying aspects of what's going inside their head are certainly an important part of what is being communicated. Analogously if you're interacting with a computer the control and display of the images and audio and the text, etc. by the computer program all point to the hidden aspects of the program itself and I think this happens whether one wants it to or not simply because the program is responding to the present. If the program is trivial, then an aspect of the communication will seem trite.

Another related way to try and understand why there is meaning within a program is to look at some older mediums. If you're watching a film or looking at a painting, the images that you see reference the past in a static way. In an interactive work the images that you see are dynamically referencing the past.

If the new element to film was time, then I think that the new element to interactivity is the present. And it is the program that connects the present to the past.

There is one very simple thing that a computer program can do that our minds can not. Flip a coin. This ability of a program to make a truly arbitrary decision, an unmotivated decision is often used to model many naturally occurring processes, but it is really an inaccurate model of anything. Typically the only characteristic in common with the process being modelled is unpredictability. Irrational behavior for example is unpredictable, but it is anything but arbitrary. If many irrational interactions occur within a communication and the actions point to the same set of hidden motivating forces they will begin to reveal what these motivations are. A number of random actions will point in many different directions creating nothing but confusion. Unpredictability need not be confusing but can actually be revealing if the right models are used in the programs.

Most programs are trivial with randomness thrown in to make them seem complex. What this does is make the communication shallow and confusing.


Sensory Inputs:

One of the ways that people tend to think about interfaces is ways of getting discrete and accurate information into the computer because computers process numbers. Input interfaces are ways of converting the real world into numbers(digitizing), but the world is continuous not discrete. Going back to the idea that interactive works can be put on a spectrum from controllable to responsive, interfaces can be put on a similar spectrum with command on one end and measurement on the other and these usually correspond to discrete interfaces on the command end and continuous on the measurement end. And again where an interface fits on this spectrum has not just to do with its implementation but also with its perceived structure.

A few examples: A foot switch is mounted under a carpet near a video monitor. If the viewer walks up to the monitor and steps on the carpet the switch is closed and triggers an image and sound to start playing on the monitor. When the viewer leaves the carpet the image and sound stop. If the viewer wants to see the image again they step on the carpet. It's a command to start the image. The viewer is not interacting with the image or what's behind it, the program, they are interacting with the foot switch. There is no dialogue. It's a discrete interface, like an icon, the switch is on or off, the image is on or off.

A second example: 100 foot switches are mounted in a row under a carpet to create a position detector that measures a viewer's distance from the same video monitor as above . The system can differentiate 100 possible distances from 0 feet to 20 feet. The image is at maximum brightness and the sound is at maximum volume when the viewer is 20 feet away, but as the viewer walks closer the image and sound fade to nothing reaching nothing when the viewer is 1 foot away. A viewer will find that the optimal positions for image intelligibility and sound intelligibility will be different. Different viewers might respond in any of a number of ways from oscillating between these 2 optimal positions to compromising to prioritizing, but an important point is that their action will be based on what they are seeing and hearing, not on where their feet are. This illustrates the fundamental difference between discrete interfaces and continuous ones, namely that In discrete interfaces the interaction is between the viewer and the interface and in continuous interfaces the interaction is between the viewer and the work or the program.

Even though the above interface is discrete as all digital representations by definition are, it will be perceived of as being continuous because the difference between any 2 of the 100 levels is imperceptible... Of course one could display the distance as a number between 1 and 100 on the screen along with the image and this would turn the perceived continuous interface into a discrete one causing the viewer to interact with the number, like a slider bar on a mac or windows program.

Interfaces that involve discrete choices leave little room for intuition. Discrete choices generally cause the viewers to look for a logical reason to make the correct choice based on what they might think the consequences will be. Unless it's a game there is no correct choice.

It has been my experience that intuitive interaction through an interface can only be possible if that interface is able to understand any input of its type. A couple of examples: If the interface to the computer is word recognition, then the computer should have a reasonable understanding of anything that might be said to it, not just a few words. If the interface to the computer is a distance measuring device then the computer should understand distance in any direction that it is approached from, not just from straight on. If an interface has holes in it with regard to its structure, then it will be disregarded by a viewer simply as metaphorical and any interaction that does occur between a viewer and the work will get stuck at the interface. The interaction that occurs will be between the viewer and the interface not between the viewer and the work or the program, just as I suggested earlier with discrete interfaces. A transparent interface is a continuous one that is perceivably complete over its type of structure.



Like the program the memory in a computer is also invisible, but even more so because for information to get into or out of memory it needs to travel through the program. This process which connects the real world with the internal memory must involve transformation but may or may not involve interpretation. It usually does not. For example a moving image is stored as a moving image and later played back as the same moving image. In this case transformation takes place at the input device, the camera system, by digitizing the image and at the output device by undigitizing the image and displaying it. This process is simply the regurgitation of raw data. It doesn't have to be. The current structure of the computer allows for the possibility of interpreting an input and subsequently storing this interpretation in memory. The original data need not be stored at all. This potential of the computer to be able to extract information from an input and store it not as raw data but as associated data is one the fundamental characteristics that allows for a work to be able to change and grow with time and even change its vocabulary with time. To me this is one of the most exciting and unique possibilities in computer art and very little work has been done in this area. Works that perceivably never repeat themselves. Works that respond to their environment not just in a short term way, but in a long term way unpredictably and meaningfully. Easier said then done.


In Conclusion:

The difference between an interactive game and an interactive work of art is not just in the subject matter. It's also in the program and interface, which are an important part of the expression of a work. Artists working in this field will continue to be odds with the models and directions of the multimedia industry.

Interactive art that uses a computer is not there yet. It is certainly the first medium in history in which the expression of an emotion or an idea or a concept has to be reduced to a mathematical form. Probably the only meaningful dialogues that occur while interacting with a work, are between the viewers and themselves. Responses from the work that are altered reflections of the viewer's responses. The limitations that we are up against at this point are no longer technological. Possibly as we understand more about communication, it will be possible to express not a thought but a fragment of a way of thinking and growing.