BREAKING GLASS
SPATIAL FABULATIONS & OTHER TALES OF REPRESENTATION IN VIRTUAL REALITY
SAC
JOURNAL
6
4
EDITORIAL
BREAKING GLASS
8
INTRODUCTION JOHAN BETTUM
AN ECOLOGY OF SIGNALS & THE SPATIALISED IMAGE
14
A CONVERSATION SANFORD KWINTER & DANIEL BIRNBAUM
THE THIRD GLASS
28
ESSAY FREDRIK HELLBERG & LARA LESMES
TWEETING OUT LOUD IN THE SQUARE
34
ESSAY MARTINE BEUGNET
VIRTUAL REALITY AND THE GAZE
ON FRAMELESSNESS, PANOPTIC VISION,
AND PARRAGIRLS’ MEMORY WORK
42
ESSAY CURTIS ROTH
GESTURING ELSEWHERE
48
ESSAY MICHAEL YOUNG
EXCESSIVE RELIEF
58
TRIALOGUE JONATAN HABIB ENGQVIST, CHRISTER LUNDAHL & MARTINA SEITL
AN ACCEPTABLE LEVEL OF REALITY
68
A CONVERSATION
DANIEL BIRNBAUM & SVEN-OLOV WALLENSTEIN
A NEW CURATORIAL TOOLBOX
72
ESSAY FABRIZIA BANDI
THE ARCHITECTURAL RELEVANCE OF VIRTUAL REALITY
82
PORTFOLIO KLING KLANG KLONG & ONFORMATIVE
COLLIDE
90
PORTFOLIO AINSLEE ALEM ROBSON
FERENJ A GRAPHIC MEMOIR IN VR
96
PORTFOLIO FOLLY FEAST LAB
MEDITERRANEAN SEA DIARIES
ON THE AFTERLIFE OF OVERPRODUCTION
104
PORTFOLIO PAISLEY SMITH & LAWRENCE PAUL YUXWELUPTUN
UNCEDED TERRITORIES
110
PORTFOLIO RACHEL ROSSIN
I CAME AND WENT AS A GHOST HAND
118
PORTFOLIO SPACE POPULAR
FREESTYLE
126
PORTFOLIO TIMUR SI-QIN
A NEW PROTOCOL VR
132
PORTFOLIO MARSHMALLOW LASER FEAST
WE LIVE IN AN OCEAN OF AIR
138
PORTFOLIO CURTIS ROTH
GESTURING ELSEWHERE
144
PORTFOLIO LUNDAHL & SEITL
SYMPHONY
152
PORTFOLIO MARCO BRAMBILLA
HEAVEN’S GATE (MEGAPLEX)
160
PORTFOLIO JACOLBY SATTERWHITE
WE ARE IN HELL WHEN WE HURT EACH OTHER
167
INTRODUCTION
THE AIV MASTER THESIS PRIZE 2019
168
AIV MASTER THESIS PRIZE 2019 HAEWOOK JEONG
PATCHED CITY
THE ARCHAEOLOGY OF THE DEBRIS
178
AIV MASTER THESIS PRIZE 2019 ARUNA ANANTA DAS
METABOLISM MUTATING NEW YORK
188
AIV MASTER THESIS PRIZE 2019 YEON JOO OH
SATURATED SPACE
DESIGN PERFORMANCE IN VIRTUAL REALITY
198
PROJECT AND IMAGE CREDITS
199
COLOPHON
CONTENTS
EDITORIAL
BREAKING GLASS:
SPATIAL FABULATIONS & OTHER TALES OF REPRESENTATION IN VIRTUAL REALITY
4
For architecture and the arts, technology is a double-edged sword. The negative effects include those that typically deliver generic standardisation, degraded aesthetic qualities and uncritical consumption. In this respect, the rate at which technology becomes adopted in the disciplines does not help, and the influx of computerised processes and mediation during the past three decades is a case in point.
The grand upside of technology, whether in the form of new ways of working, novel material systems, construction or design procedures, comprises of opportunities for formal and/or aesthetic innovation. Thus, as the technology supplying the media of Extended Reality (XR), here principally in the form of Augmented and Virtual Reality (AR and VR), is increasingly adopted in architecture and the arts, there are good reasons to ask what this entails beyond reinforcing established ways of working.
With this sixth issue in the publication series of the Städelschule Architecture Class, the SAC Journal, we address some of the history and issues that are at stake at a crucial moment for architecture and the arts.
The use of AR and VR raises questions far beyond immediate disciplinary concerns as the media are inscribed in a larger field of speculations than those raised by the technology alone. For instance, they profoundly concern the production of images, our relationship to these regardless of technology, and thus how these feed our becoming subjects under the spell of the massive, contemporary influx of information. Thus, this radical momentum pertains to the image as a building block for AR and VR, but it goes beyond the assumed centrality of sight also to include the other senses. The media prompt us to reevaluate the strict division between that which we refer to as “real” and “not real.” This “real” is given through perception, and recent findings in neurophysiology enable a new understanding of the underlying processes for sensorial experience through which we relate to the world and construct our sense of reality.
With AR and VR the experiencing subject is situated in a novel and potentially interactive relation to a partly or fully simulated environment. For architecture this entails re-inscribing the human subject in discursive and practical contexts with a rejuvenated arsenal of insights, tools and processes. This re-inscription does not restore a conventional phenomenological approach to architecture but fills that subject’s absence in the annals of recent discourse and could eventually prompt the emergence of a new formal approach to architectural design.
5
In the arts, the media open a new space for conveying art which may destabilise the art world’s dominant market-driven and institutional forms. This destabilisation is partly precipitated by the intimacy between the immersed subject and the art object as well as the interactive relation that AR and VR make possible.
All of the above is encompassed by the central and yet evasive phenomenon of virtual space that AR and VR present. Its pervasive immersive quality make these media the ultimate architectural tools. Hence, they are already generally used to present us with known spatial constructs and thus extend the existing presentation kits in architecture, the arts and the entertainment industries. However, they can also be employed to explore, research and experiment with the more intangible qualities of space. This presents us with a crucial opportunity given that we live in an extraordinary time when a global health crisis radically has constricted the spatial spheres that we previously assumed were ours and perhaps took for granted.
In sum, AR and, in particular, VR presents us with an unprecedented occasion for furthering our understanding of what space is and can be. Or, as phrased by Sanford Kwinter: ‘It completely redirects our conception of space [and allows us to] re-understand how we experience the world and design for that re-understanding.’1
The subject, the image and space comprise the thematic concerns for the work with AR and VR undertaken in the Städelschule Architecture Class since 2015. This includes both the space that architecture and art present us with but equally important the space that architects and artists occupy when they design. For both, the technologically based media establish new inroads into fundamental aspects that underlie the formation of the disciplines themselves.
As an entry into the vast realm of topics at stake herein, this issue opens with a conversation between Sanford Kwinter and Daniel Birnbaum hosted at the Städelschule in 2017.2 The conversation took place during the academy’s annual open house exhibition, Rundgang, in which the Städelschule Architecture Class presented its Third Glass, a humorous interpretation of Marcel Duchamp’s Large Glass in AR and VR. Kwinter and Birnbaum’s exchange lays bare easily neglected historical strands of the media and presents a series of concerns which could have a radical impact on architecture and the arts.
In anticipation of the increasing importance of the virtual in our lives, Fredrik Hellberg and Lara Lesmes (Space Popular) in their contribution to this publication relate the virtual and the real worlds to one another and examine our use of spatial references, architectural metaphors and analogies for the virtual. Their interest revolve around social life and the affordances that architectural elements and spaces offer. Despite the virtual’s uncanny ability to simulate the real, they insist on their differences and call for precaution as we proceed.3
In her contribution, Martine Beugnet addresses the contemporary issue of surveillance in relation to the panopticon, the immersive condition of VR, and the frameless image that VR experience is based on. With her erudite background in film theory, Beugnet elaborates details pertaining to, amongst other things, ‘the freedom of the spectator turned experienced and examines a specific project to make a case for how VR enables ‘polyphonic narration’ and evoke memory. Her contribution maps a political and cultural territory far beyond the immediate technological.
The economy that is implied in our relationship to the image regimes so central to AR and VR, is of a mixed nature given the corporate control of the technology and social media. This control exploits our shifting attention within the saturated visual field where bodily movement is subtly connected to if not regulated by the visual input. Curtis Roth examines this mixed economy and makes a case for the connection between attention,
1) As stated in conversation with the editors.
2) Kwinter’s contribution has far exceeded that of a participant as he has advised on the unfolding work for five years. The conversation in 2017 was the first of five Städelschule AR and VR related events that have featured Kwinter as a participant.
3) Space Popular also contributes a project in the portfolio section herein.
4) Beugnet’s contribution was presented in a lecture during Breaking Glass II - The Virtual Image in 2019.
6
EDITORIAL
wrist movement and the subject in relation to the image. Historically and technically anchored in how we engage with mediated imagery, his call is for resistance through creative engagement with the technologically driven media given the value of gestural movements - thus, sensory attention in immersive space.5
Returning to write for this series of publications, Michael Young tackles the quintessential architectural dyad of form and space. With great attention to detail, Young rereads Adolf Hildebrand’s seminal text The Problem of Form in the Fine Arts from 1893 and concludes that our challenge is to understand the problem of depth. He links this problem to contemporary techniques for designing, and his insightful and provocative text is an essential contribution to a renewed take on the relation between form, space and the contemporary technological image.
Addressing the role of the virtual in their own artwork, Martina Seitl and Christer Lundahl are joined by Jonatan Habib Engqvist to argue that ‘virtual reality is an ability rather than a form of technology.’ While VR is fully inscribed in the production of the artwork, it is never centrally thematised. Rather, they make a case for our organising perception into a living reality by activating our imagination and multi-facetted sensory capacity. Bodily experience and everything outside of it are no longer discrete binaries but open onto an “in-between” filled with friction and enriched experience.
Daniel Birnbaum, upon whose suggestion the engagement with AR and VR in the Städelschule Architecture Class commenced, enters into another conversation herein, this time with the philosopher Sven-Olov Wallenstein. Their conversation addresses the virtual as a phenomenon not invented with the latter-day technology. Their talk helps to situate current work in AR and VR in the context of a difficult philosophical phenomenon that at once has deep roots within the history of human thought and yet continues to be conceptually developed given contemporary technology.
Finally, with her background in philosophy and aesthetics, Fabrizia Bandi supplies the last text herein. Addressing ‘the architectural relevance of virtual reality,’ Bandi explores three avenues through which the virtual and the real relate and are not antithetical. Both realms are anchored in bodily experience and its sensory apparatus, and Bandi concludes that ‘the technology of VR enables us to experience and inhabit that architecture which is conceived, imaginable, but not yet physically manifest.’
In the second of its two sections, the first comprising of the texts briefly accounted for above, this issue presents a collection of VR projects by various architects and artists. This entails the documentation of different approaches to working with VR, and the portfolios may be enjoyed for their beauty alone. However, notwithstanding their quality, obviously no documentation based on printed images alone does justice to the profound and often provocative experience with AR or VR. Hence, the visual part of this publication can only suggest the experience of what these projects offer.
Lastly, this issue includes student projects in the Städelschule Architecture Class that were awarded the AIV Master Thesis Prize in 2019.
Breaking Glass: Spatial Fabulations and Other Tales of Representation in Virtual Reality is published on the occasion of the third and last conference in Städelschule’s series, Breaking Glass, which addresses the emerging role of AR and VR in architecture and the arts. Breaking Glass I - Virtual Reality and Subjectification in Art and Architecture took place in 2018; Breaking Glass II - The Virtual Image in 2019; and now Breaking Glass III - Virtual Space ends the series. This publication neither summarises the conferences to date nor prefaces Breaking Glass III. Rather, it is presented as a supplement to the latter and aims to give a glimpse into the vast intellectual and creative opportunities that AR and VR open up to.
5) Curtis Roth’s project, Gesturing Elsewhere, is also featured in the portfolio section of this publication.
7
INTRODUCTION
AN ECOLOGY OF
SIGNALS & THE SPATIALISED IMAGE
JOHAN BETTUM
8
“Reality” is a funny word when used in architecture and the arts. It suggests a shared, essential experience, something that cannot be disproven and exists as a fact. Conventionally, this “real” knows nothing of a notional or imaginative other.
However, whereas architectural design and the arts are devoted to the real, they are so only in the form of what does not yet exist or resists and queries “reality” as given. Each in their own way and architectural design in particular are entirely devoted to imagining what is not yet there. Whence, the business of architecture and the arts is to imagine alternative realities through invention, augmentation, critique and supplement to what is already here. Accordingly, technology that capacitates architects and artists to produce such “realities” presents the respective disciplines with a desirable and powerful tool to pursue their ends. And this is why the coming of Augmented and Virtual Reality (AR and VR) in architecture and the arts raises so many expectations and questions.
Both creative fields currently see a rapid adoption of the mediums of AR and VR for a wide variety of uses. With AR, artists can superimpose their work on any given setting through the handheld devices of their audience, likewise museums can provide their exhibited work with an associative or explanatory context, and VR offers an immersive and fully simulated theatre for narrational and compositional freedom. In architecture, AR is explored in construction to safeguard and optimise the outcome in relation to different qualities and time tables, and VR takes the clients and stakeholders in a project through the fully rendered building that is planned - all with the option to give the user realtime interaction with the computer-generated elements.
Yet, AR and VR raise questions and creative opportunities far beyond those tied to reifying established practice. These opportunities are concomitant to imagining the new, that “real” that does not yet exist.
BEYOND VERISIMILITUDE
The technology used for AR and VR is not new; it comprises of computer software and hardware which gradually emerged in the 1950s and -60s.1 During the last few years the technology and hardware have greatly improved, including that for monitors or screens as well as miniature accelerometers and gyroscopes for measuring velocity and spatial orientation. These improvements enable a tighter integration of the computer generated elements or environment with the user’s physical setting, the position of her/his body as well as visual attention within the simulated field.
1) Latter-day AR and especially VR is typically referenced to the 1960s when Morton Heilig patented a multi-sensorial immersive apparatus for experiencing films in three dimensions. From 1966 and on, Ivan Sutherland and colleagues at MIT conducted the first experiments with head-mounted displays to generate immersive experience. From then on until the 1990s, VR was mainly used in medical science, the car industry, and for flight simulations and military training. Only since the 1990s has the technology become commercially available with rapid industrial and technological development since 2010.
9
AR and VR constitute different forms or mixtures of the same technological output and can be placed on the “reality-virtuality continuum” which extends from the completely real to the completely virtual environment.2 AR typically superimposes virtual objects over a live image given by a camera, such as on a mobile device, or smart glasses. The computer generated elements blend with the physical surrounding and can be interactive. VR, on the other hand, immerses the user in a fully simulated setting with the use of a head-mounted display (HMD) or multi-projected environment. Select objects or the environment can be interactive. Whereas AR always combines an image of the physical environment with virtual elements, which means that AR is near synonymous with so-called Mixed Reality (MR),3 VR can align the physical and the virtual environments through technology for positional tracking and highly detailed and accurate three-dimensional, computer generated models. The dominating sensory information conveyed by AR and VR is visual, yet auditory, including dynamic binaural input, as well as other types of sensory feedback with the use of haptic technology are possible.
The two mediums and especially VR are used in a wide range of fields, obviously by the entertainment industry, but also in education and - in social sciences, medicine and psychology, for select forms of therapy. In many if not most of these applications, the central issue is the systems’ capacity to generate verisimilitude with the known physical world. In other words, the success of the application depends on the degree to which the simulated environment appears true in reference to the appearance and/or behaviour of the physical environment.
For architectural design and the arts, however, novelty and invention - and therefore imaginative and critical projection beyond the horizon of the known - are essential and ingrained in the disciplines’ respective histories. This reflects how the phenomena of the “virtual” and the “augmented” are themselves not tied to technology.4 They present us with experiential dimensions that include the possible and supplemental to what we think we know and want to trust as they present us with a sometimes seductive, sometimes subtle and sometimes provocative disruption of the fine yet nervous economy of lived experience.
The idea of a “reality-virtuality continuum” suggests a fluid movement between the “real” and the “virtual” and therefore a possible flattening of different streams of sensory input. While this may be neurologically true, it also lessens the hypothetical productive tension between the physical and the digital, the “real” and the “virtual.” This tension is lodged in the corporeal presence of the user or experiencer. While the technology can be seen as a prosthetic extension to that presence, the difference between them must be maintained as a disjunctive force in their productive exchange.5
It is too easy to relegate technology to known regimes of thought and submit it to service existing systems of production. AR and VR in architecture and the arts are precisely at this crossroads: They may reify the existing or set free vast opportunities for the disciplines and what they produce. It is the human subject that occupies this juncture, not the technology; it is the human subject with its sensory capabilities that facilitates both that which we refer to as “real” and “virtual.”
AN ECOLOGY OF SIGNALS
Insofar as AR and VR concern a digitally produced realm, the user of these media constitute a physical one. This foregrounds the human subject rather than the technol-ogy, the experiencer’s perceptional capacity, imagination and creativity rather than the computational capacity to simulate that which we routinely know. It also ushers in an unprecedented intimacy between the technology, its hardware or devices, and the immersed subject - an intimacy that is predicated on an ecology of signals in the immer-
2) The “reality-virtuality continuum” was formulated by the engineer Paul Milgram and his colleagues in 1994. Extended Reality (XR) denotes this entire spectrum between the real and the virtual.
3) Mixed Reality refers to a hybrid version of the real and the virtual, comprising of both physical and digital objects that can interact in real-time.
4) The playwright Antonin Artaud is typically invoked to demonstrate this. Already in 1938 Artaud used the term ‘virtual reality’ in his book, The Theatre and Its Double.
5) The notion of a disjunctive force is loosely taken from Gilles Deleuze and Felix Guattari’s description of a ‘productive synthesis.’ See: Gilles Deleuze and Félix Guattari. Anti-Oedipus: Capitalism and Schizophrenia, trans. Robert Hurley, Mark Seem, and Helen R. Lane (Minneapolis: University of Minnesota Press, 1983), 75-76.
10
JOHAN BETTUM INTRODUCTION
sive environment. These signals traverse the boundary between the machinic and the corporeal, the digital and the physical.
The centrality of the immersed subject with regard to architecture and spatial perception has been observed by the philosopher Elizabeth Grosz: ‘[T]he body … is already there, albeit shrouded in latency or virtuality. Bodies […] reain architecture’s unspoken condition […].’6 Elsewhere she elaborates to suggest that possible spaces arise from our corporeal presence:
‘I would contend that space and time are not, as Kant suggests, a priori mental or conceptual categories that precondition and make possible our concepts; rather, they are a priori corporeal categories, whose precise features and idiosyncrasies parallel the cultural and historical specificities of bodies […]. The limits of possible spaces are the limits of possible modes of corporeality: the body’s infinite pliability is a measure of the infinite plasticity of the spatiotemporal universe in which it is housed and through which bodies become real, are lived, and have effects.’7
Space, then, is not a pre-given and fixed entity but continuously constructed by the inhabiting subject. Being in space is to be in constant exchange with the environment, and critical advances in neurophysiology suggest that our sensing the world is far more malleable than previously thought. The plasticity of the brain and neural processes handle input from the surroundings in a far more complex, intricate and interrelated manner than what was previously thought.
This challenges the idea that we relate to the world as something entirely external to ourselves. Based on research on neural processes and modifiable synapses in the complex network of the brain, the neurophysiologist Wolf Singer describes a continuous negotiation between previous experiences and sensory input as incessant neural labour. His description of ‘cross modal integration’ accounts for multiple sensory sources and inputs being neurally and cerebrally negotiated to produce our sense of “reality.”8
This malleability and multiplicity in neurophysiological processes break down the clear distinction between the self and the space it inhabits. It expresses a dynamic relationship between the subject and its surroundings where physical movement is an integral component: Pulses, shifts and turns engender perpetual change, shifts in visual attention and an experience of depth.
There are different ways to structure the relationship between the ingredients at play in this drama. One is to think of the technology supplying the immersive simulation as a transactional realm between inner and outer “realities.” VR is then a third transitory realm between inner experience and outer setting, echoing how the British psychoanalyst Donald Winnicott spoke - not of Virtual Reality - but of an ‘area of individual development and experience’ given by ‘transitional objects [and] phenomena.’ His work was in part dedicated to conceptualise this transitional space, an ‘intermediate area of experience […] between primary activity and projection of what has already been introjected, …’9.
Winnicott spoke of ‘precursors,’ which echoes the term ‘priors’ used in contemporary neurophysiology, and thought that both inner reality and external life contribute to this third realm. Addressing inter-personal relationships, in particular that between mother and child, he was concerned with ‘… the intermediate area between the subjective and that which is objectively perceived.’10 The French philosopher Bernard Stiegler has said of Winnicott’s third space: ‘In this beyond or beneath of both the exterior and the interior, there is something that holds between the mother and her child, and which nevertheless does not exist. What takes hold between the mother and child in not existing, but in passing through the transitional object, and which therefore finds itself con-
6) Elizabeth Grosz, “Embodying Space: An Interview”, in: Architecture From the Outside: Essays on Virtual and Real Space, (Cambridge: The MIT Press, 2001), 13-14.
7) Ibid., 32-33.
8) Wolf Singer, “The Constructivist Nature of Perception: Why Virtual Reality Works”, in his presentation at Breaking Glass I - Virtual Reality and Subjectification in Art and Architecture (Städelschule, Frankfurt am Main, 2018).
9) Donald Woods Winnicott, Playing and Reality (London: Routledge, 2005), 2-3.
10) Ibid, 4. Winnicott goes on to suggests that ‘it is … only in playing [and] being creative that the individual discovers the self.’, 72-74
11
stituted by it, links and attaches them to one another through a wonderful relationship: a relation of love, of amour fou [crazy love].’11
The transitional object is a node of convergence and transactions, much like corporeal presence is a node for transactions between the virtual and the real. Yet, invoking the human body as a central node in these economies of exchange and experience is not concomitant with a reductive phenomenological approach that singles out individual experience and elevates individual perception to a philosophical phenomenon.
We are faced with a technical question that partly centres on the nature and serial structure of signals. These signals are given by digital technology as much as by neurological processes. They travel between the digital and the physical, the machine and the human, and infuse the space defined by technology and flesh with productive tension. An open-ended exchange between technological, corporeal and cultural systems ensues. The question then becomes: other than minuscule electrical pulses and chemical discharges across small differentials in molecular distribution, what is their shared currency?
THE SPATIALISED IMAGE
There are signals everywhere, and when gathered, serialised and directed, they engender images.12 In AR and particularly VR, this signal-based status of the image is pervasive. While AR drapes its simulated elements over images of our surroundings, the space of VR is all image. It wraps itself 360 degrees around the immersed subject whose visual experience at any moment is only limited by bodily orientation and the visual cone. When immersed, we visually occupy this image; it has become spatialised and inhabitable.
Already more than a decade ago, the art historian and media theoretician Oliver Grau noted: ‘We are witnessing the transformation of the image into a computer-generated, virtual, and spatial entity that seemingly is capable of changing “autonomously” and representing a lifelike, visual sensory sphere. Interactive media are changing our perception and concept of the image in the direction of a space for multi-sensory, interactive experience with a temporal dimension.’13
The implications of this idea of the image as a spatial entity are formidable and open up for new conceptual paths and formal inventions. Borrowing from Edmond Couchot, we can imagine how this image no longer presents the subject with a fixed viewpoint, such as given in perspectival construction. Subjectivity can be conceptualised as spatially distributed,14 which in turn reflects the distribution of machinic and neurological signals and their production of images. The interactive capacity embedded in these images make them ‘autonomous visual artefacts capable of reacting to the gestures of [the immersed user and] reinstate the decisive importance of the body, in all its mysterious complexity, at the core of aesthetic relations. […] An unprecedented artistic situation springs from the choreographic interaction between real and virtual beings.’15
Gone is the opportunity for thinking in terms of “this and that,” in binary oppositional terms. A new construct must be invented for the relation between subject and object, between form and space. Signal, calculation, image, attention, incentive and action are one productive whole and contribute to the intimate, unfolding choreography in the image-space.
Yet, we should not be surprised by the centrality of the image in this perceptional economy. Consider, for instance, its role in Henri Bergson’s writing when he more than a hundred years ago addressed perception and memory: