Ircam Forum Workshops

Moving Sound Pictures: Hommage

Content creation for art mediation through  

VR technologies

Presented by : Konstantina Orlandatou

Biography 

Abstract

Hommage is an interactive VR installation in which three artworks have been adapted to a virtual three-dimensional environment. Explore Dali’s living room inspired by Mae West’s face, interact with Matisse’s spray of leaves, and play with Picasso’s mandolin and guitar. “Hommage” is a tribute to friendship, common respect, and admiration between artists, who are milestones in art history of the 20th century. Users are invited to explore artworks in a 3D space by interacting with the objects of these artworks. Through interaction music emerges and the artworks become musical instruments for the user. The installation is part of the Moving Sound Pictures project which mission is the usage of VR technologies for art mediation and transfer of knowledge.

INTRODUCTION

In the past years the development of software and hardware in the area of XR technologies has rapidly blossomed opening incredible possibilities in different fields. Especially VR has become a permanent feature for immersive learning experiences providing, in many fields, the possibility of recreating real life settings and simulations of work challenges. This has been particularly beneficial in areas that deal with purpose-created scenarios and crisis management, hands-on experiences, skills analysis and decision making, and remote training. Why not then to use VR technologies for art mediation and education?
From the perspective of a classical composer time is an inevitable element of music. In visual arts time doesn’t exist. The painter doesn’t tell a story in a strict time line but rather catches a glimpse of a moment in a frame. What would it be if paintings become alive and come out of their two-dimensional space? What if Kandinsky could make music with his circles, lines and triangles? What if Mondrian could come out of the canvas’ frame?
All these questions are dealt in the project Moving Sound Pictures [1]; a project in which users can interactively explore paintings by famous and contemporary visual artists through playful actions using VR technologies. Paintings are transformed into three-dimensional spaces in which the user has the opportunity to touch, move or enlarge the objects in the painting and to create music through these actions. Additionally, the user gets information about the artwork and its history. While the user interacts with the VR environment, the transfer of knowledge takes place on this virtual stage at the same time. Furthermore the user has the opportunity to discover the artwork from another perspective in a unique immersive experience. These imaginary worlds are based on how I see the artworks and try to add my artistic interpretation creating a connection between visual arts and music through (visual) story-telling. 
In a previous VR environment, entitled The Abstract Painters [2], four paintings of the Abstract Era between 1916 - 1924 by Kasimir Malevich, El Lissitzky, Piet Mondrian and Wassily Kandinsky were transferred into a VR environment and dressed up musically. The user enters a gallery designed in the style of the 1920s, with a wallpaper in wine red color and a shiny brown wooden floor. The paintings hang on the walls like in an exhibition and act as gateways into the walk-in virtual rooms: in the VR version of Malevich’s painting Suprematism you can make the objects sound with two mallets, in the VR version of Kandinsky’s Merry Structure you can touch the objects with your hands and thus create music. In the main room of this virtual gallery the user can get – through voice-over in Englisch, German or Chinese language – information about the painting and the painter.
Following the same paradigm, the interactive VR environment Hommage hosts three artworks from painters who are considered to be milestones of the 20^th century and more specifically: The Face of May West (1934-34) by Salvador Dali, Spray of Leaves (1953) by Henri Matisse and Mandolin and Guitar (1924) by Pablo Picasso.  Hommage is literally a tribute to friendship, common respect, and admiration between these artists.

Content Creation & Development

• Creative content

Reading art history books many would notice that throughout the years many relationships (friendships or rivalries) have been established between famous artists. Salvador Dali was fascinated by the actress and sex icon Mae West. Henri Matisse felt at the beginning threatened by the younger painter Pablo Picasso but as time passed a creative rivalry developed into a respectful artistic relationship between them.  What would it be, if their paintings could be combined in one interactive space?
A big three-dimensional space (approximately 100m²) of an interesting architecture serves as an apartment. The starting point is a corridor where photos of the paintings and their titles are hanging on the walls. From there the user has a view to the living room which is a 3D reproduction of Dali’s The face of Mae West. The corridor leads to the main large room where the artworks - now transformed as three-dimensional reconstructions - are placed as parts of an apartment. However these parts are not separated from each other through wall but rather the one artwork overlaps with the other.
There are three (3) different areas, one for each artwork, where different interactions can happen. Dali’s The Face of Mae West is set to middle of the room, representing a living room, where the user can “touch” the sofa or play percussion with the clock positioned on the top of a furniture, which resembles a nose.  Matisses’s oversized leaves have been positioned on the right side of the room. Here the colored leaves have been placed on the floor and by going through the leaves, the leaves start twisting and swinging producing different alternating sounds. On the left side of the room, one can take in his/her hands and play with Picasso’s dissonant Mandolin and Guitar.
The music and the sound designed have been inspired by the colors and the size of the objects based on the knowledge gained by cross modal correspondences and sound-color synaesthesia. Among others brighter colors are connected to darker sounds or big objects are assigned to low-pitched sounds.

• Technical development

When developing an interactive VR installation there are many aspects one should consider. Since the project serves the transfer of knowledge for a wide audience (ideally in a museum where the analog meets the digital  artwork accessible to audiences of any age) the environments have to be designed in such a way that they are appropriate for small children up to elderly people or people with handicap – so all visitors of a museum.
Through storytelling and visual cues Hommage has been intuitively designed. Grabbable objects are placed in a middle height, so to be reachable by small children, and divers type of interaction is possible by pressing one button or trigger of a controller at a time.
For the installation, visuals and graphics play a very important role for the perception of the experience. Therefore the whole virtual environment, graphics, lightening and interactions have been developed using Unreal Engine. Raw material for the music and the sound design has been composed and produced outside the engine (using recordings, Cubase or Max/MSP etc), however sound implementation is developed using FMOD and the FMOD plug-in for the engine.
The environment/installation is walkable and can be explored through a VR headset/toolkit. It is developed as a PCVR version and appropriate both for MetaQuest 3 or HTC Vive Pro 2/Focus headsets.

CONCLUSION

Hommage is a VR installation dedicated to visual artists who have contributed in an exceptional and unique way. With the usage of VR technology artworks are presented in a total different way combining music and interaction to mediate art from another perspective. Innovative content VR creation and technology can lead to a unique individualistic immersive arts experience for museums and institutions.