CAPES Fellowship: Reflections on time spent embedded at KDL and CMCI in 2024
Between the months of May and October of 2024, I had the valuable opportunity to develop part of my PhD thesis as a visiting post-graduate researcher at King’s College London (KCL), through a scholarship granted by the Doctorate Sandwich Program from CAPES, an agency from the Brazilian Ministry of Education. Under the guidance of Dr. Stephanie Janes, from the Department of Culture, Media and Creative Industries, and supported by Neil Jakeman, and the King’s Digital Lab (KDL) team under the direction of Dr.Arianna Ciula, I was able to conduct activities that immensely benefited the final, analytical phase of my research, originally based in the post-graduate course of design of the Faculty of Architecture, Urbanism and Design of University of São Paulo - FAU USP.
Supervised by Prof. Giselle Beiguelman, my doctoral research focuses on the investigation of spatial aspects derived from immersive experiences, provided by the advances in audiovisual and computational technologies of extended reality (XR). Identified as a recent multidisciplinary field under gradual development, the research aims to organize a framework of design elements and practices for XR experiences, that can assist in activating spaces and creating social relations, while at the same time analyzing and stimulating the possibilities of autonomous creation of such immersive experiences, not only as a way of contributing to the growing field of Digital Humanities, but also for it to be understood as a form of decolonial practice in response to the massively concentrated development of these technologies by Big Tech enterprises and their opaque agendas.
The research is based on conceptual and practical investigation through design in the recent context of spatial computing developments, starting from the first historical manifestations of technical images used to represent tridimensionality in bidimensional surfaces, and reaching the contemporary moment of a global immersion inside ubiquitous computing, allowed by the extensive use of smartphones in a timespan of less than 20 years. This rapid development triggered a progressively stronger dependence of the population on them for executing everyday tasks, and a notable shift in our spatial and social behaviours. In this sense, it might be said that we are witnessing the early stage of a transition process, from the use of portable handheld devices to their gradual substitution by optical media, a path that can be identified by a growing development and production of visors, such as mixed reality (MR) headsets and augmented reality (AR) glasses, predominantly produced by the main Big Tech companies of the global north. This production is followed by the continuous launch of new spatial computing experiences, from games and entertainment to day-by-day applications, all composed with virtual, animated and interactive 3D modelled elements.
In order to foster a deeper understanding of the conceptual, technical and sensitive elements necessary to design such immersive (XR) experiences, a more practical part of the research was developed under the methodology called Research through Design, characterised by the activities of conceptualizing and coordinating both the design and development processes of a series of different immersive experiences, that explored different modalities of XR design, that were effectively implemented in the city of São Paulo. As the final part of the design framework developed under this methodology, it was important that feedback could be collected from the users of these experiences, in order to validate the success of the design elements created and implemented, and to gather beneficial information for their improvement.
This final part of the research informed the planning of activities that was executed at KCL during the period of my scholarship and, among the experiences implemented, the following projects were selected to be reproduced, play-tested and later analyzed during the workshop sessions held:
-
The Instagram City: a VR experience for smartphones adapted into headsets by cardboard visors, transforming the addictive action of doom-scrolling into a spatial experience;
-
Playing With Trust: a modular game structure with AR interface designed during the pandemic isolation period in order to reestablish social interactions in public spaces, based on a human game board and a set of original ludic activities especially designed for it; and,
-
demonumenta: a two-year multidisciplinary research held at FAU USP, involving graduate students and post-graduate researchers, that investigated the history of a series of public monuments from the city of São Paulo, in order to resignify them through the lenses of decoloniality.
Originally implemented as an exhibition for a public square in São Paulo with an AR gamified interface, the project demonumenta presented the original 3D remodelling of 22 monuments that could be virtually instantiated in space and tri-dimensionally experienced through the smartphone screen with an audio guide containing revised narratives of their origins and meanings through time. In order for this project to be experienced in the local context of London through a new version of the application, three chosen local monuments were selected to be problematized, for paying homage to public figures that have aspects of their history related to slavery.
In addition to having the monuments remodelled in 3D virtual versions for the immersive experience, this version featured an amazing collaboration with the King’s Past project, coordinated by Dr.Érika Melek Delgado from the Department of History, and carried out by a group of enthusiastic students who helped researching and creating the original texts related to these monuments, and also recorded the audio guides. Besides the AR application, the same original material created by this research, could be turned into an additional experience, called demonumentaMR: a mixed reality experience for a Meta Quest 3 headset, that was kindly lent by KDL, in which the user could experience the 3D reconstitution of the monuments by walking around them.
With the supervision of Prof. Stephanie Janes, the kind support of Dr.Gabriele Salciute Civiliene from Department of Digital Humanities, and valuable assistance of student Jamila Lui, a whole week in the month of September was dedicated to XR playtest sessions with volunteers, and it was organized based on the outstanding technological infrastructure of Macadam 3.1 room inside the Strand Campus, through a layout designed to create separate zones where the participants could test the different spatial experiences one at a time. Through an open call for participants released to the King’s community, each playtest session included an average of 6 people, all with different backgrounds and previous experiences with the technologies presented, that individually tested a sequence with all the different XR immersive activities, followed by an online survey with linear scale questions for collecting quantitative data. The end of each session day was reserved for focal group activities, during which the concept and design process of each experience could be explained with more detail, followed by collective discussions where they could give provide feedback with their thoughts, critiques and suggestions for improvements.
During the playtest sessions, all the participants were also equipped with Sony Mocopi motion capture sensors, through which we could collect their physical movements in tri-dimensional space, making it possible to also analyze the behaviour of their bodies in space, and their relation to the real and virtual elements designed for each experience. This allowed for an unprecedented perspective that added great value to my research, and from which the sensibility of a designer of XR immersive experiences could immensely benefit. Beyond the practice of defining and giving shape to the environmental and objectual XR elements, either physical or virtual, the visualization of the body movement through space may reveal itself as the actual central chore of the design process, in such way that the user’s movement during the experience could be understood as choreography that could be predicted, controlled and conducted. From this perspective, what before could only be understood as an empty or negative space, can now be transformed, through the spatial and geometrical aspects of the body, into a design object itself, and become the main reference for a design starting point, from which all the other elements should be derived, and thus introducing the possibility of a new design process.
I’m very thankful for all the King’s Digital Lab team for being so kind, receiving and interested in following up the progress of the activities that I developed during my period as a visiting researcher at KCL. They systematically created room for presentations and conversations, and were always open for suggestions, clarifying technical doubts, helping to come up with smart ways of solving problems, and providing easy access to their facilities and electronic equipment. It was an immensely great experience, that I sincerely wish that more lucky people could be benefited from it as much as I was.
Footnote
It was a real pleasure for us to have the opportunity to host Luis last year and to learn so much from his research. He produced some wonderfully realised visualisations with enviable impact value and he was always willing to share his experiences through seminars and in informal settings and his contributions enriched our debates about the many and varied uses of XR, and he continues to enrich that dialogue from back in São Paulo.
The KDL Team