A playful immersive installation, ‘Only A Game (?)’ is an interactive game experience where audiences can use their body movements to offset the gradual increase of global warming and co-create generative art in the process.
The text is split into two sections: the first, is a dialogue between Tanya Saxena, a dancer and choreographer from India, and Fabian Raith, a media artist and theatre-maker from Germany; and the second section is a comprehensive reflection by Pritha, a movement and visual artist from India.
Fabian & Tanya: Conversation, Collaboration, and AI
Tanya: Before attending the BeFantastic Fellowship I knew very little about TechArt. Now, four months later, I find myself in the throws of an ambitious project combining movement, sound, and technology. A sense of irony and wonder pervades my writing, whoever reading this is forewarned!
As someone who takes great joy in moving, anything that can inspire others with a similar joy is exciting to me. What I wasn’t prepared for was that my involvement in the project would challenge me to collaborate with the most unexpected colleague — Artificial Intelligence.
Fabian: I started to work with technology a few years ago but I would still describe myself as a performing artist. For me, it was very interesting to combine technology and physical performances and during the fellowship, I learned a lot about dance and movement.
Tanya: Almost as soon as Pritha proposed the idea of designing a game that responded to movement, I knew that I wanted to work on it. As someone who takes great joy in moving, anything that can inspire others with a similar joy is exciting to me. What I wasn’t prepared for was that my involvement in the project would challenge me to collaborate with the most unexpected colleague — Artificial Intelligence.
Tanya: In essence, our project involves designing a game where the body acts as the controller, and the game environment changes according to a pre-designed almanac of movements. The game is based on rendering climate action and prompts the player(s) to try and save the Earth by combating rising temperatures and water levels. My role in the project involves designing the almanac of movements in a way that feels intuitive and inclusive to a wide range of players. This involves two aspects:
- Researching global warming and actions that we can take to combat them, subsequently translating these climate actions into movements; and
- Experimenting with motion tracking to understand how the AI responds to movement
The way the AI responds to movement and pose recognition is sometimes so unexpected; it comes as a complete surprise. Whereas initially, this would throw me off, I am beginning to enjoy these surprises. It is in this chasm between humans and AI where I feel the possibility and excitement of creation lies.
For the purpose of the game design, we have been working with pose tracking and I have been experimenting with Teachable Machine. For me, this is really where things got interesting. As movement artists, when we create or collaborate, we tend to rely on devised processes. So we take an idea, a starting point, and keep moving and experimenting to see what kind of shape emerges, this is a largely intuitive process. When working with AI, however, that sense of intuition is challenged in a completely different way. The way the AI responds to movement and pose recognition is sometimes so unexpected; it comes as a complete surprise. Whereas initially, this would throw me off, I am beginning to enjoy these surprises. It is in this chasm between humans and AI where I feel the possibility and excitement of creation lies. I am still continuing to explore this relationship, and (perhaps indulgently) foresee myself spending quite a bit of time playing around with this, while simultaneously continuing to read about climate change.
Fabian: At the same time, the AI model can also be limiting in a way. We hand our perception of the world to a machine, which is not able to see movement the way we do. It is less sensual; it reduces the movement to re-detectable poses and is not really able to pick up on minute details. The communication with it is always one-sided: we provide the input, and the AI generates the output.
Tanya: One aspect of climate change that has particularly stood out for me over my research is its close connection with social justice. Climate change affects everyone, but often the people who are most severely impacted by it are the ones who have contributed to it the least. In every report, article or video I watch this fact becomes more evident. As someone who lives in a tropical, developing country I can see this imbalance in climate justice affecting me and those around me. This is an aspect that we have been finding ways to incorporate into our game through movement and through the game design itself.
We hand our perception of the world to a machine, which is not able to see movement the way we do. It is less sensual; it reduces the movement to re-detectable poses and is not really able to pick up on minute details. The communication with it is always one-sided: we provide the input, and the AI generates the output.
Fabian: Yes, this is also very weird to me. We see devastating images from all around the world of increasingly worse floods, for instance. Coming from Germany, one of the richest and also leading industrial countries, I can see that it is already facing drastic changes in its climate, yet the conversation on climate is still focused on dry summers, and so little on transformation. Reality is hitting hard, yet old imaginations still survive. In some sense, it is metaphorically like an AI: we use existing imageries from the past to make predictions of a tomorrow, yet our tomorrow is not shaped by yesterday, but by the current realities.
Tiz, Fabian and Tanya in an online meeting
Tanya: It has also been fascinating to see and learn from my team – Pritha, Tiz, Fabian and David. At our very core, we are all artists. However, because we are all used to working with different disciplines when we meet we bring to the table many different ways of approaching the creative process. I’ve enjoyed seeing how each of them works/thinks about their work, and I am quite grateful for their kindness and patience.
Fabian: We are a team of five people, each one of us with very different knowledge about technology, digitality, movement and performing arts. An important aspect of working on this project was to connect the different ways we approach these aspects while making sense of our experiences. So, even as we are working on the piece, we are still figuring out our own ways of working and sharing the workload among us.
Reality is hitting hard, yet old imaginations still survive. In some sense, it is metaphorically like an AI: we use existing imageries from the past to make predictions of a tomorrow, yet our tomorrow is not shaped by yesterday, but by the current realities.
Pritha: Looking Back and Reflecting
I applied to the BeFantastic Within Fellowship at a time in my artistic practice when I was seeking creative intersections of all things that drive me — creative coding, tech, ecology, movement arts, visual arts, community engagement, and pedagogy. It was around March 2022 and almost two years of having confined my movement practice, photography, and filmmaking to the compound of my house and not having any human collaborators in the same physical space had pushed me to consider the idea of working with the only accessible collaborator: my computer.
During that time I learnt how to build AI-driven, interactive programs which would dance with me and create visuals with me. It was also a time when I truly started appreciating how powerful embodied experiences can be to connect with the world around us and empathize with the beings that inhabit it. It was a time when I began to consider how technology can have a critical role in creating empathetic, equitable, accessible spaces where communities can connect and find platforms to represent their narratives and know that of others.
At the time, for me, two major thoughts were driving my practice:
- How art installations/performances mostly have pre-determined outcomes and trajectories and how it is something a member of the audience only experiences in observation. I was working on creating spaces where artists share their practice with the audience with the intent of inviting them to take an active part in the creation and in the process, understand the art. For me, embodied/somatic practices are extremely powerful and I wanted to build a space where everyone who steps into it gets to experience that.
- How inaccessible artistic and scientific practices are to those who are not “artists” or “scientists”, how much arts, science, and tech are considered to be different from one another, and how very pertinent, influential, and all-pervasive arts-sci-tech are in our lives. I feel if we had a way to engage with it more and understand why the world around us is the way it is (in the context of changing climate, for example), we’d understand our role in it so much more, dispelling the indifference or disinterest that exists in our minds about it.
All of these thoughts mashed and moulded into the idea for Only a Game (?) when I first pitched it to the others. I knew from the start that its scope stretched beyond what could be accomplished during the fellowship. However, I was eager to kickstart the process – in its simplest version – at the very least. And then came along these four incredible artists and technologists who actually wanted to build this together!
I began to consider how technology can have a critical role in creating empathetic, equitable, accessible spaces where communities can connect and find platforms to represent their narratives and know that of others.
The discussions, dialogs, mentor sessions, and every other resource BeFantastic curated for us in the course of this fellowship have been instrumental for us in understanding the scope, limitations, and slippery slopes of developing this project within the context of AI, performing arts, and climate action.
Tanya and Fabian have delved into some of the technical challenges we are encountering so I’d like to write about some conceptual roadblocks and insights we have had. I feel that having five different minds and their diverse perspectives has helped us conceptualize the goals, trajectory, and outcome of the game to reflect what was in our minds as closely as possible. Often, when I have something I’d like to add but have no words to describe it, Fabian would know of a software that could do it! We can rely on David’s input to help us create an immersive soundscape for the game where, I for one, would have been quite lost. Tanya and I share some similarities in our creative vocabulary – in that we both come from backgrounds that involve Bharatanatyam – and are enthusiastic to start working on figuring out what kind of movements can be developed for the gameplay. Tiz has been working on developing the details of the gameplay and the user experience.
One of the things we would like to pay close attention to at this stage of the process is the congruence of the movements with the narrative, the journey we build, and what thoughts we leave the participants with. For me, that involves a mix of research on existing games, climate narratives, hours in a dance studio, and generally thinking about stage design and lighting! Right now, a lot of my work involves transcribing and organizing notes I make into concrete ideas so that the others on the team really know what I have been up to.
This has also been a time when each of us has had unavoidable commitments that keep dragging us away from this work – no matter how much we would want to stick to our schedules! Having four other people on the team ready to pick up the slack is incredibly reassuring.