Give Me A Sign: Developing The Narrative

Give Me a Sign combines aspects of Indian philosophy and practices with narratives around Artificial Intelligence and climate change, through an illusionary AI entity called ‘Shunya’ who responds to mudras with visions, dreams, stories, concerns and hopes they have about our changing planet.

Artists: Diane Edwards(UK) , Upasana Nattoji Roy (India)
Supported By: Hasan S

Intention

We wish to highlight the significance and power of the physical gesture in expressing meaning with relation to our physical, living environment by repurposing mudras to reconnect with nature and convey the climate concerns we face. Highlighting the granular, tactile, and organic disconnect that purely virtual experiences create, we aim to not only build a bridge between human and machine, but also to build a bridge between cultures.  Gestures or mudras are powerful tools in telling stories,  they go beyond verbal communication,  opening up to a more emotive, intuitive, inner and even spiritual space;  a form of expression that an AI system may never fully decipher.  Although technological advancements and AI may help us deal with aspects of climate change, it is essential that we do not disregard other forms of knowledge and practices, which may aid in rethinking our way through the impending climate disaster. We envision this work as a form of acceptance, a tool to think through new imaginaries. 

Climate & Human Impact

The narrative is focused around human accelerated climate change. Shunya manifests as a fantastical digital entity, whose insights provide an alternative lens in challenging our understanding of climate change. Keeping in mind the tradition of science fiction it can be useful to think about such difficult and overwhelming situations through illusionary means, probing into and confronting our insecurities of what change is coming and how might we live in this new and potentially hostile environment.

Story Flow

We are continuing to develop our script for the character of Shunya. When an audience member selects and presents a mudra, one out of three outputs will appear on a screen in the form of a short animated video, accompanied by sound and text. Each of these clips will address a certain theme which branches from the original meaning of the Indian dance mudra. Shunya will respond with a statement, philosophy or a leading question which we are scripting in parts collaborating with ChatGPT3.

Interaction Diagram

We envision this work as a form of acceptance, a tool to think through new imaginaries. 

Mudras & the Theme of Climate Action

We have been researching and interpreting the meaning of the mudras further by using them as tools to explore narratives and urgent conversations around possible futures and the climate crisis.

Similar to our first prototype we are using 6 mudras to converse with Shunya:

Curated Mudras & Themes

  • Mookoola: This sign is interpreted as the bud of a flower. It can also mean to demonstrate the action of eating, or used to show the navel of a human, animal or God. Here the navel is meant to represent the connection of life — the umbilical cord. 
  • Prithvi or Mayura: Mayura means ‘Peacock’ in indian dance. And in yoga the same mudra is called Prithvi, which denotes the Earth.
  • Shikara: Shikhara is the human element.  It translates into the word ‘peak’. The peak of a mountain, the peak of civilization, the top of the food chain, man or the masculine, male Gods, Shiva, leader of the tribe/ pack / society, a phallic symbol.
  • Soochi: Soochi literally means ‘needle’, it can be used to indicate or point to something or symbolise a tool or technology. 
  • Mushthi: This sign takes the form of a fist, which can represent power, grasping at something stolen or holding a weapon. 
  • Pataka: The Pataka mudra means’ flag’, it can be used as a sign to stop,  or as a blessing. In dance depending on the way it is performed it can denote the sea, forest, clouds, river, wind, shield, among many others. 

Interaction

This project is a successor from an earlier prototype where we used Google’s Teachable Machine. The key problem with the first version that we wanted to address was the issue of running the project successfully no matter who is sitting in front of the camera, what they are wearing, and their surroundings. 

For this new iteration of Give Me a Sign we have been learning and experimenting with more robust gesture recognition Machine Learning techniques. We are exploring the possibility of hand pose recognition, where instead of the computer classifying images based on matching pixels, it can detect the position of a hand. 

Animating GIF : MediaPipeHand

Interaction using media pipe hand pose

The hand is broken up into ‘landmark’ points which the Machine Learning model can be trained on, mapping the arrangement of these points to our desired triggered outputs such as sound or animation. We began using libraries such as the Tensorflow/MediaPipe Handpose Model (which we found to be a better version of the ML5 Handpose model), p5.js, and MI5 Neural Network. The project needed to function fully offline, while running video and real-time skeletal hand recognition. 

Instead of the computer classifying images based on matching pixels, it can detect the position of a hand.

We want the user to have an experience beyond the basic human computer interaction, generating the perception of communication and collaboration with the machine. By feeding it a gestural meaning and receiving a unique response, that builds a conversation between the user and the machine; a story that can be narrated depending on which gesture is presented and in which order.

User Interface: Holding Page

The holding page is a key element of the project, acting as a User Interface (UI) for interaction. Our speculative/fantastical character Shunya, is one that is learning from us and sharing their knowledge back with the user.

Holding Page Design Language Exploration

Currently we are aiming for this holding page to have an animated hand that cyclically runs through our six selected gestures, until an audience member is prompted to choose a gesture to hold up to the webcam, thus activating Shunya. 

Visual Language

Visual language is the key to communicating the key themes of each of the selected Mudras meanings in relation to the climate change narrative.

Visual language_Tobias_Bjork

The elements we are designing:

  • 3D Character 
  • Mask 
  • Environment – stable diffusion, void, fog, after effects
  • Video transitions

The aim is to experience beyond the basic human-computer interaction, generating the perception of communication and collaboration with the machine.

Masks

We are designing 3D mask models for each of gesture, reflecting on the meaning and message we wish to convey with the gesture. 

Mask Designs

For instance, Prithvi (Earth) can correspond to ecological awareness, which corresponds to entanglement, vines, roots, and mycelium.

Sketches

Early sketch of a mask which could be modelled and used for the gesture, Prithvi.

Shunya sketch 1

Instead of a human head – deconstructing our theme into elements that evoke them and using those to construct Shunya. 

Character Design

Early experiments creating our bespoke character within Unreal Engine’s Metahuman. We have decided not to go in this direction. 

Shunya Unreal design test

Shunya Mask and Hands sketch

Layout Mockup

Instead of a full humanoid figure we will have an invisible body and head, with 6 alternate masks and corresponding floating hands. Text will also appear on the screen, which matches the statements, philosophy and questions which will be audible. 

Hands

For instance Prithvi (Earth) can correspond to ecological awareness, which corresponds to entanglement, vines, roots, and mycelium.

For the holding page we are planning on using a wireframe single hand. Shunya however will have hands which are textured to correspond to the aesthetic of their mask. 

Hands 3D Mesh

Shunya Unreal Hands Tests

Unreal arms and texture map

Deepmotion

We are using Deepmotion to animate our character. Shunya will perform the mudra that the audience member has presented, and the way the mudra is performed takes inspiration from Indian dance and other more contemporary uses.

Deepmotion test for holding page mudras

Sound

We are working with a sound designer, bringing together a mixture of traditional indian instruments with a more ambient synth spacey soundscape, and layered spoken word elements.  

Installation Design

Mock up for BIC

Mock up for BIC 2

Portable mockup for HOME Manchester

Early iterations of the mockup

Challenges

Having such a small team has been a challenge, it has been difficult to balance the technical aspects of the project alongside the more artistic and design elements. We now have Hasan on board to help us complete the programming, which has allowed us to really get sucked into the visuals, narrative and storytelling. 

CREDITS

Created and designed by : Diane Edwards & Upasana Nattoji Roy

Unreal and 3D : Diane Edwards

2D animation, deep motion & dance  : Upasana Nattoji Roy

Programming : Diane Edwards & Hasan S

Sound design : Sound: & Tim Garratt & Ugendran Padiachey

Handpose guidance : Ambika Joshi

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: