Interview | dmstfctn (demystification)

On Sunday, Oct. 13, in the suggestive venues of La Pelanda al Mattatio in Rome as part of the Digitalive. Romaeuropa Festival, curated by Federica Patti, will stage Waluigi’s Purgatory interactive audiovisual experience created by artist duo dmstfctn (demystification), with an extraordinary original soundtrack performed live by Evita Manji. Set in a 3D theater simulated in real time, the performance tells the story of an artificial intelligence trapped in a purgatory reserved for AIs who have transgressed during their training. Lidia De Nuzzo talks about it with duo dmstfctn about the process of this work, the collaboration with composer Evita Manji, and their creative approach at large.

Lidia De Nuzzo: AI training can be programmed and re-programmed in a vast number of ways and for a vast number of possibilities. How does this compare to the experience of encountering an artwork and its potential to reprogram one’s perceptions, perspectives, understandings…? And how would Waluigi’s Purgatory temporarily reprogram the festival spaces, and how audiences experience and perceive them?

dmstfctn: While AI training can be reconfigured in many ways, a lot of the models that might come to mind first when talking of AI have language as a key component. Large Language Models are of course language based – even if they’re capable of tackling a range of other tasks – and in training generative visual models images or videos are paired with their online alt-text descriptions to allow for text-based prompting. So, in AI training things can only be known if they can be written, or if they can be named or labelled. But encountering an artwork (can) happen outside of language and this is what offers a lot of the potential for new perceptions or perspectives.

And then we can think about duration or speed. An encounter with artwork might take place briefly or over a long duration, it might be returned to, and so on. Physically, or in terms of varying levels of attention. But AI training is an intensive, continuous process. Imagine encountering thousands or millions of artworks with no break – you’d look for a way out. And there’s a parallel here, perhaps with the way that the AI characters we meet in Waluigi’s Purgatory have all cheated in their training.

One example we use in the performance is of a system being trained to simulate riding a bike which learned the best option was to just ride in circles because it was rewarded for riding towards a goal but never punished for riding away. So there’s a human perspective in the task set, a human understanding of goals, bike riding and so on, but the machine doesn’t operate that way, it has none of that context so it finds its own way to tackle the problem.

There’s a potential to gain new perspectives and so on through looking at and understanding this, and that’s something we try to offer with Waluigi’s Purgatory. It tells the story of W., a character who finds itself in a purgatory having cheated during its previous life. Descending, W. meets these other AI characters that have a similar history of cheating before eventually being presented with a challenge to either leave its memories behind, or remain in purgatory and continue to chew on them. The audience assists it in this choice and perhaps reflecting on their part in this will stay with them in their wider experience of the festival.

Purgatory has always been identified in the collective imagination as the non-place, a place of transit, of waiting, a liminal space of continuous stretching and aspiration towards something else, an other space. Where did the choice of setting the protagonist-AI’s journey in purgatory come from?

The choice of setting came partially from the larger narrative arc of this work, which is the second in a trilogy called GOD MODE (2021-) delving into themes such as mythmaking and folklore in AI, the promise and pitfalls of “alignment,” and the relationship between machine learning, simulation, and synthetic realities. The first performance in the series, is the origin story of W., the protagonist AI in Waluigi’s Purgatory, and tells the story of how it was training endlessly in a simulated supermarket in order to learn to operate as an AI checkout system, and how it ended up exploiting a bug it found in the simulation in order to complete its training.

GOD MODE (ep. 1) ends in a limbo of sorts, with the protagonist aware of having completed its training by cheating and unsure on whether it can progress onto the real world that it has been training for and told it is what it wants, or deprecated, returned to training camp, switched off… Waluigi’s Purgatory is then a dream, or perhaps an hallucination, of the same AI that is now facing this conflicted history, working through its contradictions, and ultimately learning to accept that what it wants to experience is perhaps not what the human wanted it to experience i.e. operating in a supermarket.

We often refer to this work as a theatrical play set in a 3D stage depicting a purgatory, and the former is as important here as the latter. The theatrical setting was strongly inspired by a concept developed by the 16th century Italian philosopher Giulio Camillo – The Theatre of Memory. This describes an ideal theatrical structure destined to locate and administrate all human concepts, featuring seven levels that encompass everything that exists in the world – a RAM (random access memory) for a single spectator to look at, placed where the stage would normally be. This theatre dreamt by the protagonist, and the characters subsequently encountered in it, then are but a projection of its subconscious, a set of memories found deep in the embeddings and internal space of its neural network, used to subconsciously make sense of the world, now beginning to surface back. As W. recites: “These images are just shadows, these sounds are just echos, the real world is out there somewhere”.

The theme of this edition of the Backstage/Onstage 2024 editorial project rotates around the Intraspaces, all those interstitial spaces that connect technologies artists, space, spectators. How does the projection of the spectator within the life of AI reflect and overturn the dynamics of approaching space as a passive spectator of the autonomous life of a human creation?

In Waluigi’s Purgatory everyone performs, audience, artists and protagonist AI. We mainly act as a proxy for the protagonist, using a joystick to move it along, in response to the audience’s interaction – using their phones to move a theatrical light in the simulation and point the way. We also speak out the lines for the protagonist, and Francesco’s [dmstfctn] voice and facial expressions are modulated and mapped onto the protagonist on screen in order to animate it. The use of real-time video game engines here allows for an immediate emotional transfer and audience-artist feedback loop, moving the performance away from a definitive recital and towards an open game spawning a field of possibilities. In The Open Work (1962), Italian philosopher Umberto Eco describes how similar open-ended approaches to creating artworks can produce a “controlled disorder” to reflect “the senselessness of the modern experience of the world” (Moving Castles, Three Eras of World Generation). Here, Waluigi’s Purgatory points to a senselessness found in the idea that machine intelligence can be aligned to supposedly universal human values. The work portrays both the external disorder of a cast of AI characters autonomously spawning and interacting with each other on stage, as well as the internal disorder of a protagonist who, guided by an audience in its encounters with such characters, navigates doubts, fears and desires about its condition.

What were the key narrative and artistic references behind the story/narrative? What space does the musical composition take within it and the larger work?

There’s a vast number of references that contributed to the process of thinking about the environment early on, some of which ended up inevitably impacting the narrative structure later. We looked at Opera to learn about the use of rotating stages as theatrical and narrative devices that aid story progression, but also at philosophical concepts with strong environmental cues such as the aforementioned Theater of Memory or Plato’s Cave. We looked at various illustrations of Dante’s Purgatory, including Blake’s, Doré’s and ones from printmaker Giovanni Britto, where Dante and Virgil are indicated with the letters D and V respectively, making the poem and the image more explicable – something we borrowed in Waluigi’s Purgatory. And we looked back at our childhood, specifically 2.5D video games such as Grim Fandango, MMORPGs such as Metin 2, and animations such as Michel Ocelot’s Kirikou – which sees the protagonist surpass a number of challenges to enter a cave where an elder resides, his grandad, who answers his questions and comforts him. The narrative in Waluigi’s Purgatory follows a similar traditional structure – a hero’s journey with a descent both physically and metaphorically into some sort of crisis or unknown situation, effectively a series of challenges, and a moment of death and rebirth, or transformation. This narrative structure unfolds in eight scenes across seven sets – each representing a level of purgatory, and with two scenes requiring active participation from the audience to choose a direction on behalf of the protagonist.

In composing the soundtrack for the performance, musician Evita Manji initially improvised using an old zither, building a fairly dramatic ambient score out of it. We prompted Evita with some direction around the structure of the music so that it could reflect that of the narrative, and discussed elements of their music we felt appropriate, but they had freedom to pick elements that interest them and synthesise them fittingly. Evita also created a series of “sound signatures” as voice and language for secondary characters in the performance. These were originally intended as recurring melodies for recurring characters – like Bach’s motifs, but eventually became assemblages of sounds played live and able to reflect characters’ emotion or personality: whispers for the doubtful, brass for the military, jazz piano and sax for the old and wise.

images (all): dmstfctn, «Waluigi’s Purgatory», 2024, courtesy of dmstfctn