I called the session “We are who we pretend to be,” after the moral of story in one of my favorite novels, Mother Night, by Kurt Vonnegut. The novel is about an American spy in World War II who does too good a job at his cover role of being a Nazi propagandist. The moral is: “We are who we pretend to be, so we must be careful about what we pretend to be.”
We chose to talk about the ethics and promise of virtual humans, or artificial people created for video games and other experiences. Saatchi is the organizer of The Virtual Beings Summit. His event explores what it means to create virtual beings, and it has been held a couple of times since mid-2019, and Saatchi is organizing a new event for June.
I did a rehearsal talk with Saatchi where we talked about other things as well, and I’ve included quotes from that conversation in this story, as well as quotes from the embedded video that aired at our event.
From Saatchi’s view, AI is the next great art form. It is fighting for its legitimacy now, just as virtual reality, video games, films, comics, and other things had to fight for their legitimacy in the past.
“As game developers start to explore natural language processing, computer vision, and synthetic speech, we could move away from the slightly repetitive releases we have had recently,” said Saatchi, who is trying to bring together game developers and AI technologists through his summit. “I think there’s a lot of opportunity in developers exploring machine learning and artificial intelligence as if it were an art form.”
Above: A rogue Android kidnaps a girl in Detroit: Become Human.
Image Credit: Sony
I opened with a question about the usual dark vision associated with artificial people, going back to Mary Shelley’s Frankenstein. Today, we’ve got Black Mirror, the Terminator, Westworld, Detroit: Become Human, and more. I asked, “What are we so afraid of?”
We can see how fresh these fears are with Neon, a Samsung-backed spinoff that touted its artificial humans as assistants for people at CES 2020, the big tech trade show that took place in January. You could rent these people, who could become a doctor, chef, stewardess or some kind of replacement for a human.
It’s part of what Saatchi has called the “replacement fear narrative” that we have about artificial intelligence, or AI, replacing us. Westworld raises another “valid” fear about how AI could come to dominate humans, while another fear is loneliness, based on the worry that we could trick ourselves into having relationships with these beings out of loneliness, he said.
“There are quite a few movies and books, all with possible negative outcomes. That’s not what I believe, but it’s a pretty cool canon,” Saatchi said.
In Detroit: Become Human, there’s massive unemployment, caused by the economic impact of robots replacing people. It’s a video game where we see the psychological impact, where people can inflict their darker impulses on artificial beings that we tell themselves aren’t real people, Saatchi said.
Then there’s Lucy
“For us, there’s a world where virtual beings could help us become better people,” Saatchi said. “Beings who aren’t selfish, who aren’t motivated by greed or envy, who are able to listen to us and validate us and see us, could help us be kinder and gentler toward other real people. We all walk around with a huge amount of emotions, anxiety, things we can’t express to ourselves or to others.”
But we could say these things to a virtual being.
Lucy is a cute girl who believes there are wolves in the walls of her house. She’s also a companion who looks you in the eye and talks with you, remembering the choices that you’ve made.
“Lucy is a kid, our first being, from a Neil Gaiman story,” Saatchi said. “You will be able to have video chats quite soon, interact with her, learn from her. We haven’t yet seen a connection between a character’s life you can follow on Instagram or YouTube, but also be able to talk to that character, check in with them, talk about what is going on with you, build empathy. For us, this is the first time you can interact with a character and build a set of memories and an emotional relationship with that character.”
Fable Studio is building out that interactive side of Lucy, as well as with other characters as well. Maybe Lucy will show up in Zoom meetings in the future, helping create the illusion that this character is real.
“She sees things in a hopeful and optimistic way,” Saatchi said. “In the conversations we have with Lucy, you get a sense of her as a real, three-dimensional character that is hopefully connective. In this coronavirus period, for us, it is always important for us to wake up in the morning feeling like you are doing something useful. Building a virtual being you can have conversations with, have a video chat with, and communicate with, feels even more important because the loneliness and feeling of isolation — the stuff we think about metaphorically — is real.”
In the story of Lucy, the fact that no one listens to Lucy becomes a metaphor for what is going on in the family. When the wolves do come out of the walls, a crisis happens, and she has to take charge and lead her family to love each other and fight the wolves.
“We want you to feel like you can communicate with Lucy as your imaginary friend,” Saatchi said. “We want you to feel that there is a universe we have created and a logic to why you are communicating with each other that is not transactional. It’s not a virtual assistant relationship. It’s more of a story-driven, playful relationship.”
Do digital humans have to look realistic?
Above: The Unreal Engine 5 will produce outstanding imagery for the PlayStation 5.
Image Credit: Epic Games
Other visions of digital humans tend to push the envelope on realism. Epic Games’ is creating highly realistic demos of digital humans (like with its recent Unreal Engine 5 demo for the PlayStation 5) that will make use of the growing processing power of PCs and next-generation consoles.
But, as noted, Lucy is a cartoon. Fable’s DNA includes veterans of Pixar (like cofounder Jessica Yaffa Shamash) and Dreamworks (Peter Billington), as well as artificial intelligence experts. And both are necessary to move virtual beings forward. Saatchi said it is easier to animate a character like Lucy. It’s still hard to get characters who look super-realistic, like Magic Leap’s Mica, to look real when they’re speaking to you.
With Lucy, Fable is going beyond the visuals to integrate a wide variety of AI aspects, including machine learning, computer vision, synthetic speech, memory, and computer animation. The idea is to have a narrative with Lucy leading the storyline, but also to create a real companion for you.
“We think memory is the thing that everyone should be working on or exploring in virtual beings because it might unlock some pretty profound things about how humans relate to each other, and eventually to a virtual being,” Saatchi said.
The visuals have improved over the years, Saatchi said, but the behavior, or the brain of the artificial character, hasn’t kept pace. Fable Studio spends less time on the visuals, and it spends more time on “memory” and the deepening one-to-one relationship between the human (you) and the character. The character should know who you are, with a memory that goes back in time about choices you have made.
The explosion of virtual beings
Above: Lil Miquela
Image Credit: @Lilmiquela on Instagram
Since Saatchi’s first Virtual Beings Summit in July 2019, there has been an explosion of these projects that are taking very different forms. Replika is a text messaging bot that has millions of users who believe the bot is like a virtual best friend. It’s like a form of therapy with enormous potential, Saatchi said.
“You have an ongoing digital conversation with your Replika,” Saatchi said.
Investor Cyan Bannister of Founders Fund has bankrolled a lot of these projects. She saw a virtual concert put on by a Japanese virtual character, Hatsune Miku, and was fascinated with how she could draw crowds.
One of her investments is Brud. On Instagram, you can follow the virtual life of Brud’s Lil Miquela, an artificial influencer who has 2.3 million followers on Instagram.
Beyond that, you can have a conversation with Deepak Chopra via a project being created by AI Foundation. The celebrity musician Grimes created a digital avatar of herself. MuseNet is an AI that creates its own music, like a new Mozart composition. Genies lets celebrities cash in on animated clones of themselves. Saatchi thinks there is enormous potential for virtual beings in this space.
Virtual Immortality is resurrecting deceased actor James Dean for a computer-generated imaging (CGI) performance in a new film.
“I just think that’s fantastic,” Saatchi said. “A lot of people obviously think that’s terrible. Artistically, that’s so fascinating to think about a new performance.”
The Wave has virtual concerts where a real performer like John Legend is driving a motion-captured animated character. Netflix will likely move deeper into interactive, as it did with Black Mirror: Bandersnatch, with interactive experiences that are part game and part live action. Facade is an example of an interactive story game that is very conversational.
On the not-so-good side, deep fakes are being used to make real people say things they never said.
“I think this is an interesting moment for us where we can see the beginning of a new movement,” Saatchi said. “Each of these comes with controversy.”
The intersection of virtual beings and games
Above: Motion capture for digital character Lucy of Wolves in the Walls.
Image Credit: Fable Studios
Bannister said at the first Virtual Beings Summit that she would love to have a conversation with her grandmother again, but her grandmother passed away. The startup HereAfter is trying to make that kind of conversation happen, across the divide of death. Saatchi noted that his own children won’t have a chance to meet his mother, because she has passed away.
“It will change human psychology, quite a lot,” Saatchi said, if you could have a conversation with someone who died.
These interactions could yield powerful emotional moments associated with virtual beings, and that could benefit games in particular, Saatchi said.
“We’re pretty close to being at the point where we are able to get these things done,” Saatchi said. “Interactive entertainment is getting more cinematic and story driven. There are empathy games and narrative games. That’s going to be very exciting. When you see how much people connect to or love interactive characters like Ellie in The Last of Us, you can see that looking to the virtual beings community will be very important.”
SpiritAI is helping companies create non-player characters who appear to be like real humans, so the hundreds of characters you may come upon in a game will make it feel more immersive, not fake. Speech Graphics is creating facial animation that is AI driven.
Game designers have focused on “emergent gameplay,” where the events in a game aren’t scripted but emerge from what the player does. These types of games are more immersive because the player feels like they’re in a real world, where anything can happen. But Saatchi said much of that is based on environment-based gameplay, like how enemies might spawn in a part of the world where you don’t expect.
But “character-based gameplay” is more of a rarity, and that’s something the virtual beings community is focused on, he said.
“It would be cool to see emergent gameplay where the characters are changing in an unpredictable way,” Saatchi said.
The ethics of creating AI characters
Above: Dean Takahashi of GamesBeat talks with Edward Saatchi of Fable Studio about virtual beings.
Image Credit: GamesBeat
Richard Bartle, professor of computer science at Essex University in the United Kingdom, is a well-known online gaming expert. He believes that game developers should think about how to ethically create AI characters who are “sapient,” or have their own free will. He said game developers should view themselves as gods who are creating people, and that the ethical consequences of what they do matters.
For myself, I feel like how well we treat those virtual characters depends on how well they are done.
“For us, we want them to appear to have their own destiny,” he said. “Just as in Westworld, they program in a core motivator. Or core trauma. For realism’s sake, you need to have that. It will, of itself, I hope, change the way we treat the character.”
Saatchi said the ethical challenge reminds him of The Truman Show, the Jim Carey movie where Truman’s life is engineered by people who are watching him, and he doesn’t know that he’s being manipulated. Saatchi believes there should be some way for the virtual being to step out of a relationship, if necessary. He also believes the creators should use their authorial intent, rather than just make them mirrors of us, so that these are clearly works of art, rather than feats of technology.
“The memories form the character’s destiny,” Saatchi said. “It doesn’t so much mean free will and what are we forcing [a character] to do. It’s more about what memories have we implanted and whether you have their destiny already laid out for them.”
Saatchi believes that for characters to feel real, you need to creates memories for the characters. To design those memories, you need storytellers. You have to then design a story that takes into account what users can do with that character. You could have a character driven by traumas of the past that are inescapable. The character designer, then, is the one that limits the virtual being’s free will.
Saatchi said that it’s interesting to see discussions between the AI experts and the game designers, as Fable Studio has both of those kinds of people under one roof. Saatchi believes the AI needs to be subordinate to the storytelling, as in some ways AI isn’t advanced enough to be the complete star of the experience.
Peter Molyneux’s Black and White game from 2001 is a good example of thinking about how players should or should not be allowed to treat virtual characters, Saatchi said. In that game, the player is a god, with subjects who are characters in the game. It raises a lot of questions. Are they property? Or are they people?
In this case, the developers hold sway on whether the players can allow themselves to distance themselves from the virtual being, like in Detroit: Become Human. Players may have the impulse to behave badly toward the virtual beings, just because they know it’s a game and being bad is an option they can pursue to unravel a whole different side of the game.
Some users might want to get to know game characters, the same way they want to get to know music celebrities through both their work at massive concerts as well as their day-to-day personalities as they appear on Instagram.
“You would figure out what your role is, just as I would figure out at a concert that I shouldn’t feel bad that Travis Scott isn’t talking to me,” Saatchi said. “You understand the context. The user needs to be trained in each circumstance which role to pretend to take on. That’s what gets us back to what we pretend to be. The mutual cosplay of it quickly takes over. You will know what to do.”
The next operating system
Above: Pete Billington and Jessica Yaffa Shamash talk about Lucy at the Virtual Beings Summit.
Image Credit: Dean Takahashi
Over time, Saatchi believes we will go down the path of Her, the Joaquin Phoenix movie where a man falls in love with an operating system that takes the form of a female character, or Blade Runner 2049, with its own virtual character, Joi.
“When I look at Miquela, she is real to me,” Saatchi said.
The community has discussed issues such as trying to do a fake bank robbery, as once suggested by philosopher Jean Baudrillard, Saatchi said.
“You’ll quickly find out you are doing a real bank robbery,” he said, because of the way that people would react to your actions. “You’ll be arrested.”
“What that means to people in the community is that if you are able to take enough of the symbols of what you are pretending to be, like with Lil Miquela, … it’s so similar to how a real person would do it that she is what she pretends to be,” Saatchi said. “She would pass through. I would encourage any game developer to study what Miquela has done. She is a great example of being what she pretends to be.”
Asked where he wants to see this go, Saatchi said, “My No. 1 vision and goal is to create the next operating system. It’s so boring. iOS 14 now. Mac and Windows updates. Alexa and Siri. It feels like they are the same as they were five years ago. I used to think the paradigm was going to be spatial computing, or AR and VR. Now I think the next paradigm will be a virtual being, a character who is our friend who helps us through life and helps us organize our digital world. That’s where I want to go.”
He added, “I think everyone is in very different ways unlocking a piece of the puzzle to for this to be the next paradigm of computing. A virtual being is a representative of all computers. We saw that in Her. We saw that in lots of movies. Ethically, it’s obvious that level of power is terrifying. If you think of Blade Runner 2049, the company that created Joi, the name of the virtual being, would be the Microsoft, Google, Amazon of that day. All of those combined. We would have to hope that consumers would reward companies that have invested their virtual beings with a moral sense, and that care about what their virtual beings are doing to people. You could do so much with addicting people to virtual beings. Ultimately, I have to have faith in the market forces that consumers will be able to detect the intentions of the people creating these virtual beings. Just as you get a vibe from a person, you also get a vibe from a virtual being.”
That brought us back to our beginning. We are what we pretend to be, so we must be careful about what we pretend to be.