Amy Winehouse in 2007. Roger Kisby/Getty Images
Amy Winehouse is going on tour, as reported by Reuters last week.
A hologram of the artist, who died in July 2011, will tour North America next year. Her father Mitch Winehouse is the primary caretaker of her estate, and says he’s been working with Las Vegas-based company Base Entertainment’s nascent hologram studio to make sure that this digital Winehouse is an accurate recreation. And, as is always the case when a new hologram of a dead celebrity is announced, the reaction from fans has been mixed.
“Like she hasn’t been exploited enough??! Let her rest in peace and stop tarnishing her legacy,” one wrote on Twitter. Fair!
“People are mad about Amy Winehouse’s hologram tour. Can’t relate, I never got to see Amy perform live AND it’s raising money to help the Amy Winehouse Foundation which provides rehab facilities for young people who can’t afford them,” wrote another. Also fair!
What is not fair is that we are forced to consider, yet again, whether the dead celebrity hologram industry is just a niche technological novelty that will soon go out of fashion, or the future of entertainment.
The birth of the dead celebrity hologram
It’s been six years since Tupac Shakur (then 12 years dead) appeared on stage at Coachella to perform alongside Snoop Dogg and Dr. Dre.
This hologram, created by special effects company Digital Domain, was not a hologram. It was a trick of light founded on 150-year-old theater prop standard called “Pepper’s Ghost,” which uses glass and some knowledge of refraction to create an otherworldly-looking figure. It was invented by British scientists Henry Dircks and John Henry Pepper, and first used to add a “real ghost” to an 1862 production of Charles Dickens’ other creepy Christmas novella, The Haunted Man and the Ghost’s Bargain.
In a May feature for Wired, Jimi Famurewa called it a “low-tech piece of razzmatazz” and pointed out that it is still in use in Disney World’s Haunted Mansion ride and in stage productions of the horrible musical adaptation of the 1990 film Ghost.
The technologies that enable detailed 3D modeling, capture super-intimate facial tics, and allow neural nets to mimic highly-specific manners of speaking have evolved plenty over the past couple of years. But the technology behind projecting those images to create the illusion of a three-dimensional body has evolved very little since its invention. The biggest change: In 1995, German inventor Uwe Maass patented a version of it that swapped glass for — basically — Saran wrap, and subbed in HD video.
It was this basic tech that was used to make Tupac, married with Digital Domain’s cutting-edge CGI capabilities. The company was, at the time, best known for fabricating Brad Pitt’s face in The Curious Case of Benjamin Button and for being owned by James Cameron, though it had actually been purchased by dot-com millionaire (and Michael Bay’s college roommate) John Textor in 2006.
Textor’s company won a special award at Cannes Lions for the Tupac hologram that June, and filed for bankruptcy three months later, around the same time Textor was sued for $80 million by the state of Florida. Update: Textor tells Vox via email: “[Digital Domain] failed for reasons that have plagued the entire VFX industry in the US — virtually every American VFX company has moved to Canada, Singapare (others) or shut down. The lawsuit came later, in response to our failure, and, importantly, though we are not at all comfortable with our failure, I did prevail in the lawsuit and was awarded the largest settlement out of the lawsuit.”
Tupac Shakur’s “hologram” at Coachella Christopher Polk/Getty Images
In 2015, Rolling Stone’s Andrew Leonard chronicled the bitter (and very spicy!) war between the two biggest hologram companies: Hologram USA and Pulse Evolution. Hologram USA is owned by Greek billionaire Alki David; it was responsible for the Whitney Houston hologram tour that was shot down by her family in 2016, and it was also responsible for the breach of contract lawsuit filed against Whitney Houston’s family in 2017.
It owns the exclusive rights to resurrect Patsy Cline, Buddy Holly, Billie Holiday, and Jackie Wilson, and it is managing Chicago rapper Chief Keef’s current tour, which he is not attending because only his hologram is attending, and because he is legally not allowed to go to London.
Pulse Evolution is made up of the leftover parts of Digital Domain and it owns the rights to the digitally-enabled ghosts of Elvis Presley, Marilyn Monroe, and Selena. Hologram USA sued the company for patent violation days before Pulse was set to debut its biggest accomplishment: Michael Jackson performing a previously-unreleased song at the 2014 Billboard Music Awards. In March 2016, Pulse settled with Hologram USA for an undisclosed sum.
Base Hologram, which is working on the Winehouse tour, is actually a relatively new and minor player in the world of dead celebrity holograms. The company, which is best known for producing Vegas staples such as Magic Mike Live, Phantom, and Criss Angel’s “Mindfreak” magic show, partnered with laser projection company Epson to launch its hologram division in January of this year. It has plans for nationwide tours, like with Winehouse, and for permanent theatrical residences.
The technology has gotten more sophisticated, but not really
The January press release announcing Base Hologram’s existence promised that the new company would represent “one of the most aggressive and unique approaches” to the field. Executive producer Marty Tudor told Vox in an email that his company’s tours use “cutting-edge proprietary techniques” to combine audio with “digital and laser imaging, CGI techniques, and spectacular showmanship.”
This claim was put to the test with the Roy Orbison production, which debuted in LA earlier this month, and was reviewed positively by the Los Angeles Times. Orbison’s son also told the paper that his father would have liked the show, having been tickled by the holographic Princess Leia in Star Wars: Episode IV — A New Hope.
Last year, in a longform consideration of the question of whether the government could commission a CGI Donald Trump to perform the president’s duties whenever he was busy with other concerns — such as golf, or television, or not feeling like it — New York Magazine’s Brian Feldman spoke to “digital facial expert” Mike Seymour, a special effects industry blogger and associate lecturer at the University of Sydney.
Seymour told him: “This technology is growing nearly daily and deep learning and neural nets are contributing not only to modeling and facial tracking, but soon to fine detail, high-frequency animation also. The actual computer rendering of the face has already become remarkably complex, producing digital skin that accurately represents the way light both reflects off skin and is partially absorbed and scattered.”
There are also private research labs that can create 3D models of a human face using just one high-quality reference image, Seymour told Feldman, and it’s only a matter of time until that technology is commercially available.
But to make Winehouse’s hologram (or light creature, I guess), Tudor tells Vox that Base will just have to hire an actress who resembles her physically to create a bank of movements. (While some companies use motion-capture suits for this, Tudor says Base will use simple video recordings.) Tudor didn’t mention if Winehouse ever had a 3D scan taken of her face, but it seems unlikely, so his company will also have to work through an elaborate process to make a digital recreation of that. Given that she was a very famous person who was photographed literally thousands of times, it may be tedious, but it won’t be difficult.
Famurewa’s Wired piece explains how the Tupac team “[Worked] around round the clock for two months in a room plastered with pictures of the rapper,” making “an entire bank of facial movements and expressions” to pull from when sequencing the entire pre-programmed Coachella performance. The VFX team basically just used the bank to piece together a video that bounced of some mirrors and translucent foils to make it look 3D; what seemed very high-tech and futuristic was actually just very labor intensive.
Making digital recreations of dead celebrities who can talk and ad lib during a show is also fairly easy now, thanks to chatbots modeled using artificial intelligence. All it requires really is that these companies can find and license a large enough sample of the person talking.
Buying the rights to dead people is still complicated, and expensive
The big question is how to obtain all the materials you need to make a dead celebrity hologram. The legal issues around that rights and archives gathering are still pretty murky.
For example: One of the more notable legal battles in this space happened in 2012, when Marilyn Monroe’s estate threatened to sue obscure startup Digicon Media, which had patented “VM2 — The Virtual Marilyn, the first visual actress to live and work from Cyberspace.” Digicon claimed that Virtual Marilyn — who you can still visit on an incredibly retro, net art-heavy website — “did not in any way infringe upon the purported IP rights that were asserted by the Estate.” Digicon eventually gave up on its plans to have VM2 perform live, and seems to have stopped making new work.
In general, to create a hologram of a dead musician, any entertainment company first has to license the artist’s music and videos the same way it would to use those things in any other, more traditional show. It also has to license any images of the celebrity’s likeness necessary to create the visual recreation, which likely means dealing with the family or estate of the celebrity in question. From there, a contract has to be drawn up between all of the relevant parties to figure out how to dole out the profits.
VM2 — Virtual Marilyn! Digicon Media
But that’s not exactly the end of it! A celebrity’s likeness is subject to copyright and trademark, and something else that’s even slippier: “right to publicity.” This is similar to a normal person’s right to privacy, and means that while a person is living, they hold the primary right to make money off of their own existence and face and voice and even mannerisms. But after they die, what happens to that right depends entirely on the state they lived in.
There are 23 states in which a right to publicity lives on postmortem, and the time frames vary widely, from 10 years in Washington to 100 in Oklahoma. In California, the right to publicity is held by the person’s family or estate for 70 years. In New York, it doesn’t currently exist at all, with the right to publicity halting at the moment of death. Nearly every year, a bill is proposed that would extend the right to publicity in New York to 40 years after a person’s death — primarily to guarantee that a celebrity’s family or estate would be able to profit off of it.
As more of a legal scaffolding is built up around the still relatively niche industry, it will become more and more expensive for those looking to get in the business of buying up life rights and putting on new shows.
The normalization of holograms might actually hurt the dead celeb industry
When Justin Timberlake hinted that he may perform with a hologram of Prince at last year’s Super Bowl, Prince fans furiously circulated gossipy tidbits about how much Prince hated Timberlake, and wouldn’t have wanted to share a stage with him, even from the grave. They also circulated a quote from a 1998 interview with Guitar World, in which Prince responded to a question about whether he would ever perform with a holographic version of say … Duke Ellington:
“That’s the most demonic thing imaginable. Everything is as it is, and it should be. If I was meant to jam with Duke Ellington, we would have lived in the same age. That whole virtual reality thing... it really is demonic. And I am not a demon.”
But holograms have been normalized, significantly, since 1998. Even before the Tupac performance — landmark primarily because of its shock value and the intimacy of the details programmed into its performance (like thanking the audience for applause, and interacting with the other performers) — Kate Moss walked as a hologram at an Alexander McQueen show in 2006, Will.i.am appeared as a very confused hologram on CNN in 2008, Frank Sinatra performed with Alicia Keys at the 2008 Grammys, Elvis Presley performed with Celine Dion on American Idol in 2009, and Mariah Carey showed up to a T-Mobile event in Poland in 2011.
Not all of these people were dead, obviously, but the “wow!” of making any famous person appear to be someplace they couldn’t actually physically be still dimmed a little more with each instance. When Feist — best known at the time as one of many indistinguishable members of Broken Social Scene — “performed” in three Canadian cities at the same time in 2013, the possibility of holographic concert tours that would be easy and weird and possibly profitable seemed fairly close. Yet, no real pop star has tried it.
Japanese virtual pop star Hatsune Miku went on her first wildly successful tour of North America in 2016; fans paid as much to see her as they did to see Taylor Swift the summer before. Not because they love the surprise of a hologram, but because they love her music and this is the form in which she exists. It hardly even seems odd.
And then, last year, The Verge’s Lauren Goode reported on startup 8i, asking if holograms are “the future of how we capture memories.” The New Zealand and LA-based company has been working on setting up studios that resemble classic department store family photo studios, but use an arrangement of 41 cameras and proprietary software to render holographic video vignettes of whoever steps inside.
“Everybody wants holograms of their kids,” Nicole St. Jean, 8i’s vice president of content, told Goode. And chief executive Steve Raymond promised her, “People are going to want to put their loved ones into their phones for lots of reasons… [It] will become something you can just buy on Amazon and have it installed in your living room and make it at home.” The point of improving the technology isn’t to make anyone think the holograms are real, but to make them feel a more irresistible and automatic emotional connection. You don’t love a pricey computer-generated “vignette;” you love your toddler.
When Wired spoke to celebrity estate manager Jeff Jampol earlier this year, he said that the technology itself was basically worthless: “You can’t move around it, it can’t interact with you other than from a distance. It’s the equivalent of a used VHS tape."
The novelty of a “hologram,” he argued, would wear off, and only our emotional ties to dead celebrities we’ve loved would remain to keep us interested in a dusty, hokey experiment. If the intensity of feeling that comes with the unexpected intimacy of these tricks of light wears off, this entire hotly debated industry will be little more than a couple-decade trend — as forgettable and inconsequential as the once-beloved VHS.
Correction: The actress who models the Amy Winehouse hologram will be recorded using video, not a motion-capture suit as previously stated.