Creating Valuable (and Trusted) Experiences With Digital Personas

Have you interacted with a digital persona yet? At the Museum of Art & Photography in Bangalore, you can have a deep and engaging exchange with one that represents the late artist M.F. Husain — considered the “Picasso of India” by many. This avatar is eager to talk art. And if you ask him whether he’s real, he will look straight at you and say, “As close to real, enough to impress you.”

Digital personas — digitally created “doubles” of individuals that reflect their style, personalities, and attitudes — are becoming more common in everyday life. They also represent a step change in the way people can study and understand history. In fact, studies show that the use of immersive technologies associated with virtual reality in educational settings can boost engagement rates by a factor of 10 and significantly improve recall as well. But businesses are also catching on to potential use cases — to help with advertising, and to make shopping experiences more entertaining and compelling for customers in online and physical settings. Snack brand Cheetos, for example, recently deployed a suite of celebrity avatars (a form of digital identity) to promote its brand. And Reactive Reality’s Pictofit app enables users to create their own avatars in just three minutes so they can “try on” clothes sold online.

Many business leaders interested in this still-developing technology see its potential to support their company’s brand, educate consumers, and champion social or environmental change. But they’re also finding that in order to realize the benefits of digital personas, they must overcome two significant challenges.

Complex Creations

The first hurdle for companies pursuing a digital persona strategy is technological. Producing a simple deepfake — synthetic media in which a person in an existing image, video, or audio file is replaced with someone else’s likeness or voice — with face-swapping and voice synthesis software is relatively easy, given the advances in technology and the range of free software available for it. An interactive persona that can engage in two-way communication, on the other hand, is much harder to develop than a simple deepfake or a static persona programmed to give a limited set of responses. And for a high level of communication with an audience, a persona needs to be trained with in-depth knowledge — a task that may require significant upfront investment.

Making a “living” avatar of a person for screenless interaction presents a considerable challenge. To create avatars of still-living Holocaust survivors for the Illinois Holocaust Museum & Education Center, designers asked each participant more than 2,000 questions. They then transferred that content to the avatar using natural language understanding so that machine learning tools could respond properly when the avatar engages with museum visitors. (Natural language understanding is a subfield of natural language processing, focusing primarily on machine reading comprehension.) Designers had to ensure that these holographic avatars would accurately represent the person’s history; they also had to try to ensure that the avatars would be able to answer just about any question visitors might ask — even on complex topics such as the psychological impact of slavery.

Creating an interactive digital persona of someone who is no longer living is even more complex. The technology behind digital personas includes an AI tool called an autoencoder that first encodes facial features and then decodes them on the target face. This can be further enhanced using a generative adversarial network, a machine learning framework that pits two neural networks against each other. Here, a “generator” neural net creates fake data to try to fool the “evaluator” neural net into believing it’s real until it passes the test. In creating Husain’s persona, this meant using publicly available videos of the artist to superimpose his face on the target personality. In addition, people with expertise in his art had to be part of the project to hone the content and ensure accurate responses to museum-visitor questions.

Given that the technological challenges associated with digital personas vary depending on the context, not all personas are created equal. For example, the challenge associated with creating an avatar of a living person may be to “democratize” this technology to be available to all your customers rather than a select few — as in the case of the Pictofit “try on” technology. By understanding the context well, businesses are less likely to underestimate the technological challenges associated with the creation of digital personas.

Correcting for Misuse and Abuse

The second challenge of developing digital personas is the associated risk and potential harm that deepfake technology poses to society. Sadly, many examples of misuse have already gained widespread attention. In 2019, The Wall Street Journal reported that the CEO of a U.K.-based energy firm, believing he was on the phone with his boss, transferred $243,000 to the bank account of a Hungarian supplier. In Gabon, uncertainty about the authenticity of a video showing the president of Gabon alive and in reasonably good health sparked an attempted coup. And the AI company DeepTrace estimated that out of the 15,000 deepfake videos online in September 2019, 96% were pornographic, mostly involving female celebrities.

As these examples suggest, the technology used for creating digital personas can also be weaponized to commit significant harm across society and business. Companies, however, should not be overly fearful of the potential risks and reputational pitfalls. Many new technologies have raised similar concerns — consider photo editing and 3D printing, to name just two. Digital personas can deliver immense social and business benefits, but businesses do need to develop and execute a well-thought-out strategy to mitigate risks if they intend to use digital personas to advance business and social goals.

1. Develop an authentic persona in context. At the museum in Bangalore, leaders made the decision to produce a holographic projection of M.F. Husain only after following a design-thinking process to explore all the issues that might arise in the museum setting. For example, how many people could interact simultaneously with the persona? Would the persona be accessible to persons with disabilities?

One key tool was the application of cognitive ergonomics, by which systems are made easy to use by analyzing human perception and mental processing. Context was also critical during the programming process. Designers had to ensure that the lighting would be consistent and that the hair, skin tone, beard, and facial structure — which are difficult to reproduce — would be properly represented. Even slight imperfections can “break” a user’s experience.

They also had to ensure that the avatar could perform at its best in front of its intended audience and meet the business goal of educating visitors. In developing the artist’s persona, that meant running pilot tests with mock visitors and iterating based on those tests.

In other contexts, different features might be more important. For example, if a company is planning to let customers develop their own personas to try on clothes virtually, the most critical part of the authentic persona is that it accurately represents the person’s physical features so he or she can get an accurate idea of look and fit. If the persona will be used as a gaming avatar, then the key may be rapidly creating a good approximation that customers can relate to and develop a bond with. A gaming persona may also include “aspirational” features instead of being an accurate replica of the person.

2. Create a persona that can adapt to what users want. Most current examples of digital personas are static and unresponsive; they broadcast their messages and cannot adapt to users’ changing interests. That level of functionality won’t be enough to create an engaging customer experience. Instead, a persona will have to be able to relate and adapt to changing conditions in much the same way that a comedian performing a stand-up routine assesses the audience and shifts gears to keep people interested in the material.

That’s where natural language processing and natural language understanding technologies come into play. Adding a multilingual feature can further enhance the experience for both global and local audiences. Again, though, it’s important to understand the context in which the avatar will be “performing.” Accurate natural language understanding can be a challenge in a noisy environment, especially when accents vary and speakers interrupt or talk over one another.

The use of emotion analytics is also valuable. The mood of an audience can be gauged via nonverbal cues. For example, at the Bangalore museum, if eye movements among visitors around the avatar suggest that their engagement level is dropping below 50%, the avatar will shift the conversation to a different topic.

Collecting that data of course means using video cameras, which can present their own contextual challenges, particularly in the low-light environments common in museums. Because this data is for real-time use only, there is no need to store the data.

3. Proof your persona against bias and hacking. Given that digital personas are, in most cases, virtual twins, the “fake” experience must be ethical by design. To minimize negative consequences, digital personas need to be deployed in a controlled environment with consent from all stakeholders prior to deploying the persona. Further, the organization needs to obtain consent from all stakeholders prior to deploying the persona.

Business leaders must also take care to avoid gender or racial bias. For example, facial recognition software has been known to work poorly for people of color. When letting customers develop their own personas, businesses need to ensure that they do not have inherent biases in their software and that it works equally well for different skin tones, facial structures, and other characteristics across groups.

One way to protect digital personas from threats of hacking is through digital watermarks. Researchers at New York University are developing a digital watermark technique that uses neural networks to authenticate image integrity. The watermark breaks if any attempt is made to alter the image. Also, consider tamperproof digital fingerprinting, which can alert a company if another party is potentially misusing the original content. Digital fingerprinting relies on a combination of AI, computer vision, neural networks, and blockchain to detect spatial tampering, such as face-swapping edits, and temporal tampering — cutting or modifying content to change someone’s speech, for example.

Companies can do a lot to reduce risks, but they still need to plan for contingencies when things do go wrong. Doing so includes developing an explicit communication strategy for your business. It also means educating customers and employees alike on how to minimize risks in the first place.

The more sophisticated the technology, the wider the scope of its applications. Soon, digital personas could find a place in corporate board meetings or public policy discussions. In the post-pandemic world, board members could meet in a 3D virtual office, with their realistic digital personas, or even holographic projections, interacting with one another. They could potentially use face-to-face translation software to convert a language they are comfortable speaking into another language for the audience, with appropriate lip-syncing. Digital personas could be powerful “twin” sources of information — viable tools to bridge physical distances and enable executives and teams to be present in more than one location at a time. While the risks of misuse are serious and need to be understood, this technology can be of significant value to businesses. It’s time to explore the ways to make it happen for yours.