AR/VR Special Feature 2019: Augmenting pharma

 

Med Ad News spoke with digital guru Fabio Gratton about the present and future of augmented and virtual reality technologies in pharma and healthcare.

 

Med Ad News: What AR/VR work have you seen out in the wild of pharma that is interesting?

Fabio Gratton: The large majority of what people are doing falls into two key areas. One is training and education and the other is tradeshows. If you go to ASCO or ASH, most of the big companies are using AR or VR. But a majority of the time, it’s still being used as an engagement gimmick, not necessarily because this was the best way the information could be communicated but because, “Hey, this is going to get people into our booth.” It’s effective as an attractor. We’ve done some AR ourselves in the past and you could certainly get people to line up at your booth. I’ve seen people putting on the goggles and standing in front of screens and walking around flinging at things, “Grabbing the cells” or doing other things you don’t really need to do in the real world – but it’s a good distraction from the regular tradeshow stuff and something interesting. So most of what I’ve seen has been aimed more towards creating an interesting way to get people to interact with your brand or to interact with your science than exploring the full capabilities of the technology. I think there’s – while I’m using the word gimmicky, I’m not trying to make it sound insignificant. Engagement is always important. People remember the things that they’re engaged with, right? So if I’m more likely to remember your mechanism of action from a tradeshow because I put on these goggles versus the person who just had it playing on a screen as they walk by, well, that’s important. So, it does serve a purpose beyond just being gimmicky distractor at tradeshows. But I just don’t think I’ve seen many substantive applications in marketing, parallel to, say, conducting a surgery through VR, with the doctor controlling a robot from 1,200 miles away. That’s interesting. That’s really genuinely saving a life. When you consider AR/VR applications like that, you want to see that kind of value on the marketing side too, applications that drive some new kind of value, and we just aren’t seeing very many of them yet.

Med Ad News: Are there areas or disease states where marketers are perhaps a little further along towards that goal?

Fabio Gratton: Some of the best applications I’ve seen are when people are trying to simulate the perspective of the patient – for example, in ophthalmology. Ophthalmology is about the visual field. So to being able to have a doctor put on glasses and see what a cataract might look like or what a treatment might look like or what a patient sees, that can be very powerful because it’s actually connected to the visual field – the virtual reality is adding something really substantive. It’s hard to simulate what a patient sees. If you can do that, you can create empathy, a connection to not only what the products do but what a patient is feeling. I’ve seen good uses of that. There were some good examples at the Cannes Awards, where they’re trying to immerse the user in the patient’s perspective, showing what a patient might see when they, say, have a heart attack, or other experiences like that. Those sorts of applications of AR and VR are creating more of a human connection, as opposed to, “Hey, reach your hand into the cell and spin this virus.” But we haven’t seen as many of those because they have to be a bit more clinically accurate for medical-legal-regulatory to be comfortable. So they’re more time- and cost-intensive. And considering that immersive is still considered relatively innovative and cutting-edge, people aren’t allocating the dollars to those kinds of things. We don’t see as many really relevant applications because they’re more complicated and expensive to execute.
MedAdNews: Aside from cost, what else is slowing down the adoption of those sorts of AR/VR applications?

Fabio Gratton: It’s actually a lot like the classic chasm I saw in digital back when we were starting our agency during 2000 – the people that know the tech well do not know the health well yet and vice versa. We’ve got extraordinary creative and scientific people in healthcare agencies, but they do not yet have their hands or minds around this technology. And we’ve got people who are incredible at developing immersive experiences technologically, but they are in outside, non-healthcare shops, not sitting in healthcare agencies, and they do not usually have that core healthcare expertise.

Med Ad News: So we have to wait for the health shops’ technology departments to catch up and close that gap.

Fabio Gratton: Yes, or build stronger partnerships between the technology shops and the agencies getting the work. Agencies are selling the work and then they are trying to figure out a way to do it to maximize their revenue versus looking to the outside partnerships with the tech people that know how to do it really well. If I were an agency, I would not want to build that capability in-house because it’s complex and sophisticated and it’s changing all the time too. So, why would you want to keep chasing all the new hardware? Unless you’re going to create an entire offering based on immersive technologies, why not just continue to be the creative shops that companies want you to be and find good partners who are keeping up with the tech. But agencies sometimes do not like to do that because they do not like to give up revenue, unfortunately.

Med Ad News: So where do you see AR and VR sneaking in to the message in a substantive rather than gimmicky way? What are other spaces or disease areas, aside from ophthalmology?

Fabio Gratton: As long as there’s a hardware requirement, a specialized hardware requirement, we’re going to have some challenges. As long as some piece of hardware is required – goggles, perhaps – that is not native in the smartphone devices we’re already buying, then we’re going to have a gap because not everybody can access it. If we’re limited by the availability of that hardware, we’ll be stuck with very unique circumstances like tradeshows, like sales reps, and like events where you can control the hardware experience. But the place where we might see more advancements is in augmented technologies, augmented realities, rather than virtual reality. To distinguish, AR can be an overlay on our world that helps us augment it through things like the Holo Lens, the Microsoft Holo Lens, versus the Oculus where you’re pretty much immersing yourself in a complete virtual world. So augmented probably has some more near-term applications because everything already has a camera on it, and everything with a camera comes with the ability for that camera to view the world, be a viewfinder into the world, and then for algorithms to be able to do object recognition and then to be able to say “OK.” For example, if you have an insulin injection pen or some kind of medical device that a patient needs to use, the opportunity to have a patient just open up an app and hover over that device and be able to see its various features and click on the virtual application, but while they’re looking and holding the actual real thing so they understand how to use it better. It’s perfect for training patients how to use a device that they have to use at home or to remind them how to use something if they run into an issue. Those kinds of possibilities, I think we’re going to see a lot more of them just because the technology is in our hands already.

Med Ad News: That being the case, do you think we will be seeing more of it out in the wild?

Fabio Gratton: I know companies have been pitching it. I know I’ve seen demos. I know we’ve looked at prototypes in the past and they’re super cool. But we’re probably not seeing it as much because it’s still kind of in the realm of “innovation,” and people are not putting the money into the – if you think something like I just described is practical, if it’s practical, then you want it to be absolutely accurate. Then, it goes back to, “If it’s absolutely accurate, how does it get validated?” Then, you have to go through the whole process within a company of like “Hey, if this is how we’re going to do training, then we can’t screw this up. People need to have access to it. It can’t ever have a glitch.” So, I think people are just really nervous about new technology that is not validated. Again, we see it at tradeshows because tradeshows are the perfect place to do something for the purpose of demonstration but not let anybody take anything home. And I know that a lot of agencies put together demos to show off their technology prowess for companies.

Med Ad News: The date that you take out but are not quite ready to bring home.

Fabio Gratton: Virtual reality has been used a lot because it can be an attractor and an engager, but it is still being used in a gimmicky way. I think our short-term path to more adoption in immersive technologies in general is going to be through augmented reality. So what’s going to happen is, probably, mixed reality is going to be the best place to go where you can have a virtual reality experience at a conference, you can have an augmented reality experience outside of a conference. And then as that sinks in, brand managers will start thinking when brands are developing their marketing programs, from the very beginning they’ll be thinking about, “What’s our total media reality that we want to create?” That’s where people are going to go. The example I like to use is, if you think about Disney, Pixar, or Avengers, when they’re sitting at a room at the script phase, you’ve got the toy people, you’ve got the theme park people, you’ve got the movie people, you’ve got the franchise, all the different merchandising people. They’re all sitting in a room together. You’ve got the video game team. You’ve got people thinking holistically about what the total media experience is going to be for this property. At some point in the future, we’re going to see that kind of mentality when thinking about a brand, having all the disciplines and thinking about, what’s the augmented experience, what’s the virtual experience, what’s with the application of all these different ways that we can touch customers, and how they are all going to work together. Because then, you’re not just doing it ad hoc. There is just going to have to be more planning upfront involved, where we’re going to start seeing this as core to the brand experience.

Med Ad News: Setting aside AR and VR, is that something that brand managers even do now with all their other channels, sit down with the web people, the social media people, and the print people, and have everybody in the same room talk about planning the brand’s entire appearance? I’ve never really gotten a sense that that’s even the case now.

Fabio Gratton: I do not think they are. So, that’s how behind we are. I do not think that kind of multi-sensory total experience perspective really exists yet. The closest we’re getting to it short-term is companies hiring one person who’s supposed to be their omni-channel expert. That person is supposed to represent all perspectives. They show up to a meeting. If they’re really good, they might be able to take good enough input to be able to walk it back to the subject matter experts. The reason we don’t see all of this stuff taking place in the real world is, we don’t see the multi-sensory experience, is probably because that is not happening like you just said. Regardless of AR and VR, is it happening across all media the way it should be? No.

Med Ad News: What is AR/VR in healthcare going to look like in five years? What are the things that are going to be normal for brands?

Fabio Gratton: Well, the complex part of that question is for brands because brands are focused on sort of the – I mean, I think we’re going to see a lot more of it on training. I think we’re going to see a lot more on education. I think we’re going to see the ability for people to augment a – we were doing this five years or eight years ago on brochures, but the ability to – do you remember the Ikea catalogue example, the kind of famous for its augmented reality app and catalogue that allowed people to basically interact with the catalogue and allow these constructions to come to life? It’s amazing to me that a rep can come in with a brochure and that brochure is not augmented, that somebody can’t on their own time hover over a little icon on the brochure and a little thought leader comes up and explains the data to you, or there’s a way, a picture, a snapshot of a picture in a printed piece that you can just pull out your phone and bring that to life or get more information about it. The fact that that bridge hasn’t been closed between the technology in our hands and in our pockets and in the real world, I think it’s the easiest place to go first, the real world being brought into this kind of virtual world by using the devices already in our hand. Hopefully we’ll see more of that. Do you remember when the QR code was supposed to be everything? Well, that was the promise, right? “Hey, put a QR code in everything.” What was that supposed to do? You pull out your phone, you hover over the QR code, and then you were supposed to get some incredible thing. The problem is that first of all, people didn’t understand that. It was super ugly. The scanners were widely – you had to install the app sometimes and people just didn’t know how to use it. Then, what was being delivered on the other end had no extra value. It was like, “Now, you’ve signed up for my email newsletter.”

Med Ad News: Which most people could have done on their own.

Fabio Gratton: Yes. But the idea, the idea of object recognition, the idea that you could pull up your phone with hardly any effort and be able to interact with an object like a medical device, a brochure, or a picture of a drug, that has to be the first step to bridge things because we can’t and don’t need to have really complicated eyewear at first. We already have something in our hand that’s one of the most powerful computers in our lives. So, that to me is the first step, creating this new behavior which we don’t even see that frequently in consumer life. We get to create the behavior where people understand to use their phones as augmenters of the real world. That to me is a must step. Because once we retrain people to believe that any time they open up their camera they can get more information from the real world, then we’re going to start to see behaviors where people expect to be able to do that with everything. Most people don’t do that today, which tells us that we’re not there yet.

Med Ad News: Not just with healthcare but with everything.

Fabio Gratton: Right. So to me that is the logical first step, where healthcare will be in three to five years on the marketing side, when that idea of bridging the virtual and the real worlds through the devices in our hands is going to be second nature.

Med Ad News: Augmenting everything, augmenting virtually every brand interaction or drug interaction, whatever and wherever it might be. Having some kind of augmented experience that you can delve into with just your phone whether you’re a physician or even a patient.

Fabio Gratton: Right, once that behavior is there. If you think about it, let’s just say that it’s already a habit. Let’s just say your fun thing like Pokémon Go, where you open up your camera, your augmenter, you’re walking down and you see a store. Instead of like going to Yelp, instead of looking at your location, instead of clicking a button, you literally just bring up your phone in front of yourself. It immediately recognizes the sign. It tells you they have a special there and you walk in. That’s an easy bridge to behavior. But, imagine once you start doing that all the time. Then, you start just getting curious. You hover it over a car to see, “Hey, how much does that cost? It’s a cool car.” You hover it over a bus stop because you want to see the most recent schedule and how far the buses are. After you get into that habit, you just open up your phone and say, “Oh, I like that. Let me get that.” So, once you get into that habit, you have a Tylenol in front of you, you’ve got Motrin in front of you, you’ve got your prescription in front of you. You’ve got all these things when you’re walking by your doctor’s office and you think, “Hey, I wonder what his appointment schedule looks like”. But then, we’re in the habit of expecting to interact with healthcare in the same way. We’re not there yet in the everyday world so it’s going to be a bit of a gap before we’re there in the healthcare world. But, you’re starting to see some of it, right? Some of the diabetes companies. I’ve seen some apps out there that you can take a picture of your food and get the calorie count and the carb count and the sugar. There’s a little bit of that now but it’s still not a natural behavior.

Med Ad News: From a technological standpoint rather than a behavioral standpoint, how challenging is creating that sort of augmentation? I would think it’s not nearly as challenging as creating a whole virtual environment with the eyewear and everything else.

Fabio Gratton: It’s not challenging at all. We already have third party developer kits where you can develop things by just taking 10 or 20 pictures of anything, of a tree, of an object, of a bottle, and then the app knows to recognize the bottle and it’s simply just a question of, what does it do with it? Once it recognizes it, do you want to pull up a website? Do you want to show a video? What do you want to do with that information? So, the technology is already there. Facial recognition, data recognition, object recognition, the tech is there. There are a lot of standard libraries already being developed. Facebook, Google, and Amazon, they’re all creating these object recognition libraries. So, I think the technology’s very close. But it hasn’t been deployed in a way that makes it consumer accessible yet.

Med Ad News: So if a company wanted to augment everything on its menu that a patient or HCP might interact with, the technological challenge of doing so would not be significant?

Fabio Gratton: No. It’s just a matter of working your way through everything, deciding what augmentations you want to add. When you think about how interesting it is, people take billions and billions of pictures a day, right? They post them but they don’t expect anything from taking those pictures, right? You take a picture to share it with somebody and that’s it. We have yet to have an expectation that our camera is going to do more than snap a picture and share it. There’s going to be a day, and that day is coming soon, when a picture you take, you’ll have an expectation. I take pictures of bugs all the time and send them to my brother because he’s an entomologist and I want to know what kind of bug it is. But, there’s going to be a day when I don’t need my brother.

Med Ad News: Your phone will figure it out on its own.

Fabio Gratton: Yes. That technology is already developed but people aren’t thinking about the world in that way yet. But even in medicine, there are a couple of apps out there. You take pictures of a pill and it tells you what the pill is. But it’s not seamless. Remember how Snapchat changed the world? It changed the world because you opened the app on the camera and it was like, “This is the new way to think about photography. This is the new way to think about the picture.” So, they shifted our mindset to camera-first. One day, we’re going to be thinking in terms of a camera-first world or an augment-first world when we – maybe it’ll take a company, one specific company, to take the lead on that. Remember, Pokémon Go was actually incredibly successful at demonstrating that the everyday person would participate in a virtual game, in a real-world immersive game, augmented game. Tens of millions if not hundreds of millions of people played and continue to play today. That wasn’t an object-based game. That was a geo-based game, knowing that I’m at this spot in the world and it’s going to render this Pokémon and I need to capture it. But we see what happens when somebody does something really simple, effective, and engaging: people play it. Maybe it’ll take something like that in the practical everyday world before it becomes a habit. But again, the tech is there.

Med Ad News: Are there specific hypothetical patient-focused augmentations that occur to you?

Fabio Gratton: My sister is a social worker in the UK. She works with a lot of Alzheimer’s and elderly patients that have – I guess for lack of a better word, sort of age-proof homes. As people get older, they get more and more challenges not only with dementia, Parkinson’s, and other neuromuscular disorders and neurodegenerative disorders. They also just simply get older and weaker. So apparently in the UK there’s a free service to have people come into your home and make recommendations and adjustments and even cover some of the costs in making adjustments, to “age-proof” the home. To me, that’s a perfect sign-up application. If we think about the home of the elderly, we think about things like dementia, we think about things like obstacles that may pose dangers to the people that are getting older, that’s a big problem, big issue, big leading cause of a lot of hospitalizations, huge cost. Right now, in America anyway, there’s not an easy way to age-proof your home. We child-proof our homes but there’s no way to age-proof your home. So, to me, one obvious place is, what if instead of worrying about how to use augmented reality to design your perfect living room, how about making a home more well-adjusted for people with various disorders, right? Being able to make people safer.

I think about safety a lot versus the marketing part because you’ve got to start at the services that protect people, the things that have high value because most people are not hurt not because of the medications they take. They get hurt because of all these other things that happen. So, augmenting the overall patient environment and experience is a huge opportunity. It’s already happening in clinical trials through wearables, right? We’re collecting patient data instead of self-reporting to a survey that you take when you go to the doctor’s office a month after you’ve had an experience. We’re collecting data in real-time from wearables, not only sleep monitors, Fitbits and everything else. Even your mobile phone is technically a wearable as long as you’re carrying it around. That’s already happening. Wearables are going to help augment the patient world really well because you’re not having to do a discrete activity and you’re not having to do it by memory. You’re doing it in the real world. Imagine the capacity for clinical data to be collected, whether for clinical trials to get a drug approved or post-marketing surveillance, the ability to understand treatment efficacy. For example if I have Parkinson’s and I’m struggling to walk across the room and I’m taking the medication, what better way to know whether the treatment is effective than to have a wearable that is immediately sending messages to the cloud to a doctor who’s able to monitor me in real-time and say, “Wow, this is working,” or “This isn’t working. Let’s do something about it.” Patients’ lives have immense opportunities for being augmented. The means to that end is going to be both the phone in our pocket and the wearables in our lives that are going to become more and more common, and all the dumb sensors too. Motion sensors, light sensors, sound sensors, all these things – camera sensors, anything that’s becoming so cheap to implement all these devices into our homes and our lives that the ability for all these devices to be able to keep us safe and protected or even adherent to medications, to me that’s a form of augmentation. We often miss the smart speaker, which is an incredibly visible and real way that we’ve augmented reality through our technology, being able to ask and speak and get information from anywhere, from the cloud, or get reminded when to take a blood sugar reading, to eat a certain meal, to exercise, or to take our medication. That’s technology augmenting our current lives. It’s around us all the time.

Med Ad News: It sounds like you are excited and worried at the same time.

Fabio Gratton: Yes. I worry that people will continue to think of virtual reality as Oculus or something gimmicky when it has the potential to have so much impact in so many different ways. I worry that people think of it as something that’s extremely expensive and complicated. Bridging the real world to existing data and assets is relatively simple. I worry about the fact that we’re not pushing the envelope, that we’re thinking of virtual reality in very binary ways and it’s either this whole immersive experience that we’ve created or we’re not doing it. That kind of thinking is a detriment to the industry because we should be asking the question of how do we augment our world, the world of patients and physicians – that should be a part of everything we do. It shouldn’t be an afterthought from the innovation group. It should be something we ask ourselves from the very beginning because it’s truly going to have a massive impact in the real world today. So, if there’s anything I lose sleep over, it’s that I hate when technologies get pigeonholed into these one-off unique uses and then the whole world believes that that’s the only application for it. Or, put another way, the technology is mature, but our ability to implement it and our appreciation of its potential role is not mature, not yet.