[LIVEWORX THEME]
Welcome back to LiveWorx 2020.
My name is Andia Winslow and I'm your host
for today's livestream event.
Already we've been hearing from some
of the top technology thought leaders
from the industrial world.
Now, if you weren't able to attend those earlier sessions,
be sure to check them out on the LiveWorx on-demand catalog,
which is available to you from today through June 19th.
This online library has over 100 additional sessions
of great content.
Also, keep an eye out for an upcoming email
to gain access to our content archive for an entire year.
Now, let's kick off our next session--
artificial emotional intelligence or emotion
AI, what it is and why it matters.
Dr. Rana el Kaliouby is the CEO and co-founder of Affectiva.
An MIT Media Lab spinoff, Affectiva created and defined
the emotion AI category.
Software that can detect nuanced human emotions
and complex cognitive states from the face and voice.
Dr. el Kaliouby is a pioneer in emotion AI and the author
of Girl Decoded: A Scientist's Quest
to Reclaim our Humanity by Bringing Emotional
Intelligence to Technology.
I am super excited to welcome Dr. Rana el Kaliouby.
Thank you, Andia.
It's a pleasure to be with you all today.
I'm on a mission to humanize technology
before it dehumanizes us by building emotional intelligence
and empathy into our devices and our technologies.
And in doing so my goal is to re-imagine
human-computer interfaces as well as
human to human connection.
We've all been catapulted into this universe where
we're working virtually.
That's how we're connecting with our team members.
We're learning online, I mean, that's
how my kids are learning.
And we're connecting with friends and families virtually.
However, something is missing.
It's not quite the same.
And it's really this concept of all of our nonverbal signals
missing from these virtual environments.
AI is taking on roles that were traditionally done by humans,
such as assisting with driving our cars,
assisting with our health care, helping us be more productive,
and perhaps even hiring your next co-worker.
The problem is we need a new social contract between humans
and AI.
One that is based on mutual and reciprocal trust.
Of course, we need to trust in the AI
and there's a lot of conversations around that.
But more importantly, AI needs to trust in us humans.
After all, we don't always have a perfect track record
of doing the right thing.
Unfortunately, there are already numerous examples where
this trust has gone wrong.
A Twitter chatbot that has turned racist overnight,
self-driving cars getting into fatal accidents,
and facial recognition technology that discriminates
against certain populations--
especially women of color.
To rebuild this trust, let's look at how humans do it.
Every day we make thousands of decisions
that involve trusting each other,
both in our personal and our professional relationships.
Sometimes this trust is based on legalize
and terms and conditions.
But more often, it's based on the implicit, nonverbal, subtle
cues that we exchange with one another with empathy
at the core of building this trust.
Technology today has a lot of IQ--
a lot of cognitive intelligence, but no EQ--
no emotional intelligence.
It's the missing component.
So my entire career, I've been asking,
what if technology could identify human emotions just
as we can?
What if your computer could tell the difference
between a smile and a smirk?
Both involve the lower half of the face,
but they have very different meanings.
So how can we build AI that understands human?
The way to do this is to look at how humans do it.
Only 7% of how we communicate our mental states
is based on the actual choice of words we use.
93% is non-verbal, split between our facial expressions
and our gestures--
I do a lot of these--
and 38% of vocal intonations, how fast
are you speaking, how much energy is in your voice?
A lot of my career has been focused on the face.
It's a very powerful canvas of communicating human emotion.
The science of facial emotions has existed for over 200 years.
This guy, Duchenne, used to electrically stimulate
our facial muscles to map how these facial muscles move.
We don't do that anymore, thankfully.
And then in the late 1970s, Paul Ekman and his team
published the Facial Action Coding System--
an objective method of mapping each facial muscle movement
to an action unit, to a code.
So, for example, when you smile-- and try this with me--
when you smile, you're pulling the zygomaticus muscle
and that's basically the lip corner pull or action unit 12.
When you furrow or frown, this is the action unit 4,
it's the brow furrow.
And it's typically an indicator of a negative emotion
like confusion or anger.
There's about 45 of these facial muscles and it
takes about 100 hours of training
to become a certified fax coder or face reader--
very laborious, very time intensive.
And to code every minute of video
takes you about five minutes of watching the video
in slow motion and say, oh, ah-ha, I see a,
you know, an eyebrow raise or a squint.
We don't need to do that anymore.
Instead, we use computer vision and machine learning and deep
learning to automatically train algorithms to detect
these facial expressions.
We use hundreds of thousands of examples
of people smiling and smirking and furrowing their eyebrows
to train these algorithms.
And the deep learning network is able to distill
what is common between all these smiles, what's
common between all these frowns, and that's how it learns.
And so, to simplify it, the first step of the process
is to triangulate where the face is,
find facial landmarks like your eyebrows, your mouth,
or your nose.
And then you feed that region into a deep neural network
which is able to distill what expressions are happening
on the face and mapping those into a number
of emotional and cognitive states.
Everything from joy, surprise, anger, disgust
to more complex states like fatigue, attention,
cognitive overload, confusion, and more.
In our work over the past number of years,
we have amassed the world's largest emotion repository.
9 and 1/2 million face videos that we've collected--
with everybody's opt-in and consent--
in 90 countries around the world.
This roughly translates to about 5 billion facial frames.
It's by far the largest repository
of actual emotional responses out there.
And we use that data to train, but also
validate our algorithms.
There are so many applications of this technology
that transforms industries.
My big vision is that in the next few years,
we're going to see emotion AI becoming the de
facto human-machine interface.
So basically we would be interacting with our devices
just the way we interact with one another.
Through conversation.
We're already seeing that with conversational devices
like Alexa and Siri.
Through perception.
Again, we're already starting to see devices
that have cameras on them.
But perhaps most importantly, through empathy and emotional
intelligence.
There's a lot of applications.
I'm going to focus on a few.
The first application is around quantifying
how consumers emotionally engage with products and brands
around them.
The way this works is we send out surveys to people,
we ask them to watch a piece of content.
It could be an online video ad, it could be a movie trailer,
it could be an actual TV show, it could be learning content.
But the idea is we want to capture
the emotional engagement and the emotional response people
have to this content.
We asked people to turn their cameras on--
consent and opt-in is really important--
and then we're able to capture these moment-by-moment
responses.
Our technology is being used by 25% of the Fortune Global 500,
as well as leading market research firms that
use our technology to quantify the emotional response
that consumers and viewers have to their content.
I thought it would be fun to actually
show you one of the video ads that we have tested
and we have permission to share publicly.
So let's watch together.
[MUSIC - TONY DALLARA, "COME PRIMA"]
[ITALIAN SINGING]
Uh, mom got there first.
So today, we have tested over 50,000 ads worldwide.
And this particular ad scores in the top 10%
of all of these ads in the ninetieth percentile.
It garners very strong emotional engagement, which
is our expressiveness score.
It garners a lot of smiles.
And you can see the moment by moment smile curve
for everybody that's watched that ad that we
were able to record data from.
What is really fascinating about this particular ad
is if you compare the first time people saw it,
which is the solid green line.
With the second time, people saw it,
which is the dotted green line.
You can actually see that people have memory of the ad.
They're anticipating when it's really funny
and they're laughing even before that scene starts, which
is exactly what you want.
And, more importantly, we're taking the viewers
on an emotional journey which culminates
with a very positive response at the end which coincides
with the brand reveal.
And, again, we know from our research
that these kinds of metrics are very positive.
I love this particular example, because it's
an example of where brands like Coca-Cola and Unilever
are using this technology to really push online video
advertising and become more inclusive and more progressive.
But they have to do that in a very thoughtful way.
And they're able to capture these subconscious visceral
responses to their ads.
We also know, from a lot of work we've done with these brands
where we've correlated the emotional responses
to actual consumer behavior, that this kind
of emotional journey and positive emotional responses
correlate very highly with things
like sales lift, purchase intent, and vitality.
And because we've tested 50,000 ads worldwide,
we're able to create benchmarks.
So, for example, we know that in the US,
pet care and baby care ads elicit the most enjoyment.
Interestingly, in Canada, it's the cereal ads
that elicit the most enjoyment.
And, unfortunately, telecom ads elicit almost no enjoyment
at all.
They're pretty boring to watch.
So this data is really, really insightful,
and provides brands and marketers
with novel insights that helps them
make decisions around media spend
as well as how to optimize their advertising content.
And it's not just advertising, you
can generate that same type of data with TV shows as well.
And this is one of my favorite examples.
It's a sitcom that we tested a while back for CBS.
It was called Friends With Better Lives.
I don't think it aired for long.
It didn't do very well.
But here, again, you're seeing the smile curve.
And we superimpose the characters
of the scene on that curve.
And you can see that there are two particular characters,
every time they show up they're just not funny.
They're basically the trough in this smile curve.
And, again, very, very interesting data
that the producers were able to use
to switch out these characters.
Moving on from the world of understanding
the emotional engagement people have
with their content to another area that we're
spending a lot of time in.
And it's the future of mobility and transportation.
When we first started working in the space,
we got approached by a number of automakers
around the world who wanted to repurpose our emotions sensing
technology for the car.
And so to do that we wanted to see, well,
how did people behave in the car anyways, especially drivers.
And so we set out to collect some data.
A lot of us in the company were very skeptical that perhaps
because people knew that there was
going to be a camera in their dashboard
that there wouldn't be any interesting emotional
or cognitive responses.
And we were wrong.
So I'm going to show you some examples of video clips.
Again, these people knew that the camera was
in their vehicle, they installed it,
they consented, and yet we saw very interesting driving
behaviors.
In this particular case, this dad is driving the vehicle.
He's extremely drowsy.
There are four levels of drowsiness.
He's basically asleep.
And you can also see that his toddler
daughter is in the backseat.
So very, very unsafe driving.
And I can certainly empathize with that.
There are many times back when we actually
traveled when I'd be back from a long trip and jet-lagged
and exhausted while I'm driving my kids.
So I certainly can resonate with that.
But that's such an easy example that the technology can pick.
And you can see here that the technology was
able to detect things like eye closure, mouth open,
and levels of drowsiness.
And you could imagine how the car can interject in that case.
Here's another example.
So she's driving.
She's also texting while driving, so very distracted.
But, oh, she has two phones in her hand,
so she's very distracted.
Her eyes are off the road.
Her hands are off the wheel.
You do not want to be driving next to this woman.
But, again, that's an example that technology can pick up
by just looking at her glance behavior,
her head pose information, but also combining it
with things like object detection.
So we can detect that she has not only one, but two cell
phones in her hand.
We can flag that as a very highly distracted driver.
And, again, the car can intervene one way or another.
You can imagine if this were a Tesla
and it can jump into semi-autonomous mode,
it can basically say, hang on a second,
I'm going to be a better driver at this moment in time
than you are, I'm taking control.
We have done a lot of research in collaboration
with MIT'S advanced vehicle technology group,
where we did the first-ever large scale study looking
at driver behavior over time.
And so in this particular case, you're
looking at participant number 66.
She's female.
She's 22 years old.
And all her Monday drives are concatenated in the Monday row.
And all her Tuesday drives are concatenated on Tuesday,
and so on.
And you can see that she's pretty tired during the week.
There's a lot of yawning happening
at the beginning of the day.
And during the weekend and also towards the end of the day,
there's a lot more positive expressions.
They say a picture is worth a thousand words.
I believe a video is worth even more.
So I'll just show you examples of her PM versus AM versus PM
driving behaviors.
So, again, these were all mined using our algorithms.
So we basically invoked the yarn classifier
to detect all of her yawning behaviors
across all of her driving sessions.
So you can see that she's often very exhausted in the morning.
And you can imagine, if that's a recurring pattern,
then the car already knows that she's
going to come in exhausted.
Maybe there are suggestions proactively
that the car can make to the driver given her profile.
Now contrast that to her PM driving behaviors.
You can see that she's already looking much more awake.
She's a lot more animated, a lot more smiles, a lot more
positive expressions.
And once again if the car had her driver profile
it could customize and personalize
the driving experience based on her emotional experiences.
So we're working very closely with a number of automakers
to re-imagine this transportation experience.
Not only in cars today where we can
focus on the driver, but also vehicles of the future where
we really have this vision of an incumbent sensing solution that
looks at the driver but the other occupants in the vehicle
as well.
As well as other objects in the car.
Is there a child left behind?
Did you leave your phone behind?
How many occupants are in the vehicle?
What is their state?
Can you personalize the music, the content, the lighting,
and so on?
But, of course, a lot of us are not
spending a lot of time in our car
anymore during this global pandemic.
And, in fact, a lot of us are spending much more of our time
on video conferences and on virtual events.
Making human connection via technology is really hard.
And it's primarily because the main method of communication
is nonverbal.
And sometimes when we are on these virtual conferences
and virtual events and video events,
you're not really tapping into the energy and the expressions
of the audience.
I certainly feel that myself.
I just launched my book Girl Decoded as I mentioned.
And I had to pivot from a book tour
where I was supposed to be traveling nonstop March, April,
and May, to all of these virtual book tours and virtual book
talks.
And it's so different because in a live environment
I can see the audience.
I can see you all and I can riff off of your energy.
And I can customize and personalize
and adapt my content based on how you're engaging with me.
When I do this virtually, it's often a one-way conversation
or at least it feels to me like it's a one-way conversation.
And I find it really painful and unsettling.
But I think there's an opportunity to change that.
What if we integrate it, emotion AI,
as a way to aggregate audience response,
capture all of your facial expressions anonymously.
I don't need to see everybody's faces.
In fact, that would probably be overwhelming as I'm presenting.
But if I was able to see a moment by moment trace of how
positive or negative did you--
are you engaging with this content,
that would be very powerful information
that would allow me to customize and get
a sense of the level of engagement of the audience.
And so I wanted to show you one example.
Just from an internal team meeting on zoom where
we recorded teams' responses.
And you're able to see here the aggregated response.
Of course, there's different ways to visualize this data,
but we're able to track everybody
and we can aggregate people's smiles, responses.
They were actually talking about-- this
was early in March, just when the pandemic broke.
And we were talking about what's going
to happen and work from home.
And people were sharing their stories.
And there was a lot of empathy and smiling behavior.
So we're able to track that.
Now compare that to a really flat, boring meeting
where everybody's just not engaging.
You can tell the difference.
And we're able to quantify that data for real-time engagement,
but also do that after the fact as post analytics as well.
Similarly, in an online learning environment,
what if a teacher could measure the emotional engagement
of his or her students just as he or she would
in a live classroom.
That's what an awesome teacher does, right?
You riff off of the engagement of the students
and then you can personalize the learning experience.
And maximize learning outcomes as well.
Similarly, there's applications in mental health as well.
Today when you walk into a doctor's office
they don't ask you for your temperature
or they don't ask you for your blood pressure,
they just measure it.
So what if doctors could objectively
measure how you are feeling the way they
measure other vital signs?
Unfortunately, the gold standard in mental health
is still on a scale from 1 to 10--
how depressed are you?
How suicidal are you?
How much pain are you in?
And we can bring emotion AI and apply it
in a way that brings objective measures of mental health
conditions.
One area that is very near and dear to my heart,
it's the very first application of emotion AI that I explored
is autism.
Individuals on the autism spectrum struggle
with reading and understanding non-verbal signals.
They find the face, in particular, very overwhelming.
And sometimes they avoid it altogether.
They avoid face and eye contact altogether.
So we are partnered with a company called Brain Power.
They use Google Glass and our technology to help individuals
on the autism spectrum learn about these non-verbal signals
in a very fun and gamified way.
So I'm going to show you a short video demonstrating that.
What do you see on-screen?
Mom.
8-year-old Matthew Krieger has been diagnosed with autism.
A lot of the trouble he gets into with other kids
is he thinks he's funny.
And doesn't read at all that he's not
or that they're annoyed or angry.
Matthew's mother, Laura, signed him
up for a clinical trial being conducted by Ned Sahin.
I want to know what's going on inside the brain of someone
with autism.
And it turns out parents want to know that too.
You get points for looking for a while
and then even for looking away and then looking back.
Sahin's company, Brain Power, uses Affectiva's software
in programs Matthew sees through Google Glass.
These games are trying to help him understand
how facial expressions correspond to emotions
and learn social cues.
One of the key life skills is understanding
the emotions of others.
And another is looking in their direction
when they're speaking.
Looking at your mom and while it's green you're getting
points when it starts to get orange and red you're--
you slow down with the points.
I am looking at you.
You are looking me.
Just a few minutes later, the difference in Matthew's gaze
overwhelmed his mother.
I want to cry.
Why?
Because when you look at me it makes
me think you haven't really before because you're
looking at me differently.
So Brain Power has about 400 of these Google Glass systems
deployed in families across the United States.
And the main question they're trying to answer-- we're
already seeing improvement in terms
of the kids' social and non-verbal understanding,
while they're wearing the glasses.
The key question is what happens when they take off the glasses.
Does this learning generalize?
There's also other applications of this technology
in mental health.
For example, a system for early detection of Parkinson's.
Erin Smith-- she's now a student at Stanford.
Way back when she was a junior in high school,
she emailed us and she said I've been watching
this documentary about Parkinson's and I
want to use your technology.
How much does it cost?
And I remember our head of sales asked me what do I tell her?
She can't afford our technology.
And I said just give it to her for free.
What is she going to do with it anyways?
And Erin disappeared for a couple of months and came back.
She had partnered with the Michael J.
Fox foundation and built this system to identify facial
biomarkers of Parkinson's.
And she's continued to work on this research.
Very inspiring young woman.
And I feel very proud that we are playing
a small part of her journey.
We also know that there are facial and vocal biomarkers
of depression.
And there's a lot of work being done to flag suicidal intent
based on these signals.
This is work that we are collaborating
with Professor Steven Benoit and others as well.
But this brings up a very important topic,
which is, OK, there's a lot of applications
of this technology, where do we draw the line?
And I feel very passionately about this concept
of the ethical development and deployment of AI.
It's not just about acknowledging that there's
so much potential for good, but it's also
acknowledging where this can be abused
and the unintended consequences of this technology.
A number of years ago when we were raising
money for the company, we got approached
by an agency that wanted to give us a lot of money--
$40 million at the time, which was a lot of money
for our little startup--
on condition that they wanted to use the technology for lie
detection and surveillance.
And that was really not in line with our core values
of respecting people's privacy and consent.
And acknowledging that, as a user,
this data is very personal.
And if I'm going to be sharing it,
that I need to know exactly who's using it,
how is it being used, and also what's in it for me?
So we spent a lot of time thinking
about this power asymmetry--
what value am I getting in return
for sharing this very personal data.
What's really cool about this is that the industry
is taking a lead in defining these best
practices and guidelines.
So we are part of a consortium called the Partnership on AI.
It was started by the tech giants--
Amazon, Google, Facebook, Microsoft.
And they have since invited a number of startups
like Affectiva.
But also other stakeholders like ACLU and Amnesty International.
And I'm part of the FATE committee,
which is fair, accountable, transparent, and equitable AI.
And our task is to come up with these guidelines
around thoughtful regulation.
We need regulation, but it needs to be thoughtful.
We don't want to completely squander innovation,
but at the same time, we really need
to think through where do we draw the line
and what does this thoughtful regulation look like.
And, at the end of the day, my mission
is to humanize technology before it dehumanizes us.
And I want us to put the emphasis back on the human,
not on the artificial.
And to wrap up, if this made you a little bit more curious
about emotion AI and its applications
and its implications, I am giving away a signed book
to the first three people who post on my social media--
LinkedIn, Facebook, Twitter, Instagram-- using the hashtags
#LIVEWORX and #GIRLDECODED With a comment or a question
about this presentation.
Thank you.
[LIVEWORX THEME]
Thank you for joining us Dr. el Kaliouby.
What a compelling presentation.
So that concludes our third session of the day.
Next up at the top of the hour, we
will hear how companies are leveraging SAAS-based CAD, PLM,
and augmented reality as key tools for them
to not only survive disruption but embrace this new normal.
We'll be right back after a short break.
[LIVEWORX THEME]