U.S. patent application number 17/140821 was filed with the patent office on 2021-07-01 for system and method for gathering and analyzing biometric user feedback for use in social media and advertising applications.
The applicant listed for this patent is Nielsen Consumer LLC. Invention is credited to Ravi Kanth V. Kothuri, Brian Levine, Carl Marci.
Application Number | 20210196188 17/140821 |
Document ID | / |
Family ID | 1000005459144 |
Filed Date | 2021-07-01 |
United States Patent
Application |
20210196188 |
Kind Code |
A1 |
Levine; Brian ; et
al. |
July 1, 2021 |
SYSTEM AND METHOD FOR GATHERING AND ANALYZING BIOMETRIC USER
FEEDBACK FOR USE IN SOCIAL MEDIA AND ADVERTISING APPLICATIONS
Abstract
Systems and methods for measuring biologically and behaviorally
based responses to social media, locations, or experiences and
providing instant and continuous feedback in response thereto are
disclosed. An example system includes a first sensor to determine
an emotional response of a user exposed to a social media
application, a second sensor to determine a current activity of the
user, and a third sensor to determine an environment of the user.
The example system also establishes a priority schedule based on
the emotional response, the current activity, and the environment.
The system also correlates, based on the priority schedule, an
advertisement with at least one of the emotional response,
activity, or the environment. In addition, the example system
presents the advertisement based on the priority schedule and the
correlation of the advertisement with the at least one of the
activity, the environment, or the emotional response.
Inventors: |
Levine; Brian; (Needham,
MA) ; Marci; Carl; (Boston, MA) ; Kothuri;
Ravi Kanth V.; (Nashua, NH) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Nielsen Consumer LLC |
New York |
NY |
US |
|
|
Family ID: |
1000005459144 |
Appl. No.: |
17/140821 |
Filed: |
January 4, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15400287 |
Jan 6, 2017 |
10881348 |
|
|
17140821 |
|
|
|
|
13779497 |
Feb 27, 2013 |
9569986 |
|
|
15400287 |
|
|
|
|
61603528 |
Feb 27, 2012 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
A61B 5/389 20210101;
A61B 5/02405 20130101; A61B 5/0077 20130101; G09B 25/00 20130101;
A61B 5/11 20130101; A61B 5/486 20130101; G06Q 50/01 20130101; A61B
5/165 20130101; A61B 3/112 20130101; A61B 5/369 20210101; A61B
2503/12 20130101; A61B 5/02055 20130101; A61B 3/113 20130101; A61B
5/0533 20130101; A61B 5/0816 20130101; G09B 19/00 20130101; G06N
20/00 20190101 |
International
Class: |
A61B 5/00 20060101
A61B005/00; G09B 25/00 20060101 G09B025/00; G09B 19/00 20060101
G09B019/00; G06N 20/00 20060101 G06N020/00; G06Q 50/00 20060101
G06Q050/00; A61B 5/369 20060101 A61B005/369; A61B 5/389 20060101
A61B005/389; A61B 3/11 20060101 A61B003/11; A61B 3/113 20060101
A61B003/113; A61B 5/0205 20060101 A61B005/0205; A61B 5/16 20060101
A61B005/16 |
Claims
1. (canceled)
2. A system comprising: a sensor to collect biometric data from a
user; a holder to secure the sensor to the user; a memory; and a
processor to: determine a first activity of the user, the first
activity related to use of an application on a mobile device;
determine a second activity of the user, the second activity
related to a motion data of the user; determine an emotional state
of the user based on the biometric data; and provide a media
content recommendation based on the emotional state of the user and
at least one of the first activity or the second activity.
3. The system of claim 2, wherein the processor is to prioritize
one of the first activity or the second activity over the other of
the first activity or the second activity based on a machine
learning model.
4. The system of claim 3, wherein the machine learning model
includes prioritization of the first activity or the second
activity based on a prior use of the mobile device and a
prioritization scale that provides a first weight to a first
activity type and a second weight to a second activity type, the
second activity type different from the first activity type.
5. The system of claim 2, wherein the application is a first
application, and the processor is to operate a second application
to train the machine learning model to prioritize the first
activity or the second activity.
6. The system of claim 2, wherein the sensor is a first sensor to
collect a first biometric data, the biometric data including one of
a galvanic skin response, a heart rate, a skin temperature, an eye
movement, an electroencephalogram, an electromyogram, or a pupil
dilation, the system further including a second sensor to collect a
second biometric data, the second biometric data different than the
first biometric data, the processor to determine the emotional
state of the user based on the first and second biometric data.
7. The system of claim 2, wherein the holder includes at least one
of a wrist band, an arm band, or a headpiece.
8. The system of claim 2, wherein the motion data includes velocity
and the processor is to provide the media content recommendation
based on the velocity.
9. The system of claim 8, wherein the media content recommendation
includes a first level of activity for a first velocity and a
second level of activity for a second velocity, the first level of
activity lower than the second level of activity and the first
velocity lower than the second velocity.
10. The system of claim 2, wherein the motion data includes a
direction, and the processor is to: determine a location of the
user based on the direction; and determine the media content
recommendation based on the location.
11. A tangible computer readable storage medium comprising
instructions that, when executed, cause at least one processor to
at least: determine a first activity of a user, the first activity
related to use of an application on a mobile device; determine a
second activity of the user, the second activity related to a
motion data of the user; determine an emotional state of the user
based on biometric data; and provide a media content recommendation
based on the emotional state of the user and at least one of the
first activity or the second activity.
12. The storage medium of claim 11, wherein the instructions cause
the processor to prioritize one of the first activity or the second
activity over the other of the first activity or the second
activity based on a machine learning model.
13. The storage medium of claim 12, wherein the machine learning
model includes prioritization of the first activity or the second
activity based on a prior use of the mobile device and a
prioritization scale that provides a first weight to a first
activity type and a second weight to a second activity type, the
second activity type different from the first activity type.
14. The storage medium of claim 11, wherein the motion data
includes velocity and the instructions cause the processor to
provide the media content recommendation based on the velocity.
15. The storage medium of claim 14, wherein the media content
recommendation includes a first level of activity for a first
velocity and a second level of activity for a second velocity, the
first level of activity lower than the second level of activity and
the first velocity lower than the second velocity.
16. The storage medium of claim 11, wherein the motion data include
a direction, and the instructions cause the processor to: determine
a location of the user based on the direction; and determine the
media content recommendation based on the location.
17. A method comprising: determining, by executing instructions
with a processor, a first activity of a user, the first activity
related to use of an application on a mobile device; determining,
by executing instructions with the processor, a second activity of
the user, the second activity related to a motion of the user;
determining, by executing instructions with the processor, an
emotional state of the user based on a biometric data; and
providing, by executing instructions with the processor, a media
content recommendation based on the emotional state of the user and
at least one of the first activity or the second activity.
18. The method of claim 17, wherein the application is a first
application, the method further including operating a second
application to train a machine learning model to prioritize one of
the first activity or the second activity over the other of the
first activity or the second activity based on a prior use of the
mobile device and a prioritization scale that provides a first
weight to a first activity type and a second weight to a second
activity type, the second activity type different from the first
activity type.
19. The method of claim 17, wherein the motion data includes
velocity, the method further including providing the media content
recommendation based on the velocity, the media content
recommendation including a first level of activity for a first
velocity and a second level of activity for a second velocity, the
first level of activity lower than the second level of activity and
the first velocity lower than the second velocity.
20. The method of claim 17, wherein the motion data include a
direction, the method further including: determining a location of
the user based on the direction; and determining the media content
recommendation based on the location.
Description
RELATED APPLICATIONS
[0001] This patent arises from a continuation of U.S. application
Ser. No. 15/400,287 (now U.S. Pat. No. 10,881,348), titled "SYSTEM
AND METHOD FOR GATHERING AND ANALYZING BIOMETRIC USER FEEDBACK FOR
USE IN SOCIAL MEDIA AND ADVERTISING APPLICATIONS," filed Jan. 6,
2017, which is a continuation of U.S. application Ser. No.
13/779,497 (now U.S. Pat. No. 9,569,986), titled "SYSTEM AND METHOD
FOR GATHERING AND ANALYZING BIOMETRIC USER FEEDBACK FOR USE IN
SOCIAL MEDIA AND ADVERTISING APPLICATIONS," filed on Feb. 27, 2013,
which claims priority to U.S. Provisional Application No.
61/603,528, titled "SYSTEM AND METHOD FOR GATHERING AND ANALYZING
BIOMETRIC USER FEEDBACK FOR USE IN SOCIAL MEDIA AND ADVERTISING
APPLICATIONS," filed Feb. 27, 2012. U.S. application Ser. No.
15/400,287; U.S. application Ser. No. 13/779,497; and U.S.
Provisional Application No. 61/603,528 are incorporated herein by
this reference in their entireties.
FIELD OF THE DISCLOSURE
[0002] The present invention relates to systems and methods for
measuring biologically and behaviorally based responses to social
media, locations, or experiences and providing instant and
continuous feedback in response thereto. In addition, the system
and methods of the invention are capable of monitoring stress
levels and well-being. The system and methods of the invention may
be implemented using a cloud-based infrastructure for remote
monitoring.
BACKGROUND
[0003] There are many different kinds of audio, visual and
audio-visual presentations and activities that people are exposed
to every day. These presentations serve as sensory experiences that
stimulate our senses and are known to result in biologically based
responses that can be measured electronically and mechanically (for
example, heart rate, respiration rate, blood pressure, and skin
conductance).
[0004] Likewise, people now have the ability to provide instant and
continuous feedback in response to various social media such as
pictures, websites, and the like. Such feedback can be provided on
computers, tablets, smart phones, and other devices that access the
internet. For example, "like" is a way to give positive feedback or
to connect with things a person is interested in on the popular
social media site Facebook.RTM.. In particular, the "like" button
on Facebook.RTM. is a button a user may click on after looking at
most content on Facebook.RTM., which is then reported in newsfeeds
to "friends". Websites unrelated to Facebook.RTM. may also use a
"like" button that enables a website visitor to click on the button
to let his/her friends know that they like the site. For example,
after clicking on the website's "like" button, a pop-up will
request login to Facebook.RTM. (or sign-up if not already a member)
and a post on the user's Facebook.RTM. page will let his/her
friends know that he/she likes the site. When used on a mobile
device, such as a smart phone, the "like" button is merely an
integrated hardware "Facebook.RTM." button on the phone that does
nothing more than take the user to Facebook.RTM. when the button is
pressed.
[0005] Similarly, the "Pin It" button on a computer or mobile
device allows users to grab images and videos from around the web
and add them to an on-line pinboards created by the users. Other
users can view the pinboards, comment, and "re-pin".
[0006] Capabilities have also been introduced to allow people to
use mobile devices to interact with their environment. For example,
location-based social networking websites allow users to "check-in"
at venues using a mobile website, text messaging, or a
device-specific application by selecting from a list of venues the
application locates nearby. The location is based on GPS hardware
in the mobile device or the network location provided by the
application. Each check-in awards the user points or other types of
rewards.
[0007] Even with these advances in technology, the ability to
measure and evaluate the user experience, effectiveness, and the
usability of social media, locations, or experiences has been
limited. In fact, current methodologies for measuring or evaluating
user experience, effectiveness, and usability of websites and other
interactive internet and software media has thus far been limited
to traditional self-report, i.e., relying on the user to use the
"like" button and to accurately reflect his/her actual response to
the social media, which may be subject to error, bias, or low
compliance.
[0008] Thus, a need in the art exists for a system and method that
integrates passive biometric sensors into smart phones or other
portable devices to collaborate with or eliminate the "like" button
and replace it with a continuous stream of emotional responses
across all experiences. A need also exists in the art for a
biometrically enabled suite of applications that are built into
smart phones, tablets, and other social media enabled devices to
determine when a user unconsciously likes (or dislikes) their
current experience, e.g., a web page, "app", song, video, location,
or other experience, and also to remotely monitor the user's stress
levels and well-being.
SUMMARY OF THE INVENTION
[0009] The present invention is related to a system and method that
integrates passive biometric sensors into smart phones or other
portable devices to collaborate with or eliminate the "like" button
and replace it with a continuous stream of emotional responses
across all experiences. In one embodiment of the invention,
biometric responses may be automatically detected using
physiological signal measurement, via an explicit button on the
mobile device, or via a continuous biometric data collector. In one
embodiment, the biometric data collector is associated with the
mobile device, such as a case that incorporates sensors to collect
data from the hands of the user.
[0010] In another embodiment of the invention, the response may be
automatically associated with specific content stimuli, not only in
the mobile phone, but also in the surrounding environment of the
consumer using a specialized content-associating system
(predicting/associating by "proximity" of the response to the
variety of content/activities).
[0011] In yet another embodiment, the system and method of the
invention provides content recommendations by transforming
emotional responses to music, video, and other content into more
effective and pervasive recommendations. In still another
embodiment, the system and method of the invention enhances
advertisement targeting by combining biometric sensing and
location-based data based on anticipated emotional responses. In
yet another embodiment, the system and method of the invention add
emotional intelligence to social networks by generating a complex
dataset of interests from which users can share and connect with
friends. In still another embodiment, the system and method of the
invention continually performs background searches based on
emotional reactions to any activity in order to push more relevant
information to consumers.
[0012] The present invention is also directed to a system for
determining the emotional response of a user to social media
including: a plurality of biometric sensors in a first device
operable to measure a plurality of biometric parameters for a user
of the first device when exposed to a social media application,
marketing or advertising applications, or a combination thereof; a
computer system operable to receive data representative of the
plurality of biometric parameters, wherein the computer system
further includes a memory capable of storing the data, wherein the
system is capable of determining an emotional response of the user,
identifying the causation of the emotional response, and delivering
the emotional response and the causation to a listening application
for appropriate action.
[0013] In one embodiment, the plurality of biometric parameters
include at least two of galvanic skin response, heart response,
motion, skin temperature, breathing, EEG, EMG, pupil dilation, eye
tracking, facial expressions, or a combination thereof. In another
embodiment, the first device includes a device capable of accessing
a social media application, a marketing or advertising application,
or a combination thereof. For example, the first device may be
selected from the group consisting of mobile devices, tablets,
laptops, desktops, goggles, watches, and combinations thereof.
[0014] In another embodiment, the first device further includes
image acquisition functionality, and wherein the image acquisition
functionality includes static photographs, videos, or a combination
thereof. For example, the image acquisition functionality may
include at least one camera, and wherein the at least one camera is
capable of acquiring a plurality of images including images in the
user's view, images of the user, or a combination thereof. The
system may also be capable of image recognition of the images in
the user's view. In yet another embodiment, the system may be
capable of processing facial recognition using the images, videos,
or a combination thereof of the user.
[0015] The present invention is also directed to a system for
determining the emotional state of a user to social media
including: a first device capable of accessing a social media
application marketing or advertising applications, or a combination
thereof a second device including at least one biometric sensor
operatively connected to the first device, wherein the at least one
biometric sensor is operable to measure at least one biometric
parameter for a user of the first device when exposed to social
media; a computer system operable to receive data representative of
the at least one biometric parameter, wherein the computer system
further includes a memory capable of storing the data, wherein the
system is capable of determining an emotional response of the user
to the social media, and wherein the system is capable of
delivering the emotional response to a targeted application for
appropriate action.
[0016] In one embodiment, the at least one biometric parameter
includes galvanic skin response, heart response, motion, skin
temperature, breathing, EEG, EMG, pupil dilation, eye tracking,
facial recognition, or a combination thereof. In another
embodiment, the second device includes at least two biometric
sensors. In still another embodiment, the at least two biometric
sensors are operable to measure at least two biometric parameters
selected from the group consisting of galvanic skin response, heart
response, motion, skin temperature, breathing, EEG, EMG, pupil
dilation, eye tracking, facial recognition, and combinations
thereof.
[0017] The first device may further include image acquisition
functionality. In one embodiment, the image acquisition
functionality includes at least one camera, and wherein the at
least one camera is capable of acquiring a plurality of images
including at least one image in the user's view, at least one image
of the user, or a combination thereof. The system may be capable of
image recognition of the at least one image in the user's view,
facial recognition of the at least one images of the user, or a
combination thereof.
[0018] In another embodiment, the system may be capable of
collecting motion data to determine activity level, direction, or a
combination thereof. In yet another embodiment, the system is
capable of determining a current activity of the user. In still
another embodiment, the system may be capable of feeding the
current activity and the emotional state to listening applications,
wherein the listening applications reside on the device or on a
central server. The listening application may be capable of
providing tailored content to the user based on the emotional state
and current activity. The listening application may also be capable
of processing feeds of the emotional state and current activity for
social media.
BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Further features and advantages of the invention can be
ascertained from the following detailed description that is
provided in connection with the drawing(s) described below:
[0020] FIG. 1 is a side view of a suitable biometric data collector
for mobile devices according to an embodiment of the invention;
[0021] FIG. 2 is a front and rear view of the suitable biometric
data collector for mobile devices shown in FIG. 1;
[0022] FIGS. 3A and 3B are front views of a suitable biometric data
collector for mobile devices according to another embodiment of the
invention;
[0023] FIGS. 4A and 4B are a perspective views illustrating a
suitable biometric data collector in position on a mobile device
user's wrist according to an embodiment of the invention;
[0024] FIG. 5 is a front view illustrating a suitable biometric
data collector according to an embodiment of the invention;
[0025] FIG. 6 is a perspective view of a biometric data collector
with a mobile device in place according to an embodiment of the
invention;
[0026] FIG. 7 is a perspective view of a suitable biometric data
collector according to an embodiment of the invention; and
[0027] FIG. 8 is a perspective view of a suitable biometric data
collector according to an embodiment of the invention.
DETAILED DESCRIPTION
[0028] The present invention is directed to a social media
interface that will execute social media functions when pressed,
rather than simply open a social media application. In particular,
the system and method of the invention will accomplish at least one
of the following:
[0029] (a) determine a user's current activities;
[0030] (b) among the current activities, select the one that is the
most likely activity that the user wants to take action on;
[0031] (c) undertake a specific defined social media action;
and/or
[0032] (d) serve related information to the user based on the
social media action taken.
[0033] The result of the system and method of the invention is the
creation of a continuous stream of social media data that provides
a richer and more usable experience than active and discreet
"liking" in existing social media applications.
[0034] In addition to measuring relevant emotional responses, the
system and method of the invention is also capable of computing
overall stress and other vital signals of the user to establish the
general stress levels and well-being of the user. In one embodiment
of the invention, the system and method of the invention is
implemented at least in part with a cloud-based infrastructure that
allows remote monitoring of the user's vital signs and biological
signatures for well-being and derived parameters thereof. This
information can then be combined with the user's location and
activities over time to further define the drivers of well-being
(and the opposite thereof (i.e., stress)). In this aspect of the
invention, the system may also include the creation of web/mobile
interfaces for sharing the user's well-being and derived parameters
thereof with the user's specified social network, specified
contacts or groups, as well as with designated family or designated
health care providers.
[0035] In one embodiment, the system and method of the invention
incorporates at least one affinity button or software application
on a mobile device, such as a smart phone, a tablet, or other
device that is capable of accessing a social media application. For
example, to determine the user's current activities, when the
affinity button is pressed or the application is started, the
device will use some or all software and sensors available to
determine the possible set of activities the user may be engaged
in. This includes, but is not limited to, all or a subset of the
following:
[0036] (1) The current application being used;
[0037] (2) The user's current location to determine what is located
at that location;
[0038] (3) Images and/or video within the user's view and/or images
of the user;
[0039] (4) Neighboring devices; and/or
[0040] (5) Motion data.
[0041] In particular, with regard to (1), a software layer may
determine if the mobile device is currently running any
applications and which application is in the foreground. For
example, in one embodiment, the process/application using the most
central processing unit (cpu) in the current time (e.g., past 15
seconds, past 60 seconds, or another predetermined time interval)
may be the application in the foreground. In this aspect, the
following steps may be performed:
[0042] (a) Determination of the application currently running (to
the degree the OS allows);
[0043] (b) If the foreground application is a web browser,
determining at least (i) the current URL, (ii) text characters and
image tags on the current page, and (iii) text characters and
images in view (accounting for any scroll in the pages, and the
like);
[0044] (c) If the foreground activity is a photo viewer (either
native or embedded in another app), determining the subject, time,
date, and the like by gathering and analyzing the metadata in the
photo and leveraging image recognition capabilities (either
developed specifically for this application or commercially
available);
[0045] (d) If music, videos, or eBooks are being played or viewed
on the mobile device, return all accessible information about the
content;
[0046] (e) If the audio/video is external to the mobile device,
allowing the user to record a "snapshot" and capture an affinity to
the snapshot such that the system may then go back to the internet
to extract metadata for the audio/video content (using tools/apps
that already exist for audio/video);
[0047] (f) If a map or navigation application, determine which
location is being researched; and/or
[0048] (g) If other applications, gather and analyze any data the
application allows the operating system to access. For example,
application vendors will likely be encouraged to use software hooks
which the social media button can use to determine the exact
activity within an application is being accessed.
[0049] With regard to (2), the mobile device may use the cellular
networks, accessible Wi-Fi and/or GPS to (a) determine the user's
current location and (b) search to determine what activities may be
available at that location. In particular, the determination of the
available activities may favor businesses, parks, addresses in the
users' address book, tourist spots, or other activities
identifiable through the internet. For example, in one embodiment,
locations at the GPS coordinates that have the most details online
or in point-of-interest databases will be favored.
[0050] With regard to (3), the mobile device may acquire a
plurality of snapshots and/or video upon a button press or, if
programmed, automatically. The plurality of snapshots and/or video
may include images in view of the user and images of the user. In
one embodiment, the plurality of images/videos may then be run
through image recognition software created for the application to
determine the image in view (e.g., a product, location, logo,
person, etc.). In another embodiment, the plurality of
images/videos may then be run through existing software (such as
Google Goggles or the like) to determine the image in view (e.g., a
product, location, logo, person, etc.). Facial detection software
may also be used to determine which specific facial-emotion is
being expressed for the activity.
[0051] With regard to (4), the mobile device may use Bluetooth to
determine whether there are any known devices in the area. In one
embodiment, pressing the affinity button near a second user with
the same capability on his/her mobile device will identify that
device and add it to the identifiable device list as well as the
second user's information to the primary user's phone's contact
database.
[0052] With regard to (5), motion data may be obtained to determine
whether the mobile device is manipulated in such a way to indicate
that the user is attempting to capture an image. If such action is
determined, the image recognition process described with respect to
(3) will be given preference in the activity determination. Motion
data may also be used to determine whether the mobile device is in
"heavy" motion or "light" motion. In particular, if the mobile
device is determined to be in "heavy" motion, i.e., walking
briskly, running, and the like, preference will be given to
activities requiring more action. Conversely, "light" movement or
no movement may indicate less active current activities, such as
web browsing or movie watching. Accordingly, preference will be
given to less active activities. Motion data may also be used to
determine the direction of the mobile device such that, in
conjunction with (2) above, an appropriate landmark may be
identified. In one embodiment, the motion data may be obtained
using at least one of an accelerometer, a magnometer, a gyroscope,
or a combination thereof.
[0053] With regard to (1)-(5), the system and method of the
invention then predict the current activity using a set of likely
rules. In particular, the system may use a set of likely rules to
determine the most likely relevant activity in the mobile device
and/or surrounding environment by monitoring the increases in
"recent" time window in the (1) the cpu usage, (2) the memory
usage, (3) the decibel (volume) usage, (3) the images being viewed,
(4) the online social activity, (5) the online web page
interactions, (6) the online or web game interactions, and other
similar categories.
[0054] More specifically, in one embodiment of the invention, the
system and method of the invention will also be capable of
selecting a likely activity from the collected set. In particular,
machine learning will be employed to determine which potential
activity is the activity meant to be tagged via social media.
Examples of machine learning in accordance with the present
invention include, but are not limited to, the following:
[0055] (a) A training period. When the user initiates the affinity
button push, a list of possible primary activities is provided. The
user will then select the intended activity. Over time, this trains
the system to prioritize the selected activity based on prior use.
For example, if the user tends to select the song being played, the
system will prioritize music as the likely item of interest.
Alternatively, a more passive training model may be followed. In
this aspect, the system would suggest a single likely activity and
the user would respond whether this was correct or for the system
to try again. The system would then suggest an alternative (or an
open field for the user to suggest an alternative) and the process
would continue. Through machine learning, the system would create
appropriate weighting to possible activities of interest to hone
its suggestions.
[0056] (b) Background service. In another embodiment, the software
may run as a background service, regularly querying the device to
determine the current activity through all of, or a subset of, the
activities covered in the previous section. When the user then
expressly activates this software, it will first compare the
current state to the previous state(s) to determine what has
changed and then determine the desired activity of focus by
combining the state change information with (1) any previous
training, if it exists (2) compare to previous items the user has
taken action on to determine if it fits with the user profile and
(3) prioritize actions taken on the cell phone (browser, apps, new
music, video or other content being played) over environmental
processes like Bluetooth and GPS.
[0057] (c) Recent items. Instead of querying multiple sensors, in
one embodiment, the machine training involves a determination of
the most recently opened item on the mobile device and acts upon
it. If no suitable application has been opened or changed, it will
then look to the other sensors and activities.
[0058] (d) Prioritization. Machine training may involve the
prioritization of activities. For example, in one embodiment,
activities taken on the mobile device over any environmental
activities (GPS, Bluetooth, external audio) have first priority,
camera action has second priority, GPS and audio recordings have
third priority, and Bluetooth and other sensors have fourth
priority. As would be understood by one of ordinary skill in the
art, these priorities may be appropriately altered based on any
additional information on the activity of the user to create a
prioritization system for "guesstimating" the activity of a user.
The "guesstimation" may then be associated with extracted
biological/emotional/facial responses based on all and sundry
information available on the usage of the background/foreground
apps, in-view objects and current events pertaining to the user (at
his/her location).
[0059] The above-described forms of machine learning may be used
independently or together.
[0060] Once an activity is determined, the system will then take
the desired social media action(s). Suitable social media actions
include, but are not limited to:
[0061] (a) Execution of a function defined in the system that may
or may not be editable by the user. For example, the function may
be a Facebook.RTM. "like", a Twitter post, a Facebook.RTM. update,
a Pinterest.RTM. addition, a Foursquare check-in, a Google+, or any
other standard action within social media interfaces (whether it be
directly actions in current platforms, future platforms, or a
separate platform specifically created for this functionality);
[0062] (b) Execution of a custom action such as a "dislike", a
"yum" for food related items, or other actions directly related to
the current activity; and/or
[0063] (c) Execution of more standard functions like a bookmark, a
favorite, a pin, or a web search.
[0064] In one embodiment, (c) may include a web search based on any
or all of the current possible activities. For example, a user is
at a sports bar in Location A watching a Soda 1 commercial
featuring Singer X with a couple of friends on his/her mobile
device. During the commercial, the user holds up the phone to a
Picture Z on the wall and presses the affinity button. The system
will go through its normal process of determining the liked action,
but will also (i) perform a web search on each item returning a
search results page split into sections showing information on
Location A, Picture Z, the sports bar, their friends, Singer Z and
Soda 1 and (ii) flag all of these items in a social media list
stored on the mobile device for future search and reference. In
other words, (c) may be a form of social media in a natural
environment bookmark.
[0065] Once these actions are taken, the information can be stored
in a form of customer relationship management (CRM) database
enabling advertisers to target messages to these users based on
items they have flagged using this application. Over time,
advertising within the app or across the operating system will
become increasingly more directed at the user.
[0066] Biometrics may also be integrated into or associated with
the device in order to ascertain the emotional state of the phone
user. Biometrics include, but are not limited to galvanic skin
response (GSR) to measure emotional arousal, heart response
(approach/avoid), motion to determine activity level), skin
temperature, breathing, EEG, EMG, pupil dilation, eye tracking, and
facial recognition. The sensor(s) used to collect the biometrics
may be integrated into the hardware platform through additional
sensors.
[0067] In one embodiment, the biometric data collection is
performed through a sensor-array case on the back, sides, and/or
front of the existing mobile device. For example, as shown in FIGS.
1 and 2, the case 10 has sides 12a and 12b, a front 30, a back 20,
and a top and bottom (not shown). The case 10 does not cover the
display of the mobile device, as indicated by void 24. The
biometrics discussed above may be captured via a plurality of
sensors 14 and/or 32. A subset of the biometrics, i.e., heart
response, breathing, pupil dilation, eye tracking, and facial
recognition, may be captured via sensors 14, built-in camera 22,
and/or sensors 32. While FIG. 1 shows sensors 14 only on one side,
i.e., side 12a, side 12b may also be equipped with one or more
sensors.
[0068] FIG. 3A shows another embodiment of a biometric data
collector for use with a mobile device. In particular, the case 40
has an opening for the mobile device 42 and the built-in camera 48.
At least one sensor for biometric data collection is incorporated
into at least one of the finger holes 46a-d. While FIG. 3A shows
four finger holes 46a-d, one of ordinary skill in the art would
appreciate that more or less finger holes may be used in the case
design so as to facilitate biometric data collection. For example,
as shown the case 40 may incorporate five finger holes 46a-e such
that the thumb hole 46e is on an opposite side of case 40 than
finger holes 46a-d. In addition, as with the case shown and
described in FIGS. 1-2, a subset of the biometrics, i.e., heart
response, breathing, pupil dilation, eye tracking, and facial
recognition, may be captured via the sensors in finger holes 46,
the built-in camera 42, and/or sensors built into the back of case
40 (not shown). Similarly, one of ordinary skill in the art would
appreciate that, while FIGS. 3A-B illustrate a case 40 intended for
right-handed users, the case could be modified to accommodate
left-handed users by changing the side(s) on which the finger holes
46a-d (and optionally 46e) reside.
[0069] In another embodiment, the biometric data collector is
incorporated into a wrist or arm band, such as those shown in FIGS.
4A-B and FIG. 5. In particular, a wrist band 50 may utilize a hook
and loop fastener system 58, 56 to attach the mobile device 52 or
its case 54. At least one sensor 60 may be incorporated into the
wrist band 50. In one embodiment, a plurality of sensors may be
incorporated into the wrist band. The wrist band 50 may also
include an emitter that transmits the data to a wireless or wired
receiver in the mobile device 52. Similarly, arm band 70 with
straps 70a and 70b include at least one sensor 72 for biometric
data collection (four sensors 72a-d are shown here as a
non-limiting example of the plurality of sensors that may be used
in accordance with the invention). The front 76a and back 76b of
case 76 may house the mobile device. The case 76 may have an
opening 78a for the mobile device display and an opening 78b for a
camera built in to the mobile device. As with the previous
embodiments, the arm band may include an emitter to transmit the
biometric data to a wired or wireless receiver in the mobile
device.
[0070] In yet another embodiment, the biometric data collector is
incorporated into headgear, such as the hat illustrated in FIG. 6.
Headgear of any style may be incorporated into this embodiment of
the present invention. In this instance, a winter hat is used with
a pocket portion 82 attached such that in use at least one sensor
(shown here as 86) in the hat 80 will properly be aligned with the
user's temple. The sensor(s) is in direct communication with the
mobile device 84 via a wired connection 88. However, as one of
ordinary skill in the art would appreciate, the wired connection
may be substituted for a wireless connection such that the
biometric data collected via the one or more sensors 86 will be
received by the mobile device 84.
[0071] In still another embodiment, the biometric data collector
may be incorporated into a smart watch, as generally shown in FIG.
7. For example, a smart watch 90 may include a base 92, a wristband
94 coupled to the base 92, a flip up portion 96, a hinge 98 coupled
to the base 92 and the flip up portion 96, and a camera 100. The
base 92 may include at least one housing 102 that includes a
processor, a wireless transceiver, and at least one sensor (not
shown) coupled to the housing 102. The sensors and wireless
transceiver may be coupled to the housing and in communication with
the processor. In one embodiment, a tactile user interface in
communication with the processor and coupled to the housing is
accessible via flip up portion 96 but viewable through display 104.
The wireless transceiver can provide a connection to a wireless
network. As with the previous embodiments, the at least one sensor
in the housing 102 is capable of collecting biometric data.
[0072] In another embodiment, the biometric data collector may be
incorporated into a steering wheel of a vehicle. For example, as
shown in FIG. 8, a mobile device 112 may be housed in holder 110.
Holder 110 may be connected to sensors 116 via a wired or wireless
connection such that biometric data may be collected.
[0073] Furthermore, while additional sensors may be added directly
through integrated hardware, a subset of the sensors may require an
external monitoring device (due to sensor needs on other parts of
the body). In this aspect of the invention, suitable sensors for
collecting EEG and EMG include, but are not limited to, an armband,
a pair of glasses, a watch, and similar sensors. In one embodiment,
all of the remaining sensors are built into a secondary device.
Some, such as a breathing sensor, may work more effectively in a
secondary device that communicates with the device of the
invention.
[0074] In an alternative embodiment, the system of the invention
incorporates a device that is not a handheld device. In other
words, instead of incorporating the sensors into a mobile phone,
the device used is a pair of glasses with attached sensors to the
scalp, the forehead, or the like, which are designed with all of
the required sensors and equipment as most handheld technology
devices. In one aspect, the glasses include a visible screen and
sensors (e.g., sensors to the central-parietal regions of the head
for EEG, to the forehead, chin or neck for the skin-conductance and
to the neck for heart-rate, or other optimal locations on the
face/neck of a person for these biological signals), that may have
otherwise required a secondary device. The collection of biometric
data distinguishes the glasses described in this embodiment from
conventional eye tracking goggles.
[0075] The biometric data thus collected is intended to determine
the emotional state of the user. This emotional state may then be
used to activate the social media function with limited button
presses. Thus, in one aspect of the invention, the affinity button
or application may be used in conjunction with the biometric data.
In this aspect, the mobile device (via sensors incorporated
directly into the mobile device, into the biometric data collector
holding the mobile device, and/or other external secondary devices)
is capable of gathering biometric data to determine current
physical and emotional state. Affinity buttons may be selected to
perform the following steps:
[0076] 1) ascertain the current activity the user is engaged in as
described in the previous section;
[0077] 2) determine the degree of acceptance of the current
activity (i.e., whether the current activity is liked or disliked);
and/or
[0078] 3) execute the actions described in the previous
section.
[0079] The affinity buttons exist on the mobile device solely for
training the system on the user's biometric profile. In particular,
as a particular affinity button is pressed, a biometric snapshot is
taken of the user to determine the emotional state. The snapshot
may then be stored as a biometric fingerprint associated with both
the self-described state (positive or negative) and the current
activity. As the user builds this dataset, the system begins to
suggest content (e.g., music, video, books), products, locations,
and the like for future biometric states (without actual button
presses. Indeed, at a predetermined stage, the button presses may
be completely removed. For example, if every time a user presses
the positive affinity button when their heart rate and GSR spike
above a threshold while listening to Elvis music, when the user has
similar future spikes, the system may recommend listening to
similar music.
[0080] In addition, the affinity buttons assist in training the
system/device. For example, the user presses the particular
affinity buttons to train the system that biometric patterns are
associated with liking and disliking. In one embodiment of the
invention, the system will be able to continuously monitor the
user's biometrics. When the system senses a biometric
profile/signature pattern matching "like" or "dislike", the system
automatically takes the associated social media action, which would
then execute the series of steps listed in the previous function on
a continuous basis without the user ever having to press a button.
The buttons or applications may be removed or disabled after the
system is adequately trained.
[0081] In an alternate embodiment of the system, the training
algorithm occurs completely outside the biometric device (using a
simulated environment), or through predictions of valence and
arousal, such as through the methods described in U.S. Pat. No.
8,296,172 and U.S. Patent Publication Nos. 2010/0211439 and
2010/0004977, the entire disclosures of which are incorporated
herein by reference.
[0082] Alternatively, the system may be a device that has no
affinity buttons and only relies on the biometric sensors to
accomplish the end goal. In this aspect, no additional training is
necessary (existing techniques as described in U.S. Pat. No.
8,296,172 and U.S. Patent Publication Nos. 2010/0211439 and
2010/0004977 for predicting valence, arousal, and engagement, and
resonance may be used). In this embodiment, the system continuously
determines emotional/biometric state such that whenever a specific
emotional state of interest, such as an approach or avoid response,
is identified, the series of activities previously described are
followed, i.e., determining current activity and taking desired
social media actions.
[0083] Other embodiments are within the scope and spirit of the
invention. For example, functions described above can be
implemented and/or automated using software, hardware, firmware,
hardwiring, or combinations of any of these. Features implementing
functions may also be physically located at various positions,
including being distributed such that portions of functions are
implemented at different physical locations.
[0084] Further, while the description above refers to the
invention, the description may include more than one invention.
* * * * *