U.S. patent application number 15/482544 was filed with the patent office on 2017-10-12 for methods and systems for obtaining, analyzing, and generating vision performance data and modifying media based on the vision performance data.
The applicant listed for this patent is Vizzario, Inc.. Invention is credited to Syed Khizer Rahim Khaderi, Kyle Christopher McDermott, Mohan Komalla Reddy.
Application Number | 20170293356 15/482544 |
Document ID | / |
Family ID | 59999063 |
Filed Date | 2017-10-12 |
United States Patent
Application |
20170293356 |
Kind Code |
A1 |
Khaderi; Syed Khizer Rahim ;
et al. |
October 12, 2017 |
Methods and Systems for Obtaining, Analyzing, and Generating Vision
Performance Data and Modifying Media Based on the Vision
Performance Data
Abstract
The present specification describes methods and systems for
modifying a media, such as Virtual Reality, Augmented Reality, or
Mixed Reality (VR/AR/MxR) media based on a vision profile and a
target application. In embodiments of the specification, a Sensory
Data Exchange (SDE) is created that enables identification of
various vision profiles for users and user groups. The SDE may be
utilized to modify one or more media in accordance with each type
of user and/or user group.
Inventors: |
Khaderi; Syed Khizer Rahim;
(Venice, CA) ; Reddy; Mohan Komalla; (Fremont,
CA) ; McDermott; Kyle Christopher; (Los Angeles,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Vizzario, Inc. |
Venice |
CA |
US |
|
|
Family ID: |
59999063 |
Appl. No.: |
15/482544 |
Filed: |
April 7, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62425736 |
Nov 23, 2016 |
|
|
|
62381784 |
Aug 31, 2016 |
|
|
|
62363074 |
Jul 15, 2016 |
|
|
|
62359796 |
Jul 8, 2016 |
|
|
|
62322741 |
Apr 14, 2016 |
|
|
|
62319825 |
Apr 8, 2016 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/147 20130101;
A61B 5/04842 20130101; A61B 3/024 20130101; G09G 2320/0693
20130101; A61B 5/165 20130101; A63F 13/25 20140902; G02B 27/0172
20130101; G06F 3/0346 20130101; A61B 3/028 20130101; G09G 5/006
20130101; A63F 13/212 20140902; G02B 2027/0138 20130101; G09G
2354/00 20130101; A61B 3/113 20130101; G16H 40/60 20180101; A61B
3/10 20130101; G06F 3/013 20130101; A61B 5/11 20130101; A61B 5/0476
20130101; G02B 27/0093 20130101; G16H 50/20 20180101; A61B 5/04845
20130101; A61B 3/0091 20130101; A61B 5/024 20130101 |
International
Class: |
G06F 3/01 20060101
G06F003/01; G02B 27/00 20060101 G02B027/00; A61B 3/113 20060101
A61B003/113; G09G 5/00 20060101 G09G005/00; A63F 13/212 20060101
A63F013/212; A63F 13/25 20060101 A63F013/25; G06F 3/0346 20060101
G06F003/0346; G02B 27/01 20060101 G02B027/01; A61B 5/0476 20060101
A61B005/0476 |
Claims
1. A method of improving or treating a condition experienced by a
user, while said user is experiencing media using a computing
device with a display comprising acquiring a first value for at
least one of a plurality of data using said computing device;
acquiring a second value for the at least one of the plurality of
data using said computing device; using said first value and second
value, determining a change in at least one of the plurality of
data over time; based upon said change in the at least one of the
plurality of data over time, determining a degree of said
condition; and based upon determining a degree of said condition,
modifying said media.
2. The method of claim 1 wherein the computing device is a virtual
reality, augmented reality, or mixed reality view device.
3. The method of claim 2 wherein the virtual reality, augmented
reality, or mixed reality view device comprises at least one of a
camera configured to acquire eye movement data, a sensor configured
to detect a rate and/or direction of head movement, a sensor
configured to detect a heart rate, and an EEG sensor to detect
brain waves.
4. The method of claim 3 wherein the eye movement data comprises
rapid scanning, saccadic movement, blink rate data, fixation data,
pupillary diameter, and palpebral fissure distance.
5. The method of claim 2 wherein the condition is at least one of
comprehension, fatigue, engagement, performance, symptoms
associated with visually-induced motion sickness secondary to
visual-vestibular mismatch, symptoms associated with post-traumatic
stress disorder, double vision related to accommodative
dysfunction, vection due to unintended peripheral field
stimulation, vergence-accommodation disorders, fixation disparity,
blurred vision and myopia, headaches, difficulties in focusing,
disorientation, postural instability, visual discomfort, eyestrain,
dry eye, eye tearing, foreign body sensation, feeling of pressure
in the eyes, aching around the eyes, nausea, stomach discomfort,
potential phototoxicity from overexposure to screen displays,
hormonal dysregulation arising from excessive blue light exposure,
heterophoria, decrease in positive emotions, and increase in
negative emotions.
6. The method of claim 2 wherein the plurality of data comprises at
least one of rapid scanning, saccadic movement, fixation, blink
rate, pupillary diameter, speed of head movement, direction of head
movement, heart rate, motor reaction time, smooth pursuit,
palpebral fissure distance, degree and rate of brain wave activity,
degree of convergence, and degree of convergence.
7. The method of claim 2 wherein the modifying of media comprises
at least one of increasing a contrast of the media, decreasing a
contrast of the media, making an object of interest that is
displayed in the media larger in size, making an object of interest
that is displayed in the media smaller in size, increasing a
brightness of the media, decreasing a brightness of the media,
increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view, decreasing an amount of
an object of interest displayed in the media shown in a central
field of view and increasing said object of interest in a
peripheral field of view, changing a focal point of content
displayed in the media to a more central location, removing objects
from a field of view and measuring if a user recognizes said
removal, increasing an amount of color in said media, increasing a
degree of shade in objects shown in said media changing RGB values
of said media based upon external data, demographic or trending
data.
8. The method of claim 2 wherein the condition is
comprehension.
9. The method of claim 2 wherein the change is at least one of
increased rapid scanning, increased saccadic movement, decreased
fixation, increased blink rate, increased pupillary diameter,
increased head movement, increased heart rate, decreased reaction
time, decreased separation of the eyelids, changes in brain wave
activity, and increased smooth pursuit.
10. The method of claim 9 wherein the degree of the condition is a
decreased comprehension of the user.
11. The method of claim 10 wherein, based on said decreased
comprehension of the user, said media is modified by at least one
of increasing a contrast of the media, making an object of interest
that is displayed in the media larger in size, increasing a
brightness of the media, increasing an amount of an object of
interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view, changing a focal point of content displayed in the media to a
more central location, removing objects from a field of view and
measuring if a user recognizes said removal, increasing an amount
of color in said media, increasing a degree of shade in objects
shown in said media, and changing RGB values of said media based
upon external data, demographic or trending data.
12. The method of claim 2 wherein the condition is fatigue.
13. The method of claim 12 wherein the change is at least one of
decreased fixation, increased blink rate, and changes in
convergence and divergence.
14. The method of claim 13 wherein the degree of the condition is
an increased fatigue of the user.
15. The method of claim 14 wherein, based on said increased fatigue
of the user, said media is modified by at least one of increasing a
contrast of the media, making an object of interest that is
displayed in the media larger in size, increasing a brightness of
the media, and increasing or introduction more motion.
16. A method of improving comprehension experienced by a user,
while the user is experiencing media through a virtual reality,
augmented reality, or mixed reality view device, the method
comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of reduced comprehension of the
user; and modifying media based upon determining a degree of
reduced comprehension.
17. The method of claim 16 wherein acquiring the first value and
the second value of the plurality of data comprises acquiring at
least one or more of: a sensor configured to detect basal body
temperature, heart rate, body movement, body rotation, body
direction, body velocity, or body amplitude; a sensor configured to
measure limb movement, limb rotation, limb direction, limb
velocity, or limb amplitude; a pulse oximeter; a sensor configured
to measure auditory processing; a sensor configured to measure
gustatory and olfactory processing; a sensor to measure pressure;
an input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; an
electroencephalograph; an electrocardiograph; an electromyograph;
an electrooculograph; an electroretinography; and a sensor
configured to measure galvanic skin response.
18. The method of claim 16 wherein the plurality of data comprises
at least one or more of: palpebral fissure, blink rate, pupil size,
pupil position, gaze direction, gaze position, vergence, fixation
position, fixation duration, fixation rate; fixation count; saccade
position, saccade angle, saccade magnitude, pro-saccade,
anti-saccade, inhibition of return, saccade velocity, saccade rate,
screen distance, head direction, head fixation, limb tracking,
weight distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
19. The method of claim 16 wherein modifying the media comprises
modifying by at least one of: increasing a contrast of the media,
making an object of interest that is displayed in the media larger
in size, increasing a brightness of the media, increasing an amount
of an object of interest displayed in the media shown in a central
field of view and decreasing said object of interest in a
peripheral field of view, changing a focal point of content
displayed in the media to a more central location, removing objects
from a field of view and measuring if a user recognizes said
removal, increasing an amount of color in said media, increasing a
degree of shade in objects shown in said media, and changing RGB
values of said media based upon external data (demographic or
trending data).
20. The method of claim 16 wherein the modifying the media
comprises modifying to provide a predefined increase in
comprehension.
Description
CROSS-REFERENCE
[0001] The present application relies on, for priority, the
following United States Provisional Patent Applications, which are
also herein incorporated by reference in their entirety:
[0002] U.S. Provisional Patent Application No. 62/425,736, entitled
"Methods and Systems for Gathering Visual Performance Data and
Modifying Media Based on the Visual Performance Data" and filed on
Nov. 23, 2016;
[0003] U.S. Provisional Patent Application No. 62/381,784, of the
same title and filed on Aug. 31, 2016;
[0004] U.S. Provisional Patent Application No. 62/363,074, entitled
"Systems and Methods for Creating Virtual Content Representations
Via A Sensory Data Exchange Platform" and filed on Jul. 15,
2016;
[0005] U.S. Provisional Patent Application No. 62/359,796, entitled
"Virtual Content Representations" and filed on Jul. 8, 2016;
[0006] U.S. Provisional Patent Application No. 62/322,741, of the
same title and filed on Apr. 14, 2016; and U.S. Provisional Patent
Application No. 62/319,825, of the same title and filed on Apr. 8,
2016.
FIELD
[0007] The present specification relates generally to virtual
environments, and more specifically to methods and systems for
modifying media, such as virtual reality-based, augmented
reality-based, or mixed reality-based (VR/AR/MxR) media, based on
an individual's vision profile and/or a target application.
BACKGROUND
[0008] In recent years, Virtual Reality (VR) environments,
Augmented Reality (AR), and Mixed Reality (MxR) applications have
become more common. While VR is a non-invasive simulation
technology that provides an immersive, realistic, three-dimensional
(3D) computer-simulated environment in which people perform tasks
and experience activities as if they were in the real world; AR
depicts a real world environment that is augmented or supplemented
by computer generated media. The most direct experience of
VR/AR/MxR is provided by fully immersive VR/AR/MxR systems, and the
most widely adopted VR/AR/MxR systems display their simulated
environment through special wearable Head-Mounted visual Displays
(HMDs). HMDs typically consist of screens and lenses fitted into
glasses, helmets or goggles, with a display that may be monocular
(display seen by one eye only), binocular (both eyes view a single
screen), or dichoptic (each eye views a different screen or image
that can be stereoscopic, which gives additional depth cues).
[0009] Although HMDs have recently been introduced to the general
public, they are not a new phenomenon. As early as the 1960s,
computer graphics pioneer Ivan Sutherland developed the first HMD,
which made it possible to overlay virtual images on the real world.
HMD technology gradually evolved through the 1970s with use across
military, industry, scientific research and entertainment domains.
The early commercially available HMDs, such as the Virtual Research
Flight Helmet.TM., Virtual I/O I-Glasses.TM., and Dynovisor.TM.,
had limited applications due to their narrow field-of-view (FOV)
and inherent cumbersomeness in weight, physical restrictions, and
system parameters. Recent advancements have been directed toward
making HMDs more comfortable for longer duration of use. Recent HMD
products including Google Glass.TM., Epson Moverio.TM., Vuzix
Wrap.TM., and Oculus Rift.TM. have become commercially available
and increasingly commonplace as a result of technical advancements.
For example, one version of the Oculus Rift.TM., the Development
Kit 2 (DK2), has a high resolution, high refresh rate (i.e., the
frequency with which a display's image is updated), low persistence
(which aids in removing motion blur), and advanced positional
tracking for lower latency and precise movement, when compared to
its predecessors. HMD technology advancement and cost reduction has
increased its potential for widespread use.
[0010] Unfortunately, a number of vision-related conditions are
associated with the use of such technology. For example, visually
induced motion sickness (VIMS) or simulation sickness, which is
related to visual-vestibular mismatch, has been attributed to
significant systemic and perceptual problems inherently associated
with the use of HMDs and remains an obstacle to the widespread
adoption and commercial development of technologies associated with
VR/AR/MxR-based HMDs. The systemic and perceptual problems with
HMDs, not typically associated with traditional displays, include
nausea, stomach discomfort, disorientation, postural instability
and visual discomfort.
[0011] It is commonly accepted that the symptoms of nausea and
instability result from various sensory input conflicts, including
conflicting position and movement cues, leading to a disharmonious
effect on the visual and vestibular systems. In addition, specific
types of HMDs and also other apparatuses that provide virtual
environments, may have mismatch problems with the user's visual
system due to improper optical design, resulting in
convergence-accommodation conflict and visual discomfort or
fatigue. Other studies have reported high incidence of visual
discomfort including eyestrain, dry eye, tearing, foreign body
sensation, feeling of pressure in the eyes, aching around the eyes,
headache, blurred vision, and difficulty in focusing. Other visual
problems such as myopia, heterophoria, fixation disparity,
vergence-accommodation disorders, and abnormal tear break-up time
(TBUT) also have been reported. Using HMDs may cause accommodative
spasm that in turn may lead to a transient myopia. Continued
conflict between convergence-accommodation, the user's
inter-pupillary distance (IPD), and/or the systems' inter-optical
distance (IOD) may lead to heterophoria and fixation disparity
changes. Moreover, visual symptoms are not necessarily limited to
the time of actual virtual environment (VE) immersion; rather,
visual changes including visual fatigue, reduced visual acuity and
heterophoria may continue after terminating exposure to HMD-based
VE. Users are often required to avoid driving or operating heavy
machinery after exposure to VR/AR/MxR until VIMS and postural
instability resolve. Complex visual tasks and reading during and
after exposure to VR/AR/MxR may increase severity of VIMS.
[0012] Advances in HMD technology have provided the potential for
its widespread use in VR/AR/MxR. However, VIMS still remains an
obstacle to public adoption and commercial development of this
technology. Visual discomfort induced by VR/AR/MxR in VE may be
reduced by optimizing quality and design of VR/AR/MxR apparatuses
such as HMDs. However, there is still a need for methods and
systems that can resolve visual-vestibular mismatch and adapt
VR/AR/MxR to the visual capacity of a user and/or a group of users
in order to minimize and/or eliminate VIMS. Current visual measures
and rating systems for VR/AR/MxR are qualitative in nature. There
is also a need to establish quantitative measures to improve the
quality of the user experience in VR/AR/MxR environments.
[0013] What is also needed is a system that is capable of grouping
individuals based on demographic or other common factors to
identify an acceptable modifications of visual media, thus reducing
the levels of discomfort. A system is also needed that may adapt to
an identified user or group in order to modify and present
VR/AR/MxR media that reduces discomfort. A system is also needed
that may identify delay and other forms of data including biometric
data, and their patterns, to recommend and/or automate or
dynamically change a VR/AR/MxR environment based on the data and
the patterns.
SUMMARY
[0014] In some embodiments, the present specification is directed
toward a method of improving or treating a condition experienced by
a user, while said user is experiencing media using a computing
device with a display, such as, but not limited to, a conventional
laptop, mobile phone, tablet computer, desktop top computer, gaming
system, virtual reality, augmented reality, and mixed reality view
device. The method comprises acquiring a first value for at least
one of the plurality of data using said computing device; acquiring
a second value for the at least one of the plurality of data using
said computing device; using said first value and second value,
determining a change in at least one of the plurality of data over
time; based upon said change in the at least one of the plurality
of data over time, determining a degree of said condition; and
based upon determining a degree of said condition, modifying said
media.
[0015] Optionally, the computing device is a virtual reality,
augmented reality, or mixed reality view device.
[0016] Optionally, the virtual reality, augmented reality, or mixed
reality view device comprises at least one of a camera configured
to acquire eye movement data, a sensor configured to detect a rate
and/or direction of head movement, a sensor configured to detect a
heart rate, and an EEG sensor to detect brain waves.
[0017] Optionally, the eye movement data comprises rapid scanning,
saccadic movement, blink rate data, fixation data, pupillary
diameter, and palpebral fissure distance.
[0018] Optionally, the condition is at least one of comprehension,
fatigue, engagement, performance, symptoms associated with
visually-induced motion sickness secondary to visual-vestibular
mismatch, symptoms associated with post-traumatic stress disorder,
double vision related to accommodative dysfunction, vection due to
unintended peripheral field stimulation, vergence-accommodation
disorders, fixation disparity, blurred vision and myopia,
headaches, difficulties in focusing, disorientation, postural
instability, visual discomfort, eyestrain, dry eye, eye tearing,
foreign body sensation, feeling of pressure in the eyes, aching
around the eyes, nausea, stomach discomfort, potential
phototoxicity from overexposure to screen displays, hormonal
dysregulation arising from excessive blue light exposure,
heterophoria, decrease in positive emotions, and increase in
negative emotions.
[0019] Optionally, the plurality of data comprises at least one of
rapid scanning, saccadic movement, fixation, blink rate, pupillary
diameter, speed of head movement, direction of head movement, heart
rate, motor reaction time, smooth pursuit, palpebral fissure
distance, degree and rate of brain wave activity, degree of
convergence, and degree of convergence.
[0020] Optionally, the modifying of media comprises at least one of
increasing a contrast of the media, decreasing a contrast of the
media, making an object of interest that is displayed in the media
larger in size, making an object of interest that is displayed in
the media smaller in size, increasing a brightness of the media,
decreasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, decreasing an amount of an object of interest
displayed in the media shown in a central field of view and
increasing said object of interest in a peripheral field of view,
changing a focal point of content displayed in the media to a more
central location, removing objects from a field of view and
measuring if a user recognizes said removal, increasing an amount
of color in said media, increasing a degree of shade in objects
shown in said media changing RGB values of said media based upon
external data, demographic or trending data.
[0021] Optionally, the condition is comprehension.
[0022] Optionally, the change is at least one of increased rapid
scanning, increased saccadic movement, decreased fixation,
increased blink rate, increased pupillary diameter, increased head
movement, increased heart rate, decreased reaction time, decreased
separation of the eyelids, changes in brain wave activity, and
increased smooth pursuit.
[0023] Optionally, the degree of the condition is a decreased
comprehension of the user.
[0024] Optionally, based on said decreased comprehension of the
user, said media is modified by at least one of increasing a
contrast of the media, making an object of interest that is
displayed in the media larger in size, increasing a brightness of
the media, increasing an amount of an object of interest displayed
in the media shown in a central field of view and decreasing said
object of interest in a peripheral field of view, changing a focal
point of content displayed in the media to a more central location,
removing objects from a field of view and measuring if a user
recognizes said removal, increasing an amount of color in said
media, increasing a degree of shade in objects shown in said media,
and changing RGB values of said media based upon external data,
demographic or trending data.
[0025] Optionally, the condition is fatigue.
[0026] Optionally, the change is at least one of decreased
fixation, increased blink rate, and changes in convergence and
divergence.
[0027] Optionally, the degree of the condition is an increased
fatigue of the user.
[0028] Optionally, based on said increased fatigue of the user,
said media is modified by at least one of increasing a contrast of
the media, making an object of interest that is displayed in the
media larger in size, increasing a brightness of the media, and
increasing or introduction more motion.
[0029] In some embodiments, the present specification is directed
toward a method of improving comprehension experienced by a user,
while the user is experiencing media through a virtual reality,
augmented reality, or mixed reality view device, the method
comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of reduced comprehension of the
user; and modifying media based upon determining a degree of
reduced comprehension.
[0030] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0031] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0032] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0033] Optionally, modifying the media comprises modifying to
provide a predefined increase in comprehension.
[0034] In some embodiments, the present specification is directed
toward a method of decreasing fatigue experienced by a user, while
the user is experiencing media through a virtual reality, augmented
reality, or mixed reality view device, the method comprising:
acquiring a first value for a plurality of data; acquiring a second
value for the plurality of data; using the first value and the
second value to determine a change in the plurality of data over
time; based upon the change in the plurality of data over time,
determining a degree of increased fatigue of the user; and
modifying media based upon determining a degree of increased
fatigue.
[0035] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring using at least one or
more of: using one or more of: a sensor configured to detect basal
body temperature, heart rate, body movement, body rotation, body
direction, body velocity, or body amplitude; a sensor configured to
measure limb movement, limb rotation, limb direction, limb
velocity, or limb amplitude; a pulse oximeter; a sensor configured
to measure auditory processing; a sensor configured to measure
gustatory and olfactory processing; a sensor to measure pressure;
an input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; an
electroencephalograph; an electrocardiograph; an electromyograph;
an electrooculograph; an electroretinography; and a sensor
configured to measure galvanic skin response.
[0036] Optionally, the plurality data of comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0037] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0038] Optionally, the modifying the media comprises modifying to
provide a predefined decrease in fatigue.
[0039] In some embodiments, the present specification is directed
toward a method of increasing engagement of a user, while the user
is experiencing media through a computing device with a display,
the method comprising: acquiring a first value for a plurality of
data; acquiring a second value for the plurality of data; using the
first value and the second value to determine a change in the
plurality of data over time; based upon the change in the plurality
of data over time, determining a degree of decreased engagement of
the user; and modifying media based upon determining a degree of
decreased engagement.
[0040] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0041] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0042] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0043] Optionally, the modifying the media comprises modifying to
provide a predefined increase in engagement.
[0044] In some embodiments, the present specification is directed
toward a method of improving performance of a user, while the user
is experiencing media through a computing device with a display,
including a virtual reality, augmented reality, or mixed reality
view device, the method comprising: acquiring a first value for a
plurality of data; acquiring a second value for the plurality of
data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of
improvement in performance of the user; and modifying media based
upon determining a degree of improved performance.
[0045] Optionally, the acquiring the first value and the second
value of the plurality of data comprises acquiring at least one or
more of: sensor configured to detect basal body temperature, heart
rate, body movement/rotation/direction/velocity/amplitude; sensor
configured to measure limb
movement/rotation/direction/velocity/amplitude; sensor configured
to measure pulse rate, and other parameters similar to a pulse
oximeter; sensor configured to measure auditory processing; sensor
configured to measure gustatory and olfactory processing; sensor to
measure pressure; input device such as a traditional keyboard and
mouse and or any other form of controller to collect manual user
feedback; electroencephalography; electrocardiography;
electromyography; electrooculography; electroretinography; and
sensor configured to measure Galvanic Skin Response.
[0046] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0047] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0048] Optionally, the modifying the media comprises modifying to
provide a predefined increase in performance.
[0049] In some embodiments, the present specification is directed
toward a method of decreasing symptoms associated with
Visually-Induced Motion Sickness (VIMS) secondary to
visual-vestibular mismatch, of a user, while the user is
experiencing media through a computing device with a display,
including a virtual reality, augmented reality, or mixed reality
view device, the method comprising: acquiring a first value for a
plurality of data; acquiring a second value for the plurality of
data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in VIMS symptoms of the user; and modifying media based upon
determining a degree of decrease in VIMS symptoms.
[0050] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0051] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0052] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0053] Optionally, the modifying the media comprises modifying to
provide a predefined decrease in VIMS symptoms.
[0054] In some embodiments, the present specification is directed
toward a method of decreasing symptoms associated with
Post-Traumatic Stress Disorder (PTSD), of a user, while the user is
experiencing media through a computing device with a display
including a virtual reality, augmented reality, or mixed reality
view device, the method comprising: acquiring a first value for a
plurality of data; acquiring a second value for the plurality of
data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in PTSD symptoms of the user; and modifying media based upon
determining a degree of decrease in PTSD symptoms.
[0055] Optionally, the method further includes combining at least
one of image processing methods, machine learning methods,
electronic stimulation, and chemical stimulation, with the change
in the plurality of data over time, wherein the combining is used
for purposes of neuro-programming.
[0056] Optionally, the method further comprises combining at least
one of image processing methods, machine learning methods,
electronic stimulation, and chemical stimulation, with the change
in the plurality of data over time, wherein the combining is used
to modify light stimuli while the user is experiencing media
through the virtual reality, augmented reality, or mixed reality
view device.
[0057] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0058] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0059] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0060] Optionally, modifying the media comprises modifying to
provide a predefined decrease in PTSD symptoms.
[0061] In some embodiments, the present specification is directed
toward a method of decreasing double vision related to
accommodative dysfunction of a user, while the user is experiencing
media through a computing device with a display, including a
virtual reality, augmented reality, or mixed reality view device,
the method comprising: acquiring a first value for a plurality of
data; acquiring a second value for the plurality of data; using the
first value and the second value to determine a change in the
plurality of data over time; based upon the change in the plurality
of data over time, determining a degree of decrease in double
vision of the user; and modifying media based upon determining a
degree of decrease in double vision.
[0062] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0063] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0064] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0065] Optionally, modifying the media comprises modifying to
provide a predefined decrease in double vision.
[0066] In some embodiments, the present specification is directed
toward a method of decreasing vection due to unintended peripheral
field stimulation of a user, while the user is experiencing media
through a computing device with a display, including a virtual
reality, augmented reality, or mixed reality view device, the
method comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of decrease in vection of the user;
and modifying media based upon determining a degree of decrease in
vection.
[0067] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0068] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0069] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0070] Optionally, modifying the media comprises modifying to
provide a predefined decrease in vection.
[0071] In some embodiments, the present specification is directed
toward a method of decreasing hormonal dysregulation arising from
excessive blue light exposure of a user, while the user is
experiencing media through a computing device with a display,
including a virtual reality, augmented reality, or mixed reality
view device, the method comprising: acquiring a first value for a
plurality of data; acquiring a second value for the plurality of
data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in hormonal dysregulation; and modifying media based upon
determining a degree of decrease in hormonal dysregulation.
[0072] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0073] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0074] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0075] Optionally, modifying the media comprises modifying to
provide a predefined decrease in hormonal dysregulation.
[0076] In some embodiments, the present specification is directed
toward a method of decreasing phototoxicity from overexposure to
screen displays of a user, while the user is experiencing media
through a computing device with a display, including a virtual
reality, augmented reality, or mixed reality view device, the
method comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of decrease in phototoxicity; and
modifying media based upon determining a degree of decrease in
phototoxicity.
[0077] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0078] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0079] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0080] Optionally, modifying the media comprises modifying to
provide a predefined decrease in phototoxicity.
[0081] In some embodiments, the present specification is directed
toward a method of decreasing nausea and stomach discomfort of a
user, while the user is experiencing media through a computing
device with a display, including a virtual reality, augmented
reality, or mixed reality view device, the method comprising:
acquiring a first value for a plurality of data; acquiring a second
value for the plurality of data; using the first value and the
second value to determine a change in the plurality of data over
time; based upon the change in the plurality of data over time,
determining a degree of decrease in nausea and stomach discomfort;
and modifying media based upon determining a degree of decrease in
nausea and stomach discomfort.
[0082] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0083] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0084] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0085] Optionally, modifying the media comprises modifying to
provide a predefined decrease in nausea and stomach discomfort.
[0086] In some embodiments, the present specification is directed
toward a method of decreasing visual discomfort of a user,
including at least one of eyestrain, dry eye, eye tearing, foreign
body sensation, feeling of pressure in the eyes, or aching around
the eyes, while the user is experiencing media through a computing
device with a display, including a virtual reality, augmented
reality, or mixed reality view device, the method comprising:
acquiring a first value for a plurality of data; acquiring a second
value for the plurality of data; using the first value and the
second value to determine a change in the plurality of data over
time; based upon the change in the plurality of data over time,
determining a degree of decrease in visual discomfort; and
modifying media based upon determining a degree of decrease in
visual discomfort.
[0087] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0088] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0089] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0090] Optionally, modifying the media comprises modifying to
provide a predefined decrease in visual discomfort.
[0091] In some embodiments, the present specification is directed
toward a method of decreasing disorientation and postural
instability of a user, while the user is experiencing media through
a computing device with a display, including a virtual reality,
augmented reality, or mixed reality view device, the method
comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of decrease in disorientation and
postural instability; and modifying media based upon determining a
degree of decrease in disorientation and postural instability.
[0092] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: using one or more of: a sensor configured to detect basal body
temperature, heart rate, body movement, body rotation, body
direction, body velocity, or body amplitude; a sensor configured to
measure limb movement, limb rotation, limb direction, limb
velocity, or limb amplitude; a pulse oximeter; a sensor configured
to measure auditory processing; a sensor configured to measure
gustatory and olfactory processing; a sensor to measure pressure;
an input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; an
electroencephalograph; an electrocardiograph; an electromyograph;
an electrooculograph; an electroretinography; and a sensor
configured to measure galvanic skin response.
[0093] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0094] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0095] Optionally, modifying the media comprises modifying to
provide a predefined decrease in disorientation and postural
instability.
[0096] In some embodiments, the present specification is directed
toward a method of decreasing headaches and difficulties in
focusing of a user, while the user is experiencing media through a
computing device with a display, including a virtual reality,
augmented reality, or mixed reality view device, the method
comprising: acquiring a first value for a plurality of data;
acquiring a second value for the plurality of data; using the first
value and the second value to determine a change in the plurality
of data over time; based upon the change in the plurality of data
over time, determining a degree of decrease in headaches and
difficulties in focusing; and modifying media based upon
determining a degree of decrease in headaches and difficulties in
focusing.
[0097] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0098] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0099] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0100] Optionally, modifying the media comprises modifying to
provide a predefined decrease in headaches and difficulties in
focusing.
[0101] Optionally, the present specification is directed toward a
method of decreasing blurred vision and myopia of a user, while the
user is experiencing media through a computing device with a
display, including a virtual reality, augmented reality, or mixed
reality view device, the method comprising: acquiring a first value
for a plurality of data; acquiring a second value for the plurality
of data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in blurred vision and myopia; and modifying media based upon
determining a degree of decrease in blurred vision and myopia.
[0102] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0103] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0104] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0105] Optionally, modifying the media comprises modifying to
provide a predefined decrease in blurred vision and myopia.
[0106] In some embodiments, the present specification is directed
toward a method of decreasing heterophoria of a user, while the
user is experiencing media through a computing device with a
display, including a virtual reality, augmented reality, or mixed
reality view device, the method comprising: acquiring a first value
for a plurality of data; acquiring a second value for the plurality
of data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in heterophoria; and modifying media based upon determining a
degree of decrease in heterophoria.
[0107] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0108] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0109] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0110] Optionally, modifying the media comprises modifying to
provide a predefined decrease in heterophoria.
[0111] In some embodiments, the present specification is directed
toward a method of decreasing fixation disparity of a user, while
the user is experiencing media through a computing device with a
display, including a virtual reality, augmented reality, or mixed
reality view device, the method comprising: acquiring a first value
for a plurality of data; acquiring a second value for the plurality
of data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in fixation disparity; and modifying media based upon determining a
degree of decrease in fixation disparity.
[0112] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0113] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0114] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0115] Optionally, modifying the media comprises modifying to
provide a predefined decrease in fixation disparity.
[0116] In some embodiments, the present specification is directed
toward a method of decreasing vergence-accommodation disorders of a
user, while the user is experiencing media through a computing
device with a display, including a virtual reality, augmented
reality, or mixed reality view device, the method comprising:
acquiring a first value for a plurality of data; acquiring a second
value for the plurality of data; using the first value and the
second value to determine a change in the plurality of data over
time; based upon the change in the plurality of data over time,
determining a degree of decrease in vergence-accommodation
disorders; and modifying media based upon determining a degree of
decrease in vergence-accommodation disorders.
[0117] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0118] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0119] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, changing RGB values of said media
based upon external data (demographic or trending data), increasing
use of longer viewing distances when possible, matching simulated
distance with focal distance more closely, moving objects in and
out of depth at a slower pace, and making existing object conflicts
less salient.
[0120] Optionally, the modifying the media comprises modifying to
provide a predefined decrease in vergence-accommodation
disorders.
[0121] In some embodiments, the present specification is directed
toward a method of increasing positive emotion of a user, while the
user is experiencing media through a computing device with a
display, including a virtual reality, augmented reality, or mixed
reality view device, the method comprising: acquiring a first value
for a plurality of data; acquiring a second value for the plurality
of data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of increase
in positive emotion; and modifying media based upon determining a
degree of increase in positive emotion.
[0122] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0123] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0124] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0125] Optionally, modifying the media comprises modifying to
provide a predefined increase in positive emotion.
[0126] In some embodiments, the present specification is directed
toward a method of decreasing negative emotion of a user, while the
user is experiencing media through a computing device with a
display, including a virtual reality, augmented reality, or mixed
reality view device, the method comprising: acquiring a first value
for a plurality of data; acquiring a second value for the plurality
of data; using the first value and the second value to determine a
change in the plurality of data over time; based upon the change in
the plurality of data over time, determining a degree of decrease
in negative emotion; and modifying media based upon determining a
degree of decrease in negative emotion.
[0127] Optionally, acquiring the first value and the second value
of the plurality of data comprises acquiring at least one or more
of: a sensor configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, body velocity,
or body amplitude; a sensor configured to measure limb movement,
limb rotation, limb direction, limb velocity, or limb amplitude; a
pulse oximeter; a sensor configured to measure auditory processing;
a sensor configured to measure gustatory and olfactory processing;
a sensor to measure pressure; an input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; an electroencephalograph; an
electrocardiograph; an electromyograph; an electrooculograph; an
electroretinography; and a sensor configured to measure galvanic
skin response.
[0128] Optionally, the plurality of data comprises at least one or
more of: palpebral fissure, blink rate, pupil size, pupil position,
gaze direction, gaze position, vergence, fixation position,
fixation duration, fixation rate; fixation count; saccade position,
saccade angle, saccade magnitude, pro-saccade, anti-saccade,
inhibition of return, saccade velocity, saccade rate, screen
distance, head direction, head fixation, limb tracking, weight
distribution, frequency domain (Fourier) analysis,
electroencephalography output, frequency bands, electrocardiography
output, electromyography output, electrooculography output,
electroretinography output, galvanic skin response, body
temperature, respiration rate, oxygen saturation, heart rate, blood
pressure, vocalizations, inferred efferent responses, respiration,
facial expression, olfactory processing, gustatory processing, and
auditory processing.
[0129] Optionally, modifying the media comprises modifying by at
least one of: increasing a contrast of the media, making an object
of interest that is displayed in the media larger in size,
increasing a brightness of the media, increasing an amount of an
object of interest displayed in the media shown in a central field
of view and decreasing said object of interest in a peripheral
field of view, changing a focal point of content displayed in the
media to a more central location, removing objects from a field of
view and measuring if a user recognizes said removal, increasing an
amount of color in said media, increasing a degree of shade in
objects shown in said media, and changing RGB values of said media
based upon external data (demographic or trending data).
[0130] Optionally, modifying the media comprises modifying to
provide a predefined decrease in negative emotion.
[0131] In some embodiments, the present specification is directed
toward a method of performing a transaction with a user, while said
user is experiencing media using a computing device with a display,
including a virtual reality, augmented reality, or mixed reality
view device comprising obtaining at least one of psychometric,
sensory, and biometric information from the user, the at least one
of psychometric, sensory, and biometric information comprising one
or more values for at least one of the plurality of data using said
virtual reality, augmented reality, or mixed reality view device;
rewarding the user for the obtained at least one of psychometric,
sensory, and biometric information; using said one or more values
to determine a change in at least one of the plurality of data over
time; based upon said change in the at least one of the plurality
of data over time, modifying said media.
[0132] The aforementioned and other embodiments of the present
shall be described in greater depth in the drawings and detailed
description provided below.
BRIEF DESCRIPTION OF THE DRAWINGS
[0133] These and other features and advantages of the present
specification will be appreciated, as they become better understood
by reference to the following detailed description when considered
in connection with the accompanying drawings, wherein:
[0134] FIG. 1A shows a block diagram illustrating user interaction
with an exemplary Sensory Data Exchange Platform (SDEP), in
accordance with an embodiment of the present specification;
[0135] FIG. 1B illustrates an exemplary breakdown of functions
performed by a data ingestion system and a data processing
system;
[0136] FIG. 1C illustrates an exemplary machine learning system, in
accordance with an embodiment of the present specification;
[0137] FIG. 2 is a block diagram illustrating processing of a
sensor data stream before it reaches a query processor, in
accordance with an embodiment of the present specification;
[0138] FIG. 3 illustrates an overview of sources of digital data,
in accordance with an embodiment of the present specification;
[0139] FIG. 4A illustrates characteristic metrics for visual data,
in accordance with an embodiment of the present specification;
[0140] FIG. 4B provides a graphical presentation of color pair
confusion components, in accordance with an embodiment of the
present specification;
[0141] FIG. 4C shows a graph illustrating how luminance may be
found for a given chromaticity that falls on the top surface of the
display gamut projected into 3D chromoluminance space;
[0142] FIG. 5 illustrates characteristic metrics for auditory
information, in accordance with an embodiment of the present
specification;
[0143] FIG. 6 illustrates characteristic metrics for eye tracking,
in accordance with an exemplary embodiment of the present
specification;
[0144] FIG. 7 illustrates characteristic metrics for manual input,
in accordance with an embodiment of the present specification;
[0145] FIG. 8 illustrates characteristic metrics for head tracking,
in accordance with an embodiment of the present specification;
[0146] FIG. 9 illustrates characteristic metrics for
electrophysiological and autonomic monitoring data, in accordance
with an embodiment of the present specification;
[0147] FIG. 10A illustrates an exemplary process of image analysis
of building curated data, in accordance with an embodiment of the
present specification;
[0148] FIG. 10B illustrates an exemplary process of image analysis
of building curated data, in accordance with an embodiment of the
present specification;
[0149] FIG. 10C illustrates an exemplary process of image analysis
of building curated data, in accordance with an embodiment of the
present specification;
[0150] FIG. 10D illustrates an exemplary process of image analysis
of building curated data, in accordance with an embodiment of the
present specification;
[0151] FIG. 11A illustrates pupil position and size and gaze
position over time;
[0152] FIG. 11B illustrates pupil position and size and gaze
position over time;
[0153] FIG. 12 is an exemplary outline of a data analysis
chain;
[0154] FIG. 13 provides a table containing a list of exemplary
metrics for afferent and efferent sources, in accordance with some
embodiments of the present specification;
[0155] FIG. 14 is an exemplary flow chart illustrating an overview
of the flow of data from a software application to the SDEP;
[0156] FIG. 15 is an exemplary outline of a pre-processing portion
of a process flow, in accordance with an embodiment of the present
specification;
[0157] FIG. 16 is an exemplary outline of a python scripting
portion of the analysis chain;
[0158] FIG. 17 is a flow chart illustrating a method of modifying
media, in accordance with an embodiment of the present
specification;
[0159] FIG. 18 is a flow chart illustrating a method of modifying
media, in accordance with another embodiment of the present
specification;
[0160] FIG. 19 illustrates a flow chart describing an exemplary
process for improving comprehension, in accordance with some
embodiments of the present specification;
[0161] FIG. 20 illustrates a flow chart describing an exemplary
process for decreasing fatigue, in accordance with some embodiments
of the present specification;
[0162] FIG. 21 illustrates a flow chart describing an exemplary
process for increasing engagement, in accordance with some
embodiments of the present specification;
[0163] FIG. 22 illustrates a flow chart describing an exemplary
process for improving performance, in accordance with some
embodiments of the present specification;
[0164] FIG. 23 illustrates a flow chart describing an exemplary
process for decreasing symptoms associated with visually-induced
motion sickness secondary to visual-vestibular mismatch, in
accordance with some embodiments of the present specification;
[0165] FIG. 24 illustrates a flow chart describing an exemplary
process for decreasing symptoms associated with post-traumatic
stress disorder (PTSD), in accordance with some embodiments of the
present specification;
[0166] FIG. 25 illustrates a flow chart describing an exemplary
process for decreasing double-vision related to accommodative
dysfunction, in accordance with some embodiments of the present
specification;
[0167] FIG. 26 illustrates a flow chart describing an exemplary
process for decreasing vection due to unintended peripheral field
stimulation, in accordance with some embodiments of the present
specification;
[0168] FIG. 27 illustrates a flow chart describing an exemplary
process for decreasing hormonal dysregulation arising from
excessive blue light exposure, in accordance with some embodiments
of the present specification;
[0169] FIG. 28 illustrates a flow chart describing an exemplary
process for decreasing potential phototoxicity from overexposure to
screen displays, in accordance with some embodiments of the present
specification;
[0170] FIG. 29 illustrates a flow chart describing an exemplary
process for decreasing nausea and stomach discomfort, in accordance
with some embodiments of the present specification;
[0171] FIG. 30 illustrates a flow chart describing an exemplary
process for decreasing visual discomfort, including at least one of
eyestrain, dry eye, eye tearing, foreign body sensation, feeling of
pressure in the eyes, or aching around the eyes, in accordance with
some embodiments of the present specification;
[0172] FIG. 31 illustrates a flow chart describing an exemplary
process for decreasing disorientation and postural instability, in
accordance with some embodiments of the present specification;
[0173] FIG. 32 illustrates a flow chart describing an exemplary
process for decreasing headaches and difficulties in focusing, in
accordance with some embodiments of the present specification;
[0174] FIG. 33 illustrates a flow chart describing an exemplary
process for decreasing blurred vision and myopia, in accordance
with some embodiments of the present specification;
[0175] FIG. 34 illustrates a flow chart describing an exemplary
process for decreasing heterophoria, in accordance with some
embodiments of the present specification;
[0176] FIG. 35 illustrates a flow chart describing an exemplary
process for decreasing fixation disparity, in accordance with some
embodiments of the present specification;
[0177] FIG. 36 illustrates a flow chart describing an exemplary
process for decreasing vergence-accommodation disorders, in
accordance with some embodiments of the present specification;
[0178] FIG. 37 illustrates a flow chart describing an exemplary
process for increasing positive emotion, in accordance with some
embodiments of the present specification;
[0179] FIG. 38 illustrates a flow chart describing an exemplary
process for decreasing negative emotion, in accordance with some
embodiments of the present specification; and
[0180] FIG. 39 is a flow chart describing an exemplary process for
modifying media while enabling a micro-transaction, in accordance
with some embodiments of the present specification.
DETAILED DESCRIPTION
[0181] In various embodiments, the present specification provides
methods and systems for enabling modification of media in
accordance with a visual profile of a user and/or group of
users.
[0182] In another embodiment, the present specification describes
methods, systems and software that is provided to third party
developers of media (advertising and entertainment) who then use
the software and data to optimize the presentation of that media
for a user's specific vision characteristics.
[0183] In yet another embodiment, the present specification
describes methods, systems, and software for directly providing
media (advertising and entertainment) that already incorporates
software and data that, when experienced by a user, can be
optimized for that user's specific vision characteristics in
real-time.
[0184] In one embodiment, a Sensory Data Exchange Platform (SDEP)
is provided, wherein the SDEP may enable developers of media for
Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality
(MxR) systems and/or software to optimize the media for a user
and/or a group of users. In embodiments, users may include
programmers and developers of mobile platforms and web sites. In
embodiments, the VR, AR, and/or MxR media is presented to an
end-user through one or more electronic media devices including
computers, portable computing devices, mobile devices, or any other
device that is capable of presenting VR, AR, and/or MxR media.
[0185] In an embodiment, a user interacts with a software program
embodying at least a portion of the SDEP in a manner that enables
the software to collect user data and provided it to the SDEP. In
an embodiment, the user may interact directly or indirectly with a
SDEP to facilitate data collection. In an embodiment, the SDEP is a
dynamic, two-way data exchange platform with a plurality of sensory
and biometric data inputs, a plurality of programmatic instructions
for analyzing the sensory and biometric data, and a plurality of
outputs for the delivery of an integrated visual assessment.
[0186] In some embodiments, the SDEP outputs as a general
collective output a "visual data profile" or a "vision performance
index" (VPI). The visual data profile or vision performance index
may be used to optimize media presentations of advertising, gaming,
or content in a VR/AR/MxR system or a conventional laptop, mobile
phone, desktop or tablet computing environment. In embodiments, the
platform of the present specification is capable of taking in a
number of other data sets that may enhance the understanding of a
person's lifestyle and habits. In addition, machine learning,
computer vision, and deep learning techniques are employed to help
monitor and predict health outcomes through the analysis of an
individual's data.
[0187] In an embodiment, the SDEP is used via an operating system
executed on hardware (such as mobile, computer or Head Mounted
Display (HMD)). In another embodiment, the SDEP is used by one or
more content developers. In one embodiment, both hardware and
content developers use the SDEP. The SDEP may enable collection of
data related to how the user is interfacing with the content
presented, what aspects of the content they are most engaged with
and how engaged they are. Data collected through the SDEP may be
processed to create a profile for the user and or groups of users
with similar demographics. The content may be represented, for a
particular profile, in a way that conforms to the hardware
capabilities of the VR/AR/MxR system in a manner to optimize
experience of that user and other users with a similar profile.
[0188] For example, the experience may be optimized by representing
the media in a manner that may decrease phoria
movement--specifically, long periods of convergence with
simultaneous head movement to minimize visual vestibular mismatch;
blending optical zones/focal zones of objects in the VE to minimize
accommodative decoupling/dysfunction; disabling large peripheral
stimuli during central stimuli engagement, to decrease the
experience of vection; among other methods that enable an enhanced
VR/AR/MxR experience.
[0189] The present specification is directed towards multiple
embodiments. The following disclosure is provided in order to
enable a person having ordinary skill in the art to practice the
invention. Language used in this specification should not be
interpreted as a general disavowal of any one specific embodiment
or used to limit the claims beyond the meaning of the terms used
therein. The general principles defined herein may be applied to
other embodiments and applications without departing from the
spirit and scope of the invention. Also, the terminology and
phraseology used is for the purpose of describing exemplary
embodiments and should not be considered limiting. Thus, the
present invention is to be accorded the widest scope encompassing
numerous alternatives, modifications and equivalents consistent
with the principles and features disclosed. For purpose of clarity,
details relating to technical material that is known in the
technical fields related to the invention have not been described
in detail so as not to unnecessarily obscure the present
invention.
[0190] The term "and/or" means one or all of the listed elements or
a combination of any two or more of the listed elements.
[0191] The terms "comprises" and variations thereof do not have a
limiting meaning where these terms appear in the description and
claims.
[0192] Unless otherwise specified, "a," "an," "the," "one or more,"
and "at least one" are used interchangeably and mean one or more
than one.
[0193] For any method disclosed herein that includes discrete
steps, the steps may be conducted in any feasible order. And, as
appropriate, any combination of two or more steps may be conducted
simultaneously.
[0194] Also herein, the recitations of numerical ranges by
endpoints include all whole or fractional numbers subsumed within
that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, 5,
etc.). Unless otherwise indicated, all numbers expressing
quantities of components, molecular weights, and so forth used in
the specification and claims are to be understood as being modified
in all instances by the term "about." Accordingly, unless otherwise
indicated to the contrary, the numerical parameters set forth in
the specification and claims are approximations that may vary
depending upon the desired properties sought to be obtained by the
present invention. At the very least, and not as an attempt to
limit the doctrine of equivalents to the scope of the claims, each
numerical parameter should at least be construed in light of the
number of reported significant digits and by applying ordinary
rounding techniques.
[0195] Notwithstanding that the numerical ranges and parameters
setting forth the broad scope of the invention are approximations,
the numerical values set forth in the specific examples are
reported as precisely as possible. All numerical values, however,
inherently contain a range necessarily resulting from the standard
deviation found in their respective testing measurements.
[0196] It should be noted herein that any feature or component
described in association with a specific embodiment may be used and
implemented with any other embodiment unless clearly indicated
otherwise.
[0197] It should be further appreciated that all the afferent data
presented herein and efferent data collected are performed using a
hardware device, such as a mobile phone, laptop, tablet computer,
or specialty hardware device, executing a plurality of programmatic
instructions expressly designed to present, track, and monitor
afferent data and to monitor, measure, and track efferent data, as
further discussed below.
General Problem
[0198] Potential inciting factors of VIMS may be broadly
categorized into the following factor areas: Hardware, User, Task,
and Service. Methods and systems are needed that may optimize one
or a combination of these factors to ease development and adoption
of VR/AR/MxR. These factors are briefly discussed here.
Hardware Factors
[0199] Hardware device system variables impacting VIMS include
viewing mode (e.g. monocular, binocular or dichoptic), headset
design (e.g. fit, weight), optics (e.g. misalignment in the optics;
contrast, luminance), field of view (FOV), and time lag (i.e.
transport delay). HMD weight has been associated with the
experience of visual discomfort and injury. Thus, hardware factors
include field of view, resolution and/or frame rate, and time lag
or latency, among other variables.
[0200] For example, field of view (FOV), may be implicated in
producing visual discomfort symptoms. The FOV studies show that
narrow FOV (<50 degrees) reduces the perception of self-motion
and wide FOV (>100 degrees) may increase the presence and level
of simulator sickness. For a full immersion experience, a FOV of at
least 60.degree. is recommended. The sense of immersion can be
provided by parsing horizontal and vertical FOVs, which allows for
flexibility in the content presentation. In flight simulation
applications, for example, segmenting object presentation within a
horizontal FOV of 40.degree. by a vertical FOV of 30.degree.
improves the ergonomics and improves pilot performance.
[0201] Aside from influencing overall image quality, resolution may
also affect a user's experience of VIMS. It is often uncomfortable
to view low-quality images that are noisy or blurry. The visual
resolution in humans is 1 minute of arc and is a technological
limitation to many HMD systems. Depending on perceived distance in
the VR environment, increased resolution mitigates "pixel
perception" as objects gets closer. Having said that, it is
important to provide the highest possible resolution in the design
process in order to better accomplish immersive tasks in virtual
reality (and AR and MxR) applications. Refresh or frame rate is
another factor affecting visual comfort.
[0202] Most VR/AR and mixed environment systems have similar
problems related to motion sickness and visual discomfort. One of
the sources of these problems is latency or delay. Delay refers to
the time it takes from when a user triggers an action to when the
results of the user-triggered action are visible through the
VR/AR/MxR media. Delay may also be attributed to the screens that
are used to display VR/AR/MxR media. Specifically, features of a
screen such as screen refresh and response time may attribute to a
measure of delay. Measures of an acceptable level of delay, or
delay that may not result in motion sickness or other forms of
discomfort, may vary for different individuals.
[0203] Time lag between the individual and the system's action and
reaction potentially could influence a user's experience of VIMS
symptoms, as it affects human perception of visual and vestibular
cues. Therefore, reducing the sensor error of HMD systems may
minimize the VIMS experience. HMD optical characteristics, such as
eye relief (a fixed distance from the eyepiece lens to its exit
pupil), convergence demand, horizontal disparity, vertical
misalignment of displays, inter-ocular rotation difference,
vertical-horizontal magnification differences, luminance, focus
differences, temporal asynchrony, focal distance, field curvature
difference and inter-pupillary distance (IPD), are all potential
factors that can induce visual discomfort and headache when they
are misaligned or not optimally calibrated.
User Factors
[0204] Another factor related to the impact of VIMS is user
characteristics because it is known that individuals differ in
their susceptibility to VIMS. User characteristics may include,
among others, age and gender; visual deficits; plasticity; and
posture.
[0205] Age has been shown to have a significant relationship with
HMD-related eyestrain symptoms. Children 2-12 years of age have
immature visual systems and binocular function that is worse than
that of adults; this makes children more susceptible to both visual
discomfort caused by HMDs and oculomotor side effects including
reduced visual acuity, amblyopia, or strabismus. Adults with
limited fusional ranges experienced more visual discomfort,
specifically with convergent eye movement in response to stimuli in
VEs. Therefore, age effect on HMDs needs to be further studied and
incorporated into the design of future HMDs. In regards to gender,
females reported more simulator sickness and more often withdrew
from HMD-based VEs when compared to male participants. This
difference may be due to under-reporting from self-reports by males
(so-called "macho effect") or hormonal effects. Another possibility
is the gender difference in FOV, with female having a wider FOV,
increasing the risk for flicker perception, vection and motion
sickness susceptibility.
[0206] People with visual deficits may have an increased
susceptibility to oculomotor side effects compared to those without
such deficits, though more studies are needed in this area. A past
history of motion sickness or conditions that preclude these
symptoms (migraines), are also notable for predicting
susceptibility to motion sickness in HMD-based VEs. Individuals may
habituate or adapt to HMD-based VEs (i.e. plasticity), with
improvement in symptoms after repeated exposure to virtual
environments. However, this habituation is variable among groups,
with certain individuals adapting much more readily than others
following repeated exposures to stimuli.
[0207] Individuals with more plasticity may be less likely to
experience VIMS, though there may be variability in the amount of
time needed to adapt to the VE. Greater plasticity does not
translate to reduction or lack of initial symptoms, but rather the
ability to improve susceptibility to symptoms quicker, typically
following repeated exposures to VEs.
[0208] Based on the postural instability theory, an individual's
posture may also contribute to VIMS. Postural instability has a
unique role in VIMS, as it can be a cause for and a result of VIMS.
Studies have noted individuals with stable posture are less
susceptible to VIMS, suggestive of an inverse relationship between
postural stability and VIMS. Postural stability is a confluence of
sensory inputs that are visual, vestibular and somatosensory in
nature. Two neural reflexes involved in postural stability include
the vestibular-ocular reflex (stabilizes objects on the retina) and
the vestibular-spinal reflex (stabilizes posture while body in
motion). When visual and vestibular inputs are not in synchrony,
the result is postural instability, VIMS, or both. Postural
instability may last for several hours after exposure. Special
considerations for HMD user safety, as related to the risk of
postural instability, must be kept in mind. A recommendation that
HMD users allow for a reorientation/recovery time prior to engaging
in potentially dangerous activities such as driving or sports may
be in order.
Task Factors
[0209] Task characteristics have been also identified as
potentially affecting VIMS. Among these tasks, duration of time in
Virtual Environments (VE) is most notable. Longer exposure to VE
increases the incidence of VIMS. These symptoms may persist up to
60 minutes after exposure. Another important factor shown to
influence VIMS is vection (i.e. an illusion of self-motion), with
faster vection resulting in greater sickness symptoms. Viewing
HMD-based VR in a sitting position may reduce symptoms, as sitting
reduces the demands on postural control. More complicated tasks,
such as reading, may induce total symptom severity scores and
oculomotor-related symptom scores that are significantly higher
than those observed with movies or games. These findings imply that
more demanding tasks probably will create some degree of eyestrain.
Increased reading sensitivity, when compared to watching a movie or
playing a game, might be due to activation of different areas of
the brain, which may make reading more complex than other tasks.
Alternatively, reading can affect attention and blink rate, which
may also contribute to an increase in VIMS. Moreover, inappropriate
vertical gaze angle may cause increased oculomotor changes and
visual discomfort.
Service Factors
[0210] Technical advances in hardware components will reduce
physiologic ergonomic issues including HMD weight, system time
delay, and luminance. However, given the multifactorial nature of
VIMS, the conflict between visual and vestibular input remains a
significant problem. Further reduction in VIMS needs to take into
consideration how content is created and how it influences service
factors. Service factors, in turn, need to take into consideration
the VIMS effect on the viewer.
[0211] Services can be created intended for a wide audience shared
experience (e.g. Broadcast) or for narrow niche audience experience
(e.g. longtail content in video on demand systems). For large scale
audiences, further VIMS reduction will be highly dependent on
content creation, where FOV creates an immersive environment where
watching a movie or playing a game would be more preferred rather
than reading. User factors could be mitigated by reducing visual
cues that create sensory input conflicts. This could be done by
making the viewer more of a detached fixed observer (i.e. reduced
vection) and making the content more of a scenery type which has
already been made popular in viewing UHD/HDR material.
[0212] Content may be transmitted via linear broadcast, on-demand,
or downloaded among other avenues. Service factors for VR need to
be cognizant of the bits delivered to HMDs. This stream of bits
constrains resolutions and frame rates while also affecting
time-lag or latency where frame rate is a gating factor to latency.
Ways to create a service around this dimension are to permit
progressive download, on-demand streaming, or real-time linear
streaming deliveries. This progression tracks like the evolution of
Video streaming where progressive download was initially done until
codec compression efficiencies and bandwidth expansion advanced
enough to allows for online streaming and ultimately real-time
encoding/streaming to occur.
[0213] VR environments hinge on some level of interactivity between
the viewer and the observed environment. The accepted amount of
interactivity can be designed into the service and should consider
a reduction of VIMS effects.
[0214] Optimized service factors can reduce VIMS effects by
creating and making aware to users and the content community a set
of guidelines of optimized visual ergonomics for HMD use.
[0215] Next generation computing will be dominated by immersive
platforms that represent a new level of synergy between users,
content, and technology. VR/AR/MxR is the fourth major platform
shift after PC, web, and email. VR/AR/MxR is also a part of
technology commonly known as Brain-Machine (Computer) Interface
(BMI/BCI). BMI/BCI has clinical applications, such as and not
limited to EEG, MRI, fMRI, and Ultrasound. Most of these clinical
interfaces are modular in nature. These can be invasive, or
non-invasive; portable, or non-portable. Non-clinical applications
may include gaming, military, and others. Among these different
potential interfaces, the division in clinical and non-clinical
context, is in part limited to the portability of the interfaces,
with non-clinical being traditionally more portable. It is expected
that an area of most intensive future development and investment is
likely to be portable non-invasive systems such as gaming,
especially incorporating non-invasive BMI with AR, VR, and MxR.
There is a need developing to standardize BMI for clinical and
non-clinical applications.
[0216] Overall BMI standardization requires standardizing the
interoperability, connectivity, and modularity of multiple sensory
interfaces with the brain, with many being closed-looped. There is
thus a need for methods and systems that, given the current
limitations of closed-loop systems, can support a standardization
of these requirements.
[0217] Current health risks of BMI include visually-induced motion
sickness secondary to visual-vestibular mismatch, double vision
related to accommodative dysfunction, vection to unintended
peripheral field stimulation, hormonal dysregulation (circadian
rhythm) from blue light exposure, and potential phototoxicity from
overexposure to screen displays. HMDs are also influenced by user
factors including gender, inter-pupillary distance variances,
accommodative amplitude (age-dependent), postural stability and the
type of software content being displayed--as more visually-task
oriented content tend to be more disruptive.
Definitions
[0218] The term "Virtual Reality" or "VR" is used throughout this
specification, and, in embodiments, refers to immersive
computer-simulated reality, or the computer-generated simulation of
a three-dimensional image or environment that can be interacted
with in a seemingly real or physical way by a person using special
electronic equipment, such as a helmet with a screen inside and/or
gloves fitted with sensors.
[0219] In embodiments, Augmented Reality (AR), also used along with
VR throughout this specification, is a technology that superimposes
a computer-generated image on a user's view of the real world, thus
providing a composite view. In embodiments, a common helmet-like
device is the HMD, which is a display device, worn on the head or
as part of the helmet, that has a small display optic in front of
one (monocular HMD) or each eye (binocular HMD). In embodiments,
the SDEP is a cloud-based service that any party can access in
order to improve or otherwise modify a visually presented product
or service.
[0220] Further, in embodiments, Mixed Reality (MxR), is also used
with VR and AR throughout this specification. MxR, also referred to
as hybrid reality, is the merging of VR and/or AR environments with
the real environment to produce new levels of visual-experiences
where physical and digital objects co-exist and interact in real
time.
[0221] In embodiments, VR, AR, and MxR devices could include one or
more of electronic media devices, computing devices, portable
computing devices including mobile phones, laptops, personal
digital assistants (PDAs), or any other electronic device that can
support VR, AR, or MxR media. It should be noted herein that while
the present specification is disclosed in the context of Virtual
Reality, any and all of the systems and methods described below may
also be employed in an Augmented Reality environment as well as
Mixed Reality environments. So, where a Virtual Reality (VR) system
is described, it should be understood by those of ordinary skill in
the art that the same concepts may apply to an Augmented Reality
(AR) and a Mixed Reality (MxR) system.
Eye-Tracking Definitions
[0222] In terms of performance, several eye tracking measures could
be put into the context of Vision Performance Index (VPI)
components, which are defined and described in detail in subsequent
section of the specification. Blink rate and vergence measures can
feed into measures of Fatigue and Recovery. Gaze and, more
specifically, fixation positions can be used to estimate Reaction
and Targeting measures. Continuous error rates during pursuit eye
movements can also become targeting measures.
[0223] Various examples of physical measures for eye tracking may
be available with desired standard units, expected ranges for
measured values and/or, where applicable, thresholds for various
states or categories based on those measures. Some references are
provided through sections that discuss various components and
subcomponents of eye tracking.
[0224] The following terms are associated with eye-tracking
measures as made from a combination of video recording and image
processing techniques; expert human scoring; and/or from
Electrooculography (EOG) recording. Video eye tracking (VET)
techniques may use explicit algorithmic analysis and/or machine
learning to estimate proportional eyelid opening/closure, pupil
size, pupil position (relative to the face) and gaze direction
independently for each eye. EOG recording may be used to estimate
eyelid and eye motion and, with limited precision, eye gaze
direction. Both recording modalities may sample at rates of tens to
thousands of times per second and allow for analysis of position,
velocity, direction, and acceleration for the various measures.
Comparison between the two eyes allows for measures of vergence
which in turn allows for a three-dimensional (3D) gaze direction to
be estimated.
[0225] Palpebral Fissure refers to the opening of the eyelids.
While typically about 30 millimeters (mm) wide by 10 mm tall, most
measurements can be relative to baseline distances measured on
video. Of particular interest is the height (interpalpebral fissure
height) as it relates to the following terms:
[0226] Percent Open (p.sub.eye open) refers to how open the left
(p.sub.left eye open), right (p.sub.right eye open), or both
(p.sub.both eyes open) eyes are, relative to the maximum open
distance and typically measured over a predefined period of
time.
[0227] Proportion Open (P.sub.eyes open) refers to the proportion
of time the eyes are open over a span of time (for example, during
a session (P_(eyes open|session))). The threshold for `open` may be
variable (for example, P.sub.eyes open(where p.sub.both eyes
open.gtoreq.25%)).
[0228] Blink can be defined as a complete closure of both eyes
(p.sub.both eye open=0%) for between roughly 10 to 400 milliseconds
(ms), with a specific measured blink closure time being based on
differences among users and the eye tracking method.
[0229] Blink Rate (Frequency) (f.sub.blink) refers to the average
number of blinks per second (s.sup.-1 or Hz) measured for all
blinks and/or blinks over a period of time (e.g. f_(blink|target
present)). The blink rate may be referred to as a rate of change of
the blink rate or a ratio of partial blinks to full blinks.
[0230] Blink Count Number (N_blink) refers to the number of blinks
measured for all blinks and/or blinks over a period of time (e.g.
N_(blink|target present)).
[0231] Pupil Size (S_pupil) refers to the size of the pupil,
typically the diameter in millimeters (mm).
[0232] Pupil Position ([x, y]_pupil) refers to the position of the
left ([x, y]_(left pupil)) or right ([x, y]_(right pupil)) pupil
within the fixed reference frame of the face, typically as a
function of time. The pupil position definition includes, and is
dependent upon, an initial pupil position and a final pupil
position.
[0233] Gaze Direction ([.theta.,.phi.]_gaze) refers to the
direction in 3D polar coordinates of left ([.theta.,.phi.]_(left
gaze)) or right ([.theta.,.phi.]_(right gaze)) eye gaze relative to
the face, typically as a function of time. This is a measure of
where the eyes are facing without regard to what the eyes see. It
may be further classified as relevant or irrelevant depending on a
task or a target.
[0234] Gaze Position ([x, y, z]_gaze or [r, .theta., .phi.]_gaze)
refers to the position (or destination) of gaze in the environment
in Cartesian or spherical 3D coordinates, typically as a function
of time. The reference frame may be with respect to the user,
device or some other point in space, but most commonly the origin
of a coordinate space will be the user's eyes (one or the other or
a point halfway between). The gaze position definition includes,
and is dependent upon, an initial gaze position and a final gaze
position.
[0235] Vergence is derived from estimated gaze direction and may be
quantified as the difference in angle of the two eyes (positive
differences being divergence and negative being convergence). When
derived from gaze position, vergence contributes to and may be
quantified as the distance of the gaze position from the eyes/face.
Convergence and divergence may each be defined by their duration
and rate of change.
[0236] Fixation Position ([x, y, z].sub.fixation or [r, .theta.,
.phi.].sub.fixation) is the position of a fixation in Cartesian or
spherical 3D space measured as the estimated position of the user's
gaze at a point in time. The fixation position definition includes,
and is dependent upon, an initial fixation position and a final
fixation position.
[0237] Fixation Duration (D.sub.fixation) is the duration of a
fixation (i.e. the time span between when the gaze of the eye
arrives at a fixed position and when it leaves), typically measured
in milliseconds or seconds (s). The average duration is denoted
with a bar D.sub.fixation and may represent all fixations,
fixations over a period of time (e.g. _D_(fixation|target present))
and/or fixations within a particular region (e.g.
_D_(fixation|display center)). The fixation duration definition
includes, and is dependent upon, a rate of change in fixations.
[0238] Fixation Rate (Frequency) (f_fixation) refers to the average
number of fixations per second (s (-1) or Hz) measured for all
fixations, fixations over a period of time (e.g. f_(fixation|target
present)) and/or fixations within a particular region (e.g.
f_(fixation|display center)).
[0239] Fixations Count (Number) (N.sub.fixation) refers to the
number of fixations measured for all fixations, fixations over a
period of time (e.g. N_(fixation|target present)) and/or fixations
within a particular region (e.g. N_(fixation|display center)).
[0240] Saccade Position ([x.sub.1, y.sub.1, z.sub.1|x.sub.2,
y.sub.2, z.sub.2].sub.saccade or [r.sub.1, .theta..sub.1,
.phi..sub.1|r.sub.2, .theta..sub.2, .theta..sub.2,
.phi..sub.2].sub.saccade) refers to the starting (1) and ending (2)
positions of a saccadic eye movement in Cartesian or spherical 3D
space. The reference frame will generally be the same, within a
given scenario, as that used for gaze position. The saccade
position definition includes, and is dependent upon, a rate of
change, an initial saccade position, and a final saccade
position.
[0241] Saccade Angle (.THETA..sub.accade) refers to an angle
describing the 2-dimensional (ignoring depth) direction of a
saccade with respect to some reference in degrees (.degree.) or
radians (rad). Unless otherwise specified the reference is
vertically up and the angle increases clockwise. The reference may
be specified (e.g. (.THETA..sub.accade target) to denote the
deviation of the saccade direction from some desired direction
(i.e. towards a target). The average saccade direction is denoted
with a bar .THETA..sub.saccade and may represent all or a subset of
saccades (e.g. _.THETA._(saccade|target present)); because the
direction is angular (i.e. circular) the average direction may be
random unless a relevant reference is specified (e.g.
_.THETA._(saccade-target|target present)). The saccade angle may be
used to determine how relevant a target is to a user, also referred
to as a context of relevancy towards a target.
[0242] Saccade Magnitude (M.sub.saccade) refers to the magnitude of
a saccade relating to the distance traveled; this may be given as a
visual angle in degrees (.degree.) or radians (rad), a physical
distance with regard to the estimated gaze position (e.g. in
centimeters (cm) or inches (in)) or a distance in display space
with regard to the estimated gaze position on a display (e.g. in
pixels (px)). In reference to a particular point (P) in space, the
component of the saccade magnitude parallel to a direct line to
that point may be given as:
M.sub.saccade-P=M.sub.saccadecos(.THETA..sub.saccade-P)
[0243] where M.sub.saccade is the magnitude of the saccade and
.THETA..sub.accade-P is the angle between the saccade direction and
a vector towards point P. The average saccade magnitude is denoted
with a bar M.sub.saccade, and this notation may be applied to all
saccades and/or a subset in time or space and with regard to
saccade magnitudes or the components of saccade magnitude relative
to a designated point.
[0244] Pro-Saccade refers to movement towards some point in space,
often a target, area of interest or some attention-capturing event.
By the above terminology a pro-saccade would have a relatively
small saccadic angle and positive magnitude component relative to a
designated position.
[0245] Anti-Saccade refers to movement away from some point in
space, often due to aversion or based on a task (instruction to
look away). By the above terminology an anti-saccade would have a
relatively large saccadic angle (around .+-.180.degree. or
.+-..pi.rad) and a negative magnitude component relative to a
designated position.
[0246] Inhibition of Return (IOR) is related to anti-saccades and
describes a tendency during search or free viewing to avoid
recently fixated regions which are less informative. IOR reflects a
general strategy for efficient sampling of a scene. It may be
furthered defined by, or a function of, anti-saccades.
[0247] Saccade Velocity (v.sub.saccade) or the velocity of a
saccade is taken as the change in magnitude over time (and not
generally from magnitude components towards a reference point).
Based on the degree of magnitude and direction of the saccade
velocity, it may be indicative of a degree of relevancy of the
target to the user. The average saccade velocity is denoted with a
bar v.sub.saccade and may be applied to all saccades or a subset in
time and/or space.
[0248] Saccade Rate (Frequency) (f.sub.saccade) denotes the average
number of saccades per second (s.sup.-1 or Hz) measured for all
saccades, saccades over a period of time (e.g. f_(saccade|target
present)), saccades within a particular region (e.g.
f_(saccade|display center)) and/or saccades defined by their
direction (e.g. f_(saccade|towards target)).
[0249] Saccade Count (Number) (N.sub.saccade) is the number of
saccades measured for all saccades, saccades over a period of time
(e.g. N_(saccade|target present)), saccades within a particular
region (e.g. N_(saccade|display center)) and/or saccades defined by
their direction (e.g. N_(saccade|towards target)).
[0250] Pursuit Eye Movements (PEM) is used to refer to both smooth
pursuit eye movements where gaze tracks a moving object through
space and vestibulo-ocular movements that compensate for head or
body movement. It may be further defined by data indicative of an
initiation, a duration, and/or a direction of smooth PEM. Also
included are compensatory tracking of stationary objects from a
moving frame of reference. PEM generally do not consist of
fixations and saccades but rather continuous, relatively slow
motion interrupted by occasional error-correcting saccades. The
smooth and saccadic portions of a PEM trace may be subtracted and
analyzed separately.
Body Tracking Definitions
[0251] Body tracking entails measuring and estimating the position
of the body and limbs as a function of time and/or discrete events
in time associated with a class of movement (e.g. a nod of the
head). Information sources include video tracking with and without
worn markers to aid in image processing and analysis, position
trackers, accelerometers and various hand-held or worn devices,
platforms, chairs, or beds.
[0252] Screen Distance (d.sub.screen) refers to the distance
between the user's eyes (face) and a given display device. As a
static quantity, it is important for determining the direction
towards various elements on the screen (visual angle), but as a
variable with time, screen distance can measure user movements
towards and away from the screen. Screen distance is dependent upon
a rate of change, an initial position, and a final position between
the user's eyes (face) and a given display device. Combined with
face detection algorithms, this measure may be made from device
cameras and separate cameras with known position relative to
displays.
[0253] Head Direction (Facing) ([.theta., .phi.].sub.facing) refers
to the direction in 3D polar coordinates of head facing direction
relative to either the body or to a display or other object in the
environment. Tracked over time this can be used to derive events
like nodding (both with engagement and fatigue), shaking, bobbing,
or any other form of orientation. Head direction is dependent upon
a rate of change, an initial position, and a final position of head
facing direction relative to either the body or to a display or
other object in the environment.
[0254] Head Fixation, while similar to fixations and the various
measures associated with eye movements, may be measured and
behavior-inferred. Generally head fixations will be much longer
than eye fixations. Head movements do not necessarily indicate a
change in eye gaze direction when combined with vestibulo-ocular
compensation. Head fixation is dependent upon a rate of change, an
initial position, and a final position of head fixations.
[0255] Head Saccade, while similar to saccades and their various
measures associated with eye movements, may be measured as rapid,
discrete head movements. These will likely accompany saccadic eye
movements when shifting gaze across large visual angles. Orienting
head saccades may also be part of auditory processing and occur in
response to novel or unexpected sounds in the environment.
[0256] Head Pursuit, while similar to pursuit eye movements, tend
to be slower and sustained motion often in tracking a moving object
and/or compensating for a moving frame of reference.
[0257] Limb Tracking refers to the various measures that may be
made of limb position over time using video with image processing
or worn/held devices that are themselves tracked by video,
accelerometers or triangulation. This includes pointing devices
like a computer mouse and hand-held motion controllers. Relative
limb position may be used to derive secondary measures like
pointing direction. Limb tracking is dependent upon a rate of
change, an initial position, and a final position of the limbs.
[0258] Weight Distribution refers to the distribution of weight
over a spatial arrangement of sensors while users stand, sit or lie
down can be used to measure body movement, position and posture.
Weight distribution is dependent upon a rate of change, an initial
position, and a final position of weight.
[0259] Facial expressions including micro-expressions, positions of
eyebrows, the edges, corners, and boundaries of a person's mouth,
and the positions of a user's cheekbones, may also be recorded.
Electrophysiological and Autonomic Definitions
[0260] Electrophysiological measures are based on recording of
electric potentials (voltage) or electric potential differences
typically by conductive electrodes placed on the skin. Depending on
the part of the body where electrodes are placed various
physiological and/or behavioral measures may be made based on a set
of metrics and analyses. Typically voltages (very small--microvolts
.mu.V) are recorded as a function of time with a sample rate in the
thousands of times per second (kHz). While electrophysiological
recording can measure autonomic function, other methods can also be
used involving various sensors. Pressure transducers, optical
sensors (e.g. pulse oxygenation), accelerometers, etc. can provide
continuous or event-related data.
[0261] Frequency Domain (Fourier) Analysis allows for the
conversion of voltage potentials as a function of time (time
domain) into waveform energy as a function of frequency. This can
be done over a moving window of time to create a spectrogram. The
total energy of a particular frequency or range of frequencies as a
function of time can be used to measure responses and changes in
states.
[0262] Electroencephalography (EEG) refers to electrophysiological
recording of brain function. Time averaged and frequency domain
analyses (detailed below) provide measures of states. Combined with
precise timing information about stimuli, event-related potentials
(EEG-ERP) can be analyzed as waveforms characteristic of a
particular aspect of information processing.
[0263] Frequency Bands are typically associated with brain activity
(EEG) and in the context of frequency domain analysis different
ranges of frequencies are commonly used to look for activity
characteristic of specific neural processes or common states.
Frequency ranges are specified in cycles per second (s.sup.-1 or
Hz): [0264] Delta--Frequencies less than 4 Hz. Typically associated
with slow-wave sleep. [0265] Theta--Frequencies between 4 and 7 Hz.
Typically associated with drowsiness. [0266] Alpha--Frequencies
between 8 and 15 Hz. [0267] Beta--Frequencies between 16 and 31 Hz.
[0268] Gamma--Frequencies greater than 32 Hz.
[0269] Electrocardiography (ECG) refers to electrophysiological
recording of heart function. The primary measure of interest in
this context is heart rate.
[0270] Electromyography (EMG) refers to electrophysiological
recording of muscle tension and movement. Measures of subtle muscle
activation, not necessarily leading to overt motion, may be made.
Electrodes on the face can be used to detect facial expressions and
reactions.
[0271] Electrooculography (EOG) refers to electrophysiological
recording across the eye. This can provide sensitive measures of
eye and eyelid movement, however with limited use in deriving pupil
position and gaze direction.
[0272] Electroretinography (ERG) refers to electrophysiological
recording of retinal activity.
[0273] Galvanic Skin Response (GSR) (Electrodermal response) is a
measure of skin conductivity. This is an indirect measure of the
sympathetic nervous system as it relates to the release of
sweat.
[0274] Body Temperature measures may be taken in a discrete or
continuous manner. Relatively rapid shifts in body temperature may
be measures of response to stimuli. Shifts may be measured by
tracking a rate of change of temperature, an initial temperature,
and a final temperature.
[0275] Respiration Rate refers to the rate of breathing and may be
measured from a number of sources including optical/video,
pneumography and auditory and will typically be measured in breaths
per minute (min.sup.-1 Brief pauses in respiration (i.e. held
breath) may be measured in terms of time of onset and duration.
[0276] Oxygen Saturation (S.sub.O.sub.2) is a measure of blood
oxygenation and may be used as an indication of autonomic function
and physiological state.
[0277] Heart Rate is measured in beats per minute (min.sup.-1nd may
be measured from a number of sources and used as an indication of
autonomic function and physiological state.
[0278] Blood Pressure is typically measured with two values: the
maximum (systolic) and minimum (diastolic) pressure in millimeters
of mercury (mm Hg). Blood pressure may be used as an indication of
autonomic function and physiological state.
Efferent Audio Recording Definitions
[0279] Audio recording from nearby microphones can measure
behavioral and even autonomic responses from users. Vocal responses
can provide measures of response time, response meaning or content
(i.e. what was said) as well as duration of response (e.g. "yeah"
vs. "yeeeeeeeaaaah"). Other utterances like yawns, grunts or
snoring might be measured. Other audible behaviors like tapping,
rocking, scratching or generally fidgety behavior may be measured.
In certain contexts, autonomic behaviors like respiration may be
recorded.
[0280] Vocalizations, such as spoken words, phrases and longer
constructions may be recorded and converted to text strings
algorithmically to derive specific responses. Time of onset and
duration of each component (response, word, syllable) may be
measured. Other non-lingual responses (yelling, grunting, humming,
etc.) may also be characterized. Vocalizations may reflect a range
of vocal parameters including pitch, loudness, and semantics.
[0281] Inferred Efferent Responses refer to certain efferent
responses of interest that may be recorded by audio and indicate
either discrete responses to stimuli or signal general states or
moods. Behaviors of interest include tapping, scratching, repeated
mechanical interaction (e.g. pen clicking) bouncing or shaking of
limbs, rocking and other repetitive or otherwise notable
behaviors.
[0282] Respiration, such as measures of respiration rate, intensity
(volume) and potentially modality (mouth vs. nose) may also be
made.
Afferent Classification/Definitions
[0283] The states discussed below are generally measured in the
context of or response to various stimuli and combinations of
stimuli and environmental states. A stimulus can be defined by the
afferent input modality (visual, auditory, haptic, etc.) and
described by its features. Features may be set by applications
(e.g. setting the position, size, transparency of a sprite
displayed on the screen) or inferred by image/audio processing
analysis (e.g. Fourier transforms, saliency mapping, object
classification, etc.).
[0284] Regions of interest as discussed below may be known ahead of
time and set by an application, may be defined by the position and
extent of various visual stimuli and/or may be later derived after
data collection by image processing analysis identifying
contiguous, relevant and/or salient areas. In addition to stimulus
features, efferent measures may be used to identify regions of
interest (e.g. an area where a user tends to fixate is defined by
gaze position data). Likewise both afferent and efferent measures
may be used to segment time into periods for summary analysis (e.g.
total number of fixations while breath is held).
Sensory Data Exchange Platform Overview
[0285] Reference is made to FIG. 1A, which shows a block diagram
100 illustrating user interaction with an exemplary SDEP, in
accordance with an embodiment of the present specification. In an
embodiment, a user 102 interfaces with a VR/AR/MxR system 104.
VR/AR/MxR system 104 may include devices such as HMDs, sensors,
and/or any other forms of hardware elements 106 that present
VR/AR/MxR media to the user in the form of a stimulus, and enables
collection of user response data during user interaction with the
presented media. The media may be communicated by a server, through
a network, or any other type of content platform that is capable of
providing content to HMDs. Sensors may be physiological sensors,
biometric sensors, or other basic and advanced sensors to monitor
user 102. Additionally, sensors may include environmental sensors
that record audio, visual, haptic, or any other types of
environmental conditions that may directly or indirectly impact the
vision performance of user 102. VR/AR/MxR system 104 may also
include software elements 108 that may be executed in association
with hardware elements 106. Exemplary software elements 108 include
gaming programs, software applications (apps), or any other types
of software elements that may contribute to presentation of a
VR/AR/MxR media to user 102. Software elements 108 may also enable
the system to collect user response data. Collected data may be
tagged with information about the user, the software application,
the game (if any), the media presented to the user, the session
during which the user interacted with the system, or any other
data. A combination of hardware elements 106 and software elements
108 may be used to present VR/AR/MxR media to user 102.
[0286] In an embodiment, stimulus and response data collected from
user's 102 interaction with VR/AR/MxR system 104 may constitute
data sources 110. Data sources 110 may be created within an SDEP
118 based on an interaction between software elements 108 and SDEP
118. Software elements 108 may also interact with SDEP 118 through
proprietary function calls included in a Software Development Kit
(SDK) for developers (i.e. the developers may send/receive data
to/from SDEP 118 using predefined functions). SDEP 118 may include
storage and processing components and could be a computing system.
The functionality of SDEP 118 may largely reside on one or more
servers and the data stored and retrieved from cloud services.
Sources of data may be in the form of visual data, audio data, data
collected by sensors deployed with VR/AR/MxR system 104, user
profile data, or any other data that may be related to user 102.
Visual data may largely include stimulus data and may be sourced
from cameras (such as cell phone cameras or other vision
equipment/devices), or from other indirect sources such as games
and applications (apps). Sensors may provide spatial and time
series data. User data may pertain to login information, or other
user-specific information derived from their profiles, from social
media apps, or other personalized sources. In embodiments, data
sources are broadly classified as afferent data sources and
efferent data sources, which are described in more detail in
subsequent sections of the specification. In an embodiment, user
profile data may be collected from another database, or may be
provided through a different source. In an exemplary embodiment
user profile data may be provided by service providers including
one or more vision care insurance provider. In other embodiments,
the user profile data may be collected from other sources including
user's device, opt-in options in apps/games, or any other
source.
[0287] Data sources 110 may be provided to a data ingestion system
112. Data ingestion system 112 may extract and/or transform data in
preparation to process it further in a data processing system 114.
Data adapters, which are a set of objects used to communicate
between a data source and a dataset, may constitute data ingestion
system 112. For example, an image data adapter module may extract
metadata from images, and may also process image data. In another
example, a video data adapter module may also extract metadata from
video data sources, and may also include a video transcoder to
store large volumes of video into distributed file system. In
another example, a time series data adapter module parses sensor
data to time series. In another embodiment, a spatial data adapter
module may utilize data from relatively small areas such as skin,
and spatially transform the data for area measurements. In another
example, a user profile data adapter module may sort general user
data, such as through a login, a social media connect API, unique
identifiers on phone, and the like.
[0288] SDEP 118 may further comprise a data processing system 114
that receives conditioned data from data ingestion system 112. A
machine learning module 152 within data processing system 114 may
communicate with a storage and a real time queue to output data to
a data serving system 116, which may include an Application Program
Interface (API). In embodiments, the machine learning system may
implement one or more known and custom models to process data
output from data ingestion system 112.
[0289] FIG. 1B illustrates an exemplary process of breakdown of
functions performed by data ingestion system 112 and data
processing system 114. In embodiments, at 172, an application
residing at system 104 collects stimulus and response data. The
stimulus and response data is forwarded in the form of information
related to display, color, light, image, position, time, user,
session, and other data related to the user interaction. Data may
be represented in the application in the raw form used for
presenting images on a display, playing sounds through speakers and
taking in user input information relevant to the running of the
application. Additional telemetry information and video and sound
recording from more advanced systems (i.e. VR/AR/MxR) may also be
included.
[0290] At 174, a software toolkit may take in the raw programmatic
information from the application and apply various conversions to
represent data in a more physically and/or physiologically relevant
form. Images and video, combined with information about the display
hardware, may be converted from red, green and blue (RGB) values
into CIE 1931 chromoluminance values (and/or some other
physiologically relevant chromoluminance space). Spatial display
information (horizontal and vertical pixel coordinates), combined
with estimates of physical display size and user viewing distance,
may be converted into head-centric visual angle and distance. The
data may be combined further with estimated gaze direction from eye
tracking and this may be further converted into retinal
coordinates. Likewise user interface markers (e.g. mouse cursor)
may have their position converted. In embodiments, some other
relevant data may pass through without conversion. In some
applications, information about the current and previous
interactions may be utilized by the toolkit to provide the
application with suggestions for efficient sampling towards
estimating psychometric parameters (shown as Bracketing
information).
[0291] At 176, image processing and analysis, relying on machine
learning or deep learning applications, may break down image or
audio information into relevant features (for example, edges,
contours, textures, and others) and objects for which parameters
like identity, spatial location and extent, motion, and the like,
may be estimated.
[0292] At 178, raw, physical parameters of stimuli and responses
may be combined and analyzed into psychometric estimates of
detection, discrimination, reaction, accuracy, memory, and other
derivative measures. In embodiments derivative measure may include
measures for trend analysis.
[0293] User and session data may be forwarded throughout the
process illustrated in FIG. 1B to tag stimulus, response, and
analysis data, in order to provide context for later presentation
and/or analysis.
[0294] In embodiments, data output from the analysis at 174 may
include graphs for targeting, reaction, detection, discrimination,
and other parameters that are useful to process and present vision
data.
[0295] FIG. 1C illustrates an exemplary machine learning system
152, in accordance with an embodiment of the present specification.
As described above, input data in the forms of visual data, audio
data, sensor data, and user data, interfaces with SDEP 118 and is
pre-processed through data integration system 112.
Processed/transformed data is provided to machine learning system
152. In embodiments, machine learning (ML) system processes
transformed data using one or more known and customized data
models, such as but not limited to naive Bayes, decision trees, and
others. In embodiments, ML system 152 creates a data pipeline based
on software framework such as Keystone ML and Velox. Modelled data
may be stored in a database 154. In an embodiment, a combination of
NoSQL (Accumulo/HBase), SQL (MySQL), and object storage (for raw
image and video data) is used. In embodiments, cell-level security
is provided to storage 154 in compliance with HIPAA.
[0296] In an embodiment, a real time queue 156 communicates with ML
system 152 to stream processing pipelines. In an embodiment, real
time queue 156 functions using a distributed, publish-subscribe
messaging system such as Kafka. In an embodiment, a Kafka agent
collects the images, videos, and time series data, from sources at
a desired frequency and these are then processed using various
OpenCV and custom image processing libraries at runtime.
[0297] SDEP 118 may be used via a hardware operating system of a
user device (for example, HMD), and/or by content developers. In
one example, both hardware and content developers may use the SDEP.
In this example, data may be collected about how the user is
interfacing with the content presented, what aspects of the content
they are most engaged with and how engaged they are. Furthermore,
engagement may be increased based on what is known of that user
and/or similar users within the same demographic. The content may
present in a way to conform to the hardware capabilities in a
manner to optimize the experience from an ergonomic standpoint.
[0298] In embodiments, SDEP 118 may further include a module 120
for backend analytics that feeds another API 122. API 122 may, in
turn, interface with user 102, providing modified media to user
102.
[0299] FIG. 2 is a block diagram illustrating processing of a
sensor data stream before it reaches a query processor, in
accordance with an embodiment of the present specification. In an
embodiment, FIG. 2 illustrates a lambda architecture 200 for a
sensor data stream received by a SDEP. Data processing architecture
200 may be designed to handle large quantities of data by parallel
processing of data stream and batch. In an embodiment, a sensor
data stream 202 comprising sensor data collected from users in real
time is provided to a real time layer 204. Real time layer 204 may
receive and process online data through a real time processor 214.
Data collected in batches may be provided to a batch layer 206.
Batch layer 206 comprises a master data set 222 to receive and
utilize for processing time stamped events that are appended to
existing events. Batch layer 206 may precompute results using a
distributed processing system involving a batch processor 216 that
can handle very large quantities of data. Batch layer 206 may be
aimed at providing accurate data by being able to process all
available sensor data, to generate batch views 218. A bulk uploader
220 may upload output to be stored in a database 210, with updates
completely replacing existing precomputed batch views. Processed
data from both layers may be uploaded to respective databases 208
and 210 for real time serving and batch serving. Data from
databases 208 and 210 may subsequently be accessed through a query
processor 212, which may be a part of a serving layer. Query
processor 212 may respond to ad-hoc queries by returning
precomputed views or building views from the processed data. In
embodiments, real-time layer 204, batch layer 206, and serving
layer may be utilized independently.
Data Acquisition
[0300] Events may be coded within the stream of data, coming
potentially from the app, the user and environmental sensors, and
may bear timestamps indicating when things happen. Anything with an
unambiguous time of occurrence may qualify as an "event". Most
events of interest may be discrete in time, with time stamps
indicating either the start or the end of some state. As an
exception, electrophysiological data may be recorded continuously
and generally analyzed by averaging segments of data synchronized
in time with other events or by some other analysis.
[0301] In an embodiment, data collected from interactions with user
102 is broadly classified as afferent data and efferent data,
corresponding to afferent events and efferent events. In the
peripheral nervous system, an afferent nerve fiber is the nerve
fiber (axon) of an afferent neuron (sensory neuron). It is a long
process (projection) extending far from the nerve cell body that
carries nerve impulses from sensory receptors or sense organs
toward the central nervous system. The opposite direction of neural
activity is efferent conduction. Conversely, an efferent nerve
fiber is the nerve fiber (axon) of an efferent neuron (motor
neuron). It is a long process (projection) extending far from the
nerve cell body that carries nerve impulses away from the central
nervous system toward the peripheral effector organs (mainly
muscles and glands).
[0302] A "stimulus" may be classified as one or more events,
typically afferent, forming a discrete occurrence in the physical
world to which a user may respond. A stimulus event may or may not
elicit a response from the user and in fact may not even be
consciously perceived or sensed at all; thus, if an event occurred,
it is made available for analysis. Stimulus event classes may
include "Application Specific Events" and "General and/or Derived
Stimulus Events".
[0303] Application Specific Events may include the many stimulus
event classes that may be specific to the sights, sounds, and other
sensory effects of a particular application. All of the art assets
are potential visual stimuli, and all of the sound assets are
potential auditory stimuli. There may be other forms of input
including, but not limited to gustatory, olfactory, tactile, along
with physiologic inputs--heart rate, pulse ox, basal body
temperature, along with positional data--accelerometer,
visual-motor--limb movement, gyroscope--head movements/body
movement--direction, force, and timing. The sudden or gradual
appearance or disappearance, motion onset or offset, playing or
pausing or other change in state of these elements will determine
their specific timestamp. Defining these stimulus event classes may
require an app developer to collaborate with the SDE, and may
include specific development of image/audio processing and analysis
code.
[0304] General and/or Derived Stimulus Events are those stimulus
events that may be generic across all applications. These may
include those afferent events derived from video (e.g. head mounted
camera) or audio data recorded of the scene and not coming directly
from the app (which itself will provide a more accurate record of
those events). Device specific, but not app specific, events may
also be classified. Likewise calibration and other activities
performed for all apps may be considered general (though perhaps
still able to be categorized by the app about to be used).
[0305] Some stimulus events may not be apparent until after a large
volume of data is collected and analyzed. Trends may be detected
and investigated where new stimulus event classes are created to
explain patterns of responding among users. Additionally,
descriptive and predictive analysis may be performed in order to
facilitate real-time exchange of stimuli/content depending on the
trends/patterns so as to personalize user-experience.
[0306] A "response" may be classified as one or more events,
typically efferent, forming a discrete action or pattern of actions
by the user, potentially in response to a perceived stimulus (real
or imagined). Responses may further include any changes in
physiological state as measured by electrophysiological and/or
autonomic monitoring sensors. Responses may not necessarily be
conscious or voluntary, though they will be identified as
conscious/unconscious and voluntary/involuntary whenever possible.
Response events classes may include discrete responses, time-locked
mean responses, time derivative responses, and/or derived response
events.
[0307] "Discrete Responses" represent the most common response
events associated with volitional user behavior and are discrete in
time with a clear beginning and end (usually lasting on the order
of seconds or milliseconds). These include, among others, mouse or
touch screen inputs, vocalizations, saccadic and pursuit eye
movements, eye blinks (voluntary or not), head or other body part
movement and electrophysiologically detected muscle movements.
[0308] Due to the noisy nature of some data recording, notably
electrophysiological recording, it is difficult to examine
responses to individual stimulus events. A Time-Locked Mean
Response refers to the pattern of responding to a particular
stimulus event, which may be extracted from numerous stimulus
response events by averaging. Data for a length of time (usually on
the order of seconds) immediately following each presentation of a
particular stimulus is put aside and then averaged over many
"trials" so that the noise in the data (presumably random in
nature) cancels itself out leaving a mean response whose
characteristics may be measured.
[0309] Time Derivative Responses reflect that some responses,
particularly autonomic responses, change slowly over time;
Sometimes too slowly to associate with discrete stimulus events.
However the average value, velocity of change or acceleration of
velocity (and other derived measures) within certain periods of
time may be correlated with other measured states (afferent or
efferent).
[0310] As with stimulus events, some response events may not be
apparent before data collection but instead reveal themselves over
time. Whether through human or machine guided analysis, some
characteristic responses may emerge in the data, hence may be
termed Inferred Response Events.
[0311] Whenever possible, responses will be paired with the stimuli
which (may have) elicited them. Some applications may make explicit
in the data stream how stimuli and responses are paired (as would
be the case in psychophysical experimentation). For the general
case, stimulus event classes will be given a set period of time,
immediately following presentation, during which a response is
reasonably likely to be made. Any responses that occur in this time
frame may be paired with the stimulus. If no responses occur then
it will be assumed the user did not respond to that stimulus event.
Likewise response events will be given a set period of time,
immediately preceding the action, during which a stimulus is likely
to have caused it. Windows of time both after stimuli and before
responses may be examined in order to aid in the discovery of new
stimulus and response event classes not previously envisioned.
[0312] Stimulus and Response Event Classes may be defined and
differentiated by their features (parameters, values, categories,
etc.). Some features of an event class may be used to establish
groups or categories within the data. Some features may (also) be
used to calculate various metrics. Features may be numeric in
nature, holding a specific value unique to the event class or the
individual instance of an event. Features may be categorical,
holding a named identity either for grouping or potentially being
converted later into a numerical representation, depending on the
analysis.
[0313] The features of stimulus events may primarily constitute a
physical description of the stimulus. Some of these features may
define the event class of the stimulus, and others may describe a
specific occurrence of a stimulus (e.g. the timestamp). The named
identity of a stimulus (e.g. sprite file name) and state
information (e.g. orientation or pose) are stimulus features. The
pixel composition of an image or waveform of a sound can be used to
generate myriad different descriptive features of a stimulus. Some
stimulus features may require discovery through data analysis, just
as some stimulus event classes themselves may emerge from
analysis.
[0314] Response features may generally include the type or category
of response made, positional information (e.g. where the mouse
click occurred or where a saccade originated/landed, a touch, a
gaze, a fixation, turn of head, turn of body, direction and
velocity of head, or body/limb movement) and timing information.
Some derived features may come from examining the stimulus to which
a response is made; for example: whether the response was "correct"
or "incorrect".
[0315] FIG. 3 illustrates an overview 300 of sources of digital
data. In embodiments, afferent data 304 may be collected from
sources that provide visual information 307, auditory information
308, spatial information 310, or other environmentally measured
states including and not limited to temperature, pressure, and
humidity. Sources of afferent data 304 may include events that are
meant to be perceived by a user 302. User 302 may be a user
interfacing with a VR/AR/MxR system in accordance with various
embodiments of the present specification.
[0316] Afferent and efferent data may be collected for a plurality
of people and related to demographic data that correspond to the
profiles for each of the plurality of people, wherein the
demographic data includes at least the sex and the age of each of
the plurality of people. Once such a database is created, visual
content, electronic advertisements, and other personalized services
can be created that are targeted to a group of people having at
least one particular demographic attribute by causing the media
content of that service to have a greater impact on the
retino-geniculo-cortical pathway of the targeted group.
Afferent Data
[0317] Afferent (stimulus) events may be anything happening on a
display provided to user 302 in the VE, events coming from speakers
or head/earphones, or haptic inputs generated by an app. Data may
also be collected by environment sensors including and not limited
to head-mounted cameras and microphones, intended to keep a record
of things that may have been seen, heard, or felt by user 302 but
not generated by the app itself. Afferent data 304 may be a form of
stimulus, which may be broken down into raw components (features or
feature sets) that are used to build analytic metrics.
[0318] In embodiments, an afferent (stimulus) event is paired with
an efferent (response) event. In the pairing, each of the component
stimulus features may be paired with each of the component response
features for analysis. In some cases pairs of stimulus features or
pairs of response features may also be examined for correlations or
dependencies. Stimulus/response feature pairs are at the root of
most of the conceivable metrics to be generated. All analyses may
be broken down by these feature pairs before being grouped and
filtered according to various other of the event features
available. In embodiments, for all data sources including afferent
304 and efferent 306 data sources, timing information is required
to correlate inputs to, and outputs from, user's 302 sensory
system. The correlations may be utilized to identify characteristic
metrics or psychophysical metrics for the user. For example, if
VR/AR/MxR system 104 records that an object was drawn on a screen
at time tS (stimulus), and also that a user pressed a particular
key at a time tR (response), the time it took the user to respond
to the stimulus may be derived by subtracting tR-tS. In alternate
embodiments, the user may press a key, or make a gesture, or
interact with the AR/VR/MxR environment through a touch or a
gesture. This example correlates afferent data 304 and efferent
data 306.
[0319] An example that correlates two types of afferent data 304
may be if a gaze tracker indicates that the gaze position of a user
changed smoothly over a given period of time indicating that the
user was tracking a moving object. However, if a head tracker also
indicates smooth motion in the opposite direction, at the same
time, it might also indicate that the user was tracking a
stationary object while moving their head.
[0320] Another example that correlates two types of afferent data
304 may be if visual object appears at time t1, and a sound file is
played at time t2. If the difference between t1 and t2 is small (or
none), they may be perceived as coming from the same source. If the
difference is large, they may be attributed to different
sources.
[0321] The data taken from accumulated response events may be used
to describe patterns of behavior. Patterns of responding,
independent of what stimuli may have elicited them, can be used to
categorize various behavioral or physiological states of the user.
Grouping responses by the stimuli that elicited them can provide
measures of perceptual function. In some cases analyses of stimulus
events may provide useful information about the apps themselves, or
in what experiences users choose to engage. The analysis may
include following parameters: unique events, descriptive
statistics, and/or psychometric functions.
[0322] Unique Events represent instances where raw data may be of
interest. Some uncommon stimulus or response events may not provide
opportunities for averaging, but instead are of interest because of
their rarity. Some events may trigger the end of a session or time
period of interest (e.g. the user fails a task and must start over)
or signal the beginning of some phase of interaction.
[0323] Descriptive Statistics provide summarized metrics. Thus, if
multiple occurrences of an event or stimulus/response event or
feature pairing may be grouped by some commonality, measures of
central tendency (e.g. mean) and variability (e.g. standard
deviation) may be estimated. These summarized metrics may enable a
more nuanced and succinct description of behavior over raw data.
Some minimal level of data accumulation may be required to be
reasonably accurate.
[0324] Psychometric Functions may form the basis of measures of
perceptual sensitivity and ability. Whenever a particular class of
stimulus event is shown repeatedly with at least one feature
varying among presentations there is an opportunity to map users'
pattern of responses against that stimulus feature (assuming
responding varies as well). For example, if the size (stimulus
feature) of a particular object in a game varies, and sometimes the
user finds it and sometimes they don't (response feature), then the
probability of the user finding that object may be plotted as a
function of its size. This may be done for multiple
stimulus/response feature pairs for a single stimulus/response
event pairing or for many different stimulus/response event pairs
that happen to have the same feature pairing (e.g. size/detection).
When a response feature (detection, discrimination, preference,
etc.) plotted against a stimulus feature (size, contrast, duration,
velocity, etc.) is available with mean responses for multiple
stimulus levels, a function to that data (e.g. detection vs. size)
may be fitted. The variables that describe that function can
themselves be descriptive of behavior. Thresholds may be defined
where on one side is failure and the other side success, or on one
side choice A and the other side choice B, among others.
Visual Data
[0325] Referring back to FIG. 3, in an embodiment, for an
application, visual information data 307 from physical display(s)
and the visual environment is in the form of still image files
and/or video files captured by one or more cameras. In an
embodiment, data is in the form of instructions for drawing a
particular stimulus or scene (far less data volume required, some
additional time in rendering required).
[0326] FIG. 4A is a block diagram 400 illustrating characteristic
metrics for visual data, in accordance with an embodiment of the
present specification. Characteristic metrics may characterize a
user session and may be time-averaged. Referring to FIG. 4A, scope
402 may refer to whether the visual data is for an entire scene
(the whole visual display or the whole image from a
user-head-mounted camera). Physical attributes 404 may refer to
objective measures of the scene or objects within it. They may
include location relative to the retina, head and body, an
orthogonal 3-D chromoluminance; and contrast vs. spatial frequency
vs. orientation. Categorical attributes 406 may be named properties
of the image, which may include named identity of an object, and/or
the group identity.
[0327] Visual stimuli may generally be taken in as digital, true
color images (24-bit) either generated by an application (image
data provided by app directly) or taken from recorded video (e.g.
from a head mounted camera). Images and video may be compressed in
a lossy fashion; where weighted averaging of data may account for
lossy compression, but otherwise image processing would proceed the
same regardless. A developer may choose to provide information
about the presentation of a stimulus which may allow for the
skipping of some image processing steps and/or allow for post hoc
rendering of scenes for analysis. Visual stimuli may include, but
are not limited to the following components: objects, size,
chromatic distance, luminance contrast, chromatic contrast, spatial
feature extraction, saliency maps and/or temporal dynamics.
[0328] Objects (stimuli) may be identified in an image (or video
frame) either by information from the application itself or found
via machine learning (Haar-like features classification cascade, or
similar). Once identified, the pixels belonging to the object
itself (or within a bounding area corresponding to a known size
centered on the object) will be tagged as the "object". The pixels
in an annulus around the object (necessarily within the boundaries
of the image/scene itself) with the same width/height of the object
(i.e. an area 3.times. the object width and 3.times. the object
height, excluding the central area containing the object) will be
tagged as the "surround". If another image exists of the same exact
area of the surround, but without the object present (thus showing
what is "behind" the object), that entire area without the object
may be tagged as the "background". Metrics may be calculated
relative to the surround and also relative to the background when
possible. Object segments or parts may be used to break objects
down into other objects and may also be used for identity or
category variables. Objects need not correspond to physical objects
and may include regions or boundaries within a scene or comprise a
single image feature (e.g. an edge).
[0329] Object size is an important feature for determining acuity,
or from known acuity predicting whether a user will detect or
correctly identify an object. The object size may be defined as a
width and height, either based on the longest horizontal and
vertical distance between pixel locations in the object or as the
width and height of a rectangular bounding box defining the
object's location. Smaller features that may be necessary to
successfully detect or discriminate the object from others may be
located within the object. It may be assumed that the smallest
feature in an object is 10% of the smaller of its two dimensions
(width and height). It may also be assumed the smallest feature
size is proportional to the size of a pixel on the display for a
given viewing distance. The smallest feature size may be more
explicitly found either by analysis of a Fourier transform of the
image or examining key features from a Harr-like feature
classification cascade (or similar machine learning based object
detection) trained on the object.
[0330] The first of two breakdowns by color, chromatic distance is
a measure of the color difference between the object and its
surround/background, independent of any luminance differences. Red,
green and blue values may be independently averaged across all
pixels of the object and all pixels of the surround/background.
These mean RGB values will be converted into CIE Tristimulus values
(X, Y and Z) and then into CIE chromaticity (x and y) using either
standard conversion constants or constants specific to the display
used (when available). In an embodiment, conversion constants for
conversion from RGB to XYZ, taken from Open CV function `cvtColor`
based on standard primary chromaticities, a white point at D65, and
a maximum, white luminance of 1, is:
[ X Y Z ] .rarw. [ 0.412453 0.357580 0.180423 0.212671 0.715160
0.072169 0.019334 0.119193 0.950227 ] [ R G B ] ##EQU00001##
[0331] In this embodiment, RGB is converted to xy using the
following:
x = X X + Y + Z ##EQU00002## y = Y X + Y + Z ##EQU00002.2##
[0332] The absolute distance between the chromaticity of the object
and that of the surround/background will be logged as the chromatic
distance. Next, a line will be drawn from the midpoint between the
two chromaticities and each of the three copunctal points for L, M
and S cones. These lines are confusion lines for L, M and S cone
deficiencies, along which someone missing one of those cone types
would be unable to discriminate chromaticity. The component of the
line between object and surround/background chromaticity parallel
to each of these three confusion lines will be logged as the L, M
and S specific chromatic distances.
[0333] FIG. 4B provides a graphical presentation of color pair
confusion components, in accordance with an embodiment of the
present specification. Referring to the figure, a line 1308 is
drawn between the two chromaticities given. As seen in the figure,
three large dots--red 410, green 412, and blue 414 are copunctal
points for L, M and S cones, respectively. From each dot extends a
similarly color-coded, dashed line. Bold line 416 has a mid-point
where the three, dashed lines intersect. Based on the angle between
line 416 and the lines drawn from the midpoint to each of the
copunctal points, the parallel component of that line for each of
the three resulting confusion lines is determined. In embodiments,
the closer to the parallel line between the colors is to a
particular confusion line, the more difficult it will be for
someone with a deficiency of the corresponding cone to
discriminate. The component length divided by the total length (the
quotient will be in the interval [0,1]) would be roughly the
probability of the colors being confused.
[0334] FIG. 4C shows a graph illustrating how luminance may be
found for a given chromaticity that falls on the top surface of the
display gamut projected into 3D chromoluminance space. The graph
shows a projection of a full display gamut for a computer screen
into CIE 1931 chromoluminance space. While the RGB space used to
define the color of pixels on a display can be represented by a
perfect cube, the actual physical property of luminance is somewhat
complexly derived from those values, represented by the shape seen
in FIG. 6. Luminance contrast may be defined in three ways.
Generally the context of an analysis will suggest which one of the
three to use, but all three may be computed for any object and its
surround/background. For instances where a small object is present
on a large, uniform background (e.g. for text stimuli), Weber
contrast may be computed using the CIE Tristimulus values Y
(corresponding to luminance) calculated from the mean RGB of the
object and of the surround/background. Here it is assumed that the
average luminance is roughly equal to the surround luminance. Weber
contrast can be positive or negative and is theoretically
unbounded. For object/surrounds that are periodic in nature, and
especially with gradients (e.g. a sine wave grating), Michelson
contrast may be computed from the minimum and maximum luminance
values in the stimulus. Michelson contrast will always be a value
between 0 and 1. For most cases it will be necessary to compute
contrast from all of the pixel values, instead of from a mean or
from the minimum and maximum. The RMS contrast (root mean square,
or standard deviation) can be found by taking the standard
deviation of the CIE Tristimulus value Y for all pixels. The RMS
contrast of the object is one measure. The RMS contrast of the
object relative to the RMS contrast of the surround/background is
another. Finally, the RMS contrast of the object and surround
together is yet a third measure of RMS contrast that can be
used.
[0335] Chromatic contrast may be calculated on any pair of
chromaticity values, independently, in all of the ways described
above for luminance contrast. The most useful of these will either
be the a* and b* components of CIELAB color space, or the L vs. M
and S vs. LM components of cone-opponent color space. For any pair
of dimensions, the Weber, Michelson and/or RMS contrast may be
calculated, depending on the type of stimulus being analyzed. In
addition, RMS contrast will be calculated for L, M and S cone
deficiencies. CIE chromaticity values for all pixels will be
converted into three sets of polar coordinates centered on the L, M
and S copunctal points. In an embodiment, the following equation is
used to convert Cartesian coordinates to polar coordinates, with an
option to provide center points other than [0,0]:
.theta. = tan - 1 ( y - y c x - x c ) ##EQU00003## Radius = ( y - y
c ) 2 + ( x - x c ) 2 ##EQU00003.2##
[0336] RMS contrast may be calculated based on the radius
coordinates for each conversion.
[0337] In addition to finding objects, algorithms may also identify
prominent features present in a scene, or within objects, that may
capture attention, be useful for a task the user is performing or
otherwise be of interest as independent variables to correlate with
behavior. Edges, those inside identified objects and otherwise, may
be targets for fixations or other responses and their positions may
be responsible for observed positional errors in responding and be
worth correlating with correct and incorrect responses. Regions,
contours, surfaces, reflections, shadows and many other features
may be extracted from this data.
[0338] Saliency Maps refer to data that are collected from user
interactions to inform models of saliency for future analysis of
stimulus scenes. Edges, contours and other image features may be
used to measure saliency and predict where user responses,
including eye gaze fixations, may fall. Multiple algorithms may be
applied to highlight different types of features in a scene.
[0339] Temporal Dynamics are also important because features of a
visual display or environment, and any objects and object features
thereof, may change over time. It will be important to log the time
of any change, notably: appearance/disappearance or change in
brightness/contrast of objects or features, motion start/stop or
abrupt position change (in x, y, z planes), velocity change (or
acceleration or any higher order time derivative of position) and
any and all changes in state or identity of objects or features.
Changes in chromaticity or luminance of objects or features should
also be logged. Secondary changes in appearance resulting from
changes in orientation or pose of an object or the object's
position relative to the surround/background may also be
logged.
Auditory Data
[0340] Referring back to FIG. 3, auditory information 308 may be
received from audio output such as speakers, and the environment by
using microphones. In an embodiment auditory information 308 may be
available in raw, waveform files or in more descriptive terms (e.g.
this audio file played at this time).
[0341] FIG. 5 illustrates characteristic metrics 500 for auditory
information 308 (shown in FIG. 3), in accordance with an embodiment
of the present specification. Referring to FIG. 5, a positional
reference 502 may be noted to identify the location of sounds. The
position, relative to a user's head, of an object or speaker in the
environment will vary as they move their head. The position of a
virtual source perceived through headphones may not change as the
user turns their head (unless head tracking and sound processing
work together to mimic those changes).
[0342] The physical attributes 504 of sound may include their
location (derived from intensity, timing and frequency differences
between the ears), frequency composition (derived from the
waveform), and the composition of different sources. Categorical
attributes 506 may be named properties of the image, which may
include named identity of an object, and/or the group identity and
may follow a similar description as for visual stimuli.
[0343] Auditory (Sound) stimuli may generally be taken in as
digital waveforms (with varying spatial and temporal resolution or
bitrate and possible compression) either generated by an
application or taken from recorded audio (e.g. head mounted
microphones, preferably binaural). Compression parameters, if any,
may be recorded. Developers may choose to provide information about
the presentation of a stimulus which may allow for the skipping of
some processing. Visual information may be used to model the audio
environment so that sound reflections or obscurations can be taken
into account. Audio stimuli may be broken down to include the
following parameters: Fourier Decomposition, Head-Centric Position,
Sound Environment, and/or Objects.
[0344] Fourier Decomposition may be performed to break sound waves
into components based on sound objects. Time-domain waveform data
may be transformed into the frequency domain such that the
amplitude and phase of different audio frequencies over time may be
analyzed. This will allow the utilization of sound parameters (e.g.
frequency, amplitude, wavelength, shape and envelope, timbre,
phase, etc.) as independent variables.
[0345] Head-Centric Position or head tracking data may be necessary
for environmental sounds. The position of sound sources relative to
a user's ears may be derived, and whenever possible the sound
waveforms as they exist at the user's ears may be recorded (ideally
from binaural, head-mounted microphones). Binaural headset sound
sources (e.g. headphones/earphones) may obviate the necessity for
this.
[0346] Similarly, tracking data for body and/or limbs may be
necessary for environmental sounds. The position of sound sources
relative to a user's body and limbs may be derived. This data may
be related to head tracking data identified for environmental
sounds. The data may enable understanding of how body and limbs
react with the movement of head.
[0347] Sound Environment is not critical in most common use cases
(e.g. sound is coming from headset or from directly in front of the
user), but will be important for considering environmental sounds
to which users are anticipated to respond. Objects in the
environment that reflect and/or block sound (commonly frequency
specific) may change the apparent source location and other
frequency dependent features of a sound. It may be useful to
roughly characterize the physical environment as it affects the
propagation of sound from its sources to the user.
[0348] Audio objects may be detected and segmented out using the
same type of machine learning algorithms (Haar-like feature
classification cascades or similar) that are used for detecting and
segmenting out visual objects. This should be used whenever
possible to obtain accurate audio event details and may also be
useful for extracting audio parameters used by the auditory system
for localization.
[0349] Most analysis may revolve around visual and (to a lesser
extent) auditory stimuli occurring discretely in time. Other
stimuli may include those sensed in other modalities (e.g. touch,
taste, smell, etc.) or general environmental state variables that
define the context of user interactions with applications (e.g.
ambient lighting and background audio).
[0350] Examples of other stimuli may include the following:
[0351] Haptic Stimuli, where developers may choose to use haptic
feedback mechanisms and, if they so choose, provide details about
the nature and timing of those events. Haptic stimulation may also
be derived via direct recording (unlikely) or derived from other
sources (e.g. hearing the buzz of a physical vibration via
microphone).
[0352] Other Modality Stimuli, where developers may be able to
initiate smell, taste, temperature, pressure, pain or other
sensation at discrete times creating stimulus events not already
discussed. As with haptic stimuli, any record of such stimulation
would best come directly from the application itself via function
calls.
[0353] Environmental Stimuli, or stimuli that do not occur
discretely in time and are either of constant state or steadily
repeating, may provide important context for the discrete stimuli
and responses that occur in a session. Ambient light levels may
affect contrast sensitivity, baseline pupil size, circadian
patterns and other physiological states of the user. Ambient sounds
may affect auditory sensitivity, may mask certain auditory stimuli
and also affect physiological and other states of the user. The
time of day may also be an important variable for categorization
and correlation. Though perhaps not readily recorded by an
application, user input could provide information about sleep
patterns, diet and other physiologically relevant state variables
as well as categorical descriptions of the space including
temperature, pressure, humidity (which may also be derived from
location and other services).
Spatial Information
[0354] Referring back to FIG. 3, in an embodiment, spatial
information 310 may consist of descriptions of the setting around
user 302. This may include spatial orientation of user 302 and
physical space around user 302.
[0355] In an embodiment, setting is an environment in which
interactions between user 302 and the app take place. Setting data
may refer to things that are mostly static during a session
including the physical setting, ambient light levels, room
temperature, and other types of setting information. In
embodiments, spatial information 310 is a part of the setting data.
Setting data may generally be constant throughout a session with
user 302 and therefore may not be broken down into "events" as
described earlier. Setting data may pertain to a physical setting
or may relate to personal details of user 302.
[0356] Physical setting data may correspond to any description of
the physical space, such as and not limited to a room or an outdoor
setting, and may be useful to categorize or filter data. In an
exemplary embodiment, physical setting data such as the ambient
lighting present, may directly affect measures of pupil size,
contrast sensitivity and others. Lighting may affect quality of
video eye tracking, as well as any afferent events derived from
video recording of a scene. Similarly, environmental sounds may
affect users' sensitivity as well as the ability to characterize
afferent events derived from audio recording.
[0357] Personal details of a user may pertain to any personal,
largely demographic, data about the user or information about their
present physiological or perceptual state (those that will remain
largely unchanged throughout the session). This data may also be
useful for categorization and filtering. Personal details may
include any information regarding optics of the user's eyes (for
example, those derived from knowledge of the user's eyeglass or
contact prescription). Personal details may also include diet
related information, such as recent meal history. Further, time,
duration, and quality of most recent sleep period, any psychoactive
substances recently taken in (e.g. caffeine) and recent exercise or
other physical activity may all impact overall data.
Efferent Data
Eye Tracking
[0358] Video eye tracking and electrooculography provide
information about eye movements, gaze direction, blinking and pupil
size. Derived from these are measures of vergence, fatigue,
arousal, aversion and information about visual search behavior.
Information pertaining to eye movements include initiation,
duration, and types of pro-saccadic movements (toward targets),
anti-saccadic movements (toward un-intended target), the amount of
anti-saccadic error (time and direction from intended to unintended
target), smooth pursuit, gaze with fixation duration, pupil changes
during movement and during fixation, frequency and velocity of
blink rate, as well as frequency and velocity of eye movements.
Information pertaining to vergence may include both convergence and
divergence--in terms of initiation and duration. Combined with
information about the visual scene, measures of accuracy, search
time and efficiency (e.g. minimizing number of saccades in search)
can be made.
[0359] Autonomic measures derived from video eye tracking data may
be used to guide stimulus selection towards those that increase or
decrease arousal and/or aversion. Summary information about gaze
position may indicate interest or engagement and likewise be used
to guide stimulus selection.
[0360] Referring to FIG. 3, efferent data sources 306 may include
video eye tracking data 312. Data 312 may measure gaze direction,
pupil size, blinks, and any other data pertaining to user's 302
eyes that may be measured using a Video Eye Tracker (VET) or an
electrooculogram. This is also illustrated in FIG. 6, which shows
characteristic metrics 600 for eye tracking, in accordance with an
exemplary embodiment of the present specification. Video eye
tracking 602 generally involves recording images of a user's eye(s)
and using image processing to identify the pupil and specific
reflections of known light sources (typically infrared) from which
may be derived measures of pupil size and gaze direction. The
angular resolution (of eye gaze direction) and temporal resolution
(frames per second) may limit the availability of some measures.
Some measures may be recorded as discrete events, and others
recorded over time for analysis of trends and statistics over
epochs of time.
Gaze Direction
[0361] Software, typically provided with the eye tracking hardware,
may provide calibrated estimates of gaze direction in coordinates
tied to the display used for calibration. It may be
possible/necessary to perform some of this conversion separately.
For head mounted units with external view cameras the gaze position
may be in head centric coordinates or in coordinates relative to
specific objects (perhaps provided reference objects) in the
environment. It is assumed that gaze direction will be provided at
some rate in samples per second. Most of the following metrics will
be derived from this stream of gaze direction data: saccade,
pursuit, vergence, patterns, and/or microsaccades.
[0362] Saccade: Prolonged periods of relatively fixed gaze
direction separated by rapid changes in gaze (over a matter of
milliseconds) may be logged as "fixations" and the jumps in between
as "saccades". Fixations will be noted for position, start and end
time and duration. In some cases they may also be rated for
stability (variability of gaze direction during fixation). Saccades
will be noted for their direction (angle), speed and distance. It
is worth noting, and it will generally be assumed, that there is a
period of cortical suppression during saccades when visual
information is not (fully) processed. This saccadic suppression may
be exploited by developers to alter displays without creating a
percept of motion, appearance or disappearance among display
elements.
[0363] Pursuit: Pursuit eye movements may be characterized by
smooth changes in gaze direction, slower than typical saccades (and
without cortical suppression of visual processing). These smooth
eye movements generally occur when the eyes are pursuing/tracking
an object moving relative to head facing direction, a stationary
object while the head moves or moving objects while the head also
moves. Body or reference frame motion can also generate pursuit eye
movements to track objects. Pursuit can occur in the absence of a
visual stimulus based on the anticipated position of an invisible
or obscured target.
[0364] Vergence: This measure may require relatively fine
resolution gaze direction data for both eyes simultaneously so that
the difference in gaze direction between eyes can be used to
determine a depth coordinate for gaze. Vergence is in relation to
the distance of the object in terms of the user to measure objects
between the near point of convergence and towards infinity in the
distance--all of which may be modelled based off the measurements
of vergence between convergence and divergence.
[0365] Patterns: Repeated patterns of eye movements, which may be
derived from machine learning analysis of eye gaze direction data,
may be used to characterize response events, states of user
interaction or to measure effects of adaptation, training or
learning. Notable are patterns during visual search for targets or
free viewing of scenes towards the completion of a task (e.g.
learning of scene details for later recognition in a memory task).
Eye movement patterns may also be used to generate models for
creating saliency maps of scenes, guiding image processing.
[0366] Microsaccades: With relatively sensitive direction and time
resolution it may be possible to measure and characterize
microsaccadic activity. Microsaccades are generally present during
fixation, and are of particular interest during rigid or prolonged
fixation. Feedback into a display system may allow for creating
images that remain static on the retina resulting in Troxler
fading. Microsaccades are not subject to conscious control or
awareness.
[0367] Sample questions concerning eye tracking metrics that may be
answered over a period of time may include: where are users looking
the most (potentially in response to repeating events), how fast
and accurate are saccadic eye movements, how rapidly are users
finding targets, are users correctly identifying targets, how
accurate is pursuit/tracking, are there preferences for certain
areas/stimuli.
[0368] During free viewing or search, fixations (relatively stable
eye gaze direction) between saccades typically last on the order of
200-300 milliseconds. Saccades have a rapidly accelerating
velocity, up to as high as 500 degrees per second, ending with a
rapid deceleration. Pursuit eye movements occur in order to
steadily fixate on a moving object, either from object motion or
head motion relative to the object or both. Vergence eye movements
are used to bring the eyes together to focus on near objects.
Vestibular eye movements are compensatory eye movements derived
from head and/or body movement.
[0369] Reference is made to WO2015003097A1 entitled "A Non-Invasive
Method for Assessing and Monitoring Brain", which has at least
partial common inventorship with the present specification. In an
example, a pro-saccade eye tracking test is performed. The
pro-saccade test measures the amount of time required for an
individual to shift his or her gaze from a stationary object
towards a flashed target. The pro-saccade eye tracking test may be
conducted as described in The Antisaccade: A Review of Basic
Research and Clinical Studies, by S. Everling and B. Fischer,
Neuropsychologia Volume 36, Issue 9, 1 Sep. 1998, pages 885-899
("Everling"), for example.
[0370] The pro-saccade test may be performed while presenting the
individual with a standardized set of visual stimuli. In some
embodiments, the pro-saccade test may be conducted multiple times
with the same or different stimuli to obtain an average result. The
results of the pro-saccade test may comprise, for example, the
pro-saccade reaction time. The pro-saccade reaction time is the
latency of initiation of a voluntary saccade, with normal values
falling between roughly 200-250 ms. Pro-saccade reaction times may
be further sub-grouped into: Express Pro-Saccades: 80-134 ms; Fast
regular: 135-175 ms; Slow regular: 180-399 ms; and Late: (400-699
ms).
[0371] Similarly, an anti-saccade eye tracking test may be
performed. The anti-saccade test measures the amount of time
required for an individual to shift his or her gaze from a
stationary object away from a flashed target, towards a desired
focus point. The anti-saccade eye tracking test can be conducted as
described in Everling, for example. In some examples, the
anti-saccade test may also measure an error time and/or error
distance; that is, the amount of time or distance in which the eye
moves in the wrong direction (towards the flashed target). The
anti-saccade test may be performed using the standardized set of
visual stimuli. The results of the anti-saccade test may comprise,
for example, mean reaction times as described above for the
pro-saccade test, with typical mean reaction times falling into the
range of roughly 190 to 270 ms. Other results may include initial
direction of eye motion, final eye resting position, time to final
resting position, initial fovea distance (i.e., how far the fovea
moves in the direction of the flashed target), final fovea resting
position, and final fovea distance (i.e., how far the fovea moves
in the direction of the desired focus point).
[0372] Also, a smooth pursuit test may be performed. The smooth
pursuit test evaluates an individual's ability to smoothly track
moving visual stimuli. The smooth pursuit test can be conducted by
asking the individual to visually follow a target as it moves
across the screen. The smooth pursuit test may be performed using
the standardized set of visual stimuli, and may be conducted
multiple times with the same or different stimuli to obtain an
average result. In some embodiments, the smooth pursuit test may
include tests based on the use of fade-in, fade-out visual stimuli,
in which the target fades in and fades out as the individual is
tracking the target. Data gathered during the smooth pursuit test
may comprise, for example, an initial response latency and a number
of samples that capture the fovea position along the direction of
motion during target tracking. Each sampled fovea position may be
compared to the position of the center of the target at the same
time to generate an error value for each sample.
[0373] For more sensitive tracking hardware, it may also be
possible to measure nystagmus (constant tremor of the eyes), drifts
(due to imperfect control) and microsaccades (corrections for
drift). These will also contribute noise to gross measurements of
gaze position; as a result fixations are often characterized by the
mean position over a span of relatively stable gaze position
measures. Alternatively, a threshold of gaze velocity
(degrees/second) can be set, below which any small movements are
considered to be within a fixation.
[0374] Saccades require time to plan and execute, and a delay, or
latency, of at least 150 ms is typical after, for example, the
onset of a visual stimulus eliciting the saccade. Much can be said
about the latency before a saccade and various contexts that may
lengthen or shorten them. The more accurate information we have
regarding the relative timing of eye movements and events occurring
in the visual scene, the more we can say about the effect of
stimulus parameters on saccades.
[0375] Although usually correlated, shifts in attention and eye
gaze do not necessarily have to happen together. In some contexts
it may be efficient for the user to direct attention to a point in
their visual periphery, for example to monitor one location while
observing another. These scenarios may be useful for generating
measures related to Field of View and Multi-Tracking.
[0376] It is possible to use image processing techniques to
highlight regions within a scene of greater saliency based on
models of the visual system. For example areas of greater
high-spatial-frequency contrast (i.e. edges and lines) tend to
capture attention and fixations. It is possible within a specific
context to use eye gaze direction to develop custom saliency maps
based on the information available in the visual scene combined
with whatever tasks in which an observer may be engaged. This tool
can be used to highlight areas of interest or greater
engagement.
Pupil Size
[0377] Pupil size may be measured as part of the image processing
necessary to derive gaze direction. Pupil size may generally change
in response to light levels and also in response to certain
stimulus events via autonomic process. Pupil responses are not
subject to conscious control or awareness (except secondarily in
the case of extreme illumination changes). Sample questions
concerning eye tracking metrics that may be answered over a period
of time may include: how are the pupils responding to different
stimuli, how are the pupils behaving over time.
[0378] Pupil diameter generally falls between 2 and 8 mm at the
extremes in light and dark, respectively. The pupil dilates and
constricts in response to various internal and external stimuli.
Due to differences in baseline pupil diameter, both among observers
and due to ambient lighting and physiological state, pupil
responses may generally be measured as proportions of change from
baseline. For example, the baseline pupil diameter might be the
diameter at the moment of an external stimulus event (image
appears), and the response is measured by the extent to which the
pupil dilates or constricts during the 1 second after the stimulus
event. Eye color may affect the extent of constriction, and age may
also be a factor.
[0379] In addition to responding to light, accommodation for
distance and other spatial and motion cues, pupil diameter will
often be modulated by cognitive load, certain imagery and reading.
Pupil diameter may be modulated during or at the termination visual
search. Proportional changes can range from a few to tens of
percentage points.
[0380] Thresholds for determining computationally if a response has
been made will vary depending on the context and on the sensitivity
of the hardware used. Variations in ambient lighting and/or the
mean luminance of displays will also have a large influence on
pupil diameter and proportional changes, so thresholds will need to
be adaptable and likely determined by the data itself (e.g.
threshold for dilation event itself being a percentage of the range
of pupil diameter values recorded within a session for one
user).
[0381] Reference is again made to WO2015003097A1 titled "A
Non-Invasive Method for Assessing and Monitoring Brain", which has
at least partial common inventorship with the present
specification. In an example, pupillary response is assessed.
Pupillary response is often assessed by shining a bright light into
the individual's eye and assessing the response. In field settings,
where lighting is difficult to control, pupillary response may be
assessed using a standardized set of photographs, such as the
International Affective Picture System (TAPS) standards. These
photographs have been determined to elicit predictable arousal
patterns, including pupil dilation. The pupillary response test may
be performed using a variety of stimuli, such as changes to
lighting conditions (including shining a light in the individual's
eyes), or presentation of photographs, videos, or other types of
visual data. In some embodiments, the pupillary test may be
conducted multiple times with the same or different stimuli to
obtain an average result. The pupillary response test may be
conducted by taking an initial reading of the individual's pupil
diameter, pupil height, and/or pupil width, then presenting the
individual with visual stimuli to elicit a pupillary response. The
change in pupil dilation (e.g., the change in diameter, height,
width, and/or an area calculated based on some or all of these
measurements) and the time required to dilate are measured. The
results of the pupillary response test may include, for example, a
set of dilation (mydriasis) results and a set of contraction
(miosis) results, where each set may include amplitude, velocity
(speed of dilation/constriction), pupil diameter, pupil height,
pupil width, and delay to onset of response.
Blinks
[0382] Video eye trackers, as well as less specialized video
imaging of a user's face/eye region, may detect rapid or prolonged
periods of eye closure. Precautions may be taken as loss of
acquisition may also be a cause for periods of data loss. Blink
events, conscious or reflexive, and blink rates over time related
to measures of fatigue or irritation may be recorded. Sample
questions concerning eye tracking metrics are mentioned in FIG. 6.
In embodiments, these are questions that may be answered over a
period of time and may include: are the users blinking in response
to the onset of stimuli, is the blink rate changing in response to
the stimuli, is the blink rate changing overall, does the blink
rate suggest fatigue.
[0383] Normal blinking rates among adults are around 10 blinks per
minute at rest, and generally decreases to around 3 blinks per
minute during focused attention (e.g. reading). Other properties of
blinks, for example distance/speed of eyelid movement and durations
of various stages within a blink, have been correlated with error
rates in non-visual tasks (for example, using auditory stimulus
discrimination) and other measures; whenever possible it may be
advantageous to use video recordings to analyze eyelid position in
detail (i.e. automated eyelid tracking). Blink durations longer
than 150 ms may be considered long-duration blinks.
[0384] As with most measures, proportional changes from baseline
may be more valuable than absolute measures of blink frequency or
average duration. Generally, significance can be assigned based on
statistical measures, meaning any deviation is significant if it is
larger than the general variability of the measure (for example as
estimated using a t-test).
Manual Inputs
[0385] Referring back to FIG. 3, another efferent data source 306
may be manual input 314. Which have been a traditional tool of
computer interaction and may be available in many forms. Exemplary
manual inputs 314 of interest include input identity (key pressed),
any other gesture, position coordinates (x, y, z) on a touch screen
or by a mouse, and/or (video) tracking of hand or other limb. FIG.
7 illustrates characteristic metrics 700 for manual inputs 702, in
accordance with an embodiment of the present specification.
[0386] Sample questions concerning manual input metrics that may be
answered over a period of time may include: where are the users
clicking/touching the most (potentially in response to repeating
events), how fast and accurate are the clicks/touches, how rapidly
are users finding targets, are users correctly identifying targets,
how accurate is tracking, are there preferences for certain
areas/stimuli, what kind of grasping/touching motions are the users
making, how is the hand/eye coordination, are there reflexive
actions to virtual stimuli.
[0387] Responses made with the fingers, hands and/or arms, legs, or
any other part of the body of users may generally yield timing,
position, trajectory, pressure and categorical data. These
responses may be discrete in time, however some sustained or state
variable may be drawn from manual data as well. Following analytic
response metrics may be derived from manual responses: category,
identity, timing, position, and/or trajectory.
[0388] Category: In addition to categories like click, touch, drag,
swipe and scroll there may be sub categories like double click, tap
or push, multi-finger input, etc. Any variable that differentiates
one action from another by category that is detectable by an
application may be important for differentiating responses (and
will likely be used for that purpose by developers).
[0389] Identity: Whenever multiple input modalities exist for the
same type of response event, most notably the keys on a computer
keyboard, or any other gesture that may be possible in a VR/AR/MxR
environment, the identity of the input may be recorded. This also
includes directions indicated on a direction pad, mouse buttons
clicked and, when possible, the area of a touchpad touched
(independent of cursor position), or any other gesture.
[0390] Timing: The initiation and ending time of all responses may
be recorded (e.g. a button press will log both the button-down
event and the button-up event), and from that response durations
can be derived. This timing information will be key to connecting
responses to the stimuli that elicited them and correlating events
in time.
[0391] Position: For visual interfaces, the position may be in
display coordinates. Positions may be singular for discrete events
like clicks or continuously recorded at some reasonable rate for
tracing, dragging, etc. When possible these may also be converted
to retinal coordinates (with the combination of eye gaze tracking).
By understanding position, a topography of the retina may be done,
and areas of the retina may be mapped in relationship to their
specific functions further in relationship to the brain, body,
endocrine, and autonomic systems. For gestures recorded by
video/motion capture the body-centric position will be recorded
along with the location of any cursor or other object being
controlled by the user.
[0392] Trajectory: For swipe, scroll and other dynamic gestures it
may be possible to record the trajectory of the response (i.e. the
direction and speed as a vector) in addition to any explicit
position changes that occur. This will, in fact, likely be derived
from an analysis of rapid changes in position data, unless the
device also provides event types for these actions.
Head Tracking
[0393] Head tracking measures are largely associated with virtual,
augmented, and mixed reality displays. They can provide measures of
synchrony with displayed visual environments, but also of users'
reactions to those environments. Orienting towards or away from
stimuli, compensatory movements in line or not in line with the
displayed visual environments and other motion behavior can be used
to derive similar, though less precise, measures similar to those
from eye tracking. Those derived measures associated with arousal,
fatigue and engagement can be modified as previously stated.
[0394] If head movements, particularly saccadic head movements,
prove to be a source of mismatch and discomfort for users it may be
desirable to modify displays to reduce the number of such head
movements. Keeping display elements within a region near
head-center and/or encouraging slower changes in head-facing may
reduce large head movements. With regards to individual
differences: some users will move their heads more than others for
the same scenario. It may be possible to train head movers to
reduce their movements.
[0395] Referring back to FIG. 3, head tracking data 316 may be
another form of efferent data 306 source. Head tracking data 316
may track user's 302 head orientation and physical position from
either video tracking (VET or otherwise) or position sensors
located on HMDs, headsets, or other worn devices. In addition to
tracking user's 302 head, their body may be tracked. The position
of users' 302 bodies and parts thereof may be recorded, likely from
video based motion capture or accelerometers in wearable devices.
This position data would commonly be used to encode manual response
data (coming from finger, hand or arm tracking) and/or head
orientation relative to the environment to aid in eye gaze
measurements and updating of the user's visual environment. Head
position data may also be used to model the effect of head shadow
on sounds coming from the environment. FIG. 8 illustrates
characteristic metrics 800 for head tracking, which may include
head orientation 802 and/or physical position 804, in accordance
with an embodiment of the present specification.
[0396] Sample questions concerning head tracking metrics that may
be answered over a period of time may include: where are the users
looking most (potentially in response to repeating events), how
fast and accurate are head movements, how accurate is
pursuit/tracking, is there preference for certain areas/stimuli,
are users accurately coordinating head and eye movements to direct
gaze and/or track objects, are head movements reduced due to the
hardware, are users making many adjustments to the hardware, are
users measurably fatigued by the hardware.
[0397] Head movements may be specifically important in the realms
of virtual, augmented, and mixed reality, and may generally be
correlated with eye movements, depending upon the task. There is
large individual variability in propensity for head movements
accompanying eye movements. During tasks like reading, head
movement can account for 5% to 40% of shifting gaze (combined with
eye movements). The degree to which a user normally moves their
head may prove a key indicator of susceptibility to sickness from
mismatch of visual and vestibular sensation.
[0398] It is likely that saccadic and pursuit head movements may be
qualitatively different in those two modalities. For example, a
mismatch may be less jarring if users follow an object from body
front, 90 degrees to the right, to body side using a pursuit
movement as opposed to freely directing gaze from forward to the
right. If the velocity of a pursuit object is relatively steady
then the mismatch would be imperceptible through most of the
motion.
[0399] Referring back to FIG. 3, a user's 302 vocal responses may
also be tracked via microphone. Speech recognition algorithms would
extract semantic meaning from recorded sound and mark the time of
responses (potentially of individual words or syllables). In less
sophisticated scenarios the intensity of vocal responses may be
sufficient to mark the time of response. In embodiments, voice and
speech data is correlated with several other forms of data such as
and not limited to head tracking, eye-tracking, manual inputs, in
order to determine levels of perception.
Electrophysiology/Autonomous Recording
[0400] Electrophysiological and autonomic measures fall largely
outside the realm of conscious influence and, therefore,
performance. These measures pertain largely to states of arousal
and may therefore be used to guide stimulus selection. Recounted
for convenience here, the measures of interest would come from
electroencephalography (EEG--specifically the activity of various
frequency bands associated with arousal states), galvanic skin
response (GSR--also associated with arousal and reaction to
emotional stimuli), heart rate, respiratory rate, blood
oxygenation, and potentially measures of skeletal muscle
responses.
[0401] Reference is again made to WO2015003097A1 titled "A
Non-Invasive Method for Assessing and Monitoring Brain", which has
at least partial common inventorship with the present
specification. In an example, brain wave activity is assessed by
performing an active brain wave test. The active brain wave test
may be conducted using EEG (electroencephalography) equipment and
following methods known in the art. The active brain wave test may
be performed while the individual is presented with a variety of
visual stimuli. In some embodiments, the active brain wave test is
conducted while presenting a standardized set of visual stimuli
that is appropriate for assessing active brain wave activity. In
some embodiments, the active brain wave test may be conducted
multiple times, using the same or different visual stimuli, to
obtain an average result. The results of the active brain wave test
may comprise, for example, temporal and spatial measurements of
alpha waves, beta waves, delta waves, and theta waves. In some
embodiments, the results of the active brain wave test may comprise
a ratio of two types of brain waves; for example, the results may
include a ratio of alpha/theta waves.
[0402] Similarly, a passive brain wave test may be performed. The
passive brain wave test may be conducted using EEG
(electroencephalography) equipment to record brain wave data while
the individual has closed eyes; i.e., in the absence of visual
stimuli. The results of the passive wave brain wave test may
comprise, for example, temporal and spatial measurements of alpha
waves, beta waves, delta waves, and theta waves, for example. In
some embodiments, the results of the passive brain wave test may
comprise a ratio of two types of brain waves; for example, the
results may include a ratio of alpha/theta waves. In some
embodiments, the passive brain wave test may be conducted multiple
times to obtain an average result.
[0403] When possible, and reliant upon precise timing information
for both electric potentials and stimulus displays/speakers,
time-averaged responses can be generated from repeated trials.
Characteristic waveforms associated with visual or auditory
processing (Event Related Potentials, ERP) can be measured and
manipulated in various ways. As these do not require volitional
behavior from users they represent a lower-level, arguably more
pure measure of perception.
[0404] Referring back to FIG. 3, electrophysiological data 318 may
be yet another efferent data source 306, which may generally be
available in the form of voltage potentials recorded at a rate on
the order of kHz. This may include any and all measurements of
voltage potentials among electrodes placed on the skin or other
exposed tissue (notably the cornea of the eye). Most use cases
would presumably involve noninvasive recording, however
opportunities may arise to analyze data from implanted electrodes
placed for other medically valid purposes. Data may generally be
collected at rates in the hundreds or thousands of samples per
second. Analyses may focus on either time-locked averages of
responses to stimulus events to generate waveforms or on various
filtered representations of the data over time from which various
states of activity may be inferred. For example,
Electroencephalogram (EEG) may be used to gather electrode
recording from the scalp/head, to reveal electrical activity of the
brain and other neural activity. Recording may focus on areas of
primary sensory processing, secondary and later sensory processing,
cognitive processing or response generation (motor processing,
language processing). An Electrooculogram (EOG) may be utilized to
gather electrode recording from near the eye to measure changes in
field potential due to relative eye position (gaze direction) and
can also measure properties of retinal function and muscle
activity. EOG may provide a low spatial resolution substitute for
video eye tracking. An Electroretinogram (ERG) may be used to
gather electrode recording from the cornea (minimally invasive) to
capture neural activity from the retina. Correlation with chromatic
and spatial properties of stimuli may allow for the
characterization of responses from different cone types and
locations on the retina (this is also the case with visual evoked
potentials recorded via EEG). An Electrocardiogram (ECG) may be
used to gather neuromuscular activity corresponding to cardiac
function and provide measures of autonomic states, potentially in
response to stimuli. Measurement of neuromuscular potentials may
involve electrodes placed anywhere to record neuromuscular activity
from skeletal muscle flex and/or movement of body and limb
(including electromyogram, or EMG). Measurement of Galvanic Skin
Response (GSR) may involve electrodes that can measure potential
differences across the skin which are subject to conductance
variations due to sweat and other state changes of the skin. These
changes are involuntary and may reveal autonomic responses to
stimuli or scenarios.
[0405] Another source of efferent data 306 may be autonomic
monitoring data 320, including information about heart rate,
respiratory rate, blood oxygenation, skin conductance, and other
autonomic (unconscious) response data from user 302 in forms
similar to those for electrophysiological data 318. Pressure
transducers or other sensors may relay data about respiration rate.
Pulse oximetry can measure blood oxygenation. Pressure transducers
or other sensors can also measure blood pressure. Any and all
unconscious, autonomic measures may reveal responses to stimuli or
general states for categorization of other data. FIG. 9 illustrates
characteristic metrics 900 for electrophysiological monitoring data
902 and autonomic monitoring data 904, in accordance with an
embodiment of the present specification.
[0406] Sample questions concerning electrophysiological metrics 902
and autonomic metrics 904 that may be answered over a period of
time may include: what are the characteristics of time-averaged
responses to events, how do various frequency bands or other
derived states change over time or in response to stimuli.
[0407] Sensors for collecting data may be a part of hardware 106,
described above in context of FIG. 1A. Some sensors can be
integrated into an HMD (for example, sensors for
electroencephalography, electrooculography, electroretinography,
cardiovascular monitoring, galvanic skin response, and others).
Referring back to FIG. 3, some data may require sensors elsewhere
on the body of user 302. Non-contact sensors (even video) may be
able to monitor some electrophysiological data 318 and autonomic
monitoring data 320. In embodiments, these sensors could be smart
clothing and other apparel. It may be possible to use imaging data
for users, to categorize users or their present state. Functional
imaging may also provide data relating to unconscious responses to
stimuli. Imaging modalities include X-Ray/Computed Tomography (CT),
Magnetic Resonance Imaging (MRI), Ophthalmic Imaging, Ultrasound,
and Magnetoencephalography (MEG). Structural data derived from
imaging may be used to localize sources of electrophysiological
data (e.g. combining one or more of structural, MRI EEG, and MEG
data).
[0408] Metrics may be broken into direct measures that can be
inferred from these stimulus/response feature pairs, and indirect
measures that can be inferred from the direct measures. It should
be understood that in most cases individual occurrences of
stimulus/response feature pairings may be combined statistically to
estimate central tendency and variability. There is potential value
in data from a single trial, from descriptive statistics derived
from multiple repeated trials of a particular description and from
exploring stimulus and/or response features as continuous variables
for modelling and prediction.
Facial Pattern Recognition Machine Learning
[0409] The SDEP may utilize its models and predictive components in
combination with a product to enable development of a customized
predictive component for the product. The SDEP predictive
components may be built through a collection process by which a
large dataset of vision data from naturalistic or unconstrained
settings from both primary and secondary sources may be curated and
labeled. The dataset may include photographs, YouTube videos,
Twitch, Instagram, and facial datasets that are available through
secondary research, such as through the Internet. The curated and
labeled data may be utilized for further engagement, and to build a
custom-platform for the product.
[0410] FIGS. 10A to 10D illustrate an exemplary process of image
analysis of building curated data. The illustrations describe an
exemplary mobile-based version of the model. In other embodiments,
the model may be executed on the cloud. FIG. 10A illustrates an
exemplary image of a subject for whom a customized predictive
component may be developed. FIG. 10B illustrates an image of the
subject where the SDEP identifies the eyes for eye tracking, blink
detection, gaze direction, and other parameters and/or facial
attributes. In embodiments, the eyes are continually identified for
tracking purposes through a series of images or through a video of
the subject.
[0411] FIG. 10C illustrates a dataset 1002 of vision data from
naturalistic or unconstrained settings, which may be used for
extracting face attributes in the context of eye tracking, blink,
and gaze direction. In embodiments, the SDEP system is trained with
a large data set 1002 under different conditions where the frames
are extracted from videos. Different conditions may include among
other, complex face variations, lighting conditions, occlusions,
and general hardware used. In embodiments, various computer vision
techniques and Deep Learning are used to train the system.
Referring to FIGS. 10C and 10D, image 1004 is selected to extract
face attributes for analyzing emotions of the subject. In
embodiments, images from the dataset, including image 1004, are
curated and labelled.
[0412] The following steps outline an exemplary data curation and
labelling process: [0413] 1. Identify desirable data sources [0414]
2. Concurrently, develop a pipeline to perform facial key point
detection from video and still images. This may be achieved by
leveraging facial key point localization to segment and select the
ocular region from faces. Further key point features may be used to
determine rotation, pitch, and lighting of images, as possible
dimensions to marginalize over in downstream analysis. Facial
expressions may be identified to analyze emotions. Blinks, eye
movements, and microsaccades may also be identified as part of the
key point detection system. [0415] 3. Scrapes of data sources may
be identified and fed through the SDEP to obtain a normalized set
of ocular region images. Final images may be segmented/cropped to
include only the ocular region, such that information on pitch,
rotation, and lighting is available upon return. [0416] 4. Output
from the above processing may be combined with a product to label
blink, coloration, strabismus, and other metrics of interest to the
product.
[0417] The above-mentioned collected and labelled data may be
leveraged to develop custom predictive models of the ocular region.
Customized machine learning algorithms may be created to predict
key parameters ranging from blink rate, fatigue, emotions, gaze
direction, attention, phorias, convergence, divergence, fixation,
gaze direction, pupil size, and others. In addition, multimodal
approaches may leverage the SDEP in order to benefit from pixel
level information in digital stimuli and jointly learn
relationships with ocular response. The pixel level information may
be broken down to RGB, luminance to fuse the same with existing
visual modeling algorithms.
[0418] In embodiments, eye tracking parameters are extracted from
eye tracking algorithms. In an embodiment, pupil position, relative
to the face, provides one measure from which to classify eye
movements as fixations, pursuits and saccades. In an embodiment,
pupil size is also measured, independently for both eyes. In an
embodiment, gaze direction is estimated from relative pupil
position. Gaze position may be measured in 3D space using data from
both eyes and other measures (i.e. relative position of the face
and screen), including estimates of vergence. Gaze Position
provides another measure from which to classify eye movements.
[0419] FIGS. 11A and 11B illustrate pupil position and size and
gaze position over time. While FIG. 11A illustrates pupil position
and size and gaze position in 3D 1104A and 2D 1110A, at a first
time; FIG. 11B illustrates pupil position and size and gaze
position in 3D 1104B and 2D 1110B, at a second time. In an
embodiment the second time is later than the first time. At any
given point in the image there is (up to) 1 second of data being
shown, with older data shown in a different color, such as blue.
The light blue square represents the display at which the observer
was looking. Physical dimensions are not to scale (e.g. the viewing
distance was greater than it appears to be in the left panel). The
left panel 1104A and 1104B shows a 3D isometric view of space with
user's eyes 1106 to the left and the display 1108 to the right.
[0420] On the left side, gaze position is shown in 3D 1104A and
1104B. A line is drawn from the surface of the observer's display
1108 to the gaze position; red indicates gaze position behind the
display 1108 and green indicates gaze position in front of the
display 1108. Three circles convey information about the eyes 1106:
[0421] 1. The largest, dark grey outline circle represents the
average position of the eyes and face, relatively fixed in space.
[0422] 2. The light grey outline within represents the average
pupil size and pupil position relative to the face (moves but
doesn't change size). [0423] 3. The black filled circle shows
relative pupil size as well as pupil position relative to the face
(moves and changes size).
[0424] When the pupil information is missing it may be assumed that
the eyes are closed (or otherwise obscured).
[0425] Gaze position in 3D 1104A and 1104B is shown by a black dot
(connected by black lines), with gaze direction emanating from both
eyes. Depth of gaze from the display is further indicated by a
green (front) or red (behind) line from the display to the current
gaze position.
[0426] On the right side, gaze position 1110A and 1110B is shown in
2D. Here information about the pupils is absent. Also, information
classifying eye movements is added: [0427] 1. Black indicates
fixation during which a grey outline grows indicating relative
duration of the fixation. [0428] 2. Blue indicates pursuit. [0429]
3. Green (with connecting lines) indicates saccades with lines
connecting points during the saccade.
Brain-Machine Interfaces
[0430] Overall Brain-Machine (Computer) Interface (BMI/BCI)
standardization requires standardizing the interoperability,
connectivity, and modularity of multiple sensory interfaces with
the brain, with many being closed looped. Embodiments of the
present specification provide an exchange platform, given the
current limitations of closed-loop symptoms, in supporting a
standardization of these requirements.
[0431] Additionally, current measures and rating systems for
VR/AR/MxR are qualitative in nature. Embodiments of the present
specification aid in establishing quantitative measures to improve
the quality of the user experience in VR/AR/MxR environments. This
standardization is necessary, as these HMDs are becoming more
pervasive in the non-clinical settings.
[0432] Current BMI/BCI interfaces, including but not limited to
EEG, MRI, EOG, MEG, fMRI, ultrasound, and microwaves, are modular
in nature. Among these different potential interfaces, the division
in clinical and non-clinical context, is in part limited to the
portability of the interfaces, with non-clinical being
traditionally more portable. Vision data may be learned and
utilized within the SDEP for different means of connectivity and
interoperability, that will translate to the larger equipment
involved in BMI/BCI interfaces, including but not limited to MRI,
MEG, and others.
[0433] Embodiments of the present specification describe exemplary
use cases that may be utilized for standardization of both the
hardware components that make up HMDs and the software requirements
for apps used in AR/VR environments.
[0434] Referring to the hardware components, features of HMDs may
be key for standardization. For example, HMD devices have built in
cameras very well suited to capture vision related data and extract
various parameters to glean information in ways which was not
possible before. This, combined with contextual information and
data from other allied sensors may provide a unique opportunity to
study the data and put perceptual computing into BMI/BCI systems.
Defining of minimal specifications of cameras may be required to
achieve this type of data capture for perceptual computing.
[0435] Referring to software components, displaying stimuli in
VR/AR/MxR environments the embodiments of present specification
provide systems and methods for being cognizant of focal point
position, crowding, vection, accommodative mismatch, prolonged
convergence and divergence, chroma-luminance, frame rate,
sequencing, and other factors. Ignoring them may lead to multiple
points for visually-induced motion sickness, headaches and/or
computer vision syndrome. Quantification methods to establish data
collection and pattern recognition for best in practice developer
design methods of software is needed and are part of embodiments of
the present specification.
[0436] HMDs do provide conflict between visual and vestibular
stimuli because of the variations in foveal and peripheral vision.
Also HMD image does not move with the head motion of the wearer.
Embodiments of the present specification may be able to measure
torsional eye movements, vergence eye movements, gaze detection and
pupillary response from the data captured by native eye-tracking
components in a non-invasive way. Data capturing for data like the
minimal frame rate and pupil capture, may thus be standardized for
the SDEP, in accordance to the various embodiments.
[0437] The SDEP built as part of above embodiments may comprise
data ingestion modules from different sources such as HMD, mobile,
various eye trackers, image and video sources, traditional imaging
systems such as fMRI, X-ray, and the like, and data from EEG, EOG,
and others.
[0438] The machine learning modules may process data in batch and
real-time mode and expose the same as API so that it can be
integrated and interfaced with multiple applications. The machine
learning system may use Deep Convolutional neural networks to
detect pupillary metrics, blink detection and gaze accurately from
any image or video source. The other machine learning components
may then correlate this data with sensory data inputs such as EEG,
EOG, EMG, head movement data, haptic data and build comprehensive
perceptual models of human vision.
Vision Performance Index
[0439] An important class of metrics may be those relating to
performance. The performance of a user may be determined in the
form of Vision Performance Index (VPI), which is described in
detail subsequently in embodiments of the present
specification.
[0440] Referring back to FIG. 1A, in an embodiment, data collected
from user 102, such as by media system 104, may be processed to
identify a Vision Performance Index (VPI) for user 102 (also
referring to 1210 of FIG. 12). The VPI may indicate a level of
vision performance of user 102 assessed during user's 102
interaction with VR/AR/MxR system 104. The VPI may be used to
identify a group of users for user 102 that have a similar VPI. The
VPI may be further utilized to modify VR/AR/MxR media for user 102
in order to minimize Visually Induced Motion Sickness (VIMS), or
any other discomfort arising from the virtual experience. In an
embodiment, media is modified in real time for user 102. In another
embodiment, VPI is saved and used to modify presentation of
VR/AR/MxR media to subsequent users with a similar VPI, or
subsequently to user 102.
[0441] VPI may be measured and manipulated in various ways. In
general, the goal may be to improve user's vision performance,
however manipulations may also be aimed at increasing challenge
(e.g. for the sake of engagement) which may, at least temporarily,
decrease performance. In alternate embodiments, performance indices
other than or in addition to that related to vision may be measured
and manipulated. For example, other areas such as design,
engagement, and the like, may be measured and manipulated through
performance indices.
[0442] Referring again to FIG. 12, an exemplary outline of a data
analysis chain is illustrated. The data analysis begins at the
lowest level at 1202 where data level may not be simplified
further. At 1202, parameters of a single stimulus can be used for
multiple measures based on different independent variables, which
correspond to direct features of a stimulus. Parameters of a single
response can be used for multiple measures based on different
dependent variables. At 1204 independent and dependent variables
may be paired to extract a measure of a user's vision performance,
or combined with others and fit to a model to generate measures of
the user's vision performance. In embodiments, pairing may involve
combining a response event to one or more stimulus events through
correlation or other statistical/non-statistical methods.
Individual pairs may be filtered to arrive at 1206, where, for a
given type of interaction, many pairs of independent and dependent
variables can be used to either estimate the parameters of a model
distribution or estimate descriptive statistics. In embodiments, a
model distribution is an expectation of how often a measure will be
a specific value. In some instances a normal distribution, which
has the classic shape of a `Bell curve`, may be used. Once the
process of descriptive statistics or model fitting is completed, at
1208, an individual estimate of a physical measure of a property of
user's vision may be generated. The individual user estimate may be
based on a single interaction or a summary measure from multiple
interactions. The measures of at least one physical property may be
normalized to contribute to sub-components of VPI, at 1210. At
1212, multiple VPI sub-components scores may be combined (for
example, averaged) to generate component scores. In embodiments,
component scores may be further combined to generate overall VPI.
VPI, its subcomponents, and components are discussed in greater
detail in subsequent sections of the present specification.
[0443] In embodiments, measures of vision performance may be
presented as a normalized "score" with relative, but not absolute,
meaning, to the users. This is also illustrated at 1210 and 1212 in
context of FIG. 12. Users may be able to gauge their level of
performance against the general population, or specific subsets
thereof. Due to the presumed high degree of measurement noise
associated with data recording from non-specialized hardware (i.e.
mobile devices used outside of a controlled experimental setting),
precise measures of efferent phenomena (e.g. pupil size, gaze
direction, blink detection) and afferent parameters (e.g. display
chromoluminance, viewing distance, audio intensity) are
unavailable. It may therefore be required to rely on estimates of
central tendency (i.e. mean) and variability (i.e. standard
deviation) from the accumulated data of all users to define
"typical" ranges for each measure and to set reasonable goals for
increasing or decreasing those measures.
[0444] Scores may be normalized independently for each type of
measure, for each of a variety of types of tasks and generally for
each unique scenario or context. This may enable easy comparison
and averaging across measures taken in different units, to
different stimuli, and from different kinds of user responses.
Additionally, for any and all scores, performance may be
categorized as being marginally or significantly above or below
average. Set descriptive criteria may be decided based on
percentiles (assuming a given measure will be distributed normally
among the general population). The examples in the following
sections use 10% and 90%, however the percentiles may be
arbitrarily chosen and can be modified for specific contexts. It
may be assumed that 10% of users' scores will fall in the bottom or
top 10% of scores, and therefore be `abnormally` low or high,
respectively.
[0445] In an embodiment, VPI may be a combination of one or more of
the following parameters and sub-parameters, which may be both
afferent and efferent in nature. Direct measures generally relate a
single response feature to a single stimulus feature. Whenever
possible a psychometric function may be built up from the pattern
of responses (average response, probability of response or
proportion of a category of responses) as the stimulus feature
value changes. Direct measure may include the following: detection,
discrimination, response time, and/or error.
[0446] Indirect measures may be the higher level interpretations of
the direct measures and/or combinations of direct measures. These
may also generally include descriptions of direct measures within
or across specific contexts and the interactions among variables.
Indirect measures may include the following: multi-tracking,
fatigue/endurance, adaptation/learning, preference, memory, and/or
states.
[0447] In embodiments, other vision-related parameters may be used
to calculate the VPI, and may include, but are not limited to field
of view (F), accuracy (A), multi-tracking (M), endurance (E),
and/or detection/discrimination (D), together abbreviated as FAMED,
all described in greater detail below.
Field of View (F)
[0448] Referring back to FIG. 1A, the Field of View (F) may be
described as the extent of visual world seen by user 102 at any
given moment. Central vision represents a central part of the field
of view of user 102, where user 102 has the greatest acuity which
is important for things like reading. Peripheral Vision is the
external part of the field of view of user 102, which is important
for guiding future behavior and catching important events outside
of user's 102 focus.
[0449] Field of View measures the relative performance of users
when interacting with stimuli that are in their Central or
Peripheral fields of view based on measures of Accuracy and
Detection. It is assumed that performance should generally be worse
in the periphery due to decreased sensitivity to most stimulus
features as visual eccentricity increases. The ratio of performance
with Central and Peripheral stimuli will have some mean and
standard deviation among the general population; as with other
measures, the normalized scores will be used to determine if users
have abnormally low or high Field of View ability.
[0450] If a user's Field of View score is abnormally low it may be
improved by increasing the Accuracy and Detection scores for
stimuli presented in the periphery. This generally would entail
increasing consistency of timing and position, increasing
chromaticity and luminance differences (between and within
objects), increasing the size of objects and slowing any moving
targets when presented in the periphery.
Accuracy (A)
[0451] Referring back to FIG. 1A, accuracy (A) may be a combination
of making the right choices and being precise in actions performed
by user 102. Measures of accuracy may be divided into two
subcomponents: Reaction and Targeting. Reaction relates to the time
it takes to process and act upon incoming information. Reaction may
refer to ability of the user 102 to make speedy responses during
the VR/AR/MxR experience. Reaction may be measured as the span of
time between the point when enough information is available in the
stimulus to make a decision (i.e. the appearance of a stimulus) and
the time when the user's response is recorded. For a speeded
response this will usually be less than one second.
[0452] If a user's Reaction is abnormally slow (abnormally low
score) it may be that the task is too difficult and requires
modification of stimulus parameters discussed later in the context
of Targeting and Detection. In an embodiment, a model distribution
for any given measure (for example, a log-normal distribution for
reaction times) is estimated. A cut-off may be determined from the
estimate, above which 5% (or any other percentage) slowest time
spans are found. Any incoming measure of reaction time that is
equal or greater to the cut-off is considered `slow` (or
`significantly slow`). However, if reaction alone is abnormally
low, when other scores are normal, it may be a sign of poor
engagement with the task or a distraction. It may be helpful to
reduce the number of items presented simultaneously or add
additional, congruent cues to hold attention (e.g. add a sound to
accompany the appearance of visual stimuli). If the user is
required to respond to the location of a moving object, it may be
that they require longer to estimate trajectories and plan an
intercepting response; slowing of the target may improve
reaction.
[0453] Response Time may be important for detection related
measures, but is relevant to any response to a stimulus. Response
time is generally the time span between a stimulus event and the
response to that event. Response time may be used to measure the
time necessary for the brain to process information. As an example,
the appearance of a pattern on a display may lead to a certain
pattern of responding from the retina measurable by ERG. At some
point after the stimulus processing is evident from an averaged ERG
waveform, the processing of that same stimulus will become evident
in an average visual evoked potential (VEP) waveform recorded from
the back of the head. At some point after that the average time to
a button press response from the user indicates that the stimulus
was fully processed to the point of generating a motor response.
Though multiple timestamps may be generated by stimulus and
response events, the response time should generally be taken as the
time between the earliest detectable change in the stimulus
necessary to choose the appropriate response to the earliest
indication that a response has been chosen. For example, if an
object begins moving in a straight line towards some key point on
the display, that initial bit of motion in a particular direction
may be enough for the user to know where the object will end up.
They need not wait for it to get there. Likewise the initiation of
moving of the mouse cursor (or any other gesture acceptable in a
VR/AR/MxR environment) towards a target to be clicked may indicate
that a response has been chosen, well before the click event
actually occurs.
[0454] In embodiments, other changes in patterns of responding,
including improvements, decrements and general shifts, may occur as
the result of perceptual adaptation, perceptual learning and
training (higher order learning). Considering adaptation and
learning by the user may account for any variability in responses
that can be explained, and thereby reduce measures of statistical
noise and improve inferential power.
[0455] Patterns in responding, and changes thereof, may also be
related to high order processes within the system. Users have an
occasional tendency to change their minds about how they perform a
task while they're doing it. Therefore, in embodiments, every
choice made by users is analyzed for preferences, regardless of
whether it informs models of visual processing.
[0456] In embodiments, responses are used by the system to measure
recall or recognition by a user. Recall is the accurate generation
of information previously recorded. Recognition is the correct
differentiation between information previously recorded and new
information.
[0457] Derived from measures over time and in specific contexts,
measures of memory recall and recognition and memory capacity can
be made. These may generally fall under the performance category
and users may improve memory performance with targeted practice.
Recall and recognition are often improved by semantic similarity
among stimuli. Memory span may, likewise, be improved by learning
to associate items with one another. The span of time over which
items must be remembered may also be manipulated to alter
performance on memory tasks. Distracting tasks, or lack thereof,
during the retention span may also heavily influence
performance.
[0458] For long term memory there may be exercises to enhance
storage and retrieval, both of specific items and more generally.
It may also be possible to derive measures associated with muscle
memory within the context of certain physical interactions.
Perceptual adaptation and perceptual learning are also candidates
for measurement and manipulation.
[0459] Targeting relates to measures of temporal and positional
precision in the user's actions. Referring back to FIG. 1A,
targeting may relate to the precision of the responses of user 102
relative to the position of objects in the VE. Targeting is
measured as the error between the user's responses and an optimal
value, in relation to stimuli. The response could be a click,
touch, gesture, eye movement, pupil response, blink, head movement,
body/limb movement, or any other. If the user is expected to
respond precisely in time with some event (as opposed to acting in
response to that event, leading to a Reaction measure), they may
respond too early or too late. The variability in the precision of
their response yields a Targeting time error measure (usually on
the order of one second or less). Additionally the position of the
user's responses may have either a consistent bias (mean error)
and/or level of variability (standard deviation of error) measured
in pixels on the screen or some other physical unit of
distance.
[0460] In embodiments, the system analyzes data related to user
errors, including incorrect choices and deviations made by the user
from the ideal or an optimum response. Most commonly these may be
misidentification of stimuli, responding at inappropriate times
(false positive responses), failing to respond at appropriate times
(false negatives) and inaccuracy of timing or position of
responses. Variability in responses or measures of response
features may also be indications of error or general inaccuracy or
inconsistency.
[0461] If a user's targeting score is abnormally low it may be that
targets are too small or variability of location is too great. For
timing of responses, more consistent timing of events makes
synchronizing responses easier. This may be in the form of a
recurring rhythm or a cue that occurs at some fixed time before the
target event. For position, errors can be reduced by restricting
the possible locations of targets or, in the case of moving
targets, using slower speeds. Particularly for touch interfaces or
other contexts where responses may themselves obscure the target
(i.e. finger covering the display), making the target larger may
improve targeting scores.
Multi-Tracking (M)
[0462] Multi-tracking (M) may generally refer to instances in which
users are making multiple, simultaneous responses and/or are
responding to multiple, simultaneous stimuli. They also include
cases where users are performing more than one concurrent task, and
responses to stimulus events that occur in the periphery
(presumably while attention is focused elsewhere). Combination
measures of peripheral detection (detection as a function of
eccentricity) and other performance measures in the context of
divided attention may be included.
[0463] Multi-tracking (M) may represent the ability of the user to
sense multiple objects at the same time. Divided attention tasks
may require user to act upon multiple things happening at once.
Multi-Tracking measures the relative performance of users when
interacting with stimuli that are presented in the context of
Focused or Divided Attention. With focused attention, users
generally need to pay attention to one part of a scene or a limited
number of objects or features. In situations requiring divided
attention, users must monitor multiple areas and run the risk of
missing important events despite vigilance. As with Field of View,
measures of Accuracy and Detection are used to determine a user's
performance in the different Multi-Tracking contexts.
[0464] If a user's Multi-Tracking score is abnormally low it may
indicate that they are performing poorly with tasks requiring
Divided Attention, or exceptionally well with tasks requiring
Focused Attention. Therefore, making Divided Attention tasks easier
or Focused Attention tasks more difficult may improve the
Multi-Tracking score. In the context of Divided Attention, reducing
the perceptual load by decreasing the number of objects or areas
the user needs to monitor may help. Increasing durations (object
persistence) and slowing speeds in Divided Attention may also
improve scores.
Fatigue/Endurance (E)
[0465] Performance measures may become worse over time due to
fatigue. This may become evident in reductions in sensitivity
(detection), correct discrimination, increase in response time and
worsening rates or magnitudes of error. The rate of fatigue (change
over time) and magnitude of fatigue (maximum reduction in
performance measures) may be tracked for any and all measures. The
delay before fatigue onset, as well as rates of recovery with rest
or change in activity, may characterize endurance.
[0466] Endurance (E) may be related to the ability of user to
maintain a high level of performance over time. Endurance measures
relate to trends of Accuracy and Detection scores over time. Two
measures for Endurance are Fatigue and Recovery.
[0467] Fatigue is a measure of how much performance decreases
within a span of time. Fatigue is the point at which the
performance of user may begin to decline, with measures of a rate
of decline and how poor the performance gets. The basic measure of
fatigue may be based on the ratio of scores in the latter half of a
span of time compared to the earlier half. We assume that, given a
long enough span of time, scores will decrease over time as users
become fatigued and therefore the ratio will be less than 1. A
ratio of 1 may indicate no fatigue, and a ratio greater than 1 may
suggest learning or training effects are improving performance
along with a lack of fatigue. If a user's Fatigue score is
abnormally low then they may want to decrease the length of
uninterrupted time in which they engage with the task. Taking
longer and/or more frequent breaks may improve Fatigue scores.
Generally decreasing the difficulty of tasks should help as
well.
[0468] Recovery is a measure of performance returning to baseline
levels between spans of time, with an assumed period of rest in the
intervening interval. Recovery may relate to using breaks provided
to user effectively to return to optimum performance. The basic
measure of recovery currently implemented is to compare the ratio
of scores in the latter half of the first of two spans of time to
the scores in the earlier half of the second span of time. The
spans of time may be chosen with the intention of the user having
had a bit of rest between them. We assume that, given long enough
spans of time to ensure some fatigue is occurring, scores will be
lower before a break compared to after and therefore the ratio will
be less than 1. A ratio of 1 indicates no effect of taking a break,
and a ratio greater than 1 may indicate a decrease in engagement
after the break or the presence of fatigue across, and despite, the
break.
[0469] If a user's Recovery score is abnormally low, they may want
to take longer breaks. It's possible they are not experiencing
sufficient fatigue in order for there to be measurable recovery.
Challenging the user to engage for longer, uninterrupted spans of
time may improve recovery scores. Likewise an increase in task
difficulty may result in greater fatigue and more room for
recovery.
Detection/Discrimination (D)
[0470] Detection/Discrimination (D) may refer to the ability of the
user to detect the presence of an object, or to differentiate among
multiple objects. This parameter may depend on the sensitivity of
user to various attributes of the object. Whenever a response event
signals awareness of a stimulus event it may be determined that a
user detected that stimulus. Unconscious processing, perhaps not
quite to the level of awareness, may also be revealed from
electrophysiological or other responses. Detection can be revealed
by responding to the location of the stimulus or by a category of
response that is congruent with the presence of that stimulus (e.g.
correctly identifying some physical aspect of the stimulus). The
magnitude of a stimulus feature parameter/value necessary for
detection may define the user's detection threshold. Any feature of
a stimulus may be presumed to be used for detection, however it
will only be possible to exclusively attribute detection to a
feature if that feature was the only substantial defining
characteristic of the stimulus or if that stimulus feature appears
in a great variety of stimuli to which users have made
responses.
[0471] Whenever users correctly identify a stimulus feature
parameter/value or make some choice among multiple alternatives
based on one or more stimulus features that interaction may
contribute towards a measure of discrimination. In many cases the
measure of interest may be how different two things need to be
before a user can tell they are different (discrimination
threshold). Discrimination measures may indicate a threshold for
sensitivity to certain features, but they may also be used to
identify category boundaries (e.g. the border between two named
colors). Unlike detection measures, discrimination measures need
not necessarily depend upon responses being correct/incorrect.
Discrimination measures may indicate subjective experience instead
of ability.
[0472] Measures of Detection/Discrimination may be divided into
three subcomponents: measures related to detecting and/or
discriminating Color (chromoluminance), Contrast (chromoluminant
contrast), and Acuity measures based on the smallest features of a
stimulus. These afferent properties, in combination with efferent
measures from manual or vocal responses, eye tracking measures
(initiation of pro-saccade, decrease in anti-saccade, sustained
fixation and decreased blink response), gaze direction, pupil size,
blinks, head tracking measures, electrophysiological and/or
autonomously recorded measures, measures from facial pattern
recognition and machine learning, and others are used to determine
sensitivity. All measures may be based on a user's ability to
detect faintly visible stimuli or discriminate nearly identical
stimuli. These measures are tied to the different subcomponents
based on differences (between detected objects and their
surroundings or between discriminated objects) in their features.
Stimulus objects can differ in more than one feature and therefore
contribute to measures of more than one subcomponent at a time.
[0473] Color differences may refer specifically to differences in
chromaticity and/or luminance. If a user's Color score is
abnormally low, tasks can be made easier by increasing differences
in color. Specific color deficiencies may lead to poor color scores
for specific directions of color differences. Using a greater
variety of hues will generally allow specific deficiencies to have
a smaller impact and stabilize scores.
[0474] Contrast differs from Color in that contrast refers to the
variability of chromaticity and/or luminance within some visually
defined area, whereas measures relating to Color in this context
refer to the mean chromaticity and luminance. If a user's Contrast
score is abnormally low it may be improved by increasing the range
of contrast that is shown. Contrast sensitivity varies with spatial
frequency, and so increasing or decreasing spatial frequency
(making patterns more fine or coarse, respectively) may also help.
Manipulations that improve Color scores will also generally improve
Contrast scores.
[0475] Acuity measures derive from the smallest features users can
use to detect and discriminate stimuli. It is related to contrast
in that spatial frequency is also a relevant physical feature for
measures of acuity. If a user's Acuity score is abnormally low it
may be that objects are generally too small and should be enlarged
overall. It may also help to increase differences in size, increase
contrast and decrease spatial frequency. More so with Acuity than
Color or Contrast, the speed of moving stimuli can be a factor and
slowing moving targets may help improve Acuity scores.
[0476] The above parameters are all based on measuring features. In
embodiments, their patterns may be noted over time. Trends and
patterns may enable predictive analytics and also help personalize
the user experience based on detection capabilities and other
VPI/FAMED capabilities of the end user.
[0477] A great many general states of being may be inferred from
the direct measures discussed. States may be estimated once per
session, for certain segments of time or on a continuous basis, and
in response to stimulus events. These may commonly relate to rates
of responding or changes in behavior. FIG. 13 provides a table
containing a list of exemplary metrics for afferent and efferent
sources, in accordance with some embodiments of the present
specification. The table illustrates that an afferent source may
result in a stimulus event and feature. The combination of afferent
source, stimulus events and feature, when combined further with a
response (efferent source), may indicate a response event and
feature. These combinations may hint at a psychometric measure. In
the last column, the table provides a description for each
psychometric measure derived from the various combinations.
[0478] FIG. 14 is an exemplary flow diagram illustrating an
overview of the flow of data from a software application to the
SDEP. At 1402, a software application that may provide an interface
to a user for interaction. The app may be designed to run on an
HMD, or any other device capable of providing a VR/AR/MxR
environment for user interaction. Information collected by the
application software may be provided to a Software Development Kit
(SDK) at 1404. The SDK works with a group of software development
tools to generate analytics and data about use of the application
software. At 1406 the data is provided as session data from the SDK
to the SDEP. At 1408, session data is pre-processed at the SDEP,
which may include organizing and sorting the data in preparation
for analysis. At 1410, stimulus and response data that has been
pre-processed is generated and passed further for analysis and
processing. At 1412, data is analyzed and converted to performance
indices or scores or other measures of perceivable information,
such as VPI scores. At 1414, the analyzed data is sent back to the
SDK and/or application software in order to modify, personalize, or
customize the user experience. In embodiments data is passed from
1402, from application software through the chain of analysis, and
back to the application software non-intrusively, in real time.
[0479] FIG. 15 illustrates an exemplary outline 1500 of a
pre-processing part of the process flow (1408, FIG. 14).
[0480] FIG. 16 is an exemplary representation 1600 of the
programming language implementation of a data processing function
responsible for taking in raw data (pre-processed), choosing and
implementing the appropriate analysis, sending and receiving
summary measures based on the analysis to temporary and long-term
stores for estimates of `endurance` measures and score
normalization, respectively, and computing scores to be sent back
to the application for display to the end user. In embodiments, the
programming language used is Python. The figure shows application
of several Python functions to FAMED data in order to derive VPI
scores. The figure illustrates color-coded processes for each FAMED
function. In an embodiment, FOV functions are in Red, Accuracy in
Green, Multi-Tracking in Purple, Endurance in Orange, and Detection
in Blue. In an embodiment, parallelograms represent variables;
rounded rectangles represent functions; elements are color coded
for user/session data, which are shown in yellow.
[0481] Referring to the figure, contents of a large red outline
1602 represent the processing function (va_process_data), which
includes three main sections--a left section 1604, a middle section
1606 and a right section 1608. In an embodiment, left section 1604
takes in raw data and applies either Accuracy or
Detection/Discrimination analysis functions to the data yielding a
single measure summarizing the incoming data. That is sent to
middle-level functions 1606 for measures of Field of View and
Multi-Tracking as well as to an external store. That first external
store, or cache, returns similar measures from the recent past to
be used for measures of Endurance. The output from the middle-level
functions 1606 are sent to another external store that accumulates
measures in order to estimate central tendency (i.e. arithmetic
mean) and variability (i.e. standard deviation) for normalization.
Data from this second, external store are combined with the present
measurements to be converted into Scores in the right-level section
1608. The figure also illustrates a small sub-chart 1610 in the
lower left of the figure to show the placement of analysis portion
1600 in the broader chain.
[0482] FIG. 17 provides a flowchart illustrating a method for
modifying media, in accordance with an embodiment of the present
specification. In an embodiment, the method is implemented within
SDEP 118 described above in context of FIG. 1A and the various
embodiments. A user is presented with a media, for example,
VR/AR/MxR media. In embodiments, media is presented through an HMD
or any other type of VR/AR/MxR media rendering device. While the
user experiences the media, at 1702, the SDEP receives vision data
of the user. In an embodiment, the user is in accordance to user
102 of FIG. 1A. In an embodiment, vision data is received from
various afferent and efferent data sources that are engaged within
the media environment. At 1704, the vision data is used to process
the media and modify it. In embodiments, vision data is processed
in real time. The media may be modified to enable the user to
experience the media in an optimal manner. The media may be
modified to improve the user experience in order to minimize VIMS
or any other problems that may be induced otherwise based on the
visual capacity of the user. In embodiments, the media is modified
differently for different applications. For example, a media may be
modified differently for users of games and differently for users
who are potential customers, where the media is respectively
presented through a game and through an advertisement. In other
examples, the media may be presented differently for users of a
specific application, and content experience. In an embodiment, the
vision data is processed in real time to modify it. The vision data
here is the vision model/profile/persona developed through both
batch and real-time analysis and contextualizing of afferent and
efferent data, along with autonomic data input. Alternatively, the
vision data is stored and analyzed for modification, in batches.
The different forms of processing media are described above in
context of FIG. 2. The VPI may also be used to process the media
for its modification in accordance to various metrics. At 1706,
modified media is represented to the user. In an embodiment,
modified media is presented to a group of users at the same time or
at different times, where the users in the group may correspond to
similar vision data. The resulting media may be in continuation to
previously presented media, modified in accordance to certain
metrics determined for the user.
[0483] FIG. 18 provides a flowchart illustrating a method for
modifying media, in accordance with another embodiment of the
present specification. In an embodiment, the method is implemented
within SDEP 118 described above in context of FIG. 1A and the
various embodiments. A user is presented with a media, for example,
VR/AR/MxR media. In embodiments, media is presented through an HMD
or any other type of VR/AR/MxR media rendering device. While the
user experiences the media, at 1802, the SDEP receives vision data
of the user. In an embodiment, the user is in accordance to user
102 of FIG. 1A. In an embodiment, vision data is received from
various afferent and efferent data sources that are engaged within
the media environment. At 1804, the vision data is used to identify
metrics that may affect the visual experience of the user directly
or indirectly. The metrics, as described above in context of the
table of FIG. 13, may aid in deconstructing psychometrics that
impact user's experience. At 1806, information derived from the
metrics may be used to process the media and modify it. The media
may be modified to enable the user to experience the media in an
optimal manner. The media may be modified to improve the user
experience in order to minimize VIMS or any other problems that may
be induced otherwise based on the visual capacity of the user. In
embodiments, the media is modified or personalized differently for
different applications. For example, a media may be personalized
differently for users of games and differently for users who are
potential customers, where the media is respectively presented
through a game and through an advertisement. In an embodiment, the
vision data is processed in real time to modify it. Alternatively,
the vision data is stored and analyzed for modification, in
batches. The different forms of processing media are described
above in context of FIG. 2. At 1808, modified media is represented
to the user. In an embodiment, modified media is presented to a
group of users at the same time or at different times, where the
users in the group may correspond to similar vision data. The
resulting media may be in continuation to previously presented
media, modified in accordance to certain metrics determined for the
user.
Examples of Use
[0484] Data generated by the SDEP in accordance with various
embodiments of the present specification may be used in different
forms. In embodiments, data output by the SDEP may be packaged
differently for gamers, for advertisers, and others.
[0485] The sensory inputs determined and analyzed by the system may
eventually drive work and play engagements. In embodiments, sensory
information may be purchased from users and used to create sensory
data exchanges after adding value to the data through platforms
such as the SDEP. In embodiments of the present specification, the
senses of individuals and potential consumers may be measured and
monitored with the SDEP.
[0486] In embodiments, the SDEP allows for advantageously using
data generated from technologies such as smart devices, wearables,
eye-tracking tools, EEG systems, and virtual reality and augmented
reality HMDs. For example, EEG bands may be used to track eye
movement against electrodes in the brain as well as game-based
applications designed to create vision benchmarks and, ultimately,
help improve visual acuity over time.
[0487] In embodiments, data output by the SDEP may be packaged
differently for medical use (visual acuity, eye strain, traumatic
brain injury, and sports vision performance), for athletes/sports,
and others. For example, applications include the ability to track
the effects of digital eye strain over a period of time or to
screen for traumatic brain injury in contact sports such as
football by measuring key areas of the eye-brain connection.
[0488] In embodiments, systems and methods of the present
specification are used to develop deep learning systems and used to
model artificial neural networks. Artificial Neural Networks and
its advanced forms, including Deep Learning, have varying
applications such as and not limited to image/speech recognition,
language processing, Customer Relationship Management (CRM),
bioinformatics, facial expression recognition, among others. In
embodiments, the SDEP uses feedback loops between efferent and
afferent information to model the human sensory system. In further
embodiments, the SDEP sources information from additional sensors
to combine with the afferent and efferent sources of
information.
Example Use 1: Modifying Media In Order To Improve A User's
Comprehension Of Information In That Media
[0489] In one embodiment, a user's degree of comprehension is
measured by testing knowledge and understanding, namely by
presenting through a display a plurality of questions, receiving
inputs from the user in response to those questions, and
determining to what extent the user answered the questions
correctly. A user's degree of comprehension may also be inferred
from the user's behavior, including subtle changes in behavior, and
from the user's autonomic and electrophysiological measures.
[0490] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to improve that user's comprehension of information being
communicated by that media. FIG. 19 illustrates a flow chart
describing an exemplary process for modifying media in order to
improve comprehension, in accordance with some embodiments of the
present specification. At 1902, a first value for a plurality of
data, as further described below, is acquired. In embodiments, data
is acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [0491] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [0492] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [0493] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [0494]
4. One or more sensors configured to measure auditory processing;
[0495] 5. One or more sensors configured to measure gustatory and
olfactory processing; [0496] 6. One or more sensors to measure
pressure; [0497] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [0498] 8. A device to perform
electroencephalography; [0499] 9. A device to perform
electrocardiography; [0500] 10. A device to perform
electromyography; [0501] 11. A device to perform
electrooculography; [0502] 12. A device to perform
electroretinography; and [0503] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0504] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken into
account, either individually or in combination.
[0505] To probe a user's comprehension of some spatially defined
process in media, the system applies measures of where the user is
looking, measuring the proportion of time the user spends looking
at relevant areas within the media vs. irrelevant areas within the
media. The system can also measure the degree to which the user is
focusing its attention on specific areas as compared to less
focused sampling of the visual space.
[0506] In one embodiment, the system determines a Ratio of Relevant
Fixation R.sub.Rel.Fix. defined as the ratio of fixation number,
frequency or average duration in relevant areas of interest to
irrelevant areas:
R Rel . Fix . = N fixation relevant N fixation irrelevant .apprxeq.
f fixation relevant f fixation irrelevant .apprxeq. D _ fixation
relevant D _ fixation irrelevant ##EQU00004##
[0507] If the user is looking about randomly, and relevant and
irrelevant areas are roughly equal in size, this ratio should be
around 1. If the user is focused more on relevant areas, the ratio
should be greater than 1, and the greater this ratio the more
comprehension the system attributes to the user. The system
determines if the user is comprehending, or not comprehending,
media content based upon said ratio. If the ratio is below a
predetermined threshold, then the system determines the user is not
focused, is looking around randomly, and/or is not comprehending
the media. If the ratio is above a predetermined threshold, then
the system determines the user is focused, is not looking around
randomly, and/or is comprehending the media.
[0508] In one embodiment, the system determines measures derived
from saccade parameters showing more eye movements towards relevant
areas compared to irrelevant areas. The system may determine if the
user is comprehending, or not comprehending, media content in a
VR/AR/MxR environment based upon saccadic movements.
[0509] With regard to saccade angle, the mean of the absolute angle
relative to relevant regions |.theta.|.sub.saccade-relevant should
be much less than 90.degree. if the user is looking towards
relevant areas more often, around 90.degree. if the user is looking
around randomly and greater than 90.degree. if the user is
generally looking away from relevant areas. If the user is looking
more towards relevant areas, the angle would be less than
90.degree., and the system may determine that the user is focused
and is able to comprehend the media. If the angle is near
90.degree. or more, then the system may determine that the user is
not focused, is looking around randomly, and/or has low
comprehension or is not comprehending the media.
[0510] With regard to saccade magnitude, the mean magnitude
component relative to relevant regions M.sub.saccade-relevant
should be significantly positive if the user is looking towards
relevant areas more often, around 0 if the user is looking around
randomly, and significantly negative if the user is generally
looking away from relevant areas. Here, a positive mean magnitude
would imply a significant level of comprehension, whereas around 0,
or a negative value would imply a low level of comprehension of
media content in a VR/AR/MxR environment, by the user. In
embodiments, the system uses this information to modify displayed
media in the VR, AR and/or MxR environment or a conventional
laptop, mobile phone, desktop or tablet computing environment, in
order to improve that user's comprehension of information being
communicated by that media
[0511] The Ratio of Relevant Fixation definition may also be
expanded to include saccade parameters, although it may be assumed
that these are generally equivalent to the fixation parameters:
R Rel . Fix . = N saccade relevant N saccade irrelevant .apprxeq. f
saccade relevant f saccade irrelevant ##EQU00005##
[0512] In another embodiment, the system determines a measure that
exploits the fact that eye movements will frequently go back and
forth between related words or objects in a scene.
[0513] The system defines Fixation Correlations C.sub.fixation
between areas (A and B) known to be related, as a measure of
comprehension:
C.sub.fixation=cor(N.sub.fixation|A,N.sub.fixation|B).apprxeq.cor(f.sub.-
fixation|A,f.sub.fixation|B).apprxeq.cor(D.sub.fixation|A,D.sub.fixation|B-
)
[0514] In one embodiment, the system defines
[0515] Saccade Correlations C.sub.saccade based on saccades with
angles generally toward areas A and B
(.theta..sub.saccade-A.fwdarw.0 and
.theta..sub.saccade-B.fwdarw.0):
C.sub.saccade=cor(N.sub.saccade|towards A,N.sub.saccade|towards
B).apprxeq.cor(f.sub.saccade|towards A,f.sub.saccade|towards B)
[0516] The greater these kinds of correlations, the system
determines that the more users are monitoring the behavior of two
related objects in the scene suggesting greater comprehension. The
system determines if the user is comprehending, or not
comprehending, media content based upon the correlations. If the
correlations are below a predetermined threshold, then the system
determines the users are not focused, are looking around randomly,
and/or are not comprehending the media. If the correlations are
above a predetermined threshold, then the system determines the
users are focused, are not looking around randomly, and/or are
comprehending the media. It should be appreciated that the system
may be engineered such that the reverse is true instead: If the
correlations are above a predetermined threshold, then the system
determines the users are not focused, are looking around randomly,
and/or are not comprehending the media. If the correlations are
below a predetermined threshold, then the system determines the
users are focused, are not looking around randomly, and/or are
comprehending the media.
[0517] Even without knowledge of the scene and what areas are
relevant or should be correlated to signal comprehension, the
system may assume that more or less focused attention within a
scene is indicative of a degree of comprehension. In combination
with more direct comprehension measures (i.e. questioning), a
measure of focus can be used to take a simple correct/incorrect
measure and assign to it some magnitude.
[0518] In some embodiments, comprehension is also gleaned from eye
movement data of a listener compared to a speaker when both are
viewing the same thing. When the eye movements of a listener are
determined to be correlated to the eye movements of a speaker while
explaining something going on in a shared visual scene, the system
may determine that the user is able to comprehend. The system may
attribute greater comprehension when the delay between the eye
movements of the speaker and the corresponding eye movements of the
listener is lower.
[0519] Correlation of the listener's eye movements may be
calculated as:
C.sub.listening=cor([x,y,z].sub.fixation|speaker(t)[x,y,z].sub.fixation|-
listener(t+.tau.))
using fixation position as a function of time for the speaker and
listener with a delay of .tau. seconds. In embodiments, the above
correlation peaks at around .tau.=2 s. The greater the correlation,
the system determines that the more users are monitoring the
behavior of two related objects in the scene, thereby suggesting
greater comprehension. The system determines if the users are
comprehending, or not comprehending, media content based upon the
correlation. If the correlation is below a predetermined threshold,
then the system determines the users are not focused, are looking
around randomly, and/or are not comprehending the media. If the
correlations are above a predetermined threshold, then the system
determines the users are focused, are not looking around randomly,
and/or are comprehending the media. It should be appreciated that
the system may be engineered such that the reverse is true instead:
If the correlations are above a predetermined threshold, then the
system determines the users are not focused, are looking around
randomly, and/or are not comprehending the media. If the
correlations are below a predetermined threshold, then the system
determines the users are focused, are not looking around randomly,
and/or are comprehending the media.
[0520] In embodiments, an Area of Focus A.sub.focus is determined
as the minimum area of the visual field (square degrees of visual
angle) that contains some proportion p (e.g. 75%) of the fixations
(N.sub.fixation) or the total fixation duration
(.SIGMA.D.sub.fixation) for a given span of recording. This may be
found algorithmically by estimating the smallest circular area
containing all of a subset of fixations, defined by number or
duration, and repeating for all possible subsets and keeping the
smallest area among all subsets. In embodiments, the determined
area may indicate the area that offers greater comprehension to
users.
Looking Away
[0521] The action of users looking away (averting gaze) is
correlated with cognitive load, indicative of a mechanism to limit
the amount of detailed information coming into the visual system to
free up resources. The system determines that the greater the
amount of time users look away (.SIGMA.D.sub.fixation|away) from
the display, the more demanding could be the task; which in turn is
determined to be evidence of greater comprehension (as looking away
correlates with cognitive load as described above). In additional
embodiments, if it is observed that the user looks more towards
areas of less high-spatial-frequency contrast, the system may again
determine that the given task is more demanding, leading to a need
for the user to look away, and therefore evidence of greater
comprehension. In embodiments, head movements associated with the
action of looking away may be used in place of or in addition to
the eye movements, to determine comprehension.
Reading
[0522] In embodiments, the system determines comprehension levels
of a user by tracking the eyes during reading. The system has
knowledge of the text being read, which is used to determine
measures of comprehension. In embodiments, the measures are
determined based on fixation durations on specific words, the
number or frequency of regressive saccades (in an exemplary case,
for English, leftward saccades on the same line/within the same
sentence are determined; direction of regressive saccades may be
different and measured accordingly for different languages) and
fixation durations on specific sentence parts. The lower the
durations of fixation and/or the greater the frequency of
regressive saccades, the system determines that the greater is a
user's comprehension. The system determines if the users are
comprehending, or not comprehending, media content based upon the
fixation durations. If the duration is above a predetermined
threshold, then the system determines the users is facing
difficulty in comprehending the media. If the durations are below a
predetermined threshold, then the system determines the users are
focused, and/or are comprehending the media.
[0523] It has also been shown that blink rates (f.sub.blink)
decrease during reading which also leads to an increase in
variability in the time between blinks. In embodiments, the system
determines a slowed blink rate relative to an established baseline
(f.sub.blink significantly less than f.sub.blink). The act of
blinking slowly relative to a baseline may be determined by
calculating a deviation from a mean. The greater the deviation from
the mean, the system determines that greater is the comprehension
of a user.
Electrophysiology
[0524] In embodiments, electrophysiological recordings yield
measures of degree of comprehension within certain contexts. For
example, using electroencephalographic event-related potentials
(EEG-ERP) there may be seen increases in amplitudes of some
cognitive potentials (N2, N400, P300, P600) for unexpected and/or
incongruous words while reading. In embodiments, the system
determines, in response to a significant amplitude magnitude
increases in cognitive EEG potentials (N2, N44, P300, P600)
resulting from infrequent, novel or unexpected stimuli, to be an
indication of greater comprehension. In general, the system may
conclude that the magnitude of amplitude changes compared to a
baseline are proportional to a degree of comprehension. A positive
change in amplitude may be attributed to greater comprehension
while a lower or negative change may be attributed to lower ability
to focus and/or comprehend.
[0525] The presence or absence of these amplitude changes,
depending upon what's being read, aids the system to determine
whether a user is correctly interpreting what they're reading. More
generally, some EEG-ERP components can be used to measure cognitive
load which may itself, within certain contexts, be an analog of
comprehension. Galvanic skin response (GSR-ERP) can also be used,
with increasing values indicating increasing cognitive load, and
therefore greater comprehension. In embodiments, the system
determines a significant increase in GSR-ERP as a signal of
comprehension.
[0526] In embodiments, the system uses general
electroencephalographic activity to measure comprehension. For
example, increased activity in a beta/gamma frequency band (20-70
Hz) can be linked to various cognitive processes including
associative learning. The system determines significant increases
in energy of an EEG in beta and gamma frequency bands (.gtoreq.16
Hz) to be a signal of increased comprehension.
Timing
[0527] Response times will generally be faster for correct
responses when users know they are correct as compared to when they
guessed correctly. Additionally, the system uses the relative
timing of behavior to an introduction of key components of a scene
or narrative, to measure comprehension. In embodiments, the system
determines increases in comprehension when elements in a scene
change from ambiguous to congruent, or a task/problem goes from
unsolvable to solvable, at a given time. In embodiments, the system
determines this information specifically in the moments after the
new information is made available. The system may further select
appropriate spans of time, based on the moments after the new
information is made available, to analyze other measures described
in various embodiments of the present specification.
Onset of Comprehension
[0528] The onset of comprehension can be revealed by state changes
from various measures including those listed above for measuring
the degree of comprehension. Measures of comprehension onset may
not necessarily be as precise as reaction time data; instead of
identifying the moment in time when comprehension begins, the
following measures may indicate at what point in a sequence of
stimulus and response events the user gains new understanding. For
example, if relying on correct/incorrect responses the system uses
the point in time when percentage of correct responses jumps from a
low baseline to a higher level.
[0529] The system determines the onset of comprehension as the
point in time where, when applicable, the percent of correct
responses increases significantly. In an embodiment, onset of
comprehension is determined using a t-test to compare percent
correct responses from a second time frame relative to a first time
frame or from the current N responses to the previous N responses,
for a number N that is sufficiently large to be statistically
significant.
Target Detection
[0530] In embodiments, the system determines an onset of
comprehension when a user detect and/or selects a target where
knowing a target's identity requires comprehension. Additionally,
the system determines that the user is able to appropriately detect
and/or select based on vocal or manual responses indicating
identity of the target and/or pointing, gestures, facing or gaze
directed at the target location. In embodiments, the system
determines the onset of comprehension as the point when a target in
a VR/AR/MxR media is correctly identified and or located. When
applicable this should be at a rate or degree greater than that
possible by chance. The system determines a rate of onset of
comprehension greater that a specific threshold to be an indication
of greater comprehension. On the other hand, the system may
attribute a rate of onset of comprehension equal to or lower that
the specific threshold to reduced or lack of comprehension. The
system determines if the user is comprehending, or not
comprehending, media content based upon the rate of onset of
comprehension.
Fixation Duration
[0531] Initial sampling of a scene takes in all of the necessary
information to solve a problem, but if users get stuck the duration
of fixations increases as focus turns inward. Once a solution is
discovered fixation durations drop as users resume normal scanning
and verify their solution by checking the available information.
Therefore, we can estimate the onset of comprehension by looking
for a peak in fixation duration over time and finding any sudden
decline thereafter. If the user is found to resume normal scanning
that is associated with low fixation durations after an initial
sampling that is associated with an increased fixation duration,
the system determines such instances to be an indication of onset
of comprehension. The instances are further combined with the
information displayed in the corresponding VR/AR/MxR media, to
determine onset of comprehension. If the instances are low or do
not exist, then the system determines the user is not focused, is
looking around randomly, and/or is not comprehending the media. If
the instances exists and/or are high, then the system determines
the user is focused, is not looking around randomly, and/or is
comprehending the media. In embodiments the system determines the
onset of comprehension as the end of a period of significantly
longer fixation durations (D.sub.fixation).
Pupil Dilation
[0532] In another embodiment, the system uses pupil dilation to
detect the onset of comprehension. The magnitude and time to peak
pupil dilation can be used to signal degree and onset of
comprehension, respectively. The system may observe any significant
dilation of the pupil, as compared to a few to several seconds
prior, as a noteworthy event, in general. The system determines if
the user is, at the onset, comprehending, or not comprehending,
media content based upon the magnitude and time to peak pupil
dilation.
[0533] In embodiments, the system determines a rapid and
significant increase in pupil diameter (S.sub.pupi1) as the onset
of comprehension. In embodiments, the context of the media rendered
in the VR/AR/MxR environment is factored-in with the observations
pertaining pupil dilation. An exemplary context is when the user is
tasked with deriving novel meaning from a stimuli provided through
the media.
Facial Cues
[0534] In embodiments, the system determines transient changes in
facial expression as indications of the onset of states like
confusion, worry, and concentration. In an exemplary embodiment, a
user squinting is an indication of confusion, whereas releasing the
squint is an indication that comprehension has occurred. In
embodiments, the system determines a transition from partially to
completely open eyes (significant increase in p.sub.both eyes open
from a non-zero baseline) as the onset of comprehension in the
appropriate contexts. The system determines if the user is
comprehending, or not comprehending, media content based upon the
transition from partially to completely open eyes.
Sudden Increase in Degree Measures
[0535] In various embodiments, the system uses one or a combination
of the above-described measures to determine an onset of
comprehension. The system samples the above-described measures for
this purpose. In an example, if users go suddenly from a low, or
baseline, degree of comprehension to a heightened degree of
comprehension, the system can determine when comprehension began.
In embodiments, the degree of comprehension is based on a
combination of one or more of the measures described above to
identify levels and onset of comprehension. Algorithmically this
may be established by finding a time with the greatest, and also
statistically significant, difference in degree of comprehension
measures before and after. The system determines if the user is
comprehending, or not comprehending, media content based upon the
sudden differences in the degree measures. A sudden difference in a
parameter is one which is greater than a predefined standard
deviation of historical values for that parameter. When the a
measure passes a predefined standard deviation, the parameter is
deemed to have suddenly changed, thereby indicating a change in
comprehension state (either increased or decreased) based on the
value range of the parameter.
Failure or Lack of Comprehension
[0536] In embodiments, the system determines some measures that
indicate a failure or general lack of comprehension. An absence of
expected changes in degree of comprehension or any signs of
comprehension onset may indicate a failure of comprehension. The
system may identify some new behavior that is determined to signal
the onset of a frustration or a search. In an embodiment, an
increase in body temperature and/or heart rate may signal
frustration during a comprehension task.
[0537] In an embodiment, the system determines a significant
increase in body temperature and/or heart rate in association with
delayed response to a question of understanding, as a signal of a
lack of comprehension. The system determines if the user is
comprehending, or not comprehending media content based upon a
predefined increase in body temperature and/or heart rate in
association with delayed response to a question of understanding,
as a signal of lack of comprehension.
[0538] Eye gaze positions that seem uncorrelated with
comprehension, particularly non-specific search characterized by
brief fixation durations (D_fixation significantly less than
_D_fixation) and saccades sampling the entire task space with large
jumps (M_saccade significantly greater than _M_saccade) may
indicate a desperate search for some missing clue. In embodiments,
the system attributes such instances to lack of comprehension.
[0539] In embodiments, the system determines random or un-focused
search characterized by significantly brief fixation durations and
significantly large saccade magnitudes as indicative of a lack of
comprehension in appropriate contexts. The system determines if the
user is comprehending, or not comprehending media content based
upon the random or un-focused search characterized by significantly
brief fixation durations and significantly large saccade
magnitudes.
Other Correlations
[0540] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous comprehension measures like making correct choices from
among multiple alternatives. In some examples, some of the measures
described above are context specific and may be more or less robust
or even signal the opposite of what is expected. However the
ability of users to respond correctly at rates better than expected
by chance can be taken as a sign of comprehension and
understanding. The system can correlate all available measures and
look for trends in comprehension. Accordingly, the media presented
in a VR/AR/MxR environment is modified, in order to increase its
comprehensibility, for the user and/or a group of users.
[0541] At 1904, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 1906, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect reduced comprehension by the
user of the VR, AR, and/or MxR media: [0542] 1. Decrease in
palpebral fissure height [0543] 2. Increased blink rate [0544] 3.
Increased rate of change for blink rate [0545] 4. Increased ratio
of partial blinks to full blinks [0546] 5. Decreased target
relevancy for pupil initial and final position [0547] 6. Decreased
target relevancy for gaze direction [0548] 7. Decreased target
relevancy for gaze initial and final position [0549] 8. Decreased
target relevancy for fixation initial and final position [0550] 9.
Increased fixation duration rate of change [0551] 10. Decreased
target relevancy for saccade initial and final position [0552] 11.
Decreased target relevancy for saccade angle [0553] 12. Increased
ratio of anti-saccade/pro-saccade [0554] 13. Increased inhibition
of return [0555] 14. Increased screen distance [0556] 15. Decreased
target relevant head direction [0557] 16. Decreased target relevant
head fixation [0558] 17. Decreased target relevant limb movement
[0559] 18. Shift in weight distribution [0560] 19. Decreased
alpha/delta brain wave ratio [0561] 20. Increased alpha/theta brain
wave ratio [0562] 21. Increased body temperature [0563] 22.
Increased respiration rate [0564] 23. Decrease in comprehension
[0565] 24. Low oxygen saturation [0566] 25. Increased heart rate
[0567] 26. Changes in blood pressure [0568] 27. Increased
vocalizations [0569] 28. Increased reaction time
[0570] The system may determine a user has an increased degree of
comprehension of the media in a VR, AR, and/or MX environment based
upon the following changes: [0571] 1. Increased rate of change for
pupil size [0572] 2. Increased rate of convergence [0573] 3.
Increased rate of divergence [0574] 4. Increased fixation rate
[0575] 5. Increased fixation count [0576] 6. Increased saccade
velocity [0577] 7. Increased saccade rate of change [0578] 8.
Increased saccade count (number of saccades) [0579] 9. Increased
smooth pursuit
[0580] Additionally, the system may conclude that a user is
experiencing increased or decreased comprehension of media based
upon the following changes in combination with a specific type of
user task. Accordingly, the system analyzes both the data types
listed below together with the specific type of task being engaged
in to determine whether the user is increasing or decreasing his or
her level of comprehension. [0581] 1. Increased fixation duration
[0582] 2. Decreased saccade magnitude (distance of saccade)
[0583] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; and change in olfactory
processing.
[0584] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to improve comprehension, these
lists are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[0585] At 1908, the changes in the plurality of data determined
over time may be used to determine a degree of change in
comprehension levels of the user. The change in comprehension
levels may indicate either enhanced comprehension or reduced
comprehension.
[0586] At 1910, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) comprehension. In
embodiments, the media may be modified to address all the changes
in data that reflect reduced comprehension. In embodiments, a
combination of one or more of the following modifications may be
performed: [0587] 1. Increasing a contrast of the media [0588] 2.
Making an object of interest that is displayed in the media larger
in size [0589] 3. Increasing a brightness of the media [0590] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [0591] 5. Changing a
focal point of content displayed in the media to a more central
location [0592] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [0593] 7. Increasing an
amount of color in said media [0594] 8. Increasing a degree of
shade in objects shown in said media [0595] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[0596] One or more of the following indicators may be observed to
affirm an improvement in comprehension: increase in palpebral
fissure height; decrease in blink rate; decrease in the blink
rate's rate of change; decreased ratio of partial blinks to full
blinks; increased target relevancy for gaze direction; increased
target relevancy for gaze initial and final position; increased
target relevancy for fixation initial and final position; increased
fixation duration where task requires; decreased rate of change for
fixation duration; increased target relevancy for saccade initial
and final position; increased target relevancy for saccade angle;
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased screen distance; increased target relevant
head direction; increased target relevant head fixation; increased
target relevant limb movement; decrease in shifts of weight
distribution; increased alpha/delta brain wave ratio; decreased
alpha/theta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; targeted facial
expressions; and targeted gustatory processing.
[0597] In embodiments, a specific percentage or a range of
improvement in comprehension may be defined. In embodiments, an
additional value for data may be acquired at 1912, in order to
further determine change in data over time at 1914, after the
modifications have been executed at 1910. At 1916, a new
degree/percentage/range of improvement in comprehension may be
acquired. At 1918, the system determines whether the improvement in
comprehension is within the specified range or percentage. If it is
determine that the improvement is insufficient, the system may loop
back to step 1910 to further modify the media. Therefore, the media
may be iteratively modified 1910 and comprehension may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
Example Use 2: Modifying Media in Order to Decrease a User's
Experience of Fatigue from Information in that Media
[0598] In one embodiment, a user's experience of fatigue and/or a
severity of fatigue is measured by observing visible signs of
fatigue as well as physiological measures that indicate fatigue. A
user's degree of fatigue may also be inferred from the user's
behavior, including subtle changes in behavior, and from the user's
autonomic and electrophysiological measures. Depending on the
information source, measures of fatigue may inform that the user is
fatigued or that the user is becoming fatigued. Some behaviors like
yawning, nodding off, or closed eyes and certain
electrophysiological patterns can signal with little ambiguity that
a user is fatigued, even at the beginning of a session of data
recording. In embodiments, a baseline for comparison is determined
that accounts for individual variability in many other measures, in
order to conclude whether a person is becoming fatigued. For
example, a user's reaction time may be slow for a number of reasons
that have little or nothing to do with fatigue. In the example, an
observation that user's reaction times are becoming slower over a
window of time, may signal fatigue.
[0599] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease measures of fatigue from information being
communicated by that media. FIG. 20 illustrates a flow chart
describing an exemplary process for modifying media in order to
decrease fatigue, in accordance with some embodiments of the
present specification. At 2002, a first value for a plurality of
data, as further described below, is acquired. In embodiments, data
is acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [0600] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [0601] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [0602] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [0603]
4. One or more sensors configured to measure auditory processing;
[0604] 5. One or more sensors configured to measure gustatory and
olfactory processing; [0605] 6. One or more sensors to measure
pressure; [0606] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [0607] 8. A device to perform
electroencephalography; [0608] 9. A device to perform
electrocardiography; [0609] 10. A device to perform
electromyography; [0610] 11. A device to perform
electrooculography; [0611] 12. A device to perform
electroretinography; and [0612] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0613] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken into
account, either individually or in combination.
[0614] To probe a user's measure of fatigue, the system applies
measures of changes in comprehension, engagement or other derived
states based on these measures detailed previously. An important
tool in disambiguating the overlap of these states with general
fatigue is a consideration of time. Time of day is expected to
influence performance and the measures of fatigue, and is taken
into consideration when measures from varying times of day is
available. Duration of a session, or more generally, the duration
of a particular behavior or performance of a given task, is also
considered while measuring fatigue. Therefore, depending on time of
day and duration of task, the probability that a measure will
signal fatigue increases.
[0615] Some measures of fatigue may generally be classified as
direct measures of fatigue, and the measures that indicate a
transition to a state of fatigue (transitional measures of
fatigue).
[0616] Direct measures of fatigue may be measured independent of
baseline comparisons. In some embodiments, these are behaviors and
measures typically associated with sleepiness or transitional
states between wakefulness and sleep. Examples of direct measures
may include visible signs of fatigue and physiological measures of
fatigue.
Visible Signs
[0617] Visible signs of fatigue or sleepiness may largely be
measured by imaging of the face and head. Video eye trackers may be
used to obtain these measures, and EOG recording may also capture
some behaviors.
[0618] Nodding of the head, notably with a slow downward and rapid
upward pattern are likewise potential signals of fatigue. In an
embodiment, head nodding with a slow downward movement, followed by
rapid upward movement is considered indicative of fatigue.
[0619] Closed or partially closed eyes, especially for extended
periods, can be yet another sign of fatigue. Prolonged periods of
(mostly) closed eyes can be considered indicative of fatigue. For
example, when the proportion of time that the eyes are at least 50%
open is less than 75% (P.sub.eyes open(where P.sub.both eyes
open.gtoreq.50%)<0.75), the system considers a user to be
fatigued.
[0620] In embodiments, ocular fatigue is correlated with dry eyes.
An abnormally low tear-break-up-time can be a signal of fatigue. In
some embodiments, special imaging methods are used to measure
ocular fatigue. The system considers significant signs of dry eye
(such as low tear-break-up-time) as indicative of ocular
fatigue.
[0621] In embodiments, yawning or other pronounced and discrete
respiration is indicative of fatigue. Yawning or other isolated,
deep inhalation of air can signal a fatigued state, and may be
noted both for time and rate of occurrence.
[0622] One visible sign of transition to fatigue is determined
through eye movements. In an embodiment, the system determines
decrease in saccade velocity and magnitude, and decrease in
frequency of fixations, to be a sign of slow eye movements, and
therefore a sign of an onset of or increase in fatigue.
[0623] Also, in an embodiment, transitions to shorter and higher
frequency of blinks is considered as an indication of fatigue
onset. In this condition, user's eyes begin to close, partially or
completely, and blinking goes from the normal pattern to a series
of small, fast rhythmic blinks.
[0624] In another embodiment, sudden vertical eye movements is
considered as indicative of fatigue.
[0625] A user transitioning to a state of fatigue may display a
depth of gaze that drifts out towards infinity (zero convergence)
and eye movements that may no longer track moving stimuli or may
not respond to the appearance of stimuli. Therefore, in another
embodiment, a 3-D depth of gaze towards infinity for extended
periods is considered as indicative of fatigue.
Physiological Measures
[0626] Physiological sensors may be used to determine physiological
indication of fatigue.
[0627] In an embodiment, significant decreases in heart rate and/or
body temperature is associated with sleep, and is considered as
indication of fatigue when the user displays these signs when
awake.
[0628] In an embodiment, increased energy in low frequency EEG
signals (for example, slow-wave sleep patterns) are interpreted as
a signal of fatigue. For example, a trade-off where low frequency
(<10 Hz) EEG energy increases and high frequency (.gtoreq.10 Hz)
EEG energy decreases is an indication of fatigue.
[0629] Transitions to fatigue may be determined from changes in
behavior and other states over time, based on a significant
deviation from an established baseline. Transitional measures of
fatigue may be observed through visible signs as well as through
behavioral measures.
Behavioral Measures
[0630] An increase in time taken by the user to react to stimuli
may be considered indicative of fatigue. Additionally, measures of
precision and timing in user responses to degrade, may be
proportional to a level of fatigue.
[0631] In some embodiments, reductions in `performance` metrics
over extended periods of activity is considered as indicative of
fatigue.
[0632] In some cases, user's vigilance decreases, leading to
increasing lapse rates in responding to stimuli. In these cases,
decreasing proportion of responding, in appropriate contexts, is
considered as indicative of fatigue.
[0633] In one embodiment, the system determines significant
reductions in comprehension, engagement and other excitatory states
in the context of prolonged activity as signals of fatigue. In an
embodiment, distractibility increases with decrease in
comprehension and/or engagement, also signaling user's
disengagement from the media experience. A prolonged duration of
time may be defined based on the nature of the activity, but may
generally range from tens of minutes to hours.
Other Correlations
[0634] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral, physiological
and/or visible measures, the less ambiguous measures of fatigue
like sleepiness, and ability to characterize behavior after
prolonged periods and at certain times of day. The system can
correlate all available measures and look for trends in fatigue.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to reduce fatigue, for the user and/or a group
of users.
[0635] At 2004, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2006, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increased level of fatigue
experienced by the user of the VR, AR, and/or MxR media: [0636] 1.
Decrease in palpebral fissure rate of change [0637] 2. Low distance
palpebral fissure resting state [0638] 3. Low distance palpebral
fissure active state [0639] 4. Increased ratio of partial blinks to
full blinks [0640] 5. Decreased target relevancy for pupil initial
and final position [0641] 6. Decreased target relevancy for gaze
direction [0642] 7. Decreased target relevancy for gaze initial and
final position [0643] 8. Increased rate of divergence [0644] 9.
Decreased relevancy for fixation initial and final position [0645]
10. Increased fixation duration [0646] 11. Decreased target
relevancy for saccade initial and final position [0647] 12.
Decreased target relevancy for saccade angle [0648] 13. Decreased
saccade magnitude (distance of saccade) [0649] 14. Increased ratio
of anti-saccade/pro-saccade [0650] 15. Increased smooth pursuit
[0651] 16. Increased screen distance [0652] 17. Decreased target
relevant head direction [0653] 18. Decreased target relevant head
fixation [0654] 19. Decreased target relevant limb movement [0655]
20. Decreased alpha/delta brain wave ratio [0656] 21. Low oxygen
saturation [0657] 22. Changes in blood pressure [0658] 23.
Increased reaction time
[0659] The system may determine a user is experiencing reduced
levels of fatigue while interacting with the media in a VR, AR,
and/or MxR environment based upon the following changes: [0660] 1.
Increased blink rate [0661] 2. Increased rate of change for blink
rate [0662] 3. Increased rate of change for pupil size [0663] 4.
Increased rate of convergence [0664] 5. Increased fixation duration
rate of change [0665] 6. Increased fixation rate [0666] 7.
Increased fixation count [0667] 8. Increased inhibition of return
[0668] 9. Increased saccade velocity [0669] 10. Increased saccade
rate of change [0670] 11. Increased saccade count (number of
saccades) [0671] 12. Shift in weight distribution [0672] 13.
Increased alpha/theta brain wave ratio [0673] 14. Increased body
temperature [0674] 15. Increased respiration rate [0675] 16.
Increased heart rate [0676] 17. Increased vocalizations
[0677] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[0678] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease fatigue, these lists
are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[0679] At 2008, the changes in the plurality of data determined
over time may be used to determine a degree of change in levels of
fatigue of the user. The change in fatigue levels may indicate
either enhanced fatigue or reduced fatigue.
[0680] At 2010, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) fatigue. In
embodiments, the media may be modified to address all the changes
in data that reflect increased fatigue. In embodiments, a
combination of one or more of the following modifications may be
performed: [0681] 1. Increasing a contrast of the media [0682] 2.
Making an object of interest that is displayed in the media larger
in size [0683] 3. Increasing a brightness of the media [0684] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [0685] 5. Changing a
focal point of content displayed in the media to a more central
location [0686] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [0687] 7. Increasing an
amount of color in said media [0688] 8. Increasing a degree of
shade in objects shown in said media [0689] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[0690] One or more of the following indicators may be observed to
affirm a decrease in fatigue levels: increase in palpebral fissure
height; decreased ratio of partial blinks to full blinks; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; decreased rate of divergence;
increased relevancy for fixation initial and final position;
decreased fixation rate; decreased fixation count; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude; decreased
ratio of anti-saccade/pro-saccade; decreased smooth pursuit;
decreased screen distance; increased target relevant head
direction; increased target relevant head fixation; increased
target relevant limb movement; decreased shift in weight
distribution; increased alpha/delta brain wave ratio; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal blood pressure; target relevant facial expressions; task
relevant gustatory processing; task relevant olfactory processing;
and task relevant auditory processing.
[0691] In embodiments, a specific percentage or a range of increase
in fatigue may be defined. In embodiments, an additional value for
data may be acquired at 2012, in order to further determine change
in data over time at 2014, after the modifications have been
executed at 2010. At 2016, a new degree/percentage/range of
increase in levels of fatigue may be acquired. At 2018, the system
determines whether the increase in levels of fatigue is within the
specified range or percentage. If it is determine that the increase
is greater that the specified range, the system may loop back to
step 2010 to further modify the media. Therefore, the media may be
iteratively modified 2010 and levels of fatigue may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[0692] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[0693] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[0694] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[0695] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 3: Modifying Media in Order to Improve a User's
Engagement with Information in that Media
[0696] In one embodiment, a user's degree of engagement is derived
from a number of measures. User's engagement may be determined as a
binary state--whether or not the user is engaging. The binary
engagement of the user with a particular application or task can be
directly measured by their responding, or lack thereof, to events.
Further, measures of comprehension and fatigue detailed above can
be used to indicate such engagement as a prerequisite of
comprehension and/or fatigue. Behavior oriented away from an
application, device and/or task towards other things in the
environment (i.e. engagement with something else) can also signal
whether user is engaged with the application, device and/or task. A
user's level of engagement can be derived by observing the user's
focused attention, as opposed to divided attention, and measures of
time-on-task. Users' behaviors that enhance their perception of
stimuli may also indicate enhanced engagement. The behaviors that
indicate an enhancement in perception of stimuli, may include
leaning in, slowing blink rate, eye gaze vergence signaling focus
at the appropriate depth of field, among others.
[0697] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment or a
conventional laptop, mobile phone, desktop or tablet computing
environment, in order to improve that user's engagement with
information being communicated by that media. FIG. 21 illustrates a
flow chart describing an exemplary process for modifying media in
order to improve engagement, in accordance with some embodiments of
the present specification. At 2102, a first value for a plurality
of data, as further described below, is acquired. In embodiments,
data is acquired by using at least one camera configured to acquire
eye movement data (rapid scanning and/or saccadic movement), blink
rate data, fixation data, pupillary diameter, palpebral (eyelid)
fissure distance between the eyelids. Additionally, the VR, AR,
and/or MxR device can include one or more of the following sensors
incorporated therein: [0698] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [0699] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [0700] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [0701]
4. One or more sensors configured to measure auditory processing;
[0702] 5. One or more sensors configured to measure gustatory and
olfactory processing; [0703] 6. One or more sensors to measure
pressure; [0704] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [0705] 8. A device to perform
electroencephalography; [0706] 9. A device to perform
electrocardiography; [0707] 10. A device to perform
electromyography; [0708] 11. A device to perform
electrooculography; [0709] 12. A device to perform
electroretinography; and [0710] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0711] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken into
account, either individually or in combination.
[0712] To probe a user's engagement with some spatially defined
process in media, the system applies measures of where the user is
looking, measuring the proportion of time the user spends looking
at relevant areas within the media vs. irrelevant areas within the
media.
[0713] In one embodiment, any measure signaling significant
comprehension, or onset of comprehension, as is used to determine a
signal of engagement with a task. Measures of comprehension can be
used to indicate such engagement as a prerequisite of
comprehension.
Engagement as a Binary State
[0714] Assigning a binary state, as a function of time, as to
whether or not a user is engaged with an application may depend on
less ambiguous cues. Measures (or cues) that indicate user
engagement in a binary state may include the following:
[0715] 1. Discrete, Conscious Responding
[0716] In an embodiment, a response rate of less than 100%, or
another lower, baseline rate of responding, is determined to be an
indication of an unengaged state. The response rate may depend on a
context, such as but not limited to a situation where a brief
duration of time is allowed for response that users may miss. In
some embodiments, a user may be expected to respond through a
manual interaction such as a mouse click or a screen touch. The
system notes whether the user responded at any point in time when
the user was expected to do so. If the user fails to respond then
it is likely that the user is not engaged with the application.
[0717] In some embodiments, rapid changes in performance are noted
by the system through a total percentage of correct responses
provided by the users. The rapid changes in performance may signal
engagement (with increase in rapid changes in performance) or
disengagement (with decrease in rapid changes in performance). The
system may exclude other causes for such performance changes,
including but not limited to--little to no change in difficulty,
and discounting learning/training effects for performance
improvements. Therefore, a significant upward or downward deviation
from average percent of correct responding is considered signaling
engagement or disengagement, respectively. The system may determine
if the user is engaging, not engaging, from media content in a
VR/AR/MxR environment based upon presence or absence of responsive
behavior.
[0718] 2. Distraction
[0719] In an embodiment, distracted behavior can signal
disengagement, or a shift of engagement away from one thing and
towards another. Device inputs not related to the task at hand
indicate onset and, potentially, duration of disengagement.
Orienting of head or eyes away from an application, device or task
likewise may indicate disengagement. Returning to the application,
device or task may signal re-engagement. The system determines
interactions away from a particular task or stimulus as indicating
lack of engagement, or disengagement. The system determines
disengagement due to distraction through measures of user attention
based on body and eye tracking measures indicating that user is
oriented away from the media. The system may determine if the user
is engaging, not engaging, or disengaging, from media content in a
VR/AR/MxR environment based upon distracted behavior.
Level of Engagement
[0720] A user's level of engagement further measures along a
continuous dimension the degree to which users are engaged with an
application, device or task to the exclusion of anything else. This
relative measure may be normalized to the range of values recorded.
In an example, fixation duration on the current item compared to
the distribution of fixation durations is used as relative measure.
Alternatively, in the presence of a distracting stimuli, the ratio
of time or effort spent on one item versus another, is used to
determine a level of engagement.
[0721] 1. Time-on-Task
[0722] In one embodiment, user interactions are measured with more
than one application or task. In embodiments, in such cases, the
level of engagement with any application or task is taken as the
ratio of time spent interacting with it compared to time spent not
interacting with it. Therefore, the system may determine relative
time-on-task as the proportion of time spent performing a task or
processing a stimulus compared to the time not spent performing the
task or processing the stimulus. Engagement is proportional to the
value of relative time-on-tasks. The system determines greater
engagement with tasks or applications where the user spends
relatively greater time performing them. The system may determine
if the user is engaging, not engaging, or disengaging, from media
content in a VR/AR/MxR environment based upon relative
time-on-tasks.
[0723] In an embodiment, users switch between applications or
tasks, and responses are recorded for each application or task. In
this case, the system uses the number and/or duration of
interactions with each application/task to determine the level of
engagement with them. Therefore, the system determines the ratio of
interactions among available tasks as indicative of time-on-task
for each as a relative measure of engagement with each task.
[0724] In an embodiment, the system performs eye tracking. In
embodiments, the ratio of fixation count and/or duration between an
application, device or task, and anything outside of it is used as
a measure of level of engagement. Therefore, the system determines
the ratio of fixation count and/or duration among stimuli and/or
visual regions as indicative of time-on-task as a relative measure
of engagement with each stimulus or visual region.
[0725] 2. Enhancing Perception
[0726] Some behaviors allow users to better perceive stimuli and
can signal their level of engagement with them. In some
embodiments, face and/or body tracking is used by the system to
measure when and the extent to which users lean towards a display
or, while held, bring the display closer to their face. In an
embodiment, the system determines significant shortening of the
distance between a visual stimulus and the user's eyes as an
indication of onset of engagement, and a proportional deviation
from a baseline as an indication of level of engagement. The system
may determine a level of user's engagement with media content in a
VR/AR/MxR environment based upon significant shortening of the
distance between a visual stimulus and the user's eyes.
[0727] In an embodiment, tracking of gaze direction of both eyes is
used to measure the extent that users are converging their gaze
position at the appropriate depth to view stimuli (this is on top
of whether gaze is in the appropriate direction). In an embodiment,
the system determines adjustment of 3-D gaze position towards the
appropriate depth (here considered separately from direction of
gaze) to view a stimulus as a signal of engagement with that
stimulus. The system may determine a level of user's engagement
with stimuli in media content in a VR/AR/MxR environment based upon
adjustment of 3-D gaze position towards the appropriate depth to
view the stimuli.
[0728] In some embodiments, a relative measure of engagement is
indicated when user maintains more rigid or steady fixation on a
stimulus, which can aid in spotting subtle changes. In an
embodiment, the system determines rigid fixation in the context of
monitoring for subtle changes or motion, or the precise onset of
any change or motion, as indicative of engagement. The system may
determine a level of user's engagement with stimuli in media
content in a VR/AR/MxR environment based upon rigid fixation in the
context of monitoring for subtle changes or motion.
[0729] Greater sampling near and around a stimulus may indicate
increasing engagement as a user studies the details of the
stimulus. In an embodiment, the system determines a level of
engagement based on an Area of Focus (also described in context of
Tomprehension'), where the area of focus is correlated with the
spatial extent of the stimulus in question. The system may
determine a level of user's engagement with stimuli in media
content in a VR/AR/MxR environment based upon the user's area of
focus within the media.
[0730] In an embodiment, the system determines a blink rate that is
significantly less than a baseline, as indicative of engagement
with an ongoing task. Alternatively, the system determines given
eye gaze position estimation of the user within a fixated region,
to be an indication of a level of engagement with the ongoing task.
A decrease in blink rate can indicate increasing level of
engagement. The system may determine a level of user's engagement
with media content in a VR/AR/MxR environment based upon the user's
blink rate and/or eye gaze position.
[0731] Sometimes a user may hold breathing to reduce body motion in
order to focus on the media, for monitoring for subtle changes in
stimuli within the media. In an embodiment, the system determines
reduced or held respiration in the context of monitoring as
indicative of engagement. The system may determine a level of
user's engagement with media content in a VR/AR/MxR environment or
a conventional laptop, mobile phone, desktop or tablet computing
environment based on user's breathing while monitoring for changes
in a stimuli within the media.
[0732] 3. Preference
[0733] A user may prefer some object(s) over the other when two or
more alternatives are presented within a media. Even with only one
object of interest, given appropriate sources of data, the system
may determine following measures in the context of comparing the
object of interest with everywhere else in the user's immediate
environment, to derive a level of engagement.
[0734] In addition to generally looking more at objects for which
users have preference, some other eye tracking measures can be used
to estimate preference and, by extension, engagement. In an
embodiment, the system predicts that, just before making a choice,
a user's last fixation is on the item of choice. Therefore, the
system determines that when a choice is made by a user, the
duration of last fixation on the selected stimulus of choice, is
defined as proportional to level of engagement of the user with
that choice and with the selected task. The choice, in this case,
is tied to a following selection, and eliminates cases of the
instance of `choice` that itself does not indicate a preference
(for example, the user may choose to continue without an explicit
selection). Alternatively, at the beginning of the decision making
process, the duration of the first fixation is correlated with the
ultimate selection. The system may determine a level of user's
engagement with media content in a VR/AR/MxR environment or a
conventional laptop, mobile phone, desktop or tablet computing
environment based on the duration of first fixation on any stimulus
when a choice is made by the user, where the duration is determined
to be proportional to level of engagement with the selected
task.
[0735] In embodiments, broader patterns of eye gaze can reveal
choices before they are made, and patterns of eye gaze can
influence choice even when stimuli are not present. Also, in
embodiments the system uses preference for features within objects
to predict preference for novel objects with similar features.
[0736] In addition to gaze direction, other measures of
preference/engagement may be made based on other eye tracking data.
In an embodiment, the system determines pupil dilation will
increase during decision making in favor of a task of choice, to
measure user's level of engagement. The system may determine a
level of user's engagement with media content in a VR/AR/MxR
environment or a conventional laptop, mobile phone, desktop or
tablet computing environment based on the pupil dilation observed
from the eye tracking data, while making a decision.
[0737] Another measure of preference/engagement may be derived from
blinking. While it has been discussed that blinking is inhibited
when the user is engaged with visual stimuli from very early in
development, the system may also determine increased blinking,
along with fewer fixations on task-relevant areas, to be associated
with disengagement. The disengagement may also be measured by
observing subsequent errors, post the significant increase in
blinking. The system may determine a level of user's disengagement
with media content in a VR/AR/MxR environment or a conventional
laptop, mobile phone, desktop or tablet computing environment based
on a significant increase in blinking of the eyes.
[0738] In addition to event-related signals mentioned previously in
the context of comprehension that may indicate attention to
stimuli, the system may determine some more generalized measures
that can indicate decision making and/or choice. Such measures can
be assumed to be proportional to engagement in certain contexts. In
an embodiment, the system determines increased bilateral phase
synchrony of EEG activity during choice tasks as indicative of
increased level of engagement with the task. The system may
determine a level of user's engagement with media content in a
VR/AR/MxR environment or a conventional laptop, mobile phone,
desktop or tablet computing environment based on
electrophysiological measurements such as EEG.
[0739] In addition to EEG, other physiological and autonomic
measures may be used by the system to determine a level of
engagement. In an embodiment, the system determines an increased
level of engagement to be proportional to an increase in heart
rate. Similar changes in blood pressure, oxygen saturation, and
respiration rate may be used by the system, along with changes in
skin conductance (GSR). Therefore, in embodiments, the system
determines an increase in autonomic arousal as indicative of
increasing engagement, and decreases in arousal as disengagement.
The system may determine a level of user's engagement with media
content in a VR/AR/MxR environment or a conventional laptop, mobile
phone, desktop or tablet computing environment based on changes in
autonomic arousal.
Other Correlations
[0740] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous engagement signs like consistent interaction,
proportionately high time-on-task, and perceptually enhancing
behaviors. In some examples, some of the measures described above
are context specific and may be more or less robust or even signal
the opposite of what is expected. Measures with significant
correlations with the less ambiguous signals of engagement may
therefore become less ambiguous themselves and become new ways of
identifying engagement. Accordingly, the media presented in a
VR/AR/MxR environment is modified, in order to increase its
engagement factor, for the user and/or a group of users.
[0741] At 2104, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2106, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increased engagement of the
user with the VR, AR, and/or MxR media: [0742] 1. Decreased
palpebral fissure rate of change [0743] 2. Increased rate of change
for blink rate [0744] 3. Increased rate of change for pupil size
[0745] 4. Increased rate of convergence [0746] 5. Increased rate of
divergence [0747] 6. Increased fixation duration rate of change
[0748] 7. Increased fixation rate [0749] 8. Increased fixation
count [0750] 9. Increased inhibition of return [0751] 10. Increased
saccade velocity [0752] 11. Increased saccade rate of change [0753]
12. Increased saccade count (number of saccades) [0754] 13.
Increased smooth pursuit [0755] 14. Increased alpha/theta brain
wave ratio
[0756] The system may determine a user has reduced level of
engagement with the media in a VR, AR, and/or MX environment based
upon one or more of the following changes: [0757] 1. Low distance
palpebral fissure resting state [0758] 2. Low distance palpebral
fissure active state [0759] 3. Increased blink rate [0760] 4.
Increased ratio of partial blinks to full blinks [0761] 5.
Decreased target relevancy for pupil initial and final position
[0762] 6. Decreased target relevancy for gaze direction [0763] 7.
Decreased target relevancy for gaze initial and final position
[0764] 8. Decreased relevancy for fixation initial and final
position [0765] 9. Reduced fixation duration [0766] 10. Decreased
target relevancy for saccade initial and final position [0767] 11.
Decreased target relevancy for saccade angle [0768] 12. Decreased
saccade magnitude (distance of saccade), depending on the task
[0769] 13. Increased ratio of anti-saccade/pro-saccade [0770] 14.
Increased screen distance [0771] 15. Decreased target relevant head
direction [0772] 16. Decreased target relevant head fixation [0773]
17. Decreased target relevant limb movement [0774] 18. Shift in
weight distribution [0775] 19. Decreased alpha/delta brain wave
ratio [0776] 20. Increased body temperature [0777] 21. Increased
respiration rate [0778] 22. Low oxygen saturation [0779] 23.
Increased heart rate [0780] 24. Low blood pressure [0781] 25.
Increased vocalizations
[0782] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[0783] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to improve engagement, these
lists are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[0784] At 2108, the changes in the plurality of data determined
over time may be used to determine a degree of change in engagement
levels of the user. The change in engagement levels may indicate
either enhanced engagement or reduced engagement.
[0785] At 2110, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) engagement. In
embodiments, the media may be modified to address all the changes
in data that reflect decrease in engagement. In embodiments, a
combination of one or more of the following modifications may be
performed: [0786] 1. Increasing a contrast of the media [0787] 2.
Making an object of interest that is displayed in the media larger
in size [0788] 3. Increasing a brightness of the media [0789] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [0790] 5. Changing a
focal point of content displayed in the media to a more central
location [0791] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [0792] 7. Increasing an
amount of color in said media [0793] 8. Increasing a degree of
shade in objects shown in said media [0794] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[0795] One or more of the following indicators may be observed to
affirm an improvement in comprehension: increase in palpebral
fissure height; decrease in blink rate; decreased ratio of partial
blinks to full blinks; increased target relevancy for pupil initial
and final position; increased target relevancy for gaze direction;
increased target relevancy for gaze initial and final position;
increased relevancy for fixation initial and final position;
decreased fixation duration depending on task; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude based on
task; decreased ratio of anti-saccade/pro-saccade; decreased screen
distance; increased target relevant head direction; increased
target relevant head fixation; increased target relevant limb
movement; decrease in shifts of weight distribution; increased
alpha/delta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; task relevant
facial expressions; decreased reaction time; task relevant
gustatory processing; task relevant olfactory processing; and task
relevant auditory processing.
[0796] In embodiments, a specific percentage or a range of
improvement in engagement may be defined. In embodiments, an
additional value for data may be acquired at 2112, in order to
further determine change in data over time at 2114, after the
modifications have been executed at 2110. At 2116, a new
degree/percentage/range of improvement in engagement may be
acquired. At 2118, the system determines whether the improvement in
engagement is within the specified range or percentage. If it is
determine that the improvement is insufficient, the system may loop
back to step 2110 to further modify the media. Therefore, the media
may be iteratively modified 2110 and engagement may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[0797] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[0798] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[0799] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[0800] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 4: Modifying Media in Order to Improve a User's Overall
Performance while Interacting with that Media
[0801] In an embodiment, user's overall performance is measured,
and a media is modified in order to improve the performance of the
user. The performance of a user may be determined in the form of
user's vision performance, ability to comprehend, engagement
levels, fatigue, and various other parameters, in combination,
which directly or indirectly affect the overall performance of the
user while interacting with the media including media in a
VR/AR/MxR environment or a conventional laptop, mobile phone,
desktop or tablet computing environment.
[0802] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, may be processed to determine
the overall performance of the user. The data may indicate a level
of performance of assessed during user's interaction with the
media. The data may be further utilized to modify VR/AR/MxR media
for the user in order to optimize overall performance, such as but
not limited to by minimizing visual, or any other discomfort
arising from the media experience. In an embodiment, media is
modified in real time for the user. In another embodiment, data is
saved and used to modify presentation of VR/AR/MxR media or
conventional laptop, mobile phone, desktop or tablet computing
media to subsequent users with a similar data, or subsequently to
the user.
[0803] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to improve that user's overall performance while interacting
with that media. FIG. 22 illustrates a flow chart describing an
exemplary process for modifying media in order to improve overall
performance, in accordance with some embodiments of the present
specification. At 2202, a first value for a plurality of data, as
further described below, is acquired. In embodiments, data is
acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids.
[0804] Additionally, the VR, AR, and/or MxR device can include one
or more of the following sensors incorporated therein: [0805] 1.
One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [0806] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[0807] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [0808] 4. One or more sensors configured
to measure auditory processing; [0809] 5. One or more sensors
configured to measure gustatory and olfactory processing; [0810] 6.
One or more sensors to measure pressure; [0811] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [0812] 8.
A device to perform electroencephalography; [0813] 9. A device to
perform electrocardiography; [0814] 10. A device to perform
electromyography; [0815] 11. A device to perform
electrooculography; [0816] 12. A device to perform
electroretinography; and [0817] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0818] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken into
account, either individually or in combination.
[0819] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous performance measures. In some examples, some of the
measures described above are context specific. The system can
correlate all available measures and look for trends in overall
performance. Accordingly, the media presented in a VR/AR/MxR
environment or a conventional laptop, mobile phone, desktop or
tablet computing environment is modified, in order to improve or
optimize performance, for the user and/or a group of users.
[0820] At 2204, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2206, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect overall improvement of the
user's performance while interacting with the VR, AR, and/or MxR
media: [0821] 1. Decreased Palpebral Fissure Rate of Change [0822]
2. Increased Rate of Change for Pupil Size [0823] 3. Increased Rate
of Convergence [0824] 4. Increased Rate of Divergence [0825] 5.
Increased Fixation Duration Rate of Change [0826] 6. Increased
Fixation Rate [0827] 7. Increased Fixation Count [0828] 8.
Increased Saccade Velocity [0829] 9. Increased Saccade Count
(Number of Saccades) [0830] 10. Increased Smooth Pursuit
[0831] The system may determine a user has a reduced overall
performance levels while interacting with the media in a VR, AR,
and/or MxR environment based upon one or more of the following
changes: [0832] 1. Low Distance Palpebral Fissure Resting State
[0833] 2. Low Distance Palpebral Fissure Active State [0834] 3.
Increased Blink Rate [0835] 4. Increased Rate of Change for Blink
Rate [0836] 5. Increased Ratio of Partial Blinks to Full Blinks
[0837] 6. Decreased Target Relevancy for Pupil Initial and Final
Position [0838] 7. Decreased Target Relevancy for Gaze Direction
[0839] 8. Decreased Target Relevancy for Gaze Initial and Final
Position [0840] 9. Decreased Relevancy for Fixation Initial and
Final Position [0841] 10. Changes in Fixation Duration, based on
the context [0842] 11. Decreased Target Relevancy for Saccade
Initial and Final Position [0843] 12. Decreased Target Relevancy
for Saccade Angle [0844] 13. Decreased Saccade Magnitude (Distance
of Saccade) [0845] 14. Increased Ratio of Anti-Saccade/Pro-Saccade
[0846] 15. Increased Inhibition of Return [0847] 16. Increased
Saccade Rate of Change [0848] 17. Increased Screen Distance [0849]
18. Decreased Target Relevant Head Direction [0850] 19. Decreased
Target Relevant Head Fixation [0851] 20. Decreased Target Relevant
Limb Movement [0852] 21. Shift in Weight Distribution [0853] 22.
Decreased Alpha/Delta Brain Wave ratio [0854] 23. Increased
Alpha/Theta Brain Wave ratio [0855] 24. Increased Body Temperature
[0856] 25. Increased Respiration Rate [0857] 26. Low Oxygen
Saturation [0858] 27. Increased Heart Rate [0859] 28. Low Blood
Pressure [0860] 29. Increased Vocalizations [0861] 30. Increased
Reaction Time
[0862] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; and change in olfactory
processing.
[0863] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to improve overall user
performance, these lists are not exhaustive and may include other
data acquisition components, types of data, and changes in
data.
[0864] At 2208, the changes in the plurality of data determined
over time may be used to determine a degree of change in overall
performance levels of the user. The change in overall performance
levels may indicate either enhanced performance or reduced
performance.
[0865] At 2210, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) performance. In
embodiments, the media may be modified to address all the changes
in data that reflect reduced performance. In embodiments, a
combination of one or more of the following modifications may be
performed: [0866] 1. Increasing a contrast of the media [0867] 2.
Making an object of interest that is displayed in the media larger
in size [0868] 3. Increasing a brightness of the media [0869] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [0870] 5. Changing a
focal point of content displayed in the media to a more central
location [0871] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [0872] 7. Increasing an
amount of color in said media [0873] 8. Increasing a degree of
shade in objects shown in said media [0874] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[0875] One or more of the following indicators may be observed to
affirm an improvement in comprehension increase in palpebral
fissure height; decrease in blink rate; decreased rate of change
for blink rate; increased target relevant pupil initial and final
position; increased target relevant gaze direction; increased
target relevant gaze initial and final position; increased target
relevant fixation initial and final position; decreased fixation
duration; increased target relevancy for saccade initial and final
position; increased target relevancy for saccade angle; increased
saccade magnitude (task relevant); decreased ratio of
anti-saccade/pro-saccade; decreased inhibition of return; decreased
saccade rate of change; decreased screen distance; increased target
relevant head direction; increased target relevant head fixation;
increased target relevant limb movement; decrease in shifts of
weight distribution; increased alpha/delta brain wave ratio; normal
body temperature; normal respiration rate; 90-100% oxygen
saturation; normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; and task relevant auditory processing.
[0876] In embodiments, a specific percentage or a range of
improvement in overall performance may be defined. In embodiments,
an additional value for data may be acquired at 2212, in order to
further determine change in data over time at 2214, after the
modifications have been executed at 2210. At 2216, a new
degree/percentage/range of improvement in overall performance may
be acquired. At 2218, the system determines whether the improvement
in overall performance is within the specified range or percentage.
If it is determine that the improvement is insufficient, the system
may loop back to step 2210 to further modify the media. Therefore,
the media may be iteratively modified 2210 and overall performance
may be measured, until a percentage of improvement of anywhere from
1% to 10000%, or any increment therein, is achieved.
[0877] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[0878] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[0879] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[0880] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 5: Modifying Media in Order to Decrease Symptoms
Associated with Visually-Induced Motion Sickness Secondary to
Visual-Vestibular Mismatch
[0881] In an embodiment, user's symptoms of Visually-Induced Motion
Sickness (VIMS) secondary to visual-vestibular mismatch, is
measured, and media is modified in order to decrease the symptoms
for the user.
[0882] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, may be processed to determine
the symptoms of VIMS, experienced by the user. The data may
indicate a level of symptoms shown during or after user's
interaction with the media. The data may be further utilized to
modify VR/AR/MxR media for the user in order to decrease the VIMS
symptoms, such as but not limited to by minimizing visual, or any
other discomfort arising from the media experience. In an
embodiment, media is modified in real time for the user. In another
embodiment, data is saved and used to modify presentation of
VR/AR/MxR media to subsequent users with a similar data, or
subsequently to the user.
[0883] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease user's symptoms of VIMS secondary to
visual-vestibular mismatch, during or after interaction with that
media. FIG. 23 illustrates a flow chart describing an exemplary
process for modifying media in order to decrease symptoms of VIMS
secondary to visual-vestibular mismatch, in accordance with some
embodiments of the present specification. At 2302, a first value
for a plurality of data, as further described below, is acquired.
In embodiments, data is acquired by using at least one camera
configured to acquire eye movement data (rapid scanning and/or
saccadic movement), blink rate data, fixation data, pupillary
diameter, palpebral (eyelid) fissure distance between the eyelids.
Additionally, the VR, AR, and/or MxR device can include one or more
of the following sensors incorporated therein: [0884] 1. One or
more sensors configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, and/or body
velocity; [0885] 2. One or more sensors configured to measure limb
movement, limb rotation, limb direction, and/or limb velocity;
[0886] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [0887] 4. One or more sensors configured
to measure auditory processing; [0888] 5. One or more sensors
configured to measure gustatory and olfactory processing; [0889] 6.
One or more sensors to measure pressure; [0890] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [0891] 8.
A device to perform electroencephalography; [0892] 9. A device to
perform electrocardiography; [0893] 10. A device to perform
electromyography; [0894] 11. A device to perform
electrooculography; [0895] 12. A device to perform
electroretinography; and [0896] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0897] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken into
account, either individually or in combination.
[0898] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous symptomatic measures. In some examples, some of the
measures described above are context specific. The system can
correlate all available measures and look for trends in overall
display of VIMS symptoms. Accordingly, the media presented in a
VR/AR/MxR environment is modified, in order to decrease the VIMS
symptoms secondary to visual-vestibular mismatch, for the user
and/or a group of users.
[0899] At 2304, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2306, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in VIMS symptoms while
interacting with the VR, AR, and/or MxR media: [0900] 1. Decreased
Palpebral Fissure Rate of Change [0901] 2. Low Distance Palpebral
Fissure Resting State [0902] 3. Low Distance Palpebral Fissure
Active State [0903] 4. Increased Ratio of Partial Blinks to Full
Blinks [0904] 5. Decreased Target Relevancy for Pupil Initial and
Final Position [0905] 6. Decreased Target Relevancy for Gaze
Direction [0906] 7. Decreased Target Relevancy for Gaze Initial and
Final Position [0907] 8. Decreased Relevancy for Fixation Initial
and Final Position [0908] 9. Changes in Fixation Duration or
Increased Convergence Duration [0909] 10. Decreased Target
Relevancy for Saccade Angle [0910] 11. Decreased Saccade Magnitude
(Distance of Saccade) [0911] 12. Increased Ratio of
Anti-Saccade/Pro-Saccade [0912] 13. Increased Inhibition of Return
[0913] 14. Increased Smooth Pursuit [0914] 15. Increased Screen
Distance [0915] 16. Decreased Target Relevant Head Direction [0916]
17. Decreased Target Relevant Head Fixation [0917] 18. Decreased
Target Relevant Limb Movement [0918] 19. Shift in Weight
Distribution [0919] 20. Decreased Alpha/Delta Brain Wave ratio
[0920] 21. Increased Body Temperature [0921] 22. Increased
Respiration Rate [0922] 23. Low Oxygen Saturation [0923] 24.
Increased Heart Rate [0924] 25. Changes in Blood Pressure [0925]
26. Decreased Reaction Time
[0926] The system may determine decrease in VIMS symptoms for a
user while interacting with the media in a VR, AR, and/or MX
environment based upon one or more of the following changes: [0927]
1. Increased Blink Rate [0928] 2. Increased Rate of Change for
Blink Rate [0929] 3. Increased Rate of Change for Pupil Size [0930]
4. Increased Rate of Convergence [0931] 5. Increased Rate of
Divergence [0932] 6. Increased Fixation Duration Rate of Change
[0933] 7. Increased Fixation Rate [0934] 8. Increased Fixation
Count [0935] 9. Decreased Target Relevancy for Saccade Initial and
Final Position [0936] 10. Increased Saccade Velocity [0937] 11.
Increased Saccade Rate of Change [0938] 12. Increased Saccade Count
(Number of Saccades) [0939] 13. Increased Alpha/Theta Brain Wave
ratio [0940] 14. Increased Vocalizations
[0941] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[0942] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease user's symptoms of
VIMS, these lists are not exhaustive and may include other data
acquisition components, types of data, and changes in data.
[0943] At 2308, the changes in the plurality of data determined
over time may be used to determine a degree of change in user's
VIMS symptoms. The change in VIMS symptoms may indicate either
reduced symptoms or enhanced symptoms.
[0944] At 2310, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) symptoms. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in VIMS symptoms. In embodiments, a
combination of one or more of the following modifications may be
performed: [0945] 1. Increasing a contrast of the media [0946] 2.
Making an object of interest that is displayed in the media larger
in size [0947] 3. Increasing a brightness of the media [0948] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [0949] 5. Changing a
focal point of content displayed in the media to a more central
location [0950] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [0951] 7. Increasing an
amount of color in said media [0952] 8. Increasing a degree of
shade in objects shown in said media [0953] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[0954] One or more of the following indicators may be observed to
affirm a decrease in VIMS symptoms: increased palpebral fissure
height; decreased ratio of partial blinks to full blinks; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; increased relevancy for fixation
initial and final position; decreased fixation duration; increased
target relevancy for saccade initial and final position; increased
target relevancy for saccade angle; increased saccade magnitude
(task relevant); decreased ratio of anti-saccade/pro-saccade;
decreased inhibition of return; decreased smooth pursuit; decreased
screen distance; increased target relevant head direction;
increased target relevant head fixation; increased target relevant
limb movement; decrease in shifts of weight distribution; increased
alpha/delta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; task relevant
facial expressions; decreased reaction time; task relevant
gustatory processing; task relevant olfactory processing; and task
relevant auditory processing.
[0955] In embodiments, a specific percentage or a range of decrease
in symptoms associated with visually-induced motion sickness
secondary to visual-vestibular mismatch, may be defined. In
embodiments, an additional value for data may be acquired at 2312,
in order to further determine change in data over time at 2314,
after the modifications have been executed at 2310. At 2316, a new
degree/percentage/range of decrease in symptoms associated with
visually-induced motion sickness secondary to visual-vestibular
mismatch may be acquired. At 2318, the system determines whether
the decrease in symptoms associated with visually-induced motion
sickness secondary to visual-vestibular mismatch is within the
specified range or percentage. If it is determine that the decrease
is insufficient, the system may loop back to step 2310 to further
modify the media. Therefore, the media may be iteratively modified
2310 and overall performance may be measured, until a percentage of
improvement of anywhere from 1% to 10000%, or any increment
therein, is achieved.
[0956] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[0957] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[0958] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[0959] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 7: Modifying Media in Order to Decrease Symptoms
Associated with Post Traumatic Stress Disorder (PTSD)
[0960] Post-Traumatic Stress Disorder (PTSD) is a condition that is
developed by an individual after they are exposed to a traumatic
event. In embodiments of the present specification, the SDEP allows
for collection of biometric information from hardware and software
sources, utilizes machine and deep learning techniques, combined
with image processing and machine learning to understand how
multiple sensory and physiologic inputs and outputs affect both
visual and human behavior, in conditions such as PTSD. Through
these learnings, one may understand a person at a deeply intimate
neurophysiological state. The learnings may be utilized to modify
media in order to address a user's symptoms associated with PTSD.
In further embodiments, the learnings are used to modulate light
stimuli through HMDs in order to allow/enable perceiving
information by the human body through neurophysiologic+/-electronic
stimulation, such as through neuro-programming. In embodiments of
the present specification, the communication is modulated through
neurophysiologic+/-electronic stimulation, through the use of
direct, reflected, diffracted, refracted light/sound, with both
amplitude, depth, area, and frequency. Additionally, the
communication is modulated through neurophysiologic+/-electronic
stimulation+/-chemical stimulation, through the use of direct,
indirect touch/taste/smell, with both amplitude, frequency, depth
and area.
[0961] In embodiments of the present specification, the direction,
location, amplitude, frequency, depth, pattern, and combination of
these light fields, based on the same principles as stated with
bio-electronic implants, allow for stimulation of certain visual
and accessor visual channels of the retino-geniculo-cortical
system, allowing in part for activation and encoding of different
aspects of vision, including but not limited to stereopsis (depth),
color, contrast, size discrimination, object/face recognition,
border detection, oculomotor function, pupillary function, field of
view, visual memory, for neuroplasticity by bypassing injured
channels in favor of intact channels and/or developing new
channels, and for neuro-programming for therapeutic approaches for
the neural, cardiac, auditory, olfactory, tactile, gustatory,
muscular, endocrine (hormone regulation--e.g. retinal ganglion cell
subtype stimulation for circadian rhythm reset), metabolic, immune,
psychology/psychiatric systems.
[0962] Embodiments of the present specification are also applicable
to different types of vision implants, such as and not limited to
PRIMA, IRIS I and IRIS II, and other vision implants that may be
fixed under or outside the retina.
[0963] In an embodiment, an individual with PTSD interfaces with
the system to understand how multiple sensory and physiologic
inputs and outputs affect both visual and human behavior of the
individual. In the example, SDEP database may be utilized to
develop a benchmark for PTSD, including increased saccadic eye
movements, pupillary dilation, color sensitivity to longer
wavelengths of color/red RGB, increased heart rate, increased basal
body temperature, auditory sensitivity to elevated intensity levels
in binaural states, and increased sensitivity to patterns in
images/videos/scenes with high RGB, decreased background/foreground
luminance with multiple object recognition requirements. In an
example, increase in anti-saccadic error with minimal pupillary
reactivity and increased sensitivity to blue light (RGB of 0,0,1),
with increased heart rate >100 bpm, and basal body temperature
greater than 98.6 degrees F., may be benchmarked as PTSD related
anxiety. Based on the data, presentation of therapeutics through
the use of titrated visual and non-visual stimuli through the SDEP,
real-time dynamic exchange of stimuli (RDES) can stimulate the
neurophysiologic retina via communication with a bio-electronic
implant, via communication through an HMD.
[0964] The therapeutic effect of such stimulation may be measured
against the benchmark for decreased/normalization of saccadic eye
movements, pupil reaction, color sensitivity, heart rate, basal
body temperature, auditory sensitivity, and image sensitivity.
[0965] In an embodiment, user's symptoms of PTSD, is measured, and
media is modified in order to decrease the symptoms for the
user.
[0966] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, may be processed to determine
the symptoms of PTSD, experienced by the user. The data may
indicate a level of symptoms shown during or after user's
interaction with the media. The data may be further utilized to
modify VR/AR/MxR media for the user in order to decrease the PTSD
symptoms, such as but not limited to by minimizing visual, or any
other discomfort arising from the media experience. In an
embodiment, media is modified in real time for the user. In another
embodiment, data is saved and used to modify presentation of
VR/AR/MxR media to subsequent users with a similar data, or
subsequently to the user.
[0967] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease user's symptoms of PTSD, during interaction with
that media. FIG. 24 illustrates a flow chart describing an
exemplary process for modifying media in order to decrease symptoms
of PTSD, in accordance with some embodiments of the present
specification. At 2402, a first value for a plurality of data, as
further described below, is acquired. In embodiments, data is
acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [0968] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [0969] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [0970] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [0971]
4. One or more sensors configured to measure auditory processing;
[0972] 5. One or more sensors configured to measure gustatory and
olfactory processing; [0973] 6. One or more sensors to measure
pressure; [0974] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [0975] 8. A device to perform
electroencephalography; [0976] 9. A device to perform
electrocardiography; [0977] 10. A device to perform
electromyography; [0978] 11. A device to perform
electrooculography; [0979] 12. A device to perform
electroretinography; and [0980] 13. One or more sensors configured
to measure Galvanic Skin Response.
[0981] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[0982] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous symptomatic measures. In some examples, some of the
measures described above are context specific. The system can
correlate all available measures and look for trends in overall
display of PTSD symptoms. Accordingly, the media presented in a
VR/AR/MxR environment is modified, in order to decrease the PTSD
symptoms, for the user and/or a group of users.
[0983] At 2404, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2406, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in occurrences of PTSD
symptoms while interacting with the VR, AR, and/or MxR media:
[0984] 1. Increased Blink Rate [0985] 2. Increased Rate of Change
for Blink Rate [0986] 3. Increased Ratio of Partial Blinks to Full
Blinks [0987] 4. Increased Rate of Change for Pupil Size [0988] 5.
Decreased Target Relevancy for Pupil Initial and Final Position
[0989] 6. Decreased Target Relevancy for Gaze Direction [0990] 7.
Decreased Target Relevancy for Gaze Initial and Final Position
[0991] 8. Decreased Relevancy for Fixation Initial and Final
Position [0992] 9. Increased Fixation Duration Rate of Change
[0993] 10. Decreased Target Relevancy for Saccade Initial and Final
Position [0994] 11. Decreased Target Relevancy for Saccade Angle
[0995] 12. Decreased Saccade Magnitude (Distance of Saccade) [0996]
13. Increased Ratio of Anti-Saccade/Pro-Saccade [0997] 14.
Increased Saccade Velocity [0998] 15. Increased Saccade Rate of
Change [0999] 16. Increased Saccade Count (Number of Saccades)
[1000] 17. Increased Screen Distance [1001] 18. Decreased Target
Relevant Head Direction [1002] 19. Decreased Target Relevant Head
Fixation [1003] 20. Decreased Target Relevant Limb Movement [1004]
21. Shift in Weight Distribution [1005] 22. Increased Alpha/Theta
Brain Wave ratio [1006] 23. Increased Body Temperature [1007] 24.
Increased Respiration Rate [1008] 25. Increased Heart Rate [1009]
26. Increased Vocalizations [1010] 27. Increased Reaction Time
[1011] The system may determine decrease in occurrences of PTSD
symptoms for a user while interacting with the media in a VR, AR,
and/or MX environment based upon one or more of the following
changes: [1012] 1. Decreased Palpebral Fissure Rate of Change
[1013] 2. Low Distance Palpebral Fissure Resting State [1014] 3.
Low Distance Palpebral Fissure Active State [1015] 4. Increased
Rate of Convergence [1016] 5. Increased Rate of Divergence [1017]
6. Increased Fixation Duration [1018] 7. Increased Fixation Rate
[1019] 8. Increased Fixation Count [1020] 9. Increased Smooth
Pursuit [1021] 10. Decreased Alpha/Delta Brain Wave ratio
[1022] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased inhibition of return; low oxygen saturation; low blood
pressure; change in facial expression (may be dependent on specific
expression); change in gustatory processing; change in olfactory
processing; and change in auditory processing.
[1023] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease occurrences of
user's symptoms of PTSD, these lists are not exhaustive and may
include other data acquisition components, types of data, and
changes in data.
[1024] At 2408, the changes in the plurality of data determined
over time may be used to determine a degree of change in
occurrences of user's PTSD symptoms. The change in occurrences of
user's PTSD symptoms may indicate either reduced occurrences or
enhanced occurrences.
[1025] At 2410, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) occurrences of PTSD
symptoms. In embodiments, the media may be modified to address all
the changes in data that reflect increase in occurrences of PTSD
symptoms. In embodiments, a combination of one or more of the
following modifications may be performed: [1026] 1. Increasing a
contrast of the media [1027] 2. Making an object of interest that
is displayed in the media larger in size [1028] 3. Increasing a
brightness of the media [1029] 4. Increasing an amount of an object
of interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view [1030] 5. Changing a focal point of content displayed in the
media to a more central location [1031] 6. Removing objects from a
field of view and measuring if a user recognizes said removal
[1032] 7. Increasing an amount of color in said media [1033] 8.
Increasing a degree of shade in objects shown in said media [1034]
9. Changing RGB values of said media based upon external data
(demographic or trending data)
[1035] One or more of the following indicators may be observed to
affirm a decrease in PTSD symptoms: increased palpebral fissure
height; decreased ratio of partial blinks to full blinks; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; increased relevancy for fixation
initial and final position; decreased fixation duration; increased
target relevancy for saccade initial and final position; increased
target relevancy for saccade angle; increased saccade magnitude
(task relevant); decreased ratio of anti-saccade/pro-saccade;
decreased inhibition of return; decreased smooth pursuit; decreased
screen distance; increased target relevant head direction;
increased target relevant head fixation; increased target relevant
limb movement; decrease in shifts of weight distribution; increased
alpha/delta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; task relevant
facial expressions; decreased reaction time; task relevant
gustatory processing; task relevant olfactory processing; and task
relevant auditory processing.
[1036] In embodiments, a specific percentage or a range of decrease
in symptoms associated with PTSD, may be defined. In embodiments,
an additional value for data may be acquired at 2412, in order to
further determine change in data over time at 2414, after the
modifications have been executed at 2410. At 2416, a new
degree/percentage/range of decrease in symptoms associated with
PTSD may be acquired. At 2418, the system determines whether the
decrease in symptoms associated with PTSD is within the specified
range or percentage. If it is determine that the decrease is
insufficient, the system may loop back to step 2410 to further
modify the media. Therefore, the media may be iteratively modified
2410 and overall performance may be measured, until a percentage of
improvement of anywhere from 1% to 10000%, or any increment
therein, is achieved.
[1037] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1038] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1039] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1040] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 8: Modifying Media in Order to Decrease Double Vision
Related to Accommodative Dysfunction
[1041] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine an
extent of double vision related to accommodative dysfunction,
experienced by the user. The data may be further utilized to modify
VR/AR/MxR media for the user in order to decrease the double vision
related to accommodative dysfunction, such as but not limited to by
minimizing visual, or any other discomfort arising from the media
experience. In an embodiment, media is modified in real time for
the user. In another embodiment, data is saved and used to modify
presentation of VR/AR/MxR media to subsequent users with a similar
data, or subsequently to the user.
[1042] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease user's double vision related to accommodative
dysfunction, during interaction with that media. FIG. 25
illustrates a flow chart describing an exemplary process for
modifying media in order to decrease user's double vision, in
accordance with some embodiments of the present specification. At
2502, a first value for a plurality of data, as further described
below, is acquired. In embodiments, data is acquired by using at
least one camera configured to acquire eye movement data (rapid
scanning and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1043]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1044] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1045] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1046] 4. One or more sensors configured
to measure auditory processing; [1047] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1048] 6.
One or more sensors to measure pressure; [1049] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1050] 8.
A device to perform electroencephalography; [1051] 9. A device to
perform electrocardiography; [1052] 10. A device to perform
electromyography; [1053] 11. A device to perform
electrooculography; [1054] 12. A device to perform
electroretinography; and [1055] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1056] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1057] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous symptomatic measures. In some examples, some of the
measures described above are context specific. The system can
correlate all available measures and look for trends in user's
experience of double vision related to accommodative dysfunction.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease user's double vision, for the user
and/or a group of users.
[1058] At 2504, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2506, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in double vision while
interacting with the VR, AR, and/or MxR media: [1059] 1. Increased
Blink Rate [1060] 2. Increased Rate of Change for Blink Rate [1061]
3. Increased Ratio of Partial Blinks to Full Blinks [1062] 4.
Increased Rate of Change for Pupil Size [1063] 5. Decreased Target
Relevancy for Pupil Initial and Final Position [1064] 6. Decreased
Target Relevancy for Gaze Direction [1065] 7. Decreased Target
Relevancy for Gaze Initial and Final Position [1066] 8. Increased
Rate of Convergence [1067] 9. Decreased Relevancy for Fixation
Initial and Final Position [1068] 10. Increased Fixation Duration
[1069] 11. Increased Fixation Duration Rate of Change [1070] 12.
Increased Fixation Rate [1071] 13. Increased Fixation Count [1072]
14. Decreased Target Relevancy for Saccade Initial and Final
Position [1073] 15. Decreased Target Relevancy for Saccade Angle
[1074] 16. Decreased Saccade Magnitude (Distance of Saccade) [1075]
17. Increased Ratio of Anti-Saccade/Pro-Saccade [1076] 18.
Increased Inhibition of Return [1077] 19. Increased Saccade
Velocity [1078] 20. Increased Saccade Rate of Change [1079] 21.
Increased Saccade Count (Number of Saccades) [1080] 22. Decreased
Target Relevant Head Direction [1081] 23. Decreased Target Relevant
Head Fixation [1082] 24. Decreased Target Relevant Limb Movement
[1083] 25. Shift in Weight Distribution [1084] 26. Decreased
Alpha/Delta Brain Wave ratio [1085] 27. Increased Alpha/Theta Brain
Wave ratio [1086] 28. Increased Body Temperature [1087] 29.
Increased Respiration Rate [1088] 30. Low Oxygen Saturation [1089]
31. Increased Heart Rate [1090] 32. Low Blood Pressure [1091] 33.
Increased Reaction Time
[1092] The system may determine decrease in double vision for a
user while interacting with the media in a VR, AR, and/or MX
environment based upon one or more of the following changes: [1093]
1. Decreased Palpebral Fissure Rate of Change [1094] 2. Low
Distance Palpebral Fissure Resting State [1095] 3. Low Distance
Palpebral Fissure Active State [1096] 4. Increased Rate of
Divergence [1097] 5. Increased Smooth Pursuit [1098] 6. Increased
Screen Distance
[1099] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease double vision, these
lists are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[1100] At 2508, the changes in the plurality of data determined
over time may be used to determine a degree of change in user's
double vision. The change in double vision may indicate either
reduced double vision or enhanced double vision, related to
accommodative dysfunction.
[1101] At 2510, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) double vision. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in double vision related to
accommodative dysfunction. In embodiments, a combination of one or
more of the following modifications may be performed: [1102] 1.
Increasing a contrast of the media [1103] 2. Making an object of
interest that is displayed in the media larger in size [1104] 3.
Increasing a brightness of the media [1105] 4. Increasing an amount
of an object of interest displayed in the media shown in a central
field of view and decreasing said object of interest in a
peripheral field of view [1106] 5. Changing a focal point of
content displayed in the media to a more central location [1107] 6.
Removing objects from a field of view and measuring if a user
recognizes said removal [1108] 7. Increasing an amount of color in
said media [1109] 8. Increasing a degree of shade in objects shown
in said media [1110] 9. Changing RGB values of said media based
upon external data (demographic or trending data)
[1111] One or more of the following indicators may be observed to
affirm a decrease in double vision: decreased blink rate; decreased
rate of change for blink rate; decreased ratio of partial blinks to
full blinks; decreased rate of change for pupil size; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; decreased rate of convergence;
increased relevancy for fixation initial and final position;
decreased fixation duration; decreased fixation duration rate of
change; decreased fixation rate; decreased fixation count;
increased target relevancy for saccade initial and final position;
increased target relevancy for saccade angle; increased saccade
magnitude (task relevant); decreased ratio of
anti-saccade/pro-saccade; decreased inhibition of return; decreased
saccade velocity; decreased saccade rate of change; decreased
saccade count; ocular re-alignment or improvement in alignment and
coordinated ocular motility; increased target relevant head
direction; increased target relevant head fixation; increased
target relevant limb movement; decrease in shifts of weight
distribution; increased alpha/delta brain wave ratio; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; and task relevant auditory processing.
[1112] In embodiments, a specific percentage or a range of decrease
in symptoms associated with double vision, may be defined. In
embodiments, an additional value for data may be acquired at 2512,
in order to further determine change in data over time at 2514,
after the modifications have been executed at 2510. At 2516, a new
degree/percentage/range of decrease in symptoms associated with
double vision may be acquired. At 2518, the system determines
whether the decrease in double vision related to accommodative
dysfunction is within the specified range or percentage. If it is
determined that the decrease is insufficient, the system may loop
back to step 2510 to further modify the media. Therefore, the media
may be iteratively modified 2510 and overall performance may be
measured, until a percentage of improvement of anywhere from 1% to
10000%, or any increment therein, is achieved.
[1113] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1114] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1115] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1116] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 9: Modifying Media in Order to Decrease Vection Due to
Unintended Peripheral Field Stimulation
[1117] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine an
extent of vection due to unintended peripheral field stimulation,
experienced by the user. The data may be further utilized to modify
VR/AR/MxR media for the user in order to decrease the vection, such
as but not limited to by minimizing visual, or any other discomfort
arising from the media experience. In an embodiment, media is
modified in real time for the user. In another embodiment, data is
saved and used to modify presentation of VR/AR/MxR media to
subsequent users with a similar data, or subsequently to the
user.
[1118] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease user's experience of vection due to unintended
peripheral field stimulation, during interaction with that media.
FIG. 26 illustrates a flow chart describing an exemplary process
for modifying media in order to decrease vection, in accordance
with some embodiments of the present specification. At 2602, a
first value for a plurality of data, as further described below, is
acquired. In embodiments, data is acquired by using at least one
camera configured to acquire eye movement data (rapid scanning
and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1119]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1120] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1121] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1122] 4. One or more sensors configured
to measure auditory processing; [1123] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1124] 6.
One or more sensors to measure pressure; [1125] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1126] 8.
A device to perform electroencephalography; [1127] 9. A device to
perform electrocardiography; [1128] 10. A device to perform
electromyography; [1129] 11. A device to perform
electrooculography; [1130] 12. A device to perform
electroretinography; and [1131] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1132] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1133] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in user's experience of
vection due to unintended peripheral field vision. Accordingly, the
media presented in a VR/AR/MxR environment is modified, in order to
decrease user's vection, for the user and/or a group of users.
[1134] At 2604, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2606, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in vection while
interacting with the VR, AR, and/or MxR media: [1135] 1. Increased
Rate of Change for Blink Rate [1136] 2. Increased Ratio of Partial
Blinks to Full Blinks [1137] 3. Increased Rate of Change for Pupil
Size [1138] 4. Decreased Target Relevancy for Pupil Initial and
Final Position [1139] 5. Decreased Target Relevancy for Gaze
Direction [1140] 6. Decreased Target Relevancy for Gaze Initial and
Final Position [1141] 7. Increased Rate of Convergence [1142] 8.
Decreased Relevancy for Fixation Initial and Final Position [1143]
9. Increased Fixation Duration [1144] 10. Increased Fixation
Duration Rate of Change [1145] 11. Decreased Target Relevancy for
Saccade Initial and Final Position [1146] 12. Decreased Target
Relevancy for Saccade Angle [1147] 13. Decreased Saccade Magnitude
(Distance of Saccade) [1148] 14. Increased Ratio of
Anti-Saccade/Pro-Saccade [1149] 15. Increased Inhibition of Return
[1150] 16. Increased Saccade Velocity [1151] 17. Increased Saccade
Rate of Change [1152] 18. Increased Smooth Pursuit [1153] 19.
Decreased Target Relevant Head Direction [1154] 20. Decreased
Target Relevant Head Fixation [1155] 21. Decreased Target Relevant
Limb Movement [1156] 22. Shift in Weight Distribution [1157] 23.
Low Oxygen Saturation [1158] 24. Increased Heart Rate [1159] 25.
Low Blood Pressure [1160] 26. Increased Reaction Time
[1161] The system may determine decrease in vection for a user
while interacting with the media in a VR, AR, and/or MX environment
based upon one or more of the following changes: [1162] 1.
Decreased Palpebral Fissure Rate of Change [1163] 2. Low Distance
Palpebral Fissure Resting State [1164] 3. Low Distance Palpebral
Fissure Active State [1165] 4. Increased Blink Rate [1166] 5.
Increased Rate of Divergence [1167] 6. Increased Fixation Rate
[1168] 7. Increased Fixation Count [1169] 8. Increased Saccade
Count (Number of Saccades) [1170] 9. Increased Screen Distance
[1171] 10. Decreased Alpha/Delta Brain Wave ratio [1172] 11.
Increased Alpha/Theta Brain Wave ratio
[1173] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased body temperature; increased respiration rate; increased
vocalizations; change in facial expression (may be dependent on
specific expression); change in gustatory processing; change in
olfactory processing; and change in auditory processing.
[1174] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease vection, these lists
are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[1175] At 2608, the changes in the plurality of data determined
over time may be used to determine a degree of change in user's
vection. The change in vection may indicate either reduced vection
or enhanced vection, due to unintended peripheral field vision.
[1176] At 2610, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) vection. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in vection due to unintended
peripheral field vision. In embodiments, a combination of one or
more of the following modifications may be performed: [1177] 1.
Increasing a contrast of the media [1178] 2. Making an object of
interest that is displayed in the media larger in size [1179] 3.
Increasing a brightness of the media [1180] 4. Increasing an amount
of an object of interest displayed in the media shown in a central
field of view and decreasing said object of interest in a
peripheral field of view [1181] 5. Changing a focal point of
content displayed in the media to a more central location [1182] 6.
Removing objects from a field of view and measuring if a user
recognizes said removal [1183] 7. Increasing an amount of color in
said media [1184] 8. Increasing a degree of shade in objects shown
in said media [1185] 9. Changing RGB values of said media based
upon external data (demographic or trending data)
[1186] One or more of the following indicators may be observed to
affirm a decrease in vection: decreased rate of change for blink
rate; decreased ratio of partial blinks to full blinks; decreased
rate of change for pupil size; increased target relevancy for pupil
initial and final position; increased target relevancy for gaze
direction; increased target relevancy for gaze initial and final
position; decreased rate of convergence; increased relevancy for
fixation initial and final position; decreased fixation duration;
decreased fixation duration rate of change; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude (task
relevant); decreased ratio of anti-saccade/pro-saccade; decreased
inhibition of return; decreased saccade velocity; decreased saccade
rate of change; decreased smooth pursuit; increased target relevant
head direction; increased target relevant head fixation; increased
target relevant limb movement; decrease in shifts of weight
distribution; increased alpha/delta brain wave ratio; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; and task relevant auditory processing.
[1187] In embodiments, a specific percentage or a range of decrease
in vection due to unintended peripheral field stimulation, may be
defined. In embodiments, an additional value for data may be
acquired at 2612, in order to further determine change in data over
time at 2614, after the modifications have been executed at 2610.
At 2616, a new degree/percentage/range of decrease in vection due
to unintended peripheral field stimulation may be acquired. At
2618, the system determines whether the decrease in vection is
within the specified range or percentage. If it is determined that
the decrease is insufficient, the system may loop back to step 2610
to further modify the media. Therefore, the media may be
iteratively modified 2610 and overall performance may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[1188] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1189] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1190] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1191] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 10: Modifying Media in Order to Decrease Hormonal
Dysregulation Arising from Excessive Blue Light Exposure
[1192] Blue light exposure has been shown to impact health.
Elongated exposure to the waves transmitted through screen devices
can disrupt circadian rhythm and impact health in various ways,
including an impact on the hormones. The effect of blue light is
believed to cause a decrease in the bodies' production of
melatonin. Prolonged exposure to blue light is also believed to
negatively impact ocular health.
[1193] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine an
extent of hormonal dysregulation arising from excessive blue light
exposure, experienced by the user. The data may be further utilized
to modify VR/AR/MxR media for the user in order to decrease the
hormonal dysregulation, such as but not limited to by minimizing
visual, or any other discomfort arising from the media experience.
In an embodiment, media is modified in real time for the user. In
another embodiment, data is saved and used to modify presentation
of VR/AR/MxR media to subsequent users with a similar data, or
subsequently to the user.
[1194] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease user's hormonal dysregulation arising from
excessive blue light exposure, during interaction with that media.
FIG. 27 illustrates a flow chart describing an exemplary process
for modifying media in order to decrease hormonal dysregulation, in
accordance with some embodiments of the present specification. At
2702, a first value for a plurality of data, as further described
below, is acquired. In embodiments, data is acquired by using at
least one camera configured to acquire eye movement data (rapid
scanning and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1195]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1196] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1197] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1198] 4. One or more sensors configured
to measure auditory processing; [1199] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1200] 6.
One or more sensors to measure pressure; [1201] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1202] 8.
A device to perform electroencephalography; [1203] 9. A device to
perform electrocardiography; [1204] 10. A device to perform
electromyography; [1205] 11. A device to perform
electrooculography; [1206] 12. A device to perform
electroretinography; and [1207] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1208] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1209] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in user's hormonal
dysregulation arising from excessive blue light exposure.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease hormonal dysregulation, for the user
and/or a group of users.
[1210] At 2704, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2706, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in hormonal
dysregulation while interacting with the VR, AR, and/or MxR media:
[1211] 1. Decreased Palpebral Fissure Rate of Change [1212] 2. Low
Distance Palpebral Fissure Resting State [1213] 3. Low Distance
Palpebral Fissure Active State [1214] 4. Increased Ratio of Partial
Blinks to Full Blinks [1215] 5. Decreased Target Relevancy for
Pupil Initial and Final Position [1216] 6. Decreased Target
Relevancy for Gaze Direction [1217] 7. Decreased Target Relevancy
for Gaze Initial and Final Position [1218] 8. Increased Rate of
Divergence [1219] 9. Decreased Relevancy for Fixation Initial and
Final Position [1220] 10. Increased Fixation Duration [1221] 11.
Decreased Target Relevancy for Saccade Initial and Final Position
[1222] 12. Decreased Target Relevancy for Saccade Angle [1223] 13.
Decreased Saccade Magnitude (Distance of Saccade) [1224] 14.
Increased Ratio of Anti-Saccade/Pro-Saccade [1225] 15. Increased
Inhibition of Return [1226] 16. Increased Smooth Pursuit [1227] 17.
Increased Screen Distance [1228] 18. Decreased Target Relevant Head
Direction [1229] 19. Decreased Target Relevant Head Fixation [1230]
20. Decreased Target Relevant Limb Movement [1231] 21. Shift in
Weight Distribution [1232] 22. Decreased Alpha/Delta Brain Wave
ratio [1233] 23. Increased Body Temperature [1234] 24. Increased
Respiration Rate [1235] 25. Low Oxygen Saturation [1236] 26.
Increased Heart Rate [1237] 27. Low Blood Pressure [1238] 28.
Increased Reaction Time
[1239] The system may determine decrease in hormonal dysregulation
for a user while interacting with the media in a VR, AR, and/or MX
environment based upon one or more of the following changes: [1240]
1. Increased Blink Rate [1241] 2. Increased Rate of Change for
Blink Rate [1242] 3. Increased Rate of Change for Pupil Size [1243]
4. Increased Rate of Convergence [1244] 5. Increased Fixation
Duration Rate of Change [1245] 6. Increased Fixation Rate [1246] 7.
Increased Fixation Count [1247] 8. Increased Saccade Velocity
[1248] 9. Increased Saccade Rate of Change [1249] 10. Increased
Saccade Count (Number of Saccades) [1250] 11. Increased Alpha/Theta
Brain Wave ratio
[1251] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased vocalizations; change in facial expression (may be
dependent on specific expression); change in gustatory processing;
and change in olfactory processing.
[1252] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease hormonal
dysregulation, these lists are not exhaustive and may include other
data acquisition components, types of data, and changes in
data.
[1253] At 2708, the changes in the plurality of data determined
over time may be used to determine a degree of change in user's
hormonal dysregulation. The change in hormonal dysregulation may
indicate either reduced hormonal dysregulation or enhanced hormonal
dysregulation, arising from excessive blue light exposure.
[1254] At 2710, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) hormonal
dysregulation. In embodiments, the media may be modified to address
all the changes in data that reflect increase in hormonal
dysregulation arising from excessive blue light exposure. In
embodiments, a combination of one or more of the following
modifications may be performed: [1255] 1. Increasing a contrast of
the media [1256] 2. Making an object of interest that is displayed
in the media larger in size [1257] 3. Increasing a brightness of
the media [1258] 4. Increasing an amount of an object of interest
displayed in the media shown in a central field of view and
decreasing said object of interest in a peripheral field of view
[1259] 5. Changing a focal point of content displayed in the media
to a more central location [1260] 6. Removing objects from a field
of view and measuring if a user recognizes said removal [1261] 7.
Increasing an amount of color in said media [1262] 8. Increasing a
degree of shade in objects shown in said media [1263] 9. Changing
RGB values of said media based upon external data (demographic or
trending data)
[1264] One or more of the following indicators may be observed to
affirm a decrease in hormonal dysregulation: increased palpebral
fissure height; decreased ratio of partial blinks to full blinks;
increased target relevancy for pupil initial and final position;
increased target relevancy for gaze direction; increased target
relevancy for gaze initial and final position; decreased rate of
divergence; increased relevancy for fixation initial and final
position; decreased fixation duration; increased target relevancy
for saccade initial and final position; increased target relevancy
for saccade angle; increased saccade magnitude (task relevant);
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased smooth pursuit; decreased screen distance;
increased target relevant head direction; increased target relevant
head fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1265] In embodiments, a specific percentage or a range of decrease
in hormonal dysregulation arising from excessive blue light
exposure, may be defined. In embodiments, an additional value for
data may be acquired at 2712, in order to further determine change
in data over time at 2714, after the modifications have been
executed at 2710. At 2716, a new degree/percentage/range of
decrease in hormonal dysregulation arising from excessive blue
light exposure may be acquired. At 2718, the system determines
whether the decrease in hormonal dysregulation is within the
specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 2710 to
further modify the media. Therefore, the media may be iteratively
modified 2710 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1266] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1267] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1268] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1269] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 11: Modifying Media in Order to Decrease Potential
Photo-Toxicity from Over-Exposure to Screen Displays
[1270] Prolonged exposure to screen displays is believed to
increase the potential for phototoxicity for a user. In an
embodiment, data collected from the user, such as by HMDs, or any
other VR/AR/MxR system, is processed to determine the potential of
phototoxicity, which could be experienced by the user. The data may
be further utilized to modify VR/AR/MxR media for the user in order
to decrease the potential of phototoxicity, such as but not limited
to by minimizing visual, or any other discomfort arising from the
media experience. In an embodiment, media is modified in real time
for the user. In another embodiment, data is saved and used to
modify presentation of VR/AR/MxR media to subsequent users with a
similar data, or subsequently to the user.
[1271] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease potential phototoxicity from over-exposure to
screen displays, during interaction with that media. FIG. 28
illustrates a flow chart describing an exemplary process for
modifying media in order to decrease the potential for
phototoxicity, in accordance with some embodiments of the present
specification. At 2802, a first value for a plurality of data, as
further described below, is acquired. In embodiments, data is
acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [1272] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [1273] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [1274] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [1275]
4. One or more sensors configured to measure auditory processing;
[1276] 5. One or more sensors configured to measure gustatory and
olfactory processing; [1277] 6. One or more sensors to measure
pressure; [1278] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [1279] 8. A device to perform
electroencephalography; [1280] 9. A device to perform
electrocardiography; [1281] 10. A device to perform
electromyography; [1282] 11. A device to perform
electrooculography; [1283] 12. A device to perform
electroretinography; and [1284] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1285] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1286] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in phototoxicity from
over-exposure to screen displays. Accordingly, the media presented
in a VR/AR/MxR environment is modified, in order to decrease
phototoxicity, for the user and/or a group of users.
[1287] At 2804, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2806, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in phototoxicity while
interacting with the VR, AR, and/or MxR media: [1288] 1. Decreased
Palpebral Fissure Rate of Change [1289] 2. Low Distance Palpebral
Fissure Resting State [1290] 3. Low Distance Palpebral Fissure
Active State [1291] 4. Increased Blink Rate [1292] 5. Increased
Rate of Change for Blink Rate [1293] 6. Increased Ratio of Partial
Blinks to Full Blinks [1294] 7. Increased Rate of Change for Pupil
Size [1295] 8. Decreased Target Relevancy for Pupil Initial and
Final Position [1296] 9. Decreased Target Relevancy for Gaze
Direction [1297] 10. Decreased Target Relevancy for Gaze Initial
and Final Position [1298] 11. Increased Rate of Divergence [1299]
12. Decreased Relevancy for Fixation Initial and Final Position
[1300] 13. Increased Fixation Duration [1301] 14. Increased
Fixation Duration Rate of Change [1302] 15. Decreased Target
Relevancy for Saccade Initial and Final Position [1303] 16.
Decreased Target Relevancy for Saccade Angle [1304] 17. Decreased
Saccade Magnitude (Distance of Saccade) [1305] 18. Increased Ratio
of Anti-Saccade/Pro-Saccade [1306] 19. Increased Inhibition of
Return [1307] 20. Increased Saccade Velocity [1308] 21. Increased
Saccade Rate of Change [1309] 22. Increased Smooth Pursuit [1310]
23. Increased Screen Distance [1311] 24. Decreased Target Relevant
Head Direction [1312] 25. Decreased Target Relevant Head Fixation
[1313] 26. Decreased Target Relevant Limb Movement [1314] 27. Shift
in Weight Distribution [1315] 28. Increased Alpha/Theta Brain Wave
ratio [1316] 29. Increased Body Temperature [1317] 30. Increased
Respiration Rate [1318] 31. Increased Heart Rate [1319] 32. Low
Blood Pressure [1320] 33. Increased Reaction Time
[1321] The system may determine decrease in phototoxicity while
interacting with the media in a VR, AR, and/or MX environment based
upon one or more of the following changes: [1322] 1. Increased Rate
of Convergence [1323] 2. Increased Fixation Rate [1324] 3.
Increased Fixation Count [1325] 4. Increased Saccade Count (Number
of Saccades) [1326] 5. Decreased Alpha/Delta Brain Wave ratio
[1327] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: low
oxygen saturation; increased vocalizations; change in facial
expression (may be dependent on specific expression); change in
gustatory processing; change in olfactory processing; and change in
auditory processing.
[1328] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease phototoxicity, these
lists are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[1329] At 2808, the changes in the plurality of data determined
over time may be used to determine a degree of change in
phototoxicity. The change in phototoxicity may indicate either
reduced phototoxicity or enhanced phototoxicity, from over-exposure
to screen displays.
[1330] At 2810, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) phototoxicity. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in phototoxicity from over-exposure
to screen displays. In embodiments, a combination of one or more of
the following modifications may be performed: [1331] 1. Increasing
a contrast of the media [1332] 2. Making an object of interest that
is displayed in the media larger in size [1333] 3. Increasing a
brightness of the media [1334] 4. Increasing an amount of an object
of interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view [1335] 5. Changing a focal point of content displayed in the
media to a more central location [1336] 6. Removing objects from a
field of view and measuring if a user recognizes said removal
[1337] 7. Increasing an amount of color in said media [1338] 8.
Increasing a degree of shade in objects shown in said media [1339]
9. Changing RGB values of said media based upon external data
(demographic or trending data)
[1340] One or more of the following indicators may be observed to
affirm a decrease in phototoxicity: increased palpebral fissure
height; decreased blink rate; decreased rate of change for blink
rate; decreased ratio of partial blinks to full blinks; decreased
rate of change for pupil size; increased target relevancy for pupil
initial and final position; increased target relevancy for gaze
direction; increased target relevancy for gaze initial and final
position; decreased rate of divergence; increased relevancy for
fixation initial and final position; decreased fixation duration;
decreased fixation duration rate of change; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude (task
relevant); decreased ratio of anti-saccade/pro-saccade; decreased
inhibition of return; decreased saccade velocity; decreased saccade
rate of change; decreased smooth pursuit; decreased screen
distance; increased target relevant head direction; increased
target relevant head fixation; increased target relevant limb
movement; decrease in shifts of weight distribution; increased
alpha/delta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; task relevant
facial expressions; decreased reaction time; task relevant
gustatory processing; task relevant olfactory processing; and task
relevant auditory processing.
[1341] In embodiments, a specific percentage or a range of decrease
in phototoxicity from over-exposure to screen displays, may be
defined. In embodiments, an additional value for data may be
acquired at 2812, in order to further determine change in data over
time at 2814, after the modifications have been executed at 2810.
At 2816, a new degree/percentage/range of decrease in phototoxicity
from over-exposure to screen displays may be acquired. At 2818, the
system determines whether the decrease in phototoxicity is within
the specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 2810 to
further modify the media. Therefore, the media may be iteratively
modified 2810 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1342] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1343] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1344] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1345] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 12: Modifying Media in Order to Decrease Nausea and/or
Stomach Discomfort
[1346] Prolonged exposure to screen displays may result in nausea
and/or stomach discomfort. In an embodiment, data collected from
the user, such as by HMDs, or any other VR/AR/MxR system, is
processed to determine the extent of nausea and/or stomach
discomfort, which could be experienced by the user. The data may be
further utilized to modify VR/AR/MxR media for the user in order to
decrease the nausea and/or stomach discomfort, such as but not
limited to by minimizing visual, or any other discomfort arising
from the media experience. In an embodiment, media is modified in
real time for the user. In another embodiment, data is saved and
used to modify presentation of VR/AR/MxR media to subsequent users
with a similar data, or subsequently to the user.
[1347] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease nausea and/or stomach discomfort, during
interaction with that media. FIG. 29 illustrates a flow chart
describing an exemplary process for modifying media in order to
decrease nausea and/or stomach discomfort, in accordance with some
embodiments of the present specification. At 2902, a first value
for a plurality of data, as further described below, is acquired.
In embodiments, data is acquired by using at least one camera
configured to acquire eye movement data (rapid scanning and/or
saccadic movement), blink rate data, fixation data, pupillary
diameter, palpebral (eyelid) fissure distance between the eyelids.
Additionally, the VR, AR, and/or MxR device can include one or more
of the following sensors incorporated therein: [1348] 1. One or
more sensors configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, and/or body
velocity; [1349] 2. One or more sensors configured to measure limb
movement, limb rotation, limb direction, and/or limb velocity;
[1350] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1351] 4. One or more sensors configured
to measure auditory processing; [1352] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1353] 6.
One or more sensors to measure pressure; [1354] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1355] 8.
A device to perform electroencephalography; [1356] 9. A device to
perform electrocardiography; [1357] 10. A device to perform
electromyography; [1358] 11. A device to perform
electrooculography; [1359] 12. A device to perform
electroretinography; and [1360] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1361] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1362] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in nausea and/or stomach
discomfort. Accordingly, the media presented in a VR/AR/MxR
environment is modified, in order to decrease nausea and/or stomach
discomfort, for the user and/or a group of users.
[1363] At 2904, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 2906, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in nausea and/or
stomach discomfort while interacting with the VR, AR, and/or MxR
media: [1364] 1. Decreased Palpebral Fissure Rate of Change [1365]
2. Low Distance Palpebral Fissure Resting State [1366] 3. Low
Distance Palpebral Fissure Active State [1367] 4. Increased Ratio
of Partial Blinks to Full Blinks [1368] 5. Decreased Target
Relevancy for Pupil Initial and Final Position [1369] 6. Decreased
Target Relevancy for Gaze Direction [1370] 7. Decreased Target
Relevancy for Gaze Initial and Final Position [1371] 8. Increased
Rate of Divergence [1372] 9. Decreased Relevancy for Fixation
Initial and Final Position [1373] 10. Increased Fixation Duration
[1374] 11. Decreased Target Relevancy for Saccade Initial and Final
Position [1375] 12. Decreased Target Relevancy for Saccade Angle
[1376] 13. Decreased Saccade Magnitude (Distance of Saccade) [1377]
14. Increased Ratio of Anti-Saccade/Pro-Saccade [1378] 15.
Increased Inhibition of Return [1379] 16. Increased Smooth Pursuit
[1380] 17. Increased Screen Distance [1381] 18. Decreased Target
Relevant Head Direction [1382] 19. Decreased Target Relevant Head
Fixation [1383] 20. Decreased Target Relevant Limb Movement [1384]
21. Shift in Weight Distribution [1385] 22. Increased Body
Temperature [1386] 23. Increased Respiration Rate [1387] 24. Low
Oxygen Saturation [1388] 25. Increased Heart Rate [1389] 26. Low
Blood Pressure [1390] 27. Increased Vocalizations [1391] 28.
Increased Reaction Time
[1392] The system may determine decrease in nausea and/or stomach
discomfort while interacting with the media in a VR, AR, and/or MX
environment based upon one or more of the following changes: [1393]
1. Increased Blink Rate [1394] 2. Increased Rate of Change for
Blink Rate [1395] 3. Increased Rate of Change for Pupil Size [1396]
4. Increased Rate of Convergence [1397] 5. Increased Fixation
Duration Rate of Change [1398] 6. Increased Fixation Rate [1399] 7.
Increased Fixation Count [1400] 8. Increased Saccade Velocity
[1401] 9. Increased Saccade Rate of Change [1402] 10. Increased
Saccade Count (Number of Saccades) [1403] 11. Decreased Alpha/Delta
Brain Wave ratio [1404] 12. Increased Alpha/Theta Brain Wave
ratio
[1405] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[1406] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease nausea and/or
stomach discomfort, these lists are not exhaustive and may include
other data acquisition components, types of data, and changes in
data.
[1407] At 2908, the changes in the plurality of data determined
over time may be used to determine a degree of change in nausea
and/or stomach discomfort. The change in nausea and/or stomach
discomfort may indicate either reduced nausea and/or stomach
discomfort or enhanced nausea and/or stomach discomfort.
[1408] At 2910, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) nausea and/or stomach
discomfort. In embodiments, the media may be modified to address
all the changes in data that reflect increase in nausea and/or
stomach discomfort. In embodiments, a combination of one or more of
the following modifications may be performed: [1409] 1. Increasing
a contrast of the media [1410] 2. Making an object of interest that
is displayed in the media larger in size [1411] 3. Increasing a
brightness of the media [1412] 4. Increasing an amount of an object
of interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view [1413] 5. Changing a focal point of content displayed in the
media to a more central location [1414] 6. Removing objects from a
field of view and measuring if a user recognizes said removal
[1415] 7. Increasing an amount of color in said media [1416] 8.
Increasing a degree of shade in objects shown in said media [1417]
9. Changing RGB values of said media based upon external data
(demographic or trending data)
[1418] One or more of the following indicators may be observed to
affirm a decrease in nausea and/or stomach discomfort: increased
palpebral fissure height; decreased ratio of partial blinks to full
blinks; increased target relevancy for pupil initial and final
position; increased target relevancy for gaze direction; increased
target relevancy for gaze initial and final position; decreased
rate of divergence; increased relevancy for fixation initial and
final position; decreased fixation duration; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude (task
relevant); decreased ratio of anti-saccade/pro-saccade; decreased
inhibition of return; decreased smooth pursuit; decreased screen
distance; increased target relevant head direction; increased
target relevant head fixation; increased target relevant limb
movement; decrease in shifts of weight distribution; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; task relevant auditory processing.
[1419] In embodiments, a specific percentage or a range of decrease
in nausea and/or stomach discomfort, may be defined. In
embodiments, an additional value for data may be acquired at 2912,
in order to further determine change in data over time at 2914,
after the modifications have been executed at 2910. At 2916, a new
degree/percentage/range of decrease in nausea and/or stomach
discomfort may be acquired. At 2918, the system determines whether
the decrease in nausea and/or stomach discomfort is within the
specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 2910 to
further modify the media. Therefore, the media may be iteratively
modified 2910 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1420] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1421] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1422] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1423] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 13: Modifying Media in Order to Decrease Visual
Discomfort
[1424] Prolonged exposure to screen displays may result in visual
discomfort, including at least one of eyestrain, dry eye, eye
tearing, foreign body sensation, feeling of pressure in the eyes,
or aching around the eyes. In an embodiment, data collected from
the user, such as by HMDs, or any other VR/AR/MxR system, is
processed to determine the extent of visual discomfort, including
at least one of eyestrain, dry eye, eye tearing, foreign body
sensation, feeling of pressure in the eyes, or aching around the
eyes, which could be experienced by the user. The data may be
further utilized to modify VR/AR/MxR media for the user in order to
decrease the visual discomfort, such as but not limited to by
minimizing visual, or any other discomfort arising from the media
experience. In an embodiment, media is modified in real time for
the user. In another embodiment, data is saved and used to modify
presentation of VR/AR/MxR media to subsequent users with a similar
data, or subsequently to the user.
[1425] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease visual discomfort, during interaction with that
media. FIG. 30 illustrates a flow chart describing an exemplary
process for modifying media in order to decrease the visual
discomfort, in accordance with some embodiments of the present
specification. At 3002, a first value for a plurality of data, as
further described below, is acquired. In embodiments, data is
acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [1426] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [1427] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [1428] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [1429]
4. One or more sensors configured to measure auditory processing;
[1430] 5. One or more sensors configured to measure gustatory and
olfactory processing; [1431] 6. One or more sensors to measure
pressure; [1432] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [1433] 8. A device to perform
electroencephalography; [1434] 9. A device to perform
electrocardiography; [1435] 10. A device to perform
electromyography; [1436] 11. A device to perform
electrooculography; [1437] 12. A device to perform
electroretinography; and [1438] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1439] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1440] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in visual discomfort.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease visual discomfort, for the user
and/or a group of users.
[1441] At 3004, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3006, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in visual discomfort
while interacting with the VR, AR, and/or MxR media: [1442] 1.
Decreased Palpebral Fissure Rate of Change [1443] 2. Low Distance
Palpebral Fissure Resting State [1444] 3. Low Distance Palpebral
Fissure Active State [1445] 4. Increased Blink Rate [1446] 5.
Increased Rate of Change for Blink Rate [1447] 6. Increased Ratio
of Partial Blinks to Full Blinks [1448] 7. Increased Rate of Change
for Pupil Size [1449] 8. Decreased Target Relevancy for Pupil
Initial and Final Position [1450] 9. Decreased Target Relevancy for
Gaze Direction [1451] 10. Decreased Target Relevancy for Gaze
Initial and Final Position [1452] 11. Increased Rate of Divergence
[1453] 12. Decreased Relevancy for Fixation Initial and Final
Position [1454] 13. Increased Fixation Duration [1455] 14.
Increased Fixation Duration Rate of Change [1456] 15. Increased
Fixation Rate [1457] 16. Increased Fixation Count [1458] 17.
Decreased Target Relevancy for Saccade Initial and Final Position
[1459] 18. Decreased Target Relevancy for Saccade Angle [1460] 19.
Decreased Saccade Magnitude (Distance of Saccade) [1461] 20.
Increased Ratio of Anti-Saccade/Pro-Saccade [1462] 21. Increased
Inhibition of Return [1463] 22. Increased Saccade Velocity [1464]
23. Increased Saccade Rate of Change [1465] 24. Increased Saccade
Count (Number of Saccades) [1466] 25. Increased Smooth Pursuit
[1467] 26. Increased Screen Distance [1468] 27. Decreased Target
Relevant Head Direction [1469] 28. Decreased Target Relevant Head
Fixation [1470] 29. Decreased Target Relevant Limb Movement [1471]
30. Shift in Weight Distribution [1472] 31. Increased Body
Temperature [1473] 32. Increased Respiration Rate [1474] 33. Low
Oxygen Saturation [1475] 34. Increased Heart Rate [1476] 35. Low
Blood Pressure [1477] 36. Increased Vocalizations [1478] 37.
Increased Reaction Time
[1479] The system may determine decrease in visual discomfort while
interacting with the media in a VR, AR, and/or MX environment based
upon one or more of the following changes: [1480] 1. Increased Rate
of Convergence [1481] 2. Decreased Alpha/Delta Brain Wave ratio
[1482] 3. Increased Alpha/Theta Brain Wave ratio
[1483] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[1484] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease visual discomfort,
these lists are not exhaustive and may include other data
acquisition components, types of data, and changes in data.
[1485] At 3008, the changes in the plurality of data determined
over time may be used to determine a degree of change in visual
discomfort. The change in visual discomfort may indicate either
reduced visual discomfort or enhanced visual discomfort.
[1486] At 3010, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) visual discomfort. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in visual discomfort. In embodiments,
a combination of one or more of the following modifications may be
performed: [1487] 1. Increasing a contrast of the media [1488] 2.
Making an object of interest that is displayed in the media larger
in size [1489] 3. Increasing a brightness of the media [1490] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [1491] 5. Changing a
focal point of content displayed in the media to a more central
location [1492] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [1493] 7. Increasing an
amount of color in said media [1494] 8. Increasing a degree of
shade in objects shown in said media [1495] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[1496] One or more of the following indicators may be observed to
affirm a decrease in visual discomfort: increased palpebral fissure
rate of change; decreased blink rate; decreased rate of change for
blink rate; decreased ratio of partial blinks to full blinks;
increased target relevancy for pupil initial and final position;
increased target relevancy for gaze direction; increased target
relevancy for gaze initial and final position; decreased rate of
divergence; increased relevancy for fixation initial and final
position; increased target relevancy for saccade initial and final
position; decreased fixation duration; decreased fixation duration
rate of change; decreased fixation rate; decreased fixation count;
increased target relevancy for saccade angle; increased saccade
magnitude (task relevant); decreased ratio of
anti-saccade/pro-saccade; decreased inhibition of return; decreased
saccade velocity; decreased saccade rate of change; decreased
saccade count; decreased smooth pursuit; decreased screen distance;
increased target relevant head direction; increased target relevant
head fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1497] In embodiments, a specific percentage or a range of decrease
in visual discomfort, may be defined. In embodiments, an additional
value for data may be acquired at 3012, in order to further
determine change in data over time at 3014, after the modifications
have been executed at 3010. At 3016, a new degree/percentage/range
of decrease in visual discomfort may be acquired. At 3018, the
system determines whether the decrease in visual discomfort is
within the specified range or percentage. If it is determined that
the decrease is insufficient, the system may loop back to step 3010
to further modify the media. Therefore, the media may be
iteratively modified 3010 and overall performance may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[1498] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1499] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1500] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1501] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 14: Modifying Media in Order to Decrease Disorientation
and Postural Instability
[1502] Prolonged exposure to screen displays may result in
disorientation and postural instability. In an embodiment, data
collected from the user, such as by HMDs, or any other VR/AR/MxR
system, is processed to determine the extent of disorientation and
postural instability, which could be experienced by the user. The
data may be further utilized to modify VR/AR/MxR media for the user
in order to decrease the disorientation and postural instability,
such as but not limited to by minimizing visual, or any other
discomfort arising from the media experience. In an embodiment,
media is modified in real time for the user. In another embodiment,
data is saved and used to modify presentation of VR/AR/MxR media to
subsequent users with a similar data, or subsequently to the
user.
[1503] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease disorientation and postural instability, during
interaction with that media. FIG. 31 illustrates a flow chart
describing an exemplary process for modifying media in order to
decrease disorientation and postural instability, in accordance
with some embodiments of the present specification. At 3102, a
first value for a plurality of data, as further described below, is
acquired. In embodiments, data is acquired by using at least one
camera configured to acquire eye movement data (rapid scanning
and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1504]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1505] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1506] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1507] 4. One or more sensors configured
to measure auditory processing; [1508] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1509] 6.
One or more sensors to measure pressure; [1510] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1511] 8.
A device to perform electroencephalography; [1512] 9. A device to
perform electrocardiography; [1513] 10. A device to perform
electromyography; [1514] 11. A device to perform
electrooculography; [1515] 12. A device to perform
electroretinography; and [1516] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1517] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1518] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in disorientation and
postural instability. Accordingly, the media presented in a
VR/AR/MxR environment is modified, in order to decrease
disorientation and postural instability, for the user and/or a
group of users.
[1519] At 3104, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3106, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in disorientation and
postural instability while interacting with the VR, AR, and/or MxR
media: [1520] 1. Decreased Palpebral Fissure Rate of Change [1521]
2. Low Distance Palpebral Fissure Resting State [1522] 3. Low
Distance Palpebral Fissure Active State [1523] 4. Increased Blink
Rate [1524] 5. Increased Rate of Change for Blink Rate [1525] 6.
Increased Ratio of Partial Blinks to Full Blinks [1526] 7.
Increased Rate of Change for Pupil Size [1527] 8. Decreased Target
Relevancy for Pupil Initial and Final Position [1528] 9. Decreased
Target Relevancy for Gaze Direction [1529] 10. Decreased Target
Relevancy for Gaze Initial and Final Position [1530] 11. Increased
Rate of Divergence [1531] 12. Decreased Relevancy for Fixation
Initial and Final Position [1532] 13. Increased Fixation Duration
Rate of Change [1533] 14. Increased Fixation Rate [1534] 15.
Increased Fixation Count [1535] 16. Decreased Target Relevancy for
Saccade Initial and Final Position [1536] 17. Decreased Target
Relevancy for Saccade Angle [1537] 18. Decreased Saccade Magnitude
(Distance of Saccade) [1538] 19. Increased Ratio of
Anti-Saccade/Pro-Saccade [1539] 20. Increased Inhibition of Return
[1540] 21. Increased Saccade Velocity [1541] 22. Increased Saccade
Rate of Change [1542] 23. Increased Saccade Count (Number of
Saccades) [1543] 24. Decreased Target Relevant Head Direction
[1544] 25. Decreased Target Relevant Head Fixation [1545] 26.
Decreased Target Relevant Limb Movement [1546] 27. Shift in Weight
Distribution [1547] 28. Increased Body Temperature [1548] 29.
Increased Respiration Rate [1549] 30. Low Oxygen Saturation [1550]
31. Increased Heart Rate [1551] 32. Low Blood Pressure [1552] 33.
Increased Vocalizations Increased Reaction Time
[1553] The system may determine decrease in disorientation and
postural instability while interacting with the media in a VR, AR,
and/or MX environment based upon one or more of the following
changes: [1554] 1. Increased Rate of Convergence [1555] 2.
Increased Fixation Duration [1556] 3. Increased Smooth Pursuit
[1557] 4. Increased Screen Distance [1558] 5. Decreased Alpha/Delta
Brain Wave ratio [1559] 6. Increased Alpha/Theta Brain Wave ratio
[1560] 7. Stable Weight Distribution
[1561] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[1562] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease disorientation and
postural instability, these lists are not exhaustive and may
include other data acquisition components, types of data, and
changes in data.
[1563] At 3108, the changes in the plurality of data determined
over time may be used to determine a degree of change in
disorientation and postural instability. The change in
disorientation and postural instability may indicate either reduced
disorientation and postural instability or enhanced disorientation
and postural instability.
[1564] At 3110, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) disorientation and
postural instability. In embodiments, the media may be modified to
address all the changes in data that reflect increase in
disorientation and postural instability. In embodiments, a
combination of one or more of the following modifications may be
performed: [1565] 1. Increasing a contrast of the media [1566] 2.
Making an object of interest that is displayed in the media larger
in size [1567] 3. Increasing a brightness of the media [1568] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [1569] 5. Changing a
focal point of content displayed in the media to a more central
location [1570] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [1571] 7. Increasing an
amount of color in said media [1572] 8. Increasing a degree of
shade in objects shown in said media [1573] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[1574] One or more of the following indicators may be observed to
affirm a decrease in disorientation and postural instability:
increased palpebral fissure height; decreased blink rate; decreased
rate of change for blink rate; decreased ratio of partial blinks to
full blinks; decreased rate of change for pupil size; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; decreased rate of divergence;
increased relevancy for fixation initial and final position;
decreased fixation duration rate of change; decreased fixation
rate; decreased fixation count; increased target relevancy for
saccade initial and final position; increased target relevancy for
saccade angle; increased saccade magnitude (task relevant);
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased saccade velocity; decreased saccade rate of
change; decreased saccade count; increased target relevant head
direction; increased target relevant head fixation; increased
target relevant limb movement; decrease in shifts of weight
distribution; increased alpha/delta brain wave ratio; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; task relevant auditory processing.
[1575] In embodiments, a specific percentage or a range of decrease
in disorientation and postural instability, may be defined. In
embodiments, an additional value for data may be acquired at 3112,
in order to further determine change in data over time at 3114,
after the modifications have been executed at 3110. At 3116, a new
degree/percentage/range of decrease in disorientation and postural
instability may be acquired. At 3118, the system determines whether
the decrease in disorientation and postural instability is within
the specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 3110 to
further modify the media. Therefore, the media may be iteratively
modified 3110 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1576] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1577] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1578] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1579] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 15: Modifying Media in Order to Decrease Headaches and
Difficulties in Focusing In an embodiment, data collected from the
user, such as by HMDs, or any other
[1580] VR/AR/MxR system, is processed to determine the extent of
headaches and difficulties in focusing, which could be experienced
by the user. The data may be further utilized to modify VR/AR/MxR
media for the user in order to decrease the headaches and
difficulties in focusing, such as but not limited to by minimizing
visual, or any other discomfort arising from the media experience.
In an embodiment, media is modified in real time for the user. In
another embodiment, data is saved and used to modify presentation
of VR/AR/MxR media to subsequent users with a similar data, or
subsequently to the user.
[1581] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease headaches and difficulties in focusing, during
interaction with that media. FIG. 32 illustrates a flow chart
describing an exemplary process for modifying media in order to
decrease headaches and difficulties in focusing, in accordance with
some embodiments of the present specification. At 3202, a first
value for a plurality of data, as further described below, is
acquired. In embodiments, data is acquired by using at least one
camera configured to acquire eye movement data (rapid scanning
and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1582]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1583] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1584] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1585] 4. One or more sensors configured
to measure auditory processing; [1586] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1587] 6.
One or more sensors to measure pressure; [1588] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1589] 8.
A device to perform electroencephalography; [1590] 9. A device to
perform electrocardiography; [1591] 10. A device to perform
electromyography; [1592] 11. A device to perform
electrooculography; [1593] 12. A device to perform
electroretinography; and [1594] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1595] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1596] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in headaches and
difficulties in focusing. Accordingly, the media presented in a
VR/AR/MxR environment is modified, in order to decrease headaches
and difficulties in focusing, for the user and/or a group of
users.
[1597] At 3204, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3206, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in headaches and
difficulties in focusing while interacting with the VR, AR, and/or
MxR media: [1598] 1. Decreased Palpebral Fissure Rate of Change
[1599] 2. Low Distance Palpebral Fissure Resting State [1600] 3.
Low Distance Palpebral Fissure Active State [1601] 4. Increased
Blink Rate [1602] 5. Increased Rate of Change for Blink Rate [1603]
6. Increased Ratio of Partial Blinks to Full Blinks [1604] 7.
Increased Rate of Change for Pupil Size [1605] 8. Decreased Target
Relevancy for Pupil Initial and Final Position [1606] 9. Decreased
Target Relevancy for Gaze Direction [1607] 10. Decreased Target
Relevancy for Gaze Initial and Final Position [1608] 11. Increased
Rate of Divergence [1609] 12. Decreased Relevancy for Fixation
Initial and Final Position [1610] 13. Increased Fixation Duration
Rate of Change [1611] 14. Increased Fixation Rate [1612] 15.
Increased Fixation Count [1613] 16. Decreased Target Relevancy for
Saccade Initial and Final Position [1614] 17. Decreased Target
Relevancy for Saccade Angle [1615] 18. Decreased Saccade Magnitude
(Distance of Saccade) [1616] 19. Increased Ratio of
Anti-Saccade/Pro-Saccade [1617] 20. Increased Inhibition of Return
[1618] 21. Increased Saccade Velocity [1619] 22. Increased Saccade
Rate of Change [1620] 23. Increased Saccade Count (Number of
Saccades) [1621] 24. Increased Screen Distance [1622] 25. Decreased
Target Relevant Head Direction [1623] 26. Decreased Target Relevant
Head Fixation [1624] 27. Decreased Target Relevant Limb Movement
[1625] 28. Shift in Weight Distribution [1626] 29. Decreased
Alpha/Delta Brain Wave ratio [1627] 30. Increased Body Temperature
[1628] 31. Increased Respiration Rate [1629] 32. Low Oxygen
Saturation [1630] 33. Increased Heart Rate [1631] 34. Changes in
Blood Pressure [1632] 35. Increased Vocalizations [1633] 36.
Increased Reaction Time
[1634] The system may determine decrease in headaches and
difficulties in focusing while interacting with the media in a VR,
AR, and/or MX environment based upon one or more of the following
changes: [1635] 1. Increased Rate of Convergence [1636] 2.
Increased Fixation Duration [1637] 3. Increased Smooth Pursuit
[1638] 4. Increased Alpha/Theta Brain Wave ratio
[1639] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to: change
in facial expression (may be dependent on specific expression);
change in gustatory processing; change in olfactory processing; and
change in auditory processing.
[1640] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease headaches and
difficulties in focusing, these lists are not exhaustive and may
include other data acquisition components, types of data, and
changes in data.
[1641] At 3208, the changes in the plurality of data determined
over time may be used to determine a degree of change in headaches
and difficulties in focusing. The change in headaches and
difficulties in focusing may indicate either reduced headaches and
difficulties in focusing or enhanced headaches and difficulties in
focusing.
[1642] At 3210, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) headaches and
difficulties in focusing. In embodiments, the media may be modified
to address all the changes in data that reflect increase in
headaches and difficulties in focusing. In embodiments, a
combination of one or more of the following modifications may be
performed: [1643] 1. Increasing a contrast of the media [1644] 2.
Making an object of interest that is displayed in the media larger
in size [1645] 3. Increasing a brightness of the media [1646] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [1647] 5. Changing a
focal point of content displayed in the media to a more central
location [1648] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [1649] 7. Increasing an
amount of color in said media [1650] 8. Increasing a degree of
shade in objects shown in said media [1651] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[1652] One or more of the following indicators may be observed to
affirm a decrease in headaches and difficulties in focusing:
increased palpebral fissure height; decreased blink rate; decreased
rate of change for blink rate; decreased ratio of partial blinks to
full blinks; decreased rate of change for pupil size; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; decreased rate of divergence;
increased relevancy for fixation initial and final position;
decreased fixation duration rate of change; decreased fixation
rate; decreased fixation count; increased target relevancy for
saccade initial and final position; increased target relevancy for
saccade angle; increased saccade magnitude (task relevant);
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased saccade velocity; decreased saccade rate of
change; decreased saccade count; decreased screen distance;
increased target relevant head direction; increased target relevant
head fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1653] In embodiments, a specific percentage or a range of decrease
in headaches and difficulties in focusing, may be defined. In
embodiments, an additional value for data may be acquired at 3212,
in order to further determine change in data over time at 3214,
after the modifications have been executed at 3210. At 3216, a new
degree/percentage/range of decrease in headaches and difficulties
in focusing may be acquired. At 3218, the system determines whether
the decrease in headaches and difficulties in focusing is within
the specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 3210 to
further modify the media. Therefore, the media may be iteratively
modified 3210 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1654] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1655] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1656] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1657] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 16: Modifying Media in Order to Decrease Blurred Vision
and Myopia
[1658] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of blurred vision and myopia, which could be experienced by
the user. The data may be further utilized to modify VR/AR/MxR
media for the user in order to decrease the blurred vision and
myopia, such as but not limited to by minimizing visual, or any
other discomfort arising from the media experience. In an
embodiment, media is modified in real time for the user. In another
embodiment, data is saved and used to modify presentation of
VR/AR/MxR media to subsequent users with a similar data, or
subsequently to the user.
[1659] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease blurred vision and myopia, during interaction
with that media. FIG. 33 illustrates a flow chart describing an
exemplary process for modifying media in order to decrease blurred
vision and myopia, in accordance with some embodiments of the
present specification. At 3302, a first value for a plurality of
data, as further described below, is acquired. In embodiments, data
is acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [1660] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [1661] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [1662] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [1663]
4. One or more sensors configured to measure auditory processing;
[1664] 5. One or more sensors configured to measure gustatory and
olfactory processing; [1665] 6. One or more sensors to measure
pressure; [1666] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [1667] 8. A device to perform
electroencephalography; [1668] 9. A device to perform
electrocardiography; [1669] 10. A device to perform
electromyography; [1670] 11. A device to perform
electrooculography; [1671] 12. A device to perform
electroretinography; and [1672] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1673] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1674] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in blurred vision and
myopia. Accordingly, the media presented in a VR/AR/MxR environment
is modified, in order to decrease blurred vision and myopia, for
the user and/or a group of users.
[1675] At 3304, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3306, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in blurred vision and
myopia while interacting with the VR, AR, and/or MxR media: [1676]
1. Decreased Palpebral Fissure Rate of Change [1677] 2. Low
Distance Palpebral Fissure Resting State [1678] 3. Low Distance
Palpebral Fissure Active State [1679] 4. Increased Blink Rate
[1680] 5. Increased Rate of Change for Blink Rate [1681] 6.
Increased Ratio of Partial Blinks to Full Blinks [1682] 7.
Increased Rate of Change for Pupil Size [1683] 8. Decreased Target
Relevancy for Pupil Initial and Final Position [1684] 9. Decreased
Target Relevancy for Gaze Direction [1685] 10. Decreased Target
Relevancy for Gaze Initial and Final Position [1686] 11. Increased
Rate of Convergence [1687] 12. Decreased Relevancy for Fixation
Initial and Final Position [1688] 13. Increased Fixation Duration
Rate of Change [1689] 14. Increased Fixation Rate [1690] 15.
Increased Fixation Count [1691] 16. Decreased Target Relevancy for
Saccade Initial and Final Position [1692] 17. Decreased Target
Relevancy for Saccade Angle [1693] 18. Decreased Saccade Magnitude
(Distance of Saccade) [1694] 19. Increased Ratio of
Anti-Saccade/Pro-Saccade [1695] 20. Increased Inhibition of Return
[1696] 21. Increased Saccade Velocity [1697] 22. Increased Saccade
Rate of Change [1698] 23. Increased Saccade Count (Number of
Saccades) [1699] 24. Increased Screen Distance [1700] 25. Decreased
Target Relevant Head Direction [1701] 26. Decreased Target Relevant
Head Fixation [1702] 27. Decreased Target Relevant Limb Movement
[1703] 28. Shift in Weight Distribution [1704] 29. Decreased
Alpha/Delta Brain Wave ratio [1705] 30. Increased Body Temperature
[1706] 31. Increased Respiration Rate [1707] 32. Low Oxygen
Saturation [1708] 33. Increased Heart Rate [1709] 34. Low Blood
Pressure [1710] 35. Increased Reaction Time
[1711] The system may determine decrease in blurred vision and/or
myopia while interacting with the media in a VR, AR, and/or MX
environment based upon one or more of the following changes: [1712]
1. Increased Rate of Divergence [1713] 2. Increased Fixation
Duration [1714] 3. Increased Smooth Pursuit [1715] 4. Increased
Alpha/Theta Brain Wave ratio
[1716] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased vocalizations; change in facial expression (may be
dependent on specific expression); change in gustatory processing;
change in olfactory processing; and change in auditory
processing.
[1717] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease blurred vision
and/or myopia, these lists are not exhaustive and may include other
data acquisition components, types of data, and changes in
data.
[1718] At 3308, the changes in the plurality of data determined
over time may be used to determine a degree of change in blurred
vision and/or myopia. The change in blurred vision and/or myopia
may indicate either reduced blurred vision and/or myopia or
enhanced blurred vision and/or myopia.
[1719] At 3310, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) blurred vision and/or
myopia. In embodiments, the media may be modified to address all
the changes in data that reflect increase in blurred vision and/or
myopia. In embodiments, a combination of one or more of the
following modifications may be performed: [1720] 1. Increasing a
contrast of the media [1721] 2. Making an object of interest that
is displayed in the media larger in size [1722] 3. Increasing a
brightness of the media [1723] 4. Increasing an amount of an object
of interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view [1724] 5. Changing a focal point of content displayed in the
media to a more central location [1725] 6. Removing objects from a
field of view and measuring if a user recognizes said removal
[1726] 7. Increasing an amount of color in said media [1727] 8.
Increasing a degree of shade in objects shown in said media [1728]
9. Changing RGB values of said media based upon external data
(demographic or trending data)
[1729] One or more of the following indicators may be observed to
affirm a decrease in blurred vision and/or myopia: increased
palpebral fissure height; decreased blink rate; decreased rate of
change for blink rate; decreased ratio of partial blinks to full
blinks; decreased rate of change for pupil size; increased target
relevancy for pupil initial and final position; increased target
relevancy for gaze direction; increased target relevancy for gaze
initial and final position; decreased rate of convergence;
increased relevancy for fixation initial and final position;
decreased fixation duration rate of change; decreased fixation
rate; decreased fixation count; increased target relevancy for
saccade initial and final position; increased target relevancy for
saccade angle; increased saccade magnitude (task relevant);
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased saccade velocity; decreased saccade rate of
change; decreased saccade count; decreased screen distance;
increased target relevant head direction; increased target relevant
head fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1730] In embodiments, a specific percentage or a range of decrease
in blurred vision and/or myopia, may be defined. In embodiments, an
additional value for data may be acquired at 3312, in order to
further determine change in data over time at 3314, after the
modifications have been executed at 3310. At 3316, a new
degree/percentage/range of decrease in blurred vision and/or myopia
may be acquired. At 3318, the system determines whether the
decrease in blurred vision and/or myopia is within the specified
range or percentage. If it is determined that the decrease is
insufficient, the system may loop back to step 3310 to further
modify the media. Therefore, the media may be iteratively modified
3310 and overall performance may be measured, until a percentage of
improvement of anywhere from 1% to 10000%, or any increment
therein, is achieved.
[1731] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1732] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1733] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1734] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 17: Modifying Media in Order to Decrease
Heterophoria
[1735] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of heterophoria, which could be experienced by the user. The
data may be further utilized to modify VR/AR/MxR media for the user
in order to decrease the heterophoria, such as but not limited to
by minimizing visual, or any other discomfort arising from the
media experience. In an embodiment, media is modified in real time
for the user. In another embodiment, data is saved and used to
modify presentation of VR/AR/MxR media to subsequent users with a
similar data, or subsequently to the user.
[1736] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease heterophoria, during interaction with that media.
FIG. 34 illustrates a flow chart describing an exemplary process
for modifying media in order to decrease heterophoria, in
accordance with some embodiments of the present specification. At
3402, a first value for a plurality of data, as further described
below, is acquired. In embodiments, data is acquired by using at
least one camera configured to acquire eye movement data (rapid
scanning and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [1737]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [1738] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[1739] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1740] 4. One or more sensors configured
to measure auditory processing; [1741] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1742] 6.
One or more sensors to measure pressure; [1743] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1744] 8.
A device to perform electroencephalography; [1745] 9. A device to
perform electrocardiography; [1746] 10. A device to perform
electromyography; [1747] 11. A device to perform
electrooculography; [1748] 12. A device to perform
electroretinography; and [1749] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1750] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1751] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in heterophoria.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease heterophoria, for the user and/or a
group of users.
[1752] At 3404, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3406, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in heterophoria while
interacting with the VR, AR, and/or MxR media: [1753] 1. Decreased
Palpebral Fissure Rate of Change [1754] 2. Low Distance Palpebral
Fissure Resting State [1755] 3. Low Distance Palpebral Fissure
Active State [1756] 4. Increased Blink Rate [1757] 5. Increased
Rate of Change for Blink Rate [1758] 6. Increased Ratio of Partial
Blinks to Full Blinks [1759] 7. Increased Rate of Change for Pupil
Size [1760] 8. Decreased Target Relevancy for Pupil Initial and
Final Position [1761] 9. Decreased Target Relevancy for Gaze
Direction [1762] 10. Decreased Target Relevancy for Gaze Initial
and Final Position [1763] 11. Increased Rate of Divergence [1764]
12. Decreased Relevancy for Fixation Initial and Final Position
[1765] 13. Increased Fixation Duration Rate of Change [1766] 14.
Increased Fixation Count [1767] 15. Decreased Target Relevancy for
Saccade Initial and Final Position [1768] 16. Decreased Target
Relevancy for Saccade Angle [1769] 17. Decreased Saccade Magnitude
(Distance of Saccade) [1770] 18. Increased Ratio of
Anti-Saccade/Pro-Saccade [1771] 19. Increased Inhibition of Return
[1772] 20. Increased Saccade Velocity [1773] 21. Increased Saccade
Rate of Change [1774] 22. Increased Saccade Count (Number of
Saccades) [1775] 23. Increased Screen Distance [1776] 24. Decreased
Target Relevant Head Direction [1777] 25. Decreased Target Relevant
Head Fixation [1778] 26. Decreased Target Relevant Limb Movement
[1779] 27. Shift in Weight Distribution [1780] 28. Decreased
Alpha/Delta Brain Wave ratio [1781] 29. Low Oxygen Saturation
[1782] 30. Low Blood Pressure [1783] 31. Increased Reaction
Time
[1784] The system may determine decrease in heterophoria while
interacting with the media in a VR, AR, and/or MX environment based
upon one or more of the following changes: [1785] 1. Increased Rate
of Convergence [1786] 2. Increased Fixation Duration [1787] 3.
Increased Fixation Rate [1788] 4. Increased Smooth Pursuit [1789]
5. Increased Alpha/Theta Brain Wave ratio [1790] 6. Increased
Ocular Alignment
[1791] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased body temperature; increased respiration rate; increased
heart rate; increased vocalizations; change in facial expression
(may be dependent on specific expression); change in gustatory
processing; change in olfactory processing; and change in auditory
processing.
[1792] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease heterophoria, these
lists are not exhaustive and may include other data acquisition
components, types of data, and changes in data.
[1793] At 3408, the changes in the plurality of data determined
over time may be used to determine a degree of change in
heterophoria. The change in heterophoria may indicate either
reduced heterophoria or enhanced heterophoria.
[1794] At 3410, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) heterophoria. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in heterophoria. In embodiments, a
combination of one or more of the following modifications may be
performed: [1795] 1. Increasing a contrast of the media [1796] 2.
Making an object of interest that is displayed in the media larger
in size [1797] 3. Increasing a brightness of the media [1798] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [1799] 5. Changing a
focal point of content displayed in the media to a more central
location [1800] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [1801] 7. Increasing an
amount of color in said media [1802] 8. Increasing a degree of
shade in objects shown in said media [1803] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[1804] One or more of the following indicators may be observed to
affirm a decrease in heterophoria: increased palpebral fissure
height; decreased blink rate; decreased rate of change for blink
rate; decreased ratio of partial blinks to full blinks; decreased
rate of change for pupil size; increased target relevancy for pupil
initial and final position; increased target relevancy for gaze
direction; increased target relevancy for gaze initial and final
position; increased rate of divergence; increased relevancy for
fixation initial and final position; decreased fixation duration
rate of change; decreased fixation count; increased target
relevancy for saccade initial and final position; increased target
relevancy for saccade angle; increased saccade magnitude (task
relevant); decreased ratio of anti-saccade/pro-saccade; decreased
inhibition of return; decreased saccade velocity; decreased saccade
rate of change; decreased saccade count; decreased screen distance;
increased target relevant head direction; increased target relevant
head fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1805] In embodiments, a specific percentage or a range of decrease
in heterophoria, may be defined. In embodiments, an additional
value for data may be acquired at 3412, in order to further
determine change in data over time at 3414, after the modifications
have been executed at 3410. At 3416, a new degree/percentage/range
of decrease in heterophoria may be acquired. At 3418, the system
determines whether the decrease in heterophoria is within the
specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 3410 to
further modify the media. Therefore, the media may be iteratively
modified 3410 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1806] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1807] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1808] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1809] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 18: Modifying Media In Order To Decrease Fixation
Disparity
[1810] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of fixation disparity, which could be experienced by the
user. The data may be further utilized to modify VR/AR/MxR media
for the user in order to decrease the fixation disparity, such as
but not limited to by minimizing visual, or any other discomfort
arising from the media experience. In an embodiment, media is
modified in real time for the user. In another embodiment, data is
saved and used to modify presentation of VR/AR/MxR media to
subsequent users with a similar data, or subsequently to the
user.
[1811] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease fixation disparity, during interaction with that
media. FIG. 35 illustrates a flow chart describing an exemplary
process for modifying media in order to decrease fixation
disparity, in accordance with some embodiments of the present
specification. At 3502, a first value for a plurality of data, as
further described below, is acquired. In embodiments, data is
acquired by using at least one camera configured to acquire eye
movement data (rapid scanning and/or saccadic movement), blink rate
data, fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [1812] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [1813] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [1814] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [1815]
4. One or more sensors configured to measure auditory processing;
[1816] 5. One or more sensors configured to measure gustatory and
olfactory processing; [1817] 6. One or more sensors to measure
pressure; [1818] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [1819] 8. A device to perform
electroencephalography; [1820] 9. A device to perform
electrocardiography; [1821] 10. A device to perform
electromyography; [1822] 11. A device to perform
electrooculography; [1823] 12. A device to perform
electroretinography; and [1824] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1825] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1826] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in fixation disparity.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease fixation disparity, for the user
and/or a group of users.
[1827] At 3504, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3506, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in fixation disparity
while interacting with the VR, AR, and/or MxR media: [1828] 1.
Decreased Palpebral Fissure Rate of Change [1829] 2. Low Distance
Palpebral Fissure Resting State [1830] 3. Low Distance Palpebral
Fissure Active State [1831] 4. Increased Blink Rate [1832] 5.
Increased Rate of Change for Blink Rate [1833] 6. Increased Ratio
of Partial Blinks to Full Blinks [1834] 7. Increased Rate of Change
for Pupil Size [1835] 8. Decreased Target Relevancy for Pupil
Initial and Final Position [1836] 9. Decreased Target Relevancy for
Gaze Direction [1837] 10. Decreased Target Relevancy for Gaze
Initial and Final Position [1838] 11. Decreased Relevancy for
Fixation Initial and Final Position [1839] 12. Increased Fixation
Duration Rate of Change [1840] 13. Decreased Target Relevancy for
Saccade Initial and Final Position [1841] 14. Decreased Target
Relevancy for Saccade Angle [1842] 15. Decreased Saccade Magnitude
(Distance of Saccade) [1843] 16. Increased Ratio of
Anti-Saccade/Pro-Saccade [1844] 17. Increased Inhibition of Return
[1845] 18. Increased Saccade Velocity [1846] 19. Increased Saccade
Rate of Change [1847] 20. Increased Saccade Count (Number of
Saccades) [1848] 21. Increased Screen Distance [1849] 22. Decreased
Target Relevant Head Direction [1850] 23. Decreased Target Relevant
Head Fixation [1851] 24. Decreased Target Relevant Limb Movement
[1852] 25. Shift in Weight Distribution [1853] 26. Decreased
Alpha/Delta Brain Wave ratio [1854] 27. Low Oxygen Saturation
[1855] 28. Low Blood Pressure [1856] 29. Increased Reaction
Time
[1857] The system may determine decrease in fixation disparity
while interacting with the media in a VR, AR, and/or MxR
environment based upon the following changes: [1858] 1. Increased
Rate of Convergence [1859] 2. Increased Rate of Divergence [1860]
3. Increased Fixation Duration [1861] 4. Increased Fixation Rate
[1862] 5. Increased Fixation Count [1863] 6. Increased Smooth
Pursuit [1864] 7. Increased Alpha/Theta Brain Wave ratio
[1865] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased body temperature; increased respiration rate; increased
heart rate; increased vocalizations; change in facial expression
(may be dependent on specific expression); change in gustatory
processing; change in olfactory processing; and change in auditory
processing.
[1866] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease fixation disparity,
these lists are not exhaustive and may include other data
acquisition components, types of data, and changes in data.
[1867] At 3508, the changes in the plurality of data determined
over time may be used to determine a degree of change in fixation
disparity. The change in fixation disparity may indicate either
reduced fixation disparity or enhanced fixation disparity.
[1868] At 3510, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) fixation disparity. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in fixation disparity. In
embodiments, a combination of one or more of the following
modifications may be performed: [1869] 1. Increasing a contrast of
the media [1870] 2. Making an object of interest that is displayed
in the media larger in size [1871] 3. Increasing a brightness of
the media [1872] 4. Increasing an amount of an object of interest
displayed in the media shown in a central field of view and
decreasing said object of interest in a peripheral field of view
[1873] 5. Changing a focal point of content displayed in the media
to a more central location [1874] 6. Removing objects from a field
of view and measuring if a user recognizes said removal [1875] 7.
Increasing an amount of color in said media [1876] 8. Increasing a
degree of shade in objects shown in said media [1877] 9. Changing
RGB values of said media based upon external data (demographic or
trending data)
[1878] One or more of the following indicators may be observed to
affirm a decrease in fixation disparity: increased palpebral
fissure height; decreased blink rate; decreased rate of change for
blink rate; decreased ratio of partial blinks to full blinks;
decreased rate of change for pupil size; increased target relevancy
for pupil initial and final position; increased target relevancy
for gaze direction; increased target relevancy for gaze initial and
final position; increased relevancy for fixation initial and final
position; decreased fixation duration rate of change; increased
target relevancy for saccade initial and final position; increased
target relevancy for saccade angle; increased saccade magnitude
(task relevant); decreased ratio of anti-saccade/pro-saccade;
decreased saccade velocity; decreased saccade rate of change;
decreased saccade count; decreased screen distance; increased
target relevant head direction; increased target relevant head
fixation; increased target relevant limb movement; decrease in
shifts of weight distribution; increased alpha/delta brain wave
ratio; normal body temperature; normal respiration rate; 90-100%
oxygen saturation; normal heart rate; normal blood pressure; task
relevant vocalizations; task relevant facial expressions; decreased
reaction time; task relevant gustatory processing; task relevant
olfactory processing; and task relevant auditory processing.
[1879] In embodiments, a specific percentage or a range of decrease
in fixation disparity, may be defined. In embodiments, an
additional value for data may be acquired at 3512, in order to
further determine change in data over time at 3514, after the
modifications have been executed at 3510. At 3516, a new
degree/percentage/range of decrease in fixation disparity may be
acquired. At 3518, the system determines whether the decrease in
fixation disparity is within the specified range or percentage. If
it is determined that the decrease is insufficient, the system may
loop back to step 3510 to further modify the media. Therefore, the
media may be iteratively modified 3510 and overall performance may
be measured, until a percentage of improvement of anywhere from 1%
to 10000%, or any increment therein, is achieved.
[1880] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1881] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1882] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1883] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 19: Modifying Media in Order to Decrease
Vergence-Accommodation Disorder
[1884] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of vergence-accommodation disorder, which could be
experienced by the user. The data may be further utilized to modify
VR/AR/MxR media for the user in order to decrease the
vergence-accommodation disorder, such as but not limited to by
minimizing visual, or any other discomfort arising from the media
experience. In an embodiment, media is modified in real time for
the user. In another embodiment, data is saved and used to modify
presentation of VR/AR/MxR media to subsequent users with a similar
data, or subsequently to the user.
[1885] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease vergence-accommodation disorder, during
interaction with that media. FIG. 36 illustrates a flow chart
describing an exemplary process for modifying media in order to
decrease vergence-accommodation disorder, in accordance with some
embodiments of the present specification. At 3602, a first value
for a plurality of data, as further described below, is acquired.
In embodiments, data is acquired by using at least one camera
configured to acquire eye movement data (rapid scanning and/or
saccadic movement), blink rate data, fixation data, pupillary
diameter, palpebral (eyelid) fissure distance between the eyelids.
Additionally, the VR, AR, and/or MxR device can include one or more
of the following sensors incorporated therein: [1886] 1. One or
more sensors configured to detect basal body temperature, heart
rate, body movement, body rotation, body direction, and/or body
velocity; [1887] 2. One or more sensors configured to measure limb
movement, limb rotation, limb direction, and/or limb velocity;
[1888] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [1889] 4. One or more sensors configured
to measure auditory processing; [1890] 5. One or more sensors
configured to measure gustatory and olfactory processing; [1891] 6.
One or more sensors to measure pressure; [1892] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback; [1893] 8.
A device to perform electroencephalography; [1894] 9. A device to
perform electrocardiography; [1895] 10. A device to perform
electromyography; [1896] 11. A device to perform
electrooculography; [1897] 12. A device to perform
electroretinography; and [1898] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1899] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1900] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in vergence-accommodation
disorder. Accordingly, the media presented in a VR/AR/MxR
environment is modified, in order to decrease
vergence-accommodation disorder, for the user and/or a group of
users.
[1901] At 3604, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3606, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in
vergence-accommodation disorder while interacting with the VR, AR,
and/or MxR media: [1902] 1. Decreased Palpebral Fissure Rate of
Change [1903] 2. Low Distance Palpebral Fissure Resting State
[1904] 3. Low Distance Palpebral Fissure Active State [1905] 4.
Increased Blink Rate [1906] 5. Increased Rate of Change for Blink
Rate [1907] 6. Increased Ratio of Partial Blinks to Full Blinks
[1908] 7. Increased Rate of Change for Pupil Size [1909] 8.
Decreased Target Relevancy for Pupil Initial and Final Position
[1910] 9. Decreased Target Relevancy for Gaze Direction [1911] 10.
Decreased Target Relevancy for Gaze Initial and Final Position
[1912] 11. Decreased Relevancy for Fixation Initial and Final
Position [1913] 12. Increased Fixation Duration Rate of Change
[1914] 13. Decreased Target Relevancy for Saccade Initial and Final
Position [1915] 14. Decreased Target Relevancy for Saccade Angle
[1916] 15. Decreased Saccade Magnitude (Distance of Saccade) [1917]
16. Increased Ratio of Anti-Saccade/Pro-Saccade [1918] 17.
Increased Inhibition of Return [1919] 18. Increased Saccade
Velocity [1920] 19. Increased Saccade Rate of Change [1921] 20.
Increased Saccade Count (Number of Saccades) [1922] 21. Increased
Screen Distance [1923] 22. Decreased Target Relevant Head Direction
[1924] 23. Decreased Target Relevant Head Fixation [1925] 24.
Decreased Target Relevant Limb Movement [1926] 25. Shift in Weight
Distribution [1927] 26. Decreased Alpha/Delta Brain Wave ratio
[1928] The system may determine decrease in vergence-accommodation
disorder while interacting with the media in a VR, AR, and/or MxR
environment based upon one or more of the following changes: [1929]
1. Increased Rate of Convergence [1930] 2. Increased Rate of
Divergence [1931] 3. Increased Fixation Duration [1932] 4.
Increased Fixation Rate [1933] 5. Increased Fixation Count [1934]
6. Increased Smooth Pursuit [1935] 7. Increased Alpha/Theta Brain
Wave ratio
[1936] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased body temperature; increased respiration rate; low oxygen
saturation; increased heart rate; low blood pressure; increased
vocalizations; change in facial expression (may be dependent on
specific expression); increased reaction time; change in gustatory
processing; change in olfactory processing; and change in auditory
processing.
[1937] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease
vergence-accommodated disorder, these lists are not exhaustive and
may include other data acquisition components, types of data, and
changes in data.
[1938] At 3608, the changes in the plurality of data determined
over time may be used to determine a degree of change in
vergence-accommodated disorder. The change in vergence-accommodated
disorder may indicate either reduced vergence-accommodated disorder
or enhanced vergence-accommodated disorder.
[1939] At 3610, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) vergence-accommodated
disorder. In embodiments, the media may be modified to address all
the changes in data that reflect increase in vergence-accommodated
disorder. In embodiments, a combination of one or more of the
following modifications may be performed: [1940] 1. Increasing a
contrast of the media [1941] 2. Making an object of interest that
is displayed in the media larger in size [1942] 3. Increasing a
brightness of the media [1943] 4. Increasing an amount of an object
of interest displayed in the media shown in a central field of view
and decreasing said object of interest in a peripheral field of
view [1944] 5. Changing a focal point of content displayed in the
media to a more central location [1945] 6. Removing objects from a
field of view and measuring if a user recognizes said removal
[1946] 7. Increasing an amount of color in said media [1947] 8.
Increasing a degree of shade in objects shown in said media [1948]
9. Changing RGB values of said media based upon external data
(demographic or trending data) [1949] 10. Increase use of longer
viewing distances when possible [1950] 11. Match simulated distance
with focal distance more closely [1951] 12. Move objects in and out
of depth at a slower pace [1952] 13. Make existing object conflicts
less salient
[1953] One or more of the following indicators may be observed to
affirm a decrease in vergence-accommodated disorder: increased
palpebral fissure rate of change; decreased blink rate; decreased
rate of change for blink rate; decreased ratio of partial blinks to
full blinks; decreased rate of change for pupil size; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; increased relevancy for fixation
initial and final position; decreased fixation duration rate of
change; increased target relevancy for saccade initial and final
position; increased target relevancy for saccade angle; increased
saccade magnitude (task relevant); decreased ratio of
anti-saccade/pro-saccade; decreased inhibition of return; decreased
saccade velocity; decreased saccade rate of change; decreased
saccade count; decreased screen distance; increased target relevant
head direction; increased target relevant head fixation; increased
target relevant limb movement; decrease in shifts of weight
distribution; increased alpha/delta brain wave ratio; normal body
temperature; normal respiration rate; 90-100% oxygen saturation;
normal heart rate; normal blood pressure; task relevant
vocalizations; task relevant facial expressions; decreased reaction
time; task relevant gustatory processing; task relevant olfactory
processing; and task relevant auditory processing.
[1954] In embodiments, a specific percentage or a range of decrease
in vergence-accommodate disorder, may be defined. In embodiments,
an additional value for data may be acquired at 3612, in order to
further determine change in data over time at 3614, after the
modifications have been executed at 3610. At 3616, a new
degree/percentage/range of decrease in vergence-accommodate
disorder may be acquired. At 3618, the system determines whether
the decrease in vergence-accommodate disorder is within the
specified range or percentage. If it is determined that the
decrease is insufficient, the system may loop back to step 3610 to
further modify the media. Therefore, the media may be iteratively
modified 3610 and overall performance may be measured, until a
percentage of improvement of anywhere from 1% to 10000%, or any
increment therein, is achieved.
[1955] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[1956] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[1957] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[1958] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 20: Modifying Media in Order to Increase Positive
Emotion
[1959] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of positive emotion, which could be experienced by the user.
The data may be further utilized to modify VR/AR/MxR media for the
user in order to increase the positive emotion, such as but not
limited to by minimizing visual, or any other discomfort arising
from the media experience. In an embodiment, media is modified in
real time for the user. In another embodiment, data is saved and
used to modify presentation of VR/AR/MxR media to subsequent users
with a similar data, or subsequently to the user.
[1960] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to increase positive emotion, during interaction with that
media. FIG. 37 illustrates a flow chart describing an exemplary
process for modifying media in order to increase positive emotion,
in accordance with some embodiments of the present specification.
At 3702, a first value for a plurality of data, as further
described below, is acquired. In embodiments, data is acquired by
using at least one camera configured to acquire eye movement data
(rapid scanning and/or saccadic movement), blink rate data,
fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [1961] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [1962] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [1963] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [1964]
4. One or more sensors configured to measure auditory processing;
[1965] 5. One or more sensors configured to measure gustatory and
olfactory processing; [1966] 6. One or more sensors to measure
pressure; [1967] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [1968] 8. A device to perform
electroencephalography; [1969] 9. A device to perform
electrocardiography; [1970] 10. A device to perform
electromyography; [1971] 11. A device to perform
electrooculography; [1972] 12. A device to perform
electroretinography; and [1973] 13. One or more sensors configured
to measure Galvanic Skin Response.
[1974] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[1975] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in positive emotion.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to increase positive emotion, for the user
and/or a group of users.
[1976] At 3704, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3706, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect decrease in positive emotion
while interacting with the VR, AR, and/or MxR media: [1977] 1.
Decreased Palpebral Fissure Rate of Change [1978] 2. Low Distance
Palpebral Fissure Resting State [1979] 3. Low Distance Palpebral
Fissure Active State [1980] 4. Increased Blink Rate [1981] 5.
Increased Rate of Change for Blink Rate [1982] 6. Increased Ratio
of Partial Blinks to Full Blinks [1983] 7. Decreased Target
Relevancy for Pupil Initial and Final Position [1984] 8. Decreased
Target Relevancy for Gaze Direction [1985] 9. Decreased Target
Relevancy for Gaze Initial and Final Position [1986] 10. Decreased
Relevancy for Fixation Initial and Final Position [1987] 11.
Increased Fixation Duration [1988] 12. Increased Fixation Duration
Rate of Change [1989] 13. Decreased Target Relevancy for Saccade
Initial and Final Position [1990] 14. Decreased Target Relevancy
for Saccade Angle [1991] 15. Decreased Saccade Magnitude (Distance
of Saccade) [1992] 16. Increased Ratio of Anti-Saccade/Pro-Saccade
[1993] 17. Increased Inhibition of Return [1994] 18. Increased
Saccade Rate of Change [1995] 19. Increased Smooth Pursuit [1996]
20. Decreased Target Relevant Head Direction [1997] 21. Decreased
Target Relevant Head Fixation [1998] 22. Decreased Target Relevant
Limb Movement [1999] 23. Shift in Weight Distribution [2000] 24.
Decreased Alpha/Delta Brain Wave ratio [2001] 25. Increased Body
Temperature [2002] 26. Increased Respiration Rate [2003] 27. Low
Oxygen Saturation [2004] 28. Increased Heart Rate [2005] 29. Low
Blood Pressure [2006] 30. Increased Reaction Time
[2007] The system may determine increase in positive emotion while
interacting with the media in a VR, AR, and/or MxR environment
based upon one or more of the following changes: [2008] 1.
Increased Rate of Change for Pupil Size [2009] 2. Increased
Fixation Rate [2010] 3. Increased Fixation Count [2011] 4.
Increased Saccade Velocity [2012] 5. Increased Saccade Count
(Number of Saccades) [2013] 6. Increased Screen Distance [2014] 7.
Increased Alpha/Theta Brain Wave ratio [2015] 8. Increased
Vocalizations
[2016] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to:
increased rate of convergence; increased rate of divergence; change
in facial expression (may be dependent on specific expression);
increased reaction time; change in gustatory processing; change in
olfactory processing; and change in auditory processing.
[2017] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to increase positive emotion,
these lists are not exhaustive and may include other data
acquisition components, types of data, and changes in data.
[2018] At 3708, the changes in the plurality of data determined
over time may be used to determine a degree of change in positive
emotion. The change in positive emotion may indicate either reduced
positive emotion or enhanced positive emotion.
[2019] At 3710, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) positive emotion. In
embodiments, the media may be modified to address all the changes
in data that reflect decrease in positive emotion. In embodiments,
a combination of one or more of the following modifications may be
performed: [2020] 1. Increasing a contrast of the media [2021] 2.
Making an object of interest that is displayed in the media larger
in size [2022] 3. Increasing a brightness of the media [2023] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [2024] 5. Changing a
focal point of content displayed in the media to a more central
location [2025] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [2026] 7. Increasing an
amount of color in said media [2027] 8. Increasing a degree of
shade in objects shown in said media [2028] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[2029] One or more of the following indicators may be observed to
affirm an increase in positive emotion: increased palpebral fissure
height; decreased blink rate; decreased rate of change for blink
rate; decreased ratio of partial blinks to full blinks; increased
target relevancy for pupil initial and final position; increased
target relevancy for gaze direction; increased target relevancy for
gaze initial and final position; increased relevancy for fixation
initial and final position; decreased fixation duration; decreased
fixation duration rate of change; increased target relevancy for
saccade initial and final position; increased target relevancy for
saccade angle; increased saccade magnitude (task relevant);
decreased ratio of anti-saccade/pro-saccade; decreased inhibition
of return; decreased saccade rate of change; decreased smooth
pursuit; increased target relevant head direction; increased target
relevant head fixation; increased target relevant limb movement;
decrease in shifts of weight distribution; increased alpha/delta
brain wave ratio; normal body temperature; normal respiration rate;
90-100% oxygen saturation; normal heart rate; normal blood
pressure; task relevant vocalizations; task relevant facial
expressions; decreased reaction time; task relevant gustatory
processing; task relevant olfactory processing; and task relevant
auditory processing.
[2030] In embodiments, a specific percentage or a range of increase
in positive emotion, may be defined. In embodiments, an additional
value for data may be acquired at 3712, in order to further
determine change in data over time at 3714, after the modifications
have been executed at 3710. At 3716, a new degree/percentage/range
of increase in positive emotion may be acquired. At 3718, the
system determines whether the increase in positive emotion is
within the specified range or percentage. If it is determined that
the increase is insufficient, the system may loop back to step 3710
to further modify the media. Therefore, the media may be
iteratively modified 3710 and overall performance may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[2031] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[2032] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[2033] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[2034] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 21: Modifying Media in Order to Decrease Negative
Emotion
[2035] In an embodiment, data collected from the user, such as by
HMDs, or any other VR/AR/MxR system, is processed to determine the
extent of negative emotion, which could be experienced by the user.
The data may be further utilized to modify VR/AR/MxR media for the
user in order to decrease the negative emotion, such as but not
limited to by minimizing visual, or any other discomfort arising
from the media experience. In an embodiment, media is modified in
real time for the user. In another embodiment, data is saved and
used to modify presentation of VR/AR/MxR media to subsequent users
with a similar data, or subsequently to the user.
[2036] More specifically, the present specification describes
methods, systems and software that are provided to the user for
modifying displayed media in a VR, AR and/or MxR environment in
order to decrease negative emotion, during interaction with that
media. FIG. 38 illustrates a flow chart describing an exemplary
process for modifying media in order to decrease negative emotion,
in accordance with some embodiments of the present specification.
At 3802, a first value for a plurality of data, as further
described below, is acquired. In embodiments, data is acquired by
using at least one camera configured to acquire eye movement data
(rapid scanning and/or saccadic movement), blink rate data,
fixation data, pupillary diameter, palpebral (eyelid) fissure
distance between the eyelids. Additionally, the VR, AR, and/or MxR
device can include one or more of the following sensors
incorporated therein: [2037] 1. One or more sensors configured to
detect basal body temperature, heart rate, body movement, body
rotation, body direction, and/or body velocity; [2038] 2. One or
more sensors configured to measure limb movement, limb rotation,
limb direction, and/or limb velocity; [2039] 3. One or more sensors
configured to measure pulse rate and/or blood oxygenation; [2040]
4. One or more sensors configured to measure auditory processing;
[2041] 5. One or more sensors configured to measure gustatory and
olfactory processing; [2042] 6. One or more sensors to measure
pressure; [2043] 7. At least one input device such as a traditional
keyboard and mouse and or any other form of controller to collect
manual user feedback; [2044] 8. A device to perform
electroencephalography; [2045] 9. A device to perform
electrocardiography; [2046] 10. A device to perform
electromyography; [2047] 11. A device to perform
electrooculography; [2048] 12. A device to perform
electroretinography; and [2049] 13. One or more sensors configured
to measure Galvanic Skin Response.
[2050] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[2051] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in negative emotion.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, in order to decrease negative emotion, for the user
and/or a group of users.
[2052] At 3804, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3806, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. In embodiments of the current use case scenario, one or more
of the following changes, tracked and recorded by the hardware and
software of the system, may reflect increase in negative emotion
while interacting with the VR, AR, and/or MxR media: [2053] 1.
Decreased Palpebral Fissure Rate of Change [2054] 2. Low Distance
Palpebral Fissure Resting State [2055] 3. Low Distance Palpebral
Fissure Active State [2056] 4. Increased Rate of Change for Blink
Rate [2057] 5. Increased Ratio of Partial Blinks to Full Blinks
[2058] 6. Decreased Target Relevancy for Pupil Initial and Final
Position [2059] 7. Decreased Target Relevancy for Gaze Direction
[2060] 8. Decreased Target Relevancy for Gaze Initial and Final
Position [2061] 9. Increased Rate of Divergence [2062] 10.
Decreased Relevancy for Fixation Initial and Final Position [2063]
11. Increased Fixation Duration [2064] 12. Decreased Target
Relevancy for Saccade Initial and Final Position [2065] 13.
Decreased Target Relevancy for Saccade Angle [2066] 14. Decreased
Saccade Magnitude (Distance of Saccade) [2067] 15. Increased Ratio
of Anti-Saccade/Pro-Saccade [2068] 16. Increased Inhibition of
Return [2069] 17. Increased Smooth Pursuit [2070] 18. Decreased
Target Relevant Head Direction [2071] 19. Decreased Target Relevant
Head Fixation [2072] 20. Decreased Target Relevant Limb Movement
[2073] 21. Shift in Weight Distribution [2074] 22. Decreased
Alpha/Delta Brain Wave ratio [2075] 23. Increased Alpha/Theta Brain
Wave ratio [2076] 24. Increased Body Temperature [2077] 25.
Increased Respiration Rate [2078] 26. Low Oxygen Saturation [2079]
27. Increased Heart Rate [2080] 28. High Blood Pressure [2081] 29.
Increased Reaction Time
[2082] The system may determine decrease in negative emotion while
interacting with the media in a VR, AR, and/or MxR environment
based upon one or more of the following changes: [2083] 1.
Increased Blink Rate [2084] 2. Increased Rate of Change for Pupil
Size [2085] 3. Increased Rate of Convergence [2086] 4. Increased
Fixation Duration Rate of Change [2087] 5. Increased Fixation Rate
[2088] 6. Increased Fixation Count [2089] 7. Increased Saccade
Velocity [2090] 8. Increased Saccade Rate of Change [2091] 9.
Increased Saccade Count (Number of Saccades) [2092] 10. Increased
Screen Distance [2093] 11. Increased Vocalizations
[2094] Other changes may also be recorded and can be interpreted in
different ways. These may include, but are not limited to change in
facial expression (may be dependent on specific expression);
increased reaction time; change in gustatory processing; change in
olfactory processing; and change in auditory processing.
[2095] It should be noted that while the above-stated lists of data
acquisition components, types of data, and changes in data may be
used to determine variables needed to decrease negative emotion,
these lists are not exhaustive and may include other data
acquisition components, types of data, and changes in data.
[2096] At 3808, the changes in the plurality of data determined
over time may be used to determine a degree of change in negative
emotion. The change in negative emotion may indicate either reduced
negative emotion or enhanced negative emotion.
[2097] At 3810, media rendered to the user may be modified on the
basis of the degree of reduced (or enhanced) negative emotion. In
embodiments, the media may be modified to address all the changes
in data that reflect increase in negative emotion. In embodiments,
a combination of one or more of the following modifications may be
performed: [2098] 1. Increasing a contrast of the media [2099] 2.
Making an object of interest that is displayed in the media larger
in size [2100] 3. Increasing a brightness of the media [2101] 4.
Increasing an amount of an object of interest displayed in the
media shown in a central field of view and decreasing said object
of interest in a peripheral field of view [2102] 5. Changing a
focal point of content displayed in the media to a more central
location [2103] 6. Removing objects from a field of view and
measuring if a user recognizes said removal [2104] 7. Increasing an
amount of color in said media [2105] 8. Increasing a degree of
shade in objects shown in said media [2106] 9. Changing RGB values
of said media based upon external data (demographic or trending
data)
[2107] One or more of the following indicators may be observed to
affirm a decrease in negative emotion: increased palpebral fissure
height; decreased rate of change for blink rate; decreased ratio of
partial blinks to full blinks; increased target relevancy for pupil
initial and final position; increased target relevancy for gaze
direction; increased target relevancy for gaze initial and final
position; decreased rate of divergence; increased relevancy for
fixation initial and final position; decreased fixation duration;
increased target relevancy for saccade initial and final position;
increased target relevancy for saccade angle; increased saccade
magnitude (task relevant); decreased ratio of
anti-saccade/pro-saccade; decreased inhibition of return; decreased
smooth pursuit; increased target relevant head direction; increased
target relevant head fixation; increased target relevant limb
movement; decrease in shifts of weight distribution; increased
alpha/delta brain wave ratio; normal body temperature; normal
respiration rate; 90-100% oxygen saturation; normal heart rate;
normal blood pressure; task relevant vocalizations; task relevant
facial expressions; decreased reaction time; task relevant
gustatory processing; task relevant olfactory processing; and task
relevant auditory processing.
[2108] In embodiments, a specific percentage or a range of decrease
in negative emotion, may be defined. In embodiments, an additional
value for data may be acquired at 3812, in order to further
determine change in data over time at 3814, after the modifications
have been executed at 3810. At 3816, a new degree/percentage/range
of decrease in negative emotion may be acquired. At 3818, the
system determines whether the decrease in negative emotion is
within the specified range or percentage. If it is determined that
the decrease is insufficient, the system may loop back to step 3810
to further modify the media. Therefore, the media may be
iteratively modified 3810 and overall performance may be measured,
until a percentage of improvement of anywhere from 1% to 10000%, or
any increment therein, is achieved.
[2109] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[2110] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation.
[2111] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[2112] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
Example Use 22: Modifying Media Resulting from
Micro-Transactions
[2113] The sensory inputs determined and analyzed by the system may
eventually drive work and play engagements. In embodiments, sensory
information may be purchased from users and used to create sensory
data exchanges after adding value to the data through platforms
such as the SDEP. In embodiments of the present specification, the
senses of individuals and potential consumers may be measured and
monitored with the SDEP. The SDEP may provide data analysis
regarding trends on user-behavior based on sensory data, user data,
context of environment and location. Embodiments of SDEP may use
machine learning and deep learning techniques to develop predictive
recommendations in real-time and to allow the ability for a company
to use a real-time dynamic change to the content/advertisement to
personalize the experience to the consumer.
[2114] In embodiments, a user interfacing with an HMD or a similar
device is monitored. The user may be offered an option to share
their psychometric/sensory/biometric data, which may be further
used to better understand and customize the user's experience in
terms of type of content and other show suggestions. Assuming that
the user opts to share the data, in an embodiment, during the
interfacing, the SDEP determines the user to have a first sensory
state. In an embodiment, the sensory states include a first blink
rate, a first degree of pupil dilation, a first degree of saccadic
movement, any other eye movement, inter-palpebral fissure distance,
facial expression, and/or one or more other parameters such as the
ones discussed in the following User Case Scenarios. Additionally,
image processing of the content presented to the user at a certain
rate (frames/per second) may deconstruct the content in to core
psychometric raw data including color (RGB) values, contrast, size
and location of objects. Further, smart devices such as a fitness
monitoring band or a watch may obtain and provide data pertaining
heart rate and basal body temperature; smart clothing may provide
data pertaining respiratory rate and motion of body and limbs;
smart shoes may provide data pertaining weight/pressure
distribution. Similarly, there may be other sources that provide
various psychometric/sensory/biometric data of the user. Examples
of combinations of measures include comprehension levels, fatigue
levels, engagement levels, among others. Furthermore, the SDEP
determines a second sensory state, wherein the second sensory state
indicates changes in the measured psychometric/sensory/biometric
data relative to the first state. The
psychometric/sensory/biometric data measured by the SDEP may be
combined with characteristics of the visual data rendered to the
user in the same duration. The combination may be further used to
change a set of visuals and/or characteristics of the visuals in
order to receive a desired psychometric/sensory/biometric result,
indicating greater engagement, from the same user or across groups
of users that have a similar profile as the user discussed in this
embodiment. The SDEP thus utilizes each vision metric and
subsequent weight of each vision metric to develop a conversion
metric. Further, conversion metrics may be developed by the SDEP
with additional value with a successful gesture toward a desired
product. The user profiles may be grouped according to
demographics, or any other parameters.
[2115] In an example, a user watching a sports event and an in-ad
display using an HMD, is shown a branded shirt that is a certain
shade of the color blue (RGB value, luminance level). The user may
have elected to share advanced data settings in his HMD system.
During the initial 1 minute period--SDEP noted meta-data trends of
red at a certain RGB value was trending in males within user's
demographic. The trend may be AB tested in real time with the user
during the ad display by changing the Blue shirt to Red, and
further changing the Red to a specific shade of color Red. The
specific shade of color Red could be personalized to the user based
on the personal trends noted of the user that were shared through
the settings enabled in his HMD. The personal trends noted by the
SDEP, through the user's HMD may include quantifiable metrics for
user engagement, such as but not limited to decreased blink rate,
decrease saccadic movements, including anti-saccadic error prior to
full fixation of vision, dilation of pupil from steady state,
movement of head in relationship to where ad is placed in VR/AR/MxR
environment, with increase in heart rate, temperature and movement
toward the object.
[2116] In embodiments, the SDEP may interface with the user to
provide regular (periodic) updates to a separate entity, such as a
third party or the content provider, about the
psychometric/sensory/biometric user data shared with the SDEP. The
proportion of psychometric/biometric/sensory data shared and
duration of this share, may be used as a basis for a
micro-transaction or series of micro-transactions that occur
between the user and the separate entity. In embodiments, the SDEP
provides a platform to enable such micro-transactions with the
user. In some embodiments, user's revenue share may be proportional
to the amount of the psychometric/sensory/biometric data shared
regularly with the SDEP.
[2117] Therefore, in an embodiment, data collected from the user,
such as by HMDs, or any other VR/AR/MxR system, is processed to
determine the extent of data optionally shared by the user for the
knowledge of a separate entity, such as a third party or the source
of the content, which could be experienced by the user. The data
may be further utilized to modify VR/AR/MxR media for the user.
Additionally, the extent and duration of shared data may be
utilized to transact with the user. In one embodiment, a
transaction is in the form of a financial reward, where the amount
of the reward is proportional to the extent and duration of shared
data. In an embodiment, media is modified in real time for the
user. In another embodiment, data is saved and used to modify
presentation of VR/AR/MxR media to subsequent users with a similar
data, or subsequently to the user.
[2118] More specifically, the present specification describes
methods, systems and software that are provided to enable
micro-transactions with the user, involving rewards in exchange of
psychometric/sensory/biometric data, while also modifying displayed
media in a VR, AR and/or MxR environment, during interaction with
that media. FIG. 39 illustrates a flow chart describing an
exemplary process for modifying media while enabling a
micro-transaction, in accordance with some embodiments of the
present specification. At 3902, a first value for a plurality of
data, such as psychometric/sensory/biometric data of the user, is
acquired. In embodiments, data is acquired by using at least one
camera configured to acquire eye movement data (rapid scanning
and/or saccadic movement), blink rate data, fixation data,
pupillary diameter, palpebral (eyelid) fissure distance between the
eyelids. Additionally, the VR, AR, and/or MxR device can include
one or more of the following sensors incorporated therein: [2119]
1. One or more sensors configured to detect basal body temperature,
heart rate, body movement, body rotation, body direction, and/or
body velocity; [2120] 2. One or more sensors configured to measure
limb movement, limb rotation, limb direction, and/or limb velocity;
[2121] 3. One or more sensors configured to measure pulse rate
and/or blood oxygenation; [2122] 4. One or more sensors configured
to measure auditory processing; [2123] 5. One or more sensors
configured to measure gustatory and olfactory processing; [2124] 6.
One or more sensors to measure pressure; [2125] 7. At least one
input device such as a traditional keyboard and mouse and or any
other form of controller to collect manual user feedback.
[2126] In embodiments, the data acquired by a combination of these
devices may include data pertaining to one or more of: palpebral
fissure (including its rate of change, initial state, final state,
and dynamic changes); blink rate (including its rate of change
and/or a ratio of partial blinks to full blinks); pupil size
(including its rate of change, an initial state, a final state, and
dynamic changes); pupil position (including its initial position, a
final position); gaze direction; gaze position (including an
initial position and a final position); vergence (including
convergence vs divergence based on rate, duration, and/or dynamic
change); fixation position (including its an initial position, a
final position); fixation duration (including a rate of change);
fixation rate; fixation count; saccade position (including its rate
of change, an initial position, and a final position); saccade
angle (including it relevancy towards target); saccade magnitude
(including its distance, anti-saccade or pro-saccade); pro-saccade
(including its rate vs. anti-saccade); anti-saccade (including its
rate vs. pro-saccade); inhibition of return (including presence
and/or magnitude); saccade velocity (including magnitude, direction
and/or relevancy towards target); saccade rate, including a saccade
count, pursuit eye movements (including their initiation, duration,
and/or direction); screen distance (including its rate of change,
initial position, and/or final position); head direction (including
its rate of change, initial position, and/or final position); head
fixation (including its rate of change, initial position, and/or
final position); limb tracking (including its rate of change,
initial position, and/or final position); weight distribution
(including its rate of change, initial distribution, and/or final
distribution); frequency domain (Fourier) analysis;
electroencephalography output; frequency bands; electrocardiography
output; electromyography output; electrooculography output;
electroretinography output; galvanic skin response; body
temperature (including its rate of change, initial temperature,
and/or final temperature); respiration rate (including its rate of
change, initial rate, and/or final rate); oxygen saturation; heart
rate (including its rate of change, initial heart rate, and/or
final heart rate); blood pressure; vocalizations (including its
pitch, loudness, and/or semantics); inferred efferent responses;
respiration; facial expression (including micro-expressions);
olfactory processing; gustatory processing; and auditory
processing. Each data type may hold a weight when taken in to
account, either individually or in combination.
[2127] In embodiments, the system uses machine learning to be able
to discover new correlations between behavioral,
electrophysiological and/or autonomic measures, and the less
ambiguous measures. In some examples, some of the measures
described above are context specific. The system can correlate all
available measures and look for trends in negative emotion.
Accordingly, the media presented in a VR/AR/MxR environment is
modified, for the user and/or a group of users.
[2128] At 3904, a second value for the plurality of data, described
above, is acquired. In embodiments, the first value and the second
value are of the same data types, including the data types
described above. At 3906, the first and second values of data are
used to determine one or more changes in the plurality of data over
time. Different types of data changes may be recorded and can be
interpreted in different ways.
[2129] At 3908, the determined changes in the plurality of data
determined over time may be stored in a database. The
psychometric/sensory/biometric data measured by the SDEP may be
combined with characteristics of the visual data rendered to the
user in the same duration. The database may be maintained by the
SDEP and/or a separate entity such as a third party, a company, or
the content-provider of the content presented to the user. The data
is further processed in accordance with the various embodiments
described in the present specification, to model user behaviour and
modify the media. The combination may be further used to change a
set of visuals and/or characteristics of the visuals in order to
receive a desired psychometric/sensory/biometric result, indicating
greater engagement, from the same user or across groups of users
that have a similar profile as the user discussed in this
embodiment. The SDEP thus utilizes each vision metric and
subsequent weight of each vision metric to develop a conversion
metric. Further, conversion metrics may be developed by the SDEP
with additional value with a successful gesture toward a desired
product. The user profiles may be grouped according to
demographics, or any other parameters.
[2130] At 3910, the quantity and duration of changes in data
determined over time may be used to reward the user. The reward may
be provided by the separate entity in lieu of the user opting to
share their psychometric/sensory/biometric data.
Weighing Sources of Information
[2131] In one embodiment, the system applies a numerical weight or
preference to one or more of the above-described measures based on
their statistical significance for a given application, based on
their relevance and/or based on their degree of ambiguity in
interpretation.
[2132] In one embodiment, the system first determines a media
context within which the above listed data is collected. A context
may include a type of application, such as a puzzle game, action
game, movie, advertisement, strategy game, social network, or other
form of media application. Context appropriate measures that may
signal a particular state preference is given to those measures
with the fewest alternative interpretations. It is also preferable
to favor more general (less specific) states when interpreting
measures. For example, an increase in heart rate suggests at least
a heightened state of arousal, if not an increase in
comprehension.
[2133] In another embodiment, a particular condition or state is
determined independent of any other condition or state. The
condition or state may include fatigue, engagement, performance,
comprehension, symptoms associated with visually-induced motion
sickness secondary to visual-vestibular mismatch, symptoms
associated with post-traumatic stress disorder, double vision
related to accommodative dysfunction, vection due to unintended
peripheral field stimulation, vergence-accommodation disorders,
fixation disparity, blurred vision and myopia, headaches,
difficulties in focusing, disorientation, postural instability,
visual discomfort, eyestrain, dry eye, eye tearing, foreign body
sensation, feeling of pressure in the eyes, aching around the eyes,
nausea, stomach discomfort, potential phototoxicity from
overexposure to screen displays, and hormonal dysregulation arising
from excessive blue light exposure. In another embodiment, a
particular condition or state is determined in correlation with any
other state since the states are potentially correlated in certain
scenarios. For example, in certain applications, comprehension
requires engagement. However, in other applications, engagement may
not, necessarily, require comprehension. As fatigue increases,
engagement and comprehension will likely decrease. Engagement and
comprehension may also decrease without increasing fatigue if users
simply become disinterested. Accordingly, the measuring of these
states should be done independently and in parallel, followed by
considerations of the interaction of those measures.
[2134] The system preferably arithmetically weights the various
measures based upon context and a predefined hierarchy of measures,
wherein the first tier of measures has a greater weight than the
second tier of measures, which has a greater weight than the third
tier of measures. The measures are categorized into tiers based on
their degree of ambiguity and relevance to any given contextual
situation. It would be apparent to one skilled in the art that the
following measures are exemplary, and not exhaustive. Other
measures and/or combinations of measures may be used in different
tiers.
1. First Tier
[2135] a. Eye tracking measures of comprehension: The may include a
combination of measures of comprehension such as relevant fixation
(R_(Rel. Fix.)); mean of the absolute angle relative to relevant
regions (|.theta.|.sub.saccade-relevant) mean magnitude; component
relative to relevant regions (M.sub.saccade-relevent); fixation
correlations (C.sub.fixation); saccade correlations
(C.sub.saccade); Correlation of the listener's eye movements; an
area of focus (A.sub.focus). It may also include a level of
engagement based on the area of focus (A.sub.focus) where the area
of focus is significantly correlated with the spatial extent of the
stimulus in question. [2136] b. A significant amplitude magnitude
increases in cognitive EEG potentials (N2, N44, P300, P600)
resulting from infrequent, novel or unexpected stimuli [2137] c. A
transition from partially to completely open eyes (significant
increase in p.sub.both eyes open from a non-zero baseline) [2138]
d. Random or un-focused search characterized by significantly brief
fixation durations and significantly large saccade magnitudes
[2139] e. Combination of measures of engagement such as response
rate of less than 100% (or some lower, baseline rate of responding,
depending on context); and measure of fatigue such as reductions in
`performance` metrics over extended periods of activity and
decreasing proportion of responding, in appropriate contexts [2140]
f. Interactions away from a particular task or stimulus as
indicating lack of or disengagement [2141] g. Other measures of
engagement including relative time-on-task as the proportion of
time spent performing a task or processing a stimulus compared to
not; the ratio of interactions among available tasks as indicative
of time-on-task for each as a relative measure of engagement with
each task; and the ratio of fixation count and/or duration among
stimuli and/or visual regions as indicative of time-on-task as a
relative measure of engagement with each stimulus or visual region
[2142] h. Combination of measures of engagement and fatigue such as
significant shortening of the distance between a visual stimulus
and the user's eyes as an indication of engagement onset, and the
proportional deviation from baseline as an indication of level of
engagement; and yawning or other pronounced and discrete
respiration [2143] i. Measures of fatigue such as prolonged periods
of (mostly) closed eyes; and sudden vertical eye movements
2. Second Tier
[2143] [2144] a. Combination of measures such as a slowed blink
rate relative to an established baseline (f.sub.blink significantly
less than f.sub.blink) and a blink rate significantly less than
baseline. Also, combination of measures of increased blink rate,
such as significant increase in blink rate and transitions to
shorter and more frequency blinks. [2145] b. Onset of comprehension
as the point in time where, when applicable, the percent of correct
responses increases significantly [2146] c. Onset of comprehension
as the point when a target in a VR/AR/MxR media is correctly
identified and or located [2147] d. Combination of measures related
to the onset of comprehension as the end of a period of
significantly longer fixation durations (D.sub.fixation); the
duration of last fixation on the selected stimulus; and when a
choice is made, the duration of first fixation on any stimulus
[2148] e. Combination of measures such as a rapid and significant
increase in pupil diameter (S.sub.pupil), and significant pupil
dilation in the context of a choice task [2149] f. A significant
upward or downward deviation from average percent correct
responding as signaling engagement or disengagement, respectively
[2150] g. Adjustment of 3D gaze position towards the appropriate
depth (here considered separately from direction of gaze) to view a
stimulus as a signal of engagement with that stimulus; and 3D depth
of gaze towards infinity for extended periods as indicative of
fatigue [2151] h. Rigid fixation in the context of monitoring for
subtle changes or motion, or the precise onset of any change or
motion, as indicative of engagement; and reduced or held
respiration in the context of monitoring [2152] i. Changes in eye
movement patterns characterized by reduced saccade magnitude and
fixation frequency
3. Third Tier
[2152] [2153] a. Significant increase in GSR-ERP [2154] b.
significant increases in energy of an EEG in beta and gamma
frequency bands 16 Hz); increased bilateral phase synchrony of EEG
activity during choice tasks; and tradeoff where low frequency
(<10 Hz) EEG energy increases and high frequency (.gtoreq.10 Hz)
EEG energy decreases [2155] c. A significant increase in body
temperature and/or heart rate in association with delayed response
to a question of understanding. Also, measures related to increase
in autonomic arousal as indicative of increasing engagement, and
decreases in arousal as disengagement; and significant decreases in
heart rate and/or body temperature as indicative of fatigue [2156]
d. Any measure signaling significant comprehension, or onset of
comprehension; and significant reductions in comprehension,
engagement and other excitatory states in the context of prolonged
activity [2157] e. Significant signs of dry eye (e.g. low
tear-break-up-time) as indicative of ocular fatigue
[2158] The system further tracks any and all correlations of
different states, such as correlations between engagement,
comprehension and fatigue, to determine timing relationships
between states based on certain contexts. These correlations may be
immediate or with some temporal delay (e.g. engagement reduction is
followed after some period of time by fatigue increase). With the
embodiments of the present specification, any and all correlations
may be found whether they seem intuitive or not.
[2159] For any significant correlations that are found, the system
models the interactions of the comprising measures based on a
predefined algorithm that fits the recorded data. For example,
direct measures such as user's ability to detect, discriminate,
accuracy for position, accuracy for time, and others, are required
across various application. Indirect measures such as and not
limited to fatigue and endurance are also monitored across various
applications. However, a gaming application may find measures of
user's visual attention, ability to multi-track, and others, to be
of greater significance to determine rewards/points. Meanwhile, a
user's ability to pay more attention to a specific product or color
on a screen may lead to an advertising application to lay greater
significance towards related measures.
[2160] The above examples are merely illustrative of the many
applications of the system of present invention. Although only a
few embodiments of the present invention have been described
herein, it should be understood that the present invention might be
embodied in many other specific forms without departing from the
spirit or scope of the invention. Therefore, the present examples
and embodiments are to be considered as illustrative and not
restrictive, and the invention may be modified within the scope of
the appended claims.
* * * * *