U.S. patent application number 16/132656 was filed with the patent office on 2019-05-09 for gaze and saccade based graphical manipulation.
This patent application is currently assigned to Tobii AB. The applicant listed for this patent is Tobii AB. Invention is credited to Anders Clausen, Denny Ronngren, Robin Thunstrom.
Application Number | 20190138091 16/132656 |
Document ID | / |
Family ID | 59560287 |
Filed Date | 2019-05-09 |
![](/patent/app/20190138091/US20190138091A1-20190509-D00000.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00001.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00002.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00003.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00004.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00005.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00006.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00007.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00008.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00009.png)
![](/patent/app/20190138091/US20190138091A1-20190509-D00010.png)
View All Diagrams
United States Patent
Application |
20190138091 |
Kind Code |
A1 |
Thunstrom; Robin ; et
al. |
May 9, 2019 |
GAZE AND SACCADE BASED GRAPHICAL MANIPULATION
Abstract
According to the invention, a system for presenting graphics on
a display device is disclosed. The system may include an eye
tracking device for determining a gaze point of a user on a display
device. The system may also include a graphics processing device
for causing graphics to be displayed on the display device. The
graphics displayed on the display device may be modified such that
the graphics in an area including the gaze point of the user have
at least one modified parameter relative to graphics outside the
area. The size of the area may be based at least in part on an
amount of noise in the gaze point over time and at least one other
secondary factor.
Inventors: |
Thunstrom; Robin; (Danderyd,
SE) ; Ronngren; Denny; (Danderyd, SE) ;
Clausen; Anders; (Danderyd, SE) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Tobii AB |
Danderyd |
|
SE |
|
|
Assignee: |
Tobii AB
Danderyd
SE
|
Family ID: |
59560287 |
Appl. No.: |
16/132656 |
Filed: |
September 17, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15476650 |
Mar 31, 2017 |
10082870 |
|
|
16132656 |
|
|
|
|
15270783 |
Sep 20, 2016 |
9898081 |
|
|
15476650 |
|
|
|
|
14197171 |
Mar 4, 2014 |
9665171 |
|
|
15270783 |
|
|
|
|
61772366 |
Mar 4, 2013 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G09G 2350/00 20130101;
G09G 5/373 20130101; G09G 5/363 20130101; G09G 2354/00 20130101;
G06F 3/013 20130101; G09G 2340/0407 20130101; G09G 5/00 20130101;
G06T 5/00 20130101; G06F 3/039 20130101; G09G 2360/144 20130101;
G09G 5/37 20130101; G06F 2203/011 20130101 |
International
Class: |
G06F 3/01 20060101
G06F003/01; G09G 5/37 20060101 G09G005/37; G06T 5/00 20060101
G06T005/00; G09G 5/373 20060101 G09G005/373; G06F 3/039 20060101
G06F003/039 |
Claims
1. A system for presenting graphics on a display device, wherein
the system comprises: an eye tracking device for determining a gaze
point of a user on a display device; a graphics processing device
for causing graphics to be displayed on the display device; wherein
the graphics displayed on the display device are modified such that
the graphics in an area including the gaze point of the user have
at least one modified parameter relative to graphics outside the
area; and wherein the size of the area is based at least in part on
an amount of noise in the gaze point over time and at least one
other secondary factor.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 15/476,650, filed Mar. 31, 2017, which is a
continuation-in-part of U.S. patent application Ser. No.
15/270,783, filed Sep. 20, 2016, which is a continuation-in-part of
U.S. patent application Ser. No. 14/197,171, filed Mar. 4, 2014,
which claims priority to Provisional U.S. Patent Application No.
61/772,366, filed Mar. 4, 2013, the entire disclosures of which are
hereby incorporated by reference, for all purposes, as if fully set
forth herein.
BACKGROUND OF THE INVENTION
[0002] Graphical items may be used on a display to show data and
information to a viewer. These graphical items may include text,
images, and video. Graphical items in the area of computing are
well known and have been in use for many years. Recently, showing
three dimensional (3D) graphical items on a display has increased
in importance in areas such as gaming, modeling and movies.
[0003] When displaying graphical items, a system such as a computer
uses a processor in combination with memory to display the items on
a screen or other display device. Methods for displaying graphical
items vary, but typically they rely on a computer interpreting
commands to control a graphics processing device that provides
graphical items for display. The graphics processing device
typically contains custom hardware for this purpose including a
processor and memory. In some computer systems the graphics
processing device is fully integrated, and in others it is provided
as a separate component known as a graphics card.
[0004] Graphics processing devices have limits on their processing
power, usually quantified in terms of the amount of graphical items
that can be displayed on a screen at any given time. This is
typically limited by the capabilities of the hardware embodying the
graphics processing device, including processors, memory, and
communication channels. Additionally, the amount of graphical items
able to be displayed on a screen at a given point can be limited by
communication limits between the graphics processing device and
computer.
[0005] In many scenarios that require graphical items be displayed
on a screen, a user only focuses on a portion of the screen, and
therefore only a portion of the graphical items, an any given time.
Meanwhile, other graphical items continue to be displayed on the
remaining portions of the screen, which the user is not focused on.
This wastes valuable graphics processing device resources to
produce graphical items that cannot be fully appreciated by the
user because the visual acuity of a human drops dramatically
outside those images immediately focused on.
BRIEF DESCRIPTION OF THE INVENTION
[0006] In some embodiments, a system for presenting graphics on a
display device is provided. The system may include an eye tracking
device for determining a gaze point of a user on a display device.
The system may also include a graphics processing device for
causing graphics to be displayed on the display device. The
graphics displayed on the display device may be modified such that
the graphics in an area including the gaze point of the user have
at least one modified parameter relative to graphics outside the
area. The size of the area may be based at least in part on an
amount of noise in the gaze point over time and at least one other
secondary factor.
[0007] In another embodiment, a method for presenting graphics on a
display device is provided. The method may include determining,
with an eye tracking device, a gaze point of a user on a display
device. The method may also include causing, with a graphics
processing device, graphics to be displayed on the display device.
The graphics displayed on the display device may be modified such
that the graphics in an area including the gaze point of the user
have at least one modified parameter relative to graphics outside
the area. The size of the area may be based at least in part on an
amount of noise in the gaze point over time and at least one other
secondary factor.
[0008] In another embodiment, a non-transitory machine readable
medium having instructions thereon for presenting graphics on a
display device is provided. The instructions may be executable by
one or more processors to perform a method. The method may include
determining a gaze point of a user on a display device. The method
may also include causing graphics to be displayed on the display
device. The graphics displayed on the display device may be
modified such that the graphics in an area including the gaze point
of the user have at least one modified parameter relative to
graphics outside the area. The size of the area may be based at
least in part on an amount of noise in the gaze point over time and
at least one other secondary factor.
BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The present invention is described in conjunction with the
appended figures:
[0010] FIG. 1 is a block diagram of one possible system of the
invention for modifying an image based on a user's gaze point;
[0011] FIG. 2 is a view of a display device of the invention in
which image modification is occurring in response to a user's gaze
point;
[0012] FIG. 3A is a diagram of how image quality may continuously
vary within a modified image area;
[0013] FIG. 3B is a diagram of how image quality may vary in steps
within a modified image area;
[0014] FIG. 4 is a view of a display device of the invention in
which image modification is occurring in response to a detected
change in a user's gaze point;
[0015] FIG. 5 is flow diagram of one possible method of the
invention for modifying an image based on a user's gaze point;
[0016] FIG. 6 is a block diagram of an exemplary computer system
capable of being used in at least some portion of the devices or
systems of the present invention, or implementing at least some
portion of the methods of the present invention;
[0017] FIGS. 7A and 7B are diagrams showing a change in the
location of an area on a display device according to the change in
a user's gaze point;
[0018] FIG. 8 is a diagram demonstrating an eccentricity angle;
[0019] FIG. 9 is a flow diagram of another possible method of the
invention for modifying an image based on a user's gaze point and
other secondary factors;
[0020] FIG. 10 is a display screen showing one embodiment of the
invention where different regions may be rendered in varying
quality based at least on a user's gaze point; and
[0021] FIG. 11 is a display screen showing another embodiment of
the invention where different size and/or shaped regions may be
rendered in varying quality based at least on a user's gaze
point.
[0022] In the appended figures, similar components and/or features
may have the same numerical reference label. Further, various
components of the same type may be distinguished by following the
reference label by a letter that distinguishes among the similar
components and/or features. If only the first numerical reference
label is used in the specification, the description is applicable
to any one of the similar components and/or features having the
same first numerical reference label irrespective of the letter
suffix.
DETAILED DESCRIPTION OF THE INVENTION
[0023] The ensuing description provides exemplary embodiments only,
and is not intended to limit the scope, applicability or
configuration of the disclosure. Rather, the ensuing description of
the exemplary embodiments will provide those skilled in the art
with an enabling description for implementing one or more exemplary
embodiments. It being understood that various changes may be made
in the function and arrangement of elements without departing from
the spirit and scope of the invention as set forth in the appended
claims.
[0024] Specific details are given in the following description to
provide a thorough understanding of the embodiments. However, it
will be understood by one of ordinary skill in the art that the
embodiments may be practiced without these specific details. For
example, circuits, systems, networks, processes, and other elements
in the invention may be shown as components in block diagram form
in order not to obscure the embodiments in unnecessary detail. In
other instances, well-known circuits, processes, algorithms,
structures, and techniques may be shown without unnecessary detail
in order to avoid obscuring the embodiments.
[0025] Also, it is noted that individual embodiments may be
described as a process which is depicted as a flowchart, a flow
diagram, a data flow diagram, a structure diagram, or a block
diagram. Although a flowchart may describe the operations as a
sequential process, many of the operations can be performed in
parallel or concurrently. In addition, the order of the operations
may be re-arranged. A process may be terminated when its operations
are completed, but could have additional steps not discussed or
included in a figure. Furthermore, not all operations in any
particularly described process may occur in all embodiments. A
process may correspond to a method, a function, a procedure, a
subroutine, a subprogram, etc. When a process corresponds to a
function, its termination corresponds to a return of the function
to the calling function or the main function. Any detail present in
one discussed embodiment may or may not be present in other
versions of that embodiment or other embodiments discussed
herein.
[0026] The term "machine-readable medium" includes, but is not
limited to portable or fixed storage devices, optical storage
devices, wireless channels and various other mediums capable of
storing, containing or carrying instruction(s) and/or data. A code
segment or machine-executable instructions may represent a
procedure, a function, a subprogram, a program, a routine, a
subroutine, a module, a software package, a class, or any
combination of instructions, data structures, or program
statements. A code segment may be coupled to another code segment
or a hardware circuit by passing and/or receiving information,
data, arguments, parameters, or memory contents. Information,
arguments, parameters, data, etc. may be passed, forwarded, or
transmitted via any suitable means including memory sharing,
message passing, token passing, network transmission, etc.
[0027] Furthermore, embodiments of the invention may be
implemented, at least in part, either manually or automatically.
Manual or automatic implementations may be executed, or at least
assisted, through the use of machines, hardware, software,
firmware, middleware, microcode, hardware description languages, or
any combination thereof. When implemented in software, firmware,
middleware or microcode, the program code or code segments to
perform the necessary tasks may be stored in a machine readable
medium. A processor(s) may perform the necessary tasks.
[0028] In some embodiments of the invention, and with reference to
FIG. 1, a system 100 for presenting graphics on a display device
110 is provided. System 100 may include an eye tracking device 120
and a graphics processing device 130. In some embodiments, the
system may also include a processor/computer 140 which communicates
with, and controls, graphics processing device 130. In some
embodiments, any function of graphics processing device 130 may be
performed, in whole or in part, by processor/computer 140. Merely
by way of example, eye tracking device 120 may be provided integral
to, or in addition to, a personal computer 140 having graphics
processing device 130 and a central processing unit (in some
configurations, graphics processing device 130 and the central
processing unit are integrated). In other embodiments, eye tracking
device 120 may be provided integral to, or in addition to, a gaming
console 140 or other device having graphics processing device 130
and a central processing unit. Examples of gaming consoles include
those produced and available from Microsoft.TM., Nintendo.TM., or
Sony.TM.. In other embodiments, the eye tracking device 120 may be
provided integral to, or in addition to, a wearable headset such as
a Virtual Reality (VR) or Augmented Reality (AR) or the like.
Examples of wearable headsets include those produced and available
under the names Oculus Rift.TM., HTC Vive.TM., Sony
PlaystationVR.TM. and Fove.TM.. Thus, embodiments of the invention
may be applied to the presentation of graphics in any number of
possible devices and applications, including video display, video
games, video production and editing, video communications, computer
aided drafting and design, etc.
[0029] Eye tracking device 120 may be for determining at least one
of a gaze point of a user on display device 110, or a change in the
gaze point of the user on display device 110. Eye tracking devices
and methods, sometimes referred to as gaze detection systems and
methods, include, for example, products produced and available from
Tobii Technology AB, and which operate by using infrared
illumination and an image sensor to detect reflection from the eye
of a user. An example of such a gaze detection system is described
in U.S. Pat. No. 7,572,008, which is hereby incorporated by
reference, for all purposes, as if fully set forth herein. Other
alternative gaze detection systems may also be employed by the
invention, regardless of the technology behind the gaze detection
system. Eye tracking device 120 may employ its own processor or the
processor of another device (i.e., processor/computer 140) to
interpret and process data received. When an eye tracking device is
referred to herein, both possible methods of processing data are
referred to.
[0030] Graphics processing device 130 employed by various
embodiments of the invention may be for causing an image to be
displayed on display device 110. Graphics processing device 130 may
modify what image is displayed on display device 110 based at least
in part on the gaze point of the user on display device 110, or a
change in the gaze point of the user on display device 110, as
determined by eye tracking device 120. While in some embodiments a
separate non-included or non-integrated display device will be
controlled by the graphics processing device 130, other embodiments
may include or integrate display device 110.
[0031] The way in which the image displayed on display device 110
may be modified by graphics processing device 130 may vary
depending on the embodiment, but regardless, the way in which the
image is displayed may be intended to increase the image quality of
portions of the image on which a user's gaze, or focused gaze, is
directed, relative to those portions of the image to which the
user's gaze, or focused gaze, is not directed. In this manner, the
use of available resources of graphics processing device 130,
and/or other system resources, are maximized to deliver image
quality where it matters most on display device 110. To
demonstrate, FIG. 2 illustrates a display device 110 showing a
user's gaze point 210 and an area 220 around user's gaze point 210
in which embodiments of the invention may increase the quality of
the image relative to the remaining area 230 of the display device
110. Thus, in various embodiments of the invention, the quality of
the image produced across display device 110 may be increased in
area 220 relative to remaining area 230.
[0032] When "modification" of an image presented on display device
110 is discussed herein, it shall be understood that what is
intended is that a subsequent image displayed on display device
110, is different than a prior image displayed on display device
110. Thus, graphics processing device 130 and display device 110,
or other device(s) discussed herein, "modify" an image by causing a
first image to be displayed and then a second image to be displayed
which is different than the first image. Any other change of an
image discussed herein, for example, increasing or decreasing of
image quality, shall also be understood to mean that a subsequent
image is different than a prior image. Note that a change or
modification of an image may include changing or modifying only a
portion of the image. Thus, some portions of a prior image may be
the same as a subsequent image, while other portions may be
different. In other situations, the entirety of a prior image may
be different than a subsequent image. It shall be understood that
the modification of an area or an entirety of an image does not
necessarily mean every finite portion of the area or entirety are
changed (for example, each pixel), but rather that the area or
entirety may be changed in some potentially consistent, predefined,
or ordered manner (for example, the quality of the image is
changed).
[0033] Increasing the quality of the image may include increasing
the quality of any one or more of the below non-exclusive list of
graphical characteristics, and/or modifying the content of the
graphics, in addition to other possible characteristics known in
the art: [0034] Resolution: The number of distinct pixels that may
be displayed in one or more dimensions. For example,
"1024.times.768" means 1024 pixels displayed in height and 768
pixels displayed in width. [0035] Shading: Variation of the color
and brightness of graphical objects dependent on the artificial
lighting projected by light sources emulated by graphics processing
device 130. [0036] Texture-mapping: The mapping of graphical images
or "textures" onto graphical objects to provide the objects with a
particular look. The resolution of the textures influence the
quality of the graphical object to which they are applied. [0037]
Bump-mapping: Simulation of small-scale bumps and rough gradients
on surfaces of graphical objects. [0038] Fogging/participating
medium: The dimming of light when passing through non-clear
atmosphere or air. [0039] Shadows: Emulation of obstruction of
light. [0040] Soft shadows: Variance in shadowing and darkness
caused by partially obscured light sources. [0041] Reflection:
Representations of mirror-like or high gloss reflective surfaces.
[0042] Transparency/opacity (optical or graphic): Sharp
transmission of light through solid objects. [0043] Translucency or
Opacity: Highly scattered transmission of light through solid
objects. [0044] Refraction: Bending of light associated with
transparency. [0045] Diffraction: Bending, spreading and
interference of light passing by an object or aperture that
disrupts the light ray. [0046] Indirect illumination: Surfaces
illuminated by light reflected off other surfaces, rather than
directly from a light source (also known as global illumination).
[0047] Caustics (a form of indirect illumination): Reflection of
light off a shiny object, or focusing of light through a
transparent object, to produce bright highlights on another object.
[0048] Anti-aliasing: The process of blending the edge of a
displayed object to reduce the appearance of sharpness or jagged
lines. Typically an algorithm is used that samples colors around
the edge of the displayed object in to blend the edge to its
surroundings. [0049] Frame rate: For an animated image, the number
of individual frames presented during a certain period of time to
render movement within the image. [0050] 3D: Visual and temporal
characteristics of an image which cause the image to appear to be
three dimensional to a viewer. [0051] Animation quality: When an
animated image is presented, the detail of the animated image is
decreased in the periphery of view. [0052] Post processing effects
quality [0053] Refraction quality [0054] Three dimensional object
quality: Polygons shown based on the distance of the object to the
virtual camera
[0055] Other parameters beyond the quality of specific graphic
content which could be modified include the content of the graphics
presented itself. For instance, if normally a collection of objects
would be displayed in the periphery, fewer objects might be
displayed in the periphery under foveated rendering conditions
discussed herein. In some applications, this may reduce processing
requirements if objects and or activity occurring in the periphery
would not be sufficiently cognizable to a user under foveated
rendering conditions. In these embodiments, content of a certain
kind could be removed from peripheral presentation. For example,
objects having certain virtual characteristics (i.e., characters in
a video game, objects in a video game, an environment in a video
game, species of objects in other software applications, etc.),
certain geometric characteristics (i.e., shape, size, etc.),
certain virtual geographic characteristics (i.e., virtual
location), or any other specific characteristics might be displayed
or not displayed in peripheral rendering under foveated rendering
conditions.
[0056] The size and shape of the area of the image which may be
modified to appear in greater quality can vary depending on the
embodiment. Merely by way of example, the shape of the area may be
circular, oval, square, rectangular, or polygonal. In some
embodiments, the quality of the image within the area may be
uniformly increased. In other embodiments, the increase in quality
of the image may be greatest at the center of the area (i.e.,
proximate to the gaze point), and decrease towards the edges of the
area (i.e., distal to the gaze point), perhaps to match the quality
of the image surrounding the area. To demonstrate, FIG. 3A shows
how image quality may decrease in a linear or non-liner continuous
manner from the center of a gaze area outward, while FIG. 3B shows
how image quality may decrease in a stepped manner from the center
of a gaze area outward.
[0057] In some embodiments, modifying the image displayed on
display device 110 may occur in response to the detection of a
change in the gaze point. This may occur in a number of fashions,
some of which are described below.
[0058] In some embodiments, an entirety of the image may be
modified during the period of change in the gaze point of the user,
and once the change in the gaze point of the user ceases, either
the area around end gaze point of the user or a remainder of the
image (portions of the image not around the end gaze point) may be
modified. Merely by way of example, in one embodiment, the quality
of the entire image may be increased during movement of the user's
gaze (sometimes referred to as a saccade), but the increase in
quality may only be sustained in an area around the user's end gaze
point once the saccade is complete (i.e., the quality of the
remainder of the image may be decreased upon completion of the
saccade). In a different embodiment, the quality of the entire
image may be decreased during a saccade, but the decrease in
quality may only be sustained areas besides around the user's end
gaze point once the saccade is complete (i.e., the quality of the
area of the image around the user's end gaze point may be increased
upon completion of the saccade).
[0059] Additionally, the use of other system resources, including
for example processor/computer 140 and related resources, may also
be modified during a user's saccade. For example, non-graphical
operations may be supplemented by the resources of
processor/computer 140 and graphics processing device 130, during a
saccade. More specifically, during a saccade, non-graphical
calculations necessary for other system operations may proceed at
greater speed or efficiency because additional resources associated
with processor/computer 140 and graphics processing device 130 are
made available for such operations.
[0060] In some embodiments, modifying the image displayed on
display device 110 may include modifying a portion of the image in
an area around an anticipated gaze point of the user, potentially
by increasing the quality thereof. The anticipated gaze point may
be determined based on the change in the gaze point of the user. To
determine the anticipated gaze point of a user, eye tracking device
120 and/or another processor (i.e., the computer or game consoler's
processor), may determine a rate of the change in the gaze point of
the user on display device 110, and determine the anticipated gaze
point based at least in part on this rate of the change.
[0061] The rate of change of the gaze point of the user, also
referred to as the velocity or speed of a saccade by the user is
directly dependent on the total change in the gaze point of the
user (often referred to as the amplitude of the saccade). Thus, as
the intended amplitude of a user's saccade increases, so does the
speed of the saccade. While the saccade of a human user can be as
fast as 900.degree./second in humans, for saccades of less than or
about 60.degree., the velocity of a saccade is generally linearly
and directly dependent on the amplitude of the saccade. For
example, a 10.degree. amplitude is associated with a velocity of
300.degree./second and a 30.degree. amplitude is associated with a
velocity of 500.degree./second. For saccades of greater than
60.degree., the peak velocity starts to plateau toward the maximum
velocity attainable by the eye (900.degree./second). In response to
an unexpected stimulus, a saccade normally takes about 200
milliseconds (ms) to be initiated and then lasts from about 20 to
about 200 ms. Based on these relationships between saccade speed
and amplitude, embodiments of the invention may determine
anticipated gaze points based on saccade velocity. Other
predetermined models of mathematical relationships between saccade
speed and amplitude may also be employed by various embodiments of
the invention to determine an anticipated gaze point.
[0062] In some embodiments, the portion of the image modified
around the anticipated gaze point may also include the portion of
the image around the original gaze point (i.e., the gaze point from
which the user's saccade started). While the shape of the portion
of the image modified may be any of those shapes described above,
in some embodiments it may be a triangle or a trapezoidal shape
having a progressively greater width perpendicular to a direction
of the saccade as shown in FIG. 4.
[0063] In FIG. 4, display device 110 is shown, and an initial user
gaze point 410 is shown thereon. Prior to any change in initial
gaze point 410, embodiments of the invention may provide increased
graphics quality in area 420. When a user saccade, represented by
arrow 430, is detected by eye tracking device 120, the size and
shape of area 420 may change to accommodate both initial gaze point
410 and anticipated gaze point 440. The changed area 450, while
being triangular and/or trapezoidal in this embodiment, may be
shaped and sized differently in other embodiments. Merely by way of
example, an entire side of display device 110 from the initial gaze
point to the edges of the display in the direction of the saccade
may also be included in changed area 450 to account for more
possibilities of where the user's gaze point may end. In other
embodiments, a circular, oval, or square changed area 450 may be
provided. In yet other embodiments, changed area 450 may include
separate and distinct areas around the initial gaze point 410 and
anticipated gaze point 440.
[0064] In some embodiments, the size or shape of the area around
the gaze point for which an image is modified (or which remains
unmodified from a heightened quality in various embodiments), is
dynamic. This may occur based at least in part on any number of
factors, including the current location of the gaze point relative
to the image or display device. Merely by way of example, if a user
moves their gaze point to a certain portion of the screen, a
predefined portion of the screen may be modified via increased
quality therein (for example, a corner portion of the display
having a map of a virtual area in a video game). In some
embodiments, if enough user saccades having one or more predefined
characteristics are detected in predefined amount of time, the
entirety of the display may be modified to be rendered in greater
quality.
[0065] The performance of a computing system may be directly
influenced by the consumption of resources the system has at its
disposal. These resources include, but are not limited to,
processing power, memory size, memory access speed, and computer
bus speed. The display of information such as images and other
graphical items may directly require the use of such resources. The
higher the quality of this information, as has been previously
described, the greater the amount of resources required, or the
greater level of strain on existing resources. The present
invention seeks to decrease the consumption of these resources by
allocating graphical processing and execution resources first and
primarily to areas of display device 110 that can be readily
perceived in high definition by a user, as the areas in which high
definition information is actually displayed. Other areas of the
display device, which will not be, or cannot easily be, perceived
in high definition by a user may be allocated a lesser or remaining
amount of resources. Due to latency between the output of
information to display device 110 and the speed at which a user can
move their eyes and perceive information, it may be desirable to
provide a system in which the user does not perceive that there is
any change to the quality or definition of information being
displayed on the display device 110.
[0066] In some embodiments, the gaze point information may be
determined by, or based on information from, eye tracking device
120, and may be filtered to minimize the change in areas around the
gaze point for which an image may be modified. Referring to FIG. 7A
the embodiment includes display device 110 comprising an area 800
containing a sub-area 810. An objective of this and other
embodiments may be to maintain high definition and/or other
improved graphical rendering qualities in sub-area 810 and/or area
800 around any determined gaze points 210. An image may thus be
modified in area 800 such that it contains greater quality graphics
or other modified parameters as previously described. If it is
determined that a gaze point 210 remains within sub-area 810, the
quality of the graphics or the like in area 800 may be modified
such that the graphical quality of the images in area 800 are
displayed at a higher quality or the like than images outside of
area 800 on display device 110. If it is determined that the gaze
point is located outside of sub-area 810, as shown in FIG. 7B, a
new area 800A is defined containing a new sub-area 810A. New area
800A may then be modified to contain higher quality graphics or
other parameters. The invention then repeats, in that if it is
determined that the gaze point remains within new sub-area 810A,
area 800A remains constant, however if the gaze point is detected
outside of new sub-area 810A, area 800A is redefined.
[0067] In these or other embodiments, filtering of gaze information
may performed to ensure that relocation of area 800 is necessary.
For example, the system may determine a gaze point 210 is located
outside the sub-area 810 however it may perform no action (such as
relocating the area 800) until a predetermined number of gaze
points 210 are located outside the sub-area (for example 2, 5, 10,
50). Alternatively, the system could temporarily enlarge area 800
until it is certain the gaze point 210 is located within a certain
area. Additionally, predefined time periods may be established to
determine if gaze points 210 have moved outside of sub-area 810 for
at least those time periods prior to enlarging or changing area
800.
[0068] As described herein, when the amount of noise in a gaze
point determination is low, and gaze points 210 are located close
together, then area 800 and/or sub-area 810 may be smaller, and
closely correlated to the actual size of the field of gaze points
210. Conversely, if the amount of noise in a gaze point
determination is high, and gaze points 210 are dispersed and/or not
located close together, then area 800 and sub-area 810 may be
larger. Noise may include, merely by way of example, (1) errors or
differences between (a) the calculated/determined gaze point
location and (b) the actual gaze point location, as well as (2)
drift of the user's actual gaze point even when the user's
attention is actually directed toward a particular point on the
display device 110. Thus, when noise in a set of gaze point
determinations is low, there may be high precision in the gaze
point determination, and when noise in a set of gaze point
determinations is high, there may be low precision in the gaze
point determination.
[0069] In some embodiments, a number of different factors can
affect the amount of noise in the gaze point 210 determinations.
Consequently, the amount of noise in gaze point 210 determinations
can affect the size of area 800 and/or sub-area 810. Noise in gaze
point 210 determinations can be software and/or hardware based,
resulting from inaccuracies in determining the actual gaze point of
the user consistently, and/or can be user-based, resulting from
drift of the user's gaze from an actual point of interest that the
user is consciously focusing on. In either or both cases, multiple
and/or continuous gaze point 210 determinations will result in a
pattern of gaze points 210 as shown in FIGS. 7A and 7B.
[0070] Rather than attempting to determine which of the determined
gaze points 210 is the actual gaze point 210 of the user, instead,
as described above, area 800 and/or sub-area 810 may be sized to
contain all determined gaze points, or a large sub-total thereof
(i.e., perhaps excluding certain extreme outliers using statistical
and/or error checking routines such as a standard deviation method,
Z-score, and/or other methods known in the art). Because certain
conditions or "secondary factors" may be known before even a first
gaze point 210 is determined, or after only a minimal number of
gaze points 210 have been determined, it may be possible to at
least initially, if not continually, size area 800 and/or area 810
based on a number of secondary factors, thereby allowing for
rendering of area 800 and sub-area 810 immediately around the first
(or first few) determined gaze point(s) 210, without the need to
obtain additional gaze point 210 data.
[0071] Some secondary factors or conditions which may inform an
initial or continuing determination as to the size of area 800
and/or sub-area 810 may include (1) which user is viewing the
display, (2) environmental factors, (3) content of display device
110, and (4) where on display device 110 the user is gazing.
[0072] If the gaze determination system 100 is informed of which
user is using system 100, then that user may have a previously
determined amount of noise in their gaze data. Different users may
have inherently different and/or distinct noise levels due to the
particular characteristics of how their gaze wanders on display
device 110. From prior usage of system 100, it may thus be known
what size of area 800 and/or area 810 will be sufficient for areas
of increased graphical quality, without requiring analysis of
initial, or additional, gaze point 210 determinations.
[0073] Characteristics from the user's body, face, and/or eyes, as
detected by the eye tracking device imaging sensors or other
sensors, may also inform system 100 of likely characteristics of
the user's gaze, and therefore also be used to anticipate the
necessary size of area 800 and/or sub-area 810. For example,
slumped shoulders and/or squinted eyes may indicate the user is
tired (or in some other state), and therefore inform system 100
that area 800 and sub-area 810 should be adjusted accordingly
(i.e., made smaller or larger).
[0074] Furthermore, the pairing of the user with system 100 may
also be relevant to the determination. This meaning that the same
user operating two different gaze determination systems may have
different noise levels in their gaze point 210 determinations.
Thus, USER-A using SYSTEM-A may be associated with a first amount
of gaze point 210 noise, while the same USER-A using SYSTEM-B may
be associated with a second amount of gaze point 210 noise, with
each amount of gaze point 210 noise corresponding to a differently
sized area 800 and/or sub-area 810. Additionally, the shape of area
800 and/or sub-area 810 may also correspond to different users
and/or different user/system pairs.
[0075] Environmental factors may also play a part in an initial or
continuing determination of the necessary size for area 800 and/or
sub-area 810. Merely by way of example, if system 100 determines
that the environment around the user is bright, dim, humid, dry,
polluted (particulate matter in the air), windy, subject to extreme
or unusual temperatures, or oscillating between various states
(e.g., there are flashing lights in the area), system 100 may
adjust the size of area 800 and sub-area 810 based upon such
information. Thus system 100 may be able to anticipate a certain
amount of noise in gaze point 210 determination under such
conditions, and anticipatorily adjust the size of area 800 and/or
sub-area 810. Other environmental characteristics such as time of
day may also affect anticipated noise. Perhaps because users are
more likely to be tired during morning or evening hours, and have
more or less drift associated with their gaze.
[0076] The content displayed on display device 110 may also may
also inform system 100 as to the likely noise in gaze point 210
determination. Merely by way of example, brighter or darker images
displayed on display device 110 may be known to cause more or less
noise in gaze determination. In other example, text may make more
or less noise likely for a given user. Other examples include the
nature of the graphical content. Merely by way of example, dynamic
fast moving images may produce more or less noise than static or
slow moving images. System 100 may be able to take this into
account to anticipate what size area 800 and sub-area 810 are
necessary.
[0077] Finally, where a user is gazing on the display device 110
may also affect likely noise levels. Merely by way of example, when
gaze point 210 of the user is near the boundaries of display device
110, noise may be amplified compared to when the user's gaze point
210 is near the center of display device 110. Noise may also vary
less or more greatly depending on whether gaze point 210 is to the
left or right of center versus being above or below center of the
display.
[0078] Thus, in one embodiment, as shown in FIG. 9, a method of one
embodiment of the invention may include, at block 910, displaying
graphics on a display device. At block 920, and continually
thereafter, one or more gaze points of the user may be determined.
At block 930, secondary factors as discussed above may be
determined. At block 940, the graphics displayed may be modified
based at least on the determined gaze point(s) and the determined
secondary factor(s).
[0079] In some embodiments, another method of determining the size
of area 800 and/or sub-area-810 may be to employ predefined regions
on display device 110 which would dictate the size and/or shape of
area 800 and/or sub-area 810. Thus, display device 110 may be
divided into a plurality of predefined regions within the display
area. For example, as shown in FIG. 10, display device 110 may be
divided into sixteen predefined regions 1010. Predefined regions
1010 may be fixed by hardware and/or software, or may be manually
set by the user before the usage. Different software programs may
also be able to predefine regions 1010 based on the particular
needs of the software program.
[0080] If all of the user's gaze points 210 are located within a
particular predefined region 1010, then the particular predefined
region 1010 is rendered in a higher quality than compared to the
other predefined regions 1010 in which no gaze position is
included. In another example, if the gaze positions are located in
more than one predefined region 1010, then all predefined regions
1010 in which gaze positions are located are rendered in a higher
quality than compared to the other predefined regions 1010 in which
no gaze position is included. In other related embodiments,
variations in the above examples may be present. Merely by way of
example, the predefined regions 1010 may be divided from just a
portion of display device 110 instead of the entire display area of
display device 110.
[0081] Such methods of using predefined regions 1010 for foveated
rendering may have particular merit in some virtual and/or
augmented reality headsets that are not equipped with especially
accurate and/or powerful eye tracking systems. Low quality eye
tracking in such embodiments may exist in some part due to a low
maximum frame rate of the eye tracking camera employed. In such
situation, the system may only provide a rough determination of
gaze positions, which may result in the low precision eye tracking
data (i.e., the determined gaze positions are widely spread). In
order to provide foveated rendering for such low quality eye
tracking situations, we predefined regions 1010 may be
employed.
[0082] In an alternative embodiment shown in FIG. 11, the method
may determine gaze positions 1110 first, and then a high quality
region 1120 (size and shape) is determined based on at least a
portion (for example, majority) of the gaze positions. A plurality
of surrounding regions 1130, surrounding initial high quality
region 1120 are then determined based on the determined size and
shape of the high quality region 1120. The display quality of those
surrounding regions are lower than the determined high quality
region 1120. In a special case, each of the surrounding regions
1130 may have differing display quality, depending on various
rendering conditions (e.g. displayed object characteristics,
displayed object positioning, distance to the high quality region
1120, etc.).
[0083] In another possible embodiment, if a sub-area 810 represents
a certain zone where the gaze point 210 of the user is located, and
to which a first degree of increased rendering quality is applied,
surrounding zones to may be represented by area 800 to which a
second degree of increased rendering quality is applied (the second
degree being less significant than the first degree).
[0084] In some embodiments, the size of area 810 may be based on an
eccentricity angle of about 1-5 degrees from the currently
determined gaze point. In exemplary embodiments, the angle may be
about 3 degrees. Although the concept of an eccentricity angle
would be well understood by a person of skill in the art, for
demonstration purposes, its use in the present embodiment will now
be described with reference to FIG. 8.
[0085] The eccentricity angle .theta. represents a person's fovea
vision. In some embodiments, it may be preferable that area 800 is
larger than the area labeled "object" in FIG. 8. This means that
there may be a high probability that while a person's gaze point
210 remains within the sub-area 810, that person will not be able
to perceive information outside the area 800 at high quality. The
size of the area labeled "object" on display device 110 is
primarily dependent on the physical size of display device 110 and
the distance between the eye(s) and display device 110.
[0086] The size of area 800 may be modified to affect performance,
as the greater the size of area 800, the more system resources are
required to render graphics at higher quality. By way of example,
the area 800 may be of such size so as to fit within the display 2
to 12 times. This size may be optionally linked directly to the
size of display device 110, or to the eccentricity angle, such that
the size may scale efficiently. In a further embodiment, gaze point
210 may be determined after adjusting information obtained by eye
tracking device 120 to remove noise. Different individuals may have
different levels of noise in the gaze information obtained by eye
tracking device 120 (for example, due to wandering of their eye(s)
about the gaze point). If an individual has a low level of noise,
area 800 may be smaller, and thus performance of the system on
which the present embodiment is being practiced may be
increased.
[0087] In another embodiment of the invention, a non-transitory
computer readable medium having instructions thereon for presenting
graphics on display device 110 is provided. The instructions may be
executable by one or more processors to at least display an image
on display device 110. The instructions may also be executable to
receive information from eye tracking device 120 indicative of at
least one of a gaze point of a user on display device 110, or a
change in the gaze point of the user on display device 110. The
instructions may further be executable to cause graphics processing
device 130 to modify the image displayed on display device 110
based at least in part on the gaze point of the user on display
device 110, or the change in the gaze point of the user on display
device 110. Thus, a non-transitory computer readable medium able to
implement any of the features described herein in relation to other
embodiments is also provided.
[0088] In another embodiment of the invention, a method 500 for
presenting graphics on display device 110 is provided as shown in
FIG. 5. At step 510, method 500 may include displaying an image on
display device 110. At step 520, method 500 may also include
receiving information from eye tracking device 120 indicative of at
least one of a gaze point of a user on display device 110, or a
change in the gaze point of the user on display device 110. At step
530, method 500 may further include causing graphics processing
device 130 to modify the image displayed on display device 110
based at least in part on the gaze point of the user on display
device 110, or the change in the gaze point of the user on display
device 110. Step 530 may include, at step 533, increasing the
quality of the image in an area around the gaze point of the user,
relative to outside the area. Step 530 may also include, at step
536, decreasing the quality of the image outside an area around the
gaze point of the user, relative to inside the area. Thus, a method
to implement any of the features described herein in relation to
other embodiments is also provided.
[0089] In some embodiments, the systems and methods described
herein may be toggled on and off by a user, possibly to account for
multiple additional viewers of display device 110 being present. In
other embodiments, the systems and methods described herein may
automatically toggle on when only one user is viewing display
device 110 (as detected by eye tracking device 120), and off when
more than one user is viewing display device 110 (as detected by
eye tracking device 120). Additionally, in some embodiments, the
systems and methods described herein may allow for reduction in
rendering quality of an entire display device 110 when no viewers
are detected, thereby saving system resources and power consumption
when display device 110 is not the primary focus of any viewer.
[0090] In other embodiments, the systems and methods described
herein may allow for modifying multiple portions of an image on
display device 110 to account for multiple viewers as detected by
eye tracking device 120. For example, if two different users are
focused on different portions of display device 110, the two
different areas of the image focused on may be rendered in higher
quality to provide enhanced viewing for each viewer.
[0091] In yet other embodiments, data associated with an image may
inform the systems and methods described herein to allow prediction
of which areas of an image may likely be focused on next by the
user. This data may supplement data provided by eye tracking device
120 to allow for quicker and more fluid adjustment of the quality
of the image in areas likely to be focused on by a user. For
example, during viewing of a sporting event, a picture-in-picture
of an interview with a coach or player may be presented in a corner
of the image. Metadata associated with the image feed may inform
the systems and methods described herein of the likely importance,
and hence viewer interest and likely focus, in the sub-portion of
the image.
[0092] FIG. 6 is a block diagram illustrating an exemplary computer
system 600 in which embodiments of the present invention may be
implemented. This example illustrates a computer system 600 such as
may be used, in whole, in part, or with various modifications, to
provide the functions of eye tracking device 120, graphics
processing device 130, the game console, the processor/computer
140, and/or other components of the invention such as those
discussed above. For example, various functions of eye tracking
device 120 and associated processors may be controlled by the
computer system 600, including, merely by way of example, tracking
a user's gaze point, determining an anticipated gaze point,
controlling graphics processing device 130, etc.
[0093] The computer system 600 is shown comprising hardware
elements that may be electrically coupled via a bus 690. The
hardware elements may include one or more central processing units
610, one or more input devices 620 (e.g., a mouse, a keyboard,
etc.), and one or more output devices 630 (e.g., a display device,
a printer, etc.). The computer system 600 may also include one or
more storage device 640. By way of example, storage device(s) 640
may be disk drives, optical storage devices, solid-state storage
device such as a random access memory ("RAM") and/or a read-only
memory ("ROM"), which can be programmable, flash-updateable and/or
the like.
[0094] The computer system 600 may additionally include a
computer-readable storage media reader 650, a communications system
660 (e.g., a modem, a network card (wireless or wired), an
infra-red communication device, Bluetooth.TM. device, cellular
communication device, etc.), and working memory 680, which may
include RAM and ROM devices as described above. In some
embodiments, the computer system 600 may also include a processing
acceleration unit 670, which can include a digital signal
processor, a special-purpose processor and/or the like.
[0095] The computer-readable storage media reader 650 can further
be connected to a computer-readable storage medium, together (and,
optionally, in combination with storage device(s) 640)
comprehensively representing remote, local, fixed, and/or removable
storage devices plus storage media for temporarily and/or more
permanently containing computer-readable information. The
communications system 660 may permit data to be exchanged with a
network, system, computer and/or other component described
above.
[0096] The computer system 600 may also comprise software elements,
shown as being currently located within a working memory 680,
including an operating system 684 and/or other code 688. It should
be appreciated that alternate embodiments of a computer system 600
may have numerous variations from that described above. For
example, customized hardware might also be used and/or particular
elements might be implemented in hardware, software (including
portable software, such as applets), or both. Furthermore,
connection to other computing devices such as network input/output
and data acquisition devices may also occur.
[0097] Software of computer system 600 may include code 688 for
implementing any or all of the function of the various elements of
the architecture as described herein. For example, software, stored
on and/or executed by a computer system such as system 600, can
provide the functions of eye tracking device 120, graphics
processing device 130, the game console, the processor/computer,
and/or other components of the invention such as those discussed
above. Methods implementable by software on some of these
components have been discussed above in more detail.
[0098] The invention has now been described in detail for the
purposes of clarity and understanding. However, it will be
appreciated that certain changes and modifications may be practiced
within the scope of the appended claims.
* * * * *