U.S. patent application number 16/428686 was filed with the patent office on 2019-09-19 for attention-based rendering and fidelity.
The applicant listed for this patent is SONY INTERACTIVE ENTERTAINMENT AMERICA LLC. Invention is credited to Andres Ramos Cevallos, Ryan Halvorson, Paul Timm.
Application Number | 20190286216 16/428686 |
Document ID | / |
Family ID | 52582479 |
Filed Date | 2019-09-19 |
United States Patent
Application |
20190286216 |
Kind Code |
A1 |
Timm; Paul ; et al. |
September 19, 2019 |
ATTENTION-BASED RENDERING AND FIDELITY
Abstract
Methods and systems for attention-based rendering on an
entertainment system are provided. A tracking device captures data
associated with a user, which is used to determine that a user has
reacted (e.g., visually or emotionally) to a particular part of the
screen. The processing power is increased in this part of the
screen, which increases detail and fidelity of the graphics and/or
updating speed. The processing power in the areas of the screen
that the user is not paying attention to is decreased and diverted
from those areas, resulting in decreased detail and fidelity of the
graphics and/or decreased updating speed.
Inventors: |
Timm; Paul; (San Diego,
CA) ; Cevallos; Andres Ramos; (San Diego, CA)
; Halvorson; Ryan; (San Diego, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
SONY INTERACTIVE ENTERTAINMENT AMERICA LLC |
San Mateo |
CA |
US |
|
|
Family ID: |
52582479 |
Appl. No.: |
16/428686 |
Filed: |
May 31, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15659254 |
Jul 25, 2017 |
10310583 |
|
|
16428686 |
|
|
|
|
15180275 |
Jun 13, 2016 |
9715266 |
|
|
15659254 |
|
|
|
|
14014199 |
Aug 29, 2013 |
9367117 |
|
|
15180275 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G09G 2330/021 20130101;
Y02D 10/153 20180101; G09G 5/363 20130101; G06F 3/015 20130101;
G06F 3/013 20130101; G09G 2354/00 20130101; G06F 1/3265 20130101;
Y02D 10/00 20180101; G06F 1/3231 20130101; G09G 5/391 20130101;
G09G 2340/0407 20130101; Y02D 10/173 20180101 |
International
Class: |
G06F 1/3231 20060101
G06F001/3231; G06F 1/3234 20060101 G06F001/3234; G09G 5/36 20060101
G09G005/36; G06F 3/01 20060101 G06F003/01; G09G 5/391 20060101
G09G005/391 |
Claims
1. A method for updating displayed content, the method comprising:
identifying a gaze direction of a user while an eye of the user is
focused on displayed content in a display based on data received by
an optical sensor, the gaze direction corresponding to a first
location within the display; tracking movement of the eye of the
user to a second location within the display based on changes in
the identified gaze direction of the user; detecting that the user
has indicated an instruction while the gaze direction corresponds
to the second location; and updating at least the displayed content
at the second location according to the instruction indicated by
the user.
2. The method of claim 1, wherein detecting that the user has
indicated the instruction comprises receiving vocal input from the
user, wherein updating the displayed content is based on the vocal
input.
3. The method of claim 1, further comprising increasing processing
power to render an object at the second location with greater
detail or speed.
4. The method of claim 1, further comprising decreasing processing
power used to render displayed content at another location within
the display, wherein the displayed content at the other location is
rendered with less detail than the displayed content at the second
location.
5. The method of claim 1, wherein detecting that the user has
indicated the instruction is based on acceleration data detected by
an accelerometer of a control device; and identifying the indicated
instruction includes identifying a position of the control device
in three dimensional (3D) space based on the acceleration sensor
data.
6. The method of claim 3, wherein detecting that the user has
indicated the instruction is further based on infrared sensor data
detected by an infrared sensor, wherein identifying the position of
the control device in 3D space is further based on the received
infrared sensor data.
7. The method of claim 1, wherein detecting that the user has
indicated the instruction is based on movement data detected by a
sensor.
8. The method of claim 1, wherein detecting that the user has
indicated the instruction is based on facial recognition data
detected by a camera.
9. The method of claim 1, wherein detecting that the user has
indicated the instruction includes identifying that the user has
performed a gesture.
10. An apparatus for updating displayed content, the apparatus
comprising: a display device that displays content; and one or more
sensors that: receive data identify a gaze direction of a user
while an eye of the user is focused on the displayed content, the
gaze direction corresponding to a first location within the
display, track movement of the eye of the user to a second location
within the display based on changes in the identified gaze
direction of the user; and detect that the user has indicated an
instruction while the gaze direction corresponds to the second
location, wherein the display device updates at least the displayed
content at the second location according to the instruction
indicated by the user.
11. The system of claim 10, wherein the sensors include a
microphone that detects that the user has indicated the instruction
by receiving vocal input from the user, wherein the display device
updates the displayed content based on the vocal input.
12. The system of claim 10, further comprising a processor that
increases processing power to render an object at the second
location with greater detail or speed.
13. The system of claim 10, further comprising a processor that
decreases processing power used to render displayed content at
another location within the display, wherein the displayed content
at the other location is rendered with less detail than the
displayed content at the second location.
14. The system of claim 10, wherein the sensors include an
accelerometer of a control device that detects that the user has
indicated the instruction by detecting acceleration data and that
further identifies the indicated instruction by identifying a
position of the control device in three dimensional (3D) space
based on the acceleration sensor data.
15. The system of claim 3, wherein the sensors include an infrared
sensor that detects that the user has indicated the instruction by
detecting infrared sensor data and that further identifies the
position of the control device in 3D space based on the received
infrared sensor data.
16. The system of claim 10, wherein the sensors detect that the
user has indicated the instruction is based on detected movement
data.
17. The system of claim 10, wherein the sensors include a camera
that detects that the user has indicated the instruction based on
facial recognition data.
18. The system of claim 10, wherein the sensors include a camera
that detects that the user has indicated the instruction by
identifying that the user has performed a gesture.
19. A non-transitory computer-readable storage medium, having
embodied thereon a program executable by a processor to perform a
method for updating displayed content, the method comprising:
identifying a gaze direction of a user while an eye of the user is
focused on displayed content in a display based on data received by
an optical sensor, the gaze direction corresponding to a first
location within the display; tracking movement of the eye of the
user to a second location within the display based on changes in
the identified gaze direction of the user; detecting that the user
has indicated an instruction while the gaze direction corresponds
to the second location; and updating at least the displayed content
at the second location according to the instruction indicated by
the user.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation and claims the priority
benefit of U.S. patent application Ser. No. 15/659,254 filed Jul.
25, 2017, now U.S. Pat. No. 10,310,583, which is a continuation and
claims the priority benefit of U.S. patent application Ser. No.
15/180,275 filed Jun. 13, 2016, now U.S. Pat. No. 9,715,266, which
is a continuation and claims the priority benefit of U.S. patent
application Ser. No. 14/014,199 filed Aug. 29, 2013, now U.S. Pat.
No. 9,367,117, the disclosures of which are incorporated herein by
reference.
BACKGROUND OF THE INVENTION
Field of the Invention
[0002] This invention relates generally to electronic systems and
more particularly to a system and method for utilizing tracking to
identify reactions to content.
Description of the Related Art
[0003] In electronic systems, particularly entertainment and gaming
systems, a user typically controls the behavior or actions of at
least one character in a game program. The users' perspective, as
determined by the camera angle, varies depending on a variety of
factors, including hardware restrictions, such as the processing
power of the system. In games with two-dimensional graphics,
typical user perspectives include a top-down view (or "helicopter"
view), where the user views the game from a third-person
perspective, and a side-scrolling view, where the user views the
characters from a third-person perspective as they move across the
screen from left to right. These perspectives require lower levels
of detail, and thus, require lower processing power from the
processing units of the system.
[0004] In games with three-dimensional graphics, typical user views
include a fixed 3D view, where the objects in the foreground are
updated in real time against a static background, and the
perspective of the user remains fixed, a first-person view (i.e.,
the user views the game from the perspective of a game character),
and third-person view, where the user views the game character from
a distance away from the game character, such as above or behind
the character. The views depend on the sophistication of the camera
system of a game. Three types of camera systems are typically used:
a fixed camera system, a tracking camera system that follows the
game character, and an interactive camera system that allows the
user to control the camera angle.
[0005] Although the three-dimensional perspectives are more
realistic for the user, they require more processing power, and,
thus, the level of detail in rendering can suffer as a result of
the drain in processing power to create the three-dimensional
view.
[0006] Therefore, there is a need for a system and method for
improving the balance between providing rendering detail and
conservation of processing power by tracking where the user focuses
his attention during game play.
SUMMARY OF THE CLAIMED INVENTION
[0007] Embodiments of the present invention provide methods and
systems for attention-based rendering on an entertainment system
are provided. A tracking device captures tracking data associated
with a user. The tracking data is utilized to determine that the
user reacted to at least one area displayed on a display device
connected to the entertainment system. A processor communicates the
determination to a graphics processing unit and instructs it to
alter the processing power used for rendering graphics in the area
of the display device. If the user is paying attention to the area,
the processing power is increased, which in turn increases the
detail and fidelity of the graphics and/or increases the speed with
which objects within the area are updated. If the user is not
paying attention to the area, processing power is diverted from the
area, resulting in decreased detail and fidelity of the graphics
and/or decreased updating speed of the objects within the area.
[0008] Various embodiments of the present invention include methods
for attention-based rendering on an entertainment system. Such
methods may include receiving tracking data from at least one user
by a tracking device, wherein the tracking data is captured in
response to a reaction of the user to at least one area displayed
on a display device. The tracking data is sent by way of the
tracking device to a processor. The processor executes instructions
stored in memory, wherein execution of the instructions by a
processor utilizes the tracking data to determine that the user
reacted to the at least one area and communicates to a graphics
processing unit to alter processing power used for rendering
graphics. A further embodiment includes the steps of receiving a
selection by the user indicating a preference for initiating a
power-saving mode, storing the selection in memory, and initiating
a power-saving mode when the tracking data indicates a lack of
attention to the display device by the user.
[0009] Further embodiments include systems for attention-based
rendering. Such systems may include a memory and a display device
connected to an entertainment system. A tracking device captures
tracking data associated with a user. A processor executes
instructions stored in memory, wherein execution of the
instructions by the processor utilizes the tracking data to
determine that the user reacted to the at least one area displayed
on the display device and communicates to a graphics processing
unit to alter processing power used for rendering graphics.
[0010] Some embodiments of the present invention further include
computer-readable storage media having embodied thereon programs
executable by processors to perform methods for attention-based
rendering.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1 is a block diagram of an exemplary electronic
entertainment system;
[0012] FIG. 2 is a flowchart of method steps for utilizing tracking
to identify reactions to content.
[0013] FIG. 3A is a screenshot of an exemplary entertainment system
environment showing a standard level of detail.
[0014] FIG. 3B is a screenshot of an exemplary entertainment system
environment showing a low level of detail in areas in which a user
is not focusing attention.
[0015] FIG. 3C is a screenshot of an exemplary entertainment system
environment showing a high level of detail in areas in which a user
is focusing attention.
DETAILED DESCRIPTION
[0016] FIG. 1 is a block diagram of an exemplary electronic
entertainment system 100. The entertainment system 100 includes a
main memory 102, a central processing unit (CPU) 104, at least one
vector unit 106, a graphics processing unit 108, an input/output
(I/O) processor 110, an I/O processor memory 112, a controller
interface 114, a memory card 116, a Universal Serial Bus (USB)
interface 118, and an IEEE 1394 interface 120, an auxiliary (AUX)
interface 122 for connecting a tracking device 124, although other
bus standards and interfaces may be utilized. The entertainment
system 100 further includes an operating system read-only memory
(OS ROM) 126, a sound processing unit 128, an optical disc control
unit 130, and a hard disc drive 132, which are connected via a bus
134 to the I/O processor 110. The entertainment system 100 further
includes at least one tracking device 124.
[0017] The tracking device 124 may be a camera, which includes
eye-tracking capabilities. The camera may be integrated into or
attached as a peripheral device to entertainment system 100. In
typical eye-tracking devices, infrared non-collimated light is
reflected from the eye and sensed by a camera or optical sensor.
The information is then analyzed to extract eye rotation from
changes in reflections. Camera-based trackers focus on one or both
eyes and records their movement as the viewer looks at some type of
stimulus. Camera-based eye trackers use the center of the pupil and
light to create corneal reflections (CRs). The vector between the
pupil center and the CR can be used to compute the point of regard
on surface or the gaze direction. A simple calibration procedure of
the viewer is usually needed before using the eye tracker.
[0018] Alternatively, more sensitive trackers use reflections from
the front of the cornea and that back of the lens of the eye as
features to track over time. Even more sensitive trackers image
features from inside the eye, including retinal blood vessels, and
follow these features as the eye rotates.
[0019] Most eye tracking devices use a sampling rate of at least 30
Hz, although 50/60 Hz is most common. Some tracking devises run as
high as 1250 Hz, which is needed to capture detail of very rapid
eye movement.
[0020] A range camera may instead be used with the present
invention to capture gestures made by the user and is capable of
facial recognition. A range camera is typically used to capture and
interpret specific gestures, which allows a hands-free control of
an entertainment system. This technology may use an infrared
projector, a camera, a depth sensor, and a microchip to track the
movement of objects and individuals in three dimension. This system
employs a variant of image-based three-dimensional
reconstruction.
[0021] The tracking device 124 may include a microphone integrated
into or attached as a peripheral device to entertainment system 100
that captures voice data. The microphone may conduct acoustic
source localization and/or ambient noise suppression.
[0022] Alternatively, tracking device 124 may be the controller of
the entertainment system. The controller may use a combination of
built-in accelerometers and infrared detection to sense its
position in 3D space when pointed at the LEDs in a sensor nearby,
attached to, or integrated into the console of the entertainment
system. This design allows users to control a game with physical
gestures as well as button-presses. The controller connects to the
console using wireless technology that allows data exchange over
short distances (e.g., 30 feet). The controller may additionally
include a "rumble" feature (i.e., a shaking of the controller
during certain points in the game) and/or an internal speaker.
[0023] The controller may additionally or alternatively be designed
to capture biometric readings using sensors in the remote to record
data including, for example, skin moisture, heart rhythm, and
muscle movement.
[0024] Preferably, the entertainment system 100 is an electronic
gaming console. Alternatively, the entertainment system 100 may be
implemented as a general-purpose computer, a set-top box, or a
hand-held gaming device. Further, similar entertainment systems may
contain more or less operating components.
[0025] The CPU 104, the vector unit 106, the graphics processing
unit 108, and the I/O processor 110 communicate via a system bus
136. Further, the CPU 104 communicates with the main memory 102 via
a dedicated bus 138, while the vector unit 106 and the graphics
processing unit 108 may communicate through a dedicated bus 140.
The CPU 104 executes programs stored in the OS ROM 126 and the main
memory 102. The main memory 102 may contain pre-stored programs and
programs transferred through the I/O Processor 110 from a CD-ROM,
DVD-ROM, or other optical disc (not shown) using the optical disc
control unit 132. The I/O processor 110 primarily controls data
exchanges between the various devices of the entertainment system
100 including the CPU 104, the vector unit 106, the graphics
processing unit 108, and the controller interface 114.
[0026] The graphics processing unit 108 executes graphics
instructions received from the CPU 104 and the vector unit 106 to
produce images for display on a display device (not shown). For
example, the vector unit 106 may transform objects from
three-dimensional coordinates to two-dimensional coordinates, and
send the two-dimensional coordinates to the graphics processing
unit 108. Furthermore, the sound processing unit 130 executes
instructions to produce sound signals that are outputted to an
audio device such as speakers (not shown).
[0027] A user of the entertainment system 100 provides instructions
via the controller interface 114 to the CPU 104. For example, the
user may instruct the CPU 104 to store certain game information on
the memory card 116 or instruct a character in a game to perform
some specified action.
[0028] Other devices may be connected to the entertainment system
100 via the USB interface 118, the IEEE 1394 interface 120, and the
AUX interface 122. Specifically, a tracking device 124, including a
camera or a sensor may be connected to the entertainment system 100
via the AUX interface 122, while a controller may be connected via
the USB interface 118.
[0029] FIG. 2 is an exemplary flowchart 200 for utilizing tracking
to identify user reactions to content. In step 202, tracking data
is received from the at least one user by the tracking device that
is captured in response to a reaction of a user to at least one
area displayed on the display device. The tracking data may be
based on any type of tracking methodology, including but not
limited to gesture-based tracking using a sensor and a range camera
or a controller containing an accelerometer and infrared detection,
eye tracking using a specialized camera or optical sensor using
infrared light, audio-based tracking using an audio sensor or a
microphone, and/or biometric tracking using a controller containing
biometric sensors. In step 204, the tracking data is sent by the
tracking device to the CPU 104 (FIG. 1).
[0030] In step 206, the CPU 104 executes a software module stored
in main memory 102 (FIG. 1) with instructions to utilize the
tracking data to determine the reaction of the user to the at least
one area displayed on the display device. The software module may
be custom-made for different game titles, or it may be native to
the gaming platform. Alternatively, the software module may have
different tracking functionalities for different types of
interfaces (e.g., audio tracking, video tracking, or gesture
tracking). The software module may also be installed into main
memory 102 by way of a digital data storage device (e.g., an
optical disc) being inserted into entertainment system 100 using
optical disc control unit 132. The reaction may be a visual
reaction, determined by, for example, movement of the eyes of the
user toward or away from the area. The visual reaction may be
captured by an integrated or peripheral camera connected to
entertainment system 100. Alternatively, the reaction may be an
emotional reaction by the user. An emotional reaction may include,
for example and limited to, a vocal reaction by the user captured
by a microphone, or a biometric reaction captured by the controller
interface 114 (FIG. 1). An emotional reaction may occur, for
example, when a user is surprised by an event occurring within the
game (e.g., the user shouts or exclaims), or when a user is
frightened or anxious because his game character is in danger
(e.g., the user sweats or his pulse increases).
[0031] In step 208, when the user reaction indicates that the user
is focusing his attention on the area of the display on the display
device, the CPU 104 communicates with the main memory 102 (FIG. 1)
and instructs the graphics processing unit 108 (FIG. 1) to increase
processing power to render greater detail and fidelity in that area
and/or to increase the speed with which objects within the area are
updated in real-time.
[0032] Alternatively, in step 210, when the user reaction indicates
that the user is not focusing his attention on the area of the
display, the CPU 104 communicates with the main memory 102 and
instructs the graphics processing unit 108 (FIG. 1) to decrease
processing power to render detail and fidelity in that area and/or
to decrease the speed with which objects within the area are
updated in real-time.
[0033] Thus, greater processing power is diverted to areas of the
display on the display device where the user is focusing most of
his attention. For example, when a special effect is displayed on
the display device, the user is likely to focus attention on the
area of the screen in which the special effect is occurring.
Meanwhile, areas of the display that the user is not focusing on
(e.g., when these areas are only in the peripheral vision of user),
less detail is needed and, therefore, less processing power is
needed for rendering graphics. This allows the entertainment system
to conserve processing power in areas that are not the focus of the
attention of the user, and improve the graphical details of areas
on which the user is currently focusing.
[0034] In another embodiment of the present invention, at step 212,
the user may optionally select a power-saving preference in a
preference module. The CPU 104 (FIG. 1) executes the preference
module and instructs it to receive the selection by the user and
store it in main memory 102 (FIG. 1) of the entertainment system
100. When selected, the power-saving preference initiates, at step
214, a power-saving mode when the tracking data indicates a lack of
attention to the display device by a user. The power-saving mode
may include, for example and not by way of limitation, initiation
of a screen saver on the display device. Alternatively, the
power-saving mode may require the entertainment system 100 to shut
down.
[0035] FIGS. 3A-3C illustrate exemplary interfaces for transferring
a ticket from one party to another on a mobile device, where both
parties have access to and accounts with the same ticketing
application.
[0036] Referring now to FIG. 3A, a screenshot of an exemplary
entertainment system environment 300 showing a standard level of
detail is shown, which may occur in a game on an entertainment
system that does not employ a tracking device. In this environment,
no additional detail is added or diminished because no processing
power has been diverted to a certain area of the screen based on
the attention of the user.
[0037] FIG. 3B is a screenshot of environment 300, showing a low
level of detail in areas in which a user is not focusing attention.
The focus area 310 is identified by the tracking device as the area
on which the user is focusing. Focus area 310 has a normal level of
detail, such as that shown in FIG. 3A. The remainder of the
environment 300 has diminished detail because processing power has
been diverted from these areas, which are likely only visible in
the peripheral vision of the user. Therefore, a lower level of
rendering is necessary.
[0038] FIG. 3C is a screenshot of environment 300 showing a high
level of detail in areas in which a user is focusing attention.
Focus area 310 has a higher level of detail because the processing
power has been diverted from the remainder of the screen because
the tracking device has recognized that the user is focusing
attention only on focus area 310. An event, such as the vehicle
crash visible in focus area 310, is one example of an event in a
gaming environment that may draw the attention of the user to a
particular area of a screen. Thus, a higher level of rendering is
necessary in an area such as focus area 310 to improve the gaming
experience for the user.
[0039] The invention has been described above with reference to
specific embodiments. It will, however, be evident that various
modifications and changes may be made thereto without departing
from the broader spirit and scope of the invention as set forth in
the appended claims. The foregoing description and drawings are,
accordingly, to be regarded in an illustrative rather than a
restrictive sense.
* * * * *