U.S. patent application number 15/322413 was filed with the patent office on 2017-05-18 for gaze detection device.
The applicant listed for this patent is FOVE, INC.. Invention is credited to Bakui Chou, Lochlainn Wilson.
Application Number | 20170140224 15/322413 |
Document ID | / |
Family ID | 54937600 |
Filed Date | 2017-05-18 |
United States Patent
Application |
20170140224 |
Kind Code |
A1 |
Wilson; Lochlainn ; et
al. |
May 18, 2017 |
GAZE DETECTION DEVICE
Abstract
Problem: The purpose is to reduce the size of a gaze detection
device. Solution: A gaze detection device that is used with a
head-mounted display and comprises an invisible light emitter
device for illuminating the user's eye with invisible light, an
invisible light mirror that reflects the invisible light and
transmits visible light, a filter that transmits the invisible
light but blocks visible light, and an imaging device that can
capture images with the invisible light, wherein the invisible
light mirror is positioned between a display and the user's eye,
and between the display and the imaging device at an angle between
the surface of the invisible light mirror and the image display
surface of the display that is greater than or equal to 0 degrees
and less than 45 degrees, reflecting the invisible light emitted by
the invisible light emitter device and reflected from the user's
eye, the filter is positioned between the display and the imaging
device, the imaging device is positioned so that at least a part of
the image display surface of the display is included in the field
of view of the imaging device, allowing images of the user's eye to
be captured using the invisible light reflected by the invisible
light mirror.
Inventors: |
Wilson; Lochlainn; (Tokyo,
JP) ; Chou; Bakui; (Tokyo, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
FOVE, INC. |
San Mateo |
CA |
US |
|
|
Family ID: |
54937600 |
Appl. No.: |
15/322413 |
Filed: |
November 14, 2014 |
PCT Filed: |
November 14, 2014 |
PCT NO: |
PCT/JP2014/080254 |
371 Date: |
December 27, 2016 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G02B 27/0093 20130101;
G06K 9/00604 20130101; G02B 2027/0138 20130101; H04N 5/247
20130101; G06F 3/013 20130101; G02B 2027/0187 20130101; H04N 5/33
20130101; G02B 27/0172 20130101 |
International
Class: |
G06K 9/00 20060101
G06K009/00; G06F 3/01 20060101 G06F003/01; G02B 27/00 20060101
G02B027/00; H04N 5/33 20060101 H04N005/33; H04N 5/247 20060101
H04N005/247; G02B 27/01 20060101 G02B027/01 |
Foreign Application Data
Date |
Code |
Application Number |
Jun 27, 2014 |
JP |
PCT/JP2014/067197 |
Claims
1. A gaze detection device for use with a head-mounted display that
comprises at least one display and is used while secured to the
head of the user, comprising: at least one invisible light emitter
device for illuminating an eye of the user with invisible light; at
least one invisible light mirrors that reflects the invisible light
and transmits visible light; at least one filter that transmits the
invisible light and does not transmit visible light; and at least
one imaging device that can capture images formed by the invisible
light, wherein at least one of the invisible light mirrors is
disposed both between at least one of the displays and an eye of
the user, and between the display and at least one of the imaging
devices, the angle between the image display surface of the display
and the surface of the invisible light mirror is defined in
reference to two orthogonal axes within the plane that includes the
image display surface of the display, wherein the first rotation
angle of the invisible light mirror around one of the two axes and
the second rotation angle of the invisible light mirror around the
other axis are both set to angles greater than or equal to 0
degrees and less than 45 degrees; the invisible light mirror
reflects invisible light emitted by at least one of the invisible
light emitter devices and reflected from the user's eye; at least
one of the filters is disposed between at least one of the displays
and at least one of the imaging devices; at least one of the
imaging devices is oriented so as to include at least a part of the
image display surface of at least one of the displays in the field
of view of the imaging device, allowing the imaging device to
capture an image of the user's eye using the invisible light
reflected by the invisible light mirror; and the imaging device is
positioned so as to make the crossing angle between the optical
axis of the imaging device reflected from the invisible light
mirror, and the ocular axis that is orthogonal to at least one of
the one or more displays and passes through the corneal apex,
greater than zero when the user has mounted the head-mounted
display.
2. The device of claim 1, wherein at least one of the invisible
light mirrors comprises two invisible light mirrors, one of which
is disposed between at least one of the displays and one of the
eyes of the user and oriented by setting each of the first rotation
angle and the second rotation angle so as to reflect the invisible
light reflected from the eye of the user towards one imaging device
of the one or more imaging devices; and the other of the two
invisible mirrors is disposed between at least one of the displays
and the other eye of the user and oriented by setting each of the
first rotation angle and the second rotation angle so as to reflect
the invisible light reflected from the other eye of the user
towards the same imaging device.
3. The device of claim 1 that further comprises a gaze analysis
device connected to at least one imaging device, wherein one or
more imaging devices capture images of the user's eye and send the
image data to the gaze analysis device that detects the gaze of the
user on the basis of the captured image data of the user's
eyes.
4. The device of claim 1, wherein the invisible light is infrared
light and the invisible light mirror is a hot mirror.
5. The device of claim 1 that further comprises at least one lens
disposed between at least one of the displays and an eye of the
user.
6. The device of claim 5, wherein the distances between at least
one of the lenses, at least one of the displays and an eye of the
user are set on the basis of at least the user's visual acuity and
the user's focal length.
7. The device of claim 5, wherein the lens is detachable from the
gaze detection device.
8. The device of claim 1, wherein the device is detachable from the
head-mounted display.
9. The device of claim 1, wherein at least one of the imaging
devices is disposed at a location that allows the user's eye to be
imaged from a lower direction.
10. A head-mounted display comprising the device of claim 1.
11. A system that comprises a head-mounted display of claim 10,
wherein the head-mounted display comprises at least one display;
and a video control system comprising a video control device that
is connected to the gaze analysis device and at least one display,
wherein the gaze analysis device sends the user's gaze detection
information to the video control device; the video control device
generates video data for at least one display based on the users
gaze information received from the gaze analysis device; and at
least one display shows a video based on the video data.
12. The system of claim 11 that further comprises a user input
device connected to the video control device, wherein the user
input device send the user inputs to the video control device and
the video control device sends to the display video data based on
the user input.
13. The device of claim 5 that further comprises an adjustment unit
for adjusting the distance between the lens and the eye of the
user.
14. The device of claim 1, wherein the device comprises a plurality
of invisible light emitter devices.
Description
TECHNICAL FIELD
[0001] The present invention relates to a gaze detection device. In
particular, the invention relates to a gaze detection device used
in conjunction with a head-mounted display (HMD).
BACKGROUND ART
[0002] Conventional gaze detection devices that estimate a user's
gaze direction from images of the user's eye comprise, as
illustrated in FIG. 8, a lens 61, an infrared emitter device 62, a
hot mirror 63 that reflects infrared light and transmits visible
light, a camera 64, and a display 65. The hot mirror 63 is
positioned at an angle of about 45 degrees relative to the optical
axis 66 that is perpendicular to the surface of the display 65,
with the angle .theta. between the surface of the display and the
hot mirror being 45 degrees. The camera 64 is positioned below the
optical axis 66 and is oriented perpendicular to and pointing
towards the optical axis 66. The infrared light emitted by the
infrared emitter device 62 reflects from the eye 60 of the user,
then reflects from the hot mirror 63, and reaches the camera 64.
The infrared light optical path connects the camera 64 and the
user's eye 60 through a right angle reflection in the hot mirror
63. An example of the aforementioned structure is disclosed in FIG.
8 of patent document 1. Further, patent document 2 discloses a
medical eye camera comprising a hot mirror that is tilted at an
angle of 45 degrees. In contrast, as illustrated in FIG. 9 herein,
patent document 1 discloses a configuration wherein the space
wasted by the hot mirror is eliminated by making the angle y
between the optical axis 66 and the dichroic mirror (hot mirror) 63
larger than 45 degrees and the angle .theta. between the display
and the optical axis smaller than 45 degrees.
[0003] Technology for another known method for gaze detection is
shown in FIG. 10, wherein a camera 64 is pointed towards an eye 60
of the user and an image of the user's eye 60 can be captured
directly where the captured image is used for gaze detection.
[0004] Patent document 3 discloses a gaze detection device that
uses a half mirror. The half mirror in patent document 3 changes
the optical path by reflecting light incident from a specific
direction and transmitting light incident from directions other
than the specific direction (Patent Document 3 [0015]). In this
gaze detection device, a lens is positioned in front of the user's
eye, a liquid crystal display (LCD) is positioned orthogonally to
the user's gaze direction, and a half mirror is oriented at a 45
degree angle with respect to the user's gaze direction. Light
emitted from the LCD is reflected by the half mirror, reflected
again by the lens, and reaches the eye of the user after being
transmitted through the half mirror. Light from the LCD emitted
towards the gaze detection device that faces the LCD is blocked by
the half mirror and therefore cannot enter the gaze detection
device.
[0005] Techniques for changing the video shown by the video display
in a HMD in response to user's head movement are disclosed in
patent document 4. Other technologies related to HMDs or gaze
detection are disclosed in patent documents 5 to 7.
PATENT REFERENCES
[0006] Patent document 1: JP Application No. PH08-9205 (U.S. Pat.
No. 6,018,630)
[0007] Patent document 2: JP Application No. PH05-076497
[0008] Patent document 3: JP Application No. 2001-108933
[0009] Patent document 4: JP Application No. PH08-179239
[0010] Patent document 5: JP Application No. 2014-21272
[0011] Patent document 6: JP Application No. 2006-163383 (USP
2006/0103591)
[0012] Patent document 7: JP Application No. P11-155152 (U.S. Pat.
No. 6,611,283)
SUMMARY OF THE INVENTION
Problems Solved by the Invention
[0013] In existing technology described in FIG. 8, a hot mirror is
tilted at an angle of 45 degrees relative to the optical axis 66
that is perpendicular to the display 65 surface. Therefore the
large space required by the hot mirror is the main reason why the
size of a head-mounted display cannot be
[0014] An existing technology to solve this problem, as illustrated
in FIG. 9, is to positon the hot mirror at a larger angle than 45
degrees with respect to the optical axis 66. However, because the
angle is restricted, it is impossible to achieve the desired
reduction in size. In this structure, the optical axis of the
camera 64 that captures images of the user's eye 60 is
substantially horizontal with respect to the surface of the
display. Light emitted from the display 65 adds a noise component
to the image of the user's eye used for gaze detection, and it is
thus necessary to orient the camera 64 in a direction where light
emitted by the display 65 does not enter the camera. For this
reason, even if the angle .theta. between the display 65 and the
hot mirror 63 is made smaller than 45 degrees, the angle .theta.
still has to be considerably large. Thus, the distance between the
lens and the display still cannot be sufficiently reduced and
miniaturization is impossible.
[0015] If a smaller hot mirror is used to obtain a more compact
device, edges of the hot mirror between the user's eye and the
display would create a visual disruption due to unintended
reflection or refraction of light, disturbing the video
presentation.
[0016] In a conventional configuration described in FIG. 10, the
camera 64 can directly capture images of the user's eye. This means
that the camera 64 is in the user's field of view, disrupting the
user's immersion in the virtual realty world created by the
head-mounted display.
[0017] In the configuration using a half mirror as described in
patent document 3, the half mirror has to be position at an angle
of 45 degrees to the gaze direction of the user, wasting space and
making miniaturization of a head-mounted display impossible.
[0018] Therefore, the purpose of the present invention is to
provide a gaze detection device that allows a head-mounted display
(HMD) to be miniaturized, or a HMD or another system employing the
same.
Means of Solving the Problem
[0019] The present invention has been made in view of the above
considerations, and has the following features. The gaze detection
device of the present invention is used in conjunction with a
head-mounted display that is used while secured to the head of a
user, wherein the head-mounted display comprises at least one
display, at least one invisible light emitter illuminating an eye
of the user with invisible light, at least one invisible light
mirror that reflects the invisible light and transmits visible
light, at least one filter that transmits the invisible light and
does not transmit visible light, and at least one imaging device
that can capture images using the invisible light, wherein at least
one of the invisible light mirrors is positioned between at least
one of the displays and an eye of the user and between at least one
of the imaging devices and at least one of the displays, wherein at
least one of the invisible light mirrors is positioned at an angle
relative to the surface of at least one of the displays, wherein of
the two perpendicular axes defined in the surface plane of at least
one of the displays, the first rotation angle of at least one of
the invisible light mirrors around one of the two axes, and the
second rotation angle of the invisible light mirror around the
other of the two axes, are both larger than or equal to 0 and less
than 45 degrees, such that when invisible light emitted by at least
one of the invisible light emitter devices illuminating an eye of
the user is reflected from an eye of the user, at least one of the
filters is positioned between at least one of the displays and at
least one of the imaging devices, and one or more imaging devices
are oriented so that at least a part of the image display surface
of at least one of the displays is in the imaging range of the
imaging device, at least one of the imaging devices can capture an
image of an eye of the user by using invisible light reflected in
at least one of the invisible light mirrors.
[0020] In one aspect of the present invention, one or more of the
invisible light mirrors may be implemented as a combination of two
invisible light mirrors, wherein one of the two invisible light
mirrors is positioned between at least one of the displays and an
eye of the user, and each of the first rotation angle and the
second rotation angle of the invisible light mirror are set to
reflection angles that direct invisible light reflected from an eye
of the user towards one of the one or more imaging devices, while
the other of the two invisible light mirrors is positioned between
at least one display and another eye of the user, and each of the
first rotation angle and the second rotation angle of the invisible
light mirror are set to reflection angles that direct invisible
light reflected from the user's other eye towards the one imaging
device.
[0021] Also, the gaze analysis device may be connected to at least
one imaging device, wherein one or more imaging devices transfer
the information related to the captured images of an eye of the
user to the gaze analysis device, and the gaze analysis device may
detect the user's gaze based on the information related to the
captured images of the user's eyes.
[0022] Furthermore, if the non-visible light is infrared light, a
hot mirror may be used as the invisible light mirror.
[0023] On or more lenses may be disposed between an eye of the user
and one or more displays.
[0024] The distances between at least one lens, at least one
display, and the user's eye may be determined based on at least the
user's visual acuity and the focal length of the user's eyes.
[0025] The lens may be detachable from the device.
[0026] The gaze detection device of the present invention may be
detachable from the head-mounted display.
[0027] At least one imaging device may be disposed at a position
allowing imaging of the user's eye from a lower direction.
[0028] The head-mounted display of the present invention comprises
the aforementioned gaze detection device.
[0029] Furthermore, the system of the present invention includes a
head-mounted display described above, the head-mounted display
comprises at least one display, the video control system comprises
a gaze analysis device and a video control device connected to one
or more displays, wherein the gaze analysis device transmits the
user's gaze detection information to the video control device, and
based on the user's gaze detection information received from the
gaze analysis device, the video control device generates a video
signal that is sent to one or more displays, and one or more
displays display a video based on the video signal.
[0030] Furthermore, the image control device may be connected to a
user input device, wherein the user input device transmits user
inputs to the video control device and, based on the user input,
the video control device may generate appropriate image information
to be transmitted to the display.
Advantageous Effects of the Invention
[0031] In the aforementioned configuration of the present
invention, the imaging device is positioned and oriented so as to
allow the field of view of the imaging device to include at least a
part of the display surface. The degree of freedom in selecting the
angle between the invisible light mirror and the display is
increased, making it possible to further miniaturize the
head-mounted display. Further, since there is no need to use a
smaller-sized invisible light mirror to miniaturize the device,
there is no effect on the image presentation due to the edges of
the invisible light mirror entering the field of view. Therefore,
it is possible to miniaturize the device and maintain high video
presentation quality.
[0032] In the present invention, when the user's eye is illuminated
with infrared light from an infrared emitter device, the infrared
light reflects from the eye of the user, is then further reflected
by the invisible light mirror, and can be delivered to an imaging
device that is positioned outside of the user's field of view. That
is, by adopting such a configuration, since it is possible to
position the imaging device outside of the user's field of view,
the imaging device does not impede the visibility of the video
image for the user and the immersion of the user in the virtual
reality world created by the head-mounted display can be
improved.
[0033] When the user slightly lowers an eyelid, the eyelid conceals
the upper half of the eyeball. Since an imaging device installed
above the eye acquires images of a user's eye from the upper
direction an eyelid or eyelashes may obstruct imaging by a camera.
In case the imaging device for capturing images of an eye of the
user is positioned so as to capture images of an eye of the user
from a lower direction, an eyelid or eyelashes have no effect and
images of an eye of the user can be reliably captured.
[0034] By enclosing the gaze detection device of the present
invention in a casing that is detachable from a head-mounted
display, the gaze detection function can be added to a head-mounted
display that does not have a built-in gaze detection device.
Further, by enclosing in a casing a gaze detection device with an
optical configuration tailored to a specific user, there is no need
to purchase whole head-mounted display units for each user.
Further, by setting the distances between at least one lens, at
least one display and an eye of the user on the basis of the visual
acuity and focal distance of the user's eye, head-mounted displays
can be tailored to accommodate each user.
BRIEF DESCRIPTION OF THE DRAWINGS
[0035] FIG. 1 is a schematic side view of the HMD according to the
first embodiment of the present invention.
[0036] FIG. 2 is an explanatory diagram of the X-axis and Y-axis of
the HMD.
[0037] FIG. 3 is a schematic plan view of a HMD according to the
first embodiment of the present invention.
[0038] FIG. 4 is a schematic plan view of a HMD according to the
second embodiment of the present invention.
[0039] FIG. 5 is a schematic plan view of a HMD according to the
third embodiment of the present invention.
[0040] FIG. 6 is a schematic side view of the HMD according to the
fourth embodiment of the present invention.
[0041] FIG. 7 is an example of a display screen according to the
fourth embodiment of the present invention.
[0042] FIG. 8 is a schematic side view of a conventional gaze
detection device.
[0043] FIG. 9 is a schematic side view of a conventional gaze
detection device.
[0044] FIG. 10 is a schematic side view of a conventional gaze
detection device.
DETAILED DESCRIPTION OF THE EMBODIMENTS
The First Embodiment
[0045] A schematic side view of the head-mounted display according
to the first embodiment is shown in FIG. 1. The head-mounted
display 1 that can be secured to the head of a user contains a
display unit 14 and the display unit 14 has an image display
surface 16. Further, the gaze detection device installed in the
head-mounted display comprises invisible light emitter devices 20
that illuminate the eye 10 of the user with invisible light, an
invisible light mirror 13 that reflects the invisible light and
transmits visible light, an imaging device 12 that can capture
images with the invisible light, and a filter 15 that transmits the
invisible light and does not transmit visible light. The gaze
detection device further comprises a lens 11 that is positioned
between the display unit 14 and the eye 10 of the user.
[0046] Although in the present embodiment infrared emitter devices
20-1.about.20-4 are used as the invisible light emitter device 20,
as long as the emitted light is invisible, the use of a light
source with a different wavelength is possible as well. Further, as
described in FIG. 1, in the present embodiment, the invisible light
emitter device 20 is implemented as four infrared emitter devices
20-1.about.20-4 that are positioned at the top, bottom, left, and
right edges of a lens. However, as long as the emitted infrared
light can reach the imaging device 12 after being reflected from
the eye 10 of the user, other arrangements of the light emitter
positions are also possible, while a smaller number of light
emitters, for example one, may be used. The number of installed
infrared light sources may be larger as well. The imaging device 12
in FIG. 1 is positioned in the lower part of the head-mounted
display 1, but may be positioned in the upper part, sides, or
elsewhere. In the present embodiment, the invisible light mirror 13
is a hot mirror that reflects infrared light and transmits visible
light. In case the invisible light emitter device 20 is a light
emitter with a wavelength outside of the infrared range, the
invisible light mirror should reflects light at the operating
wavelength of the invisible light emitter device. A hot mirror is
disposed at a position where it is between an eye of the user and a
display, and between the display and the imaging device. In this
embodiment, a hot mirror 13 is positioned between the display 14
and the lens 11. As shown in FIG. 1, the angle .theta. between the
hot mirror 13 and the image display surface 16 of the display unit
14 (the hot mirror side of the display 14) is larger than 0 degrees
and less than 45 degrees, preferably in the range from 0 degrees to
less than 20 degrees. Smaller angle .theta. leads to greater
reduction of the head-mounted display thickness and weight. For
example, the edges of the hot mirror 13 and the display 14 may be
in contact.
[0047] As shown in FIG. 2, the angle between the hot mirror surface
and the image display surface 16 of the display 14 can be defined
by the rotation angle of the hot mirror 13 around the X-axis and
the rotation angle of the hot mirror 13 around the Y-axis. Here,
the axes are defined in the plane of the image display surface 16
of the display 14 such that when the head-mounted display is
mounted on the head, the X-axis points along the left-right
direction and the Y-axis points along the up-down direction. In the
present embodiment, the Z-axis points in the direction of the user.
The angle .theta. in FIG. 1 is the rotation angle of the hot mirror
13 around the X-axis. In the present embodiment, as shown in FIG.
3, the hot mirror 13 rotation angle .alpha. around the Y-axis is 0
degrees. The hot mirror 13 thus reflects the infrared light emitted
by the infrared emitter device 20 and reflected from the eye 10 of
the user towards the imaging device 12. When both angles .alpha.
and .theta. are small, the wasted space can be reduced and the
head-mounted display can be made smaller.
[0048] Filter 15 is positioned between the display 14 and the
imaging device 12. The filter 15 is a visible light cut-off filter
that transmits infrared light and blocks visible light. The filter
15 may be any filter that can block visible light and transmit the
invisible light emitted by the invisible light emitter device 20
and reflected from the eye of the user. The imaging device 12 is
oriented in a direction that places at least a part of the image
display surface 16 in the field of view of the imaging device and
the imaging device captures images of the user's eye using infrared
light reflected by the hot mirror 13. That is, the imaging device
12 is positioned in a direction that allows light emitted from the
image display surface 16 of the display 14 to reach the imaging
device directly, while infrared light reaches the imaging device
after being reflected in the hot mirror 13.
[0049] Next, the operation according to the present embodiment will
be explained. First, the user secures the head-mounted display 1 to
the head. The image display surface 16 of the display 14 is
directed towards the eyes of the user and displays a video. The
video may represent a computer-generated animation, a video game, a
movie, etc. The image display surface 16 of the display 14 displays
a video by emitting visible light. This emitted visible light is
transmitted through the hot mirror 13, and reaches the eye 10 of
the user through the lens 11. The lens 11 magnifies the image on
the display 14 for viewing by the user. The lens can also
contribute to vision correction for the user. The lens 11 is not
essential for the operation of the gaze detection device.
[0050] The infrared emitter devices 20-1.about.20-4 illuminate the
eye 10 of the user with infrared light at a fixed illumination
angle. The illuminating infrared light is reflected from the eye 10
of the user, passes through the lens 11, and reaches the hot mirror
13. The hot mirror 13 reflects the infrared light towards the
imaging device 12. In the present embodiment, the hot mirror 13 is
in the field of view 21 of the user and preferably covers the
entire surface of the display 14. It is thereby possible to view
the display without the edges of the hot mirror creating boundaries
within the field of view 21. However, the hot mirror 13 does not
necessarily have to cover the entire surface of the display 14. The
imaging device 12 captures an image of the user's eye, wherein the
image is formed by the infrared light reflected from the eye 10 of
the user. It is possible to determine the user's gaze direction
based on the captured image of the eye of the user. By reflecting
the infrared light, the hot mirror 13 makes it possible for an
imaging device located outside of the user's field of view 21 to
acquire images of the user's eye. The user can view the video shown
on the display without being obstructed by the imaging device.
[0051] In the present embodiment, using a plurality of infrared
emitter devices 20-1.about.20-4 allows the intensity of the
infrared light reflected from the eye 10 of the user and reaching
the imaging device 12 to be increased. It is possible to obtain
sufficient received infrared light intensity with a single infrared
emitter device as well. Increasing the infrared light emission
intensity of the single infrared emitter device may be used to
increase the intensity of the reflected infrared light reaching the
imaging device 12. However, in that case strong infrared light
irradiates a particular part of the user's eye, increasing the risk
of injury to the eye of the user. By using a plurality of infrared
emitter devices, the relatively weak infrared irradiation does not
damage the eyes of the user, while sufficient infrared light
intensity reaches the imaging device 12 for gaze detection.
[0052] Since the imaging device 12 is positioned such that at least
a part of the surface of the display 14 is included in the field of
view of the imaging device, visible light emitted from the image
display surface 16 of the display 14 passes through the hot mirror
13 and illuminates the imaging device 12. However, the light
emitted from the display 14 is visible light, which is blocked by
the visible light cut-off filter 15, positioned between the display
14 and the imaging device 12, preventing the emitted light from
reaching the imaging device 12. In contrast, the infrared light
reflected by the hot mirror 13 passes through the visible light
cut-off filter 15 and reaches the imaging device.
[0053] In existing technology, because visible light emitted from a
display can enter the imaging device, noise is added to the user's
gaze detection and it is therefore necessary to direct the imaging
device in a direction that prevents visible light emitted by the
display from entering the imaging device, preferably in a vertical
direction. For this reason, the tilt angle of the hot mirror with
respect to the display has to be large.
[0054] In contrast, by employing the configuration of the present
invention, even though the imaging device 12 may be oriented so
that at least a part of the image display surface of the display 14
is included in the field of view of the imaging device, the visible
light emitted by the display 14 that reaches the imaging device 12
does not give rise to noise and the received infrared light can be
used for capturing images of the user's eye. Therefore, there is
greater freedom in the positioning direction of the imaging device,
and also greater freedom in setting the angle .theta. between the
hot mirror 13 and the display 14. For example, the angle .theta.
may be set to 0 degrees. Accordingly, the head-mounted display can
be made thinner and smaller.
[0055] Further, as shown in FIG. 1, the imaging device 12 is
preferably arranged at a position that allows capturing images of
an eye of the user from below. When imaging the eye of the user
from below, without being obstructed by the eyelid or the
eyelashes, reliable capturing of images of the user's eye is
possible.
The Second Embodiment
[0056] FIG. 4 shows a schematic top-view diagram of the
configuration of the head-mounted display according to the second
embodiment. In the following description, explanations are omitted
for components that are identical to those described in the first
embodiment.
[0057] The configuration of the present embodiment is similar to
the configuration of the first embodiment but applied for each eye
of the user. In other words, for the right eye 10a of the user, the
head-mounted display 1 comprises a display 14a, a hot mirror 13a
that is positioned between the display 14a and the right eye 10a,
and between the display 14a and the imaging device 12a, a lens 11a
that is positioned between the display 14a and the user's right eye
10a, infrared emitter devices 20a-1.about.20a-4 that illuminate the
right eye 10a of the user with infrared light, an imaging device
12a that is oriented so that at least a part of the image display
surface of the display 14a is included in the field of view of the
imaging device, and a visible light cut-off filter 15a that is
positioned between the imaging device 12a and the display 14a. In
this embodiment, as in the first embodiment, illustrated in FIG. 1,
the hot mirror 13a is rotated around an X-axis defined within the
image display surface 16a plane of the display 14a, wherein the
rotation angle .theta. (referred to as .theta.1 in this embodiment)
between the surface of the hot mirror 13a and the display 14a is
greater than or equal to 0 degrees and less than 45 degrees, but
preferably greater than or equal to 0 degrees and less than 20
degrees. On the other hand, as shown in FIG. 4, for a rotation of
the hot mirror 13a around the Y-axis, the angle .alpha. between the
image display surface 16a of the display 14a and the surface of the
hot mirror 13a is 0 degrees.
[0058] Similar to the above, for the user's left eye 10b, the gaze
detection device further comprises a display 14b, a hot mirror 13b,
a lens 11b, infrared emitter devices 20b-1 20b-4, a visible light
cut-off filter 15b, and an imaging device 12b. The hot mirror 13b
is rotated around the X-axis defined within the image display
surface 16b plane of the display 14b, wherein the rotation angle
.theta. (referred to as .theta.2 in this embodiment) between the
hot mirror 13b and the display 14b is greater than or equal to 0
degrees and less than 45 degrees, but preferably greater than or
equal to 0 degrees and less than 20 degrees, while the rotation
angle f between the surface of the hot mirror 13b and the image
display surface 16b of the display 14b for a rotation of the hot
mirror 13b around the Y-axis is 0 degrees. The angles .theta.1 and
.theta.2 do not need to be equal to each other.
[0059] To apply the method described in the first embodiment to
both, images formed by infrared light of the user's right eye 10a
and the left eye 10b are captured by the infrared imaging devices
12a and 12b, respectively.
[0060] In the present embodiment, a display 14c and a display 14d
are located on the sides of the front part of the head-mounted
display. Since the user can see the video images on the displays
14c and 14d in addition to the displays 14a and 14b that are
directly facing the user, the user may gain close to a real-world
experience. The visible light cut-off filters 15a and 15b are
disposed between the side-mounted displays 14c, 14d and the imaging
devices 12a, 12b, blocking the visible light emitted from these
displays, making it possible to properly image the motion of the
user's eye. For example, a visible light cut-off filter that covers
the lens of the imaging device may be used. It is also possible to
use a plurality of visible light cut-off filters to block light
from a plurality of displays.
[0061] Similarly, displays may be located along the top surface or
the bottom surface of the front part of the head-mounted display
1.
The Third Embodiment
[0062] FIG. 5 shows a schematic top view of the configuration of
the head-mounted display according to the present embodiment. The
explanation will focus on differences from the second embodiment.
For the sake of simplifying the explanation of the display, only a
configuration with two front displays 14a and 14b will be
considered. Further displays may be added on the sides and at the
bottom. The displays 14a and 14b may be implemented as a single
display unit.
[0063] In the present embodiment, a single common imaging device 12
is used for the left eye and the right eye. This single imaging
device is positioned in a direction that allows at least a part of
the image display surface of at least one of the displays 14a and
14b to be included in the field of view of the imaging device.
Further, two hot mirrors 13a and 13b are used, each of which is
positioned between one of the eyes of the user and one of the two
displays 14a and 14b. As shown in FIG. 5, the angle .alpha. between
the image display surface 16a of the display 14a and the surface of
the hot mirror 13a is the rotation angle of the hot mirror 13a
around the Y-axis of the display 14a.
[0064] A schematic side view of this embodiment has the same
configuration as that shown in FIG. 1. For rotations of the hot
mirror 13a around the X-axis defined in the plane of the image
display surface of the display 14a, the rotation angle between the
image display surface 16a of the display 14a and the surface of the
hot mirror 13a is .theta. (referred to as .theta.3). The angles
.alpha. and .theta.3 are set so as to reflect the infrared light
reflected from the right eye of the user towards the common imaging
device 12.
[0065] Similarly, the other hot mirror 13b is positioned between
the display 14b and the user's left eye 10b. The angle between the
hot mirror surface and the image display surface 16b of the display
14b, wherein the rotation angle of the hot mirror 13b around the
Y-axis defined in the plane of the image display surface of the
display 14b is .beta. and the rotation angle of the hot mirror 13b
around the X-axis is .theta.(referred to as .theta.4), is set so as
to reflect infrared light reflected from the left eye of the user
towards the imaging device 12. The angles .alpha. and .beta. of
this embodiment are set to predetermined values that are greater
than 0. The angles .alpha., .beta., .theta.3, and .theta.4 are not
necessarily equal to each other.
[0066] The imaging device 12 receives infrared light reflected from
each of the hot mirrors 13a and 13b, and captures images of both
the right and the left eyes 10a and 10b of the user.
[0067] Since the imaging device is directed in a direction that
allows at least a portion of the image display surface of a display
to be included in the field of view of the imaging device, and the
hot mirror is rotated not only around the horizontal axis of the
head-mounted display, but also around the vertical axis to incline
the hot mirror relative to the display, a single imaging device can
be used to capture images of the gaze of both eyes. Therefore, the
head-mounted display can be made more compact and manufactured at a
lower cost.
The Fourth Embodiment
[0068] FIG. 6 shows a structure diagram of the video control system
of the present embodiment. The figure shows a schematic side view
of the head-mounted display 1a.
[0069] In this embodiment, the head mounted-display 1a further
comprises a video control device 24. The head-mounted display 1a is
a variation of the head-mounted display 1 described in the first
embodiment. In addition to the configuration of the gaze detection
device of the first embodiment, the gaze detection device of the
head-mounted display 1a further comprises a gaze analysis device 22
that is connected to the imaging device 12. The gaze analysis
device 22 is connected to a video control device 24, and the video
control device 24 is connected to the display 14. Further, a user
input device 25 for receiving input from the user is connected to
the video control device 24. These connections may be wired or
wireless. The video control device 24 of the present embodiment is
a video game machine for controlling a video game, the user input
device is a controller 25 connected to the video game machine.
[0070] The gaze analysis device of the present embodiment is placed
in a casing 23 that is detachable from the head-mounted display 1a.
Since the gaze detection device is detachable from the head-mounted
display, it is possible to add the gaze detection function to an
existing head-mounted display that does not include a gaze
detection device.
[0071] The selection of the lens 11 and the distances between the
lens 11, the display 14, and the user's eye 10 are preferably
determined based on the user's visual acuity and the user's focal
length. Since the optical system of the gaze detection device is
designed based on the user's eye and the position of the lens, an
entire head-mounted display unit would need to be purchased for
each user to provide a customized configuration suitable for the
user, placing a large burden on the user.
[0072] With the detachable gaze detection device of the present
embodiment, the lens 11, and the distances between the lens 11, the
display 14, and the user's eye 10 can be selected based on the
user's eyesight and the user's focal length, which means that the
user would only need to purchase the gaze detection device and
mount it on a head-mounted display. Thus, it is possible to provide
head-mounted displays with gaze detection devices suitable for each
user at a lower cost.
[0073] Further, by adjusting the respective distances between the
lens 11, the display 14, and user's eye 10 appropriately, it is
possible to set an optimal optical configuration for each user in
the gaze detection device. The adjustment can be done, for example,
by fixing the distance between the lens and the display, and
changing the distance between the lens and the user's eye. Further,
if the distance between the lens and the user's eye is fixed, the
adjustment can be done by moving the display. It is also possible
to fix the distance between the display and the user's eye, and
perform the adjustment by moving the lens in between. Specifically,
as shown in FIG. 6, for example, the lens 11 may be attached to a
rail 26 that is disposed between the eye 10 of the user and the
display 14, wherein the lens can slide along the rail 26, allowing
the lens 11 position to be adjusted to suit the user. The same
configuration can be used for the display 14. Alternatively, the
lens and the display may be configured to slide simultaneously.
[0074] For a head-mounted display, depending on the build of the
cheekbones of the user and the shape of the user's nose, the
distance between the lens 11 and the eye of the user may be
significantly different for each user. By adopting the
configuration described in this embodiment, it is possible to build
a head-mounted display that fits every user. Further, it is
possible to use a detachable lens to configure the gaze detection
device for each user by replacing the lens.
[0075] Although in the present embodiment, the video control device
24 is described as being a device that is external to the
head-mounted display, it is also possible to provide a device that
is integrated in a head-mounted display. Further, although the gaze
analysis device 22 has is described as being a part of the gaze
detection device, it is also possible to implement it as an
external device for a head-mounted display or a gaze detection
device.
[0076] The imaging device 12 transmits the image data of the
captured images of the user's eye to the gaze analysis device 22,
and based on such image information, the gaze analysis device 22
analyzes the user's gaze, eye movement frequency, nystagmus, etc.
In the present embodiment, the gaze analysis device 22 determines
the gaze direction and speed, and transmits the gaze information to
the video control device 24. Based on the user's gaze information
received from the gaze analysis device 22, the video control device
24 generates the video information to be displayed on the display
14. In the present embodiment, the user plays a shooting game using
the head-mounted display 1a. A video game machine 24, functioning
as a video control device, displays the video of the shooting game
on the image display surface 16 of the display 14. Based on the
user's gaze information from the analysis by the gaze analysis
device 21, a cursor is displayed on the image display surface of
the display 14, marking the firing point in the vicinity of the
user's gaze location, while the cursor moves in accordance with the
gaze movement. For example, as shown in FIG. 7, a circular cursor
32 is shown on the display in the video image 30 around the
position 31 of the user's gaze. When the user's gaze location moves
to the right, the cursor 32 is also moved to the right. The
position and the moving speed of the cursor are calculated on the
basis of the position and movement speed of the gaze location.
After the user moves the cursor to the desired position, the firing
command is input from the controller 25, and according to this
command, the video game machine 24 generate a video that is shown
on the display 14. Instead of the user input received through the
controller 25, if the user keeps the gaze fixed at one location for
a predetermined period of time, the game machine 24 may decide that
a user input action has occurred.
[0077] If instead of a video game machine the user is using a
personal computer, the gaze location can be used to control the
mouse pointer position. In this case the user input device 25 is a
mouse. Also, the image control device 24 used for displaying a
movie etc., in accordance with the movement of the gaze position,
may performs a control function such as scrolling the image. In
this case a user input device 25 is not necessarily required.
[0078] The above embodiments have been presented as examples for
the purpose of explaining the present invention. The preset
invention is not limited to these embodiments. The present
invention, without departing from the gist thereof can be
implemented in various forms.
REFERENCE LIST
[0079] 1 HMD [0080] 10 User's eye [0081] 11 Lens [0082] 12 Imaging
device [0083] 13 Invisible light mirror [0084] 14 Display [0085] 15
Filter [0086] 16 Image display surface [0087] 20 Invisible light
emitter device [0088] 21 Field of view [0089] 22 Gaze analysis unit
[0090] 23 Casing [0091] 24 Video control device [0092] 25 User
input unit [0093] 26 Rail [0094] 30 Display surface plane [0095] 31
Gaze location [0096] 32 Cursor [0097] 60 User's eye [0098] 61 Lens
[0099] 62 Infrared emitter device [0100] 63 Hot mirror [0101] 64
Camera [0102] 65 Display [0103] 66 Optical axis
* * * * *