U.S. patent application number 15/563444 was filed with the patent office on 2018-03-08 for head mounted display.
The applicant listed for this patent is FOVE, INC.. Invention is credited to Bakui Chou, Keiichi Seko, Lochlainn Wilson.
Application Number | 20180067306 15/563444 |
Document ID | / |
Family ID | 57004015 |
Filed Date | 2018-03-08 |
United States Patent
Application |
20180067306 |
Kind Code |
A1 |
Wilson; Lochlainn ; et
al. |
March 8, 2018 |
HEAD MOUNTED DISPLAY
Abstract
A head mounted display is used in a state of being mounted on a
user's head and includes a convex lens disposed at a position
facing the user's cornea when the head mounted display is mounted.
An infrared light source emits infrared light toward the convex
lens. A camera captures an image including the user's cornea in a
subject. A housing houses the convex lens, the infrared light
source, and the camera. The convex lens is provided with a
plurality of reflection regions that reflects infrared light in an
inside of the convex lens. The infrared light source causes a
pattern of infrared light to appear on the user's cornea by
emitting infrared light to each of the plurality of reflection
regions provided in the convex lens.
Inventors: |
Wilson; Lochlainn; (Tokyo,
JP) ; Chou; Bakui; (Tokyo, JP) ; Seko;
Keiichi; (Tokyo, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
FOVE, INC. |
San Mateo |
CA |
US |
|
|
Family ID: |
57004015 |
Appl. No.: |
15/563444 |
Filed: |
February 29, 2016 |
PCT Filed: |
February 29, 2016 |
PCT NO: |
PCT/JP2016/056078 |
371 Date: |
September 29, 2017 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
H04N 13/332 20180501;
G06K 9/00604 20130101; G02B 27/34 20130101; G06K 9/2036 20130101;
G02B 27/0093 20130101; G02B 27/0955 20130101; G02B 19/0028
20130101; G06K 9/0061 20130101; G06T 7/215 20170101; H04N 5/2254
20130101; G02B 2027/0138 20130101; G02B 2027/014 20130101; G06F
3/013 20130101; H04N 5/33 20130101; G06K 2009/4666 20130101; G06T
2207/30196 20130101; H04N 13/344 20180501; G06K 9/4661 20130101;
G02B 27/0172 20130101; G06F 3/0346 20130101; H04N 17/002 20130101;
G02B 17/086 20130101; G02B 19/009 20130101; H04N 13/383
20180501 |
International
Class: |
G02B 27/00 20060101
G02B027/00; G02B 27/01 20060101 G02B027/01; G02B 27/09 20060101
G02B027/09; H04N 5/33 20060101 H04N005/33; H04N 13/04 20060101
H04N013/04 |
Foreign Application Data
Date |
Code |
Application Number |
Apr 1, 2015 |
JP |
PCT/JP2015/060398 |
Claims
1. A line-of-sight detection system comprising a head-mounted
display and a line-of-sight detection device, the head-mounted
display includes an image display element that includes a plurality
of pixels, each pixel including sub-pixels that emit red, green,
blue, invisible light, and displays an image to be viewed by a
user; an imaging unit that images an eye of the user wearing the
head-mounted display on the basis of the invisible light emitted
from the sub-pixel that emits the invisible light; and a
transmission unit that transmits a captured image captured by the
imaging unit, and the line-of-sight detection device includes a
reception unit that receives the captured image; and a
line-of-sight detection unit that detects a line of sight of the
eye of the user based on the captured image.
2. The line-of-sight detection system according to claim 1, wherein
the head-mounted display further include a control unit that
selects the pixel that emits invisible light among the plurality of
pixels constituting the image display element and causes the
selected pixel to emit light.
3. The line-of-sight detection system according to claim 2, wherein
the control unit changes the pixel that emits the invisible light
when a predetermined time elapses.
4. The line-of-sight detection system according to claim 2, wherein
the control unit switches and controls a light emission timing of a
sub-pixel that emits the invisible light and the sub-pixel other
than the sub-pixel that emits the invisible light.
5. A head-mounted display mounted on a head of a user and used, the
head-mounted display comprising: a convex lens disposed in a
position facing a cornea of the user when the head-mounted display
is mounted; an image display element that includes a plurality of
pixels, each pixel including sub-pixels that emit red, green, blue,
invisible light, and displays an image to be viewed by a user; a
camera that images a video including the cornea of the user as a
subject; and a housing that houses the convex lens, the image
display element, and the camera.
6. The head-mounted display according to claim 5, further
comprising a control unit that selects the pixel that emits
invisible light among the plurality of pixels constituting the
image display element and causes the selected pixel to emit
light.
7. The head-mounted display according to claim 6, wherein the
control unit changes the pixel that emits the invisible light when
a predetermined time elapses.
8. The head-mounted display according to claim 6, wherein the
control unit switches and controls a light emission timing of a
sub-pixel that emits the invisible light and the sub-pixel other
than the sub-pixel that emits the invisible light.
9. A line-of-sight detection method using a line-of-sight detection
system comprising a head-mounted display including an image display
element that includes a plurality of pixels, each pixel including
sub-pixels that emit red, green, blue, invisible light, and
displays an image to be viewed by a user, and a line-of-sight
detection device, the line-of-sight detection method comprising: an
irradiation step of emitting, by the head-mounted display,
invisible light from sub-pixels of the plurality of pixels and
irradiating a cornea of the user with the invisible light; an
imaging step of imaging, by the head-mounted display, a subject
radiated with invisible light emitted from the sub-pixel that emits
the invisible light and including a cornea of the user viewing the
image displayed on the image display element to generate a captured
image; a transmission step of transmitting, by the head-mounted
display, the captured image to the line-of-sight detection device;
a reception step of receiving the captured image by the
line-of-sight detection device; and a detection step of detecting,
by the line-of-sight detection device, a direction of a line of
sight of the user on the basis of the captured image.
10. A line-of-sight detection system comprising a head-mounted
display and a line-of-sight detection device, wherein the
head-mounted display includes an image display element that
displays an image to be viewed by a user; an imaging unit that
images an eye of the user in which the image displayed on the image
display element is reflected; a transmission unit that transmits a
captured image captured by the imaging unit, and the line-of-sight
detection device includes a reception unit that receives the
captured image; and a line-of-sight detection unit that detects a
direction of a line of sight of the eye of the user on the basis of
a feature point of the image displayed on the image display element
included in the captured image and a feature point of the image
displayed on the image display element.
Description
TECHNICAL FIELD
[0001] This disclosure relates to a head mounted display.
BACKGROUND ART
[0002] A technique is known in which the eyesight direction of a
user is detected by emitting non-visible light such as
near-infrared light to the user's eyes, and analyzing an image of
the user's eyes including reflected light. Information of the
detected eyesight direction of the user is reflected on the monitor
of, for example, a PC (Personal Computer), a game console or the
like, and thus use as a pointing device has been realized.
CITATION LIST
Patent Literature
[Patent Literature 1]
[0003] Japanese Unexamined Patent Application, First Publication
No. H2-264632
SUMMARY OF INVENTION
Technical Problem
[0004] A head mounted display is an image display device that
presents a three-dimensional image to a user wearing the device.
Generally, the head mounted display is used in a state of being
mounted to cover the visual range of a user. For this reason, a
user wearing the head mounted display has an external image
shielded. When the head mounted display is used as a display device
of an image of a moving picture, a game or the like, it is
difficult for a user to visually recognize an input device such as
a controller.
[0005] Therefore, the usability of a head mounted display as a
substitute for a pointing device by detecting the eyesight
direction of a user wearing the display is of convenience.
Particularly, the acquisition of geometric information (information
of spatial coordinates or a shape) of a user's cornea in a state
where the user wears a head mounted display is useful in estimating
the eyesight direction of the user.
[0006] It could therefore be helpful to provide a technique of
detecting geometric information of the cornea of a user wearing a
head mounted display.
Solution to Problem
[0007] Provided is a line-of-sight detection system including a
head-mounted display and a line-of-sight detection device, the
head-mounted display includes an image display element that
includes a plurality of pixels, each pixel including sub-pixels
that emit red, green, blue, invisible light, and displays an image
to be viewed by a user; an imaging unit that images an eye of the
user wearing the head-mounted display on the basis of the invisible
light emitted from the sub-pixel that emits the invisible light;
and a transmission unit that transmits a captured image captured by
the imaging unit, and the line-of-sight detection device includes a
reception unit that receives the captured image; and a
line-of-sight detection unit that detects a line of sight of the
eye of the user based on the captured image.
[0008] Further, in order to resolve the problem, a head-mounted
display according to an aspect of the present invention is a
head-mounted display mounted on a head of a user and used, and
includes a convex lens disposed in a position facing a cornea of
the user when the head-mounted display is mounted; an image display
element that includes a plurality of pixels, each pixel including
sub-pixels that emit red, green, blue, invisible light, and
displays an image to be viewed by a user; a camera that images a
video including the cornea of the user as a subject; and a housing
that houses the convex lens, the image display element, and the
camera.
[0009] Further, in the head-mounted display, the head-mounted
display may further include a control unit that selects the pixel
that emits invisible light among the plurality of pixels
constituting the image display element and causes the selected
pixel to emit light.
[0010] Further, the control unit may change the pixel that emits
the invisible light when a predetermined time elapses.
[0011] Further, the control unit may switch and control a light
emission timing of the sub-pixel that emit the invisible light and
the sub-pixel other than the sub-pixels that emit the invisible
light.
[0012] Meanwhile, any combination of the aforementioned components,
and implementation of our displays in the form of methods, devices,
systems, computer programs, data structures, recording mediums, and
the like may be considered part of this disclosure.
[0013] It is thus possible to provide a technique of detecting
geometric information of the cornea of a user wearing a head
mounted display.
BRIEF DESCRIPTION OF DRAWINGS
[0014] FIG. 1 is a diagram schematically illustrating a general
view of an example of our image system.
[0015] FIG. 2 is a diagram schematically illustrating an optical
configuration of an image display system housed by a housing.
[0016] FIGS. 3(a) and 3(b) are schematic diagrams illustrating a
reflection region.
[0017] FIGS. 4(a) and 4(b) are diagrams schematically illustrating
an example of dot patterns generated by a reflection region of a
convex lens.
[0018] FIG. 5 is a schematic diagram illustrating a relationship
between a captured dot pattern and a structure of a subject.
[0019] FIG. 6 is a diagram schematically illustrating a functional
configuration of an image reproducing device.
[0020] FIG. 7 is a schematic diagram illustrating an eyesight
direction of a user.
[0021] FIG. 8 is a schematic diagram illustrating calibration in an
eyesight direction which is executed by a head mounted display.
[0022] FIG. 9 is a diagram illustrating an example of a pixel
configuration of an image display element.
[0023] FIG. 10 is a diagram illustrating an example in which an eye
of a user is directly irradiated with image light, which includes
infrared light.
[0024] FIG. 11A and FIG. 11B illustrate an example in which a
line-of-sight detection is performed on the basis of an image
reflected in the eye of the user.
DESCRIPTION OF EMBODIMENTS
[0025] FIG. 1 is a diagram schematically illustrating a general
view of an image system 1 according to an example. The image system
1 includes a head mounted display 100 and an image reproducing
device 200. As shown in FIG. 1, the head mounted display 100 is
used in a state of being mounted on the head of a user 300.
[0026] The image reproducing device 200 generates an image
displayed by the head mounted display 100. Although not limited, as
an example, the image reproducing device 200 is a device capable of
reproducing an image of a stationary game console, a portable game
console, a PC, a tablet, a smartphone, a phablet, a video player, a
television or the like. The image reproducing device 200 connects
to the head mounted display 100 in a wireless or wired manner. In
an example shown in FIG. 1, the image reproducing device 200 is
wirelessly connected to the head mounted display 100. The wireless
connection of the image reproducing device 200 to the head mounted
display 100 can be realized using, for example, a wireless
communication technique such as known WI-FI (Registered Trademark)
or BLUETOOTH (Registered Trademark). Although not limited, as an
example, transmission of an image between the head mounted display
100 and the image reproducing device 200 is executed according to
the standard of MIRACAST (Trademark), WIGIG (Trademark), WHDI
(Trademark), or the like.
[0027] Meanwhile, FIG. 1 illustrates an example when the head
mounted display 100 and the image reproducing device 200 are
different devices. However, the image reproducing device 200 may be
built into the head mounted display 100.
[0028] The head mounted display 100 includes a housing 150, a
mounting fixture 160, and a headphone 170. The housing 150 houses
an image display system such as an image display element that
presents an image to the user 300, or a wireless transmission
module such as a WI-FI module or a BLUETOOTH (Registered Trademark)
module which is not shown. The mounting fixture 160 mounts the head
mounted display 100 on the head of the user 300. The mounting
fixture 160 can be realized by, for example, a belt, an elastic
band or the like. When the user 300 mounts the head mounted display
100 using the mounting fixture 160, the housing 150 is disposed at
a position where the eyes of the user 300 are covered. For this
reason, when the user 300 mounts the head mounted display 100, the
visual range of the user 300 is shielded by the housing 150.
[0029] The headphone 170 outputs a voice of an image reproduced by
the image reproducing device 200. The headphone 170 may be fixed to
the head mounted display 100. Even in a state where the user 300
mounts the head mounted display 100 using the mounting fixture 160,
the user can freely attach and detach the headphone 170.
[0030] FIG. 2 is a diagram schematically illustrating an optical
configuration of an image display system 130 housed by the housing
150. The image display system 130 includes a near-infrared light
source 103, an image display element 108, a hot mirror 112, a
convex lens 114, a camera 116, and an image output unit 118.
[0031] The near-infrared light source 103 is a light source capable
of emitting light of a near-infrared (approximately 700 nm to 2,500
nm) wavelength band. The near-infrared light is light of a
wavelength band of non-visible light which is not able to be
generally observed with a naked eye of the user 300.
[0032] The image display element 108 displays an image for
presentation to the user 300. The image displayed by the image
display element 108 is generated by a GPU (Graphic Processing
Unit), not shown, within the image reproducing device 200. The
image display element 108 can be realized using, for example, a
known LCD (Liquid Crystal Display), an organic EL display (Organic
Electro-Luminescent Display) or the like.
[0033] When the user 300 mounts the head mounted display 100, the
hot mirror 112 is disposed between the image display element 108
and the cornea 302 of the user 300. The hot mirror 112 has a
property of transmitting visible light generated by the image
display element 108, but reflecting near-infrared light.
[0034] The convex lens 114 is disposed on the opposite side to the
image display element 108 with respect to the hot mirror 112. In
other words, when the user 300 mounts the head mounted display 100,
the convex lens 114 is disposed between the hot mirror 112 and the
cornea 302 of the user 300. That is, when the head mounted display
100 is mounted to the user 300, the convex lens 114 is disposed at
a position facing the cornea 302 of the user 300.
[0035] The convex lens 114 condenses image display light that
passes through the hot mirror 112. For this reason, the convex lens
114 functions as an image enlargement unit that enlarges an image
generated by the image display element 108 and presents the
enlarged image to the user 300. Meanwhile, for convenience of
description, only one convex lens 114 is shown in FIG. 2, but the
convex lens 114 may be a lens group configured by combining various
lenses, and may be configured such that one lens has a curvature
and the other lens is a planar one-sided convex lens.
[0036] The near-infrared light source 103 is disposed at the
lateral side of the convex lens 114. The near-infrared light source
103 emits infrared light toward the inside of the convex lens 114.
The convex lens 114 is provided with a plurality of reflection
regions that reflect the infrared light inside the lens. These
reflection regions can be realized by providing fine regions having
different refractive indexes in the inside of the convex lens 114.
Meanwhile, providing the regions having different refractive
indexes in the convex lens 114 can be realized using a known laser
machining technique. The reflection region is provided at a
plurality of places in the inside of the convex lens 114.
[0037] Near-infrared light emitted toward the inside of the convex
lens 114 by the near-infrared light source 103 is reflected from
the reflection region inside the convex lens 114 and directed to
the cornea 302 of the user 300. Meanwhile, since the near-infrared
light is non-visible light, the user 300 is almost not able to
perceive the near-infrared light reflected from the reflection
region. In addition, the reflection region is a region which is as
large as a pixel constituting the image display element 108 or is
finer. For this reason, the user 300 is almost not able to perceive
the reflection region, and is able to observe image light emitted
by the image display element 108. Meanwhile, the details of the
reflection region will be described later.
[0038] Although not shown, the image display system 130 of the head
mounted display 100 includes two image display elements 108, and
can generate an image for presentation to the right eye of the user
300 and an image for presentation to the left eye independently of
each other. For this reason, the head mounted display 100 can
present a parallax image for the right eye and a parallax image for
the left eye, respectively, to the right eye and the left eye of
the user 300. Thereby, the head mounted display 100 can present a
stereoscopic image having a sense of depth to the user 300.
[0039] As described above, the hot mirror 112 transmits visible
light, and reflects near-infrared light. Therefore, image light
emitted by the image display element 108 passes through the hot
mirror 112 and reaches the cornea 302 of the user 300. In addition,
infrared light emitted from the near-infrared light source 103 and
reflected from the reflection region inside the convex lens 114
reaches the cornea 302 of the user 300.
[0040] The infrared light reaching the cornea 302 of the user 300
is reflected from the cornea 302 of the user 300, and directed to
the direction of the convex lens 114 again. This infrared light
passes through the convex lens 114, and is reflected from the hot
mirror 112. The camera 116 includes a filter that shields visible
light, and captures near-infrared light reflected from the hot
mirror 112. That is, the camera 116 is a near-infrared camera that
captures near-infrared light emitted from the near-infrared light
source 103 and reflected from the cornea of the user 300.
[0041] The image output unit 118 outputs an image captured by the
camera 116 to an eyesight detection unit that detects the eyesight
direction of the user 300. The image output unit 118 also outputs
the image captured by the camera 116 to a cornea coordinate
acquisition unit that acquires spatial coordinates of the user's
cornea. Specifically, the image output unit 118 transmits the image
captured by the camera 116 to the image reproducing device 200. The
eyesight detection unit and the cornea coordinate acquisition unit
will be described later, but can be realized by an eyesight
detecting program and a cornea coordinate acquiring program
executed by a CPU (Central Processing Unit) of the image
reproducing device 200. Meanwhile, when the head mounted display
100 has a computing resource of a CPU, a memory or the like, the
CPU of the head mounted display 100 may execute a program to
operate the eyesight detection unit.
[0042] Although a detailed description will be given later, in the
image captured by the camera 116, a bright point of the
near-infrared light reflected by the cornea 302 of the user 300 and
an image of the eye including the cornea 302 of the user 300
observed at a near-infrared wavelength band are captured.
[0043] In the convex lens 114, a plurality of reflection regions
are formed so that a pattern of infrared light appearing on the
cornea 302 of the user 300 forms structured light. The "structured
light" refers to light used in one method of three-dimensional
measurement of an object called a structured light method. More
specifically, the structured light is light emitted to cause a
light pattern having a special structure to appear on the surface
of an object to be measured. Various patterns caused to appear
through the structured light are present, but include as an
example, a plurality of dot patterns arrayed in a lattice shape,
stripe-shaped patterns disposed at equal intervals, a lattice
pattern, and the like. In addition, the structured light may
include not only single-color light, but also multi-color (such as,
for example, red, green and blue) light.
[0044] The structured light method is a known technique, and thus a
detailed description thereof will not be given, but the structured
light formed by the reflection region provided inside the convex
lens 114 causes a pattern formed by a plurality of infrared light
dots to appear in a region including the cornea 302 of the user
300.
[0045] FIGS. 3(a) and 3(b) are schematic diagrams illustrating
reflection regions 120. FIG. 3(a) is a diagram illustrating when
the convex lens 114 is seen from the lateral side (outer canthus of
the user 300), and FIG. 3(b) is a diagram illustrating when the
convex lens 114 is seen from the upper side (top of the head of the
user 300).
[0046] As shown in FIG. 3(a), the near-infrared light source 103
includes a plurality of LEDs 104. To avoid becoming complicated, in
FIG. 3(a), only one reference numeral 104 is shown, but the
rectangles of broken lines indicate the LEDs 104. The LED 104 emits
infrared light toward the inside of the convex lens 114.
[0047] As shown in FIG. 3(b), a plurality of reflection regions 120
are provided inside the convex lens 114. To avoid becoming
complicated, in FIG. 3(b), only one reference numeral 120 is shown,
but regions shown by diagonal segments in the drawing indicate the
reflection regions 120.
[0048] As described above, the reflection region 120 is a region
having a different refractive index as compared to other regions in
the convex lens 114. For this reason, the infrared light incident
from the LED 104 is totally reflected from the reflection region
120 and directed to the cornea 302 of the user 300. Since the
reflection region 120 is provided in a plurality of places in the
convex lens 114, as much infrared light as the reflection region
120 is directed to the cornea 302 of the user 300. Thereby, dot
patterns according to an installation shape of the reflection
region 120 can be formed on the cornea 302 of the user 300.
Meanwhile, providing a region having a refractive index in the
convex lens 114 can be realized using a known laser machining
technique.
[0049] As described above, the infrared light reaching the cornea
302 of the user 300 is reflected from the cornea 302 of the user
300, and directed to the direction of the convex lens 114 again. In
this case, when the infrared light reaches the reflection region
120, the infrared light is reflected by the reflection region 120
and is not able to pass through the convex lens 114. However, each
of the reflection regions 120 is a narrow region, and a relative
position between the reflection region 120 and the cornea 302 of
the user 300 continually changes with a change in the eyesight of
the user 300. For this reason, the probability of the infrared
light reflected from the cornea 302 of the user 300 and directed to
the convex lens 114 being reflected by the reflection region 120 is
small, which does not lead to a problem.
[0050] Even when it is assumed that the infrared light reflected
from the cornea 302 of the user 300 and directed to the convex lens
114 is reflected in the reflection region 120 at a certain timing,
the relative position between the reflection region 120 and the
cornea 302 of the user 300 changes at another timing, and thus the
infrared light is not reflected. Therefore, even when reflected
light from the cornea 302 of the user 300 is reflected by the
reflection region 120 at a certain moment by capturing the infrared
light in the camera 116 over time, the camera 116 can capture an
image at another moment, which does not lead to a problem.
[0051] FIG. 3(b) illustrates the reflection regions 120 present in
a certain horizontal cross section of the convex lens 114. The
reflection regions 120 are also present on other horizontal cross
sections of the convex lens 114. Therefore, the infrared light
emitted from the near-infrared light source 103 and reflected by
the reflection region 120 is distributed two-dimensionally in the
cornea 302 of the user 300 and forms dot patterns.
[0052] FIGS. 4(a) and 4(b) are diagrams schematically illustrating
an example of dot patterns generated by the reflection region 120
of the convex lens 114. More specifically, FIG. 4(a) is a schematic
diagram illustrating dot patterns emitted from the convex lens 114.
Meanwhile, in the drawing shown in FIG. 4(a), A to H are symbols
shown for convenience to describe the lineup of dots, and are not
dot patterns caused to appear in reality.
[0053] In FIG. 4(a), black circles and white circles are lined up
in a longitudinal direction at equal intervals. Meanwhile, the
longitudinal direction in FIG. 4 is a direction linking the top of
the head of the user to the chin, and is a vertical direction when
the user stands upright. Hereinafter, a dot row longitudinally
lined up from the symbol A may be described as a row A. The same is
true of the symbols B to H.
[0054] In FIG. 4(a), the black circle indicates that a dot (bright
point) caused by infrared light appears, and the white circle
indicates that a dot does not appear in reality. In the convex lens
114, a reflection region is provided so that a different dot
pattern appears at a different position of the cornea 302 of the
user 300. In an example shown in FIG. 4(a), all the dot rows are
formed as different dot patterns from the row A to the row H.
Therefore, the cornea coordinate acquisition unit can uniquely
specify each dot row by analyzing dot patterns in an image captured
by the camera 116.
[0055] FIG. 4(b) is a schematic diagram illustrating dot patterns
reflected from a region including the cornea 302 of the user 300,
and is a diagram schematically illustrating an image captured by
the camera 116. As shown in FIG. 4(b), the dot patterns captured by
the camera 116 are dot patterns mainly reaching the ocular surface
of the user 300, and dot patterns reaching the skin of the user 300
have a tendency to be captured. This is because the dot patterns
reaching the skin of the user 300 are diffusely reflected from the
skin, and the amount of light reaching the camera 116 is reduced.
On the other hand, dot patterns reaching the ocular surface of the
user 300 are subject to reflection close to specular reflection due
to the influence of tears or the like. For this reason, the amount
of light of the dot patterns reaching the camera 116 also
increases.
[0056] FIG. 5 is a schematic diagram illustrating a relationship
between a captured dot pattern and a structure of a subject. An
example shown in FIG. 5 shows a state where the row D in FIG. 4(a)
is captured.
[0057] As shown in FIG. 5, the camera 116 captures the cornea 302
of the user 300 from a downward direction (that is, direction of
the user's mouth). Generally, the human cornea has a shape
protruding in an eyesight direction. For this reason, even when
equally-spaced dot patterns appear in the cornea 302 of the user
300, an interval between each of the dots in the dot pattern
captured by the camera 116 changes depending on the shape of the
cornea 302 of the user 300. In other words, an interval between the
dot patterns appearing in the cornea 302 of the user 300 reflects
information in a depth direction (that is, direction in which the
infrared light is emitted to the cornea 302 of the user 300). This
interval between the dot patterns is analyzed, and thus the cornea
coordinate acquisition unit can acquire the shape of the cornea 302
of the user 300. Meanwhile, the above is not limited to when the
camera 116 captures the cornea 302 of the user 300 from the
downward direction. Light paths of infrared light incident on the
cornea 302 of the user 300 and infrared light reflected from the
cornea 302 of the user 300 may shift from each other, and the
camera 116 may capture, for example, the cornea 302 of the user 300
from a transverse direction or an upward direction.
[0058] FIG. 6 is a diagram schematically illustrating a functional
configuration of the image reproducing device 200. The image
reproducing device 200 includes a reception and transmission unit
210, an image generation unit 220, an eyesight detection unit 230,
and a cornea coordinate acquisition unit 240.
[0059] FIG. 6 illustrates a functional configuration to operate an
image generation process, an eyesight detection process, and a
cornea coordinate detection process performed by the image
reproducing device 200, and other configurations are omitted. In
FIG. 6, each component described as functional blocks that perform
various processes can be constituted by a CPU (Central Processing
Unit), a main memory, and other LSIs (Large Scale Integrations) in
a hardware manner. In addition, each component is realized by
programs or the like loaded into the main memory in a software
manner. Meanwhile, the programs may be stored in a computer
readable recording medium, and may be downloaded from a network
through a communication line. It is understood by those skilled in
the art that these functional blocks can be realized in various
forms by hardware only, software only, or a combination thereof,
and are not limited to any particular one.
[0060] The reception and transmission unit 210 executes the
transmission of information between the image reproducing device
200 and the head mounted display 100. The reception and
transmission unit 210 can be realized by a wireless communication
module according to the standard of MIRACAST (Trademark), WIGIG
(Trademark), WHDI (Trademark), or the like described above.
[0061] The image generation unit 220 generates an image displayed
on the image display element 108 of the head mounted display 100.
The image generation unit 220 can be realized using, for example,
the GPU or the CPU described above.
[0062] The cornea coordinate acquisition unit 240 analyzes the
interval between the dot patterns appearing in the cornea 302 of
the user 300, and thus acquires a three-dimensional shape of the
cornea 302 of the user 300. Thereby, the cornea coordinate
acquisition unit 240 can also estimate position coordinates of the
cornea 302 of the user 300 in a three-dimensional coordinate system
using the camera 116 as an origin.
[0063] Meanwhile, the camera 116 may be a monocular camera, and may
be a stereo camera including two or more imaging units. In this
case, the cornea coordinate acquisition unit 240 analyzes the
parallax image of the cornea 302 of the user 300 which is captured
by the camera 116, and thus can more accurately estimate the
position coordinates of the cornea 302 of the user 300 in the
three-dimensional coordinate system using the camera 116 as an
origin.
[0064] FIG. 7 is a schematic diagram illustrating an eyesight
direction of the user 300. As described above, the dot patterns
appearing in the cornea 302 are analyzed, and thus the cornea
coordinate acquisition unit 240 can acquire the shape of the cornea
302 of the user 300. Thereby, as shown in FIG. 6, the eyesight
detection unit 230 can detect a peak P of the cornea 302 of the
user 300 having an approximately hemisphere shape. Subsequently,
the eyesight detection unit 230 sets a plane 304 that comes into
contact with the cornea 302 at the peak P. In this case, the
direction of a normal line 306 of the plane 304 at the peak P is
set to the eyesight direction of the user 300.
[0065] Meanwhile, the cornea 302 of the user 300 is generally
aspherical rather than spherical. For this reason, in the above
method in which the cornea 302 of the user 300 is assumed to be
spherical, an estimation error may occur in the eyesight direction
of the user 300. Consequently, the eyesight detection unit 230 may
provide calibration for an eyesight direction estimation in advance
of the user 300 starting to use the head mounted display 100.
[0066] FIG. 8 is a schematic diagram illustrating calibration in an
eyesight direction executed by the eyesight detection unit 230. The
eyesight detection unit 230 generates nine points from points
Q.sub.1 to Q.sub.9 as shown in FIG. 8 in the image generation unit
220, and displays these points on the image display element 108.
The eyesight detection unit 230 causes the user 300 to keep
observation on these points in order from the point Q.sub.1 to the
point Q.sub.9, and detects the aforementioned normal line 306. In
addition, when the user 300 keeps observation on, for example, the
point Q1, the central coordinates (that is, coordinates of the peak
P described above with reference to FIG. 7) of the cornea 302 of
the user 300 are set to P.sub.1. In this case, the eyesight
direction of the user 300 is set to a direction P.sub.1-Q.sub.1
linking the point P1 to the point Q.sub.1 in FIG. 8. The eyesight
detection unit 230 compares the acquired direction of the normal
line 306 with the direction P.sub.1-Q.sub.1, and stores the error
thereof.
[0067] Hereinafter, similarly, the user 300 stores errors with
respect to nine directions P.sub.1-Q.sub.1, P.sub.2-Q.sub.2, . . .
, P.sub.9-Q.sub.9 of the point Q.sub.1 to the point Q.sub.9, and
thus the eyesight detection unit 230 can acquire a correction table
to correct the direction of the normal line 306 obtained by
calculation. The eyesight detection unit 230 acquires the
correction table in advance through calibration, and corrects the
direction of the normal line 306 obtained in the aforementioned
method, thereby allowing higher-accuracy eyesight direction
detection to be realized.
[0068] It is also considered that, after the user 300 mounts the
head mounted display 100 on the head and the eyesight detection
unit 230 performs calibration, a relative positional relationship
between the head of the user 300 and the head mounted display 100
changes. However, when the eyesight direction of the user 300 is
detected from the shape of the cornea 302 of the user 300 described
above, the relative positional relationship between the head of the
user 300 and the head mounted display 100 does not influence the
accuracy of detection of the eyesight direction. Therefore, it is
possible to realize robust eyesight direction detection with
respect to a change in the relative positional relationship between
the head of the user 300 and the head mounted display 100.
[0069] Regarding the above, a method in which the eyesight
detection unit 230 detects the eyesight direction of the user 300
using a geometric method has been described. The eyesight detection
unit 230 may execute eyesight direction detection based on an
algebraic method using coordinate transformation described below,
instead of the geometric method.
[0070] In FIG. 8, the coordinates of the point Q.sub.1 to the point
Q.sub.9 in the two-dimensional coordinate system which is set in a
moving image displayed by the image display element 108 are set to
Q.sub.1(x.sub.1, y.sub.1).sup.T, Q.sub.2(x.sub.2, y.sub.2).sup.T .
. . , Q.sub.9(x.sub.9, x.sub.9).sup.T, respectively. In addition,
the coordinates of the position coordinates P.sub.1 to P.sub.9 of
the cornea 302 of the user 300 when the user 300 keeps observation
on the point Q.sub.1 to the point Q.sub.9 are set to
P.sub.1(X.sub.1, Y.sub.1, Z.sub.1).sup.T, P.sub.2(X.sub.2, Y.sub.2,
Z.sub.2).sup.T, . . . , P.sub.9(Z.sub.9, Y.sub.9, Z.sub.9).sup.T,
respectively. Meanwhile, T indicates the transposition of a vector
or a matrix.
[0071] A matrix M having a size of 3.times.2 is defined as
Expression (1).
[ Math . 1 ] M = ( m 11 m 12 m 13 m 21 m 22 m 23 ) ( 1 )
##EQU00001##
[0072] In this case, when the matrix M satisfies Expression (2),
the matrix M becomes a matrix to project the eyesight direction of
the user 300 onto a moving image surface displayed by the image
display element 108.
P.sub.N=MQ.sub.N (N=1, . . . ,9) (2)
[0073] When Expression (2) is specifically written, Expression (3)
is established.
[ Math . 2 ] ( x 1 x 2 x 9 y 1 y 2 y 9 ) = ( m 11 m 12 m 13 m 21 m
22 m 23 ) ( X 1 X 2 X 9 Y 1 Y 2 Y 9 Z 1 Z 2 Z 9 ) ( 3 )
##EQU00002##
[0074] When Expression (3) is deformed, Expression (4) is
obtained.
[ Math . 3 ] ( x 1 x 2 x 9 y 1 y 2 y 9 ) = ( X 1 Y 1 Z 1 0 0 0 X 2
Y 2 Z 2 0 0 0 X 9 Y 9 Z 9 0 0 0 0 0 0 X 1 Y 1 Z 1 0 0 0 X 2 Y 2 Z 2
0 0 0 X 9 Y 9 Z 9 ) ( m 11 m 12 m 13 m 21 m 22 m 23 ) ( 4 ) [ Math
. 4 ] y = ( x 1 x 2 x 9 y 1 y 2 y 9 ) , A = ( X 1 Y 1 Z 1 0 0 0 X 2
Y 2 Z 2 0 0 0 X 9 Y 9 Z 9 0 0 0 0 0 0 X 1 Y 1 Z 1 0 0 0 X 2 Y 2 Z 2
0 0 0 X 9 Y 9 Z 9 ) , x = ( m 11 m 12 m 13 m 21 m 22 m 23 )
##EQU00003##
[0075] When the following expression is set, Expression (5) is
obtained.
Y=Ax (5)
[0076] In Expression (5), the elements of a vector y are the
coordinates of the points Q.sub.1 to Q.sub.9 displayed on the image
display element 108 by the eyesight detection unit 230, and thus
the elements are known. In addition, the elements of a matrix A are
coordinates of the peak P of the cornea 302 of the user 300
acquired by the cornea coordinate acquisition unit 240. Therefore,
the eyesight detection unit 230 can acquire the vector y and the
matrix A. Meanwhile, a vector x which is a vector obtained by
arranging the elements of the transformation matrix M is unknown.
Therefore, when the vector y and the matrix A are known, a problem
of estimating the matrix M becomes a problem of obtaining the
unknown vector x.
[0077] In Expression (5), when the number of expressions (that is,
the number of points Q presented to the user 300 when the eyesight
detection unit 230 performs calibration) is larger than the number
of unknowns (that is, the number of elements of the vector x is 6),
a priority determination problem occurs. In the example shown in
Expression (5), the number of expressions is nine, which leads to a
priority determination problem.
[0078] An error vector between the vector y and a vector Ax is set
to a vector e. That is, the relation of e=y-Ax is established. In
this case, in the meaning of minimizing a square sum of the
elements of the vector e, an optimum vector x.sub.opt is obtained
by Expression (6).
[0079] x.sub.opt=(A.sup.TA).sup.-1A.sup.Ty (6) wherein "-1"
indicates an inverse matrix.
[0080] The eyesight detection unit 230 constitutes the matrix M of
Expression (1) by using the elements of the obtained vector
x.sub.opt. Thereby, the eyesight detection unit 230 uses the matrix
M and the coordinates of the peak P of the cornea 302 of the user
300 acquired by the cornea coordinate acquisition unit 240, and
thus can estimate where on the moving image surface displayed by
the image display element 108 the user 300 keeps observation on
according to Expression (2).
[0081] It is also considered that, after the user 300 mounts the
head mounted display 100 on the head and the eyesight detection
unit 230 performs calibration, a relative positional relationship
between the head of the user 300 and the head mounted display 100
changes. However, the position coordinates of the peak P of the
cornea 302 constituting the matrix A described above are values
estimated by the cornea coordinate acquisition unit 240 as position
coordinates in the three-dimensional coordinate system using the
camera 116 as an origin. Even when it is assumed that the relative
positional relationship between the head of the user 300 and the
head mounted display 100 changes, a coordinate system based on the
position coordinates of the peak P of the cornea 302 does not
change. Therefore, even when the relative positional relationship
between the head of the user 300 and the head mounted display 100
changes slightly, coordinate transformation according to Expression
(2) is considered to be effective. Consequently, eyesight detection
executed by the eyesight detection unit 230 can improve robustness
with respect to a shift of the head mounted display 100 during
mounting.
[0082] As described above, according to the head mounted display
100, it is possible to detect geometric information of the cornea
302 of the user 300 wearing the head mounted display 100.
[0083] Particularly, the head mounted display 100 can acquire the
three-dimensional shape and the position coordinates of the cornea
302 of the user 300, it is possible to estimate the eyesight
direction of the user 300 with good accuracy.
[0084] As stated above, our displays have been described on the
basis of our examples. The examples have been described for
exemplary purposes only, and it can be readily understood by those
skilled in the art that various modifications may be made by a
combination of each of these components or processes, which are
also encompassed by the scope of this disclosure.
[0085] In the above, a description has been given of an example
when the convex lens 114 is provided with the reflection regions
120 so that different dot patterns appear at different positions of
the cornea 302 of the user 300. Dots having different blinking
patterns may be caused to appear at different positions of the
cornea 302 of the user 300, instead thereof or in addition thereto.
This can be realized by forming, for example, the near-infrared
light source 103 by a plurality of different light sources, and
changing a blinking pattern in each light source.
[0086] Further, although the near-infrared light is radiated from
the near-infrared light source 103 in the above-described
embodiment, a light source that radiates near-infrared light may be
included in each pixel constituting the image display element 108.
That is, generally, one pixel is constituted by RGB, and a light
emitting element that emits near-infrared light is provided in
addition to the light emitting elements that emit red light, green
light, and blue light. When a sub-pixel that emits near-infrared
light is included as a sub-pixel in the image display element, the
near-infrared light source 103 may not be provided in the
head-mounted display 100.
[0087] FIG. 9 is a diagram illustrating a configuration example of
the screen display element 108. The image display element 108
includes pixels constituting an image and sub-pixels constituting
the pixels. One pixel will be described by way of example. A pixel
900 that is one pixel constituting the image display element 108
includes sub-pixels 900r, 9009, 900b, and 900i. The sub-pixel 900r
is a pixel that emits red light (including light emission of a
backlight, light emission of the pixel itself, or light emission of
both), the sub-pixel 900g is a pixel that emits green light, and
the sub-pixel 900b is a pixel that emits blue light. In the case of
a normal display device, sub-pixels including three colors are set
as one pixel, or sub-pixels with white light added thereto are set
as one pixel. However, in the case of the screen display element
108 according to the present invention, the sub-pixel 900i is
included as a sub-pixel.
[0088] The sub-pixel 900i is a pixel that emits near-infrared
light. It is determined whether or not the sub-pixel 900i of each
pixel 900 emits the near-infrared light according to an instruction
from the video output unit 224, and information indicating whether
or not the sub-pixel 900i of each pixel 900 emits the near-infrared
light is included in display image data that the video output unit
224 outputs to the head-mounted display 100. Thus, an emission
pattern of the near-infrared light desired by an operator can be
formed. Therefore, in a video to be displayed at that time, for
example, formation of a pattern can also be realized such that the
near-infrared light is not emitted according to content of the
image in a pixel that strongly emits red light. The light emission
of the sub-pixel 900i may be executed by a display unit included in
the head-mounted display shown in the above embodiment, or may be
executed by an irradiation unit that controls the sub-pixel 900i
that emits near-infrared light.
[0089] A configuration for emitting near-infrared light in the
image is effective regardless of a type of the display device, and
can be applied to various display devices such as an LCD, a plasma
display, an organic EL display. Further, even when a sub-pixel for
the near-infrared light is included in the pixel, the user do not
feel uncomfortable when viewing the image by setting a wavelength
of the near-infrared light to be radiated to be outside a range of
wavelengths that can be perceived by a person with respect to an
actually displayed image.
[0090] Further, control of the sub-pixel 900i of which of the
respective pixels of the image display element 108 is caused to
emit light may be executed by the display unit or the irradiation
unit of the head-mounted display 100, or may be executed by the
display unit or the irradiation unit of the head-mounted display
according to designation of the video generation unit 220. Thus,
structural light shown in the above embodiment can be realized.
Further, at this time, turn-on of the sub-pixel 900i may be
appropriately changed. Particularly, for example, when a moving
image is displayed on the image display element 108, the sub-pixel
900i that emits the near-infrared light may be changed each time a
predetermined time elapses in time series. Here, the predetermined
time may be defined by the number of seconds. In the case of a
moving image, the predetermined time may be defined by the number
of frames, or the predetermined time may be defined for each
blanking period. In this case, the frame number of the moving image
and coordinate position information of the image display element
108 of the sub-pixel 900i that emits near-infrared light at that
time are stored in the head-mounted display or a line-of-sight
detection device in association with each other, such that a
line-of-sight detection can be appropriately executed each time.
Further, a blinking pattern of the sub-pixel 900i that emits the
near-infrared light may be changed in a predetermined period.
[0091] Further, a timing at which the sub-pixel 900i is turned on
and a timing at which the sub-pixel 900r, the sub-pixel 900g, and
the sub-pixel 900b are turned on may be different timings. The
camera 116 may be configured to execute imaging only at a timing at
which the sub-pixel 900i is turned on. Further, as a scheme for
realizing this configuration, for example, the configuration may be
realized so that a blanking period of the sub-pixel 900r, the
sub-pixel 900g, and the sub-pixel 900b and a blanking period of the
sub-pixel 900i are set to be different time zones. More
specifically, it is preferable for the blanking period of the
sub-pixel 900r, the sub-pixel 900g, and the sub-pixel 900b to be
set as a turn-on period of the sub-pixel 900i and for the blanking
period of the sub-pixel 900i to be set as the turn-on period of the
sub-pixel 900r, the sub-pixel 900g, and the sub-pixel 900b.
[0092] Further, in the above embodiment, the image is displayed on
the image display element 108 provided on the head-mounted display
100 and the video is provided to the user, but the present
invention is not limited thereto. FIG. 10 illustrates an example in
which the video is provided to the user without being displayed on
the image display element 108 that may be adopted by the
head-mounted display 100.
[0093] A display system 1000 illustrated in FIG. 10 is a display
system in a case in which the image displayed on the image display
element 108 is not caused to be visually recognized by the user,
but the video is directly projected onto the eyes of the user. In
recent years, the development of a virtual retinal display that
directly projects the video on the eyes of the user has been
remarkable. Image light can be directly projected without causing
any adverse effect on the eyes of the user, such that the image can
be caused to be recognized by the user. This technology can also be
applied to the head-mounted display 100 according to the present
invention. As illustrated in FIG. 10, the display system 1000
directly projects image data transmitted from the video output unit
224 onto the eyes 303 of the user via a convex lens 114 using an
optical fiber 1001. In this case, an invisible light source is also
included in a pattern as shown in the above embodiment in the image
1102 and is also radiated on the eyes 303 of the user. Therefore,
line-of-sight detection can be realized by imaging a cornea that
reflects the invisible light radiated on the eyes of the user using
the camera 116. Although an example in which the image is directly
projected from the optical fiber 1001 to the eyes of the user is
shown in FIG. 10, the image light from the optical fiber 1001 may
be reflected by a hot mirror or the like and projected onto the
eyes 303 of the user via the convex lens 114.
[0094] In the above embodiment, the example in which the
line-of-sight detection is assumed, a marker image is displayed,
the marker image is caused to be gazed by the user, mapping
information indicating a relationship between the cornea and a
monitor obtained by calibration is stored, and the line-of-sight
detection for the user when the user views an actual video is
performed is shown. However, it goes without saying that a
line-of-sight detection scheme is not limited to the above
algorithm. Line-of-sight detection using the following scheme is
also included in the idea of the present invention.
[0095] FIG. 11(a) and FIG. 11(b) illustrate an example of a
line-of-sight detection method. A line-of-sight detection unit 230
stores a position of a corneal center of the user when the user
views a center of the image. FIG. 11(a) is a diagram illustrating
an image 1100 obtained by imaging a visible light image including a
left eye of the user. Here, although the left eye is used, the same
applies to a right eye. It is generally known that a landscape that
the user views is reflected in the eyes of the user. As illustrated
in FIG. 11(a), an image 1110 displayed on the image display element
108 illustrated in FIG. 11(b) is reflected in the eyes of the
user.
[0096] When line-of-sight detection using an image reflected in the
eyes is realized, a visible light camera is used as the camera 116.
Accordingly, an image based on normal visible light can be imaged,
and an image as illustrated in FIG. 11(a) can be acquired. The
line-of-sight detection unit 230 specifies a feature point by
performing image analysis such as edge analysis and corner analysis
on the obtained image. In FIG. 11(a), for example, feature points
1101a, 1101b, and 1101c can be detected. The line-of-sight
detection unit 230 compares the detected feature point with a
position of the corneal center of the user, specifies the amount of
movement from the position of the corneal center of the user when
the user views the center of the image stored in advance, and
detects a place (line-of-sight direction) at which the user is
gazing. Such a configuration may be used. When the line-of-sight
detection using the image reflected in the eyes of the user is
performed, the head-mounted display 100 needs to include a half
mirror that partly reflects visible light and partially transmits
the visible light in place of the hot mirror 112 or needs to
include a visible light camera that directly images the eyes of the
user separately from the camera 116.
[0097] Further, although the position of the corneal center of the
user when the user is viewing the center of the image is stored in
the above description, the line-of-sight detection can be performed
without storing the position information. That is, the feature
point is detected from a first frame of the moving image output by
the video output unit 224 and a second frame following the first
frame (the second frame may not be a frame immediately after the
first frame, but at least a part of the same object as an object to
be displayed in the first frame is required to be displayed).
Further, a position of a corneal center of the user gazing at the
first frame at that time and a position of a corneal center of the
user gazing at the second frame are detected. The line-of-sight
detection unit 230 may be configured to detect a point
(line-of-sight direction) of the second frame at which the user is
gazing on the basis of a movement distance and a movement direction
on the screen display element 108 from the feature point in the
first frame to the corresponding feature point in the second frame,
and a movement distance and a movement direction on the screen
display element 108 from the position of the corneal center of the
user in the first frame to the position of the corneal center of
the user in the second frame.
[0098] According to these schemes, it is not necessary to execute
the calibration by displaying the marker image shown in the above
embodiment. Therefore, prior preparation for performing the
line-of-sight detection using the head-mounted display 100 may not
be performed, and convenience of the user can be improved.
INDUSTRIAL APPLICABILITY
[0099] The present invention is applicable to a head mounted
display.
REFERENCE SIGNS LIST
[0100] 1 Image system [0101] 100 Head mounted display [0102] 103
near-infrared light source [0103] 104 LED [0104] 108 Image display
element [0105] 112 Hot mirror [0106] 114 Convex lens [0107] 116
Camera [0108] 118 Image output unit [0109] 120 Reflection region
[0110] 130 Image display system [0111] 150 Housing [0112] 160
Mounting fixture [0113] 170 Headphone [0114] 200 Image reproducing
device [0115] 210 Reception and transmission unit [0116] 220 Image
generation unit [0117] 230 Eyesight detection unit
* * * * *