U.S. patent application number 16/020158 was filed with the patent office on 2020-01-02 for gesture-based content sharing in artifical reality environments.
The applicant listed for this patent is Facebook Technologies, LLC. Invention is credited to Daniel Andersen, Albert Peter Hwang, Kenrick Cheng-Kuo Kin.
Application Number | 20200004401 16/020158 |
Document ID | / |
Family ID | 69007569 |
Filed Date | 2020-01-02 |
![](/patent/app/20200004401/US20200004401A1-20200102-D00000.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00001.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00002.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00003.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00004.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00005.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00006.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00007.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00008.png)
![](/patent/app/20200004401/US20200004401A1-20200102-D00009.png)
United States Patent
Application |
20200004401 |
Kind Code |
A1 |
Hwang; Albert Peter ; et
al. |
January 2, 2020 |
GESTURE-BASED CONTENT SHARING IN ARTIFICAL REALITY ENVIRONMENTS
Abstract
Embodiments are directed to a near eye display (NED) system for
displaying artificial reality content to a user. In some
embodiments, multiple users may be in a local area, each using a
different NED. A first user of a first NED may view virtual content
using a first NED. The first NED may comprise an imaging device
capable of capturing images of the local area, allowing the first
NED to identify gestures performed by the first user and/or by
other users in the local area. In some embodiments, the first NED
may, in response to detecting one or more predetermined gestures
performed by the first user, share virtual content displayed to the
first user with a second user using a second NED, allowing the
second user to view the virtual content through the second NED.
Inventors: |
Hwang; Albert Peter;
(Seattle, WA) ; Andersen; Daniel; (Redmond,
WA) ; Kin; Kenrick Cheng-Kuo; (Vancouver,
WA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Facebook Technologies, LLC |
Menlo Park |
CA |
US |
|
|
Family ID: |
69007569 |
Appl. No.: |
16/020158 |
Filed: |
June 27, 2018 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/017 20130101;
G06T 19/006 20130101; G06K 9/00335 20130101; G02B 27/0093 20130101;
G06F 3/013 20130101; G06K 9/00355 20130101; G02B 2027/0178
20130101; G02B 2027/0138 20130101; G06F 3/1454 20130101; G06F
3/0346 20130101; G06F 3/04815 20130101; G06F 3/011 20130101; G06F
3/1423 20130101; G09G 2354/00 20130101; G06K 9/00671 20130101; G06F
3/0482 20130101; G06K 9/00261 20130101; G06T 2219/024 20130101;
G06F 2203/0381 20130101; G02B 27/0172 20130101 |
International
Class: |
G06F 3/0481 20060101
G06F003/0481; G06F 3/01 20060101 G06F003/01; G06F 3/14 20060101
G06F003/14 |
Claims
1. A system comprising: a near eye display (NED) comprising: an
optical assembly having an electronic display that is configured to
display images in accordance with display instructions; an imaging
device configured to capture one or more images of a portions of a
local area of the NED; and a controller configured to: determine a
position of a hand of a first user of the NED located within the
local area using the captured one or more images; identify a second
user within the local area of the NED, the second user associated
with a second NED; display to the first user using the NED a first
virtual object corresponding to a first content item; transmit a
first message to the second NED to cause the second NED to display
to the second user a representation of the first virtual object,
wherein the contents of the first content item corresponding to the
first virtual object are obscured to the second user; identify a
gesture performed by the hand of the first user based upon one or
more determined positions of the hand, the gesture corresponding to
a predetermined sequence of one or more hand movements or hand
positions; and responsive to identifying the gesture, transmit a
second message to the second NED enabling the second NED to access
the first content item, and to cause the second NED to display to
the second user a second representation of the first virtual object
where the contents of the first content item corresponding to the
first virtual object are visible to the second user.
2. The system of claim 1, wherein the controller is configured to
display the first virtual object corresponding to the first content
item to the first user at a location in a virtual environment
corresponding to a particular part of the first user's body.
3. The system of claim 2, wherein the particular part of the first
user's body is a hand of the first user.
4. The system of claim 1, wherein the gesture corresponds to an
arcing movement a first hand of the first user starting from within
a threshold distance of a second hand of the first user and moving
in a substantially horizontal direction over an arc-shaped
path.
5. The system of claim 1, wherein the first message to the second
NED causes the second NED to display to the second user the
representation of the first virtual object at a location in a
virtual environment corresponding to a particular part of the first
user's body.
6. The system of claim 5, wherein the controller is further
configured to: identify a second gesture performed by the first
user; responsive to identifying a performance of the second
gesture, transmit a third message to the second NED to cause the
second NED to display the second representation of the first
virtual object at a location in the virtual environment
corresponding to a particular part of the second user's body.
7. The system of claim 5, wherein the third message to the second
NED is configured to grant permission to the second user of the
second NED to perform a manipulation of the first content item.
8. The system of claim 5, wherein the second gesture comprises a
first movement in which a hand of the first user moves on a path
towards a hand of the second user, such that the hand of the first
user is moved to within a threshold distance of the hand of the
second user.
9. The system of claim 8, wherein the second gesture further
corresponds to a hand of the second user moving on a path towards
the hand of the first user.
10. The system of claim 8, wherein the second gesture further
corresponds to the hand of the first user oriented such that a
distal end of the first user's index finger and a distal end of the
first user's thumb on the hand of the first user are within a
threshold distance of each other during the first movement.
Description
BACKGROUND
[0001] The present disclosure generally relates to a gesture and
eye tracking system, and specifically gesture-based content sharing
in artificial reality environments. Augmented reality systems
typically rely on wearable devices that have smaller form factors
than classical virtual reality (VR) head mounted devices. The use
of augmented reality systems presents new challenges in user
interaction. Previous methods of user interaction with the local
area may not be sufficient or optimal in an augmented reality
system. For example, without the use of augmented reality, a user
may need to interact physically with a device in a local area in
order to enable a change in that device. However, with the user of
augmented reality, both the device and the user experience may be
upgraded to allow the user to cause a change in the device using
methods other than simply physical interaction. However, such
changes in user experience should be intuitive for the user to
understand and should be technically feasible. Current method of
user interaction in augmented reality are not readily intuitive and
do not exploit the technical capabilities of an augmented reality
system, and thus are not optimal for use.
SUMMARY
[0002] Embodiments are directed to a near eye display (NED) system
for displaying artificial reality content to a user. In some
embodiments, multiple users may be in a local area, each using a
different NED. A first user of a first NED may view virtual content
using a first NED. The first NED may comprise an imaging device
capable of capturing images of the local area, allowing the first
NED to identify gestures performed by the first user and/or by
other users in the local area. In some embodiments, the first NED
may, in response to detecting one or more predetermined gestures
performed by the first user, share virtual content displayed to the
first user with a second user using a second NED, allowing the
second user to view the virtual content through the second NED.
[0003] In some embodiments, the system comprises a NED that
includes an optical assembly having an electronic display that is
configured to display images in accordance with display
instructions and an imaging device configured to capture one or
more images of a portions of a local area of the NED. The system
further comprises a controller configured to determine a position
of a hand of a first user of the NED located within the local area
using the captured one or more images and to identify a second user
within the local area of the NED who is associated with a second
NED. The controller is configured to display to the first user
using the NED a first virtual object corresponding to a first
content item.
[0004] In some embodiments, the first user may share the first
content item with the second user, allowing the second user to view
and/or manipulate the content item using the second NED, by
performing one or more gestures that can be detected by the imaging
device of the system. For example, the controller may transmit a
first message to the second NED to cause the second NED to display
to the second user a representation of the first virtual object,
wherein the contents of the first content item corresponding to the
first virtual object are obscured to the second user, and identify
a gesture performed by the hand of the first user based upon one or
more determined positions of the hand, the gesture corresponding to
a predetermined sequence of one or more hand movements or hand
positions. Responsive to identifying the gesture, the controller
transmits a second message to the second NED enabling the second
NED to access the first content item, and to cause the second NED
to display to the second user a second representation of the first
virtual object where the contents of the first content item
corresponding to the first virtual object are visible to the second
user. In some embodiments, the gesture may comprise a movement by a
first hand of the first user starting from within a threshold
distance of a second hand of the first user over a substantially
horizontal arcing path, similar to opening a book. In some
embodiments, the gesture may comprise a movement by a first hand of
the first user towards a hand of the second user. In addition, the
first NED may also detect a movement of the hand of the second user
towards the first hand of the first user, simulating the passing of
an object between the first and second users.
BRIEF DESCRIPTION OF THE DRAWINGS
[0005] FIG. 1 is a diagram of an eyewear device, in accordance with
one or more embodiments.
[0006] FIG. 2 is a cross section of the eyewear device of FIG. 1,
in accordance with one or more embodiments.
[0007] FIG. 3 is a block diagram of a NED system with an eye
tracker, in accordance with one or more embodiments.
[0008] FIG. 4 illustrates an example of a virtual menu that may be
displayed to a user by an eyewear device, in accordance with one or
more embodiments.
[0009] FIG. 5 is a flowchart of an example process for manipulating
an artificial reality environment based upon identified gestures,
in accordance with one or more embodiments.
[0010] FIG. 6A illustrates a view where another user has content
not viewable by the first user, in accordance with one or more
embodiments.
[0011] FIG. 6B illustrates a view after the other user has shared
the content with the first user, such that the content is visible
to the first user, in accordance with one or more embodiments.
[0012] FIG. 6C illustrates a view where the other user shares
content through the use of a pointing device, in accordance with
one or more embodiments.
[0013] FIG. 7 illustrates another example of sharing data, in
accordance with one or more embodiments.
[0014] FIG. 8 illustrates a flowchart of an example process for
sharing data between different users of NED systems, in accordance
with one or more embodiments.
[0015] The figures depict embodiments of the present disclosure for
purposes of illustration only. One skilled in the art will readily
recognize from the following description that alternative
embodiments of the structures and methods illustrated herein may be
employed without departing from the principles, or benefits touted,
of the disclosure described herein.
DETAILED DESCRIPTION
[0016] Embodiments of the invention may include or be implemented
in conjunction with an artificial reality system. Artificial
reality is a form of reality that has been adjusted in some manner
before presentation to a user, which may include, e.g., a virtual
reality (VR), an augmented reality (AR), a mixed reality (MR), a
hybrid reality, or some combination and/or derivatives thereof.
Artificial reality content may include completely generated content
or generated content combined with captured (e.g., real-world)
content. The artificial reality content may include video, audio,
haptic feedback, or some combination thereof, and any of which may
be presented in a single channel or in multiple channels (such as
stereo video that produces a three-dimensional effect to the
viewer). Additionally, in some embodiments, artificial reality may
also be associated with applications, products, accessories,
services, or some combination thereof, that are used to, e.g.,
create content in an artificial reality and/or are otherwise used
in (e.g., perform activities in) an artificial reality. The
artificial reality system that provides the artificial reality
content may be implemented on various platforms, including a
head-mounted display (HMD) connected to a host computer system, a
standalone HMD, a mobile device or computing system, or any other
hardware platform capable of providing artificial reality content
to one or more viewers.
[0017] In some embodiments, the near eye display (NED) system is
configured to project artificial reality content (e.g., AR content)
to a user of the NED system. The NED system further includes a hand
pose and gesture tracking system to determine hand positions and/or
gestures performed by the user of the NED system and/or by other
individuals with a local area of the NED system. Based upon the
identified gestures, the NED system is able to perform various
functions on content items displayed in the artificial reality
environment, such as sharing displayed content items with other
users of the NED system or of other NED systems. For example, a
first user of the NED system viewing a displayed content item may
wish to share the content item with a second user in a local area.
Similar to how the first user may be able to share a real-world
object with the second user by handing the object to the second
user, the first user may perform a "pass" gesture that simulates
the handing of an object to the second user. In addition, the NED
system may detect as part of the "pass" gesture, the second user
performing a gesture simulating receipt of the object from the
first user, and in response transmit the content item to the second
user, allowing the second user to view the content item. As such,
the first user of the NED system is able to share content items
viewed in a virtual environment with other users within the local
area, by performing gestures which are intuitive and similar to
interactions with real world objects.
[0018] As used herein, the local area of the NED system may refer
to an area surrounding a NED of the NED system. In some
embodiments, the local area of the NED system may refer to a
360.degree. area surrounding a location of a NED of the NED system
that would be potentially visible to a user of the NED at the
location. In addition, while the follow discussion refers primarily
to AR content, it is understood that in other embodiments the NED
system may display to the user VR content in a VR environment.
Near Eye Display System Overview
[0019] FIG. 1 is a diagram of an eyewear device 100, in accordance
with one or more embodiments. In some embodiments, the eyewear
device 100 is a near eye display (NED) for presenting media to a
user. Examples of media presented by the eyewear device 100 include
one or more images, text, video, audio, or some combination
thereof. In some embodiments, audio is presented via an external
device (e.g., speakers and/or headphones) that receives audio
information from the eyewear device 100, a console (not shown), or
both, and presents audio data based on the audio information. The
eyewear device 100 can be configured to operate as an artificial
reality NED. In some embodiments, the eyewear device 100 may
augment views of a physical, real-world environment with
computer-generated elements (e.g., images, video, sound, etc.).
[0020] The eyewear device 100 shown in FIG. 1 includes a frame 105
and an optical assembly 110, which is surrounded by a rim 115. The
optical element 110 is substantially transparent (e.g., allows a
percentage transmittance) in the visible spectrum and may also
include a substantially transparent electronic display. The frame
105 is coupled to one or more optical elements. In some
embodiments, the frame 105 may represent a frame of eye-wear
glasses. The optical assembly 110 may be configured for users to
see content presented by the eyewear device 100. For example, the
eyewear device 110 can include at least one waveguide display
assembly (not shown) for directing one or more image light to an
eye of the user. A waveguide display assembly includes, e.g., a
waveguide display, a stacked waveguide display, a stacked waveguide
and powered optical elements, a varifocal waveguide display, or
some combination thereof. For example, the waveguide display may be
monochromatic and include a single waveguide. In some embodiments,
the waveguide display may be polychromatic and include a single
waveguide. In yet other embodiments, the waveguide display is
polychromatic and includes a stacked array of monochromatic
waveguides that are each associated with a different band of light,
i.e., are each sources are of different colors. A varifocal
waveguide display is a display that can adjust a focal position of
image light emitted from the waveguide display. In some
embodiments, a waveguide display assembly may include a combination
of one or more monochromatic waveguide displays (i.e., a
monochromatic waveguide display or a stacked, polychromatic
waveguide display) and a varifocal waveguide display. Waveguide
displays are described in detail in U.S. patent application Ser.
No. 15/495,373, incorporated herein by references in its
entirety.
[0021] In some embodiments, the optical assembly 110 may include
one or more lenses or other layers, such as lenses for filtering
ultraviolet light (i.e., sunglass lenses), polarizing lenses,
corrective or prescription lenses, safety lenses, 3D lenses, tinted
lenses (e.g., yellow tinted glasses), reciprocal focal-plane
lenses, or clear lenses that do not alter a user's view. The
optical assembly 110 may include one or more additional layers or
coatings, such as protective coatings, or coatings for providing
any of the aforementioned lens functions. In some embodiments, the
optical assembly 110 may include a combination of one or more
waveguide display assemblies, one or more lenses, and/or one or
more other layers or coatings.
[0022] FIG. 2 is a cross-section 200 of the eyewear device 100
illustrated in FIG. 1, in accordance with one or more embodiments.
The optical assembly 110 is housed in the frame 105, which is
shaded in the section surrounding the optical assembly 110. A
user's eye 220 is shown, with dotted lines leading out of the pupil
of the eye 220 and extending outward to show the eye's field of
vision. An eyebox 230 shows a location where the eye 220 is
positioned if the user wears the eyewear device 100. The eyewear
device 100 includes an eye tracking system.
[0023] The eye tracking system determines eye tracking information
for the user's eye 220. The determined eye tracking information may
include information about a position of the user's eye 220 in an
eyebox 230, e.g., information about an angle of an eye-gaze. An
eyebox represents a three-dimensional volume at an output of a
display in which the user's eye is located to receive image
light.
[0024] In one embodiment, the eye tracking system includes one or
more light sources to illuminate the eye at a particular wavelength
or within a particular band of wavelengths (e.g., infrared). The
light sources may be placed on the frame 105 such that the
illumination from the light sources are directed to the user's eye
(e.g., the location of the eyebox 230). The light sources may be
any device capable of producing visible or infrared light, such as
a light emitting diode. The illumination of the user's eye by the
light sources may assist the eye tracker 240 in capturing images of
the user's eye with more detail. The eye tracker 240 receives light
that is emitted from the light sources and reflected off of the eye
220. The eye tracker 240 captures images of the user's eye, and the
eye tracker 240 or an external controller can analyze the captured
images to measure a point of gaze of the user (i.e., an eye
position), motion of the eye 220 of the user (i.e., eye movement),
or both. The eye tracker 240 may be a camera or other imaging
device (e.g., a digital camera) located on the frame 105 at a
position that is capable of capturing an unobstructed image of the
user's eye 220 (or eyes).
[0025] In one embodiment, the eye tracking system determines depth
information for the eye 220 based in part on locations of
reflections of the light sources. Additional discussion regarding
how the eye tracker 240 determines depth information is found in,
e.g., U.S. application Ser. No. 15/456,383 and U.S. application
Ser. No. 15/335,634, both of which are hereby incorporated by
reference. In another embodiment, the eye tracker 240 does not
include light sources, but instead captures images of the user's
eye 220 without additional illumination.
[0026] The eye tracker 240 can be embedded in an upper portion of
the frame 105, but may be located at any portion of the frame at
which it can capture images of the user's eye. While only one eye
tracker 240 is shown in FIG. 2, the eyewear device 100 may include
multiple eye trackers 240 per eye 220.
[0027] By tracking the user's gaze, the eye tracker 240 can be used
to determine where a user is looking at. This can be combined with
a system, described below, which determines the gesture of the same
user's hands. The combination of both a gaze and a particular
gesture may be detected by the system and, in response, the system
may perform some action based on the combination of gesture and
gaze.
[0028] FIG. 3 is a block diagram of a NED system 300, in accordance
with one or more embodiments-. The NED system 300 shown by FIG. 3
comprises a NED 305 coupled to a controller 310, with the
controller 310 coupled to an imaging device 315. While FIG. 3 shows
an example NED system 300 including one NED 305 and one imaging
device 315, in other embodiments any number of these components may
be included in the NED system 300. In alternative configurations,
different and/or additional components may be included in the NED
system 300. Similarly, functionality of one or more of the
components can be distributed among the components in a different
manner than is described here. For example, some or all of the
functionality of the controller 310 may be contained within the NED
305. The NED system 300 may operate in an artificial reality
environment.
[0029] The NED 305 presents content to a user. In some embodiments,
the NED 305 is the eyewear device 100. Examples of content
presented by the NED 305 include one or more images, video, audio,
text, or some combination thereof. In some embodiments, audio is
presented via an external device (e.g., speakers and/or headphones)
that receives audio information from the NED 305, the controller
310, or both, and presents audio data based on the audio
information. In some embodiments, the NED 305 operates as an
artificial reality NED. In some embodiments, the NED 305 may
augment views of a physical, real-world environment with
computer-generated elements (e.g., images, video, sound, etc.).
[0030] The NED 305 includes an optical assembly 320 for each eye,
an eye tracker 325, an inertial measurement unit (IMU) 330, one or
more position sensors 335, and a depth camera array (DCA) 340. Some
embodiments of the NED 305 have different components than those
described here. Similarly, the functions can be distributed among
other components in the NED system 300 in a different manner than
is described here. In some embodiments, the optical assembly 320
displays images to the user in accordance with data received from
the controller 310. In one embodiment, the optical assembly 320 is
substantially transparent (e.g., by a degree of transmittance) to
electromagnetic radiation in the visible spectrum.
[0031] The eye tracker 325 tracks a user's eye movement. The eye
tracker 325 includes a camera for capturing images of the user's
eye. An example of the placement of the eye tracker is shown in eye
tracker 240 as described with respect to FIG. 2. Based on the
detected eye movement, the eye tracker 325 may communicate with the
controller 310 for further processing.
[0032] In some embodiments, the eye tracker 325 allows a user to
interact with content presented to the user by the controller 310
based on the detected eye movement. Example interactions by the
user with presented content include: selecting a portion of content
presented by the controller 310 (e.g., selecting an object
presented to the user), movement of a cursor or a pointer presented
by the controller 310, navigating through content presented by the
controller 310, presenting content to the user based on a gaze
location of the user, or any other suitable interaction with
content presented to the user.
[0033] In some embodiments, NED 305, alone or conjunction with the
controller 310 or another device, can be configured to utilize the
eye tracking information obtained from the eye tracker 325 for a
variety of display and interaction applications. The various
applications include, but are not limited to, providing user
interfaces (e.g., gaze-based selection), attention estimation
(e.g., for user safety), gaze-contingent display modes, metric
scaling for depth and parallax correction, etc. In some
embodiments, based on information about position and orientation of
the user's eye received from the eye tracking unit, a controller
(e.g., the controller 310) determines resolution of the content
provided to the NED 305 for presentation to the user on the optical
assembly 320. The optical assembly 320 may provide the content in a
foveal region of the user's gaze (and may provide it at a higher
quality or resolution at this region).
[0034] In another embodiment, the eye tracking information obtained
from the eye tracker 325 may be used to determine the location of
the user's gaze in the local area. This may be used in conjunction
with a gesture detection system to allow the system to detect
various combinations of user gesture and gazes. As described in
further detail below, different combinations of user gaze and
gestures, upon detection by the controller 310, may cause the
controller 310 to transmit further instructions to devices or other
objects in the local area, or execute additional instructions in
response to these different combinations.
[0035] In some embodiments, the eye tracker 325 includes a light
source that is used to project light onto a user's eye or a portion
of the user's eye. The light source is a source of the light that
is reflected off of the eye and captured by the eye tracker
325.
[0036] The IMU 330 is an electronic device that generates IMU
tracking data based on measurement signals received from one or
more of the position sensors 335. A position sensor 335 generates
one or more measurement signals in response to motion of the NED
305. Examples of position sensors 335 include: one or more
accelerometers, one or more gyroscopes, one or more magnetometers,
another suitable type of sensor that detects motion, a type of
sensor used for error correction of the IMU 330, or some
combination thereof. The position sensors 335 may be located
external to the IMU 330, internal to the IMU 330, or some
combination thereof.
[0037] Based on the one or more measurement signals from one or
more position sensors 335, the IMU 330 generates IMU tracking data
indicating an estimated position of the NED 305 relative to an
initial position of the NED 305. For example, the position sensors
335 include multiple accelerometers to measure translational motion
(forward/back, up/down, left/right) and multiple gyroscopes to
measure rotational motion (e.g., pitch, yaw, and roll). In some
embodiments, the IMU 330 rapidly samples the measurement signals
and calculates the estimated position of the NED 305 from the
sampled data. For example, the IMU 330 integrates the measurement
signals received from the accelerometers over time to estimate a
velocity vector and integrates the velocity vector over time to
determine an estimated position of a reference point on the NED
305. Alternatively, the IMU 330 provides the sampled measurement
signals to the controller 310, which determines the IMU tracking
data. The reference point is a point that may be used to describe
the position of the NED 305. While the reference point may
generally be defined as a point in space; however, in practice the
reference point is defined as a point within the NED 305 (e.g., a
center of the IMU 330).
[0038] The depth camera assembly (DCA) 340 captures data describing
depth information of a local area surrounding some or all of the
NED 305. The data that may be captured may include information
captured from a structured light pattern projected on the local
area, stereo images, time of flight data, or depth information
captured using other depth measurement techniques. The DCA 340 can
compute the depth information using the data (e.g., based on a
captured portion of a structured light pattern), or the DCA 340 can
send this information to another device such as the controller 710
that can determine the depth information using the data from the
DCA 340.
[0039] The DCA 340 includes a light generator, an imaging device
and a controller. The light generator of the DCA 340 is configured
to illuminate the local area with illumination light in accordance
with emission instructions. The imaging device of the DCA 340
includes a lens assembly, and a detector. The lens assembly is
configured to receive light from a local area surrounding the
imaging device and to direct at least a portion of the received
light to the detector. The controller of the DCA 340 generates the
emission instructions and provides the emission instructions to the
light generator. The controller of the DCA 340 further determines
depth information for the one or more objects based in part on the
captured one or more images.
[0040] The imaging device 315 may be used to capture a
representation of the user's hands over time for use in tracking
the user's hands (e.g., by capturing multiple images per second of
the user's hand). To achieve a more accurate capture, the imaging
device 315 may be able to capture depth data of the local area or
environment. This may be achieved by various means, such as by the
use of computer vision algorithms that generate 3D data via
detection of movement in the scene, by the emission of a grid
pattern (e.g., via emission of an infrared laser grid) and
detection of depth from the variations in the reflection from the
grid pattern, from computation of time-of-flight of reflected
radiation (e.g., emitted infrared radiation that is reflected),
and/or from the user of multiple cameras (e.g., binocular vision,
stereophotogrammetry). The imaging device 315 may be positioned to
capture a large spatial area, such that all hand movements within
the spatial area are captured. In one embodiment, more than one
imaging device 315 is used to capture images of the user's hands.
As described in further detail below, the captured images of the
user's hands may be used to identify various gestures for the user.
Upon detecting these gestures in combination with the satisfaction
of other conditions, a controller may execute certain associated
actions.
[0041] In another embodiment, the imaging device 315 may also
capture images of one or more objects in the local area, and in
particular the area encompassing the field of view of a user
wearing an eyewear device that includes the NED 305. The imaging
device 315 may also capture depth data of these one or more objects
in the local area according to any of the methods described
above.
[0042] Although the imaging device 315 is illustrated in FIG. 3 as
being separate from the NED 305, in some embodiments the imaging
device is attached to the NED 305, e.g., attached to the frame 105,
and may also be part of the DCA 340.
[0043] The imaging device 315 may include one or more cameras,
imaging sensors, one or more video cameras, any other device
capable of capturing images, or some combination thereof.
Additionally, the imaging device 315 may include one or more
hardware and software filters (e.g., used to increase signal to
noise ratio). Image tracking data is communicated from the imaging
device 315 to the controller 310, and the imaging device 315
receives one or more calibration parameters from the controller 310
to adjust one or more imaging parameters (e.g., focal length,
focus, frame rate, ISO, sensor temperature, shutter speed,
aperture, etc.).
[0044] In some embodiments, the NED 305 may further comprise audio
sensors 345. The audio sensors 345 may comprise a microphone or
other type of sensor configured to capture audio data from the
local area surrounding the NED 305. The audio data may include
speech by the user of the NED system 300 or by other individuals
within the local area, other types of noises within the local area,
and/or the like. In some embodiments, the audio sensors 345 may be
located separately from the NED 305.
[0045] The controller 310 provides content to the NED 305 for
presentation to the user in accordance with information received
from the imaging device 315 or the NED 305. In the example shown in
FIG. 3, the controller 310 includes an input interface 350, an
application store 355, a tracking module 360, a gesture
identification (ID) module 365, a speech recognition module 370,
and a communication module 375, a user profile store 380, and an
execution engine 385. Some embodiments of the controller 310 have
different modules than those described herein. Similarly, the
functions further described below may be distributed among
components of the controller 310 in a different manner than is
described herein. In one embodiment, the controller 310 is a
component within the NED 305.
[0046] In one embodiment, the controller 310 includes an input
interface 350 to receive additional external input. These external
inputs may be action requests. An action request is a request to
perform a particular action. For example, an action request may be
to start or end an application or to perform a particular action
within the application. The input interface 350 may receive input
from one or more input devices. Example input devices include: a
keyboard, a mouse, a game controller, or any other suitable device
for receiving action requests. In another embodiment, the input
interface 350 receives input from one or more radio frequency (RF)
signal receivers. These may be used to receive radio signals from
RF identifiers in the local area, and in some cases to determine a
distance (based on signal strength) and position (based on
triangulation or other method) of the RF identifier. After
receiving an action request, the controller 310 performs an action
corresponding to the action request. In some embodiments, the
action performed by the controller 310 may include haptic feedback,
which may be transmitted via the input interface 350 to haptic
feedback devices. In some embodiments, the input interface 350 may
comprise a separate component in communication with the controller
310, such as a mobile device able to communicate wirelessly with
the controller 310.
[0047] The application store 355 stores one or more applications
for execution by the controller 310. An application is a group of
instructions, that when executed by a processor, generates content
for presentation to the user. Content generated by an application
may be in response to inputs received from the user via movement of
the NED 305, the input interface 350, the eye tracker 325, and/or
the audio sensors 345. Examples of applications include: gaming
applications, conferencing applications, video playback
application, or other suitable applications. In some embodiments,
the application store 355 may further store content generated by
the one or more applications. In other embodiments, generated
content may be stored in a separate content data store (not shown)
implemented as part of the controller 310 or as an external device
in communication with the controller 310.
[0048] The tracking module 360 tracks movements of the NED 305 and
the hands of the user wearing the NED 305. To track the movement of
the NED 305, the tracking module 360 uses information from the DCA
340, the imaging device 315, the one or more position sensors 335,
the IMU 330, or some combination thereof. For example, the tracking
module 360 determines a position of a reference point of the NED
305 in a mapping of a local area based on information from the NED
305. The tracking module 360 may also determine positions of the
reference point of the NED 305 using data indicating a position of
the NED 305 from the IMU 330. Additionally, in some embodiments,
the tracking module 360 may use portions of data indicating a
position or the NED 305 from the IMU 330 as well as representations
of the local area from the DCA 340 to predict a future location of
the NED 305. The tracking module 360 may provide the estimated or
predicted future position of the NED 305 to the execution engine
385.
[0049] As noted, the tracking module 360 also tracks the user's
hands, and the digits of the user's hands, in order to recognize
various poses for the user's hand. Each pose indicates a position
of a user's hand. By detecting a combination of multiple poses over
time, the tracking module 360 is able to determine a gesture for
the user's hand. These gestures may in turn translate into various
inputs to the system. For example, a movement using a single digit
in one direction may translate into a button press input in the
system.
[0050] In one embodiment, the tracking module 360 uses a deep
learning model to determine the poses of the user's hands. The deep
learning model may be a neural network, such as a convolutional
neural network, or a residual neural network. The neural network
may take as input feature data extracted from raw data from the
imaging device 315 of the hand, e.g., depth information of the
user's hand, or data regarding the location of locators on any
input device worn on the user's hands. The neural network may
output the most likely pose that the user's hands are in.
Alternatively, the neural network may output an indication of the
most likely positions of the joints of the user's hands. The joints
are positions of the user's hand, and may correspond to the actual
physical joints in the user's hand, as well as other points on the
user's hand that may be needed to sufficiently reproduce the motion
of the user's hand in a simulation.
[0051] If the neural network outputs the positions of joints, the
tracking module 360 additionally converts the joint data into a
pose, e.g., using inverse kinematics principles. For example, the
position of various joints of a user's hand, along with the natural
and known restrictions (e.g., angular, length, etc.) of joint and
bone positions of the user's hand allow the tracking module 360 to
use inverse kinematics to determine a most likely pose of the
user's hand based on the joint information. The pose data may also
include an approximate structure of the user's hand, e.g., in the
form of a skeleton, point mesh, or other format.
[0052] The neural network is trained using training data. In one
embodiment, the training data is generated from a multiple camera
array, such as multiple imaging devices 315, that captures hand
movements in different poses with different hands from different
users, and/or the locators on input devices worn by the different
hands. The ground truth for this training data indicates joint
positions and/or poses for the hands, and may be generated using
human verification.
[0053] In one embodiment, the tracking module 360 is used to track
movement of the digits of the user's hands and the hands themselves
in order to recognize various gestures and poses for the user's
hand. Each pose indicates a position of a user's hand. By detecting
a combination of multiple poses over time, the tracking module 360
is able to determine a gesture for the user's hand. These gestures
may in turn translate into various inputs to the system. For
example, a movement using a single digit in one direction may
translate into a button press input in the system.
[0054] An additional neural network may be used by the tracking
module 360 to determine a gesture from a particular series of
poses. Such a neural network may be similarly trained, using as
input data computed poses (or joints) and with output data
indicating the most likely gesture. Other methods may be used by
the tracking module 360 to determine the gesture from the pose,
such as a measurement of the distances and positions between the
digits of the hand and the positions of a series of poses in 3D
space. If these distances and positions of each pose fall within
certain thresholds, the tracking module 360 may indicate that a
particular gesture is present.
[0055] Using such a method, the tracking module 360 is able to
determine the likely poses of a user's hands, and with the
determination of the poses, the tracking module 360 may be able to
match the movement of the user's hands with predefined gestures.
These gestures may be used to indicate various actions in an
augmented reality environment.
[0056] The gesture ID module 365 identifies the gestures of a
user's hand based on the poses determined by the tracking module
360. The gesture ID module 365 may utilize a neural network to
determine a gesture from a particular series of poses. Such a
neural network may be trained using as input data computed poses
(or joints) and with output data indicating the most likely
gesture. Other methods may be used by the gesture ID module 365 to
determine the gesture from the pose, such as a measurement of the
distances and positions between the digits of the hand and the
positions of a series of poses in 3D space. If these distances and
positions of each pose fall within certain thresholds, the gesture
ID module 365 may indicate that a particular gesture is present. In
one embodiment, the gesture ID module 365 identifies a pinch
gesture of the user. A pinch gesture is formed when distal ends of
a user's index finger and thumb on one of the user's hands move
within a threshold distance of each other. When the series of poses
which form this gesture are recognized by the gesture ID module
365, the gesture ID module 365 determines that a pinch gesture is
formed. As described in further detail below, the controller 310
may execute certain actions when an identified gesture is detected
along with other conditions, such as a particular gaze direction of
the user's eyes as determined by the eye tracker 325, audio data
captured by the audio sensors 345, and/or the like.
[0057] Using such a method, the tracking module 360 is able to
determine the likely poses of a user's hands, and with the
determination of the poses, the gesture ID module 365 matches the
movement of the user's hands with predefined gestures. These
gestures may be used to indicate various actions in an artificial
reality environment.
[0058] Additional details regarding the tracking and determination
of hand positions using imaging devices and input devices are
described in U.S. application Ser. No. 15/288,453, filed Oct. 7,
2016, and U.S. application Ser. No. 15/668,418, filed Aug. 3, 2017,
both of which are incorporated by reference in their entirety.
[0059] In another embodiment, the tracking module 360 is also
configured to recognize objects in images captured by the imaging
device 315. To perform this function, the tracking module 360 may
first be trained on a large corpus of labeled object data, or be
coupled to a pre-trained image recognition system, which may be on
an online system. In the former case, the tracking module 360
includes a machine learning model (e.g., a convolutional neural
network) and is trained on a standard image-object library (e.g.,
ImageNet), or on a large set of user-provided images from an online
system. These user-provided images may include a large number of
images of objects, as well as a labeling of these objects (e.g.,
using captions, etc.). Alternatively, in the latter case, the
online system itself already includes a machine learning model
trained on the aforementioned user-provided and labeled images. For
example, the online system may already have an object recognition
system which receives images and outputs a label for each. The
model on the online system is used instead of any model on the
controller 310 to perform the object recognition in this case.
After recognizing an object, the tracking module 360 may be able to
track the location of the object in the field of view provided by
the NED 305 to the user. This may be achieved by continuously
recognizing users in each frame captured by the imaging device 315.
Once an object is recognized, the tracking module 360 can indicate
the location of the object, and the boundaries of the object (e.g.,
the pixels corresponding to the recognized object) in the captured
image. This can be translated to a location of the object in the
user's field of view provided by the NED 305 through the optical
assembly 310.
[0060] In some embodiments, the controller 310 comprises a speech
recognition module 370 configured to transcribe audio data received
via one or more audio sensors 345. In some embodiments, the audio
sensors 345 are able to record audio data corresponding to speech
by the user of the NED system 300 or other individuals within the
local area of the NED 305. The speech recognition module 370 uses
one or more audio transcription algorithms to parse to received
audio data and transcribe a transcription of the detected speech.
In some embodiments, the transcription may be used to create or
manipulate one or more content items, to display one or more
virtual objects to the user of the NED 305, mapped to an executable
command, and/or the like.
[0061] In some embodiments, the controller 310 comprises a
communication module 375 for communication with other systems, such
as other NED systems, allowing the NED system 300 to transmit and
receive commands and/or content items with other systems through
the communication module 375. For example, different users of
different NED systems may be able to access and view different
content items in their respective AR environments. The
communication module 375 may allow for the first user of a first
NED system 300 to transmit and share one or more content items with
a second user of a second NED system (which may have a structure
similar to that of the first NED system 300), allowing both users
to view the same content. In some embodiments, the communication
module 375 may transmit one or more commands to an external system
based upon inputs received from a user (e.g., through the input
interface 350, or through the performance of one or more
gestures).
[0062] In some embodiments, the NED system 300 may be used by a
plurality of different users. Each user of the NED system 300 may
be associated with an account or user profile stored in a user
profile store 385. The user profile store 385 stores user
information corresponding to each user, such as user attributes,
user preferences, content associated with the user, and/or the
like. In some embodiments, the user profile store 385 stores
permission information indicating content that may be viewed and/or
modified by different users of the NED system 300. As such,
different users of the NED system 300 may be able to view different
content through the NED 305. For example, a first user of the NED
system 300 may be able to view through an AR environment projected
by the NED 305 virtual objects depicting images associated with a
user profile of the first user, which would not be visible to a
second user of the NED system 300 associated with a different user
profile.
[0063] In some embodiments, instead of user profile information
being stored in the user profile store 380, the NED system 300 may,
using the communication module 375, access an online system such as
a social network platform in order to access user profile
information associated with different users. In some embodiments,
the NED system 300 accesses the social network platform to identify
social network connections between different users of the NED
system 300. Based upon the social network connections, the NED
system 300 may determine which types of content can be displayed to
which users (e.g., content associated with a first user may be
viewed by a second user having a "friend" relationship with the
first user, but not by a third user having no social network
connection to the first user).
[0064] In some embodiments, different users may concurrently use
the NED system 300 via different NEDs 305. For example, the
controller 310 may transmit display instructions to a first NED 305
to display a first AR environment to a first user, and to a second
NED to display a second AR environment to a second user. As such,
each of the first and second users may be able to view different
virtual objects through their respective AR environments. In some
embodiments, the NED system 300 may cause different content to be
displayed in the first or second AR environments in response to
detecting one or more predetermined gestures performed by the first
and/or second users. For example, as discussed in greater detail
below, the NED system 300 may, in response to detecting a
particular gesture by the first user, display to the second user a
visual flair emphasizing the gesture in the second AR environment.
In some embodiments, the NED system 300 may, in response to a
gesture by the first user, change the permissions of a content item
such that the content item can be viewed by the second user in the
second AR environment.
[0065] In one embodiment, the controller 310 additionally includes
an execution engine 385. The execution engine 385 executes
applications within the NED system 300 and receives position
information, acceleration information, velocity information,
predicted future positions, or some combination thereof, from the
NED 305, input interface 350, and/or the tracking module 360. Based
on the received information, the execution engine 385 determines
content to provide to the NED 305 for presentation/display to the
user. For example, if the received information indicates that the
user has looked to the left, the execution engine 385 generates
content for the NED 305 that is based off the user's movement in
the artificial reality environment. Similarly, if information
received from the tracking module 360 indicates the user's hand
makes a particular gesture, the execution engine 385 generates
content based on the identified gesture. In addition, if the
information received from the NED 305 indicates a particular gaze
of the user, the execution engine 385 may generate content based on
that gaze. This content may include an update to the optical
assembly 320 in the NED 305, such that content displayed to a user
wearing the NED 305 changes.
[0066] The execution engine 385 may also perform an action within
an application executing on the controller 310 in response to an
action request received from the input interface 350 and provides
feedback to the user that the action was performed. The provided
feedback may be visual or audible feedback via the NED 305. For
example, the execution engine 385 may receive an action from the
input interface 350 to open an application, and in response, the
execution engine 385 opens the application and presents content
from the application to the user via the NED 305.
[0067] In one embodiment, the execution engine 370 determines the
current pose of the user's hand based on raw joint data received
from the tracking module 360. In another embodiment, the execution
engine 370 detects the gestures of the user's hand based on the
pose data. The methods in which the execution engine 370 determines
poses and gestures are similar to the methods described above for
the tracking module 360. In one embodiment, the execution engine
370 performs object recognition similar to the method described
above for the tracking module 360.
[0068] In addition to determining the current pose of the user's
hand(s), the execution engine 385 may also provide output to the
optical assembly 320 in accordance with a set of display
instructions (e.g., pixel data, vector data, etc.). This output to
the electronic display of the optical assembly 320 may include a
virtual recreation (using computer graphics) of the user's hands,
as well as other objects (virtual or otherwise), such as outlines
of objects in the local area, text, graphics, other elements that
coincide with objects within a field of view of a user wearing the
NED 305, and so on. In some embodiments, the execution engine 385
provides output to the optical assembly 320 to cause the optical
assembly 320 to display one or more virtual objects to be displayed
to user at locations corresponding to the position of the user's
hands. For example, if a user performs a "pinch" gesture with their
hand, the execution engine 385 may provide instructions to the
optical assembly 320 to display a particular virtual object such
that a corner of the virtual object remains between the thumb and
index finger of the user's hand, even as the user moves their
hand.
[0069] The execution engine 385 may receive from the tracking
module 360 an indication of a tracked object. The tracking module
360 may be previously configured to recognize and track certain
objects. These objects may provide some control functionality or
may be associated with additional details or information. Upon
receiving the indication of the tracked object, the execution
engine 385 transmits display instructions to the optical assembly
320 to cause the optical assembly 320 to display various elements,
such as contextual menus, control user interface elements,
informational menus, and so on, to the user. These displayed
elements may be shown at a threshold distance from the tracked
object as viewed by the user in the augmented or artificial reality
environment presented by the NED 305.
[0070] In one embodiment, the execution engine 385 may first
recognize the recognizable objects in a local area as captured by
the imaging device 315. An object may be recognized according to a
pre-programmed recognition pattern. A recognition pattern may
include a unique identifier of the object as generated by the
object recognition system of the tracking module 360. The
recognition pattern may include the values of the output parameters
generated by the object recognition system that caused the tracking
module 360 to recognize the object (e.g., the confidence weights
generated by the object recognition system). In another embodiment,
the recognition pattern may be some other fingerprint, pattern,
identifier, or other data that is able to be used to recognize the
object again under different orientation and lighting. When the
object is encountered, the object recognition system of the
tracking module 360 may generate another identifier based on the
characteristics of the object. This identifier is compared to the
stored recognition pattern for the object, and if a match occurs,
the object is recognized as the object associated with the stored
recognition pattern.
[0071] Additionally, in some embodiments, the execution engine 385
further utilizes additional tracking indicators in the local area
to assist in the recognition of objects. As noted above, the
objects in the environment may have RF identifiers, which may be
received by the input interface 350 via one or more RF receivers.
The execution engine 385, via the signals received from the RF
receivers, and through various signal source locating mechanisms
(e.g., triangulation, time-of-flight, Doppler shift), may determine
the position of an object that has an RF identifier using the RF
signals from the object. This information may be used to augment
(e.g., adjust for error) the image based object recognition system,
or may be used in place of the image based object recognition
system (e.g., in the case where the image based object recognition
system fails or has high error/uncertainty). Other tracking
indicators, such as retroreflectors (which may respond to a
non-visible light signal from the eyewear device 100), high
contrast locators, QR codes, barcodes, identifying image patterns,
and so on, may also be used by the execution engine 385 to assist
in recognizing the object, and this information may be stored in
the recognition pattern for the object. Upon recognition of the
enhanced object, the execution engine 385 may update the display
instructions of the optical assembly 320 to present additional
simulated or virtual elements related to the enhanced object in the
artificial reality environment presented by the NED system 300. The
virtual elements may be positioned in the artificial reality
environment at a threshold distance (e.g., 1 cm) of the enhanced
object. The execution engine 385 may compute the position of the
enhanced object in 3D space and project the virtual elements on the
display such that they appear to be within the 3D space and near to
the enhanced object (within the threshold distance). Upon detection
of movement of the enhanced object, the execution engine 385 may
submit updated display instructions to move the virtual elements
based on the movement of the enhanced object.
[0072] In some embodiments, the one or more tracked objects may
correspond to other individuals within the local area, such as
users of other NEDs. The execution engine 385 may, as part of
tracking the other individuals, identify gestures performed by the
other individuals, and update the display instructions of the
optical assembly 320 based upon the identified gestures. For
example, the execution engine 385 may instruct the optical assembly
320 to display visual flair corresponding to the identified
gestures. In some embodiments where the other individuals are users
of other NED systems, the execution engine 385 may transmit or
receive content with the other NED systems, based upon identified
gestures performed by the user of the NED system 300 or by the
users of the other NED systems.
[0073] The execution engine 385, in response to receiving from the
tracking module 360 an indication of certain gestures, may execute
certain instructions related to one or more of the recognized
objects. This may allow for the user to select and manipulate
various virtual objects in the AR environment intuitively through
the use of gestures. For example, the user may select a virtual
object corresponding to a content item within the AR environment
using a pinching or grabbing gesture, cast the virtual object onto
different locations of the AR environment, share the content
associated with the virtual object with other users, and/or
manipulate the virtual object using one or more virtual interface
elements. In some embodiments, the instructions may be further
based upon an eye-gaze direction of the user (as determined by the
eye tracker 325), transcribed speech of the user or other
individual in the local area (as determined by the speech
recognition module 370), and/or some combination thereof.
Additional examples of such gestures are described with further
detail below with reference to FIGS. 4-8. FIGS. 4-5 describe
gestures for selecting virtual interface objects on a displayed
virtual menu. FIGS. 6A-6C, 7, and 8 describe gestures for sharing
content items between different users of an NED system or of
different NED systems.
[0074] Although the description below is made primarily with
reference to an artificial reality (e.g., an augmented reality)
environment, the method described herein can also be applied to a
virtual reality environment as well. In the virtual reality
environment, the user wears a head mounted device that has an
electronic display that does not allow light from the local area to
strike the user's eyes. Instead, the user's view includes only what
is displayed by the electronic display. In such a case, the eye
gaze of the user may be towards a virtual object rather than an
object in the local area, and the user may see a virtual
representation of his or her hands in the virtual reality
environment instead of his or her actual hands.
Virtual Interface in AR
[0075] In some embodiments, a near eye display (NED) system (e.g.,
NED system 300) having object recognition and gesture tracking
capabilities may allow the NED system 300 to perform manipulations
of the AR environment based upon detected user hand gestures. In
some embodiments, the NED 305 presents a user of the NED system
with a virtual interface in the AR environment, which may be
manipulated by the user using hand gestures.
[0076] FIG. 4 illustrates an example of a virtual menu that may be
displayed to a user by an eyewear device 410, according to an
embodiment. The view in FIG. 4 is of the eyewear device 410 and the
AR environment. In one embodiment, the eyewear device 410 includes
the NED system 300 described with reference to FIG. 3, and thus
includes the controller 310, optical assembly 365, and so on. In
other embodiments, the eyewear device 410 corresponds to the NED
305 illustrated in FIG. 3, and does not include the controller 310.
The optical assembly 365 of the eyewear device 410 may display the
virtual menu 430 in the artificial environment to the user. The
virtual menu 430 is represented here by a ring of circular objects,
with each circle representing a virtual menu option. However, in
other embodiments, they may take on other shapes and features. For
example, they could be spheres, cubes, text, or shaped (e.g.,
skeuomorphically) like real-world objects such as buttons, etc.
[0077] The controller 310 may further detect that the user's hand
440 is performing a touch gesture with one of the virtual menu
options. A touch gesture is detected when the controller 310
detects that the distal end of one of the user's fingers is within
a threshold distance of one of the virtual menu options (e.g., 2
mm) in the artificial reality environment. Once the touch gesture
is detected, the controller 310 may execute an option corresponding
to the selected or touched virtual menu option.
[0078] In some embodiments, the controller 310 is configured to be
able to identify different types of gestures that may be performed
by the user. As used herein, a "gesture" may refer to any series of
one or more movements or positions of the user's hand 440 able to
be identified or categorized by the controller 310. In some
embodiments, a gesture comprises a sequence of multiple motions of
the user's hand 440. In some embodiments, a gesture also
corresponds to a particular position or orientation of the user's
hand 440. The controller 310 may, in response to an identified
gesture, execute one or more operations manipulating the artificial
reality environment, based upon the specific gesture identified. As
used herein, motions or positions of the user's hand may encompass
particular motions or positions of the fingers/digits of the user's
hand.
[0079] In some embodiments, the controller 310 detects a pinch-pull
gesture with one of the virtual menu options. The pinch-pull
gesture is detected when the distal ends of the user's index finger
and thumb are within a threshold distance of each other (e.g., 5
mm) and they surround one of the virtual menu options. Furthermore,
the user's hand moves in a direction towards the eyewear device 410
(i.e., that reduces the distance between the user's hand and the
eyewear device). This type of gesture may be used for a virtual
menu option that indicates a status indicator for the user. Upon
detecting such a gesture, the controller 310 may apply the status
indicator indicated by the selected virtual menu option to the
user. For example, the virtual menu option may indicate a status
indicator of "busy." If the user pinches this option and pulls it
towards the user, the controller 310 may apply the "busy" status
indicator to the user, and transmit this status to an online
system. In some embodiments, other users with eyewear devices can
subsequently see this updated status (which may be updated in real
time) for the user. In other embodiments, the virtual menu options
are used to manipulate the AR environment displayed to the user,
such as by displaying a content item corresponding to the virtual
menu option, perform an operation corresponding to the virtual menu
option on a displayed content item, and/or the like.
[0080] FIG. 5 is a flowchart of an example process for manipulating
an artificial reality environment based upon identified gestures,
in accordance with one or more embodiments. As discussed above, in
some embodiments, a NED system (e.g., the NED system 300) may
comprise an imaging device (e.g., imaging device 315) or other type
of sensor able to capture a representation of the user's hands over
time for use in tracking the user's hands. A controller (e.g., the
controller 310) may perform one or more manipulations of the
artificial reality environment displayed to the user through the
eyewear device, based upon the tracked position and movement of the
user's hands (e.g., using the tracking module 360).
[0081] The controller determines 510 a position of a hand of the
user associated with the NED system. In some embodiments, the
position of the hand determined by the controller may comprise a
position of the user's hand relative to the user's body, a position
of the user's hand relative to their other hand, a position of the
fingers of the user's hand(s), or any combination thereof.
[0082] The controller identifies 520 a first gesture, based upon a
first set of determined hand positions. In some embodiments the
controller tracks the position of the user's hand over time to
determine a movement of the user's hand. The first gesture may
correspond to a sequence of one or more determined movements of the
user's hand. In some embodiments, the controller identifies a
pulling gesture or a pushing gesture when it detects movement of
the user's hand on a path towards or away from the user's body,
respectively.
[0083] The first gesture may be identified based upon a determined
position or pose of the user's hand(s). In an embodiment, the
controller, in response to determining that the user's hands are on
either side of their body, with their palms turned upwards,
identifies a shrug gesture.
[0084] In some embodiments, identifying the first gesture comprises
identifying that the user's hand moves to a particular location
relative to a real-world object (e.g., the user's body or some
other object within the local area) or a virtual object displayed
in the artificial reality environment during a particular portion
of the gesture, and/or achieves a particular pose at the particular
location. In some embodiments, the controller, as part of
identifying a pulling gesture, determines that the position of the
user's hand at the start of the pulling gesture corresponds to a
particular displayed virtual object.
[0085] The controller performs 530 a manipulation of the artificial
reality environment, based upon the identified gesture. In some
embodiments, the controller causes the eyewear device to display a
new virtual object as part of the artificial reality environment,
change an existing virtual object (e.g., how the virtual object is
displayed, and/or a setting of the virtual object), and/or the
like. In some embodiments, the controller, in response to the
identified gesture, changes a setting associated with the NED
system 300 (e.g., a status of the user, a setting of a piece of
data associated with the user, and/or the like), communicate with
another system (e.g., through the communication module 375) such as
a controller associated with a real-world object in the local area
or another NED system 300, and/or the like. In some embodiments,
the manipulation is based upon a position of the user's hand a
particular point in time during the gesture (e.g., in relation to a
virtual object, a real-world object, and/or the like). Examples of
specific gestures and their effects are described in greater detail
below.
Sharing Data in Artificial Reality Environments
[0086] In some embodiments, users having access to certain content
may wish to share the content with other users, or be able to
receive content shared by other users. The NED system 300 may be
used to quickly and easily share content between different users,
while restricting unauthorized users from viewing the content.
[0087] FIGS. 6A-6C illustrate an example of sharing data in an
artificial reality environment, in accordance with one or more
embodiments. FIGS. 6A-6C show images from the point of view of a
first user of a first NED system.
[0088] FIG. 6A illustrates a view where a second user has content
not viewable by the first user, in accordance with one or more
embodiments. The first user of the first NED system (e.g., NED
system 300) is able to see the second user 605 within a first AR
environment displayed by the first NED system 300. The second user
605 may be a user wearing a second NED 610 of a second NED system
and as such may be viewing a second AR environment displayed by the
second NED system through the second NED 610. In some embodiments,
virtual objects displayed by the second NED 610 to the second user
605 in the second AR environment are not visible to the first user
in the first AR environment, and vice versa.
[0089] While, for purpose of discussion, the first and second users
are referred to as being associated with first and second NED
systems, it is understood that in some embodiments, both users may
use the same NED system. For example, the NED system 300 may
comprise multiple NEDs 305 worn by different users, each user
having access to different content (e.g., through different
accounts or profiles corresponding to each user).
[0090] In some embodiments, the second user 605 may be viewing,
within the second AR environment displayed by the second NED 610, a
virtual object corresponding a content item that can be potentially
shared with the first user in the first AR environment, hereinafter
referred to as being "shareable." In some embodiments, the sharable
content item may comprise any content item viewable by the second
user 605 in the second AR environment. In some embodiments, the
second user 605 may manually specify (e.g., via one or more
gestures or through an input interface of the second NED system)
which content items can be potentially shared with the first user.
In other embodiments, whether one or more content items viewable by
the second user 605 are sharable may be based upon a relationship
between the first and second users (e.g., a social network
connection between the first and second users).
[0091] The second NED system may, in response to the second user
605 performing a particular gesture, display the virtual object to
the second user 605 containing a shareable content item viewable by
the second user 605 in the second AR environment. As illustrated in
FIG. 6A, the second user 605 performs a "viewing data on wrist"
gesture by maintaining their forearm 615 in a horizontal position
at a height above the waist of the second user 605, such that their
gaze direction (as determined by the eye tracker of the second NED
system) is within a predetermined distance of their wrist 620
(e.g., intersects a point within a predetermined range above the
wrist 620). In response, the second ND system may display a virtual
object corresponding to the content item to the second user 605
above their wrist 620. In addition, if the second NED system is a
separate NED system from the first NED system, the second NED
system may transmit an indication to the first NED system 300 used
by the first user indicating the presence of the sharable content
item being viewed by the second user 605. The indication may
comprise a message transmitted wirelessly through the
communications module of the second NED system. In some
embodiments, the second NED system transmits the indication to the
first NED system 300 directly using near-field communication,
Bluetooth, or other type of direction transmission technique. In
other embodiments, the second NED system may transmit the
indication to the first NED system 300 through an intermediate
platform (e.g., through a social network platform). For example,
the first and second NED system may both access a social network
platform through a wireless internet connection, which may be used
to transmit data between the first and second NED systems.
[0092] In response to receiving through the communication module
375 the indication that the second user 605 is viewing content that
can potentially be shared with the first user, the first NED system
300 displays to the first user in the first AR environment a
virtual object 625 indicating that the second user 605 is currently
viewing a sharable content item. The virtual object 625 may be
displayed at a location based upon the location of the second user
605, such as a predetermined distance above (i.e., adjacent to) the
second user's 605 wrist 620, and may have a shape corresponding to
the virtual object corresponding to the shareable content item that
is being viewed by the second user 605. Because the content being
viewed by the second user 605 has not yet been shared with the
first user, the virtual object 625 does not contain the actual
content being viewed by the second user 605. Instead, the virtual
object 625 may comprise a cross or other symbol indicating that the
content item being viewed by the second user 605 is not currently
being shared. As such, the first user is made aware that the second
user 605 is viewing a shareable content item, but is unable to view
the actual content item.
[0093] In some embodiments, the second NED system identifies the
first NED system based upon a pose of the second user 605. An
imaging device of the second NED system (not shown) may be
configured to identify the first user and/or the first NED 305 worn
by the first user. As such, if the second user 605 is oriented such
that the first user falls with a field of view the second NED 610
worn by the second user 605, the second NED system may transmit the
indication of shareable data being viewed by the second user 605 to
the first NED system.
[0094] In some embodiments, the second NED system identifies the
first NED system 300 in response to the NED 305 of the first NED
system worn by the first user being within a certain proximity of
the NED of the second NED 610 of the second NED system worn by the
second user 605. For example, the NED 305 of the first NED system
300 may transmit a limited-range signal, such the RF signal
described above with reference to FIG. 3, able to be detected by
the second NED 610 of the second NED system. In other embodiments,
the first user or the second user 605 may initiate a connection
(e.g., using a mobile device, not shown) between the first and
second NED systems, allowing for exchange of content items between
the first and second NED systems.
[0095] FIG. 6B illustrates a view after the second user 605 has
shared the content item with the first user, such that the content
item is visible to the first user. The second user 605 may opt to
share the content item with the first user by performing one or
more gestures. For example, the second user 605 may perform an
"open book" gesture wherein the second user 605 uses a first hand
630 to "pinch" a corner of the virtual object corresponding to the
shareable content item being viewed by the second user 605 within
the second AR environment. In some embodiments, the corner may
correspond to a specific corner of the virtual object (e.g., bottom
left corner). The "pinch" motion may correspond to the second user
605 holding at least their thumb and index finger together (or
within a threshold distance) at a location corresponding to a
corner (e.g., a vertex) of the virtual object. In response, the
second NED system may continue displaying the virtual object at a
particular location associated with the first hand 630 (e.g., such
that the corner of the virtual object corresponds to the location
of the "pinch" between the user's thumb and index finger) as long
as the second user 605 continues to hold their thumb and index
fingers together (or within the threshold distance), even if the
first hand 630 of the user moves.
[0096] The second user 605 then performs an "opening" gesture with
their second hand 635, comprising an arcing movement 640 with the
second hand 635 starting within a threshold distance of their
pinched first hand 630 and moving in a substantially horizontal
direction over an arc-shaped path, as if turning the pages of a
book. In some embodiments, the second user 605 must perform the
"opening" gesture within a threshold time of the initial "pinch"
gesture.
[0097] In response to detecting the "pinch" and "opening" gestures
described above, the second NED system may transmit the shareable
content to the first NED system 300 such that the virtual object
625 is transformed to a virtual object 645 that displays the
shareable content to the first user. As such, the first user of the
first NED system 300 is able to view the content. In embodiments
there the first and second NED systems are the same NED system, the
NED system may change a permission of the shareable content item,
such that the first user is able to view the content item within
the first AR environment. Once the first user is able to view the
content item, the shareable content item may be referred to as a
"shared content item."
[0098] In some embodiments, after the second user 056 has performed
the gestures allowing the first user to view the shared content
through the virtual object 645 (e.g., the "pinch" and "opening"
gestures), the virtual object 645 may remain "attached" to the
second user 605, in that the location of the virtual object 645
displayed in the first AR environment remains fixed relative to the
second user 605 (e.g., at the "pinch" location of the first hand
630 of the second user 605). The virtual object 645 may be
displayed at a location in the first AR environment mirroring the
location of the virtual object corresponding to the shareable
content item displayed in the second AR environment, but translated
in 3D space to match the perspective of the first NED 305. As the
second user 605 continues to pinch the virtual object in the second
AR environment, the virtual object 645 may continue to be displayed
in the first AR environment mirroring the virtual object in the
second AR environment (e.g., such that the virtual object 645
appears to be pinched by the second user 605). As such, it may be
difficult for the first user to be able to clearly view the shared
content 620, especially if the second user 605 is moving.
[0099] FIG. 6C illustrates a view where the second user 605
"passes" shared content to the first user, allowing the first user
to more clearly view the shared content. In some embodiments, the
second user 605 "passes" the shared content to the first user in
response to the first NED system 300 detecting the first user and
the second user 605 performing a "pass" gesture. The "pass" gesture
may correspond to a hand 650 of the first user moving on a path
towards a hand of the second user (e.g., first hand 630) at the
same time as the hand 630 of the second user is moving on a path
towards the hand 650 of the first user, such that the hands 650 and
630 come within a threshold distance of each other (e.g., 5 cm). In
some embodiments, the hand 630 of the second user is oriented in
the "pinch" gesture described above during an initial portion of
the movement, and the virtual object 645 continues to be displayed
at the location of the "pinch" (as described above), in order for
the "pass" gesture to be identified by the first and second NED
systems. As such, as the second user 605 moves her hand 630 towards
the hand 650 of the first user, the virtual object 645 moves along
with the hand 630. In some embodiments, the hand 650 of the first
user is also be in a certain orientation during the "pass" gesture,
such as in a "pinch" gesture or with the palm facing upwards.
[0100] In response to detection of the "pass" gesture, the first
NED system 300 displays the virtual object 645 corresponding to the
shared content at a location corresponding to the hand 650 of the
first user (e.g., at a "pinch" location of the hand 650, above the
palm of the hand 650, and/or the like) instead of to the second
user 605. In some embodiments, the virtual object 645 is only
passed to the first user if the second NED system associated with
the second user 605 also detects the "pass" gesture, and transmits
an indication to the first NED system 300 indicating that the
"pass" gesture has been detected. In some embodiments, the second
user 605 may perform the "pass" gesture without having performed
the "opening" gesture illustrated in FIG. 6B. The "pass" gesture
may function as an implied share action, causing the second NED
system to transmit the shareable content to the first NED system
300 and display the virtual object 645 at the location
corresponding to the hand 650 of the first user.
[0101] In some embodiments, once the virtual object 645 displayed
in the first AR environment by the first NED system 300 has been
passed to the first user and displayed at a location corresponding
to the first user (e.g., at the hand 650 of the first user), the
first user may perform one or more manipulations on the virtual
object 645 or the shared content corresponding to the virtual
object 645 (e.g., perform one or more gestures to change the
content, manipulate how the content is displayed, and/or the like).
In some embodiments, handing the virtual object 645 from the second
user 605 to the first user may cause the first NED system 300 to
store a copy of the shared content into a persistent memory, or
perform some other action on the content. In some embodiments, when
virtual object 645 corresponding to the shared content is passed
from the second user 605 to the first user, the second NED system
transmits a message to the first NED system 300 granting the first
user permission to manipulate the shared content. In some
embodiments, the first and second NED systems may communicate with
each other, such that manipulations performed on the shared content
by the first user in the first AR environment of the first NED
system 300 are shared and synchronized with the second AR system,
allowing the second user 605 to view the content as it is
manipulated by the first user. In other embodiments, the
manipulations performed by the first user are stored in a local
copy of the content in the first NED system (e.g., within the
application store 355 and/or a separate content data store), while
the copy of the content maintained the second NED system remains
unchanged.
[0102] FIG. 9 illustrates another example of sharing content, in
accordance with some embodiments. As illustrated in FIG. 9, a
second user 705 wearing a second NED 710 associated with a second
NED system may be visible within a field of view of a first user of
a first NED system 300. The second user 705 may be associated with
a virtual object 715 corresponding to shareable content displayed
to the first user at a location near a hand 720 of the second user
705. In some embodiments, the second user 705 may be holding a
pointing device 725 (e.g., a mobile phone, a remote control, a
laser pointer, or other device), wherein the virtual object 715 is
displayed at a location relative to the pointing device 725 (e.g.,
the virtual object 715 displayed such that a designated location of
the virtual object, such as the center, intersects a directed
pointed to by the pointing device 725). The pointing device 725 may
correspond to an input interface 350 as illustrated in FIG. 3.
[0103] In some embodiments, the first NED system further displays
in the first AR environment a pointer line 730 extending from the
pointing device 725, corresponding to a direction pointed to by the
pointing device 725. In some embodiments, the pointer line 730 is
aligned (e.g., parallel) with a longitudinal axis of the pointing
device 725 corresponding to a longest dimension of the device.
[0104] In some embodiments, the second user 705 of the second NED
system may share sharable content items with the first user using
the pointing device 725. In some embodiments, the second user 705
views through the second AR environment a virtual object
corresponding to shareable content displayed at a location
corresponding to the pointing device 720. The virtual object (not
visible to the first user) may be displayed such that the pointer
line 730 intersects a center of the virtual object.
[0105] The second NED system may transmit to the first NED system
300 an indication that the second user 705 is viewing a virtual
object corresponding to a sharable content item that can be
potentially shared with the first user. In response, the first NED
system 300 displays to the first user in the first AR environment
the virtual object 715 indicating that the second user 705 is
viewing a shareable content item. However, because the content has
not been shared with the first user, the virtual object 715 does
not display the actual content of the sharable content item. In
some embodiments, the virtual object 715 is displayed at a location
within the first AR environment equivalent to the location in the
second AR environment in which the virtual image being viewed by
the second user 705 is displayed, but translated in 3D space to
match the perspective of the first NED 305.
[0106] The second user 705 may perform a gesture using the pointing
device 725 in order to share the content with the first user. In
some embodiments, if the second user 705 points the pointing device
725 at the first user such that the pointer line 730 intersects
with the first user for at least a threshold period of time, the
second NED system may identify the first user or first NED system
300 based upon the orientation of the pointer line 730, and
transmit to the first NED system 300 the shareable content being
viewed by the second user 705. In some embodiments, the second user
705 may also "pass" the shareable content to the first user by
providing an input on an interface element of the pointing device
720 (e.g., pressing a button on the pointing device 720). In some
embodiments, the second user 705 may share the content using an
"opening" gesture (e.g., as illustrated in FIG. 6B). In some
embodiments, the second user 705 may share the content by
performing a "pass" gesture in which the second user 705 extends an
arm holding a virtual object representing the shareable content
(e.g., using a "pinch" gesture) towards the first user for at least
a threshold period of time, similar to how the second user 705
would point the pointing device 725 towards the first user.
[0107] FIG. 10 illustrates a flowchart of an example process for
sharing data between different users of NED systems, in accordance
with some embodiments. In FIG. 10, a second user of a second NED
system (e.g., the second user 605 wearing the second NED 610
illustrated in FIGS. 6A-6C) is able to access sharable content
items and may wish to share one or more of the sharable content
items with a first user of the first NED system (e.g., the NED
system 300). As discussed above, in some embodiments, the first and
second NED systems may be the same NED system that maintains
different accounts or profiles corresponding to the first and
second users, such that content viewable by the second user is not
necessarily viewable by the first user, based upon one or more
permissions associated with the content.
[0108] The second NED system determines 810 a position of a hand of
the second user of the second NED system. In addition, the second
NED system identifies 820 the first user in a local area associated
with the second NED system with which content may be shared.
[0109] The second NED system displays 830 a virtual object
associated with the shareable content to the second user (e.g.,
within a second AR environment). In some embodiments, the virtual
object is displayed at a location relative to the hand of the
second user. For example, the second NED system may determine that
the second user is performing a "viewing data on wrist" gesture as
described above, and display the virtual object at a location above
the second user's wrist.
[0110] In addition, the second NED system transmits instructions to
the first NED system 300 associated with the first user, causing
the first NED system 300 to display 840 to the first user in the
first AR environment a virtual object indicating that the second
user is currently viewing shareable content. However, because the
shareable content has not yet been shared with the first user, the
virtual object will not contain the actual shareable content. The
virtual object may be displayed at a location in the first AR
environment corresponding to the virtual object associated with the
shareable content being viewed by the second user in the second AR
environment. The virtual object may comprise one or more visual
indicators, such as a cross, question mark, or other symbol,
indicating that the shareable data being viewed by the second user
is currently not being shared, and thus not visible in the first AR
environment.
[0111] The second NED system identifies 850 a predetermined gesture
based upon one or more determined hand positions of the second
user. In some embodiments, the second NED system identifies the
second user performing a "pinch" gestured followed by an "opening"
gesture as described above. In some embodiments, the second user
performs a "pass" gesture, a "pointing" gesture using a pointing
device, and/or other gesture indicating that the second user wishes
to share the shareable content associated with the virtual image
displayed in the second AR environment with the first user. In some
embodiments, the first NED system 300 may also identify the gesture
performed by the second user, and, in response to identifying the
gesture, prepare to receive shareable content from the second NED
system.
[0112] The second NED system transmits 860 the shareable content to
the first NED system 300. The first NED system 300, in response to
receiving the content, displays 870 to the first user a virtual
object corresponding to the shareable content, such that the
shareable content is visible to the first user.
Additional Configuration Information
[0113] The foregoing description of the embodiments of the
disclosure has been presented for the purpose of illustration; it
is not intended to be exhaustive or to limit the disclosure to the
precise forms disclosed. Persons skilled in the relevant art can
appreciate that many modifications and variations are possible in
light of the above disclosure.
[0114] Some portions of this description describe the embodiments
of the disclosure in terms of algorithms and symbolic
representations of operations on information. These algorithmic
descriptions and representations are commonly used by those skilled
in the data processing arts to convey the substance of their work
effectively to others skilled in the art. These operations, while
described functionally, computationally, or logically, are
understood to be implemented by computer programs or equivalent
electrical circuits, microcode, or the like. Furthermore, it has
also proven convenient at times, to refer to these arrangements of
operations as modules, without loss of generality. The described
operations and their associated modules may be embodied in
software, firmware, hardware, or any combinations thereof.
[0115] Any of the steps, operations, or processes described herein
may be performed or implemented with one or more hardware or
software modules, alone or in combination with other devices. In
one embodiment, a software module is implemented with a computer
program product comprising a computer-readable medium containing
computer program code, which can be executed by a computer
processor for performing any or all of the steps, operations, or
processes described.
[0116] Embodiments of the disclosure may also relate to an
apparatus for performing the operations herein. This apparatus may
be specially constructed for the required purposes, and/or it may
comprise a general-purpose computing device selectively activated
or reconfigured by a computer program stored in the computer. Such
a computer program may be stored in a non-transitory, tangible
computer readable storage medium, or any type of media suitable for
storing electronic instructions, which may be coupled to a computer
system bus. Furthermore, any computing systems referred to in the
specification may include a single processor or may be
architectures employing multiple processor designs for increased
computing capability.
[0117] Embodiments of the disclosure may also relate to a product
that is produced by a computing process described herein. Such a
product may comprise information resulting from a computing
process, where the information is stored on a non-transitory,
tangible computer readable storage medium and may include any
embodiment of a computer program product or other data combination
described herein.
[0118] Finally, the language used in the specification has been
principally selected for readability and instructional purposes,
and it may not have been selected to delineate or circumscribe the
inventive subject matter. It is therefore intended that the scope
of the disclosure be limited not by this detailed description, but
rather by any claims that issue on an application based hereon.
Accordingly, the disclosure of the embodiments is intended to be
illustrative, but not limiting, of the scope of the disclosure,
which is set forth in the following claims.
* * * * *