U.S. patent application number 15/062104 was filed with the patent office on 2016-09-08 for systems and methods for augmented reality.
This patent application is currently assigned to Magic Leap, Inc.. The applicant listed for this patent is Magic Leap, Inc.. Invention is credited to Michael J. Woods.
Application Number | 20160259404 15/062104 |
Document ID | / |
Family ID | 56848278 |
Filed Date | 2016-09-08 |
United States Patent
Application |
20160259404 |
Kind Code |
A1 |
Woods; Michael J. |
September 8, 2016 |
SYSTEMS AND METHODS FOR AUGMENTED REALITY
Abstract
Configurations are disclosed for presenting virtual reality and
augmented reality experiences to users. An augmented reality
display system comprises a handheld component housing an
electromagnetic field emitter, the electromagnetic field emitter
emitting a known magnetic field, the head mounted component coupled
to one or more electromagnetic sensors that detect the magnetic
field emitted by the electromagnetic field emitter housed in the
handheld component, wherein a head pose is known, and a controller
communicatively coupled to the handheld component and the head
mounted component, the controller receiving magnetic field data
from the handheld component, and receiving sensor data from the
head mounted component, wherein the controller determining a hand
pose based at least in part on the received magnetic field data and
the received sensor data.
Inventors: |
Woods; Michael J.; (Mountain
View, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Magic Leap, Inc. |
Dania Beach |
FL |
US |
|
|
Assignee: |
Magic Leap, Inc.
Dania Beach
FL
|
Family ID: |
56848278 |
Appl. No.: |
15/062104 |
Filed: |
March 5, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62128993 |
Mar 5, 2015 |
|
|
|
62292185 |
Feb 5, 2016 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G01S 19/42 20130101;
G02B 2027/0187 20130101; G06F 3/012 20130101; G06F 3/0304 20130101;
G02B 27/017 20130101; G06T 19/006 20130101; G06F 3/017 20130101;
G06F 3/0346 20130101 |
International
Class: |
G06F 3/01 20060101
G06F003/01; G01S 19/42 20060101 G01S019/42; G06T 7/00 20060101
G06T007/00; G06T 19/00 20060101 G06T019/00; G06F 3/03 20060101
G06F003/03 |
Claims
1. An augmented reality (AR) display system, comprising: an
electromagnetic field emitter to emit a known magnetic field; an
electromagnetic sensor to measure a parameter related to a magnetic
flux at the electromagnetic sensor as a result of the emitted known
magnetic field, wherein world coordinates of the electromagnetic
sensor are known; a controller to determine pose information
relative to the electromagnetic field emitter based at least in
part on the measured parameter related to the magnetic flux at the
electromagnetic sensor; and a display system to display virtual
content to a user based at least in part on the determined pose
information relative to the electromagnetic field emitter.
2. The AR display system of claim 1, wherein the electromagnetic
field emitter resides in a mobile component of the AR display
system.
3. The AR display system of claim 2, wherein the mobile component
is a hand-held component.
4. The AR display system of claim 2, wherein the mobile component
is a totem.
5. The AR display system of claim 2, wherein the mobile component
is a head-mounted component of the AR display system.
6. The AR display system of claim 1, further comprising a
head-mounted component that houses the display system, wherein the
electromagnetic sensor is operatively coupled to the head-mounted
component.
7. The AR display system of claim 1, wherein the world coordinates
of the electromagnetic sensor is known based at least in part on
SLAM analysis performed to determine head pose information, wherein
the electromagnetic sensor is operatively coupled to a head-mounted
component that houses the display system.
8. The AR display system of claim 7, further comprising one or more
cameras operatively coupled to the head-mounted component, and
wherein the SLAM analysis is performed based at least on data
captured by the one or more cameras.
9. The AR display system of claim 1, wherein the electromagnetic
sensors comprise one or more inertial measurement units (IMUs).
10. The AR display system of claim 1, wherein the pose information
corresponds to at least a position and orientation of the
electromagnetic field emitter relative to the world.
11. The AR display system of claim 1, wherein the pose information
is analyzed to determine world coordinates corresponding to the
electromagnetic field emitter.
12. The AR display system of claim 1, wherein the controller
detects an interaction with one or more virtual contents based at
least in part on the pose information corresponding to the
electromagnetic field emitter.
13. The AR display system of claim 12, wherein the display system
displays virtual content to the user based at least in part on the
detected interaction.
14. The AR display system of claim 1, wherein the electromagnetic
sensor comprises at least three coils to measure magnetic flux in
three directions.
15. The AR display system of claim 14, wherein the at least three
coils are housed together at substantially the same location, the
electromagnetic sensor being coupled to a head-mounted component of
the AR display system.
16. The AR display system of claim 14, wherein the at least three
coils are housed at different locations of the head-mounted
component of the AR display system.
17. The AR display system of claim 1, further comprising a control
and quick release module to decouple the magnetic field emitted by
the electromagnetic field emitter.
18. The AR display system of claim 1, further comprising additional
localization resources to determine the world coordinates of the
electromagnetic field emitter.
19. The AR display system of claim 18, wherein the additional
localization resources comprises a GPS receiver.
20. The AR display system of claim 18, wherein the additional
localization resources comprises a beacon.
21. The AR display system of claim 1, wherein the electromagnetic
sensor comprises a non-solid ferrite cube.
22. The AR display system of claim 1, wherein the electromagnetic
sensor comprises a stack of ferrite disks.
23. The AR display system of claim 1, wherein the electromagnetic
sensor comprises a plurality of ferrite rods each having a polymer
coating.
24. The AR display system of claim 1, wherein the electromagnetic
sensor comprises a time division multiplexing switch.
25. A method to display augmented reality, comprising: emitting,
through an electromagnetic field emitter, a known magnetic field;
measuring, through an electromagnetic sensor, a parameter related
to a magnetic flux at the electromagnetic sensor as a result of the
emitted known magnetic field, wherein world coordinates of the
electromagnetic sensor are known; determining pose information
relative to the electromagnetic field emitter based at least in
part on the measured parameter related to the magnetic flux at the
electromagnetic sensor; and displaying virtual content to a user
based at least in part on the determined pose information relative
to the electromagnetic field emitter.
26. The method of claim 25, wherein the electromagnetic field
emitter resides in a mobile component of the AR display system.
27. The method of claim 26, wherein the mobile component is a
hand-held component.
28. The method of claim 26, wherein the mobile component is a
totem.
29. The method of claim 26, wherein the mobile component is a
head-mounted component of the AR display system.
30. The method of claim 25, further comprising housing the display
system in a head-mounted component, wherein the electromagnetic
sensor is operatively coupled to the head-mounted component.
31. The method of claim 25, wherein the world coordinates of the
electromagnetic sensor is known based at least in part on SLAM
analysis performed to determine head pose information, wherein the
electromagnetic sensor is operatively coupled to a head-mounted
component that houses the display system.
32. The method of claim 31, capturing image data through one or
more cameras that are operatively coupled to the head-mounted
component, and wherein the SLAM analysis is performed based at
least on data captured by the one or more cameras.
33. The method of claim 25, wherein the electromagnetic sensors
comprise one or more inertial measurement units (IMUs).
34. The method of claim 25, wherein the pose information
corresponds to at least a position and orientation of the
electromagnetic field emitter relative to the world.
35. The method of claim 25, wherein the pose information is
analyzed to determine world coordinates corresponding to the
electromagnetic field emitter.
36. The method of claim 25, further comprising detecting an
interaction with one or more virtual contents based at least in
part on the pose information corresponding to the electromagnetic
field emitter.
37. The method of claim 36, further comprising displaying virtual
content to the user based at least in part on the detected
interaction.
38. The method of claim 25, wherein the electromagnetic sensor
comprises at least three coils to measure magnetic flux in three
directions.
39. The method of claim 38, wherein the at least three coils are
housed together at substantially the same location, the
electromagnetic sensor being coupled to a head-mounted component of
the AR display system.
40. The method of claim 38, wherein the at least three coils are
housed at different locations of the head-mounted component of the
AR display system.
41. The method of claim 25, further comprising decoupling the
magnetic field emitted by the electromagnetic field emitter through
a control and quick release module.
42. The method of claim 25, further comprising determining the
world coordinates of the electromagnetic field emitter through
additional localization resources.
43. The method of claim 42, wherein the additional localization
resources comprises a GPS receiver.
44. The method of claim 42, wherein the additional localization
resources comprises a beacon.
45. The method of claim 25, wherein the electromagnetic sensor
comprises a non-solid ferrite cube.
46. The method of claim 25, wherein the electromagnetic sensor
comprises a stack of ferrite disks.
47. The method of claim 25, wherein the electromagnetic sensor
comprises a plurality of ferrite rods each having a polymer
coating.
48. The method of claim 25, wherein the electromagnetic sensor
comprises a time division multiplexing switch.
49. An augmented reality display system, comprising: a handheld
component housing an electromagnetic field emitter, the
electromagnetic field emitter emitting a known magnetic field; a
head mounted component having a display system that displays
virtual content to a user, the head mounted component coupled to
one or more electromagnetic sensors that detect the magnetic field
emitted by the electromagnetic field emitter housed in the handheld
component, wherein a head pose is known; and a controller
communicatively coupled to the handheld component and the head
mounted component, the controller receiving magnetic field data
from the handheld component, and receiving sensor data from the
head mounted component, wherein the controller determines a hand
pose based at least in part on the received magnetic field data and
the received sensor data, wherein the display system modifies the
virtual content displayed to the user based at least in part on the
determined hand pose.
50. The AR display system of claim 49, wherein the handheld
component is mobile.
51. The AR display system of claim 50, wherein the handheld
component is a totem.
52. The AR display system of claim 50, wherein the handheld
component is a gaming component.
53. The AR display system of claim 49, wherein the head pose is
known based at least in part on SLAM analysis.
54. The AR display system of claim 53, further comprising one or
more cameras operatively coupled to the head-mounted component, and
wherein the SLAM analysis is performed based at least on data
captured by the one or more cameras.
55. The AR display system of claim 49, wherein the electromagnetic
sensor comprises one or more inertial measurement units (IMUs).
56. The AR display system of claim 49, wherein the head pose
corresponds to at least a position and orientation of the
electromagnetic sensor relative to the world.
57. The AR display system of claim 49, wherein the hand pose is
analyzed to determine world coordinates corresponding to the
handheld component.
58. The AR display system of claim 49, wherein the controller
detects an interaction with one or more virtual contents based at
least in part on the determined hand pose.
59. The AR display system of claim 58, wherein the display system
displays the virtual content to the user based at least in part on
the detected interaction.
60. The AR display system of claim 49, wherein the electromagnetic
sensor comprises at least three coils to measure magnetic flux in
three directions.
61. The AR display system of claim 60, wherein the at least three
coils are housed together at substantially the same location.
62. The AR display system of claim 60, wherein the at least three
coils are housed at different locations of the head-mounted
component.
63. The AR display system of claim 49, further comprising a control
and quick release module to decouple the magnetic field emitted by
the electromagnetic field emitter.
64. The AR display system of claim 49, further comprising
additional localization resources to determine the hand pose.
65. The AR display system of claim 64, wherein the additional
localization resources comprises a GPS receiver.
66. The AR display system of claim 64, wherein the additional
localization resources comprises a beacon.
67. The AR display system of claim 49, wherein at least one of the
one or more electromagnetic sensors comprises a non-solid ferrite
cube.
68. The AR display system of claim 49, wherein at least one of the
one or more electromagnetic sensors comprises a stack of ferrite
disks.
69. The AR display system of claim 49, wherein at least one of the
one or more electromagnetic sensors comprises a plurality of
ferrite rods each having a polymer coating.
70. The AR display system of claim 49, wherein at least one of the
one or more electromagnetic sensors comprises a time division
multiplexing switch.
Description
CROSS-REFERENCE TO RELATED APPLICATION(S)
[0001] This application claims priority to U.S. Provisional
Application Ser. No. 62/128,993 filed on Mar. 5, 2015 entitled
"ELECTROMAGNETIC TRACKING SYSTEM AND METHOD FOR AUGMENTED REALITY,"
under attorney docket number ML.30031.00, and U.S. Provisional
Application Ser. No. 62/292,185 filed on Feb. 5, 2016 entitled
"SYSTEMS AND METHODS FOR AUGMENTED REALITY," under attorney docket
number ML.30062.00. The content of the aforementioned patent
applications is hereby expressly incorporated by reference in its
entirety for all purposes.
BACKGROUND
[0002] Modern computing and display technologies have facilitated
the development of systems for so called "virtual reality" or
"augmented reality" experiences, wherein digitally reproduced
images or portions thereof are presented to a user in a manner
wherein they seem to be, or may be perceived as, real. A virtual
reality, or "VR", scenario typically involves presentation of
digital or virtual image information without transparency to other
actual real-world visual input. An augmented reality, or "AR",
scenario typically involves presentation of digital or virtual
image information as an augmentation to visualization of the actual
world around the user.
[0003] For example, referring to FIG. 1, an augmented reality scene
4 is depicted wherein a user of an AR technology sees a real-world
park-like setting 6 featuring people, trees, buildings in the
background, and a concrete platform 1120. In addition to these
items, the user of the AR technology may also perceive a robot
statue 1110 standing upon the real-world platform 1120, and a
cartoon-like avatar character 2 flying around the park. Of course,
the virtual elements 2 and 1110 do not exist in the real world, but
the user perceives these virtual objects as being part of, and as
interacting with objects of the real world (e.g., 6, 1120, etc.).
It should be appreciated, the human visual perception system is
very complex, and producing such an AR scene that facilitates a
comfortable, natural-feeling, rich presentation of virtual image
elements amongst other virtual or real-world imagery elements is
challenging.
[0004] For instance, head-worn AR displays (e.g., helmet-mounted
displays, or smart glasses) may be coupled to a user's head, and
thus may move when the user's head moves. If the user's head
motions are detected by the display system, the data being
displayed can be updated to take the change in head pose into
account. The head pose may be utilized to appropriately render
virtual content to the user. Thus any small variation may disrupt
and/or diminish the delivery or timing of virtual content that is
delivered to the user's AR display.
[0005] As an example, if a user wearing a head-worn display views a
virtual representation of a three-dimensional (3-D) object on the
display and walks around the area where the 3-D object appears,
that 3-D object can be re-rendered for each viewpoint, giving the
user the perception that he or she is walking around an object that
occupies real space. If the head-worn display is used to present
multiple objects within a virtual space (for instance, a rich
virtual world), measurements of head pose (i.e., the location and
orientation of the user's head) can be used to re-render the scene
to match the user's dynamically changing head location and
orientation, and provide an increased sense of immersion in the
virtual space. Similarly, when a user of AR technology is
interacting with the virtual world, he or she may use an object or
his/her hand to point to objects or to select options. In order for
this interaction to occur, localization of the object or the user's
hand to an accurate degree is also important. Thus, both head pose,
and "hand pose" are both crucial, and localization techniques must
be used in order to accurately depict virtual content to the
user.
[0006] In AR systems, detection and/or calculation of head pose
and/or hand pose can facilitate the AR display system to render
virtual objects such that they appear to occupy a space in the real
world in a manner that is congruent to the objects of the real
world. Presenting an AR scene realistically such that the virtual
content does not seem jarring/disorienting in relation to one or
more real objects improves the user's enjoyment of the AR
experience. In addition, detection of the position and/or
orientation of a real object, such as a handheld device (which also
may be referred to as a "totem"), haptic device, or other real
physical object, in relation to the user's head or AR system may
also facilitate the display system in presenting display
information to the user to enable the user to interact with certain
aspects of the AR system efficiently.
[0007] It should be appreciated that in AR applications, placement
of virtual objects in spatial relation to physical objects (e.g.,
presented to appear spatially proximate a physical object in two or
three dimensions) is a non-trivial problem. For example, head
movement may significantly complicate placement of virtual objects
in a view of an ambient environment. This may be true whether the
view is captured as an image of the ambient environment and then
projected or displayed to the end user, or whether the end user
perceives the view of the ambient environment directly. For
instance, head movement may cause the field of view of the user to
change. This may, in turn, require an update to where various
virtual objects are displayed in the field of view of the end user.
Similarly, movement of the hand (in case of a handheld object) when
used to interact with the system provides the same challenge. These
movements may be fast and typically need to be accurately detected
and localized at a high refresh rate and low latency.
[0008] Additionally, head and/or hand movements may occur at a
large variety of ranges and speeds. The speed may vary not only
between different types of head movements, but within or across the
range of a single movement. For instance, speed of head movement
may initially increase (e.g., linearly or otherwise) from a
starting point, and may decrease as an ending point is reached,
obtaining a maximum speed somewhere between the starting and ending
points of the head movement. Rapid movements may even exceed the
ability of the particular display or projection technology to
render images that appear uniform and/or as smooth motion to the
end user.
[0009] Head or hand tracking accuracy and latency (i.e., the
elapsed time between when the user moves his or her head/hand and
the time when the image gets updated and displayed to the user)
have been challenges for VR and AR systems. Especially for display
systems that fill a substantial portion of the user's visual field
with virtual elements, it is critical that the accuracy of tracking
is high and that the overall system latency is very low from the
first detection of motion to the updating of the light that is
delivered by the display to the user's visual system. If the
latency is high, the system can create a mismatch between the
user's vestibular and visual sensory systems, and generate a user
perception scenario that can lead to motion sickness or simulator
sickness. If the system latency is high, the apparent location of
virtual objects may appear unstable during rapid head motions.
[0010] In addition to head-worn display systems, other display
systems can also benefit from accurate and low-latency head pose
detection. These may include head-tracked display systems in which
the display is not worn on the user's body, but is, e.g., mounted
on a wall or other surface. The head-tracked display may act like a
window onto a scene, and as a user moves his head relative to the
"window" the scene is re-rendered to match the user's changing
viewpoint. Other systems may include a head-worn projection system,
in which a head-worn display projects light onto the real
world.
[0011] Additionally, in order to provide a realistic AR experience,
AR systems may be designed to be interactive with the user. For
example, multiple users may play a ball game with a virtual ball
and/or other virtual objects. One user may "catch" the virtual
ball, and throw the ball back to another user. In another
embodiment, a first user may be provided with a totem (e.g., a
physical "bat" communicatively coupled to the AR system) to hit the
virtual ball. In other embodiments, a virtual user interface may be
presented to the AR user to allow the user to select one of many
options. The user may use totems, haptic devices, wearable
components, or simply touch the virtual screen to interact with the
system.
[0012] Detecting a pose and an orientation of the user (e.g., the
user's head and hand), and detecting a physical location of real
objects in space may enable the AR system to display virtual
content in an effective and enjoyable manner. However, such
accurate detection of head and hand pose may be difficult to
achieve. In other words, the AR system must recognize a physical
location of a real object (e.g., user's head, totem, haptic device,
wearable component, user's hand, etc.) and correlate the physical
coordinates of the real object to virtual coordinates corresponding
to one or more virtual objects being displayed to the user. This
process can be improved by highly accurate sensors and sensor
recognition systems that track a position and orientation of one or
more objects at rapid rates. Current approaches do not perform
localization at satisfactory speed or precision standards.
[0013] There, thus, is a need for a better localization system in
the context of AR and VR devices.
SUMMARY
[0014] Embodiments of the present invention are directed to
devices, systems and methods for facilitating virtual reality
and/or augmented reality interaction for one or more users.
[0015] In one aspect, an augmented reality (AR) display system
comprises an electromagnetic field emitter to emit a known magnetic
field, an electromagnetic sensor to measure a parameter related to
a magnetic flux measured at the electromagnetic sensor as a result
of the emitted known magnetic field, wherein world coordinates of
the electromagnetic sensor are known, a controller to determine
pose information relative to the electromagnetic field emitter
based at least in part on the measure parameter related to the
magnetic flux measured at the electromagnetic sensor, and a display
system to display virtual content to a user based at least in part
on the determined pose information relative to the electromagnetic
field emitter.
[0016] In one or more embodiments, the electromagnetic field
emitter resides in a mobile component of the AR display system. In
one or more embodiments, the mobile component is a hand-held
component. In one or more embodiments, the mobile component is a
totem.
[0017] In one or more embodiments, the mobile component is a
head-mounted component of the AR display system. In one or more
embodiments, the AR display system further comprises a head-mounted
component that houses the display system, wherein the
electromagnetic sensor is operatively coupled to the head-mounted
component. In one or more embodiments, the world coordinates of the
electromagnetic sensor is known based at least in part on SLAM
analysis performed to determine head pose information, wherein the
electromagnetic sensor is operatively coupled to a head-mounted
component that houses the display system.
[0018] In one or more embodiments, the AR display further comprises
one or more cameras operatively coupled to the head-mounted
component, and wherein the SLAM analysis is performed based at
least on data captured by the one or more cameras. In one or more
embodiments, the electromagnetic sensors comprise one or more
inertial measurement units (IMUs).
[0019] In one or more embodiments, the pose information corresponds
to at least a position and orientation of the electromagnetic field
emitter relative to the world. In one or more embodiments, the pose
information is analyzed to determine world coordinates
corresponding to the electromagnetic field emitter. In one or more
embodiments, the controller detects an interaction with one or more
virtual contents based at least in part on the pose information
corresponding to the electromagnetic field emitter.
[0020] In one or more embodiments, the display system displays
virtual content to the user based at least in part on the detected
interaction. In one or more embodiments, the electromagnetic sensor
comprises at least three coils to measure magnetic flux in three
directions. In one or more embodiments, the at least three coils
are housed together at substantially the same location, the
electromagnetic sensor being coupled to a head-mounted component of
the AR display system.
[0021] In one or more embodiments, the at least three coils are
housed at different locations of the head-mounted component of the
AR display system.
[0022] The AR display system of claim 1, further comprising a
control and quick release module to decouple the magnetic field
emitted by the electromagnetic field emitter. In one or more
embodiments, the AR display system further comprises additional
localization resources to determine the world coordinates of the
electromagnetic field emitter. In one or more embodiments, the
additional localization resources comprises a GPS receiver. In one
or more embodiments, the additional localization resources
comprises a beacon.
[0023] In one or more embodiments, the electromagnetic sensor
comprises a non-solid ferrite cube. In one or more embodiments, the
electromagnetic sensor comprises a stack of ferrite disks. In one
or more embodiments, the electromagnetic sensor comprises a
plurality of ferrite rods each having a polymer coating. In one or
more embodiments, the electromagnetic sensor comprises a time
division multiplexing switch.
[0024] In another aspect, a method to display augmented reality
comprises emitting, through an electromagnetic field emitter, a
known magnetic field, measuring, through an electromagnetic sensor,
a parameter related to a magnetic flux measured at the
electromagnetic sensor as a result of the emitted known magnetic
field, wherein world coordinates of the electromagnetic sensor are
known, determining pose information relative to the electromagnetic
field emitter based at least in part on the measured parameter
related to the magnetic flux measured at the electromagnetic
sensor, and displaying virtual content to a user based at least in
part on the determined pose information relative to the
electromagnetic field emitter.
[0025] In one or more embodiments, the electromagnetic field
emitter resides in a mobile component of the AR display system. In
one or more embodiments, the mobile component is a hand-held
component. In one or more embodiments, the mobile component is a
totem. In one or more embodiments, the mobile component is a
head-mounted component of the AR display system.
[0026] In one or more embodiments, the method further comprises
housing the display system in a head-mounted component, wherein the
electromagnetic sensor is operatively coupled to the head-mounted
component. In one or more embodiments, the world coordinates of the
electromagnetic sensor is known based at least in part on SLAM
analysis performed to determine head pose information, wherein the
electromagnetic sensor is operatively coupled to a head-mounted
component that houses the display system.
[0027] In one or more embodiments, further comprises capturing
image data through one or more cameras that are operatively coupled
to the head-mounted component, and wherein the SLAM analysis is
performed based at least on data captured by the one or more
cameras. In one or more embodiments, the electromagnetic sensors
comprise one or more inertial measurement units (IMUs).
[0028] In one or more embodiments, the pose information corresponds
to at least a position and orientation of the electromagnetic field
emitter relative to the world. In one or more embodiments, the pose
information is analyzed to determine world coordinates
corresponding to the electromagnetic field emitter. In one or more
embodiments, the method further comprises detecting an interaction
with one or more virtual contents based at least in part on the
pose information corresponding to the electromagnetic field
emitter.
[0029] In one or more embodiments, the method further comprises
displaying virtual content to the user based at least in part on
the detected interaction. In one or more embodiments, the
electromagnetic sensor comprises at least three coils to measure
magnetic flux in three directions. In one or more embodiments, the
at least three coils are housed together at substantially the same
location, the electromagnetic sensor being coupled to a
head-mounted component of the AR display system. In one or more
embodiments, the at least three coils are housed at different
locations of the head-mounted component of the AR display
system.
[0030] In one or more embodiments, the method further comprises
decoupling the magnetic field emitted by the electromagnetic field
emitter through a control and quick release module. In one or more
embodiments, the method further comprises determining the world
coordinates of the electromagnetic field emitter through additional
localization resources. In one or more embodiments, the additional
localization resources comprises a GPS receiver. In one or more
embodiments, the additional localization resources comprises a
beacon.
[0031] In yet another aspect, an augmented reality display system,
comprises a handheld component housing an electromagnetic field
emitter, the electromagnetic field emitter emitting a known
magnetic field, a head mounted component having a display system
that displays virtual content to a user, the head mounted component
coupled to one or more electromagnetic sensors that detect the
magnetic field emitted by the electromagnetic field emitter housed
in the handheld component, wherein a head pose is known, and a
controller communicatively coupled to the handheld component and
the head mounted component, the controller receiving magnetic field
data from the handheld component, and receiving sensor data from
the head mounted component, wherein the controller determines a
hand pose based at least in part on the received magnetic field
data and the received sensor data, wherein the display system
modifies the virtual content displayed to the user based at least
in part on the determined hand pose.
[0032] In one or more embodiments, the handheld component is
mobile. In one or more embodiments, the handheld component is a
totem. In one or more embodiments, the handheld component is a
gaming component. In one or more embodiments, the head pose is
known based at least in part on SLAM analysis.
[0033] In one or more embodiments, the AR display system further
comprises one or more cameras operatively coupled to the
head-mounted component, and wherein the SLAM analysis is performed
based at least on data captured by the one or more cameras. In one
or more embodiments, the electromagnetic sensor comprises one or
more inertial measurement units (IMUs).
[0034] In one or more embodiments, the head pose corresponds to at
least a position and orientation of the electromagnetic sensor
relative to the world. In one or more embodiments, the hand pose is
analyzed to determine world coordinates corresponding to the
handheld component. In one or more embodiments, the controller
detects an interaction with one or more virtual contents based at
least in part on the determined hand pose.
[0035] In one or more embodiments, the display system displays the
virtual content to the user based at least in part on the detected
interaction. In one or more embodiments, the electromagnetic sensor
comprises at least three coils to measure magnetic flux in three
directions. In one or more embodiments, the at least three coils
are housed together at substantially the same location. In one or
more embodiments, the at least three coils are housed at different
locations of the head-mounted component.
[0036] In one or more embodiments, the AR display system further
comprises a control and quick release module to decouple the
magnetic field emitted by the electromagnetic field emitter. In one
or more embodiments, the AR display system further comprises
additional localization resources to determine the hand pose. In
one or more embodiments, the additional localization resources
comprises a GPS receiver. In one or more embodiments, the
additional localization resources comprises a beacon.
[0037] Additional and other objects, features, and advantages of
the invention are described in the detail description, figures and
claims.
[0038] Additional and other objects, features, and advantages of
the invention are described in the detail description, figures and
claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The drawings illustrate the design and utility of various
embodiments of the present invention. It should be noted that the
figures are not drawn to scale and that elements of similar
structures or functions are represented by like reference numerals
throughout the figures. In order to better appreciate how to obtain
the above-recited and other advantages and objects of various
embodiments of the invention, a more detailed description of the
present inventions briefly described above will be rendered by
reference to specific embodiments thereof, which are illustrated in
the accompanying drawings. Understanding that these drawings depict
only typical embodiments of the invention and are not therefore to
be considered limiting of its scope, the invention will be
described and explained with additional specificity and detail
through the use of the accompanying drawings in which:
[0040] FIG. 1 illustrates a plan view of an AR scene displayed to a
user of an AR system according to one embodiment.
[0041] FIGS. 2A-2D illustrate various embodiments of wearable AR
devices
[0042] FIG. 3 illustrates an example embodiment of a user of a
wearable AR device interacting with one or more cloud servers of
the AR system.
[0043] FIG. 4 illustrates an example embodiment of an
electromagnetic tracking system.
[0044] FIG. 5 illustrates an example method of determining a
position and orientation of sensors, according to one example
embodiment.
[0045] FIG. 6 illustrates an example diagram of utilizing an
electromagnetic tracking system to determine head pose.
[0046] FIG. 7 illustrates an example method of delivering virtual
content to a user based on detected head pose.
[0047] FIG. 8 illustrates a schematic view of various components of
an AR system according to one embodiment having an electromagnetic
transmitter and electromagnetic sensors.
[0048] FIGS. 9A-9F illustrate various embodiments of the control
and quick release module.
[0049] FIG. 10 illustrates one simplified embodiment of the AR
device.
[0050] FIGS. 11A and 11B illustrate various embodiments of
placement of the electromagnetic sensors on the head-mounted AR
system.
[0051] FIGS. 12A-12E illustrate various embodiments of a ferrite
cube to be coupled to the electromagnetic sensors.
[0052] FIG. 13A-13C illustrate various embodiments of circuitry of
the electromagnetic sensors.
[0053] FIG. 14 illustrates an example method of using an
electromagnetic tracking system to detect head and hand pose.
[0054] FIG. 15 illustrates another example method of using an
electromagnetic tracking system to detect head and hand pose.
DETAILED DESCRIPTION
[0055] Referring to FIGS. 2A-2D, some general componentry options
are illustrated. In the portions of the detailed description which
follow the discussion of FIGS. 2A-2D, various systems, subsystems,
and components are presented for addressing the objectives of
providing a high-quality, comfortably-perceived display system for
human VR and/or AR.
[0056] As shown in FIG. 2A, an AR system user 60 is depicted
wearing a head mounted component 58 featuring a frame 64 structure
coupled to a display system 62 positioned in front of the eyes of
the user. A speaker 66 is coupled to the frame 64 in the depicted
configuration and positioned adjacent the ear canal of the user (in
one embodiment, another speaker, not shown, is positioned adjacent
the other ear canal of the user to provide for stereo/shapeable
sound control). The display 62 may be operatively coupled 68, such
as by a wired lead or wireless connectivity, to a local processing
and data module 70 which may be mounted in a variety of
configurations, such as fixedly attached to the frame 64, fixedly
attached to a helmet or hat 80 as shown in the embodiment of FIG.
2B, embedded in headphones, removably attached to the torso 82 of
the user 60 in a backpack-style configuration as shown in the
embodiment of FIG. 2C, or removably attached to the hip 84 of the
user 60 in a belt-coupling style configuration as shown in the
embodiment of FIG. 2D.
[0057] The local processing and data module 70 may comprise a
power-efficient processor or controller, as well as digital memory,
such as flash memory, both of which may be utilized to assist in
the processing, caching, and storage of data, which may be (a)
captured from sensors which may be operatively coupled to the frame
64, such as image capture devices (such as cameras), microphones,
inertial measurement units, accelerometers, compasses, GPS units,
radio devices, and/or gyros; and/or (b) acquired and/or processed
using the remote processing module 72 and/or remote data repository
74, possibly for passage to the display 62 after such processing or
retrieval. The local processing and data module 70 may be
operatively coupled (76, 78), such as via a wired or wireless
communication links, to the remote processing module 72 and remote
data repository 74 such that these remote modules (72, 74) are
operatively coupled to each other and available as resources to the
local processing and data module 70.
[0058] In one embodiment, the remote processing module 72 may
comprise one or more relatively powerful processors or controllers
configured to analyze and process data and/or image information. In
one embodiment, the remote data repository 74 may comprise a
relatively large-scale digital data storage facility, which may be
available through the internet or other networking configuration in
a "cloud" resource configuration. In one embodiment, all data may
be stored and all computation may be performed in the local
processing and data module, allowing fully autonomous use from any
remote modules.
[0059] Referring now to FIG. 3, a schematic illustrates
coordination between the cloud computing assets 46 and local
processing assets, which may, for example reside in head mounted
componentry 58 coupled to the user's head 120 and a local
processing and data module 70, coupled to the user's belt 308;
therefore the component 70 may also be termed a "belt pack" 70, as
shown in FIG. 3. In one embodiment, the cloud 46 assets, such as
one or more cloud server systems 110 are operatively coupled 115,
such as via wired or wireless networking (wireless being preferred
for mobility, wired being preferred for certain high-bandwidth or
high-data-volume transfers that may be desired), directly to (40,
42) one or both of the local computing assets, such as processor
and memory configurations, coupled to the user's head 120 and belt
308 as described above. These computing assets local to the user
may be operatively coupled to each other as well, via wired and/or
wireless connectivity configurations 44, such as the wired coupling
68 discussed below in reference to FIG. 8. In one embodiment, to
maintain a low-inertia and small-size subsystem mounted to the
user's head 120, primary transfer between the user and the cloud 46
may be via the link between the subsystem mounted at the belt 308
and the cloud, with the head mounted subsystem 120 primarily
data-tethered to the belt-based subsystem 308 using wireless
connectivity, such as ultra-wideband ("UWB") connectivity, as is
currently employed, for example, in personal computing peripheral
connectivity applications.
[0060] With efficient local and remote processing coordination, and
an appropriate display device for a user, such as the user
interface or user display system 62 shown in FIG. 2A, or variations
thereof, aspects of one world pertinent to a user's current actual
or virtual location may be transferred or "passed" to the user and
updated in an efficient fashion. In other words, a map of the world
may be continually updated at a storage location which may
partially reside on the user's AR system and partially reside in
the cloud resources. The map (also referred to as a "passable world
model") may be a large database comprising raster imagery, 3-D and
2-D points, parametric information and other information about the
real world. As more and more AR users continually capture
information about their real environment (e.g., through cameras,
sensors, IMUs, etc.), the map becomes more and more accurate and
complete.
[0061] With a configuration as described above, wherein there is
one "model" of the world that can reside on cloud computing
resources and be distributed from the cloud server, such a world
can be "passable" to one or more users in a relatively low
bandwidth form. This may be preferable to transferring real-time
video data or similar complex information from one AR system to
another. The augmented experience of the person standing near the
statue (i.e., as shown in FIG. 1) may be informed by the
cloud-based world model, a subset of which may be passed down to
the person's local display device to complete the view. A person
sitting at a remote display device (e.g., a personal computer
sitting on a desk), can efficiently download that same section of
information from the cloud and have it rendered on the personal
computer display. In yet another embodiment, yet another user may
be present in real-time at the park, and may take a walk in that
park, with a friend (e.g., the person at the personal computer)
joining the user through a shared AR and/or VR experience. In order
to render the park scene to the friend, the AR system may detect a
location of the street, a location of the trees in the park, a
location of the statue, etc. This location may be uploaded to the
passable world model in the cloud, and the friend (at the personal
computer) can download the portion of the passable world from the
cloud, and then start "walking along" with the AR user in the park.
Of course, in some embodiments, the friend may be rendered as an
avatar in the passable world model to the AR user in the park such
that the AR user can walk alongside the virtual friend in the
park.
[0062] More particularly, in order to capture details of the world
such that it can be passed on to the cloud (and subsequently to
other AR users) 3-D points pertaining to various objects may be
captured from the environment, and the pose (i.e., vector and/or
origin position information relative to the world) of the cameras
that capture those images or points may be determined. These 3-D
points may be "tagged", or associated, with this pose information.
It should be appreciated that there may be a large number of AR
systems capturing the same points in any given environment. For
example, points captured by a second camera (of a second AR system)
may be utilized to determine the head pose of the second camera. In
other words, one can orient and/or localize a second camera based
upon comparisons with tagged images from a first camera. Then, this
information may be utilized to extract textures, make maps, and
create one or more virtual copies of the real world.
[0063] In one or more embodiments, the AR system can be utilized to
capture both 3-D points and the 2-D images that produced the
points. As discussed above, these points and images may be sent out
to the cloud storage and processing resource (e.g., the servers 110
of FIG. 3), in some embodiments. In other embodiments, this
information may be cached locally with embedded pose information
(i.e., the tagged images) such that tagged 2-D images are sent to
the cloud along with 3-D points. If a user is observing a dynamic
scene, the user may also send additional information up to the
cloud servers. In one or more embodiments, object recognizers may
run (either on the cloud resource or on the local system) in order
to recognize one or more objects in the captured points. More
information on object recognizers and the passable world model may
be found in U.S. patent application Ser. No. 14/205,126, entitled
"SYSTEM AND METHOD FOR AUTMENTED AND VIRTUAL REALITY", which is
incorporated by reference in its entirety herein, along with the
following additional disclosures, which related to augmented and
virtual reality systems such as those developed by Magic Leap, Inc.
of Fort Lauderdale, Fla.: U.S. patent application Ser. No.
14/641,376; U.S. patent application Ser. No. 14/555,585; U.S.
patent application Ser. No. 14/212,961; U.S. patent application
Ser. No. 14/690,401; U.S. patent application Ser. No. 13/663,466;
and U.S. patent application Ser. No. 13/684,489.
[0064] In order to capture points that can be used to create the
"passable world model," it is helpful to accurately know the user's
location, pose and orientation with respect to the world. More
particularly, the user's position must be localized to a granular
degree, because it may be important to know the user's head pose,
as well as hand pose (if the user is clutching a handheld
component, gesturing, etc.). In one or more embodiments, GPS and
other localization information may be utilized as inputs to such
processing. Highly accurate localization of the user's head,
totems, hand gestures, haptic devices etc. are desirable in
processing images and points derived from a particular AR system,
and also in order to displaying appropriate virtual content to the
user.
[0065] One approach to achieve high precision localization may
involve the use of an electromagnetic field coupled with
electromagnetic sensors that are strategically placed on the user's
AR head set, belt pack, and/or other ancillary devices (e.g.,
totems, haptic devices, gaming instruments, etc.). Electromagnetic
tracking systems typically comprise at least an electromagnetic
field emitter and at least one electromagnetic field sensor. The
electromagnetic sensors may measure electromagnetic fields with a
known distribution. Based on these measurements a position and
orientation of a field sensor relative to the emitter is
determined.
[0066] Referring now to FIG. 4, an example system of an
electromagnetic tracking system (e.g., such as those developed by
organizations such as the Biosense.RTM. division of Johnson &
Johnson Corporation, Polhemus.RTM., Inc. of Colchester, Vt., and
manufactured by Sixense.RTM. Entertainment, Inc. of Los Gatos,
Calif., and other tracking companies) is illustrated. In one or
more embodiments, the electromagnetic tracking system comprises an
electromagnetic field emitter 402 which is configured to emit a
known magnetic field. As shown in FIG. 4, the electromagnetic field
emitter 402 may be coupled to a power supply 410 (e.g., electric
current, batteries, etc.) to provide power to the electromagnetic
field emitter 402.
[0067] In one or more embodiments, the electromagnetic field
emitter 402 comprises several coils (e.g., at least three coils
positioned perpendicular to each other to produce a field in the x,
y and z directions) that generate magnetic fields. These magnetic
fields are used to establish a coordinate space. This may allow the
system to map a position of the sensors 404 in relation to the
known magnetic field, which, in turn, helps determine a position
and/or orientation of the sensors 404. In one or more embodiments,
the electromagnetic sensors 404a, 404b, etc. may be attached to one
or more real objects. The electromagnetic sensors 404 may comprise
smaller coils in which current may be induced through the emitted
electromagnetic field. Generally, the "sensor" components 404 may
comprise small coils or loops, such as a set of three
differently-oriented (i.e., such as orthogonally oriented relative
to each other) coils coupled together within a small structure such
as a cube or other container, that are positioned/oriented to
capture incoming magnetic flux from the magnetic field emitted by
the electromagnetic emitter 402. By comparing currents induced
through these coils, and by knowing the relative position and
orientation of the coils relative to each other, a relative
position and orientation of a sensor 404 relative to the
electromagnetic emitter 402 may be calculated.
[0068] One or more parameters pertaining to a behavior of the coils
in the electromagnetic tracking sensors 404 and the inertial
measurement unit ("IMU") components operatively coupled to the
electromagnetic tracking sensors 404 may be measured in order to
detect a position and/or orientation of the sensor 404 (and the
object to which it is attached to) relative to a coordinate system
to which the electromagnetic field emitter 402 is coupled. Of
course this coordinate system may be translated into a world
coordinate system, in order to determine a location or pose of the
electromagnetic field emitter in the real world. In one or more
embodiments, multiple sensors 404 may be used in relation to the
electromagnetic emitter 402 to detect a position and orientation of
each of the sensors 404 within the coordinate space associated with
the electromagnetic field emitter 402.
[0069] It should be appreciated that in some embodiments, head pose
may already be known based on sensors on the headmounted component
of the AR system, and SLAM analysis performed based on sensor data
and image data captured through the headmounted AR system. However,
it may be important to know a position of the user's hand (e.g., a
handheld component like a totem, etc.) relative to the known head
pose. In other words, it may be important to know a hand pose
relative to the head pose. Once the relationship between the head
(assuming the sensors are placed on the headmounted component) and
hand is known, a location of the hand relative to the world (e.g.,
world coordinates) can be easily calculated.
[0070] In one or more embodiments, the electromagnetic tracking
system may provide 3-D positions (i.e., X, Y and Z directions) of
the sensors 404, and may further provide location information of
the sensors 404 in two or three orientation angles. In one or more
embodiments, measurements of the IMUs may be compared to the
measurements of the coil to determine a position and orientation of
the sensors 404. In one or more embodiments, both electromagnetic
(EM) data and IMU data, along with various other sources of data,
such as cameras, depth sensors, and other sensors, may be combined
to determine the position and orientation of the electromagnetic
sensors 404.
[0071] In one or more embodiments, this information may be
transmitted (e.g., wireless communication, Bluetooth, etc.) to a
controller 406. In one or more embodiments, pose information (e.g.,
position and orientation) corresponding to the sensors 404 may be
reported at a relatively high refresh rate to the controller 406.
Conventionally, an electromagnetic emitter 402 may be coupled to a
relatively stable and large object, such as a table, operating
table, wall, or ceiling, etc. and one or more sensors 404 may be
coupled to smaller objects, such as medical devices, handheld
gaming components, totems, frame of the head-mounted AR system, or
the like.
[0072] Alternatively, as described below in reference to FIG. 6,
various features of the electromagnetic tracking system may be
employed to produce a configuration wherein changes or deltas in
position and/or orientation between two objects that move in space
relative to a more stable global coordinate system may be tracked.
In other words, a configuration is shown in FIG. 6 wherein a
variation of an electromagnetic tracking system may be utilized to
track position and orientation changes between a head-mounted
component and a hand-held component, while head pose relative to
the global coordinate system (say of the room environment local to
the user) is determined otherwise, such as by simultaneous
localization and mapping ("SLAM") techniques using
outward-capturing cameras which may be coupled to the head mounted
component of the AR system.
[0073] Referring back to FIG. 4, the controller 406 may control the
electromagnetic field emitter 402, and may also capture measurement
data from the various electromagnetic sensors 404. It should be
appreciated that the various components of the system may be
coupled to each other through any electro-mechanical or
wireless/Bluetooth means. The controller 406 may also store data
regarding the known magnetic field, and the coordinate space in
relation to the magnetic field. This information may then be used
to detect the position and orientation of the sensors 404 in
relation to the coordinate space corresponding to the known
electromagnetic field, which can then be used to determined world
coordinates of the user's hand (e.g., location of the
electromagnetic emitter).
[0074] One advantage of electromagnetic tracking systems is that
they can produce highly accurate tracking results with minimal
latency and high resolution. Additionally, the electromagnetic
tracking system does not necessarily rely on optical trackers,
thereby making it easier to track sensors/objects that are not in
the user's line-of-vision.
[0075] It should be appreciated that the strength of the
electromagnetic field ("v") drops as a cubic function of distance
("r") from a coil transmitter (e.g., electromagnetic field emitter
402). One or more algorithms may be formulated based on a distance
of the sensors from the electromagnetic field emitter. The
controller 406 may be configured with such algorithms to determine
a position and orientation of the sensor/object at varying
distances away from the electromagnetic field emitter. Given the
rapid decline of the strength of the electromagnetic field as one
moves farther away from the electromagnetic emitter, improved
results, in terms of accuracy, efficiency and low latency, may be
achieved at closer distances. In typical electromagnetic tracking
systems, the electromagnetic field emitter is powered by electric
current (e.g., plug-in power supply) and has sensors located within
a 20 ft. radius away from the electromagnetic field emitter. A
shorter radius between the sensors and field emitter may be more
desirable in many applications, including AR applications.
[0076] Referring now to FIG. 5, an example flowchart describing a
functioning of a typical electromagnetic tracking system is briefly
described. At 502, a known electromagnetic field is emitted. In one
or more embodiments, the electromagnetic field emitter may generate
a magnetic field. In other words, each coil of the emitter may
generate an electric field in one direction (e.g., x, y or z). The
magnetic fields may be generated with an arbitrary waveform. In one
or more embodiments, each of the axes may oscillate at a slightly
different frequency.
[0077] At 504, a coordinate space corresponding to the
electromagnetic field may be determined. For example, the
controller 406 of FIG. 4 may automatically determine a coordinate
space around the electromagnetic emitter based on parameters of the
electromagnetic field. At 506, a behavior of the coils at the
sensors (which may be attached to a known object) may be
detected/measured. For example, a current induced at the coils may
be measured. In other embodiments, a rotation of a coil, or other
quantifiable behavior may be tracked and measured. At 508, this
measurement may be used to determine/calculate a position and
orientation of the sensor(s) and/or known object. For example, the
controller may consult a mapping table that correlates a behavior
of the coils at the sensors to various positions or orientations.
Based on these calculations, the position and orientation of the
sensors (or object attached thereto) within the coordinate space
may be determined. In some embodiments, the pose/location
information may be determined at the sensors. In other embodiment,
the sensors communicate data detected at the sensors to the
controller, and the controller may consult the mapping table to
determined pose information relative to the known magnetic field
(e.g., coordinates relative to the handheld component).
[0078] In the context of AR systems, one or more components of the
electromagnetic tracking system may need to be modified in order to
facilitate accurate tracking of mobile components. As described
above, tracking the user's head pose and orientation is helpful in
many AR applications. Accurate determination of the user's head
pose and orientation allows the AR system to display the right
virtual content to the user in the appropriate position in the AR
display. For example, the virtual scene may comprise a monster
hiding behind a real building. Depending on the pose and
orientation of the user's head in relation to the building, the
view of the virtual monster may need to be modified such that a
realistic AR experience is provided.
[0079] In other embodiments, a position and/or orientation of a
totem, haptic device or some other means of interacting with a
virtual content may be important in enabling the AR user to
interact with the AR system. For example, in many gaming
applications, the AR system must detect a position and orientation
of a real object in relation to virtual content. Or, when
displaying a virtual interface, a position of a totem, user's hand,
haptic device or any other real object configured for interaction
with the AR system must be known in relation to the displayed
virtual interface in order for the system to understand a command,
etc. Conventional localization methods including optical tracking
and other methods are typically plagued with high latency and low
resolution problems, which makes rendering virtual content
challenging in many AR applications.
[0080] In one or more embodiments, the electromagnetic tracking
system, discussed above may be adapted to the AR system to detect
position and orientation of one or more objects in relation to an
emitted electromagnetic field. Typical electromagnetic systems tend
to have large and bulky electromagnetic emitters (e.g., 402 in FIG.
4), which may make them less-than-ideal for use in AR applications.
However, smaller electromagnetic emitters (e.g., in the millimeter
range) may be used to emit a known electromagnetic field in the
context of the AR system.
[0081] Referring now to FIG. 6, an electromagnetic tracking system
may be incorporated into an AR system as shown, with an
electromagnetic field emitter 602 incorporated as part of a
hand-held controller 606. In one or more embodiments, the hand-held
controller may be a totem to be used in a gaming application. In
other embodiments, the hand-held controller may be a haptic device
that may be used to interact with the AR system (e.g., via a
virtual user interface). In yet other embodiments, the
electromagnetic field emitter may simply be incorporated as part of
the belt pack 70, as shown in FIG. 2D. The hand-held controller 606
may comprise a battery 610 or other power supply that powers the
electromagnetic field emitter 602.
[0082] It should be appreciated that the electromagnetic field
emitter 602 may also comprise or be coupled to an IMU component 650
that is configured to assist in determining position and/or
orientation of the electromagnetic field emitter 602 relative to
other components. This may be useful in cases where both the
electromagnetic field emitter 602 and the sensors 604 (discussed in
further detail below) are mobile. In some embodiments, placing the
electromagnetic field emitter 602 in the hand-held controller
rather than the belt pack, as shown in the embodiment of FIG. 6,
ensures that the electromagnetic field emitter does not compete for
resources at the belt pack, but rather uses its own battery source
at the hand-held controller 606.
[0083] In one or more embodiments, electromagnetic sensors 604 may
be placed on one or more locations on the user's headset 58, along
with other sensing devices such as one or more IMUs or additional
magnetic flux capturing coils 608. For example, as shown in FIG. 6,
sensors 604, 608 may be placed on either side of the head set 58.
Since these sensors 604, 608 are engineered to be rather small (and
hence may be less sensitive, in some cases), it may be important to
include multiple sensors in order to improve efficiency and
precision of the measurements.
[0084] In one or more embodiments, one or more sensors 604, 608 may
also be placed on the belt pack 620 or any other part of the user's
body. The sensors 604, 608 may communicate wirelessly or through
Bluetooth.RTM. with a computing apparatus 607 (e.g., the
controller) that determines a pose and orientation of the sensors
604, 608 (and the AR headset 58 to which they are attached) in
relation to the known magnetic field emitted by the electromagnetic
field emitter 602. In one or more embodiments, as shown in FIG. 6,
the computing apparatus 607 may reside at the belt pack 620. In
other embodiments, the computing apparatus 607 may reside at the
headset 58 itself, or even the hand-held controller 604. In one or
more embodiments, the computing apparatus 607 may receive the
measurements of the sensors 604, 608, and determine a position and
orientation of the sensors 604, 608 in relation to the known
electromagnetic field emitted by the electromagnetic filed emitter
602.
[0085] In one or more embodiments, a mapping database 632 may be
consulted to determine the location coordinates of the sensors 604,
608. The mapping database 632 may reside in the belt pack 620 in
some embodiments. In the illustrated embodiment, the mapping
database 632 resides on a cloud resource 630. As shown in FIG. 6,
the computing apparatus 607 communicates wirelessly to the cloud
resource 630. The determined pose information in conjunction with
points and images collected by the AR system may then be
communicated to the cloud resource 630, and then be added to the
passable world model 634.
[0086] As described above, conventional electromagnetic emitters
may be too bulky for use in AR devices. Therefore, the
electromagnetic field emitter may be engineered to be compact,
using smaller coils compared to traditional systems. However, given
that the strength of the electromagnetic field decreases as a cubic
function of the distance away from the field emitter, a shorter
radius between the electromagnetic sensors 604 and the
electromagnetic field emitter 602 (e.g., about 3-3.5 ft.) may
reduce power consumption while maintaining acceptable field
strength when compared to conventional systems such as the one
detailed in FIG. 4.
[0087] In one or more embodiments, this feature may be utilized to
prolong the life of the battery 610 that powers the controller 606
and the electromagnetic field emitter 602. Alternatively, this
feature may be utilized to reduce the size of the coils generating
the magnetic field at the electromagnetic field emitter 602.
However, in order to get the same strength of magnetic field, the
power of the electromagnetic field emitter 602 may be need to be
increased. This allows for an electromagnetic field emitter unit
602 that may fit compactly at the hand-held controller 606.
[0088] Several other changes may be made when using the
electromagnetic tracking system for AR devices. In one or more
embodiments, IMU-based pose tracking may be used. In such
embodiments, maintaining the IMUs as stable as possible increases
an efficiency of the pose detection process. The IMUs may be
engineered such that they remain stable up to 50-100 milliseconds,
which results in stable signals with pose update/reporting rates of
10-20 Hz. It should be appreciated that some embodiments may
utilize an outside pose estimator module (because IMUs may drift
over time) that may enable pose updates to be reported at a rate of
10-20 Hz. By keeping the IMUs stable for a reasonable amount of
time, the rate of pose updates may be dramatically decreased to
10-20 Hz (as compared to higher frequencies in conventional
systems).
[0089] Yet another way to conserve power of the AR system may be to
run the electromagnetic tracking system at a 10% duty cycle (e.g.,
only pinging for ground every 100 milliseconds). In other words,
the electromagnetic tracking system operates for 10 milliseconds
out of every 100 milliseconds to generate a pose estimate. This
directly translates to power savings, which may, in turn, affect
size, battery life and cost of the AR device.
[0090] In one or more embodiments, this reduction in duty cycle may
be strategically utilized by providing two hand-held controllers
(not shown) rather than just one. For example, the user may be
playing a game that requires two totems, etc. Or, in a multi-user
game, two users may have their own totems/hand-held controllers to
play the game. When two controllers (e.g., symmetrical controllers
for each hand) are used rather than one, the controllers may
operate at offset duty cycles. The same concept may also be applied
to controllers utilized by two different users playing a
multi-player game, for example.
[0091] Referring now to FIG. 7, an example flow chart describing
the electromagnetic tracking system in the context of AR devices is
described. At 702, the hand-held controller 606 emits a magnetic
field. At 704, the electromagnetic sensors 604 (placed on headset
58, belt pack 620, etc.) detect the magnetic field. At 706, a
position and orientation of the headset/belt is determined based on
a behavior of the coils/IMUs 608 at the sensors 604. In some
embodiments, the detected behavior of the sensors 604 is
communicated to the computing apparatus 607, which in turn
determines the position and orientation of the sensors 604 in
relation to the electromagnetic field (e.g., coordinates relative
to the hand-held component). Of course, it should be appreciated
that these coordinates may then be converted to world coordinates,
since the head pose relative to the world may be known through SLAM
processing, as discussed above.
[0092] At 708, the pose information is conveyed to the computing
apparatus 607 (e.g., at the belt pack 620 or headset 58). At 710,
optionally, the passable world model 634 may be consulted determine
virtual content to be displayed to the user based on the determined
head pose and hand pose. At 712, virtual content may be delivered
to the user at the AR headset 58 based on the correlation. It
should be appreciated that the flowchart described above is for
illustrative purposes only, and should not be read as limiting.
[0093] Advantageously, using an electromagnetic tracking system
similar to the one outlined in FIG. 6 enables pose tracking at a
higher refresh rate and lower latency (e.g., head position and
orientation, position and orientation of totems, and other
controllers). This allows the AR system to project virtual content
with a higher degree of accuracy, and with lower latency when
compared to optical tracking techniques for calculating pose
information.
[0094] Referring to FIG. 8, a system configuration is illustrated
featuring many sensing components, similar to the sensors described
above. It should be appreciated that the reference numbers of FIGS.
2A-2D, and FIG. 6 are repeated in FIG. 8. A head mounted wearable
component 58 is shown operatively coupled 68 to a local processing
and data module 70, such as a belt pack (similar to FIG. 2D), here
using a physical multicore lead which also features a control and
quick release module 86 as described below in reference to FIGS.
9A-9F. The local processing and data module 70 may be operatively
coupled 100 to a hand held component 606 (similar to FIG. 6). In
one or more embodiments, the local processing module 70 may be
coupled to the hand-held component 606 through a wireless
connection such as low power Bluetooth.RTM.. In one or more
embodiments, the hand held component 606 may also be operatively
coupled 94 directly to the head mounted wearable component 58, such
as by a wireless connection such as low power Bluetooth.RTM..
[0095] Generally, where IMU data is passed in order to detect pose
information of various components, a high-frequency connection may
be desirable, such as in the range of hundreds or thousands of
cycles/second or higher. On the other hand, tens of cycles per
second may be adequate for electromagnetic localization sensing,
such as by the sensor 604 and transmitter 602 pairings. Also shown
is a global coordinate system 10, representative of fixed objects
in the real world around the user, such as a wall 8. Cloud
resources 46 also may be operatively coupled 42, 40, 88, 90 to the
local processing and data module 70, to the head mounted wearable
component 58, to resources which may be coupled to the wall 8 or
other item fixed relative to the global coordinate system 10,
respectively. The resources coupled to the wall 8 or having known
positions and/or orientations relative to the global coordinate
system 10 may include a Wi-Fi transceiver 114, an electromagnetic
emitter 602 and/or receiver 604, a beacon or reflector 112
configured to emit or reflect a given type of radiation, such as an
infrared LED beacon, a cellular network transceiver 110, a RADAR
emitter or detector 108, a LIDAR emitter or detector 106, a GPS
transceiver 118, a poster or marker having a known detectable
pattern 122, and a camera 124.
[0096] The head mounted wearable component 58 features similar
components, as illustrated, in addition to lighting emitters 130
configured to assist the camera 124 detectors, such as infrared
emitters 130 for an infrared camera 124. In one or more
embodiments, the head mounted wearable component 58 may further
comprise one or more strain gauges 116, which may be fixedly
coupled to the frame or mechanical platform of the head mounted
wearable component 58 and configured to determine deflection of
such platform in between components such as electromagnetic
receiver sensors 604 or display elements 62, wherein it may be
valuable to understand if bending of the platform has occurred,
such as at a thinned portion of the platform, such as the portion
above the nose on the eyeglasses-like platform depicted in FIG.
8.
[0097] The head mounted wearable component 58 may also include a
processor 128 and one or more IMUs 102. Each of the components
preferably are operatively coupled to the processor 128. The hand
held component 606 and local processing and data module 70 are
illustrated featuring similar components. As shown in FIG. 8, with
so many sensing and connectivity means, such a system is likely to
be heavy, large, relatively expensive, and likely to consume large
amounts of power. However, for illustrative purposes, such a system
may be utilized to provide a very high level of connectivity,
system component integration, and position/orientation tracking.
For example, with such a configuration, the various main mobile
components (58, 70, 606) may be localized in terms of position
relative to the global coordinate system using Wi-Fi, GPS, or
Cellular signal triangulation; beacons, electromagnetic tracking
(as described above), RADAR, and LIDIR systems may provide yet
further location and/or orientation information and feedback.
Markers and cameras also may be utilized to provide further
information regarding relative and absolute position and
orientation. For example, the various camera components 124, such
as those shown coupled to the head mounted wearable component 58,
may be utilized to capture data which may be utilized in
simultaneous localization and mapping protocols, or "SLAM", to
determine where the component 58 is and how it is oriented relative
to other components.
[0098] Referring to FIGS. 9A-9F, various aspects of the control and
quick release module 86 are depicted. Referring to FIG. 9A, two
outer housing 134 components are coupled together using a magnetic
coupling configuration which may be enhanced with mechanical
latching. Buttons 136 for operation of the associated system may be
included. FIG. 9B illustrates a partial cutaway view with the
buttons 136 and underlying top printed circuit board 138 shown.
Referring to FIG. 9C, with the buttons 136 and underlying top
printed circuit board 138 removed, a female contact pin array 140
is visible. Referring to FIG. 9D, with an opposite portion of
housing 134 removed, the lower printed circuit board 142 is
visible. With the lower printed circuit board 142 removed, as shown
in FIG. 9E, a male contact pin array 144 is visible.
[0099] Referring to the cross-sectional view of FIG. 9F, at least
one of the male pins or female pins are configured to be
spring-loaded such that they may be depressed along each pin's
longitudinal axis. In one or more embodiments, the pins may be
termed "pogo pins" and may generally comprise a highly conductive
material, such as copper or gold. When assembled, the illustrated
configuration may mate 46 male pins with female pins, and the
entire assembly may be quick-release decoupled in half by manually
pulling it apart and overcoming a magnetic interface 146 load which
may be developed using north and south magnets oriented around the
perimeters of the pin arrays 140, 144.
[0100] In one embodiment, an approximate 2 kg load from compressing
the 46 pogo pins is countered with a closure maintenance force of
about 4 kg. The pins in the arrays 140, 144 may be separated by
about 1.3 mm, and the pins may be operatively coupled to conductive
lines of various types, such as twisted pairs or other combinations
to support USB 3.0, HDMI 2.0, I2S signals, GPIO, and MIPI
configurations, and high current analog lines and grounds
configured for up to about 4 amps/5 volts in one embodiment.
[0101] Referring to FIG. 10, it is helpful to have a minimized
component/feature set to be able to minimize the weight and bulk of
the various components, and to arrive at a relatively slim head
mounted component, for example, such as that of head mounted
component 58 featured in FIG. 10. Thus various permutations and
combinations of the various components shown in FIG. 8 may be
utilized.
[0102] Referring to FIG. 11A, an electromagnetic sensing coil
assembly (604, e.g., 3 individual coils coupled to a housing) is
shown coupled to a head mounted component 58. Such a configuration
adds additional geometry (i.e., a protrusion) to the overall
assembly which may not be desirable. Referring to FIG. 11B, rather
than housing the coils in a box or single housing as in the
configuration of FIG. 11A, the individual coils may be integrated
into the various structures of the head mounted component 58, as
shown in FIG. 11B. For example, x-axis coil 148 may be placed in
one portion of the head mounted component 58 (e.g., the center of
the frame). Similarly, the y-axis coil 150 may be placed in another
portion of the head mounted component 58 (e.g., either bottom side
of the frame). Similarly, the z-axis coil 152 may be placed in yet
another portion of the head mounted component 58 (e.g., either top
side of the frame).
[0103] FIGS. 12A-12E illustrate various configurations for
featuring a ferrite core coupled to an electromagnetic sensor to
increase field sensitivity. Referring to FIG. 12A, the ferrite core
may be a solid cube 1202. Although the solid cube may be most
effective in increasing field sensitivity, it may also be the most
heavy when compared to the remaining configurations depicted in
FIGS. 12B-12E. Referring to FIG. 12B, a plurality of ferrite disks
1204 may be coupled to the electromagnetic sensor. Similarly,
referring to FIG. 12C, a solid cube with a one axis air core 1206
may be coupled to the electromagnetic sensor. As shown in FIG. 12C,
an open space (i.e., the air core) may be formed in the solid cube
along one axis. This may decrease the weight of the cube, while
still providing the necessary field sensitivity. In yet another
embodiment, referring to FIG. 12D, a solid cube with a three axis
air core 1208 may be coupled to the electromagnetic sensor. In this
configuration, the solid cube is hollowed out along all three axes,
thereby decreasing the weight of the cube considerably. Referring
to FIG. 12E, ferrite rods with plastic housing 1210 may also be
coupled to the electromagnetic sensor. It should be appreciated
that the embodiments of FIGS. 12B-12E are lighter in weight than
the solid core configuration of FIG. 12A and may be utilized to
save mass, as discussed above.
[0104] Referring to FIGS. 13A-13C, time division multiplexing
("TDM") may be utilized to save mass as well. For example,
referring to FIG. 13A, a conventional local data processing
configuration is shown for a 3-coil electromagnetic receiver
sensor, wherein analog currents come in from each of the X, Y, and
Z coils (1302, 1304 and 1306), go into a separate pre-amplifier
1308, go into a separate band pass filter 1310, a separate
pre-amplifier 1312, through an analog-to-digital converter 1314,
and ultimately to a digital signal processor 1316.
[0105] Referring to the transmitter configuration of FIG. 13B, and
the receiver configuration of FIG. 13C, time division multiplexing
may be utilized to share hardware, such that each coil sensor chain
doesn't require its own amplifiers, etc. This may be achieved
through a TDM switch 1320, as shown in FIG. 13B, which facilitates
processing of signals to and from multiple transmitters and
receivers using the same set of hardware components (amplifiers,
etc.). In addition to removing sensor housings, and multiplexing to
save on hardware overhead, signal to noise ratios may be increased
by having more than one set of electromagnetic sensors, each set
being relatively small relative to a single larger coil set. Also,
the low-side frequency limits, which generally are needed to have
multiple sensing coils in close proximity, may be improved to
facilitate bandwidth requirement improvements. It should be noted
that there may be a tradeoff with multiplexing, in that
multiplexing generally spreads out the reception of radiofrequency
signals in time, which results in generally coarser signals. Thus,
larger coil diameters may be required for multiplexed systems. For
example, where a multiplexed system may require a 9 mm-side
dimension cubic coil sensor box, a non-multiplexed system may only
require a 7 mm-side dimension cubic coil box for similar
performance. Thus, it should be noted that there may be tradeoffs
in minimizing geometry and mass.
[0106] In another embodiment wherein a particular system component,
such as a head mounted component 58 features two or more
electromagnetic coil sensor sets, the system may be configured to
selectively utilize the sensor and electromagnetic emitter pairing
that are closest to each other to optimize the performance of the
system.
[0107] Referring to FIG. 14, in one embodiment, after a user powers
up his or her wearable computing system 160, a head mounted
component assembly may capture a combination of IMU and camera data
(the camera data being used, for example, for SLAM analysis, such
as at the belt pack processor where there may be more RAW
processing horsepower present) to determine and update head pose
(i.e., position and orientation) relative to a real world global
coordinate system 162. The user may also activate a handheld
component to, for example, play an augmented reality game 164, and
the handheld component may comprise an electromagnetic transmitter
operatively coupled to one or both of the belt pack and head
mounted component 166. One or more electromagnetic field coil
receiver sets (e.g., a set being 3 differently-oriented individual
coils) coupled to the head mounted component may be used to capture
magnetic flux from the electromagnetic transmitter. This captured
magnetic flux may be utilized to determine positional or
orientational difference (or "delta"), between the head mounted
component and handheld component 168.
[0108] In one or more embodiments, the combination of the head
mounted component assisting in determining pose relative to the
global coordinate system, and the hand held assisting in
determining relative location and orientation of the handheld
relative to the head mounted component, allows the system to
generally determine where each component is located relative to the
global coordinate system, and thus the user's head pose, and
handheld pose may be tracked, preferably at relatively low latency,
for presentation of augmented reality image features and
interaction using movements and rotations of the handheld component
170.
[0109] Referring to FIG. 15, an embodiment is illustrated that is
somewhat similar to that of FIG. 14, with the exception that the
system has many more sensing devices and configurations available
to assist in determining pose of both the head mounted component
172 and a hand held component 176, 178, such that the user's head
pose, and handheld pose may be tracked, preferably at relatively
low latency, for presentation of augmented reality image features
and interaction using movements and rotations of the handheld
component 180.
[0110] Specifically, after a user powers up his or her wearable
computing system 160, a head mounted component captures a
combination of IMU and camera data for SLAM analysis in order to
determined and update head pose relative a real-world global
coordinate system. The system may be further configured to detect
presence of other localization resources in the environment, like
Wi-Fi, cellular, beacons, RADAR, LIDAR, GPS, markers, and/or other
cameras which may be tied to various aspects of the global
coordinate system, or to one or more movable components 172.
[0111] The user may also activate a handheld component to, for
example, play an augmented reality game 174, and the handheld
component may comprise an electromagnetic transmitter operatively
coupled to one or both of the belt pack and head mounted component
176. Other localization resources may also be similarly utilized.
One or more electromagnetic field coil receiver sets (e.g., a set
being 3 differently-oriented individual coils) coupled to the head
mounted component may be used to capture magnetic flux from the
electromagnetic transmitter. This captured magnetic flux may be
utilized to determine positional or orientational difference (or
"delta"), between the head mounted component and handheld component
178.
[0112] Thus, the user's head pose and the handheld pose may be
tracked at relatively low latency for presentation of AR content
and/or for interaction with the AR system using movement or
rotations of the handheld component 180.
[0113] Various exemplary embodiments of the invention are described
herein. Reference is made to these examples in a non-limiting
sense. They are provided to illustrate more broadly applicable
aspects of the invention. Various changes may be made to the
invention described and equivalents may be substituted without
departing from the true spirit and scope of the invention. In
addition, many modifications may be made to adapt a particular
situation, material, composition of matter, process, process act(s)
or step(s) to the objective(s), spirit or scope of the present
invention. Further, as will be appreciated by those with skill in
the art that each of the individual variations described and
illustrated herein has discrete components and features which may
be readily separated from or combined with the features of any of
the other several embodiments without departing from the scope or
spirit of the present inventions. All such modifications are
intended to be within the scope of claims associated with this
disclosure.
[0114] The invention includes methods that may be performed using
the subject devices. The methods may comprise the act of providing
such a suitable device. Such provision may be performed by the end
user. In other words, the "providing" act merely requires the end
user obtain, access, approach, position, set-up, activate, power-up
or otherwise act to provide the requisite device in the subject
method. Methods recited herein may be carried out in any order of
the recited events which is logically possible, as well as in the
recited order of events.
[0115] Exemplary aspects of the invention, together with details
regarding material selection and manufacture have been set forth
above. As for other details of the present invention, these may be
appreciated in connection with the above-referenced patents and
publications as well as generally known or appreciated by those
with skill in the art. The same may hold true with respect to
method-based aspects of the invention in terms of additional acts
as commonly or logically employed.
[0116] In addition, though the invention has been described in
reference to several examples optionally incorporating various
features, the invention is not to be limited to that which is
described or indicated as contemplated with respect to each
variation of the invention. Various changes may be made to the
invention described and equivalents (whether recited herein or not
included for the sake of some brevity) may be substituted without
departing from the true spirit and scope of the invention. In
addition, where a range of values is provided, it is understood
that every intervening value, between the upper and lower limit of
that range and any other stated or intervening value in that stated
range, is encompassed within the invention.
[0117] Also, it is contemplated that any optional feature of the
inventive variations described may be set forth and claimed
independently, or in combination with any one or more of the
features described herein. Reference to a singular item, includes
the possibility that there are plural of the same items present.
More specifically, as used herein and in claims associated hereto,
the singular forms "a," "an," "said," and "the" include plural
referents unless the specifically stated otherwise. In other words,
use of the articles allow for "at least one" of the subject item in
the description above as well as claims associated with this
disclosure. It is further noted that such claims may be drafted to
exclude any optional element. As such, this statement is intended
to serve as antecedent basis for use of such exclusive terminology
as "solely," "only" and the like in connection with the recitation
of claim elements, or use of a "negative" limitation.
[0118] Without the use of such exclusive terminology, the term
"comprising" in claims associated with this disclosure shall allow
for the inclusion of any additional element--irrespective of
whether a given number of elements are enumerated in such claims,
or the addition of a feature could be regarded as transforming the
nature of an element set forth in such claims. Except as
specifically defined herein, all technical and scientific terms
used herein are to be given as broad a commonly understood meaning
as possible while maintaining claim validity.
[0119] The breadth of the present invention is not to be limited to
the examples provided and/or the subject specification, but rather
only by the scope of claim language associated with this
disclosure.
* * * * *