U.S. patent application number 17/170255 was filed with the patent office on 2021-08-12 for gesture tracking for mobile rendered augmented reality.
The applicant listed for this patent is Krikey, Inc.. Invention is credited to Yusuf Olanrewaju Olokoba, Jhanvi Samyukta Lakshmi Shriram, Ketaki Lalitha Uthra Shriram.
Application Number | 20210247846 17/170255 |
Document ID | / |
Family ID | 1000005400852 |
Filed Date | 2021-08-12 |
United States Patent
Application |
20210247846 |
Kind Code |
A1 |
Shriram; Ketaki Lalitha Uthra ;
et al. |
August 12, 2021 |
GESTURE TRACKING FOR MOBILE RENDERED AUGMENTED REALITY
Abstract
An augmented reality (AR) system hosted and executed on a mobile
client enables control of AR objects using gestures. The system
receives, from the mobile client, an image from a camera view
(e.g., the mobile client's camera) of an environment, where the
image depicts a user's hand. The system applies a machine learning
model to the received image. The machine learning model identifies
a formation of the hand. The system determines to render an AR
object based on the identified formation. For example, a user
forming a first with his hand may cause an AR ball to move upward
within the screen of the mobile client.
Inventors: |
Shriram; Ketaki Lalitha Uthra;
(San Francisco, CA) ; Shriram; Jhanvi Samyukta
Lakshmi; (San Francisco, CA) ; Olokoba; Yusuf
Olanrewaju; (Hanover, NH) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Krikey, Inc. |
Menlo Park |
CA |
US |
|
|
Family ID: |
1000005400852 |
Appl. No.: |
17/170255 |
Filed: |
February 8, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62971766 |
Feb 7, 2020 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/6267 20130101;
G06F 3/017 20130101; G06T 19/006 20130101; G06N 20/00 20190101 |
International
Class: |
G06F 3/01 20060101
G06F003/01; G06T 19/00 20060101 G06T019/00; G06N 20/00 20060101
G06N020/00; G06K 9/62 20060101 G06K009/62 |
Claims
1. A non-transitory computer readable storage medium comprising
stored instructions, the instructions when executed by a processor
cause the processor to: receive from a mobile client an image from
a camera view of an environment, the image depicting a hand of a
user; apply a machine learning model to the image, the machine
learning model trained on training image data representative of a
plurality of hand formations, the machine learning model configured
to identify a formation of the hand in the image as one of the
plurality of hand formations; provide for display on the mobile
client, responsive to identification of the formation of the hand
as a first hand formation of the plurality of hand formations, an
augmented reality (AR) object in a first state from an AR engine in
the camera view of the environment; and provide for display on the
mobile client, responsive to the formation of the hand identified
as a second hand formation of the plurality of hand formations, the
AR object in a second state from the AR engine in the camera view
of the environment.
2. The non-transitory computer readable storage medium of claim 1,
wherein the machine learning model is a first machine learning
model, and wherein the instructions further comprise instructions
that when executed by the processor cause the processor to identify
a location of the hand, the instructions to identify location of
the hand further comprising instructions that when executed by the
processor cause the processor to: apply a second machine learning
model to the received image, the second machine learning model
configured to classify real-world objects in the environment, the
real-world objects including the hand; receive a plurality of
feature points associated with the real-world objects in the
environment; generate a three-dimensional (3D) virtual coordinate
space based the plurality of feature points; and identify, based on
a classification of the hand by the second machine learning model,
the location of the hand associated with corresponding coordinates
in the generated 3D virtual coordinate space.
3. The non-transitory computer readable storage medium of claim 2,
wherein the AR engine rendered object is provided for display based
on the identified location.
4. The non-transitory computer readable storage medium of claim 2,
wherein the instructions further comprise instructions that when
executed by the processor cause the processor to remove data of a
given image from memory responsive to the second machine learning
model classifying an absence of any hand depicted within the given
image.
5. The non-transitory computer readable storage medium of claim 1,
wherein the instructions further comprise instructions that when
executed by the processor cause the processor to train the machine
learning model using the training image data, the instruction to
train the machine learning model further comprising instructions
that when executed by the processor cause the processor to: receive
a plurality of images of the plurality of hand formations; and
apply a respective label to each of the plurality of images of the
plurality of hand formations, the training image data comprising
the labeled plurality of images.
6. The non-transitory computer readable storage medium of claim 5,
wherein each respective label corresponds to a computer executable
command.
7. The non-transitory computer readable storage medium of claim 5,
wherein the plurality of hand formations includes a user-customized
hand formation.
8. The non-transitory computer readable storage medium of claim 7,
wherein the instructions further comprise instructions that when
executed by the processor cause the processor to prompt the user to
provide a user-specified state of the AR engine rendered object,
wherein an identification of the user-customized hand formation by
the machine learning model indicates that the AR engine rendered
object is to be provided for display in the user-specified
state.
9. The non-transitory computer readable storage medium of claim 1,
wherein the machine learning model is further configured to output
a confidence score associated with the identified formation of the
hand, and wherein providing for display the AR engine rendered
object in the first state or in the second state is further
responsive to the confidence score exceeding a threshold confidence
score.
10. The non-transitory computer readable storage medium of claim 9,
wherein the instructions further comprise instructions that when
executed by the processor cause the processor to receive a
selection of a user-specified threshold confidence score.
11. The non-transitory computer readable storage medium of claim 1,
wherein the AR engine is a game engine.
12. A computer system comprising: a gesture tracking module
configured to: receive from a mobile client an image from a camera
view of an environment, the image depicting a hand of a user; and
apply a machine learning model to the image, the machine learning
model trained on training image data representative of a plurality
of hand formations, the machine learning model configured to
identify a formation of the hand in the image as one of the
plurality of hand formations; and a rendering module of the gesture
tracking application configured to: provide for display on the
mobile client, responsive to identification of the formation of the
hand as a first hand formation of the plurality of hand formations,
an augmented reality (AR) object in a first state from an AR engine
in the camera view of the environment; and provide for display on
the mobile client, responsive to the formation of the hand
identified as a second hand formation of the plurality of hand
formations, the AR object in a second state from the AR engine in
the camera view of the environment.
13. The system of claim 12, wherein the machine learning model is a
first machine learning model, wherein the gesture tracking module
is further configured to identify a location of the hand within the
image by being further configured to: apply a second machine
learning model to the received image, the second machine learning
model configured to classify real-world objects in the environment,
the real-world objects including the hand; receive a plurality of
feature points associated with the real-world objects in the
environment; generate a three-dimensional (3D) virtual coordinate
space based the plurality of feature points; and identify, based on
a classification of the hand by the second machine learning model,
the location of the hand associated with corresponding coordinates
in the generated 3D virtual coordinate space.
14. The system of claim 13, wherein the AR engine rendered object
is provided for display based on the identified location.
15. The system of claim 13, wherein the gesture tracking module is
further configured to remove data of a given image from memory
responsive to classification of an absence of any hand depicted
within the given image by the second machine learning model.
16. A computer-implemented method comprising: receiving from a
mobile client an image from a camera view of an environment, the
image depicting a hand of a user; applying a machine learning model
to the image, the machine learning model trained on training image
data representative of a plurality of hand formations, the machine
learning model configured to identify a formation of the hand in
the image as one of the plurality of hand formations; providing for
display on the mobile client, responsive to identification of the
formation of the hand as a first hand formation of the plurality of
hand formations, an augmented reality (AR) object in a first state
from an AR engine in the camera view of the environment; and
providing for display on the mobile client, responsive to the
formation of the hand identified as a second hand formation of the
plurality of hand formations, the AR object in a second state from
the AR engine in the camera view of the environment.
17. The computer-implemented method of claim 16, wherein the
machine learning model is a first machine learning model, further
comprising identifying a location of the hand within the image by:
applying a second machine learning model to the received image, the
second machine learning model configured to classify real-world
objects in the environment, the real-world objects including the
hand; receiving a plurality of feature points associated with the
real-world objects in the environment; generating a
three-dimensional (3D) virtual coordinate space based the plurality
of feature points; and identifying, based on a classification of
the hand by the second machine learning model, the location of the
hand associated with corresponding coordinates in the generated 3D
virtual coordinate space.
18. The computer-implemented method of claim 16, wherein the AR
engine rendered object is provided for display based on the
identified location.
19. The computer-implemented method of claim 16, further comprising
removing data of a given image from memory responsive to the second
machine learning model classifying an absence of any hand depicted
within the given image.
20. A computer-implemented method comprising: receiving from a
mobile client an image from a camera view of an environment, the
image depicting a portion of a body of a user; providing the image
to a machine learning model configured to identify a formation of
the portion of the body in the image; and providing for display on
the mobile client, based on an identification of the formation by
the machine learning model, an augmented reality (AR) object in the
camera view of the environment.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application No. 62/971,766, filed Feb. 7, 2020, which is
incorporated by reference in its entirety.
TECHNICAL FIELD
[0002] The disclosure generally relates to the field of mobile
rendered augmented reality and more specifically to object control
based on gesture tracking in mobile rendered augmented reality
environments.
BACKGROUND
[0003] Conventional augmented reality (AR) systems use handheld
controllers to track a user's hands and determine whether the user
is making a hand gesture. However, the tracked gestures are limited
to what gestures can be made with a controller in the user's hands.
For example, the user cannot make a "five" gesture (i.e., palm open
and fingers/thumb extended) without dropping the controller. Mobile
devices may execute AR applications without handheld controllers,
but execution of the applications slows. This is because a mobile
device has limited processing bandwidth and/or limited battery life
necessary for intensive processing. Hence, there lacks an AR system
that can track user gestures without demanding large processing
bandwidth or excessively consuming a device's powers.
BRIEF DESCRIPTION OF DRAWINGS
[0004] The disclosed embodiments have other advantages and features
which will be more readily apparent from the detailed description,
the appended claims, and the accompanying figures (or drawings). A
brief introduction of the figures is below.
[0005] Figure (FIG. 1 illustrates an augmented reality (AR) system
environment, in accordance with at least one embodiment.
[0006] FIG. 2 is a block diagram of the gesture tracking
application of FIG. 1, in accordance with at least one
embodiment.
[0007] FIG. 3 is a flowchart illustrating a process for controlling
an AR object using gesture detection, in accordance with at least
one embodiment.
[0008] FIG. 4 is a flowchart illustrating a process for controlling
the AR object using gesture detection based on the process of FIG.
3, in accordance with at least one embodiment.
[0009] FIGS. 5A and 5B illustrate user interactions with an AR
application that integrates gesture tracking to control AR objects,
in accordance with at least one embodiment.
[0010] FIG. 6 illustrates a block diagram including components of a
machine able to read instructions from a machine-readable medium
and execute them in a processor (or controller), in accordance with
at least one embodiment.
DETAILED DESCRIPTION
[0011] The Figures (FIGS.) and the following description relate to
preferred embodiments by way of illustration only. It should be
noted that from the following discussion, alternative embodiments
of the structures and methods disclosed herein will be readily
recognized as viable alternatives that may be employed without
departing from the principles of what is claimed.
[0012] Reference will now be made in detail to several embodiments,
examples of which are illustrated in the accompanying figures. It
is noted that wherever practicable similar or like reference
numbers may be used in the figures and may indicate similar or like
functionality. The figures depict embodiments of the disclosed
system (or method) for purposes of illustration only. One skilled
in the art will readily recognize from the following description
that alternative embodiments of the structures and methods
illustrated herein may be employed without departing from the
principles described herein.
Configuration Overview
[0013] In one example embodiment of a disclosed system, method and
computer readable storage medium, an augmented reality (AR) object
is rendered on a mobile client based on a user's gestures
identified within a camera view of the mobile client. Conventional
gesture tracking solutions for AR are designed for handheld
hardware to track where a user's hands are and determine if a
gesture is being made. Accordingly, described is a configuration
that enables gesture tracking to control an AR object in a mobile
rendered AR system while optimizing for the power and processing
constraints of the mobile client.
[0014] In one example configuration, a camera coupled with the
mobile client (e.g., integrated with the mobile client or
wirelessly or wired connection with the mobile client) captures a
camera view of an environment. The environment may correspond to
the physical world, which may include a portion of a user's body,
that may be positioned with a field of view of the camera. A
processor (e.g., of the mobile device) processes program code that
causes the processor to execute specified functions as are further
described herein. Accordingly, the processor receives an image from
the camera view of the environment and applies a machine learning
model to the received image. Using the machine learning, the
processor identifies a gesture, which may also be referred to
herein as a "formation," made by the hand depicted within the
image. The processor determines, based on the identified gesture, a
state in which to render an AR object (e.g., from an AR engine).
The processor provides for display (e.g., transmits instructions
(e.g., program code or software) to render on a screen) the
rendered object in the environment (e.g., with the user's hand) to
the mobile client. In some embodiments, a user interacts with the
displayed AR objects.
[0015] In some embodiments, if the processor does not detect a
gesture or a hand within a received image, the processor may
determine not to store the image into memory or remove the image
from memory such that the processor will not expend further
processing resources on the frame (e.g., due to its lack of
depicting an informative object). In some embodiments, the machine
learning model for detecting gestures may require more processing
resources than a machine learning model for detecting hands. In
these circumstances, the processor may optimize the mobile client's
processing resources by applying a hand detection model to an image
first rather than directly applying a gesture tracking model.
[0016] Gesture tracking allows the user to use a mobile device to
control and interact with AR objects without dedicated hardware as
though they are interacting with reality around the user,
presenting an immersive gaming experience for the user. In
particular, the methods described herein allow for AR object
control using gesture tracking on a mobile client that does not
consume too much processing and/or battery power.
Augmented Reality System Environment
[0017] Figure (FIG. 1 illustrates an augmented reality (AR) system
environment, in accordance with at least one embodiment. The AR
system environment enables AR applications on a mobile client 100,
and in some embodiments, presents immersive experiences to users
via gesture tracking. The system environment includes a mobile
client 100, an AR system 110, an AR engine 120, a gesture tracking
application 130, a database 140, and a network 150. The AR system
110, in some example embodiments, may include the mobile client
100, the AR engine 120, the gesture tracking application 130, and
the database 140. In other example embodiments, the AR system 110
may include the AR engine 120, the gesture tracking application
130, and the database 140, but not the mobile client 100, such that
the AR system 110 communicatively couples (e.g., wireless
communication) to the mobile client 100 from a remote server.
[0018] The mobile client 100 is a mobile device that is or
incorporates a computer. The mobile client may be, for example, a
relatively small computing device in which network, processing
(e.g., processor and/or controller) and power resources (e.g.,
battery) may be limited and have a formfactor size such as a
smartphone, tablet, wearable device (e.g., smartwatch) and/or a
portable internet enabled device. The limitations of such device
extend from scientific principles that must be adhered to in
designing such products for portability and use away from constant
power draw sources.
[0019] The mobile client 100 may be a computing device that
includes the components of the machine depicted in FIG. 6. The
mobile client 100 has general and/or special purpose processors,
memory, storage, networking components (either wired or wireless).
The mobile client 100 can communicate over one or more
communication connections (e.g., a wired connection such as
ethernet or a wireless communication via cellular signal (e.g.,
LTE, 5G), WiFi, satellite) and includes a global positioning system
(GPS) used to determine a location of the mobile client 100.
[0020] The mobile client 100 also includes one or more cameras 102
that can capture forward and rear facing images and/or videos. To
capture images for gesture or hand detection, the camera 102 may be
a two-dimensional (2D) camera as opposed to a stereo camera or a
three-dimensional (3D) camera. That is, the machine-learned
detection described herein does not necessarily require a 3D image
or depth to classify a hand or a gesture depicted within an
image.
[0021] The mobile client 100 also includes a screen (or display)
103 and a display driver to provide for display interfaces on the
screen 103 associated with the mobile client 100. The mobile client
100 executes an operating system, such as GOOGLE ANDROID OS and/or
APPLE iOS, and includes the screen 103 and/or a user interface that
the user can interact with. In some embodiments, the mobile client
100 couples to the AR system 110, which enables it to execute an AR
application (e.g., the AR client 101).
[0022] The AR engine 120 interacts with the mobile client 100 to
execute the AR client 101 (e.g., an AR game). For example, the AR
engine 120 may be a game engine such as UNITY and/or UNREAL ENGINE.
The AR engine 120 displays, and the user interacts with, the AR
game via the mobile client 100. For example, the mobile client 100
may host and execute the AR client 101 that in turn accesses the AR
engine 120 to enable the user to interact with the AR game.
Although the AR application refers to an AR gaming application in
many instances described herein, the AR application may be a retail
application integrating AR for modeling purchasable products, an
educational application integrating AR for demonstrating concepts
within a learning curriculum, or any suitable interactive
application in which AR may be used to augment the interactions. In
some embodiments, the AR engine 120 is integrated into and/or
hosted on the mobile client 100. In other embodiments, the AR
engine 120 is hosted external to the mobile client 100 and
communicatively couples to the mobile client 100 over the network
150. The AR system 110 may comprise program code that executes
functions as described herein.
[0023] In some example embodiments, the AR system 110 includes the
gesture tracking application 130. The gesture tracking application
enables gesture tracking in the AR game such that AR objects (e.g.,
virtual objects rendered by the AR engine 120) and their behaviors
or states may be controlled by the user. The user may capture an
image and/or video of an environment captured within a camera view
of the camera 102 of the mobile client 100. An image from the
camera view may depict a portion of the user's body such as the
user's hand. The AR engine 120 renders an AR object, where the
rendering may be based on gestures that the gesture tracking
application 130 has determined that the user is performing (e.g., a
fist). The gesture tracking application 130 may detect or track
(e.g., detecting changes in gestures over time) a variety of
gestures such as a five (i.e., an open palm facing away from the
user), a fist, pointing, waving, facial gestures (e.g., smiles,
open mouth, etc.), lifting a leg, bending, kicking, or any suitable
movement made by any portion of the body to express an intention.
While the gesture tracking application 130 is described herein as
primarily tracking hand gestures, the gesture tracking application
130 may detect various gestures as described above. As referred to
herein, "gesture" and "formation" may be used interchangeably.
[0024] During use of the AR client 101 (e.g., during game play),
the gesture tracking application 130 identifies a body part (e.g.,
a hand) within an image from a camera view captured by the camera
102, determines a gesture made by the body part (e.g., a five), and
renders an AR object based on the determined gesture. The gesture
tracking application 130 may instruct the camera 102 to capture
image frames periodically (e.g., every three seconds). In some
embodiments, the state in which the AR engine 120 displays the AR
object depends on input from the user during game play. For
example, the direction of the AR object's movement may change
depending on the user's gestures. FIGS. 5A and 5B, described
further herein, provide details on how gesture tracking may be used
control AR objects in the AR system 110. In some embodiments, the
AR system 110 includes applications instead of and/or in addition
to the gesture tracking application 130. In some embodiments, the
gesture tracking application 130 may be hosted on and/or executed
by the mobile client 100. In other embodiments, the gesture
tracking application 130 is communicatively coupled to the mobile
client 100.
[0025] The database 140 stores images or videos that may be used by
the gesture tracking application 130 to detect a user's hand and
determine the gesture the hand is making. The mobile client 100 may
transmit images or videos collected by the camera 102 during the
execution of the AR client 101 to the database 140. The data stored
within the database 140 may be collected from a single user (e.g.,
the user of the mobile client 100) or multiple users (e.g., users
of other mobile clients that are communicatively coupled to the AR
system 110 through the network 150). The gesture tracking
application 130 may use images and/or videos of gestures stored in
the database 140 to train a model (e.g., a neural network). In
particular, the machine learning model training engine 210 of the
gesture tracking application 130 may access the database 140 to
train a machine learning model. This is described in further detail
in the description of FIG. 2.
[0026] The database 140 may store a mapping of gestures to AR
objects and/or states in which the AR objects may be rendered. The
gesture tracking application 130 may determine a gesture made by a
user's hand as captured within a camera view of the mobile client
100 and access the database 140 to determine, using the determined
gesture, that an AR object should be rendered in a particular state
(e.g., a AR ball floating upward). The database 140 may store one
or more user profiles, each user profile including user
customizations or settings that personalize the user's experience
using the AR client 101. For example, a user profile stored within
the database 140 may store a user-specified name of a custom hand
gesture, images of the customized hand gesture (e.g., taken by the
user using the mobile client 100), and a user-specified mapping of
a gesture to the customized hand gesture.
[0027] The network 150 transmits data between the mobile client 100
and the AR system 110. The network 150 may be a local area and/or
wide area network that uses wired and/or wireless communication
systems, such as the internet. In some embodiments, the network 150
includes encryption capabilities to ensure the security of data,
such as secure sockets layer (SSL), transport layer security (TLS),
virtual private networks (VPNs), internet protocol security
(IPsec), etc.
Example Gesture Tracking Application
[0028] FIG. 2 is a block diagram of the gesture tracking
application 130 of FIG. 1, in accordance with at least one example
embodiment. The gesture tracking application 130 includes a machine
learning model training engine 210, a gesture tracking module 220,
an AR object state module 230, and a rendering module 240. The
gesture tracking module 220 further includes, a gesture detection
model 221 and a hand detection model 222. In some embodiments, the
gesture tracking application 130 includes modules other than those
shown in FIG. 2. The modules may be embodied as program code (e.g.,
software comprised of instructions stored on non-transitory
computer readable storage medium and executable by at least one
processor such as the processor 602 in FIG. 6) and/or hardware
(e.g., application specific integrated circuit (ASIC) chips or
field programmable gate arrays (FPGA) with firmware. The modules
correspond to at least having the functionality described when
executed/operated.
[0029] The process of detecting a gesture and modifying the state
of an AR object based on the detected gesture may begin with the
gesture tracking module 220 receiving an image from the mobile
client 100. In one example, the image may be taken by camera 102
and transmitted to the gesture tracking application 130 by the AR
client 101 to determine whether a gesture is depicted within the
image to control an AR object. The gesture tracking module 220 may
apply one or more of trained models such as the gesture detection
model 221 and the hand detection model 222 to determine the gesture
depicted in the received image. The models may be trained by the
machine learning model training engine 210. After classifying the
gesture (e.g., determining the user is making a five), the gesture
tracking module 220 may provide the classification to the AR object
state module 230, which subsequently determines a state in which an
AR object should be rendered. The AR object state module 230
provides the determined state to the rendering module 240, which
may request an AR object in a particular state from the AR engine
120 and provide the AR object received from the AR engine 120 at
the screen 103 of the mobile client 100.
[0030] The machine learning model training engine 210 applies
training data sets to the gesture detection model 221 or the hand
detection model 222. The training engine 210 may create training
data sets based on data from the database 140. The training data
sets may include positive or negative samples of hand gestures or
hands. The training data sets may be labeled according to the
presence, or lack thereof, of a hand gesture or hand. The labels
may be provided to the training engine 210 from a user (e.g., using
a user input interface of the mobile client 100). This may enable
the training engine 210 to train, for example, the gesture
detection model 221 to classify an image of a custom user gesture
according to a user-specified label.
[0031] For the gesture detection model 221 to classify customized
gestures, the machine learning model training engine 210 may create
a training data set using images of a custom gesture. The gesture
tracking module 220 may prompt (e.g., through the screen 103) the
user to make a custom gesture and instruct the user to capture
multiple images (e.g., in different positions or angles) of the
gesture using the camera 102. The gesture tracking module 220
receives the captured images and transmit them to the database 140
for storage. The machine learning model training engine 210 may use
the captured images to train the gesture detection model 221 or the
hand detection model 222.
[0032] The machine learning model training engine 210 may train a
machine learning model in multiple stages. In a first stage, the
training engine 210 may use generalized data representing hands or
hand gestures taken from multiple users. In a second stage, the
training engine 210 may use user-specific data representing the
hand or hand gestures of the user of mobile client 100 to further
optimize the gesture tracking performed by the gesture tracking
module 220 to a user.
[0033] The gesture detection model 221 and the hand detection model
222 may be machine learning models. The models 221 and 222 may use
various machine learning techniques such as linear support vector
machine (linear SVM), boosting for other algorithms (e.g.,
AdaBoost), neural networks, logistic regression, naive Bayes,
memory-based learning, random forests, bagged trees, decision
trees, boosted trees, boosted stumps, a supervised or unsupervised
learning algorithm, or any suitable combination thereof.
[0034] The gesture detection model 221 classifies gestures within
images or videos. The gesture detection model 221 is trained using
various images of one or more hand gestures (e.g., by machine
learning model training engine 210). In some embodiments, data
representing an image or video captured by the camera 102 is input
into the gesture detection model 221. The gesture detection model
221 classifies one or more objects within the image or video. For
example, the gesture detection model 221 classifies a hand making a
five within an image whose data is input into the model. In some
embodiments, the gesture detection model 221 detects a gesture
performed over time (e.g., a hand wave). For example, consecutively
captured images are input into the gesture detection model 221 to
determine that the images represent a five at varying positions
within the camera view, which the gesture tracking module 220 may
then classify as a hand wave.
[0035] The hand detection model 222 classifies the presence of
hands within images or videos. The hand detection model 222 is
trained using various images of hands (e.g., by machine learning
model training engine 210). Similar to the gesture detection model
221, data representing an image or video captured by the camera 102
may be input into the hand detection model 222. The hand detection
model 222 classifies an object within the image or video as a hand
or outputs an indication that no hand was detected. In some
embodiments, the hand detection model 222 uses clusters of feature
points identified from the camera view, where the feature points
are associated with a physical object in the environment and the
cluster corresponds to a physical surface of the object (e.g., a
hand). The hand detection model 222 may use the feature points
(e.g., a combination of three-dimensional Cartesian coordinates and
the corresponding depths) to identify clusters corresponding to a
hand or to what is not a hand.
[0036] The gesture tracking module 220 may use the output of the
gesture detection model 221 or the hand detection model 222 to
generate instructions or prompts for display to the user (e.g., via
the screen 103 of the mobile client 100). The instructions may
include a suggested movement that the user should make or position
that the user should be in within the camera view of the mobile
client 100 to detect a hand gesture. For example, the gesture
tracking module 220 may generate an outline of a hand or hand
gesture on the screen 103 to guide a user to form a hand gesture.
The camera 102 captures images of the user's hand as it aligns with
the displayed guide. The captured images may be input into the
gesture detection model 221 or the hand detection model 222. If a
gesture or hand is not detected within the images, the gesture
tracking module 220 may generate a notification at the mobile
client 100 that the detection failed, instructions to guide the
user to make a desired hand gesture, or a combination thereof.
[0037] One or more of the gesture detection model 221 or the hand
detection model 222 may output a classification based on a success
threshold. The models may determine that the likelihood of a
classification (e.g., a confidence score) of a particular gesture
or of the presence of hand must meet or exceed a success threshold.
For example, the gesture detection model 221 determines that a
"five" gesture is being made with 60% likelihood and determines,
based on a success threshold of 90%, that there is not a "five"
gesture present in the input image or video. In another example,
the hand detection model 222 determines that a hand is present with
95% likelihood and determines, based on a success threshold of 80%,
that there is indeed a hand present in the input image or video.
The success threshold for each model of the gesture tracking module
220 may be different. In some embodiments, a success threshold may
be user-specified or adjustable. For example, a user may lower the
success threshold used by the models of the gesture tracking module
220 to increase the likelihood that his hand gesture will be
identified. In this example, the user may increase the flexible
afforded when the user's hand is not placed within the camera view
at a sufficiently proper angle or position, and a model may be more
likely to determine the user's gesture as one that the user is
indeed making.
[0038] In some embodiments, the gesture detection model 221 and the
hand detection model 222 may be one model. For example, the gesture
detection model 221 may be used to both detect the presence of a
hand within an image and classify the gesture made by the present
hand. In some embodiments, use of the hand detection model 222 may
improve processing bandwidth and thus, power consumption, of the
mobile client 100 by enabling the gesture tracking application 130
(e.g., the gesture tracking module 220) to discard image frames of
videos where a hand is not present. Video processing consumes a
large amount of a mobile client's processing bandwidth, and by
discarding frames that do not have information relevant to the
gesture tracking application 130, subsequent processing of image
frames are reserved for images that contain valuable information
(e.g., a hand gesture being made).
[0039] The application of both the hand detection model 222 and the
gesture detection model 221 may reduce the processing cycles and
power consumption of the mobile client 100. In particular, applying
the hand detection model 222 to an image before applying the
gesture detection model may save processing cycles. In some
embodiments, the gesture detection model 221 may be able to
classify a large number of gestures (e.g., both still and moving
gestures) and thus, requires complex processing with each
application to an image in order to determine which of the many
gestures is present within the image. By contrast, the hand
detection model 222 may be simpler, as detecting a hand by its
outline may be simpler than determining the gesture being made.
Accordingly, the initial application of the hand detection model
222 that requires less processing bandwidth prior to the subsequent
application, if the hand has been detected by the model 222, of the
gesture detection model 221 that requires more processing bandwidth
allows the mobile client 100 to reserve its processing
resources.
[0040] The AR object state module 230 may hold in memory the
current state of the AR object or previous states of the AR objects
to determine a subsequent state based on the user's hand gesture
and the current or previous states of the AR object. A conditional
decision algorithm may be used by the AR object state module 230 to
determine a state to instruct the rendering module 240 to render
the AR object in. The AR Object State Module 240 may receive custom
mappings of states for AR objects and hand gestures from the user.
For example, a user may specify that an AR object corresponding to
a shield is to be rendered during game play when a user waves his
hand and that the rendering module 240 should stop rendering the
shield when a user waves his hand again. States of an AR object and
the application of the AR object state module 230 in determining
those states using detected hand gestures is further described in
the descriptions of FIGS. 4, 5A, and 5B.
[0041] The rendering module 240 provides for display, on the mobile
client 100, an augmented reality (AR) object that may be controlled
by a user's gestures. The AR engine 120 generates the AR object.
When the AR object is at a location on a virtual coordinate space
that represents the surfaces within the environment captured by the
camera view of the camera 102, the rendering module 240 displays
the AR object in a state based on a detected hand gesture. As
referred to herein, a "state" of an AR object refers to a position,
angle, shape, or any suitable condition of appearance that may
change over time. For example, the rendering module 240 may render
a ball in a first state where the ball is not aflame or in a second
state where the ball is aflame. In another example, the rendering
module 240 may render the ball in a first state where the ball is
located at a first set of Cartesian coordinates and a first depth
in a virtual coordinate space corresponding to the environment or
in a second state where the ball is located at a second set of
Cartesian coordinates and a second depth in the virtual coordinate
space.
Processes for Controlling Ar Objects Using Gesture Detection
[0042] FIG. 3 is a flowchart illustrating a process 300 for
controlling an AR object using gesture detection, in accordance
with at least one example embodiment. The process 300 may be
performed by the gesture tracking application 130. The gesture
tracking application 130 may perform operations of the process 300
in parallel or in different orders, or may perform different,
additional, or fewer steps. For example, prior to receiving 302 the
image, the gesture tracking application 130 may generate for
display on the mobile client an instruction to position a user's
hand within a certain area of the camera view of the mobile client
(e.g., in order to successfully capture the hand within the
image).
[0043] The gesture tracking application 130 receives 302, from a
mobile client, an image from a camera view of an environment, where
the image depicts a portion of a body of a user. The gesture
tracking module 220 may perform the receiving 302. For example, the
mobile client 100 provides an image of a user's hand, from the
camera view captured by the camera 102, within an environment of
the user's living room.
[0044] The gesture tracking application 130 provides 304 the image
to a machine learning model configured to identify a formation of
the portion of the body. For example, the gesture tracking module
220 may apply the image data of the received 302 image to the
gesture detection model 221. The gesture detection model 221 may
identify a formation being made by the hand within the image and
classify the formation into one of multiple potential formations
(e.g., as specified by a user through labels corresponding to the
potential formations that were used to train the gesture detection
model 221).
[0045] The gesture tracking application 130 provides 306 for
display on the mobile client, based on an identification of the
formation by the machine learning model, an AR object in the camera
view of the environment. The rendering module 240 may perform the
providing 306. The gesture tracking application 130 may determine a
state that the AR object is to be rendered in. For example, the AR
object state module 230 uses a conditional decision tree that
indicates that, if a particular formation is identified, then the
rendering module 240 is to display a corresponding AR object and in
a corresponding state.
[0046] FIG. 4 is a flowchart illustrating a process 400 for
controlling the AR object using gesture detection based on the
process 300 of FIG. 3, in accordance with at least one example
embodiment. The process 400 includes subprocesses of the process
300. Like the process 300, the process 400 may be performed by the
gesture tracking application 130. The gesture tracking application
130 may perform operations of the process 400 in parallel or in
different orders, or may perform different, additional, or fewer
steps. For example, after determining 406 what formation was
identified, the gesture tracking application 130 may access a
previous formation identified and determine, using both the
previous and the current formation, a state in which to generate
the AR object.
[0047] The gesture tracking application 130 receives 402, from a
mobile client, an image from a camera view of an environment, the
image depicting a hand of a user. The hand captured within the
received 402 image is one example of a portion of the body of the
user received 302 in the process 300.
[0048] The gesture tracking application 130 applies 404 a machine
learning model to the image, the machine learning model trained on
training image data representative of hand formations, the machine
learning model configured to identify a formation of the hand in
the image as one of the hand formations. The machine learning model
applied 404 is one example of a machine learning model applied 304
in the process 300. The machine learning model may be a
convolutional neural network trained (e.g., by the machine learning
model training engine 210) using various images of hands making
particular formations (e.g., fists and fives). In one example, if
the image received 402 depicts the hand making a fist, the machine
learning model may output a classification that the identified
formation is a first (e.g., palm covered over by fingers curled
in).
[0049] The gesture tracking application 130 determines 406 whether
a first or a second hand formation is identified by the machine
learning model. The gesture tracking application may make this
determination using a mapping between formations and AR object
states, as described in the description of the AR object state
module 230. If a first hand formation is identified, the gesture
tracking application 130 provides 408, for display on the mobile
client, the AR object in a first state from the AR engine in the
camera view of the environment. If a second hand formation is
identified, the gesture tracking application 130 provides 410, for
display on the mobile client, the AR object in a second state from
the AR engine in the camera view of the environment. To provide the
AR object for display in a particular state, the gesture tracking
application may use the rendering module 240 which receives
instructions to render a particular AR object in a particular state
from the AR object state module 230 and transmits instructions to
the AR engine 120 to render the AR object accordingly at the screen
103 of the mobile client 100. The determination 406 and either
provided 408 or 410 AR object for display may be one example of the
provided 306 AR object for display of the process 300.
Example AR Application with Gesture Tracking to Control AR
Objects
[0050] FIGS. 5A and 5B illustrate user interactions with an AR
application that integrates gesture tracking to control AR objects,
in accordance with at least one embodiment. FIG. 5A shows a first
user interaction 500a where a user's hand 510a is in a first state
and is captured, by the mobile client 100, within an image (e.g.,
camera-captured hand 510b). The gesture tracking application 130
renders an AR object 520 (e.g., a ball) for display on the mobile
client 100 such that the AR object 520 appears integrated into the
environment with the user's hand 510a captured within the camera
view.
[0051] In the first user interaction 500a, the AR object 520 is
rendered for display (e.g., by the rendering module 240) in a first
state where it is at a position overlaying the user's hand. The
gesture tracking application 130 may determine a position within a
virtual coordinate space that the user's hand 510a is located and
determine a corresponding location to render the AR object 520
(e.g., a set of coordinates causing the AR object 520 to appear
above the user's hand 510a).
[0052] The gesture tracking application 130 provides a gesture
indicator 530 for display that indicates the gesture identified by
the gesture tracking application (e.g., by the gesture detection
model 221). In the first user interaction 500a, the gesture
indicator 530 indicates that the user's hand 510a is making a five.
The gesture tracking application 130 may highlight, circle, or
otherwise visually distinguish a gesture indicator from other
indicators to inform the user of presently detected gestures within
the camera view.
[0053] FIG. 5B shows a second user interaction 500b where the
user's hand 510a is in a second state and is captured, by the
mobile client 100, within an image (e.g., camera-captured hand
510b). The gesture tracking application 130 provides a gesture
indicator 540 for display that indicates the gesture identified by
the gesture detection model 221 is a first during the second user
interaction 500b. The gesture tracking application 130 renders the
AR object 520 for display in a second state where the object
appears to move upward within the camera view, as indicated by the
state change indicator 550 included in FIG. 5B for clarity and not
necessarily rendered by the gesture tracking application 130 for
display to the user. In some embodiments, the gesture tracking
application 130 may use a previous state and the present gesture of
a user's hand to determine a subsequent state in which to render an
AR object (e.g., using a mapping table or conditional decision
tree). For example, the gesture tracking application 130 determines
that a combination of the identified first and the existing
position of the AR object 520 over the user's hand indicates that
the next state in which the AR object is to be rendered is
appearing to move upward.
Computing Machine Architecture
[0054] FIG. 6 is a block diagram illustrating components of an
example machine able to read instructions from a machine-readable
medium and execute them in a processor (or controller).
Specifically, FIG. 6 shows a diagrammatic representation of a
machine in the example form of a computer system 600 within which
program code (e.g., software) for causing the machine to perform
any one or more of the methodologies discussed herein may be
executed. The program code may correspond to functional
configuration of the modules and/or processes described with FIGS.
1-5B. The program code may be comprised of instructions 624
executable by one or more processors 602. In alternative
embodiments, the machine operates as a standalone device or may be
connected (e.g., networked) to other machines. In a networked
deployment, the machine may operate in the capacity of a server
machine or a client machine in a server-client network environment,
or as a peer machine in a peer-to-peer (or distributed) network
environment.
[0055] The machine may be a portable computing device or machine
(e.g., smartphone, tablet, wearable device (e.g., smartwatch))
capable of executing instructions 624 (sequential or otherwise)
that specify actions to be taken by that machine. Further, while
only a single machine is illustrated, the term "machine" shall also
be taken to include any collection of machines that individually or
jointly execute instructions 624 to perform any one or more of the
methodologies discussed herein.
[0056] The example computer system 600 includes a processor 602
(e.g., a central processing unit (CPU), a graphics processing unit
(GPU), a digital signal processor (DSP), one or more application
specific integrated circuits (ASICs), one or more radio-frequency
integrated circuits (RFICs), or any combination of these), a main
memory 604, and a static memory 606, which are configured to
communicate with each other via a bus 608. The computer system 600
may further include visual display interface 610. The visual
interface may include a software driver that enables displaying
user interfaces on a screen (or display). The visual interface may
display user interfaces directly (e.g., on the screen) or
indirectly on a surface, window, or the like (e.g., via a visual
projection unit). For ease of discussion the visual interface may
be described as a screen. The visual interface 610 may include or
may interface with a touch enabled screen. The computer system 600
may also include alphanumeric input device 612 (e.g., a keyboard or
touch screen keyboard), a cursor control device 614 (e.g., a mouse,
a trackball, a joystick, a motion sensor, or other pointing
instrument), a storage unit 616, a signal generation device 618
(e.g., a speaker), and a network interface device 620, which also
are configured to communicate via the bus 608.
[0057] The storage unit 616 includes a machine-readable medium 622
on which is stored instructions 624 (e.g., software) embodying any
one or more of the methodologies or functions described herein. The
instructions 624 (e.g., software) may also reside, completely or at
least partially, within the main memory 604 or within the processor
602 (e.g., within a processor's cache memory) during execution
thereof by the computer system 600, the main memory 604 and the
processor 602 also constituting machine-readable media. The
instructions 624 (e.g., software) may be transmitted or received
over a network 626 via the network interface device 620.
[0058] While machine-readable medium 622 is shown in an example
embodiment to be a single medium, the term "machine-readable
medium" should be taken to include a single medium or multiple
media (e.g., a centralized or distributed database, or associated
caches and servers) able to store instructions (e.g., instructions
624). The term "machine-readable medium" shall also be taken to
include any medium that is capable of storing instructions (e.g.,
instructions 624) for execution by the machine and that cause the
machine to perform any one or more of the methodologies disclosed
herein. The term "machine-readable medium" includes, but not be
limited to, data repositories in the form of solid-state memories,
optical media, and magnetic media.
Additional Configuration Considerations
[0059] While using an AR application, a user may desire a more
seamless experience by controlling AR objects with his hands (e.g.,
using hand gestures), as the user is accustomed to doing in
reality. While tracking hand gestures for conventional AR systems
achieve the user's desire with dedicated, handheld controllers, a
wired power source, and enough hardware real-estate to accommodate
for powerful, power-hungry processors, mobile clients do not share
those similar specifications to afford gesture tracking in that
conventional manner. Rather, a mobile device is limited in its
power and processing resources. The embodiments herein optimize for
a mobile device's power and processing constraints by limiting
image frames processed during the gesture detection (e.g.,
discarding image frames that a machine learning model determines
does not depict a hand). Thus, the methods described herein enable
gesture tracking for AR object control on mobile client rendered AR
systems without consuming excessive amounts of processing power and
present an immersive AR experience to the user.
[0060] Throughout this specification, plural instances may
implement components, operations, or structures described as a
single instance. Although individual operations of one or more
methods are illustrated and described as separate operations, one
or more of the individual operations may be performed concurrently,
and nothing requires that the operations be performed in the order
illustrated. Structures and functionality presented as separate
components in example configurations may be implemented as a
combined structure or component. Similarly, structures and
functionality presented as a single component may be implemented as
separate components. These and other variations, modifications,
additions, and improvements fall within the scope of the subject
matter herein.
[0061] Certain embodiments are described herein as including logic
or a number of components, modules, or mechanisms. Modules may
constitute either software modules (e.g., code embodied on a
machine-readable medium or in a transmission signal) or hardware
modules. A hardware module is tangible unit capable of performing
certain operations and may be configured or arranged in a certain
manner. In example embodiments, one or more computer systems (e.g.,
a standalone, client or server computer system) or one or more
hardware modules of a computer system (e.g., a processor or a group
of processors) may be configured by software (e.g., an application
or application portion) as a hardware module that operates to
perform certain operations as described herein.
[0062] In various embodiments, a hardware module may be implemented
mechanically or electronically. For example, a hardware module may
comprise dedicated circuitry or logic that is permanently
configured (e.g., as a special-purpose processor, such as a field
programmable gate array (FPGA) or an application-specific
integrated circuit (ASIC)) to perform certain operations. A
hardware module may also comprise programmable logic or circuitry
(e.g., as encompassed within a general-purpose processor or other
programmable processor) that is temporarily configured by software
to perform certain operations. It will be appreciated that the
decision to implement a hardware module mechanically, in dedicated
and permanently configured circuitry, or in temporarily configured
circuitry (e.g., configured by software) may be driven by cost and
time considerations.
[0063] Accordingly, the term "hardware module" should be understood
to encompass a tangible entity, be that an entity that is
physically constructed, permanently configured (e.g., hardwired),
or temporarily configured (e.g., programmed) to operate in a
certain manner or to perform certain operations described herein.
As used herein, "hardware-implemented module" refers to a hardware
module. Considering embodiments in which hardware modules are
temporarily configured (e.g., programmed), each of the hardware
modules need not be configured or instantiated at any one instance
in time. For example, where the hardware modules comprise a
general-purpose processor configured using software, the
general-purpose processor may be configured as respective different
hardware modules at different times. Software may accordingly
configure a processor, for example, to constitute a particular
hardware module at one instance of time and to constitute a
different hardware module at a different instance of time.
[0064] Hardware modules can provide information to, and receive
information from, other hardware modules. Accordingly, the
described hardware modules may be regarded as being communicatively
coupled. Where multiple of such hardware modules exist
contemporaneously, communications may be achieved through signal
transmission (e.g., over appropriate circuits and buses) that
connect the hardware modules. In embodiments in which multiple
hardware modules are configured or instantiated at different times,
communications between such hardware modules may be achieved, for
example, through the storage and retrieval of information in memory
structures to which the multiple hardware modules have access. For
example, one hardware module may perform an operation and store the
output of that operation in a memory device to which it is
communicatively coupled. A further hardware module may then, at a
later time, access the memory device to retrieve and process the
stored output. Hardware modules may also initiate communications
with input or output devices, and can operate on a resource (e.g.,
a collection of information).
[0065] The various operations of example methods described herein
may be performed, at least partially, by one or more processors
that are temporarily configured (e.g., by software) or permanently
configured to perform the relevant operations. Whether temporarily
or permanently configured, such processors may constitute
processor-implemented modules that operate to perform one or more
operations or functions. The modules referred to herein may, in
some example embodiments, comprise processor-implemented
modules.
[0066] Similarly, the methods described herein may be at least
partially processor-implemented. For example, at least some of the
operations of a method may be performed by one or processors or
processor-implemented hardware modules. The performance of certain
of the operations may be distributed among the one or more
processors, not only residing within a single machine, but deployed
across a number of machines. In some example embodiments, the
processor or processors may be located in a single location (e.g.,
within a home environment, an office environment or as a server
farm), while in other embodiments the processors may be distributed
across a number of locations.
[0067] The one or more processors may also operate to support
performance of the relevant operations in a "cloud computing"
environment or as a "software as a service" (SaaS). For example, at
least some of the operations may be performed by a group of
computers (as examples of machines including processors), these
operations being accessible via a network (e.g., the Internet) and
via one or more appropriate interfaces (e.g., application program
interfaces (APIs).)
[0068] The performance of certain of the operations may be
distributed among the one or more processors, not only residing
within a single machine, but deployed across a number of machines.
In some example embodiments, the one or more processors or
processor-implemented modules may be located in a single geographic
location (e.g., within a home environment, an office environment,
or a server farm). In other example embodiments, the one or more
processors or processor-implemented modules may be distributed
across a number of geographic locations.
[0069] Some portions of this specification are presented in terms
of algorithms or symbolic representations of operations on data
stored as bits or binary digital signals within a machine memory
(e.g., a computer memory). These algorithms or symbolic
representations are examples of techniques used by those of
ordinary skill in the data processing arts to convey the substance
of their work to others skilled in the art. As used herein, an
"algorithm" is a self-consistent sequence of operations or similar
processing leading to a desired result. In this context, algorithms
and operations involve physical manipulation of physical
quantities. Typically, but not necessarily, such quantities may
take the form of electrical, magnetic, or optical signals capable
of being stored, accessed, transferred, combined, compared, or
otherwise manipulated by a machine. It is convenient at times,
principally for reasons of common usage, to refer to such signals
using words such as "data," "content," "bits," "values,"
"elements," "symbols," "characters," "terms," "numbers,"
"numerals," or the like. These words, however, are merely
convenient labels and are to be associated with appropriate
physical quantities.
[0070] Unless specifically stated otherwise, discussions herein
using words such as "processing," "computing," "calculating,"
"determining," "presenting," "displaying," or the like may refer to
actions or processes of a machine (e.g., a computer) that
manipulates or transforms data represented as physical (e.g.,
electronic, magnetic, or optical) quantities within one or more
memories (e.g., volatile memory, non-volatile memory, or a
combination thereof), registers, or other machine components that
receive, store, transmit, or display information.
[0071] As used herein any reference to "one embodiment" or "an
embodiment" means that a particular element, feature, structure, or
characteristic described in connection with the embodiment is
included in at least one embodiment. The appearances of the phrase
"in one embodiment" in various places in the specification are not
necessarily all referring to the same embodiment.
[0072] Some embodiments may be described using the expression
"coupled" and "connected" along with their derivatives. It should
be understood that these terms are not intended as synonyms for
each other. For example, some embodiments may be described using
the term "connected" to indicate that two or more elements are in
direct physical or electrical contact with each other. In another
example, some embodiments may be described using the term "coupled"
to indicate that two or more elements are in direct physical or
electrical contact. The term "coupled," however, may also mean that
two or more elements are not in direct contact with each other, but
yet still co-operate or interact with each other. The embodiments
are not limited in this context.
[0073] As used herein, the terms "comprises," "comprising,"
"includes," "including," "has," "having" or any other variation
thereof, are intended to cover a non-exclusive inclusion. For
example, a process, method, article, or apparatus that comprises a
list of elements is not necessarily limited to only those elements
but may include other elements not expressly listed or inherent to
such process, method, article, or apparatus. Further, unless
expressly stated to the contrary, "or" refers to an inclusive or
and not to an exclusive or. For example, a condition A or B is
satisfied by any one of the following: A is true (or present) and B
is false (or not present), A is false (or not present) and B is
true (or present), and both A and B are true (or present).
[0074] In addition, use of the "a" or "an" are employed to describe
elements and components of the embodiments herein. This is done
merely for convenience and to give a general sense of the
invention. This description should be read to include one or at
least one and the singular also includes the plural unless it is
obvious that it is meant otherwise.
[0075] Upon reading this disclosure, those of skill in the art will
appreciate still additional alternative structural and functional
designs for a system and a process for gesture tracking in an
augmented reality environment executed on a mobile client through
the disclosed principles herein. Thus, while particular embodiments
and applications have been illustrated and described, it is to be
understood that the disclosed embodiments are not limited to the
precise construction and components disclosed herein. Various
modifications, changes and variations, which will be apparent to
those skilled in the art, may be made in the arrangement, operation
and details of the method and apparatus disclosed herein without
departing from the spirit and scope defined in the appended
claims.
* * * * *