U.S. patent application number 15/868167 was filed with the patent office on 2018-05-17 for virtual dressing room.
The applicant listed for this patent is eBay Inc.. Invention is credited to Eric J. Farraro, Krystal Rose Higgins, Kumaresan Manickavelu, Saurav Mukherjee, John Tapley.
Application Number | 20180137515 15/868167 |
Document ID | / |
Family ID | 50025026 |
Filed Date | 2018-05-17 |
United States Patent
Application |
20180137515 |
Kind Code |
A1 |
Higgins; Krystal Rose ; et
al. |
May 17, 2018 |
VIRTUAL DRESSING ROOM
Abstract
A method and system facilitate recognition of commands to
initiate actions within an electronic marketplace on behalf of a
user. A machine detects a first command issued by a user. The
machine, based on the detecting of the first command, identifies a
request to generate a model of a physical object in the environment
of the user. The machine generates the model based on a set of
spatial data pertaining to the physical object. The machine, based
on the model, causes a display of a graphical representation of the
physical object in a user interface of a client device. The machine
identifies a request to initiate an action with respect to the
physical object via the client device on behalf of the user. The
machine automatically initiates the action via the client device on
behalf of the user. The action uses the graphical representation of
the physical object.
Inventors: |
Higgins; Krystal Rose;
(Campbell, CA) ; Farraro; Eric J.; (San Jose,
CA) ; Tapley; John; (San Jose, CA) ;
Manickavelu; Kumaresan; (Port Blair, IN) ; Mukherjee;
Saurav; (Kolkata, IN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
eBay Inc. |
San Jose |
CA |
US |
|
|
Family ID: |
50025026 |
Appl. No.: |
15/868167 |
Filed: |
January 11, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13679498 |
Nov 16, 2012 |
9898742 |
|
|
15868167 |
|
|
|
|
61679630 |
Aug 3, 2012 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 2210/16 20130101;
G06Q 30/0641 20130101; G06T 19/00 20130101; G06T 17/00 20130101;
G06Q 30/00 20130101 |
International
Class: |
G06Q 30/00 20060101
G06Q030/00; G06Q 30/06 20060101 G06Q030/06; G06T 17/00 20060101
G06T017/00; G06T 19/00 20060101 G06T019/00 |
Claims
1. (canceled)
2. A method comprising: detecting, using one or more hardware
processors, a first command issued by a user; based on the
detecting of the first command, identifying a request to generate a
model of a physical object in the environment of the user; in
response to the identifying of the request to generate the model,
generating the model of the physical object in the environment of
the user based on a set of spatial data pertaining to the physical
object; based on the generating of the model of the physical
object, causing a display of a graphical representation of the
physical object in a user interface of a client device; based on
detecting a second command issued by the user, identifying a
request to initiate an action with respect to the physical object
via the client device on behalf of the user; and in response to the
identifying of the request to initiate the action, automatically
initiating the action with respect to the physical object via the
client device on behalf of the user, the action using the graphical
representation of the physical object.
3. The method of claim 2, wherein the request to initiate the
action with the respect to the physical object via the client
device on behalf of the user includes a request to obtain input
regarding the physical object via an electronic marketplace.
4. The method of claim 3, wherein the input is a curated
recommendation pertaining to the physical object.
5. The method of claim 2, wherein the physical object is an item of
furniture, wherein the request to initiate the action with the
respect to the physical object via the client device on behalf of
the user includes a request to move the item of furniture to a
different location within the environment of the user, and wherein
the automatic initiating of the action with the respect to the
physical object via the client device on behalf of the user
includes modifying a location of the graphical representation of
the physical object in the user interface of the client device.
6. The method of claim 2, further comprising: determining one or
more measurements of the physical object based on the set of
spatial data pertaining to the physical object; and storing, in a
database record, the one or more measurements of the physical
object in association with at least one of an identifier of the
physical object, the graphical representation of the physical
object, an identifier of the user, or an identifier of the client
device.
7. The method of claim 2, wherein the second command issued by the
user includes a command to list the physical object for sale on an
electronic marketplace, and wherein the method further comprises:
based on the second command, generating a listing of the physical
object; and publishing the listing of the physical object on the
electronic marketplace.
8. The method of claim 7, wherein the listing of the physical
object includes the graphical representation of the physical
object.
9. The method of claim 2, wherein the graphical representation
includes a three-dimensional representation of the physical
object.
10. The method of claim 2, wherein the causing of the display of
the graphical representation of the physical object in the user
interface of the client device includes: accessing, at a record of
a database, one or more measurements of the physical object; and
causing a display of the one or more measurements of the physical
object in association with the graphical representation of the
physical object in the user interface of the client device, wherein
the request to initiate the action with the respect to the physical
object via the client device on behalf of the user includes a
request to obtain input regarding the physical object via an
electronic marketplace, and wherein the method further comprises
receiving a curated recommendation pertaining to a further physical
object based on the graphical representation of the physical object
and the one or more measurements of the physical object.
11. The method of claim 2, wherein at least one of the first
command or the second command includes a gesture, by the user,
pertaining to the physical object.
12. A system comprising: one or more hardware processors; and a
non-transitory machine-readable medium for storing instructions
that, when executed by the one or more hardware processors, cause
the one or more hardware processors to perform operations
comprising: detecting a first command issued by a user; based on
the detecting of the first command, identifying a request to
generate a model of a physical object in the environment of the
user; in response to the identifying of the request to generate the
model, generating the model of the physical object in the
environment of the user based on a set of spatial data pertaining
to the physical object; based on the generating of the model of the
physical object, causing a display of a graphical representation of
the physical object in a user interface of a client device; based
on detecting a second command issued by the user, identifying a
request to initiate an action with respect to the physical object
via the client device on behalf of the user; and in response to the
identifying of the request to initiate the action, automatically
initiating the action with respect to the physical object via the
client device on behalf of the user, the action using the graphical
representation of the physical object.
13. The system of claim 12, wherein the request to initiate the
action with the respect to the physical object via the client
device on behalf of the user includes a request to obtain input
regarding the physical object via an electronic marketplace.
14. The system of claim 13, wherein the input is a curated
recommendation pertaining to the physical object.
15. The system of claim 12, wherein the physical object is an item
of furniture, wherein the request to initiate the action with the
respect to the physical object via the client device on behalf of
the user includes a request to move the item of furniture to a
different location within the environment of the user, and wherein
the automatic initiating of the action with the respect to the
physical object via the client device on behalf of the user
includes modifying a location of the graphical representation of
the physical object in the user interface of the client device.
16. The system of claim 12, wherein the operations further
comprise: determining one or more measurements of the physical
object based on the set of spatial data pertaining to the physical
object; and storing, in a database record, the one or more
measurements of the physical object in association with at least
one of an identifier of the physical object, the graphical
representation of the physical object, an identifier of the user,
or an identifier of the client device.
17. The system of claim 12, wherein the second command issued by
the user includes a command to list the physical object for sale on
an electronic marketplace, and wherein the operations further
comprise: based on the second command, generating a listing of the
physical object; and publishing the listing of the physical object
on the electronic marketplace.
18. The system of claim 17, wherein the listing of the physical
object includes the graphical representation of the physical
object.
19. The system of claim 12, wherein the graphical representation
includes a three-dimensional representation of the physical
object.
20. The system of claim 12, wherein the causing of the display of
the graphical representation of the physical object in the user
interface of the client device includes: accessing, at a record of
a database, one or more measurements of the physical object; and
causing a display of the one or more measurements of the physical
object in association with the graphical representation of the
physical object in the user interface of the client device, wherein
the request to initiate the action with the respect to the physical
object via the client device on behalf of the user includes a
request to obtain input regarding the physical object via an
electronic marketplace, and wherein the operations further comprise
receiving a curated recommendation pertaining to a further physical
object based on the graphical representation of the physical object
and the one or more measurements of the physical object.
21. A non-transitory machine-readable storage medium comprising
instructions that, when executed by one or more hardware processors
of a machine, cause the one or more hardware processors to perform
operations comprising: detecting a first command issued by a user;
based on the detecting of the first command, identifying a request
to generate a model of a physical object in the environment of the
user; in response to the identifying of the request to generate the
model, generating the model of the physical object in the
environment of the user based on a set of spatial data pertaining
to the physical object; based on the generating of the model of the
physical object, causing a display of a graphical representation of
the physical object in a user interface of a client device; based
on detecting a second command issued by the user, identifying a
request to initiate an action with respect to the physical object
via the client device on behalf of the user; and in response to the
identifying of the request to initiate the action, automatically
initiating the action with respect to the physical object via the
client device on behalf of the user, the action using the graphical
representation of the physical object.
Description
RELATED APPLICATION
[0001] The present application is a continuation of and claims the
benefit of priority to U.S. patent application Ser. No. 13/679,498,
filed Nov. 16, 2012, which claims the benefit of priority to U.S.
Provisional Patent Application No. 61/679,630, filed Aug. 3, 2012,
which are hereby incorporated by reference herein in their
entirety.
TECHNICAL FIELD
[0002] The subject matter disclosed herein generally relates to the
processing of data. Specifically, the present disclosure addresses
systems and methods to facilitate provision of a virtual dressing
room.
BACKGROUND
[0003] A number of e-commerce sites exist on the Internet. An
electronic marketplace, such as eBay.com, is an example of such an
e-commerce site. Many of these e-commerce sites facilitate
commercial transactions between users of these sites. Some users
may list items for sale on such e-commerce sites, while other users
may buy or make bids for the items listed for sale.
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Some embodiments are illustrated by way of example and not
limitation in the figures of the accompanying drawings.
[0005] FIG. 1 is a chart illustrating examples of possible actions
in a virtual dressing room, according to some example
embodiments.
[0006] FIG. 2 is a conceptual diagram discussing an initial welcome
state, according to some example embodiments.
[0007] FIGS. 3-4 are conceptual diagrams discussing a virtual rack
of clothes, according to some example embodiments.
[0008] FIG. 5 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0009] FIG. 6 is a conceptual diagram illustrating an action,
according to some example embodiments.
[0010] FIG. 7 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0011] FIG. 8 is a conceptual diagram illustrating a gesture and an
action, according to some example embodiments.
[0012] FIG. 9 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0013] FIG. 10 is a conceptual diagram illustrating an action,
according to some example embodiments.
[0014] FIG. 11 is a conceptual diagram illustrating gestures,
according to some example embodiments.
[0015] FIG. 12 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0016] FIGS. 13-14 are conceptual diagrams illustrating an action,
according to some example embodiments.
[0017] FIG. 15 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0018] FIG. 16 is a conceptual diagram illustrating an action,
according to some example embodiments.
[0019] FIG. 17 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0020] FIG. 18 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0021] FIGS. 19-20 are conceptual diagrams illustrating actions,
according to some example embodiments.
[0022] FIG. 21 is a conceptual diagram illustrating issuance of a
voice command, according to some example embodiments.
[0023] FIGS. 22-23 are conceptual diagrams illustrating the use of
the present method and system in the context of curated
recommendations and group virtual dressing rooms, according to some
example embodiments.
[0024] FIG. 24 is a conceptual diagram illustrating a position of a
user in relation to a depth sensor (e.g., Microsoft.TM. Kinect.TM.)
of the present method and system, according to some example
embodiments.
[0025] FIG. 25 is a conceptual diagram illustrating a gesture,
according to some example embodiments.
[0026] FIG. 26 is a conceptual diagram illustrating data flow in an
example system and a model, according to some example
embodiments.
[0027] FIG. 27 is a conceptual diagram illustrating the use of the
present method and system in the context of home and garden
decorating and curated recommendations, according to some example
embodiments.
[0028] FIG. 28 is a network diagram illustrating a network
environment suitable for action initiation on behalf of a user
based on gesture recognition, according to some example
embodiments.
[0029] FIG. 29 is a block diagram illustrating components of an
action machine suitable for action initiation on behalf of a user
based on gesture recognition, according to some example
embodiments.
[0030] FIGS. 30-34 are flowcharts illustrating operations of the
action machine in performing a method of action initiation on
behalf of a user based on gesture recognition, according to some
example embodiments.
[0031] FIG. 35 is a block diagram illustrating components of a
machine, according to some example embodiments, able to read
instructions from a machine-readable medium and perform any one or
more of the methodologies discussed herein.
DETAILED DESCRIPTION
[0032] Example methods and systems are directed to facilitating
provision of a virtual dressing room. Provision of a virtual
dressing room may include recognition of one or more gestures
representing commands to take actions, as well as initiating one or
more actions on behalf of a user, within an electronic marketplace.
Examples merely typify possible variations. Unless explicitly
stated otherwise, components and functions are optional and may be
combined or subdivided, and operations may vary in sequence or be
combined or subdivided. In the following description, for purposes
of explanation, numerous specific details are set forth to provide
a thorough understanding of example embodiments. It will be evident
to one skilled in the art, however, that the present subject matter
may be practiced without these specific details.
[0033] Some people shop online because on-line shopping may save
them time or increase their shopping options. Others want to avoid
the hassle of going to the mall, while other shoppers would rather
shop "on the go" (e.g., using mobile devices). However, what many
online clothing shoppers have in common is their desire to try on
the clothing they see online before they choose to buy an item. The
subject matter described herein may allow a user of a system, using
a gesture, to issue a command to the system to engage in (e.g., to
initiate) an action on an electronic marketplace such as eBay,
Inc., on behalf of the user.
[0034] One example of such action is the system displaying (e.g.,
on a TV screen, computer screen, or a mobile device screen) a
representation of a clothing item (hereinafter, also "virtual
clothing" or "item of virtual clothing") mapped to a representation
of the user, thus allowing the user to virtually try on the virtual
clothing item in a virtual dressing room. The user may also
virtually try on a virtual item that relates to clothing, such as
footwear or accessories (hereinafter, also "clothing") in the
virtual dressing room. In some example embodiments, the virtual
clothing that the user may virtually try on is representative of
real clothing available for purchase using an electronic
marketplace. The user may virtually try on the virtual clothing
item while the user is in his home (e.g., using the TV in his
living room) or away from home (e.g., using his mobile phone).
Another example of an action on an electronic marketplace is the
system initiating a purchase order using the electronic
marketplace, on behalf of the user, in response to receiving a
gesture from the user, the gesture representing a command to buy an
item.
[0035] With the use of skeletal and depth tracking technology, for
example implemented in depth sensors (e.g., the Microsoft.TM.
Kinect.TM., hereinafter, "Kinect", stereo cameras, mobile devices,
and any other device that may capture depth data) spatial data can
be gathered (e.g., by a machine of the system) about objects
located in the physical environment external to the depth sensor
(e.g., the user's living room). For example, an infrared
(hereinafter, also "IR") emitter of the system, located in the
user's living room, may project (e.g., emit or spray out) beams of
infrared light into surrounding space. The projected beams of IR
light may hit and reflect off objects that are located in their
path (e.g., the user or a physical object in the user's living
room). A depth sensor (e.g., located in the user's living room) may
capture (e.g., receive) spatial data about the surroundings of the
depth sensor based on the reflected beams of IR light. In some
example embodiments, the captured spatial data may be used to
create (e.g., represent, model, or define) a 3D field of view that
may be displayed on a screen (e.g., of a TV set, computer, or
mobile device). Examples of such spatial data include the location
and shape of the objects within the room where the spatial sensor
is located. In some example embodiments, based on measuring how
long it takes the beams of IR light to reflect off objects they
encounter in their path and be captured by the depth sensor, the
system may determine the location (e.g., the distance from the
depth sensor) of the objects off which the beams of IR light
reflected (e.g., the user, a furniture piece, or a wall). In
various example embodiments, based on the received spatial data,
the system may determine details of the objects in the room, such
as spatial measurements of the objects in the room (e.g., of the
user's body or of the furniture in the room). In some example
embodiments, based on the received spatial data, the system may
determine gestures made by the user. Some devices (e.g., that may
include a depth sensor and a camera) may detect other details of
the objects in the room (e.g., texture, color, or pattern of the
clothing worn by the user or of a wall of the room).
[0036] In various example embodiments, the system may generate a
first model based on a first set of spatial data received at a
first time. The first model may be representative of a user of an
electronic marketplace, at the first time. The system may also
generate a second model based on a second set of spatial data
received at a second time, and the second model may also be
representative of the same user, but at the second time. Then, the
system may detect a difference between the first model and the
second model. The system may also determine that the difference
between the first and second models corresponds to a gesture
performed by the user between the first and the second times, and
that the gesture represents a command to initiate an action within
the electronic marketplace on behalf of the user. In some example
embodiments, the system may prompt the user to confirm that the
gesture represents the command (e.g., the user intended to issue
the command when making the gesture). Based on the user confirming
that the gesture represents the command, the system may initiate
the action within the electronic marketplace on behalf of the user.
Examples of such actions that correspond to user gestures include
displaying a representation of a clothing item mapped to a
representation of the user, initiating a purchase order of an
instance of the clothing item whose representation has been mapped
to the representation of the user, listing an item for sale on the
electronic marketplace, bidding for an item available for purchase
on the electronic marketplace, and generating a representation of a
physical object displayed by the user to the depth sensor. Examples
of the user confirming that the gesture was intended as a command
to initiate an action include repeating the gesture, making a
different gesture, and issuing a voice command.
[0037] In various example embodiments, the user may engage with the
system using gestures when the user is located in the space
external to the depth sensor (e.g., in a user's room where the
depth sensor is located). In some example embodiments, the user may
engage with the system when the user is away from home (e.g., using
a mobile device). In certain example embodiments, a mobile device
may be used as a depth sensor. In such example embodiments, a
camera of a mobile device (e.g., an iPhone or Android phone) may be
used to capture an image of the user and to show the captured image
of the user on a TV screen. The mobile device may be mounted on a
TV set and may be in communication with the TV set to transmit the
captured image of the user from the mobile device to the TV set.
Thus, the captured image of the user may be communicated to the TV
screen and displayed thereon. In other words, the TV screen may act
as a mirror for the user. The system may then superimpose a
representation of a clothing item onto the image of the user and
show the representation superimposed on the image on the TV screen.
Using one or more gestures, the user may browse through a number of
virtual clothing items or virtual outfits (e.g., a representation
of a set of garments and accessories that may be worn together)
that the system may suggest (e.g., by displaying them on the TV
screen) or that may be found in an electronic marketplace
catalogue. Furthermore, the user may also employ a gesture or voice
command to save a reference to an item within a shopping cart or
purchase the item using the electronic marketplace.
[0038] In some example embodiments, the depth sensor may have an
infrared emitter to emit a pattern of IR dots (e.g., spray the room
with infrared light) and a camera to capture spatial data (e.g., an
IR image) containing data about the size, brightness, and
distortion of the dots as viewed in space. The captured spatial
data may be descriptive of (e.g., include or comprise data about)
the shape of the user's body, measurements of the user's body, the
user's gestures, space around the user, including other people or
objects in the user's vicinity or physical environment. Using the
captured spatial data, the system may generate a model of an object
located in the physical environment of the spatial sensor. Examples
of an object include a person, an animal, and an inanimate item. In
some example embodiments, the model may illustrate the shape of the
object. The model of an object may be a two-dimensional
(hereinafter, also "2D") visual representation or a
three-dimensional (hereinafter, also "3D") visual representation of
the object. In some example embodiments, the model of the user's
body may include a skeletal representation of the user's body. In
various example embodiments, the model may include data about the
position of the user at the time the spatial data used to generate
the model was captured. In some example embodiments, the model of
the object may have additional data that corresponds to the model
(e.g., text data and audio data). In various example embodiments,
the system may store the model of the object, as well as the data
that may pertain to the model (e.g., a dimension, shape, size,
texture, color, text, and audio) in a database (e.g., a 3D or 2D
asset library). The dimensions of the user's body, determined from
the captured spatial data, may be used (e.g., by a machine of the
system) to create an avatar (e.g., of the user) that may have a
shape that resembles the shape of the user's body. The avatar may
be two-dimensional or three-dimensional. Similarly, the dimensions
of an inanimate object (e.g., a clothing item), determined from the
captured spatial data, may be used (e.g., by a machine of the
system) to create a 3D representation of the inanimate object. In
some example embodiments, the avatar of the user's body and the 3D
representation of the inanimate object may be stored in a database
and each may reference other stored data that pertains to it. The
storing of representations of the users' bodies and of clothing
items may allow the system to reuse them and, thus, avoid spending
resources in creating them each time a particular user engages with
the system or a representation of a clothing item is to be
displayed to the user. Furthermore, in some example embodiments,
the system may allow the stored representations to be reused by
users to virtually try on different virtual clothing items and to
avoid having to create a separate representation each time a
virtual clothing item is being tried on.
[0039] In some example embodiments, the system may determine the
user's body measurements (e.g., height or shoulder width) from the
captured spatial data and may generate (e.g., build) an avatar of
the user using the knowledge of the user's body measurements. In
certain example embodiments, the system may generate the user's
avatar dynamically. The avatar may be stored (e.g., by the system
or by the user) in a memory of a machine (e.g., the user's mobile
device) for reuse by the user. Thus, the user may virtually try on
virtual clothing from anywhere without having to stand in front of
the depth sensor every time he or she wants to virtually try on
virtual clothing. In various example embodiments, the user may use
this avatar to engage in mobile clothes shopping (e.g., when not
using the depth sensor in his home).
[0040] In some example embodiments, as noted above, the avatar is
two-dimensional. In other example embodiments, the avatar is a
three-dimensional representation of the user's body. In certain
example embodiments, the system may generate a 3D representation of
a clothing item (e.g., available for purchase using the electronic
marketplace), size the representation of the clothing item
according to the user's body measurements (e.g., find the size of
the clothing item that is closest to the user's body measurements),
and display the 3D representation of clothing overlaid on the 3D
avatar. The determination of the user's body measurements, the
generation of an avatar based on the user's body measurements, and
the sizing of the virtual clothing to the user's measurement may,
for example, allow the user to better understand how the real
clothing item would fit him or her without physically trying on the
real clothing item.
[0041] In some example embodiments, the electronic marketplace
operator may recommend to the user a virtual clothing item or a
virtual outfit to virtually try on. In certain example embodiments,
the system may determine one or more of the user's body
measurements based on the received spatial data. Based on the
determined user's body measurements, the system may recommend to
the user only virtual clothing items that fit the user's body
measurements. Not having to consider virtual clothing items that do
not fit may save the user time while shopping on the electronic
marketplace. In certain example embodiments, the system provides
the user with a choice for changing the size of the virtual
recommended item.
[0042] In various examples, the system may recommend a clothing
item by displaying a representation of the clothing item overlaid
on a representation of the user. As noted above, the representation
of the item may be two-dimensional or three-dimensional. The
representation of the user may also be two-dimensional or
three-dimensional. In various examples, the creation of a 3D
representation of the user's body or of an item of clothing based
on spatial data captured by a depth sensor may be rotated, which
may allow the user to view the 3D representation of the user's body
or of the clothing item from different perspectives. This may help
the user understand better how the fabric of the clothing item
would move when the user wearing the clothing item would move. In
some example embodiments, the system maps a 3D virtual clothing
item onto the user's 3D avatar. In various example embodiments, the
2D virtual clothing is displayed overlaid on a 2D image of the user
(e.g., a video feed of the user).
[0043] In various example embodiments, the user may employ a
particular gesture to change the color, pattern, or texture of a
representation of the clothing item the user is virtually trying
on. In some example embodiments, the user may use a gesture to
communicate a command to the system to create a 3D representation
of an item of real clothing the user is wearing. The user may use
another gesture to create a listing of the item of real clothing
the user is wearing. In various example embodiments, the listing
may include the 3D representation of the clothing item the user is
wearing as part of the description of the clothing item. In certain
example embodiments, the listing describes the clothing item being
offered for sale. In some example embodiments, the system may add
the listing that describes the clothing item into a catalogue of
items. In some example embodiments, the catalogue listings may
contain 3D representations of items. In various example
embodiments, the user may make a gesture toward an item, and the
depth sensor may capture spatial data about the user's gesture and
about the item toward which the user gestured. Then, the system may
interpret the gesture in relation to the item toward which the user
gestured. In certain example embodiments, based on interpreting the
gesture (e.g., determining the action that corresponds to the
gesture), the system may create a 3D representation of the item. In
some example embodiments, the user may make another gesture to
command the system to store the 3D representation of the item in a
database. In certain example embodiments, the system may receive a
record (e.g., an inventory entry or a description) of an item to be
used in the creation of a catalogue of items. Such a record may
include an image of the item. In some example embodiments, based on
one or more rules, the system may remove from the image certain
content (e.g., background) and may use the remaining content to
generate a representation of the item that may become part of a
catalogue of items. In some example embodiments, the representation
of the item is a 3D representation of the item generated based on
the content of the image received in the record.
[0044] In some example embodiments, upon either obtaining spatial
data about the user's body for the first time or recognizing the
user based on previously received spatial data about the user's
body, the system may retrieve, from a database, a recommended
virtual clothing item or virtual outfit in the user's size and may
display it to the user. In certain example embodiments, the system
may present an item of virtual clothing in response to a voice
command (e.g., by the user). In some example embodiments, the
system may adjust the size of the virtual clothing item to fit the
user's body measurements after retrieving the virtual clothing item
from the database and before it presents the virtual clothing item
to the user. In certain example embodiments, the system may present
just one item that relates to clothing.
[0045] When the user is in the virtual dressing room (e.g., when
the user interacts with the system described herein), in addition
to virtually trying on virtual items of clothing, the user may add
a virtual outfit to his or her closet, search for a specific item
of virtual clothing in the user's virtual closet, search for a real
item of clothing on the electronic marketplace, mix and match a
recommended virtual clothing item with virtual outfits or single
items saved in the user's virtual closet, buy, sell, or bid for a
clothing item on the electronic marketplace, or hold a social
network virtual fashion show by presenting the virtual outfit on a
virtual runway. The system may provide the user the choice to
create a virtual closet to store virtual clothing items. In some
example embodiments, the user may get feedback from the user's
friends over a social network service (e.g., Twitter*, hereinafter
also "Twitter", or Facebook.RTM., hereinafter, also "Facebook".)
The user's friends may comment on the virtual outfits, give advice,
and even coordinate virtual outfits. If they have access to a depth
sensor, such as a Kinect, the user's friends may join the user in a
group virtual dressing room. In some example embodiments, a group
virtual dressing room may be useful as a fitting room for a group
of people coordinating outfits for an event, such as a wedding.
[0046] In some example embodiments, the system may allow the user
to virtually try on virtual clothing while away from the depth
sensor (e.g., located in the user's living room) by displaying an
avatar of the user's body within the user interface of a mobile
device. In certain example embodiments, the avatar is generated
based on the user's body measurements derived from spatial data
descriptive of the user's body, which spatial data was previously
received from the depth sensor. In various example embodiments, the
system may use spatial data that may include data about the shape
of the user's body in generating the avatar of the user's body. In
some example embodiments, the avatar is generated by the system
based on data provided by the user (e.g., photographs, sizes, or
body measurements).
[0047] In some example embodiments, the use of garment recognition
technology in the present system may allow for easier and more
accurate selling of clothes (e.g., using the electronic
marketplace). The system may recognize a particular clothing item
based on depth data received from the depth sensor and may create a
visual representation of the clothing item. In certain example
embodiments, after recognizing the item, the system may retrieve a
visual representation of the clothing item from a database (e.g.,
from an asset library). In some example embodiments, the system may
include the visual representation of the item in a listing for sale
on an electronic marketplace. In various example embodiments, the
visual representation of the item is three-dimensional. In certain
example embodiments, the visual representation is
two-dimensional.
[0048] In some example embodiments, the system may be described to
receive a spatial data feed descriptive of the body of the user in
a 3D physical space. The spatial data feed may be received from a
depth sensor. The system may generate a model of the user based on
the spatial data feed descriptive of the body of the user. The
model of the user may include positional data descriptive of a
movement of a body part of the user. The model of the user may also
include a first shape of the user's body based on the spatial data
received at a first time and a second shape of the user's body
based on the spatial data received at a second time. Then, the
system may determine a gesture of the user based on the model of
the user. In some example embodiments, the determining of the
gesture may be performed by analyzing a difference between the
first and second shapes of the model of the user. The system may
also determine an action that corresponds to the gesture of the
user. In various example embodiments, the system may perform the
action that corresponds to the gesture.
[0049] In certain example embodiments, the system may be described
to receive a stream of spatial data, which stream of spatial data
may include a delineation of a contour of the user's body in a
three-dimensional physical space. The stream of spatial data may be
received from a depth sensor. The system may generate a model of
the user based on the received spatial data. The model may include
positional data that describes a movement of a body part of the
user. The model may also include a first shape of the user's body
based on the spatial data received at a first time and a second
shape of the user's body based on the spatial data received at a
second time. The system may determine a gesture based on the model
of the user by analyzing a difference between the first and second
shapes of the model of the user. The system may also interpret the
gesture as a command to perform an action that corresponds to the
gesture. In some example embodiments, the system may perform the
action that corresponds to the gesture.
[0050] In various example embodiments, the user may engage and
interact with the system using gestures. In some example
embodiments, the system presents virtual interface buttons which
the user may virtually push. An example of the virtually pushing of
a virtual button is the user making a pushing motion with a hand
toward a virtual button on a screen of the system. In certain
example embodiments, the system, using gesture recognition
technology, may determine that the user made a motion with a part
of the user's body, may interpret that motion to be a gesture, may
further determine what gesture the user intended, and may perform
the action associated with the particular gesture. In some example
embodiments, upon determining which action corresponds to the
captured gesture, the system may request the user to perform
another gesture or issue a voice command to confirm that the user
intended the system to perform the action that corresponds to the
gesture. Examples of actions that the user may request the system
to perform in response to the user's gestures are virtual entering
the virtual dressing room, displaying virtual clothing items,
displaying the virtual clothing items on a virtual clothing rack,
creating virtual outfits from separate virtual clothing items,
shuffling the virtual outfits on the virtual clothing rack,
spreading the virtual clothing items on the virtual clothing rack,
displaying the price of a real clothing item that corresponds to a
selected virtual clothing item, displaying the price of all the
real clothing items that correspond to the presented virtual
clothing items, buying an instance of a real clothing item that
corresponds to a selected virtual clothing item, adding a virtual
clothing item to the user's virtual closet, presenting a color
picker, texture selector, or pattern selector for the user to have
a choice to change the color, texture, or pattern of a presented
virtual clothing item, selecting a color, texture, or pattern,
invoking a color picker, texture selector, or pattern selector,
starting a virtual fashion show, sharing images of virtual clothing
displayed with the user on a social network site, requesting
comments pertaining to the displayed virtual clothing items from
friends on social networks, invoking a camera, and taking a
snapshot of a virtual outfit displayed on the virtual catwalk of
the virtual fashion show.
[0051] Examples of gestures that the user may employ while
interacting with the system are taking an initial welcome pose to
begin using the system (e.g., virtually enter the virtual dressing
room), making a swipe motion to move virtual clothing items on the
virtual clothing rack, making a grabbing-and-dragging motion to
select a virtual outfit off the rack, virtually pulling on the
virtual clothing item to display the virtual clothing item overlaid
on the image of the user, jumping to randomize the virtual clothing
items presented, showing a thumbs up gesture to communicate "I like
this item; show me more like this", showing a thumbs down gesture
to communicate "I don't like this, don't show it again", virtually
hanging a virtual hanger on a virtual rack to communicate "Put it
in my closet", and making a discarding motion to communicate "Next
(e.g., show me another item)!" Other examples of gestures that the
user may employ while interacting with the system are tapping an
item (e.g., a shirt the user is wearing) to select the item,
scrubbing an area of the color picker to select a color, moving
hands away from each other from an initial almost touching position
where only the thumbs and pointing fingers are open to invoke the
color picker, virtually pushing a virtual control button to start a
virtual fashion show, and pushing a virtual control button to
request the display of comments from friends on social networks. In
some example embodiments, the user may use voice commands to
communicate with the system (e.g., user may say "cheese!" or
"shoot!" to activate a camera.)
[0052] In some example embodiments, the system using a depth sensor
may determine the user's body measurements based on positional data
points determined by the system based on the spatial data captured
by the depth sensor. The system may pre-filter the clothing items
whose representations should be presented to the user such that
only representations of clothing items matching the user's body
measurements may be presented to the user. For example, if the
system determines that the user wears a medium-sized shirt based on
the measurements of the user's torso, only representations of
shirts of a medium size may be displayed to the user. In certain
example embodiments, before a seller lists an item for sale, the
seller may display the item within the depth sensor's field of
spatial data capture so that the depth sensor may capture the
dimensions of that item. The system may determine the size of the
item based on the captured dimensions of the item. The captured
dimensions and determined size may be provided to a buyer (e.g., by
a machine) to help a buyer ascertain the fit of an item. In some
example embodiments, the seller may provide 2D images of an item
for sale to the system and the system may generate a 3D
representation of the item based on the 2D images of the item. In
some example embodiments, the system may translate (e.g.,
correlate) different sizes of items to absolute dimensions. This
may assist buyers and sellers in obtaining and delivering more
detailed information about the items listed for sale. In various
example embodiments, the system may use cloth physics technology to
generate a 3D representation of a clothing item that accurately
reflects the features of the fabric from which the clothing item
was made. For example, clothing items made from different fabrics
may hang or move differently based on the type of fabric used to
manufacture the particular clothing item. Thus, using cloth physics
technology in generating 3D representation of clothing items may
allow the user to see how the real physical item of clothing would
move when worn by the user.
[0053] In certain example embodiments, the system may determine the
user's affinity for a type of item (e.g., skirts, dresses, shoes,
or jewelry) or characteristic of an item (e.g., color, pattern, or
texture) based on tracking the amount of time the user may spend
examining a representation of an item displayed to the user. In
certain example embodiments, the system may determine that a user
has an affinity for an item based on the user viewing a
representation of the item more than once. The system may recommend
certain items based on the determined affinity of the user for
particular items. In various example embodiments, the system may
provide curated recommendations to the user (e.g., recommendations
by a stylist or another expert). These recommendations may be, for
example, style-based.
[0054] In certain example embodiments, the depth sensor may be used
to perform image recognition. For example, the depth sensor may
recognize the items of clothing a person is wearing, as well as the
color and texture of those items. The system may present to the
user a virtual clothing item that complements another virtual
clothing item the user is already virtually trying on. For example,
the system may present the user with a representation of a pair of
trousers that matches a representation of a shirt the user is
virtually trying on. In some example embodiments, the system may
extract data about a texture of a clothing item worn by the user
and apply that texture to a representation of another clothing item
such that the user may visualize the texture on a different
clothing item. For example, if the user is wearing a shirt that has
a polka dot pattern, the system may extract the respective polka
dot pattern from the user's shirt, may apply the polka dot pattern
to a representation of a skirt, and may display the representation
of the skirt having the polka dot pattern to the user. Thus, the
user may be able to visualize how that polka dot pattern would look
on the skirt whose representation has been displayed to the user.
In certain example embodiments, the system may also have
applicability to a home and garden scenario. For example, the
system may capture a color from a paint color swatch displayed in
the field of vision of a device (e.g., that includes a depth sensor
and a camera) and may apply the respective color to a
representation of a wall that is displayed to the user on a screen
to help him visualize how the wall would look if painted that
color.
[0055] In various example embodiments, the system may allow the
creation of a virtual group dressing room. In some example
embodiments, the depth sensor may recognize up to four people in
the same field of vision (e.g., the depth sensor's field of capture
of the spatial data). In other words, the system may capture
spatial data about the bodies of multiple users standing inside the
room where the depth sensor is located and may allow them to
virtually try on virtual clothing by displaying representations of
the multiple users and their virtual clothing in the same image on
a screen. In some example embodiments, the system may enable users
who are not in the same physical room, but who have access to depth
sensors, to connect via a network such that the depth sensors
capture the positional information (e.g., spatial data) of
different users standing in front of different depth sensors. In
certain example embodiments, the networking of depth sensors may
allow more than four users to be recognized in the same field of
vision. In some example embodiments, the representations of the
users' bodies may be displayed together on each user's screen
(e.g., inserted into each view) such that each user in front of a
depth sensor may see all users of the group virtual dressing room.
For example, a wedding party (e.g., a group of people coordinating
outfits for a wedding) that is scattered geographically may connect
via networked depth sensors to virtually try on virtual bridesmaid
dresses together, so that everyone can see how the wedding party
would look together. In some example embodiments, the system may
allow the group of people to place an aggregate order for the items
that they virtually tried on in the virtual group dressing room.
The order may reflect the sizes of the physical clothing items
based on the system determining the body measurements of each
person virtually trying on the virtual clothing items based on the
spatial data captured by the depth sensors.
[0056] In some example embodiments, the system may allow the user
to solicit fashion opinions via social sharing (e.g., on Twitter or
Facebook). Often, a user who is shopping for clothing would like to
solicit a friend's opinion about a clothing item the user is
interested in. When the user is shopping by herself, this is not an
option. In some example embodiments, the system may allow the user
to share a video feed of the user virtually trying on virtual
clothing items with his or her friends on social networks, thereby
enabling the friends to provide opinions of how the virtual
clothing item appears on the user. The video feed may be shared via
existing social media channels (e.g., Twitter and Facebook). In
some example embodiments, feedback may be provided via audio, text,
social media (e.g., Twitter or Facebook), or a separate video feed
displayed to the user. In certain example embodiments, the system
may include a social voting mechanism to enable the user's friends
to communicate to the system (e.g., by gesture, voice, or text)
their approval or disapproval of a virtual clothing item as being
virtually tried on by the user. In certain example embodiments, the
system may tabulate the votes and display the aggregated vote
totals to the user.
[0057] In various example embodiments, the system may allow the
user to engage in market transactions (e.g., using the electronic
market) using gestures recognized by the system. Examples of market
transactions are offering an item for sale, bidding for an item,
and buying an item. In some example embodiments, to prevent an
accidental gesture from being interpreted as the user intending to
enter into a transaction, the system may prompt the user to confirm
that the gesture was intentional (e.g., by engaging in an action).
In some example embodiments, the user may issue a voice command to
confirm that his gesture was intentional. In other example
embodiments, the user may express his or her intent to enter into a
transaction by speaking a voice command together with repeating the
gesture. In some example embodiments, the user may perform
sequential gestures, simultaneous gestures, or a subsequent gesture
to express an intent to enter into a transaction.
[0058] In various example embodiments, the system may allow the
user to streamline the number of steps needed to list an item for
sale. For example, the user may tap or touch an item (e.g., a
physical object) and the depth sensor may capture the spatial data
pertaining to the user's movement in relation to the item. In some
example embodiments, the system may interpret the captured spatial
data to recognize the item and the item's dimensions. Some devices
(e.g., that may include a depth sensor and a camera) of the system
may detect other features of the item, such as color style, or
texture. Based on the captured spatial data that relates to the
item, the system may recognize one or more features of the item and
determine that it already has some data that pertains to the item,
which data may be stored in a database. In certain example
embodiments, the system may populate a search listing page with
recognized features of the item. In some example embodiments, the
system may map different gestures recognized by the depth sensor to
different actions (e.g., actions that relate to transactions using
the electronic marketplace). In certain example embodiments, the
user jumping up and down once could communicate to the system that
the user wants to virtually try on a random virtual outfit pulled
from the electronic marketplace operator's inventory. In various
example embodiments, the user performing the touchdown gesture
could communicate to the system to begin calibration. In some
example embodiments, the user touching an item (e.g., a clothing
item) and drawing (e.g., tracing) a line around the item from a
start point to an end point could instruct the system to capture
spatial data that relates to the item, generate a representation of
the item, and store the representation of the item (e.g., in the
user's virtual closet) for a later use. In some example
embodiments, the user may make a gesture with his or her hand
resembling a bidding paddle to instruct the system to make a bid on
an item listed with an electronic marketplace operator. In some
example embodiments, the user may hold up a wallet or a payment
card to instruct the system to make a payment on the user's behalf.
In some example embodiments, the system will request the user to
confirm the user's identity (e.g., using biometrics). In some
example embodiments, the user may point to an item (e.g., a
physical item in the field of view of the depth sensor or a visual
representation of an item on a screen) and the system may generate
a 3D model of the item either using spatial data descriptive of the
item or data that the system already has stored in a database. In
certain example embodiments, the system may determine the size of a
box to be sent to a seller to use for shipping an item for sale
based on the system determining the spatial measurements of the
item for sale from the spatial data descriptive of the item for
sale captured by a depth sensor.
[0059] A product may be manufactured by a manufacturer and
available for purchase from a seller. For example, the product may
take the form of a good (e.g., a physical object), a service (e.g.,
performed by a service provider), information (e.g., digital
media), a license (e.g., authorization to access something), or any
suitable combination thereof. An item may be a specimen (e.g., an
individual instance) of the product, and multiple items may
constitute multiple specimens of the product. Accordingly, a seller
of a product may seek to merchandise one or more items as specimens
of the product.
[0060] In merchandising an item, the seller may use a network-based
system to present the item to the user of the network-based system
(e.g., a potential buyer of the item). Examples of network-based
systems include commerce systems (e.g., shopping websites),
publication systems (e.g., classified advertisement websites),
listing systems (e.g., auction websites), and transaction systems
(e.g., payment websites). The item may be presented within a
document (e.g., a webpage) that describes the item or product. In
shopping for an item, one or more users may search the
network-based system (e.g., by submitting queries) for such
documents or similar information regarding details of the item or
product.
[0061] FIG. 1 is a chart illustrating examples of possible actions
in a virtual dressing room 105, according to some example
embodiments. The electronic marketplace operator, in some example
embodiments, may recommend a virtual outfit or a virtual clothing
item to the user to virtually try on. In certain example
embodiments, the size of a recommended virtual clothing item may
correspond to the user's body measurements (e.g., provided by the
user or determined based on the spatial data captured by the depth
sensor). For example, a recommended real clothing item may be
presented to the user as a representation of the recommended real
clothing item, adjusted (e.g., scaled) to the clothing item's brand
size closest to the user's body measurements, being overlaid (e.g.,
mapped) on a representation of the user. In some example
embodiments, the representation of the item of clothing may be
scaled to correspond to the particular user's body
measurements.
[0062] As noted above, the representation of the item may be
two-dimensional or three-dimensional. The representation of the
user may also be two-dimensional (e.g., a video feed, a photograph
of the user, or a 2D avatar of the user) or three-dimensional
(e.g., a 3D avatar of the user). In some example embodiments, the
representation of the recommended clothing item may be displayed on
a virtual clothing rack. In certain example embodiments, the user
may have a choice to view the virtual outfit, either displayed on
the virtual clothing rack or overlaid on a representation of the
user.
[0063] The user may look at a recommended virtual clothing item or
virtual outfit, or may browse through a number of suggested virtual
outfits 110 or virtual clothing items (e.g., based on deals,
stylists, or popularity). In some example embodiments, the user may
change one piece of a virtual outfit 115 for another. Because the
user may prefer a color different from the color of the recommended
virtual clothing item or virtual outfit, the user may change the
color 120 (e.g., by making a gesture to invoke a color picker menu
or by issuing a voice command) of the virtual clothing item or
virtual outfit that has been recommended by the system. In some
example embodiments, the user may choose to change another detail
125 (e.g., texture, pattern, or size) of the presented virtual
clothing item or virtual outfit. The user may also want to share
130 (e.g., in a virtual fashion show using a social media channel)
the recommended virtual outfit or virtual clothing item with the
user's friends (e.g., to get their opinion on the recommendation or
on the fit of the virtual outfit or virtual clothing item, or to
make a further recommendation to the user's friends). As noted
above, the sharing 130 may occur using public social media
platforms (e.g., Facebook or Twitter) or a private network limited
to a group of users.
[0064] In some example embodiments, the user may be interested in
finding a specific clothing item 135 (e.g., to create an outfit or
to dress a bridal party). The present method and system may allow
the user to match something the user owns 140 to an additional
clothing item that may be listed on the electronic marketplace
(e.g., a dress the user owns to a pair of shoes for sale on the
electronic marketplace). In various example embodiments, the user
may match the item he already owns to a desired item by color,
size, or other detail 145 (e.g., texture, pattern, or style) using
the virtual dressing room 105. Furthermore, the user may match the
clothing items by occasion 150 (e.g., a formal occasion, such as a
wedding). The user may also collaborate on an outfit 155 with
others (e.g., a friend on Facebook or Twitter, or a stylist). In
some example embodiments, several users may coordinate outfits 160
(e.g., outfits for a bridal party or for a family photograph).
[0065] In some example embodiments, the user may choose to create a
virtual closet 165. For example, the user may use the system to
create representations of clothing items in the user's physical
closet and save them in the user's virtual closet. In some example
embodiments, this would allow the user to remotely access the
user's virtual closet. For example, using a mobile device, the user
may look up items of virtual clothing in the user's virtual closet
to match to real items of clothing while shopping in a real store.
In another example, the user may match items of clothing available
for sale on an electronic marketplace to the clothes in the user's
virtual closet without physically trying on clothes or standing in
front of a depth sensor.
[0066] Accordingly, the creation of a virtual closet 165 may allow
the user to add virtual clothing items to the user's virtual closet
170. In some example embodiments, the user's gesture representing
adding a virtual clothing item to the user's virtual closet may be
correlated to the action of placing an order to buy the physical
item of clothing that corresponds to the virtual clothing item
using the electronic marketplace. In other example embodiments, the
user's gesture representing adding an item of virtual clothing to
the user's virtual closet may be correlated to adding the item of
virtual clothing to a "like" list, also known as a wish list (e.g.,
a list of items to be considered for purchasing in the future).
[0067] In addition, the user may also search the user's virtual
closet 175. In some example embodiments, the user may organize the
representations of the virtual clothing items in the user's virtual
closet (e.g., by type of item, color, style, texture, size, or
occasion).
[0068] Furthermore, in certain example embodiments, the user may
sell his or her clothes 180 (e.g., using the electronic
marketplace). For example, the user, wearing a clothing item he or
she wants to sell, may stand in front of the depth sensor and make
a gesture that is correlated to a command to list the clothing item
for sale on the electronic marketplace. Based on determining that
the user's gesture corresponds to the command to list the clothing
item for sale on the electronic marketplace, the system may
initiate the listing of the clothing item for sale on the
electronic marketplace on behalf of the user. In some example
embodiments, the system may create a 3D representation of the
clothing item for sale and include the 3D representation of the
clothing item in the listing that describes the clothing item for
sale. In other example embodiments, the user may display the real
clothing item for sale to the depth sensor without wearing the real
clothing item for sale and the system may create a listing for the
displayed clothing item, which listing may include a representation
(e.g., 2D or 3D) of the displayed clothing item.
[0069] The system may allow for different ways to display the user
on a screen. In some example embodiments, the user interface may
display a live camera feed showing an image of the user. In other
example embodiments, a 2D or 3D avatar of the user may be
displayed. FIG. 2 is a conceptual diagram discussing an initial
welcome state, according to some example embodiments. In various
example embodiments, the system may present a graphical user
interface to allow the user to activate the system. In some example
embodiments, the user may assume an initial welcome state to
activate the system. In other example embodiments, the user may
make an initial welcome gesture to activate the system. In certain
example embodiments, the user may activate the system with the use
of a voice command.
[0070] Upon activation, in some example embodiments, the depth
sensor may acquire (e.g., capture or receive) spatial data about
the user standing in front of the depth sensor. The system may
display a representation of the user within the virtual dressing
room (e.g., using a live camera feed or an avatar) on a screen
(e.g., of a TV set, computer monitor, or mobile device), according
to some example embodiments. Because the acquired spatial data may
include data that may allow the system to determine the user's body
measurements, the representation of the user mimics closely the
user's body shape. In some example embodiments, the system may also
determine the user's one or more sizes (e.g., the user's size may
be different for different brands of clothing) based on the
acquired spatial data. While the system loads outfits in the user's
size, the system may communicate to the user a status update. For
example, the user may be presented with a statement "Loading
outfits in your size . . . " and an information plane 1.1 that may
rotate back and forth while loading outfits. Also, a loading ring
1.2 may be shown on the ground plane. Further, the user may be
presented with a contextual control, such as a virtual navigation
button 1.3 to the user's virtual closet (e.g., "My closet") or a
virtual share button 1.4 (e.g., "Fashion Show").
[0071] FIGS. 3-4 are conceptual diagrams discussing a virtual rack
of clothes, according to some example embodiments. In various
example embodiments, the user may employ gestures to interact with
virtual clothing items in a virtual dressing room. As shown in FIG.
3, the user may make a gesture to cause the system to load virtual
clothing outfits on a virtual clothing rack by spreading (e.g.,
opening out) from the center. For example, the representation of
the user is centrally located and the virtual clothes are being
loaded behind the user's representation and spread out as more
virtual clothes are loaded.
[0072] As shown in FIG. 4, in some example embodiments, when the
virtual outfits finish loading on the virtual rack, a virtual
outfit may automatically appear overlaid on the representation of
the user. The user may be presented with an option to buy 3.1 the
outfit or item (e.g., presented or selected), find out the price
3.2 of the outfit or item, add the virtual outfit or virtual
clothing item to the user's virtual closet 3.3, or select the
highlighted item 3.4.
[0073] FIG. 5 is a conceptual diagram illustrating a gesture,
according to some example embodiments. The user may make a swiping
movement which may correspond to a command to move a virtual outfit
on the virtual rack. In some example embodiments, a longer swipe
may move a larger number of virtual outfits with one swipe and a
shorter swipe may move a smaller number of virtual outfits with one
swipe. In some example embodiments, the contextual controls (e.g.,
Buy 3.1, price 3.2, and Add to Closet 3.3) may be dimmed between
virtual outfit selections.
[0074] FIG. 6 is a conceptual diagram illustrating an action,
according to some example embodiments. In response to the swiping
gesture as described above with respect to FIG. 5, the system may
auto-increment by one virtual outfit, rotating clockwise (e.g., may
move the virtual outfits on the rack clockwise one at a time).
[0075] FIG. 7 is a conceptual diagram illustrating a gesture,
according to some example embodiments. When the user sees a virtual
outfit he or she may want to virtually try on, the user may make a
grab and drag gesture directed to the respective virtual outfit.
The gesture may represent a command to the system to initiate the
corresponding action (e.g., grab or select at item on the
rack).
[0076] FIG. 8 is a conceptual diagram illustrating a gesture and an
action, according to some example embodiments. In response to the
grab and drag gesture as described above with respect to FIG. 7,
the system may map the virtual outfit selected by the user to the
representation of the user (e.g., by overlaying the representation
of the outfit on the representation of the user). In some example
embodiments, the mapping of the selected virtual outfit to the
representation of the user (e.g., pulling or putting the virtual
outfit on) may cause the current virtual outfit and selected
virtual outfit to swap places (e.g., the user wears the selected
virtual outfit and the current virtual outfit is moved to the rack,
replacing the selected virtual outfit).
[0077] FIG. 9 is a conceptual diagram illustrating a gesture,
according to some example embodiments. The user may employ a
gesture to request the system to mix the order in which the virtual
outfits are displayed on the virtual clothing rack. For example,
the user may jump up and down to cause the system to randomize the
presentation order of the virtual outfits (e.g., to mix the virtual
outfits in a random fashion). In certain example embodiments, the
system may mix the items based on knowledge it may have about the
user (e.g., the user's preferences in clothing or the user's
upcoming scheduled events).
[0078] FIG. 10 is a conceptual diagram illustrating an action,
according to some example embodiments. In response to the gesture
to mix the displayed virtual outfits as described above with
respect to FIG. 9, the system may display the presented virtual
outfits in a different order.
[0079] FIG. 11 is a conceptual diagram illustrating gestures,
according to some example embodiments. As shown, the user may
employ additional gestures to engage with the virtual clothing
items in the user's virtual dressing room. For example, the user
may use the thumbs up or thumbs down gestures to communicate to the
system whether the user likes or dislikes, respectively, a
recommended item of virtual clothing. In some example embodiments,
the thumbs up gesture further communicates to the system a request
to show more items similar to the liked item. In some example
embodiments, the thumbs down gesture further communicates to the
system a request to not show items similar to the disliked item
again.
[0080] Another example of a gesture the user may employ in the
virtual dressing room is the "hang it" gesture, as shown in FIG.
11. The "hang it" gesture directed to a particular item of virtual
clothing may command the system to place the virtual clothing item
in the user's virtual closet. In some example embodiments, the
"hang it" gesture indicating a physical item of clothing may cause
the system to generate a representation (e.g., 3D or 2D) of the
physical item of clothing and to place the representation of the
physical item of clothing into the user's virtual closet. In
certain example embodiments, the user may employ a gesture to list
the physical item of clothing for sale on the electronic
marketplace and include the representation of the physical item of
clothing as part of the description of the physical item of
clothing in the listing.
[0081] A further example of a gesture the user may employ in the
virtual dressing room is the "discard" gesture, as shown in FIG.
11. This gesture may be used to represent the user's intention to
move to (e.g., consider) a next item.
[0082] FIG. 12 is a conceptual diagram illustrating a gesture,
according to some example embodiments. In some example embodiments,
tapping an item (e.g., a shirt the user is wearing) may signal a
request to select that item.
[0083] FIGS. 13-14 are conceptual diagrams illustrating an action,
according to some example embodiments. As shown in FIG. 13, based
on the user tapping his or her shirt as described above with
respect to FIG. 12, the virtual outfit rail may split and rotate
off the stage as a virtual shirt rail may expand out from the
center (e.g., the user position). In some example embodiments, a
text plane 11.1 stating "Loading shirts . . . " may be displayed to
textually communicate a status update to the user.
[0084] FIG. 14 illustrates a virtual rail of virtual shirts that
have finished loading. This view may show that the focus is on the
individual virtual clothing item (e.g., virtual shirt) versus the
virtual outfit (e.g., which has moved off the display area). In
some example embodiments, the system may present the user with
another contextual control, the color picker 12.1. In some example
embodiments, the system may also present another contextual
control, such as a pattern selector. The functionality of the color
picker 12.1 is described below with respect to FIGS. 16-17.
[0085] FIG. 15 is a conceptual diagram illustrating a gesture,
according to some example embodiments. Upon being presented with
the color picker 12.1 as described above with respect to FIG. 14,
the user may select the color picker 12.1 (e.g., by tapping the
color picker 12.1) to view a range of colors.
[0086] FIG. 16 is a conceptual diagram illustrating an action,
according to some example embodiments. As a result of the user
selecting the color picker 12.1 as described above with respect to
FIG. 15, the color picker 12.1 may expand open to display a range
of colors. In some example embodiments, the user may make a
scrubbing motion over the color picker 12.1 to cause the system to
scroll through the display of colors. In certain example
embodiments, a circle 14.1 may highlight the current selection on
the color picker 12.1 (e.g., the currently chosen color). In
certain example embodiments, when the user selects a color on the
color picker 12.1 (e.g., by tapping the color), all the virtual
shirts displayed on the virtual shirt rail in the virtual dressing
room may change their color to show off the potential for search
(e.g., by color or pattern).
[0087] FIG. 17 is a conceptual diagram illustrating a gesture,
according to some example embodiments. In addition to the color
picker 12.1 being displayed when the user focuses on a particular
type of item as shown and described above with respect to FIG. 14,
the user may invoke the color picker 12.1 at any point while in the
virtual dressing room. In certain example embodiments, the user may
make the gesture illustrated in FIG. 17 (e.g., using the thumb and
forefinger of each hand, the user may start the gesture with the
respective digits touching and move the hands apart in an arc) to
cause the system to display the color picker 12.1.
[0088] FIG. 18 is a conceptual diagram illustrating a gesture,
according to some example embodiments. The user may want to have a
virtual fashion show for the user's social network friends from the
user's virtual dressing room (e.g., to get the friends' opinions as
to a virtual outfit before the user buys the outfit). The virtual
fashion show could be based on the current virtual outfit or the
user's closet of saved virtual outfits. To invoke the virtual
fashion show option the user may virtually hit the virtual share
button (e.g., the contextual control in the upper right corner) 1.4
described above with respect to FIG. 2.
[0089] FIGS. 19-20 are conceptual diagrams illustrating actions,
according to some example embodiments. As shown in FIG. 19, in
response to the user selecting the virtual share button 1.4, the
system may load tweets and virtual outfits. In some example
embodiments, a text plane stating "Loading Fashion Show" may be
displayed to textually communicate a status update to the user.
[0090] As shown in FIG. 20, the system displays a selected virtual
outfit mapped to the representation of the user in the context of a
virtual fashion show with a virtual catwalk (e.g., to show a
virtual clothing item to the user's friends on social networks,
thereby enabling the friends to provide opinions on how the virtual
clothing item appears on the user). In some example embodiments,
the system may use an avatar of the user on which the selected
virtual outfit is overlaid. In other example embodiments, the
system may show a video feed of the user virtually trying on
virtual clothing. The video feed may be shared via existing social
media channels (e.g., Twitter and Facebook). In some example
embodiments, feedback may be provided via audio, text, social media
channels or separate video feeds displayed to the user (e.g.,
Tweets 17.3). In certain example embodiments, the system may
include a social voting ability (e.g., mechanism) to enable users
to communicate to the system (e.g., by gesture, voice, or text)
their approval or disapproval of a virtual clothing item displayed
on the representation of the user. In certain example embodiments,
the system may tabulate the votes and display the aggregated vote
totals to the user.
[0091] Some example embodiments may allow the user to have a
snapshot taken by activating the camera contextual control 17.2
with a gesture. Alternately, the user may activate the camera with
a voice command (e.g., by saying "Cheese!") Also, the virtual
fashion show with the virtual catwalk view may have a contextual
control 17.1, the activation of which may take the user back to a
prior view (e.g., the virtual outfit picker view).
[0092] FIG. 21 is a conceptual diagram illustrating issuance of a
voice command, according to some example embodiments. As described
above with respect to FIG. 20, the user may invoke a camera using a
voice command rather than using a gesture. This may allow the user
to have a picture taken of the user virtually trying on virtual
clothes without ruining his or her pose.
[0093] FIGS. 22-23 are conceptual diagrams illustrating the use of
the present method and system in the context of curated
recommendations and group virtual dressing rooms, according to some
example embodiments. In various example embodiments, the system may
provide curated recommendations to the user (e.g., recommendations
by a stylist or another expert). These recommendations may be, for
example, style-based. In certain example embodiments, these curated
recommendations may be based on the system learning about the
user's likes and dislikes. For example, the system may determine
the user's affinity for a type of item (e.g., skirts, dresses,
shoes, or jewelry) or characteristic of an item (e.g., color,
pattern, or texture) based on tracking the amount of time the user
spends examining a representation of an item displayed to the user.
In another example, the system may determine the affinity for an
item based on the user viewing a representation of the item more
than once. The system, in certain example embodiments, may
recommend certain items based on the determined affinity of the
user for particular items.
[0094] As shown in FIG. 22, in some example embodiments, the system
may send a notification to the user's mobile device informing the
user of an available curated recommendation (e.g., the statement
"your stylist has an outfit idea for your friend's wedding!") In
some example embodiments, the system may allow for a calendar
synchronization option. In various example embodiments, a stylist
may show the user some recommended styles (e.g., the statement "Let
me show you some styles for your wedding.") In certain example
embodiments, the system may provide sizing information (e.g., the
user's body measurements) based on storing the user's body
measurement data during the user's prior interaction with the
system (e.g., user's body measurement data determined based on the
spatial data captured by the system's depth sensor). Also, in
certain example embodiments, the user may both receive curated
recommendations on the user's mobile device and use the mobile
device to engage with the virtual dressing room.
[0095] As further shown in FIG. 22, the user may be presented with
different styles, including subset virtual outfits that were
coordinated by a professional stylist. The user may also invite
friends on social networks to view the user's virtual fashion show
and communicate to the user what they think about a particular
virtual outfit.
[0096] As shown in FIG. 23, in various example embodiments, a group
of people may use the depth sensing technology of the present
system to coordinate their outfits (e.g., for a wedding dress
party). As noted above, in some example embodiments, the depth
sensor may recognize up to four people in the same field of vision
(e.g., the depth sensor's field of capture). The system may allow
multiple people to virtually try on virtual clothing items and may
display representations of the multiple people with their virtual
clothing in the same image. In some example embodiments, the system
may enable different users who are not in the same physical room
but who have a depth sensor to connect via a network such that the
depth sensors capture the positional information of different users
standing in front of different depth sensors. The captured data may
be superimposed (e.g., inserted) into each view such that each
person in front of a depth sensor may see all users connected via
the network. In certain example embodiments, the networking of
depth sensors may allow more than four users to be recognized in
the same field of vision. In various example embodiments, the
system may allow multiple people to virtually try on virtual
outfits together. For example, a wedding party (e.g., a group of
people coordinating outfits for a wedding) that is scattered
geographically may connect via networked depth sensors to virtually
try on virtual bridesmaid dresses together. This may allow each
person of the respective wedding party to see how the wedding party
would look together. As further shown in FIG. 23, in some example
embodiments, the professional recommendations may extend to color
suggestions (e.g., "This shade of pink goes better with your eyes .
. . ") and shoe suggestions (e.g., "These shoes aren't wide enough
for your feet.")
[0097] FIG. 24 is a conceptual diagram illustrating a position of
the user in relation to a depth sensor (e.g., the Kinect) of the
present system, according to some example embodiments. In certain
example embodiments, the depth sensor that may be used with the
system is the Microsoft Kinect. To allow the Kinect to capture the
spatial data about the user and the user's surroundings, the user
may be positioned inside a predetermined distance range from the
Kinect (e.g., 6-8 foot clearance). Using the Kinect, a 3D
representation of the environment may be generated by emitting a
matrix of infrared dots. In various example embodiments, by
observing how the size, brightness, and distortion of the dots are
viewed in the space, models (e.g., 3D or 2D representations) of the
body of the user may be generated at different points in time.
Additional details about the models of the body of the user are
described below with respect to FIGS. 25-27 and 30-34.
[0098] FIG. 25 is a conceptual diagram illustrating a gesture,
according to some example embodiments. In some example embodiments,
a calibration pose (e.g., user has both arms up), as shown in FIG.
25, may be used to help the Kinect acquire spatial data that may
include data about the body of the user and to begin tracking the
body of the user. In other example embodiments, the depth sensor
can acquire the spatial data about the user without the aid of the
calibration pose (e.g., the user is in a sitting position). As
described above with respect to FIG. 24, the system may receive
spatial data about the environment external to the depth sensor at
different points in time. Capturing spatial data about the user in
the 3D space at different times may allow the system to track
motions of the user (e.g., gestures made by the user). In some
example embodiments, the limbs of the user may be represented as
bones (e.g., symbolic bones in a skeleton or stick figure) in the
models of the user's body. The user's body may also be represented
in 3D models (e.g., 3D relief). In certain example embodiments,
skeletal tracking of the differences (e.g., changes) between the
models of the user generated based on spatial data captured at
different times may allow the system to capture and interpret
gestures made by the user.
[0099] For example, as shown in FIG. 25, the user, represented by a
stick figure, has both arms up at the first time and one arm up and
the other arm down at a second time. The system may generate a
first model for when the user has both arms up (e.g., a first time)
and a second model for when the user has one arm up and one arm
down (e.g., a second time), based on spatial data received at a
first and second times, respectively. By detecting and analyzing
the difference between the first and second models, the system may
determine that the difference (e.g., change) between the models
corresponds to a gesture performed by the user between the first
and second times. The system then may determine that the gesture
represents a command to initiate an action within the electronic
marketplace on behalf of the user. Examples of such an action are
generating a representation (e.g., 2D or 3D) of an item to which
the user is pointing, making a bid for an item listed for sale on
the electronic marketplace, or creating a listing to sell an item
of real clothing the user owns.
[0100] Furthermore, skeletal tracking of the models of the user's
body may allow the system to track a midpoint of the user's body
(e.g., in relation to the depth sensor) to properly represent an
image of the user displayed on a screen. For example, as the user
gets closer to the camera (e.g., the user moves from point one at a
first time to point two at a second time), the user's body midpoint
is closer to the camera and, thus, the skeleton representation
generated when the user is closer to the camera becomes larger
compared to the skeleton representation generated when the user was
farther from the camera. As a result, the size of the
representation of the user on the screen increases. In some example
embodiments, skeletal tracking may be used to proportionally
increase the texture size in the display of the virtual clothing
items the user virtually tries on such that the virtual clothing
texture is appropriately sized and properly tracks the user (e.g.,
when the user moves in the 3D space). Additional details about
modeling of representations of items of clothing are described
below with respect to FIG. 26.
[0101] FIG. 26 is a conceptual diagram illustrating data flow in an
example system and a model, according to some example embodiments.
In some example embodiments, the flow of spatial data may be
symbolically represented as being received at the depth sensor
(e.g., the Kinect). From the Kinect, the data may flow to the
Laptop (e.g., symbolically representing a system described herein)
where models of the user body may be generated based on the
received spatial data. Based on these models, the system may
determine whether the user made a gesture and, if so, determine
what action is correlated to the gesture. Upon determining the
action that is correlated to the gesture and that the action is to
be initiated by the system, the system initiates the action on
behalf of the user whose spatial data was received by the Kinect.
An example of an action, as illustrated in FIG. 26, is facilitating
a real-time Twitter feed to obtain fashion opinions from social
network friends. Another example of an action is displaying a 3D
representation of an item of clothing on a 3D avatar of the user on
a screen.
[0102] Also shown in FIG. 26 is an example of an illustration of a
model of a virtual outfit in relation to the body of the user. In
some example embodiments, the virtual outfit is broken up (e.g.,
divided) into six different textures (e.g., areas or boxes, each
box representing one .png texture with alpha, where "alpha" is the
alpha channel of the image, which may allow some parts of the image
to be transparent). Certain points of each texture may be locked to
the skeleton representation such that each texture can scale and
rotate independently from the other textures. If, for example, the
user wearing the virtual outfit moves only the user's arm closer to
the camera, the system scales only the respective arm texture
without scaling the other textures (e.g., leaves the other textures
alone). This may allow for a more realistic representation of user
movement and of virtual clothing while displaying a representation
of the user trying on virtual clothes in the virtual dressing
room.
[0103] FIG. 27 is a conceptual diagram illustrating the use of the
present method and system in the context of home and garden
decorating and of curated recommendations, according to some
example embodiments. In some example embodiments, the present
system including depth sensing technology may be used to decorate
the user's room. For example, the user may request the
recommendation of an interior decorator. The depth sensor in the
user's room may capture spatial data about the room and, based on
the captured spatial data, the system may create a 3D view of the
room (e.g., a model of the room). Based on the 3D view of the room,
a decorator may make recommendations to the user on how styles may
work together (e.g., "Red and white would complement your couch" or
"Here is a table that will fit"). In various example embodiments,
the generated model of the room may assist the decorator to only
recommend items that fit the size of the room or work with other
furniture that may already be in the room.
[0104] FIG. 28 is a network diagram illustrating a network
environment 2800 suitable for action determination and initiation
on behalf of the user, within an electronic marketplace, based on
gesture recognition, according to some example embodiments. The
network environment 2800 includes an action machine 2810, a
database 2815, an electronic marketplace 2820, and devices 2830 and
2840, all communicatively coupled to each other through a network
2850. In some example embodiments, the database 2815 may house an
asset library to store 2D or 3D representations of users' bodies,
users' body measurements, or both, as well as 2D or 3D
representations of items (e.g., clothing or other objects). As
shown in FIG. 28, the action machine 2810, the database 2815, the
electronic marketplace 2820, some or all of them, may form all or
part of a network-based system 2805. The action machine 2810, the
database 2815, the electronic marketplace 2820, and the devices
2830 and 2840 may each be implemented in a computer system, in
whole or in part, as described below with respect to FIG. 35.
[0105] Also shown in FIG. 28 are users 2832 and 2842. One or both
of the users 2832 and 2842 may be a human user (e.g., a human
being), a machine user (e.g., a computer configured by a software
program to interact with the device 2830 or device 2840), or any
suitable combination thereof (e.g., a human assisted by a machine
or a machine supervised by a human). The user 2832 is not part of
the network environment 2800, but is associated with the device
2830 and may be the user of the device 2830. For example, the
device 2830 may be a desktop computer, a vehicle computer, a tablet
computer, a navigational device, a portable media device, or a
smart phone belonging to the user 2832. Likewise, the user 2842 is
not part of the network environment 2800, but is associated with
the device 2840. As an example, the device 2840 may be a desktop
computer, a vehicle computer, a tablet computer, a navigational
device, a portable media device, or a smart phone belonging to the
user 2842.
[0106] Also shown in FIG. 28 are the depth sensors 2834, 2836, and
2838 (e.g., a Kinect; a mobile device, such as a cell phone,
tablet, or PDA; or a camera; hereinafter, also "depth sensor(s)").
The system (e.g., the network environment 2800) may include one or
more depth sensors. In some example embodiments, one or more of the
depth sensor 2834, 2836, or 2838 may be part of the device 2830. In
other example embodiments, the depth sensors 2834, 2836, or 2838
may be external to the device 2830. Each depth sensor 2834, 2836,
and 2838 may capture (e.g., receive, gather, or collect) spatial
data about the physical space external to the depth sensor (e.g.,
spatial data about user 2832) and transmit the captured spatial
data to the device 2830, which in turn may transmit some or all of
the spatial data captured by the depth sensor 2834, 2836, or 2838
to the action machine 2810 via network 2850. In some example
embodiments, the depth sensor 2834, 2836, or 2838 may communicate
with and send the captured spatial data to the action machine 2810
via network 2850 without first sending the spatial data to the
device 2830.
[0107] Any of the machines, databases, or devices shown in FIG. 28
may be implemented in a general-purpose computer modified (e.g.,
configured or programmed) by software to be a special-purpose
computer to perform the functions described herein for that
machine, database, or device. For example, a computer system able
to implement any one or more of the methodologies described herein
is discussed below with respect to FIG. 35. As used herein, a
"database" is a data storage resource and may store data structured
as a text file, a table, a spreadsheet, a relational database
(e.g., an object-relational database), a triple store, a
hierarchical data store, or any suitable combination thereof.
Moreover, any two or more of the machines, databases, or devices
illustrated in FIG. 28 may be combined into a single machine, and
the functions described herein for any single machine, database, or
device may be subdivided among multiple machines, databases, or
devices.
[0108] The network 2850 may be any network that enables
communication between or among machines, databases, and devices
(e.g., the action machine 2810 and the device 2830). Accordingly,
the network 2850 may be a wired network, a wireless network (e.g.,
a mobile or cellular network), or any suitable combination thereof.
The network 2850 may include one or more portions that constitute a
private network, a public network (e.g., the Internet), or any
suitable combination thereof.
[0109] FIG. 29 is a block diagram illustrating components of the
action machine 2810, according to some example embodiments. The
action machine 2810 is shown as including a generation module 2910,
a detection module 2920, a gesture module 2930, a communication
module 2940, an action module 2950, and a recommendation module
2960, all configured to communicate with each other (e.g., via a
bus, shared memory, or a switch). The functions of these modules
2910-2960 are described below with respect to FIGS. 30-34.
[0110] Any one or more of the modules described herein may be
implemented using hardware (e.g., a processor of a machine) or a
combination of hardware and software. For example, any module
described herein may configure a processor to perform the
operations described herein for that module. Moreover, any two or
more of these modules may be combined into a single module, and the
functions described herein for a single module may be subdivided
among multiple modules. Furthermore, according to various example
embodiments, modules described herein as being implemented within a
single machine, database, or device may be distributed across
multiple machines, databases, or devices.
[0111] FIGS. 30-34 are flowcharts illustrating operations of the
action machine 2810 in performing a method 3000 of action
determination and initiation on behalf of the user, within an
electronic marketplace, based on gesture recognition, according to
some example embodiments. Operations in the method 3000 may be
performed by the action machine 2810 of FIGS. 28-29, using modules
described above with respect to FIG. 29. As shown in FIG. 30, the
method 3000 includes operations 3010, 3020, 3030, and 3040.
[0112] In operation 3010, the generation module 2910 generates a
first model based on (e.g., from) a first set of spatial data
received at a first time. For example, the first model may be
representative of the user 2832 of the electronic marketplace 2820
who may be positioned in the field of view of the depth sensor
2834. In some example embodiments, the first model may be a 2D
representation (e.g., a silhouette, outline, profile, contour, or a
skeletal representation) of the body of the user 2832. In other
example embodiments, the first model may be a 3D representation
(e.g., a relief showing the form or shape of the body in 3D,
including differences in elevation and slope between different
areas) of the body of the user 2832. Further details of operation
3010 are discussed below with respect to FIG. 31.
[0113] In operation 3020, the generation module 2910 generates a
second model based on a second set of spatial data received at a
second time (e.g., 1.5 seconds after the first time). For example,
the second model may be representative of the user 2832 of the
electronic marketplace 2820 who may be positioned in the field of
view of the depth sensor 2834. In some example embodiments, the
second model may be a 2D representation of the body of the user
2832, as described above with respect to operation 3010. In other
example embodiments, the second model may be a 3D representation of
the body of the user 2832, as described above with respect to
operation 3010. Further details of operation 3020 are discussed
below with respect to FIG. 31.
[0114] In operation 3030, the detection module 2920 detects a
difference between the first and second models generated in
operations 3010 and 3020, respectively. Further details of
operation 3030 are discussed below with respect to FIG. 31.
[0115] In operation 3040, the gesture module 2930 determines that
the difference between the first and second models, detected in
operation 3030, corresponds to a gesture performed by the user
between the first time and second time (e.g., the user started the
motion representing the gesture at or after the first time and
finished the motion by the second time). As noted above, the
gesture may represent a command to initiate an action within the
electronic marketplace on behalf of the user. Further details of
operation 3040 are discussed below with respect to FIG. 31.
[0116] As shown in FIG. 31, the method 3000 may include one or more
of operations 3110, 3120, 3130, 3140, and 3150. Operation 3110 may
be performed as part (e.g., a precursor task, a subroutine, or a
portion) of operation 3010, in which the generation module 2910
generates a first model based on a first set of data received at a
first time.
[0117] In operation 3110, the generation module 2910 of the action
machine 2810 receives a first set of spatial data from a depth
sensor 2834 at a first time. Accordingly, the depth sensor 2834 may
be configured to capture spatial data and to transmit the captured
spatial data to the generation module 2910. In some example
embodiments, the depth sensor 2834 may automatically send the
captured spatial data to the generation module 2910. In other
example embodiments, the depth sensor 2834 may respond to a request
for spatial data, the request made by the action machine 2810.
[0118] Operation 3120 may be performed as part (e.g., a precursor
task, a subroutine, or a portion) of operation 3020, in which the
generation module 2910 generates a second model based on a second
set of data received at a second time.
[0119] In operation 3120, the generation module 2910 of the action
machine 2810 receives a second set of spatial data from a depth
sensor 2834 at a second time. Accordingly, the depth sensor 2834
may be configured to capture spatial data and to transmit the
captured spatial data to the generation module 2910. In some
example embodiments, the depth sensor 2834 may automatically send
the captured spatial data to the generation module 2910. In other
example embodiments, the depth sensor 2834 may respond to a request
for spatial data, where the request is made by the action machine
2810.
[0120] Operations 3130 and 3140 may be performed as part (e.g., a
precursor task, a subroutine, or a portion) of operation 3030, in
which the detection module 2920 detects a difference between the
first and second models.
[0121] In some example embodiments, the first and second models of
the body of the user are 2D representations (e.g., silhouettes) of
the body of the user. In operation 3130, the detection module 2920
of the action machine 2810 compares a first 2D silhouette of the
body of the user and a second 2D silhouette of the body of the user
to discover the existence of a difference between the first and
second models (e.g., silhouettes). The first 2D silhouette was
generated by the generation module 2910 in operation 3010 based on
the first set of data received at the first time in operation 3110.
The second 2D silhouette was generated by the generation module
2910 in operation 3020 based on the second set of data received at
the second time in operation 3120.
[0122] In other example embodiments, the first and second models of
the body of the user are 3D representations (e.g., reliefs) of the
body of the user. In operation 3140, the detection module 2920 of
the action machine 2810 compares a first 3D representation of the
body of the user and a second 3D representation of the body of the
user to discover the existence of a difference between the first
and second models (e.g., 3D representations or reliefs of the body
of the user). The first 3D representation of the body of the user
was generated by the generation module 2910 in operation 3010 based
on the first set of data received at the first time in operation
3110. The second 3D representation of the body of the user was
generated by the generation module 2910 in operation 3020 based on
the second set of data received at the second time in operation
3120.
[0123] Operation 3150 may be performed as part (e.g., a precursor
task, a subroutine, or a portion) of operation 3040, in which the
gesture module 2930 determines that the difference between the
first and second models corresponds to a gesture performed by the
user between the first time and the second time. As noted above,
the gesture represents a command to initiate an action within the
electronic marketplace on behalf of the user.
[0124] In operation 3150, the gesture module 2930 accesses a data
record. In some example embodiments, the data record is hosted in
the database 2815 and accessed from the database 2815. The data
record may correlate the gesture to an action to be initiated by
the action module 2950, and the gesture module 2930 may determine
the action based on the data record correlating the gesture to the
action. In various example embodiments, a first gesture of the user
may be correlated to a command to initiate a bid for an item listed
for sale on the electronic marketplace 2820 without the user using
a keyboard or mouse to interact with the user interface to place
the bid. In certain example embodiments, a second gesture of the
user indicating a physical object in the user's environment may be
correlated to a command to create a 3D representation of the
physical object. In some example embodiments, a third gesture of
the user may be correlated to a command to include the 3D
representation of the physical object in a listing describing the
physical object being for sale on the electronic marketplace
2820.
[0125] As shown in FIG. 32, the method 3000 may include one or more
of operations 3210, 3220, 3230, 3240, and 3250. Operation 3210 may
be performed as part (e.g., a subroutine or a portion) of operation
3040. In operation 3210, the communication module 2940 may prompt
the user to confirm that the gesture represents a command to
initiate the action determined in operation 3150. In some example
embodiments, operation 3210 may be implemented to prevent an
accidental gesture by the user from being interpreted as a command
(e.g., to initiate a transaction on the electronic marketplace). A
further action by the user may confirm or deny a particular
interpretation of the gesture by the action machine 2810.
[0126] Operation 3220 may be performed as part (e.g., a subroutine
or a portion) of, or subsequently to, operation 3210. In operation
3220, the gesture module 2930 may detect that the user made a
different gesture (e.g., a confirmation gesture that indicates an
affirmative response to the prompting performed in operation 3210)
in response to the prompting by the communication module 2940. In
some example embodiments, the generation module 2910 generates
additional models of the body of the user based on further sets of
spatial data collected after the prompting by the communication
module 2940. These additional models may be compared in operation
3130 or operation 3140 to detect a difference that may correspond
to a different gesture by the user. In some example embodiments,
the different gesture indicates that the user has confirmed the
gesture representing the command.
[0127] Operation 3230 may be performed as part (e.g., a subroutine
or a portion) of, or subsequently to, operation 3210. In operation
3230, the gesture module 2930 may detect that the user repeated the
gesture (e.g., made again the gesture representing the command) in
response to the prompting by the communication module 2940. In some
example embodiments, the repeating of the gesture representing the
command indicates that the user has confirmed the gesture
representing the command.
[0128] Operation 3240 may be performed as part (e.g., a subroutine
or a portion) of, or subsequently to, operation 3210. In operation
3240, the gesture module 2930 may detect that the user issued a
voice command (e.g., a vocalization that indicates an affirmative
response to the prompting performed in operation 3210) in response
to the prompting by the communication module 2940. In some example
embodiments, a voice command by the user in response to the
prompting indicates that the user has confirmed the gesture
representing the command.
[0129] Operation 3250 may be performed as part (e.g., a subroutine
or a portion) of, or subsequently to, operation 3210. In operation
3250, the action module 2950 initiates the action within the
electronic marketplace 2820 on behalf of the user, based on the
user confirming that the gesture represents the command. Further
details of operation 3250 are discussed below with respect to FIGS.
33 and 34.
[0130] As shown in FIG. 33, the method 3000 may include one or more
of operations 3310, 3320, 3330, 3340, 3350, and 3360. One or more
of operations 3310, 3320, and 3330 may be performed as part of
operation 3250, in which the action module 2950 initiates the
action within the electronic marketplace 2820 on behalf of the
user, which initiation of the action may be based on the user
confirming that the gesture represents the command. In operation
3310, the action module 2950 initiates an order to buy an item for
sale on the electronic marketplace 2820. In some example
embodiments, the user's gesture indicates (e.g., is directed to)
the item for sale displayed within a UI. In some example
embodiments, the user identifies the item with a voice command.
[0131] In operation 3320, the action module 2950 lists an item for
sale on the electronic marketplace 2820. In various example
embodiments, the user's gesture indicates (e.g., it is directed to)
the item (e.g., a physical object in the user's environment or an
item displayed on a UI) to be listed for sale on the electronic
marketplace 2820. In some example embodiments, the user identifies
the item with a voice command. In various example embodiments, when
the user indicates a physical object in the user's environment to
be listed for sale on the electronic marketplace 2820, the action
machine 2810 determines one or more characteristics (e.g., spatial
measurements) of the physical object based on the spatial data
captured by one or more of the depth sensors 2834, 2836, or 2838
and transmitted to the action machine 2810.
[0132] In operation 3330, the action module 2950 generates a
representation of a physical object based on the first set of
spatial data, the second set of spatial data, or both. The physical
object may be indicated by the user's gesture. As noted above, the
representation of the physical object may be two-dimensional or
three-dimensional. In some example embodiments, the generated
representation of the physical object is stored (e.g., as a data
record) in the database 2815.
[0133] One or more of operations 3340, 3350, and 3360 may be
performed as part of operation 3330, in which the action module
2950 generates the representation of the physical object indicated
by the gesture of the user. In operation 3340, the action module
2950 lists the physical object (e.g., as being available for sale)
on the electronic marketplace 2820. The resultant listing of the
physical object may include a 3D representation of the physical
object. In some example embodiments, the action module 2950 creates
a listing for the physical object indicated by the user, retrieves
a 3D representation of the physical object from a 3D asset library
(e.g., the database 2815), and includes the 3D representation of
the physical object in the listing. The listing may be published on
the electronic marketplace 2820. In some example embodiments, the
listing may be stored as part of a catalogue stored in the database
2815.
[0134] In operation 3350, the action module 2950 determines a
characteristic of the physical object based on the first set of
spatial data, the second set of spatial data, or both. An example
of a characteristic includes the physical object's spatial
dimensions.
[0135] In operation 3360, the action module 2950 includes a
description of the characteristic in a listing of the physical
object on the electronic marketplace 2820. In various example
embodiments, the characteristic is a spatial dimension of the
physical object, and the description of the characteristic may
quantify the spatial dimension (e.g., describe the size of the
physical object) in the listing.
[0136] As shown in FIG. 34, the method 3000 may include one or more
of operations 3410, 3420, and 3430. One or more of operations 3410,
3420, and 3430 may be performed as part of operation 3250, in which
the action module 2950 initiates the action within the electronic
marketplace 2820 on behalf of the user, which initiation of the
action may be based on the user confirming that the gesture
represents the command. In operation 3410, the action module 2950
creates a 3D representation of an item of clothing based on the
first set of spatial data, the second set of spatial data, or
both.
[0137] In operation 3420, the action module 2950 sizes (e.g.,
adjusts the size of) the 3D representation of the item of clothing
according to a measurement of the user's body. The measurement may
be based on the first set of spatial data, the second set of
spatial data, or both.
[0138] In operation 3430, the action module 2950 displays the 3D
representation of the item of clothing as being mapped to an image
of the user. The 3D representation of the item of clothing may be
mapped to the appropriate area of the body of the user, as depicted
in the image of the user. For example, a 3D representation of a
pair of trousers may be mapped to the legs of the user in the image
of the user. In some example embodiments, the image of the user
shows a 2D representation (e.g., a photograph or video feed) of the
user. In other example embodiments, the image of the user shows a
3D representation (e.g., a 3D model or 3D avatar) of the user.
[0139] According to various example embodiments, one or more of the
methodologies described herein may facilitate action determination
based on gesture recognition from spatial data. Moreover, one or
more of the methodologies described herein may facilitate action
initiation in an electronic marketplace on behalf of the user based
on the action determination. Hence, one or more of the
methodologies described herein may facilitate generation of
information using gesture recognition, the generated information
being used during the initiation of actions on an electronic
marketplace on behalf of the user.
[0140] When these effects are considered in aggregate, one or more
of the methodologies described herein may obviate a need for
certain efforts or resources that otherwise would be involved in
action determination and initiation in an electronic marketplace
based on gesture recognition. Computing resources used by one or
more machines, databases, or devices (e.g., within the network
environment 2800) may be reduced. Examples of such computing
resources include processor cycles, network traffic, memory usage,
data storage capacity, power consumption, and cooling capacity.
[0141] FIG. 35 is a block diagram illustrating components of a
machine 3500, according to some example embodiments, able to read
instructions from a machine-readable medium (e.g., a
machine-readable storage medium, a computer-readable storage
medium, or any suitable combination thereof) and perform any one or
more of the methodologies discussed herein, in whole or in part.
Specifically, FIG. 35 shows a diagrammatic representation of the
machine 3500 in the example form of a computer system and within
which instructions 3524 (e.g., software, a program, an application,
an applet, an app, or other executable code) for causing the
machine 3500 to perform any one or more of the methodologies
discussed herein may be executed, in whole or in part. In
alternative embodiments, the machine 3500 operates as a standalone
device or may be connected (e.g., networked) to other machines. In
a networked deployment, the machine 3500 may operate in the
capacity of a server machine or a client machine in a server-client
network environment, or as a peer machine in a distributed (e.g.,
peer-to-peer) network environment. The machine 3500 may be a server
computer, a client computer, a personal computer (PC), a tablet
computer, a laptop computer, a netbook, a set-top box (STB), a
personal digital assistant (PDA), a cellular telephone, a
smartphone, a web appliance, a network router, a network switch, a
network bridge, or any machine capable of executing the
instructions 3524, sequentially or otherwise, that specify actions
to be taken by that machine. Further, while only a single machine
is illustrated, the term "machine" shall also be taken to include a
collection of machines that individually or jointly execute the
instructions 3524 to perform all or part of any one or more of the
methodologies discussed herein.
[0142] The machine 3500 includes a processor 3502 (e.g., a central
processing unit (CPU), a graphics processing unit (GPU), a digital
signal processor (DSP), an application specific integrated circuit
(ASIC), a radio-frequency integrated circuit (RFIC), or any
suitable combination thereof), a main memory 3504, and a static
memory 3506, which are configured to communicate with each other
via a bus 3508. The machine 3500 may further include a graphics
display 3510 (e.g., a plasma display panel (PDP), a light emitting
diode (LED) display, a liquid crystal display (LCD), a projector,
or a cathode ray tube (CRT)). The machine 3500 may also include an
alphanumeric input device 3512 (e.g., a keyboard), a cursor control
device 3514 (e.g., a mouse, a touchpad, a trackball, ajoystick, a
motion sensor, or other pointing instrument), a storage unit 3516,
a signal generation device 3518 (e.g., a speaker), and a network
interface device 3520.
[0143] The storage unit 3516 includes a machine-readable medium
3522 on which is stored the instructions 3524 embodying any one or
more of the methodologies or functions described herein. The
instructions 3524 may also reside, completely or at least
partially, within the main memory 3504, within the processor 3502
(e.g., within the processor's cache memory), or both, during
execution thereof by the machine 3500. Accordingly, the main memory
3504 and the processor 3502 may be considered as machine-readable
media. The instructions 3524 may be transmitted or received over a
network 3526 (e.g., network 2850) via the network interface device
3520.
[0144] As used herein, the term "memory" refers to a
machine-readable medium able to store data temporarily or
permanently and may be taken to include, but not be limited to,
random-access memory (RAM), read-only memory (ROM), buffer memory,
flash memory, and cache memory. While the machine-readable medium
3522 is shown in an example embodiment to be a single medium, the
term "machine-readable medium" should be taken to include a single
medium or multiple media (e.g., a centralized or distributed
database, or associated caches and servers) able to store
instructions. The term "machine-readable medium" shall also be
taken to include any medium, or combination of multiple media, that
is capable of storing instructions for execution by a machine
(e.g., machine 3500), such that the instructions, when executed by
one or more processors of the machine (e.g., processor 3502), cause
the machine to perform any one or more of the methodologies
described herein. Accordingly, a "machine-readable medium" refers
to a single storage apparatus or device, as well as "cloud-based"
storage systems or storage networks that include multiple storage
apparatus or devices. The term "machine-readable medium" shall
accordingly be taken to include, but not be limited to, one or more
data repositories in the form of a solid-state memory, an optical
medium, a magnetic medium, or any suitable combination thereof.
[0145] Throughout this specification, plural instances may
implement components, operations, or structures described as a
single instance. Although individual operations of one or more
methods are illustrated and described as separate operations, one
or more of the individual operations may be performed concurrently,
and nothing requires that the operations be performed in the order
illustrated. Structures and functionality presented as separate
components in example configurations may be implemented as a
combined structure or component. Similarly, structures and
functionality presented as a single component may be implemented as
separate components. These and other variations, modifications,
additions, and improvements fall within the scope of the subject
matter herein.
[0146] Certain embodiments are described herein as including logic
or a number of components, modules, or mechanisms. Modules may
constitute either software modules (e.g., code embodied on a
machine-readable medium or in a transmission signal) or hardware
modules. A "hardware module" is a tangible unit capable of
performing certain operations and may be configured or arranged in
a certain physical manner. In various example embodiments, one or
more computer systems (e.g., a standalone computer system, a client
computer system, or a server computer system) or one or more
hardware modules of a computer system (e.g., a processor or a group
of processors) may be configured by software (e.g., an application
or application portion) as a hardware module that operates to
perform certain operations as described herein.
[0147] In some example embodiments, a hardware module may be
implemented mechanically, electronically, or any suitable
combination thereof. For example, a hardware module may include
dedicated circuitry or logic that is permanently configured to
perform certain operations. For example, a hardware module may be a
special-purpose processor, such as a field programmable gate array
(FPGA) or an ASIC. A hardware module may also include programmable
logic or circuitry that is temporarily configured by software to
perform certain operations. For example, a hardware module may
include software encompassed within a general-purpose processor or
other programmable processor. It will be appreciated that the
decision to implement a hardware module mechanically, in dedicated
and permanently configured circuitry, or in temporarily configured
circuitry (e.g., configured by software) may be driven by cost and
time considerations.
[0148] Accordingly, the phrase "hardware module" should be
understood to encompass a tangible entity, be that an entity that
is physically constructed, permanently configured (e.g.,
hardwired), or temporarily configured (e.g., programmed) to operate
in a certain manner or to perform certain operations described
herein. As used herein, "hardware-implemented module" refers to a
hardware module. Considering embodiments in which hardware modules
are temporarily configured (e.g., programmed), each of the hardware
modules need not be configured or instantiated at any one instance
in time. For example, where a hardware module comprises a
general-purpose processor configured by software to become a
special-purpose processor, the general-purpose processor may be
configured as respectively different special-purpose processors
(e.g., comprising different hardware modules) at different times.
Software may accordingly configure a processor, for example, to
constitute a particular hardware module at one instance of time and
to constitute a different hardware module at a different instance
of time.
[0149] Hardware modules can provide information to, and receive
information from, other hardware modules. Accordingly, the
described hardware modules may be regarded as being communicatively
coupled. Where multiple hardware modules exist contemporaneously,
communications may be achieved through signal transmission (e.g.,
over appropriate circuits and buses) between or among two or more
of the hardware modules. In embodiments in which multiple hardware
modules are configured or instantiated at different times,
communications between such hardware modules may be achieved, for
example, through the storage and retrieval of information in memory
structures to which the multiple hardware modules have access. For
example, one hardware module may perform an operation and store the
output of that operation in a memory device to which it is
communicatively coupled. A further hardware module may then, at a
later time, access the memory device to retrieve and process the
stored output. Hardware modules may also initiate communications
with input or output devices, and can operate on a resource (e.g.,
a collection of information).
[0150] The various operations of example methods described herein
may be performed, at least partially, by one or more processors
that are temporarily configured (e.g., by software) or permanently
configured to perform the relevant operations. Whether temporarily
or permanently configured, such processors may constitute
processor-implemented modules that operate to perform one or more
operations or functions described herein. As used herein,
"processor-implemented module" refers to a hardware module
implemented using one or more processors.
[0151] Similarly, the methods described herein may be at least
partially processor-implemented, a processor being an example of
hardware. For example, at least some of the operations of a method
may be performed by one or more processors or processor-implemented
modules. Moreover, the one or more processors may also operate to
support performance of the relevant operations in a "cloud
computing" environment or as a "software as a service" (SaaS). For
example, at least some of the operations may be performed by a
group of computers (as examples of machines including processors),
with these operations being accessible via a network (e.g., the
Internet) and via one or more appropriate interfaces (e.g., an
application program interface (API)).
[0152] The performance of certain of the operations may be
distributed among the one or more processors, not only residing
within a single machine, but deployed across a number of machines.
In some example embodiments, the one or more processors or
processor-implemented modules may be located in a single geographic
location (e.g., within a home environment, an office environment,
or a server farm). In other example embodiments, the one or more
processors or processor-implemented modules may be distributed
across a number of geographic locations.
[0153] Some portions of the subject matter discussed herein may be
presented in terms of algorithms or symbolic representations of
operations on data stored as bits or binary digital signals within
a machine memory (e.g., a computer memory). Such algorithms or
symbolic representations are examples of techniques used by those
of ordinary skill in the data processing arts to convey the
substance of their work to others skilled in the art. As used
herein, an "algorithm" is a self-consistent sequence of operations
or similar processing leading to a desired result. In this context,
algorithms and operations involve physical manipulation of physical
quantities. Typically, but not necessarily, such quantities may
take the form of electrical, magnetic, or optical signals capable
of being stored, accessed, transferred, combined, compared, or
otherwise manipulated by a machine. It is convenient at times,
principally for reasons of common usage, to refer to such signals
using words such as "data," "content," "bits," "values,"
"elements," "symbols," "characters," "terms," "numbers,"
"numerals," or the like. These words, however, are merely
convenient labels and are to be associated with appropriate
physical quantities.
[0154] Unless specifically stated otherwise, discussions herein
using words such as "processing," "computing," "calculating,"
"determining," "presenting," "displaying," or the like may refer to
actions or processes of a machine (e.g., a computer) that
manipulates or transforms data represented as physical (e.g.,
electronic, magnetic, or optical) quantities within one or more
memories (e.g., volatile memory, non-volatile memory, or any
suitable combination thereof), registers, or other machine
components that receive, store, transmit, or display information.
Furthermore, unless specifically stated otherwise, the terms "a" or
"an" are herein used, as is common in patent documents, to include
one or more than one instance. Finally, as used herein, the
conjunction "or" refers to a non-exclusive "or," unless
specifically stated otherwise.
* * * * *