U.S. patent application number 16/046434 was filed with the patent office on 2018-11-22 for augmented reality system, method, and apparatus for displaying an item image in a contextual environment.
This patent application is currently assigned to eBay Inc.. The applicant listed for this patent is eBay Inc.. Invention is credited to Eric J. Farraro, John Tapley.
Application Number | 20180336734 16/046434 |
Document ID | / |
Family ID | 44150414 |
Filed Date | 2018-11-22 |
United States Patent
Application |
20180336734 |
Kind Code |
A1 |
Tapley; John ; et
al. |
November 22, 2018 |
Augmented Reality System, Method, and Apparatus for Displaying an
Item Image in a Contextual Environment
Abstract
Method, apparatus, and system for providing an item image to a
client for display in a contextual environment are described. In
some embodiments, the user may select an item for display in the
contextual environment, and the user may position a camera coupled
to a processing system to capture the contextual environment. A
placeholder may be generated and associated with an item selected
by a user. In an embodiment, the generated placeholder may be
placed in a location within the contextual environment, and the
user's processing system may send a visual data stream of the
camera-captured environment to a server. In an embodiment, the
user's processing device may receive a modified data stream
including an image of the item, and the user's processing device
may display the item image in the same location as the
placeholder.
Inventors: |
Tapley; John; (San Jose,
CA) ; Farraro; Eric J.; (San Jose, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
eBay Inc. |
San Jose |
CA |
US |
|
|
Assignee: |
eBay Inc.
San Jose
CA
|
Family ID: |
44150414 |
Appl. No.: |
16/046434 |
Filed: |
July 26, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
14868105 |
Sep 28, 2015 |
|
|
|
16046434 |
|
|
|
|
12644957 |
Dec 22, 2009 |
9164577 |
|
|
14868105 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/011 20130101;
G06F 3/0325 20130101; G06F 16/58 20190101; G06T 19/006
20130101 |
International
Class: |
G06T 19/00 20060101
G06T019/00; G06F 3/01 20060101 G06F003/01; G06F 3/03 20060101
G06F003/03; G06F 17/30 20060101 G06F017/30 |
Claims
1. A method comprising: visually capturing, by a computing device,
an environment including a placeholder, the placeholder including a
first portion and a second portion, the first portion being
visually distinct from the second portion; determining, by the
computing device, a placement of the placeholder in the
environment, the placement describing an orientation of the
placeholder and a distance of the placeholder relative to the
computing device; identifying, by the computing device, an item
corresponding to the placeholder; retrieving, by the computing
device, an image of the item corresponding to the placeholder; and
generating, by the computing device, a visual display of the
environment that includes the retrieved image of the item, the
retrieved image of the item being positioned and oriented in the
visual display based on the determined placement information of the
placeholder in the environment.
2. The method of claim 1, further comprising receiving user input
selecting the item corresponding to the placeholder and requesting
the placeholder in response to receiving the user input.
3. The method of claim 1, wherein the visually capturing the
environment including the placeholder is performed using a camera
of the computing device.
4. The method of claim 1, wherein the first portion of the
placeholder includes at least two shaded objects of dissimilar
color.
5. The method of claim 1, wherein identifying the item
corresponding to the placeholder is performed using visual
information included in the second portion of the placeholder.
6. The method of claim 1, wherein the retrieved image of the item
is scaled in the visual display of the environment based on the
determined placement information of the placeholder in the
environment.
7. The method of claim 1, further comprising determining one or
more physical dimensions of the item, wherein retrieving the image
of the item corresponding to the placeholder is performed in
response to determining the one or more physical dimensions of the
item.
8. A system comprising: at least one processor; and a
computer-readable storage media having instructions stored thereon
that are executable by the at least one processor to perform
operations comprising: visually capturing an environment including
a placeholder, the placeholder including a first portion and a
second portion, the first portion being visually distinct from the
second portion; determining a placement of the placeholder in the
environment, the placement describing an orientation of the
placeholder and a distance of the placeholder relative to the
computing device; identifying an item corresponding to the
placeholder; retrieving an image of the item corresponding to the
placeholder; and generating a visual display of the environment
that includes the retrieved image of the item, the retrieved image
of the item being positioned and oriented in the visual display
based on the determined placement information of the placeholder in
the environment.
9. The system of claim 8, the operations further comprising
receiving user input selecting the item corresponding to the
placeholder and requesting the placeholder in response to receiving
the user input.
10. The system of claim 8, wherein the first portion of the
placeholder includes at least two shaded objects of dissimilar
color.
11. The system of claim 8, wherein identifying the item
corresponding to the placeholder is performed using visual
information included in the second portion of the placeholder.
12. The system of claim 8, wherein the retrieved image of the item
is scaled in the visual display of the environment based on the
determined placement information of the placeholder in the
environment.
13. The system of claim 8, the operations further comprising
determining one or more physical dimensions of the item, wherein
retrieving the image of the item corresponding to the placeholder
is performed in response to determining the one or more physical
dimensions of the item.
14. A computer-readable storage medium having instructions stored
thereon that are executable by a computing device to perform
operations comprising: visually capturing an environment including
a placeholder, the placeholder including a first portion and a
second portion, the first portion being visually distinct from the
second portion; determining a placement of the placeholder in the
environment, the placement describing an orientation of the
placeholder and a distance of the placeholder relative to the
computing device; identifying an item corresponding to the
placeholder; retrieving an image of the item corresponding to the
placeholder; and generating a visual display of the environment
that includes the retrieved image of the item, the retrieved image
of the item being positioned and oriented in the visual display
based on the determined placement information of the placeholder in
the environment.
15. The computer-readable storage medium of claim 14, the
operations further comprising receiving user input selecting the
item corresponding to the placeholder and requesting the
placeholder in response to receiving the user input.
16. The computer-readable storage medium of claim 14, wherein the
visually capturing the environment including the placeholder is
performed using a camera of the computing device.
17. The computer-readable storage medium of claim 14, wherein the
first portion of the placeholder includes at least two shaded
objects of dissimilar color.
18. The computer-readable storage medium of claim 14, wherein
identifying the item corresponding to the placeholder is performed
using visual information included in the second portion of the
placeholder.
19. The computer-readable storage medium of claim 14, wherein the
retrieved image of the item is scaled in the visual display of the
environment based on the determined placement information of the
placeholder in the environment.
20. The computer-readable storage medium of claim 14, the
operations further comprising determining one or more physical
dimensions of the item, wherein retrieving the image of the item
corresponding to the placeholder is performed in response to
determining the one or more physical dimensions of the item.
Description
RELATED APPLICATIONS
[0001] This application is a continuation of and claims the benefit
of priority to U.S. patent application Ser. No. 14/868,105, filed
Sep. 28, 2015, which claims priority under 35 U.S.C. .sctn. 120 to
U.S. patent application Ser. No. 12/644,957, filed on Dec. 22,
2009, the benefit of priority of which is claimed hereby, and which
is incorporated by reference herein in its entirety.
BACKGROUND
[0002] Item listings on online shopping and auction websites (e.g.,
e-commerce companies) may often provide images related to the item
for sale to supplement a text description of the item. The images
may be in the form of a digital photograph, a drawing, or in some
cases, a video clip or any combination of these. For some items,
there may be multiple images (e.g., digital photos) providing
different angles and views of the item. In some examples, the
images may be in a contextual environment, such as a sofa in a
living room. In other examples, the item image may merely depict
the item without a contextual environment.
BRIEF DESCRIPTION OF DRAWINGS
[0003] Embodiments of the present invention are illustrated by way
of example and not limitation in the Figures of the accompanying
drawings, in which like reference numbers indicate similar elements
and in which:
[0004] FIG. 1 is a diagrammatic representation of a network within
which an example augmented reality system for providing modified
visual data to a processing system may be implemented;
[0005] FIG. 2 is a block diagram of an example augmented reality
module, in accordance with one example embodiment;
[0006] FIG. 3 is a flow chart of a method for providing modified
visual data, in accordance with an example embodiment;
[0007] FIG. 4 is an example of a placeholder for use with the
augmented reality system, in accordance with an example
embodiment;
[0008] FIGS. 5A-B are example user interfaces illustrating a
placeholder in a camera-captured environment and the resulting
augmented reality display in a contextual environment, in
accordance with an example embodiment; and
[0009] FIG. 6 is a diagrammatic representation of an example
machine in the form of a computer system within which a set of
instructions, for causing the machine to perform any one or more of
the methodologies discussed herein, may be executed.
DETAILED DESCRIPTION
[0010] Systems and methods for displaying an item image on a screen
as the item would appear positioned at a location in a
camera-captured environment are discussed. In the following
description, for purposes of explanation, numerous specific details
are set forth in order to provide a thorough understanding of an
embodiment of the present invention. It will be evident, however,
to one skilled in the art that the present invention may be
practiced without these specific details. The following detailed
description includes reference to the accompanying drawings, which
form a part of the detailed description. The drawings show
illustrations in accordance with example embodiments. These example
embodiments, which are also referred to herein as "examples," are
described in enough detail to enable those skilled in the art to
practice the present subject matter. The embodiments may be
combined, other embodiments may be utilized, or structural, logical
and electrical changes may be made without departing from the scope
of what is claimed. The following detailed description is,
therefore, not to be taken in a limiting sense, and the scope is
defined by the appending claims and their equivalents.
[0011] Augmented reality generally refers to creating a mixed
virtual and real world environment though computer-generated
imagery. Elements of the real world are mixed with virtual images
in a real-time environment to provide a user with information and a
visual effect that may not be entirely available otherwise.
[0012] In an online publication system such as an online trading
platform, where a seller may post items for sale, it may be
desirable to view how a particular item would appear in a location
specified by a user to aid the user's purchase decision. For
example, a user considering purchasing a sofa for a living room may
need to determine where to position the sofa, and generally, how
the sofa would appear in the living room. The user may wish to view
the sofa as it would appear in the living room without actually
possessing the sofa. Furthermore, the user may prefer to compare
various sofas, view how the sofa would look from different angles
in the room, or even surround the sofa with other items to create a
furniture ensemble without physically moving the actual furniture.
In an example embodiment, an augmented reality system may provide a
user with the ability to arrange and rearrange a virtual sofa (and
additional items, if desired) in an environment (e.g., a living
room) captured by a camera instead of requiring physical movement
of an actual sofa.
[0013] In an example embodiment, the method allows a user to place
a marker (e.g., a placeholder such as a piece of paper with
contrasting markings and unique objects) in a location where the
actual item would be positioned. The method records an image of the
market positioned in the location using, e.g., a video camera,
sends the captured image to the augmented reality system, and
receives an image of the actual item positioned at the chosen
location. In various embodiments, the marker (e.g., placeholder)
can be a sheet of paper (8.5''.times.11'') with unique markings and
that a user can download, print and placed at the location. A
server receives the video stream of the placeholder in the location
from the user's system, detects the placeholder, and sends a
modified video stream with an item in place of the marker back to
the user's system for display.
[0014] In various example embodiments, an input including item
selection is received from a client to generate a placeholder and
information for generating the placeholder is provided to the input
source (e.g., the client) or a processing system with output
capabilities. Video data of a camera-captured environment is
received, in which the placeholder may be located. The input is
analyzed to detect the placeholder and an image is retrieved
according to the analysis. The visual data is modified to include
an image data associated with the image, which, in some
embodiments, may be an overlay of the image, and the modified
visual data may be provided to the input source (e.g., a user's
processing system) or another processing system. In some example
embodiment, the retrieved image may be modified in accordance with
the analysis to scale and orient the item image overlay in
accordance with the environment and placement information
determined from the placeholder information. In one example
embodiment, the placeholder may be a printed sheet of paper with
unique markings (e.g., bar code, contrastable image,
two-dimensional data code) printable by the user from a home
computer system. In some example embodiments, the sheet of paper
may also be recognizable by other processing systems. In some
embodiments, a single placeholder may be generic to multiple items
and used interchangeably to represent more than one item. For
example, when a user wishes to compare a first item with a second
item in the same location, the placeholder may interchangeably
represent both the first and second item, with needing to generate
two placeholders. In some embodiments, a single placeholder can be
used without associating the placeholder with any particular item.
An example augmented reality system is discussed with reference to
FIG. 1.
[0015] FIG. 1 is a diagrammatic representation of a network within
which an example augmented reality system 100 for providing
modified visual data to a processing system 110 may be implemented.
As shown, the augmented reality system 100 may include a processing
system 110 and a server 120. The server 120, in one example
embodiment, may host an on-line trading platform. The processing
system 110 may run an application 116, such as a web browser
application and may have access to the server 120 via a network
130. The network 130 may be a public network (e.g., the Internet, a
wireless network, etc.) or a private network (e.g., a local area
network (LAN), a wide area network (WAN), intranet, etc.).
[0016] In one embodiment, the network-based trading platform may
provide one or more marketplace applications, payment applications,
and other resources. The market place application may provide a
number of marketplace functions and services to users that access
the marketplace. The payment applications, likewise, may provide a
number of payment services and functions to users. The
network-based trading platform may display various items listed on
the trading platform. The embodiments discussed in the
specification are not limited to network-based trading platforms,
however. In additional embodiments, other publishing platforms,
such as social networking websites, news aggregating websites, web
portals, network-based advertising platforms, or any other system
that display items to users, may be used.
[0017] The processing system 110 may include a camera source 112
(e.g., a video camera or digital still camera) for capturing a
contextual environment (e.g., a living room) and a visual output
114 (e.g., an LCD display or the like) to present the contextual
environment and, in an example embodiment, to present an augmented
environment. The server 120 includes an item image source 122 and
an augmented reality application 124. In some embodiments, the
server 120 may be communicatively coupled with one or more
databases 140, which may store item images 142.
[0018] In an example embodiment, the processing system 110 may send
a request to the server 120 to modify a contextual environment
captured by a camera source 112. In some embodiments, the request
from the processing system 110 may include a user-selected item. In
an embodiment, an augmented reality application 124 at the server
120 may detect the request, and provide placeholder generation
information to the processing system 110 to generate a placeholder
associated with the user-selected item. Upon receiving the
placeholder generation information, a user in control of the
processing system 110 may produce the placeholder, e.g., by
printing an image of the placeholder provided by the augmented
reality application 124. In some example embodiments, the
placeholder may be an ordinary letter-sized sheet of paper (e.g.,
8.5''.times.11'') with unique markings, an example of which is
illustrated in FIG. 4 and described below. The unique markings may
be utilized by the augmented reality application 124 to identify
the placeholder in the contextual environment and to determine a
desired placement of the user-selected item within the contextual
environment captured by the camera. In some embodiments, the
placeholder may be generated using an auxiliary display device
(e.g., mobile phone display) displaying the placeholder with unique
markings. In other embodiments, the user may use the placeholder
generation information to physically create the placeholder by
drawing and/or coloring the unique markings onto an object, such as
a piece of paper, or a three-dimensional object. In some
embodiments, in addition to the user-selected item, the placeholder
generation information and associated unique markings may be
customized to accommodate elements of the environment based on
received visual data collected by a camera. For example, the
placeholder generation information may be produced by taking into
considerations dimensions of the environment (e.g., room).
[0019] In another example embodiment, the augmented reality
application 124 may receive and process visual data received from
the processing system 110 and the camera source 112 communicatively
coupled to the processing system 110, and provides modified visual
data back to the processing system 110. For example, the camera
source 112 may collect visual data of a contextual environment
within which a placeholder is located. In some embodiments, one of
the applications 116 may receive visual data. The application 116
may provide the visual data to the augmented reality application
124 of the server 120 via network 130. The augmented reality
application 124 may analyze the visual data to detect placeholder
information associated with a placeholder located within the
captured contextual environment, and may also determine placement
information from the placeholder information. The augmented reality
application 124 may retrieve an image from an item image source 122
of the server 120 based on the placeholder information, placement
information or any combination thereof. In some embodiments, the
augmented reality application 124 may retrieve the image from the
item images 142 stored in a database 140 that is communicatively
coupled to the server 120. The augmented reality application 124
may then modify the visual data to include image data associated
with the retrieved item image. In some embodiments, the augmented
reality application 124 may modify the visual data with modified
image data based on the determined placement information. In some
embodiments, the augmented reality application 124 then provides
modified visual data to the processing system 110 for visual
display on a visual output 114.
[0020] In some embodiments, the processing system 110 may include
an item image source, such as a library of item images stored on
the processing system 110. In an example embodiment, a user may
create a library of images of one or more items by way of the
camera source 112 (e.g., digital camera). In one embodiment, an
application hosted on the server 120 may be used to identify an
item depicted in the image (see U.S. patent application Ser. No.
12/371,882) and the augmented reality application 124 may provide
placeholder generation information from the server 120 to the
processing system. In some example embodiments, the processing
system 110 may host the augmented reality application 124 and may
provide modified visual data, such as to the processing system 110
without use of the network 130 and the server 120.
[0021] FIG. 2 is a block diagram 200 of an example augmented
reality module 210, in accordance with one example embodiment. As
shown in FIG. 2, augmented reality module 210 comprises in input
receiver module 220, an information provider module 222, a visual
data receiver module 224, a detection module 226, a placement
determination module 228, a retrieval module 230, and a
modification module 232. These modules of the augmented reality
system may be communicatively coupled to other modules via a bus
240.
[0022] An input receiver module 220 may be configured to receive a
request to commence an augmented reality application (e.g.,
augmented reality application 124 of FIG. 1). The request may
include item information associated with a selected item. In one
embodiment, the input receiver module 220 may be configured to
receive the request (e.g., an input) from a source, such as, for
example, the processing system 110 of FIG. 1. The input receiver
module 220 may be configured to receive additional information from
the processor system, such as, e.g., camera specifications,
lighting and dimensional information of a captured contextual
environment, data transmission rates, frame refresh rate, among
other specifications of the processing system 110, and environment
parameters known to those of ordinary skill in the art.
[0023] An information provider module 222 may be configured to
determine placeholder generation information based on the received
request and item information from the input receiver module 220.
Placeholder generation information may include instructions (e.g.,
machine readable instructions) to generate a placeholder. In some
embodiments, the placeholder generation information may be
machine-readable information, such as, for example, an electronic
document. In an embodiment, the information provider module 222 may
produce a file deliverable to the requesting source (e.g.,
processing system 110) or any other processing system. For example,
the information provider module 222 may generate an electronic
document or an image file in any one or combination of formats
(e.g., Portable Document Format (PDF), Microsoft.RTM. Word
document, Joint Photographic Experts Group (JPEG) or the like). In
operation, a user may interface with a processing system to select
an item listing. The user may send a request to generate a
placeholder for placement within an environment. The input receiver
module 220 may receive the request, which, in some embodiments, may
include information identifying the item listing selected by the
user, and the information provider module 222 may generate an
electronic document and deliver the document to the requesting
source (e.g., processing system 110). In some embodiments, the
request to generate the placeholder may include the item listing
selected. In some embodiment, a user may receive placeholder
generation instruction without selecting or sending a request to
generate a placeholder.
[0024] A visual data receiver module 224 receives visual data from
a client, such as the processing system 110, communicatively
coupled to a camera, such as a digital camera, video camera (e.g.,
web cam), or the like. In one example embodiment, a user may
position the camera to capture and view a contextual environment in
which the user may want to view an item as it would appear in that
environment. In some embodiments, the visual data receiver module
224 may process, some or all of the camera-captured visual data to
determine parameters of the visual environment. For example, the
visual data receiver module 224 may determine parameters of the
captured environment including lighting, angles, dimensions,
gravitational orientation, and the like, and may determine
specifics of the camera, such as frame refresh rate, zoom
capabilities, camera model information, and the like.
[0025] A detection module 226 detects placeholder information that
may be included with the received visual data. In an embodiment,
the placeholder information (e.g., machine-readable instructions to
generate a document) is associated with a placeholder placed within
a viewable location of a camera (e.g., within the camera-captured
contextual environment), the camera communicatively coupled with a
processing system (e.g., client device), the processing system
providing visual data received by the camera to the augmented
reality module 210 at the visual data receiver module 224. In one
example, unique markings of the placeholder may be captured by a
camera coupled to the processing system, and these unique markings
may be detected by the detection module 226 as representing a
placeholder in the captured environment.
[0026] In one embodiment, the detection module 226 may be
configured to parse visual data received from the visual data
receiver module 224 to determine whether the received visual data
includes placeholder information within the visual data. In some
embodiments, the detection module 226 may verify the placeholder
information, such as to determine whether the placeholder
information matches with the selected item received at input
receiver module 220. In some embodiments, the detection module 226
may be configured to notify the client (e.g. processing system 110)
when the placeholder information does not match the selected item.
In an example, the detection module 226 may provide suggestions to
the client source, such as to recreate the placeholder, reposition
the placeholder, or the like.
[0027] A placement determination module 228 may be configured to
determine placement information of a placeholder using the
placeholder information received with the visual data. In one
embodiment, the placement determination module 228 may determine
various characteristics associated with the placeholder based on
the placeholder information. For example, the placement
determination module 228 may determine characteristics including
coordinates and general location of the placeholder in the
contextual environment captured by the camera, and may determine an
orientation, scale factor, lighting effects, and any obstructions
obstructing the placeholder in the contextual environment, or the
like. In one embodiment, the placement determination module 228 may
be configured to provide the placement information including
characteristics to a retrieval module 230. In one example
embodiment, the placement determination module 228 may be
configured to determine the placement information of a placeholder
based on unique markings of the placeholder. These unique markings
may be detected by the detection module 226 as described above.
[0028] A retrieval module 230 may be configured to retrieve an
image of the item upon detection of the image of the placeholder
captured by the camera. The retrieval module 230 may retrieve the
image based on the unique markings of the placeholder. The
retrieval module 230 may incorporate information received from the
detection module 226 and the placement determination module 228
when retrieving an image of the selected item. In one example, the
retrieval module 230 may select an image from a plurality of images
of the item based on determined characteristics of the placeholder
determined from the placeholder information and received from the
placement determination module 228, as described above. In an
example embodiment, the retrieval module 230 may use the determined
characteristics to select a best fit image from a plurality of
images. In some embodiments, the retrieval module 230 may retrieve
or determine physical dimensions of the item, such as the height,
the length, and the width, and may select a best fit image from the
plurality of images according to the physical dimensions of the
item and/or the characteristics of the placeholder. In one example,
the retrieval module 230 may be configured to modify image data
associated with the retrieved image according to the physical
dimensions of the item and the characteristics of the placeholder,
such as to represent a scaled and oriented image within the
contextual environment.
[0029] A modification module 232 may be configured to modify the
received visual data to include image data associated with the
retrieved image into the visual data according to the placement
information. In some embodiments, the modification module 232 may
be configured to modify the received visual data to include the
modified visual data generated by retrieval module 230. In one
example embodiment, the modification module 232 may return the
modified visual data to the client (e.g. processing system 110)
providing the item selection and/or the selection to augment
reality.
[0030] It will be noted, that while FIG. 2 shows particular modules
independent of other modules, other embodiments may include one or
more modules in a single component. Additionally, embodiments may
be provided where a component that is shown in FIG. 2 as a single
module maybe implemented as two or more components. Various
operations performed by the augmented reality module 210 may be
discussed with reference to FIGS. 3-6.
[0031] FIG. 3 is a flow chart of a method 300 for providing
modified visual data, in accordance with an example embodiment. The
method 300 may be performed by processing logic that may comprise
hardware (e.g., dedicated logic, programmable logic, microcode,
etc.) software (such as run on a general purpose computer system or
a dedicated machine), or a combination of both. In one example
embodiment, the processing logic may reside at a server 120 as
shown in FIG. 1 in the form of an augmented reality application 124
of FIG. 1.
[0032] As shown in FIG. 3, the method 300 commences at operation
310 when the server 120 receives an input from a client (e.g., the
processing system 110 of FIG. 1), the input including an item
selection. At operation 320, the method 300 may provide generation
information for generating a placeholder based on the received
input. The generation information may be provided to the processing
system 110 to permit a user to generate the placeholder using a
printer, or view the placeholder using a visual display device
(e.g., a mobile phone screen, a camera screen or the like). An
example of a placeholder is depicted in FIG. 4 and described
below.
[0033] At operation 330, the method 300 receives visual data from
the client (e.g., processing system 110). In one embodiment, the
visual data may be captured by a camera communicatively coupled
with the client. At operation 340, the method 300 detects
placeholder information included with the visual data; the
placeholder information is associated with a placeholder placed in
a contextual environment captured by the camera. At operation 350,
the method 300 determines placement information using the
placeholder information. At operation 360, the method 300 retrieves
an image of an item associated with the placeholder based on the
placeholder. At operation 370, the method 300 modifies the received
visual data to include the image data associated with the retrieved
image into the visual data with reference to the placement
information. In an example, the retrieved image may be modified to
reconcile the respective dimensions, orientation, etc. of the
captured environment with the dimensions, orientation etc. of an
item represented by the retrieve image. In some embodiments, the
operation 370 of method 300 may provide the modified visual data to
the client (e.g., processing system 110) providing the input at
operation 310. Now the user may view the item as it would appear in
the environment.
[0034] FIG. 4 is an example of a physical placeholder 400 for use
with the augmented reality system 100 (e.g. processing system 110,
server 120, and network 130 all of FIG. 1), in accordance with an
example embodiment. In one example, upon receiving an input
including an item selection, the augmented reality system 100
provides generation information to generate a placeholder 400 with
unique markings 402-424 to a client source from which the input
originated or, in some embodiment, any other processing system
(e.g., a mobile phone). As previously described, the generation
information may be a document, image, computer-readable
instructions, or the like, for generating the placeholder 400 by
use of rendering instrument such as, for example, a printer (e.g.,
in the case of a printable sheet) or a display screen (e.g., a
display screen of a mobile phone, in the case of a non-printable
digital file). In some embodiments, the placeholder 400 may be a
standard sheet of paper (e.g., dimension of about 8.5''.times.11'')
with the unique markings 402-424. In other embodiments, the
placeholder 400 may be an electronically displayable set of
instructions for display on a mobile phone or an auxiliary display
screen. In other embodiments, the placeholder 400 may be a three
dimensional object.
[0035] The placeholder 400 may include any one or more of arrows
402, 404, 406, and 408, which provide the augmented reality system
100 an upside and a downside orientation for the placeholder 400,
such as to orient the placeholder 400 when captured by the camera.
In one embodiment, the augmented reality system 100 may calibrate a
camera source 112 of FIG. 1 of the processing system 110 with any
combination of arrows 402, 404, 406, and 408. In an example
embodiment, the arrows 402, 404, 406, and 408 may be formed in a
clockwise direction such as to allow the augmented reality system
100 to first determine a top side and bottom side orientation. When
the page is flipped, the orientation determined by the augmented
reality system 100 may be flipped by 180.degree.. In one example
embodiment, any directionally distinguishable shape may be
substituted for one or more of the arrows 402, 404, 406, and 408.
In one embodiment, the augmented reality system 100 may
differentiate between the four corners of the placeholder 400 with
use of the arrows 402, 404, 406, and 408 when rotating the page
across various axes from an original orientation. In an example,
when a user rotates the placeholder 400 in the camera-captured
environment, the augmented reality system 100 detects the rotation
and rotates the corresponding retrieved item image when presenting
the item in the contextual environment on a visual output 114 of
FIG. 1.
[0036] In one example embodiment, the placeholder 400 includes a
middle rectangle 410, which may provide for an initial detection of
the placeholder 400. In some embodiments, the augmented reality
system 100 may initially detect the middle rectangle 410 before
other unique markings when the placeholder 400 is placed a great
distance away from the camera in an open environment, such as for
example, an auditorium or in an outdoor environment. In other
example embodiments, the middle rectangle 410 may be substituted
with any design relatively large in size with relation to the
placeholder 400 and generally located near the middle of the
placeholder 400.
[0037] As depicted in FIG. 4, the placeholder 400 includes shaded
objects 412 and 414 shaded with dissimilar colors to provide
distinct regions of the placeholder 400 for detecting an
orientation, scale, and distance of the placeholder 400. The
augmented reality system may employ the shaded objects 412 and 414
for determining a left-right and depth dimensional orientation.
[0038] The placeholder 400 may further include line markings 416,
418, 420 and 422 to assist with augmented reality system in
refining a third dimension for the placeholder 400. In one
embodiment, the lines within each of a first and second set of
lines 416 and 418 are equidistant for each set. The first and
second sets of lines 416 and 418 may provide a depth of the object
and a scale factor. By known methods, the augmented reality system
may orient and scale the image based on any or all of the arrows
402, 404, 406, and 408; middle rectangle 410; shaded objects 412
and 414 and line markings 416, 418, 420, and 422.
[0039] In one embodiment, the unique markings of placeholder 400
may include identification markings 424, which may be item
identification information used when verifying whether the
generated placeholder 400 matches with a selected item. In other
embodiments, these additional identification markings 424 may
include or may reference additional information related to the
selected item, such as for example, price, inventory, and suggested
items related to the selected item. In some example embodiments,
the additional information may be presentable along with the image
of the item in the contextual environment. In some embodiments, the
additional identification markings 424 may aid in uniquely
identifying more than one placeholder 400 in the environment, and
thereby, allow simultaneously presentation of a plurality of items
in the contextual environment. In some embodiments, the
identification markings 424 may be a one-dimensional barcode, two
dimensional barcode, or any other marking uniquely associable with
a selected item. In some embodiments, a placeholder may be
generated that may not be associable with one particular item and
may be interchangeably used to represent more than one item.
[0040] FIGS. 5A-B are screen shots 500A and 500B of example user
interfaces 502A and 502B illustrating a placeholder 550 in a
camera-captured environment and the resulting augmented reality
image 560 displayed in the contextual environment, in accordance
with an example embodiment. In an example, a user may direct a
camera toward a location such as the user's living room, which
provides the contextual environment for the augmented reality. The
user interface 502A illustrates a visual frame 504A of the
camera-captured environment (e.g., a living room). In some
embodiments, the user interface 502A and 502B may include selection
buttons 506, 508, and 510. Selection button 506 may be selected to
augment reality, selection button 508 may be selected to purchase
an item, and selection button 510 may be selected to place a bid on
an item.
[0041] For example, a user may select an item, such as a lamp, and
orient a camera to view an environment in which the user may want
to place the lamp (e.g., a bedroom). The user may select the
augment reality selection button 506 at a client (e.g., the
processing system 110 of FIG. 1) to provide an input along with the
item selection information to a server (e.g., server 120 of FIG.
1). The server 120 may provide generation information to the client
(e.g., processing system 110) to generate a placeholder, such as
the placeholder 400 depicted in FIG. 4 and the user may place the
placeholder in the camera-captured environment. Screen shot 500A
illustrates a user interface 502A of FIG. 5A with a placeholder 550
viewable within a camera-captured environment in visual frame
504A.
[0042] The augmented reality application 124 of FIG. 1 at the
server 120 receives the visual data of the camera-captured
environment (shown in visual frame 504A including the placeholder
550) viewable from the client device (e.g., processing system 110).
The augmented reality application 124 detects the placeholder
information associated with the placeholder 550 included with the
visual data and determines placement information from the
placeholder information. The augmented reality application 124 then
retrieves an image of the item associated with the placeholder 550.
In some embodiments, the unique markings of the placeholder 550 may
be used to determine which item is associated with the placeholder,
or which image of the item may be selected from a plurality of
images of an item. The augmented reality application 124 provides
the client device (e.g., processing system 110) with modified
visual data, which includes the image data associated with the
retrieved image of the selected item. In some embodiments, the
augmented reality application 124 replaces the placeholder
information with the image data, such as to modify the received
visual data and to display a replacement image of the item within
the camera-captured viewable in visual frame 504B. In an example
embodiment, as shown in FIG. 5B, the augmented reality application
124 may modify the received visual data by providing overlay image
data in place of the placeholder information, such as to display an
overlay of the item image (e.g., augmented reality image 560)
within the camera-captured viewable in visual frame 504B.
[0043] FIG. 6 shows a diagrammatic representation of a machine in
the example form of a computer system 600 within which a set of
instructions, for causing the machine to perform any one or more of
the methodologies discussed herein, may be executed. In alternative
embodiments, the machine operates as a stand-alone device or may be
connected (e.g., networked) to other machines. In a networked
deployment, the machine may operate in the capacity of a server or
a client machine in a server-client network, or as a peer machine
in a peer-to-peer (or distributed) network. The machine may be a
personal computer (PC), a tablet PC, a set-top box (STB), a
Personal Digital Assistant (PDA), a cellular telephone, a web
appliance, a network router, switch or bridge, or any machine
capable of executing a set of instructions (sequential or
otherwise) that specify actions to be taken by that machine.
Further, while only a single machine is illustrated, the term
"machine" shall also be taken to include any collection of machines
that individually or jointly execute a set (or multiple sets) of
instructions to perform any one or more of the methodologies
discussed herein.
[0044] The example computer system 600 includes a processor 602
(e.g., a central processing unit (CPU), a graphics processing unit
(GPU) or both), a main memory 604 and a static memory 606, which
communicate with each other via a bus 608. The computer system 600
may further include a video display unit 610 (e.g., a liquid
crystal display (LCD) or a cathode ray tube (CRT)). The computer
system 600 also includes an alpha-numeric input device 612 (e.g., a
keyboard), a user interface (UI) navigation device 614 (e.g., a
cursor control device), a disk drive unit 616, a signal generation
device 618 (e.g., a speaker) and a network interface device
620.
[0045] The disk drive unit 616 includes a machine-readable medium
622 on which is stored one or more sets of instructions and data
structures (e.g., software 624) embodying or utilized by any one or
more of the methodologies or functions described herein. The
software 624 may also reside, completely or at least partially,
within the main memory 604 and/or within the processor 602 during
execution thereof by the computer system 600, with the main memory
604 and the processor 602 also constituting machine-readable
media.
[0046] The software 624 may further be transmitted or received over
a network 626 via the network interface device 620 utilizing any
one of a number of well-known transfer protocols (e.g., Hyper Text
Transfer Protocol (HTTP)).
[0047] While the machine-readable medium 622 is shown in an example
embodiment to be a single medium, the term "machine-readable
medium" should be taken to include a single medium or multiple
media (e.g., a centralized or distributed database, and/or
associated caches and servers) that store the one or more sets of
instructions. The term "machine-readable medium" shall also be
taken to include any medium that is capable of storing, encoding or
carrying a set of instructions for execution by the machine and
that cause the machine to perform any one or more of the
methodologies of embodiments of the present disclosure, or that is
capable of storing, encoding or carrying data structures utilized
by or associated with such a set of instructions. The term
"machine-readable medium" shall accordingly be taken to include,
but not be limited to, solid-state memories, optical and magnetic
media, and carrier wave signals. Such media may also include,
without limitation, hard disks, floppy disks, flash memory cards,
digital video disks, random access memory (RAMs), read only memory
(ROMs), and the like.
[0048] The embodiments described herein may be implemented in an
operating environment comprising software installed on a computer,
in hardware, or in a combination of software and hardware.
[0049] Thus, an augmented reality system has been described.
Although embodiments have been described with reference to specific
example embodiments, it will be evident that various modifications
and changes may be made to these embodiments without departing from
the broader spirit and scope of the inventive subject matter.
Accordingly, the specification and drawings are to be regarded in
an illustrative rather than a restrictive sense.
* * * * *