U.S. patent application number 15/453880 was filed with the patent office on 2017-08-10 for system and method for object extraction.
The applicant listed for this patent is EYECUE VISION TECHNOLOGIES LTD.. Invention is credited to Shai FEDER, Ronen HOROVITZ, Ran KAFTORY.
Application Number | 20170228880 15/453880 |
Document ID | / |
Family ID | 56554555 |
Filed Date | 2017-08-10 |
United States Patent
Application |
20170228880 |
Kind Code |
A1 |
HOROVITZ; Ronen ; et
al. |
August 10, 2017 |
SYSTEM AND METHOD FOR OBJECT EXTRACTION
Abstract
A system and method for capturing images from various angles of
an object that is resting on an orientation indication background,
using a camera of a mobile device and visible light. The images are
processed to construct a three-dimensional image model and a
representation of the three-dimensional image model are displayed
on an electronic display screen and manipulated or embedded into a
video game.
Inventors: |
HOROVITZ; Ronen; (Zichron
Yaakov, IL) ; KAFTORY; Ran; (Kiryat Tivon, IL)
; FEDER; Shai; (Haifa, IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
EYECUE VISION TECHNOLOGIES LTD. |
Yokneam |
IL |
US |
|
|
Family ID: |
56554555 |
Appl. No.: |
15/453880 |
Filed: |
March 8, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15097950 |
Apr 13, 2016 |
9595108 |
|
|
15453880 |
|
|
|
|
13201512 |
Aug 15, 2011 |
9498721 |
|
|
PCT/US2010/044343 |
Aug 4, 2010 |
|
|
|
15097950 |
|
|
|
|
62275270 |
Jan 6, 2016 |
|
|
|
61231216 |
Aug 4, 2009 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 7/70 20170101; A63F
13/213 20140902; G06T 7/507 20170101; G06T 7/85 20170101; G06T
7/194 20170101; G06T 2200/08 20130101; G06K 9/4604 20130101; A63F
13/65 20140902; G06T 17/00 20130101; G06K 9/00208 20130101; G06T
7/55 20170101; G06T 2200/04 20130101 |
International
Class: |
G06T 7/55 20060101
G06T007/55; G06T 7/80 20060101 G06T007/80; G06T 7/194 20060101
G06T007/194 |
Claims
1. A computer-implemented method comprising: (A) capturing, with a
two-dimensional (2-D) image capture device, multiple distinct 2-D
images of an arbitrary object, said object at a fixed position on a
region having a known background pattern thereon, said multiple 2-D
images corresponding to multiple orientations or positions of said
2-D image capture device relative to said object; (B) in at least
some of said multiple 2-D images, separating the object from the
known background pattern to form multiple silhouettes corresponding
to said object; (C) obtaining a three-dimensional (3D) image model
of said object, said obtaining using at least some of the multiple
silhouettes; and then (D) embedding a digital representation of
said three-dimensional image model into an electronic display of
said object.
2. The method of claim 1 wherein the capturing in (A) uses visible
light.
3. The method of claim 1 wherein the known background pattern is
visible.
4. The method of claim 1 wherein, in (B), said separating uses
spectral features of the object or the background pattern.
5. The method of claim 1 wherein, in (B), said separating uses
spatial features of the object or the background pattern.
6. The method of claim 1 wherein, in (B), said separating uses
color features of the object or the background pattern.
7. The method of claim 1 wherein the object made from material
selected from: plastic, paper, clay, and metal.
8. The method of claim 1 wherein the region having a known
background pattern thereon is a mat.
9. The method of claim 1 wherein the region having a known
background pattern thereon comprises a retroreflective
material.
10. The method of claim 9 wherein the 2-D image capture device uses
a lighting source to perform said capturing in (A).
11. The method of claim 10 wherein said lighting source is an
LED.
12. The method of claim 1 wherein the known background pattern
comprises at least two colors.
13. The method of claim 12 wherein the known background pattern
comprises a grid.
14. The method of claim 13 wherein the known background pattern
comprises two color grid.
15. The method of claim 1 wherein the known background pattern
comprises a visible orientation indicator.
16. The method of claim 1 wherein the region having a known
background pattern thereon comprises known features including one
or more of: at least one specific color, and at least one specific
spatial pattern.
17. The method of claim 1 wherein the known background pattern has
known spatial features.
18. The method of claim 1 wherein the region having a known
background pattern thereon comprises a color calibration
palette.
19. The method of claim 1 further comprising: color calibrating
said three-dimensional image model using said known background
pattern.
20. The method of claim 1 further comprising: calibrating said
image capture device during said capturing.
21. The method of claim 1 wherein obtaining the 3D image model of
the object in (C) comprises: performing voxel carving on said at
least some of said multiple silhouettes to obtain the 3D image
model of said object.
22. The method of claim 21 wherein said voxel carving is performed
in (C) while 2-D images of the object are being captured in
(A).
23. The method of claim 22 wherein said voxel carving is performed
in (C) in real time while 2-D images of the object are being
captured.
24. The method of claim 1 where said separating in (B) comprises
removing the known background pattern from said at least some of
said multiple 2-D images.
25. The method of claim 1 wherein said separating in (B) is
performed simultaneously or in real time with the capturing of the
images in (A)
26. The method of claim 1, wherein said electronic display is for a
video game, and wherein the digital representation can be
manipulated by a user in said video game.
27. The method of claim 1 wherein the 2-D image capture device is
in a device selected from: a notebook, a netbook, a personal
digital assistant (PDA), a mobile phone, a tablet, a pad, and a
computer.
28. The method of claim 1 wherein said multiple 2-D images are
captured in uncontrolled lighting conditions.
29. The method of claim 1 wherein the known background pattern is
printed on a substrate.
30. The method of claim 29 wherein the substrate is paper or
cardboard.
31. The method of claim 1 wherein the region having the known
background pattern is on a rotatable surface and wherein, during
said capturing in (A), said object is rotated with said rotatable
surface to obtain multiple views of said object on said region.
Description
RELATED APPLICATIONS
[0001] This application is a continuation of U.S. application Ser.
No. 15/097,950, filed Apr. 13, 2016, which application claims the
benefit of U.S. Provisional Application No. 62/275,270, filed Jan.
6, 2016, and which is a continuation-in-part of U.S. application
Ser. No. 13/201,512, filed Aug. 15, 2011, which is the U.S.
National Stage of International Application PCT/US2010/044343,
filed Aug. 4, 2010, which claims the benefit of U.S. Provisional
Application No. 61/231,216, filed Aug. 4, 2009, all of which are
fully incorporated herein by reference for all purposes.
FIELD OF THE INVENTION
[0002] This application relates to an interactive computer imaging
system and method, and in particular to a system and method for
identifying an object extracted from an image.
BACKGROUND
[0003] Construction sets may include standardized interlocking
pieces that allow for the construction of a variety of different
models or shapes. These pieces may not require special training or
design time to construct complex systems. Interlocking pieces may
be suitable for creating temporary structures for use as a toy for
children. One example of an interlocking construction set is
LEGO.RTM. (LEGO Juris A/S Corporation, Denmark), which can include
colorful interlocking plastic bricks and an accompanying array of
gears, minifigures and various other parts. These interlocking
bricks can be assembled and connected in many ways, to construct
such objects as vehicles, buildings, and even working robots.
Anything constructed by these interlocking bricks can be taken
apart, and the pieces may be used to make other objects.
SUMMARY OF THE INVENTION
[0004] In one aspect, the invention is a method of extracting
images of a real-world object and displaying a representation of a
three-dimensional model of the object in such images in an
electronic display of the object. In this aspect, the invention
comprises: capturing, using an image capture device and visible
light, a plurality of two dimensional images of said real world
object, said object at a fixed position relative to an orientation
indicator visible in at least some of said captured two-dimensional
images, a first image of said plurality of said captured two
dimensional images captured from a first orientation of said object
to the image capture device capturing said plurality of images, and
a second image of said plurality of images captured from a second
orientation of said object to said image capture device;
calculating said first orientation of said object from a position
of said orientation indicator in said first image, and calculating
said second orientation of said object from a position of said
orientation indicator in said second image; constructing a
three-dimensional image model of said object from said plurality of
captured images; deleting data of said orientation indicator prior
to said construction; and embedding a representation of said
three-dimensional image model into an electronic display of said
object.
[0005] In another aspect, the invention is a system for extracting
images of a real-world object and displaying a representation of a
three-dimensional model of said object in such images in an
electronic display of the object. In this aspect, the system
comprises: an image capture device, a memory; a processor; and a
mat having visible thereon an orientation indicator, and configured
to support said object; wherein said image capture device is
configured to capture a plurality of images of said object using
visible light, a first image of said plurality of images from a
first orientation relative to said object, and a second image of
said plurality of images from a second orientation relative to said
object; and said processor is configured to calculate said first
orientation of said object from a position of said orientation
indicator in said first image, and calculate said second
orientation of said object from a position said orientation
indicator in said second image; construct a three-dimensional image
model of said object from said plurality of captured images; delete
data of said orientation indicator prior to said construction; and
embed a representation of said three-dimensional image model into
an electronic display of said object.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIG. 1 depicts a schematic diagram of a system in accordance
with an embodiment of the invention;
[0007] FIG. 2 depicts a schematic diagram of components of the
system of FIG. 1;
[0008] FIG. 3 depicts an image captured by the system depicted in
FIG. 1;
[0009] FIG. 4 depicts a method in accordance with an embodiment of
the invention;
[0010] FIG. 5 depicts a representation of an image in accordance
with a step of the method depicted in FIG. 4;
[0011] FIG. 6 depicts a representation of another step in the
method depicted in FIG. 4;
[0012] FIG. 7 depicts a close up of a portion of the image of FIG.
5;
[0013] FIG. 8 depicts an alternate embodiment of an element of the
image of FIG. 3;
[0014] FIGS. 9A, 9B and 9C depict a representation of another step
in the method depicted in FIG. 4;
[0015] FIG. 10 depicts an extracted image from the image of FIG. 3
in accordance with the method depicted in FIG. 4;
[0016] FIG. 11 depicts a method in accordance with an embodiment of
the invention;
[0017] FIG. 12 depicts a method in accordance with an embodiment of
the invention;
[0018] FIG. 13A depicts an assortment of real world objects;
[0019] FIG. 13B depicts the assortment of objects of FIG. 11A
embedded in a video game;
[0020] FIG. 14 depicts a representation of a result in accordance
with the method of FIG. 2 presented on a handheld mobile
device;
[0021] FIG. 15 depicts, a schematic diagram of an object to be
imaged resting on a mat in accordance with an embodiment of the
invention; and
[0022] FIG. 16 is a flow diagram of a method of extracting images
of a real-world object and embedding representation of such images
in an electronic display of the object, in accordance with an
embodiment of the invention.
DETAILED DESCRIPTION
[0023] In the following description, various embodiments of the
invention will be described. For purposes of explanation, specific
examples are set forth in order to provide a thorough understanding
of at least one embodiment of the invention. However, it will also
be apparent to one skilled in the art that other embodiments of the
invention are not limited to the examples described herein.
Furthermore, well-known features may be omitted or simplified in
order not to obscure embodiments of the invention described
herein.
[0024] Unless specifically stated otherwise, as apparent from the
following discussions, it is appreciated that throughout the
specification, discussions utilizing terms such as "selecting,"
"evaluating," "processing," "computing," "calculating,"
"associating," "determining," "designating," "allocating" or the
like, refer to the actions and/or processes of a computer, computer
processor or computing system, or similar electronic computing
device, that manipulate and/or transform data represented as
physical, such as electronic, quantities within the computing
system's registers and/or memories into other data similarly
represented as physical quantities within the computing system's
memories, registers or other such information storage, transmission
or display devices.
[0025] The processes and functions presented herein are not
inherently related to any particular computer, network or other
apparatus. Embodiments of the invention described herein are not
described with reference to any particular programming language,
machine code, etc. It will be appreciated that a variety of
programming languages, network systems, protocols or hardware
configurations may be used to implement the teachings of the
embodiments of the invention as described herein. In some
embodiments, one or more methods of embodiments of the invention
may be stored as instructions or code in an article such as a
memory device, where such instructions upon execution by a
processor or computer result in the execution of a method of an
embodiment of the invention.
[0026] This application relates to interactive games and computer
vision imaging systems that may extract and identify an object
captured in an image. The object may be created by assembling
interlocking bricks (e.g., LEGO.RTM. bricks or building units) or
other pieces or building blocks. In one embodiment, the bricks or
construction objects used have or conform to a known set of
dimensions (e.g., a known and limited set of shapes and
dimensions). The image may be captured in uncontrolled lighting
conditions. The identification of the object may be based on shape
analysis and/or shape comparison. Identification can be done by
recognizing and classifying the object by comparison to a record in
a predefined database of objects. By embedding the extracted image
of the physical object within an interactive video game, a user may
experience a level of interaction with the extracted object.
[0027] FIG. 1 is a schematic diagram of system 100 in accordance
with an embodiment of the invention. System 100 may include
computer such as a workstation or personal computer 110 and digital
imaging device 190. Personal computer 110 may include a processor,
a display 292 (FIG. 2), a user interface(s) input/output device 296
(FIG. 2) (e.g., keyboard, arrow keys, and/or mouse), and memory 245
(FIG. 2). Digital imaging device 190 may be, by way of example, an
imager, a webcam or a digital camera. Connected to personal
computer 110 may be datastore 170 containing a database of images,
and other data (e.g., software or code). Datastore 170 may be
implemented by a memory or another storage device, such as a hard
disk drive.
[0028] System 100 may be a dedicated, stand-alone device having a
processor, a display, a user interface, memory, database, and a
digital imaging device. System 100 may be embodied in other
computing devices, such as notebook or netbook 120, personal
digital assistant (PDA) 130, mobile phone 140, or tablet (pad)
computer 150. System 100 may include an integral imaging device in,
for example, PDA 130 and mobile phone 140. A digital imaging device
190, 192, 194 may be connected, respectively, to personal computer
110, notebook or netbook 120, and tablet computer 150.
[0029] System 100 may include a computer program application stored
in non-volatile memory, or computer-readable medium (e.g., hard
drive, flash memory, CD ROM, magnetic media, etc.). The computer
program application may include code or executable instructions
that when executed may instruct or cause the controller or
processor of system 100 to perform methods discussed herein such as
a method of extracting, identifying, or embedding the object.
[0030] In another embodiment, computing device 110, 120, 130, 140,
150 of system 100 may each be configured as client devices
connected through electronic communication network 160 to a remote
server 180. Electronic communication network 160 may be the
Internet, a local area network, a wide area network, or other
suitable configurations of an electronic communication network.
Client device computing devices 110, 120, 130, 140, 150 may have a
local client application, and remote server 180 may include a
remote server application. In combination, the client application
and remote server application may provide the instructions for
system 100 to perform methods discussed herein such as a method of
extracting, identifying, and/or embedding the object. A datastore
172 containing a database of images may be connected to remote
server 180. In one embodiment, stand-alone datastore 174 containing
a database of images may be connected to electronic communication
network 160. Datastore 174 may be accessed through electronic
communication network 160 by computing devices 110, 120, 130, 140,
150 and/or remote server 180.
[0031] System 100 may provide an interactive system that can detect
and extract an image of a physical object from an image of a real
world scene captured by digital imaging device 190, 192, 194.
System 100 may model the extracted object on a computer display for
visualization and embedding in a computer video game. The object
may be constructed from interlocking bricks, or other items, or
other materials. In one embodiment, system 100 may provide
instructions to a user on a suggested shape or configuration for
the object to be constructed--for example, a plane, an automobile,
a house, a character, etc. For example, a processor or controller
may select a shape and present the shape on a display possibly
along with an instruction to a user to construct the displayed
shape from the bricks, so as to create the physical object. The
suggested configurations can be stored in, for example, datastores
170, 172, 174. After a suggested object is extracted, system 100
may compare the extracted image to the suggested configuration and
may compute a rating or metric that is representative of the
comparison results, or of the correspondence between the image or
the physical object to the suggested configuration or shape.
[0032] FIG. 2 depicts a schematic of components of system 100.
System 100 may include controller or central processor unit 270
that may be connected to an internal bus 230. Digital imaging
device 190 may be connected to CPU 270 via an input/output port
(not shown). Also connected to CPU 270 via an input/output port may
be memory or datastore 170. In an alternate embodiment, the shape
configurations stored within the datastore may be stored in memory
245 coupled to internal bus 230, thus, reducing the need for
datastore 170.
[0033] CPU 270 may provide background detector 240 with a digital
image provided by the digital imaging device. Background detector
240 may be implemented by dedicated hardware, software modules,
and/or firmware, where CPU 270 executes the instructions. Other
units discussed herein may also be implemented by dedicated
hardware units, software modules, and/or firmware, where CPU 270
executes the instructions. The boundaries and edges of background
220 may be calculated and extracted by background detector 240
using an edge detector process or algorithm. To discriminate
background 220 (FIG. 3) from image 205 (FIG. 3) (e.g., a digital
image) of a physical object, e.g., made of bricks or other building
units, the curvature of the edges may be calculated and locations
where the curvature is about 90 degrees may be marked as corners.
Each detected corner may be associated with the curvature of edges
that are connected to it. The background detector may provide its
result to CPU 270 for storage in internal memory 245, or an
external memory unit, for example as a record in datastore 170.
Background 220 may include a grid, pattern, or visual cues to
enable extraction of the image of the physical object. A background
need not be used.
[0034] Perspective transformation correction (PTC) unit 250 may
compute a corrected virtual grid substantially aligned to the image
of the object 210. Color calibration extractor unit 260 may use the
perspective corrected grid to locate color calibration palette(s)
formed on background 220. The color value of the associated pixel
from image 205 corresponding to the bricks of the palette may be
extracted and converted to another color space representation by
the color calibration extractor unit. Also, a few calibration
points from the background field of background 220, chosen to
represent the field color, may also be extracted and converted to
the same HSV (hue, saturation, and value) color space by color
calibration extractor unit 260. Other color spaces may be used, and
color need not be used.
[0035] Brick identifier unit 280 may sample and extract the value
of associated pixels of image 205 in a few different places. These
values may be converted to for example the HSV color space. The hue
value of the converted data may be compared with the hue values of
the color calibration palette, and optionally the field of
background 220. The color with the smallest difference is chosen by
brick identifier unit 280 to represent the color of this grid
location. Brick identifier unit 280 compares the intensity and
saturation levels of the pixels to determine if the brick is
colored, black or white. If the determined color of the brick is
not the color of the background, then brick identifier unit 280
discerns that this brick is part of the constructed object.
[0036] Comparison unit 290 may compare the shape of the detected
image with a record stored in a database. The comparison is done by
comparison unit using for example a correlation function.
[0037] FIG. 3 depicts image 205 of object 210 placed on background
220. Object 210 may be a real world, physical object made of
building bricks or other units. Similarly, background 220 may be a
real-world, physical object made of building bricks or other units,
or printed on paper or cardboard. A background need not be
used.
[0038] The object to be extracted can be captured on a background
having a predefined pattern. This predefined pattern may have
predetermined known spatial features that can discriminate the
pixels belonging to object 210 from the pixels belonging to
background 220. For example, FIG. 3 depicts background 220 having a
square grid. The spatial pattern of background 220 may be
discriminated from pixels of image 205 containing object 210 made
from interlocking building bricks, which are typically solid and
rigid. The predefined grid need not be a square grid, and other
implementations of a predefined background pattern may be used.
[0039] With a result similar to the chroma key technique, where
foreground objects are separated from a background using a
bluescreen or greenscreen, system 100 may extract object 210 from
image 205 by using the extracted object's spatial and morphological
features, and may be done independent of color recognition.
[0040] The captured image containing object 210 may be analyzed to
extract the object from background 220. Once extracted, object 210
may be analyzed for shape identification by comparison to a
predefined database of objects. Once the shape of object 210 is
identified, the digital extracted object can be used in a variety
of dynamic interactions with a player.
[0041] FIG. 4 depicts process 400 in accordance with an embodiment
of the invention. A mask of object 210 may be extracted using a
morphological operator such as a 2D bottom hat operator, which may
give an image that can then be filtered using a combination of a
threshold techniques and other morphological operators (e.g.,
closing and opening) to remove spurious artifacts. The resultant
mask created by applying these morphological operators may
represent object 210 detected and isolated from background 220.
[0042] In one embodiment, process 400 for extracting and
identifying object 210 may be performed as follows:
[0043] Image 205 may be obtained, step 410, by arranging object 210
on background 220, and taking a digital image of the arrangement.
The image may be taken using digital imaging device 190 connected
to computing device 110, a personal computer. Alternatively, the
image may be taken using other computing devices, described above,
and either external or internal digital imaging devices associated
with these other computing devices.
[0044] Background Object Detection:
[0045] The background object on which the bricks may be placed can
be any surface with known features such as a specific color, a
specific spatial pattern, or other spectral/spatial feature(s) that
may aid in the detection and extraction of the object. Background
object 220 may be, for example, a printed paper or cardboard, or a
surface formed from interlocking bricks. If for example, an
interlocking brick background is used as a background, process 400
may include detecting four high contrast corners, which can be
created by using, for example, white perimeter bricks encompassing
a dark color field. The corners can be used to find the background
field's boundaries in the following way:
[0046] The boundaries of the image may be calculated and extracted
using any edge detector which can detect edges in images, for
example, the Canny algorithm is one such multi-stage algorithm. The
Canny algorithm may use an edge detection operator that may include
four filters to detect horizontal, vertical and diagonal edges in
an image. The edge detection operator may return a value for the
first derivative in the horizontal direction and the vertical
direction. From these derivatives an edge gradient and direction
can be determined. Other non-maximal suppression edge detector
techniques may also be used by process 400.
[0047] To discriminate background 220 from image 205, the curvature
of the edges may be calculated and locations where the curvature is
about 90 degrees may be marked as corners, step 420. FIG. 5 depicts
an interlocking brick background and object with corners identified
by process 400 and indicated by a "+" mark in the figure. Each of
the marked corners may be considered a candidate corner of the
background field.
[0048] With reference to FIG. 5, each detected candidate corner may
be associated with the curvature of edges that are connected to it.
These edge curvatures may be on a straight line if they are found,
within a predetermined threshold, to be close to zero. In such a
case, the corners may be marked as candidates for being the
background field's corners. The slope of each associated edge of
the candidate corners is calculated and its line equation is
extracted. If the distance between a first candidate corner and one
of the line equations associated with a second candidate corner is
smaller than some predetermined threshold, then those two corners
are accounted as being on the same line. If four such corners are
found where the lines form a quadrangle the background field's
corners are assumed to be found. A candidate corner that is not on
the same line with other corners may be removed from consideration
as a background corner.
[0049] Perspective Transformation Correction:
[0050] Background 220 may be rectangular or square, or other
shapes, but due to the fact that the background may not necessarily
be orientated perpendicular to the camera lens' axis, the acquired
image may not be square and true. The position of the detected
corners of background 220 may be used to perform a perspective
transformation calculation, step 430. This perspective
transformation may be used to compute a corrected virtual grid that
is substantially aligned with the image of the interlocking bricks
forming object 210 as depicted in image 205. FIG. 6 depicts a
representation of image 205 with a superimposed true grid after the
perspective transformation calculation.
[0051] Calibration Color Extraction:
[0052] FIG. 7 depicts a close up of one corner of background 220
depicting a color calibration palette 610. Background 220 may have
color calibration palette 610 located in one or more of its four
corners. Each color calibration palette may include, for example,
four different colored blocks--e.g., red, yellow, green, and blue.
Other numbers of colors and other colors may be used. The colored
blocks may be formed, for example, using interlocking bricks when
background 220 is assembled.
[0053] Using the perspective corrected grid obtained in step 430,
the location of each of these colored bricks forming color
calibration palette 610 may be found. The color value of the
associated pixel from image 205 corresponding to the bricks of the
palette may be extracted, step 440, and converted to another color
space representation, such as HSV color space. Also, a few
calibration points from the background field of background 220,
chosen to represent the field color, may also be extracted and
converted to the same HSV color space.
[0054] Colored building blocks can undergo additional analysis, and
classification of their colors to obtain a color model of the
object can be made.
[0055] FIG. 8 depicts an alternate embodiment of background 220. In
this embodiment, background 220 may be printed on a suitable flat
surface and material, such as paper or cardboard. Here, background
220 may include color calibration palette 620 which may include a
combination of colors arranged in a specific spatial arrangement.
Color calibration palette 620 may allow for the calibration of
colors to compensate for white balance, dynamic range and color
consistency.
[0056] Brick Identification:
[0057] For each grid location, the value of the associated pixels
of image 205 may be sampled in a few different places and
extracted, step 450. These values may be converted to HSV color
space. The hue value of the converted data may be compared with the
hue values of the color calibration palette, and optionally the
background field, obtained during step 440. The color with the
smallest difference may be chosen to represent the color of this
grid location. The intensity level of the associated pixels may be
compared with the intensity level and the saturation level of the
black and white pixels associated with the bricks used in the four
corners of background 220. If the saturation level is closer to the
saturation level of the white color than to the saturation level of
the color palette and the background; and if its intensity level is
also high, then the color of the grid location may be marked as
white. If the intensity level is very low and close to the
intensity level of the black color, then the grid location may be
marked as black. Otherwise, the color of the grid location is
assigned the corresponding color from the color calibration
palette.
[0058] FIGS. 9A-C depicts interim results of step 440. A
morphological operator such as a 2D bottom hat operator will give
an image (see FIG. 9A) which can then be filtered using one, or
more, threshold techniques (see FIG. 9B) and other morphological
operators--e.g., closing and opening, to remove spurious artifacts.
The resultant mask created by applying these morphological
operators may represent object 210 isolated from background 220
(see FIG. 9C). FIG. 10 depicts an example of an image of object 210
after extraction from image 205 by process 400.
[0059] After the image of object 210 is separated from background
220 and extracted from image 205, process 400 may perform further
analysis of the orientation of the extracted image by applying
binary large object (BLOB) analysis, labeling, and moments
calculations to obtain a calculated angle that can be used to
rotate the object to a more desired angle.
[0060] In one embodiment, further analysis of the spatial and
spectral features associated with the detected object can be used
to train system 100 to recognize the object in the future by, for
example, examining the color histogram of the detected object and
relations of colored segments within the detected object. This
information may be stored in datastore 170, 172, and 174.
[0061] Shape Comparison and Scoring:
[0062] The shape of the extracted object from image 205 may be
compared, steps 460, 470 with a record of a database stored within
datastore 170, 172 or 174. The extracted object shape can be
compared to the input shape from the database by using, for
example, a normalized correlation function, cor:
f(cor)=(stored configuration, extracted shape)
f(cor)=1, if stored configuration=extracted shape
where: the stored configuration may be within a datastore, and the
extracted shape may be provided, for example, by process 400.
[0063] Other operations can be used. In one embodiment, a game can
be created based on the extracted image and its classification. The
computer may engage the user in an interactive manner to create
real world objects which are than provided to the computer in an
image for extraction. A computer may present a record from a
database on a computer display and ask a player to build the
displayed object. The displayed object may be, for example, a car.
After the object is assembled by the player, an image of the object
placed on a suitable background (as discussed above) can be
obtained. The computer may than extract an image of the object, and
compare it to the database record. FIG. 13A depicts a variety of
objects that can be built. By way of example, FIG. 13A depicts real
world objects in the form of character 1010, car 1020, tree 1030,
airplane 1040, face 1050, and house 1060. The following text is an
example of an interactive script which may be outputted to a user
and the corresponding actions in accordance with this embodiment:
[0064] Computer: Welcome, players, let's go on a fun storytelling
adventure! [0065] Computer: First, we need a character . . . . Can
you build a boy character? [0066] [Player builds and shows a body
of a character on the background] [0067] Computer [after extracting
and comparing the object]: Cool, now let's give him a face; can you
build a face for him? [0068] [Player builds and shows a face on the
background] [0069] Computer: Fantastic . . . mmm . . . . Let's
build him a house . . . can you help do that? [0070] [Player builds
and shows a house on the background] [0071] Computer: That's a
really nice house. [0072] Computer: Hey, what's that sound? Is that
an airplane? Can you build an airplane? [0073] [Player builds and
shows an airplane on the background] [0074] Computer: And what is
this sound? Is that a car honking? Let's build one . . . . [0075]
[Player builds and shows a car on the background] [0076] If showing
the wrong thing than computer responds with: [0077] Computer:
That's a cool [face/house/airplane/car/tree] but let's try
again.
[0078] By obtaining an image of the constructed object, the
computing device may detect the presence of a shape and extract the
shape showing a digital representation of the extracted shape on a
display screen. The user can create a virtual world by manipulating
the location or other aspects of the digital representation of the
shape on the screen.
[0079] The extraction can also involve automatic recognition of the
shape by using image processing techniques. Using automatic
recognition, the computer can ask the user to build a specific
shape and may give some feedback based on a comparison with a
predefined image in the database.
[0080] FIG. 11 depicts process 700 which may implement a game, for
example, including interactive script presented above. A computer
instructs (e.g., via a display or monitor) a player, step 710, to
construct an object. The instruction can be provided, for example,
by audio (verbal or sound) through a sound card, or visually by
presenting written words on a computer display. An image of the
object may or may not be displayed. By not displaying an image of
the instructed object, the player's knowledge, word skill, and
imagination may be exercised. The object may be chosen at random
from a database, or may be chosen in a predetermined sequence. The
predetermined sequence may be selected so that the player's
construction skills are developed by the sequence of object
selection.
[0081] After the player constructs the object, the player may place
the object on a suitable background, as describe above, and take a
digital image of the object and background using a digital imaging
device. A background need not be used. The computer obtains, step
720, the digital image of the physical object as an image file from
the digital imaging device.
[0082] The image of the constructed physical object may be
extracted, step 730, from the digital image. The extraction may be
performed using, for example, the method of process 400 described
above with reference to FIG. 4.
[0083] The object's extracted image may be compared to a database
record that corresponds to the instructed object, step 740. In one
embodiment of the invention, the comparison may provide a rating or
metric that may be indicative of the constructed object's
conformance with the instructed object.
[0084] The extracted image may be embedded, step 750, in a video
game and utilized within the game parameters by the player.
[0085] Other steps or series of steps may be used.
[0086] In another embodiment, the computer may randomly choose a
shape (e.g., a predetermined shape taken from a memory or a
database) and show or display the shape on the display device. A
player may try and construct that shape using bricks or other
construction objects, for example, under timed conditions, as part
of a game. Once the player finishes constructing the shape, he uses
the digital imager to upload an image of his creation to the
computer. After the computer extracts and identifies the object, it
can be compared to the original shape shown on the device. The
result of comparison can be shown to indicate a success or failure
in the construction task as part of an interactive competition.
[0087] A user may be presented with a selected shape composed from
bricks or other building units appearing on the screen of a
computing device. This selected shape may also be accompanied by a
timer that starts when the user first sees the shape. The timer may
be used to measure how fast the user can successfully build the
shape using the interlocking bricks.
[0088] The user tries to build the shape using regular bricks and
may place the object he built on the area designated as the
background area, if a background is used. The computing device may
constantly analyze images acquired from a digital camera and may
detect the presence of an object on the background area.
[0089] If an object is detected, an embodiment of the method may
extract the shape of the object the user created by using processes
in embodiments of the present invention, for example, process 400
described above.
[0090] The process may compare the extracted image of the object
built by the user to the selected shape that was presented as the
target shape to determine how successful the user was in building
the shape. The comparison may provide a rating or metric based on
how accurate the user's object corresponds to the selected shape.
This rating or metric may also include a component indicative of
the time spent to construct the object, where the component varies
based on the complexity of the object.
[0091] FIG. 12 depicts process 800 which may implement the timed
construction described above. A computer may retrieve a record,
step 810, from a database containing records representing
predetermined shapes. A visual image of the retrieved shape may be
displayed, step 820, on a computer display.
[0092] The computer may provide a message requesting, step 830, the
player to construct the object seen on the display. Optionally, the
computer may also present a clock timer on the screen. The timer
(whether displayed or not) may be a count up, or a count down timer
so as to track the time elapsed for the user to construct the
physical object.
[0093] After receiving an indication (e.g., via a user input
device) that the user has completed construction, or waiting for a
predetermined time period to elapse, step 840, the computer
instructs the player to place the object on a suitable background,
as describe above, and take a digital image of the object and
background using a digital imaging device. A background need not be
used. The computer may obtain, step 850, the digital image as an
image file from the digital imaging device.
[0094] The image of the constructed object may be extracted, step
860, from the digital image. The extraction may be performed using,
for example, the method of process 400 described above with
reference to FIG. 4.
[0095] The object's extracted image may be compared to the
retrieved database record, step 870. The comparison may provide a
rating or metric that may be indicative of the constructed object's
conformance with the retrieved record.
[0096] The extracted image may be embedded, step 880, in a video
game and utilized within the game parameters by the player.
[0097] Other steps or series of steps may be used.
[0098] The extracted image (or multiple extracted images of various
physical objects) can be digitally represented on the display
device as part of a virtual world, or video game, where the objects
inhibiting the virtual world and/or video game, were designed and
built from the construction set in the real-world. FIG. 13B depicts
the extracted images of the multiple real world objects of FIG. 13A
embedded in a computer graphic.
[0099] FIG. 14 depicts extracted image 110, after being processed
from an image of real world combination 1020 containing an
interlocking brick background and object. FIG. 14 depicts extracted
image on mobile device 130, 140 equipped with an internal
camera.
[0100] Reference is made to FIG. 15, a schematic diagram of an
object to be imaged resting on and supported by a mat in accordance
with an embodiment of the invention. In some embodiments, an object
1500 such as a toy or model may be placed upon a mat 1502 or other
surface. Mat 1502 may include a visible pattern 1504 that may
appear in an image captured of the mat 1502 and the object 1500 and
that may indicate in such image an orientation of the mat 1502 and
the object 1500 resting on the mat 1502. In some embodiments, mat
1502 may be placed upon a wheel or other rotatable surface so that
mat 1502 and object 1500 may be rotated in view of a camera of a
portable device while images of the mat 1502 and object 1500 are
captured from a reasonably steady platform and uniform height and
distance.
[0101] In some embodiments, object 1500 may be or include a
construction block but may be other than a construction block. For
example, object 1500 may be or include a figure, action figure,
toy, doll, door knob or any object made from any material such as
plastic, paper, clay or metal. In some embodiments, object 1500 to
be imaged and extracted may be placed on or against mat 1500 that
may have pattern 1504 printed thereon so that one or more images of
the object 1500 against the mat 1502 may indicate an orientation
from which the image was captured. Images of the object 1500 placed
or resting on the background mat 1502 may be captured from various
angles and processed, by known methods such as for example:
[0102] Control points recognition--Using an algorithm for detecting
corners or other control points such as Harris corner detector or
keypoints recognition to find corners or control points in the
images of the object 1500.
[0103] Recognition of significant control points such as main
corners by recognizing two, three or more control points of a high
contrast frame in the printed pattern.
[0104] Determining a perspective transformation and rectifying the
image by using a 2D to 2D transformation, also known as homography
that maps the captured pattern to rectify coordinates.
[0105] Verification of the pattern by checking for example colored
areas or other a priori known patterns, and/or by checking specific
points located relative to a high contrast frame.
[0106] Finding an orientation and position of the pattern on the
mat or background based on specific known features located in one
or more areas of the pattern: Once an orientation is derived and a
position of a first corner is understood, there may be defined for
example a closest corner to the specific known feature of the
object 1500. Other features may then be plotted in coordinates of
for example x, y, z=0,0,0 in a world coordinate system. Coordinates
of one or more control points may be determined, and imposed on
coordinates of the captured images (screen coordinates).
[0107] Using the relations between corner positions in the world
coordinate system and screen coordinates, a camera matrix (one or
more of intrinsic and extrinsic) may be extracted. This may be
performed for example using SVD (for first guess) and square error
minimization algorithms. A camera matrix may provide a position and
rotation of the camera relative to the world coordinates.
[0108] To separate the scanned object from the printed background,
a background subtraction process may be used based on color
features (like a green screen effect) or spatial features or both
may be used. If the pattern itself is extracted with the silhouette
it may be removed in the voxel carving sequence.
[0109] Voxel carving may be performed on for example some or all of
the voxels in the region of interest (which may be any cubic
D.times.D.times.D millimeters). For each voxel subject to the
carving, determining its 8 vertices and back projecting their 3D
coordinates into the screen coordinates is performed. A
determination may be made as to whether or not those screen
coordinates are inside the silhouette. If all 8 vertices of a voxel
are outside of the silhouette, then this voxel may be "carved" or
"turned off".
[0110] The image capture device may be rotated around the object
1500 or the mat 1502 may be rotated with the object resting thereon
and still or video images may be captured. A full digital
representation of the object may thereby be created.
[0111] In some embodiments one or more of the following processes
may be performed:
[0112] Color information may be gathered from the image about
vertices of the contour voxels in one or more of the images in
which such voxels are visible. A color of the vertices may be
determined from a most saturated pixel, or median color or average
color or any other relevant methods.
[0113] In some embodiments a smoothing of the mesh may be
implemented using a smoothing algorithm such as a marching cubes
algorithm.
[0114] Texturing of triangles may be achieved by gathering color
data of a back projection of one or triangles that face the camera
to create a texture to the mesh.
[0115] Stereoscopic 3D reconstruction may be performed by creating
depth maps from couples of captured images to further carve the
voxels and have models with cavities being determined.
[0116] Objects may be classified as being for example a model type,
such as humanoid, four-legged animal, car, etc. A smart rig or
skeleton for the model type may be created or designated and a
determination may be made as to which vertices of the imaged models
are to be attributed to one or more skeleton part. Digitally
animating the model may be performed with predefined animations
such as walking, jumping, flying etc.
[0117] Image models may be presented in for example a CAD
environment that allows a user to show, rotate, move and zoom
in/out of the extracted image of the object. Image models may be
presented in an augmented reality format (with or without
animations) such as over the mat, or over other models/patterns.
Models may be presented in a virtual reality environment: showing
the model in 3D by rendering the scene from at least two angles and
showing the images side by side on a same screen so it will be
compatible with for example Google cardboard or Samsung gear
VR.
[0118] In some embodiments an extracted and digital representation
of a scanned object whether smoothed or not, may be transmitted to
for example a 3D printer to create a physical copy of the object
1500.
[0119] Some or all of the images of the object may be captured
using a 2D camera as may be included in a mobile device 1520 such
as a smart telephone, laptop or table computer that may include a
processor 1522 and a memory 1524. The mobile device 1520 may for
example be moved around the object 1500 to be imaged, or the mobile
device 1520 may be held or placed in a stand and the object 1500 on
the mat 1502 may be turned as the images are captured. Processing
of the images into the modeled object may also be performed in real
time using processor 1522 or some other processor. In some
embodiments, camera calibration may be performed during the image
capture process. The pattern 1504 on the mat 1502 or background may
be removed simultaneously or in real time with the capturing of the
images or at any time prior to the construction of the 3D
representation of the captured images. Image capture and scanning
of the object 1500 may be performed in free hand scanning with
freedom of the angles from which the images are captured in all
axis. Such scanning and image capture may allow images to be
captured by for example a hand-held device rather than requiring
that the imager be mounted in a fixed height and orientation. In
some embodiments carving of the voxels may be performed in real
time while the images of the object are being captured by the
imager associated with the mobile device. In some embodiments mat
1502 may be constructed from a highly retroreflective material and
the imager used to capture images of the object 1500 will use a
lighting source such as an LED to create an image where the object
1500 may be easily extracted or differentiated from mat 1502.
[0120] Reference is made to FIG. 16, a flow diagram of a method of
extracting images of a real-world object and embedding for example
3D or other representation of such images in an electronic display
of the object, in accordance with an embodiment of the
invention.
[0121] In block 1600, a mobile imager in for example a housing of a
mobile device may capture images of a real world object using
visible light. The object may be resting on or in front of and at a
fixed position relative to an orientation indicator that appears in
at least some of the captured images. A first of the images may be
captured from a first orientation or angle of the object to camera,
and a second of the images may be captured from a second
orientation or angle of the object to the camera. Other numbers of
images may captured, such that for example images of the object
from many sides or orientations may be collected and subject to the
processes below.
[0122] In block 1602 an orientation or position of the camera
during the capture of the first image relative to one or more of
the object and the orientation indicator may be calculated. The
orientation of the camera during the capture of the second image
relative to one or more of the object and the orientation indicator
may be also be calculated.
[0123] In block 1604, a processor may develop or construct a
three-dimensional image model of the object from the captured
images.
[0124] In block 1606, data about the orientation indicator as may
have appeared in the captured images may be deleted from the
process of constructing the three-dimensional image model.
[0125] In block 1608, data from the three-dimensional image model
may be embedded into a representation the three-dimensional image
model and such representation may be displayed on an electronic
display.
[0126] In some embodiments a real world copy of the object may be
constructed by a three-dimensional printer, using data from the
three-dimensional image model.
[0127] In some embodiments, the camera and its housing may be
rotated around the real world object during the capturing of the
images.
[0128] In some embodiments, the real world object may be rotated
along with the mat while the camera is held still during the image
capturing.
[0129] In some embodiments, the constructing of the
three-dimensional image model is performed by the processor in the
housing of the device that includes the imager.
[0130] In some embodiments deleting image data of the orientation
indicator as was captured in the first image is performed prior to
capturing of the second image.
[0131] Some embodiments may include detecting one or more of the
same control points of the real world object that appear in a first
image and in the second image, and determining a perspective
transformation of the control points between the first image and
the second image.
[0132] Some embodiments may include manipulating the representation
of the three-dimensional image model on the electronic display. In
some embodiments the three-dimensional image model may be embedded
in view of a video game that displayed on the electronic
display.
[0133] While there have been shown and described fundamental novel
features of the invention as applied to several embodiments, it
will be understood that various omissions, substitutions, and
changes in the form, detail, and operation of the illustrated
embodiments may be made by those skilled in the art without
departing from the spirit and scope of the invention. Substitutions
of elements from one embodiment to another are also fully intended
and contemplated. The invention is defined solely with regard to
the claims appended hereto, and equivalents of the recitations
therein.
* * * * *