U.S. patent application number 09/877198 was filed with the patent office on 2001-11-15 for object extracting method using motion picture.
This patent application is currently assigned to LG Electronics Inc.. Invention is credited to Kim, Hyeon-June, Lee, Jin-Soo.
Application Number | 20010040982 09/877198 |
Document ID | / |
Family ID | 19523669 |
Filed Date | 2001-11-15 |
United States Patent
Application |
20010040982 |
Kind Code |
A1 |
Kim, Hyeon-June ; et
al. |
November 15, 2001 |
Object extracting method using motion picture
Abstract
An object extracting method using a motion picture more
accurately and rapidly extracts a specific object by using a
difference image frame of a pair of still image frames obtained
from a motion picture and color information which defines a color
of the object. The object extracting method using a motion picture
includes the steps of: obtaining a difference image frame by
getting a pair of still image frames having a predetermined time
interval from a motion picture and obtaining a color image frame
which satisfies color information defining a color of a particular
object from one of the still image frames; performing a grid
processing of a logic image frame which is obtained from the
difference image frame and the color image frame at a predetermined
size and obtaining connected components using direction connection
information and defining minimum areas each includes the connected
components; comparing each of the minimum areas with predetermined
conditions, thereby selecting the minimum areas which satisfy the
conditions as object area candidates; and selecting and optimizing
a largest object area candidate among the object area
candidates.
Inventors: |
Kim, Hyeon-June; (Sungnam,
KR) ; Lee, Jin-Soo; (Seoul, KR) |
Correspondence
Address: |
FLESHNER & KIM, LLP
P.O. Box 221200
Chantilly
VA
20153-1200
US
|
Assignee: |
LG Electronics Inc.
|
Family ID: |
19523669 |
Appl. No.: |
09/877198 |
Filed: |
June 11, 2001 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
09877198 |
Jun 11, 2001 |
|
|
|
09157948 |
Sep 22, 1998 |
|
|
|
Current U.S.
Class: |
382/103 |
Current CPC
Class: |
G06V 40/161
20220101 |
Class at
Publication: |
382/103 |
International
Class: |
G06K 009/00 |
Foreign Application Data
Date |
Code |
Application Number |
Oct 29, 1997 |
KR |
55956/1997 |
Claims
What is claimed is:
1. An object extracting method using a motion picture, said method
comprising the steps of: obtaining a difference image frame by
getting a pair of still image frames having a predetermined time
interval from a motion picture and obtaining a color image frame
which satisfies color information defining a color of a particular
object from one of the still image frames; performing a grid
processing of a logic image frame which is obtained from the
difference image frame and the color image frame at a predetermined
size and obtaining connected components using direction connection
information and defining minimum areas each includes the connected
components; comparing each of the minimum areas with predetermined
conditions, thereby selecting the minimum areas which satisfy the
conditions as object area candidates; and extracting an object by
selecting and optimizing an object area candidate which has a
largest size among the object area candidates.
2. The method of claim 1, wherein the predetermined time difference
is a time that motion change of the particular object can be
detected in the motion picture.
3. The method of claim 1, wherein the difference image frame only
represents an area where there is the motion change of the
particular object between the two still image frames.
4. The method of claim 1, wherein the particular object is a human
face.
5. The method of claim 1, wherein the color information is a color
which a user arbitrarily defines by adjusting color condition of
the object in specific color space.
6. The method of claim 1, wherein the color information represents
a skin color.
7. The method of claim 1, wherein the color image frame is defined
by RGB.
8. The method of claim 1, wherein the color image frame is obtained
from one of the still image frames.
9. The method of claim 1, wherein the logic image frame is obtained
by ANDing the difference image frame and the color image frame.
10. The method of claim 1, wherein the direction connection
information indicates a connection state of a grid to other grids
with respect to eight directions.
11. The method of claim 1, wherein each of the connected components
is composed of grids which are gathered according to the direction
connection information.
12. The method of claim 1, wherein the minimum area is a rectangle
which includes the connected components.
13. The method of claim 1, wherein the predetermined conditions
are: (1) A size of a variable RECT[i]>a threshold value of a
size of the minimum area; (2) A minimum value <the row/column
ratio of the variable RECT[i]<a maximum value; (3) A density of
the variable RECT[i]<a threshold value of a density, wherein the
threshold value of the size of the minimum area, and wherein the
minimum value and the maximum value of the row/column ratio of the
variable RECT[i] are previously defined values in accordance with
the object, and the density of the variable RECT[i] is a value of
which the density of the connected components is divided by an area
size of the variable RECT[i].
14. The method of claim 1, wherein the largest object candidate
area is located nearest to a camera.
15. The method of claim 1, wherein in the step of optimizing the
selected object area candidate, each row or each column of the
object area candidate obtains its density and each density is
compared with the defined threshold value of the density and the
row or column of which density is smaller than the threshold value
is deleted, thus being optimized to the human face.
16. The method of claim 15, wherein the density is in the minimum
area variable RECT[i] which includes the connected components the
density is a value of which the number of grids in a row or a
column of one of the connected components is divided by a length of
the row or the column thereof.
17. The method of claim 1, wherein the step of extracting the
object comprises optimizing each of the object area candidates and
selecting the largest area candidate thereamong.
Description
BACKGROUND OF THE INVENTION
[0001] 1. Field of the Invention
[0002] The present invention relates to a technique of extracting a
specific object in a motion picture, and more particularly to an
object extracting method using motion pictures which more
accurately and rapidly extracts an object using information of a
moving object in a motion picture and color information which
defines a color of the object.
[0003] 2. Description of the Conventional Art
[0004] A motion picture is composed of image frames, each frame
carries information with respect to an object. Recently, a
conventional technique which extracts a specific object using the
motion picture can be divided into two ways; one is to extract the
object using only color information which defines a color thereof
and the other is to extract the object using information of motion
changes of the object.
[0005] First, the conventional object extracting method using the
color information will be explained.
[0006] After obtaining a still image frame from a motion picture, a
preprocessing step is performed, which generates a color histogram
by converting a negative image of the still image frame to a
positive image.
[0007] Next, red, green and blue (RGB) color domains included in
the histogram are converted to hue saturation values (HSV), and
color information of the object such as skin-color pixels are
obtained. To the color pixels, segmentation occurs through edge
detection, hole filtering and gap filtering. Finally, the segments
and a predetermined object domain are compared, thus the specific
object is extracted.
[0008] However, because the conventional object extracting method
using only the color information obtains one still image frame
using the color information which defines the color of the object
and thus extracts the object therefrom, the object may not be
accurately obtained. Accordingly, the above object extracting
method using the color information requires a considerable
operation time to extract the object because RGB color space is
converted to HSV color space with respect to the inaccurate object
domain.
[0009] A face-it method which is the other object extracting method
extracts a specific object only using motion information of the
object on a motion picture, without using color information
thereof. Particularly, the face-it method which extracts a human
face designates an area in which there is a movement of a specific
object as a face domain candidate and carries out a grey image
process for the domain candidate, thereby obtaining information
with respect to the corresponding human face. Accordingly, the
face-it which extracts the human face with insufficient information
has difficulty of accurately extracting a human face.
SUMMARY OF THE INVENTION
[0010] Accordingly, an object of the present invention is to
provide an object extracting method using a motion picture which
accurately and rapidly extracts an object using information of a
moving object in a motion picture and color information which
define a color of the object.
[0011] To achieve these and other advantages and in accordance with
the purpose of the present invention, as embodied and broadly
described, an object extracting method includes the steps of:
obtaining a difference image frame by getting a pair of still image
frames having a predetermined time difference from a motion picture
and obtaining a color image frame which satisfies color information
defining a color of a particular object from one of the still image
frames; performing a grid processing of a logic image frame which
is obtained from the difference image frame and the color image
frame at a predetermined size and obtaining connected components
using direction connection information and defining minimum areas
each includes the connected components; comparing each of the
minimum areas with predetermined conditions, thereby selecting the
minimum areas which satisfy the conditions as object area
candidates; and selecting and optimizing a largest object area
candidate among the object area candidates.
[0012] It is to be understood that both the foregoing general
description and the following detailed description are exemplary
and explanatory and are intended to provide and further explanation
of the invention as claimed.
DETAILED DESCRIPTION OF THE INVENTION
[0013] The accompanying drawings, which are included to provide a
further understanding of the invention and are incorporated in and
constitute a part of this specification, illustrate embodiments of
the invention and together with the description serve to explain
the principles of the invention.
[0014] In the drawings:
[0015] FIG. 1 is a flowchart which illustrates an object extracting
method using a motion picture according to the present
invention;
[0016] FIG. 2 is a diagram illustrating an example of a connected
component which is generated in accordance with 8-direction
connection information;
[0017] FIG. 3A is a picture which illustrates a first still image
frame obtained from a motion picture;
[0018] FIG. 3B is a picture which illustrates a second still image
frame obtained from a motion picture after a predetermined time
elapses from the time at which the first still image frame is
obtained;
[0019] FIG. 4 illustrates a difference image frame using the first
still image frame and the second still image frame in FIGS. 3A and
3B;
[0020] FIG. 5 illustrates a skin color image frame obtained from
the second still image frame of FIG. 3B;
[0021] FIG. 6 illustrates a logic image frame which is obtained by
ANDing the difference image frame of FIG. 4 and the skin color
image frame of FIG. 5;
[0022] FIG. 7 illustrates a grid image frame obtained by which a
grid process is applied to the logic image frame of FIG. 6;
[0023] FIG. 8 illustrates minimum rectangles each includes a
connected component; and
[0024] FIG. 9 is a picture which illustrates a human face extracted
from a specific object in a motion picture according to the present
invention.
DETAILED DESCRIPTION OF THE INVENTION
[0025] Now, the object extracting method using a motion picture
according to the present invention will be described in detail with
reference to the accompanying drawings.
[0026] FIG. 1 is a flowchart which illustrates the object
extracting method using a motion picture according to the present
invention. In a first step (S1), a first still image frame IMAGE
(t) is obtained at a time (t) from a motion picture. After a
predetermined time (.DELTA.t) elapses from the time (t), a second
still image frame IMAGE(t+.DELTA.t) is obtained in a second step
(S2). In a third step (S3), a difference image frame which has
information with respect to motion changes of an object is obtained
from the first still image frame IMAGE(t) and the second still
image frame IMAGE(t+.DELTA.t).
[0027] While, in a fourth step (S4) a color image frame which
satisfies color information with respect to the object is obtained
from the second still image frame IMAGE(t+.DELTA.t), Here, the
color image frame can be obtained from the first still image frame
IMAGE(t). The color image frame which has been obtained from the
first still image frame IMAGE(t) or the second still image frame
IMAGE(t+.DELTA.t) outputs an identical result in the object
extracting method according to the present invention.
[0028] In a fifth step (S5) the color image frame and the
difference image frame are ANDed, thus generating a logic image
frame which has the color information and the motion change
information of the object and then a grid image frame is generated
by performing a grid process with respect to the logic image frame.
Here, the grid process for the logic image frame reduces operation
capacity and time compared with which the logic image frame is
processed by the pixel. Specifically, the grid process divides the
logic image frame into multiple grids, each has a fixed size, and
compares a predetermined value with a value representing pixels of
the grid, and expresses a value of the grid which is larger than
the predetermined value as a binary grid image, thereby reducing
the operation capacity and time to process the logic image
frame.
[0029] However, because each grid indicates a part of the object,
the grids which are gathered may have a shape similar to the
object. Thus, in a sixth step (S6) using direction connection
information it is determined whether the grids are connected with
other grids, and if connected, the grids are defined as a connected
component and thus the logic image frame includes a plurality of
connected components. In addition, in the sixth step (S6), minimum
rectangles each includes each of the connected components are
obtained. Here, the minimum rectangles are defined as a variable
RECT[i] wherein i is an integer number indicates a number of
minimum rectangles. The minimum rectangle represents a candidate of
a specific object to be extracted.
[0030] In a seventh step (S7), only each of the minimum rectangles,
the candidates of the specific object and minimum rectangles
satisfying conditions for the specific object. Here, the conditions
which define the object are as follows.
[0031] (1) A size of a variable RECT[i]>a threshold value of a
size of the minimum rectangle
[0032] (2) A minimum value of a row/column ratio of the variable
RECT[i]<the row/column ratio of the variable RECT[i]<a
maximum value of the row/column ratio of the variable RECT[i]
[0033] (3) A density of the variable RECT[i]<a threshold value
of a density.
[0034] Here, the threshold value of the size of the minimum
rectangle and the minimum and maximum values of the row/column
ratio of the variable RECT[i] are well known in the relevant field
of the present invention, and in the variable RECT[i] of a minimum
rectangle which includes connected components the density is a
value of which the number of grids in a row or a column of one of
the connected components is divided by a length of the row or the
column thereof. While, the density of the variable RECT[i] is a
value of which the density of the connected components is divided
by an area size of the variable RECT[i].
[0035] In an eighth step (S8), among the minimum rectangles the
minimum rectangle having the maximum size is selected. Finally, in
a ninth step (S9), the maximum sized rectangle is optimized to
correspond to the object. In addition, if an image considerably has
noise, the ninth step may be performed first before the eighth step
is carried out.
[0036] FIG. 2 is a diagram which illustrates an example of a
connected component which is generated in accordance with
8-direction connection information, which explains the sixth step
(S6) of FIG. 1.
[0037] When each of G1, G2, G3 and G4 represents a grid and D1, D2,
. . . ,D8 respectively indicate 8 directions, direction connection
information show a condition in which a grid is connected with
other grids in accordance with 4 directions or 8 directions
thereof. That is, a grid G3 is connected with a grid G2 in the
direction of D4 and connected with a grid G4 in the direction of
D8. Accordingly, the grids G3, G2, G4 constitute a connected
component. Similarly, when each connection condition is detected
with respect to the grids G1, G2, G4, the grids G1, G2, G3, G4
constitute another connected component.
[0038] Now, a face extracting method which applies the object
extracting method using the motion picture according to the present
invention will be explained.
[0039] FIGS. 3A and 3B illustrate a pair of still image frames
having a predetermined time interval, wherein FIG. 3A is a first
still image frame IMAGE(t) at a time (t) and FIG. 3B is a second
still image frame IMAGE(t+.DELTA.t) after a predetermined time
(.DELTA.t) elapses from the time (t).
[0040] First, in order to find the motion change of the face in
accordance with the time change, a difference value between the
first still image frame IMAGE(t) and the second still image frame
IMAGE(t+.DELTA.t) is obtained by each pixel and the resultant
values thereof are compared with the threshold value which has been
previously defined. Accordingly, when the resultant values are
greater than the threshold value, a difference image frame can be
obtained as shown in FIG. 4.
[0041] On the other hand, FIG. 5 illustrates a skin color image
frame obtained from the second still image frame IMAGE(t+.DELTA.t)
of FIG. 3B. As described above, the skin color image frame may be
obtained from the first still image frame, and although the skin
color image frame obtained from the first still image frame is
applied to the object extracting method according to the present
invention, the result is the same as a result of the skin color
image frame obtained from the second still image frame.
[0042] According to the object extracting method of the present
invention, a logic image frame can be obtained by ANDing the
difference image frame of FIG. 4 and the skin color image frame of
FIG. 5 and the logic image frame includes the motion change
information and the skin color information of the face. Here, since
the logic image frame is composed of pixel units, numerous
computation processes are required to extract a shape of the face.
Thus, the grid process is applied to obtain a grid image frame as
shown in FIG. 7. Here, when the 8-direction connection information
is applied to the grid image frame composed of grids which are
dispersed therein, a connected component corresponding to an area
of the face can be generated.
[0043] In FIG. 7, there are seven connected components, and FIG. 8
illustrates seven rectangles each defines a minimum area which
includes each of the connected components. Here, when the minimum
area is expressed as a variable RECT[i] wherein i indicates a
number of minimum areas, the minimum areas are RECT[1], RECT[2], .
. . , RECT[7]. Thus, the conditions of the specific object are
compared with each of the minimum areas and the minimum area which
satisfies the conditions thereof can be obtained.
[0044] For instance, the variable RECT[3] of the minimum area is
compared with the conditions of the object as follows.
[0045] (1) A size of the RECT[3]>a threshold value of a size of
the minimum area
[0046] (2) A minimum value of a row/column ratio of the
RECT[3]<the row/column ratio of the RECT[3]<a maximum value
of the row/column ratio of the RECT[3]
[0047] (3) A density of the RECT[3]<a threshold value of a
density.
[0048] Here, the threshold value of the size of the face, and the
minimum value and the maximum value of the row/column ratio of the
variable RECT[3] are the values which are respectively defined in
accordance with the object. While, the density of the variable
RECT[3] is a value of which the density of the connected components
is divided by the area of the variable RECT[3].
[0049] When the above-described method is applied to all of the
seven minimum areas, there are remained several minimum areas which
will be face area candidates (not shown).
[0050] The minimum area RECT[3] which has the largest size among
the remaining minimum areas is selected as the face area. In the
variable RECT[3] of a minimum rectangle which includes connected
components the density indicates a value of which the number of
grids in a row or a column of one of the connected components is
divided by a length of the row or the column thereof. Since, the
minimum area RECT[3] is a rectangle, each row or column of the
minimum area obtains its density and each density is compared with
a threshold value of the density and the row or column of which
density is smaller than the defined threshold value is deleted,
thereby optimizing the minimum area to become the shape of the
face.
[0051] As described above, the object extracting method using the
motion picture according to the present invention rapidly and
accurately extracts the particular object using the information of
the motion change of the object in the motion picture and the color
information which define the color of the object.
[0052] Although the human face is taken as the present invention,
the object extracting method of the present invention can be
applied to any object which has change of its motion and color
information thereof, if the color information thereof is
differently defined. Further, if there are a plurality of objects
to be extracted in the motion picture, an object which is nearest
to a camera, that is the object which has a largest size
thereamong, is determined as a specific object. However, when using
a templete or a neural network, the specific object can be
accurately and rapidly extracted even though there are a plurality
of objects in a motion picture.
[0053] It will be apparent to those skilled in the art that various
modifications and variations can be made in the object extracting
method using the motion picture of the present invention without
departing from the spirit or scope of the invention. Thus, it is
intended that the present invention cover the modifications and
variations of this invention provided they come within the scope of
the appended claims and their equivalents.
* * * * *