U.S. patent application number 14/463109 was filed with the patent office on 2014-12-04 for apparatus and method for configuring screen for video call using facial expression.
The applicant listed for this patent is Samsung Electronics Co., Ltd.. Invention is credited to Sung-Dae Cho, Dong-Hoon Jang, Jin-Ho Kim, Jong-Man Kim, Chul-Hwan Lee, Kee-Hyon Park, Ji-Young Yi.
Application Number | 20140359486 14/463109 |
Document ID | / |
Family ID | 46019250 |
Filed Date | 2014-12-04 |
United States Patent
Application |
20140359486 |
Kind Code |
A1 |
Yi; Ji-Young ; et
al. |
December 4, 2014 |
APPARATUS AND METHOD FOR CONFIGURING SCREEN FOR VIDEO CALL USING
FACIAL EXPRESSION
Abstract
Apparatus and method for configuring a screen for a video call
using a facial expression by recognizing a face from an image,
calculating facial expression information for an expression of the
recognized face, and determining whether there is a change in
expression of the recognized face by comparing the calculated
facial expression information with reference expression information
preset to determine a change in expression of the face. If there is
a change in expression of the recognized face, the apparatus and
method selects a video image corresponding to the changed
expression in the video call screen, and reconfigures the video
call screen using the selected video image, making it possible for
a user to conveniently select an image of the interested person
without taking extensive action, and preventing a wrong image from
being selected due to the unintended user facial movement.
Inventors: |
Yi; Ji-Young; (Seoul,
KR) ; Cho; Sung-Dae; (Gyeonggi-do, KR) ; Park;
Kee-Hyon; (Gyeonggi-do, KR) ; Kim; Jong-Man;
(Gyeonggi-do, KR) ; Kim; Jin-Ho; (Seoul, KR)
; Lee; Chul-Hwan; (Seoul, KR) ; Jang;
Dong-Hoon; (Gyeonggi-do, KR) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Samsung Electronics Co., Ltd. |
Gyeonggi-do |
|
KR |
|
|
Family ID: |
46019250 |
Appl. No.: |
14/463109 |
Filed: |
August 19, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13293720 |
Nov 10, 2011 |
8810624 |
|
|
14463109 |
|
|
|
|
Current U.S.
Class: |
715/753 |
Current CPC
Class: |
G06F 3/0481 20130101;
H04N 7/141 20130101; G06K 9/00302 20130101; H04N 7/14 20130101;
G06F 3/04842 20130101 |
Class at
Publication: |
715/753 |
International
Class: |
G06K 9/00 20060101
G06K009/00; G06F 3/0481 20060101 G06F003/0481; H04N 7/14 20060101
H04N007/14; G06F 3/0484 20060101 G06F003/0484 |
Foreign Application Data
Date |
Code |
Application Number |
Nov 10, 2010 |
KR |
10-2010-0111791 |
Claims
1. An apparatus for configuring a screen for a video call using a
facial expression, comprising: a facial expression information
calculator for recognizing a face from an image, and calculating
facial expression information for an expression of the recognized
face; a facial expression determiner for determining whether there
is a change in expression of the recognized face by comparing the
calculated facial expression information with reference expression
information preset to determine a change in expression of the face;
a screen configurer for configuring a video call screen including
multiple video images received for the video call; and an image
selector for selecting a video image corresponding to the changed
expression in the video call screen if there is a change in
expression, wherein the screen configurer reconfigures the video
call screen using the selected video image.
2. The apparatus of claim 1, wherein the video call screen includes
a screen of a main image received from a camera and a screen of at
least one sub image received outside of the video call.
3. The apparatus of claim 2, wherein the facial expression
information calculator recognizes a face in the main image,
extracts facial features of the recognized face, and calculates a
face angle of the recognized face based on the extracted facial
features.
4. The apparatus of claim 3, wherein the facial expression
information calculator calculates a face angle from an image
received before the video call, and sets the calculated face angle
as a reference face angle.
5. The apparatus of claim 4, wherein the facial expression
determiner determines whether a difference between the calculated
face angle and the reference face angle is greater than or equal to
a preset threshold by comparing the calculated face angle with the
reference face angle.
6. The apparatus of claim 5, wherein the image selector estimates a
face direction corresponding to the calculated face angle if the
difference between the calculated face angle and the reference face
angle is greater than or equal to the threshold and selects a sub
image corresponding to the estimated face direction in a face area
of the main image on the video call screen.
7. The apparatus of claim 6, wherein the screen configurer
reconfigures the video call screen by switching between the screen
of the main image and the screen of the selected sub image on the
video call screen.
8. A method for configuring a screen for a video call using a
facial expression, comprising: configuring a video call screen
including multiple video images received for the video call;
recognizing a face from an image, and calculating facial expression
information for an expression of the recognized face; determining
whether there is a change in expression of the recognized face by
comparing the calculated facial expression information with
reference expression information preset to determine a change in
expression of the face; selecting a video image corresponding to
the changed expression in the video call screen if there is a
change in expression; and reconfiguring the video call screen using
the selected video image.
9. The method of claim 8, wherein the video call screen includes a
screen of a main image received from a camera and a screen of at
least one sub image received outside of the video call.
10. The method of claim 9, wherein calculating facial expression
information further comprises: recognizing a face in the main
image; extracting facial features of the recognized face; and
calculating a face angle of the recognized face based on the
extracted facial features.
11. The method of claim 10, further comprising calculating a face
angle from an image received before the video call, and setting the
calculated face angle as a reference face angle.
12. The method of claim 11, wherein determining whether there is a
change in expression of the recognized face further comprises
determining whether a difference between the calculated face angle
and the reference face angle is greater than or equal to a preset
threshold by comparing the calculated face angle with the reference
face angle.
13. The method of claim 12, wherein selecting a video image
corresponding to the changed expression comprises: estimating a
face direction corresponding to the calculated face angle if the
difference between the calculated face angle and the reference face
angle is the threshold value or above; and selecting a sub image
corresponding to the estimated face direction in a face area of the
main image on the video call screen.
14. The method of claim 13, wherein reconfiguring the video call
screen comprises switching between the screen of the main image and
the screen of the selected sub image on the video call screen.
Description
PRIORITY
[0001] This application is a Continuation Application of U.S.
application Ser. No. 13/293,720, filed at the U.S. Patent and
Trademark Office on Nov. 10, 2011, now U.S. Pat. No. 8,810,624,
issued on Aug. 19, 2014, which claims priority under 35 10 U.S.C.
.sctn.119(a) to a Korean Patent Application filed in the Korean
Intellectual Property
[0002] Office on Nov. 10, 2010 and assigned Serial No.
10-2010-0111791, the entire disclosure of which is incorporated
herein by reference.
BACKGROUND OF THE INVENTION
[0003] 1. Field of the Invention
[0004] The present invention relates generally to a screen
configuring apparatus and method, and more particularly, to an
apparatus and method for configuring a screen for a video call by
selecting an image of an interested user among multiple users.
[0005] 2. Description of the Related Art
[0006] In a video call, a picture of a caller is taken using a
camera, displayed on a screen, and images of the persons with whom
the caller wants to have a telephone conversation are displayed in
a specific location of the screen, for a video call.
[0007] A multipoint video call (or video conference call)
technique, which allows a user to have a video call with multiple
persons on a mobile terminal, automatically identifies a speaking
party by lip movement recognition, and displays an image of the
speaker at the center of the screen, making it possible to talk
with multiple persons.
[0008] In a multipoint video call apparatus, a display includes a
main screen having the largest area on a video call screen, and at
least one sub screen.
[0009] However, the conventional multipoint video call technique
may malfunction when several users move their lips at the same
time.
[0010] Additionally, it is difficult for a user to select an image
of another user other than the speaker, when a user has an interest
in having a conversation with the other user.
SUMMARY OF THE INVENTION
[0011] Accordingly, the present invention has been made to solve
the above-mentioned problems occurring in the prior art, and the
present invention provides a video call apparatus and method for
estimating a facial expression during a video call with multiple
users, selecting an image of an interested person, and allowing a
user to have a video call with the selected interested person.
[0012] According to one aspect of the present invention, there is
provided an apparatus for configuring a screen for a video call
using a facial expression which includes a facial expression
information calculator for recognizing a face from an image, and
calculating facial expression information for an expression of the
recognized face; a facial expression determiner for determining
whether there is a change in expression of the recognized face by
comparing the calculated facial expression information with
reference expression information preset to determine a change in
expression of the face; a screen configurer for configuring a video
call screen including multiple video images received for the video
call; and an image selector for selecting a video image
corresponding to the changed expression in the video call screen if
there is a change in expression. The screen configurer may
reconfigure the video call screen using the selected video
image.
[0013] According to another aspect of the present invention, there
is provided a method for configuring a screen for a video call
using a facial expression by configuring a video call screen
including multiple video images received for the video call;
recognizing a face from an image, and calculating facial expression
information for an expression of the recognized face; determining
whether there is a change in expression of the recognized face by
comparing the calculated facial expression information with
reference expression information preset to determine a change in
expression of the face; selecting a video image corresponding to
the changed expression in the video call screen if there is a
change in expression; and reconfiguring the video call screen using
the selected video image.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The above and other aspects, features and advantages of
various embodiments of the present invention will be more apparent
from the following description taken in conjunction with the
accompanying drawings, in which:
[0015] FIG. 1 is a diagram illustrating a structure of a screen
configuring apparatus according to an embodiment of the present
invention;
[0016] FIG. 2 is a flowchart illustrating a process of extracting
reference expression information used to estimate changes in facial
expression in a screen configuring apparatus according to an
embodiment of the present invention;
[0017] FIG. 3 is a diagram illustrating images obtained in a
process of extracting reference expression information according to
an embodiment of the present invention;
[0018] FIG. 4 is a flowchart illustrating a process of
reconfiguring a video call screen corresponding to changes in
facial expression during a video call in a screen configuring
apparatus according to an embodiment of the present invention;
and
[0019] FIGS. 5 to 7 are diagrams illustrating images obtained in a
process of configuring a video call screen according to an
embodiment of the present invention.
[0020] Throughout the drawings, the same drawing reference numerals
will be used to refer to the same elements, features and
structures.
DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION
[0021] Various embodiments of the present invention will be
described in detail with reference to the accompanying drawings. In
the following description, specific details such as detailed
configuration and components are merely provided to assist the
overall understanding of various embodiments of the present
invention. Therefore, it will be apparent to a person having
ordinary skill in the art of the present invention that various
changes and modifications of the embodiments described herein can
be made without departing from the scope and spirit of the
invention. In addition, descriptions of well-known functions and
constructions are omitted for clarity and conciseness.
[0022] FIG. 1 is a diagram illustrating a structure of a screen
configuring apparatus according to an embodiment of the present
invention.
[0023] Referring to FIG. 1, the screen configuring apparatus
includes a facial expression information calculator 100, a facial
expression determiner 110, an image selector 120, and a screen
configurer 130.
[0024] The facial expression information calculator 100 calculates
facial expression information within a frame of an input image
received from a camera during a video call, or of input images
received outside the video call.
[0025] The facial expression information calculator 100 presets
reference expression information that is used to determine changes
in facial expression during a video call from an input image
received from the camera before the video call.
[0026] The facial expression information calculator 100 includes a
face recognizer 101, a facial feature extractor 102, and a face
angle calculator 103.
[0027] The face recognizer 101 uses a general face recognition
technique in recognizing a face area in an input image, for
example, recognizing an area corresponding to a preset facial skin
color in an input image, as a face area.
[0028] The facial feature extractor 102 extracts facial features in
the recognized face area. For the extraction of the facial
features, a general facial feature extraction technique is used.
The facial features as used herein may refer to facial feature
components such as eyes, nose, mouth and chin.
[0029] The face angle calculator 103 calculates a reference face
angle based on the extracted facial features. Specifically, the
face angle calculator 103 draws polygonal sides by connecting the
calculated facial features, and calculates an angle of the
recognized face based on the drawn polygonal sides. For the
calculation of the face angle, a general face angle calculation
technique is used.
[0030] When a video call begins, the screen configurer 130
configures a video call screen for the video call, using at least
one input image received during the video call and a user image
received from a camera. The at least one input image is defined as
at least one sub image, and the user image received from a camera
is defined as a main image.
[0031] That is, the face configurer 130 displays a main image in an
area with a preset size on the video call screen, and displays at
least one sub image in the remaining area except for the area where
the main image is displayed. The screen configurer 130 sets a size
of the area where the main image is displayed on the video call
screen, to be greater than a size of the area where the at least
one sub image is displayed.
[0032] The facial expression determiner 110 determines whether
there is a change in facial expression by comparing facial
expression information in the main image, calculated by the facial
expression information calculator 100 during a video call, with
preset reference expression information.
[0033] Specifically, the facial expression determiner 110
determines whether there is a change in face angle by comparing the
face angle in the main image calculated by the face angle
calculator 103 with a preset reference face angle.
[0034] If there is a change in facial expression information, the
image selector 120 selects a sub image corresponding to the changed
facial expression from among multiple sub images located on the
video call screen.
[0035] That is, if a difference between the calculated face angle
in the main image and the preset reference face angle is greater
than or equal to a preset value, the image selector 120 estimates a
face direction corresponding to the face angle in the main image,
and selects a sub image corresponding to the estimated face
direction in a face area of the main image on the video call
screen.
[0036] If a change in facial expression is continuously recognized
for a preset time, the screen configurer 130 reconfigures a video
call screen corresponding to the changed facial expression using
the sub image selected by the image selector 120, and displays the
reconfigured video call screen.
[0037] Specifically, if the face direction estimated by the image
selector 120 is continuously recognized for a preset time, the
screen configurer 130 switches between a screen of the main image
and a screen of the selected sub image on the video call
screen.
[0038] As such, the screen configuring apparatus estimates a facial
expression of a user and selects an image of an interested person
on the video call screen, making it possible for the user to
conveniently select an image of the interested person without
taking extensive action.
[0039] FIG. 2 is a diagram illustrating a process of setting
reference expression information in a screen configuring apparatus
according to an embodiment of the present invention.
[0040] Referring to FIG. 2, upon receiving an image from a camera
in step 200, the facial expression information calculator 100
recognizes a face in the received image in step 210. As described
above, for the recognition of a face in an image, a general face
recognition technique is used, and a technique of learning a skin
color and recognizing an area corresponding to the learned skin
color as a face area may also be used. For example, with reference
to FIG. 3, the facial expression information calculator 100
recognizes a face area 301 in an input image 300.
[0041] In step 220, the facial expression information calculator
100 extracts facial features in the recognized face. As represented
by reference numeral 310, the facial expression information
calculator 100 extracts facial features at the locations
corresponding to eyes, nose, mouth and chin in the face area.
[0042] In step 230, the facial expression information calculator
100 calculates a face angle of the recognized face based on the
extracted facial features. For example, the facial expression
information calculator 100 calculates a face angle 321 in an image
320 using an area of a polygon by connecting the facial features,
and then ends setting the reference expression information.
[0043] As such, the screen configuring apparatus may recognize
changes in facial expression in an image received during a video
call and reconfigure a video call screen corresponding to the
changed facial expression.
[0044] FIG. 4 is a diagram illustrating a process of reconfiguring
a video call screen corresponding to changes in facial expression
during a video call in a screen configuring apparatus according to
an embodiment of the present invention.
[0045] According to an embodiment of the present invention, a user
image received from a camera is defined as a main image, and at
least one input image received from outside is defined as at least
one sub image. An embodiment of the present invention will be
described with reference to FIGS. 5 to 7.
[0046] FIGS. 5 to 7 are diagrams illustrating images obtained in a
process of configuring a video call screen according to an
embodiment of the present invention.
[0047] Referring to FIG. 4, if a video call begins in step 400, the
screen con figurer 130 configures and displays a video call screen
including a screen of a main image and a screen of at least one sub
image in step 401.
[0048] The screen configurer 130 displays a main image in an area
with a preset size on the video call screen, and displays at least
one sub image in the remaining area except for the area where the
main image is displayed. The screen configurer 130 sets a size of
the area where the main image is displayed on the video call
screen, to be greater than a size of the area where the at least
one sub image is displayed.
[0049] The displayed video call screen may be as illustrated in
FIG. 5.
[0050] In step 402, the facial expression information calculator
100 recognizes a face in the main image, calculates facial features
of the recognized face, and calculates a face angle based on the
calculated facial features.
[0051] Specifically, the face recognizer 101 recognizes a face area
in the main image using the general face recognition technique, for
example, by recognizing an area corresponding to a preset facial
skin color in an input image, as a face area.
[0052] Thereafter, the facial feature extractor 102 extracts facial
features in the recognized face area, and the face angle calculator
103 calculates a reference face angle based on the extracted facial
features.
[0053] In step 403, the facial expression determiner 110 compares
the face angle in the main image calculated by the facial
expression information calculator 100, with a preset reference face
angle.
[0054] In step 404, the facial expression determiner 110 determines
whether there is a change in face angle. If there is a change in
face angle, the image selector 120 proceeds to step 405. Otherwise,
the screen configurer 130 continuously displays the video call
screen in step 401.
[0055] In step 405, the image selector 120 selects a sub image that
is located on the video call screen to correspond to the face angle
of the main image.
[0056] That is, if a difference between the calculated face angle
in the main image and the preset reference face angle is greater
than or equal to a preset value, the image selector 120 estimates a
face direction corresponding to the face angle in the main image,
and selects a sub image corresponding to the estimated face
direction in a face area of the main image on the video call
screen.
[0057] Referring to FIGS. 5 and 6, the image selector 120 estimates
a face direction 502 corresponding to a face angle of a main image
500, and selects a sub image 501 corresponding to the estimated
face direction 502 in the face area.
[0058] As illustrated in FIG. 5, the screen configurer 130 may
further display, on the video call screen, face direction arrow
icons for allowing the user to recognize face directions
corresponding to face angles. These face direction arrow icons may
be displayed to overlap the screen of the main image.
[0059] To emphasize that the selected sub image is a selected
image, the screen configurer 130 may display the edge of the
selected sub image to be bold, or may display the selected sub
image to be greater in size than other sub images.
[0060] In step 406, the screen configurer 130 determines whether a
change in face angle is continuously recognized for a preset time.
If the change in face angle is continuously recognized, the screen
configurer 130 proceeds to step 407. Otherwise, the screen
configurer 130 continuously displays the video call screen in step
401.
[0061] The reason why the screen configurer 130 determines whether
a change in face angle continues for a preset time is to prevent a
wrong sub image from being selected due to the unintended user
facial movement.
[0062] In step 407, the screen configurer 130 reconfigures a video
call screen corresponding to the changed facial expression using
the sub image selected by the image selector 120, and displays the
reconfigured video call screen. For example, as illustrated in FIG.
6, the screen configurer 130 reconfigures the video call screen
such that the screen of the main image and the screen of the
selected sub image are switched in terms of the location, thereby
displaying the selected sub image in the area of the main
image.
[0063] As illustrated in FIG. 7, the screen configurer 130 may to
place the screen of the sub image in the area of the main image by
switching between a screen 700 of the main image and a screen 701
of the sub image on the video call screen.
[0064] In step 408, the screen configurer 130 determines if the
video call has been completed. If the video call has been
completed, the screen con figurer 130 ends the video call
operation. Otherwise, the screen configurer 130 returns to step 401
and repeats its succeeding steps.
[0065] According to embodiments of the present invention, the
user's facial expression is estimated and an image of the user's
interested person on the video call screen is selected, thereby
allowing the user to conveniently select an image of the interested
person without taking extensive action. The embodiments of the
present invention may set a time margin for selecting an image of
an interested person, to prevent a wrong image from being selected
due to unintended user facial movement.
[0066] In addition, the embodiments of the present invention
provide accurate, intuitive and convenient functions according to
the display screen of the video call apparatus, thereby increasing
user convenience.
[0067] While the invention has been shown and described with
reference to various embodiments thereof, it will be understood by
those skilled in the art that various changes in form and detail
may be made therein without departing from the spirit and scope of
the invention as defined by the appended claims.
* * * * *