U.S. patent application number 11/797526 was filed with the patent office on 2008-07-10 for image processing apparatus using the difference among scaled images as a layered image and method thereof.
This patent application is currently assigned to InterVideo, Digital Technology Corporation. Invention is credited to Tsung-Wei Lin.
Application Number | 20080166069 11/797526 |
Document ID | / |
Family ID | 39594360 |
Filed Date | 2008-07-10 |
United States Patent
Application |
20080166069 |
Kind Code |
A1 |
Lin; Tsung-Wei |
July 10, 2008 |
Image processing apparatus using the difference among scaled images
as a layered image and method thereof
Abstract
The present invention is to provide an image processing
apparatus using the difference among scaled images as a layered
image and a method thereof, which utilize the Gaussian and
Laplacian pyramid theory to convert an original image into a
plurality of scaled images of different scales, and the difference
among scaled images of two adjacent different scales as a layered
image of the corresponding layer, so that the edge and line
characteristics of a scene of the original image for each layered
image can be displayed in different levels sequentially from a
clear level to a vague level, and provide a layered image display
interface and an image characteristic editing interface for users
to examine each layered image through the layered image display
interface and edit or perform special effect to each layered image,
so as to simulate different visual effects based on different
vision models.
Inventors: |
Lin; Tsung-Wei; (Taipei,
TW) |
Correspondence
Address: |
BACON & THOMAS, PLLC
625 SLATERS LANE, FOURTH FLOOR
ALEXANDRIA
VA
22314
US
|
Assignee: |
InterVideo, Digital Technology
Corporation
Taipei City
TW
|
Family ID: |
39594360 |
Appl. No.: |
11/797526 |
Filed: |
May 4, 2007 |
Current U.S.
Class: |
382/302 |
Current CPC
Class: |
G06T 3/40 20130101 |
Class at
Publication: |
382/302 |
International
Class: |
G06K 9/54 20060101
G06K009/54 |
Foreign Application Data
Date |
Code |
Application Number |
Jan 8, 2007 |
TW |
096100638 |
Claims
1. A method of processing an image by using the difference among
scaled images as a layered image, being applied in a computer, and
said method comprising the steps of: converting an original image
into a plurality of scaled images with different scales in a visual
model, and using the difference among said scaled images of two
adjacent different scales as a layered image of a corresponding
layer, such that said each layered image can be displayed in a
sequence of different levels from a clear level to a vague level
for showing the characteristics of an edge and a line of a scene in
said original image; providing a layered image display interface,
for displaying said each layered image in a visual model of a
different scale in said original image; and providing an image
characteristic editing interface, for editing or performing a
special effect for the characteristics of said each layered
image.
2. The method of claim 1, further comprising the step of providing
a layered image editing interface for displaying an editing
composed image comprised of said each layered image or said layered
images.
3. The method of claim 2, further comprising the steps of providing
an image output interface for integrating said edited layered
images into a new image, and converting said new image into an
image file having a specific format.
4. An image processing apparatus using the difference among scaled
images as a layered image, comprising: a scaled image conversion
interface, for reading an original image, and converting said
original image into a plurality of scaled images in a visual model
with a different scale, and using an image difference among said
scale images of two adjacent different scales for a computation to
find said image difference as a layered image of a corresponding
layer, such that said each layered image can be shown in a sequence
of levels from a clear level to a vague level to show the
characteristics of an edge and a line of said original image; a
layered image display interface, including a plurality of layer
control panels, each having a layer display window and an editing
start button, wherein said layer display window is provided for
displaying a corresponding layered image on said layer to examine a
layered image in said visual model with a different scale of said
original image through said layer display window; and said editing
start button is provided for starting an editing program for
editing a layered image displayed on said layer display window; and
an image characteristic editing interface, including a plurality of
image characteristic menus, for editing and adjusting a
characteristic value of said layered image of a started editing
program.
5. The apparatus of claim 4, further comprising a layered image
editing interface, and said layered image editing interface
comprising: a layered image editing window, for displaying said
each editing layered image or a composed image composed of said
layered images; at least one layered image characteristic adjusting
button, installed at the periphery of said layered image editing
window, for adjusting the contrast or Gaussian of said layered
image; at least one size adjusting button, for adjusting a process
of zooming in or zooming out of said layered image or said composed
image; and at least one image composition switch button, for
switching said layered image editing window to selectively show
said each layered image or composed image.
6. The apparatus of claim 5, further comprising an image output
interface for integrating said each edited layered image into a new
image, and converting said new image into an image file with a
specific format for saving, displaying or printing the output of
said image file.
7. The apparatus of claim 4, wherein said each layer control panel
further comprises a layer display button, for controlling whether
or not to display a corresponding layered image on said layered
image editing window.
8. The apparatus of claim 4, wherein said each layer control panel
further comprises a mask display window and a mask menu, and said
mask menu is provided for users to click and use said mask, and
display said mask on said mask display window, for displaying a
mask of said corresponding layer.
9. The apparatus of claim 8, wherein said each layer control panel
further comprises a mask display button, for controlling whether or
not to display said mask in said layer editing window.
10. The apparatus of claim 4, wherein said each layer control panel
further comprises a blurring or fine-tune switching button, for
adjusting a blurring or a fine-tune of pixels of said layered
image.
11. The apparatus of claim 4, wherein said each layer control panel
further comprises a noise reduction button, for effectively
preventing said layered image from being interfered by a noise.
12. The apparatus of claim 4, wherein said each image
characteristic editing interface further comprises a profile menu
used for determining whether or not to access said existing profile
of said layered image.
13. The apparatus of claim 4, wherein said each image
characteristic editing interface further comprises a LMS channel
menu, for selecting a LMS channel.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to an image processing
apparatus and method, more particularly to an image processing
apparatus and method by using the difference among scaled images as
a layered image for enabling a user to edit or perform special
effect to the edge and line characteristics of the layered images
of a scene of an original image, so as to simulate different visual
effects based on different vision models.
BACKGROUND OF THE INVENTION
[0002] In general, an image processing software usually comes with
a layer processing function that allows each image to be resided at
a layer all the time as shown in FIG. 1, and thus users usually
need to create a layer 111, 121 or add a layer before processing
(including editing or composing) an image 110, 120 in a file, and a
canvas 130 is situated at the lowest layer of all layers 111, 121
but the canvas 130 is not a layer by itself. Therefore, the layers
111, 121 are similar to a stack of transparent films, and the
layers 111, 121 can help a user to organize the images 110, 120 in
a file, so that the user can edit the image 110 on the layer 111
without affecting the image 120 on the other layer 121. If the
layer 111 contains no image 110, then the user will be able to see
the image 120 on the layer 121 through the layer 111. In addition,
a user can view the stack sequence of the layers and images on a
layer display panel provided by different image processing
software, and such stack sequence is also the sequence of images
appeared in the document. In general, various different image
processing software stack the layers according to such sequence,
and the lately produced layer is placed at the top of the stack,
and the stack sequence determines how to stack an image of a layer
on an image of another layer, such that users can rearrange the
sequence of layers and images through a control interface of the
image processing software to change the content of the images in a
file.
[0003] Referring to FIG. 2 for an image processing software called
"Photoshop" available in the market, a user can create two images
210, 220 on two separate layers from a layer display panel provided
by the image processing software, if the user wants to compose two
images 210, 220. A composed image 310 is created as shown in FIG.
3, after the size and position of each image 210, 220 are adjusted
according to the user's requirements. Referring to FIG. 4, a user
can click on an [Add Mask] button 450 to create a mask 440 for a
first image 410 on the first layer 411, if the user wants to edit
the first image 410 on the layer display panel 400. In FIG. 5, a
brush tool 471 is selected from a tool menu 470. In FIGS. 5 and 6,
the selected brush tool 471 is used to paint the mask 440, such
that the dark color position of a second image 420 at a second
layer 421 corresponding to the mask 440 is set to a black color. If
the user selects to apply a gentle pressure of the brush tool 471,
the user needs to adjust the transparency of the black color, such
that the image 410 (or foreground) at the first layer 411 is merged
with the second image 420 (or background) at the second layer 421
to form a composed image 430 with the best composition effect.
[0004] From the description above, the image processing software
available in the market can use the concept and technique of a
layer to provide a tool for editing the image on each layer, or
rearranging the image sequence of each layer, and can use the image
composition technology to simulate a digital dark room, for editing
and composing images, but these image processing software cannot
show the effect of different vision models for the image.
[0005] As LMS is a color space used for indicating the response of
three kinds of cones of human eyes, which refer to the sensitivity
of color lights with a long wavelength, a medium wavelength and a
short wavelength respectively, and the cross-section of a human
retinal-cortical system includes complicated neural links. Only the
LMS cone is known in general, but the profound structure of the
retinal-cortical system still includes tree-structured
constructions of cells strains, and the root of these constructions
link several cones together to form a so-called "ganglion cell",
for making several receptive fields. Although neurophysists already
have relatively high understanding on the visual imaging method of
a retinal-cortical system of human eyes, yet this understanding is
limited to an edge enhancement effect only, since the ganglion
cells of different sizes are distributed increasingly denser from a
fovea to the peripheral areas of a cornea, and thus the basic
visual imaging principle decreases the vision from the center of
the visual line to the peripheral areas. While human eyes are
viewing a scene, an image sensed by a fixation point or a
perceptual field of an eye cannot be the same as a camera or
camcorder as shown in FIG. 7 for treating each position on the
image 510 the same way. In fact, human eyes as shown in FIG. 8 only
perceive an area (such as the cross area) interested to the eyes
for a longer time, but the fixation point or perceptive field
varies and the level of fixation is different. As a result, an
image 530 with the effect of a different vision model can be
produced in an imaging area of a human brain as shown in FIG.
9.
[0006] In recent years, unsharp-masking (USM) has been used
extensively in different traditional image processing software and
provided for users to perform special effects for an image.
Regardless of the computation program used by these image
processing software, the core technology uses the Laplacian of
Gaussian edge enhancement technology and concept to simulate the
effects of a receptive field of a human vision, but the computation
carries out a single-level processing for the image only, and it
cannot show the effect of different vision models for the
image.
SUMMARY OF THE INVENTION
[0007] In view of the foregoing shortcomings of the prior art, the
inventor of the present invention based on years of experience in
the related industry to conduct extensive researches and
experiments, and finally developed an image processing apparatus
using the difference among scaled images as a layered image and its
method in accordance with the present invention.
[0008] It is a primary objective of the present invention to
provide an image processing apparatus using the difference among
scaled images as a layered image and its method to simulate an
image with an effect of different vision models of the same scene
in an imaging area of a human brain according to the fixation
point, perceptive field or level of fixation of a human eye that
views the scene, while a user is editing the image.
[0009] Another objective of the present invention is to use the
Gaussian and Laplacian pyramid theory to convert an original image
into a plurality of scaled images of different scales, and the
difference among scaled images of two adjacent different scales as
a layered image of the corresponding layer, so that the edge and
line characteristics of a scene of the original image for each
layered image can be displayed in different levels sequentially
from a clear level to a vague level, and provide a layered image
display interface and an image characteristic editing interface for
users to examine each layered image through the layered image
display interface. According to actual requirements, the image
characteristic editing interface is used for editing or performing
another special effect for each layered image, so as to simulate
different visual effects based on different vision models.
[0010] A further objective of the present invention is to provide a
layered image editing interface, and the layered image editing
interface includes a layered image editing window for displaying
each editing layered image or a composed image composed of the
layered images, and its periphery has at least one layered image
characteristic adjusting button for users to click to adjust the
contrast or Gaussian variance of the layered image; at least one
size adjusting button for users to click to zoom in or out the
layered image or composed image; and at least one image composition
switch button for users to click to switch the layered image
editing window and select each layered image or composed image, so
that a user can click the image composition switch button to browse
each editing layered image or composed image.
[0011] Another further objective of the present invention is to
provide an image output interface, for integrating the edited
layered image into a new image, and converting the new image into
an image file having a specific format for saving, displaying or
printing the image file.
[0012] To make it easier for our examiner to understand the
objective, technical characteristics and effects of the present
invention, preferred embodiment will be described with accompanying
drawings as follows:
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1 is a schematic view of images and layers in various
different image files of a prior art;
[0014] FIG. 2 is a schematic view of two images;
[0015] FIG. 3 is a schematic view of using Photoshop image
processing software to compose two images as depicted in FIG. 2
into one image;
[0016] FIG. 4 is a schematic view of using Photoshop image
processing software to add a mask operation to the composed image
as depicted in FIG. 3;
[0017] FIG. 5 is a schematic view of a tool menu of Photoshop image
processing software;
[0018] FIG. 6 is a schematic view of using Photoshop image
processing software to add a new mask to the image as depicted in
FIG. 3 for a composition;
[0019] FIG. 7 is a schematic view of a scene captured by a camera
or a video camera;
[0020] FIG. 8 is a schematic view of an area (such as a cross area)
of a certain scene attracted to human eyes;
[0021] FIG. 9 is a schematic view of simulating an image with an
area interested to and viewed by a human eye as shown in FIG. 8 for
a longer time;
[0022] FIG. 10 is a schematic view of simulating five visual model
spaces according to human vision sensing experience;
[0023] FIG. 11 is a schematic view of five levels of visual model
spaces as shown in FIG. 10;
[0024] FIG. 12 is a schematic view of converting an original image
into a plurality of scaled images of different scales by using the
Gaussian and Laplacian pyramid theory;
[0025] FIG. 13 is a schematic view of using an image difference
among scaled images of two adjacent different scales to compute a
layered image corresponding to a layer;
[0026] FIG. 14 is a schematic view of a structure of an image
processing apparatus in accordance with a preferred embodiment of
the present invention;
[0027] FIG. 15 is a schematic view of a layered image display
interface of an image processing apparatus as depicted in FIG.
14;
[0028] FIG. 16 is a schematic view of an image characteristic
editing interface of an image processing apparatus as depicted in
FIG. 14;
[0029] FIG. 17 is a schematic view of a layered image editing
interface of an image processing apparatus as depicted in FIG.
14;
[0030] FIG. 18 is a schematic view of a layered image display
interface in accordance with a preferred embodiment of the present
invention;
[0031] FIG. 19 is a schematic view of a layer control panel in
accordance with another preferred embodiment of the present
invention; and
[0032] FIG. 20 is a schematic view of a layer control panel in
accordance with a further preferred embodiment of the present
invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033] In general, a human visual system is used for identifying an
edge and a line of an image, which is an easy task for many
occasions except for a camera system. Even though many relatively
complicated theories and algorithms have been adopted, it is not
easy to simulate the recognition capability of a human visual
system due to the following factors:
[0034] (1) An error caused by the quantization of the original
image and a detected noise will cause an area of an edge or a line
of the image to disappear as the detected brightness varies;
[0035] (2) The precise positions of an edge and a line of an image
may be affected by quantized errors and noises to produce a
deviation; and
[0036] (3) An edge and a line of an object in an image should show
the change of a sharp brightness theoretically, due to the
high-frequency characteristic, and thus any smooth filtered wave
for reducing the noise of an image will cause a blurring to the
signals at the edge areas. Since a vast majority of edge detection
methods adopt the derivative processing, not only amplifying the
high-frequency signals, but also amplifying the noises as well,
therefore it is necessary to perform a smoothing filter to the
signals, and the smoothing filter effect depends on the size of the
filter and the scale of the filtered wave. The larger the scale of
filter, the broader is the range of brightness, but the precision
detected at the edge positions also becomes lower as well. The
smaller the scale of filter, the more precise is the edge position,
but it is easier to generate errors at the edge points.
Furthermore, the influence on different scaled images will be
different, and thus it is not easy to appropriately find the best
scale for all edges of an image with all edges, not even mentioning
about finding an appropriate scale of all images.
[0037] In the present image processing area, the generally
acknowledge best edge detection method is called a multi-scale edge
detection, and the main concept of the method is to applied a
smoothing filter (such as a Gaussian filter) with different scales
to perform a convolution with the original image to obtain a
filtered image with a different scale and then take the edge of the
filtered image for every scale, and finally stack all edges with
all scales to form an edge image.
[0038] As an object can be expressed in different scales and
displayed in an image. For instance, if a camera is shooting a
photograph of a person walking towards the camera and capturing the
images continuously, the scale of a human face falls within a range
of size from 3.times.3 pixels to 300.times.300 pixels or even a
larger range for a plurality of captured images. As the scale of
the human face varies, a change of characteristics of the human
face will show up in the captured images, which is equivalent to
the characteristics of the human face being changed continuously on
the photographer's retina, so that the photographer's visual cortex
at a high level perception area senses the quantum jumps to form
visual models with different scales of a different generation.
[0039] Referring to FIGS. 10 and 11, the complexity of a graph
structure will be increased if the size of an image 600 is
enlarged, so that the characteristics of the image 600 become more
significant. If a different generation model with a different scale
can be created, then a series of models can be used for defining
perceptual model spaces. From the experience of human perceptual
sensing, the perceptual model spaces can be defined into five major
regimes:
[0040] (1) A texture regime 610: If a viewer views a person at a
distance, the viewer cannot identify an image of the human face of
that person easily, and thus the color of skin is generally
identified to segment the human face in an image;
[0041] (2) A PCA regime 620: This regime was proven as a regime
capable of showing the characteristics of a scene of a mid-scale
image the best;
[0042] (3) A parts regime 630: This regime has a higher resolution
for clearly identify the image with five face features (including
eyes, nose and mouths), and thus it can identify the movements of
the five face features (including the closing or opening movement
of an eye or a nose);
[0043] (4) A sketch regime 640: This regime has a much higher
resolution, for displaying characteristics in more details to
identify the five human face features including eyelids, eyebrows,
and crow's feet; and
[0044] (5) A super-resolution regime 650: This regime has a much
higher resolution for displaying characteristics in more
details.
[0045] From the description above, any image can generate a
different visual model according to a different scale of its
perceptual space. The present invention adopts this concept
together with the Gaussian and Laplacian pyramid theory as
illustrated in FIG. 12 to convert an original image into a
plurality of scaled images 700, 710, 720, 730, 740, 750 in a visual
model with a different scale (as shown in FIG. 13,) and the image
difference among scaled images 700, 710, 720, 730 of two adjacent
different scales is used for the computation, and a computed image
difference is used as a layered image 701, 711, 721, 731 of the
corresponding layer, such that each layered image 701, 711, 721,
731 can be shown in a sequence of different levels from a clear
level to a vague level for displaying the characteristics of an
edge and a line of the original image. The present invention also
provides a layered image display interface and an image
characteristic editing interface, so that a user can examine each
layered image 701, 711, 721, 731 in a visual model of a different
scale of the original image through the layered image display
interface, and edit or perform a special effect for each layered
image 701, 711, 721, 731 through the image characteristic editing
interface, so as to simulate different visual effects based on
different vision models and the user's requirements.
[0046] It is noteworthy to point out that the mathematical
algorithm for computing the image difference among scaled images of
two adjacent different scales have been disclosed in many technical
literatures and journals. These algorithms vary according to actual
needs and objectives, but the basic algorithm generally adopts the
visual model of the Gaussian and Laplacian pyramid theory to obtain
the user's expected edge and line characteristics of an original
image in a visual model of a different scale. Since these
algorithms and mathematical models are not intended to be covered
in the patent claims of the present invention, therefore they will
not be described here.
[0047] In a preferred embodiment of the present invention as shown
in FIG. 14, the image processing apparatus includes the following
interfaces:
[0048] (1) A scaled image conversion interface 800: This interface
as shown in FIG. 12 is provided for reading an original image and
converting the original image into a plurality of scaled images
700, 710, 720, 730, 740, 750 in a visual model of a different
scale. In FIG. 13, an image difference among scaled images 700,
710, 720, 730 of two adjacent different scales is computed and used
as a layered image 701, 711, 721, 731 of a corresponding layer,
such that each layered image 701, 711, 721, 731 can be shown in a
sequence of different levels from a clear level to a vague level
for displaying the characteristics including an edge and a line of
the original image;
[0049] (2) A layered image display interface 810: The layered image
display interface 810 as shown in FIG. 15 comprises a plurality of
layer control panels 811, each having a layer display window 812
and an editing start button 813, wherein the layer display window
812 is provided for displaying a layered image of a corresponding
layer, such that a user can examine a layered image in a visual
model with a different scale in the original image through the
layer display window 812, and determining whether or not to edit or
perform a special effect for each layered image; and the editing
start button 813 is provided for starting an editing program that
allows users to edit a layered image by the editing program
displayed on the layer display window 812;
[0050] (3) An image characteristic editing interface 820: The image
characteristic editing interface 820 as shown in FIG. 16 comprises
a plurality of image characteristic menus 821 for users to click to
edit and adjust the characteristics including contrast, highlight,
midtone, shadow and white balance of the layered image of a started
editing program;
[0051] (4) A layered image editing interface 830: The layered image
editing interface 830 as shown in FIG. 17 comprises a layered image
editing window 831 for displaying each editing layered image or a
composed image composed of the layered images; at least one layered
image characteristic adjusting button 832 disposed at the periphery
of the layered image editing interface 830 for users to click to
adjust the contrast or Gaussian variance of the layered image; at
least one size adjusting button 833 for users to click to adjust
the process of zooming in or out the layered image or the composed
image; and at least one image composition switch button 834 for
users to click to switch the layered image editing window 831 to
select and show each layered image or composed image, such that a
user can click the image composition switch button 834 to browse
each editing layered image or composed image; and
[0052] (5) An image output interface 840: The image output
interface 840 is provided for integrating each edited layered image
into a new image as shown in FIG. 14 and converting the new image
into an image file having a specific format for saving, displaying
and printing the image file.
[0053] Referring to FIGS. 15 and 17 for another preferred
embodiment of the present invention, each layer control panel 811
further installs a layer display button 814 for users to click to
control whether or not to display a corresponding layered image in
the layered image editing window 831. Therefore, users can click a
layer display button 814 on the layer control panel 811 according
to actual requirements to select to open or close the corresponding
layered image in the layer control panel 811, and click an editing
start button 813 of the layer control panel 811 to determine
editing or performing a special effect for the layered images.
[0054] Referring to FIGS. 16 and 18 for another preferred
embodiment of the present invention, each layer control panel 861
further adds a mask display window 862 and a mask menu 863, wherein
the mask menu 863 is provided for users to click the desired mask
and display the mask on the mask display window 862, for displaying
a mask on a corresponding layer, such that users can click a
plurality of image characteristic menus 821 in the image
characteristic editing interface 820 to edit the mask.
[0055] Referring to FIGS. 17 and 19 for another preferred
embodiment of the present invention, each layer control panel 871
adds a mask display button 872 for users to click to control
whether or not to display the mask in the layered image editing
window 831.
[0056] Referring to FIG. 16 for another preferred embodiment of the
present invention, the image characteristic editing interface 820
further adds a profile menu 822, for users to click, to determine
whether or not to perform an access by the existing profile of the
layered image.
[0057] In another preferred embodiment of the present invention as
shown in FIG. 16, the image characteristic editing interface 820
adds a LMS channel menu 823, for users to click to select a LMS
channel.
[0058] In another preferred embodiment of the present invention as
shown in FIG. 20, the layer control panel 881 adds a blurring or
fine-tune switching button 882 for users to click to adjust the
blurring or fine-tune of the pixels of the layered image.
[0059] In another preferred embodiment of the present invention as
shown in FIG. 20, the layer control panel 881 adds a noise
reduction button 883, for users to click to effectively prevent the
interference of a noise to the layered image.
[0060] It is noteworthy to point out that the editing and
composition described in the preferred embodiment is used for the
illustration purposes only, and the persons skilled in the art
should be able to use the Gaussian and Laplacian pyramid theory and
the concept of the present invention to convert an original image
into a plurality of scaled images and display each layered image in
visual models of different scales in the original image through a
layered image display interface, so that users can edit or perform
a special effect to each layered image through an image
characteristic editing interface. Based on different vision models,
different visual effects can be simulated. All mathematical
conversion programs or editing and composition programs of this
sort are covered in the scope of the patent claims of the present
invention.
* * * * *